A book on pragmatic software architecture, patterns and frameworks?

One of the major items on my wish list – that is on the professional half of it – is to write a book that displays my ideas on software architecture, patterns and frameworks. Yes I know, there are many books on software architecture, and there are many books that explain patterns, and yes there also are a lot of books that show you how to work with framework A, B or C.

So you would say, why on earth would you want to write a book on these subjects? Well there’s three major reasons. First, there’s not a lot of books out there that combine all these subjects into one single coherent story that will actually help you to write your applications better. Second, maybe I’m a bit arrogant on this, but I feel like I have something to all the stuff that’s out there. And last but not least, I finally need to get this stuff out of my head! Maybe ambitious, but let´s see!

Order from the menu

To be honest, you might like or not like all of the ideas in this book, or even may totally disagree with some of them. Actually, you probably will. But on the whole, I hope this book will be able to inspire you and illuminate you in how to build software. It´s often a matter of taste. So consider this book as a menu. Just order from it what you like. You don´t need to copy the C# and .NET code examples from the book, but rather I hope that use can use them to shape your own ideas on software architecture, frameworks and patterns. Take from it what you like, and forget about the few things you don’t like. 

A first step

What’s the first step towards writing this book? I’ve been teaching about software architecture, patterns and frameworks for .NET for quite a number of years, and the material from these courses will be a good start, similar to the way I shaped my book on UML – from iterating over the material I had for my UML courses.

PIC091019003 

Along with the course material comes the work I’ve done in projects, and as a result the work we (the core team) do on our ever-evolving and improving ADF.NET framework.

Architecture and frameworks

So here’s the overview of the material in the software architecture and patterns course:

  • Introduction. All the usual bla bla and thanks you stuff.
  • How to set up your software architecture? A mild and brief introduction. Thinking about layers, elements in layers,  collaborations (yes CRC). And of cause you can not start without summing up some of the good practices, such as
  • (Software architecture != List<Framework>). How people mix up architecture and frameworks.
  • How frameworks can kill your project. It’s all about dependencies guys. This chapter will be a write out of my talk with the same name.
  • Some patterns to help avoid getting killed. Explaining the different variations of layer super types, my fine descriptor pattern, and the use of dependency injection and the manager-provider pattern.
  • Glue 1.0. My first framework. Short introduction into setting up a minimal framework, using your architecture and the above patterns, minimizing dependencies between your application code and the frameworks you use. It should provide you with the glue to keep your “regular” developers in the loop.
  • Extension methods and fluent interfaces. Have got a bit (not all by far!) to say on how to use them to enable your application developers to code easier and more natural using your first framework.
  • Lasagna and ingredients. What layers would you need in your application? Suggesting presentation, process, domain and data/services. And suggesting a lot of elements in the different layers.

Presentation

In the next chapters I will slowly go through the individual layers. First, let’s investigate the presentation layer:

  • Panels and lists. The basic patterns of presentation layer elements. And a straightforward implementation of a user interface independent panel. Just for the fun of it.
  • The master of detail. Introducing some typical user interface patterns, such as manage, select, search, master-detail, list-detail and the effect of those patterns on your architectural and Glue 1.0 framework requirements.
  • MVWTF. Looking at the different variations in model-view patterns, and discussing the relevance. But briefly, there’s enough said on this stuff.
  • Binding and persisting. Coupling and de-coupling your models, view-models, domain objects, whatever to and from the user interface.
  • The view-model. Sense and non-sense of view-models.
  • Validations. Things to validate and things not to validate in the user interface.

Process

Next the process layer gets a rub:

  • Why do I need a process layer? Explaining the least well-understood layer in application architectures.
  • An brief introduction to smart use cases. Still an excellent unit of work, implementation and testing for projects. This chapter will follow the lines of my presentation on this particular topic.
  • Implementing smart use cases. Introducing the task pattern, and re-iterating the manager-provider pattern. Great stuff I think! Also re-visiting the MVWTF patterns again. 
  • Unit and regression testing smart use cases. How you can write your own extensions to regular unit test framework to allow regression tests over your smart use cases.

Domain

The layer it all revolves about:

  • About being domain driven. Design, code and test, do it domain driven. Now I’m not going to repeat Eric Evans’ book here, but I do repeat (briefly) the basic concepts that derive from his work. Think of entities, repositories, factories, value objects and services.
  • Augmenting your domain. However, we are using some additional patterns in our projects that are useful patterns. Here I’ll discuss the smart reference pattern, revisit value objects, and talk about cached domain objects. I might even introduce you to search objects.
  • Business rules and validations. How do you write and where do you place your business rules, and what validations do you execute in the domain layer?
  • Independent state. Domain objects, or entities, have identity and state. This state can come from a lot of place – not all being SQL Server. So the big question here is: can I keep my domain objects independent of how we get and set its state. Well yes you can, hence the internal state patterns jumps in.

Data, services and the outside world

All the way down to the bottom, in the cellar of your application, lies the data / services layer:

  • Does the data layer really exist? Hey weren’t we using NHibernate or Entity Framework? So why do we need a data layer, and if so, what is in it?
  • Map and wrap. When querying data sources it is very likely that you will need to provide some sort of mapping from your world to the outside world. I’ll show you some simple patterns of to achieve this mapping, allowing you to wrap your domain objects around the outside world’s objects.
  • Creating queries. Again, do you really need to implement a query patterns. We do have LINQ do we? Well yes, but LINQ does not cover all your possible sources of data – a lot, but not all. So a brief introduction of the query pattern – and some implementation aspects might not be a bad idea.
  • On the way out. Somehow you will need to reach out to the outside world of your application. Virtually the outside world can be just about anything. A (set of) databases of course, but what about OData, JSON, plain or not so plain XML, restful or restless web services, Twitter, Entity Framework, NHibernate, Facebook, you name it. The gateway pattern should help you here.

Concluding

    Mildly put, this little blog post could well be the introduction for this new book – except for the thank you notes. Do you miss anything, or already agree or disagree? Want to help annotating or reviewing the manuscript in progress? Don´t hesitate to become a part of the writing process. Please comment here!

November 25 & 26, 2010 – IT Works. Pragmatic modeling using UML

[Two day hands-on workshop at IT Works, Hotel Crowne Plaza, Antwerp. ]

On November 25 and 26 I will present the 32th edition of an intense two-day workshop on the pragmatic use of UML modeling techniques (and beyond) with lots of hands-on exercises.

IMAGE_073
Participants modeling activity diagrams with smart use cases

During this workshops we will go through the following subjects in depth:

  • A brief introduction of UML and its modeling techniques
  • A pragmatic YAGNI approach to modeling business processes and requirements.
  • Modeling hierarchical and chronological business processes.
  • Modeling traditional and smart use cases.
  • Modeling activity diagrams (for use cases).
  • Identifying test scenario’s for (smart) use cases using activity diagrams.
  • Modeling the user interface.
  • Modeling class diagrams.
  • Domain driven design.
  • Interaction modeling. Bringing together an applications structure and behavior, based on a pragmatic use case driven software architecture.
  • Modeling service oriented projects using smart use cases.
  • Modeling in agile projects.
  • A brief introduction to model driven software development.

Being Smart in enterprise agile

As agile is becoming more and more mainstream, organization are starting to do enterprise software development project using well-known but fairly basic lightweight agile processes.

IMAGE_089 

In many projects this has lead to surprisingly bad result, baffling the agile Certified Pokémon Trainers who are coaching these projects. The presentation below shows a number of accelerators or technique that projects can apply to tackle the challenges of enterprise agile projects.

Would be happy to hear your comments!

Spring 2010 speaking engagements

Again doing a lot of talks this spring on a wide range of subjects, from new technology, via enterprise agile to model driven development, but also about smart use cases, domain driven design, UML, and software architectures, design patterns, frameworks and .NET.

This season’s highlights? Not a difficult choice: doing talks both at Microsoft DevDays and TechEd North America is great. But revisiting SET in Zurich is not bad either. And what about the Dutch Testing Conference?

 DSC_0058

Check out my spring talks below (with links and logo’s):

  • Around the world in 80 slides.
    February 9. Keynote at Capgemini TechEd, Utrecht.
  • Pragmatic modeling using UML and beyond
    March 3-4. Two-day workshop at IT Works, Antwerp, Belgium.
    www.itworks.be
  • We’re not in Kansas anymore. New trends in Microsoft technology
    March 8. Capgemini Micrsoft Practice Meeting. Utrecht.
  • Pragmatic modeling using UML and beyond
    March 10-11. Two-day workshop at Array Seminars, Hilversum.
    www.arrayseminars.nl
  • CIO challenges. From enterprise agile to model driven development.
    March 16. Talk Capgemini customer event, Utrecht.
  • How smart use cases will change your life
    March 23. Guest lecture at Hogeschool Utrecht, Utrecht. .
  • Agile development in everyday practice
    March 24. Seminar at Array Seminars,  Nieuwegein.
    www.arrayseminars.nl
  • Do’s and don’ts in implementing extension methods
    March 30-31. Talk at Microsoft DevDays 2010. Den Haag
    www.devdays.nl 
    .
    image
  • Beyond agile testing
    April 7. Talk at Capgemini Testing Community Event, Utrecht.
  • Beyond agile testing
    Arpil 21. Talk at Dutch Testing Conference 2010, Nieuwegein.
    www.dutchtestingconference.nl
    .
    image
  • Mission impossible? How to survice agile SAP SOA projects
    May 18-20. Talk at SET 2010. Zurich, Switzerland.
    http://www.sigs-datacom.de/set/set-2010
    .
    image
  • Estimating with smart use cases
    May 26. Seminar at Array Seminars, Nieuwegein.
    www.arrayseminars.nl
  • Software architectures and patterns in .NET.
    June 2. Seminar at Array Seminars, Hilversum.
    www.arrayseminars.nl
  • How frameworks can kill your project and how to avoid getting killed
    June 7-11. Talk at TechEd US 2010, New Orleans, USA.
    northamerica.msteched.com
    .
    image
  • Pragmatic modeling using UML and beyond
    June 23-24. Two-day workshop at IT Works, Antwerp, Belgium
    www.itworks.be

Avatar, reuse and model driven software development

Last week my daughter Sam (13), her friend Joey and I went to see the movie Avatar in 3D at the IMAX theatre in Rotterdam. Following up on the hype and reading reviews I just got curious. Especially after reading an interview with director James Cameron, who previously directed Aliens and Titanic.

Being more than interested in model driven software development, this interview cracked me up. James Cameron states that there’s a model for every rock, every straw, every tree on the planet Pandora, and for every piece of skin and body of the inhabitants. That’s something for you.  On the one hand you might say: hey, this Avatar project took over 7 years to build, and it costs around 400 million dollar. No way should we do software development projects like that.

image
James Cameron: “Use the models for sequels”

But it gets more interesting. In the interview James Cameron also says that the argument that sold Avatar to studio 20 Century Fox is that these expensive and extensive models could be used for any number of sequels. To sum up why the studio put up with the length and price of the project in one simple word: it’s reuse.

Seeing is believing

I have to admit, seeing is believeing. This movie has the most incredible visual appearance I ever witnessed. And it’s all generated from the model. And that’s where I got jealous, or even a bit depressed.

There we are: enthusiastic software engineers building business software from our BPMN or UML models or our textual or visual DSL’s. Whatever abbreviation you follow, be it MDA, MDD, FMDD, or MDSD, we think we’re doing great jobs raising the level of abstraction just a little bit from hand-written code. But even this very little step  forward seems to be highly complex. The technology is hard, the meta-level thinking is harder, and convincing management and customers (the studios) that this is the way to go with software development is even more challenging.

Kindergarten stuff

But in comparison to the complexity and size of producing Avatar our whole MDx venue is kindergarten stuff. We’re not creating a highly detailed blasting 3D movie experience. We are only implementing a simple business process to change your address at your local bank. Pfff. Is that all?

image
Jake Sully: “Ha ha, changing your address at your local bank is difficult?“

Don’t get me wrong. We will simply have to move from hand-writing applications to modeling and generating applications. Raising the level of abstraction is the only way to deal with the rapidly increasing level of complexity of technology vendors present us with, and the ever increasing demand for new software, presented on new devices. There’s just no way we can keep with this demand just by hand-writing all the code, not with all the developers in the world, not with the best-of-class frameworks at hand, not with the best agile processes and techniques in place (let alone waterfall).

Even though model driven techniques are no silver bullet either it will play an increasingly important role in software development . Our work will be around simplifying customer demand, standardizing techniques and raising our levels of abstraction, writing less code and generating more code. And for those of you not yet convinced: go see Avatar or preferably one of its likely many sequels.

October 19. Talk. “Silverlight, .NET RIA Services and code generation”

SDC Conference, Papendal, Arnhem (www.sdc.nl)

Next week the annual SDC conference will take place at Papendal, Arnhem. As usual the organising SDN community has put together a long list of international appraised speakers, and challenging subjects on the matter of software development and software architecture.

SONY DSC 

A bit to my surprise, this year I was invited to do 3 talks, with very different subjects: Silverlight and code generation, smart use cases in service oriented projects, and .NET extension methods.

Putting model driven development to the test

The SDC website refers to my Silverlight talk as follows:

To be quite honest, this talk is putting model driven development (MDD) to the test. Model driven development – generating code from a model – is a concept that has long promised high productivity and quality to projects. There are many different approaches to MDD, some of them highly theoretical, some of them painfully oversimplified. Think of generating code from UML models, or using Oslo and M. And whatever happened to (graphical and textual) DSL’s?

As always Sander and his team follow a very pragmatic, straightforward approach to the subject, generating code from domain models and smart use cases. One of the benefits of true model driven development is that it allows to generate code from the same model to different target environments. Having done projects where code was generated in Windows applications, ASP.NET applications, and Java web applications, Sander will now use this approach to Silverlight line-of-business applications. In this talk Sander elaborates on how code generation works, how templates can be defined for Tobago MDA, a freely available MDD code generators, and will (try to) build up a Silverlight applications on stage.

Introducing .NET RIA Services

Unfortunately, my team was minimized due to illness and projects, and I had to figure out most of it myself – a though late night job. Especially since I want to pay attention to .NET RIA Services, a brand new way of buidling client/server applications in a distributed environment, where a lot of code generation takes place under the covers.

Let’s just I I can do some demo’s on the 19th…

Read more: Sander at SDC 2009.

October 8-9, 2009. Workshop. “Pragmatic modeling using UML and beyond”

IT Works, Crowne Plaza Hotel, Antwerp, Belgium

I will present an intense two day workshop in the Crowne Plaza in Antwerp. During this workshop I follow the pragmatic modeling approach presented in my book with the same title. During this highly interactive workshop participants will learn how the various modeling techniques from UML and beyond connect.

image

Learn everything about (smart) use cases and start working directly with UML in many exercises a pragmatic case – Dare2Date, an online dating site.

Topics include: UML overview, an agile approach to requirements, modeling smart use cases, testing use cases (with activity diagrams), user interface modeling, domain modeling (using class diagrams), modeling services, interaction modeling (with sequence diagrams), modeling in agile software development projects, and model driven development using UML.

Participants will receive a copy of the book Pragmatic Modelling with UML 2.0. This workshop always gets rave reviews. Note that this is the 29th edition of my UML workshops at IT Works!

See www.itworks.be

image

October 5, 2009. Talk. “Pragmatic model driven development”

Capgemini BAS, Faustlaan, Apeldoorn.

With Capgemini BAS joining Capgemini in 2009, some new initiatiaves in model driven development opened up, more particularly, hoping I’ve interpreted it correctly, based on naterual language. I discussed this initiative in June with Freek Bosch, manager of the unit where this natueral langauge based approach originated.

SONY DSC

During the same inspriing conversation I illustrated the pragmaitc model driven development approach in our Accelerated Delivery Platform (ADP). Immediately Freek invited me to present our approach during one of his unit’s meetings. Unfortunately my agenda left little room until now. So I will drive to Apeldoorn in the morning, and discuss our approach during a one-and-a-half hour lunch meeting.

Think I will talk about the following subjects tomorrow:

  • Brief introduction of the Accelerated Delivery Platform.
  • Our agile requirements approach.
  • Modeling smart use cases.
  • Domain driven design – including value objects, enumerations, smart references, repositories and services.
  • Our appoahc to pragmatic model driven development, as used by a growing number of organizations.
  • How our code generator Tobgao MDA works – including templates and Tobago’s DSL.
  • And of course, last but not least – a demo. Buliding a fully working ASP.NET multi-layered web application.

Writing better software faster

Published in my Interesting Things column in SDN Magazine, November 2009.

Looking back on twenty years of software development, I must have spent most of that time trying to improve the quality and productivity of software development.

Ever since I started to write small applications in Turbo Pascal in 1988 I got infected with the writing-better-software-faster virus. Right after I finished the second of fourteen of these applications, I wrote my first framework. It took some effort, but writing the following twelve applications took me less time than building the first two.

Darkening perspectives

There are good reasons for increasing productivity and quality. First of all, we have to increase productivity by a hundred-fold to meet up with current demands, says Gartner. Furthermore, technology is becoming increasingly complex. Ten years ago I was working in both Microsoft and Java projects. Anno 2009, I’ve let go of the idea that I even have an overview of either one of them. It is hard enough working with a small subset of frameworks. A Silverlight designer is not likely knowledgeable in NHibernate. Being a Spring guru does not guarantee foxy JavaFX user interfaces.

Despite this darkening perspective, there’s techniques and technologies that might keep you from drowning, by simply raising the level of abstraction. Good frameworks help. Smart use cases certainly do. And even UML assists. And then there’s model driven development a.k.a. model driven architecture a.k.a. model driven engineering a.k.a. domain specific languages. Different names for generating code and other deliverables from a model. What model? To be honest, any model will do the trick. Business process models, use case diagrams, user interface models, domain models, and last but not least data models.

Empirical start

Eight years ago I was helping a customer set up the domain model for a help desk application. After that I started to build the software in ASP.NET. Having modeled all domain objects, their properties and their services, I was typing them out in C#. So, as any decent developer would do I wrote a code generator. That helped.

I suspect most code generation efforts have a similar empirical start. That’s probably why there are so many approaches to generating code. Some really high-brow, squeezing everything-and-the-kitchen-sink into a model, and then trying to generate the whole application. From front  to back. Or, at the very low and of the spectrum, inspecting the database, generating create-read-update-delete screens in a flat two-tier architecture.

Where’s the value?

Recently I was invited in a panel on model driven development, together with some academic and commercial experts. For starters, each of the panel members introduced their position briefly to the audience. This noteworthy event learned me two things. One: never allow four enthusiastic innovators talk freely on their innovations. The word brief isn’t in their vocabulary. Two: coming from a business background, I didn’t have a clue to what model driven development does in the academic world. A whole new spectrum of approaches met my eye.

clip_image001

So where’s the value in this? Following the good old Scrum adagio I’d say: it depends. Model driven development can be highly rewarding for projects. But I’ve also witnessed projects that made a terrible mess of it.

In my opinion, being successful at model driven development requires some preconditions to be met:

  • Embed approach. Successful model driven development projects share a single principle: it’s all in a day’s work. Model driven development is never a goal. It’s just a means to an end. Making better software faster. No guru’s, no heroes.

  • Allow changes. Projects go through changes. That’s a fact of life. Model driven development should never be blocking changes, but rather should stimulate creativity, by allowing projects to show quickly what effect new or changing requirements have.

  • Stabilize architecture. Knowing how to model, and where to generate what code is key. Set up a stable (layered) software architecture, pick frameworks that match it, and only then elaborate on code generation.

  • Be pragmatic. My general advice in life also holds for model driven development. Be pragmatic. Never overdo it. Don’t try to squeeze the whole world in your model. It’s not necessarily a bad thing to still have to write some code manually.

Skepticism?

Having said all that, and having seen projects that very successfully apply model driven development, why is there still so much skepticism? After all, the paradigm really helps to deliver high quality at high productivity.

After witnessing both highly successful and extremely failing model driven development projects recently, I have given this some consideration. The vast majority of projects are in the middle of the wide spectrum of approaches. They run smoothly, being fairly practical. But it’s the failing projects at either end of the spectrum that draw the attention. Highly complex, slow moving, rigor projects on the one end, and oversimplified, bug-absorbing, underestimated projects on the other end. It’s no fun at either end.

No Jedi knight

My team and I have always adopted a highly pragmatic approach to model driven development, modeling smart use cases, applying domain driven design, run the model through our code generators (Tobago MDA) and we’re set to go. We do make better software faster.

Can life be this simple? It certainly can. We have applied this approach to all kinds of projects, and generated code in a variety of languages and frameworks, from Visual Basic to C#, from PHP and Java to Cobol. At the upcoming SDC, I’m going the check out a combination of Silverlight, .NET RIA Services, ADF and probably Fluent NHibernate. Should be fun!

To leave you with some words of wisdom (ahum). If there’s one thing I’ve learned from being successful at model driven development, it’s this: modeling and generating code should be everyday practice in your project. There’s no such thing as a “model driven development project”. There’s only software development projects. You don’t have to be a Jedi knight to steer a code generator. It’s all in a day’s work.

Pragmatic model driven development. Part III. Creating the domain model

Note. This series of posts is also published as a Capgemini group white paper and published in Software Release Magazine (in Dutch, in two parts).

The next step towards generated code, and other deliverables, is to create the domain model for the project. Next to the smart use cases, that capture the desired behavior, the domain model provides a structural view of the system.

Roughly said, the domain model captures the key concepts, types, rules, validations, and services of the business domain, expressed in customer terminology. Furthermore,  the model identifies the relationships between all major types.

In this pragmatic approach we follow and extend on the principles stated by Eric Evans in his domain driven design paradigm. As your might expect, the domain model is expressed in UML class diagrams. Primarily, the domain model models domain objects (or entities) such as Customer, Order, Subscription or Course and their relationships, expressed as associations. Next, the properties and their types are modeled.

image17

We consider five different categories of property types, that are all applicable in different situations:

  • Basic types. In general, people tend to model property types in basic types, such as string, integer, boolean or date. However, we have more knowledge on most property types than that it’s just a string or integer. In this case it’s better to use value objects.
  • Value objects. A value object holds no identity, but reflects a value. In short, value objects are good for modeling property types that can be validated, such as Bsn, Isbn or Email.
  • Enumerations. We model property types as an enumeration, if the property can only have a limited, fixed number of different values. Think of ContractType, Gender or Level.
  • Smart references. Another pattern we apply to modeling property types, is the smart reference. We have agreed to model a property as smart reference if the property can only have a limited, but possibly changing number of different values. Here you could consider properties such as Department, Prefix or Country (if no other information on countries is required).
  • Associations. If a property has a type that itself is a domain object, we model an association. Note that in UML an association is actually defined as properties on both ends of the association.

Each of these categories of property types can be used nicely in code generation. For instance, enumerations and smart references can be used to fill drop down lists, and associations of course can be used to generate master-detail behavior or object-relationship mapping configuration files. Moreover, the multiplicity, role names and composite parts of associations can all be used in code generation.