Increasing the productivity of software development

Earlier this week I presented a talk at the inspiring Software Developers Conference 2008 in Noordwijkerhout, the Netherlands. My talk had an inspiring, or at least intriguing title: How to keep our jobs. Why? Well, let me share some thoughts on the productivity of software development projects.

A 100-fold

Recent research by Gartner shows that the productivity of software development will need to increase by a 100-fold. According to Gartner, there are 2 very simple and straightforward reasons for this:

  • Not enough newbies. The number of university graduates, and other people starting new with software development is too limited.
  • Increasing demand. The demand for new software is ever increasing, for instance becomes of emerging technologies and platforms (think of mobile and web 2.0).

And from my personal experience I would add:

  • Poor quality software. I perform a lot of code audits on applications, often quite large – say of over 1.000.000 lines of code. Even with such large applications, built by respectable companies, the quality is often brain damaging poor, frequently due to failing or total lack of software architecture. Such software is therefore very hard to maintain or extend with new features.
  • Ongoing business. There is a vast amount of software present in the marketplace that will need to be maintained for a very long time. Large transactional systems with banks (nationalized or not), governmental organizations or packaged software such as SAP and PeopleSoft. Although not mind-blowing, this work still needs to be done, and limits the amount of resources available for new development.
  • No silver bullet. There is no single methodology, technology or company that is able to provide the productivity to meet up the Gartner challenge, no matter how hard vendors will try to convince you that their new tool or framework will get you there.
  • Complexity of technology. Technology is ever more increasing in complexity. As much as you might suggest that tools and technologies have evolved over the past 20 years – about as long as I have been in this business – we do not reach higher productivity in projects than we did back then during the client-server era. In good old PowerBuilder projects we reached productivities of around 2.5 hours per function point. Nowadays, when applying our full blown agile Accelerated Delivery Platform we can about get to the same level, as was recently calculated by a respectable customer. And this is 20 years later. Another quick example? Take service oriented architecture. Often presented by vendors and architects as an approach where you just model the business processes and out comes your new software. Yeah right. Building service oriented software is still hard and technologically complex work. And no matter who gave you the idea that service orientation will help you get rid of object orientation, don’t believe it. You will need good old object technology to build all this stuff architects come up with: user interfaces, process logic, business logic, interfaces to middleware. It’s just not that simple.
  • Band wagons. We developers tend to go along with every new tool, framework or language extension that comes of the Microsoft, Sun, IBM, Oracle or open source band wagons and base our software architectures on these new, unproven technologies. This results in much time being spend on solving technology issues, rather than customer issues. And I should know, I have been (re)building frameworks over the past 20 years.
  • Snail speed projects. Projects are often executed at snail speed, and have enormous amounts of slack – time wasted by people just waiting for other people to finish their work or waiting on decisions not being made. My worst case? I once spent 4 months lingering in a project that was on hold until a board high up in the organization – indeed, a large international bank – approved on the new budget. And I wasn’t the only one – there were 60 of us.

The sun always shines

As said, there is no technique or technology that will lift our productivity a 100-fold. No sir. Moreover, there is not even a combination of technologies that will do so, no matter how much vendors would like you to believe that they’ve just invented the silver bullet. We’re just not there yet. But we could try. There are promising technologies, tools and methodologies. There always have been. Some ideas will help us get underway slowly and steadily (alas):

  • Become agile. Letting go of large process and waterfall styled methodologies will produce better software a higher velocities. Agile software development is more cooperative, responsive to changes, and removes much of the air from waterfall styled projects. Not only in small, but also in very large and even distributed projects. After having evangelized agile software development for the past 10 years, I know see agile finally moving into the mainstream.
  • Become communicative. Do not design and write your software under water, and submerge after months. Keep in touch with your customer during the whole project. Preferably execute (at least a large part) of your project at the customers’. Let him or her be in the lead. Shorten your feedback cycles to a daily, if not hourly basis.
  • Become standardized. Why does every new project define its own software architecture? And why are a large number of these new architectures so poorly designed? Why do we developers go along with every new (and often unfinished) technology fad? Standardization on good, sound architecture (domain driven of course) will help create better quality software that is easier to maintain and extend in the future. But be pragmatic, travel light. Even better, standardized software architectures allow for advanced, productivity increasing techniques, such as model driven software development and further down the road domain specific languages.

Again, none of these ideas are the silver bullet to solve all our problems. And the current financial crisis doesn’t help either. But, enough said. I’m not into pessimistic locking. We geeks have the best jobs in the world. The sun always shines from our asses. Just think of it. What other job will just let us write code all day?