Offshore Agile Software Development: Does It Work?

Posted on May 15, 2013 by shoogend

Due to the ever-rising demand for seasoned software developers in the nineties, offshore software development became a compelling alternative to in-house development for many organizations. Despite the cultural, language and time differences and the geographical distance involved, more and more projects were executed with offshore development and testing, benefiting from lower rates of cost and the high availability of people, and where necessary, the ability to have teams working around the clock.

Early offshore

Particularly during the early years of offshore software development, the majority of projects were executed using rather traditional, Waterfall-style approaches. Projects were characterized as fixed-price and fixed-date. The requirements and the design were compiled onshore, after which coding and testing was done in another part of the world, be it Eastern Europe, India or South America.

The Waterfall model typically involves executing each of the activities in software development consecutively, and only once – requirements, analysis, design, code, test and deployment – to deliver a product based on the completion of each of these project milestones. Interestingly, this model has led to high failure rates in projects, even without offshoring some of the activities. At each milestone knowledge is lost and testing is executed very late – only when development is done – with exponentially rising costs of repairing bugs. Moreover, achieving completeness in requirements early on in a project is difficult and changes to requirements are costly.

Oddly enough, even with these anomalies, and even with failing local projects, many organizations still ventured into offshore software development applying the Waterfall model. Not surprisingly, many of these projects failed to deliver on time or on budget and did not deliver the required functionality. Despite the hoped-for benefits of lower costs and high availability of skilled people, offshore projects add another level of complexity due to more complex control and coordination, and because of language, cultural and time zone discrepancies.

Is Agile more suited for offshore than Waterfall?

So with Agile approaches, such as Scrum and Kanban, reaching the peaks of their popularity, an interesting question is: can Agile approaches and techniques overcome some of the shortcomings of offshore Waterfall development?

In short, Agile approaches are characterized by working in short iterations, where during each iteration a number of continuously re-prioritized work items are fully realized by a multi-disciplinary team, usually applying a similar life cycle per work item – requirements, analysis, design, code, test and deployment – as Waterfall uses over a whole project.

At the start of an Agile project the requirements are only identified, and not compiled into full detail. This list of requirements is known as the backlog, and is not designed to be complete. Rather, items from the backlog are elaborated on during iterations (also known as sprints). So all the real work is done during iterations. As a consequence, Agile teams are required to be multidisciplinary, and work together on a daily basis to implement functionality work item by work item. As such, co-location of teams, quite often at the client, works best in Agile.

So is Agile more suited than Waterfall for offshore software development? Clearly there are huge benefits. Requirements, analysis and design needn’t be finalized until a work item is actually implemented. So there is no grand, fixed, inflexible design decided upfront. Testing and deployment also take place immediately after coding the individual work items, not only at the end of the project. The total life cycle of a work item is usually no longer than a couple of days, instead of the whole project duration. That is why Agile works well in domains where it is accepted that requirements are never complete and might change, which is the case in the vast majority of projects worldwide.

So what happens when Agile approaches and techniques are applied to offshore software development to overcome Waterfall shortcomings? Apart from the apparent benefits, applying Agile to offshore also comes with consequences. Applying an Agile approach will involve close collaboration between all roles, whether on-site or offshore. It will also involve daily communication, and the ability to work on the same work item at the same time. Communication is therefore key in Agile projects, and as we all know, distance makes communication harder. Therefore offshore Agile teams need to be able to rely on other means of communication than on-site teams. Moreover, it is key that information is clear and unambiguous, which is difficult as this is exactly the bottleneck that Agile is trying to overcome, as work items are not elaborated on until implemented.

But despite some of the difficulties involved in offshore Agile projects, particularly in highly complex domains or around regulatory sensitivities, Agile approaches have the potential to offer many benefits. In my opinion, it almost goes without saying that offshore Agile is going to be more effective to most organizations than offshore Waterfall, but only when key issues around communication, overheads and language issues are overcome. I will explore some of the key strategies for making offshore Agile software development work in my next post.

This post was also published on the ISD Connect website at:

Offshore Agile Software Development: Does It Work?

The changing interpretation of agile

Posted on April 28, 2013 by shoogend

For as long as I can remember I have been evangelizing, promoting, practicing, coaching, and training agile. For me as a developer the goals for applying agile approaches and techniques are pretty clear. I want to make better software. Higher quality, better suited for use, and possibly also faster. And from my own empirical evidence I can certainly state agile helps.

Gone with the wind

So you would say the current raging popularity of everything agile would bring greater joy to my personal life. Unfortunately it doesn’t. Agile has become the new magical keyword for improving just about anything thinkable. There’s an agile mindset. The word agile is dropped on a daily basis in advertisements. There’s agile cars, agile games, and even agile washing powder. And on the fly agile seems to inspire people, communities and even movements to change the world. Great.

Yes, I know agile is characterized in dictionaries by quickness, lightness, and ease of movement or even nimble. So yes, I totally agree agile can be applied to much more than software development. But to quote Clark Gable in Gone With The Wind: Frankly, my dear, I don’t give a damn. I don’t care much for changing the world, or even changing organizations. I want to write software. Better. Faster.

Let me give you some examples where agile is going. At a recent agile conference delegates put up a board of interesting books. It contained highly agile books, such as Your Brain At Work, Tribal Leadership and Fearless Change. Without any doubt very interesting and worthwhile books. But what happened to the classics? I didn’t spot Design Patterns, Framework Design Guidelines, Code Complete, Clean Code, or Patterns Of Enterprise Application Architecture?

These days, agile is used as a adapter for all kinds of coaching and personal improvement. From Getting Things Done through neuro-linguistic programming to gamification. There is even an agile game conference in the planning. Or what about personal energy management, product owner “light bulbs” or agile kitchens? All very nice, but very little to do with programming. As said, agile is becoming an accepted synonym for any form of personal development.

Half-full or half-empty?

And it’s getting worse. In many organizations I’ve noticed that agile is both used to illustrate failure of things that have absolutely nothing to do with agile, as well as an explanation for things that go well. Also having nothing to do with agile. Let me also give some examples of this interesting use of the word agile.

First of all, agile seems to become synonymous for anything that is done just-in-time. For instance, some account manager, barely making the deadline for submitting a proposal to a client, commented on the process stating: “Great, we did this really agile,” suggesting that delivering just-in-time is really agile. Nor the proposal, nor the way it was written had anything to do with agile whatsoever. At another occasion, two people showed up late for a meeting. The organizer of this meeting, which was neither a kick-off, a retrospective nor a stand-up meeting, just smiled and said: “Well, that’s agile, isn’t it?” Well, it isn’t. So agile also has become a synonym for things are run over a deadline. The glass can be half-full or half-empty, depending how you look at it.

I suppose it is due to the liberal definition of agile that the word, and in its slipstream words such as Scrum, Kanban and Lean have become daily speech patterns. I am quite aware that we as software developers and testers have no right to claim the word agile for ourselves. And I for one certainly wouldn’t want to deprive anyone from having fun with it. Everybody is entitled to personal development, or to chasing his or her dream to change the world or even an organization, but sometimes I wish we could just go back to what we developers and testers do best: write code and test functionality.

Agile business intelligence

Posted on October 28, 2012 by shoogend

Het besparen van kosten is een veelgenoemde aanleiding voor Business Intelligence (BI) projecten. Zo wilde een bekende overheidsinstantie weten hoe effectief de bestrijding van uitkeringsfraude was. Het onderzoeken van mogelijke fraude kost de instantie geld, maar het vinden van fraudeurs levert echter direct geld op. En dus ging zoekt de instantie naar de optimale verhouding tussen het aantallen onderzoeken en het aantal opgespoorde fraudes. Kortgezegd wilde men met zo min mogelijk onderzoeken zoveel mogelijk fraudeurs vinden.

Deze doelstelling is archetypisch voor BI-projecten. Hoewel de doelstellingen vaak concreet zijn te definiëren, is het realiseren ervan niet altijd evident. Welke rapportages moeten worden ontwikkeld? Wat staat daar op? Welke bronsystemen moeten worden geraadpleegd? Vaak wanneer het project eenmaal loopt, doen zich doorlopend nieuwe inzichten voor. Bijvoorbeeld over het ontbreken van benodigde informatie in bronsystemen. En anders doet de opdrachtgever wel inspiratie op voor nieuwe wensen en eisen uit feedback over opgeleverde rapportages en analyses. BI-projecten kenmerken zich vaak door onvolledige requirements en doorlopend voortschrijdend inzicht.

Op het gebied van aanpakken voor systeemontwikkeling hebben zich de laatste jaren enorme veranderingen voorgedaan. Steeds meer organisaties en projecten stappen over op een nieuwe generatie aanpakken, die voortschrijdend inzicht niet langer schuwen, maar juist omarmen. Ook kenmerkt deze nieuwe generatie aanpakken zich door multidisciplinaire samenwerking en het frequent in korte iteraties opleveren van software. In één woord: agile. De vraag is nu of en hoe deze aanpakken ook een positieve bijdrage kunnen leveren aan het uitvoeren van BI-projecten.

Wat kenmerkt BI-projecten?

De doelstellingen van BI-projecten zijn altijd direct business gerelateerd. Denk bijvoorbeeld aan het minimaliseren van verzekeringsfraude of het behouden van klanten. Tijdens een project worden analyses en rapportages gedefinieerd die ondersteunen bij het beheersen en optimaliseren van de bedrijfsprocessen van de opdrachtgever. Deze rapportages en analyses worden – meestal dagelijks – gevoed uit een datawarehouse. Kenmerkend voor dit type projecten is dat vaak lastig is vast te stellen welke concrete bijdrage deze analyses en rapportages uiteindelijk leveren. Neem voorbeeld de eerdergenoemde overheidsinstantie, waar niet op voorhand was uit te drukken hoeveel geld men kon besparen bij het vinden van de optimale verhouding tussen het aantal onderzoeken en het aantal gevonden fraudeurs. Uiteindelijk bleek in dit voorbeeld pas na afloop van het project dat de resultaten nog beter waren dan van te voren was geschat.

Hoewel vroegtijdig in projecten nog is vast te stellen welke analyses en rapportages benodigd zijn, is de exacte invulling hiervan nauwelijks concreet te formuleren. Wat wil de opdrachtgever nu echt zien in zijn rapporten? Neem als voorbeeld een rapportage over de verhouding tussen inkomende en uitgaande berichten bij een telecom-operator. Pas toen de opdrachtgever het rapport onder ogen kreeg, bleken er diverse soorten inkomende berichten te zijn, die ook weer gekoppeld zijn aan diverse soorten uitgaande berichten. Een typisch voorbeeld van voortschrijdend inzicht.

Een interessant fenomeen is ook het extractie-, transformatie- en laadproces (ETL). Hierbij worden in een aantal stappen de gegevens uit bronsystemen verzameld, geïntegreerd en geaggregeerd tot een formaat dat voor de rapportages en analyses benodigd is. In de meeste BI-projecten beslaat dit type werk circa tachtig procent van de ontwikkeltijd.

Toch blijft dit werk meestal helaas onzichtbaar voor de opdrachtgever. Deze concentreert zich – terecht – vooral op de op te leveren rapportages en analyses. Maar doordat de bulk van het werk in een BI-project op ETL is gefocust, worden meestal ook de fasen van een dergelijk project rond ETL ingericht. Dat wil zeggen dat eerst alle extracties worden ontwikkeld, aansluitend de transformaties, om vervolgens alle gegevens te laden. Pas nadat dit is gelukt worden de rapportages en analyses gedefinieerd, zoals weergegeven in onderstaande afbeelding.

Het gevolg hiervan is dat pas in deze laatste fase van het project iets wordt opgeleverd waarmee de opdrachtgever aan de slag kan. Dit heeft een belangrijk nadeel. Voor de opdrachtgever blijft het project lang onder water. Pas nadat veel tijd en geld is geïnvesteerd, kan de opdrachtgever feedback geven op de geproduceerde rapportages en analyses. Bovendien is de ETL dan al mind of meer volledig opgeleverd. Aanpassingen aan rapportages en analyses zijn nu lastig te realiseren. Ook is het niet uitgesloten dat als gevolg van deze feedback sommige stappen uit de ETL overbodig blijken. In dit geval is er zelfs werk voor niets uitgevoerd. Ten slotte komt het voor dat voor de gewenste rapportages en analyses gegevens nodig zijn die niet direct uit de bronsystemen zijn af te leiden. Aanvullende gegevens worden dan vaak handmatig toegevoegd tijdens de ETL. Vaak worden hiervoor gaandeweg het project kleine administratieve applicaties ontwikkeld. Nog los van het feit dat deze applicaties door de verkeerde ontwikkelaars worden ontwikkeld – BI-ontwikkelaars in plaats van softwareontwikkelaars – worden deze hiaten meestal pas laat in het BI-project ontdekt. Met alle gevolgen van dien. Zo uitgevoerd zijn veel BI-projecten duurder dan strikt noodzakelijk.

Wat kenmerkt agile?

In software development is afgelopen jaren een nieuwe generatie aan aanpakken ontstaan, die de best practices van eerdere generaties koppelen aan een sterk iteratief en coöperatief karakter. Deze aanpakken, zoals DSDM, extreme programming (XP), Scrum en Smart kenmerken zich in:

Korte iteraties. Project worden uitgevoerd in korte iteraties, variërend van twee weken tot een maand. Tijdens ieder van deze iteraties wordt een klein deel van de software geanalyseerd, ontworpen, gebouwd, getest en zelfs opgeleverd aan de opdrachtgever. Pas bij de start van een iteratie wordt vastgesteld welke functionaliteit tijdens de komende iteratie wordt gerealiseerd. Projecten verkorten zo de feedback lus met hun opdrachtgever. Dit verbetert de kwaliteit van de ontwikkelde software in hoog tempo. Dit in tegenstelling tot traditionele projecten, waar de software in een big bang wordt opgeleverd aan het eind van het project.
Compacte eenheid van werk. Om dit te kunnen bereiken hanteren projecten een eenduidige en kleine eenheid van werk. Er worden altijd meerdere workitems opgeleverd per iteratie. Individuele workitems leveren direct waarde opleveren voor de opdrachtgever.
Snel en frequent opleveren van software. Tijdens agile projecten wordt ook al tijdens de eerste iteraties workitems opgeleverd aan de opdrachtgever, al dan niet direct in productie. Dit zorgt ervoor dat mogelijke problemen, bijvoorbeeld rond architectuur of infrastructuur, al snel in het project boven water komen.
Incorporeren voortschrijdend inzicht. Anders dan in traditionele projecten, waar voortschrijdend inzicht zoveel mogelijk wordt uitgebannen, is het in agile projecten mogelijk en zelfs gebruikelijk nieuwe en wijzigende requirements direct mee te nemen. Dit kan doordat steeds bij de start van een nieuwe iteratie wordt vastgesteld welke workitems worden gerealiseerd. Nieuwe workitems kunnen nu al worden meegenomen, ten faveure over al eerder benoemde workitems.
Nauwe samenwerking klant en opdrachtnemer. Het snel en frequent opleveren van software in korte iteraties vraagt een intensieve samenwerking tussen opdrachtgever en opdrachtnemer. Er vindt bij voorkeur op dagelijkse basis overleg plaats, bijvoorbeeld om de nieuwe te realiseren workitems te analyseren.
Geïntegreerde testen. Omdat software frequent en al vroegtijdig wordt opgeleverd in projecten, is het testen van de workitems van cruciaal belang vanaf dag één in een project.

Scrum

Van alle agile aanpakken is Scrum verreweg de populairste. Niet zelden wordt Scrum met agile vereenzelvigd. De helderheid van de aanpak maakt Scrum een goed uitgangspunt voor projecten.

Een Scrum-project begint zodra de lijst met workitems is vastgesteld. Dit heet de product backlog. Meestal omvat deze een verzameling user stories. De backlog wordt vastgesteld door de vertegenwoordiger van de opdrachtgever, de product owner.

Iteraties, hier sprints genoemd, duren in de regel twee tot vier weken. Bij de start van een sprint wordt tijdens de sprint planning meeting door de product owner samen met het team de te realiseren user stories vastgesteld. Het team verdeelt deze in taken en schat de hoeveelheid werk hieraan in uren in. Op basis hiervan en op basis van de snelheid in vorige sprints en de samenstelling van het team in de komende iteratie wordt bepaald hoeveel stories er in de sprint passen. Deze stories worden op de sprint backlog geplaatst. Aan het einde van een sprint vindt de sprint review meeting plaats, waarin de gerealiseerde work items worden geëvalueerd. Aansluitend vindt de retrospective plaats, waarin het team de werkwijze evalueert en verbetert.

Scrum kent slechts een beperkt aantal rollen. Het werk wordt gedaan door het team, dat in de regel vijf tot negen personen telt en waarvan de individuele rollen niet zijn beschreven. De product owner vertegenwoordigt in het project de klant. Een Scrum Master coacht de product owner en het team. De voortgang in het project wordt bewaakt op een eenvoudig dashboard.

De eenvoud en populariteit van Scrum maken de aanpak een goed raamwerk voor startende projecten. De toegepaste terminologie uit de aanpak werkt aanstekelijk. Scrum is eenvoudig toe te passen en kan gemakkelijk waar nodig worden uitgebreid met technieken uit andere agile aanpakken.

Smart

In onze ervaring kan Scrum voor BI-projecten goed worden gestructureerd met elementen uit andere agile aanpakken, in dit geval Smart. Deze van origine Nederlandse agile aanpak kent meerdere typen iteraties. Bij de start van een project wordt de backlog gevuld met work items die voorwaardelijk zijn om software te realiseren. Denk aan het vaststellen van stakeholders en doelstellingen, het modelleren van bedrijfsprocessen, rapportages en analyses in smart use cases, het maken van een schatting op basis van deze smart use cases, het opstellen van een base line architectuur, het inrichten van de ontwikkelomgeving en het maken van een projectplan. Deze workitems worden gerealiseerd tijdens de voorbereidende iteraties Propose en Scope. De iteratie Propose mondt uit in een eerste projectvoorstel. Scope eindigt met het opleveren van het projectplan.

Na deze inleidende iteraties wordt de backlog gevuld met de gemodelleerde smart use cases, die de standaard eenheid van werk zijn in Smart. De smart use cases worden gerealiseerd tijdens een of meerdere releases. Een release bestaat uit een reeks van Realize iteraties, gevolgd door een Finalize iteratie. Tijdens Realise iteraties worden de smart use cases gerealiseerd, getest en geaccepteerd. Iedere release wordt afgesloten door een Finalize iteratie, waarin de nadruk nog sterker ligt op testen en op het stabiliseren van de code.

Iedere iteratie heeft in Smart dezelfde opbouw. De iteratie start met een kick-off Plan en eindigt met de retrospective Evaluate. Daartussen worden de work items gerealiseerd tijdens Build.

In Smart telt het team meerdere rollen. De belangrijkste zijn projectsponsor, gebruiker, domeinexpert, ontwikkelaar, tester en coach. Smart doet het goed in langlopende, vaak wat complexere projecten zoals BI-projecten, waarin het meer structureren van rapportages en analyses in smart use cases beter past dan eerdergenoemde user stories. Smart use cases zijn direct gerelateerd aan de bedrijfsprocessen van de opdrachtgever. Bovendien worden ze gemodelleerd en geschat op basis van een reeks aan standaardtypen, zogenaamde stereotypes. Er zijn al stereotypes beschreven voor bijvoorbeeld stappen in ETL, rapportages en analyses.

Snel en frequent opleveren

Het snel en frequent opleveren van voor de opdrachtgever relevante software is een aspect van agile dat in BI bijzonder goed van pas komt. In plaats van de verticale fasering van traditionelere BI-projecten, kiezen we ervoor om het realiseren van analyses en rapporten juist horizontaal in te steken. Niet langer worden eerst alle extracties en aansluitend de transformaties en het laden van de gegevens opgeleverd om pas daarna de rapportages en analyses te ontwikkelen.

Liever ontwikkelen we per rapport of analyse. Hierbij kiest de opdrachtgever steeds welke rapporten of analyses de hoogste prioriteit hebben, en werkt het team uitsluitend aan de minimale set aan smart use cases die nodig zijn voor om deze te realiseren. Deze representeren de benodigde extracties en transformaties en natuurlijk het rapport zelf. Deze kanteling van werkzaamheden maakt directe feedback van de opdrachtgever mogelijk. Belangrijk bijkomend voordeel is dat de rapporten en analyses direct kunnen worden aangewend om de bedrijfsprocessen van de opdrachtgever te verbeteren. Soms al enkele weken na de start van het BI-project. Zo toont het project al op heel korte termijn benefits.

Zo wordt voortschrijdend inzicht nu eens niet uitgebannen, zoals traditionele, maar worden nieuwe inzichten direct meegenomen. Hoewel in principe tijdens een lopende iteratie de scope niet wijzigt, kunnen nieuwe requirements al tijdens een eerstvolgende iteratie op de rol worden gezet. Zodra de opdrachtgever de eerste versie van een rapportage of analyse onder ogen krijgt, formuleert hij direct zijn feedback, die dan vrijwel direct wordt geïmplementeerd – denk daarbij bijvoorbeeld aan het realiseren van data entry voor ontbrekende gegevens.

Compacte eenheid van werk

Kort gezegd is een BI-project uit te drukken in drie typen ontwikkelwerk: datamodellering en ETL, het definiëren van analyses en rapporten en het ontwikkelen van aanvullende data entry applicaties. In Smart is de smart use case de leidende eenheid van werk. Zowel voor het modelleren van bedrijfsprocessen en requirements, en voor het schatten, realiseren en testen van voor de gebruiker relevante functionaliteit. Voor het modelleren van smart use cases beschikken we over richtlijnen, die ertoe bijdragen dat smart use cases een lage granulariteit kennen en al tijdens Propose en Scope zijn te modelleren. Er is bovendien een groot aantal stereotypes beschreven die het modelleren, schatten en realiseren standaardiseren en vergemakkelijken. Voorbeelden hiervan zijn manage voor data entry, search voor het zoeken naar record of file import.

Hoewel afkomstig uit reguliere software development, blijken smart use cases ook prima aan te wenden voor agile BI-projecten. Wat betreft het definiëren van rapportages en het ontwikkelen van aanvullende data entry ligt dit voor de hand, omdat dergelijk werk niet wezenlijk verschilt van reguliere software development. Maar ook voor het vaststellen van analyses en zelfs voor het uitvoeren van ETL zijn use cases stereotypes vastgesteld, zoals collect, integrate en aggregate.

Smart use cases worden al vroeg in een agile BI-project gemodelleerd, tijdens Propose en Scope. Daarbij worden de smart use cases slechts geïdentificeerd en geschat. Details worden pas uitgewerkt tijdens latere iteraties. Aansluitend worden Realize en Finalize iteraties ingepland. Tijdens deze iteraties ligt de focus op het realiseren van individuele rapportages, op basis van de hiertoe benodigde smart use cases voor ETL, eventuele data entry en de definitie van het rapport.

De benodigde smart use cases worden zo snel mogelijk gerealiseerd en met de opdrachtgever afgestemd. Bijkomend voordeel is dat de onderliggende dataflows, die nu zijn uitgedrukt in smart use cases, goed individueel te testen zijn, en zelfs is door het modelleren van de use cases hergebruik van dataflows snel geïdentificeerd.

In BI-projecten is het snel en frequent opleveren van nieuwe rapportages en analyses van groot belang voor de opdrachtgever. Immers, ieder nieuwe rapportage kan direct worden benut in de praktijk en levert zo direct toegevoegde waarde aan het optimaliseren van de bedrijfsprocessen van de opdrachtgever. Daarnaast hebben BI-projecten veel baat bij het voortschrijdend inzicht dat ontstaat dankzij de korte iteraties in agile projecten. Beter dan dit uit te bannen, zoals traditioneel wordt getracht, is het om juist effectief gebruik te maken van deze inzichten en feedback.

Dashboards en burn-down-charts

Om agile projecten te beheersen en de voortgang te bewaken wordt in de regel gebruikt gemaakt van een tweetal pragmatische gereedschappen; een agile dashboard of taskboard en een burn-down-chart. Alle te realiseren workitems doorlopen een levenscyclus, die de stappen uit het realiseren ervan beschrijft, doorgaans in enkele dagen tijd. Voor smart use cases telt deze levenscyclus stappen als New, In Iteration, Working, Testing, Rework en Accepted. Doorgaans breiden projecten deze cyclus uit naar gelang de projectspecifieke werkwijze dit verlangt. De stappen in de levenscyclus van workitems of smart use cases vormen de kolommen op het dashboard of taskboard van een agile project.

De meeste agile projecten gebruiken voor dergelijke dashboards post-its aan de muur, of een online gereedschap. In een oogopslag is zo, ook voor de opdrachtgever, te zien wat de voortgang van de smart use cases is.

Omdat de omvang van smart use cases worden geschat in punten, is bij ieder statuswijziging snel na te gaan hoeveel werk nog nodig is om de onder handen zijnde rapportages en smart use cases te voltooien. Zodra een smart use cases is geaccepteerd, krijgt het team de bijbehorende punten. In agile BI-projecten is het daarbij belangrijk dat ook de “back end” smart use cases die stappen uit de ETL of data entry representeren door de opdrachtgever worden geaccepteerd. Dit kan bijvoorbeeld gebeuren met de opdrachtgever de resultaten van dergelijke use cases in de reporting tool te demonstreren.

Een burn-down-chart toont een dagelijkse momentopname van deze punten, uitgezet in de tijd. Met een eenvoudige extrapolatie is nu de verwachte einddatum van het project te calculeren.

In agile BI-projecten is het overigens niet alleen zeer zinvol een burn-down-chart te gebruiken voor het gehele project, maar ook om per te realiseren rapport en analyses deze voortgang te projecteren. Juist omdat de rapportages individueel worden opgeleverd, leveren deze laatstgenoemde burn-down-charts de opdrachtgever directe informatie over wanneer het nieuwe rapport is in te zetten in het besturen van zijn bedrijfsprocessen.

Snel resultaat

Agile aanpakken zoals Scrum en Smart spelen bijzonder goed in op de snel wijzigende en uitbreidende wensen en eisen van opdrachtgevers aan BI-projecten. Daarbij wordt in korte iteraties aan het “goed genoeg” opleveren van individuele rapporten en analyses gewerkt. Zo heeft de opdrachtgever al veel eerder profijt van zijn BI-project, en kan voortschrijdend inzicht sneller en goedkoper leiden tot een optimaal eindresultaat. Het toepassen van smart use cases biedt BI-projecten daarnaast een gestructureerde, maar vooral pragmatische manier om met eenzelfde eenheid van schatten, realiseren en testen te opereren, die bovendien direct te relateren is aan de bedrijfsprocessen van de opdrachtgever. De pragmatische gereedschappen die in agile projecten worden gebruikt om de voortgang te meten, zoals agile dashboards en burn-down-charts per rapport, bieden bovendien direct inzicht in de realisatie van de rapportages en analyses. Agile BI biedt daardoor snel resultaat met snel groeiende tevredenheid van de klant. En daar was het allemaal om te doen toch?

Dit artikel verschijnt in het themanummer Agile Datawarehousing van maandblad Informatie

Sander Hoogendoorn
Principal Technology Officer en Agile Thoughtleader Capgemini, auteur van de boeken Dit Is Agile (agile, Scrum, Smart) en Pragmatisch Modelleren met UML (smart use cases).

Sandra Wennemers
Principal Consultant en Data Warehouse Architect Capgemini

Zie ook www.smartusecase.com en www.ditisagile.nl

The Waterfall Dentist Anti-Pattern

Posted on October 3, 2012 by shoogend

Believe it or not, but I have a confession to make. I’m currently in a pure waterfall project. It’s my first in many, many years and despite the fact that I love the technology, I don’t like our way or working it a bit.

During the first phase of this project we are trying to deliver twenty functional work items. All of these twenty work items are analyzed first, then reviewed. After all review comments have been processed, all work items are designed. And again reviewed. After all review comments have been processed again, the developers start constructing them. And once they finish the testers have a go at the software produced.

Yes, I know this is efficient nor effective. And yes, the testers find lots of defects that could have easily been prevented using an iterative approach where we jointly work on a single work item, and only after it’s finished move on to the next. But we just don’t. And the delivery date is nearing. As a result, at the delivery date we will likely deliver twenty half-baked work items, instead of delivering maybe ten or twelve complete work items. It’s a choice.

End of discussion

You could say that the discussions about the chosen approach I have with the project manager are vivid. At the very least. “No, we’re not going to do agile,” he clearly states. “I don’t care if we call it agile or not,” I respond, “But the way we are working now is simply inefficient.” Personally I really don’t care if projects are tagged as agile, Scrum, Smart, RUP or even waterfall. I just dislike inefficient processes. “This approach is easy to explain to the client,” the project manager continues, “So we are going to stick to it. End of discussion.”

And then coming out of my girlfriends’ apartment building this morning I saw this car in the parking lot. It occurred to me immediately. We are waterfall dentists.

Imaging you’re a young dentist, fresh out of dentistry school, and you are starting a new practice. Why on earth you would name it The Waterfall I don’t know, but twenty new patients are hurdling up to be treated. Being The waterfall dentist you use an appropriate approach. During the first week you examine all twenty patients. During the second week you commence to the first part of their treatment: drilling holes. Then in the third week, all twenty patients come back and have the holes filled. In the fourth week all patients return again for a last check-up. Just testing if all holes are filled properly. Finally, in the fifth week you get to send twenty bills to twenty happy patients. Project done. Next project.

Good bookkeeping and new patients

Doesn’t that sound good? This approach resembles my current project in great detail. So why is it, although this model is easy to explain to patients, no dentist in the world would even slightly consider to apply it? Well, first of all, your patients don’t like to walk around with unfilled holes in their mouths for a least a week. Second, your patients will have to come in to your office four times, and you will have to do pretty good bookkeeping to remember each patients details when they return. If you would have filled a patients tooth right after you’ve drilled a hole, this need for documentation and good memory wouldn’t have been necessary. Moreover, what if a new patient comes in during the second or third week with a terrible toothache? You would have to decline him saying: “Sorry, I don’t take additional work now,” advising the new patient to come back after all current patients are serviced. It’s the vicious change request. And what if you become ill during? Your whole schedule will certainly slide. As a result, none of the patients will have been fully serviced at the end of the scheduled period, leaving them in uncertainty about when you will deliver. Basically, you will loose your patients as fast as they hurdled up. And you won’t be able to send any bills.

I could probably spent another two posts in explaining why the waterfall model isn’t applied by dentist anywhere in the world. But I won’t.

We just might

Dear project managers, despite the fact that the waterfall model is so easy to explain to clients, that doesn’t make it any more effective or efficient. Armed with The Waterfall Dentist Anti-Pattern, you can explain iterative, or agile, approaches to your clients just as easy. In fact you can even service your clients better, as they may send in new urgent patients during the project, and you will be able to service those patients early. And even though you may turn ill during the project, servicing the patients one by one, rather than in a returning batch, you can actually fully service a large number of them within the designated timebox, instead of having serviced all of them half when the deadline approaches. I would much rather be able to send bills to fifteen out of twenty patients, then not being able to bill at all.

So where’s my current project standing? Well, just like The waterfall dentist might actually serve twenty patients during the current phase, we might actually deliver all twenty work items. But I wouldn’t bet my gold teeth on it.

Agile anti-patterns. Yes you agile projects can and will fail too

Posted on May 8, 2012 by shoogend

Over the years I have noticed a lot of agile anti-patterns during projects. Wrongly used agile approaches, dogmatic use of agile approaches, agile-in-name-only. Recently I have presented a talk at a number of agile and software development conferences that demonstrates patterns of agile misuse. These conferences include Agile Open Holland (Dieren), Camp Digital (Manchester), GIDS (Bangalore), ACCU (Oxford) and Jazoon (Zurich). Anyway, here’s the slide deck. Enjoy.

Agile Anti-Patterns. Yes your agile projects can and will fail too.

View more PowerPoint from Sander Hoogendoorn

Evolving agile

Posted on April 4, 2012 by shoogend

Without any doubt agile is the biggest evolution in software development approaches since the introduction of waterfall back in the early seventies. And yes. Agile is an evolution rather than a revolution. The best practices and techniques in agile didn’t just pop-up. Rather they emerged from years of hard-working, real-life experience in succeeding and failing in projects.

So working in short iterations, in multi-disciplinary teams, prioritizing our work items regularly, and testing and delivering early and frequently using simplified communication, as we could capture agile in a single sentence, are improvements we’ve all introduced in our software development projects over the past forty years. Agile, and all of it flavors, such as Extreme Programming, Scrum, Smart or Kanban, have merely evolved from teams trying to do a little better every day. And have succeeded in doing that.

In the second decade of this century, due to the overall success of agile approaches, the majority of organizations executing projects will move towards agile. Either quickly or slowly. And for a variety of reasons. To raise quality. To increase predictability. To execute on-time, on-budget. To minimize time-to-market. Or simply because Gartner says so.

But with the success of agile come new challenges. Agile projects require agile coaches. Whether certified as Scrum Master, Lean Champion, or Pokémon Trainer, there is a lack of sufficiently experienced agile coaches. As a result an increasing number of projects is guided by ill-experienced coaches, often leading to a dogmatic and rigor application of agile. More over large organizations will feel the need to standardize, industrialize or institutionalize their agile efforts, resulting in ill-digestible, comprehensive agile manuals.

Although adaption and creativity are fundamental to agile, these growing pains will slow down the enormous progress we are making in the field of software development approaches. Agile is meant to evolve. Always. Continuously. Improving and professionalizing our software development is not a frog-leap process, proceeding with one giant leap every fifteen years. It is an fluid, never-ending process of many, many little steps taken on a daily basis. We learn from what we do. Yesterday, today, tomorrow, next year.

This post was written as a contribution to the upcoming book “Shaping Apps Germany”, published by Capgemini, Germany.

Death by Dogma versus Agile Assembly

Posted on November 4, 2011 by shoogend

On November 3, 2011 I presented the keynote of the Agile Open Holland Conference in Dieren. During this challenging talk I discussed the current state of affairs in agile organizations and projects and the effects of the recent strong rise in popularity of agile approaches. Let’s put it mildly: there’s a lot of work to be done.

Death by dogma

Almost all organizations, large and small, are turning towards agile to escape failing traditional software development projects. Due to this strong increase in popularity of agile approaches and techniques, many newcomers will enter the field of agile coaching. Of course without the very necessary real-life experience but proudly waving their Certified Professional Scrum Master Sensei Trainer Certificate proving they at least had two days of training.

Going through the hardship of two whole intense days of training becoming a Certified Agile Jedi Knight is worthwhile!

In my opinion, as a result many organizations and projects in the next couple of years will be coached why well-willing consultants who have barely made it through boot camp, and in the famous Shu-Ha-Ri learning model haven’t yet made it beyond copying their teacher. This will lead to very dogmatic applications of the more popular agile approaches, mostly Scrum, especially when the so-called leaders in the field themselves turn to dogma. This dogmatic thinking will block the use of more mature techniques and technology in agile projects, even when these would really improve projects, or would prevent agile projects from failing. “No, you can not do modeling in Scrum” and “Burn-down charts are mandatory” are two such simple real-life example statements that I’ve witnessed some certified agile Jedi Knight make. Due to this lack of experience and the growing dogmatism in the agile beliefs, more and more agile projects will fail. Death by dogma.

During my keynote I discussed many examples of dogmatic Scrum implementations and the drawbacks from being dogmagile, building the story up from my previous posts Scrumdamentalists and Crusaders and Flower-Power Agile Fluffiness.

But maybe even more important during the keynote I also show that there is no such thing as one-size-fits-all agile. Different organizations and different projects require different agile approaches. Sometimes lightweight agile, for instance implemented using Scrum, user stories, simple planning, simple estimation, and one co-located team using post-its on a wall is just fine. But in many projects the way of working used should rather be built up from slightly more enterprise ready approaches, for example using Smart, smart use cases, standardized estimation, multiple distributed teams and on-line dashboards.

What is agile anyway?

Implementing agile in your projects starts with establishing what it means to be in an agile project. As I demonstrate in the keynote, I consider short iterations, collaborative teams, a small unit of work, continuous planning, delivering early and often, and simplifying communication to be crucial to considering a project working in an agile way.

From there you can pick and choose from a wide variety of approaches, techniques and technology. Most of them stemming from the agile era, but some of them can also be related to older (or more mature) era’s. Concluding you might say that to be successful in implementing agile in your organization, you will need to assemble your agile approach from everything that will help you implement these six agile bare necessities. Anyway, enjoy!

Death by Dogma versus Agile Assembly

View more presentations from Sander Hoogendoorn

Flower-Power Agile Fluffiness

Posted on September 8, 2011 by shoogend

To all the dear people in the agile community and to the faint-hearted: this will not be an easy blog post. There was a time when being a software developer was a decent craft, requiring decent craftsmanship and yes also a lot of creativity, some communication, some collaboration. Still it was a decent craft. The waterfall-ish methodologies we used weren’t extremely optimal, but at least software development was a craft. Similar to a carpenter who uses his tools to craft new furniture, or a industrial designer using his tools to come up with a new model Toyota – I know this is not the best example, but at least I now have the attention of the kind folks in the lean community. And then came agile.

Now believe me, I don’t have anything against agile. I’ve been promoting agile and iterative approaches to software development since the mid-nineties, and haven’t done traditional projects ever since. Agile used to be about engineering. We were improving ourselves by using better techniques, continuous integration or continuous deployment, writing unit test, pair programming, writing smart use cases or user stories, using a bug tracker, burn-down charts and even post-its on the wall. So far, it’s still all in a days work. There was a time that as an analyst, a developer, or a tester you could be proud of being in an agile project.

But these days if I look at what’s going on at agile conferences, on twitter, in blog posts, literature and discussions on agile, Scrum, Lean, Kanban and whatever new flavors of the month are passing by, I get the feeling I’m no longer talking about craftsmanship but rather ending up in Disneyland or in San Francisco in the late sixties. I’ve got a feeling were not in Kansas anymore.

Agile coach at work.

Agile community anti-patterns

Certainly it’s a good thing everybody can join the agile community. But I witness a lot of repetitive behavior I strongly discourage. Let’s name this repetitive behavior agile community anti-patterns – not to confuse with agile anti-patterns. The latter merely describe failures in agile projects, and yes these do occur, while the former describe community failures. Let me sum some up for you – while on the fly breaking my first anti-pattern:

Metaphorizing
Zenify
Kindergarten Agile
Open Door Wisdom
Scrumdamentalism

Allow me to elaborate a bit on these agile community anti-patterns.

Metaphorizing

Although I’m not sure metaphorizing is a even good English word, I’m quite sure you get the meaning of it. Everything anybody comes up with these days about agile – or about what people think is agile – is immediately turned into a metaphor or is given a silly name.

Can we please stop talking about the Gemba Walk when we mean that it’s a good thing our manager stops by every now and then? This shouldn’t even have a name.

Japanese manager stopping by.

What does it mean when an agile specialist states that “you should verify the five why’s with the reverse loop”? And what about Feature Injection? According to a recent tweet “feature Injection is more about using examples to discover the relationships you need and missed or got wrong.” Call me old-fashioned but I totally miss what this is about.

Zenify

Yes, I know lean manufacturing started in Japan at Toyota. So there is a link between agile and Japan. But is that an argument to zenify software development?

Our new Feng Shui office space.

Why do we need to explain roles in a software development project as samurai, sensei or roshi. I thought product owner and agile coach were already abstract enough. What about re-arranging our office in a Feng Shui manner? Also the word kaizen seems to become very popular. Quoting a recent tweet: “Just write down small things on small papers. It’s your kaizen.” Although I’m all for small things what does this mean and why do I need to introduce this in my project?

Kindergarten Agile

Not sure about the average level of maturity of people in agile projects around the world, but in the projects I’ve been in the last decade or so, people where pretty mature. So why is it groups of sensible people at unconferences are discussing the Happiness Index of projects? To me this sounds much more like a weekly column in a women’s magazine than a solid engineering practice in a software development project.

And I for one certainly don’t want to have to pass a baton if the flow in my stand-up meeting or scrum isn’t achieved automagically. Still this was seriously recommended by one of the authorities in the agile community during a recent presentation at a conference.

Some tweets to further illustrate my point? If someone says “Add Ready for Celebration before the Done column on your wall board” should I start decorating the office? It makes you wonder how often his projects get things done. Even worse is “Make sure you don’t miss the agile elephant versus the waterfall elephant in the lobby.” which was tweeted from a recent agile conference. Where was this conference held? At the Toys-R-Us?

Participants at a recent agile conference strolling down the exhibition hall.

Open Door Wisdom

Often I see quotes coming from the agile community that are no more than open doors, and have been open doors in projects for decades and perhaps centuries, but that are treated by others in the community as sources of new and ultimate wisdom.

Recently a speaker at an agile conference claimed that “if your retrospectives don’t add value to your project, you should change your retrospectives.” Duh. The speaker got a loud applause from the audience, he is now considered an absolute guru, and his quote got tweeted and re-tweeted all over the world. This is not even an open door, it’s an open gate.

Scrumdamentalism

Now I’ve blogged about Scrumdamentalism before, but with the newer generations of agile converts, some communities are getting more and more populated by religious zealots who will treat their newly gained faith with deep fundamentalism. Any best practice from their own belief is treated as mandatory, while followers from other beliefs are often considered heretics. A recent blog post states that: “A technical project manager can be a good product owner if he sticks to managing the product backlog and abiding by the rules of Scrum.” I wasn’t aware that Scrum even had rules. You learn something new everyday.

And if Scrumdamentalism alone isn’t bad enough already, it is even enforced by the so-called leaders themselves, proven by the following horrible quote “the team needs to listen to god talk and follow the commandments” from one of the Great Leaders in this particular community. Dear agilists there is is no one-true-belief. There’s value in all of them. And also can we please abolish the function title Agile (or Scrum or Kanban or Lean) Evangelist. Moreover, people calling themselves Agile Sensei should be banned from conferences and projects if you ask me.

Flower-Power Agile Fluffiness

Please people can we stop adding all this new age Flower-Power fluffiness to agile. In my opinion the agile community with all it’s great ideals and best practices is slowly degrading into a worldwide Luna park. My guess is I that it won’t be long before someone somewhere will suggest to add a psychotherapist to every software development project. “How do you feel about not being able to get your user story to the Ready for Celebration column?” Or plan a clown’s visit during retrospectives to increase the Happiness Index.

Agile retrospective with product owners present.

We are slowly becoming the laughing stock of the engineering world. I long for the time that we re-introduce engineering in our trade, and all go back to work again.

Scrumdamentalists and crusaders

Posted on June 29, 2011 by shoogend

After having promoted agile and iterative approaches to software development projects for over a decade, I finally find that, like Bob Dylan says, the times they are a-changing. And for the better. Many small and large organizations and enterprises are now turning towards agile approaches, often to compensate for years and years of failing projects. You might suggest that all’s well that ends well . But…

With the growing success of agile approaches new patterns and anti/patterns arise. Implementing agile well, although promoted as being easy by many, isn’t that easy. It’s actually quite hard to get it right. Implementing agile well is a learning process, just like learning to ride a bicycle well, or learning how to master a programming language. This is where coaching on the job can and should play an important role.

Pokemon Trainers and Jedi Knights

But good agile coaches are scarce these days. So many companies jump into this growing gap and send of there consultants to far away agile training courses. Inexperienced or at most with little experience these suddenly acclaimed Scrum Masters, Lean Certified Project Managers, Pokémon Trainers and Jedi Knights are now pushed into the marketplace.

Let me predict you that this explosion of these new gurus will lead to massive agile project failure. Allow me to explain. As with most religions, people just entering the new belief tend to be far more fundamentalistic and dogmatic about their new faith than people who have lived in the same religion for years. Newbies will apply the religion’s commandments far more literally than they are meant – as the elders know. And moreover, these newbies will see there new religion as the only true source of knowledge, denying the value in other religions.

In a sense you might say we’re in the Agile Middle Ages. The new agile fundamentalists are like crusaders, full of enthusiasm, heavily armed, not very adapted or open to the new world they travel too, but with an urge to strike down all non-believers.

Agile fundamentalist in the field

Scrumdamentalism illustrated

This fundamentalism is best witnessed in the Scrum community. Personally, I like Scrum a lot, but too often I see Scrum certified coaches making remarks about what is and what isn’t Scrum, and even worse: what is and what isn’t agile. Let me illustrate my statement with some real-life examples.

In many projects, especially the more enterprise projects, I prefer modeling smart use cases and use these as unit of work, estimation, development and testing way over the more unstructured user stories. At quite a few occasions Scrumdamentalists have come up to me and claim that you must use user stories, otherwise you’re not doing Scrum, but instead you are doing waterfall. Modeling doesn’t seem to be allowed anyway in a Scrum project, if I believe my Scrumdamentalists. Now I haven’t read the little Red Book of Scrum, but I’m pretty sure Scrum doesn’t really push user stories as being mandatory, nor forbids modeling.

Another great example followed a picture I tweeted of my current project’s dashboard during a week when our tester was on holiday. Clearly the list of work items to test was larger than the other columns. My picture led to a remarkable comment from a certified and even experienced Scrum Master that all team members should automatically take over each others work when someone is offline. And if we didn’t do that we were not doing agile, but were doing waterfall instead.

Yes I do consider that taking over each others work is good practice, but to a certain extent. I don’t know about your project, but in my project testing is serious business, with serious techniques, that requires serious expertise. Testers do totally different work than analysts or developers. I don’t see any of my C# developers creating process cycle tests or writing functional test cases. Or even worse, the other way around. Would you seriously consider a tester to learn how to write C# code, just because one of the developers spends a week on the beach? So I guess I’m in a waterfall project.

IMAG0024
A non-proper agile dashboard

And if you think this was the low point of my experiences, another Scrumdamentalist overlooked the picture I posted and noticed that my project has different and more stages a work item goes through than the task board he considered obligatory. We had stages such as developer testing, testing, accepting and approving, while he said that you only have to-do, working and done on a proper task board.

And dear people, is it really necessary to call an iteration a sprint, and a stand-up meeting a scrum to make my project successful?

Process fits project

I strongly believe that this agile fundamentalism and dogmatic thinking is going to stop progress and productivity. It ignores on of the basic principles that makes agile such a strong proposition – namely to improve on your process and techniques continuously to make the process fit your project, and not the other way around.

Dear people in the agile communities, please don’t try to become holier than the pope. Being agile means not trying to be dogmatic. We are not looking for the holy grail. There is no one true agile religion I’m afraid, there’s value in all of them. As the agile manifesto clearly states: we are uncovering better ways of developing software by doing it. So go out and explore.

A recipe for enterprise agile. Mixing Scrum and Smart

Posted on October 24, 2010 by shoogend

To cut to the chase, those of you who have worked on enterprise or service oriented projects before already know this. These types of projects are characterized by a large number of organizational, functional and technically complicating factors. Enterprise software development projects are surrounded by a large number of complicating characteristics and challenges:

Many different stakeholders. Projects have many different parties in the organization that are involved. Often such project realize goals for different departments or divisions in the organization. As an example I once coached a project that served 22 different departments. Quite often these departments or division try to achieve overlapping or even contradicting goals.

IMAG0218[4]

Business processes. In general enterprise projects (re-)implement (parts of) business processes that are key to the organizations success, rather than building a straightforward web application.
Complex software architectures. Most likely, enterprise projects implement business processes in a complex IT landscape or architectures. Think of landscapes that host many different applications all tied together.

IMAGE_634[8]

Many external dependencies. Often, such complex IT landscapes contain components that are outside of the organization. Services that run outside the corporate firewall, with other parties. A well known issue to agile projects is that of availability. We need the new version of the service in this iteration, but the other party runs a waterfall project that will not deliver this year.
Changing requirements. In enterprise projects, requirements change continuously. Either by changing insights during the project, but more often due to changing legislation or regulations.
Many different types of deliverables. Traditionally these projects count many different types of deliverables. Think of service description, user interface, process models, non-functional requirements, workflow, project plans, test plans and many, many others.
Multiple roles. Large organizations count many different roles that are involved in projects. There are business analysts, enterprise architects, information analysts, testers, release managers, SAP developers, web developers, SAP functional consultants,

Thus these project often become a mission impossible, and fail in large numbers. Such projects are difficult to estimate, therefore hard to plan. For developers it’s tough to build the software, often with many new techniques and technologies, and even worse for testers, these projects are sheer Impossible to test well. This is where a structured agile approach should come to the rescue.

Being agile in enterprise projects

Independent of the approach that is applied, or the agile process that is followed, being successful in agile projects follows a number of key principles:

Multi-disciplinary teams. Instead of different roles such as analysts, designers, developer and testers working in consecutive stages in a project, all roles collaborate in implementation individual work items from heads to tales.
Short iterations. Project work in short time boxes, in most cases 2, 3 or 4 weeks. During such an iteration, a number of work items is planned, completed and evaluated – with different terminology in use with different agile processes.
A small standardized unit of work. Projects requires a small unit of work, to be able to deliver a number of them in a single iteration. Large work items such as traditional use cases therefore do not apply well.
Testing from day one. Testing is an important discipline during agile projects. Testers are involved from the very first iteration, when the first work items are realized. Note that testing in most projects goes way beyond the obvious unit testing – which is actually a developer technique.
Continuous measurement. Successful agile projects continuously measure their progress. Since they are always delivering individual work items, measurement in agile projects is easy.
Continuous (process) improvement. At the end of each iteration not only the realized work items are evaluated, but also the project approach itself is evaluated, leading to highly effective project approaches.

Scrum in a nutshell

When organizations first start with agile projects, Scrum is often the process of choice. At this point in time, Scrum is by far the most popular and best known agile approach. Scrum is a straightforward lightweight agile process that offers a collection of highly applicable principles and techniques for short iterative projects and is easy to understand. It also specifies a remarkable terminology with scrums, sprints and user stories.

image[3]

In a nutshell, Scrum is characterized as follows:

Sprints. Projects are divided into iterations, called sprints. In general these are 2 to 4 weeks.
Backlogs. Work items still to be done during a project reside in a so called project backlog. At the start of each sprint items from the backlog are placed in the sprint backlog. These are the worked items to be realized.
Sprint cycle. All iterations follow the same simple process, with an iteration kick-off, actual work and a retrospective workshop for evaluation.
User stories. The main unit of work is usually the user story, although this is not mandatory. This rather informal requirements technique is shared with the agile process extreme programming.
Planning. In general the project and sprint planning is visualized in task boards, which usually is put up on a wall using post-its. Progress is monitored by using burn down charts, a straightforward diagram that on a daily basis displays the number of points still to realize. A trend line in the burn down chart extrapolated from the points identifies the likely project end date. Again this technique is shared with a number of other agile processes, including Smart and extreme programming.
Delivery. Every iteration or sprint results in fully delivered products, software or otherwise.

Scrum has proven to be a successful approach in many (smaller scale) web projects. In larger, more complex, service or cloud oriented and enterprise projects, I think Scrum needs to be augmented to fit the project and its environment. Let me explain.

Beyond lightweight agile

When applying Scrum or other lightweight agile processes such as Crystal Clear or the set of lean principles (applied to software development that is) to more complex projects of all sorts, you might recognize a number of shortcomings:

Business goals. It is often unclear how the work items in such projects (in most cases user stories) link back to business goals, stakeholders and business processes to implement.
Analysis and design. Although many people in the so-called agile community will now think I’m promoting a waterfall approach, in most agile approaches there are hardly any analysis or design activities or deliverables defined.
For instance, it is undefined how the backlog in a Scrum project is assembled. Let me assure you, in more complex projects some analysis and design is required, albeit not in big-upfront-design mode.
Architecture. The same argument goes for software architecture. Most agile approaches lack decent software architectural activities and deliverables. Architecture is often assumed as-is. If you’re implementing business processes that run through a systems landscape of over 40 collaborating systems, you will definitively need to design your architectures, but again this need not be in big-upfront-architecture mode.
Roles. Scrum and extreme programming describe a very limited set of roles that, although looking slick, in most projects does not adhere to the standing organization. There’s often enterprise architects, information analysts, business analysts, middleware developers, application management, testers, release manager. How will a project work with all those roles? There’s a bit more to it than just saying they are allowed to participate in the daily scrum.
Testing. Apart from the obvious (developer) techniques such as unit testing, the role of testing more specifically – which is essential in larger agile projects – is not very well described in lightweight agile approaches.
User stories. Although very flexible, user stories are a highly unstructured unit of work, despite efforts in epics or story maps. Applying such an unstructured unit of work may result in low re-use, non-repeatable estimation, difficult to standardize development, testing and measurement beyond the individual projects. In a multi-project environment user stories will block progress and reuse (of for instance metrics and code).
Delivery cycle. Scrum does not specify the organization of delivery of the products to different environments such as test, acceptance and (pre)production (so called OTAP).

When executing large, more complex, service oriented or enterprise projects, I therefore strongly recommends to overcome these limitations by augmenting Scrum and the likes with additional techniques.

Smart accelerators

In my personal experience in enterprise projects I have had very good experiences with mixing the agile process Scrum with the best practices from another agile processes called Smart to overcome the drawbacks of the aforementioned approach.

Smart originated from the Netherlands in 1998 and was originally designed to implement best practices for DSDM, an iterative approach that was popular in those days. Later Smart grew out to a stand-alone agile approach that particularly combines well with other more basis agile approaches such as Scrum and extreme programming, but also aligns well with PRINCE2.

image[7]

More specifically Smart puts accents on aspects in project that are only marginally highlighted in other agile approaches. As an example Smart adds the following accelerators to projects:

Preliminary iterations. Preliminary iteration types Propose and Scope is where work items are added to the backlog such as a modeling business processes, modeling smart use cases, creating an estimate, creating a cost-benefit analysis, writing a project plan (not that obvious it seems), first-cut software architecture set-up, development environment set-up, project kick-off, team education and first-cut user interface guidelines.
Smart use cases. Next to the more “non-functional” work items mentioned above, Smart introduces the well-known smart use cases requirements technique. Smart use cases not only model the needs of the user, but also identify workflow and reusable back-end services.
This technique allows for identifying re-use, repeatable and reliable estimation, and very good traceability between requirements, architecture and code, great functional testing capabilities. An absolute must in enterprise projects.

ManageSite[2]

Estimates. Estimation of size and complexity of a project is estimated and measured in smart use case points, defined on a scale from 1 to 10. Smart use case points of course come from estimating the size or complexity of smart use cases, and is a highly pragmatic and repeatable technique..
Work item life cycle. Where Scrum’s work item life cycle contains to-do, working and done, in Smart, work items are delivered through a slightly more structured life cycle, that allows for all roles in enterprise projects to be maximally involved in a collaborative manner.
Due to the life cycle, work items need not be broken down in tasks, a cumbersome and highly unstructured elements of Scum. In Smart work items simply move through the stages of the life cycle (in most cases in one or two days), the progress of which can be shown easily on a dashboard.
Designing. During the daily stand-up meeting, it is decided whether or not a design session is required. This time-boxed design session addresses the design or a single smart use case. Most project will organize this design session directly after the daily stand-up. On a typical time scale, if the stand-up runs from 10:00 to 10:15, the design session runs from 10:15-11:00.
During such sessions all roles that need to deliver input to this particular use case participate. Of course also the developer and tester are present, so all roles create a homogeneous view of the smart use case at hand.

IMG_0247[4]

Smart testing. Working with user stories will normally not result in a very efficient test approach, as user stories do not add structure to the project. Smart describes a standardized approach on functional testing in agile projects of course based on smart use cases.
Stabilizing iteration. Usually the last iteration of a Smart project or the last one to an upcoming release is used to round off remaining work items, build issues, bugs, and remaining features. This iteration is merely used to stabilize the solution. In Smart this iteration type if referred to as Finalize.

Blending Scrum and Smart

Besides defining an obvious project life cycle, these best practices from Smart blend in really well with the more lightweight approach of Scrum and extreme programming. This leads to the following best practices:

Initial backlog. The initial backlog of the project is filled with the work items Smart recommends, rather than assuming a user story filled backlog. When prioritizing work items for the first sprint in a mixed Scrum and Smart project, it is very likely that these are picked.

As a result, after one or two iterations, a project plan for the remainder of the project is produced, containing at least an estimate, based on the smart use cases that are also defined during these iterations (of course without any analysis or design, just the diagrams, the ovals and the actors).
Iteration kick-off. After the initial work-items are cleared from the backlog, the smart use cases become the primary work item type in the project. During sprint kick-off a number of smart use cases is picked to implement in the next sprint.
Dashboard. The traditional Scrum task board is replaced by Smart’s agile dashboard, containing columns for the stages in the work item life cycle – In Iteration, Designing, Working, Testing, Rework, Accepted. If appropriate additional columns can be added, e.g. when testing is split up in Developer Testing and Customer Testing.

IMAGE_041[7]

Points. The progress in the project is not measured in story points, but rather in smart use case points.
Scrum meeting. The focus of the daily scrum meeting shifts a bit as it is also used to answer the question: what still needs to be done on the smart use case(s) we’re currently working on to get them accepted?
Design. Host the design sessions as Smart recommends, preferably right after the daily scrum meeting.
Testing. As the work item life cycle clearly stipulates, functional testing, often done manual is all in a days work. The role of the testers here becomes imminent in the project.
Finalize. Projects have to decide when to apply a Finalize iteration to stabilize the solution and round up loose ends. Depending on the type of project, these iteration can be the very last sprint, but also can be very well aligned with the organizations release calendar.

This approach of mixing best practices, techniques and tools from Scrum, Smart over the past 12 years has proven to be very successful in a variety of projects and project types, including larger Java and .NET projects. service oriented projects, business intelligence projects, and even packaged implementations.

Moreover, when augmented with more technical best practices from extreme programming, such as refactoring, continuous integration, pair programming, and test driven development (smart use cases are a very good unit for unit testing code) projects are aligned to deliver on-time, on-budget, and with just-enough documentation and fitting (reusable) features. As former US president Bush used to say: failure is not an option.