A recipe for enterprise agile. Mixing Scrum and Smart

To cut to the chase, those of you who have worked on enterprise or service oriented projects before already know this. These types of projects are characterized by a large number of organizational, functional and technically complicating factors. Enterprise software development projects are surrounded by a large number of complicating characteristics and challenges:

• Many different stakeholders. Projects have many different parties in the organization that are involved. Often such project realize goals for different departments or divisions in the organization. As an example I once coached a project that served 22 different departments. Quite often these departments or division try to achieve overlapping or even contradicting goals.

• Business processes. In general enterprise projects (re-)implement (parts of) business processes that are key to the organizations success, rather than building a straightforward web application.
• Complex software architectures. Most likely, enterprise projects implement business processes in a complex IT landscape or architectures. Think of landscapes that host many different applications all tied together.

• Many external dependencies. Often, such complex IT landscapes contain components that are outside of the organization. Services that run outside the corporate firewall, with other parties. A well known issue to agile projects is that of availability. We need the new version of the service in this iteration, but the other party runs a waterfall project that will not deliver this year.
• Changing requirements. In enterprise projects, requirements change continuously. Either by changing insights during the project, but more often due to changing legislation or regulations.
• Many different types of deliverables. Traditionally these projects count many different types of deliverables. Think of service description, user interface, process models, non-functional requirements, workflow, project plans, test plans and many, many others.
• Multiple roles. Large organizations count many different roles that are involved in projects. There are business analysts, enterprise architects, information analysts, testers, release managers, SAP developers, web developers, SAP functional consultants,

Thus these project often become a mission impossible, and fail in large numbers. Such projects are difficult to estimate, therefore hard to plan. For developers it’s tough to build the software, often with many new techniques and technologies, and even worse for testers, these projects are sheer Impossible to test well. This is where a structured agile approach should come to the rescue.

Being agile in enterprise projects

Independent of the approach that is applied, or the agile process that is followed, being successful in agile projects follows a number of key principles:

• Multi-disciplinary teams. Instead of different roles such as analysts, designers, developer and testers working in consecutive stages in a project, all roles collaborate in implementation individual work items from heads to tales.
• Short iterations. Project work in short time boxes, in most cases 2, 3 or 4 weeks. During such an iteration, a number of work items is planned, completed and evaluated – with different terminology in use with different agile processes.
• A small standardized unit of work. Projects requires a small unit of work, to be able to deliver a number of them in a single iteration. Large work items such as traditional use cases therefore do not apply well.
• Testing from day one. Testing is an important discipline during agile projects. Testers are involved from the very first iteration, when the first work items are realized. Note that testing in most projects goes way beyond the obvious unit testing – which is actually a developer technique.
• Continuous measurement. Successful agile projects continuously measure their progress. Since they are always delivering individual work items, measurement in agile projects is easy.
• Continuous (process) improvement. At the end of each iteration not only the realized work items are evaluated, but also the project approach itself is evaluated, leading to highly effective project approaches.

Scrum in a nutshell

When organizations first start with agile projects, Scrum is often the process of choice. At this point in time, Scrum is by far the most popular and best known agile approach. Scrum is a straightforward lightweight agile process that offers a collection of highly applicable principles and techniques for short iterative projects and is easy to understand. It also specifies a remarkable terminology with scrums, sprints and user stories.

In a nutshell, Scrum is characterized as follows:

• Sprints. Projects are divided into iterations, called sprints. In general these are 2 to 4 weeks.
• Backlogs. Work items still to be done during a project reside in a so called project backlog. At the start of each sprint items from the backlog are placed in the sprint backlog. These are the worked items to be realized.
• Sprint cycle. All iterations follow the same simple process, with an iteration kick-off, actual work and a retrospective workshop for evaluation.
• User stories. The main unit of work is usually the user story, although this is not mandatory. This rather informal requirements technique is shared with the agile process extreme programming.
• Planning. In general the project and sprint planning is visualized in task boards, which usually is put up on a wall using post-its. Progress is monitored by using burn down charts, a straightforward diagram that on a daily basis displays the number of points still to realize. A trend line in the burn down chart extrapolated from the points identifies the likely project end date. Again this technique is shared with a number of other agile processes, including Smart and extreme programming.
• Delivery. Every iteration or sprint results in fully delivered products, software or otherwise.

Scrum has proven to be a successful approach in many (smaller scale) web projects. In larger, more complex, service or cloud oriented and enterprise projects, I think Scrum needs to be augmented to fit the project and its environment. Let me explain.

Beyond lightweight agile

When applying Scrum or other lightweight agile processes such as Crystal Clear or the set of lean principles (applied to software development that is) to more complex projects of all sorts, you might recognize a number of shortcomings:

• Business goals. It is often unclear how the work items in such projects (in most cases user stories) link back to business goals, stakeholders and business processes to implement.
• Analysis and design. Although many people in the so-called agile community will now think I’m promoting a waterfall approach, in most agile approaches there are hardly any analysis or design activities or deliverables defined.

  For instance, it is undefined how the backlog in a Scrum project is assembled. Let me assure you, in more complex projects some analysis and design is required, albeit not in big-upfront-design mode.

• Architecture. The same argument goes for software architecture. Most agile approaches lack decent software architectural activities and deliverables. Architecture is often assumed as-is. If you’re implementing business processes that run through a systems landscape of over 40 collaborating systems, you will definitively need to design your architectures, but again this need not be in big-upfront-architecture mode.
• Roles. Scrum and extreme programming describe a very limited set of roles that, although looking slick,  in most projects does not adhere to the standing organization. There’s often enterprise architects, information analysts, business analysts, middleware developers, application management, testers, release manager. How will a project work with all those roles? There’s a bit more to it than just saying they are allowed to participate in the daily scrum.
• Testing. Apart from the obvious (developer) techniques such as unit testing, the role of testing more specifically – which is essential in larger agile projects – is not very well described in lightweight agile approaches.
• User stories. Although very flexible, user stories are a highly unstructured unit of work, despite efforts in epics or story maps. Applying such an unstructured unit of work may result in low re-use, non-repeatable estimation, difficult to standardize development, testing and measurement beyond the individual projects. In a multi-project environment user stories will block progress and reuse (of for instance metrics and code).
• Delivery cycle. Scrum does not specify the organization of delivery of the products to different environments such as test, acceptance and (pre)production (so called OTAP).

When executing large, more complex, service oriented or enterprise projects, I therefore strongly recommends to overcome these limitations by augmenting Scrum and the likes with additional techniques.

Smart accelerators

In my personal experience in enterprise projects I have had very good experiences with mixing the agile process Scrum with the best practices from another agile processes called Smart to overcome the drawbacks of the aforementioned approach.

Smart originated from the Netherlands in 1998 and was originally designed to implement best practices for DSDM, an iterative approach that was popular in those days. Later Smart grew out to a stand-alone agile approach that particularly combines well with other more basis agile approaches such as Scrum and extreme programming, but also aligns well with PRINCE2.

 

More specifically Smart puts accents on aspects in project that are only marginally highlighted in other agile approaches. As an example Smart adds the following accelerators to projects:

• Preliminary iterations. Preliminary iteration types Propose and Scope is where work items are added to the backlog such as a modeling business processes, modeling smart use cases, creating an estimate, creating a cost-benefit analysis, writing a project plan (not that obvious it seems), first-cut software architecture set-up, development environment set-up, project kick-off, team education and first-cut user interface guidelines.
• Smart use cases. Next to the more “non-functional” work items mentioned above, Smart introduces the well-known smart use cases requirements technique. Smart use cases not only model the needs of the user, but also identify workflow and reusable back-end services.

  This technique allows for identifying re-use, repeatable and reliable estimation, and very good traceability between requirements, architecture and code, great functional testing capabilities. An absolute must in enterprise projects.

• Estimates. Estimation of size and complexity of a project is estimated and measured in smart use case points, defined on a scale from 1 to 10. Smart use case points of course come from estimating the size or complexity of smart use cases, and is a highly pragmatic and repeatable technique..
• Work item life cycle. Where Scrum’s work item life cycle contains to-do, working and done, in Smart, work items are delivered through a slightly more structured life cycle, that allows for all roles in enterprise projects to be maximally involved in a collaborative manner.

  Due to the life cycle, work items need not be broken down in tasks, a cumbersome and highly unstructured elements of Scum. In Smart work items simply move through the stages of the life cycle (in most cases in one or two days), the progress of which can be shown easily on a dashboard.

• Designing. During the daily stand-up meeting, it is decided whether or not a design session is required. This time-boxed design session addresses the design or a single smart use case. Most project will organize this design session directly after the daily stand-up. On a typical time scale, if the stand-up runs from 10:00 to 10:15, the design session runs from 10:15-11:00.

  During such sessions all roles that need to deliver input to this particular use case participate. Of course also the developer and tester are present, so all roles create a homogeneous view of the smart use case at hand.

• Smart testing. Working with user stories will normally not result in a very efficient test approach, as user stories do not add structure to the project. Smart describes a standardized approach on functional testing in agile projects of course based on smart use cases.
• Stabilizing iteration. Usually the last iteration of a Smart project or the last one to an upcoming release is used to round off remaining work items, build issues, bugs, and remaining features. This iteration is merely used to stabilize the solution. In Smart this iteration type if referred to as Finalize.

Blending Scrum and Smart

Besides defining an obvious project life cycle, these best practices from Smart blend in really well with the more lightweight approach of Scrum and extreme programming. This leads to the following best practices:

• Initial backlog. The initial backlog of the project is filled with the work items Smart recommends, rather than assuming a user story filled backlog. When prioritizing work items for the first sprint in a mixed Scrum and Smart project, it is very likely that these are picked.

  As a result, after one or two iterations, a project plan for the remainder of the project is produced, containing at least an estimate, based on the smart use cases that are also defined during these iterations (of course without any analysis or design, just the diagrams, the ovals and the actors).

• Iteration kick-off. After the initial work-items are cleared from the backlog, the smart use cases become the primary work item type in the project. During sprint kick-off a number of smart use cases is picked to implement in the next sprint.
• Dashboard. The traditional Scrum task board is replaced by Smart’s agile dashboard, containing columns for the stages in the work item life cycle – In Iteration, Designing, Working, Testing, Rework, Accepted. If appropriate additional columns can be added, e.g. when testing is split up in Developer Testing and Customer Testing.

• Points. The progress in the project is not measured in story points, but rather in smart use case points.
• Scrum meeting. The focus of the daily scrum meeting shifts a bit as it is also used to answer the question: what still needs to be done on the smart use case(s) we’re currently working on to get them accepted?
• Design. Host the design sessions as Smart recommends, preferably right after the daily scrum meeting. 
• Testing. As the work item life cycle clearly stipulates, functional testing, often done manual is all in a days work. The role of the testers here becomes imminent in the project.
• Finalize. Projects have to decide when to apply a Finalize iteration to stabilize the solution and round up loose ends. Depending on the type of project, these iteration can be the very last sprint, but also can be very well aligned with the organizations release calendar.

This approach of mixing best practices, techniques and tools from Scrum, Smart over the past 12 years has proven to be very successful in a variety of projects and project types, including larger Java and .NET projects. service oriented projects, business intelligence projects, and even packaged implementations.

Moreover, when augmented with more technical best practices from extreme programming, such as refactoring, continuous integration, pair programming, and test driven development (smart use cases are a very good unit for unit testing code) projects are aligned to deliver on-time, on-budget, and with just-enough documentation and fitting (reusable) features. As former US president Bush used to say: failure is not an option.