So you’ve read the literature and you’ve decided your team needs to move to a more agile and iterative process. Great. Good for you. Now what?
If you’re a small business, it can be relatively easy to paradigm shift so long as the senior management are on-board, and the team is willing to try new things. Your first job is to sell this change to everyone internally. Agile development is centered around self-driven teams. After you convince the team to give it a go, I recommend you start with either the sales team or the guys that complain the most about your current deliverables (for example customer support or systems engineering). In small businesses, it might be best to go directly to the customer and explain this improvement you are making to get more transparency and customer focus, even if it is an internal process change – your customers will be intrigued. Critical to your shift is “Iteration” and fully completing a task including test.
If you’re in a larger business, you likely have a set methodology you are already following and moving to Agile development, SCRUM, Kanban or ScrumBan is going to be a process of changing program management’s mindset to iterate and deal with uncertainty. You are not going to plan and estimate the entire project up-front but iterate towards a point where the product is defined. Manufacturing companies struggle with this most because they have fixed systems requiring resource planning beyond just human resources and if they are manufacturing several products already, they must plan for alpha, beta and general availability to coincide with inventory levels and possibly retooling factories. In this environment where your software team is an essential part of the creation (but not the entire product) you can still move to a SCRUM model but it has to adapt within the larger costs for the project.
Moreover, significant resources are going to be assigned continuously on software projects unlike most hardware development projects. A manufacturing company finishes the development of a new product and moves on to another product. Software has continuous develop/release cycles and requires dedicated resources as a core team, with supplemental resources during a major product revision.
Square peg, round hole?
As a software manager at a semiconductor company, I faced this problem of fitting software development SCRUM into the program management waterfall methodology which was evolved to limit mask set creation (effectively the “tooling” for a new chip in the same manufacturing technology) because those mask sets cost millions to create and sample and must be handed to a Fab which adheres to complex scheduling and inventory processes. At most the company wanted an Alpha and R1 maskset and ideally, the R1 maskset directly. The same was true for the evaluation boards created on which the new processor would be evaluated by customers. The chips, the software and evaluation boards needed to be created before certifications and general availability. It is easy to see in this environment why a waterfall methodology is being used. The company is working towards better pre-silicon testing and with better simulators and emulators may be poised for an Agile shift, but I didn’t want to wait for that to be reality.
So how do you fit a scrum software development framework into a program management driven organization insisting on waterfall methodology? Well, for runtime software, we prioritized the drivers to be written or modified based on the hardware IP block changes within the silicon and built our scrum board backlog using those priorities. We automated a duplication of drivers that did not require significant change, created a process that could scrape HW designs directly for things like registers, pin signals and clocking data so we could concentrate each sprint on iterating with the updates to the hardware designs. By sprinting this way, we had time to create the primary elements for simulator/emulator testing and ensure that alpha software was available by the time alpha hardware/board samples arrived. In later sprints, we added the examples and demos needed to show off the silicon’s capabilities. You could say we were doing a pseudo agile process, but I will tell you that iterating in this manner allowed us to reduce driver costs and support more silicon products being developed in parallel while achieving more alignment with the hardware team because they were engaged and gave focus to each sprint.
Scrum had even more impact when we decided on creating new development and configuration tools to support the silicon. We divided a relatively large tools team into multiple scrum teams each with a mandate to build tools capable of being cloud accessible, with versions for use on a desktop as well with data interchange between the online and desktop tools. These new scrum teams were smaller, but iterated quickly and included virtual scrum demos. These tools teams were producing a strictly software product not necessarily aligned to hardware schedules and could use all the mechanisms of Scrum and even some of the capabilities of Kanban. But we still needed to fit into the waterfall methodology of the Program Management team.
To fit the phase gates of the typical waterfall system (concept, feasibility, planning, development, test, release, post-release) we decided to map the concept phase into our storyboarding and initial backlog timeframe and we went through the approvals from Concept to move to the Feasibility phase. When entering Feasibility, in the past, software teams would review a completed Product Requirements Document (PRD) and do the planning and scheduling up-front, detailed designs and estimates were then locked into a plan of record and committed to schedule typically 9-18 months hence with a 3-6 month Alpha/Beta cycle. After working with our software program manager we developed a plan to do a series of sprints BEFORE we reached the phase gate for planning to development. This met some resistance in management at first, but we argued that by the time we’d iterated several sprints we’d know what we wanted and could try several interface and mechanisms with transparency through demos after each sprint.
One of the most difficult changes for the organization was the fact we built our “requirements” along with each backlog item. In essence the collection of the backlog items that made it into sprints, became the detailed plan and revised the requirements, and that iterative development of requirements was very difficult for our Program Management team. We built user personas and stories up-front during the concept phase as we were building our backlog items and we grouped backlog items into epics that in concept represented our requirements but even these evolved as we did our initial sprints where we involved more people in our sprint demos.
Be transparent
My weekly report invited EVERYONE in the company to join the sprint demos with advertised date/time information. Naturally, it would have been a problem if everyone had attended but the message was transparency and it worked. There were 30-50 people sometimes attending these Virtual Scrum Demos including management, sales, field application engineers (who gave customer perspective), applications engineers (who support customers) and even hardware engineers along with the entire scrum team, the scrum master and product owner. We often had three continents represented on these calls. These Virtual Scrum Demos were hugely popular because for the first time in the company, they were involved in the development process. By the time we had iterated 6-8 times, we had a very good prototype of what we wanted to release and had clarified the backlog items needed to be completed. We then took the remaining backlog items and prioritized them into remaining sprints, mapped those sprints to silicon and development board deliverables for release testing and went to the planning/development phase gate.
Not surprisingly, as we finished the remaining sprints we got less and less attendance at the Sprint Demos. We could tell when it was time to release by the attendance roll call we made and the requests from our sales field application engineers “Can I give it to customers? Can I use it? When will it support xyz silicon?”. It was very easy to know we would have a successful software release.
One area we were still addressing when I left the organization is that of testing. While developers in the scrum team would test their sprint work and do integration testing before each sprint, we did not have release testing integrated. In larger companies that are used to Waterfall methodologies, test engineering is a separate role. In our case, the test engineers for software were located half-way around the world from the developers. We were in the process of addressing that and adding them to the SCRUM team locally. They would join the scrum daily stand-ups, and work on test plans and automated testing in parallel to the development team. Our biggest challenge in evolving to scrum was test engineering frustration over us completely revamping the UI and functionality during early sprints. I think this is especially true of new product development and less for revisions to an existing product (since backwards compatibility is always an objective). Having local, integrated test engineers in the process should make a world of difference.
To conclude, we adapted Scrum software development to both the runtime software and the software tools creation process, but fit this agile development into a program management driven waterfall model and it has been very successful. Was it pure Scrum? No. Did it adhere to the principles of Scrum? Yes. More importantly to me, it gave the team transparency within the company which made the software much better for our customers.
itsjustsoftware is my blog at blog.hemstreetsoftware.com. Please comment on this or any post I would love your input. Also, if you haven’t already please subscribe to this blog on the left panel. If you are going through software development changes on your team, let me help you. Use the Contact action on hemstreetsoftware.com to get in touch.