Small stories, legacy code, and scaffolding

Details

Created: 22 March 2013

Working with legacy code, code that is older than many of the developers working on it, can be a real challenge. I worked with some code that was written decades ago, when best practices for software design were less established. It had huge modules, with no thought to modular design and the Single Responsibility Principle.

This had a bad effect on testing, since it was not easy to build unit tests so that individual modules could be tested when they changed, as is popular with Continuous Integration.

But even trying to break stories down into sprint-sized pieces was difficult. Good Agile practice requires implementing a change in such a way that it can be demonstrated at the end of the sprint. When you have a complicated process that is all taking place in one or two huge modules, this can be really difficult. If the whole change is going to take several sprints to complete, then the amount of it that you can accomplish in a single sprint may be just enough to ensure that the product does not work at all.

There are some ways around this. The first is to add temporary messages to the product. For example, one product I worked on had a parameter library that the customer could use to customize the operation of the program. Each new parameter needed changes to a parser, several control blocks, various screens, and an editor. And all this was before we got to the module that actually used the new parameter. So how do you demonstrate that the new parameter editing is working, when you do not yet have the code that consumes the parameter? You can add a temporary message that displays the value of the parameter. Then in the sprint review, you can demonstrate that if you edit the parameter, its value changes.

But if you ship the new code with the product, you do not want the customer to see the message. You can handle this with special debug flags. There are various ways to do this. If you have a parameter library, like the product I mentioned, you can have special parameters that set debug flags. In systems like MVS (IBM mainframe), where you have control statements that map external file names to the internal names the program uses (DD statements), you can have special dummy file names that, when specified, act as flags.

Once you have a way to specify debugging flags, you can turn your temporary message on and off. And you can even change the control flow of the program. Perhaps the change to the format of a parameter causes the rest of the program (the part not yet changed) to fail. Your debug flag can make the program display the message showing the new value, then exit the program. If the flag is not set, the program processes the parameter the old way.

My experience with teams have been that they usually do not like the idea of this kind of scaffolding, because they feel it is a waste of time to put code in during one sprint, and take it out in the next. These concerns can be reduced by setting up an easy-to-use infrastructure for debugging scaffolding. If you have a way to specify debugging parameters, you can have a module that can be called anywhere in the program to find out if a particular debug flag is set. Similarly, if you have a good message infrastructure, it is easy to issue debugging messages. Unlike messages and parameters that are for the customer, these debugging aids do not have to go through the normal approvals by technical writing, support, marketing, etc., so there is not a lot of work that is undone and wasted when you remove them.

The other complaint I hear sometimes is that people question what is wrong with just leaving the story open and in-progress for several sprints. They see it as merely a bookkeeping issue. But it is more than that. A large change is going to involve a number of modules. If these are apart on the workbench for several sprints, this gets in the way of changes other team members may need to make to them.

Even more important is the issue of agility: one of the benefits of Agile development is that the team can turn on a dime when requirements or priorities change. If something of a higher priority than your multi-sprint story comes along, and you have the change half-implemented in a bunch of modules, you have to stash your half-modified code somewhere, work on the new story, and later try to resynchronize your saved copies of the modules with the ones that were changed by the new story. That is the kind of busywork that nobody needs.

Scaffolding like this can make it a lot easier to break large changes down into manageable pieces. The toughest part is getting your mindset right, and realizing that even if you are putting in a little code, only to take it back out the next sprint, the overall benefits are worth it.

Update, 2015-08-03: Here is a good article by Jim Bird about some of the hazards you have to be aware of when using feature flags and scaffolding: https://dzone.com/articles/feature-toggles-are-one-worst

Plan to throw one away

Details

Created: 10 August 2013

In Fred Brooks's classic book, The Mythical Man-Month: Essays on Software Engineering, which was first published in 1975, he suggested planning to throw the first implementation of a system away. His words: "The management question, therefore, is not whether to build a pilot system and throw it away. You will do that. The only question is whether to plan in advance to build a throwaway, or to promise to deliver the throwaway to customers." and "Hence plan to throw one away; you will, anyhow."

Brooks's concept was that you (and your customer) don't really understand the problem, let alone the solution, until you try to implement the solution. Once you have done that, you have probably learned enough about the problem to develop a workable solution.

Agile development, with its emphasis on stepwise refinement and frequent customer feedback, has the potential to eliminate the need to throw the first one away. By implementing a bit at a time, and letting the customer try it and give you feedback, you are a lot less likely to get to the end of the project and find out that you have successfully implemented the wrong thing.

But what about implementing Agile development itself?

My first introduction to Agile development was at a company that had been using waterfall methods for 20 or 30 years. Management heard about all the benefits that other companies were getting from Agile development, and wanted to start using it.

With a suite of 20 products, developers were grouped into teams, with members chosen because they worked on related products. We ended up with about 10 teams, some of which had a single product, and some of which had three or four. One person from each team volunteered to be a ScrumMaster, and the ScrumMasters were all sent to a two-day ScrumMaster class. One employee who was familiar with Scrum served as an Agile coach for all the teams.

The Agile rollout was a disaster. Management did not understand Scrum, so they would issue mandates that contradicted what the ScrumMasters had been taught about Scrum. There were all sorts of roadblocks, like legacy, non-modular code that made story breakdown difficult, and a build system that lead to blockages that made it difficult to meet Sprint promises. Management would add work to the team during a Sprint. The fact that the Product Owners were also the supervisors of the team, as well as writing code, made the lines of responsibility and authority very blurry and confusing. All this made it very difficult for the teams to accept Scrum as anything other than a management fad-of-the-day that kept them from doing their jobs.

Having read a lot about Agile development and Scrum, I had a lot of unkind thoughts about our management, and could not understand why they were doing things so stupidly.

I had this fantasy about how a company should roll out Scrum: A pilot team develops something using Scrum, and they are so much more efficient, and have so much more customer satisfaction, that all the other developers say "Wow! I want a piece of that." In my fantasy, this would work much better, because all the developers would be on board and excited about Scrum, not just doing it because it was mandated by management.

A few years later, the company looked at how poorly Agile development was working, and decided to do a restart. They assigned dedicated ScrumMasters and Product Owners, rather than ScrumMaster-coders and Product Owner-coders, they got people further up the management chain that really understood Agile and Scrum, they retrained all the ScrumMasters and Product Owners, they assigned Agile coaches, and they put an emphasis on doing Agile and Scrum right. Their efforts are still a work in progress, but they show signs of working much better this time.

I did a lot of thinking about the initial rollout, my Agile fantasies, and how the company could have done things better, and I am not sure they could have. In a large corporation, a team is not going to be given the leeway to go off and try a new development methodology--It's just not the way the company does things, and any manager who decides to go off in a different direction is likely to be putting his job on the line. And there are enough obstacles in an entrenched corporate culture that there is no guarantee that a team trying to do Agile development in that hostile environment can actually succeed. So my Agile fantasy is just that.

But how should the company have rolled out Agile development and Scrum? That's what I don't have a good answer for. In order to manage Agile development well, you have to understand it, and that is not something you can do overnight. Worse yet, the whole idea of self-organizing teams leaves managers who have spent their entire careers in a command-and-control environment feeling like all the tools they use to do their jobs have been taken away from them. Short of replacing all the lower-level management with people who understand Agile development, I'm not sure there is a good way for management to learn how to manage it without trying and failing a few times.

So I no longer judge the company's management as harshly as I once did. I now think that any introduction of Agile development (or any new development methodology) in a mature company is going to be ugly. I think all that management can do is to do their best, try to get themselves and their troops as much education about Agile development as they can, and be prepared to do a restart after things fall apart.

The down side to this is that it gives the developers a lot of negative feelings about Agile development, and this negativity is the emotional equivalent of technical debt, which slows down development (not to mention messing up morale) until it is payed down.

But I really don't have a better idea of how to do this and, based on what I have seen in the literature, I'm not convinced that anyone else does, either.

Update 2013-10-09

I've received a couple of comments on this post that got ended up in the wrong place, so I am appending them here:

On 2013-08-04, Norbert Paul said:

Hi Lynn,

Just read your post and like to share some thoughts. (I work since longtime in SAP-waterfall-projects, and just started recently with thinking about agile).

I generally agree with your realisitic (not to say disillusionated) view about the likelihood of a smooth shift to agile.

From a theoretical point for me this seems logical:

Agile development has to be implemented in an agile fashion. Not all at once, but with pilots that create so much resonance that others want to adopt, just like you described in your fantasy.

Another Approach could be to say: Plan to throw away the first attempt to implement it, that means be prepared to fail. Or better to say: use the first attempt to learn about the dynamics of the organisation. Then try again going with the organisation. I think every method must fit to the organisation and take into account where the organisation stands. The implementation of Agile is another project that requires all change management efforts, that you need for other implementations, and probably even more, because it is going to profoundly change values and behaviors.

Kind Regards,

Norbert

 

On 2013-08-11, Jay Conne said:

Introduce Agile to an enterprise top-down, bottom-up and middle-out…

Top-down have them understand the R&D nature of software development and see the necessity to focus on business value. Then share that value top to bottom.

Bottom-up, with a focus on their responsibility for independent thinking and commitment to hold their fellow team members to high standards. Also, a commitment to completing the highest priority story first and only work on the next one when they have no value to add to the prior one (s). And of course, one team – all skills in the room without titles, no my job vs your job.

Middle-out, with middle management understanding their changed role of identifying value and priority, not programming the programmers.