Monthly Archives: June 2011

How to Define a Software Development Process

Process, ,

Your software development process should be contextual, likable, forceful, and readable, and you should get it this way by iterating over it.

Defining a development process is hard

So, you want to define a software development process for your organization. Good luck. If it isn’t hard work, then you’re probably not doing it right, because you ought to take many factors into account, and those factors tend to be very organization-specific. Although you can gather lots of great ideas from books and process gurus, your ideal process will be so influenced by environmental factors that no text or outsider can tell you what it should be. At the end of the day, all of the tough calls will be on you. Also, since the development process will affect every project and stakeholder in your organization, it’s a high-stakes proposition, and something you probably can’t afford to screw up.

What should the process cover?

Actually, figuring out what kinds of things the process should govern / specify is an essential part of the process definition. (No free lunch here.) It’s not just a matter of figuring out how to answer a series of questions in an organization-specific way, which would be hard enough. You also have to figure out which questions are worth answering in the context of your organization and its goals. A larger development shop may dicker productively over the standard set of documents that will be dissected in design reviews. A one-horse development shop probably doesn’t have the same concern, as there’s nobody to do such reviews.

So, what kinds of things might a development process address? The following rough categories would make for at least a decent start:

  • Activities that will happen: The “coding” part is more or less a given, but are you going to do ROI analyses, design reviews, independent testing, user acceptance testing, management approvals, formal training, etc?
  • Order in which activities will happen: Are you going to be more waterfall-ish, with code mostly following an requirements gathering and design, or are you going to be more agile-ish, and allow your code to grow up alongside your requirements and design?
  • Conditions on activities:  What circumstances require or preclude particular activities? What approvals or phase gates govern movement from one activity to another?
  • Deliverables produced: What kinds of products do you need to create in addition to the software? Design documents? Manuals?
  • Enforcement mechanisms: How will you ensure that the requirements of the process are being respected? Will you do process reviews? Audits?

Characteristics of a good process

Consonant with the above, it’s hard to offer much advice on what your process should look like, because your process needs to be tailored to its context. However, here are some meta-level suggestions about software process definition in general:

1) Software practitioners (engineers, project managers, etc) should like the process. They don’t need to take it with them to Disney World, but they shouldn’t be bristling and gritting their teeth as they struggle under its yoke. If practitioners don’t like the process, it won’t be used as intended, and will hence be a failure. Even if a disliked process is mandatory, people will figure out ways to achieve superficial compliance, even though they are really doing things some other way. Accordingly, the process should be designed mostly by the practitioners themselves. This will presumably lead to a process that the practitioners think is valuable, and will greatly increase the likelihood of broad buy-in. Management can and should levy requirements on the process, but the practitioners should be allowed to satisfy those requirements how they see fit.

2) The process must be forceful, which is to say that it must contain mostly shalls – things people have to do (if they want to avoid a beatin’), as opposed to shoulds – things that are recommended. When a group is trying to define a process, members will often weigh in with suggestions, guidelines, and best practices that represent their views on how software engineering should be done. This may seem useful at first, but any group of experienced professionals can say a great deal about how things should be done. Before long, you’ll find yourself writing a software engineering book, filled with a big pile of shoulds that individuals will happily ignore whenever timelines get tight or when they happen to disagree with the recommendation. The payoff of this book will be too small to justify the time it takes to write it. Instead, your process should focus on what is mandatory / required, even if it is only required in certain circumstances. There is value in a big pile of shoulds, but it belongs in knowledge sharing activities (lunch and learn stuff) as opposed to a process definition.

3) The process must be extremely concise (as minimal as possible) and readable. A process that expounds on its topics in an academic and/or lengthy manner will go unread, and a process that goes unread is a process soon to be dead.

4) The process should be developed iteratively, because iteration works for everything. Don’t wait until you have the whole thing nailed down before you deliver something. Rather, start piloting elements of the nascent process as soon as possible, so that you can gain some real-world experience and adjust accordingly. Particularly if your process levies new requirements on people, you’ll want to see where people are tempted to cut corners, and if they still agree in the heat of battle to the same things they agreed to on the whiteboard.

If the process you develop is sensitive to its context, respected by the people who will use it, slanted toward mandatory practices, and right to the point, it will likely be a success. Further, you are more likely to define such a process by iterating over it early and often.

Shared Technology and Ease of Integration

Architecture, Code, Design, , , , , ,

A shared set of technologies implies very little about how easy it will be to integrate two products.

In a past organization (of course, tales of foolishness are never explicitly set in one’s present organization), we were in the midst of a multi-year effort to build THE SYSTEM, which would replace 639 other systems, save a zillion dollars, unify the thirteen colonies, and restore balance to the Force.

In spite of (or maybe as a consequence of) its inestimable virtues, THE SYSTEM (hereafter “TS”) was taking a long honking time to build. As fate would have it, my corner of the organization had a semi-urgent need for a planned part of TS, but that part was not scheduled to be delivered for another two years. Unable to persuade the makers of TS to accelerate development of the piece we needed, multiple persons in my organization became afflicted with the idea that my development group could assume responsibility for building that one chunk of TS ourselves, on a timeline more congenial to our needs.

One person sold the concept thusly – since TS was being developed in Java, if we were to write our version of the sorely needed piece in Java, then said piece could easily be grafted onto the rest of TS when the time came. This strategy would allow us to get what we needed sooner, and would save the makers of TS time, because we would have implemented a significant feature for them, and all they would need to do is integrate it. They key to all of this was that we would build on the same platform (Java), so as to make integration of the two efforts relatively easy to achieve.

That’s a lot like saying that two documents can be easily merged together because both are written in English .

The truth is that a shared technical platform doesn’t buy you very much if you need to integrate two independently developed pieces of software. In fact, it may be a liability of sorts. If I’m trying to incorporate the capabilities of a Ruby application into a C# application, everybody recognizes that I’m going to have to rewrite the Ruby stuff. However, if I’m trying to incorporate one Java application into another, independently developed application, I’m still going to have to rewrite lots of stuff, and the shared platform may actually hurt me insofar as it leads people to assume otherwise.

Like two documents, two independently developed applications may share a common language, but will inevitably differ in goals, style, assumptions, and overall organization. These differences will not be abstracted and isolated in particular places, but will be suffused throughout the code of each in ways that are difficult to systematically identify, let alone tease out.

Although the object-orientation of a language like Java does go some way toward making an application more like a set of modular and reusable components, this capacity is often over-emphasized. The promise of a set of objects that faithfully model the real world and hence transcend problem-specific solutions is usually illusory. There is often no “real world” to be seen outside of some contextual problem space, as the problem space itself heavily colors designers’ perceptions of what the world to be modeled looks like. Designers solving different problems within what is in principle the same domain will inevitably see the domain in terms of the problem, and hence will model different worlds. As such, when it comes time to integrate their code, there will be no conceptual lingua franca to complement the syntax of their shared implementation language, making it virtually impossible to combine the two codebases without either substantially rewriting one or creating some huge integration layer that exists mainly to bridge the conceptual divide.

A shared language doesn’t get you very far, especially not in a world where most technologies can now talk to each other through XML. Even if you start with a shared development language, differences in goals, styles, assumptions and organization will mitigate strongly against the ability to integrate any two codebases without significant rework, unless those codebases were intentionally designed to be integrated  in the first place.

Iteration Works for Everything

Improving, Process,

Everybody knows the benefits of iteration for developing software, but iteration should be practiced for developing just about everything.

If you’re working in software development, professional literacy / competence demands some acquaintace with iterative development techniques. From intern to architect, it has been drilled into us all – iterate, iterate, iterate. Most of us can and do carry the tune – we deride the linear ways of waterfall (at least explicitly), and preach the gospel of iteration.

So, why do we iterate? I think these are the best reasons:

  • People often don’t know what they want – It takes seeing some kind of semi-realistic product before people realize what they do and don’t want from a system. Iterative development gives people slices of functionality that they can test-drive, thereby enabling the feedback needed for future iterations.
  • People often miscommunicate – Even when they know what they want, people aren’t always so good at expressing it. Further, even if well-expressed, a requirement might be easily misunderstood. Iteration allows people to look at something in progress and say very simply and specifically what is right or wrong about it.
  • Requirements change  – Targets have a way of moving, even the ones that have already been hit. Iteration gives you a way to continually re-validate requirements and to keep pace with fluid demands.

So, we all know the above, and countenance these truths in our development practices. Yet, for some odd reason, most of us reserve the valuable practice of iteration for creating code. When we need to produce user guides, proposals, standards, COTS evaluations, slide decks, and any other non-software deliverables, we revert to the dark ages of engineering, hoarding our labors and polishing them at length, waiting for the right moment at which to unleash them upon the world.

Your non-software deliverables are just as susceptible to miscommunication, moving targets, and imperfect requirements as your code is. As such, you should iterate over those as well. Don’t wait until you have a complete and thorough document / presentation / proposal before throwing your cards on the table. Instead, get something (e.g., a table of contents, outline, or synopsis) out to other stakeholders as soon as you can, and let the magic of iteration help you to make these products as good as they can be.

Iteration works for everything.