"Do you do agile?" is a frequently-asked question from people who approach us to develop software for them. The answer is - "Yes, but."

If you squint, you can just make out a waterfall in this picture...

If there's one certainty in software development, it's that nothing has really changed in the 50-odd years of the discipline.

The industry shuffles slowly from one extreme to another. Top-down rigour emphasises planning, spreadsheets, and gantt charts. Critical path analysis favours large teams plodding towards a goal. And then there’s laissez-faire, fly-by-the-seat-of-your-pants style development which favours individual heroics and legendary hackery.

But as those seasoned pros among us know, these extremes are straw men. Every project that involves software development - and frankly, just about every phase of a larger project - lies somewhere on the continuum between the highly controlled waterfall and the loosely structured free-for-all.

To both employees and to customers I emphasise the "software engineering funnel."

Every project starts life loosely structured and more freewheeling. This is by necessity. As we explore the problem - without bogging ourselves down with process and detailed planning - patterns emerge more readily. Similarly, as you near hard delivery deadlines, every project needs to ramp up discipline and procedures as other parts of the business prepare for products to launch, and for marketing to move into high gear.

The vast majority of customers have limited budgets and strict timescales, which will necessarily mean that you'll need to make commitments to the questions, “When will it be ready?” and, “How much will it cost?”. It's rare indeed to find a customer with such patience and deep pockets that they are willing to accept the answer, "We don't know" to either of those questions, let alone both.

What's the primary driver behind discipline needing to ramp up as projects progress?

The answer lies in the complexity of software and how it is produced. At inception there is nothing, and therefore both no chance of regression and no chance of bugs. But as the code produced increases, the chances of each successive line of code causing problems in others goes up. In poorly structured code, it could go up exponentially.

It's also worth pointing out that different types of components afford different development approaches.

The funnel approach works where you have little idea of what you're producing, perhaps on a new platform or in an unfamiliar environment. But for common development patterns, a test-driven approach might be most appropriate. For very clearly, well-defined components that need to meet an external specification or standard, a waterfall approach might be best.

Be flexible, pragmatic, and willing to experiment with different methodologies rather than being particularly attached to one, and you'll come to realise that each approach has it's strengths and weaknesses.

So it's very interesting, 16 years on, to go back and re-read the "Agile Manifesto" with this in mind. There's nothing in that manifesto that suggests there's anything wrong with this more flexible approach, and it's particularly instructive to compare it to how people currently think of agile.

Agile was formulated primarily as a corrective to the overly-prescriptive software manufacturing methodologies that tried to treat engineers as replaceable widgets in a process. Instead Agile puts people at the centre, valuing participation by both customers and developers, while acknowledging that in software it's rare to know up front what you actually want built.

It's pretty scary how such a simple and unarguably worthy manifesto has been translated into armies of highly paid consultants, complex methodologies and scary process diagrams.

Wherever you sit on the continuum in your projects, I'd encourage you to cut through the fluff and read what Agile is really about - which is much more of a mentality than a list of specifics. Maybe even give it to your local agile evangelist and see what they make of it!

You can re-read the agile manifesto here.