Thursday, October 7, 2010

Agile is Not a Dirty Word

While I was writing Brown Noise in Written Language, Part 2, twice I came across the word “agile.” First, the word “agility” was in the original sentence that I was criticizing. Joel Garry picked up on it and described it as “a code word for ‘sloppy programming.’” Second, if you read my final paragraph, you might have noticed that I used the term “waterfall” to describe one method for producing bad writing. Waterfall is a reliable method for producing bad computer software too, in my experience, and for exactly the same reason. Whenever I disparage “waterfall,” I’m usually thinking fondly of “agile,” which I consider to be “waterfall’s” opposite. I was thinking fondly of “agile,” then, when I wrote that paragraph, which put me at odds with Joel’s disparaging description of the word. Such conflict usually motivates me to write something.

In my career, I’ve almost always had one foot in each of two separate worlds. These days, one foot is in the Oracle world. There, I have all my old buddies from having worked at Oracle Corporation for over a decade, from companies like Miracle and Pythian, the Oracle ACEs and ACE Directors, Oracle OpenWorld, ODTUG, and a couple dozen or so user groups that I visit every year. The other foot is in the business of software. There, I have colleagues and friends from 37signals and Fog Creek and Red Gate and Pragmatic Marketing, the Business of Software conference, and the dozens of blogs and tweets that I study every day in order to fuel a company that makes not just software that meets a list of requirements, but software that makes you feel like something magical has been accomplished when you run it.

In my Oracle world, agile is a dirty word. I have to actually be careful when I use it. To my Oracle practitioner colleagues, the A-word means, as Joel wrote, “sloppy programming.” In my business of software world, though, “agile” means wholesome golden goodness, an elegant solution to the absolutely most difficult problems in our field. I’m not being facetious one little bit here, either. The two most important influences in my professional life in the past decade have been, far and away:
  1. Eli Goldratt’s The Goal: A Process of Ongoing Improvement
  2. Kent Beck’s Extreme Programming Explained: Embrace Change (2nd Edition)
Far and away.

I don’t mention this among most of my Oracle friends. I don’t blurt out the A-word to them, any more than I’d blurt out the F-word at my parents’ dinner table. To talk with my Oracle friends about the goodness of “A-word development” would go over like an enthusiastic hour-long lecture on urophagia.

A lot of really smart people are very anti-“agile.” I’m pretty sure that it’s mostly because they’ve seen project leaders in the Oracle market segment using the A-word to sell—or justify—some really bad decisions (see Table 1). So the word “agile” itself has been co-opted in our Oracle culture now to mean sloppy, stupid, unprofessional, irresponsible, immature, or naive. That’s ok. I’ve had words taken away from me before. (Like “scalability,” which today is little more than some vague synonym for “fast” or “good”; or “methodology,” which apparently people think sounds cooler than “method.” ...Ok, I am actually a little angry at the agile guys for that one.) That doesn’t mean I can’t still use the concepts.

Table 1.
What people think agile meansWhat agile means
No written requirements specification; therefore, no disciplined way to match software to requirements.You write your requirements as computer programs that test your software instead of writing your requirements in natural language documents that a human has to read and interpret to re-test your software every time a developer commits new source code.
No testing phase; therefore, no testing.You test your software before every commit to your source code repository, by running your automated test suite.
No written design specification; therefore, developers just “design” as they go.You iterate your design along with your code, but design changes are always accompanied by changes to the automated test programs (which, remember, are the specification).
Rapid prototyping always results in the production code being a fragile—well—rapid fragile, prototype.When you can’t know how (or whether) something will work, you build it and find out—but only the parts you know you’ll really need. You use the knowledge learned from those experiences to build the one you’ll keep.

Agile is not a synonym for sloppy. On the contrary, you're not really doing agile if you’re not extraordinarily disciplined. I think that is why a lot of people who try agile hit so hard when they fail. I hope you will check out Balancing Agility and Discipline: A Guide for the Perplexed, coauthored by Barry Boehm (yes, that Barry Boehm) if you feel perplexed and in need of guidance.

As with any label, I hope you’ll realize that when you use a word that stands for a complex collection of thought, not everyone who hears or reads the word sees the same mental picture. When this happens, the word ceases being a tool and becomes part of a new problem.

17 comments:

Robyn said...

Cary,

You are absolutely dead on with this one. Agile is a positive word and being 'agile' is a necessity in the software business. The problems arise with the misuse of the terms and the methods; I see teams who claim to use Agile methods, yet their process is anything but agile. And waterfall just doesn't cut it anymore: it takes too long and doesn't recognize that customers have an expectation that a system will adapt to their needs, and do so quickly.

I overheard a conversation just yesterday in which a developer described a start up where they had worked for 2 years and written the best code he'd ever seen, but somehow, no one was ever interested in buying it when it was complete. That's the classic downfall of a waterfall approach and if you aren't focused on what a user needs, it doesn't matter how good your code is.

Agile development has been adopted most rapidly by those that are NOT database people, so most of the implementations I've seen skipped the entire database design. Classic 'database as a black box' syndrome, but that's the fault of the project leaders, not the Agile development cycle. I don't think database professionals are unwilling to work in an Agile team - they just don't get included until late in the game when changes to the database structure are difficult and complicated. After all, there's all this application code that will now have to be changed if the database changes. And when the database has not been designed properly and the application doesn't scale to all the users, it's the database's fault. Which is the incredible irony of Agile: if the developers have listened to the users, they have a product that can be sold but if it doesn't scale, all those user champions very quickly become loud and public detractors, which is the exact opposite of the start up with a fabulous system no one wanted.

We have to be agile and adapt to the customer if we want products to succeed. We also have to think about real scalability so that if we are successful, we're prepared to support the masses of users we hope to attract. There's a huge chasm between the black box approach and the designing a perfect data model before moving on to the next step in the waterfall. We need to strive for the middle ground in database development if we want to be solid applications that users will actually use and continue to champion.

And whatever development process is used, it needs to let people focus on their core function. Any process can quickly become a noose around the team's collective neck when following the process becomes more important than the work to be done.

I'm presenting on this topic at UKOUG, so it's near and dear to my heart lately but I've got a few older projects to wrap up before I focus on this again. Thanks for all the links … I'm doing my best NOT to follow them today.

cheers … Robyn

Oyvind Isene said...

Excellent post. Knowing many smart programmers using agile methods, it is not something I can ignore, but I often find it hard to fit agile methods in my database world. Maybe I should pick up Scott Ambler's books again.

Cary Millsap said...

@Robyn: Thank you very much.

@Oyvind: I think Scott Ambler's work is indeed good to revisit. You should see Dominic Delmolino's post, too.

chris said...

"but software that makes you feel like something magical has been accomplished when you run it."

i know that's not the point of this post but I found that incredibly inspirational. I read 'The Goal' after reading you 'Optimizing Oracle Performance'. It was a phenomenal book. I'll have to check out the Agile book recommended. Thanks for writing!

Cary Millsap said...

Chris, I thought that was the most important phrase in the article, too. Thank you for mentioning it.

Connor McDonald said...

I'll go out on a limb - all methodologies fail.

This is not because there is anything intrinsically wrong with them, its just that moment one is adopted it coincides with the exit of commonsense...

I've lost track of the number of times in a meeting you see:

Person #1:

Methdology Champion: "Nope, thats not how the methodoloogy works"..

For Waterfall, "intelligent suggestion" is sometimes some sort of subtle,cheap improvement , which gets shouted down as "not signed off"... and conversely for Agile, "intelligent suggestion" is some sort of preliminary thought before commencing, which gets shouted down as "waterfall"....

:-(

The best methodology: Common Sense (tm)

Connor McDonald said...

Whoops, the "person #1" line was meant to be:

Person #1: (intelligent suggestion)

Dominic said...

Cary,

Thanks for the link.

Quite frankly I'm disappointed that relational database professionals in general and Oracle ones in particular seem to not be a part of the Agile world. As you've pointed out, there's nothing in Agile that encourages sloppiness or laziness.

And yet, the Agile community doesn't have good, solid database members -- resulting in Fr'Agile database outputs. The recent Foursquare outage is a prime example of this -- a site brought down because a database design didn't accomodate changes necessitated by increasing workload. And that was a simple 150GB database sharded across 2 EC2 nodes.

As a database professional, I keep trying to "break into" Agile so I can give Agile teams a solid, rapidly responsive core database capability. I'm afraid that it's going to be a long haul -- but I think it's worth it.

This is purely speculative on my part, but I wonder if the lack of database competency in Agile has its roots in the Object/Relational disconnect...

Cary Millsap said...

Dominic, I think—as we've discussed privately—there's a SQL/Relational disconnect that unnecessarily intensifies the Object/Relational disconnect...

Joel Garry said...

Careful Cary, next thing you know you'll be Fabian Pascal ;-)

When I designed and built products for sale, I came to the conclusion that there were two conflicting sets of requirements: a base stable product, necessarily a product of classical design (and the requirements easily abstracted and improved from the competition), and an easily customizable product, which necessitated what at that time was a newfangled RAD paradigm. Through layering, this conflict could be resolved. Even now, enterprise products face this dilemma - how to be able to adapt to custom business requirements. Exadata? "We'll not give any options to the dumb customers here." Oracle Apps? "The Oracle consultants are going back to India in a month, so I voted we don't implement at the start of our heavy season." (Actual recent quote from a train buddy while we were waiting for a late train and I discovered what she worked with). And of course, all the products where, you change one piece of code like any good performance person would suggest, and goodbye support.

So again, what makes sense technically is not what succeeds from the business perspective - they may even be in opposition. How can there be any common sense with no commonality? How can you fail with no agreed definition of success? I had a strategic planning professor from the aerospace industry with a very strict definition of success (on time, to specs, under budget). I chafed at his definition (at least at it being applied beyond large aerospace projects), but it became humorous when the book he had written about a successful project at a bank was mooted by the bank going away...

Anyways, as I now deal with the frustrating effects of it being _too_ easy to change an enterprise implementation at the whims of users who don't even understand what their own needs are and can't express it properly if they do, I shouldn't have said sloppy programming. I should have said "sloppy design."

(I kid. My real view is the converse of Connor's, it's more like "all methodologies can kinda succeed, most people manage to somehow make it work - though the fails are more interesting". That is why your [I mean oakies] work towards clarity and perfection is so important, it is actually difficult to explain why or when "good enough" isn't.)

Oh well, back to fixing our db integrity problems stemming from external entities sending data with incompatible definitions of data elements and relations between their putatively standardized systems. Dang, Connor's right...

word: anetano

Charles said...

Hi there, awesome site. I thought the topics you posted on were very interesting. I tried to add your RSS to my feed reader and it a few. take a look at it, hopefully I can add you and follow.
Agile Process

Kyle Hailey said...

I'm a bit baffled at the agile backlash. I quite appreciated agile when I had a chance to work in that model. The agile model with costing was one of the most empowering facets for me as a program and product manager, unlike the willy nilly approach of "I think it will take this long to do this feature".
Plus, no development framework will make people do the right thing or prevent them from choosing an abysmal architecture.
I love agile, but then again as someone who is often involved in user interface design, finding the optimal design, unlike solving a math problem, requires prototypes. It can't just be spec'ed with the correct solution.

Cary Millsap said...

Kyle, I agree.

Robin Dymond said...

Nice Post Cary. You prompted me to write a post on the issue of DBAs and Agile with what I hope is some friendly advice on how DBAs can bridge the divide and ensure they can work with Agile teams.

Cary Millsap said...

Robin (good to have so many Rob[iy]ns!),

Thank you for your comment and for letting me know about your blog. I like your article!

Mark W. Farnham said...

Riddle:What is the difference between a terrorist and a methodologist?


While you're thinking, I'd like to suggest that the reason there is a backlash against "agile" is similar to why N Wirth found the GOTO statement harmful: It is far too easy for naive amateurs to create a Gordian Knot of spaghetti. Of course naive amateurs can also create a disaster with an if-then-else/do while language. It just takes longer. Likewise a horrible software product created by a waterfall process. Thinking Clearly (yours, I think) on the other hand can be the secret sauce that will make any reasonable method create successful programs, projects, and products. Thinking Clearly, perhaps especially with Agile, is what helps you avoid box canyons where you have to throw a lot away to get that next new feature. One of the most beautiful programs I ever saw was written by Tom Kurtz as an illustration of numerical methods. All it did was invert a matrix avoiding round-off error by creating a paired set of integer numerators and denominators. It was also a better illustration of the matrix inversion method than the textbook. (Just like your regression test being the design document [together with the right answers, I suppose]). It had a lot of GOTO statements, but of course every GOTO was part of an if-then-else or case statement simulation, a loop construct, or an error condition exit. That, of course did not prevent other people from writing a lot of spaghetti in BASIC6. A good method can make it take more effort to really screw things up, and Thinking Clearly together with a good method will tend toward success. The "dirty word" is because people who don't know what the heck they are doing try to convince people who are trying to help them that they fail to see the Emperor's New Clothes because they don't grasp the method. This is similar to Chris Dates' set piece about being told he just doesn't get why objects cannot be described in the relational model.
Sigh. You don't need to hide your liking for Agile from me. If the people plan to Think Clearly, I'll be happy to use Agile.

Answer: You can negotiate with terrorists. (I first heard this from Floyd Parks)

Victor Kahn said...

Firstly I agree with the author in regards to the differences in perception between software developers and business departments.

However, all that is happening in the world of software development is a rebranding of the Emperors clothes. The proof of the pudding is in the eating and when you scratch the surface, the same mentality continues but with shorter cycle but it's now called 'Agile' - simply by reducing the requirements per development cycle.

Nothing has changed except for the surface with the introductions of scrums etc.

Lastly, isn't Agile itself a rebranding of RAD - Rapid Application Development?