Moral Hazard: The Implacable Enemy of Agile

25 03 2012

This is from Wikipedia:

In economic theorymoral hazard is a tendency to take undue risks because the costs are not borne by the party taking the risk. The term defines a situation where the behavior of one party may change to the detriment of another after a transaction has taken place. For example, a person with insurance against automobile theft may be less cautious about locking their car, because the negative consequences of vehicle theft are now (partially) the responsibility of the insurance company. A party makes a decision about how much risk to take, while another party bears the costs if things go badly, and the party insulated from risk behaves differently from how it would if it were fully exposed to the risk.

In more general terms, moral hazard is a situation where it is the responsibility of one person or group to make a mess, and the responsibility of another person or group to clean it up.

For example, for a limited time while I was a kid, it was my job to play with the toys in my room, and my mother’s job to keep the floor uncluttered enough that she wouldn’t fall and break her neck when she had to walk across it in the dark.

So…who won, do you think?  Was the floor always clean, or was the floor always messy?

Of course the floor was always messy.  It was always messy until a rule was made–and enforced–that if my room wasn’t cleaned by a specified time, then I either wouldn’t get something I wanted or would get something I didn’t want.  (Decades later, I honestly don’t remember which.)  The trick was to take the incentive out of the wrong place and put it into the right place.

But this post isn’t about parenting, it’s about company culture.  Here are some more relevant examples of moral hazard:

  • A company has a maintenance team–or a group of them–whose job it is to jump on production problems and fix them when they crop up.  That is, it’s the development team’s job to create production problems and the maintenance team’s job to fix them.
  • Architects come up with designs, which developers then implement.  That is, it’s the architects’ job to create dozens of small but annoying and confusing incompatibilities and discrepancies between their overarching concept of how the system should work and the reality of how the system actually does work, and the development team’s job to make that design work anyway (in the process creating even more discrepancies between the architects’ picture of the system and reality).
  • Some uber-architect in the IT organization writes a Checkstyle (or other) coding standard and mandates that all code must pass the standard before it’s allowed into production.  That is, it’s the uber-architect’s job to create thousands and thousands of build errors, and the development teams’ job to clean them up.
  • Technical people outside the development team must sign off on all code before it goes into production.  That is, it’s IT’s job to create deployment delays at the last moment based on various criteria that have little or nothing to do with actual business functionality, and the product owner’s job to keep customers from leaving because of those delays.
  • Agile coaches explain all the things that each part of the Agile process is supposed to accomplish, and their clients take them seriously.  That is, it’s management’s job to spend a large portion of the development team’s time on meaningless meetings, and development’s job to deliver anyway.
  • Ideas from the dev team for making things better better or easier or faster or more efficient are submitted in writing to a special committee, which at its next meeting–or the one after that, or at the very latest the one after that–will gravely consider them and all their associated consequences.  If an idea is judged to be worthy, the machinery for imposing it on the entire organization will be set in motion as soon as the proper approvals can be applied for and awarded.  That is, it’s the committee’s job to impose incomprehensible, irrelevant, and counterproductive regulations on distant teams that already had an environment all set up and working for them, and the job of those teams to deal with the unintended consequences.

Moral hazard puts incentives in the wrong places for getting anything done.  It destroys morale, it dulls the wits, and it creates endless loops of excuses.  For example:

Product owner: “We didn’t go into production over the weekend?  Why not?”

Development team: “It was the operations team’s fault: they rejected our package.”

Product owner: “Why did you reject their package?”

Operations team: “It wasn’t our fault: the Checkstyle analysis blew up with over 700 coding-standard violations.”

Product owner: “Why did you submit a package with over 700 coding-standard violations?”

Development team: “It’s the architect’s fault: there’s no way we can track down and fix that many violations and still get your functionality into production on time.”

Product owner: “Why did you create a Checkstyle configuration that was so out of whack with the way our team codes?”

Architect: “It’s the development team’s fault: we have to have some standards, otherwise we’ll end up with a whole base of impenetrable code that nobody dares change!”

Product owner: “Why don’t you have any standards?

Development team: “Uh…”

The real answer to that one, of course, is one that the development team probably won’t give: coding standards are somebody else’s responsibility, not theirs, so they aren’t used to thinking about it and don’t really care.  Likewise things like simple design.  They’re not responsible for the design, and making official changes to the design is so painful and time-consuming that it’s much easier not to think about design at all, and just hack in fixes where the official design doesn’t fit reality.  Testing?  Sure, they’ll have to make a token effort in that direction, but as long as the Emma coverage numbers are such that the code passes the externally-imposed checks, it doesn’t really matter how good or bad the tests are, because if there are problems the maintenance team will catch them.

In a situation like this, the drive to excellence inside the team is largely extinguished, for two complementary reasons.  First, the culture is designed so that excellence is defined by people outside the team, so that any ideas the team has to make things even better are therefore by definition not excellence (or else said people outside the team would already have had them), so they are resisted or even punished.  Second, it’s just not the way things are done.  Mediocrity is a way of life, and anything completely intolerable will be taken care of by somebody else in some other layer of company bureaucracy.

Any team member who has a hunger for excellence must surmount both the why-bother and the you’re-gonna-get-in-trouble obstacles before he can accomplish anything.

The thing that has made Agile so successful, in the places where it has been successful, is the fact that it puts the incentives in the right places, not the wrong places.  To the greatest extent possible, the rewards for your successes and the consequences for your failures come directly to you, not to somebody else.  You are given not only the power but also the incentive to better yourself, the project, and the organization, and whatever stands in your way is made big and visible, and then removed with alacrity and enthusiasm.

Many organizations believe that they have adopted Agile by instituting certain of the practices that Agile coaches tell them are part of Agile, but they retain a company culture shot through with moral hazard where many of the incentives are counterproductive.

For example, in a truly Agile environment:

  • There is no separate maintenance team.  The development team is responsible for a project from conception to end-of-life.  If there’s a production problem that means somebody rolls out of bed at 3:00 in the morning, then it’s somebody from the development team responsible for that project.  If the project is so buggy that its dev team spends 80% of its time fixing production issues, then until the project is much healthier, that dev team isn’t going to get many new projects.  Hence, the dev team is going to do everything it possibly can by way of testing to make sure the project is ready for production, and by way of logging so that if there are issues forensics will be quick and easy, and by way of simple and open design so that any problems can be fixed fast.  And they’ll do this without any scowling remonstrances from a stern-faced architect outside the team, because the built-in reward for success and consequence for failure is worth much more to them than anything he could offer anyway.
  • Developers come up with designs, which they then vet–should they decide it’s necessary to avoid becoming a maintenance team–by consulting with architects who know the whole system better than they do.  The architects have no power or responsibility to approve or reject; their job is merely to act as information resources.  Hence, the designs will be better and simpler, because they spring up from reality, rather than down from an ideal.  The architects will know more about what’s really going on, because they’ll be doing more listening and less telling–and because if they don’t, development teams will prefer to interact directly with other development teams to find out what they need to know and make the highly-paid, expensive architects superfluous.
  • If Checkstyle or one of its competitors is used at all, it’ll be used because the team has decided it will be helpful, and the standard it enforces will be tailored to the team by the team, not imposed on the entire organization by somebody with a Vision who actually writes very little code.
  • The product owner, not anyone in IT, decides when the code goes into production and is responsible for the result.  If the product owner puts code in too soon, he’ll be responsible for the negative experience the customers have: but that’s appropriate, because they’re his customers and he knows what’s important to them.  At least he’s not likely to put the code in too late, because as the product owner he knows which barriers to production are important to his customers and which they don’t care about.  Wanna place any bets on where he’ll stand on Checkstyle violations?
  • The team’s cadence–iteration length, end/start day, standard meetings, impromptu meetings, etc.–is determined by the team based on the requirements of the project and the needs of the product owner.  Fly-by-night Agile coaches may pass through and provide consultation and advice, but it’s the team–not management, not IT–and the product owner that decides what works best for the project.
  • Finally, anyone on the team who has an idea to make things better or easier or faster or more efficient can directly implement that idea for the team immediately, with no submission process or lengthy approvals more involved than bringing it up after standup one day and getting thumbs-up or thumbs-down.  Whatever it is, it doesn’t have to affect or be imposed upon anyone else in the company.  If it turns out to be a good idea, other teams will notice, inquire, and imitate: voluntarily, on their own, because they’re attracted by it.  If it turns out to be a bad idea–and let’s face it, nobody can say for sure whether a newly suggested practice will have unforeseen and unintended consequences–then it will slow down only one team for an iteration or two before being abandoned, and the company will cheaply learn something valuable about what doesn’t work.

If a company labors to adopt Agile, but insists on keeping the moral hazard in its culture, the change will be limited to a few modifications in practice, but no significant increase in velocity, efficiency, quality, or morale.  Furthermore, people in an environment like this will look around and say, “So this is Agile, huh?  What a crock!  This sucks!”

But what if the company is also willing to put the incentives where they belong, as well as adopting practices that those incentives have shown to be useful in other organizations?  What if IT is told that its purpose is to serve and support development teams as they direct, and the development teams are told that if IT doesn’t support them properly, they’re to bypass it and do whatever’s necessary to get into production?  What if development teams are told that their purpose is to serve and support the product owner, and that if they don’t satisfy their product owner, he’ll go find another team?  What if the product owner is told that his purpose is to serve the business, and any demand made on him by development or IT that doesn’t move toward satisfying the needs of the business can be rejected out of hand?

In a situation like that, moral hazard will quickly become Big And Visible, and it can be dealt with promptly.

As a matter of fact, one might say that an Agile transformation should consist chiefly of a quest to expose and eliminate moral hazard, and that the various practices for which Agile is so well known will automatically trail along behind because they’re the best way anyone’s found so far to operate in a low-moral-hazard environment.

If you adopt economical driving habits, you’ll end up putting less gasoline in your tank.  But if you skip past the economical driving habits and just put less gas in your tank, you’ll end up muttering grim imprecations as you trudge down the highway with a gas can.



5 responses

25 03 2012

See, if you hadn’t had the privilege of spending the time you spent where you spent it, we would not have gotten this gem.

25 03 2012
Dan Wiebe


26 03 2012
Dan Wiebe

Ah–I forgot. Another example of moral hazard: IT approves the use of only certain third-party libraries across the enterprise, and allows only those libraries in production. That is, IT is responsible for fixing the company’s technology at a particular limited place in the past, and the dev teams are responsible for coming up with ways to hack the expanding, changing present into it–and the business is responsible for finding ways to compete, under such limitations, with smaller, faster competitors that have no such limitations.

26 03 2012
Jim Kirkbride (@jameskbride)

Hi Dan, great read. I totally agree with you about the dev team being the ones to maintain the code and provide production support (Anything we write we should be willing to maintain as well, right?), but it seems that more often than not the constraint about dev teams doing new dev and the maintenance team doing production support comes from budgeting.

The way it seems to go is “We’ve allocated X dollars for new development, which will go towards this team’s budget. We’re also forecasting (otherwise known as taking a Scientific Wild-Ass Guess) that Y amount of dollars may be necessary for production support, because something always seems to go wrong (the manager typically gives the dev team a dirty look at this point). For reasons I can’t fathom they’ve got it in their head that those two numbers should be assigned to different teams, and that if they didn’t do that the dev team’s operating budget will increase linearly over time (A false assumption, as sooner or later a given product will stabilize, but I’ve heard these exact words come from a C-level manager.).

I think that to make this model truly effective (the dev team also providing production support) a change must be made in the way that a company budgets and allocates resources… which, I suppose, is the end result of your removal of the moral hazard.

27 03 2012
Dan Wiebe

I agree that any company that decides on an Agile transformation must be willing to have its culture drastically changed, because that’s what Agile is: culture, not practice.

Several times I’ve seen a client where the suits call us in, point at the grunts, and say, “Make them Agile.” That part goes okay, but when the Agile infection begins billowing up the corporate ladder toward those very same suits, it’s suddenly, “Whoa now, wait a minute, hold on there–that’s not what we signed up for!”

I’ve never been a suit–always a grunt–but in my experience the only thing I’ve seen motivate somebody to hold onto moral hazard and fear the advance of Agile is incompetence: especially incompetence he thinks nobody else knows about.

That’s certainly what it was with me that made me hate Agile until I was forced to humble myself and learn a thing or two.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: