Riding Herd on Developers

28 03 2012

When I was a kid, I learned a song in Sunday school about motivation.  It was called The Hornet Song*, and part of it went like this:

If a nest of live hornets were brought to this room
And the creatures allowed to go free,
You would not need urging to make yourself scarce:
You’d want to get out; don’t you see?

They would not take hold and by force of their strength
Throw you out of the window; oh, no!
They would not compel you to go ‘gainst your will;
They’d just make you willing to go.

A client architect once told me, “We’ve been trying really hard to get on the software-craftsmanship bandwagon.  We’ve been having lunch-and-learns, and offering training, and everything we can think of; but maybe two or three people show up.”

When you’re a developer, and there are three or four layers of inspection and signoff between you and production (as there were in this case), there’s no real reason to care about things like software craftsmanship, unless you just naturally have a fascination with such things, in which case you’re certainly not going to be working in a place where there are three or four layers of inspection and signoff between you and production.

What makes you care about software craftsmanship is being right up against production so that the first person who sees what leaves your fingers is a customer, and the first person who gets a call when something goes blooie is you.

My architect friend said this as part of a conversation about whether IT should control the dev teams (and through the dev teams the business interests they serve) or whether the business interests should control the dev teams (and through the dev teams IT).

He objected that unless IT rides very close herd on the dev teams, they all do different things, and build engineering and operations becomes an intractable nightmare.

This struck me as a perfect case of trying to hang onto the soap by squeezing it tighter.

He was describing a classic case of moral hazard.  In his world, it was the developers’ job to make a horrible mess, and the job of build engineering and operations to clean it up; and the only way that job would ever be small enough to be kept under control by build engineering and operations was to bear down ever harder on developers.

As always in a situation like this, the real solution will turn out not to be figuring out how to cram one party into an ever tinier, more inescapable prison, but how to eliminate the moral hazard that creates the problem in the first place.

First, let’s talk about build engineering.  What is build engineering?  In this case, it was one huge farm of CI servers that knew about every project in the company, and a predetermined set of builds and test environments that every checkin from everywhere automatically went through, and a group of people to oversee and maintain this system and explain rather impatiently to dev teams why (or at least that) no, they couldn’t do that either, because the build system wasn’t set up that way.

I watched the reaction in that company to the news that a dev team was planning to set up its own team-local CI server, with a bunch of VMs to simulate the application environment for testing, so that it could run certain tests after every checkin that the company-wide system could only run once a day.  There was a moment of slack-jawed horror, followed by absolute, nonnegotiable, and emphatic prohibition–even though a team-local CI server would not require the least change or modification to any code or configuration or practice having to do with the main system.

If the build engineering team reacted this way to a suggestion that doesn’t involve them doing anything different, imagine their reaction to a suggestion that they should change something!

In their defense, though, it’s not hard to understand.  As any developer knows, build is a terrifically delicate process, especially when it involves enough configuration to get a nontrivial application to run successfully in a variety of different environments (on the order of a dozen, as I remember) with different levels of mocking each time.  Getting that all working once is already something to write home about: imagine getting it working—and keeping it working—for 150 completely unrelated applications whose natures, characteristics, and requirements you know next to nothing about!

It strikes me as being about as difficult as playing all the positions on a baseball team simultaneously.

Which is why, in the real world, we have baseball teams.  The job of playing all the positions at once doesn’t exist, because there’s no need for it, once we’ve got the first baseman playing first base and the pitcher pitching and the catcher catching and the outfielders outfielding and so on.

In my opinion, that whole huge company-wide build system should have been thrown out—or at least broken up into tiny pieces—and the individual teams should have been responsible for their own CI and testing processes.  Those processes undoubtedly would have been completely different from team to team, but that would have been a good thing, not a bad thing, because it would mean each team was doing what was best for its project.

I suspect my architect friend would have protested that a huge build system like that was the only way they could be sure that every team put their code through all the proper testing environments before it went into production.

My response would have been that the most important thing was not that all code go through every environment envisaged by the build engineers, but that all code worked properly in production.  Put the dev teams right up against production, as suggested above, and they’ll find a way to make sure their code gets whatever testing it needs, or else they’ll be doing 24/7 production support instead of software development.  (That‘ll get ’em to your lunch-and-learns, for dang sure.)

But what about operations?  Do we solve that problem by putting the dev teams in charge of operations as well?

I don’t think so—not in most cases.  It’s an issue of boundaries.  Just as it’s a moral hazard for the build team to be compelling all sorts of behavior inside the dev teams, it’s a moral hazard for the dev teams to be specifying behavior inside the operations team.

The operations team should say, “Based on our skillset, expertise, and practical constraints, here’s a short list of services we can provide to you, along with the technologies we’re willing to use to provide them.  This list is subject to change, but at our behest, not yours.”  The dev teams should design their CI processes to spit out something compatible with the services provided by the operations team, if at all possible.

When a dev team can’t do without something operations can’t provide—or could theoretically provide but not fast enough—that’s when the dev team needs to think about doing its own operations temporarily; but that’s a sign of a sick organization, and one way or another, that situation probably won’t be allowed to persist long.

To sum this all up, morale is very important on a dev team.  You don’t want developers who lie in bed staring at the brightening ceiling thinking to themselves, “I wonder what insufferable crap they’re going to make me do today.”  You want developers who say, “If my pair and I get our story done early today, I’ll get to spend some time playing with that new framework the guy tweeted about last night.”

To be in a state of high morale, developers need to be constantly in an innovative state of mind, slapping down challenges right and left and reaching for ever more velocity multiplication in the form of technology, skill, and experience.  (Notice how I didn’t mention super-high salaries or super-low hours or team-building trips.)

You don’t make or attract developers like this by insulating them from the real world with signoffs and approvals and global build systems and stupid, counterproductive rules, and then imposing a bunch of stuff on them that—even if it’s really great stuff, which is rare—they’ll resent and regard very suspiciously and cynically.

You make them by tossing them into the deep end—preferably with several senior developers who already know how to navigate the deep end comfortably—and making sure that when they grab out for something to stay afloat, it’s there for them.  (If they do the grabbing for it, they’re not going to see it as an imposition.  See?)

The term “riding herd” comes from the great cattle drives when America was young, where several dozen cowboys would drive a herd of thousands of dumb cattle a distance of thousands of miles over several months to be slaughtered.

Do you want developers like those cattle?

Then ride herd on them, like the cowboys did.

Otherwise don’t.

Update 3/30/2012:

My architect friend saw this article and got in touch with me about it.  We discussed what he said was the major reason for all the heavyweight ceremony and process in his company: Sarbanes-Oxley.  SarbOx means that the company CFO is accountable for everything individual developers do, and if they do something the government doesn’t like, he goes to prison: hence, he is pretty much required to ride herd on them.

SarbOx is an area that I haven’t yet been seriously involved in.

I understand that it’s a government’s job to kill people, devastate lives, destroy liberties, and generally cock things up as far as it can without its politicians ending up swinging from lampposts in droves; but it’s also true that for decades now software developers have been wading into cocked-up situations with the help of domain experts and making them convenient, fast, and smooth.

Is SarbOx really such a competently-conceived atrocity that even Agile developers can find no way around, through, or over it?  Somehow it seems unlikely to me that politicians could be that smart.

*The complete Hornet Song used the quoted analogy to explain the behavior of the Hivites, Canaanites, and Hittites in Exodus 23:28, and the behavior of Jonah in Jonah 3:3.




4 responses

29 03 2012
PM Hut

Hi Dan,

I have to disagree with you on this. Ensuring that the developers are properly motivated and have a sense of ownership does not mean that we have to expose them to clients – there are many, too many, crazy clients out there. Such clients can cause a lot of stress.

In fact, many developers choose to work in a company in order to avoid dealing directly with such crazy clients.

29 03 2012
Dan Wiebe

What do you mean by exposing them to clients? Do you mean having them talk directly to customers on the phone?

I agree with you that that’s probably not the best idea anyone’s ever had, at least in the general case.

What I mean comes in two halves.

First, if I’m a developer, then what I write goes right into the hands of customers–either directly, if it runs directly on the customer’s device, or as a service, if it provides functionality served from a website. There should be nobody standing between me and the customer, inspecting and signing off on what I write (other than the product owner, of course, on a story level), or else my motivation for excellence is going to be substantially reduced.

Second, if I’m on a dev team, and a customer complaint comes in to customer service about a defect in my team’s code, then my team needs to be tasked to fix that defect, once the fixing of it has been prioritized by the product owner. I don’t mind a separate customer service department that answers the phones and follows flowcharts to solve customer-error problems; but I do object to a separate maintenance team that fixes the developer-error problems the flowcharts can’t handle.

Is there anything further we can discuss in either of those assertions?

2 04 2012
Jeff Hoover (@JeffHoover)

I haven’t worked under SarBox either, but I just found a post from a ThoughtWorks person about agile under SarBox:

10 05 2016
Raul Miller

My thoughts here are:

(1) It’s easier to ask forgiveness than permission, but

(2) If it’s not broke, don’t fix it, and

(3) Government should be going after the big problems and ignoring the little ones, but the little ones are so much easier to deal with.

(4) Developers should be going after the big problems and ignoring the little ones, but the little ones are so much easier to deal with.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: