Should Agile Testers Code?

13 01 2012

This morning at CodeMash, I had an argument over breakfast with Jeff “@chzy” Morgan (more commonly known as Cheezy) of LeanDog about Agile testers.  After some advancing and retreating around the battlefield, it became clear that we seemed to generally agree about almost everything regarding how Agile teams should be run, except for this one issue.

The story test that every story card should produce when it first arrives in the Dev In Progress column: should that story test be written by a developer or by a tester?

Cheezy says a tester should write it; I say a developer should write it (after a detailed conversation with a tester or product owner, of course).

I’m going to reproduce here, as well as I can remember them, the salient points of the argument, and I’d love to have people having more experience with testing than I have address them in the comments.

Before I say what I’m going to say, though, I’d like to say what I’m not going to say.  As a developer, I’m not going to disrespect testers.  I’m not going to imply that coding is an exalted, morally superior activity that should be undertaken only by developers, since developers are the moral superiors of testers.  I live in awe of good Agile testers, and–after having tried once or twice–I have no lingering illusions that I could do what they do.  Agile testers have brought my cheese in out of the wind so many times that I’ll be indebted to them for a long time.

Essentially, my arguments will all boil down to this: people should do what they’re good at.

Finally: I’m going to do my best to be fair to Cheezy’s arguments as he presented them.  Cheezy, if you read this, feel free to comment publicly or privately about anything I get wrong, and I’ll post an update.


Cheezy says that since testers will almost always know more about the problem domain than developers will, it is the testers who should have the responsibility of translating the requirements from story-card language into executable language, like Cucumber over Ruby.

I’ve been on teams before where this was advocated.  It sounds good, both to Cheezy and to me.  What I find in the real world, though, is that writing good tests is absolutely the hardest technical part of a developer’s job.  It can sometimes take me a couple of days to get a story test written and failing for the right reasons, given all the investigation and forgotten last-minute spiking that has to go into it; once I do that, the code to make it pass is generally much easier.

So in this case you’re not just saddling testers with development work: you’re saddling them with the very hardest kind of development work.  For this, it’s not good enough to take an excellent tester and teach him a little coding.  You have to actually make him into a real honest-to-goodness developer as well as a tester, if he’s going to be successful.

I was on a team once with a tester who was really good at writing story tests.  In his case, if I remember correctly, he was writing easyb tests and backing them up with Java.  As a matter of fact, he liked it so much that he ended up crossing over and becoming a developer instead of a tester.  I was rotated off the project shortly after that, but I imagine that with both testing and development skills, he went far and fast.  If he did so, however, it was as a developer, not as a tester.

The rest of the testers on that team didn’t really cotton well to easyb or Java.  They struggled at it for awhile, frequently resorting to the developers for help, and eventually stopped writing automated tests altogether, whereupon the task fell to…that’s right, the developers.

To support Cheezy’s side of the argument, I should say that I was also on a team that used Fitnesse over Java for ATDD, and on this team the tester really did write the acceptance tests in Fitnesse, and it worked out great.  On my side of the argument, though, the Fitnesse tests had to be supported by fixtures written by the developers, and it’s somewhat of a stretch to call filling out a Fitnesse wiki table “coding.”

Cheezy says that the most dangerous part of the software development process is the place where requirements get translated from English to code; therefore it’s always safest for that to happen all inside one brain, rather than having it depend on one person understanding another person’s explanation.

I agree that that would be the best place for it if that kind of thing worked; but in my experience, in a large percentage of cases, the tester gets stuck while writing the code–remember, this is hard work, not easy work–and enlists a developer to help him.  When that happens, the developer has to look at the tester’s existing code and figure out what he’s trying to do–error-prone–talk to the tester and figure out what he wants to do–error-prone–and guide him through the rest of the process of getting his thoughts down in code–again, error-prone.

I don’t know of a way that we can eliminate errors in translation.  We can reduce them by listening really, really carefully and asking lots of questions; that’s the way Agile developers are trained to operate.  But eliminate?  You tell me.

There’s another issue here too that didn’t come up in our discussion, but which I’d like to raise here.

Many times, especially when a card represents a foray into an as-yet-unexplored wing of the project, there will be spikes preceding it to discover things like, “Once I’ve sent a set of instructions over the network to that remote node, how do I get it to tell me what it’s done in response to them?” or “How do I take control of this legacy subsystem so that I can make it behave as my tests need it to?”

Those spikes won’t be completed by testers: they’ll be completed by developers.  Hopefully they’ll be completed by the same developers who will eventually play the cards they precede.  Having those developers employ the technology they’ve just spiked out is certainly less error-prone than having them explain it to testers so that the testers can write the story tests (quite possibly in a completely different language from that of the spike) that use it.

Cheezy says that having testers automate their testing frees them from regression testing so that they can do the most valuable thing testers do: manual exploratory testing.  I agree that exploratory testing is the crowning glory of an Agile tester; I agree that manual regression testing is worse than scrubbing the gymnasium floor with a toothbrush.  But my experience with testers writing automated tests is that they spend so much time struggling with the code that it cuts into their exploratory time just as much if not more.

And as for manual regression testing, nobody should have to do that–except perhaps after a system-wide refactoring, like moving from a SQL database to a No-SQL database or switching Web frameworks, where the vulnerabilities move around.  When a tester discovers a defect during his exploratory testing, a developer needs to write a test proving that defect doesn’t exist and then make that test pass, leaving it as a permanent feature of the test suite, so that the tester never has to regress that problem again.

Cheezy pointed out that it’s very expensive when cards move backwards on the wall, and opined that having testers write story tests could eliminate that.  I’m skeptical, but if Cheezy–if anybody–can come up with a way to eliminate having cards move backwards on the wall, I suggest that the very next step should be getting rid of testing altogether: why would it be needed if a card never moves backward anyway?

A final point I’d like to make is that I’ve found that one of the reasons testers are so valuable is that they don’t instinctively think like developers; they think like users.  Making them code would force them to think like developers at least a little–that is, enough to be able to do the hardest thing developers do.  Hopefully it wouldn’t pull them completely over to the Dark Side, but in the final analysis I think I really would rather have them stare blankly at me as I yammer on about some developer excuse and then say, “But it doesn’t work.”  If they start accepting my excuses because they understand my viewpoint, that’s no good for anybody.

Thanks again, Cheezy, for the discussion this morning; it has helped me think about Agile testing in new ways.  Perhaps your comments and others will help me further.



19 responses

13 01 2012
George Dinwiddie

I find it depends on the people. But not knowing the situation, my default position is that a tester and a programmer should pair on writing the test. I’ve seen that work very well, and think it’s generally superior (in quality and speed) to having either do it alone.

14 01 2012
Dan Wiebe

I think that’s great, in some or most cases. The context I came to CodeMash in, though, was being in the middle of writing a highly technical story test, where we were still fighting through the arcane “arrange” issues of getting connected to everything we had to connect to, rather than acting or asserting.

At least for that part of the test, I think it would be much more effective to be paired with another developer than to be paired (as I was) with a tester.

14 01 2012
George Dinwiddie

I don’t know the details of your situation, but I’ve generally found that a large or complicated setup for a test is an indication that something else is wrong.

29 02 2012
Dan Wiebe

In this case, the story under development involved fairly complicated communications between four separate networked systems: two were connected to the System Under Test by JMS queues, the other was a web service.

In an initial story test for a story like that, you want the SUT to actually be reading from and writing to queues, speaking HTTP with a real socket somewhere, and doing all the real marshalling and unmarshalling. That’s pretty complicated and requires a fair amount of setup.

In the individual unit tests that you write after the story test is failing for the right reasons, you can bypass the communications-overhead boilerplate and get much simpler setup; but the way you get that simplicity is by sweeping the complexity into the story test and doing it only once.

If you try streamlining the story test by mocking too much, you’ll find yourself making incorrect assumptions about how things actually hook together in real life, so that when you get done with all your simple unit tests, you have something that works fine in your unit-test fantasy world, but has much less success in the real world.

My choice is to write a single story test that’s grounded in reality and get it failing for the right reason, and then use unit tests to drive enough code to make it fail for a different reason…and then another different reason…and another…until finally it succeeds. You won’t necessarily be done at that point, because there may be special conditions the story test doesn’t exercise, but you still get two massive benefits.

First, once you get done, your code will work in the real world.

Second, the progressively-failing story test will guide you through the development of the story and take care of the tasking for you. If you forego the story test and instead make a list of tasks, you’re going to forget something important.

Sometimes you find yourself with a story that involves several paths through the code that are distinct enough that external systems can tell the difference between them, and you’re tempted to write several story tests, one for each path. I don’t object to multiple story tests, but I think encountering that situation should suggest that the story you’re working on be split into several distinct stories so that you can write one story test per story.

14 01 2012
Michael Norton (@DocOnDev)

I’m with George Dinwiddie on this one. I’d much rather see testers and developers pair on the creation and automation of tests. Both can learn from the experience and the back-and-forth that inevitably happens can now occur in real-time.

29 02 2012
Dan Wiebe

I think it’s a good idea for developers and testers to pair on the parts of the story development that involve business logic: testers are going to understand that better than developers.

But the parts of the story development–especially the story test–that just involve figuring out how to contact this server and set up that queue and mock this database schema and so on…those are parts about which a tester has no special knowledge and in which a tester has no interest.

I’d like to pair with another developer on the technical parts of the story; once it’s ready for business logic, that’s a good time to abandon the developer and find a tester.

14 01 2012

I very much believe that agile QAs (testers) should write their own tests. The main thing for me is that if they want to build up their own regression suite then this is a great way to do that. Each time they write a test, they are effectively writing automation to help free up regression time – that is such a big thing. Writing their own tests that form part of a regression suite should help give them the motivation to write clean and maintainable code. I worked for a company that the QA team spent an entire week manually regressing the site – that was madness!

Secondly, there is the idea that if the agile QAs write their own tests, then it is a true understanding of the requirements. If they understand the requirements then that is another person who can keep a developer going down the correct development path.


14 01 2012
Lanette Creamer

I don’t understand why this matters? Testers have a huge variety of skills, just like developers do. For example, I can code some in Python, but that does no good if my team is using .net and C++. In that situation, it helps if the team understand that I know programming vocabulary and basics and will speak to me at the appropriate level. Why there must be some hard and fast mandate on how things are done is what irks me. Can’t we just say “Experiment and use the process that works best for the people you have.” Isn’t that what people over process is about in the Agile Manifesto? If we’d all follow the manifesto instead of getting too granular it would be much better for those of us who actually do the work on agile teams. That means developers and testers in all forms of Agile in practice.

Even if as a coach your experience is usually X works, you can explain now that you know of situations where Y worked better, so it might be something to try.

We are still at the START of discovering the best ways to collaborate. Trying to make final decisions on “how it will be” seems silly to me.

29 02 2012
Dan Wiebe

“Experiment and use the process that works best for the people you have” is better than “Testers shall not write code.”

But it’s also better than “Testers shall write code,” which is the mandate I was reacting against.

If you’re going to experiment, I would counsel a heavy emphasis on failing fast: you don’t want to waste a lot of time developing an infrastructure you think is going to make tests easy to write for testers and then find out it just moves the problems around without solving any of them.

14 01 2012
Doug Waddell

As Kurnosov said, we don’t want all the information residing with the testers, we also don’t want the developers to become information silos either.

I agree with both Dinwiddie & Norton, regardless of the context of the discussion in question, pairing on the test creation is the way to go. Each half of the pair brings a different dimension to the test creation that could have been missed by a single individual.

Mature development groups and most likely utilizing multiple testing methodologies anyway (unit/technical as well as acceptance/business value for example). The combination of testing approaches is what helps prevent cards from moving to the left. That doesn’t remove the need for testers, it changes the basic function of their role from _finding_ defects to helping _prevent_ defects in the first place.

14 01 2012
Diane Zajac-Woodie

I agree with Lanette…we must work with the people and the situation that we have. On my current team of 11 people, there is only 1 tester, who is ‘allocated’ 50% to our team and 50% to another team. It’s ludicrous! QA management has been trying to hire more testers, but what do we do in the meantime? Roles need to be ignored in order to get the work done.

If we’re speaking theoretically, I envision the flow as: BA pairing with QA on writing the acceptance tests and then QA pairing with devs on automating those tests. That provides continuity of knowledge from requirements to done by way of the tester.

29 02 2012
Dan Wiebe

4.5% QA is indeed ludicrous. Suggest a slogan to your product owner: “Over 95% Quality Free!”

If you have to have developers doing QA, then you have to have developers doing QA. In some cases, they may be better than nothing.

But in other cases, they may be worse than nothing. There have been occasions when I’ve been masquerading as QA when it’s been like this:

“Story’s finished!”

“How do you mean? Has it been QAed?”

“Yup, yup, QAed–I QAed it myself.”

“Whew, that’s a relief. I was afraid it hadn’t been QAed. Okay, deploy it!”

So the story goes into production and blows up spectacularly, because as a developer I suck at QA. In a case like that, my effort at QA turned out to be worse than no effort at QA: because if there had been no effort, everybody would have understood that the QA hadn’t been done, rather than thinking it had been because I said so.

14 01 2012

I would agree with George and Norton on this and also push it just a bit notch further. I would put the BA in the mix for a while as well by having a “trio” or “three amigo” session to create tests. To automate the tests we were fortunate to find an intern (who later joined the QA team) to focus on automation using Ruby / Watir.

14 01 2012
Lanette Creamer

I’ve had good results with the Three Amigo sessions, but I’ve mostly worked on teams with NO BA’s or anything similar, so in those cases a PO or in a pinch a tech support person or a UX person can help.

14 01 2012
George Dinwiddie

I’d say the Product Owner is the right person for the Three Amigos. But as long as the business viewpoint is represented by someone knowledgeable and engaged, it’ll work.

29 02 2012
Dan Wiebe

On my teams so far, there’s never been a clear distinction between BA and QA. (In the real world, I mean: on paper, there’s all kinds of distinction, but when the rubber hits the road they end up doing the same job.)

14 01 2012
Ilya Kurnosov

I’ve got a question that is not directly related to the title question of the post. You write that “exploratory testing is the crowning glory of an Agile tester”. In the same time you write that “nobody should have to do [manual regression testing]–except perhaps after a system-wide refactoring”.
This puzzles me a wee bit: do you assume there can’t be exploratory regression testing? Is it some Agile restriction (or benefit, for that matter)?

29 02 2012
Dan Wiebe

The way I’m using the term, it’s a restriction of definition, not a restriction of Agile.

Regression testing, to me, means testing things that have already been tested to make sure they still work the way they used to work.

Exploratory testing, to me, means testing things that have never been tested before to see if they work at all.

So “exploratory regression testing,” in that semantic environment, would mean something like “testing things that have never been tested before to make sure they still work the way they used to work,” which doesn’t make any sense.

It’s kind of like saying “Over and out!” at the end of a radio transmission, where “Over” means “I will await your reply,” and “Out” means “I expect no reply.”

If exploratory testing finds a problem, I think an automated test should be written to prove that that problem no longer exists, and then the problem should be squashed until the test passes, and the test should become part of the regular build process. Therefore, no manual regression testing because it’s all automated.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: