Spluttering With Indignation

26 03 2015

I got so angry at the late Sun Microsystems today that I could barely find words.  I’ll tell you why in a minute.

So…I’ve been working on some spike code for a system to generate webservice mocks (actually combination stub/spies, if I have my terminology correct) to use in testing clients.

Here’s the idea.

If you have the source code for an existing web service, or if you’re developing a new web service, you use @MockableService and @MockableOperation annotations to mark the classes and methods that you want mocks for.  Then you run a command-line utility that searches through your web service code for those annotations and generates source code for a mock version of your web service that has all the marked services and operations.

Consider, for example, one of these @MockableOperations that responds to an HTTP POST request.  The example I used for my spike was hanging ornaments on a Christmas tree.  The URL of the POST request designates a limb on a tree, and the body of the request describes in JSON the ornaments to be added to that limb.  The body of the response contains JSON describing the resulting total ornamentation of the tree. The generated mock version of this operation can accept four different kinds of POST requests:

  1. X-Mockability: clear
  2. X-Mockability: prepare
  3. X-Mockability: report
  4. [real]

“X-Mockability” is a custom HTTP header I invented just for the purpose of clandestine out-of-band communications between the automated test on the front end and the generated mock on the back end, such communications to be completely unbeknownst to the code under test.

If the incoming POST request has X-Mockability of “prepare” (type 2 above), the generated mock knows it’s from the setup portion of the test, and the body of the request contains (encapsulated first in Base64 and then in JSON) one or more HTTP responses that the mock should remember and then respond with when it receives real requests with no X-Mockability header (type 4 above) for that operation.

If the incoming POST request has X-Mockability of “report” (type 3 above), the generated mock knows it’s from the assert portion of the test, and it will send back an HTTP response whose body contains (again, encapsulated in Base64 and JSON) a list of all the real (type 4) HTTP requests that operation has recently received from the code under test.

If the incoming POST request has X-Mockability of “clear” (type 1 above), the generated mock will forget all about the client sending the request: it will throw away all pending prepared responses and all pending recorded requests.  In general, this is the first request a test will make.

And, of course, as has been mentioned, if there is no X-Mockability header at all, the generated mock knows that the request is genuine, having originated from code under test, and it responds with the prepared response that is next in line, or–and this is key for what follows–a 499 response (I made up that status code) saying, “Hey, I’m just a mock and I’m not prepared for that request!” if there is no prepared response.

Pretty cool, right?  Can you see the problem yet? No?  Don’t feel bad; I hadn’t by this point either.  Let me go a little further.

I wrote a sample client that looked at a pre-decorated Christmas tree (using GET on the tree limb and expecting exactly the same kind of response that comes from a POST) and decided whether the ornamentation was Good, TopHeavy, BottomHeavy, or Uneven.  Then I wrote an automated test for the client, meant to run on a mock service. The test used “X-Mockability: prepare” to set up an ornamentation response from the GET operation that the code under test would judge to be BottomHeavy; then it triggered the code under test; then it asserted that the code under test had indeed judged the ornamentation to be BottomHeavy.

How about now?  Do you see it?  I didn’t either.

When I ran the test, it failed with my made-up 499 status code: the generated mock thought it wasn’t prepared for the GET request. Well, that was weird.  Not too surprising, though: my generated mock has to deal with the facts that A) it may be called from a variety of different network clients, and it has to prepare for and record from each of them separately; B) the webserver in which it runs may decide to create several instances of the mock to use in thread pooling, but it still has to be able to keep track of everything; and C) each of the @MockableOperations has to be administered separately: you don’t want to send a GET and have the response you’d prepared for a POST come back, and you don’t want to send a GET to one URL and receive the response you’d prepared for a GET to a different URL.

That’s a fair amount of complexity, and I figured I’d gotten the keying logic wrong somewhere, so that my preparations were ending up somewhere that the real request couldn’t find them.

So I put in a raftload of logging statements–which I probably should have had in from the beginning: after all, it’s for testing, right?–and tried it again.

Turns out that when the real request came in, the generated mock really truly honestly wasn’t prepared for a GET: instead, it was prepared for a POST to that URL instead.

Huh?  A POST?  But I prepared it for a GET, not a POST.  Honest.

I went and looked at the code, and logged the request method several times, from the initial preparation call in the test right up to Apache’s HttpClient, which is the library I was using to contact the server.  The HTTP method was GET all the way.

Me, I was just getting confuseder and confuseder.  How about you?  Have you beaten me to the conclusion?

The problem is that while this system works just fine for POST and PUT requests, there’s a little issue with GET, HEADER, and DELETE requests.

What’s the issue?  Well, GET, HEADER, and DELETE aren’t supposed to have bodies–just headers.  It’s part of the HTTP standard.  So sending an “X-Mockability: prepare” version of any of these three kinds of requests, with a list of canned responses in the body, involves stepping outside the standard a bit.

If you try using curl to send a GET with a body, it’ll be very cross with you.  If you tell SoapUI that you’re preparing to send a GET, it’ll gray out the place where you put in the body data.  If you already have data in there, it’ll disappear.  So I figured it was fair to anticipate some recalcitrance from Apache HttpClient, but this was more than recalcitrance: somehow, somewhere, my GET was turning into a POST.

I did some packet sniffing.  Sure enough, the server was calling the POST operation because it was getting a POST request over the wire.  Everything about that POST request was exactly like the GET request I wanted to send except for the HTTP method itself.

I tried tracing into HttpClient, but there’s a lot of complexity in there, and TDD has pretty much destroyed my skill with a debugger.

I didn’t need all the capabilities of HttpClient anyway, so I tossed it out and tried a naked HttpURLConnection instead. It took me a few minutes to get everything reassembled and the unit tests passing, but once I did, the integration test displayed exactly the same behavior again: my HttpURLConnection.getMethod () was showing GET right up until the call to HttpURLConnection.getOutputStream () (which includes a call to actually send the request to the server), but the packet sniffer showed POST instead of GET.

HttpURLConnection is a little easier to step into than HttpClient, so I stepped in, and finally I found it, in sun.net.www.protocol.http.HttpURLConnection.  Here is the offending code, in a private method called from getOutputStream ():

    private synchronized OutputStream getOutputStream0() throws IOException {
        // ...
        if(this.method.equals("GET")) {
            this.method = "POST";
        // ...

See that?

Sun Microsystems has decided that if it looks like you want to add a body to a GET request, you probably meant to say POST instead of GET, so it helpfully corrects you behind the scenes.


Now, if it doesn’t want you putting a body in a GET request, it could throw an exception.

Does it throw an exception?


It could simply fail to send the request at all.

Does it simply fail to send the request at all?


It could refuse to accept a body for the GET, but give you access to a lower level of operation so that you can put the body in less conveniently and take more direct responsibility for the consequences.

Does it do that?


You can’t even reasonably subclass HttpURLConnection, because the instance is constructed by URL.openConnection () through a complicated service-provider interface of some sort.

This sort of hubris fills me with such rage that I can barely speak.  Sun presumes to decide that it can correct me?!  Even Apple isn’t this arrogant.

So I pulled out HttpURLConnection and used Socket directly, and I’ve got it working, sort of: very inconvenient, but the tests are green.

Unfortunately, Sun isn’t the only place we see practices like this.  JavaScript’s == operator frequently does things you didn’t ask for, and we’ve all experienced Apple’s and Android’s autocorrect mechanism putting words in our…thumbs…for us.

But at least JavaScript has a === operator that behaves, and you can either turn autocorrect off, or double-check your text or tweet to make sure it says what you want before you send it. Sun doesn’t consider it necessary to give you a choice; it simply pre-empts your decision on a whim.

I guess the lesson for me–and perhaps for you too, if you haven’t run into something like this already–is: don’t correct your clients.  Tell them they’re wrong, or refuse to do what they ask, or let them choose lower-level access and increased responsibility; but don’t assume you know better than they do and do something they didn’t ask you to do instead of what they did ask you to do.

They might be me, and I might know where you live.




Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: