Mocking HTTP Services in Go

Let’s explore the basics of writing tests, which access external HTTP resources. Scope of this article are APIs, but the approach can be extended to fetching other types of resources via HTTP.

This article would be useful for people learning the Go programming language from the beginning as well as for people who are coming from different OOP-gravitating ecosystems, and trying to wrap their head around Go-specific concepts.

Software mentioned in this article: Go@1.14 , jarcoal/httpmock@1.0.6 , go-resty/resty@2.3.0 .

Basics of Testing in Go

Go has a bit of a unique approach to writing tests. Instead of writing data providers and using them as input in a test method, Go encourages the concept of Table-Driven Tests, where each method under test (MUT) has usually one test function and test inputs are located inside method body.

Concept of Table Driven-Tests is proposed by authors of Go themselves. More details can be found on the official Go GitHub Wiki. In a nutshell they look like this:

var flagtests = []struct {
  in  string
  out string
}{
  {"%a", "[%a]"}, // arrange
  ...
  {"%-1.2abc", "[%-1.2a]bc"},	
}
func TestFlagParser(t *testing.T) {
  var flagprinter flagPrinter
  for _, tt := range flagtests {
    t.Run(tt.in, func(t *testing.T) {
      s := Sprintf(tt.in, &flagprinter) // act
      if s != tt.out {
        t.Errorf("got %q, want %q", s, tt.out) // assert
      }
    })
  }
}

Mocking HTTP Calls

Almost every application in one way or another communicates with external resources, which should be mocked by synthetic tests. One of the available packages for such purpose in Go ecosystem is called jarcoal/httpmock.

It is fairly straightforward to use, but the devil is in the detail, and figuring out a proper setup for a test might take some time.

Let’s look at the the hello-world example of a mocked HTTP response:

func Test_MinimalCase(t *testing.T) {
 httpmock.Activate()
 defer httpmock.DeactivateAndReset()
 httpmock.RegisterResponder(
  "GET", 
  "https://example.com", 
  httpmock.NewStringResponder(200, "resp string")
 )
 resp, _ := http.Get("https://example.com")
 defer resp.Body.Close()
 body, _ := ioutil.ReadAll(resp.Body)
 assert.Equal(t, "resp string", string(body))
}
// Result:
// ok

In this example we mock standard net/http package response to an example website, which should return us a resp string response.

Alternatives to net/http Package

Other network packages provide abstractions and convenience functions in order to simplify communication between code and the outer world. The net/http package itself is low-level and boilerplate code is needed to make it work properly.

In the following example we would use go-resty/resty package, which provides us a high level API to communicate with network resources:

func Test_MinimalResty(t *testing.T) {
 rst := resty.New()
 httpmock.ActivateNonDefault(rst.GetClient())
 defer httpmock.DeactivateAndReset()
 httpmock.RegisterResponder("GET", "https://example.com", httpmock.NewStringResponder(200, "resp string"))
 resp, _ := rst.R().Get("https://example.com")
 assert.Equal(t, "resp string", resp.String())
}

In order to mock Resty calls its client instance have to be registered with httpmock in a different way by calling the ActivateNonDefault method. Also there is no need to use ioutil to convert response to a string: it’s done now by Resty.

Difficulties of Mocking an API call

The function responsible for returning an object called Responder.
A Responder is a mock object which intercepts an HTTP call and returns a fixture as a response.httpmock comes with multiple Responders, which encourage multiple ways of mocking a response. In the next sections we will cover some (but not all) of them.

The minimum arrangement (remember AAA pattern in unit testing?) needed for a proper test which have an external dependency:

• Mock Response — to set exactly what we want to return
• Mock Request URL — a url which we want to intercept (httpmock supports wildcards as well)
• Input parameters — a set of parameters which would trigger the correct url and response from previous points

Try 1: httpmock.NewJsonResponder

NewJsonResponder creates a Responder from a given body (as an interface{} that is encoded to json) and status code.

Looks pretty easy, let’s try to use it:

httpmock.RegisterResponder(
  "GET", 
  "https://example.com/mynameis",
  httpmock.NewJsonResponderOrPanic(200, `{"name":"Oleg"}`)
)

The expectation is that you provide a valid JSON, which is supposed to be returned as a mock in a response. Nevertheless, it doesn’t work, why?

--- FAIL: Test_JSONDummyJsonResponder (0.00s)
...
expected: &dummy.JSONDummy{Name:"Oleg"}
actual  : (*dummy.JSONDummy)(nil)
...

Methods NewJsonResponder/NewJsonResponderOrPanic accept interface{} as an input and directly call json.Marshal(body) using it. This function is a standard function from the package encoding/json and encodes anything to be a valid JSON, even if the input is already a valid JSON string.

Since we’ve already provided a JSON string, thus the string is encoded for a second time. Example of such condition using only encoding/json package:

package main
import (
 "fmt"
 "encoding/json"
)
func main() {
 data, _ := json.Marshal(`{"name":"Oleg"}`)
 fmt.Println(data)
 fmt.Println(string(data[:]))
}
// Output:
// [34 123 92 34 110 97 109 101 92 34 58 92 34 79 108 101 103 92 34 125 34]
// "{\"name\":\"Oleg\"}"

Go playground: https://play.golang.org/p/F382pZzPpC0

As you can see, the second line of the output is converted back from bytes and escaped to be a valid json-encoded string. Such string cannot be decoded (unmarshalled) by thejson.Unmarshall function.

The proper way of using such function would be to provide a structure which can be marshalled to a valid json:

package main
import (
 "fmt"
 "encoding/json"
)
type Person struct {
 Name string
}
func main() {
 data, _ := json.Marshal(Person{Name: "Oleg"})
 fmt.Println(data)
 fmt.Println(string(data[:]))
}
// Output: 
// [123 34 78 97 109 101 34 58 34 79 108 101 103 34 125]
// {"Name":"Oleg"}

Go playground: https://play.golang.org/p/B6T0ldxyoDm

Changing our test input to a structure verifies the previous example and makes our example test green:

type JSONDummy struct {
  Name string `json:"name"`
}
...
httpmock.RegisterResponder(
  "GET", 
  "https://example.com/mynameis",
  httpmock.NewJsonResponderOrPanic(200,
  &JSONDummy{Name: "Oleg"})
)
...
// Result:
// ok

Passing a struct instead of a string have some disadvantages:

• It involves a conversion step to create a final response string
• It is not possible to test incorrectly typed data since structs are typed
• It is not possible to test missing / additional fields since marshalling would provide a standard object with field names

Try 2: httpmock.NewStringResponder

NewStringResponder creates a Responder from a given body (as a string) and status code.

If there is no way to avoid marshalling, we can try using different Responder functions. One of them is a StringResponder.

Let’s try to pass our JSON string to a string responder and run a test:

httpmock.RegisterResponder(
  "GET", 
  "https://example.com/mynameis", 
  httpmock.NewStringResponder(200, `{"name":"Oleg"}`)
)

The test fails, due to the response struct being empty:

--- FAIL: Test_JSONDummyStringResponder (0.00s)
...
expected: &dummy.JSONDummy{Name:"Oleg"}
actual  : &dummy.JSONDummy{Name:""}
...

The reason for the failure hides in the way StringResponder constructs a mock response. In comparison to theJsonResponder the StringResponder lacks a response header Content-Type: application/json.

If such header is missing or is not matching xml or json content types, the string response would be provided as-is and will not be unmarshalled.

Solution: Custom Responder Configuration

Seems like the package does not have a proper Responder for our needs, so let’s make one ourself and put it next to the test:

func newResponder(s int, c string, ct string) httpmock.Responder {
 resp := httpmock.NewStringResponse(s, c)
 resp.Header.Set("Content-Type", ct)
return httpmock.ResponderFromResponse(resp)
}

The function accepts one additional parameter: Content-Type header. Providing there theapplication/json header, allows us to use string as an input and get a structure on the output.

Here is how it works:

type JSONDummy struct {
 Name string `json:"name"`
}
func Test_MininalCustomJSON(t *testing.T) {
 rst := resty.New()
 httpmock.ActivateNonDefault(rst.GetClient())
 defer httpmock.DeactivateAndReset()
 httpmock.RegisterResponder(
  "GET",
  "https://example.com",
  newResponder(200, `{"name":"Oleg"}`, "application/json"),
 )
 resp, _ := rst.R().
  SetResult(&JSONDummy{}).
  Get("https://example.com")
 assert.Equal(t, &JSONDummy{Name: "Oleg"}, resp.Result().(*JSONDummy))
}
// Result:
// ok

As you can see it’s automatically unmarshalls the response to a struct.

Bonus: Mocking an XML Response

Both httpmock and (surprisingly looking at the name) Resty can unmarshall XML documents. They both use standard encoding/* packages under the hood. This allows to use XML as a response and unmarshall it to a valid struct on the output.

type XMLDummy struct {
 Name string `xml:"Name"`
}
func Test_MininalCustomXML(t *testing.T) {
 rst := resty.New()
 httpmock.ActivateNonDefault(rst.GetClient())
 defer httpmock.DeactivateAndReset()
 httpmock.RegisterResponder(
  "GET",
  "https://example.com",
  newResponder(
   200,
   `<root><Name>Oleg</Name></root>`,
   "application/xml",
  ),
 )
 resp, _ := rst.R().
  SetResult(&XMLDummy{}).
  Get("https://example.com")
 assert.Equal(t, &XMLDummy{Name: "Oleg"}, resp.Result().(*XMLDummy))
}
// Result:
// ok

Mocking Error Responses

Previous examples show only happy path examples. Let’s explore how we can mock various HTTP error responses.

Our table test setup have exactly that:

tests := map[string]struct {
  data     string
  path     string
  respCode int
  want     *XMLDummy
  wantErr  bool
 }{
  "error response": {
   data:     ``,
   path:     "https://example.com/mynameis",
   respCode: 500,
   want:     nil,
   wantErr:  true,
  },
 }

In this table we use simple wantErr flag and respCode: 500, which indicates that we expect an error and directs Responder to return a 5xx error. It is also possible to replace it with a specific error type the test expects (examples of such arrangement could be found in the full source code).

if (err != nil) != tt.wantErr {
 t.Errorf("GetXML() error = %v, wantErr %v", err, tt.wantErr)
 return
}

In the assertion part of the test we simply check for an error, and if it is not present we fail the test.

Wrap-Up

As we have seen, it might be tricky to correctly setup mocks from scratch. At the time the article was written, any error in Responder configuration would not trigger any error and just simply render an empty result.

Hope this article will save you time ;)

Full Source Code

Link to Gist

Or be

That’s all folks!