Scale Up Your E2E Tests Using Mock Server
Introduction
In this article, I will discuss the benefits of using a mock server when running E2E tests in distributed systems and introduce Cornell: record & replay mock server written in python
The Pains of E2E tests in CI/CD World
It’s pretty much established that testing is crucial for quality. Even more so if you’re implementing CI/CD. Insufficient tests or low test coverage may keep you awake dreading at night or make life hard for the unlucky on-callers in your org.
While unit tests are important during development, E2E (end to end) tests are crucial when you need to integrate with multiple web-based services in complex microservice architecture.
When you’re working with distributed systems, the test client entry point triggers a cascade of events that eventually send HTTP requests to an external server. The entry point of the test may be on a different process, thread or even machine from the actual code in the test.
Approaches to E2E test development
Use external real servers as in production
This approach is the most straightforward and simple. Your system integrate with external service? No problem, use the same service in your tests. No additional mocking effort is required.
This approach can work for very small applications that don’t require massive testing and usually don’t have CI.
Once you scale up, coupling your tests with real services will have obvious drawbacks. Running tests which call the external service every time might be slow and error prone. The tests won’t work offline and not all vendors will be happy to receive so many requests (CI running on every push to branch, might result in thousands or more request in an hour)
- Develop your own Simulator Server
In case you’re working only with a single external service, developing a dedicated robust simulator server could be a way to go for this use case.
Keep in mind that developing such a simulator will require a lot of maintenance, since you will actually be implementing a lot of logic (i.e. authentication, various api calls) similar to the original server.
In case you’re working with multiple vendors, maintaining an in house mock for them all, might quickly become tedious and entangled.
Record & replay mock server
- Using vcrpy unit tests
Before diving into the mock servers, let’s review some of the vcrpy main functionality
vcrpy is an awesome library that records and replays HTTP interactions for unit tests. Its output is saved to reusable “cassette” files.
Example for unit test with vcrpy
@vcr.use_cassette("my_cassette.yaml")
def test_url(): res = requests.post("https://www.somexyz.org/api/message/? access_token=ACCESSTOKEN", data={'subject': "test_subject", 'message': "test_message"}) res.raise_for_status() assert res.json() == {"string": "success"}
Running the above test for the first time, will interact with the external server (`”https://www.somexyz.org`) and will record the request and the response interactions in the dedicated yaml cassette in our example, it will be called my_cassette.yaml and will be saved in the test root dir.
Running the same test, after the initial recording, will use the recorded cassette and will not approach the server itself.
This lightweight and easy approach will cover most of your tests that require interactions with external web services and API.
For unit tests, the problem is solved almost perfectly, just use vcrpy ❤
- Mocking in distributed systems
But what about E2E tests you may ask. Simple decorator won’t do much good in this case. The test triggers multiple processes and starts (often) a complex flow of events, that only at some point will approach the external vendor. For this purpose you’d need to start your own mock server and replace it with the external entity.
Replacing an external web service with a mocked one, will provide the robustness, the speed and the isolation you need for your E2E tests.
Cornell: record & replay mock server in Python
By wrapping vcrpy with Flask, Cornell provides a lightweight record and replay server that can be easily used during distributed system testing and simulate all HTTP traffic needed for your tests.
With the Cornell server started, it will act as a proxy (record mode) between the outgoing HTTP requests and the external server and will record all relevant interactions. Once interactions are recorded, Cornell can work in replay mode, replacing the external server entirely, short-circuiting the calls and instead, replying back instantly with the previously recorded response.
Cornell Main Features
Record API Interaction
An HTTP proxy mode in which all interactions are recorded and saved into “cassettes” for future use.
Replay for Testing
Use your library of “cassettes” to replay pre-recorded responses instead of hitting the real API endpoint. Save time on testing cycles and avoid flakiness by using a consistent response.
Extend Cornell
Designed with flexibility in mind. Easily extend and customize the default behavior.
Cornell is super easy to use out-of-the-box. Once installed, can be used right away, without much configuration (just don’t forget to change the external URL to Cornell localhost:9000).
And there’s so much more! Check out Cornell documentation and the repo
Alternatives
Currently, there are several out-of-the-box solutions for such mock services:
GitHub — patrys/httmock: A mocking library for requests: lightweight and easy to setup mock server, that allows you to add synthetic data for responses, matched to specified request. Can be perfect for simple and a specific requests and responses
https://github.com/flickr/yakbak: server written in javascript and also inspired by ruby vcr and aimed for node js developers
Personal Note
Cornell is a python lightweight project with lots of potential. For all of you who are passionate about python, testing and open source, it can be a great opportunity for contributing your code to the community.
