The Evolution of Testing in the Age of Serverless

Introduction

As organisations adopt microservices for rebuilds and new applications, many will realise that the established ‘Test Pyramid’ approach to testing is considerably less effective than it was with traditional applications.

The Test Pyramid emphasises unit tests that focus on the application’s individual functions and code. These comprise the bulk of this testing strategy, reflected in their position in the larger bottom pyramid section. Unit tests are followed by integration tests that validate the interactions between two or more connected components. While contract testing is not officially included in this strategy, we have added it in the same section as Integration testing for awareness purposes. We will address contract testing later in this article. At the top, and the smallest part of the pyramid, are the end-to-end tests, also called E2E, which test the output of application workflows. As we move up the pyramid, the cost and effort required to develop and run each type of test increases. This is one of the reasons unit tests are preferred, as lower effort and cost typically translates to better developer productivity.

Figure 1: Typical Test Pyramid

However, when we switch to a microservices architecture, we find that unit tests do not provide sufficient application coverage. Compared to traditional applications, microservice applications tend to have less code and more workflows of many connected consumers (services that initiate a request for external services) and providers. Consumers are services that initiate a request for external services. For example, a checkout service that calls an external payment service to process payments. Providers are services that receive and process a request. In our previous example, the payment service will receive the request, process the payment, update the database and then return the payment status back to the consumer.

As such, we need to prioritise integration tests over unit tests. End-to-end or E2E tests are of similar importance to unit tests in this architecture. These preferences resulted in the development of the ‘Test Honeycomb’. In this design, the integration tests in the middle are now the largest section, and the smaller unit and E2E sections are equal in size. As with the pyramid testing strategy, contract testing is not officially mentioned, but we have again added it in the integration testing section for awareness purposes.

Figure 2: Test Honeycomb

Challenges with Integration and E2E Tests

While Integration and E2E tests can provide better microservice testing coverage, they come with their own challenges, such as dependence on external resources and state.

When a component changes the structure of its request or response payload – the messages it exchanges with other components – it is difficult to determine the change’s impact on any components that depend on it. One mitigation is to run integration tests on all connections to identify any failures resulting from the change. Depending on the architecture, this can potentially require a significant number of tests to be executed.

For example, if we have ten components interacting with one another, we could potentially end up with 90 integration points to test – if connectivity is two-way. Realistically, it will be closer to 15 integrations in most well-architected microservice designs. Still, maintaining a list of integrations and ensuring compatibility across dependencies can be challenging as the application grows.

Figure 3: Each Microservice Can Be Dependent on Multiple Providers and Have Multiple Consumer Dependencies

To add, all components, including any services they depend on, and their connections must be updated and available for the tests to pass. A common approach to microservice development is for multiple developers or small teams to work on multiple microservices in parallel. This approach further complicates the challenge of needing all components to be available for testing. Besides synchronising updates across teams, significant coordination is needed to plan deployments so the required tests can be performed before the changes go to production.

To address these challenges, we need a solution that can eliminate the dependency on external services for tests to be conducted, and at the same time, still enable developers to confidently test their changes before deploying them to production.

Consumer-Driven Contract Testing

Consumer-Driven Contract testing is an approach to test component integrations that addresses the challenges of Integration and E2E tests. When services interact with one another, the consumer is the service that sends a request. The provider receives and processes the request. Depending on the workflow design, the provider may or may not return a response to the consumer.

Contract testing uses a contract to define the structure and rules between consumers and a service provider. A contract specifies and maintains a list of templates of each request and response.

During testing, the consumer or provider runs its tests against each contract to verify that new changes do not break anything on either end. On the consumer side, it is responsible for defining the contract by writing test cases to verify itself against its expected requests and responses. The consumer will then generate and publish a contract based on the test cases once the tests are successful.

On the provider side, it then picks up all the contracts published by its consumers. The provider then picks up the contract and verifies that it is compliant with the contract. Once the tests finish, the provider then publishes the results.

With Contract Testing, developers can run all their tests in isolation. The consumer does not need to wait for the provider to finish its implementation to run its test. Instead, the consumer defines the expected responses, and the developers of the provider can implement them in their own time. While this allows developers the flexibility to manage their time during development and testing (e.g., sprint planning), there is still a hard deadline for when services (e.g., milestones) need to be running before deploying into production.

Pact

Pact is a popular open-sourced tool for running Consumer-Driven Contract Tests. Pact provides a suite of libraries for consumers and providers to build their test cases and a broker for maintaining the collection of contracts.

With Pact contract testing, the focus is on what – the request or response payload, that is sent between consumers and providers, and not the how – such as specific messaging services or formats. Pact uses REST APIs to receive and send contracts between the consumers and providers.

The value of Pact lies in it providing a platform that providers and consumers trust and, facilitating the agreement and verification of contracts between them. It aims to solve a people challenge – collaborating effectively during integration tests, rather than any particular technical challenge.

Consumers publish a contract that defines the expected response required from the provider. As a provider, it may have multiple consumers using it for various purposes. Let’s take an example of a customer making an order on an e-commerce website. When the order is submitted, several consumer microservices, such as payment, customer loyalty and warehouse inventory, would be interested to know about it. Each consumer will have its own set of values they would like to receive from the provider, which is defined in a contract. The role of the Pact broker is to collect all these contracts that were created by the consumers to send to the provider. The provider then uses these contracts to verify that the response it returns matches what is expected by the consumers.

Figure 4: Pact Workflow

With this system in mind, consumers and providers can work at their own cadence while confident that new updates will not break existing integrations within the application. The consumer developer can create and test new features without waiting for the provider, and the provider developer can prioritise the list of features, updates and fixes for the service. Again, keep in mind that this flexibility only applies within a project milestone – all service updates will still need to be deployed into production for the application to work.

Challenges

Pact grew out of a container microservices environment where a microservice-to-microservice communication pattern is more commonly used. However, such a pattern is considered an anti-pattern for Serverless microservices. In Serverless, messages from a provider are passed to a managed message router such as Amazon SQS, SNS or EventBridge, which forwards them to one or more consumers. This raises questions about where the Pact broker should sit in this architecture and which request/response payloads we should test.

Besides message routers, Serverless generally prefers using purpose-built managed services such as API Gateway and DynamoDB over developing custom code. There is limited value in writing contract tests for managed services since they have stable APIs, which we are unlikely able to influence even if the test fails. The information about the availability and implementation of these services is usually provided in their respective documentation.

Figure 5: Challenges of Contract Testing with Serverless architecture

Serverless Contract Testing

Despite the challenges mentioned, we still can find value with Contract Testing by focusing on the content of these messages instead of how and to whom the messages are being sent.

Let’s walk through a simple example of how we can implement Contract Testing using Pact to get a better idea of how to implement Contract Testing for Serverless. Do note that we are not doing a deep dive into Pact for this article, instead we are exploring the concepts of using Contract Testing for Serverless projects.

Figure 6: Focus of Contract Testing

Step 1: Consumer defines the contract

As mentioned earlier, in a Consumer-Driven Contract Test, the consumer is responsible for determining the contract between itself and the provider.

In this example, the consumer defines a test case that specifies the following when the receiveTransactionUpdate() method is executed on the consumer:

  1. Test case (“successful transaction event”) that should be triggered on the provider’s end
  2. Content that the provider should verify (transaction_id, customer_id and status)
describe("transaction complete", () => { 
  it("accepts a successful transaction", () => { 
    const pact = messagePact 
      .expectsToReceive("successful transaction event") 
      .withContent({ 
        transaction_id: like("1111-2222-3333-4444"), 
        customer_id: like("9999-8888-7777-6666"), 
        status: "SUCCESS", 
      }) 
      .verify(asynchronousBodyHandler(receiveTransactionUpdate)); 
 
    return pact 
  }); 
});

Step 2: Consumer publishes the contract

Once the developer has completed the test cases, the next step is to publish the contract to the broker. First, the Pact library will compile a list of all the test cases to create a contract, like the JSON file we see below.

Once the contract is created, the developer can add other attributes such as tags to define the feature branch/environment and the version information before uploading it to the broker.

"messages": [ 
  { 
    "description": "successful transaction event", 
    "contents": { 
      "transaction_id": "1111-2222-3333-4444", 
      "customer_id": "9999-8888-7777-6666", 
      "status": "SUCCESS" 
    }, 
    ... 
  } 
]

Step 3: Provider retrieves and verifies the contract

Whenever a consumer publishes or updates a contract, a Webhook can be configured to run automated builds or notify the provider’s developer.

To ensure that new consumer contracts do not break any existing builds, the provider can filter the tests to run based on the defined tags in the contracts. For example, we can configure the Pact library to only run tests against all consumer contracts tagged with “team_a_tests” to limit the scope of the tests.

The provider initialises the Verifier class in the Pact Library to run the test. It will automatically retrieve all the contracts associated with the provider class and run all the tests.

Here, the provider retrieves the “successful transaction event” test case published by the consumer in the contract. The provider then writes the implementation of the code and test it against the test case.

Here are two possible reasons why a contract could fail:

  1. There was a code change in the provider class that breaks existing consumers
  2. A consumer changed its expected payload, resulting in a newly-generated contract

To resolve the failed test cases, the provider will either have to modify their code or reach out to the respective consumer to re-align the requirements, such as requesting justifications for a change in contract.

const opts = { 
  ...baseOpts, 
  ...(process.env.PACT_URL ? pactChangedOpts : fetchPactsDynamicallyOpts), 
  messageProviders: { 
    "successful transaction event": () => { 
      response = createEvent(new Transaction("1111-2222-3333-4444",  
                                             "9999-8888-7777-6666",  

       "SUCCESS")) 
      return response 
    } 
  }, 
}; 
 
 
const p = new MessageProviderPact(opts); 
 
describe("successful transaction", () => { 
  it("successful transaction event", () => { 
    results = p.verify(); 
 

    return results 
  }); 
});

Step 4: Provider publishes the contract

Once the provider completes the contract, the results are published to the broker. The results of the tests can be accessed via the broker’s web interface.

Alternatively, the provider or consumer can check if their changes are ready to be deployed by running the can-i-deploy command found in the Pact library to determine if all dependencies and dependents have been successfully validated.

> pact-broker can-i-deploy \
              --pacticipant MyConsumerService \
              --broker-base-url http://localhost:8000 \
              --broker-username pact_workshop \
              --broker-password pact_workshop \
              --version 000000001
Computer says yes \o/
Conclusion

We want to emphasise that the conventional approach to testing with a strong emphasis on Unit Tests is not the best option for Serverless. Using a Microservice architecture greatly increases the number of integrations while reducing the amount of code developers need to test to ensure sufficient coverage.

Integration and E2E tests are preferred for this architecture, but they come with challenges such as all components needing to be available for testing, which can be logistically challenging when multiple teams are working on a single application. Contract testing can help address these challenges. Providers don’t need to maintain a list of consumers, and they can react to breaking changes before the code is deployed. This helps shorten development time, improve the quality of tests and increase developer productivity.

Additional Resources

Pact
More details on Pact can be found in this documentation.

Code Examples 
The following are two code examples provided by Pact which can help developers better understand how the solutions work in an Amazon Web Services (AWS) environment.

Thomas is a Lead Consultant at Sourced with over 18 years of experience in delivering digital products. He gained a passion for evangelising Serverless architecture through his enthusiasm for cloud and entry into the AWS Ambassador programme.

KangZheng is an up-and-coming AWS Cloud Engineer with two years of professional experience in architecting and implementing cloud solutions. He enjoys researching the latest technology trends and applying them to client projects.

Interested to hear more?

Menu