23 November 2023
14 Min. Read
Contract Testing
PACT Contract Testing: A Step by Step Guide
Key Highlights
In this blog, we cover the following key highlights:
Gain insights into the effectiveness of PACT contract testing as a tailored approach for testing microservices.
Follow a detailed, step-by-step guide to conducting contract testing using PACT.
Understand the challenges associated with PACT contract testing.
Explore how HyperTest addresses and resolves issues related to manual intervention in PACT testing.
In our previous contract testing article, we covered the basics of what contract testing is and how it works. Now, in this blog post, we'll introduce you to a popular tool for contract testing—PACT Contract testing.
What is PACT contract testing?
Contract tests combine the lightness of unit tests with the confidence of integration tests, and should be part of your development toolkit.
PACT is a code-based tool used for testing interactions between service consumers and providers in a microservices architecture. Essentially, it helps developers ensure that services (like APIs or microservices) can communicate with each other correctly by validating each side against a set of agreed-upon rules or "contracts".
Here's what PACT does in a nutshell:
It allows developers to define the expectations of an interaction between services in a format that can be shared and understood by both sides.
PACT provides a framework to write these contracts and tests for both the consuming service (the one making the requests) and the providing service (the one responding to the requests).
When the consumer and provider tests are run, PACT checks whether both sides adhere to the contract. If either side does not meet the contract, the tests fail, indicating an issue in the integration.
By automating these checks, PACT helps teams catch potential integration issues early and often, which is particularly useful in CI/CD environments.
So, PACT focuses on preventing breaking changes in the interactions between services, which is critical for maintaining a reliable and robust system when multiple teams are working on different services in parallel.
Importance of PACT Contract Testing
PACT reduces the complexity of the environment that is needed to verify integrations, as well as isolates changes to the specific interaction between services. This prevents cascading failures and simplifies debugging.
Decoupling for Independence: PACT enables microservices to thrive on decoupled, independent development, testing, and deployment, ensuring adherence to contracts and reducing compatibility risks during the migration from monoliths to microservices.
Swift Issue Detection: PACT's early identification of compatibility problems during development means faster feedback, with precise, interaction-focused tests that expedite feedback and streamline change sign-offs.
Enhanced Collaboration and Confidence: Clear, shared service interaction contracts reduce misunderstandings, fostering collaboration and developer confidence in releasing changes without breaking existing contracts.
Living Documentation: Pact contracts serve as dynamic, clear-cut documentation, simplifying developers' comprehension of integration points.
Reduced Service Outages: Pact contract tests swiftly highlight provider service changes that break consumer expectations, facilitating quick identification and resolution of disruptive modifications.
How does Pact implement contract testing?
Pact implements contract testing through a process that involves both the consumer and the provider of a service, following these steps:
Consumer Testing:
The consumer of a service (e.g., a client application) writes a test for the expected interaction with the provider's service.
While writing this test, Pact stubs out the actual provider service and records the expectations of the consumer—what kind of request it will make and what kind of response it expects—into a Pact file, which is a JSON file acting as the contract.
The consumer test is run with the Pact mock service, which ensures the consumer can handle the expected response from the provider.
Pact File Generation:
When the consumer tests pass, the Pact file (contract) is generated. This file includes the defined requests and the expected responses.
Provider Verification:
The provider then takes this Pact file and runs it against their service to verify that the service can meet the contract's expectations.
The provider's tests take each request recorded in the Pact file and compare it against the actual response the service gives. If they match, the provider is considered to be in compliance with the contract.
Publishing Results:
Results of the provider verification can be published to a Pact Broker, which is a repository for Pact files. This allows for versioning of contracts and tracking of the verifications.
Both the consumer and the provider use the Pact Broker to publish and retrieve Pact files. It helps to ensure that both parties in the service interaction are always testing against the latest contract.
Continuous Integration:
Pact is often integrated into the CI/CD pipeline. Whenever changes are made to the consumer or provider, the corresponding contract tests are automatically run.
This helps in identifying any breaches in the contract immediately when a change is made, ensuring that any integration issues are caught and addressed early in the development lifecycle.
Version Control:
Pact supports semantic versioning of contracts, which helps in managing the compatibility of interactions between different versions of the consumer and provider services.
By automating the creation and verification of these contracts, Pact helps maintain a reliable system of independent services by ensuring they can communicate as expected, reducing the likelihood of integration issues in a microservices architecture.
How to perform Pact Contract Testing?
Now we all know that Pact is a code-first tool for testing HTTP and message integrations using contract tests. Instead of testing the internal details of each service, PACT contract testing focus on the "contract" or the agreement between services on how their APIs should behave.
For this example, we have created a hypothetical scenario where a client app expects to fetch user data from a service.
Step 1: Define the Consumer Test
In the consumer service, you would write a test that defines the expected interaction with the provider's API.
Step 1.1: Import the Pact Library and Set Up Test Configuration
from pact import Consumer, Provider
# Define the consumer and provider objects
consumer = Consumer('ConsumerService')
provider = Provider('ProviderService')Here, we are defining the names of the consumer and provider services involved in the contract. This will be used to generate the Pact file later.
Step 1.2: Create a Pact Between Consumer and Provider
We have created an instance of a pact between our consumer and provider. We have also specified the port number where the Pact mock service will run. This mock service will simulate the provider's behavior according to the contract defined in our tests.
# Create a contract (Pact) between the consumer and provider
pact = consumer.has_pact_with(provider, port=1234)Step 1.3: Define the Expected Interaction
# Define the expected interaction
@pact.given('a user with id 1 exists')
@pact.upon_receiving('a request for user id 1')
@pact.with_request(method='GET', path='/user/1')
@pact.will_respond_with(status=200, body={'id': 1, 'name': 'John Doe', 'email': 'john.doe@example.com'})This is where we have defined the expected interaction with the Pact decorators:
@pact.given sets up the provider state required for the interaction. In this case, it's the state where a user with a specific ID exists in the system.
@pact.upon_receiving describes the expected request that the consumer will make. Here, it's a request to retrieve the user with ID 1.
@pact.with_request specifies the details of the request, including the HTTP method and the path.
@pact.will_respond_with defines the expected response from the provider, including the status code and the response body.
Step 1.4: Implement the Test Function
# Test to verify client-side of the contract
def test_get_user():
# Setup the mock service to expect the defined interaction
with pact:
# Here you would use your actual client code to make the request
# For the purpose of this example, assume 'client' is your HTTP client
result = client.get('http://localhost:1234/user/1')
# Assert that the response matches the expected result
assert result == {'id': 1, 'name': 'John Doe', 'email': 'john.doe@example.com'}This function test_get_user is the actual test that validates the consumer's ability to handle the interaction correctly. It does a few things:
It uses a context manager (with pact:) to start a mock service that will mimic the provider's API.
Inside the context, it performs the API call that we expect our consumer to make (client.get('<http://localhost:1234/user/1>')).
It then asserts that the response from the mock service matches what we've defined in our expected interaction.
Step 2: Run the Consumer Test
When this test is executed, the pact context manager starts the mock service, and the defined interaction is registered with it. Then, the test makes a request to the mock service, which checks that the request matches the registered interaction. If it does, it responds with the predefined response.
Step 3: Generate the Contract (Pact File)
If all assertions pass and the test completes successfully, Pact will generate a .json file representing the contract. This file is then used by the provider to verify that their API meets the expectations defined by the consumer.
{
"consumer": {
"name": "ConsumerService"
},
"provider": {
"name": "ProviderService"
},
"interactions": [
{
"description": "a request for user id 1",
"providerState": "a user with id 1 exists",
"request": {
"method": "GET",
"path": "/user/1"
},
"response": {
"status": 200,
"body": {
"id": 1,
"name": "John Doe",
"email": "john.doe@example.com"
}
}
}
],
"metadata": {
"pactSpecification": {
"version": "2.0.0"
}
}
}Step 4: Verify the Provider with the Pact File
The provider's test suite would use this .json Pact file to ensure their service can handle the requests and send the expected responses. The provider doesn't necessarily need to know the internals of the consumer; it just needs to satisfy the contract as outlined in the Pact file.
Here's a basic example of how the provider can verify the contract:
# This code would be part of the provider's test suite
# Assume 'provider_app' is the Flask/Django/Name-your-framework application object
from pact import Verifier
verifier = Verifier(provider='ProviderService', provider_base_url='http://localhost:1234')
# Assuming the pact file is available locally, or it could be fetched from a Pact Broker
verifier.verify_pacts('./pacts/consumer-service-provider-service.json')
# Run the provider application in a test context
with provider_app.test_client() as client:
# The actual provider verification happens here
verifier.run_provider_verification(client)The Verifier uses the pact file to make requests to the actual provider service and checks that the responses match the contract. If they do, the provider has met the contract, and you can be confident that the provider and consumer can communicate correctly.
Problems with PACT
If your primary goal is keeping contract testing simple and with lesser overheads, PACT may not be the ideal tool.
PACT contract testing has become very popular among teams off late given its simplicity and effectiveness. But it comes with its own set of challenges, making adoption at scale a challenge.
It’s not always straightforward, it demands a considerable amount of manual effort and time.
There are some obvious challenges in getting started and also the manual intervention in contract maintenance doesn’t make it the perfect fit for testing microservices alone.
👉Complex setup and high maintenance
👉CI/CD Pipeline Integration Challenges
👉High Learning Curve
👉Consumer Complexity
👉Test Data Management
Let’s get started with all of them, one-by-one.
1. Lot’s of Manual Effort Still Needed
Pact contracts need to be maintained and updated as services evolve. Ensuring that contracts accurately reflect real interactions can become challenging, especially in rapidly changing environments. Ensuring that contracts accurately reflect the expected interactions can become complex, especially when multiple consumers are involved.
Any time teams (especially producers) miss updating contracts, consumers start testing against incorrect behaviors which is when critical bugs start leaking into production.
Initial Contract Creation
Writing the first version of a contract involves a detailed understanding of both the consumer's expectations and the provider's capabilities. Developers must manually define the interactions in test code.
# Defining a contract in a consumer test
@pact.given('user exists')
@pact.upon_receiving('a request for a user')
@pact.with_request(method='GET', path='/user/1')
@pact.will_respond_with(status=200, body={'id': 1, 'name': 'John Doe'})
def test_get_user():
# Test logic hereThis change must be communicated and agreed upon by all consumers of the API, adding coordination overhead.
Maintaining Contract Tests
The test suite for both the consumer and the provider will grow as new features are added. This increased test suite size can make maintenance more difficult.
Each function represents a new contract or a part of a contract that must be maintained.
# Over time, you may end up with multiple contract tests
def test_get_user():
# ...
def test_update_user():
# ...
def test_delete_user():
# ...2. Testing Asynchronous Patterns
Pact supports non-HTTP communications, like message queues or event-driven systems, but this support varies by language and can be less mature than HTTP testing.
// A JavaScript example for message provider verification
let messagePact = new MessageProviderPact({
messageProviders: {
'a user created message': () => Promise.resolve({ /*...message contents...*/ }),
},
// ...
});This requires additional understanding of how Pact handles asynchronous message contracts, which might not be as straightforward as HTTP.
3. Consumer Complexity
In cases where multiple consumers interact with a single provider, managing and coordinating contracts for all consumers can become intricate.
Dependency Chains
Consumer A might depend on Consumer B, which in turn depends on the Provider. Changes made by Provider could potentially impact both Consumer A and the Consumer B. This chain of dependencies complicates the contract management process.
<aside>
💡 Let’s understand this with an example:
Given Services:
- Provider: User Management API.
- Consumer B: Profile Management Service, depends on the Provider.
- Consumer A: Front-end Application, depends on Consumer B.
Dependency Chain:
- `Consumer A` depends on `Consumer B`, which in turn depends on the `Provider`.
Change Scenario:
- The `Provider` adds a new mandatory field `birthdate` to its user data response.
- `Consumer B` updates its contract to incorporate `birthdate` and exposes it through its endpoint.
- `Consumer A` now has a failing contract because it doesn't expect `birthdate` in the data it receives from `Consumer B`.
Impact:
- `Consumer A` needs to update its contract and UI to handle the new field.
- `Provider` needs to coordinate changes with both the `Consumer B` and `Consumer A` to maintain contract compatibility.
- The `Provider` must be aware of how its changes affect downstream services to avoid breaking their contracts.
</aside>Coordination Between Teams
When multiple teams are involved, coordination becomes crucial. Any change to a contract by one team must be communicated to and accepted by all other teams that are consumers of that API.
# Communication overhead example
# Team A sends a message to Team B:
"We've updated the contract for the /user endpoint, please review the changes."This communication often happens outside of Pact, such as via team meetings, emails, or chat systems.
Ensuring that all consumer teams are aware of contract changes and aligned on the updates can require effective communication channels and documentation.
4. Test Data Management
Test data management in Pact involves ensuring that the data used during contract testing accurately represents real-world scenarios while maintaining consistency, integrity, and privacy. This can be a significant challenge, particularly in complex microservices ecosystems. The problems that might arise would be:
Data Generation
Creating meaningful and representative test data for all possible scenarios can be challenging. Services might need specific data states to test different interactions thoroughly.
Data Synchronization
PACT tests should use data that accurately reflects the behavior of the system. This means that the test data needs to be synchronized and consistent across different services to ensure realistic interactions. Mismatched or inconsistent data can lead to false positives or negatives during testing.
Example: If the consumer's Pact test expects a user with ID 1, but the provider's test environment doesn't have this user, the verification will fail.
Partial Mocking Limitations
Because Pact uses a mock service to simulate provider responses, it's possible to get false positives if the provider's actual behavior differs from the mocked behavior. This can happen if the provider's implementation changes without corresponding changes to the contract.
How we've fixed the biggest problem with the Pact workflow?
PACT driven integration testing has becoming very popular among teams off late given its simplicity and effectiveness. But some obvious challenges in getting started and contract maintenance still does not make it the perfect solution for integration testing.
So we at HyperTest has built an innovative approach that overcomes these shortcomings, making contract testing easy to implement and scalable.
In this approach, HyperTest builds contract tests for multiple services autonomously by monitoring actual flows from production traffic. Principally there are two modes i.e. record mode which records real world scenarios 24x7 and replay/test mode that then replays these scenarios to test the service with al external systems.
Lets explore how these two modes work:
Record mode: Automatic tests generation based on real-world scenarios
The HyperTest SDK is placed directly above a service or SUT. It observes and documents all incoming requests for traffic that the service receives. This includes recording the entire sequence of steps that the SUT takes in order to generate a response. The incoming requests represent the paths users take, and HyperTest captures them exactly as they occur.
👉This ensures that no scenarios are overlooked, resulting in a comprehensive coverage of all possible test cases. In this mode HyperTest records:
👉The incoming request to the SUT
👉The outgoing requests from the SUT to downstream services and databases. Also the response of these external systems
👉The response of the SUT which is stored (say X’)
Replay mode: Replay of recorded test scenarios with mocked dependencies
During the (replay) Test mode, integrations between components are verified by replaying the exact transaction (request) recorded during the record mode. The service then makes external requests to downstream systems, databases, or queues whose response are already mocked. HyperTest uses the mocked response to complete these calls, then compares the response of the SUT in the record mode to the test mode. If the response changes, HyperTest reports a regression.
Advantages of HyperTest over PACT
This simple approach of HyperTest takes care of all the problems with PACT. Here is how:
👉Auto-generate service contracts with no maintenance required
HyperTest observes actual calls (requests and responses) and builds contracts in minutes. If requests (consumer) or responses (producer) change breaking the contracts, respective service owners can approve changed contracts with a click for all producers or consumers rather than rewriting in PACT files. This updation of contracts (if needed) happens with every commit.
Respective consumer - provider teams are notified on slack needing no separate communication. This instant feedback on changing behavior of external systems helps developers make rapid changes to their code before it breaks in production.
👉Test Data Management
It is solved by design. HyperTest records real transactions with the real data.
For example:
When testing for login, it has several real flows captured with user trying to login.
When it tests login it will replay the same flow (with transactional data) and check if the same user is able to login to verify the right behavior of the application.
HyperTest's approach to aligning test data with real transactions and dynamically updating mocks for external systems plays a vital role in achieving zero bugs in production.
👉Dependency Management
HyperTest autonomously identifies relationships between different services and catches integration issues before they hit production.
Through a comprehensive dependency graph, teams can effortlessly collaborate on one-to-one or one-to-many consumer-provider relationships.
Notification on Disruption: Lets developer of a service know in advance when the contract between his and other services has changed.
Quick Remediation: This notification enables quick awareness and immediate corrective action.
Collaboration on Slack: This failure is pushed to a shared channel where all developers can collaborate
👉CI/CD integration for early issue detection and rollback prevention
HyperTest identifies issues early in SDLC for developers to quickly test changes or new features and ensure they integrate seamlessly with rest of the system.
Early Issue Detection
Immediate Feedback
Automatically run tests using your CI/CD pipeline when a new merge request is ready. The results can be observed on platforms like GitHub, GitLab or Bitbucket and sign-off knowing your change will not break your build in production.
👉Build Confidence
Knowing that their changes have undergone rigorous testing and integration checks, devs can sign off with confidence, assured that their code will not break the production build.
This confidence significantly reduces the likelihood of introducing bugs that could disrupt the live system.
Conclusion
While PACT undoubtedly excels in microservices contract testing, its reliance on manual intervention remains a significant drawback in today's agile environment. This limitation could potentially hinder your competitiveness.
HyperTest, comes as a better solution for testing microservices. Offering seamless collaboration between teams without the burden of manual contract creation and maintenance, it addresses the challenges of the fast-paced development landscape. Already trusted by teams at Nykaa, PayU, Urban Company, Fyers, and more, HyperTest provides a pragmatic approach to microservices testing.
To help you make an informed decision, we've compiled a quick comparison between HyperTest and PACT. Take the time to review and consider your options.
If you're ready to address your microservices testing challenges comprehensively, book a demo with us. Happy testing until then! 🙂