top of page

6 February 2024

09 Min. Read

E2E Testing

The Pros and Cons of End-to-End Testing

The Pros and Cons of End-to-End Testing

Key Highlights

In this blog, we cover following key highlights:

  1. E2E testing ensures robust software performance by identifying issues in the development process promptly.

  2. It validates that the software meets user expectations.

  3. It also requires the entire system to be kept up and running, leading to intricate setups and scenarios.

  4. Keeping E2E tests updated can be labor-intensive and challenging, learn about an alternative approach in the end.

Our whitepaper transforms the complexity in microservice testing into a manageable task, ensuring thorough testing that leaves no room for undetected bugs.

Let's talk about end-to-end testing – it's like the superhero at the top of Mike Cohn's testing pyramid! These tests are the final line of defense, and even though there aren't many of them, they're like a super-strong shield against sneaky bugs trying to get into the production party. But, here's the million-dollar question: do they really live up to all the hype?


Well, that's why we're here with this blog! We're going to talk about the pros and cons of end-to-end testing. Plus, we’ve an alternative approach to discuss yielding out the same or even better results, without you having to write any test-scripts for that. Let's dive into the world of E2E testing and find out what's really going on!


What is End-to-End testing?

So let’s get the basics clear, we start with simply explaining what is E2E tests, so that we don’t have any confusions as we go ahead with the blog. E2E are the high-level tests that are performed at the end of the testing phase. The focus is to test individual components together as a work-flow from a user’s perspective. While unit tests focuses on testing those individual components in isolation, E2E combines them together as a single working unit and run a test on that.

End-to-end testing is a methodology used to verify the completeness and correctness of a software application from start to finish. The main goal of E2E testing is to simulate real user scenarios to ensure the system behaves as expected in a fully integrated environment.

All the dependent services, third-party integrations, databases need to be kept up and running, mimicking the real scenario with all possible dependencies. It helps in evaluating the system's external interfaces and ensures all integrated components work together seamlessly to carry out any task a user might perform.


Key Features of E2E Testing:


Testing Pyramid
  • Comprehensive Coverage: Tests the application's workflow from beginning to end.


  • Real User Simulation: Mimics real user behaviors and interactions with the application.


  • Integration Verification: Ensures that all parts of the system work together correctly.


  • Environment Validation: Confirms that the application works as expected in environments that mimic production settings.


Types/ Strategies of E2E testing

End-to-end (E2E) testing strategies are essential for ensuring that software systems meet their designed functions and user expectations comprehensively. Among these strategies, horizontal and vertical E2E testing stand out for their unique approaches and focuses. While both aim to validate the complete functionality of a system, their methodologies and perspectives differ significantly.


1. Horizontal E2E Testing

Horizontal E2E testing examines the system's workflow as it would occur in its operational environment, emphasizing a wide-ranging approach that covers the system's full spectrum of functionalities. This method aligns closely with the user's perspective, traversing through the application's various interfaces and interactions just as an end user would.


  • Characteristics: It simulates real-world user scenarios, navigating through the application's user interface (UI), engaging with different features, and integrating with external systems, if applicable. The objective is to replicate the typical user journey as closely as possible.


    • In an online booking system, horizontal testing would involve steps like searching for a service, selecting an option, entering user details, proceeding through payment, and receiving a confirmation—all through the UI.


2. Vertical E2E Testing

Contrastingly, vertical E2E testing delves into the system's architecture, examining the integration and data flow between layers or components from a more technical standpoint. It is particularly effective in early development stages or for complex systems where layer-specific functionality needs thorough validation.


  • This approach tests the system's internal processes, from the database layer through business logic and up to the presentation layer, in a sequential manner. It's highly focused on backend operations, data integrity, and the integration between different system components.


    • For a cloud storage service, vertical testing might verify the process of uploading a file, ensuring that the file passes correctly from the front end, through the application logic, is stored properly in the database, and is accessible for future retrieval.

Interested in learning
how HyperTest can help you?

Get a Personalised Demo within 24 Hrs.

How to perform E2E tests for an Online E-Commerce Store?

Objective:

To conduct a thorough End-to-End (E2E) testing on an online shopping platform to ensure a seamless shopping experience from account registration to order confirmation.


Test Strategy Development:

  • Aim: To validate the complete functionality of the e-commerce platform, ensuring that all user actions lead to the expected outcomes without errors.


  • Key Customer Journey to Test: The process from creating a new account, finding products, adding items to the cart, checking out, making payment, to receiving an order confirmation.

Testing Environment Configuration:

Set up a staging environment that closely mirrors the production environment, including web servers, databases, and mock services for external integrations like payment gateways.


Test Cases Development:

  1. Account Registration

    • Purpose: To confirm that users can successfully register on the platform.

    • Procedure: Navigate to the signup page, fill out the registration form with details (username, email, password), and submit.

    • Expected Result: The user is registered and receives a confirmation email.


  2. Login Functionality

    • Purpose: To ensure that the login mechanism works correctly with valid user credentials.

    • Procedure: Go to the login page, enter valid email and password, and submit.

    • Expected Result: The user is logged into their account and directed to the homepage.


  3. Product Browsing and Selection

    • Purpose: To verify that users can browse through the product listings and access product details.

    • Procedure: Visit the product listing section, choose a category, and select a product to view its details.

    • Expected Result: The product's details page loads with all the relevant information.


  4. Adding Product to Cart

    • Purpose: To test the functionality of adding products to the shopping cart.

    • Procedure: From a product's details page, click the "Add to Cart" button.

    • Expected Result: The product is added to the cart, and the cart's item count is updated.


  5. Checkout Process

    • Purpose: To confirm the checkout process is intuitive and error-free.

    • Procedure: Access the shopping cart, click "Proceed to Checkout," enter necessary shipping and billing information, and submit.

    • Expected Result: The user is navigated to the payment page.


  6. Payment Transaction

    • Purpose: To ensure the payment process is secure and processes transactions correctly using mock payment details.

    • Procedure: Input mock payment information and submit.

    • Expected Result: The payment is processed, and an order confirmation screen is shown.


  7. Order Confirmation

    • Purpose: To verify that the order confirmation details are accurate and an email confirmation is sent.

    • Procedure: After payment, confirm the details on the order confirmation page and check for an email confirmation.

    • Expected Result: The order details are correct, and an email confirmation is received.


Preparation of Test Data:

Data Needed: User credentials for login tests, product details for browsing and selection, and mock payment information for checkout.


Perform End-to-end testing without the need to prepare test data, learn how?

Execution of Tests:

Automated testing scripts (e.g., using Selenium WebDriver) will simulate the user journey from registration to order confirmation, asserting the expected outcomes at each step.


# Example of a Python script using Selenium for automated E2E testing
from selenium import webdriver
import unittest

class E2ETesting(unittest.TestCase):
    def setUp(self):
        self.browser = webdriver.Chrome('path/to/chromedriver')
        self.addCleanup(self.browser.quit)

    def testCompleteUserJourney(self):
        # Detailed steps for each part of the test go here, including:
        # - Navigating to the site
        # - Registering a new account
        # - Logging in
        # - Browsing products and adding to cart
        # - Checking out
        # - Verifying order confirmation
        pass

if __name__ == '__main__':
    unittest.main()

Analysis of Test Results:

After executing tests, analyze logs and outputs to identify any bugs or issues with the platform.


Test Reporting:

Compile a detailed report of the testing process, findings, and recommendations. This report should include test coverage details, success rates, bugs identified, and screenshots or logs as evidence.


This comprehensive approach to E2E testing ensures the online shopping platform functions correctly across all user interactions, offering stakeholders confidence in the platform's reliability and user satisfaction.

The Pros of E2E testing

E2E tests offers the full picture of the test scenario, offering advantages like:


  1. Replicates Real-User Experience: E2E testing evaluates the system's overall functionality and its interaction with external interfaces, databases, and other systems, mirroring real-world user scenarios and behaviors.


Scenario: Testing a login feature in an application.


describe('Login Feature', () => {
  it('successfully logs in the user', () => {
    cy.visit('/login') // Navigate to the login page
      .get('input[name="email"]').type('user@example.com') // Enter email
      .get('input[name="password"]').type('password123') // Enter password
      .get('form').submit() // Submit the login form
      .get('.welcome-message').should('contain', 'Welcome back, user!'); // Verify login success
  });
});

Real-User Experience: This code simulates a user navigating to the login page, entering their credentials, and submitting the form, closely mirroring a real user's actions.


Increases Confidence: Verifying the presence of a welcome message after login actions ensures the application behaves as expected, boosting confidence in deployment.


  • 2. Identifies System-wide Issues: It helps uncover issues related to data integrity, services integration, and user interface, which might not be detected during unit or integration testing phases.


    3. Facilitates Compliance with Requirements: For applications in regulated sectors, E2E testing ensures that the software meets necessary compliance standards, including security protocols and data handling practices.


    4. Supports Continuous Integration/Continuous Deployment (CI/CD): Automated E2E tests can be integrated into CI/CD pipelines, enabling regular testing at various stages of development, which helps in identifying and addressing issues promptly.


The Cons of E2E testing

This test pyramid approach needs to be modified for testing microservices. E2E tests need to be completely dropped. Apart from taking a long time to build and maintain, E2E tests execute complete user-flows every time on the entire application, with every test.


This requires all services under the hood to be simultaneously brought up (including upstream) even when it is possible to catch the same kind and the same number of failures by testing only a selected group of services; only the ones that have undergone a change.


  1. Resource Intensive: E2E testing can be time-consuming and expensive due to the need for comprehensive test cases, the setup of testing environments that mimic production, and potentially longer execution times for tests.


Scenario: Setting up a Selenium test environment for the same login feature.


from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

# Setup WebDriver
driver = webdriver.Chrome()

# Navigate to the login page
driver.get("http://example.com/login")

# Enter login details and submit
driver.find_element(By.NAME, "email").send_keys("user@example.com")
driver.find_element(By.NAME, "password").send_keys("password123")
driver.find_element(By.NAME, "submit").click()

# Verification
assert "Welcome back, user!" in driver.page_source

# Teardown
driver.close()

Resource Intensiveness: Setting up Selenium, managing WebDriver instances, and ensuring the environment matches the production settings can be time-consuming and resource-heavy.


Complexity in Maintenance: The Selenium example requires explicit browser management (setup and teardown), which adds to the complexity, especially when scaling across different browsers and environments.


  1. Flakiness and Reliability Issues: E2E tests can sometimes produce inconsistent results due to their reliance on multiple external systems and networks, leading to flakiness in test outcomes.


  2. Slow Feedback Loop: Due to the extensive nature of E2E tests, there can be a significant delay in getting feedback, which can slow down the development process, particularly in agile environments that prioritize quick iterations.


  3. Not Suited for All Types of Testing: E2E testing is not always the best choice for detecting specific, low-level code issues, which are better identified through unit testing or integration testing.


Perform E2E Testing without test data preparation

The flakiness and complexity of End-to-End (E2E) tasks often stem from the need for test data preparation. For E2E scenarios to run smoothly, it's essential to create and maintain relevant test data.


In the context of app testing, particularly for e-commerce platforms like Nykaa or Flipkart, the process is akin to testing different states of the app.


For example, verifying if a user can apply loyalty points for a discount involves specific state testing.


  • Requirements for Test Data: To test the aforementioned scenario, a QA engineer must prepare several pieces of test data, including:

    • A valid user account

    • A valid product listing

    • Sufficient inventory for the product

    • The addition of the product to a shopping cart

    This setup is necessary before the app reaches the state where the discount via loyalty points can be applied.


The scenario described is relatively straightforward. However, an e-commerce app may contain hundreds of such flows requiring test data preparation. Managing the test data and app states for numerous scenarios significantly increases the workload and stress for QA engineers.


Fortunately, there exists a straightforward approach that allows QA engineers to test the functionality of an application without the need for extensive test data creation and management. This method focuses on testing the core functions directly, alleviating the burden of test data preparation. Click here to learn more now.


Conclusion

Concluding our discussion on the pros and cons of end-to-end (E2E) testing, it's evident that E2E testing is a critical tool in the software development but it comes at the cost of time, money and effort. They’re extremely difficult to write, maintain and update. An E2E test that actually invokes the inter service communication like a real user would catch this issue. But cost of catching this issue with a test that could involve many services would be very high, given the time and effort spent creating it.


  • imprecise because they've such a broad scope

  • needs the entire system up & running, making it slower and difficult to identify the error initiation point


The essence of navigating E2E testing successfully is choosing the right tools, automating where possible, and continuously refining testing processes to align with project needs and goals. Get in touch with us if you want to test E2E scenario’s without needing to spend any time creating and managing test data.

Frequently Asked Questions (FAQs)

1. What is E2E testing?

End-to-End (E2E) testing ensures seamless software functionality by examining the entire system's components, identifying potential issues, and verifying their integration.

2. What is the importance of end-to-end?

E2E testing is vital for detecting and preventing integration issues in software development, ensuring a smooth user experience and system reliability.

3. What are the benefits of end-to-end testing?

Benefits include early bug detection, improved system reliability, and confidence that the software meets user requirements by validating its entire functionality.
bottom of page