Case Study · Performance Engineering

ATPAR

Automated Test of Performance And Reporting — a one-click performance testing framework built to replace days of manual load-test setup with a single run.

RoleArchitect & Lead Engineer
ContextStaff SDET, Chargebee
DomainSubscription billing APIs
TypeInternal framework

The problem

Performance testing at scale meant coordinating infrastructure provisioning, realistic data setup, load generation, and metrics collection across five separate tools — by hand, before every release.

That process took days, depended on whoever ran it, and produced results that were hard to compare from one release to the next. A load run itself might finish in 10 minutes — but manually correlating infra, trace, and log signals afterward to explain why an API got slower routinely took up to three days, with no consistent, one-view answer along the way.

Architecture & flow

↔ swipe to see the full diagram

setup configure parallel collection reporting Provision Infra capacity check Data Snapshot match customer load Dynamic Load Config attached to JMeter plan Load Simulation JMeter test plans AWS Metrics infra / compute Datadog Traces perf metrics Splunk Logs application logs Consolidation + JTL reporter Benchmark Compare vs. monthly baseline Dashboard Email Report
ATPAR provisions infra, then in parallel validates a customer-matched data snapshot and dynamically computes the load profile attached to the JMeter test plan. It drives load with JMeter, collects infra, trace, and log signals from AWS, Datadog, and Splunk in parallel, then consolidates, benchmarks, and reports the result automatically.

How it works

  1. Provisions & verifies infra

    Spins up load-test infrastructure on demand and checks it against the compute capacity the run needs — before a single request fires.

  2. Prepares representative data

    Verifies — and restores where needed — data snapshots sized and shaped to match real customer load profiles, so results reflect production reality, not synthetic edge cases.

  3. Configures load dynamically

    Computes the load profile — concurrency, ramp-up, and throughput targets — for the release under test and attaches it directly to the JMeter test plan, so no test script is hand-edited between runs.

  4. Simulates load

    Drives traffic against the target APIs using the JMeter test plan, now carrying its dynamically attached load configuration.

  5. Collects signals in parallel

    While the test runs, ATPAR pulls infrastructure metrics from AWS, distributed traces and performance metrics from Datadog, and application logs from Splunk — all three at once.

  6. Consolidates & reports

    Merges all three signal streams with JMeter's JTL output into one dataset, surfaced through a single dashboard.

  7. Benchmarks automatically

    Compares each API's performance against its trailing monthly benchmark and flags deviations along with likely causes.

  8. Notifies stakeholders

    Emails a summary report to stakeholders the moment the test completes — no one has to go looking for results.

Impact

5 → 1 Tools coordinated manually, replaced by one run
3 days → 10 min Time to go from raw signals to a "why did it slow down" answer — now the same length as the test run itself
Monthly Rolling benchmark every API is automatically checked against

Where it plugs in

Live API SLA baselines Every API's performance baseline stays current against real load, instead of a one-off benchmark that quietly goes stale.
Nightly regression gating Nightly regression runs are checked against the rolling baseline automatically, before anyone has to go looking for a slowdown.
Feature-branch merge gating A branch that regresses an API's performance beyond threshold is flagged at merge time, not discovered after release.

Tools & signals

JMeter AWS Datadog Splunk JTL Reporter Automated Benchmarking One-click Orchestration
A note on code

ATPAR was built as proprietary infrastructure for Chargebee. This case study describes the architecture, the workflow, and my role designing and building it end-to-end — the implementation itself isn't something I'm able to share.