Case Study · AI-Assisted Testing

Test Pilot

A shift-left testing toolkit that turns requirements straight into reviewed, automatable test cases — with a human checkpoint at every step that matters.

RoleArchitect & Lead Engineer
ContextStaff SDET, Chargebee
ApproachShift-left, human-in-the-loop
Outcome~60% shorter QA cycle

The problem

Traditional QA starts once requirements are "done": someone reads them, writes test cases by hand, and writes automation for each one later, usually in a separate pass.

Every handoff between those stages adds delay, and test coverage quietly drifts away from what was actually specified — by the time automation exists, the requirement has often already changed again.

Architecture & flow

↔ swipe to see the full diagram

generate reuse new automation Requirements input document AI BDD Generator Given / When / Then Human Validates checkpoint 1 Step Classifier reusable vs. new Reusable Steps already automated Step Library grows every run New Steps Flagged not yet automated Human Approves checkpoint 2 Generate Code new automation
Requirements flow through an AI-generated BDD draft, a human validation gate, and a step classifier. Reusable steps rejoin the library immediately; new steps go through a second human approval before Test Pilot generates their automation code — which then grows the same library.
Human-in-the-loop checkpoint
Feedback loop into step library

How it works

  1. Reads the requirements

    Takes requirement documents as input as soon as they're ready — before a single manual test case gets written.

  2. Generates BDD test cases

    An AI model drafts Given/When/Then test cases directly from the requirement text.

  3. Human validates the cases HITL

    A reviewer checks the generated BDD cases for correctness and completeness before anything downstream happens.

  4. Sorts automation steps

    Once validated, Test Pilot scans each case's steps: steps that already exist in the automation step library are marked reusable, and steps that don't are flagged as new.

  5. Human reviews new steps HITL

    A reviewer approves — or corrects — the newly flagged steps before any code gets generated for them.

  6. Generates automation code

    On approval, Test Pilot generates the automation code for the new steps and adds them back into the reusable step library for next time.

Impact

~60% Reduction in QA cycle time
2 Human checkpoints kept in the loop by design
Compounding Step library grows automatically with every new requirement

Approach & tools

BDD Generative AI Human-in-the-loop Test Automation Shift-left Testing
A note on code

Test Pilot was built as proprietary tooling for Chargebee. This case study describes the architecture, the workflow, and my role designing it end-to-end — the implementation itself isn't something I'm able to share.