AgenticQA · case study sharadhaai.com
Engineering case study

AgenticQA Toolkit

A toolkit of independent AI agents for QA — each one solves a single real testing problem, built only where nothing good already exists.

solo-built LangChain · FastAPI · React 2 of 20+ agents shipped/in progress 18 yrs QA background
01 · Problem

QA has more real problems than any one tool can solve

Flaky tests, requirement ambiguity, coverage gaps, framework migrations, duplicate test cases, bugs that slip through a green build — each of these is a distinct, specific problem, not a feature of one grand platform.

Most "AI for QA" products pick one of these, wrap it in a dashboard, and call it a suite. AgenticQA takes the opposite approach: treat every problem as its own small, finishable build, and only build what doesn't already exist.

02 · Approach

Build-vs-reuse, checked before every agent

Before writing a line of code for a new agent, three questions get answered in order:

Decision order
  1. Does the underlying tool already solve this natively?
  2. Is there a good open-source project or agent to extend?
  3. Only build from scratch when neither of the above holds.
Example — UI self-healing locators: already solved natively by Playwright 1.56+ (>75% success rate on selector failures, per Microsoft's own benchmark). Not rebuilt. Effort went to genuinely unclaimed ground instead — like defect-leakage analysis, below.
03 · Architecture

A shared shell, not a shared pipeline

Every agent runs as its own independently deployable service — one can be demoed, shipped, or thrown away without touching the others. One shell app browses and launches all of them; it does not pass data between them.

Only three agents share a genuine dependency: a searchable corpus of existing test cases. That shared piece is built once, as its own service, and called by the agents that actually need it — everything else in the catalog stays standalone.

AgenticQA Shell — browse, launch, demo launches (no shared state) Requirement Smell shipped Test Case Generator in progress Bug-Leak RCA planned — next 17+ more (standalone) req + gaps gap-fill request dedup check coverage check Test Repo Search shared service — used by 3 agents only
The shell is a launcher, not a pipeline. Only the three agents that share a real data need — a searchable test-case corpus — talk to each other or a common service; the rest of the catalog stays fully standalone.
04 · Agents

What's built, what's in progress, what's next

Shipped

Requirement Smell Detector

Pulls a Jira ticket, flags vague wording and untestable clauses, scores confidence, and produces clarified acceptance criteria — shifting ambiguity detection left, before a test is ever written.

LangChain agentJira MCP toolconfidence score
In progress

Test Case Generator

A generator + reviewer agent pair. Turns a sanitized requirement into a confidence-scored test suite, explicitly addressing every gap the smell detector flagged — no silent drops, bounded revision cycles.

generator/reviewerper-case confidencededup-aware
Planned — next

Bug-Leak RCA Agent

Given a customer-reported bug: checks reproducibility, searches the test repo for coverage, checks whether a covering test actually ran and passed in that release, and classifies exactly why the bug leaked.

defect leakagetest-repo awaregap-fill hand-off
05 · Process

Bug-leak RCA — the process being encoded

The next agent formalizes a real QA workflow: when a customer reports a bug, the question isn't just "why did it happen" — it's "why didn't our tests catch it." The answer is a specific, traceable classification, not a guess.

Customer-reported bug Reproducible? no Not reproducible → returned to reporter yes Search test repo — covered by a test? no Add new test case → Test Case Generator Reason: coverage gap yes Check release test result for that case Ran, passed — but assertion too weak Skipped / not selected this release Flagged flaky, quarantined Reason: weak assertion Reason: not run Reason: flaky masked it
Every path — reproduced, covered or not, ran or not — ends in one of four classified leak reasons, feeding a single structured report. The taxonomy above is a first draft, not yet validated against real bug history.
06 · Stack

Standard stack across every agent

agent fw LangChain (create_agent)
backend FastAPI + Pydantic
frontend React · Vite · Tailwind
model Groq — cost-optimized default
tracing Arize
evals DeepEval
integration MCP (Jira)
07 · Principles

What holds the toolkit together

Toolkit, not monolith

Twenty-plus real QA problems, one small finishable build at a time — never one framework trying to solve all of them at once.

Reuse before building

Every agent is checked against what already ships natively or open-source first. Building is earned, not assumed.

Shell, not pipeline

One launcher app for discovery and demos. Agents stay independently deployable; only real data dependencies get wired up.

Process over prompts

Each agent encodes an actual QA practice — defect leakage analysis, requirement ambiguity review — not a generic "AI does X" pitch.