Portfolio case study
AgenticEvals
A production-grade evaluation, guardrail and observability platform being built around one real failure a live agentic system already produced — not a demo built to look finished.
The problem
A grounding failure that deterministic checks can't see
Most eval demos invent a failure case to solve. This one didn't have to — during manual testing of the target system, a sub‑agent's answer named specific tools that were never in its own retrieved evidence.
It passed every check that only looks at structure: valid JSON, correct schema, a tool call that returned 200. Nothing about the answer's shape was wrong. What was wrong is invisible to that class of check entirely — and that gap is the reason this platform evaluates content, not just format.
The tooling‑trend‑watcher sub‑agent's response named Prometheus, Grafana, and Airflow as trending tools — none of which appeared anywhere in the search snippets it had actually retrieved.
found by manual review, not by any automated gate · caught before deterministic checks would ever flag it
The system under evaluation
market-analyst-agent
A real multi‑agent research tool, not a toy Q&A bot — a job‑market‑analyst and a tooling‑trend‑watcher sub‑agent, each pulling from web/job search, GitHub trending and RSS, deduped through a persistent memory layer, publishing a weekly digest to Telegram over GitHub Actions.
It stays frozen exactly as submitted. AgenticEvals holds a pinned copy and reaches it only through its public entrypoint and a trace‑emission seam — the evaluation harness never edits the system it's evaluating.
The architecture
Four layers, one call path
Traceability feeds evaluation; evaluation gates deployment; guardrails police the live call; observability closes the loop back to the first layer. The published framework this build is scoped against:
Traceability & lineage
Every trace, prompt, tool call and config version captured under one unified ID before anything is scored.
Dual evaluation engine
Deterministic checks (schema, latency, golden diffs) and semantic/statistical checks (LLM‑as‑judge, hallucination) as CI/CD gates that can block deploy.
Runtime guardrails
Input interception and output policy wrapped around the live call path — quality enforcement doesn't stop at deployment.
Observability & feedback
Drift detection and failed traces flow back into the regression suite, closing the loop into layer one.
Where it stands
Phase 00 of 11 — foundation
Stated plainly, on purpose: this repo is a scaffold and a frozen dependency right now, not a finished platform. Each phase ships a checkable artifact before the next begins.
Planned stack
Free and open‑source by design
No paid tier required anywhere in the pipeline — every tool is self‑hostable or has a genuine free tier.