Aman Kumar BeheraResume
Back to projects
AI/MLProductAgentic AIEvaluationTrust & Safety

Nimbus Support Escalator — Human-in-the-Loop AI Support Agent

Aug 2026

An AI support agent that knows what NOT to answer on its own — auto-resolving safe tickets from policy and escalating risky ones to a human with an auditable reason. Evaluated on a 20-ticket adversarial set with zero unsafe auto-resolutions.

Nimbus Support Escalator — Human-in-the-Loop AI Support Agent

The problem

Fully autonomous support agents hallucinate refunds, invent policy, and mishandle angry customers — creating real financial and reputational risk. The unsolved product question isn't "can an agent answer?" It's "which requests can it be trusted to answer alone?"

The thesis

I designed this agent around an explicit trust boundary: it handles the safe majority automatically and hands the risky minority to a human. Crucially, that boundary is enforced by deterministic logic — not the model's judgment — so a high-risk action can never be auto-committed on the AI's word alone.

How it works

A ticket arrives via Telegram. An LLM classifies it for amount, risk flags, and emotion. Then a deterministic guardrail makes the final call:

  • Escalate if the amount exceeds ₹2,000, OR any risk flag is present (legal, chargeback, account-closure, staff complaint), OR emotion crosses a threshold.

  • Otherwise, resolve automatically from company policy.

Every escalation ships with a plain-language, auditable reason — for example: "amount ₹3000 exceeds ₹2000; risk flags: legal; high emotion (0.85)." That auditable reasoning is the core of the product.

Built with n8n, Google Gemini, and Telegram.

Results

I evaluated the system on a 20-ticket adversarial set (boundary amounts, mixed Hindi-English, soft-framed legal threats, conflicting signals):

  • Routing accuracy: 20/20

  • Precision / Recall: 1.00 / 1.00

  • Unsafe auto-resolutions: 0

(Honest caveat, which I document in the writeup: a perfect score on a small, self-authored set is weak evidence on its own — the metric I trust is zero unsafe auto-resolutions, and the real value was the qualitative failures the eval surfaced.)

What this project demonstrates

The tools are the least interesting part. The real work was the product judgment: deciding where to draw the trust boundary, making the agent's reasoning auditable, and building an evaluation to measure whether it routes correctly. As I move toward AI Product Management, this reflects how I think the role really works — not "can the AI do it," but "should it, and how do we know?"

Links

Videos