Shipped a customer-facing copilot grounded in the product knowledge base, with evaluation, guardrails, and a refusal model that reduced deflection-related complaints to near zero.
Anonymized delivery dashboard
Ticket deflection
38%
Eval pass rate
94%
Complaint rate
< 0.4%
Before
Constraint
Build
Controlled cutover
After
Measured gain
Every engagement is framed around the business situation, the constraint that made it hard, and the decision that turned delivery into a controlled path to value.
Situation
The vendor’s previous chatbot pilot was withdrawn after it confidently fabricated answers about pricing and compatibility. Leadership wanted a copilot that could deflect support tickets without ever telling a customer something untrue.
Constraint
The vendor’s previous chatbot pilot was withdrawn after it confidently fabricated answers about pricing and compatibility. Leadership wanted a copilot that could deflect support tickets without ever telling a customer something untrue.
Decision
Treated evaluation and refusal behavior as product requirements, not prompt polish, before exposing the copilot to customers.
Build
Documentation and KB articles were ingested with deliberate chunking and a hybrid keyword + dense retriever. Every response cited the source passages it relied on, and answers that could not cite a source defaulted to a human handoff.
A golden set of 600 graded questions ran in CI on every prompt change, with automatic graders for groundedness, refusal correctness, and tone. The launch criterion was a defined pass rate, not a leadership demo.
Outcome
38% of eligible support tickets deflected with verified-correct answers.
Eval pass rate steady at 94% across two model upgrades.
What changed after launch
Support leaders could tune deflection against groundedness, with traces and source citations available for every answer.
“The refusal model mattered as much as the answers. Customers trusted it because it knew when not to guess.”
The work was not abstract modernization. It changed day-to-day behavior, ownership, and the evidence leaders used to make decisions.
Before
After
The delivery plan made the system boundary explicit, then used rehearsals, gates, and telemetry to optimize safely before launch.
Delivery architecture
AI & Data control loop
Discover
Documentation and KB articles were ingested with deliberate chunking and a hybrid keyword + dense retriever. Every response cited the source passages it relied on, and answers that could not cite a source defaulted to a human handoff.
Build
A golden set of 600 graded questions ran in CI on every prompt change, with automatic graders for groundedness, refusal correctness, and tone. The launch criterion was a defined pass rate, not a leadership demo.
Launch
Prompt-injection patterns, PII redaction, and topic-scope filters wrapped the model. Every response was traced end-to-end with the retrieved context preserved for post-hoc review.
Documentation and KB articles were ingested with deliberate chunking and a hybrid keyword + dense retriever. Every response cited the source passages it relied on, and answers that could not cite a source defaulted to a human handoff.
A golden set of 600 graded questions ran in CI on every prompt change, with automatic graders for groundedness, refusal correctness, and tone. The launch criterion was a defined pass rate, not a leadership demo.
Prompt-injection patterns, PII redaction, and topic-scope filters wrapped the model. Every response was traced end-to-end with the retrieved context preserved for post-hoc review.
After launch
Support leaders could tune deflection against groundedness, with traces and source citations available for every answer.
Python
OpenAI
pgvector
LangChain
AWS Bedrock
OpenTelemetry
A similar problem?
A senior engineer will follow up within one business day with an opinionated take on the shape of the work.
