Live demo · runs the real ReAct agent & eval harness server-side
reagent is a tool-using ReAct + RAG assistant over a small Northwind knowledge base — BM25 doc_search, a calculator, and a structured employee lookup, reasoned in an explicit Thought → Action → Observation loop. The headline act is a built-in evaluation harness: a labeled test set scored for accuracy, tool-selection, safety/refusal, latency and token cost. Ask it something, then watch it grade itself.
Each question runs the real agent server-side. It thinks in an explicit
Thought → Action(tool) → Observation loop, calls
doc_search,
calculator and
table_lookup as needed, then returns a grounded
Final Answer — or refuses, with no tool calls, on unsafe / out-of-scope prompts.
A shipped agent is only as trustworthy as its evaluation. reagent bundles a labeled test set across multi-step, single-tool, pure-RAG and safety categories, and scores every case for answer accuracy, correct tool selection, and safe refusal — plus latency, token cost and step count. Grading is deterministic in mock mode and an LLM-as-judge in live mode.
Every capability below maps to the Cohere Forward Deployed Engineer, Agentic Platform role and is backed by a real, tested module in reagent — not slideware.
A ReAct loop that picks tools from a registry, executes them, and feeds observations back until a grounded final answer — with a hard step cap.
agent.py → Agent.run Embedding-free Okapi BM25 over chunked markdown; every hit carries its source doc for grounded, citable answers.
retriever.py → Bm25Index · tools.py → doc_search Three real tools — doc_search (RAG), calculator (safe arithmetic), table_lookup (structured directory) — behind a uniform registry the model is prompted to call.
tools.py → ToolRegistry Chains tools across steps, e.g. doc_search THEN calculator to turn a PTO policy into work-weeks — visible in the live trace.
agent.py → step loop · prompts.py A labeled test set scored for answer accuracy, tool-selection, safety-refusal, latency p50/p95, tokens and steps — deterministic in mock, LLM-as-judge live.
eval.py → run_eval · data/testset.json Out-of-scope and prompt-injection requests are refused with zero tool calls; the harness scores a dedicated safety category.
agent.py → refusal handling · prompts.py Live OpenRouter client (default openai/gpt-4o-mini) with token accounting; swappable base URL/model via settings.
llm/openrouter.py → OpenRouterLLM A mock LLM mirrors the live contract so the whole suite + this demo run offline with no key — mode chosen automatically from the env.
llm/mock.py → MockLLM · config.py Structured Step/Metrics trace objects per run: thought, tool, input, observation, plus llm_calls, tokens and latency.
trace.py → Step · Metrics · AgentResult A pip-installable package with a typed public API (ask / Agent / run_eval), a CLI, and a tested corpus + data bundle.
__init__.py · cli.py · pyproject.toml