Agentuity Documentation

LLM-as-judge is the right tool when the rubric needs context, nuance, or subjective comparison. It is the wrong tool when the check is structural (JSON shape, length, presence of a field). For those, validate with a schema and skip the model call.

Use case	LLM-as-judge	Code or schema
Factual grounding against retrieved sources	Yes
Tone, helpfulness, politeness	Yes
Picking a winner between two outputs	Yes
Hallucination detection in RAG	Yes
Validating JSON structure		Yes
Word count, regex match		Yes

npm install @agentuity/aigateway @agentuity/queue @agentuity/vector @agentuity/telemetry zod

Compare models with one project credential

These examples use AIGatewayClient, so the same Agentuity project credential can call the candidate and judge models through the Gateway. Keep the model IDs explicit, and validate judge output before acting on it.

Inline Quality Check

Run the judge in the same request when the score is part of the response. Fast judge models keep the added latency manageable.

typescriptsrc/lib/quality-check.ts

import { AIGatewayClient } from '@agentuity/aigateway';
import { z } from 'zod';
 
const judgmentSchema = z.object({
  score: z.number().min(0).max(1).describe('Quality score from 0 to 1'),
  passed: z.boolean().describe('Whether the response meets the threshold'),
  reason: z.string().describe('Brief explanation'),
});
 
const gateway = new AIGatewayClient();
const ANSWER_MODEL = 'deepseek/deepseek-v4-flash';
const JUDGE_MODEL = 'googleai/gemini-3.5-flash';
 
interface QualityResult {
  readonly answer: string;
  readonly judgment: z.infer<typeof judgmentSchema>;
}
 
export async function answerWithQualityCheck(question: string): Promise<QualityResult> {
  const answerResult = await gateway.completeText({
    model: ANSWER_MODEL,
    messages: [{ role: 'user', content: question }],
  });
 
  if (!answerResult.hasText) {
    throw new Error('The answer model returned no text.');
  }
 
  const { data } = await gateway.completeStructured({
    model: JUDGE_MODEL,
    messages: [
      {
        role: 'user',
        content: `Evaluate this response on a 0-1 scale.
 
Question: ${question}
Response: ${answerResult.text}
 
Consider:
- Does it directly answer the question?
- Is the information accurate?
- Is it clear and easy to read?
 
Score 0.7+ to pass.`,
      },
    ],
    response_schema: { name: 'judgment', schema: judgmentSchema },
  });
 
  return { answer: answerResult.text, judgment: judgmentSchema.parse(data) };
}

The judge model is intentionally smaller and faster than the answer model. Inline judging is for cases where the user sees the score; otherwise move it off the request path.

Model Arena

Compare two providers side-by-side and let a third model pick the winner. Run the candidates in parallel, then judge.

typescriptsrc/lib/model-arena.ts

import { AIGatewayClient } from '@agentuity/aigateway';
import { z } from 'zod';
 
const arenaJudgment = z.object({
  winner: z.enum(['anthropic', 'deepseek']),
  reasoning: z.string(),
  scores: z.object({
    anthropic: z.number().min(0).max(1),
    deepseek: z.number().min(0).max(1),
  }),
});
 
const gateway = new AIGatewayClient();
const JUDGE_MODEL = 'googleai/gemini-3.5-flash';
const ANTHROPIC_MODEL = 'anthropic/claude-opus-4-8';
const DEEPSEEK_MODEL = 'deepseek/deepseek-v4-pro';
 
interface ArenaResult {
  readonly results: ReadonlyArray<{
    readonly provider: 'anthropic' | 'deepseek';
    readonly story: string;
    readonly generationMs: number;
  }>;
  readonly judgment: z.infer<typeof arenaJudgment>;
}
 
export async function runArena(
  prompt: string,
  tone: 'whimsical' | 'suspenseful' | 'comedic'
): Promise<ArenaResult> {
  const [anthropicResult, deepseekResult] = await Promise.all([
    generateWithTiming(ANTHROPIC_MODEL, prompt, tone),
    generateWithTiming(DEEPSEEK_MODEL, prompt, tone),
  ]);
 
  const { data } = await gateway.completeStructured({
    model: JUDGE_MODEL,
    messages: [
      {
        role: 'user',
        content: `Compare these ${tone} stories and pick a winner.
 
PROMPT: "${prompt}"
 
--- Anthropic ---
${anthropicResult.story}
 
--- DeepSeek ---
${deepseekResult.story}
 
Score each on creativity, engagement, and tone match (0-1).`,
      },
    ],
    response_schema: { name: 'arena_judgment', schema: arenaJudgment },
  });
 
  return {
    results: [
      { provider: 'anthropic', ...anthropicResult },
      { provider: 'deepseek', ...deepseekResult },
    ],
    judgment: arenaJudgment.parse(data),
  };
}
 
async function generateWithTiming(
  model: string,
  prompt: string,
  tone: string
): Promise<{ story: string; generationMs: number }> {
  const start = Date.now();
  const result = await gateway.completeText({
    model,
    messages: [
      {
        role: 'system',
        content: `Write a short ${tone} story (max 200 words) with a beginning, middle, and end.`,
      },
      { role: 'user', content: prompt },
    ],
  });
  if (!result.hasText) {
    throw new Error(`The model returned no text: ${model}`);
  }
  return { story: result.text, generationMs: Date.now() - start };
}

Grounding Check for RAG

Score whether an answer is supported by the retrieved sources. Returning the unsupported claims back to the caller turns the judge into a hallucination filter.

typescriptsrc/lib/rag-grounding.ts

import { VectorClient } from '@agentuity/vector';
import { AIGatewayClient } from '@agentuity/aigateway';
import { z } from 'zod';
 
type DocumentMetadata = {
  readonly content: string;
  readonly title: string;
} & Record<string, unknown>;
 
const groundingJudgment = z.object({
  isGrounded: z.boolean(),
  score: z.number().min(0).max(1).describe('Fraction of claims supported by sources'),
  unsupportedClaims: z.array(z.string()),
  reason: z.string(),
});
 
const ANSWER_MODEL = 'anthropic/claude-opus-4-8';
const JUDGE_MODEL = 'googleai/gemini-3.5-flash';
 
interface GroundedAnswer {
  readonly answer: string;
  readonly grounding: z.infer<typeof groundingJudgment>;
  readonly sources: readonly string[];
}
 
const vector = new VectorClient();
const gateway = new AIGatewayClient();
 
export async function answerWithGrounding(question: string): Promise<GroundedAnswer> {
  const matches = await vector.search<DocumentMetadata>('knowledge-base', {
    query: question,
    limit: 3,
  });
 
  const sources = matches
    .map((match) => match.metadata?.content ?? '')
    .filter((text) => text.length > 0);
 
  const answerResult = await gateway.completeText({
    model: ANSWER_MODEL,
    messages: [
      {
        role: 'user',
        content: `Answer using these sources:\n\n${sources.join('\n\n')}\n\nQuestion: ${question}`,
      },
    ],
  });
 
  if (!answerResult.hasText) {
    throw new Error('The answer model returned no text.');
  }
 
  const { data } = await gateway.completeStructured({
    model: JUDGE_MODEL,
    messages: [
      {
        role: 'user',
        content: `Check whether this answer is supported by the sources.
 
Answer: ${answerResult.text}
 
Sources:
${sources.map((source, i) => `[${i + 1}] ${source}`).join('\n\n')}
 
Return any factual claim in the answer that is not supported by the sources.`,
      },
    ],
    response_schema: { name: 'grounding_judgment', schema: groundingJudgment },
  });
 
  return {
    answer: answerResult.text,
    grounding: groundingJudgment.parse(data),
    sources: matches.map((match) => match.key),
  };
}

Pair this with Build a RAG Agent when answers should be filtered or flagged before they reach the user.

Background Judging

For monitoring and review flows, the judge does not need to block the response. Publish to a queue and let a worker route do the scoring.

typescriptsrc/index.ts

import { Hono } from 'hono';
import { AIGatewayClient } from '@agentuity/aigateway';
import { QueueClient } from '@agentuity/queue';
import { logger } from '@agentuity/telemetry';
import { z } from 'zod';
 
const helpfulnessJudgment = z.object({
  score: z.number().min(0).max(1),
  answersQuestion: z.boolean(),
  actionable: z.boolean(),
  reason: z.string(),
});
 
const judgePayload = z.object({
  question: z.string(),
  answer: z.string(),
});
 
const queue = new QueueClient();
const gateway = new AIGatewayClient();
const app = new Hono();
const ANSWER_MODEL = 'deepseek/deepseek-v4-flash';
const JUDGE_MODEL = 'googleai/gemini-3.5-flash';
 
app.post('/api/answer', async (c) => {
  const { question } = await c.req.json<{ question: string }>();
 
  const answerResult = await gateway.completeText({
    model: ANSWER_MODEL,
    messages: [{ role: 'user', content: question }],
  });
 
  if (!answerResult.hasText) {
    return c.json({ error: 'The model returned no text.' }, 502);
  }
 
  // hand the (question, answer) pair to the worker for background scoring
  await queue.publish('answer-quality', { question, answer: answerResult.text });
 
  return c.json({ answer: answerResult.text });
});
 
app.post('/api/workers/answer-quality', async (c) => {
  const body = judgePayload.parse(await c.req.json());
 
  const { data } = await gateway.completeStructured({
    model: JUDGE_MODEL,
    messages: [
      {
        role: 'user',
        content: `Score this answer.
 
Question: ${body.question}
Answer: ${body.answer}
 
- score: 0 = useless, 1 = extremely helpful
- answersQuestion: directly addresses the question?
- actionable: can the user act on this?`,
      },
    ],
    response_schema: { name: 'helpfulness_judgment', schema: helpfulnessJudgment },
  });
 
  const output = helpfulnessJudgment.parse(data);
 
  logger.info('judge complete', {
    score: output.score,
    answersQuestion: output.answersQuestion,
  });
 
  return c.json({ ok: true });
});
 
export default app;

The worker route is fired by an HTTP destination on the answer-quality queue. See Background Work for the full request/status/worker shape.

Prompt Structure

Judge prompts work better when the rubric is explicit and the scale is fixed.

You are evaluating a {taskType} response.
 
CONTEXT:
{context}
 
RESPONSE TO EVALUATE:
{response}
 
SCORING (0.0-1.0):
- criterion1: what to look for
- criterion2: what to look for
 
CHECKS (pass/fail):
- check1: yes/no question with clear answer
- check2: yes/no question with clear answer
 
Return scores and a one-sentence reason.

pin a single scale ("0.0 to 1.0") rather than letting the model pick its own
separate score-able criteria from binary checks
include the original question or source in the prompt; the judge has no other context
prefer faster judge models for lower added latency, and keep the exact model ID in configuration

Cost Notes

LLM-as-judge doubles the model calls per request. Manage the cost by:

using smaller, faster judge models selected from the live provider or AI Gateway catalog
moving the judge off the request path with Queues when the score does not affect the response
sampling a percentage of requests in high-volume traffic
batching judgments when the rubric is the same across many items

Next Steps

Build a RAG Agent: the answer pipeline this judge plugs into
Background Work: wire the judge worker route to a queue with retries and status
AI Gateway: local dev routing and deployed provider env choices