Prompt Engineering: Production Prompt Systems with Evals, Fallbacks and Observability

01The prompt patterns: few-shot, CoT, ReAct, self-consistency

We use four core prompt patterns, each addressing a different failure mode. Few-shot prompting embeds 2–8 worked examples in the prompt to demonstrate the desired output format and reasoning style — it lifts accuracy 8–18% on classification and extraction tasks versus zero-shot. Chain-of-thought (CoT) asks the model to reason step-by-step before answering, which adds 100–400ms latency but lifts accuracy 12–30% on multi-step reasoning tasks (math, logic, causal inference).

ReAct (Reason-Act-Observe) interleaves reasoning and tool calls — the model thinks ('I need to check the user's order status first'), calls a tool, observes the result, then continues. This is the dominant pattern for agentic workflows. Self-consistency runs the same CoT prompt 3–5 times and takes the majority answer — it adds cost but reduces variance on high-stakes single-answer tasks like medical triage. We select the pattern per use case, not per preference: classification gets few-shot, math gets CoT, multi-step workflows get ReAct, regulated decisions get self-consistency.

System message

The top-level instruction that defines the model's role, constraints, and output format. Treated as privileged context — user input is appended below and treated as data, not commands.

Few-shot example

A worked input/output pair embedded in the prompt to demonstrate the desired behaviour. 2–8 examples is typical; more than 10 rarely helps and burns context.

Constrained decoding

A technique (Instructor, Outlines, llama.cpp grammars) that forces the model's output to match a JSON schema or grammar at the token level. Eliminates JSON parse errors entirely.

02Structured output: JSON schema as a first-class citizen

Asking an LLM to 'return JSON' and then parsing the result with json.loads() fails 2–8% of the time in production — the model adds prose, escapes characters wrong, or truncates on token limits. Production prompt systems enforce structured output via constrained decoding: the model's token sampler is restricted to only emit tokens that produce valid JSON conforming to a Pydantic/JSON-schema definition. This drops parse-failure rate from 5% to 0%.

We use Instructor (Python) and Outlines (Python) for constrained decoding on OpenAI and Anthropic APIs, and llama.cpp grammars for self-hosted Llama/Mistral. The schema is the contract: a prompt that emits `TicketClassification{intent, confidence, suggested_action}` cannot produce prose, cannot omit a field, cannot produce a wrong type. Downstream code treats the model's output as a typed object, not a string to be regex-parsed. This is the single highest-ROI technique in production prompt engineering — it converts LLM output from 'text that mostly parses' to 'typed data that always parses'.

Pydantic model

A Python class defining the structure of the LLM's output — field names, types, validators. Instructor compiles this into a JSON schema the model is forced to obey.

Tool/function schema

A JSON Schema definition of a function the model can call. Used in ReAct patterns where the model decides which tool to invoke and with what arguments.

Validator

A custom check on a field — e.g. 'confidence must be 0.0–1.0', 'suggested_action must be one of [escalate, auto_respond, tag]'. Failed validators trigger a retry with the error message fed back to the model.

03Evaluation: the discipline that separates toys from systems

An unmeasured prompt is unmeasurable software. We build every production prompt with a minimum of 200 test cases — drawn from real production traffic, synthetic edge cases, and adversarial red-team inputs. Each case asserts on factual accuracy (does the answer match the expected output?), format compliance (does the JSON validate?), refusal behaviour (does the model correctly decline out-of-scope queries?), and latency (P50 and P95). The suite runs on every prompt change and every model upgrade.

We use Promptfoo for offline eval (CLI-runnable, 200 cases in 90 seconds, diff against last run), LangSmith for online eval (production traces sampled and re-scored), and DSPy's automatic prompt optimizer for cases where the prompt can be compiled from a declarative signature. The eval suite is the contract: a prompt that scores 92% ships; a prompt that scores 91% does not. Without this gate, prompt changes regress accuracy by 3–8% per quarter — invisible until customer complaints arrive.

04Prompt templates, composition and programmatic optimization

Production prompts are rarely a single string. They are templates with variables (user query, retrieved context, system constraints) composed at runtime. We use LangChain's PromptTemplate, Jinja2, or DSPy's signature system to manage this composition. Templates live in git, are unit-tested, and are versioned alongside the eval suite. A change to the template triggers a full eval run before deployment.

DSPy takes this further: instead of hand-writing prompts, you declare a signature (e.g. 'ticket_text -> classification, confidence') and DSPy compiles it into a tested prompt via automated search over few-shot examples, instruction phrasing, and reasoning patterns. DSPy-compiled prompts typically score 4–12% higher than hand-written prompts on the same eval suite, because the optimizer explores variations a human would never try. We use DSPy for high-volume prompts where the 4–12% lift compounds, and hand-write + Promptfoo-test for low-volume prompts where the engineering overhead doesn't pay back.

ClickTake Production Prompt System vs. Hand-written prompts vs. No-code prompt platform vs. Fine-tuned model

Prompt pattern selection — when to use what

Model providers

Eval & observability

Structured output & optimization

CI/CD & deployment

Pricing & Timelines

Technical Specs

Safety & Compliance

Working with ClickTake