Scenarioshigh-yieldstructured-output

Scenario: Structured Data Extraction

One full lesson on pulling JSON out of documents. Few-shot fixes, schema design, validation, review, and scale.

On this page
PreviousNext
Practice questions

Scenario: Structured Data Extraction

The full picture

The exam tells one story here. A system pulls fields out of messy documents. Think invoices, contracts, and reports. It checks the output against a JSON schema. It must stay accurate.

The cast of this lesson:

  • Few-shot examples, which fix format and wording problems.
  • null and optional fields, which stop the model from guessing.
  • A tool with your schema plus tool_choice, which locks the JSON shape.
  • Human review and sampling, which keep quality high over time.
  • Batch API and chunking, which handle scale and cost.

Almost every question maps to one of three reflex answers. Learn those first.

flowchart TD
    A["Messy document<br/>invoice, contract, report"] --> B["Claude extracts the fields"]
    B --> C["Tool with your schema<br/>tool_choice forces it"]
    C --> D{"Valid JSON?"}
    D -->|"No"| E["Follow-up request<br/>with the validation error"]
    E --> C
    D -->|"Yes"| F{"Confident and clear?"}
    F -->|"No"| G["Human review"]
    F -->|"Yes"| H["Accept"]
    H --> I["Sampling and accuracy checks"]
    B -.->|"Format is wrong"| J["Few-shot examples"]
    B -.->|"Data is missing"| K["Return null, optional fields"]
Remember
  • Format problems → few-shot examples.
  • Missing data → null or optional fields. Never guess.
  • Guaranteed JSON shape → a tool with your schema plus tool_choice.
  • Add strict: true for a hard guarantee. Otherwise validate and retry.

1. The three reflex answers

These three cover most questions. Read the symptom, then pick the fix.

flowchart TD
    A{"What is broken?"} -->|"Format is inconsistent"| B["Add few-shot examples"]
    A -->|"Model invents missing values"| C["Instruct: return null when<br/>not stated in the source"]
    A -->|"Output must match a schema"| D["Define a tool whose input schema<br/>IS your target schema"]

2. Prompt fixes: format and missing data

The first two reflexes live in the prompt. Here is how the exam words them.

Problem in the questionCorrect approach
Dates and amounts come back in mixed formatsA strict output schema plus normalization rules in the prompt
Compound phrases split wrong, granularity is offFew-shot examples that show the correct handling
Informal values like "about two dozen" get converted or droppedFew-shot examples that extract the text verbatim
Varied document layouts break extractionFew-shot examples across varied structures
The model fills absent fields with guessesTell it to return null when the value is not directly stated
Remember

Few-shot examples teach shape and wording. The null rule teaches honesty.

3. Schema design

Prompts guide the model. The schema decides what the system accepts.

Problem in the questionCorrect approach
The schema rejects documents that lack optional infoMake those fields optional, not required
An enum field meets a brand new categoryAdd "other" to the enum plus a detail string field
You need a guaranteed JSON structureDefine a tool with your schema. Read the result from tool_use. Force it with tool_choice: {type: "tool", name: ...}

The strength of the guarantee matters:

  • tool_choice forces the model to call your tool.
  • strict: true gives a hard guarantee on the shape.
  • Without strict: true, you still validate and retry.

4. Validation and hard documents

Even a good schema fails sometimes. These are the recovery answers.

Problem in the questionCorrect approach
The output fails JSON validation sometimesSend a follow-up request with the validation error. Ask the model to correct it
Totals are sometimes wrongHave the model output calculated_total and stated_total. Flag mismatches for human review
Amended documents change a value later in the textThe schema captures multiple values with source location and effective date

5. Human review and quality control

Some output still needs a person. The exam asks who, and how much.

Problem in the questionCorrect approach
Which extractions go to human review?Ones with low confidence. Also ambiguous or contradictory source documents
Can we trust high-confidence extractions?Check accuracy by document type and by field. Not only the aggregate number
Ongoing quality control of high-confidence outputStratified random sampling. A fixed percent every week
How do we calibrate review thresholds?Field-level confidence scores plus a labeled validation set
Remember

Never review 100% of extractions. That costs too much. Never review 0%. That is too risky.

6. Scale, context, and cost

Quality is set. Now run it over thousands of documents.

Problem in the questionCorrect approach
Long documents get worse near the endThe tool definitions, the prompt, and the document together approach the context limit
300 of 10,000 documents failed because they were too longResubmit only the failed ones, in chunks. Merge the partial results
An overnight batch where cost mattersUse the Batch API for 50% off. Send urgent items to the real-time API

7. Traps

These answers sound sensible. On the exam they are wrong.

  • ❌ Lowering the temperature or saying "be careful" to fix format issues.
  • ❌ Making every field required "for data quality".
  • ❌ Reprocessing all 10,000 documents because 300 failed.
  • ❌ Human review for 100% of extractions (cost) or 0% (risk).

See also

Recap

  • The story: extract fields from invoices, contracts, and reports. Validate against JSON schemas. Stay accurate.
  • Format inconsistent → few-shot examples.
  • Model invents missing values → tell it to return null when not stated.
  • Output must match a schema → define a tool whose input schema is your target schema.
  • Mixed dates and amounts → strict schema plus normalization rules in the prompt.
  • Wrong splits or granularity → few-shot examples with correct handling.
  • Informal values like "about two dozen" → few-shot examples, extract verbatim.
  • Varied layouts → few-shot examples across varied structures.
  • Schema rejects documents missing optional info → make fields optional.
  • New enum category → add "other" plus a detail string field.
  • Guaranteed JSON → tool definition, read tool_use, force with tool_choice: {type: "tool", name: ...}.
  • strict: true gives a hard guarantee. Without it, validate and retry.
  • Validation fails sometimes → follow-up request with the validation error.
  • Wrong totals → output calculated_total and stated_total. Flag mismatches for human review.
  • Amended documents → schema holds multiple values with source location and effective date.
  • Send to human review: low confidence, or ambiguous or contradictory sources.
  • Trust check: accuracy by document type and field, not only aggregate.
  • Ongoing QC: stratified random sampling, a fixed percent weekly.
  • Threshold calibration: field-level confidence plus a labeled validation set.
  • Long documents degrade because tool definitions, prompt, and document approach the context limit.
  • 300 of 10,000 failed → resubmit only those, chunked, and merge results.
  • Overnight batch → Batch API, 50% off. Urgent work → real-time API.
  • Traps: temperature or "be careful" for format, all fields required, reprocessing all 10,000 docs, and 100% or 0% human review.

Next: Answer Patterns — The Complete Lesson