OurSharedCode › Prompt Engineering Studio › Guides

Writing output formats a program can parse

By Mark · 7 September 2026

The job ran nightly for six weeks and then failed at ten past three in the morning. The model had wrapped its JSON in a markdown code fence — three backticks, the word json, the object, three backticks — which it had not done once in six weeks and did that night. Nothing had changed on my side.

That is the shape of this whole problem. Getting structured output from a model is easy and you will believe you have solved it. Getting the same structure on the two-hundredth call, or after a model upgrade, is a different piece of work, and it is half prompt and half code. Anyone who tells you it is only prompt has not run one of these for a year.

Use the API's structured output if it has one

Before any prompt technique: most providers now offer a JSON or schema-constrained mode that enforces the shape at generation time rather than asking nicely. If yours does, use it. It removes the fence problem, the preamble problem and the trailing-comma problem in one move, and every trick below is a fallback for when you cannot.

What it does not do — worth being clear, because people over-trust it — is make the values right. A response can satisfy your schema perfectly and still be wrong about every field in it. Structure and truth are separate problems and only one of them is being solved for you.

Show the schema, do not describe it

A literal example of the output beats a paragraph describing the fields, every time, for the same reason examples beat instructions generally. 

Weak:
Return JSON with a category, an urgency level, a boolean for
whether a human is needed, and optionally a note.

Better:
Reply with exactly this JSON and nothing else:

{
  "category": "billing" | "account" | "technical" | "other",
  "urgency": "low" | "normal" | "high",
  "needs_human": true | false,
  "note": "one sentence, or null"
}

The alternatives written out as literals matter more than they look. "An urgency level" invites urgent, High, P1, medium-high — all reasonable, none of them in your enum. Spell the permitted values, and say what to do when none fits, or the model will invent a value rather than pick the wrong one.

Define what "nothing here" looks like

The single most common structural failure I see is a field that vanishes. The model had nothing to put in note, and rather than write something untrue it left the key out, and your code did result["note"] and threw.

Every optional field needs an explicit empty value and an instruction to use it: 

Every key above must be present in every response. When a value
is unknown or does not apply, use null — never omit the key, and
never invent a value to fill the space.

Same for lists. "Return the action items" produces [] from some models and the key missing from others when there are none. Say it: an empty list is [].

Give the urge to explain somewhere to go

Models want to introduce their output. "Here is the JSON you requested:" is not disobedience, it is helpfulness leaking into a channel where it breaks things. Forbidding it works better when you also provide an outlet: 

Output the JSON object only. No preamble, no explanation, no
markdown fences. If you need to say something about the answer,
put it in the "note" field — that is what it is for.

This is the positive-framing point from the first guide applied to formatting. "Do not explain" fights a tendency; "put the explanation here" redirects it, and redirecting wins.

Keep it flat

Deeply nested output degrades faster than flat output. Three levels of objects inside arrays inside objects, and you start seeing a bracket closed in the wrong place — rare, but rare is a nightly job failing every few weeks.

Flatten where you can. customer_name and customer_id as top-level keys are duller than a nested customer object and they survive better. If you genuinely need hierarchy, consider two calls: extract flat, assemble in code. Your code is much better at building nested structures than a language model is, and it does it the same way every time.

The code half

No prompt gets you to a hundred per cent, so the parser has to expect failure. Mine, in order:

  1. Strip fences before parsing. Look for a fenced block and take its contents; otherwise take the substring from the first { to the last }. Two lines, and they retire the 3am failure permanently.
  2. Validate against a schema, not just JSON-parse. Valid JSON with a missing key or an out-of-enum value is exactly the input that gets deep into your system before breaking.
  3. Retry once, with the error. Send back what you got and the validation message: "that was not valid — <error> — return only the corrected JSON." A large share of failures fix themselves on that pass.
  4. Log the raw text on failure. Not the exception, the response. You cannot diagnose a format failure you did not keep, and it will not reproduce.
  5. Cap the retries and have a defined giving-up path. A retry loop with no ceiling turns a bad response into a bill.

Measure it, and measure it again on upgrade

Run fifty real inputs and count how many parse and validate first time. Fifty, because a failure that happens twice in a hundred calls will not show up in five and will absolutely show up in production.

Then keep those fifty. A model upgrade is a formatting change — a new version can be better at your task and differently shaped in its output, and the release notes will not mention the thing that breaks you. The fifty inputs you already have are the cheapest regression test you will ever write.