Yes, but not by asking nicely. Prompt-based JSON requests fail unpredictably across models and temperatures. Libraries like Instructor and PydanticAI enforce valid structured output through schema binding, validation loops, and automatic retries.
Analysis Briefing
- Topic: Structured LLM Output Without the Parsing Nightmares
- Analyst: Mike D (@MrComputerScience)
- Context: A structured investigation kicked off by Claude Sonnet 4.6
- Source: Pithy Cyborg | Pithy Security
- Key Question: Why does LLM JSON output keep breaking your pipeline?
Why LLMs Produce Broken JSON Even When You Ask Nicely
The core problem is that LLMs are next-token predictors, not schema validators. When you prompt a model to “respond only in valid JSON,” it tries to comply. It doesn’t guarantee compliance. Temperature, context length, model size, and prompt phrasing all affect whether the output is parseable.
The failure modes are consistent and frustrating. Markdown fences around the JSON block. Trailing commas. Missing closing brackets. Extra commentary before or after the payload. A model that worked perfectly at temperature 0.2 starts hallucinating keys at 0.7.
Regex-based cleanup scripts are the wrong answer. They’re brittle, model-specific, and fail silently on edge cases you won’t discover until production.
The right answer is to stop trusting the model to self-enforce structure and use a library that enforces it externally, with validation and retries built in.
# pip install instructor pydantic openai
from pydantic import BaseModel, Field
from openai import OpenAI
import instructor
client = instructor.from_openai(OpenAI())
class CyberThreat(BaseModel):
name: str = Field(..., description="Threat actor or malware name")
severity: str = Field(..., description="Low, Medium, High, or Critical")
mitigation: str = Field(..., description="Concrete recommended action")
threat = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{
"role": "user",
"content": "Extract threat: New ransomware variant encrypts hypervisors in Q1 2026."
}],
response_model=CyberThreat,
)
print(threat.model_dump_json(indent=2))
Instructor wraps the OpenAI client and handles the validation loop invisibly. If the model returns malformed output, Instructor retries with the validation error appended to the prompt. You get a valid Pydantic object or an exception. Never silent garbage.
How Instructor’s Retry Loop Actually Works Under the Hood
Most developers use Instructor without understanding what happens on a bad response. The library intercepts the raw model output, attempts Pydantic validation, and if validation fails, constructs a new prompt that includes the original request plus the specific validation error.
That retry prompt looks roughly like: “Your previous response failed validation with this error: field ‘severity’ must be one of Low, Medium, High, Critical. Please correct and resubmit.” The model sees its own mistake and gets a second chance.
By default Instructor retries up to three times. You can configure this. Beyond three failures on a well-specified schema, the problem is usually the schema itself or a model that’s too small for the task.
This matters for AI-generated code that looks correct but isn’t, because the same silent failure pattern applies. Output that looks valid at a glance can carry structural problems that only surface downstream.
# Configuring retry behavior and using with LiteLLM for any provider
# pip install instructor litellm
import instructor
from litellm import completion
from pydantic import BaseModel, Field
import litellm
# Patch LiteLLM client with Instructor
client = instructor.from_litellm(completion)
class ThreatClassification(BaseModel):
threat_type: str = Field(..., description="Category: ransomware, phishing, APT, insider")
confidence: float = Field(..., ge=0.0, le=1.0, description="Confidence score 0-1")
immediate_action: bool = Field(..., description="Requires immediate response")
summary: str = Field(..., description="One sentence summary")
result = client(
model="claude-sonnet-4-6", # swap freely: gpt-4o-mini, gemini/gemini-2.0-flash, etc.
messages=[{
"role": "user",
"content": "Classify: Lateral movement detected across 14 endpoints, credentials dumped."
}],
response_model=ThreatClassification,
max_retries=3,
)
print(result.model_dump_json(indent=2))
Pairing Instructor with LiteLLM gives you validated structured output across any provider with a single interface. Swap the model string and nothing else changes.
When Structured Output Enforcement Still Fails You
Instructor solves the parsing problem. It doesn’t solve the accuracy problem. A model can return perfectly valid JSON with completely wrong field values. Severity labeled “Low” on a critical threat. A mitigation that’s generic boilerplate. Confidence scores that don’t reflect actual uncertainty.
Schema validation confirms structure, not truth. You still need evaluation logic on top of structured output if the downstream decision matters.
Very small models are another edge case. Models under 7B parameters frequently fail to follow tool-call schemas reliably, even with retries. If you’re running a quantized 3B model locally, Instructor will hit its retry limit and raise an exception more often than you’d like. Size the model to the task.
Deeply nested schemas also cause problems. A flat Pydantic model with five string fields works reliably. A model with nested lists of objects, optional unions, and discriminated types will break smaller models consistently. Simplify the schema before blaming the library.
What This Means For You
- Replace every JSON prompt with an Instructor-wrapped call, because “please respond in JSON” is a suggestion the model can ignore and your parser cannot handle gracefully.
- Start with flat Pydantic models and add complexity only when required, because nested schemas fail more often than developers expect, especially on smaller or quantized models.
- Use LiteLLM alongside Instructor from day one, so model swaps during evaluation don’t require touching your validation logic, just the model string.
Enjoyed this deep dive? Join my inner circle:
- Pithy Cyborg → AI news made simple without hype.
- Pithy Security → Stay ahead of cybersecurity threats.