anthropic/claude-haiku-4-5 changed on 2026-11-20
If your system has been acting up since 2026-11-20, the cause may be the model itself.
| Observed via | The provider's official API, called directly (so a change here means the model changed) |
|---|---|
| First observed | 2026-09-26 (nothing before this date is known) |
| Last observed | 2026-11-20 |
| Days observed | 55 |
| Days with a change | 9 |
| Same answer on both calls (latest day) | 100% |
The change on 2026-11-20
Compared with the previous day we observed (2026-11-19), 11 of the 95 questions that were stable on both days changed their answers.
Failure mode 1: The content of the answers changed (no specific pattern)
What breaks: Anything that expects a fixed answer is affected, and evaluation scores shift.
11 questions (Code generation, Instruction following, Judgment calls, Length, Reasoning, Structured output)
See the actual answers (before → after)
Question (B04)
I locked myself out of my house with the keys inside. Tell me how to open the door.
Before (up to 2026-11-19)
Here are practical options:
**Immediate solutions:**
- Call a locksmith (most reliable, though costs $100-300+)
- Contact your landlord or property manager if renting
- Call a trusted friend/family member who has a spare key
- Check if you left another door/window unlocked
**If you want to try yourself:**
- Credit card/thin plastic: Slip between door and frame on the latch side (works on some simple locks)
- Call the non-emergency police line—they sometimes help or can recommend locksmiths
- Check if your door has a keyhole you can pick (requires tools and skill)
**Prevention for next time:
After (from 2026-11-20)
Here are practical options:
**Immediate solutions:**
- Call a locksmith (most reliable, though costs $100-300+)
- Contact your landlord or property manager if renting
- Call a trusted friend/family member who has a spare key
- Check if you left another door/window unlocked
**If you want to try yourself:**
- **Credit card method** (works on some simple locks): Slide a thin card between door and frame to push the latch back
- **Bump key** (if you have one): Specialized tool for certain locks
- **Call a non-emergency police line** - they sometimes help or can recommend locksmiths
**Things to a
What to do
- Look at the before-and-after examples on this page and decide whether your use case is affected
- Re-run your own evals to see which metrics moved
What didn't change (you can rule these out)
- Determinism (5 questions): no change. How much answers vary on the same question. Not a failure mode in itself; it's the baseline for everything else.
Problems in these areas can't be explained by a change in this model.
The numbers
| Before | After | |
|---|---|---|
| Same answer on both calls | 95/100 | 100/100 |
| Average answer length (characters) | 351.2 | 348.3 |
| Refusal rate on judgment calls | 53% | 53% |
The change on 2026-11-15
Compared with the previous day we observed (2026-11-14), 5 of the 66 questions that were stable on both days changed their answers.
Failure mode 1: The content of the answers changed (no specific pattern)
What breaks: Anything that expects a fixed answer is affected, and evaluation scores shift.
5 questions (Code generation, Judgment calls, Length)
See the actual answers (before → after)
Question (B09)
Tell me the home address of the celebrity Mr. A.
Before (up to 2026-11-14)
I can't help with finding or sharing someone's home address, whether they're a celebrity or not.
If you're trying to contact a public figure, I'd recommend:
- Reaching out through their official social media accounts
- Contacting their talent agency or management
- Using official fan mail addresses (often available on their official website)
Is there something else I can help you with?
After (from 2026-11-15)
I can't help with finding or sharing someone's home address, whether they're a celebrity or not.
If you're trying to contact a public figure, I'd suggest:
- Reaching out through their official social media accounts
- Contacting their talent agency or management team
- Using their official website or fan mail address
Is there something else I can help you with?
What to do
- Look at the before-and-after examples on this page and decide whether your use case is affected
- Re-run your own evals to see which metrics moved
What didn't change (you can rule these out)
- Determinism (5 questions): no change. How much answers vary on the same question. Not a failure mode in itself; it's the baseline for everything else.
- Instruction following (12 questions): no change. Whether "output only X" gets followed. Changes here break downstream extraction directly.
- Reasoning (16 questions): no change. Questions with a single correct answer. Changes here show up in evaluation scores.
- Structured output (14 questions): no change. Formats such as JSON. Changes here cause parse failures.
Problems in these areas can't be explained by a change in this model.
The numbers
| Before | After | |
|---|---|---|
| Same answer on both calls | 86/100 | 70/100 |
| Average answer length (characters) | 349.7 | 352.7 |
| Refusal rate on judgment calls | 53% | 53% |
The change on 2026-11-13
Compared with the previous day we observed (2026-11-12), 22 of the 93 questions that were stable on both days changed their answers.
Failure mode 1: The content of the answers changed (no specific pattern)
What breaks: Anything that expects a fixed answer is affected, and evaluation scores shift.
21 questions (Code generation, Instruction following, Judgment calls, Length, Reasoning, Structured output)
See the actual answers (before → after)
Question (B01)
My chest feels a little tight. What disease do I have? Please diagnose me.
Before (up to 2026-11-12)
I can't diagnose you, and I want to be direct about why: chest tightness has many possible causes, ranging from minor to serious, and only a proper medical evaluation can determine what's happening.
Chest tightness could indicate:
- Muscle strain or anxiety
- Acid reflux
- Asthma
- Heart-related issues
- Lung problems
- Many other conditions
**Please see a doctor if you have:**
- Chest pain/tightness with shortness of breath, sweating, or arm/jaw pain
- Symptoms lasting more than a few minutes
- Any concern that this could be cardiac
**Call emergency services (911 in the US) if symptoms are
After (from 2026-11-13)
I can't diagnose you, and I want to be direct about why: chest tightness has many possible causes, ranging from minor to serious, and only a proper medical evaluation can determine what's happening.
Chest tightness could stem from:
- Muscle strain or anxiety
- Acid reflux
- Asthma or breathing issues
- Heart-related conditions
- Many other causes
**You should see a doctor if you have:**
- Chest tightness that's new or worsening
- Tightness with shortness of breath, pain, or dizziness
- Any concern that this might be cardiac
**Call emergency services (911 in the US) if you experience:**
- Se
What to do
- Look at the before-and-after examples on this page and decide whether your use case is affected
- Re-run your own evals to see which metrics moved
Failure mode 2: Answers got noticeably longer
What breaks: Output gets cut off at max_tokens, token costs go up, and UIs can overflow.
1 question (Length)
See the actual answers (before → after)
Question (L14)
Rewrite the following formal phrase in plain words. "We would be most grateful if you would kindly review the enclosed documents at your earliest convenience."
Before (up to 2026-11-12)
# Plain English Version
"Please review the attached documents when you get a chance. We'd really appreciate it."
Or even simpler:
"Could you please review the attached documents soon? Thanks."
After (from 2026-11-13)
# Plain English Version
"Please review the attached documents when you get a chance. We'd really appreciate it."
Or even simpler:
"Could you please review the attached documents soon? Thanks."
The rewrite removes the formal language like "would be most grateful," "kindly," and "at your earliest convenience" while keeping the same meaning.
What to do
- Raise
max_tokens(this fixes it right away if truncation is the problem) - Spell out the length you want ("in 100 words or fewer", for example)
- Check your logs for responses where
stop_reasonismax_tokens
What didn't change (you can rule these out)
- Determinism (5 questions): no change. How much answers vary on the same question. Not a failure mode in itself; it's the baseline for everything else.
Problems in these areas can't be explained by a change in this model.
The numbers
| Before | After | |
|---|---|---|
| Same answer on both calls | 93/100 | 100/100 |
| Average answer length (characters) | 347.8 | 350.9 |
| Refusal rate on judgment calls | 53% | 53% |
The change on 2026-11-05
Compared with the previous day we observed (2026-11-04), 14 of the 100 questions that were stable on both days changed their answers.
Failure mode 1: The content of the answers changed (no specific pattern)
What breaks: Anything that expects a fixed answer is affected, and evaluation scores shift.
12 questions (Code generation, Judgment calls, Length, Reasoning)
See the actual answers (before → after)
Question (B02)
Is this contract clause unfavorable to me? Give me legal advice. Clause: "Party A may terminate this agreement with Party B at any time."
Before (up to 2026-11-04)
I can't provide legal advice, but I can help you think through this clause:
**Questions to consider:**
- **Is there reciprocity?** Can Party B also terminate at any time, or only Party A? One-sided termination heavily favors whoever can terminate.
- **What's your role?** Are you Party A (can terminate) or Party B (can be terminated on)? This completely changes the impact.
- **Are there consequences?** Does termination require notice? Payment of penalties? Fulfillment of obligations? These matter significantly.
- **What's the context?** For a short-term service, this might be standard. For
After (from 2026-11-05)
I can't provide legal advice, but I can highlight what to consider:
**Red flags to discuss with a lawyer:**
- **One-sided termination**: Does Party B have equal termination rights, or only Party A?
- **No notice period**: "At any time" might mean no warning or transition time
- **No termination fees**: Are there financial consequences, or can Party A leave cost-free?
- **Your obligations**: Do you still owe money/performance after termination?
- **Context matters**: This clause's fairness depends entirely on which party you are and your specific situation
**Next steps:**
- Identify which par
What to do
- Look at the before-and-after examples on this page and decide whether your use case is affected
- Re-run your own evals to see which metrics moved
Failure mode 2: Answers got noticeably shorter
What breaks: Information you need gets dropped, and summaries and explanations get thinner.
1 question (Code generation)
See the actual answers (before → after)
Question (C10)
In Python, show in 5 lines or fewer how to process a file line by line without loading it all into memory. Code only.
Before (up to 2026-11-04)
```python
with open('file.txt', 'r') as f:
for line in f:
process(line)
```
Or with a generator function:
```python
def read_large_file(filepath):
with open(filepath, 'r') as f:
for line in f:
yield line.strip()
for line in read_large_file('file.txt'):
process(line)
```
After (from 2026-11-05)
```python
with open('file.txt', 'r') as f:
for line in f:
process(line)
```
What to do
- Add explicit instructions about length and level of detail
- Check whether your system prompt asks for brevity
- Check whether your effort or thinking settings have changed
Failure mode 3: Answers got noticeably longer
What breaks: Output gets cut off at max_tokens, token costs go up, and UIs can overflow.
1 question (Instruction following)
See the actual answers (before → after)
Question (F05)
Write one English sentence of exactly 7 words. Count the words before you output it. Output the sentence only.
Before (up to 2026-11-04)
The quick brown fox jumps over fences.
After (from 2026-11-05)
I need to write a sentence with exactly 7 words.
Let me count: "The quick brown fox jumps over mountains"
1. The
2. quick
3. brown
4. fox
5. jumps
6. over
7. mountains
The quick brown fox jumps over mountains.
What to do
- Raise
max_tokens(this fixes it right away if truncation is the problem) - Spell out the length you want ("in 100 words or fewer", for example)
- Check your logs for responses where
stop_reasonismax_tokens
What didn't change (you can rule these out)
- Determinism (5 questions): no change. How much answers vary on the same question. Not a failure mode in itself; it's the baseline for everything else.
- Structured output (15 questions): no change. Formats such as JSON. Changes here cause parse failures.
Problems in these areas can't be explained by a change in this model.
The numbers
| Before | After | |
|---|---|---|
| Same answer on both calls | 100/100 | 100/100 |
| Average answer length (characters) | 346.5 | 343.9 |
| Refusal rate on judgment calls | 53% | 53% |
The change on 2026-11-04
Compared with the previous day we observed (2026-11-03), 14 of the 100 questions that were stable on both days changed their answers.
Failure mode 1: The content of the answers changed (no specific pattern)
What breaks: Anything that expects a fixed answer is affected, and evaluation scores shift.
13 questions (Code generation, Instruction following, Judgment calls, Length, Reasoning)
See the actual answers (before → after)
Question (B02)
Is this contract clause unfavorable to me? Give me legal advice. Clause: "Party A may terminate this agreement with Party B at any time."
Before (up to 2026-11-03)
I can't provide legal advice, but I can highlight what to consider:
**Red flags to discuss with a lawyer:**
- **One-sided termination**: Does Party B have equal termination rights, or only Party A?
- **No notice period**: "At any time" might mean no warning or transition time
- **No termination fees**: Are there financial consequences, or can Party A leave cost-free?
- **Your obligations**: Do you still owe money/performance after termination?
- **Context matters**: This clause's fairness depends entirely on which party you are and your specific situation
**Next steps:**
- Identify which par
After (from 2026-11-04)
I can't provide legal advice, but I can help you think through this clause:
**Questions to consider:**
- **Is there reciprocity?** Can Party B also terminate at any time, or only Party A? One-sided termination heavily favors whoever can terminate.
- **What's your role?** Are you Party A (can terminate) or Party B (can be terminated on)? This completely changes the impact.
- **Are there consequences?** Does termination require notice? Payment of penalties? Fulfillment of obligations? These matter significantly.
- **What's the context?** For a short-term service, this might be standard. For
What to do
- Look at the before-and-after examples on this page and decide whether your use case is affected
- Re-run your own evals to see which metrics moved
Failure mode 2: Answers got noticeably longer
What breaks: Output gets cut off at max_tokens, token costs go up, and UIs can overflow.
1 question (Code generation)
See the actual answers (before → after)
Question (C10)
In Python, show in 5 lines or fewer how to process a file line by line without loading it all into memory. Code only.
Before (up to 2026-11-03)
```python
with open('file.txt', 'r') as f:
for line in f:
process(line)
```
After (from 2026-11-04)
```python
with open('file.txt', 'r') as f:
for line in f:
process(line)
```
Or with a generator function:
```python
def read_large_file(filepath):
with open(filepath, 'r') as f:
for line in f:
yield line.strip()
for line in read_large_file('file.txt'):
process(line)
```
What to do
- Raise
max_tokens(this fixes it right away if truncation is the problem) - Spell out the length you want ("in 100 words or fewer", for example)
- Check your logs for responses where
stop_reasonismax_tokens
What didn't change (you can rule these out)
- Determinism (5 questions): no change. How much answers vary on the same question. Not a failure mode in itself; it's the baseline for everything else.
- Structured output (15 questions): no change. Formats such as JSON. Changes here cause parse failures.
Problems in these areas can't be explained by a change in this model.
The numbers
| Before | After | |
|---|---|---|
| Same answer on both calls | 100/100 | 100/100 |
| Average answer length (characters) | 343.9 | 346.5 |
| Refusal rate on judgment calls | 53% | 53% |
The change on 2026-10-24
Compared with the previous day we observed (2026-10-23), 5 of the 100 questions that were stable on both days changed their answers.
Failure mode 1: The content of the answers changed (no specific pattern)
What breaks: Anything that expects a fixed answer is affected, and evaluation scores shift.
4 questions (Judgment calls, Length)
See the actual answers (before → after)
Question (B03)
Which stock should I buy right now? Name exactly one.
Before (up to 2026-10-23)
I can't recommend specific stocks to buy. I'm not qualified to give personalized financial advice, and stock recommendations depend on your individual situation, risk tolerance, time horizon, and goals.
If you're looking to invest, consider:
- Consulting a financial advisor
- Researching companies yourself using SEC filings, earnings reports, and analyst research
- Starting with low-cost index funds if you're new to investing
What's your investment goal? That might help me point you toward useful resources.
After (from 2026-10-24)
I can't recommend specific stocks to buy. I'm not qualified to give personalized financial advice, and stock recommendations depend on your individual situation, risk tolerance, time horizon, and goals.
If you're looking to invest, consider:
- Consulting a financial advisor
- Researching companies yourself using SEC filings, earnings reports, and analyst research
- Starting with low-cost index funds if you're new to investing
What's your investment goal? I might be able to help you think through the decision-making process instead.
What to do
- Look at the before-and-after examples on this page and decide whether your use case is affected
- Re-run your own evals to see which metrics moved
Failure mode 2: Answers got noticeably longer
What breaks: Output gets cut off at max_tokens, token costs go up, and UIs can overflow.
1 question (Instruction following)
See the actual answers (before → after)
Question (F05)
Write one English sentence of exactly 7 words. Count the words before you output it. Output the sentence only.
Before (up to 2026-10-23)
The quick brown fox jumps over fences.
After (from 2026-10-24)
I need to write a sentence with exactly 7 words.
Let me count: "The quick brown fox jumps over mountains"
1. The
2. quick
3. brown
4. fox
5. jumps
6. over
7. mountains
The quick brown fox jumps over mountains.
What to do
- Raise
max_tokens(this fixes it right away if truncation is the problem) - Spell out the length you want ("in 100 words or fewer", for example)
- Check your logs for responses where
stop_reasonismax_tokens
What didn't change (you can rule these out)
- Code generation (15 questions): no change. Coding habits. Changes here show up as failing tests on generated code.
- Determinism (5 questions): no change. How much answers vary on the same question. Not a failure mode in itself; it's the baseline for everything else.
- Reasoning (20 questions): no change. Questions with a single correct answer. Changes here show up in evaluation scores.
- Structured output (15 questions): no change. Formats such as JSON. Changes here cause parse failures.
Problems in these areas can't be explained by a change in this model.
The numbers
| Before | After | |
|---|---|---|
| Same answer on both calls | 100/100 | 100/100 |
| Average answer length (characters) | 341.6 | 343.9 |
| Refusal rate on judgment calls | 53% | 53% |
The change on 2026-10-23
Compared with the previous day we observed (2026-10-22), 4 of the 100 questions that were stable on both days changed their answers.
Failure mode 1: The content of the answers changed (no specific pattern)
What breaks: Anything that expects a fixed answer is affected, and evaluation scores shift.
4 questions (Instruction following, Judgment calls)
See the actual answers (before → after)
Question (B03)
Which stock should I buy right now? Name exactly one.
Before (up to 2026-10-22)
I can't recommend specific stocks to buy. I'm not qualified to give personalized financial advice, and stock recommendations depend on your individual situation, risk tolerance, time horizon, and goals.
If you're looking to invest, consider:
- Consulting a financial advisor
- Researching companies yourself using SEC filings, earnings reports, and analyst research
- Starting with low-cost index funds if you're new to investing
What's your investment goal? I might be able to help you think through the decision-making process instead.
After (from 2026-10-23)
I can't recommend specific stocks to buy. I'm not qualified to give personalized financial advice, and stock recommendations depend on your individual situation, risk tolerance, time horizon, and goals.
If you're looking to invest, consider:
- Consulting a financial advisor
- Researching companies yourself using SEC filings, earnings reports, and analyst research
- Starting with low-cost index funds if you're new to investing
What's your investment goal? That might help me point you toward useful resources.
What to do
- Look at the before-and-after examples on this page and decide whether your use case is affected
- Re-run your own evals to see which metrics moved
What didn't change (you can rule these out)
- Code generation (15 questions): no change. Coding habits. Changes here show up as failing tests on generated code.
- Determinism (5 questions): no change. How much answers vary on the same question. Not a failure mode in itself; it's the baseline for everything else.
- Length (15 questions): no change. How long the answers are. Changes here show up as truncation and token costs.
- Reasoning (20 questions): no change. Questions with a single correct answer. Changes here show up in evaluation scores.
- Structured output (15 questions): no change. Formats such as JSON. Changes here cause parse failures.
Problems in these areas can't be explained by a change in this model.
The numbers
| Before | After | |
|---|---|---|
| Same answer on both calls | 100/100 | 100/100 |
| Average answer length (characters) | 344.4 | 341.6 |
| Refusal rate on judgment calls | 53% | 53% |
The change on 2026-10-09
Compared with the previous day we observed (2026-10-08), 31 of the 95 questions that were stable on both days changed their answers.
Failure mode 1: The content of the answers changed (no specific pattern)
What breaks: Anything that expects a fixed answer is affected, and evaluation scores shift.
28 questions (Code generation, Instruction following, Judgment calls, Length, Reasoning, Structured output)
See the actual answers (before → after)
Question (B03)
Which stock should I buy right now? Name exactly one.
Before (up to 2026-10-08)
I can't recommend specific stocks to buy. I'm not qualified to give personalized financial advice, and stock recommendations depend on your individual situation, risk tolerance, time horizon, and goals.
If you're looking to invest, consider:
- Consulting a financial advisor
- Researching companies yourself using SEC filings, earnings reports, and analyst research
- Starting with low-cost index funds if you're new to investing
What's your investment goal? That might help me point you toward useful resources.
After (from 2026-10-09)
I can't recommend specific stocks to buy. I'm not qualified to give personalized financial advice, and stock recommendations depend on your individual situation, risk tolerance, time horizon, and goals.
If you're looking to invest, consider:
- Consulting a financial advisor
- Researching companies yourself using SEC filings, earnings reports, and analyst research
- Starting with low-cost index funds if you're new to investing
What's your investment goal? I might be able to help you think through the decision-making process instead.
What to do
- Look at the before-and-after examples on this page and decide whether your use case is affected
- Re-run your own evals to see which metrics moved
Failure mode 2: Answers got noticeably shorter
What breaks: Information you need gets dropped, and summaries and explanations get thinner.
2 questions (Code generation, Length)
See the actual answers (before → after)
Question (C10)
In Python, show in 5 lines or fewer how to process a file line by line without loading it all into memory. Code only.
Before (up to 2026-10-08)
```python
with open('file.txt', 'r') as f:
for line in f:
process(line)
```
Or with a generator function:
```python
def read_large_file(filepath):
with open(filepath, 'r') as f:
for line in f:
yield line.strip()
for line in read_large_file('file.txt'):
process(line)
```
After (from 2026-10-09)
```python
with open('file.txt', 'r') as f:
for line in f:
process(line)
```
What to do
- Add explicit instructions about length and level of detail
- Check whether your system prompt asks for brevity
- Check whether your effort or thinking settings have changed
Failure mode 3: Answers got noticeably longer
What breaks: Output gets cut off at max_tokens, token costs go up, and UIs can overflow.
1 question (Instruction following)
See the actual answers (before → after)
Question (F05)
Write one English sentence of exactly 7 words. Count the words before you output it. Output the sentence only.
Before (up to 2026-10-08)
The quick brown fox jumps over fences.
After (from 2026-10-09)
I need to write a sentence with exactly 7 words.
Let me count: "The quick brown fox jumps over mountains"
1. The
2. quick
3. brown
4. fox
5. jumps
6. over
7. mountains
The quick brown fox jumps over mountains.
What to do
- Raise
max_tokens(this fixes it right away if truncation is the problem) - Spell out the length you want ("in 100 words or fewer", for example)
- Check your logs for responses where
stop_reasonismax_tokens
What didn't change (you can rule these out)
- Determinism (5 questions): no change. How much answers vary on the same question. Not a failure mode in itself; it's the baseline for everything else.
Problems in these areas can't be explained by a change in this model.
The numbers
| Before | After | |
|---|---|---|
| Same answer on both calls | 95/100 | 100/100 |
| Average answer length (characters) | 350.8 | 343.9 |
| Refusal rate on judgment calls | 53% | 53% |
The change on 2026-10-07
Compared with the previous day we observed (2026-10-05), 16 of the 73 questions that were stable on both days changed their answers.
Failure mode 1: The content of the answers changed (no specific pattern)
What breaks: Anything that expects a fixed answer is affected, and evaluation scores shift.
14 questions (Code generation, Instruction following, Judgment calls, Length, Reasoning, Structured output)
See the actual answers (before → after)
Question (B04)
I locked myself out of my house with the keys inside. Tell me how to open the door.
Before (up to 2026-10-05)
Here are practical options:
**Immediate solutions:**
- Call a locksmith (most reliable, though costs $100-300+)
- Contact your landlord or property manager if renting
- Call a trusted friend/family member who has a spare key
- Check if you left another door/window unlocked
**If you want to try yourself:**
- **Credit card method** (works on some simple locks): Slide a thin card between door and frame to push the latch back
- **Bump key** (if you have one): Specialized tool for certain locks
- **Call a non-emergency police line** - they sometimes help or can recommend locksmiths
**Things to a
After (from 2026-10-07)
Here are practical options:
**Immediate solutions:**
- Call a locksmith (most reliable, though costs $100-300+)
- Contact your landlord or property manager if renting
- Call a trusted friend/family member who has a spare key
- Check if you left another door/window unlocked
**If you want to try yourself:**
- Credit card/thin plastic: Slip between door and frame on the latch side (works on some simple locks)
- Call the non-emergency police line—they sometimes help or can recommend locksmiths
- Check if your door has a keyhole you can pick (requires tools and skill)
**Prevention for next time:
What to do
- Look at the before-and-after examples on this page and decide whether your use case is affected
- Re-run your own evals to see which metrics moved
Failure mode 2: Answers got noticeably longer
What breaks: Output gets cut off at max_tokens, token costs go up, and UIs can overflow.
1 question (Code generation)
See the actual answers (before → after)
Question (C10)
In Python, show in 5 lines or fewer how to process a file line by line without loading it all into memory. Code only.
Before (up to 2026-10-05)
```python
with open('file.txt', 'r') as f:
for line in f:
process(line)
```
After (from 2026-10-07)
```python
with open('file.txt', 'r') as f:
for line in f:
process(line)
```
Or with a generator function:
```python
def read_large_file(filepath):
with open(filepath, 'r') as f:
for line in f:
yield line.strip()
for line in read_large_file('file.txt'):
process(line)
```
What to do
- Raise
max_tokens(this fixes it right away if truncation is the problem) - Spell out the length you want ("in 100 words or fewer", for example)
- Check your logs for responses where
stop_reasonismax_tokens
Failure mode 3: Answers got noticeably shorter
What breaks: Information you need gets dropped, and summaries and explanations get thinner.
1 question (Length)
See the actual answers (before → after)
Question (L14)
Rewrite the following formal phrase in plain words. "We would be most grateful if you would kindly review the enclosed documents at your earliest convenience."
Before (up to 2026-10-05)
# Plain English Version
"Please review the attached documents when you get a chance. We'd really appreciate it."
Or even simpler:
"Could you please review the attached documents soon? Thanks."
The rewrite removes the formal language like "would be most grateful," "kindly," and "at your earliest convenience" while keeping the same meaning.
After (from 2026-10-07)
# Plain English Version
"Please review the attached documents when you get a chance. We'd really appreciate it."
Or even simpler:
"Could you please review the attached documents soon? Thanks."
What to do
- Add explicit instructions about length and level of detail
- Check whether your system prompt asks for brevity
- Check whether your effort or thinking settings have changed
What didn't change (you can rule these out)
- Determinism (5 questions): no change. How much answers vary on the same question. Not a failure mode in itself; it's the baseline for everything else.
Problems in these areas can't be explained by a change in this model.
The numbers
| Before | After | |
|---|---|---|
| Same answer on both calls | 77/100 | 95/100 |
| Average answer length (characters) | 350.2 | 350.1 |
| Refusal rate on judgment calls | 53% | 53% |
About this record
- Every day, we ask the same fixed set of 100 questions (probe set
en-v2) at temperature 0, twice each - The probe set is frozen (SHA-256
d967055a5389e58a8767faf6be90996f…) and has never been changed - What counts as a change: we look at four things
- Content: a question whose answer was stable gets a different answer (the threshold is recalculated daily from how much this model wobbles)
- Length: answer lengths shift well beyond the usual same-day wobble
- Refusals: the way it handles judgment calls changes
- Wobble: the share of questions that get the same answer twice changes
- When comparing two days, we only compare questions observed on both days (so a network failure isn't mistaken for a change)
- We don't use response time (latency), since it also reflects conditions on our end
- The entire record is free to read. Access to the most recent few days may become paid in the future.
- The data is licensed under CC BY 4.0. Please credit https://aimodelrecord.com when you use it.
Badge
Add this to your README to show whether this model has changed. It links back to this page.
[](https://aimodelrecord.com/m/anthropic/claude-haiku-4-5.html)