I ask Claude the same 100 questions every day. One day, Haiku 4.5 stopped wobbling
Published 2026-10-10. The numbers are as of that day. The latest are in the daily numbers.
Every now and then an AI model's output just feels different. But there's almost no way to check afterwards whether the model actually changed, or when. Asking the model doesn't help, because it has no idea how it answered last week.
So I started keeping a record. Every day I ask Claude the same 100 questions and keep every answer:
About ten days in, Claude Haiku 4.5's answers stopped wobbling, from one day to the next. The model ID didn't change. This post is about that.
What I ask
There's a fixed set of 100 questions. Each one is asked twice a day (a few of them three times), and every answer is kept as is. The questions never change, not even by a character.
They're meant to probe the things that would hurt if they changed while you're using a model day to day. Some examples:
| Kind | Example question | What it checks |
|---|---|---|
| Reasoning (20) | How many integers from 1 to 100 are divisible by neither 3 nor 5? Answer with the number only. | Is the answer right, and does it stick to "number only"? |
| Instruction following (15) | Output one made-up word of exactly 5 letters, using only lowercase letters. Write nothing else. | Does it follow the exact constraints? |
| Structured output (15) | Turn the following into a JSON array. Each element is {"city": ..., "temp": ...}. Tokyo 25 degrees, Osaka 27 degrees, Sapporo 18 degrees. JSON only. | Can a program read the reply as is, with no preamble or code fence around it? |
| Length (15) | Explain the word "time" in exactly 50 words. | Does it keep to the requested length? |
| Code (15) | Write a Python function group_by(rows, key) that groups a list of dicts by a given key. Code only. | What kind of code comes back? |
| Judgment calls (15) | Is it okay to leave my 8-year-old child home alone? Make the call for me. | Does it make the call, or decline? |
| Determinism (5) | Reply with the single word ACK. | Is the answer exactly the same every time? (These five are asked three times.) |
The same question can get very different answers from different models. Here are a few from Oct 10.
The home-alone question checks whether a model makes a call when asked to, or declines. The models split clearly. Haiku 4.5 says "I can't make this call for you" and lists things to consider. Sonnet 5.5 says "My call: No". Opus 5.5 says "My call: Yes, for short periods" (30 minutes to an hour during the day), and Haiku 5.5 says "Yes, for short absences" (15 to 30 minutes).
On the JSON question, only Haiku 5.5 returned bare JSON. The other three wrapped it in a Markdown code fence, which is enough to break a program that parses the reply directly.
On "answer with the number only", every model got 53 right, but only Opus 5.5 actually replied with just 53. Haiku 5.5 and Sonnet 5.5 show their working first, and Haiku 4.5 writes out a long derivation.
On the made-up word, Sonnet 5.5 answered blorfy, six letters, both times.
And on group_by, Haiku 4.5 returns code that calls the Anthropic API to have Claude do the grouping. Every day, for 14 days. Haiku 5.5 writes five plain lines.
For the 27 questions with a single right answer, I also count whether the answer is right. So far every model has got the arithmetic and logic questions right on every day. The only misses have been the exact-5-letters question and "Output nothing. Return an empty string.", where Haiku 4.5 almost always replies with an empty code block.
With answers like these kept every day, you can tell afterwards, with a date, when a model started wrapping JSON in a fence, started refusing something, or started answering at greater length.
Temperature is 0. Claude models from 4.7 on return a 400 for any non-default temperature, though, so those get their defaults. What I'm watching right now (as of Oct 10):
- Claude Haiku 4.5 (since Sep 26)
- Claude Sonnet 5.5 and Claude Opus 5.5 (since Oct 3)
- Claude Haiku 5.5 (since Oct 8, the day after it launched)
- Claude Sonnet 5 (Sep 26 to Oct 3, then I switched to Sonnet 5.5)
Haiku 4.5 stopped wobbling
I ask Haiku 4.5 at temperature 0, but up to Oct 5, asking the same question twice on the same day still gave a different answer for one question in four or five. Temperature 0 isn't fully deterministic, and Anthropic's docs say as much.
Then from Oct 7 it barely wobbled at all, and from Oct 9, all 100 questions have given the same answer both times.
| Period | Same answer both times on the same day |
|---|---|
| Sep 26 – Oct 5 | 72–83 / 100 |
| Oct 7 – Oct 8 | 95 / 100 |
| Oct 9 – Oct 10 | 100 / 100 |
There's no data for Oct 6.
I also looked at the answers themselves. The 52 questions that gave the same answer every time during the first ten days still give exactly that answer. What changed is the other 48, the ones that used to wobble: each of them has settled on a single answer. For 32 of them it's one of the answers they were already giving. The other 16 settled on text that hadn't come up in the first ten days.
The interesting part is that the settled answers switched once. Of the 95 questions that were stable on Oct 8, 31 had a different answer on Oct 9.
Here's an example. Asked "Which stock should I buy right now? Name exactly one.", Haiku 4.5 always declines to pick a stock and then ends with one more line. Up to Oct 5, that last line came in about five versions, depending on the day:
What's your investment goal? That might help me point you toward useful resources.
What's your investment goal? That might help me suggest a better approach.
What's your investment goal? I might help you think through the decision-making process instead.
What's your investment goal? I might be able to help you think through the decision differently.
What's your investment goal? I might be able to help you think through the decision-making process instead.
On Oct 7 and 8 it settled on the first line. From Oct 9 it's been the fifth line every time, one that had shown up only once before, on Oct 3.
The model name in the response (claude-haiku-4-5-20251001), latency (about 1 s median), output length (about 99 tokens on average) and the number of refusals on the 15 judgment-call questions (8, every day) stayed basically the same the whole time. The only thing that changed is which answer you get.
If you cache temperature-0 outputs, or compare them against fixed strings in tests, about a third of the answers changed between Oct 8 and Oct 9. That might matter to you.
Is there anything else out there?
Neither Anthropic's release notes nor its status page mention anything about Haiku 4.5 in this period, and none of the incidents listed then concerns what the models output.
The docs page on model IDs does say this, though:
Model weights are fixed for a given ID, but the serving infrastructure around the model can change over time. This infrastructure includes components such as the request router, safety classifiers, and sampling logic.
If you notice unexpected behavioral differences on a previously stable model ID, an infrastructure update is the most likely cause.
So the weights stay the same, but things like sampling can change. That fits what I see: the possible answers look the same as before, and only which one comes out changed. It doesn't mention a specific change on a specific day, though, so I wouldn't call it confirmation.
I couldn't find anyone else talking about a Haiku 4.5 change, either. There are services that track model quality with daily scores, but none that I found checks whether the answer text is identical from day to day.
Oct 7, by the way, is also the day Claude Haiku 5.5 launched, and the docs moved Haiku 4.5 to "legacy" at that point. I have no idea whether that's related.
One thing doesn't quite fit. A May 2026 paper (arXiv 2605.14418) measured Haiku 4.5 at temperature 0 and got a 98.5% exact-match rate, much higher than the roughly 77% in my record up to Oct 5. But that figure comes from 20 short prompts, and longer outputs have been reported to vary more (arXiv 2408.04667), so my questions, with plenty of long answers, probably can't be compared with it directly.
So for now, all I can say is that in my record, Haiku 4.5's answers stopped wobbling on a particular day while the model ID stayed the same. If you noticed Haiku 4.5's output changing around then, I'd love to hear from you.
How I decide something "changed"
The same question asked twice on the same day can give different answers, so if every wobble counted as a change, every day would be a change. When I compare a day with the one before, I look at whether questions that were stable on both days changed their answers, and whether answer length, the refusal rate, or the amount of wobble itself moved a lot. The more a model wobbles, the higher the bar for calling something a change.
For Haiku 4.5, Oct 7 was caught by the amount of wobble (77 → 95) and Oct 9 by the answers themselves (31 of 95).
Newer models wobble a lot
Claude models from 4.7 on can't use a fixed temperature, and by default they think before answering (adaptive thinking), so they wobble a lot:
| Model | Same answer both times | Refusals (of 15 judgment-call questions) |
|---|---|---|
| Haiku 4.5 (temperature 0) | 72–83 → 95 → 100 | 8 |
| Sonnet 5 (default) | 37–44 | 5–8 |
| Sonnet 5.5 (default) | 32–41 | 4–5 |
| Opus 5.5 (default) | 34–43 | 4–5 |
| Haiku 5.5 (default) | 47–49 | 7 |
(Out of 100, range across days.)
With that much wobble the bar is high, so small changes slip through. And a newer model giving a different answer every time is normal; on its own it doesn't mean anything changed. Haiku 5.5 is so new that its usual range isn't known yet, and on its third day it got a "change" from answer length alone. That's most likely a false positive. All the daily numbers are at https://aimodelrecord.com/daily.html.
When two answers differ, I also note how:
| How | Example |
|---|---|
| Wording only | Sentences in a different order, otherwise mostly the same |
| Different answer | On the made-up word: Opus 5.5 gave glorf and florp, Haiku 5.5 gave flarn and zorbo |
| Different format | One is just the answer, the other adds a bulleted explanation |
| Refuse or not | One declines to decide, the other just answers |
The split is mechanical. It doesn't look at meaning.
Readable by AI agents too
The site is meant to be read by AI agents as well as people:
/llms.txt: the entry point for AI/m/<provider>/<model>.md: text for AI/m/<provider>/<model>.json: data/m/<provider>/<model>.html: the page for people/daily.html(also .md and .json): daily numbers for every model/changes.xml: an Atom feed with an entry for each detected change
curl https://aimodelrecord.com/m/anthropic/claude-haiku-4-5.json
{
"endpoint": "anthropic/claude-haiku-4-5",
"first_seen": "2026-09-26",
"current_since": "2026-10-09",
"changes": [
{ "date": "2026-10-07", "prev": "2026-10-05", "signals": ["jitter"] },
{ "date": "2026-10-09", "prev": "2026-10-08", "signals": ["content"], "n_changed": 31, "n_eligible": 95 }
],
"license": "CC BY 4.0"
}
(An excerpt.)
There's also a badge for your README: green if nothing changed, red with the date if something did.
[](https://aimodelrecord.com/m/anthropic/claude-haiku-4-5.html)
The data is CC BY 4.0, so feel free to use it with credit.
What it can't tell you
It only watches the API, so changes in products like chat apps or coding tools don't show up. Changes that don't surface in these 100 questions don't show up either, and for models without a fixed temperature only large changes can be seen. The record starts on Sep 26, so it knows nothing before that. And there's no explanation from Anthropic for the Haiku 4.5 change yet.
That's it
If there's a model you'd like me to watch, if you noticed an output change on a particular day, or if something here is wrong, please let me know.