Rendered at 21:20:35 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
tobwen 1 hours ago [-]
From my own experiments with various models, I suspect that LLMs have distinct/partitioned cutoff dates; for example, historical literature doesn't change (Greek history, Shakespeare, Goethe), general knowledge (updated only in certain areas), technologies (updated regularly), software also remains surprisingly stable - for example, with GIT, a basic command set is sufficient to do 99% of the jobs - new features are unknown or unnecessary, and tabloid knowledge, which is always up-to-date (politics, Taylor Swift albums).
sshh12 52 minutes ago [-]
I would have thought that as well but at least for coding vs world event facts I didn't see obvious differences in cutoffs
tobwen 49 minutes ago [-]
In my tests, I compiled the changelogs from various projects and checked how much functionality the model could recall over several test runs. In some cases, even with the latest models, the cutoff was around June 2024. Beyond that, it didn't recognize the features. However, it did know the cause of the last pope's death and even which hospital he was in (April 2025). However, the model did not know when Trump was sworn for the second time (Jan 2025) - but it did know THAT he was elected a second time (Nov 2024).
myworkaccount2 3 hours ago [-]
I wonder if this kind of analysis will give us a way to check if the frontier labs are waiting for the right moment to release their models.
To me it feels obvious that these companies are not releasing models as soon as they are done doing their post training / testing with any new model.
But there is no real way to know how much of this "waiting" any lab is doing, if we can get better estimates this way maybe we can gauge how far the open weights models really are.
simonw 2 hours ago [-]
Given how competitive the space is any form of waiting seems like it has more downside than upside to me.
If you have the new "best" model you may only have a few weeks of time in the market before some other lab releases a new model that beats yours.
This means you should get it out ASAP so you can maximize the time during which your model is "best". Once your model isn't best any more you're going to lose a lot of revenue to the new leader.
theplumber 3 hours ago [-]
It’s also worth to say that developers in general always have one more thing to fix before a release but if their hand is forced they can push it straight into main!
rad-b 4 hours ago [-]
Great read and interesting analysis! I’m less charitable toward Anthropic supposedly not distilling ChatGPT for training purposes. Maybe not today, but during the GPT-4 era when Anthropic was the underdog - I can see it happen. Packaged along with some of Amodei’s clever jumping through hoops to prove how that is, in fact, virtuous.
ddxv 5 hours ago [-]
This was great! One thing that wasn't addressed that I always assume, is that a marketing name like "Opus 5" is not a single model, but many models, versions and gets minor updates over time.
I also assumed many questions get routed to simpler models or programs to answer correctly, but it almost surprisingly didn't seem that way from the post.
Anyways, great post.
tedsanders 26 minutes ago [-]
I can explain how we do it at OpenAI.
In the API, we keep the models fixed. There are tiny caveats like rare bug fixes or models like `chat-latest`, but this is spiritually true. Suspicions of models changing over time are either human hallucinations or bugs on our end.
However, in ChatGPT, we sometimes update models without changing their names. For example, we recently launched an update to GPT-5.6 Sol in ChatGPT (https://openai.com/index/improving-gpt-5-6-sol-in-chatgpt/). Our goal isn't to be opaque or sneaky, but just to not exhaust people trying to keep track of little changes. When the changes are big, we give models a new name so that people know to expect something different.
sashank_1509 4 hours ago [-]
So Jan 2026 latest, I guess we are due to 2 OOM’s better retraining over the coming years. Curious to see the point at which AI plateaus.
But there is no real way to know how much of this "waiting" any lab is doing, if we can get better estimates this way maybe we can gauge how far the open weights models really are.
If you have the new "best" model you may only have a few weeks of time in the market before some other lab releases a new model that beats yours.
This means you should get it out ASAP so you can maximize the time during which your model is "best". Once your model isn't best any more you're going to lose a lot of revenue to the new leader.
I also assumed many questions get routed to simpler models or programs to answer correctly, but it almost surprisingly didn't seem that way from the post.
Anyways, great post.
In the API, we keep the models fixed. There are tiny caveats like rare bug fixes or models like `chat-latest`, but this is spiritually true. Suspicions of models changing over time are either human hallucinations or bugs on our end.
However, in ChatGPT, we sometimes update models without changing their names. For example, we recently launched an update to GPT-5.6 Sol in ChatGPT (https://openai.com/index/improving-gpt-5-6-sol-in-chatgpt/). Our goal isn't to be opaque or sneaky, but just to not exhaust people trying to keep track of little changes. When the changes are big, we give models a new name so that people know to expect something different.