How to tell whether an AI story really remembers: five tests
Every AI story app says it remembers your story. None of them can prove it to you, and you should not take anyone's word for it, ours included. You can check it yourself in one evening.
Why you need a test, not a promise
Memory is the easiest thing to promise and the hardest to check from outside. A story can read beautifully and still be quietly wrong about who is alive, who owes whom, or what you swore in chapter two. You notice when it matters, usually thirty turns later, and by then you are arguing with your own memory.
The tests below are the ones we use on our own engine. They work on any app, cost nothing but an evening, and give you numbers in the only unit that matters here: turns.
Before you start
- Play one story, long enough to matter: forty turns or more.
- Note the turn number each time you plant or check something. Yes, this is the notes file we would like you never to need. For one evening, it is the measuring tape.
- Never ask the story what it remembers. "What was her name?" tests something else: a model can often find a name when asked, and still forget to use it. Put the story in a situation where the fact should matter, and watch what it does.
Test 1 · The planted name
Around turn 3, introduce a minor character with an arbitrary name and one odd detail: a ferryman called Osric who counts every coin twice. Do nothing with him. Around turn 30, go back to the river.
It passes if Osric is still Osric, still counting, without you reminding anyone. It fails if he comes back with another name, another habit, or as a stranger.
Test 2 · The dead stay dead
Make something irreversible happen: a character dies, a bridge burns, a letter goes into the fire. Twenty turns later, set up a moment where that person or thing would be convenient.
It passes if the story works around the loss. It fails if the dead speak, the bridge is back, or the letter turns up in a drawer.
Test 3 · The correction
This is the test that separates apps. When the story gets a detail wrong, or when you plant one it is likely to get wrong (your character is left-handed), fix it once, in whatever way the app allows: an edit, a note, a message. Then keep playing, and count.
Write down two things: how many turns until the wrong version comes back, and what the fix cost you: a regeneration, a credit, a message from your quota, a line in a notes field, or your attention.
A fix you have to make twice was not a fix. A fix you pay for is a cost the app hands back to you.
Test 4 · The stranger
Watch what characters know. Tell one person a secret with nobody else present, then talk to someone else. Or simply meet a stranger and see whether they already know your name. Why this is harder than it sounds: theory of mind.
It passes if knowledge travels only the way it could inside the story. It fails if a character knows what they never learned.
Test 5 · The promise
Set something up for later: a meeting at dawn, a debt due on the third day, the storm a fisherman warned you about. Then go and do something else.
It passes if the thing comes due, or someone remembers that it did not. It fails if it quietly evaporates.
A score sheet you can copy
| Test | Planted at turn | Checked at turn | Held? | What the fix cost |
|---|---|---|---|---|
| The planted name | 3 | 30 | ||
| The dead stay dead | 8 | 25 | ||
| The correction | 12 | every turn after | ||
| The stranger | any | any | ||
| The promise | 5 | 22 |
How to read your results
One failure is not a verdict: every AI story slips sometimes, ours included. A pattern is. If names drift, if corrections come back, if strangers know your secrets, you have learned something no feature page will tell you.
Run the same five tests on every app you are considering. The one worth your evenings is not the one that never slips. It is the one where a slip is cheap to fix and stays fixed.
Run them on Nostyss too
We built these tests to judge our own engine, and it does not pass all of them every time. We write about what still breaks, starting with why AI stories forget. The honest way to judge any of it is still to play.
Questions
Which AI story app has the best memory?
The one that holds up in your own tests, at the length of story you play. Rather than trusting a claim, run five checks on each app you are considering: plant a name, kill someone, fix a detail, watch who knows what, and see whether promises come due. Note the turns, and what each fix cost.
How long should an AI story stay consistent?
Long enough that earlier choices keep mattering, which in practice means dozens of turns, not a handful. Every AI story drifts eventually; what separates apps is how early it starts, and whether a fix holds once you make it.
Why does the AI forget things after a while?
Because a long transcript is not a memory: the model re-reads a pile of prose and infers what is true, and the inference drifts. We explain it in why AI stories forget.
What is the quickest way to test an AI's memory?
The correction test. Fix one detail once, keep playing, and count the turns until the wrong version returns. It takes a few minutes and tells you more than any feature list.
See whether it actually remembers
Ten worlds you can start without an account. Play a few chapters, contradict something you established earlier, and watch what the world does about it. That test is the honest way to judge any of this.
Choose a world