5 Abandoning Ideas
It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.
Sir Arthur Conan Doyle
5.1 The Difficulty of Letting Go
There is a well-known tendency in business decision making called the sunk cost fallacy. Once time, effort or resources have been invested into something, it becomes much harder to away even when continuing is no longer the best option.
A classic example is watching a film you are not enjoying. Half an hour in, you realise it is not very good. The plot is weak, the dialogue is unbearable, and at least one character appears to have been written by someone who has never met another human being. But instead of turning it off, you keep watching, because you have already spent thirty minutes on it and the whole thing is only ninety.
The same thing could happen in codebreaking. After spending a long time developing an idea, testing it and trying to make it work, it becomes difficult to let go. The time you have invested starts to feel like a reason to continue even when the evidence might no longer support it.
Every year that I have competed in the challenge, this has been one of the most difficult things to deal with. Not because the problem was particularly complex, but because I was then faced with a choice: continue with an idea that feels promising, or abandon it and return to uncertainty.
5.2 Why We Hold On
It’s very natural to become attached to an idea once you have invested enough of yourself into it. And by ‘yourself’, I don’t just mean your time. I also mean your hope, concentration, and sense that this has all been leading somewhere.
The more work you have done, the more difficult it becomes to let go. Each small success reinforces the belief that you’re close to a solution, even when the overall picture does not quite fit. An idea that is ‘almost perfect’ can be quite dangerous because it creates the illusion that you are only one step away from solving the problem.
There is also a tendency to adjust the problem to fit the idea. If something does not align perfectly, it can be tempting to treat it as an exception, or to assume that it will make sense later. But over time, this can lead to increasingly complicated explanations that are difficult to justify, but equally difficult to abandon.
5.3 When an Idea just isn’t Working
Letting go of an idea becomes easier when you know what to look for. This is genuinely difficult, but gets easier with time and practice and patience.
Some common signs that an idea may not be useful include:
The explanation only works for part of the ciphertext
You need to summon exceptions to make it fit
Small inconsistencies are being ignored or explained away
The reasoning becomes more complicated over time, rather than simpler
A useful principle here is that good explanations tend to simplify the problem. So if an idea requires increasing effort to maintain, it is often a sign that it is not the right one.
5.4 Returning to the Problem
After abandoning an idea, it is important to return to the problem itself, rather than immediately searching for a replacement.
Go back to the ciphertext. Look again at what is actually there. Revisit the observations that still hold. Ask yourself what remains true even now that your favourite idea has collapsed dramatically in front of you.
By returning to the structure of the problem, you allow new ideas to emerge from what you can observe, rather than from what you hope might work.
Again, abandoning an idea does not mean starting again from nothing. Even if a particular approach does not lead to a solution, the process of exploring it may still have produced useful information. You may have ruled out certain structures, identified patterns that remain consistent, or clarified which assumptions do not hold.
The goal here is not to avoid being wrong. That would be both impossible and extremely boring. The goal is to be wrong in ways that sharpen your understanding, to fail usefully, to let the bad ideas die without dragging you down with them.
