By following clues we can find out things hidden from us. As we all know this is not an easy task, since if we are to succeed we eventually will have to recreate the things we want to know in some detail. After Sherlock Holmes has solved a crime he cannot just say so-and-so is responsible and then for ever keep silent. If he did no one would believe him. For Sherlock Holmes to be believed; for him to live up to what is expected of him; he has to explain in some detail what took place: what happened at such-and-such a time, what happened before, what happened afterwards, who is where, doing what, how this person thinks about this and that, and so on. It is with these details that he is able to build up a case. Now all this of course is done by following clues, and we know it is difficult. It is so difficult that it takes an exceptional person to do it. But why is it difficult? Should it not be impossible in the first place? After all, Sherlock Holmes was not there when the crime was committed. Not being there, how could he know what happened? And not just know, but know in detail. True, he follows clues, but can clues tell us anything? Can they tell us anything in detail? Now scientists follow clues too and the details they have been able to pile up are increasing at a tremendous rate.1 Should we believe these details? Are these details--and all that they support--nothing more than useful fiction as some say nowadays? Or do they reflect reality? In this paper I take up these questions.
This paper is divided into three sections. In the first I explain briefly why it is possible to find out in detail things hidden by following clues. In the second I give a number of reasons why this task is difficult. In the third I draw attention to a few things which make this difficult task a little easier.
To make this paper easier to read I have kept it short. This will mean the explanations I give on occasions will not be as detailed as some would like. For more details readers are asked to go to my other writings--for example, The Art of Detection.
Although human beings have been following clues for a long time even nowadays not everyone believes clues could lead to knowledge. This is understandable, for we follow clues when the things we want to know are hidden (criminals for example hide their deeds). If they are not hidden; if they are there for everybody to see; there is no need to follow clues. But if they are hidden, how can we know what they are? How can we know before they are revealed? Sherlock Holmes was not there when the crime was committed; how could he know what happened before the criminals confess? He says he finds out by following clues. This, people often say, is just an excuse for a story. Another clever person working with the same clues could easily come up with a different story (they say). Similarly with what scientists tell us. Scientists say this and that happened when the universe began. How do they know? They are not God!
Do we have to be there if we want to know? Can clues substitute for being there? What are clues that they can let us know without being there?2 Can clues let us know in detail? These are questions we want to answer in this section.
We will do well if we focus for the moment on the question, what are clues? If clues can lead to knowledge without our being there they have to be something special. We should know what this something is, so that we can better judge whether they (clues) can live up to their reputation.
A quick way to find out what clues are is to look at the way we crack ciphers. Suppose we have been given the following cryptogram to solve.
SBR SBCTU DBCKERVS FCGG WTTCXR SFH FRRJD YTHE SHUWI
Very naturally we will be looking for clues. For example, we see the first two words in the message both beginning with SB. What could SB stand for? TH? SH? And what does this tell us about clues? What are clues? Why should they be there? Why would anyone create a secret message and yet leave behind clues?
When we have a simple example in front of us it is easy to find out what clues are. From our example we can see that clues are but the characteristics of structures. The English language has a structure. Because it has a structure; because a language is not just a jumble of symbols; certain letters or groups of letters appear more frequently than others. In the English language words often begin with TH, so it is possible that SB could stand for TH. Of course there are other possibilities, but the number of possibilities is finite, so that we can try them out one by one and see which fits. The number of possibilities is finite because, as we have been saying, English has a structure. Because English has a structure, not just anything can happen.
So, clues are the characteristics of structures. Now different structures have different characteristics. This is why by following clues we can reconstruct these structures. Suppose we have constructed a sentence that has characteristics very different from those we can find in the cryptogram; we know then we have constructed the wrong sentence. But if we have constructed a sentence that has exactly the same characteristics as those in the cryptogram we know we have re-constructed it.
Can clues let us know about things in detail? We see here that the answer has to be yes. For clues are the characteristics of structures. A structure has many characteristics. The more characteristics (clues) we know the easier it is to reconstruct the structure. This means, if clues can ever let us know about anything, that knowledge has to be knowledge of that thing in some detail. If we only know that the first word of a sentence is THE we cannot reconstruct this sentence. To reconstruct this sentence we need to know its other characteristics. And of course there will be other characteristics. A sentence cannot have just one characteristic, the characteristic of beginning with THE, and no other. So, either we cannot re-construct the sentence or if we succeed, the sentence reconstructed will be replete with details.
Scientists distinguish between mature sciences and young sciences. Of the two they have much greater confidence in the former. Why? There is a simple reason. In the mature sciences, you will find, they will have discovered a huge amount of detail.
Most scientific papers are about details that laypeople will find boring even if they should be able to understand them. 'Why should anyone bother with such boring details?' laypeople often ask. We see now there is a good reason. Scientists follow clues. Clues lead to details by which they can then reconstruct the truth. The more details they have, the closer their reconstruction. That these details are boring to some is regrettable but does not alter the fact that they are necessary.
When we follow clues and are successful we will produce a lot of details. Having made this point let us next take up a question often asked. Given the same set of clues can we construct more than one structure, that is, give more than one explanation?
Some say we always can. Therefore, they say, there is no point in following clues. We do not agree. We do not deny examples abound in which more than one explanation can be given to the same set of clues, but these examples do not show that there is no point in following clues. For when we find ourselves faced with one of these situations, the simple lesson we should take from it is that we should look for more clues. If we have only one clue and it tells us that the sentence we want to reconstruct begins with THE, obviously we cannot reconstruct this sentence; there are so many sentences that are compatible with this one clue. What should we do then? Obviously again, we should look for more clues. When the number of clues in our possession is small the number of possibilities we can construct is large. When the number of clues in our possession increases, the number of possibilities decreases. As we have said, the more clues we have the closer our reconstruction is to the truth. We sometimes actually hit upon the truth by following clues, but even when we do not it is better to be closer to it than not to try at all.
To understand why it is possible to know about things to which we are not directly present by following clues it is important that we be aware that clues are the characteristics of structures. But needless to say, this is not sufficient. For one thing, clues are not just the characteristics of structures pure and simple; they are the characteristics of structures, disguised. In the cryptogram given we do not see TH; we see only SB. SB disguises a characteristic of the English language. In fact, of course, the whole cryptogram is a disguise. To find out what message is behind this cryptogram we have to strip away the disguise. This is not easy to do, as we all know. Even so, once we have understood that clues are the characteristics of structures we will have gone a long way in understanding why by following clues we can uncover things hidden.
In following clues we are trying to re-construct structures by paying attention to their characteristics. This is possible as we have seen. But as anyone with experience in following clues will know, it is difficult, for good reasons. In this section I point out some of these reasons.
In following clues we are trying to reconstruct structures. When we know enough of the characteristics of a structure we can reconstruct that structure. But, as we have just pointed out, clues are not the characteristics of structures pure and simple, but the characteristics of structures disguised. Before we can use clues to reconstruct the structures we have to poke through the disguise; we have to find out what is behind it. This we have to do on our own, without the help of those in the know. Now when the disguise is light it may not be all that difficult. Handwriting is sometimes a kind of disguise: some people's handwriting is so atrocious that we have to decipher it to find out what it is they have written. However, most of the time this is a kind of disguise we can, with patience, remove. But not all disguises are as easy to remove as this. Before the advent of modern science, how many were there who suspected the spot of light in the night sky which some call Venus was in fact a material body of tremendous size?
Even when we know the characteristics of a structure it is not necessarily easy to re-construct that structure. When these characteristics are disguised, the fact that they are, adds to the difficulty.
Jigsaw puzzles come in boxes. The pieces within each box belong together: when they are all used up they form one picture. But clues are not boxed. In an investigation the clues required are not presented to us in a neat package so that all that we have to do is figure out how they fit together. Even in solving the simple cryptogram given earlier some of the clues are not found in the cryptogram. For example, why should we assume the plaintext is in English? Now if we did it was likely because of context: an author writing in English is likely to use an English example.
In solving a murder not all the clues are necessarily confined to the crime scene: some of them could be thousands of miles away. And even at the crime scene there are many things which are not clues for the murder. Determining which are and which not is part of the investigation. Frequently, important clues are overlooked and their omissions discovered only too late.
In the game Twenty Questions there is a Question Master. Is the Hidden Object animal, vegetable, or mineral? The Question Master will answer your question and you can rely on her answer; the Question Master is not supposed to lie. But in following clues there will be many questions but there is no Question Master. Is the ash tray a clue? Does the presence of the ash tray mean that the table has been rearranged? I think the butler did it and he has confessed, but is the confession genuine? Who answers these questions? Who, but the investigators themselves?
Questions that arise when following clues are not answered by a Question Master. In this game there is no Question Master. In solving a crime, the criminals themselves are not the Question Master: it is well known that criminals often lie. In the investigation of nature scientists do not consult God.
But although there is no Question Master when following clues we can still find out what we want to know. Not all games are the same. Some games do not have Question Masters. In following clues, even though there is no Question Master, questions we raise can be answered--by ourselves. It is harder to answer questions by ourselves. Much harder. But it can be done.
Structures can be simple or complex. A word has a structure simpler than a sentence; a sentence, a structure simpler than a paragraph. With a simple structure we need fewer clues to reconstruct it. If clues tell us that the first word of a sentence has three letters and the first two letters are TH, it is likely that the third is E. But not all structures are this simple. To reconstruct more complex structures we need to know more of their characteristics; that is, we need more clues. But as everyone knows, clues are hard to find.
Moreover, when we try to reconstruct a structure, to make sure we have done it, we have to be able to distinguish it from other, neighbouring structures. For this purpose it is usually not sufficient to find one or two clues; we need many. The more clues / characteristics we have the easier it is to distinguish a structure from its neighbouring structures. But where are we to find these clues?
In following clues we are trying to reconstruct structures that already exist. But we are working in the dark: we do not know in advance whether the structure under investigation is simple or complex, and if complex how complex. We may think we have found some clues and that the underlying structure is such and such, but the structure we are dealing with could be more complicated than anything we have imagined, or can imagine.
To explain this next difficulty let me start by telling a story which I have told many times. This story is about you, so it might not be new to you at all.
In this story you are a cryptanalyst; that is, you crack ciphers. But in addition to being a cryptanalyst you are also a cryptographer; that is, you also make up new ciphers. In fact, this story is about a new cipher that you have created, a cipher of which you are rather proud.
It has taken you some time to create this new cipher but it is now finished. You have examined this cipher in detail and you have come to the conclusion that it is a good cipher. A good cipher should be easy to use. Yours is easy to use, easy because there are no instructions to be written down; there are only a few simple things to remember. Written instructions should be avoided because they can be lost. Worse, they could fall into the wrong hands. But while your cipher is easy to use it is not easy to break. Ciphers that are easy to use are usually easy to break. To make a cipher difficult to break one will have to make it complicated, and that usually means written instructions. Yours does not require written instructions; yet it is difficult to break. Why? Because it is a cipher of a completely new kind, not some variation of some known type. When one tries to invent a cipher, the farther away it is from known types the better it is. Known types have known characteristics. When one uses a cipher of a known type, it is easy for these characteristics to be detected. And once they are, the cipher is as good as broken. But your cipher does not belong to any known type. This is why you are confident it will not be easily broken, at least not in a short time. People who want to crack your cipher will try the known types first. It will be some time before they even suspect that yours does not belong to any of them.
Tomorrow you will put your new cipher in the field; that is, you will send out your new cipher to your correspondents so that they can start using it. In the meantime there is something else you have to do.
This morning you have intercepted a new message, encrypted of course, telegraphed by your opponents to their own people, a message that you very much want to read. But try as you might you cannot make head and tail of it: you cannot find a single clue! You have gone through all the different types of ciphers known but that has not further you investigation. At this moment you are still staring at this cryptogram and you are getting despondent …
But all of a sudden you notice something, something alarming, something that literally makes you sit up. You notice to your horror--which however quickly turns to glee--something that looks like a clue of the kind that your newly invented cipher would leave behind! You search through the message more carefully and you discover other clues of the same sort …
Are you going to send out your new cipher tomorrow, as you have planned? Will you put your newly invented cipher in the field?
I think you are likely to say, not on your life!
Why? Why will you not use your new cipher? You have invested so much effort in inventing it; why keep it back? Were you not proud of it just a moment ago? Are you not proud of it still?
Of course you are still proud of it. But you will not use it for the simple reason that, the coincidence of all coincidences, you and your opponents have invented the same kind of cipher. If you use your cipher tomorrow your opponents will break your cipher just as easily as you are now breaking theirs ….
What does this story, a story about coincidental co-invention, tell us about following clues? Why is this process so difficult?
One of the reasons the process is difficult is that the structure we are trying to reconstruct may not be among those that we already know; indeed, may not even resemble any structure that we already know. Clues are the characteristics of structures. But you have to know a structure first before you can tell what its characteristics are. If you did not know English you would not know what characteristics English has. Not knowing the latter you will not be able to recognise English clues. Without any clue, you cannot reconstruct the message. In our story about coincidental co-invention you had a hard time cracking your opponents' new cipher. You had a hard time because you could not find any clues based on the ciphers you know previous to the invention of your new cipher. If you had not invented the same cipher that your opponents had invented you might never have broken their cipher. But as luck would have it, you did and you won--this time.
Usually we do not find things we are not looking for. These things could be right in front of our eyes but because we do not expect to see them we in fact do not see them. You pass an old friend in the street; yet you do not recognise her--because you do not expect to see her there--until she calls out. In our story about coincidental co-invention the clues that eventually enabled you to crack the cipher were there all along. But for the longest time you did not notice them. You did not notice them because you did not expect they would be there: it is rare that two people should have invented the same thing independently of each other and then meet up right away. But eventually you did notice the clues. A bell rang; something called out to you, like the old friend in the street, except that we do not usually think of clues as capable of calling out. The expression 'ringing a bell' is common, but nobody knows who rings the bell. But while the expression is common the event is rare. In following clues we sometimes lose when we could have won because no bells ring.
It is possible, but difficult, to reconstruct hidden structures by following clues. In the last section we have set out some of the reasons why it is difficult. In this section we draw attention to some of the things that make it a little less so.
In following clues we are trying to reconstruct structures by their characteristics. But the characteristics we have to go by are not as they are, but disguised: TH does not appear as TH but as SB, for example. Now if the disguise varies every time it will be hard or even impossible to figure out what is behind it--a spy who changes her disguise constantly is hard to catch. But if the spy always wears the same mask; if TH is always SB; the task will be easier. Now the disguises we meet with when following clues will not always be as simple as wearing the same mask, but so long as the change in disguise occurs in a structured manner--say, according to a formula--the detection of what is behind the disguise can still take place. Astronomical objects do not always appear as spots of light visible to the naked eye; some of them are invisible (to the naked eye) and have to be detected by using radio-telescopes or some other means. Astronomical objects can put on different masks; however, their disguises are not random, the reason why astronomy is possible.
When asked to solve the SBR cryptogram given earlier few are likely to want to do it in one single step. If taking one single step were the only avenue open to us it probably could not be done. Instead of taking just one step it is more reasonable to allow for many. For messages could contain sentences and phrases. Sentences and phrases are made up of words. If we can find out some of the words it will make it easier to decipher some of the phrases. And when we know some of the words and phrases it should be easier to complete the sentences. Now this kind of stepwise procedure is common when following clues. It makes possible discoveries that otherwise would have been impossible. Structures often contain simpler structures as components. When this is the case the simpler structures can be attacked first. After they have been deciphered we can then take on the others. Scientists do not try to understand the whole universe at one single stroke (as some mystics would). They know the universe contains many structures, so they focus their attention on one or a few of them at a time, leaving the rest for the future. As we all know at the time scientists found out the structure of the solar system they knew nothing about the structure of the atom, even though objects in the solar system--such as the sun and the planets--are made up of atoms. The structure of the atom was discovered later on.
To reconstruct structures and to distinguish them from neighbouring structures we need many clues (characteristics). But in most investigations we have very few clues to begin with. If these few clues are the only clues we will ever have there will not be many successful investigations. But, fortunately for us, clues sometimes lead to new clues, so that we are not always confined only to the clues available to us at the beginning. In the example we have been using (reproduced below), the sixth word into the message is SFH.
SBR SBCTU DBCKERVS FCGG WTTCXR SFH FRRJD YTHE SHUWI
Clues available from the beginning tell us that it is possible that S stands for T and F for W. Now if these two hypotheses are correct, we have in the sixth word a new clue, TW?, suggesting to us what H stands for:
This clue was not there at the beginning because we did not know then what the first two letters of this three-letter word were. We develop this new clue from earlier clues, that is, those clues that tell us that S stands for T and F, W.
Clues can lead to new clues. They do not always do so but they sometimes do. Because of this, sometimes starting only with a very few clues we nevertheless manage to find those things we want to know. Following clues is difficult but sometimes when things go our way, it is a little less difficult. We therefore should not give up if at the beginning of an investigation we are not able to discover as many clues as we would like. We should press on and see if new clues could be developed. The kind of science we now have started with a very few clues, and what clues there were, were vague. But these led to others, which led to others. So many new clues have appeared since those earlier days that we now know much more than we did before, and with much greater certainty.
In the game Twenty Questions there is a Question Master. Since the Question Master is not supposed to lie or make mistakes, by relying on the answers she gives we can home in on the Mystery Object. In Twenty Questions no one expects that they can find out what the Mystery Object is by asking just one question. You are allowed twenty. By asking these questions intelligently you can narrow down the field and gradually home in on the Mystery Object. Now in following clues there is no Question Master. When we have questions--and we will have a large number, we have to answer them ourselves. This makes following clues much more difficult. But although there is no Question Master when following clues, by relying on the clues found we nevertheless can gradually zoom in on the things we want to know. How is this done? Why do we not need a Question Master?
It is a common experience in following clues that we know we are closing in on the subject of our search long before we actually have it in our grasp. How is this possible? How is it possible when there is no Question Master?
As some of my readers will know already, the answer to this question lies in the way we develop new clues. Let me explain.
In our game in cracking ciphers SFH becomes the new clue TW? :
Now some will ask, how do we know that the first two letters are TW? We 'know' from earlier clues, we answer.
Could we be wrong?
Yes, but …. And here we see the reason why we do not need a Question Master. In following clues it is easy to make mistakes, but if we had made mistakes, serious mistakes, no new clue would appear. If SF were not TW there would not have been the new clue TW? . Now that we have the new clue TW? we know we have done things correctly. The new clue confirms that SF is likely TW. When interpreting SF as TW we are heading in the right direction.
In following clues we develop new clues from old, but if we have interpreted the old clues incorrectly we will not develop new clues. From this we see when new clues appear the old clues on which they are based are likely to have been correctly interpreted. If the old clues had been incorrectly interpreted the old clues could not have led to new clues except by chance--when they lead to false clues and false clues peter out.
Now this can be generalised to all the steps we take when following clues. Only when all these steps are more or less correct will new clues appear. If new clues do not appear; if over and over again we find our way blocked for lack of new clues; it is likely that we have made serious mistakes in the steps taken. This is why in following clues we do not need a Question Master. In following clues we know on our own whether we are doing things correctly. If we are, we make progress: we discover more and more new clues. If we make serious mistakes our way forward will be blocked. Following clues is difficult; there is no Question Master; but we can nevertheless tell on our own whether we are homing in on our prey. Let me illustrate this point with yet another example.
Suppose the main suspects in a crime are the butler and the maid. The clues so far point to one or the other, but not both. The butler has confessed but some think that he does so only because he wants to protect the maid. Is this hypothesis correct? How do we decide? How do we know whether the butler may not be owning up to a crime he did not commit?
Obviously one way to answer this question is to see whether the confession given by the butler leads to new clues. If it does; if it leads to clues that give us more and more details about the crime, details that we would not have known otherwise; it is highly likely that the confession is genuine. On the other hand, if the confession leads to no new clues but the hypothesis that the maid is responsible does, it is then likely that the confession is false.
One of the things that make following clues a little easier than it otherwise might be is that in following clues we do not have to get things exactly right the first time. The reason for this is simple. Since the process will take time anyway there can be no objection if, to make things easier for ourselves, we approach the truth gradually, through a series of approximations. Indeed, most of the time if not all, this is the only way for us to proceed. For when we are following clues we are working in the dark. How likely is it that a person shooting in the dark should hit bull's eye the first time?
If we were required to hit bull's eye in everything we do when following clues the task of following clues as a means of finding out things hidden would be well nigh impossible. But there is no such requirement. In following clues we can make use of approximations. So long as our approximations lead to new clues we have the chance of bringing these approximations closer to the truth after the new clues have been interpreted. That we are allowed to use approximations when following clues is one of the things that make this difficult task a little easier.
Scientists often make use of approximations. Theories they propound are not expected to be perfect the day they are enunciated. If a theory is fruitful in providing new clues the chance exists that it can be refined when more is known. Clues lead to details, as we have explained in Section 1. More clues lead to more details. Any time we have more details we can be more accurate in what we say. This is true in science as it is in crime investigation or anywhere else. Newton made his contributions in the 16th century, over 400 years ago. Did he get everything right? No--but could we have an Einstein if there had not been a Newton?
1 So much so that no one person can master them all.
2 Some will say we would not know even if we were there. But if we can know without being there, then being there is not all that important. What is important then? What is important is that we can know without being there; that is, that we can play Sherlock Holmes.