Why is it that by following clues we can come to know things that are hidden while these things remain hidden? This, we have pointed out in the last chapter, is the central question that a textbook on the art of detection has to answer. Let us therefore take it up right away.
There is another reason why this question is urgent. As some readers will know already, there are people who believe it is impossible for human beings to have the kind of knowledge we are interested in here, that is, knowledge of things hidden while they remain hidden. The hidden, these people say, simply cannot be known. They would not accept Sherlock Holmes as evidence against their view. Sherlock Holmes, they say, is just a fictional character. But if we can explain why things hidden can be known, even if we cannot change the minds of these skeptics, we can at least carve out more space for ourselves, more space in which to explore our subject. For it is easy to deny a fact but harder to deny a fact for which there is also an explanation.1 Needless to say, the explanation we are looking for cannot depend on whether Sherlock Holmes is real or fictional. If it is possible for human beings to know about things that are hidden, it is possible whether Sherlock Holmes exists or not.
My mind rebels at stagnation. Give me problems, give me work, give me the most abstruse cryptogram or the most intricate analysis, and I am in my own proper atmosphere.
—— The Sign of the Four
Why is it that by following clues we can uncover knowledge of things hidden while these things remain hidden? To find the answer to our question, it is useful if we have an actual example in which we uncover knowledge of this kind, that is, knowledge of things hidden while they remain hidden. This way it is harder for people to deny that this kind of knowledge is possible. It is best that this example not come from Sherlock Holmes, since there are people who doubt that Sherlock Holmes is real. And it does not matter that it does not come from Sherlock Holmes, for if it is possible to uncover knowledge of things hidden while they remain hidden, anybody can do it so long as they know how; one does not have to be Sherlock Holmes. Sherlock Holmes may do it better than most people but this does not mean others cannot do it.
The example we need does not have to be an example in which some crime has been committed. Among the many mysteries Sherlock Holmes has solved there are quite a few in which no crime is involved. When a mystery has been solved some things formerly hidden and not known is now known although still hidden. The question we are asking is how this is possible. Clearly the answer cannot depend on whether anyone has broken the law.
The example we need should be simple; there is no point in making things difficult for ourselves. We want to know why things hidden can be known while remaining hidden. By analysing a simple example we have a better chance of finding the answer to our question. If we use a complicated example we could easily miss the forest for the trees. If we want to know why airplanes can fly we would not start with the 747.
But we should not use an example that is too simple. If we did we might miss some of the important trees in the forest. In our aviation example, no one would start with a 747; neither would they use a paper airplane.
What example should we use?
I suggest we take one from cryptanalysis (cracking ciphers). Cryptanalysis is an activity engaged in by all kinds of people, not necessarily by spies or by those who catch spies. Some enjoy it simply for the challenge. As it happens, Sherlock Holmes is something of an authority in this very interesting subject.
I am fairly familiar with all forms of secret writings, and am myself the author of a trifling monograph upon the subject, in which I analyse one hundred and sixty separate ciphers …
—— The Adventure of the Dancing Men
This piece of information we gather from The Adventure of the Dancing Men, in which Sherlock Holme's unravelling of the mystery benefited in no small measure from his solution of the Dancing Men Cipher. But this is incidental. Sherlock Holmes can solve ciphers, so can you, Dear Readers, as you will soon find out.
In the example I am going to use readers are given a cryptogram and are asked to find the message (the cleartext) hidden behind it. This is the cryptogram:
SBR SBCTU DBCKERVS FCGG WTTCXR SFH FRRJD YTHE SHUWI
I think I can assume at this time that those readers who are seeing this cryptogram for the first time do not see or know the message behind it. The message is hidden from them. Of course, the person who sends this message knows the message. She might still have it in her possession, written down somewhere. Or, it might already have been destroyed and she only remembers it. But even if the original message has not been destroyed; even if it still exists in a written form; I doubt that any reader can get hold of it now.
I urge readers to try to solve this cryptogram by themselves before reading on. It is quite possible that after a little while they will succeed. Of course when they do succeed, they will have done so by following clues. But even if they do not succeed, I think they will still find the experience valuable.
Since it will be useful to have a solution to refer to throughout this book, we provide below our solution to this cryptogram. The route we take in this solution may be different from the routes some readers take. This is common in cryptanalysis. In solving ciphers——as in theseologising in general, we often arrive at the same results by different routes.2
Here is our solution to the cryptogram given above:
After five steps we have deciphered our message. By following clues we have found the message hidden behind our cryptogram. This message says, THE THIRD SHIPMENT WILL ARRIVE TWO WEEKS FROM TODAY.
I have the threads of this affair all in my hand. Even if this lady should never recover consciousness, we can still reconstruct the events of last night, and insure that justice be done.
—— The Adventure of the Dancing Men
In solving our cryptogram we followed clues. Needless to say, when we follow clues we should do so correctly. Because this cryptogram is simple, it is easy to see that we have done so correctly; we have followed clues the way they should be followed.
Now an important question arises. Is the message we have arrived at the right message?
Here it seems there can be only one answer. The message is likely to be right. Why? Because we were following clues the right way. When we follow clues the right way, we arrive at results that are likely to be right. This we know from practice (we will explain presently why in theory this lesson we learn from practice is correct). It is the reason why we follow clues. If even by following clues the right way we cannot arrive at results that are likely to be right, no one would follow clues. Sherlock Holmes catches the right person by following clues the right way. Inspector Gregson follows clues too, but he makes mistakes, so he catches the wrong person (which he did in The Sign of the Four). There is a reward for doing things the way they should be done. When we are following clues, the reward for doing things right is that we obtain the right results. Following clues is not a simple business, the reason why we speak of an art. To be successful we need to master this art. When there is an art, that there is, is a clear indication that in the activity governed by this art there is a right way of doing things and there is a wrong way. In the art of playing a piano for example, it cannot be that anything we do is right: a child banging on the keys cannot produce good music; but when we have learnt the right way to play a piano we produce, if not necessarily beautiful music, then at least sufferable. It cannot be that even after we have learnt how to do things properly we still produce the same results as the child. Now in solving our cryptogram, because the example is simple we can see that we have followed clues the right way. What do we get after following clues the right way if not correct results, results that are likely to be right?
Can we say more? Instead of saying that the message we have recovered is likely to be correct, can we say we are absolutely certain that our message is correct?
It is easy to see that we cannot. After all, we do not have the original message in front of us. The original is still physically hidden: it could be locked up somewhere in a safe, or burnt, or destroyed in some other manner. In solving our cryptogram we have only recreated the hidden message; we have not physically caught hold of it.
Even so, although we have only recreated it, we do now have some knowledge of this hidden message. It is true that we cannot be absolutely certain; still, the certainty we are entitled to is high (definitely more than fifty-fifty).
Will the hidden message correspond exactly to our re-creation? Is our decipherment correct in every respect?
It is possible that our decipherment is correct in every respect. But even if it is not, it should come close, or as Sherlock Holmes would say, it is correct in essentials. This is what we can expect when we follow clues the right way. When we follow clues the right way and have obtained results, we can tell that our results will either correspond to the things we want to know or come close. We can tell even though we cannot put our results side by side with the things we want to know and compare them. This is one of the nice things about following clues. We can tell that our results will correspond, more or less, without actually making the comparison!
When we follow clues the right way we arrive at a kind of knowledge. This knowledge will not be absolutely certain even though the certainty will be high. This knowledge could be exact, but if it is not it will be close to being exact. Knowledge like this, I shall call approximate. Clues, when followed properly, will lead to approximate knowledge. In this book when I use the word 'knowledge' I will mean approximate knowledge, unless specified otherwise.
In connection with the concept of approximate knowledge I will also be using the word 'right' to describe sentences, propositions, theories, and things of that kind. By 'right' I mean 'true or close to being true'. A true theory about a cipher will tell us exactly how the cipher works. A right theory could tell us that or it could only come close. In theseology we often have to be satisfied with theories that are right; we may not be able to find out that they are true.
In following clues, when we are successful the knowledge we obtain will be approximate and it will be of things hidden. We follow clues to uncover things hidden. In our example the cleartext is hidden, as well as the cipher, which is why we have to follow clues.
Clues lead to approximate knowledge of things hidden while these things remain hidden. Those who crack ciphers do so while the ciphers remain hidden. They do not do so by going to their enemies, cap in hand, asking that they be permitted to examine the originals.
The knowledge we obtain by following clues is only approximate. Approximate knowledge, although certain to a high degree, is not absolutely certain. Approximate knowledge could be exact, but is not always so. Sometimes people find this unsatisfactory. They demand knowledge that is absolutely certain and perfectly accurate. Since approximate knowledge falls short, they think it not worth having at all. 'Why follow clues? Why pretend to be Sherlock Holmes?' they ask. 'Such diversions may satisfy the common crowd but surely not any one with loftier ambitions. There has to be a higher calling, one which requires that we dedicate ourselves to nothing less than the absolute truth, a calling that requires that we know with absolute certainty.'
But we should keep in mind, beggars cannot be choosers. Knowledge of the hidden is difficult to obtain. The way things are, it is astonishing that we could have any at all. To demand that this knowledge be perfect in every way, I think most people would say, is asking too much. Also look to the past. How many who looked only for perfect knowledge have found it? How many of these have deceived themselves into thinking that they have found it? And how many of these, after realising that perfect knowledge is impossible, have abandoned the search for knowledge altogether and turned themselves into skeptics of the most destructive sort? It is true that we should try to improve our approximate knowledge when we can, but this is very different from looking only for knowledge that is perfect in every way (see also Approximations in Chapter 3).
There is almost always elation on those occasions when we succeed in uncovering knowledge of things hidden. Readers are likely to have experienced this kind of elation in cracking our simple cipher earlier. On these occasions we cry, 'Eureka!', we throw up our arms, we bang on the table, we jump for joy, we embrace each other; at least, we smile.3 Because knowledge of things hidden is difficult; because we fail more often than we succeed when looking for it; because it seems completely impossible in the first place; when we achieve this kind of knowledge we often exaggerate. We say, 'Now I know!', 'It has to be this way.', even 'I am absolutely certain.' I suggest this is pardonable. In practice those who engage in this kind of behaviour often know that they are exaggerating. And not only they, but their peers as well.
Sherlock Holmes often exaggerates, particularly at the time of victory.
I have now in my hands … all the threads which have formed such a tangle. There are, of course, details to be filled in, but I am as certain of all the main facts … as if I had seen them with my own eyes.
—— A Study in Scarlet
At calmer moments, however, he would say,
I could only say what was the balance of probability. I did not at all expect to be so accurate.
—— The Sign of the Four
Since we have this tendency to exaggerate; since we are all human; it is, I think, all the more important that we all should have some understanding of what it is like to follow clues. If we have, it is easy to recognise this kind of exaggeration. An exaggeration does no harm when every one knows it is an exaggeration. Harm comes only when the exaggeration is taken to be the truth.4 We often say so-and-so is as thick as a mule, but no one really believes it literally.
There is nothing like first-hand evidence.
—— A Study in Scarlet
Using a simple example in cryptanalysis we have demonstrated that by following clues we can uncover knowledge of things hidden. Now some may disagree with what we have just said. They may say that the example we have been using is artificial. When we ourselves make up the cipher and then provide the solution, is it surprising that we should succeed in uncovering 'the hidden'? This is a legitimate objection but one for which there is a simple remedy. Those who think our example artificial can make up ciphers of their own and challenge others to break them. When their own ciphers are broken over and over again they will have a better idea whether the example we have been using is artificial or not. My own view is that although it is artificial, it is close enough to actual practice as would serve our purpose.
We deciphered our message by relying on clues. What are clues? Why is it that they can lead to knowledge of things hidden?
If we look to our example we can find the answer to this question easily enough. This is one of the advantages in having a simple example. From our simple example we can see that clues are the characteristics of structures, disguised. All languages have structures, this we all know. The English language has its own structure, a structure different from the structures of other languages. It is a characteristic of the English language that many words begin with TH. Knowing the many characteristics of the English language enables us to detect clues when trying to solve our cryptogram.
But clues are the characteristics of structures, disguised. TH does not appear as TH in our cryptogram; it appears as SB. TH has been disguised. To recognise that SB is in fact TH, we have to peel away the disguise. In our example the peeling away is relatively easy (another advantage of a simple example). In other cases it may not. Whether it is easy to unravel a clue depends on how heavily the underlying characteristic has been disguised.
Many things have structures besides a language. Messages have structures, crimes have structures, atoms and molecules have structures. This is why in cryptanalysis, in crime detection and in science we follow clues. By doing so we uncover knowledge we originally did not possess.
There are other areas besides the ones already mentioned in which we follow clues. One worth noticing at this time is the diagnosis of motor-car engines, a subject of which many readers are likely to have had experience either first-hand or second-hand (second-hand when they listen to mechanics talk, for example). By following clues an experienced mechanic can determine what is wrong with a motor-car engine without opening it up. Since opening up a motor-car engine is an expensive operation, this is an excellent talent to have. A motor-car engine has a large number of components put together in a structured way (a motor-car engine is not just a heap of metal, although from time to time we are driven to think that it is). When these parts are working together as designed, the behaviour of the machine has a set of characteristics. When the parts are not working together as designed; when some of them are malfunctioning; there will be a different set of characteristics. When one has learnt to detect all these many characteristics, one can find out without opening up the machine what is going on inside.
When Sherlock Holmes meets with a client for the first time, he has the habit of telling this person all kinds of things about this person, things this person would not expect him to know, since they are meeting for the first time. It is now known that this habit Sherlock Holmes has is patterned after Joseph Bell, one of Conan Doyles' teachers at medical school in Edinburgh. Joseph Bell has the same habit with his patients as Sherlock Holmes with his clients. Within the medical profession at the time, Joseph Bell is well known for being an excellent diagnostician, that is, an expert in finding out what is wrong within a patient's body simply by paying attention to the symptoms.
By a man's finger-nails, by his coat-sleeve, by his boots, by his trouser-knees, by the callosities of his forefinger and thumb, by his expression, by his shirt-cuff——By each of these things a man's calling is plainly revealed. That all united should fail to enlighten the competent inquirer in any case is almost inconceivable.
—— A Study in Scarlet
One should always look for a possible alternative and provide against it. It is the first rule of criminal investigation.
—— The Adventure of Black Peter
Structures have characteristics. Two different structures will have two different sets of characteristics. These two sets of characteristics could have some elements in common, but they cannot have all elements in common. For example, if we have two different sentences they may have some words in common, but they will not have all their words the same and arranged in the same way. If we know all the characteristics of the two sentences, we can reconstruct the two sentences.
Now clues are the characteristics of structures, disguised. If we have many clues from one and the same hidden structure and are able to peel away all their disguises, we should be able to reconstruct large parts of this hidden structure. The more clues we have, the more complete our reconstruction, and the easier it is to distinguish it from other structures. This is why by following clues we can uncover things hidden.
If all that we know of a person is the shape of her nose, we would not be able to find this person. Lots of people have noses shaped like hers. Even when we know what her ears and eyes and mouth look like, we may still not be able to find this person. Lots of people may have the same ears and eyes and mouth. But starting with information such as this, we can begin to produce a composite picture of what this person looks like. Now a person does not come into this world ex nihilo; she has relations: parents, siblings, uncles, aunts, …. She has friends, teachers, classmates, …. She may have been married, may have children of her own. She works at something, may belong to some profession or organisation, …. And this is how detectives go about looking for a person. The more they know about a person, the easier it is to find her. They do not have to have met the person at one time, or to have her picture. Now the same with reconstructing hidden structures. The more we know about them, the easier it is to reconstruct them. The fact that we have never seen them, and will never see them, makes no difference.
When following clues we reason a lot. But there is one piece of reasoning which we usually assume but do not explicitly carry out:
A and B each has an infinite number of characteristics.
If all the characteristics of A and B are identical, A and B are identical.
All the characteristics of A and B are identical.
Therefore, A and B are identical.
Now this piece of reasoning which we usually do not carry out but assume is deductive. Since it is apparently sound, instead of repeating it over and over again, we spend our time on other more pressing matters, such as looking for clues and developing new ones.
Of course, when reconstructing hidden structures we cannot compare all the characteristics of the structure constructed with all the characteristics of the hidden structure. The number of characteristics of each is infinite. We can only compare as many of them as we can or as required by the context in which the investigation is taking place. Hence in practice we do not say we are absolutely certain that A is identical to B. Instead at most we say A is likely to be identical to B.
Also in practice when we are comparing two characteristics to see whether these two characteristics are identical, we often find that they are not identical, but only close to being identical. This again is another reason for not claiming absolute certainty.
How close is close? This too often occasions debate, reminding us that detection is an art and never as precise as (pure) mathematics.
Because in reconstructing hidden structures we can only compare a limited number of characteristics, it will sometimes happen that we think we have succeeded when in fact we have not. Scientists are interested in the behaviour of gases. For a time they thought that the volume of a gas varies with its temperature and pressure according to the well-known formula,
in which P is the pressure on the gas, V its volume, R a constant and T the temperature. But as more and more characteristics of gases became known, characteristics that they have under different sorts of conditions, scientists realised that their reconstruction of the behaviour of gases as represented by this equation was not correct. Only in a limited range of temperature and pressure does a gas behave approximately in the manner prescribed by this equation. Outside this range, its behaviour is very different. For this reason the equation is nowadays called the ideal gas equation, signalling to us that no real gases actually behave in the way it prescribes except at so-called normal temperatures and pressures, and even then only approximately. The behaviour of real gases——where their volumes, temperatures and pressures are concerned——is too complex to be represented by a simple equation.
The example just cited shows us how important it is that when we try to reconstruct a hidden structure we should try to base this reconstruction on a large number of characteristics. The larger this number, the easier it is to distinguish the hidden structure from neighbouring structures.
But sometimes a structure may have an unusual characteristic, a characteristic that no neighbouring structures have. When this is the case its reconstruction is easier, just as it is easier to look for a person with some distinguishing mark.
As a rule … the more bizarre a thing is the less mysterious it proves to be. It is your commonplace, feature-less crimes which are really puzzling, just as a commonplace face is the most difficult to identify.
—— The Red-Headed League
Everybody nowadays knows that in crime investigation identifying a person by her looks, although useful, is not anywhere as useful as identifying her by fingerprints and DNA samples. Now just as this is true in identifying a person, it is also true in reconstructing structures. Not all clues are equal; some are more telling than others.
When we try to reconstruct a hidden structure by following clues, we try to reconstruct as much of it as we can, or at least as much of it as would enable us to distinguish it from its important neighbours (that is, structures similar to it). When we are trying to reconstruct a crime for example, we cannot be expected to succeed to the very last detail. And there is no need. A court of law does not require that we see things the way God sees them; the court is only interested in, say, who the murderer is. If enough is known to determine this beyond a reasonable doubt, that is sufficient so far as the court is concerned. In solving crimes Sherlock Holmes would often admit that he has not found out every single detail, but only what is 'essential'——that is, that which is sufficient to put the culprits into the hands of the law.
It will be useful if at this time we say a few words about the medium in which we carry out the reconstruction when theseologising. When we try to reconstruct a crime by following clues, we usually do so only in words: we try to describe in words what happened; we do not go out and murder the same person all over again. In some instances, to help in our description; to make it more realistic; we re-enact the crime. We let somebody be the victim, and some one else, the criminal, and we let them go through the motions to show how the crime might have occurred. Needless to say, neither the description nor the re-enactment is the real thing; all that they do is to convey to us in an abstract fashion how, structurally, the event could have occurred. Needless also to say, we cannot ensure perfect correspondence at every point between the description or the re-enactment on the one hand and the real thing on the other. Take the re-enactment for example. The person pretending to be the victim could have a somewhat different body build as compared to the actual victim. And when the pretend criminal swings his knife we could have no way of knowing that he swings it in exactly the same way as the real criminal. But we hope the differences between the real and the pretence would not matter. We hope that the description or the re-enactment tells us enough in essentials as to satisfy our reasons for wanting it, be those reasons legal, moral, historical, or whatever.
What if the description or the re-enactment does not tell us enough? What if the reconstruction is not sufficiently complete? What do we do then?
We do what we normally do in such situations. We re-open the case, we look for more clues and see if we can develop new ones. If after re-opening the case we meet with success, well and good. If not, we may have to shelve it for the moment or even for ever.
When we recreate a crime we do not recommit the crime. When we recreate the structures to be found in the natural world, we do not recreate the natural world. And just as there will be discrepancies between the recreation of a crime and the actual crime, there will be discrepancies between the recreations of structures to be found in the natural world and the actual structures in the natural world. In the case of a crime, by the time the investigation has ended we hope the discrepancies that are still present are insignificant. In the case of the investigation of nature we cannot be so easily satisfied, the reason why, as many have said, science is an unending quest.
Our only clue lay in the truncated telegram, and with a copy of this in his hand Holmes set forth to find a second link for his chain.
—— The Missing Three-Quarter
I was already firmly convinced, Watson, that there were not three separate mysteries here, but one only, and that if I could read the Musgrave Ritual aright I should hold in my hand the clue which would lead me to the truth concerning both the butler Brunton and the maid Howells.
—— The Musgrave Ritual
Clues, we have said, are the characteristics of structures. When we are looking for things hidden by following clues, we are looking for hidden structures; we are doing this with a view to recreating these structures. Now each structure has many characteristics. To recreate a hidden structure and to distinguish it from others we therefore need many clues. But the typical situation we meet with when looking for hidden structures is that we will not have many clues to begin with. Indeed, most of the time we will be happy if we find one or two. Now this is a serious situation. One or two clues will not suffice. If we are limited to only one or two, we will never find what we are looking for. What should we do? How can we overcome this problem?
Let me repeat, we have here a serious situation. We all know it is difficult to look for things hidden. One of the reasons why it is difficult is that clues are hard to find. If clues were easy to find; if clues were all over the place; if they were always there whenever we need them; finding things hidden would have been a great deal easier. But clues are not like that. In most cases we have only very few clues to start with. What can we do in such a situation? What do we in fact do?
Now there must have been something we have been doing which enables us to remedy this situation. For human beings have been theseologising for a long time. In this time they have found out many things that are hidden. So, what is this something?
We do not find this something being mentioned very often by theseologisers. As it were, it has been kept a secret. In the old days, if you were a mason you would not divulge any of the secrets of the masonry trade. This is one of your duties as a member of the guild to which you belong. In this case secrecy is intentional. In the case of theseology, the secret that I am referring to is not intentional; it is just something that theseologisers do not mention very often, probably because in practice there is no need. But the secret is an important one. If we do not bring it out in the open it will be hard to understand why by following clues we can uncover things hidden.
What is this secret?
Since we do not hear this secret being spoken, let’s look for it by ourselves. Let’s look back to what we have done in solving our cryptogram; we should find the secret there. For, in solving our cryptogram we had very few clues to begin with (although we already had more than usual), but eventually we managed to decipher the whole message. How did this come about?
Now if we would go through the process again, we will find that it came about because although we had only a few clues at the beginning, in following them we developed new ones. Old clues can lead to new clues. In our case, because new clues kept appearing, eventually we had enough to decipher the whole message.
There is a secret to theseologising, a secret that all successful theseologisers follow. When theseologising, do not just follow clues; it is important that we also develop new clues from old. In theseologising, when we have developed sufficient new clues from old, these new clues together with the old ones will allow us to recreate the hidden structures we are interested in uncovering. In solving our cryptogram we have been observing this secret, without ourselves being conscious of it. We start out with the one or two clues that we can recognise. We form hypotheses as to what they mean. We apply these hypotheses to the rest of the cryptogram, producing partial decipherments here and there. In these partial decipherments, we occasionally discover new clues. And these new clues lead to even more new clues. What a glorious time we had!
There is a secret to theseologising: Do not just follow clues, but also develop new clues from old. If we do not develop new clues from old, we will never find what we are looking for. But see how natural this secret is! In solving our cryptogram earlier, we never said a word about it.
Why would we not stop and remark upon this secret at the time? Innumerable people must have had the same experience as we; why did nobody mention it?
The reason is obvious. New clues are exhilarating. When we see new clues appearing, would we, would any one, stop the chase, take the time out, and remark upon the secret that they have just stumbled on?
In The Musgrave Ritual Sherlock Holmes starts out with clues about how to interpret the Ritual. These clues lead to the missing butler. The location of the missing butler provides Sherlock Holmes with an explanation why the maid has been behaving in such a bizarre way, which explanation leads to the discovery of a treasure whose existence no one has suspected. It is in this way that Sherlock Holmes solves this particular mystery, or this particular series of mysteries.5 He finds out not just one thing, but many, including things that he was not asked to find out. How does he do it? He does it by developing new clues from old. And this is typically the way he works. Some detectives will point fingers after one single clue. Sherlock Holmes never acts in such a rash manner. He knows that the theseological process is more complex, more demanding; he knows that we have to develop new clues from old; he knows the theseological secret.
Cryptanalysts, when they are faced with a difficult cipher, do not expect they will find many clues initially. But there is one clue they hope they will find. They call it the entrance to the cipher. If they can find it, success will be in their hands. If they cannot find it, there will be no hope. What is this one clue that they call the entrance? It is the clue that will lead to a succession of new clues. As we would expect, in practice cryptanalysts know about the theseological secret.
Cryptanalysts speak of an entrance; Sherlock Holmes, a suggestive clue; detectives nowadays, a break; scientists, a breakthrough. All these terms——entrance, suggestive clue, break, breakthrough, and others like them——indicate the importance of new clues in an investigation. An investigation is not a simple affair. It is not made up of just one or two steps, but many; and these steps are guided by clues. Since we seldom have more than one or two clues to start with, in an investigation we have to develop new clues.
In The Musgrave Ritual, after he has had all the clues——old and new——figured out Sherlock Holmes explains to Reginald Musgrave the ‘long chain of surmise and proof’ which he has arrived at. One area in which we have succeeded in arriving at ‘long chains of surmises and proofs’ is the natural sciences. The chains in this area are getting longer by the day. In crime investigation when a case is closed there is no need to develop more new clues, but in the sciences the cases are never closed: new investigations are built on top of old ones when new clues appear. Although scientists do not usually speak of the theseological secret, in practice they take it for granted, or perhaps those scientists who do not observe this secret in practice simply have nothing to publish.
When theseologising we often fall into adhering to the theseological secret automatically. But sometimes in our daily dealings we do not do this. Instead of adhering to the theseological secret, we depart from it. In reading newspapers how often have we condemned a person for doing this or that on the slim evidence typically provided by newspapers? A man was seen running away from the scene of a crime. Automatically, we assume he is guilty.
We need many clues to uncover a hidden structure and to distinguish it from others close to it. For this reason in practice we have to develop new clues. Of course, this only applies when most of the hidden structure is unknown. If most of the hidden structure is already known we may need only one or two clues to finish the task. And if these one or two clues are already in our possession, there is no need to develop new ones. In a long message in which all the letters have been deciphered except one, this last letter can be easily deciphered without developing any new clues.
One true inference invariably suggests others.
—— Silver Blaze
The proper way to theseologise requires that we do not just follow clues, but also develop new clues from old. This we say is the theseological secret. Now this secret is important not only because it provides us with the many clues we need to reconstruct a structure, it is important for yet another, perhaps even more important, reason, which I will now explain.
To develop new clues from old, the old clues have to be interpreted first. A very useful question to ask at this point is, can new clues be developed from old clues when the old clues have been misinterpreted? Clues have to be interpreted if they are to be useful to us. But when we interpret a clue we do not always do so correctly. Indeed a lot of the time we arrive at wrong interpretations. Now in an investigation we should develop new clues from old. For this purpose the old clues have to be interpreted. But the interpretation could be wrong. When the interpretation of old clues is wrong, can we develop new clues from them? This is the question we are asking.
Notice that new clues are not developed directly from old clues: we do not develop a large number of new clues from the old ones, and when they are all in front of us, both old and new, ask then and only then what each clue means. Instead we ask first of all what the old clues mean. We put forward hypotheses to answer this question. We apply these hypotheses to the evidence (the cryptogram in our example). Now in the process of doing this, we sometimes develop new clues. This is to say, we develop new clues after we have interpreted old clues. Now the question we are asking is, can new clues appear if the old clues have been misinterpreted?
If we look to actual examples; if we experiment; we will find that new clues will only appear when old clues are more or less correctly interpreted. (Why more or less? For this see the section on Approximations in the next chapter). If most of the old clues have been wrongly interpreted new clues will not appear. Sometimes by accident a false clue will appear after old clues have been wrongly interpreted, but this does not happen too often. In any case, false clues peter out.
What does this tell us? If when following clues we are able to develop new clues from old, what can we say of our interpretation of the old clues?
If we put this question to ourselves while working on an actual example (as in solving our cipher earlier), the answer is plain. If we are able to develop new clues from old, the old clues must have been more or less correctly interpreted. The more (generations of) new clues we can develop from old clues, the more likely to be correct our interpretation of these old clues. If the old clues had been seriously misinterpreted, no new clues would have appeared.
But this is to say, if we succeed in following the theseological secret; if we succeed not only in following clues but also in developing new clues from old; we will know that we are moving in the right direction. We are finding out more and more of the hidden structures that we want to know.
Why is it that the more (generations of) new clues we develop from old ones, the more likely to be correct the interpretation of the old clues? Why the more and more? Why do we not say, if the old clues lead to new ones then the interpretation of the old clues must be right, pure and simple?
The reason is not hard to understand. Old clues wrongly interpreted can sometimes by accident lead to false clues. So we cannot say right away that when new clues appear the old clues must have been correctly interpreted. But false clues peter out. When a clue does not peter out; when it leads to more and more new clues; it means it is not false. The more generations of new clues it leads to, the more likely that it is not. This is why we have to wait before we can tell that the old clues have been correctly interpreted.
There is a secret to following clues. If we want to uncover knowledge of things hidden, we should not just follow clues, but also develop new clues from old. When we can do this; when we can develop new clues from old; we know then that the old clues are likely to have been more or less correctly interpreted. The more new clues they lead to the more likely this is the case. The theseological secret is important not just because it enables us to find the many clues we need to reconstruct a hidden structure, it is important also because it provides us with a means of determining whether the results we have obtained are correct.
I shall approach this case from the point of view that what this young man says is true, and we shall see whither that hypothesis will lead us.
—— The Boscombe Valley Mystery
To uncover knowledge of things hidden we have to follow clues. But we have seen by now that this is not sufficient; we also have to develop new clues from old. So, there are two things we have to do over and over again when we want knowledge of things hidden: we have to follow clues and we have to develop new clues from old. These two reminders together——follow clues and develop new clues from old——I shall now call the theseological method.
The theseological method: Follow clues and develop new clues from old.
The theseological method tells us to do two things. The first——Follow clues!——most people already know. The second——Develop new clues from old!——we often know in practice but seldom mention. I hope from now on we will mention both often, since together they make up the theseological method.
The theseological method is a method in that as long as we are able to adhere to it; that is, as long as we are following clues and developing new clues from old; we know our older results are likely to be right. We know this even though we have no direct access to the things hidden, from which the clues originate. The method by itself provides us with the results we want. We want results that are likely to be right. The theseological method, which tells us to follow clues and develop new clues from old, provides us with results that are likely to be right. The theseological method is a method in the same sense that we have a method for adding numbers on paper (the addition algorithm). When we add numbers on paper, we know our results will be right so long as we carry out all the steps correctly. If we have carried out all the steps correctly, there is no need to take out a counting board and do the same addition all over again, this time by counting beads. The method suffices. Now the same is true of the theseological method. When we succeed in following the theseological method we can tell that our results will very likely correspond to what is the case. We can tell without putting our results side by side with what is really the case and comparing the two. We can determine to a high degree of certainty that there is correspondence without making this kind of comparison.
The existence of the theseological method answers the question, why things hidden can be known. Things hidden cannot be known directly; it is by a method that they are known; and the method in this case is the theseological method. When we succeed in following this method; when we have been following clues and developing new clues from old; the older results are likely to be right. If they had been wrong, they could not have led to more and more new clues. It is by following the theseological method that we find out things that are hidden. Without the theseological method this kind of knowledge would have been impossible.
I had heard what he had heard, I had seen what he had seen, and yet from his words it was evident that he saw clearly not only what had happened, but what was about to happen, while to me the whole business was still confused and grotesque.
—— The Red-Headed League
The theseological method tells us to follow clues and develop new clues from old. By following this method we can uncover knowledge of the hidden while the hidden remains hidden. When we have a method, the method certifies the results, obviating the need for comparing results with originals. And this is what the theseological method does. It removes the need for direct access. When we have broken a cipher we can tell on our own that we have succeeded; we do not have to go cap in hand to our enemies and beg them to show us their copy of the cipher. When Sherlock Holmes has solved a crime, it is not essential (although welcomed) that the criminals confess. When scientists have found out some of the laws governing the workings of nature, they do not take up the phone and dial God's number to ask Him to confirm their results. Now in saying that there is this method and that it obviates the need for direct verification, I am not saying the method is easy to apply. I have asked readers to try to solve our SBR cryptogram on their own. If they have, they will know it is not easy to crack a cipher, method or no method.
Now some will be puzzled. If there is a method, they will ask, should it not make cryptanalysis easy? Is that not what a method is for?
People frequently expect that if there is a method for uncovering knowledge of the hidden, this method will be easy to follow. Certainly, in the past most authors writing on method have this expectation. Unfortunately for us, reality does not always conform to our expectations. If we look around us at the many different kinds of methods we have in different areas, we will find that some of them are easy and some not; there is no reason why, if there is a method for uncovering knowledge of the hidden, this method has to be easy. We have methods for doing many different kinds of things. We have methods for putting together bicycles, automobiles, aeroplanes,…; we have methods for doing addition, subtraction, division, multiplication; we have methods for programming VCRs. Some of the many methods we have are easy, some not.
We can divide the methods we have roughly into two kinds: foolproof and non-foolproof. A popular example of a foolproof method is the instructions that come with a cake-mix, the kind we buy in supermarkets. Once upon a time to make a cake takes skill, so that not everyone can make a cake. Nowadays, even a fool can make a cake. All that she has to do is buy a cake-mix and follow instructions. These instructions are so simple that nobody can fail to follow them. And when these directions are followed a cake will result. One can tell in advance that this will occur, the reason why people buy cake-mixes. In the case of a cake-mix the method is foolproof.
But not all methods are foolproof. Before we had cake-mixes we had recipes for making cakes. Some of these recipes are not easy to follow at all: they require skills not everyone will have. For example, in many of them we find this curious piece of instruction: 'Fold sifted flour into egg mixture.' Is this an instruction a fool can carry out? I vouch that it is not.
Does it mean then that these recipes are not methods? No; a method gives directions as to what to do. These recipes give directions as to what to do. A method assures results if the directions are followed. These recipes assure results if the directions are followed.
Traditional cake recipes are methods, but they are not foolproof methods. It is not the fault of a recipe that I, lacking the requisite skills, cannot follow some of the instructions it contains. Not every method is foolproof. Some methods require skills fools do not possess. But even with these non-foolproof methods, if the method is followed the promised results will occur. Experienced bakers have no difficulty following cake recipes and produce perfect cakes every time.
The theseological method is not easy to follow; it is not foolproof. It requires skills that we often do not have, skills in detecting clues for example, and skills in interpreting them (and these are just two of the many skills involved). Still, it gives directions that, if followed, will lead to the right kind of results. And we know these results will be right so long as we have followed the method correctly; there is no need for direct verification.
Some people say nowadays there is no such thing as a scientific method. This of course depends on what one means by 'scientific method'. In order that we do not waste time arguing about words, let's divide up this question and see what the answers are, as we do below.
Is there a method that leads to knowledge of things hidden while they remain hidden?
Yes, there is such a method. I call it the theseological method, after Theseus, to remind us that the method depends on clues.
Is there a method that can lead to the kind of discoveries that scientists have been making, such as the structure of the molecule or the structure of the solar system?
Yes, there is such a method: the theseological method.
Do we know that scientists are in fact using the theseological method?
We do not know for certain that they are, but all clues point to the possibility that they are. They are interested in structures, they follow clues, they develop new clues from old, they do all of those things that people following the theseological method should do. In a word, they behave very much like Sherlock Holmes. And the results that they produce, by and large, are the kind that the theseological method can produce.
Of course, even Sherlock Holmes does not follow the theseological method at times and as a consequence pay a price. So it is possible that scientists too do not follow the theseological method at times. But when they do not, they will have to pay a price too.
Is there a method that can do all that the theseological method does but does not require clues?
I do not know of any such method.
Is there a method that can do all that the theseological method does but is simple, mechanical, and even foolproof?
I do not know of any such method.
Does the theseological method require ingenuity on the part of those who exercise it?
Yes, it does. It requires ingenuity to detect clues and to interpret them.
Does the theseological method teach us how to be ingenious?
No, the theseological method does not teach us how to be ingenious.
In what sense is the theseological method a method?
It is a method in the sense that 1)it gives directions as to what we should do if we want knowledge of things hidden; and 2)if we should succeed in following these directions the results obtained are likely to be right. The more new clues these results lead to, the more likely to be right they are.
No, no: I never guess. It is a shocking habit, -- destructive to the logical faculty.
—— The Sign of the Four
When looking for things hidden we should follow the theseological method; that is, we should follow clues and develop new clues from old. But children sometimes do not do this. When asked half way into a detective story who they think is the murderer, they would make one wild guess after another. The maid? No? The butler? The vicar? The gardener? We call these wild guesses because the child has no idea who the murderer really is. She is not following clues. She has no inkling that she should follow clues. She mentions these people only because they have occurred in the story. Now we know we cannot do this. We cannot find things hidden by making wild guesses. When a child makes a wild guess she is hoping the grown-ups will tell her whether the guess is right, but when we are looking for things hidden there is no one there to tell us whether we are right; we have to determine for ourselves. Which is why we have to follow the theseological method, that is, follow clues and develop new clues from old.
When we follow clues it does not mean that every interpretation we make of a clue will be right, but at least it has a chance, even if small. But if we make wild guesses, the chance of any one of them being right is close to nil. Moreover, even if one of them should be right we would have no way of telling that it is.
Are wild guesses completely forbidden when we are looking for knowledge of the hidden? Interestingly enough, no. Sherlock Holmes never guesses, but many experienced investigators will, sometimes. By 'guesses' we mean here 'wild guesses'. Experienced investigators will make wild guesses sometimes. They will when they run out of clues and are desperate. They know that wild guesses have next to no chance of being right, but 'next to no chance' does not mean 'no chance at all'. Once in a blue moon a wild guess could turn out to be correct.
How do we know when a wild guess is correct?
We know when the wild guess leads to more and more new clues. So, this is what experienced investigators sometimes do when they are desperate. They make wild guesses, hoping against hope that they might chance upon one that will lead them to more and more new clues. Not surprisingly, in the great majority of cases their hope is unfulfilled. But we should not be surprised if we should come across a case in which it is.
But we should repeat, making wild guesses is a desperate measure; it is not something we should do too often. Sherlock Holmes says he never does it.
Notice also that while experienced investigators will sometimes make wild guesses, they will not depend on wild guesses from beginning to end. If they did they would have no idea if any of these wild guesses are right. When experienced investigators make wild guesses, they are hoping for just one lucky break. Usually they will not have time to wait for a second. One lucky break is hard enough to come by; it is not likely that a second will follow soon after.6 After that one lucky break, therefore, experienced investigators will resume following the theseological method; that is, they will resume following clues and developing new clues from old.
However, if time is no object; if we are in no hurry whatsoever; if we do not mind taking aeons and aeons to go from one step of the investigation to the next; it will then be permissible to look for as many lucky breaks as we want; so long as eventually we resume following the theseological method. This is to say, luck can be given a bigger and bigger role in an investigation provided we do not mind spending a huge amount of time over it. Now an individual will not have this kind of time, but the human species in its evolution——both cultural and biological——does. Thus it is possible that many things we nowadays know are the result of large numbers of lucky breaks in the past.
When we follow the theseological method; that is to say, when we follow clues and develop new clues from old; we do not find out everything at the same time. Rather, we work gradually. In solving our cryptogram we start out by deciphering one or two letters, then we decipher more, then more, until eventually we have all the letters accounted for.
Also, when we follow the theseological method the results we arrive at are tentative: we will not know right away that they are correct. To find out whether a result is correct, we first ask whether it is suggested by clues. If it is, we then ask whether it leads to new clues. A result suggested by clues is infinitely better than a wild guess. A result that leads to new clues is better than one that does not.
Do results obtained by following the theseological method ever stop being tentative?
No; since in following the theseological method we never arrive at absolute certainty. When a result leads to more and more new clues, it becomes more and more certain, but never absolutely certain. Results obtained by following the theseological method are tentative. They can become less and less tentative, but they never stop being tentative.
The way we catch criminals (by following the theseological method) is often compared to tightening a noose. At the beginning of the investigation we have no idea who might be responsible for the crime. As the investigation proceeds; as we meet with success; we begin to have some rough idea. When we find out more and more, the rough idea becomes less and less rough, until eventually we are able to specify who that person is. At that point, figuratively, we tighten the noose: we name names; we point fingers.
Now not only is catching a criminal like tightening a noose, making scientific discovery can be also. Scientists are not always clear what it is they have discovered; they have to be given time to tighten the noose. They may know they are on the trail of something new but they may not know until later what that something new is. Lavoisier's discovery that de-phlogisticated air was in fact an element——which he called oxygen——did not take place at some one point in time; instead it resulted from work done over many years, in which his thinking about the kind of chemistry current at the time gradually changed.
In an investigation when new clues appear after old clues have been interpreted, the interpretation of the old clues is likely to have been correct. We can tell that this is the case while the hidden remains hidden. This is to say, when following the theseological method we do not have to wait until the end of the investigation before we can evaluate results. Once new clues appear, even though we are still some way from the end, we know we are likely to be moving in the right direction. It is proper when theseologising that we do this, that is, it is proper that we evaluate results as we proceed. This is not a sign of impatience but in fact is required by the nature of the theseological process.
Now this is worth noticing. It is worth noticing that when we are following clues we can tell that we are moving in the right direction without knowing what our destination is! Normally we have to know where we are going if we are to tell whether we are closer to it. If you do not know Scotland is your destination, you will not know you should travel north if you are starting out from London. But when we are following clues, without knowing our destination, we can tell whether we are moving in the right direction. When we are half way through in solving our cryptogram, we will not know yet what the whole cryptogram says; yet we know we are getting there! We know we are closer to the end without knowing what the end is.
Sometimes people say, when you are engaging in purposeful activity, you should be clear about the end before you start; otherwise you will be wasting your effort. If you know the end, you can then look for the means for reaching it. If you don't even know the end, how can you know what route you should take? According to this view it is irrational to start on a task without knowing, and knowing clearly, what the task is aiming at. But this view does not apply when we are following clues. When we are following clues we do not know where we will end. If we did we would not have to follow clues. If everyone knows who the murderer is, there is no need to call in Sherlock Holmes. It is just because we do not know, that we have to follow clues. But although we do not know where we will end we know what we should do: we should follow clues and develop new clues from old. And although we do not know where we will end we know whether we are getting closer to it.
So it is not always the case that we have to know the end, and know it clearly, before we can begin. Sometimes, we can begin and know that we are closer to the end without knowing in advance what the end is, let alone be clear about it.
When we follow a method we sometimes make mistakes without ourselves knowing. Suppose I have a long series of numbers to add together: 8, 7, 4, 15, 2, 41, 103, etc. So I first add 8 to 7 to get 15. Then I add 4 to 15 and, unhappily for me, arrive at 18 by mistake. Suppose I do not notice the mistake, so I go on and add 15 to 18 to get 33 and in this way continue down the series. Having made the mistake and not noticed it, the mistake is now buried in my work; all the results I obtain after the mistake will be wrong. But I will not know this since I am not aware that I have made a mistake. And as long as I keep myself to the task of adding more and more of the numbers together, I will not discover the mistake I have made. If I want to find the mistake, I will have to stop the addition and recheck what I have done. If I do not; if I have no chance to recheck; I may never know that I have made a mistake.
Now to uncover things hidden there is a method to follow, the theseological method, which tells us not just to follow clues but also develop new clues from old. But in following the theseological method we sometimes make mistakes too. For example, we could have interpreted one of the clues in the wrong way. Suppose we have done this. Suppose we have taken one of the clues to mean something very far from what it should mean. By doing this, then, we have made a mistake. But suppose we do not know that we have. What will happen from this point onwards? What will happen after we have made a serious mistake when theseologising? Will we still be able to carry on with our task? When we do additions we can carry on with the addition even after we have made mistakes: we still can get results except that they will all be wrong. In theseologising, can we carry on with our theseologising after we have made serious mistakes and, because of these mistakes, get wrong results after wrong results?
At this point, I will ask readers to experiment. I will ask them to purposely make some serious mistakes in cracking the cipher we have been using as example. If they take up my suggestion they will discover an interesting characteristic of the theseological process. In the theseological process, after we have made serious mistakes, in time, usually soon, our way forward will be blocked: we cannot obtain results any more until the mistake is corrected. This is very different from doing addition. With the latter, as we have pointed out, we can keep on obtaining result after result even after we have made mistakes.
Experienced investigators know that when serious mistakes are made in following clues, progress in the investigation will soon be impossible. But while this is known to experienced investigators, to those not familiar with the theseological process this can at times be surprising. In doing addition, right steps lead to right results; wrong steps to wrong; this is reasonable. Why then, they ask, should it not be the same for the theseological process? Why, instead of leading to more and more wrong results, should theseological progress stop altogether after a seriously wrong step?
If we have experimented and then look carefully at our experiment we will see the reason why. In the theseological process, after serious mistakes have been made clues in time will dry up. When there are no clues, there can be no progress in the investigation. When we are theseologising, new clues can only be developed from old clues if the old clues have been more or less correctly interpreted. If the old clues have been gravely misinterpreted no new clues will appear. Here we find the famous brick wall that detectives and other investigators talk about. In an investigation we need clues if we are to advance. Usually in an investigation there will not be very many clues to start with. When these are used up and serious mistakes have been made, no new clues will appear. Without new clues, the whole investigation will grind to a halt. At this point forward movement in the investigation is impossible. We cannot go through a brick wall.
In physics a brick wall occurred in the study of blackbody radiation towards the end of the 19th century. The problem confronting physicists at that time was how to account theoretically for the electromagnetic energy radiated by a perfect emitter——a blackbody——at different frequencies. People tried to provide this theoretical account in different ways, but none of the accounts they were able to come up with would fit experimental results. The situation was looking desperate; people were running out of ideas; they did not know where to turn. They had hit a brick wall. Now brick walls cannot be removed until the mistakes causing them are corrected. How was the brick wall removed in the case of blackbody radiation? It was removed by Max Planck who found that we could restore agreement between theory and observation if we were to assume that energy was not emitted continuously as traditionally thought, but in discrete packets called quanta. The idea of continuous emission turned out to be a mistake. It was because of it that scientists trying to account for blackbody radiation hit the brick wall. For his work on quantum theory Planck was awarded the Nobel Prize in 1918.
In The Man with the Twisted Lip Sherlock Holmes met with one of his most difficult cases. A man had disappeared moments after being sighted by his wife in a part of London where she did not expect to see him. An immediate search of the area turned up nothing, except for a few drops of blood and most of the man's clothing. On the surface the case should have been simple, at least, simple for Sherlock Holmes. But for a long time, tried as he might, Sherlock Holmes could make no sense of it; there was not a single glimmer of hope; all to him, he says, was dark. Sherlock Holmes is usually ahead of Watson when it comes to finding solutions, but this time Sherlock Holmes actually turned to his friend for help, with the stated hope——unfulfilled——that he, Watson, might see light where he, Sherlock Holmes, saw none. For once, Sherlock Holmes had hit the brick wall!
It seems absurdly simple, and yet, somehow I can get nothing to go upon. There's plenty of thread, no doubt, but I can't get the end of it into my hand. Now, I'll state the case clearly and concisely to you, Watson, and maybe you can see a spark where all is dark to me.
—— The Man with the Twisted Lip
In an investigation we will hit a brick wall if we have made a serious mistake. What serious mistake did Sherlock Holmes make in this case?
A colossal one. He assumed there was a murder when in fact there was none. His mistake was so big that he admitted to Watson afterwards that he had been nothing less than 'one of the most absolute fools in Europe' and that he deserved 'to be kicked from here to Charing Cross'.
In adding numbers together self-deception is possible. I keep thinking I am good with numbers. After adding together all my bank accounts I come to the conclusion I am a millionaire. In fact I am not because of all the arithmetical mistakes I have made. If I do not re-check my calculations, or if my re-checking is as bad as my original calculations, I may never wake up from my self-deception until it is too late.
But it is not so easy to deceive ourselves when theseologising. When following clues, if we make serious mistakes we will soon know, or at least be in a position to suspect. For we will hit the brick wall: clues will dry up. Without clues we cannot find anything; we cannot go through a brick wall.
Experienced investigators know what it is like to hit a brick wall. People new to investigations should wait for this to happen. This kind of experience teaches us there is a limit to self-deception. It has its best effect when it happens after we have had some success. This way we know what it is like to have been doing things correctly and also what it is like when we have made serious mistakes.
Investigations, even when successful, are usually rough journeys. Few, if any, are like the solution for our cryptogram offered earlier, in which not a single mistake is made. In real-life investigations mistakes, including serious mistakes, are common. There are often dead ends and much retracing of steps. There are long moments in which a person is completely lost and not know what to do. There are even times when mistakes are corrected and the corrections then rejected for some absurd reason not known at the time as absurd. It takes a person with great patience and fortitude to be a good investigator.
Success in investigations depends on our finding clues and figuring out what they mean. Clues are the characteristics of structures, disguised. If the things we are investigating have no structure, we cannot find out what they are by following clues because in their case, there will no clues to follow.
If there is no light, there can be no photography. Similarly, if there is no structure, there can be no clues and therefore no theseologising. Photography depends on a characteristic of the world, so does theseology. Because they do, I call them empirical arts.
Both photography and theseology are empirical arts. For them to be possible certain empirical conditions have to be satisfied. If we want to take pictures we need light. If we are to uncover knowledge of things hidden, the things hidden have to have structures (or be parts of some structure).
In previous sections I have been asking readers to experiment. Can clues wrongly interpreted lead to new clues? I have suggested that readers should experiment. Experiments are important in an empirical art. Photographers do not discover everything they should do by thinking; they experiment. Photography is an empirical art, so is theseology.
It is useful to point out that while theseology is an empirical art, the art of reasoning is not. The art of reasoning is non-empirical. Whatever the world is like, we can still reason: we can always make up premises and see what conclusions follow from them. A common example of a piece of reasoning is the following.
If pigs had wings they could fly.
Pigs had wings.
Therefore they could fly.
Pigs do not have wings, and they cannot fly, but this does not stop us from reasoning about them.
In the last chapter I pointed out that the art of detection (theseology) is distinct from reasoning by drawing attention to the fact that to practise the art of detection we need clues whereas to reason we do not. Now, if we want to, we can also alert others to the fact that the two are distinct by pointing out that whereas the art of detection is empirical, reasoning is not.
The most difficult crime to track is the one which is purposeless.
—— The Naval Treaty
In our example in cryptanalysis we have a message in English. This message is encrypted by a cipher which works in a regular way: T is always translated into S, H into B, E into S, and so on. This is to say, the cipher used has a structure. Now since theseology depends on structure, if we can construct a cipher that has no structure, a cipher that works in a totally random, unpredictable way, such a cipher will be unbreakable. It will be unbreakable because it will leave behind no clues. With such a cipher the first time TH occurs, it could be translated into XR. The second time, into KB. The third time, CT, and so on. Since all these translations are random, we can never find out that they all stand for TH.
Ciphers that are theoretically unbreakable have been in use for centuries. The best known among them is the one-time pad, sometimes mentioned in spy novels. In this case two identical pads are made, one given to the spy, and the other kept by her controller at home. On each page of these two pads is a simple cipher, with each cipher being different from all the other ciphers on the rest of the pages. Now this is what the spy will do when she has a message to send home: She will encipher the message using the cipher on the top most page. After she has done this she tears off this page and puts a match to it. Next time she has a message to send she will do the same. Her controller at home will have no problem deciphering her messages since she, the controller, can easily keep track of which page the spy is using. But so long as the spy keeps her messages short and destroys each cipher after she has used it, the messages she sends home are unbreakable. Why? The reason is simple. In our SBR example the cipher is breakable because there are clues. SB appears twice, both times at the beginning of a word, suggesting to us that it could stand for TH. Now when we use the one time pad, because we have to keep the messages short, TH is not likely to occur a second time in the same short message. Whatever it has been translated into by the cipher, the disguised version is not likely to appear more than once. Will the same translation appear in the next message? No; because for the next message the spy will be using a different cipher.
The one time pad is an unbreakable cipher, probably the best known among all unbreakable ciphers. Indeed, unbreakable ciphers are often generically referred to as one time pads, as all photocopiers are often generically referred to as Xerox machines.
A one time pad is unbreakable but cannot be used to convey long messages. Can we construct an unbreakable cipher that can convey long messages?
It is easy to see that it is not hard to do. A one time pad changes the cipher after every short message. If we have a long message and we change cipher after every letter, provided we have as many ciphers as we have letters in the message, we can encipher this message. (Of course, if we want this message to be unbreakable, we have to make sure that the change in cipher is random. If it is not; if the change follows a pattern; the message will be breakable if it is long enough. For example, suppose we keep using the same twenty ciphers over and over again, and always in the same order, even though we change cipher after every letter, our message is breakable: clues in the form of patterns will appear in the long run.)
When we have a long message and it has been encrypted by an unbreakable cipher, the cryptogram generated will be indistinguishable from a series of random symbols. In this cryptogram there will be no discernible patterns. This is to say, given a series of random symbols we cannot tell whether there is a message behind it. The series of random symbols could be just a series of random symbols, or it could be the cryptogram of a message encrypted by an unbreakable cipher.
A random series of symbols will contain no clues, but we cannot take the absence of clues to mean there is nothing hidden. We cannot take randomness as a sign that there is no structure behind the randomness.
At this point some may ask, since unbreakable ciphers are so easy to construct, why do we not use them more often, or even exclusively? Why bother with any other kind?
In practice unbreakable ciphers are not used very often at all. This is because they all suffer from one major drawback. To illustrate, let us use our spy and her one time pad as example. In her case, it is true that her messages are unbreakable. However, in order that she can keep sending messages home, she has to hang on to her one time pad and make sure that no one gets their hands on it. She has to keep the physical pad in her possession (a one-time pad is not something that she can commit to memory). This is where the weakness of this method of encryption lies. The one-time pad is ‘clumsy’. Once the physical security of a one time pad is breached, the secrecy we desire from it is lost. In the case of our spy the security she can provide for her communication is only as high as the physical security she can provide for her one time pad. If she keeps her one-time pad under lock and key, her communication is no more secure than her lock and key.
Let us look at the same problem from a different angle. Suppose our spy has finished her one time pad and asks her controller to send her a new one. How is her controller to do this? What method should she use to send her spy a new one time pad? We use a one time pad because we want secrecy, but if we have to send out a one time pad, that is where the secrecy will break down. If we send out the one time pad by registered mail, our communication from that point onwards will only be as secure as registered mail.
The weakness of unbreakable ciphers lies not with the ciphers themselves, but with how they are conveyed from one point in time and/or space to the next, or as professionals would say, with how the cipher is managed.
Can we have a cipher that is unbreakable but easy to manage? A cipher that is secure but which we can send out using, say, regular mail?
We have this kind of cipher these days. They are called public key ciphers. Public key ciphers are theoretically breakable but in practice unbreakable. They are practically unbreakable because to break them will require thousands of years even when using the fastest computers.
Public key ciphers work in the following way. Suppose I am using a public key cipher and I want you to send me a message that I do not want any one else to read. For this purpose I will send you a key, so that you can encipher your message. This is the public key. This key I can send out using whatever method I find the most convenient, even regular mail. It is called a public key because it does not matter if everybody knows it. You will encipher your message using this public key. But the encrypted message you send me cannot be deciphered by the public key. To decipher your message, I have to use my own private key. Since this key I keep to myself, no one else can read your message. Public key ciphers are sometimes called public key private key ciphers, since they use not just one key but two. They are theoretically breakable because theoretically the private key can be obtained mathematically from the public key, but to actually carry out this mathematical operation will take thousands of years even using the fastest computers.
Nowadays when we convey private information over the internet——to our banks for example, we are using public key ciphers even though we may not be aware of it. There are other kinds of ciphers which are theoretically breakable but practically not, but they are not as interesting as public key ciphers.
We uncover knowledge of things hidden by following clues. Clues are the characteristics of structures, disguised. Disguise can sometimes be so complete that the things hidden cannot be known, as in the case of the one time pad. At other times, disguise can be so complicated that even though in theory the things hidden can be known, in practice they cannot, as in the case of public key ciphers. In addition, clues can be destroyed intentionally, as they sometimes are by criminals. They can also be lost through the ravages of time, the reason why archaeological finds have to be protected. It is not possible to know everything that is hidden. No claim is made in this textbook that it is. There are limits to what the art of detection can do.
Watson thinks the world of Sherlock Holmes. To Watson Sherlock Holmes is the best detective that the world has ever seen. Yet, despite his admiration for him, Watson seems to have learnt enough about the art of detection from Sherlock Holmes to realise that no shame is involved in making known to the world that there are cases that even Sherlock Holmes cannot solve.
Somewhere in the vaults of the bank of Cox and Co., at Charing Cross, there is a travel-worn and battered tin dispatch-box with my name, John H. Watson, M.D., Late Indian Army, painted upon the lid. It is crammed with papers, nearly all of which are records of cases to illustrate the curious problems which Mr. Sherlock Holmes had at various times to examine. Some, and not the least interesting, were complete failures, and as such will hardly bear narrating, since no final explanation is forthcoming.
—— The Problem of Thor Bridge
Have you tried to drive a harpoon through a body? No? Tut, tut, my dear sir, you must really pay attention to these details.
—— The Adventure of Black Peter
Clues are the characteristics of structures. The more characteristics of a structure we know, the easier it is to distinguish it from other structures. This is the reason why in an investigation we not only have to follow clues but have to develop new clues from old. But this is just to say, when we follow the theseological method and succeed, we will have uncovered a lot of details about those structures we want to know. This is unavoidable. Clues lead to these details. In an investigation we need many clues. So, when we succeed we will have found out a lot about the details of those structures we are interested in.
Sherlock Holmes is interested in details, even details that no one else is interested in. What is it like to drive a harpoon through a body? He wants to find out. He wants to find out in detail, so that he can determine what happened when Black Peter was killed. To every one else it is sufficient that Black Peter is dead, killed by a harpoon. Not to Sherlock Holmes. He wants the details. He needs the details in order that he can solve the crime.
In the investigation of nature scientists have accumulated huge quantities of details, most of which possess no interest whatsoever to ordinary people.7 Since a lot of scientific research nowadays is supported by public funds, this has prompted questions by some politicians and their supporters. Why spend money on these minute details? Details which are of no interest except to the scientists themselves? Why use public money to support these scientists? Why don't they concentrate their research on more useful things, particularly things that can turn a quick monetary profit? This kind of request is like asking Sherlock Holmes to forego his research on the harpoon and concentrate just on the question, who killed Black Peter.
When we follow the theseological method, we will produce a lot of details when we succeed. These details are to be welcome. Without them we will not have found out anything. If we dictate in advance what details we will uncover and what not, we totally cripple ourselves.
One view of science that has some currency at the moment takes it that science is just a kind of myth, a story that we tell ourselves, a story that we make up. According to this view science does not tell us what reality is like. Reality, according to this view, is unknowable. I do not share this view. I know of no story that provides us with the quantity of details that science has provided. One of the complaints about science is that its details are boring. Would any one in their right mind create a story with such a huge quantity of boring details?
Earlier in this chapter, to acquaint readers with the theseological process we gave them a cryptogram to solve. If they had taken up the challenge they would have come across an interesting characteristic of the theseological process. In solving the SBR cryptogram, those readers who have never done this kind of thing before will find that the first few steps are quite, quite difficult. Indeed, it may even take them some time at that point just to decide what they should do. But if they are able to get through these first few steps, things will become easier and easier, so much so that towards the end the steps taken are almost effortless. This is typical of the theseological process. In an investigation in which we are following the theseological method, the beginning is always much more difficult than the end. There is a reason for this. At the beginning very little is known; at the end, so much more. When little is known it is hard to know what clues to look for, and even when we have found them, hard to know what they mean. When more is known, both tasks are easier.
With the theseological process things are difficult at the beginning but easy towards the end. But this is not always known. Often people notice how easy the steps are at the end. They then think that the whole process is just as easy. 'What is there to cracking a cipher or doing science?' they say; 'It is no more difficult than deciphering a ten-letter word in which nine is already known.' We have seen that this is not true. The theseological process is complex. The end is easy, but the beginning is not.
I mentioned earlier that, traditionally, people expect that if there is a method leading to knowledge of the hidden, this method should be simple. This expectation seems to have grown out of the same misunderstanding of the theseological process: if the process is simple towards the end, it must be simple throughout, which of course it is not.
We follow the theseological method to uncover knowledge of things hidden. The theseological method tell us to follow clues and develop new clues from old. Clues are the characteristics of structures, disguised. When something hidden is known, therefore, it is known because it has a structure (or is a part of some structure). If it has a structure (or is a part of some structure) clues can lead us to it. Knowledge of the hidden, therefore, depends on structure. No structure, no clues. No clues, no knowledge.
Knowledge of the hidden depends on structure, not on whether the hidden is already known. This is important to keep in mind in view of a popular misunderstanding of the theseological process. According to this popular misunderstanding the reason a crime can be solved is that the criminal knows what happened. The criminal knows; therefore the detective can find out. If nobody knows, neither can the detective. Confessions are very important to those who accept this mistaken view. When there is no confession, the detective may say whatever she wants, we will not know what actually happened. The detective will of course have her theories but in the absence of confessions we will not know how good her theories are. The detective knows in advance of the investigation that the criminal knows. At the end of the investigation she will have to produce the criminal so that the criminal can tell us what she, the criminal, knows. People who think this way acknowledge, of course, that detectives follow clues, but they say clues can mean so many things. Without confessions we will not know what they really mean.
This kind of thinking, we can now see, is the result of not understanding where clues come from and how we follow them. Detectives follow clues. Clues are left behind by the criminals but this does not mean only they can tell what these clues signify. A crime as an event has a structure. There is a motive for the crime. The crime happens at a certain time, in a certain place. Those things that happen during the crime do so in a certain order. And they have consequences. A person shot will bleed. If she can still move she will leave behind a trail of blood. There are some things the criminals can control, but there are also some things they cannot. Because the crime has a structure it will leave behind clues. Clues can lead to other clues. By going from clue to clue we can reconstruct what happened. It is true that a clue by itself can mean many things, but when clues lead to more and more new clues, we will have a better idea what they all really mean. Confessions are not necessary, even though they are usually welcome.
Keep in mind also that sometimes people make false confessions; that is, they admit to crimes they did not commit. Confessions therefore cannot be made into the final arbiter of what really happened. Now detectives know this. In practice detectives——at least, the better ones——do not take confessions at face value. They challenge confessions. They subject confessions themselves to investigations.
When we want to uncover knowledge of the hidden, the important question is whether the hidden is structured, not whether it is already known. A cipher that works in a random way is known to those who are in possession of it. But a cipher that works in a random way cannot be broken. That it is already known (to some) makes no difference.
However, although whether something hidden can be known does not depend on whether some one already knows it, if some one already knows it, it sometimes makes the task easier. To illustrate this point, let me tell a story and then ask readers what I should do. Suppose I am a senior cryptanalyst in charge of a cipher office. The office has been given a cipher to break. It has been done. It has been done in one single day by one of the trainees who unfortunately has gone on holiday to no one knows where and has taken the solution with her (a major lapse in security, over which actions will have to be taken later on). The solution is now urgently needed. What should I do? Should I wait for the trainee's return despite the urgency? What advice will readers give?
The advice I think everybody will give is the following. Why wait? If a trainee can crack that cipher in one day, how long will it take an experienced person? Instead of waiting for her return, why not work out the solution yourself?
From this story we see, even though advance knowledge is not essential when looking for things hidden, knowledge that somebody already knows can sometimes help in making the task easier. This it does by acting as an additional clue. If even the trainee can crack this cipher, the structure of this cipher cannot be all that complicated. In cracking this cipher therefore, as a first hypothesis, there is no need to try out the more complicated kinds.
Newton invented the integral and differential calculus, but he would not share with Leibniz his new invention. So Leibniz went home and re-invented the calculus all by himself. No doubt Leibniz's task was made a whole lot easier knowing that Newton had already done it, but this does not mean that foreknowledge is essential for invention. Before Newton no one knew anything about the calculus.
Crimes are committed by criminals. Criminals are the authors of their crimes. And just as crimes have authors, so do ciphers and secret messages. Instances like these have led some to believe that things hidden can be known because these things have authors. If they had no authors, they would not have been known, so they think. This, we can now see, is also a misunderstanding. Whether things hidden can be known depends on whether they have structures or are parts of structures, not on whether they are authored. Suppose our SBR cryptogram materialises all by itself out of thin air. Even so, we would still find the same clues in it and by following them arrive at the same decipherment.
It is true that our SBR cryptogram is not likely to materialise all by itself out of thin air. The English language is complex; it is not likely that a message written in such a complex language would appear all by itself. But there are complex structures in nature that are what they are simply as a result of evolution. Many molecules found in living organisms have complex structures. These structures result from evolution, not because some author makes them. Yet, despite the lack of authors, by following clues we can decipher these molecules and find out how they are put together.
In order that we can have knowledge of things hidden, these things that are hidden do not have to be authored. We uncover knowledge of things hidden by following clues. Clues are the characteristics of structures. When there are structures clues could exist. They could exist whether these structures are authored or not.
It is true that when a hidden structure has an author and we know who the author is, it is often easier to uncover that structure. For example, suppose we know who the author of a particular cipher is; it may well be the case then that it is easier for us to crack that cipher. For that particular person may have certain predilections when it comes to creating ciphers. If we know what her predilections are, we will know what kinds of ciphers to try first. But even in this case, when we succeed in cracking her cipher it will be because the cipher has a structure, not because it has an author. Suppose the same author has created an unbreakable cipher, even though we know who the author of this cipher is, it will still be impossible to crack.
In cracking a cipher, we determine whether our results are right by seeing whether they are obtained by following the theseological method. The theseological method tells us to follow clues and develop new clues from old. Results obtained by following clues are more likely to be right than wild guesses. Results that lead to new clues are more likely to be right than results that do not. Now it is likely that some readers will find this puzzling. Why take the trouble of saying all this? Is there not, they will ask, a simpler way for evaluating results in cryptanalysis? One that everybody already knows? Everybody already knows that in cracking a cipher, there is a simple criterion we observe when evaluating results: coherence. If at the end you arrive at a message that makes sense, a message that is coherent, you have arrived at the true message. What more is there to it? Why complicate things by talking about clues and new clues?
I often warn my audiences that the proper explanation of why things hidden can be known will clash with many of the ideas they are likely to have accepted. We have given some examples already. Here we have another. The two versions of how we should evaluate results in cryptanalysis are not compatible. The better-known version——that we evaluate results by using coherence as a criterion——leads to insurmountable difficulties and is in fact the cause of much confusion. I will not explain the confusion here. When we try to explain confusions we run the risk of adding to it. Instead I will simply explain why it cannot be the case that we evaluate results in cryptanalysis by using coherence as a criterion of truth.
1)Suppose I make up a message which is totally incoherent (it is just a random string of alphanumeric characters). I encipher this 'message' using some cipher and ask you to cryptanalyse it. Suppose you come back with a message that makes sense, a message that is perfectly coherent. By the very fact that this is the case; by the very fact that the message you arrive at is coherent; I know you have not found the true message. In this hypothetical case the true message, by design, is incoherent!
2)Consider another hypothetical case. Suppose this time you give me a cryptogram to solve. After you have handed me the cryptogram I stare at it for five seconds. In the next ten seconds I write down a message that makes perfect sense and tell you this is the solution to your cryptogram. Will you accept my verdict?
Clearly no; I have made no effort to solve the cryptogram. I have not gone about my task the proper way. I did not look for clues. I did not develop new clues from old. It is not sufficient that I stare at the cryptogram. It is not sufficient that the message I write down is coherent. It is important that I should follow the right method. If I do not follow the right method, even if the message I write down is coherent, it will still be the wrong message. Method determines whether I have found the right message, not coherence.
3)Suppose I follow the right method, will the message I arrive at always be coherent?
Some will expect me to say no but the proper answer is yes.
But if the proper answer is yes, does it not mean that coherence is the criterion after all?
This time the answer is no.
In order that I do not confuse readers further, let me first show them the following example and then explain why if I follow the right method, the message arrived at will always be coherent.
Suppose I follow the right method and have arrived at the following partial decipherment.
MEET ME AT THE T?P
There is only one letter left to be deciphered, the one represented by the question mark. Now there are at least two ways to decipher this last letter. The question mark could stand for I or O. If the question mark is I we have the message
MEET ME AT THE TIP
If the question mark is O we have the message
MEET ME AT THE TOP
Both messages make sense; they are both coherent. But we cannot have two true messages! This shows us we cannot use coherence as a criterion.
Do we know at this time which of these two messages is the true message?
We do not. We do not have enough evidence to support either solution: the cryptogram is not long enough. Moreover, the true message may not even be one of these two: the last group of letters could be an abbreviation.
Now the question, why is it that if we follow the proper method we will always arrive at a message that makes sense. Some readers, I am sure, know the answer already. A message that does not make sense; a message that is merely a random collection of symbols; even if it should be encrypted by the simplest cipher there is, cannot be cryptanalysed——because there will be no clues. There will be no clues because the message itself has no structure. The right method, the theseological method, asks us to follow clues. When there are no clues, no result can be obtained. For this reason, all messages that we can decipher by following the theseological method are coherent in some sense. They have to be if they are to be decipherable at all.
When we succeed in following the theseological method the results obtained will always be coherent, but this does not mean that in an investigation so long as the result is coherent, it must be correct.
My case is, as I have told you, almost complete; but we must not err on the side of overconfidence. Simple as the case seems now, there may be something deeper underlying it.
—— The Sign of the Four
There is a method by which things hidden can be known, gradually. This method, which we call the theseological method, tells us to follow clues and develop new clues from old. When we are able to develop new clues, it is likely that the old clues have been properly interpreted. The more generations of new clues we are able to develop, the more likely this is the case. In following the theseological method we can become more and more certain of our past results, but we can never be absolutely certain. Now we have spoken about this before, but some readers may still have questions. For example, some will ask, by the time we finish solving our SBR cryptogram, can we not be absolutely certain of our results? Can we not be certain that S = T, B = H, and so on? If they are not what we say they are, what else can they be? We have deciphered all the letters; there is nothing left. We have followed the theseological method; we have not departed from it at any point; and after all the steps we have taken, we have arrived at a message that makes sense. If we cannot be absolutely certain at this point, when can we be absolutely certain? What more can we do to be more certain?
I have to confess it appears rather certain that S = T and B = H; it is hard to imagine what else they can be. However, lack of imagination on our part cannot compel the world to behave in a certain way. There was a time when most people believe the world to be flat; they found it hard to imagine how it could be otherwise; but the world has not become flat because of their lack of imagination.
If we cannot be certain that S = T and B = H, how can we be more certain? What more can we do——other than actually getting hold of the cipher, which we have ruled out? (Even if we can get hold of the cipher, can we really be certain?)
One thing we can do if we want to be more certain is to expand the investigation. This we often do in practice. For example, with our cryptogram we can ask who the sender is, who the recipient, and what kind of shipment they are referring to. We can look for clues in order that we can answer these questions (we may not know where to look for these clues now, but if we have more information we may). If these clues produce results and these results do not require that we alter our decipherment of the cryptogram, we can then be more certain of our decipherment.
The results obtained by following the theseological method can never be absolutely certain. But they can be made more certain by expanding the investigation, or by merging it with others. Scholars and scientists know this, if not explicitly then at least at an intuitive level: they like their successful research projects to lead on to other research projects; they like this more than one that is successful but does not lead anywhere else.
In this chapter I have explained why things hidden can be known. They can be known because there is a method, the theseological method, which tells us to follow clues and develop new clues from old. When old clues lead to new clues the old clues are likely to have been correctly interpreted. The more generations of new clues the old clues lead to, the more likely to be right the interpretation of the old clues. This process, the theseological process, is not simple. If we want to understand it we have to take time to notice its various characteristics, as well as the reason behind them. Many of these characteristics, I have pointed out, are not what we would expect. I have been told by many, for example, that they are surprised to find that in this process there is such a thing as a brick wall. Others have told me that they are intrigued by the fact that in this process we can tell we are moving in the right direction. In the coming chapters I will be adding to this list of interesting characteristics.
1Some people would deny that airplanes can fly even after they have seen airplanes flying. But those who have both seen airplanes flying and understood why they can do so are less likely to engage in the same kind of denial.
2One good example of this from science is the discovery of the theory of evolution by natural selection. Charles Darwin and Alfred Russell Wallace, taking different routes, came to the same discovery independently of each other.
3Following clues can do havoc to our moods and emotions. When we succeed we are elated. When we are trapped and can't find our way out, we are depressed. As we learn from Watson, Sherlock Holmes can be bright, eager and in excellent spirits sometimes, but this often alternates with fits of the deepest depression.
4Most scientists know that the knowledge they uncover is only approximate. Lay people, however, often treat them as gods. This is where the danger lies.
5For an example of an even longer chain of clues, see Sherlock Holmes's summation of the case at the end of The Adventure of the Speckled Band.
6Not likely, but not impossible.
7How many readers of Sherlock Holmes would be interested in knowing what happens to a body when someone drives a harpoon through it?