Wordsmith.org
Posted By: shanks Etaoin shrdlu - 11/29/00 09:52 AM
Ann's postings on typesetting set me off on this. It's more a challenge than anything else, but fun anyway. What is the significance of the title of my post - both to the world of typesetting, and to that of code-breaking!

cheer

the sunshine warrior

Posted By: tsuwm Re: Etaoin shrdlu - 11/29/00 03:57 PM
"With the idea of speeding up..." research, here is Michael Quinion:
http://www.quinion.com/words/weirdwords/ww-eta1.htm

(but he only hints at the code-breaking part...)

Posted By: Bobyoungbalt Re: Etaoin shrdlu - 11/29/00 05:15 PM
It's the frequency distribution of the letters in English words. See Sherlock Holmes in "The Dancing Men". It may be noted that there are other lists with different arrangements.

Posted By: Father Steve Re: Etaoin shrdlu - 11/29/00 06:09 PM

A sophistication of the letter frequency method for cracking a simple code is provided by a knowlede of which digraphs and trigraphs appear most frequently in English, e.g. TH, HE, IN, ER, RE, AN, ON and IN; THE, AND, TIO, ATI, FOR, THA, TER and RES.


Posted By: wow Re: Etaoin shrdlu - 11/29/00 07:36 PM
Ok, OK, I got note from shanks re this subject and sent him a private note declining to be the one to introduce it. He has more courage than I do so he brought it up.
L'audace, l'audace, toujours l'audace is this Board's motto!
I have no idea about the cipher aspect, being hopeless at codes. That is all new and I will try to follow the answers on that. Good luck to me.
I ran into the phrase when my Dad first showed me a Linotype machine. Hey, I'm talking me as a toddler in the 1930s. Yes, I am an oldie.
The letters on the keyboard of a Linotype are not set up the same as a typewriter and the men who did typesetting were incredibly fast on their machines but could not transfer the skill, at speed, to a typewriter.
The Linotype makes lines of type from pigs of lead fed into the machine and heated to melt and reformed into letters and lines, the lines become banks of type that are locked into a form which later becomes part of a page. If a typesetter made a mistake he would make an X on the keyboard (starting in upper left) and the type would set as ETAOIN SHRDLU. - (My favorite swear word) - Typesetters' eyes could pick out the X'd line at a glance and remove both it and the line before it where the mistake was made. So there you go, that's my explanation.
As an aside : when visitors came through the typesetting area the operators would often set the persons name in type for them which the typesetter handed to the visitor. The men's hands were toughened to the heat but visitors got a gift of HOT type and the typesetter got a chuckle and a "gotcha" for slowing the work.
Linotype operators (typesetters) belong to one of the oldest unions. Newspapers were among the first businesses to be "automated." That's another thread for a different board.
If any of you have a newspaper near you that still uses one of the great HOE presses, ask for a tour. You'll learn a bit about how type is made, how the technique was apllied to computers, and you'll have a great time.
That's it. I am off the printing, typesetting thread for good. Why did I ever start with it? Not another word. No explanations for type lice or buckets of steam or left-handed quion keys, or chocolate kerns, or a 10 point minion, or a woolen offset blanket. And if shanks wants to go off on horseback and also intoduce the keyboards used by signalmen in WWII well, don't expect a word from me. (I heard that!)
Blessings on you all. wow

Posted By: wow Re: Etaoin shrdlu - 11/29/00 07:41 PM
This is really my last word...I read the link and I admit my mistake....it was not an X, rather the top lines of the keyboard. I was a five-year-old for heaven's sake!
wow

Posted By: shanks Re: Etaoin shrdlu - 11/30/00 09:04 AM
And is probably the only way to crack codes of the Playfair type - with doublets being transposed?

Posted By: shanks Re: Etaoin shrdlu - 11/30/00 09:08 AM
Thanks for that Ann.

The point being (just stating the obvious here) that they were on the top for the logical reason that they are the most frequently used letters - as most of the code-breakers below have appreciated. If we hadn't the historical accident of qwerty keyboards, we might well be using etaoin keyboards these days...

Posted By: FishonaBike Re: Etaoin shrdlu - 11/30/00 01:32 PM
codes of the Playfair type

Sounds like a merger between Playboy and Mayfair!

Reference or explanation please, shanks.


Posted By: wow Re: Ciphers/codes - 11/30/00 02:17 PM
May I recommend a book to those who love ciphers/codes/good story ? "From SIlk To Cyanide" by Marks. Published within the last year in US. As noted I am hopeless with codes but this is also a whacking good tale, and true. The writer is the son of the owner-founder of Marks & Co of London, a bookstore that is featured in the delightful tale "34 Charing Cross Road." The story is set in England in WWII years where he was in the business of helping spies. Anyone who has done some reading on WWII will recognize the names of many spies who became famous after the war when their work and sacrifice were finally recognized. Happy reading, wow

Posted By: Bobyoungbalt Re: Etaoin shrdlu - 11/30/00 03:11 PM
In reply to:

codes of the Playfair type



A Playfair cipher (it's not really a code)substitutes letters in pairs from a simple grid. Dorothy Sayers, who was fond of such problems, gave a detailed description of the cipher and how to crack it when she has Lord Peter and Harriet discover and solve one in Have His Carcass.

Posted By: TEd Remington Re: Etaoin shrdlu - 11/30/00 08:07 PM
WOW:

What memories you brought back. My parents were both journalists, one a reporter and the other an editor. My mother was a police and courts reporter for the Alexandria, Virginia Gazette, which boasted that it was the oldest continuous daily newspaper in the US. Now defunct, of course, much to my dismay.

I still have my line of type with my name on it that I received on a visit to the press room not quite 50 years ago. And I definitely remember the strange keyboards and the even stranger men who worked at them. They all smoked unfiltered Camels which they lit by touching the tip to the bucket of hot lead which fed the machine its raw materials.

In fact, they still set some type by hand for display ads when I first started hanging around. I too wish I had known now much the cases would be worth, because there were stacks of them, many still containing their fonts, gathering dust in a shed behind the building.

Those were glorious times, and I suspect the world is poorer for the changes that have come to the newspaper world.

Posted By: xara Re: Etaoin shrdlu - 12/01/00 06:42 AM
historical accident of qwerty keyboards

Not an accident at all. They made the qwerty so that the common patterns of keystrokes would be far apart so that the typewriter wouldn't get tangled up as easily.

it's a conspiracy, i tell you

p.s. why does AEnigma interpret qwerty as rabbit?

It's time for the human race to enter the solar system.
--Governor George W. Bush Jr.
Posted By: jmh Re: Etaoin shrdlu - 12/01/00 08:19 AM
>They all smoked unfiltered Camels which they lit by touching the tip to the bucket of hot lead

Yes, those Health & Safety people spoil all the fun these days, don't they? I remember the days when chemistry lessons had just the right number of explosions per lesson to keep everyone awake, they stopped those too.

Posted By: shanks Re: Etaoin shrdlu - 12/01/00 09:38 AM
historical accident of qwerty keyboards

Not an accident at all. They made the qwerty so that the common patterns of keystrokes would be far apart so that the typewriter wouldn't get tangled up as easily.


True enough, xara, but one presumes that there were a number of ways in which this could have been done, so the particular arrangement we have is, to that extent, a matter of chance. More importantly, however, there was the opportunity to change to a more rational scheme (a simple alphabetic one, if I remember rightly) that we missed because the winner of a touch-typing competition around 120 years ago happened to be a man who used a qwerty keyboard. As a result, qwerty became established (for speed! not for slowness!), and is now ubiquitous (except, as we can see, amongst linotype operators).



Posted By: FishonaBike Doubting Thomas - 12/01/00 10:28 AM
they are the most frequently used letters - as most of the code-breakers below have appreciated


Now, hold it here a second, folks. It just occurs to me that I've always blithely accepted statements about "the most frequently used letters" and, very closely related, "the most commonly used words" in the English language.

How are these "facts" known??

OK, in a particular context for a certain period of time you can make a broad-sweep judgement. Also it's a lot easier these days to produce accurate mathematical counts for a number of documents, and then extrapolate from there.

But but but.. like all statistics, we don't have any absolutes. In many contexts, for instance, articles will be omitted, words abbreviated and slang introduced. Surely this makes a significant difference to word and letter counts. And another thing - language evolves over time. New words and phrases appear, and old ones fade away. Sometimes (as we know) this happens very quickly.

Codebreakers would certainly need to take account of such factors, as top secret messages are only going to say what has to be said, in as condensed a form as possible. Context is absolutely crucial, as that will dictate the most frequent phrases/words/abbreviations.

I'd therefore like to ask:
Who came out with these frequency figures, how were they determined, and when?

Nice simple question!





Posted By: jmh Re: Doubting Thomas - 12/01/00 11:05 AM
So would you go with Dvorak then?

Posted By: TEd Remington Dvorak - 12/01/00 12:11 PM
No, thanks. I prefer Sibelius myself :)

Posted By: FishonaBike Re: Doubting Thomas - 12/01/00 01:36 PM
So would you go with Dvorak then?

Nah, I'll just go with the flow, Jo!

Zen Fish
(Zen bones)
(Zen - dry bones..)



Posted By: shanks Can't give chapter and verse but... - 12/01/00 02:04 PM
... Simon Singh's The Code Book is a brilliant cyphers-for-amateurs work (and there's a large-ish prize at the end, if it hasn't been claimed yet).

1. Letter frequency: Usually, I believe worked out from various corpuses, of a million words or more each. Simon Singh gives various frequency details from different corpuses. In any large enough body of words in English we find a surprisingly similar set of frequencies. Etaoin shrdlu may not be an absolute representation of relative frequencies, but it is a very useful one for a codebreaker (yes I know it's technically called a cypher, but if Simon Singh can call it code, I can too).

2. Letter frequency is not used in any absolute sense when codebreaking (ok, ok, deciphering). See, for instance, the classic story - Sherlock Holmes' "Adventure of the dancing men", to see how letter frequency combines with intuition and good ol' fashioned guesswork, to make decipherment possible. All codebreakers rely upon getting their hands on a reasonably susbtantial portion of text, and for whatever reason, I believe that type of use (telegraphic, legal, medical etc) does not skew the frequency distribution too much.

3. The technique was developed by (who else?) our old friends the Arabs (Saracens?), and the word cypher (for another cross-thread) comes from the Arabic (sifr?). In the tenth or eleventh centuries, the agents of the Caliph and others, determined to crack letter-substitution codes, slowly built up their knowledge of letter frequencies in Arabic. Whether this technique was slowly disseminated through Europe, or re-invented later, I don't know, but it has stood codebreakers in good stead for centuries. As Simon Singh points out - Mary Queen of Scots lost her head because of the efforts of a codebreaker using letter frequency tables to help him decipher her secret messages plotting against Lizzie 1.

cheer

the sunshine warrior

Posted By: wow Re: cipher problems - 12/01/00 02:29 PM
Dear FOB, Indeed, transpositions and bad spelling often caused problems for code-breakers especially with the spies sending messages under the duress of combat conditions encountered in WWII. Again, I plug "From Silk To Cyanide" where this discussed in depth with personal experience by author Marks (who was a mere stripling at the time.). ALERT: this is not yet another book about the Enigma machine at Bletchley (sp?) but a different, English, operation. The code-breakers Marks worked with were all women, for one thing! 'Nuff said.
wow

Posted By: tsuwm Re: cipher problems - 12/01/00 02:48 PM
>The code-breakers Marks worked with were all women, for one thing!

...as were many at Bletchley. http://www.secretsofwar.com/html/bletchley_park.html

Posted By: tsuwm Re: Etaoin shrdlu - 12/01/00 03:53 PM
coincidentally, today's M-W word-of-the-day is cryptography. is somebody spying on us?!

Posted By: Jazzoctopus Re: Etaoin shrdlu - 12/01/00 07:38 PM
p.s. why does AEnigma interpret qwerty as rabbit?

Apparently, AEnigma has an alphabetical list of all the words that it thinks are acceptable. If it finds a word that it doesn't recognize it just goes to the next word on the list. (This board isn't written in the most complex of computer languages, so the spell check was obviously kept simple.)

Posted By: jmh Re: rabbit - 12/01/00 10:20 PM
>to the next word on the list

In which case, the first "r" word must be rabbit. Or maybe Aenigma is recognising that it is the first day of the month.

Assuming the first version is true - Can anyone, without resorting to a dictionary, on-line or otherwise help Aenigma out with the missing words?

Posted By: tsuwm Re: rabbit - 12/01/00 10:39 PM
if you did resort to something like the OED there are probably lots, but the only significant ones I can think of are Ra and rabbi (plus rabbinical etc.)

Posted By: Jazzoctopus Re: rabbit - 12/01/00 10:43 PM
Assuming the first version is true - Can anyone, without resorting to a dictionary, on-line or otherwise help Aenigma out with the missing words?

There is no way for us as board users to add more words to Aenigma's lexicon. The only people that could do that are those who wrote the code for the board's amazing spell check feature.

Posted By: jmh Re: rabbit - 12/01/00 10:45 PM
>There is no way for us as board users to add more words to Aenigma's lexicon

I wasn't thinking of adding them, just wondering what Aenigma was missing out on. Sounds like rabbi postings are a bit of a no no.

Posted By: Capital Kiwi Re: Etaoin shrdlu - 12/02/00 01:50 PM
Actually, etaoin shrdlu is simply the order of the letters on the first two vertical rows on the left hand side of the linotype keyboard. The reason that it became so well known is also simple - bad proofreading. Often, if a linotype operator made a typing mistake, rather than being bothered to fix it s/he would simply run a finger down the first two rows of the keyboard to justify the line, with a couple of spacebands to take up the slack, then cast the line anyway. If the operator didn't diss it or the proofreaders didn't spot it, it wound up in print.

So yes, it was a cipher - but for "I screwed up", not anything more sinister!

Posted By: FishonaBike Re: Etaoin shrdlu - 12/02/00 09:01 PM
is somebody spying on us?!

"Just because you're paranaoid doesn't mean they aren't out to get you."





Posted By: shanks Re: Etaoin shrdlu - 12/04/00 12:29 PM
Actually, etaoin shrdlu is simply the order of the letters on the first two vertical rows on the left hand side of the linotype keyboard. [snip] So yes, it was a cipher - but for "I screwed up", not anything more sinister!

My point was that the reason for thie being in that position was that they tend to be (in that order) the commonest letters in the English language. This knowledge (of letter frequency) is invaluable when attempting to decipher letter-substitution methods of coding...

© Wordsmith.org