It’s unbelievable today, but there was a time when the government classed crypto as a munition and made it illegal for anyone to export or use it on national security grounds. Get that? We used to have illegal math in this country.

The National Security Agency were the real movers behind the ban. They had a crypto standard that they said was strong enough for bankers and their customers to use, but not so strong that the mafia would be able to keep its books secret from them. The standard, DES-56, was said to be practically unbreakable. Then one of EFF’s millionaire co-founders built a $250,000 DES-56 cracker that could break the cipher in two hours.

Still the NSA argued that it should be able to keep American citizens from possessing secrets it couldn’t pry into. Then EFF dealt its death-blow. In 1995, they represented a Berkeley mathematics grad student called Dan Bernstein in court. Bernstein had written a crypto tutorial that contained computer code that could be used to make a cipher stronger than DES-56. Millions of times stronger. As far as the NSA was concerned, that made his article into a weapon, and therefore unpublishable.

Well, it may be hard to get a judge to understand crypto and what it means, but it turned out that the average Appeals Court judge isn’t real enthusiastic about telling grad students what kind of articles they’re allowed to write. The crypto wars ended with a victory for the good guys when the 9th Circuit Appellate Division Court ruled that code was a form of expression protected under the First Amendment — “Congress shall make no law abridging the freedom of speech.” If you’ve ever bought something on the Internet, or sent a secret message, or checked your bank-balance, you used crypto that EFF legalized. Good thing, too: the NSA just isn’t that smart. Anything they know how to crack, you can be sure that terrorists and mobsters can get around too.

From the amazing (and edifying!) Little Brother, by Cory Doctorow. But no, this little burst of text isn’t about this (scary) modern day dystopian novel, or the EFF, or crypto, or security. It’s about a 1401 digit prime number that showed up in my feeds, via Everything2, a mystery in a puzzle in a mystery. It’s about Unix, and about peeling back layers. This prime number is illegal. I’m going to paste it anyway, then tell you why:

48565078965739782930984189469428613770744208735135792401965207366869
85134010472374469687974399261175109737777010274475280490588313840375
49709987909653955227011712157025974666993240226834596619606034851742
49773584685188556745702571254749996482194184655710084119086259716947
97079915200486670997592359606132072597379799361886063169144735883002
45336972781813914797955513399949394882899846917836100182597890103160
19618350343448956870538452085380458424156548248893338047475871128339
59896852232544608408971119771276941207958624405471613210050064598201
76961771809478113622002723448272249323259547234688002927776497906148
12984042834572014634896854716908235473783566197218622496943162271666
39390554302415647329248552489912257394665486271404821171381243882177
17602984125524464744505583462814488335631902725319590439283873764073
91689125792405501562088978716337599910788708490815909754801928576845
19885963053238234905580920329996032344711407760198471635311617130785
76084862236370283570104961259568184678596533310077017991614674472549
27283348691600064758591746278121269007351830924153010630289329566584
36620008004767789679843820907976198594936463093805863367214696959750
27968771205724996666980561453382074120315933770309949152746918356593
76210222006812679827344576093802030447912277498091795593838712100058
87666892584487004707725524970604446521271304043211826101035911864766
62963858495087448497373476861420880529443

What is it for?

When written in base 16 (hexadecimal), this 1401 digit prime number found by Phil Carmody forms a gzip file of the C-source code that decrypts the DVD Movie encryption scheme (DeCSS). This prime number is illegal in every country that the DMCA applies.

Wow.

If you’re like me, you’re firing up a terminal right now to unfurl the layers one after another, driven by burning curiosity to see what the seed contains. If you’re not… well, aren’t you a little curious? Of course your are.

Let’s get on a Unix shell and start peeling.

Getting the number

If you select all of the digits in the number above using your mouse, you’ll pull in a lot of newline (and/or carriage return) characters, even if you try copying the HTML source. If you’re going to deal with HTML source anyway, let’s make this easy on ourselves.

I read about it on Everything2, and you can use this URL to access the number from there, or you could use the URL to this page itself.

Now we pull down the HTML so we can easily parse the number:

$ curl -s http://rightshift.info/?p=463

That should spew out the HTML containing this number. If ‘curl’ is not your thing, try wget to save to a file, or use the browser itself, and replace the above by ‘cat your filename’.

Next, let’s find this block of text. It’s preformatted with either a <blockquote> or <pre> tag, so we use that in a regular expression. There may be more than one preformatted section in the page, so we use a dirty hack in the pattern matching: Let’s put in the first few digits of the number in the pattern so the other blockquotes don’t match.

$ curl -s http://rightshift.info/?p=463 | awk '/(<pre>|<blockquote>).*4856.+(<\/pre>|<\/blockquote>)/ {print}'

We could have used grep instead of awk, but we’ll get to that in a moment. If the command works, you should see the illegal prime on screen with a bunch of <br> and other tags mixed in. Let’s get rid of those, shall we?

$ curl -s http://rightshift.info/?p=463 | awk '/(<pre>|<blockquote>).*4856.+(<\/pre>|<\/blockquote>)/ {gsub(/[^[:digit:]]/,""); print}'

The gsub performs a global substitution on any character that is not a digit , /[^[:digit:]]/. This should print just the number on the screen. Things will get very unwieldy soon, so we’ll store this number (actually a string) into a variable. In Bash, the only way to do this is:

$ prime=$(curl -s http://rightshift.info/?p=463 | awk '/(<pre>|<blockquote>).*4856.+(<\/pre>|<\/blockquote>)/ \
{gsub(/[^[:digit:]]/,""); print}')

Note the ‘\’ followed by a newline, splitting the command onto two lines. If you were typing this, you don’t need to include either.

Converting it to hex

To see $prime in hex, we use bc, the handy command line calculator:

$ echo "obase=16; $prime" | bc

This should print out the number in Hex. Here are the first few bytes:

1F8B0808196C9A3A00036373732D6465736372616D626C652E6300ED56CD8E9B3010\
BEF314F412E140241B304B95984B1F618F164859205D2B4BB64AA65B56BB79F70EE1\
2736F1A1552F6DB51...

Oops. bc considered it appropriate to put end of line markers, \ splitting the text across multiple lines. Let’s get rid of them, using tr, the text replacement utility:

$ echo "obase=16; $prime" | bc | tr -d '\\\n'

There are three \’s in the text we’re deleting because one of them’s part of ‘\n’, one of them is a literal backslash that’s escaped by the shell, and the third one’s there to keep the second one from being escaped. Phew.

As a check, here’s what the gzip specification says about the header to any gzip file:

The first two bytes have the fixed values ID1 = 31 (0×1f, 037), ID2 = 139 (0×8b, 213), to identify the file as being in gzip format.

1F8B, indeed. We’re almost there.

Dumping it into a binary file

The above stream can’t be unzipped! This sure bewildered for a few minutes. I suppose this is what happens when your mental model of the computer’s storage structure is as badly broken as mine.

Of course it can’t be unzipped. It’s an ascii stream of hex characters; still an ascii stream! We need to dump the above hex code into a binary file.

How do we do that? Sure, you can write some code in C to do it. (Scan as hex and use putchar().)
But you know what they say about Unix: If you need to do it, it’s been done. Hmm. Let’s search for a bit:

$ apropos --and hex dump
hd (1)               - ASCII, decimal, hexadecimal, octal dump
hexdump (1)          - ASCII, decimal, hexadecimal, octal dump
xxd (1)              - make a hexdump or do the reverse.

Slowly I realize how true this adage is. Enter xxd. Some thinking reveals that we need to do a reverse hex dump. A hex dump is where the contents of a binary file are printed out as a stream of ascii-encoded hex characters, we’re trying to do the opposite. After reading the man page, we construct the reverse hex dump:

$ echo "obase=16; $prime" | bc | tr -d '\\\n' | xxd -r -p

If all goes well, we shoud see a stream of unprintable characters (no, not that type) desperately trying to print themselves on the screen.

On to the source

The rest is trivial. We have the gzipped stream pouring through ‘xxd‘, so let’s redirect that:

$ echo "obase=16; $prime" | bc | tr -d '\\\n' | xxd -r -p | zcat

And that’s it. If ‘zcat‘ doesn’t work, for some reason, try ‘gunzip -c‘.

The illegal source code hidden in the illegal prime should be floating on your screen now.

What, you thought I’d put it here? It’s illegal.

Probably more illegal than printing the number, anyway. If you skipped to the end hoping to glance at the code, here’s an unreadable composite command for you, instead. (Again, note the line breaking ‘\’ after awk’s first argument, added to keep this webpage from breaking.) Hide your eyes, they who fear line noise:

$ curl -s http://rightshift.info/?p=463 | awk '/(<pre>|<blockquote>).*4856.*(<\/pre>|<\/blockquote>)/ \
{gsub(/[^[:digit:]]/,""); print "obase=16;" $0}' | bc | tr -d '\\\n' | xxd -r -p | zcat

Or, you know, you could just read the code on Wikipedia. I didn’t. I don’t know anything about crypto, but I expected a certain kind of joy in deciphering this simple code, something I could actually hope to do. I wasn’t disappointed.

The wormhole

Now, the DMCA isn’t at odds with the opening excerpt: It’s all right to write code that does absolutely anything, and it may be all right to distribute it, but it’s illegal (by the DMCA) to use it to break copy-protection measures.

Except for one little thing, though. There is still illegal math.

A request, then: Someone explain that C source to me!

^{Further reading:}

• Little Brother, as described by Cory Doctorow: “This book is meant to be part of the conversation about what an information society means: does it mean total control, or unheard-of liberty? It’s not just a noun, it’s a verb, it’s something you do.”
  This book is licensed under Creative Commons, and is free to download. It’s also a cracker of a novel.

• Regular expressions are awesome. I cannot stress this enough. If you want to see a whimsical but fantastic use of regexes, try this.

• Phil Carmody’s Titanic Primes. See for yourself.

• The gzip specification: Just to see what goes into making a standard. Why did they pick 1F8B as the gzip header, for instance?

• The Unix utilities awk, bc, tr and curl. The entire GNU coreutils pack is awesome, more functionality than you can ever need in a terse, powerful set of commands.

• Hex Dump Code Golf, a game played by the Stack Overflow community where the objective is to write a reverse hex dump in the language of your choice in the fewest chars possible. Perl won.

So my parents bought a new cot made of a variety of jungle hardwood that has proven notoriously difficult to identify. A few months into the purchase, strange periodic noises emanating from within the cot began to torment their sleeping hours. It sounded like tiny creatures constantly scrapping away at the woodwork from within. No puncture marks could be detected on the cot surfaces. After a couple of months of tolerance testing, we called in the local carpenter for a check-up. He theorised that the invaders were a wood-boring species that most likely are restricted to the easily accessible soft-outer/younger parts of the wood in the cot’s central beam. So the bed was overturned and he began hacking away at it. 12 minutes later, we came away with a dozen of these.

watch?v=DhNgbjYmiqs

I managed to capture about 3 live worms and stash them in a plastic container. It was time for the analysis to begin!!!!

First Impressions

At first glance, I couldn’t help but draw parallels to the sandworms of Dune (refer: Frank Herbert’s Fiction Writing).
These little buggers were about an inch long, white, articulated and lacking in endo/exo-skeletons. Their movement seems based on hydrostatic pressure transfer through a fluid base. They were blind, had no limbs, or any other phenotype besides the mouth. They kept obsessively writhing about and munching their “jaws” on air, as if desperately ruminating.
Then it occured to me, that these hatched out of eggs laid in the timber long before the tree was cut down to make us that cot. If they are to spend a significant part of their lives in this form, buried deep within wood, then they wouldn’t need any other senses. There’s no light, and hence no need for sight. Food is plentiful, so no need for smell sensing. They are likely to bite down on wood no matter where they turn and chomp. But once exposed in this manner, suddenly, they look like darwinian losers. Genetic mistakes that can’t find delicious splinters of wood placed a centimetre away, much less navigate to it. I had to identify the species, and so I went hunting on the internets. The closest match I could find is this:

Forest Longhorn. And to think Microsoft nearly named an operating system after it. So, I could expect to watch these grow into some beetle-esque abominations. Groovy!

A little further research lead me to all sorts of woodworm species descriptions. This larval stage is meant to last months, sometimes years. And in all that duration, these organless wormies feed on dry wood. They survive on moisture in dry wood!!
That’s probably not that startling to anyone who is familiar with silverfishes surviving on paper from old books in the attic.
So at this stage, much like silkworms eating leaves, their purpose in life is to feed and accumulate enough nutrients for the transformation to occur. The worms were actually very inactive for the most part. Its not like they ingest much energy to be capable of anything other than grub.

Power Extreme

Over the course of three months, they started to die out on me. The lone survivor (a.k.a. Kwizat) had suddenly seized to feed and was shrinking in size and had grown restless, wiggling as if in a struggle. Sick with a stomach-ache perhaps. I thought his number was up.

But then, one day, I woke up to something really strange:

Not only had he grown larger, into some low-density shape, but he appeared to be growing legs and a head, with antenna and everything.

watch?v=-67hhrE1R2U

He’d wiggle in an attempt to shed his old skin.
It was time to get him out in the open.

What followed was a miraculous transformation. It was a delight, specifically because ideally, this was supposed to occur deep within wood, away from voyeuristic eyes and lenses. They say that a butterfly transformation, if interrupted by damage to the pupa, will forever be incomplete, resulting in a failed half-breed creature. But this thing was changing in the open…. How I wished I had mastered time-lapse.

The legs kept getting thicker. The eyes, darker to the point where he’d wiggle when I turned the lights on early morn. The head and mouth became more detailed, and the signature long-horn antenna curled in anticipation. Twenty days later, he was unrecognisable.
He was beginning to show some activity, as if waking from a slumber. He was rolling his own discarded skin into a ball. I wonder if it was programmed instinct. Maybe in more natural conditions, he would have some use for it (nutrition?). Also notable was the wing on his back (presumably defunct, a relic of his genetic ancestry).

watch?v=_NrGaJw2Grs

And within ten days, the transformation was complete:

I decided to release the fellow. I think they reproduce asexually, though I’m not sure if he was capable of laying eggs into polished, seasoned funiture wood in the locality. In any case, I hope its not an invasive species. He’s probably bird-food by now.

There are presumably many clever, interesting and useful tools and ideas in science and math modeling. The concept of scaling, though, is the most powerful one I’ve seen in that it makes very nontrivial predictions with just small helpings of information.

Let’s start at the top. To scale an object means to multiply its every linear dimension by the same factor. It’s what you would expect: a geometric shrinking or enlarging, like under a microscope¹.

Now, attributes of this object scale in different ways. If every linear dimension of the object is multiplied by a factor n, then the surface area and volume of the object go up by factors of and respectively. Depending on how we physically accomplish the scaling, either the mass of the object scales as or its density scales as . (Stretching the original object preserves the mass but lowers the density by the above factor; creating a scaled-up replica retains the density but increases the mass by the above factor.)²

Forces scale in different ways:

1. Let’s say you have two hot metal plates identical in shape with a ratio of surface areas of . The force of gravity on the larger of the scaled objects is times that on the smaller one.

2. If the two metal plates were tenderly rested on the surface of water, the force of surface tension on the larger one would be n times that on the smaller one, because surface tension is proportional to the length in contact with the liquid. This explains why you can’t float a steel knitting rod on the surface of water, but you can float a steel pin. The surface tension holding the steel pin up is smaller by a factor of about 10, but the pin is lighter by a factor of about 1000.

3. The force you need to break the larger plate by tugging along its length is times that needed to break the smaller one in the same way.

4. If you were to pull these plates through molasses, the forces of drag on the plates due to the molasses do not scale in an elementary way- but to a first approximation, the drag force is proportional to the surface area. This means the drag is higher on the larger plate by a factor of (approximately) .

And rates of processes scale too. The bigger one would lose heat that much faster. This does not mean that it cools faster! If they are at the same temperature to begin with, the larger plate would lose heat at a rate higher than the smaller one, because this rate is proportional to its surface area. The amount of heat it needs to lose for its temperature to go down by a degree, however, is times higher than the amount the smaller plate does, because this quantity is proportional to its mass. This means it takes the larger plate about times longer to cool down by the same degree.

The above scaling relations hold because of the simple physics that applies. Actually, when the physics gets complicated, scaling rules still hold; it’s just that geometric similarity is not a fruitful pursuit anymore. There are more abstract ways of using scales to work with the system- a bit more on this towards the end.

This is where things get really interesting. Everything, including living things, are subject to the above rules, and we can make some remarkable observations about the design of insects and animals:

The ant is known to carry several times its own weight. Why can’t we do this?

First, let’s see how an ant would fare if it were about the size of a human. Much of the ant is strong tubes filled with mushy insides. (Insects have exoskeletons, like the ants’ chitin.) If you were to breed a six foot tall ant, the weight of the ant would scale as , and the load bearing cross-section of the chitin in its legs would go up by a factor of .

This means the stress in its legs would go up by a factor of . The super-ant would break its legs just by standing!³

How about load-bearing capacity? The forces its muscles could exert would scale as the square of the linear dimension. (The force a muscle can exert is proportional to its cross sectional area). In this case, the square of (2 m/ 2 mm), which is about . A regular ant can lift at least twenty times its own weight; if the super-ant were to try this, the stress on its muscles would be times that of the regular ant.⁴ Its muscles would rip.

Not so super now, eh?

Conversely, it follows that since you can stand on our legs without collapsing, you will be at least as strong as the ant when you’re shrunk to its size. You can do the calculations; the general idea is that the capacity of a muscle-driven creature to bear it’s own body weight (or any multiple of it) goes as 1/length of the creature.

The scaling rules tell us that everything will be different when we’re shrunk by a factor of a thousand. From the example of the floating steel pin above, it follows that you should be able to walk on water, so long as it’s relatively calm. You don’t need a pitcher to carry around water; you can hold it in your hand like you would a football. It would take no effort at all to drink it- the super-powerful capillary effect would force it down your throat if you brought your mouth close to it. And yeah, you can hold your own against insects. You would be the iron man (or woman) of the insect world.

It’s not all rosy, though. From the cooling example above, it follows that your body would cool a factor of 1000 faster. (Surface area/Body mass = ) You would probably die of hypothermia unless you were consuming food all the time. Not only can you lift many times your body weight, you’d have to eat many times your body weight in a day!⁵

On the other hand, if you were scaled up by a factor of 80 (say) you would suffer the same fate as the super-ant. Your legs can only bear a load 5-6 times more than they already do when you run or jump- and the stress in your bones would be a factor of 80 higher even when mega-you were just standing still! Your knees would buckle and shatter, smashing every bone in your body as you hit the ground. Your muscles will have torn, and the hydrostatic pressure of the blood in your body acting over a height of 100 m would have forced it down your arteries before rupturing them and causing hemorrhaging. For the same reasons whales do, however, you would survive rather well in the water!

While we’re being squeamish:

As J.B.S. Haldane put it in his classic essay, “On Being the Right Size,” “You can drop a mouse down a thousand-yard mine shaft; and, on arriving on the bottom, it gets a slight shock and walks away….A rat is killed, a man broken, a horse splashes.” Haldane was being quite literal.

These facts were known to our ancestors, who used this aspect of scaling to gruesome effect—a common strategy during medieval sieges was to take a carcass of a horse, let it ripen for a few days in the sun, and then catapult it over the walls of the besieged town. On impact, the carcass would indeed splash, spreading contagion throughout the city.⁶

Wow!

The above is an extract from a wonderful article about scaling in biology that goes into much greater detail of hypothetical changes of scale- highly recommended!

This short foray into design in biology tells us two things: Evolution is, like physics, imagination in a straightjacket. Worse, it’s a straightjacket within other straightjackets, with additional constraints and intense competition to boot. Nature is the mother of all optimization engines! Second, biological design is very parsimonious; the body of an ant is strong enough to carry the crazy amounts of food its colony needs, but no stronger than it need be.⁷

There is so much scope for some hard science fiction here. In fact, a consideration of the physics of scaling is one of the things distinguishing hard and soft science fiction.⁸

The above was just one window into the concept of scaling- we saw how the properties of real world objects varied in interesting ways at different sizes. A different set of observations can be made by looking at the same object at different scales: it leads us to all kinds of interesting ideas, like fractals and universality. Perhaps the most amazing use of scaling is a technique that involves looking not at an object (real-world or abstract), but at a mathematical model for a system at different scales. This process, called the renormalization group, is touted as the most impressive use of abstraction in science. It is quite hard to segue into these other uses of scaling here; they’re fodder for Scaling II!

Footnotes:

1. Strictly speaking, a traditional microscope does not scale an object geometrically, because of the nonuniform optical properties of the lenses. I don’t know of any system of lenses that magnifies correctly along the viewer’s line of sight- the depth- of the inspected object.

2. Of course, the above are special cases. You can vary the mass (and thus the density) by whatever factor you want so long as you’re scaling it up in your head. We’re talking about magnifying the size while retaining as much of the material’s inner structure as possible: so if we wanted to scale up a clay pyramid, we would use the same kind of clay to construct a bigger pyramid, and its mass would scale like its volume does, as the cube of the linear dimension.

3. If you are trying to imagine this, pick up a thin tube of plastic- or a straw, and push on both ends of it. It kinks and buckles; this is the usual mode of failure for a slender beam or tube, like the ant’s chitinous appendages.

4. This is essentially the same calculation as in the paragraph before it.

5. This tells us something apparently paradoxical: The larger you are, the lesser you ought to eat! While this sounds like good health advice, it is of course an incomplete picture. We don’t burn all our energy as heat, we spend a large chunk of it in locomotion, a lot in circulating blood and nutrients around (and in other body maintenance tasks) and a fair bit in thinking.

6. Talk about biological warfare!

7. Because of the way natural selection works, it’s not really surprising that this is the case. If the ant couldn’t carry several times its own weight, there would be no ant colonies and no ants as we know them, right? It’s the antropic principle at work.

8. The other physics most works of sci-fi violate are, of course, conservation laws, like that of momentum and energy. There are workarounds, mind you. (PDF)

(Ahem) Even economics majors reacted with a face-palm (>>).

Humour aside, what really got me into writing about something this clichéd and media-hijacked is humanity’s blatant amnesia at display everyday on TV and in the papers. Granted that it is important not to get carried away by all the religious hype being generated. Even informed heresy is acceptable. But why are vested interest groups resorting to childish tactics?? And why is nobody able to see through them?

At this rate, Youtube will become the sole source of accessible sanity.

KVM had an interesting take on apocalyptic prophecies:

Gloom and Doom is an infallible stance, because of survivorship bias. Either people are vacationing in the Bahamas not caring about your wrong prediction, or are marveling at your foresight

But I couldn’t resist wondering at how conservatism has had its place throughout human history, and “progress” has always been associated with doom. To quote from Thief, my favourite work of fiction yet:

The world as I once knew it was a place of magic - full of mystery and
inhabited by creatures of glamour and terror. The men who lived there lit their
bonfires and wondered at what crept and lurked in the darkness outside their weak
circles of light. All their dreams, their aspirations and dreads,
come from that darkness.
Now, as the forces of "progress" cover the meadows in brick and cobblestone, as they
replace the majestic loft of tree with the blocky ponderousness of building, they also
light the world in their electric, actinic glare. With the lighting of the shadows, man
loses his ability to fear, and to dream.

The night, once the font of the unknown,
becomes only the lack of the sun.

Pagans have a way of getting things right. And stranger still, the “liberals” are fighting for a “conservative” approach, while the “conservatives” are crying foul. I love this language.

Maybe it IS a bad thing to romanticise this issue, serious though it may be. I have always believed that trying to solve this issue by balancing super-aggregated national numbers is really a case of looking at the wrong level of abstraction. This is largely a technical problem, and deserves a technical answer. Very little will come off of policy making and pledges by people who understand little. Nonetheless, it’ll be amusing to see how the leaders of men react to entire (island) nations literally disappear off the face of Midgard. The tragedy is that most people crazy enough to want to save the world from itself don’t take up technical studies, but descend into the largely ineffective field of activism.

Stands He then in the greens and festered Maw and speeds He out his judgements upon the weeps and writhing manfools!

- The Trickster of Legend

I’m still amazed and can’t quite fully digest the fact that a perfectly smooth function can be non-analytic. I mean, consider the exp(-1/x) function. If you stand at the origin, you have no clue what happens as you step forward! All the meters in your car will read zero at the origin, and yet, somehow the function bootstraps itself into rising! This never happens with analytic functions!

The function he was talking about is not exp(-1/x) per se, but

He goes on to plot it as so:

At first sight, defining a function differently for negative and positive x, and then expecting civilised behaviour felt like cheating to me. After all, exp(-1/x) is singular as . But that doesn’t make his comment any less rich, as one is lead to believe that continuity is more sacred than that. While this post doesn’t solve anything, it notices some interesting patterns and compiles them all in one place, and provides for a sweet link next time the reader is having to explain this to someone over the internet. So lets see how far we can get.

The Power Series Expansion:

I’ll confine myself to basic high-school math, so the learning curve is analytic for all readers.
For the uninitiated (I love that word), any function that is said to be “analytic” (we’ll get to that before this post ends), can be expanded as a power series in , about a point like so:

Where , the derivative of the function at , scaled down by n factorial. Since the same function can be translated along the x-axis, we’ll stick to taylor expansions about the origin for all our derivations.
This has an interesting consequence. A general single-valued function with no strings attached, in order to be completely defined, will require us to specify the value of the function at an uncountably infinite number of points on the x-axis, no matter how small the domain (as long as it is not a collection of isolated points). However, analytical functions require only that we specify one set of countably infinite values (all derivatives at any one point within every simply connected part of the domain). This is a drastic reduction. No doubt, the compression of parameter space is stronger than what that mere continuity condition imposes. Apparently, for every function representable by a power series, uniformity of convergence ensures integrability and differentiability. If you’ve understood that, than stop reading further and do more productive things with your life.

Convergence and Analyticity:

Let us examine the importance of the word domain with a few basic examples. Now, the KVM function involves stitching two separate domain definitions, so we’ll start with simpler functions. Take for example, the infinite sum of the following geometric series:

The right-hand side is the taylor expansion about the origin, where the derivative of is . Clearly, a geometric series can only converge if the multiplicative factor is less than in magnitude. So this expansion diverges for . This is no surprise, as the blows up at , and one can expect the characteristics of functional derivatives on one side to have little effect on the other side of this singular point (probably why physicists can’t define a “before” the big-bang universe). A good analogy is the vibration of a constrained string. A string held put at a point can only vibrate in modes which have a node at that point. But if we clamp down (up?) all the derivatives of the string at that point, then excitations from one side cannot travel to the other. If one were to expand as a power series about some other point greater than one, it would be valid for all . So we seem to have arrived at some sort of definition for the boundary of a domain. After all, is merely shifted and flipped. But there is an anomaly.

It turns out that this expansion also diverges for . Highly unexpected, as seems to have no bumps, kinks or black holes around . Funky behaviour on one side of the point of expansion seems to be affecting the symmetrically opposite point on the other side. The string has strings attached? Say we parity flip the function and expand again about the origin:

This again is singular at , but the expansion diverges for both , and .

Note that the two expansions are interconvertible. One is generated by flipping the signs of all alternate terms in the other. Now we invoke the concept of conditional vs. absolute convergence of a power series.

An infinite series tends to converge to a finite value if successive terms fall off fast enough as increases. Sometimes, the signs of the terms help the convergence by cancellation. But the series is said to be absolutely convergent (as opposed to conditionally convergent) if the fall off in successive term magnitude is so fast that is convergent. Absolute convergence implies conditional convergence. But conditional convergence does not imply absolute convergence. Take for example the harmonic series . It is provably divergent. But if we flip the sign of every even term, the resulting series converges to . Series with alternating signs have added help at hand, as all they require is for the absolute values of successive terms to tend monotonically to .

None of this helps our case. We are looking for conditional vs. absolute divergence relations.
Take the series . Denote the positive terms by , and the negative terms by . Then the absolute sum is , and the conditional sum is . For absolutely convergent series, both the sum of positive terms and the sum of negative terms need to be convergent. For conditional convergence, both have to be divergent. For our power series expansions, both positive term sums and negative term sums have to converge inside the domain. Only one of them needs to diverge outside the domain.

As an aside, the sum of conditionally convergent series can be changed by suitable rearrangement of the terms of the series, and it can even be made to diverge. This is not possible for absolutely convergent series.

So, what is the biggest generalisation we can make thus far?
If a power series in x converges for a value , it converges absolutely for every value x such that , and the convergence is uniform in every interval , where is any positive number less than . may lie as near as we please (in an sense).

This magically implies that,
If such that the power series diverges, then it must diverge for every value of such that ,
This defines an “interval of convergence”, centered about the point of expansion for the power series.
Since the KVM function exp(-1/x) has a singularity at , its interval of convergence for an expansion about the origin has size zero. This still doesn’t seem to demystify the stitched function weirdness.

So What Is All This Analyticity Stuff??

So why is analyticity a stronger constraint than continuity? What other conditions does it impose on the countably infinite derivatives required to completely specify the function?
It is worthwhile to recall that in calculus, the differential element of the independent variable is a linear term in x. Often in analysis and perturbation theory, we’ve neglected terms with higher order powers and managed to retain accuracy of our results. So a power series expansion of a function about a point to a very good accuracy, can be approximated by a finite degree polynomial, in a neighbourhood about that point, whose size is dependent on the desired error tolerance. This means, for an ultra-small neighbourhood, the function is a straight line. In a slightly larger, say mega-small, neighbourhood, it is a parabola, and so on. If we differentiate a power series any number of times, we again end up with a power series. So this property is not only shared by the function, but also by all its other derivatives.

Say a function f(x) has a zero of order r at , , . Then the function may be expressed as , with . g(x) cannot vanish in a suitably small neighbourhood of , or, the zeros of f(x) are “isolated”, unless of course f vanishes identically.
The same is true for . It follows that in a finite interval an analytical function (and all its derivatives) is (are) piecewise monotone(s), that is, it (they) cannot change its (their) character of monotonicity infinitely often.
As a counter-example, consider the Taylor expansion of . This tends to wildly oscillate as x approaches 0. It crosses the x-axis infinitely many times in any finite interval about the origin. So the zeros at the origin can’t be said to be isolated.

Complexity Matters:

So far, we seemed to have managed quite well on the real number line. Here comes the bomb:
Taylor expansions of:

or

These expansions also diverge of . .
It turns out that complex domain is more natural than its real subset.
We note that both and the expansion of diverge at , where . Note that the real part of the first function blows up, but for the second function, its the imaginary part that does so.

If we are willing to extend the power series expansion to complex variables, then we’ll have to upgrade our interval of convergence to circle of convergence. This would mean that if is singular at complex point , then its power series expansion about the origin definitely diverges for points farther than on the complex domain.
Take the function , which is quite close to the KVM function. All its derivatives vanish at . So we cannot expand it in a power series about the origin because diverges on the imaginary axis as it tends toward the origin.

An arbitrary point on the complex plane may be defined as . Then, . A singularity at some point affects the function expansion everywhere on the circle . The vibrating string has become a 2D membrane. And if all derivatives of the membrane are clamped at some point, then radial (and angular?) excitations from the origin cannot travel beyond the circle centred at the origin, with the clamped point on its circumference.
HSR once remarked that the EE department at IITM wrongfully exposes its students to Fourier and Laplace transforms before the concept of analytical functions has been taught. Electrical engineers who have studied filter design, should recall all those conditions for “poles lying on the left half-plane” or “outside the unit circle”, and reinterpret them in the present context.

This domain extension works wonders to our compression ratio discussion. It allows us to reduce the task of function definition from specifying two values at every point on a 2D region to specifying twice (hehe) countably infinite derivatives (real and imaginary) at any one point. There is a strange interplay between real and complex values of the power series expansion of any analytical function, that transcends the absolute vs. conditional convergence picture.

The cosine and sine functions scale the magnitudes of the real and imaginary values and affect the signs of terms depending on .
It might be worthwhile to see how a rotation transform affects power series expansions.

It is interesting that complex analysis turned out to be so fundamental to our understanding of seemingly trivial math. The Euler exponential notation was more than a calculation and expression aid. And we didn’t even need quantum mechanics! All of it has much to do with the fact that on a line, there are 2 sides to a point. There are two signs, + and -, as there are 2 kinds of electromagnetic charges. Hence the 2D domain. And a set of two harmonically conjugate solutions for the Laplacian. There cannot be a further extension of the complex domain to higher dimensions in quite the same way real was extended to complex (a la Algebraically closed fields). This was the triumph of 19th century mathematics, when rigour was born. The number 2 was found be holy.

Aside: Parity is a discrete transformation. However, in relativistic quantum mechanics, while studying chiral symmetry of Dirac particles, a parity flip is modelled as a continuous transformation using the pseudo-scalar matrix as a generator of rotations on a complex plane. The wavefunction transforms as , and an operator . Also, in QED, in perturbative expansions of charge conjugation symmetric processes (like electron-positron annihilation), the observable cannot depend on sign of charge. So it is a power series in , or , the fine structure constant. But in a universe where , like charges will attract and opposite charges will repel. This is very unstable, as the only ground-state is two oppositely charged black-holes at infinite distance from eachother. So, if could take values on a complex plane, then all negative real values are a no-no. So there is a branch cut on the complex plane all along the negative real axis, implying that the radius of convergence of a power series expansion in about the origin is zero. So even though is very small, the series don’t converge. They just happen to be “asymptotic” series, which allows physicists to do some amount of close approximation to physics. Now you know why even theoretically computed physical values (like masses) have a computable error bound, and the number of significant digits keep increasing every year.

The Search for Hidden Redundancies:

The definition of analytical functions on the complex plane is generally made using the Cauchy-Riemann differential equations. However, Weierstrass is said to have developed the theory of complex analysis from scratch from the point of view of power series expansions. That derivation seems lost to the pages of history written in European languages. Let us see how much further we can explore the compression ratio aspects and built in redundancies in function definition by specification of parameters.

So in a domain where a complex function is analytic (i.e. differentiable in an epsilon-delta limit sense), integration along any closed contour vanishes. But what if we introduce a simple pole (singularity) in the function at the point and compute the contour integral around it. We have,

In the region of analyticity, the value of a function is the average of its value in the neighbourhood of the point in question (see how this is true for real functions defined on the real line). Does this also imply that the solution to a Laplacian differential equation is unique if the boundary conditions are fixed? That is a reduction from specifying function values on a 2D domain to specifying on a 1D contour. And since it is analytic along the contour loop, it can be parameterised again by coefficients of angular basis functions, this reducing further into countably infinite space. Every analytical function can be expanded as a power series. If we introduce slightly sharper poles at points and compute contour integrals around them, we have:

The contour integral seems to circle the point times clockwise, to yield the derivative of the function. Note that in the power series expansion, the term () also circles a closed contour n times (anti-clockwise, as if to unwind the derivative coefficient). The complex plane comes with strings attached, since every point has a build-in multiplicity (there are n solutions to ).

Inverting a function converts poles to zeros and vice versa. Do you see how that affects regions of power series convergence?
If has an order zero at , then it is expressible as , where . The inverse function ( is analytic in the neighbourhood of ), can be expanded as,

A contour integral of negative index terms of around gives , and the higher negative power terms vanish. The residue of a function at a pole is therefore .
The theorem of residues, which best illustrates the information compression ratio, states that if a function is analytic in the interior of a region R and on its boundary C except at a finite number of interior poles, the integral of the function taken around C in the positive sense is equal to the sum of the residues of the function at the poles enclosed by the boundary C.

So Why Power Series??

Consider the polynomial expression,

For some real parameter , take the integral along any closed path C in the z-plane, which does not pass through any of the zeros of P(z),

Let f(z) be a constant or any polynomial in z, of a degree we shall assume to be less than , so that the integrand behaves well at infinity. The successive derivatives of with respect to under the integral sign is equivalent to multiplication of the integrand by as the case may be. If we now form the differential expression , or in symbolic operator notation, , we have,

So is a solution for the differential equation , and the arbitrary constants from the polynomial are the constants of integration.
Most differential equations engineers and scientists study can be constructed out of polynomial type operators. If

then the solution of is,

and if but

and,

and,

Balki once claimed that if all of Calculus had to be summed up in one single useful sentence, it would read, “An exponential grows faster than a polynomial, which grows faster than a logarithm”. The definition of a differential element as a linear increment of the independent variable, and the practice of solving polynomial operator type differential equations, has lead to the extensive use of power series in analysing mathematics, instead of other types of expansions. However, linear diffential operators of infinite order, like the time evolution operator in Quantum mechanics, and the generator formalism for infinitesimal transformations have been explored as well.

As for the KVM function, every derivative at vanishes because they are all products of a diverging polynomial factor and a converging (vanishing) exponential factor. If one were to blindly Taylor-expand the function about the origin, all the polynomial factors from all the terms will seem to add up to something formidable, even compared to the common vanishing exponential factor. The derivative does seem to be of order , although understandably indecisive about its sign.

[BLINK] Plugin missing

We started by stating that analytical functions can be at worst specified uniquely by a countably infinite number of parameters. In the power series expansion notation, these parameters turned out to be successive derivatives of the function at a point. However, from early lessons in probability density function theory, one may recall a similar set of parameters which are defined by the integration operator. A function on the real number line could well be uniquely specified by its mean, variance, and all the higher moments about any point. I wonder how this would translate to the complex plane, since analyticity doesn’t seem to play a role at first glance (but normalizability matters . . . . a lot . . . ). This might hint at the duality between differential equations and lagrangian formalisms. Are the invariant constants dictated by Noether’s theorem applied to the Lagrangian formalism directly related to the constants of integration arising from the differential equation formalism? Karthik’s next post promises to enlighten us regarding some aspects of this.

Epistemology, and the possible non-futility of Modelling:

KVM’s Rubel post was preceded by his disdain for all modelling. However, one must ask what it truly means to have understood a phenomena, as opposed to having merely “modelled” it. All our analysis seem to be restricted to analytical functions, but they are a small subset of all possible functions. Wiki claims that:

In a measure-theoretic sense: when the space C([0, 1]; R) is equipped with classical Wiener measure , the collection of functions that are differentiable at even a single point of [0, 1] has -measure zero. The same is true even if one takes finite-dimensional “slices” of C([0, 1]; R): the nowhere-differentiable functions form a prevalent subset of C([0, 1]; R).

How curious, that most natural processes can be easily described by analytical functions alone? Or can they? Aren’t singularities our greatest unconquered enemies? In my view, the non-rapid, piecewise monotonic behaviour appeals to an aesthetic sense of the human mind, which has evolved to expect small input changes to have small observable effects, at least in the short run. Are we then destined to know the universe only to a close approximation?

Does a mere knowledge of a handful of parameters qualify as knowledge of a process? Is the definition of the class of functions of the solution important? At what point of generalisation does a predictive DE (as defined by KVM) become a shrink DE? Is it NOT amazing that group symmetries helped define previously unobserved fundamental particles in high-energy physics? Is “intuitive feel” all we have?

Needless to say, continuity is yet to be demystified. But I hope I have motivated a domain for analysis. If this entire post is to be condensed into a single valuable sentence, it would read, “2 is a holy number”.

References:-
1> Richard Courant, Fritz john, “Introduction to Calculus and Analysis”, Vol. I. and II.
2> My class notes.
3> Wikipedia

Other suggested reading:-
1> Balki’s arguments for why 3 is a holy number.
2> Rubel’s original paper: Rubel, L. A. “A Universal Differential Equation.” Bull. Amer. Math. Soc. 4, 345-349, 1981.
3> Weierstrass’s original unpublished works, if you can find them.

The following is an essay I wrote elaborating upon the power of plain text as a digital medium. I love text based data formats; I work with plain text for nearly everything. I love working at a shell- GUIs frustrate me. I am not an authority on the subjects of GUIs and data formats—I am an end user. Paul Graham says in his inspiring
essay on essays:

An essay is something you write to try to figure something out.

Figure out what? You don’t know yet. And so you can’t begin with a thesis, because you don’t have one, and may never have one. An essay doesn’t begin with a statement, but with a question.

And that explains (in retrospect) why I sat down one evening and hammered out a few thousand words about text, of all things. I was trying to figure out why I liked text so.

He continues:

…if you want to write essays, you need two ingredients: a few topics you’ve thought about a lot, and some ability to ferret out the unexpected.

I certainly pass the first requirement, not so much the second! As a technical discourse, this essay will often be in error, and go nowhere on the whole. Read as a whimsical opinion piece about my fascination with all things plain text, however, it makes a smidgen more sense.

Verbiage

Nested firmly between screen and seat, I take stock of how much of what I see is words. I write this in a text editor, with a browser and a music player open in the background. The browser is best navigated with a mouse; nevertheless, the primary interaction with it involves typing in a URL. A text editor is (by definition) all text—but in addition, I ask things of it, and I ask by typing in commands. The music player lets me click on text naming songs, and on buttons for issuing play commands, but it is not long before I have to recourse to the keyboard to search for a song.

A remarkably large chunk of our method of instructing our personal computers is done with pointing devices, and we like seeing large friendly buttons and boxes telling us where to click. Nevertheless, one finds that many actions involve the typing of text by design, perhaps because there is no other way to design these applications to do something for you. As anyone who’s been at a computer for more than a few minutes realizes, this need is marked by that annoying phase of reaching over to the keyboard from the mouse.

The frequent fallback to the keyboard is often accepted, especially by us later generations of users brought up on "intuitive" GUIs, as a remaining vestige of the days when text was the only means of interaction with a computer. Surely this need will be dispensed with in time?

This is a one-sided view of a larger set of tradeoffs we make in our use of computers everyday: automation vs control, obfuscation vs complexity, ease of use vs power and GUIs vs text interfaces—the one that intrigues me the most.

Neal Stephenson’s In The Beginning Was The Command Line is a brilliant, meandering discourse on user interfaces, among other things. If you haven’t read it yet, you ought to stop reading this and go take a look¹. (Like, right now).

Operating systems like to present us with clever visual metaphors. Nearly everyone who’s used a computer for a while realizes that there is no micro-benchpress compressing a folder when you’re shown an animation of the same, and that pages don’t fly over across folders when you issue a move command. Of course, in reality these metaphors are multi-layered; multiple levels of abstraction, each progressively twisting and distorting our view of the innards of frighteningly complex hardware carrying out equally frightening terrible processes.

In principle, the user is free to choose one of these realms of abstraction to "reside on", to accept a comfortable level of benign deceit. In practice, most of us have had this choice made for us by others; in most cases without our awareness or consent. (Perhaps this is a good thing. Choice scares us.) This essay peels back just half a layer—a limited under-the-hood tour of how nearly everything comes down to (plain) text at some point in our interaction with computers.

A Wordy Net

Around the winter of 2004, I spent about four hours in the vicinity of a person deeply absorbed in manipulating lots of gibberish on his terminal. I asked him what it was that was forcing him to sort through bales of noise. "They’re email headers!", he said.

"Email what?"

"Headers. It’s a bunch of text part of every email.

I responded that it was strange, then, that I’d never seen one of those. There is a lot of text (often gibberish, but text nevertheless) associated with email, text that identified the sender, subject and recipient, time stamps, and every automated agent that was involved in the transfer of the email. A one line email is accompanied by more than ten times as much of (meta)information about the email! Mail services and clients go to great lengths to hide the header from the user²; this exchange of headers is a machination of the æther, designed by humans but not meant for them.

I realized in the following years that it was not just email—many of the invisible exchanges on the Internet are padded with snippets of text; snippets that, if you read carefully, appear to tell the story of their origin. At the lowest level, every data packet sent over the Internet possesses a header stating IP addresses and the format of the data—although we do not need to sniff that low.

Nearly all services offered on the Internet are composed of easily readable text. For instance, here is a snippet of the RSS feed of the site where this is published³:

 http://rightshift.info Bionic Raptors ate this taglir:...
    Sun, 07 Jun 2009 11:46:00 +0000 http://backend.userland.com/rss092
    en Blink, Morse In the novel *Cryptonomicon*, one of the lead
    characters finds himself implicated in a (comical) drug bust, and
    is placed in a jail cell under the watchful (electronic) eyes of
    hi-tech eavesdroppers. It's a scene out of a spy novel (although
    *Cryptonomicon* isn't quite that), minus the secret agents, plus
    ... http://rightshift.info/?p=138 \u201cWhy are you producing so
    few red blood cells today?\u201d An interesting note from six
    months ago that I never got around to posting. 'tis a bit vague,
    but then so is the source. Besides, I like to think that
    nebulousness has its share of merits- trickster makes this world,
    after all.

(For the uninitiated, an RSS feed is a service (and protocol) that offers the content of a website to users. Primarily, it obviates the need to visit a website to access updated content.)

There’s plenty of crud in there, but if I stare at it long enough, easily identifiable patterns emerge. I see timestamps, URLs and sample content from previous posts, arranged in order—and just like that, I begin to understand how a feed reader (aggregator) works. Retreiving content from this website, filtering out bits not meant for our eyes, and printing it to a window involves exchange of text at several levels, and little else.

Most Internet thoroughfare works the same way. HTML (and its cousins) are text-based "markup" languages, and sure enough, they begin a document with an explanatory header. Images and audio files come with text snippets attached. Who makes up these rules?

These guys do. In particular, they release abstrusely named documents called Request(s) For Comments. That is where the rules of the æther are made up. And it’s all text. For instance, here is the RFC for email messages⁴.

Why are things this way? My first thought is: How else could things be? But it is not too hard to imagine an Internet where all services talk with bits of binary data. Indeed, plain text has low information content per character, so using binary padding might conserve bandwidth. I reckon the architects stuck with text because the machinations in the æther will truly be lost to us otherwise—plain text is the most natural format for the transfer of information, albeit not the most efficient Thankfully, manageability was chosen over efficiency when the Internet was being realized.⁵

Plain Old Text

Text is the spine of the Internet.

What of it? The point is that text is a fantastic medium for information exchange between humans.

When you edit a configuration file in /etc in Linux, you’re not so much instructing the operating system as you are the person who set up the configuration in the first place. You tell him what you want your OS to do for you, in plain English, and he spells it out—he already has—in computerese. It’s the same with text and the Internet.

But all of that is just half the story.

At this point, it would be germane to elaborate upon the difference between text and plain text—at least as I use these terms in this essay.

By text, I mean any combination of ASCII (or its supersets, such as UTF-8) characters. The only common feature of entities in this category is that every character is human-readable. This would include all source code, markup code, email (and other) headers, and text like that found on this page.

Plain text refers to text written in English (or any other language, for that matter) with no characters to be interpreted specially
("escaped"). By extension, a plain text file is one that contains plain text and nothing else. Traditionally, this corresponds to the contents of .txt files.

The only type of file we’ve left out is the binary file—which can be conveniently slotted away as a file whose contents do not satisfy the above criteria. (A Microsoft Word document is a binary file, even though under the right conditions you can read plain text off of it.)

Plain text (and text in general) is a very powerful medium for storing and retreiving information. It’s the best we have today, for a number of reasons. In the light of the tradeoffs in usage mentioned earlier, the place of plain text is a very important one: Form (word processors) vs Content (text editors).

Plain text is highly portable. It doesn’t matter what OS you’re on. It doesn’t matter what software you have installed. It doesn’t matter what kind of display you use. You can read it on your phone and your ebook reader. In fact, if you can see a blinking cursor in a forty year old teletype, you can use it to read text. You could read it off a mainframe forty years ago. And when I think about it, it is quite likely that this is the only existing format we will be able to read off whatever device we use forty years from now.

It’s stable. A corrupt binary file is a nightmare to recover. By virtue of being human readable, handling a corrupt text file is a relatively pleasant prospect. As a bonus, plain text documents work the best with version control systems. In short, that means a text file is the easiest kind of file to track changes in and back up.

Text is easy to compose. There is no learning curve⁶ because you’ve been doing it all your life. Plain (unformatted) text is often easier on the eyes when you read, as well.

In a plain text document, the focus is on the content, not bling. Agreeably, there are situations when it is prudent to chose form over content, but most often (for essays such as this, say) this is not the case.

With very simple tools, plain text (and text in general) is very malleable. In the hands of a texpert⁷, plain text is amazingly efficient. No other data format comes close when it comes to manipulability.

The above reads like a checklist of reasons not to use Word, but the intended message is general in nature.⁸ For sheer power, no format I have ever used matches up. (But then I’m not a programmer, and there are several binary formats I haven’t tried.)

What plain text files are not: They aren’t good containers for large files. There is a parsing overhead involved in reading text files, and random access to some part of the file without having to read the whole thing is hard.⁹ They don’t handle associations well- text files don’t make for good spreadsheets. Of course, better ways to deal with large files also involve text, just not plain text.

For storing and exchanging information, text is the format of the Internet and the format of the OS. In retrospect, this is not as surprising—text is, after all, the natural format for exchanging information between us humans, and we built all of that.

M-x butterfly

All of which implies that we require means of generating and modifying this information in the first place. Enter the text editor—considered to be one of the pillars of a usable operating system.
This description illustrates:

In August 1969, Ken Thompson, a programmer at AT&T subsidiary Bell Laboratories, saw the month-long departure of his wife and young son as an opportunity to put his ideas for a new operating system into practice. He wrote the first version of Unix in assembly language for a wimpy Digital Equipment Corp. (DEC) PDP-7 minicomputer, spending one week each on the operating system, a shell, an editor and an assembler.

Among programmers, text editors are revered pieces of software, used and proselytised about with religious fervour. Or so the Internet suggests. I’m not a programmer^{10 11}, but I still spend an inordinate amount of time hammering away in a text editor, so I understand what the fuss is about, somewhat. Plain text is much more powerful than most people give it credit for. How efficiently it works for you depends on the editor you use, with Notepad (the one bundled with Windows) at the bottom of the utility ladder and several contenders fighting for the top spot.

Neal Stephenson is commonly quoted on his statement about text editors¹², from his above mentioned essay:

In the GNU/Linux world there are two major text editing programs: the minimalist vi (known in some implementations as elvis) and the maximalist emacs. I use emacs, which might be thought of as a thermonuclear word processor. It was created by Richard Stallman; enough said. It is written in Lisp, which is the only computer language that is beautiful. It is colossal, and yet it only edits straight ASCII text files, which is to say, no fonts, no boldface, no underlining. In other words, the engineer-hours that, in the case of Microsoft Word, were devoted to features like mail merge, and the ability to embed feature-length motion pictures in corporate memoranda, were, in the case of emacs, focused with maniacal intensity on the deceptively simple-seeming problem of editing text. If you are a professional writer—i.e., if someone else is getting paid to worry about how your words are formatted and printed—emacs outshines all other editing software in approximately the same way that the noonday sun does the stars. It is not just bigger and brighter; it simply makes everything else vanish.

Emacs is indeed comparable to Microsoft Word in scope (and little else, thankfully). If you’ve ever traversed the depths of the sub-menus in Microsoft Word and taken stock of the things you can do, you must wonder what functionality emacs could provide in handling plain text alone that compares.¹³ Rephrased, that question reads:

"How much can a text editor do with plain text anyway?"

It’s a valid question, deserving of an example-laden tour around emacs-town¹⁴, but it would be an unwelcome digression here. In short, a powerful text editor makes it a pleasure to compose text by encouraging economy of motion, automating repetitive tasks, and making it easy to (re-)configure its behaviour to suit your needs.

The best text editors are comparable to sheets of surgical steel. You can’t do much with it at first—you’ll cut your fingers on the edges if you try too hard. It needs to be tempered, beaten into shape, ground, polished and sharpened until you can use it as a scalpel. But it is, by then, your scalpel and yours alone—it fits in your palm like it belongs there (and it does). Because you spent time training with it even as you forged the instrument, you can now use the scalpel to execute deft maneuvers, to do in seconds what took minutes with the blunt knife you were using before. It will take years to test every cut the scalpel can make, but you will already have achieved a level of proficiency with it that will make you wonder how you ever managed without it.

The other powerful tool when handling text is a language for matching strings (of characters) of interest, known as regular expressions. Regular expressions let you specify a pattern of characters that you are interested in by stating rules to be followed by the expression processor. This is a much more powerful way of searching text compared to specifying a string itself—it is very nearly¹⁵ like setting an assistant on the job of, say, finding every sentence in this essay that contains exactly fourteen nonconsecutive e’s but no reference to a footnote.

Whimsical examples aside, regular expressions are used everyday to separate valuable information from torrents of raw text, but they are often stated to be a double edged sword. I love them, though. It helps that all self-respecting text editors have built-in support for regular expression syntax.

My morbid fascination with all things text is tied intricately to the tools I use to handle it. And there is no dearth of tools for handling plain text—on a standard install of the Linux operating system, there are dozens of programs (Unix shell utilities) that operate on text, and they can all be made to talk to each other, stringing together tasks to carry out complex transformations of the input. If this corpus of little tools prove insufficient, any of several programming languages (Perl, Python—also usually included) allows for writing scripts that process text in complicated (and inane) ways quite easily.

This stringing together (piping) of tools is an extremely useful idea. At a command line, programs act on strings of text passed to them as arguments—so one could, for instance, rename a thousand files at once by constructing a regular expression for the kind of filename to be renamed and the string to be renamed to, and giving the old and new names to the Unix rename-file command. As a consequence of the operating system’s text based interface, easily manipulated text corresponds to an easily controlled operating system!

This is an unexpected revelation. Every advantage that plain text enjoys is an advantage for systems that incorporate them into their control structure as well. It’s a subtle idea at first for those of us inundated with visual metaphors for using a computer. Once you begin poking under the hood, however, there’s no turning back.

Word

No one believes they lack perspective until they’ve acquired some of it. I did not realize how fractured my view of technology was until a professor at college casually mentioned that the map is one of the most underrated technologies in the history of man; it made conquest
(and imperialism) possible. One is used to thinking of cartography as an art, maybe, and definitely a skill, but not technology.

At some point, thinking about what counts as technology becomes a game of semantics- but a few (somewhat) startling observations can be made. Here is an excerpt from Wikipedia’s article on writing:

In Eurasia writing began as a consequence of the burgeoning needs of accounting. Around the 4th millennium BC, the complexity of trade and administration outgrew the power of memory, and writing became a more dependable method of recording and presenting transactions in a permanent form (Robinson, 2003, p. 36). In Mesoamerica writing may have evolved through calendrics and a political necessity for recording historical events.

When the process of instructing digital computers became a complex, enervating task, the engineers in charge instructed computers to understand words instead of numeric addresses. They constructed the first of many levels of abstraction to follow, but all of them since have been based on words. It is not surprising that this should be so—text is a mature technology in itself, all of six thousand years old.

Our primary means of interaction with programmable machines is still text, but the innards of the system are slowly being obscured by ever more intuitive and interactive graphical displays. The older, text based abstractions will stick around, though—and hopefully plain text will remain as gleeful an encoding to work with for quite some time.

After all, the recorded word is the ultimate technology.

Footnotes:

1. Incidentally, In The Beginning… is only available as a text file, which says something about the author’s preferences.

2. But provide them if you ask politely.

3. This is likely a moot exercise. Anyone familiar with RSS feeds will also know what they’re made of, so it conveys nothing to its intended audience and is irrelevant to the remaining.

4. Good luck with reading that.

5. On the other hand, text compresses better than binary data. There is doubtless more to this argument than what I could think of—I have not studied the history of the Internet.

6. If you want to get really good at writing text, this statement is patently false. A good text editor possesses as much of a learning curve as a document processor.

7. Yeah, I made that up. TeXpert is often used to describe a person proficient with TeX, but The TeXbook suggests that because of the way TeX is pronounced, TeXperts ought to be called TeXnicians instead.

8. That said, I fervently hope the day does not arise when I will be forced to use Word.

9. But not impossible, apparently.

10. And this essay isn’t aimed at programmers either. If you’re one, you already know all of this.

11. I did implement an AVL tree in C once. Phew! That was when I knew programming was not my thing. I do enjoy a little scripting every now and then, though.

12. Not all of it is accurate today. Emacs now displays italicized and bold text, for instance. But I think it remains true in spirit.

13. If you’re an emacs user, you’re probably wondering the opposite.

14. Or Vi(m) town, for that matter. I’m fairly ecumenical when it comes to the two behemoths, preferring them both for different tasks. For writing a long essay, however, emacs blows away all competition.

15. Very nearly, because regular expressions are written in a formal language, and so there are things it can’t do. No expression can be written to match a palindrome of arbitrary length, for instance.

Further Reading

• In The Beginning Was The Command Line
• Welcome To The Universe Of Fancy Colored Paper!
  

  An amusing metaphorical look at the world of proprietary data formats.

• Jamie Zawinski’s mail summary files
  

  To see how he set up a blazing fast way of accessing text files.

• The Power of Plain Text
  

  A long Wiki discussion of the merits (and demerits) of plain text.

• Demonstrations of what Emacs or Vim can do with plain text are surprisingly hard to find. The best way to see what the fuss is about would be to play around with them- but here’s a five minute screencast showcasing some of emacs’ more advanced features.
• Regular expressions explained. Also, here’s a step-by-step build of an elementary regular expression matching engine.

o_0

Apollo Optik. An expensive, and an unknown brand to me. But to my delight, I discovered that they were polarized!!

For the uninitiated, light is a self-propagating electromagnetic field disturbance. I’m tempted to call it sourceless, but fact is, that all light is generated by charges in acceleration somewhere in the universe. Physics limits the speed with which the field values of said charges update themselves to the “current” position of the charges. So these disturbances seem to travel uncorrelated from their creators, into ths distant reaches of the universe to restore the balance. Hmm, so whats all this polarization nonsense?!

The shabby diagrams section:-

Sadly, the exact quantum treatment is beyond the scope of this post. Read a book.

The reason we call light “WAVES” is because if you manage to isolate any particular frequency of the “disturbance”, the way this phenomena seems to interact with matter (such as measuring instruments) leads us to conclude that it consists of synchronised and oscillating electric and magnetic fields in mutually perpendicular planes. And the direction of propagation at any instant of time happens to be the cross product of the Electric and Magnetic Field.

So with the direction of propagation fixed, there is an extra degree of freedom in the perpendicular plane, i.e. the orientation of the Electric (or Magnetic) field. If you know how Euclidean geometry works, you’ll realise that one can define two special perpendicular polarisation states and express any other state as a linear combination of these two.

Now, the way these waves interact with matter (other charges) is similar to the way they were produced (refer Feynman Lectures in Physics, Vol. I, Chapter 28 “Electromagnetic Radiation”, equation 28.3). Any charge in an electric field will experience a force along its direction (or opposite if the charge is negative). So when an incoming EM wave is “classically” “absorbed” by an classical atom (the electron cloud version), it tends to push the negatively charged cloud to one side and the positively charged nucleus to the other (actually, given the mass ratios between electrons and nuclei, it is mostly the electrons that move. Now that the centers of charge of the positive part and the negative part are not on top of each other, they attract each-other, causing the atom to oscillate. But they won’t go on for ever since accelerating charges transmit light, thus losing energy. So our atom has become a fresh source of light, re-radiating what energy it absorbed (albeit at lower strength since it re-radiates in all directions). So in a polarisation filter (such as my shades), heterogeneous molecules are arranged relative to each other such that the oscillation along one particular direction is easier than the other. Or in some cases, in one direction all oscillations translate to re-radiated light, but in another, they may couple to vibrations of molecules and become heat. This can also be achieved with certain anisotropic crystal lattices (charge oscillations along certain bond orientations). So any polarisation filter has a defined direction of polarisation, and all the light that comes through it is polarised in that direction. So when I put two filters one behind the other, the light that comes through the first one will not get through the second one if their polarisation angles are perpendicular to each other (as can be seen in the pic). Now my particular glasses are vertically polarised. How do I know that?

Brewsters Law :-

Again, classically, if we take the case of radiation by an oscillating charge, the radiation pattern is that of a dipole. What that means is that the strength of radiation will be different when measured at different angles from the plane perpendicular to the oscillation line.

So, if I can create an object whose charges will significantly oscillate in a given plane, I know that they can’t radiate anything polarised in the normal direction. Creating a crude approximation to said object was simply a matter of creating a reflective water surface. Light reflection from a water surface occurs due to incident light being “absorbed” and “re-emitted” by charged particles in the water. The oscillations induced in the charged particles are in a plane perpendicular to direction of incident light. So, if the direction of the reflected rays is predetermined by the laws of reflection (refraction?), at a particular special angle of incidence (which depends on water’s refractive index relative to that of air), the reflected rays will propagate at a right angle to the incident ray, and will thus be polarised horizontally (parallel to the water surface). Using this, I could easily deduce the plane of polarisation of my shades.

The reflection intensity from the water is maximum when my shades are tilted vertical (and consequently, the other pair of shades kept on the table seem dark). The result flips if my shades are made horizontal.

This simple high-school level semi-classical viewpoint makes gargantuan predictions. For example, did you know that sky light (the scattered blue) is polarized?
Skylight is basically sunlight being scattered (”absorbed” and “re-emitted”, classically speaking) by air molecules. The reason its blue is because higher frequencies of oscillation get scattered more. The density of Earth’s atmosphere (and its extent) is such that more than half the scattered skylight would have suffered a single scattering event (as opposed to being scattered by multiple molecules). Now, Sunlight is not polarised. But, if one were to observe the sky in a direction perpendicular to sun-rays, a large chunk of the skylight was emitted by charges, whose oscillations were induced by the sur-rays in a plane perpendicular to the sun-ray propagation direction. So if we were to use polarisation filters, and tilt our head, we can artificially increase the contrast between the clouds, and the sky background.

The image is imperfect because I’m using an over-user-friendly digital camera that self-adjusts its aperture and exposure time, and I’m fresh out of analogue film. For more on this, refer: “Clouds in a Glass of Beer” by Craig F. Bohren. Now excuse me while I show-off my new shades outside . . . .

Allow me to guide you through your technical “higher-learning” experience with the following example.

You begin by realizing that an easy and efficient way to prepare for IIT-JEE is to memorize a set of equations such as:

Then, you memorize what name each variable is referred to as.
( Entropy, Chemical Potential, and so on.)

And then, you memorize the “conditions” under which the use of an equation is valid, such as “reversible processes for the above case”. Then you look for the same key words in the question being posed, choose the correct equation, plug-in values, and compute some number. Presto, now you are in college, have met some intellectual lecturers and intelligent peers, have been exposed to the larger portion of the real world, have realized that you don’t know jack, have a library, the internets, and accessible “consultants” at your disposal, and are suddenly faced with an abundance of free time. And then, you tend to grow a brain.

You read somewhere about the First Law of Thermodynamics:

You learn that is a perfect (exact?) differential, which means that it is path independent. And since this was explicitly mentioned, the others are not (in general). So you painstakingly learn how to compute the others consistently so as not to violate the First Law for a toy problem involving cylinders, pistons and ideal gases. Then you once again encounter entropy and reversible processes, and are tempted to rewrite the First Law like so:

This begins to look very familiar.

Here you learn to define Intensive (or Field) variables and Extensive (or State) variables. You make an analogy to electrodynamics and solid mechanics, where and stress would be Intensive and and strain would be Extensive variables. Then you treat as Field variables and as State variables.

Now you begin to view as a sort of “Thermodynamic Potential”, for a system in state . Then define

Field variables as:

Then you cross check your intuitive definitions of the Field variables for consistency. Now you define a “regime of validity” where the state changes are such (slow?, reversible?) that   behaves as a homogeneous function of the extensive variables. You express the “Thermodynamic Potential” with a bunch of first-order derivative terms, and arrive at the Euler-relation:

Now in a rather unrelated train of thought, you learn about Legendre transforms. You wiki it. You apply it in Lagrangian Mechanics.

You transform a “Lagrangian” into a “Hamiltonian” . You notice that the state variables have changed from to .

Now, armed with Legendre Transform technology, you begin ravaging the “Thermodynamic Potential” from earlier. You start to derive familiar relics from the past, such as

Gibb’s Free Energy:

Helmholtz Free Energy:

Entropy:

Grand Potential:

And other strange beasts:

You revise your list of Field and State variables for each case, and derive their respective “regimes of validity”. Suddenly, you realize that all your training has been mathematically consistent and you feel the need to write this down somewhere. And someday you graduate, at which point I stop ranting because I’m yet to experience what comes next.

Many souls in this age are unfortunate enough to get inducted into over-accelerated education programs hellbent on flooding their minds with advanced sounding semantics and arbitrary logical rules of progression. It appears that “higher-learning” is composed of recursively unlearning and relearning “old” lessons. Its probably a good way to go about it, if each level is viewed as a layer of abstraction (as hinted by fellow co-blogger elsewhere), but one would wish that said fact of abstraction be stressed early enough in place of dogmatic rhetoric, to make the unlearning a little easier.

@Neels: TeX support test #1, successful. Good job!

The rest of this lead-up contains spoilers, so skip to the image if you’re the ear-cupping-and-yelling kind.

The conspirators don’t pull punches; they’re replicating the prisoner’s laptop screen in an adjacent cell with a Van Eck phreaking setup- so everything he sees, they see. Or do they?

Food, bedding, and a computer- it sounds far too pleasant a way to imprison an inveterate hacker, so he proceeds, ostensibly, to hand over the information they need, carrying out some phony cryptanalysis in the process. And then he conjures a little script, writing out small pieces of code to a file at intermittent intervals. When executed, this script reads out text files (the Real Thing) by flashing the LEDs on his keyboard in morse.

Wow. Using intractably complex technology to simulate simple tasks is a fascinating idea, if only because it goes against design. It’s the joy of complete control over a programmable machine- articulated amazingly well in this extract (from Cory Doctorow’s Little Brother):

If you’ve never programmed a computer, you should. There’s nothing like it in the whole world. When you program a computer, it does *exactly* what you tell it to do. It’s like designing a machine — any machine, like a car, like a faucet, like a gas-hinge for a door — using math and instructions. It’s awesome in the truest sense: it can fill you with awe.

Flashing text in Morse off a keyboard LED is the sort of thing I can’t read without wanting to do myself. So I did.*

Internet, meet Morseblink. Assuming even that you’re not digging for gold in the Philippines with several governments seeking your hard drive for forgotten world war II treasures, it’s still perfect for those someone-over-shoulder moments- or minutes- when you’re trying to give the impression of being at work and want to continue catching up on your feeds.

If you know Morse, that is. At least it gives that oft-ignored scroll-lock LED something to do, the poor thing. (Alternatively, you could relegate that annoying system beep to scroll-lock by having it flash instead.It’s an old idea)

It’s fully customizable, if your binary encoding of choice happens to be different. Alternatively, you could use it to practice your Morse, which is all I’ve found it useful for (yet).

You can download it here (written in Perl 5.8, should run on all *nix systems).

Below the cut: Some usage tips and a refresher on the Morse code.

$ perl morseblink.pl [options] [File1 File2 ... | STDIN]
options: -do[t] n -da[sh] n -g[ap] n -v[erbose] -e[ncode]
         -l[ed] [1-32]

         dot:     dot duration (float, seconds)
         dash:    dash duration (float, seconds)
         gap:     gap between letters (float, seconds)
         verbose: report bits being flashed at STDOUT
         encode:  encode and print, do not flash LED
         led:     led number from 1-32. Scrollock is 3.

Use the -dot and -dash options to set the duration of the dot and dash flashes. -gap sets the duration of the gap between letters. The gap between words is twice this.

If you want to use this as a Morse tutor, run it with the -v (verbose) flag enabled. To get the encoded output to STDOUT (no flashing), use -e (encode). All of these are optional; it should work fine with the defaults. Finally, it accepts input from STDIN (terminate input with ^D), so you can use the script interactively:

$ perl morseblink.pl -v
test
^D
bit 1: "-"
bit 2: " "
bit 3: "."
bit 4: " "
bit 5: "."
bit 6: "."
bit 7: "."
bit 8: " "
bit 9: "-"
bit 10: " "
$

The morse code doesn’t cover all printable ASCII characters. There’s no !, for example. Any unrecognized character in your input is replaced with the code for 0.

” .-..-.
$ …-..-
‘ .—-.
( -.–.
) -.–.-
+ .-.-.
, –..–
- -….-
. .-.-.-
/ -..-.
0 —–
1 .—-
2 ..—
3 …–
4 ….-
5 …..
6 -….
7 –…
8 —..
9 —-.
: —…
; -.-.-.
= -…-
? ..–..
A .-
B -…
C -.-.
D -..
E .
F ..-.
G –.
H ….
I ..
J .—
K -.-
L .-..
M –
N -.
O —
P .–.
Q –.-
R .-.
S …
T -
U ..-
V …-
W .–
X -..-
Y -.–
Z –..
_ ..–.-

*Turned out to be a trivial hack, as simple to  write as it was to describe.

***

A while ago, it was thought that the trick to making a machine play chess well was to extend how far down the branching network of possible moves it could examine. Irrespective of how far they can look ahead, though, skilled human chess players can confidently confound (or at least match) most chess programs of today.

Why?

I found a fascinating account of this puzzler involving AI and human thinking in (where else?) GEB. Apparently, the reason for this was known from the 1940s; If you’ve ever played chess- or play regularly but with skill befitting a two year old, you’ll come to appreciate the reason immensely.

Chess novices and chess masters perceive a chess situation in completely different terms. The results of the Dutch psychologist Adriaan de Groot’s study (from the 1940’s) imply that chess masters perceive the distribution of pieces in chunks. (Here’s a more recent description.)

There is a higher-level description of the board than the straightforward “white pawn on K5, black rook on Q6″ type of description, and the master somehow produces such a mental image of the board. This was proven by the high speed with which a master could reproduce an actual position taken from a game, compared with the novice’s plodding reconstruction of the position, after both of them had five second glances at the board. Highly revealing was the fact that masters’ mistakes involved placing whole groups of pieces in the wrong place, which left the game strategically almost the same, but to a novice’s eyes, not at all the same. The clincher was to do the same experiment but with pieces randomly assigned to the squares on the board, instead of copied from actual games. The masters were found to be simply no better than the novices in reconstructing such random boards.

The conclusion is that in normal chess play, certain types of situation recur- certain patterns- and it is on these high-level patterns that the master is sensitive. He thinks on a different level from the novice; his set of concepts is different. Nearly everyone is surprised to find out that in actual play, a master rarely looks ahead any further than a novice does- and moreover, a master usually examines only a handful of possible moves! The trick is that his mode of perceiving the board is like a filter: he literally does not see bad moves when he looks at a chess situation- no more than chess amateurs see illegal moves when they look at a chess situation. Anyone who has played even a little chess has organized his perception so that diagonal rook-moves, forward capture by pawns, and so forth, are never brought to mind. Similarly, master-level players have built up higher levels of organization in the way they see the board; consequently, to them, bad moves are as unlikely as illegal moves are, to most people. This might be called implicit pruning of the giant branching tree of possibilities. By contrast, explicit pruning would involve thinking of a move, and after superficial examination, deciding not to pursue examining it any further.

If you pause to think about this, it comes across as an utterly spellbinding revelation.

Like the thinking of the proverbial frog in the well, mental models and levels of perception above what we are used to are very hard to digest- but they’re there, as the excerpt explains.

The “chunking into levels” is a predominant theme in all complex systems we see*, at least in the way we seek to understand and analyze them- from computer systems (Hardwired-code->machine language->Assembly language->Interpreters and Compilers) to DNA (Specifying each nucleotide atom-by-atom->Describing codons with symbols for nucleotides->Macromolecules->Cells), and even human thinking.

The last bit requires a little exposition, but first, we note the analogy between the nightmare of writing complex useful computer code in machine language and the terror of reading a virus DNA atom by atom. In both cases, we would miss out on the higher level structures that embody computer programs and virus DNA with their attributes- complex systems possess meaning on multiple levels.

The multi-level description extends to virtually every complex phenomenon. Weather systems, for instance, possess “hardware” (the earth’s atmosphere) which has certain properties hardwired into it (hardwired code) in the form of the laws that flitting air molecules obey, and “software”, which is the weather itself. Looking at the motions of individual molecules is akin to reading a huge, complicated program on the machine language level. We chunk higher level patterns into storms and clouds, pressures and winds- large scale coherent trends that emerge from the motion of astronomical number of molecules.

As for multi-level human thinking, it is illuminating to first appreciate that a higher level perception of a system does not necessarily mean an understanding of the lower levels too. One does not need to know machine language to write complex computer programs, nor is one required to be aware of individual molecule trajectories to describe or predict the weather.** In fact, a higher level of a system may itself not be “aware” of the levels it is composed of, such as AI programs that are ignorant of the operating system they are running on. The higher level descriptions are “sealed off” from the levels below them, although there is some “leakage” between the hierarchical levels of science. (This is necessarily a good thing- or people could not obtain an approximate understanding of other people without first figuring out how quarks interact.)

The title of the post is another excerpt from GEB, derived as a somewhat whimsical analogy:

The idea that “you” know all about “yourself” is so familiar from interaction with people that it is natural to extend it to the computer- after all, AI programs are intelligent enough that they can “talk” to you in English! Asking an AI program (which is compiled code) about the underlying operating system is not unlike asking a person “Why are you producing so few red blood cells today?” People do not know about that level- the “operating system level”- of their bodies.

This post owes its existence in entirety to GEB, the Big Book of Big Ideas.

***

*Where a “system” is merely something consisting of “parts”.

**The trade-off when accepting higher-level models of a system is to lose the ability to make accurate predictions. This is fairly evident, irrespective of whether we’re talking about the behaviour of weather, basketballs or people!