Larry Wall's
talk on Perl 0-5.
Interesting read.
Written by Larry Wall
Friday, 15 October 1999
They say one should never start with an apology, so here goes: I'm sorry, but this essay will be all over the map. But then, given that I too am all over the map (and by extension, so is Perl), this kind of wandering is entirely inevitable. So I'll do my best to keep this story roughly organized in a chronological order because the history of Perl is suggestive of various principles that you may see in operation elsewhere.
Or maybe not...
PERL 0
Gestation
Sex makes babies.
Like the typical human, Perl was conceived in secret, and existed for roughly nine months before anyone in the world ever saw it. Its womb was a secret project for the National Security Agency known as the "Blacker" project, which has long since closed down. The goal of that sexy project was not to produce Perl. However, Perl may well have been the most useful thing to come from Blacker. Sex can fool you that way.
Not only was Perl born of a sexy project, but its early design came from heavy cross-pollination.
Parents
Genetically speaking, of course, Perl has many more ancestors than that simple diagram above indicates. (I've left out theology and biology, for instance.) I am a synthesist at heart, and can't help throwing a little bit of everything into the pot all at once. Just in the realm of computer science,Perl draws inspiration from many languages. One could draw a similar diagram just for that
More Parents
That also is an oversimplification, since Perl has drawn from many other languages over the years. (Lisp and Ada, for instance.) But the four languages in the diagram influenced Perl the most at the beginning of its development. Well, those four, and BASIC-PLUS...
At this point, I'm talking about Perl, version 0. Only a few people in my office ever used it. In fact, the early history of Perl recorded in O'Reilly's Camel Book (Programming Perl) was written by my officemate of the time, Daniel Faigin.
He, along with my brother in law, Mark Biggar, were most influential in the early design of Perl. They were also the only users at the time. Mark talked me out of using bc as a backend expression processor, and into using normal, built in floating point operations, since they were just being standardized by the IEEE (Institute of Electrical and Electronics Engineers). Relying on that standard was one of the better decisions I ever made. Earlier scripting languages such as REXX didn't have that option, and as a result they tend to run slower.
As with a human fetus, the last several months of Perl's gestation did not produce any large differences in the appearance of Perl. Rather, Perl simply put on some weight as it prepared for the real world. I recognized that acceptance of Perl hinged on the ability of people to leverage what they already knew, so I didn't release Perl until I'd written two translators: s2p (sed to Perl) and a2p (awk to Perl).
I made one major, incompatible change to Perl just before it was born. From the start, one of my overriding design principles was to "optimize for the common case." I didn't coin this phase, of course. I learned it from people like Dennis Ritchie, who realized that computers tend to assign more values than they compare. This is why Dennis made = represent assignment and == represent comparison in his C programming language.
I'd made many such tradeoffs in designing Perl, but I realized that I'd violated the principle in Perl's regular expression syntax. It used grep's notion of backslashing ordinary characters to produce metacharacters, rather than egrep's notion of backslashing metacharacters to produce ordinary characters.
It turns out that you use the metacharacters much more frequently than you do the literal characters, so it made sense to change Perl so that /(.*)/ defined a substring that could be referenced later, while /\(.*\)/ matched a sequence inside literal parentheses.
This change broke all of Dan Faigin's scripts. I think he's forgiven me by now, even if he won't admit it.
As you might expect, another decision at this time was whether to release Perl as free software. This was essentially a no brainer. I'd already done most of my thinking about this topic earlier when I released the rn and patch programs. Occasionally people ask me whether I was surprised that Perl caught on, and then they're surprised when I say "no." But you have to remember that I already had several open source projects under my belt, and had a pretty good notion that anything I liked, other people might also like.
But it wasn't as easy deciding to do an open source project back then. These days, you can actually go to your company and make a case for releasing the source code. Back then, you had to believe in it enough to take some risks.
I knew that I didn't dare ask the company lawyers for permission, because they'd have thought about it for something like six months, and then told me "no." This is despite the fact that they wouldn't be interested in peddling it themselves. In the old days, a lot of free software was released under the principle that it's much easier to ask forgiveness than to seek permission. I'm glad things have changed -- at least to the extent that the counterculture is acknowledged these days, even if it's not quite accepted. Yet.
But Perl was actually much more countercultural than you might think. It was intended to subvert the Unix philosophy. More specifically, it was intended to subvert that part of Unix philosophy that said that every tool should do only one thing and do that one thing well.
The problem with that philosophy is that many of the tools available under Unix did not, in fact, do things very well. They had arbitrary limits. They were slow. They were non-portable. They were difficult to integrate via the shell because they had different ideas of data formats. They worked okay as long as you did what was expected, but if you wanted to do something slightly different, you had to write your own tool from scratch.
So that's what I did. Perl is just another tool in the Unix toolbox. Perl does one thing, and it does it well: it gets out of your face.
I released Perl to the comp.sources.misc newsgroup in 1987, and that was the end of Perl version 0.
But Perl was also intended to subvert some attitudes in computer science. More on that later.
PERL 1
The Baby
Perl Tim O'Reilly
A Class Act: Larry Wall's boss at O'Reilly & Associates, Tim O'Reilly.
Perl was born at an early age. I mean something specific by that.
Humans do not come from the womb with the ability to drive a stick shift. Neither did Perl. This was intentional. I've always been smart enough to realize how stupid I am, and one of the things I'm stupid about is predicting how my programs will develop over time. So Perl was equipped to learn, and have a long childhood.
We value the maturing process in our own species, but for some reason we don't like it as much in computer programs. In the absence of a handy Zeus, we like to think that computer programs should spring fully formed from our own foreheads. We want to present the world with a fait accompli. In modern terms, we want to build a cathedral.
Now let me just say that I think cathedrals have gotten a bum rap lately. Open Source advocate Eric Raymond has likened the commercial software development model to a cathedral, while he compares free software development to a bazaar.
Eric's heart is in the right place, but I think his metaphors are a little off. Most cathedrals were built in plain view with lots of volunteer labor. And most of the bazaars I've seen have produced little of lasting architectural value. Eric should have written about artists who insist on having an unveiling when their sculpture or painting is finished. Somehow I can't imagine anyone pulling a shroud off of a cathedral and saying, "Voila!"
An open source programmer is more like a public mural painter. An artist may have some idea of what he's about, but anyone walking by can make suggestions, or even help paint if the artist is agreeable. If not, he can always go paint a better mural on the other side of the freeway if he wants.
Anyway, when Perl version 1 arrived in the world, it knew it was the new arrival, not the world. It recognized that there are depths of wisdom in the world that have not yet been plumbed by computer science, let alone by Perl. So Perl played the role of a baby, and grew. Various nurturing individuals adopted Perl, and helped to raise it, because Perl was cute, and they became irrationally attached to it.
Besides, cute is cool. See penguins.
PERL 2
The Toddler
Theory is good, in moderation.
One of the ways Perl subverts computer science is by adopting theoretical axes without grinding them. The theory of regular expressions is highly developed, but typical users just want to get their work done. Perl 2 had a more powerful regular expression engine under the hood, but the idea was to make things easier for the user, not harder.
So, to effect this, I introduced many redundancies in regular expression syntax that have come to be a large part of what nowadays are known as Perl 5 regular expressions, even when they're being called from Java or Python. Yes, in Perl \d means the same thing as
[0-9]. So what? I can say "digits" in English to mean 0 through 9, without saying "0 through 9" every time. Why not in Perl too?
Moderation is good, in theory.
If you want to have a thriving open source project, then you must build a culture around it; and to build a culture, you must encourage cultural identity. That is, you must encourage a certain amount of immoderation. Call it "religion" if you will, though I think religion has gotten a bum rap too. After all, immoderation on behalf of a good cause is how saints are made.
Hence, we were immoderately evangelical about Perl. In particular, because we wanted to build a new culture, we had to pull people in from many different cultures, which oddly enough meant we had to avoid being classified as a culture ourselves for a little while.
Consequently, I refused to create a newsgroup for Perl for a long time because I wanted people to talk about how to solve their shell problems in Perl, and I didn't want Perl to become ghettoized right off the bat.
So when people would ask in the shell newsgroups how to do something, we could give them the "how to do it in Perl" answer without enduring chants of, "Take it to the Perl newsgroup!" In this way, we subverted Usenet.
(Later on, and for similar cross-cultural reasons, I started scanning my entire Usenet feed for Perl references -- I basically invented my own form of Kibology (http://www.kibo.com/) about the same time as Kibo came up with his religion of the Internet, but he got to name it. Drat. Perhaps it's just as well. "Wallogy" just doesn't quite have the right ring to it, does it?)
Anyway, I did some sneaky things to make sure Perl developed a healthy culture. While we took ourselves very seriously in some ways, we also tried to laugh at ourselves occasionally. Perl not only stands for the Practical Extraction and Report Language, but it also stands for the Pathologically Eclectic Rubbish Lister.
Anyone who can't laugh at himself is not taking life seriously enough.
PERL 3
The Child
Perl Larry Wall
Striking A Subversive Pose: Larry Wall.
I remember that when I was a child, I thought there was nothing I couldn't do if I tried hard enough.
Perl 3 was when Perl started trying harder. The early design goal of optimizing for the common case was okay as far as it went, but it didn't say anything about what to do for the uncommon case. One common solution is to say you're not going to do anything about it, and in fact, that's what I once said about binary data, based on the precedent of many Unix text processing tools that could not handle binary data. When Perl 2 was out, I said to myself, "Perl is just a text processing language. If I make Perl handle binary data, who knows where it will stop?"
Well, Perl now handles binary data, and who knows where it will stop? That's the rub -- people don't want Perl to quit just because the going gets tough. So the earlier design goal evolved into our current goal: "Easy things should be easy, and hard things should be possible." Sure, Perl is mostly a text processing language, but every time we add the ability to do something else, we make a way to solve gobs of problems that are 90 percent text processing and 10 percent something else.
Another way Perl 3 tried harder was by adopting the GNU General Public License, known as the GPL. (The copyright statement on the first two versions was woefully inadequate.) More on the GPL later...
I've been talking about the advances that marked major versions, but you should realize that most of the development of Perl was not at the version boundaries. Development was continuous between major versions, and the typical patch file would contain bug fixes mixed together with enhancements.
It's a sign of the robustness of the design of Perl that people didn't rebel at using a development version for all their work. In large part, this was because Perl has always had an extensive set of regression tests, and users could be confident that, even if they installed a new development version, at least the functionality tested by the regression tests was guaranteed not to break. And that was good enough for many, because most Perl scripts don't use the fancy stuff. I take it as a compliment that many people still use Perl 4, because that means I put a lot of the right stuff into Perl early.
PERL 4
The Preteen
It's an odd thing, but nothing much changed between Perl 3 and Perl 4. Or, I should say, between the end of Perl 3 and the beginning of Perl 4. Just as it's hard to say when a child becomes a preteen, the transition from Perl 3 to Perl 4 was rather arbitrary. What really happened was that we wrote the first edition of the Camel Book, and then changed the version number on Perl so that we could say that it documented version 4.
But there is actually a qualitative difference between how a child thinks and how a preteen thinks. There comes a time at about the age of ten when suddenly a child doesn't just think about things, but starts thinking about thinking about things. So I guess you could say that producing the Camel Book was our version of thinking about thinking about Perl. Certainly it helped a lot of people understand more about how I think about Perl. Or at least, about camels.
People keep asking me, "Why a camel?" And I have a standard litany of answers. (I'm allowed to give more than one answer, of course, because of the Perl Slogan: "There's More Than One Way To Do It!") But the fact is, there's no single left-brained answer. If I'd wanted a left-brained animal on my book, it would have been an oyster. But I wanted a right-brained animal, and the camel was it for various reasons:
A camel is ugly but useful; it may stink, and it may spit, but it'll get you where you're going.
A camel is self sufficient in a dry place. You don't need to take along food and water for your camel. You don't need a toolkit to keep your camel going. You don't need tracks or roadways, or pipes to other processes.
A camel is a horse designed by a committee. Or at least it looks like one. But appearances can be deceiving, and a camel is well adapted to its ecological niche. So is Perl. Despite the fact that it is designed by a committee.
Camels have vague Biblical associations with caravans and treasure, such as Pearls of Great Price, not to mention Pearls of Wisdom, and not to mention Pearls before Swine. Or the Pearly Gates.
Finally, no animal has a better attitude than a camel. Presuming you're looking for an animal with an attitude.
Frankly, I wanted a camel because it was countercultural. Camels aren't modern, therefore they have to be either premodern or post-modern. Maybe even both. Fortunately our postmodern culture is based on the happy contradiction that any decent culture must be countercultural. If you can't deconstruct your own program, you aren't really with the program.
Another way to look at the Camel Book is that it was a kind of bar mitzvah for Perl. Perl wasn't a real grown up language before it had a book. I remember being shocked the first time I was told that half the desks on Wall Street had a Perl book on them. I shouldn't have been. The book is what legitimized Perl in the eyes of many.
Another thing that helped legitimize Perl was the addition of the Artistic License to stand beside the GPL. Perl 3 used only the GPL, but I found that this didn't do quite what I wanted. I wanted Perl to be used, and the GPL was preventing people from using Perl. Not that I dislike the GPL myself -- it provides a set of assurances that many hackers find comforting. But business people needed a different set of assurances, and so I wrote the Artistic License to reassure them.
The really brilliant part was that I didn't require people to state which license they were distributing under, so nobody had to publicly commit to one or the other. In sociological terms, nobody had to lose face, or cause anyone else to lose face. Most everyone chose to read whichever license they preferred, and to ignore the other. That's how Perl used psychology to subvert the license wars which, as you may or may not be aware, are still going on. Ho hum.
Yet another thing that helped legitimize Perl was that there was a long period of stability for Perl 4, patch level 36. The primary cause of this was that I abandoned Perl 4 to work on Perl 5.
PERL 5
The Adolescent
Shortly after a preteen starts to think about thinking, the raging hormones enter the scene, causing a total brain meltdown and, hopefully, eventual reconstruction.
For Perl, the meltdown happened because I decided to follow the rule: "Plan to throw away your prototype, because you will anyway." Perl 5 was nearly a total reorganization. I have in times past claimed that it was a total rewrite, but that's a bit of a stretch, since I did, in fact, evolve Perl 4's runtime system into Perl 5's. (Though if you compared them, you'd see almost nothing in common.) The compiler, though, was a total rewrite.
You only get one shot at raising your kids. Similarly, I figured that this was my first and last chance to raise Perl right. So I thought of all the buzzwords I wanted Perl to be compliant with, and heroically set out to make it so. By and large, I think I succeeded. As with the typical adolescent, Perl 5 is significantly, um, sexier than Perl 4.
Among other things, Perl became more readable and more extensible. It also became more unmanageable, not because it was a bad design, but simply because it was growing up, and had too many interests of its own. With Perl 5, I began to realize something I was learning with my kids: I had to let them learn to make their own way in the world. Not that I don't still give them all a lot of advice. But the process of letting go is gradual; if you let go all of a sudden, they tend to land hard.
One manifestation of this was that I had to learn to delegate most of the work of Perl development and documentation to other people. We've developed a system of "pumpkin holders" who are responsible for the various aspects of Perl development and maintenance.
But that's not good enough. I also realized that, while the open source community is good at some things, it's lousy at other things. If Perl was to be useful to as many people as possible, I had to learn to delegate some things to the business community as well. In particular, we really needed to have a commercially packaged version of Perl for the Windows folks, because many of them were (and still are) clueless about open source. It's almost like we're doing Windows users a favor by charging them money for something they could get for free, because they get confused otherwise. (This is Linux Magazine, so I can get away with saying this, right?)
But beyond that, I was looking around for someone with some business sense to cooperate with, when lo and behold I found out that Tim O'Reilly (as in O'Reilly & Associates, my publisher) was having the same ideas about establishing a more symbiotic relationship with the open source community.
Tim is a class act. He's also a bit of a rarity: a brilliant (but not greedy) entrepreneur. His slogan is "Interesting work for interesting people." We hit it off right away, and Tim offered to pay me to take care of Perl, because anything that is good for Perl is good for O'Reilly. And from my perspective, lots of what O'Reilly does happens to be good for Perl.
But it goes beyond even that.
Tim and I both felt that there was something larger than Perl afoot. Free software has been around in various forms as long as there has been software, but something new was beginning to happen, something countercultural to the counterculture.
The various open source projects were starting to realize that, hey, we aren't just a bunch of separate projects, but we have a lot in common. We don't have a bunch of separate open source movements here. We have a single Open Source movement -- albeit one with lots of diversity of opinion as to how best to move the bandwagon forward to where more people can hop on.
In short, our counterculture was beginning to count.
When Tim hired me on three years ago, that was very much on our minds. We were preaching it a year before Netscape released the Mozilla browser code under an open source license.
Of course, both Linux folks and Perl folks recognize that Linux and Perl were here long before the bandwagon, and will (hopefully) be here long after the bandwagon has broken an axle (or more likely, has been hijacked). But in the meantime, both Linux and Perl have been growing up, and they are now mature enough to understand the utility of a bandwagon in inciting a revolution.
Frankly, I'm glad that Perl isn't competing against Linux. Quite the contrary -- now that both Perl and Linux are growing up, Perl is going out with Linux. I suppose that, as the father of Perl, this should make me nervous, but it doesn't. They have my blessing. May they prosper. May they have many happy years together. May they be fruitful and multiply. And dominate the world.
