Loading
- The Borderlands 2 Gun Discarders Club
- The Borderlands Gun Collector's Club
- Hacker News Fires Steve Yegge
- eBay Patents 10-Click Checkout
- Haskell Researchers Announce Discovery of Industry Programmer Who Gives a Shit
- Wikileaks To Leak 5000 Open Source Java Projects With All That Private/Final Bullshit Removed
- Blogger Finger
- A programmer's view of the Universe, part 3: The Death of Richard Dawkins
- Have you ever legalized marijuana?
- Story Time
- A programmer's view of the Universe, part 2: Mario Kart
- Fable II: Arguably Better than Getting Your Head Crapped On
- Ejacs: a JavaScript interpreter for Emacs
- A programmer's view of the Universe, part 1: The fish
- The Universal Design Pattern
- The Bellic School of Management Training
- Programming's Dirtiest Little Secret
- Business Requirements are Bullshit
- Done, and Gets Things Smart
- Rhinos and Tigers
- Dynamic Languages Strike Back
- XEmacs is Dead. Long Live XEmacs!
- Settling the OS X focus-follows-mouse debate
- js2-mode: a new JavaScript mode for Emacs
- Four console games you might like...
_This is basically a review of, and a pros/cons rant about, Borderlands 2. If youre not into it, just dont read it! Ill write about stuff you like some other time. Maybe._
So!
Im not the kind of person to say "I told you so." Noooo. Never. Well, never, _unless_, of course, I get to say it loudly, within hearing of a biggish stadium full of people. Which I can.
So here goes: I TOLD YOU SO. Toldya toldya toldya.
My predictions from my previous post, "The Borderlands Gun Collectors Club", all came _completely_ 100% true, with Hyperionesque accuracy, Jakobsian impact, Maliwaney inflammatoryness, Tedioric blasting and surprisingly, even Vladofish speed. I made out like a Bandit.
I predicted, as you may recall, that (A) itd be a great game ("duh"), (B) theyd screw up the token economy because they only partly understand it, and (C) as a direct result of B, players would gradually head back to Borderlands.
Three weeks after the release, I had my dreaded first "I really dont want to throw this gun away, but I have NO GODDAMN ROOM FOR IT, THANK YOU RANDY PITCHFORK" gun-discarding experience. And my reaction was, predictably, to think seriously about either creating a mule character or going back to play BL1.
I mean, I knew Id have this reaction, but I failed to predict how amazingly fast it would happen. A week playing the game, another week on playthrough 2, a final week finishing all the optional side quests, and then boom -- the farming is fundamentally broken, so lets go play something else. But I dont _waaaaant_ to! Why did they have to get this wrong? Why did I have to be so predictively correct? Argh!
Let me make this really simple and clear. You remember that famous exchange in Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb:
DR. STRANGELOVE: Of course, the whole point of a Doomsday Machine is _LOST_ if you KEEP it a SECRET! Why didnt you tell the world, EH?
AMBASSADOR DE SADESKY: It was to be announced at the Party Congress on Monday. As you know, the Premier loves surprises.
Well, if Dr. Strangelove were alive to play BL2 today, hed have said:
DR. STRANGELOVE: Of course, the whole point of 87 Bazillion Guns is _LOST_, if you DONT LET PEOPLE KEEP THEM! Why didnt you add more bank slots, EH?
I mean, at least Ambassador de Sadesky had a somewhat plausible excuse. But Gearbox has been thinking over this whole endgame-farming thing for, oh, probably eight years or more. How the hell did they arrive at the conclusion that "We should have 87 bazillion guns, and you personally should be able to keep, like, twelve of them!"
Theres only one possible answer: they are clueless. I mean, dont get me wrong: theyre also lovable, brilliant, passionate, technically astounding, and outright _visionary_. But theyre also and bumbling and clueless. Theyre the neighborhood kid who catches a lizard and thinks its really cool, and it *is* cool, except he puts it in a box and it dies.
Ill do this as a Good, Bad and Ugly post, just so you know its, like, balanced. If I were just gushing a bunch of fanboy praise, you know as well as I do that it wouldnt be as credible. You have to hear the bad with the good.
But ugh, they were so _close_. So close! The game is so amazing!
Maybe theyll cut my second prediction in half, and release a DLC or patch within 6 months that gets the folks who drifted back to Borderlands to start collecting in BL2 again.
That, or maybe Ill contribute to the BL2 player-file-editor project. Modding is stupid, stupid, stupid; its 12 year olds advertising that they are, in fact, mentally and emotionally really twelve years old. The mindset there is so juvenile that it pains me to admit that sometime in my distant past, decades ago, I probably would have thought that way myself. (That is to say: "Hey, Im going to _mod_, because its _not allowed_ so it must be _cool_, and even though a _fugging gorilla_ could figure out how to do it, and it takes any _hint_ of challenge out of the game and makes me look like I was _severely shaken_ as a baby, but Im going to go ahead and show off my modded guns as if Im some sort of _super_-gorilla." Yeah, that mindset.)
But modding will, to a very limited extent, help work around Gearboxs cluelessness by letting legit endgame farmers have _a place to put all their fugging produce_.
Very, limited, mind you. Ill tell you how it *should* be in the Ugly section. Ill talk about how to fix this situation, and, well, if Gearbox doesnt understand the fix, you can be sure some other upstart game company _will_ understand it, and itll all be the upstarts limelight soon enough. A few more years of screwing this up, and as far as Gearbox goes well be, like, "hey, remember Diablo?"
Lets hope they get the message. Randy, guys, please -- _get the message!_
THE GOOD -- NO, WAIT -- THE GREAT
Actually we should do Great first, then Good/Bad/Ugly. Because BL2 is a capital-G Great game. Best game ever? Well, no. Nobodys gonna top RDR for a while. But _dayum_, BL2 is a great game. Its actually a great test of whether youre an idiot, because if you dont like it or it doesnt appeal to you... well, you might not be an _idiot_ per se; theres bound to be _some_ explanation... I guess. In theory. There _could_ be some other reason than you being an idiot, however improbable.
Anyhoo, lets get The Big Question out of the way: Is Borderlands II better than Borderlands I? Well, the answer depends on whether you think The Empire Strikes Back was a better movie than Star Wars. Its also the answer to "Should I play BL2 if I havent played BL1?" If you think watching The Empire Strikes Back after Star Wars yields an acceptably awesome cinematic experience -- which it probably does -- then yeah, play BL2 first. Go for it!
The Lucasian comparison runs pretty deep. Borderlands is dry and dusty, has dome-like dwellings, introduces cute talking robots, features fully armored imperial bad guys with bad aim, and has a slow (but interesting) story arc up to a really dramatic finish in the last act. Whereas Borderlands 2 is lush and fast-paced and story-thick and incest-ridden and stuff, just like Ep. V. Lets just hope they dont carry the metaphor to the third installment, unless of course they want to have Patricia Tannis dressed like Princess Leia as the kept-plaything of some huge talking Thresher, in which case they have my blessing.
BTW, as an aside, and my wife agrees 100% -- all this chatter about Lilith vs. Maya vs. Moxxi is just outright silly. The answer is: Tannis. Followed, we think, by Helena Pierce, eye or no eye.
Anyway, where were we. Oh yeah, The GREAT. Where to begin?
The story is awesome. Burch was amazing. Tiny Tina is incredibly awesome. Other Burch, also amazing. Handsome Jack is so awesome that I found myself rooting for him most of the time. Threshers are way more awesome than they looked in the previews. The AIs are uniformly great, even when they occasionally make the mobs cower in the corner as if youre the Blair Witch. I dont mind. And the guns, oh the guns, they are beautiful and fascinating and a joy to behold.
The voice acting is awesome, with one noteworthy exception: Axton was mis-cast. He looks like Captain America (or Thor, or whoever, take your pick), and he possesses the competent, modest, sexily reserved flawed-hero look of Captain America (or Thor, or whoever, take your pick). But his voice and dialogue are pure Jack Black at his cheesiest. Oops. Oh well. But the rest of them are cool. Cant say much without giving the plot away, but everyones voice acting was great, and Tiny Tina stole the show. Well her, and the Goliaths.
The game balance is exquisite. THEre have BEEn some missteps, naturally, and people are now vying to slay the much-vaunted raid boss Terramorphous in the fewest number of milliseconds with 100% legit gear. But on the whole the balance is superb.
My only cause for complaint is that the game balance feels far less serendipity-prone than Borderlands often was. They made BL2 so balanced that most of the time the stuff you find is really pretty boring. No one manufacturer shines above all the others, nor is one worse than the others (though I wasnt much of a fan of Jakobs or Pangolin, by and large). All the weapon types are about equally good. It feels as if they tightened the loot-rarity bell curve so they could keep the difficulty progression smooth. The game always felt challenging, albeit without ever descending into Survival Horror territory -- there is always enough ammo around to encourage exploration.
And when you do find the occasional legendary item -- I found only four of them during my first two complete playthroughs -- it will last you a good ten levels. Oranges are the new Pearls. They got this right, I think, and only in the difficult-to-balance endgame did they encounter any issues. In short, the game is challenging in a good way.
They kept the cel shading. Yay. Cel shading helps them avoid the Uncanny Valley where most other games reside today -- they look more and more realistic without actually looking, you know, realistic. The Borderands franchise embraces the graphic-novel look, and its always stylish and fresh. Plus they dont cel-shade a lot of stuff: water, ice, atmospheric effects, weapon effects, and so on, which makes for some eye-popping moments. And just in case the poetic beauty of their rendering approach is lost on you, they also include some actual eye pops.
They kept the humor. Oh, did they ever. Id find myself giggling at 3am until I was snorting and wheezing. The humor runs the whole gamut, from the coarse and obvious to the surprisingly subtle. I love the Dr. Zed vending machines (and the voice acting, and Zeds character in general -- he may be my favorite.) And I almost lost consciousness from laughing when I realized, after my second weapon swap, exactly what The Banes curse was. Oh man, that one almost killed me. That whole mission was extraordinarily well-designed. And of course Claptrap is funny as always. Crazy Earl, too.
Not to mention the talking weapons and armor. I _love_ my Hyperion auditing sniper -- havent discarded it even though I cant really justify a precious inventory slot for it. Theres a lot of genuinely funny stuff in this game.
But I think the Best Humor award has to go to the bad guys. Handsome Jack has his moments, but its really the bandits that steal the show. Just when you think youve heard them say everything, theyll surprise you. Crazed psycho bandits running at you screaming that Pluto is still a planet, or complaining about how goddamn cold it is outside, or reciting Hamlet... it just never gets old. Bandit humor has quickly become one of the legendary defining hallmarks of the Borderlands experience.
No question about it: this game has it all. Its a huge, sprawling open-world game with an engaging story, superb balance, exciting game mechanics, outstanding writing, and absolutely unparalleled replay value.
The greatness of the game pretty much dwarfs everything else I say here. Its a worthy successor to Borderlands, and at this point its already become one of the most important franchises in gaming history.
I know the folks at Gearbox love this game and they want to keep refining it, though, so Ill weigh in my $0.02 on how it could be even more awesome next time around -- hopefully as soon as the next DLC.
THE GOOD
The game has some aspects that I feel _bordered_ on greatness without actually achieving it.
The biggest issue I have with the overall world design is that its a theme park. No other word for it. Its a _cool_ theme park, and I do love me my theme parks -- Im a Disney Vacation Club member and we go to theme parks several times a year, rain or shine. But there was some sort of dynamic going on, maybe an overreaction to the misguided criticism of the dry dustiness of BL1 (which is about as valid a criticism as saying "Star Wars had too much sand!"), that maybe made them overcompensate a little with the paint gun.
So even though the game is often beautiful, the colors are often too saturated. When they get it right, its nothing short of stunning. The Southern Shelf and Sawtooth Cauldron are standout examples. Both juxtapose bandit shantytowns with a rugged natural beauty -- but its a beauty with a relatively subdued palette, dominated by just one or two colors.
Some of the man-made places are gorgeous too -- Opportunity City comes to mind, and the Friendship Gulag. But they, too, have dominant primary colors or motifs that shape and define the visual experience into something unique and refined.
In several other locations they went a little overboard. Im not sure if its the cel-shading adding visual clutter (as seemed to be the case in Finks Slaughterhouse and in Sanctuary), or if it was actual clutter (Thousand Cuts comes to mind), or if they just went a little overboard with the fully-saturated paint gun (Wildlife Preserve, maybe, or Tundra Express). Whatever the reason, the game winds up looking a tad overdone in places. Still awesome, yes, but with colors that clash rather than harmonizing. They need to follow the basic color-matching advice from, say, Vogue or Cosmopolitan: Any three colors go together, and any more than that looks like a peacock shitting rainbows. Im pretty sure it was Cosmopolitan who said that.
The theme-park quality goes beyond the color scheme. A lot of the areas feel bowl-shaped and directly connected to other equally bowl-shaped areas with _completely different_ styling. So it feels a bit like youre walking from Adventureland to Tomorrowland to Fantasyland.
And theres a lot of... homage, lets say... to other games. It felt almost like they had other-game and/or other-movie envy, even though Borderlands is a game to be envied all on its own. So theres a dash of Skyrim (The Highlands), some Red Dead Redemption (Lynchwood), some Jurassic Park (Wildlife Preserve), and other maybe-unnecessary tributes. And Sanctuary reminds me way too much, ironically, of the towns in Rage.
So the overall world design lacked a certain cohesiveness of vision that was present in Borderlands I. It feels on the one hand like they were trying to elicit a Tolkeinesque or Homeresque journey from humble beginnings, increasing in scope, and ultimately walking into the heart of Mordor. Theres a teeny bit of that going on. But it also feels like whoever had that vision was crushed by the weight of game directors all clamoring for unrelated themed areas to show off their... their what, I dont know. Just to show off.
On the whole, though -- coming from a guy who likes theme parks -- they did a really bang-up job of creating a theme park. The individual areas all have their own distinct personality. Some of them even have world-class atmosphere. The Fridge, the Bloodshot Stronghold and Ramparts, Overlook and the Highlands, and several other areas are really memorable. And the Arid Nexus Badlands were... well, thats my favorite area of the game overall, for reasons I cant go into, but wow.
My vote for Overall Best Area Design, though, goes to the Caustic Caverns. This area stood head and shoulders above the rest of the game, in the sense of being _new_ -- who the hell has ever seen anything like that before? -- and _creepy_. I can only remember one or two times in my 35-year gaming history where I felt the sinking "I am on the WRONG side of the train tracks" feeling that I had upon entering the Nether Hive. The whole area gave me a new-found respect for -- and dread of -- the Dahl Corporation, whom I hope will be the villains of some upcoming installment. Oh, and the, uh, mission I cant give spoilers about, but it takes you to the top floor in the Caverns -- that was hands-down the best side quest of the game.
My third favorite location, after the Badlands and the Caustic Caverns, was Lynchwood. Im a sucker for that sort of thing. It didnt make any sense AT ALL -- it was a gratuitous anachronism in a game that thrives on anachronisms. But I loved it. Robbing the bank and getting out of town before the posse came: that was straight-up inspired writing. I loved the Lynchwood mini-boss and that whole plot line; I loved the Marshalls announcements; I loved the whole thing. Lynchwood may not have made much sense in the larger story, but it was unquestionably awesome.
Anyway, lets face it: the game is a theme park. Not that this is bad! Its Good. But Id argue that its not Great. I think true greatness necessitates a uniformity of vision that admits no room for tongues planted too firmly acheek. BL3 is going to have to make some hard choices about whether to be good or great.
Good is OK, though. Nothing wrong with Good.
THE BAD
Ill try to keep this short. Mostly this is stuff that could be addressed in a straightforward way in a patch or DLC.
There was no explanation as to why NPCs dont get to use the New-U stations. Just sayin. Theyd better retcon that in next time.
No in-game explanation of the Golden Key chest in Sanctuary, so I (like half the rest of the civilized world) used both my golden keys right away without realizing what they were. In retrospect I dont think it matters, since having awesome weapons is probably more useful early in the game than later on. But it should have been a conscious choice, and I, like half the rest of the civilized world, was pretty pissed off to find that Id squandered my keys without so much as a warning dialog.
They changed it so you cant open the menu if youre not on the ground -- that is, when youre jumping, or falling, or climbing a ladder, or being flung through the air by external forces (e.g. geyser, grenade), or stuck atop an enemy you had the misfortune to land on. This is hugely screwed up, so I can only imagine they did it as a last-ditch workaround for a no-holds-barred showstopper Christmas-wont-happen bug, and itll get fixed in an upcoming release. That, or they hate their customers and think theyre scum. Time will tell.
This change did help me understand that one of the habits Id truly come to enjoy in BL1 was jumping and then opening the menu while in mid-air. Seriously. It was fun. I did it on purpose, all the time. I cant really articulate why, but it was exhilarating. Its as if they took away my childhood with that one simple dick move. I sure hope it was a last-resort thing that they plan to fix.
Inventory management has taken a turn for the worse overall. Yeah, it looks slick, but when has Gearbox ever been about "looks slick" over playability? I mean, no cutscenes, right? (Or at least no cheesy prerendered ones -- they do all their cuts right there in-game, and you can usually walk away from them.)
In BL2 they put a ton of work into the look-and-feel of inventory management, but they failed to nail the usability. In an RPG, even a quasi-RPG like Borderlands, inventory management is all-important. So maybe its their shooter background at work here. I dunno. But there are a lot of serious wtfs going on. Examples:
* When you want to compare item A to other items, and you eventually navigate to item B, then close the lets-compare transaction, it leaves the selection on item B. Last I checked, this is not the way rational thought worked in any product designed by human beings with good intentions.
* You can mark items as "favorites", and then... nothing. You cant sort on them or do anything useful with them. But, alas, you CAN sell them, without any warnings or indicators that you just sold an item youd marked as a favorite. So I have accidentally sold some really, really important shit, and only realized a few areas later, when it was too late to go back and buy them back. This has happened at least four or five times in my so-far 2.5 playthroughs of the game. Thats too many for an experienced gamer. It means they have a UI problem.
* You can mark stuff as "trash", and sell it all at once. Except thats stupid. Everything should be trash by default. Most of the items you pick up ARE trash -- that is a natural outcome of the tightness of their rarity bell curve. If they had done the whole favorite/trash thing correctly, youd only need to think about it at all when you picked up a blue-or-better weapon, at which point you could mark it as a favorite to prevent accidental sale. This, friends at Gearbox, would be less error prone AND less effort. Argh.
* Theres still no way to "buy all" for ammo. Also, like in BL1, there are two concurrent views of your ammunition while youre shopping: the stores selection and your inventory levels. And, like in BL1, the two have unaccountably different sort orders. So as you move the selection cursor down the stores selection, the inventory cursor jumps around unpredictably. I cant believe they did this two games in a row.
* Unlike in BL1 (I think), when youre buying ammo by mashing buttons (because theres no bulk-buy function), you can _very easily_ scroll past the grenades and start buying shit you didnt want while youre mashing the buttons. Again, no warnings, no "are you sure?", so its really easy not to notice until a few load levels later.
* Like in BL1, they dont sort insta-health at the top of Zeds vending machines, which means if you run up to a machine to buy health in a firefight -- which is much more commmon now that theyve eliminated portable health vials -- or if youre just not paying very close attention because your dog just knocked over your glass of water, then you stand a good chance of buying some expensive class mods and maybe not noticing. Ive done this too. In all seriousness, sorting insta-health at the top is OBVIOUS, so only gross negligence can explain how it was done wrong two games in a row.
To be sure, they got a few inventory-management things right that were messed up in BL1. You can now compare items while shopping -- w00t! And the weapon cards show all the data rather than truncating. The item sorting makes a little more sense. The "examine this item" is REALLY cool, and I love just zooming and panning on my items to marvel at the intricate designs. But on the whole it was a step backwards, and it makes me very sad.
Other bad stuff... lets see. They still only let you quick-wield 4 weapons even though modern games all give you 8 slots on a wheel. In a game like BL, with elemental resistances and radically different opponent AIs, 4 slots just isnt enough. You need to be able to carry at least two different "weapon builds" with you. I dont care if we have to purchase them or work our way up, but we need more than 4 equipped-weapon slots. As things stand, swapping out weapons interrupts the otherwise smooth game flow and makes it sort of a drag. _Especially_ when they dont let you open the menu mid-air. Jesus. How can the graphics be so beautiful, and the story so awesome, and the combat so smooth, but the inventory management is so screwed up? Is it different teams? Whats going on here?
Lets see, what else, what else... oh yeah. On the PS3 version, every time you set your controller down it triggers a nuclear explosion. No, really. Well, it does if youre playing Axton with the middle skill tree. Its a side-effect of having switched the ability/grenade buttons with the zoom/fire buttons. I havent made up my mind on this one; overall I think they probably made the right choice, but it reminds me of the Fable II days when youd try to buy something from a blacksmith, hit the wrong button, destroy his house and send everyone screaming from the village for hours. Its not really ha-ha funny, at least not at the time.
Their bulk-vacuum function still sucks, so to speak. Actually the "interact with stuff" button hasnt changed behaviorally since BL1 in any significant ways. It still has all the old problems, and maybe some new ones.
For starters, they still have the horrible misfeature that holding the "pick up" button, which is used about 87 bazillion times per game session for bulk vacuuming, has different behavior if you do it on a weapon. What it does in that case is _grab it and wield it_, even though 9999 times out of ten thousand, the weapon in question is a piece of loot-crap that destined for a vending machine. Way to optimize for that 1 in 10,000 case, Gearbox. Moreover, way to keep it around for game 2.
The vacuum button still does a piss-poor job of actually vacuuming. And theyve added "auto-vacuum", which does an equally piss-poor job of auto-vacuuming. I cant tell you how many times Ive been standing there with 4 hit points, examining the item-card for a health vial on the ground, obscured only by the "pick up" text because its within reach, thinking "um, why am I able to read this?"
They really need to fix it so that every replenishing item in a ten-foot radius from your character automatically zooms to you no matter what. Otherwise it devolves into a guessing-game as to whether their algorithm will be smart enough, and of course when the gameplay is fast and furious, you have to guess conservatively -- which defeats the entire purpose of having the feature. Picking stuff up -- heck, being _near_ anything, degenerates into a button-mashfest.
And unless its just my imagination, it feels like the pick-up button is _less_ responsive than it was in BL1. When you open a chest, there is a nontrivial window during which the items _appear_ to be grabbable, but pressing the button has no effect. You have to jab at it for up to half a second, maybe a second before the game says "aw fuck, thats right, I told them Pick Up and theyre pressing the button, so maybe theyre actually trying to, you know, pick that shit up."
Is it really that hard to detect that the button is already down, once the items are actually grabbable?
And of course the whole cycle gets repeated twice per container, because the game is just as likely to ignore your button-press to _open_ the container.
The last "Bad" line-item Ill whinge about is that although the game seems really generous about accuracy, theyre real bitches about who died first, when you and the last nearby enemy expire at the same time. To illustrate how forgiving they are overall: you can be using a sniper from a thousand yards away, and pull the trigger when the cursors kinda pretty far away from the mobs head, and itll explode way more often than probability would dictate that it should. Very gratifying! No complaints here! And theyre also really nice when it comes to landing jumps that you didnt quite hit, unlike in many other games. In general the game is pretty forgiving about controller accuracy.
But if you die "at the same time" as an enemy (i.e. it happens within ~100-200 ms before or after), you go into a fight-for-your-life bleedout. Sometimes it seems very, very clear that the enemy died second, but the game didnt actually realize it, and penalizes you. It seems unfair. To avoid any suspicions of stupidity on the part of the detection algorithm, it would be nice if theyd give you a half-second window AFTER the last enemy dies before your own death results in a bleedout. Hell, even 300ms would be nice. There are already plenty of legitimate situations for aggravating bleedouts -- the classic one being when the enemy shoots you and then walks around the corner. So I dont think itll cause a balance problem to add the short grace period Im proposing.
I know for a fact that there are some issues with the code that detects whether an enemy is dead. Several of the missions have resulted in me sitting around for a long, long time (several minutes) after the mission was obviously over, except the game couldnt figure out that it was over. Examples include the last round of the Natural Selection Annex, where I just wandered around the arena hoping the game would finally notice Id won, and the plant-the-flag Sawtooth mission, where _twice_ the last enemy disappeared minutes before the Slab King noticed I was victorious.
So I suspect theres a race condition here, in which you can legitimately die _before_ the last opponent, but the game doesnt notice, and you bleed out while shaking your first at the unfairness of it all. Why go there? Just put in a short grace period, and make sure its really really clear that you died _after_ the opponent -- often from a long-fused grenade, Ive noticed. Then theres no cause for questioning the game code itself, which undermines player confidence in the fairness and quality of the engine.
Thats about it for the Bad. Inventory management woes, no menu while jumping, bulk-vacuum issues, and bleedout race conditions. Thats pretty good, all things considered. Why not just fix them all in a patch, and make it perfect?
THE UGLY
Theres only one Ugly in BL2, and its a big one. The Ugly is that for no reason whatsoever -- negative reason in fact; its flat-out anti-reason -- they dont give you enough bank slots to make farming fun for more than a few days.
They put an _astounding_ amount of effort into the endgame mechanic, folks. This was not some casual design thing for them. They put in hooks for new raid bosses, tons of one-off unique legendary weapons with custom artwork and code, and a plethora of design decisions to prevent any one raid avenue from dominating the endgame. They made it so that every one of the dozens of bosses and mini-bosses has its own legendary that it can drop, so that farming is distributed across most of the locations in the game, which breaks up the monotony. They even added formal item-twinking across characters, amazingly enough.
But for all that, its fundamentally broken. And whats more, they have this huge, gaping problem with a substance called Eridium (Im sure theyre sick of hearing about this by now), which lets you buy a limited number of carrying-capacity upgrades, including bank slots. So players are already simultaneously crying out for an Eridium-sink and more bank capacity. I mean, they should have seen this coming months ahead of their code freeze. It would have taken maybe 3 days of engineering and testing effort to make it so that Earl could sell you increased bank capacity at usury rates, even a geometric progression. And it would have been fine. Everyone would have been satisfied.
Heres the thing, though. Its not just about capacity. If Gearbox wants to do this Right, by which I mean pull their heads out and do something that nobody in the game industry has ever done before, what they really need to do is give players a database.
Thats what we want, really. You make 87 bazillion guns, and let us collect them? Well then were going to want hundreds and hundreds, maybe thousands of guns in our collections. Not twenty, or whatever stupidly low number youve given us. That just spawns modding and mule characters and leaving the game altogether -- any outlet from the collection pressure; players will use them all.
What BL1 needed was a way for you to effectively manage a collection of a thousand guns. What if you want to look at all your Mashers? Or all your weapons by type, or by elemental damage, or by manufacturer? Im not asking for a data warehouse here, or for some fancy text-based console-query UI. I mean, *_I_* would use it, but obviously we want to keep this mainstream.
If you start by formulating the basic problem as: "How do I manage a collection of a thousand guns," then your UX guys should be able to come up with something acceptable. No -- you know what? Fuck acceptable. They should be able to come up with something _awesome_, something in keeping with the innovation and forward-looking badassery that weve all come to associate with Gearbox and Borderlands.
Ironically, BL1 was better at this -- a LOT better. Of all the inventory "improvements" introduced in BL2, the only one that improves gun collecting as a hobby is the "examine this gun in 3D" feature.
I imagine Im going to do exactly what I (and everyone else) did in BL1, which is to figure out how to modify the bank-slots and inventory-slots counters in the player save files, and hope like hell that you guys can actually scale up to something reasonable without crashing or locking us out or triggering some other godawful poison-pill.
But just having a lot of slots is only a tiny part of the picture. Gearbox has created a gun-collectors game, but they havent given us a way to collect guns. How messed up is that?
I think its pretty messed up.
All this talk about BL1 has given me a major case of nostalgia. I love BL2, but I think I need to kill some Drifters to pull me out of this funk. I remember when I finally reached the point with Brick where I could walk around the sand dunes and mow down drifters -- on foot -- and live to tell the tale. BL2 doesnt have any moments like that, not yet. Nothing you had to work for like that, anyway. It took _months_ of gun collecting before I was that badass in BL1.
And I remember looking through that sepia-tinted window on entering T-Bone Junction, that window filled with promise of adventure, seeing those rowboats suspended over a forty-foot drop to the salt sand, with the scorching wind blowing the makeshift wind socks tied to the Lucasian architecture. I remember hearing Knoxx give his reports to Admiral Mikey, Mr. Shank asking if I thought I was being _stealthy_, Athena barking her ludicrous military-speak to me -- a merc -- and Thirsty the Midget asking if I could turn the power back on in the Brandywine.
I remember. And I think its time to head back. I knew this would happen. I knew Gearbox would screw us on the gun collecting, and I knew sooner or later itd be back to Knoxx and the Armory and Crawmerax.
I just didnt realize itd happen so _fast_.
Sigh.
"Ass Effect 2"
Ah, me. Anyway, Borderlands has good writing. Or as the internationally celebrated and occasionally intelligible game critic Ben "Yahtzee" Croshaw of "Zero Enunciation" fame put it, "unnecessarily good writing". Ben evidently didnt care for Borderlands, but then again he didnt make it to the endgame. Which is good for the rest of us, because now he can find time to shit all over Mass Effect 3. At least I hope he does.
_
Ben and his friends playing Borderlands_
So where were we -- oh yeah, cutscenes. Gearbox knew they had a damn good game right out of the starting gate, so they didnt need to try to bluff up an artificial sense of moneys-worthiness by padding it out with massive cutscenes. So the few cutscenes they DO have are all (a) short and (b) full of awesome. As it should be.
Dont get me wrong. Borderlands isnt perfect. There are precious few games in history that can make that claim. And heck, theres nothing wrong with a few imperfections. They can sometimes give a game more character! In fact it is in precisely that fun-loving spirit of character-inducing imperfection that Borderlands features several truly colossal vaginas. Im not just talking about the guy who implemented their driving physics, either. Good guess though.
But whatever. NONE of the awesome qualities or quirks of Borderlands really matters in the long run.
In the end, above all else, Borderlands will be remembered as the ONLY single-player title since the Diablo games to capture the fun of a Diabloesque loot system. For almost ten years people tried and failed, and then Gearbox finally came along out of nowhere and got it more or less right.
IS BORDERLANDS AN RPG?
Borderlands claims to be part shooter, part RPG. The RPG claim was IMO a minor marketing mistake, since people have preconceived notions about what it means, and Borderlands isnt an exact match. In fact I didnt play the game when it came out because I couldnt figure out what kind of game it was. It was only when Id completely run out of stuff to play -- I, Chazmina, that is, and totally not that Stevey guy whos busy doing stuff for you -- that I started playing through old "Game of the Year" titles looking desperately for something that didnt suck. And it took me a while to see why they felt they could get away with calling it an RPG.
It turns out that Borderlands has classes and skill trees and skill points and experience points and levels and specializations and support for parties of adventurers ("vault hunters") with mixed and complementary skillsets. With that in mind, its easy to see why 2K marketed it as part-RPG, knowing the game like they did.
But it might have been better to let the critics and players arrive at that conclusion, because its missing many of the other elements people have come to expect of RPGs -- elements such as "the decisions I make affect the plot outcome", "I have meaningful, stateful, persistent interactions with individual NPCs", "I put on my robe and wizard hat", "People mock me in real life", and all the other things weve come to expect from role-playing games and gamers.
I think they might have done better initially by just marketing how fun it was. Instead it took a slow-burning word-of-mouth campaign before the sales really took off.
YOU MENTIONED "FUN"? I LIKE "FUN".
Borderlands is huge on the fun factor. You hardly realize it as you play it the first or even the second time, but the team at Gearbox put a lot of stock in FUN.
For comparison, just look over at Rage, an Id Software title that came out last year. Rage is named for the emotion that new players feel when, after an hour of gameplay, they die for the first time and discover the game has no auto-save system. Rage (lets be honest here) copied a lot from the Borderlands crib sheet. Or they tried. But unfortunately all Id knows how to do well is graphics. So of course Rage has startlingly high frame rates and graphics that are more realistic than looking outside your basement window. As you play it youre all "wow man this is... uh, very _real_" as you play it. But it winds up being a disappointing (though gorgeous) slogfest.
_
Almost as pretty as Red Dead Redemption_
Which is no surprise, since Id lost their divining rod for "fun" many, many years ago. You wind up playing through the game as a chore, out of nostalgia or professional respect. And today, just a few months after its release, nobodys playing Rage anymore. The fun it offers is ephemeral: typical fire-and-forget mediocre-shooter fun. Its really sad to watch Id sinking into irrelevance. Maybe they should just focus on selling their engine. Rage offers nothing at all in the addiction department, so if they were trying to mimic the success of Borderlands they did a piss-poor job of it.
A _Token Economy_ is any system in which you are awarded meaningless but highly visible "tokens" for Good Behavior -- that is, for the behavior the creator of the system is trying to provoke in you.
TRUE BUT APPARENTLY LITTLE-KNOWN FACT: Token economies are among the most powerful drivers of human behavior. Theyre used in grade schools, prisons, mental institutions and the military to incent people to act in certain ways. And it works, boy howdy does it ever work. Once they start handing out those gold stars, youd shoot your own grandmother to get one.
Some companies think they have the whole Token Economy thing figured out, so they create a Badge system or (equivalently) a Trophy system. Badge systems are what stupid people do when they think theyve figured out Token Economies.
Hey, dont shoot the messenger here. Im just reporting facts. Its what Im known for.
TOKEN ECONOMIES NEED *SCORED* TOKENS. Thats why badge systems are lame. They can never generate the addictive pull because theres no high-score list possible, other than the overall badge count. The count itself can be reasonably addictive if there are enough players -- think "number of Facebook friends". But thats Weaksauce Flavored Sauce Substitute _(Note: contains no actual Weaksauce)_ compared to the addiction levels achievable by having multiple token categories, multiple high-score lists, and a tight bell curve for token rarity.
Some token economies let you purchase the tokens at a high cost. Some token economies even have physical tokens. Disney understands this. So does Luis Vuitton. Scroll through a few pages and try to figure out where those prices are coming from.
Token economies are fragile. If Billy breaks into the teachers desk after hours and starts handing everyone fistfuls of gold stars, they become worthless and the economy collapses, irrecoverably. In high-end fashion handbag terms, counterfeit products threaten to destroy the token value. In game terms: game balance is hard, and getting it wrong can tank the economy.
All this is just another way of saying that RARITY CREATES DESIRABILITY. Its hardwired into the human brain. There are multiple complementary parallelizable exploitable ways of creating rarity. Most people dont get this, though, and consequently they create nifty products and systems that ultimately fail to achieve any kind of stickiness: creations destined to be nothing but flashes in the pan.
If you dont already know all this stuff better than I do, then you know fuck-all about creating addiction, and its no wonder your products badge system isnt generating adoption or stickiness or 7-day actives or any of that other shit youre measuring.
It irritates me to the point of boiling rage that I have to explain this stuff -- that the people most companies put in charge of mission-critical initiatives are so completely fucking clueless, to the detriment of their companies _and_ all the rest of us. So Ill stop here before I have a heart attack. You either get it, or you dont.
Gearbox gets it.
Well, sort of. I mean, its kinda hard to tell. They definitely get _part_ of it.
DISALLOWING JUMPING IS WHAT STUPID DESIGNERS DO
I need to relax a bit, so Im going to time-out here for an utterly incongruous digression. This section has absolutely nothing to do with the rest of the post. But it has to be said.
_Jumping is fun_. Period. End of story. If playing your game involves manipulating a humanoid ragdoll in three dimensions, and it doesnt support jumping, then you suck. No, dont go pointing at Zelda. Zelda gets a bye because its *Zelda* for christs sake. But Zelda is un-fun exactly to the extent that it fails to support jumping, except off ledges which is kinda OK but not really true jumping.
Practically the first thing everyone tries in a game is jumping. If the game doesnt let you jump, then people enter a Fuck You mode that can be hard (possible, but hard) to overcome.
You kinda dont want your players to enter Fuck You mode. Just sayin. Yeah, Im going out on a limb here, but Ill contend that its probably a good idea _not_ to make a game that puts people in Fuck Everything About This mode. If youre not exactly sure what that mode looks like, well, it looks like the Acornfilms Dead Rising 2 review. Which I _heartily_ recommend watching in its entirety, but for the impatient the most relevant section is from 6:20-6:45.
If a game doesnt let you jump over a foot-high obstacle, then -- thats right, youve got the idea now -- Fuck This Game. It might be possible to recover and get people to enjoy it anyway, but youre working against a bad first impression. How fucking hard can it really be, game developers?
Borderlands (of course) lets you jump pretty high, on account of low gravity. In contrast with Rage, which lets you do this pathetic little fart-jump that accomplishes nothing except making you feel even more sorry for Id than you already felt -- and lets face it, you do feel pretty sorry for them.
Metroid -- now THAT was a game that let you *jump*. Metroid got a whole lot of things right. But right off the bat they got jumping right. You can jump _really high_ in Metroid. And thats before you find the mods that make your jumping really start to kick ass. Like Borderlands, the Metroid franchise focuses on FUN over REALISM, and on gameplay over lame cut scenes.
In fact Metroid is even better than Borderlands in some aspects, such as the important aspect of not springing a Surprise Vagina on you at the end of the game. (_No_, Samus doesnt count. Jeez people!)
And of course lets not forget Super Mario Galaxy, critically acclaimed as one of the greatest games ever created, and it was basically a feature-length exercise in fancy new jumping physics and camerawork.
Make no mistake: jumping *_puzzles*_ arent for everyone. Especially when the camera management fucking blows so hard that it singlehandedly sinks the game at review-time, before its even launched (hello and goodbye, Epic Mickey). Jumping puzzles are definitely not guaranteed to be slam-dunk in the Fun department. It comes down partly to personal taste and partly to execution quality in the games design and mechanics.
But _everyone_ likes jumping.
Im not saying games where you cant jump cant be cool. Im just saying jumping is fun. In case you care.
THE GUN COLLECTORS CLUB
Were back on topic! w00t!
Most of the people playing Borderlands today are COLLECTING GUNS. Theres no other good explanation for whats going on. The game isnt particularly social -- communication is highly limited without mics, which most players dont use. The mission replay value is modestly high, but theres only so much any game can do before familiarity and boredom set in. The only tried-and-true way to keep people coming back is with a token economy.
Even supposedly pure-social, non-gaming environments are based on token economies. This is true of every successful ecosystem -- even Facebook -- because people crave status and recognition, and those needs generally derive from actions and events that are countable and rankable. Most sites that have succeeded in becoming addictive have an obvious token economy: karma, or star ratings, or anything along those lines to encourage users to keep contributing content. Sites that dont weave at least one token economy into their fabric are left wondering why nobodys showing up to their party. Or more accurately, people show up but they dont see any reason to stick around.
In the gaming world, traditional RPG-style experience points (XP) are countable, so a lot of games have global high-score lists for experience. But the smarter designers divide up their lists by in-game demographics, geography and other differentiators so that even if you have no hope of climbing the global high score lists, you can be one of the best in your area or specialization. The more subdivisions the merrier.
Unfortunately Borderlands caps XP -- you stop earning it when you hit max level. So no XP addiction for YOU.
Instead, the primary addictive tokens in Borderlands are the guns. There are some other, weaker kinds of tokens in the game -- for instance there are collectable items dropped by Claptraps in the last DLC, and some of them are ultra-rare. So theres a small group of collectors off hunting those. But its not as addictive as hunting for guns for a number of important reasons.
Lets look a little closer.
INGREDIENTS FOR ADDICTION
Any game can _overlay_ any number of token economies. There doesnt have to be just one. You can create token economies targeted at every kind of player, in the simplistic Bartle Test sense of "kind".
If you want to hook in EXPLORERS, just keep track of visited areas, missions completed and other countable explore-ish actions taken. If you want to hook in the PVPERS, make a bunch of arenas, then keep track of a bunch of statistics about kills. (Um, or not, if youre Diablo III.) If you want to catch SOCIALIZERS in your web, keep stats on followers, likes/dislikes and all that happy social shit. For BUILDERS, keep stats on what areas theyve built and how popular they are. Etc. You rope people in by counting stuff that they like to do and reporting it somehow -- preferably to everyone.
Token economies also can be created from (or emerge naturally from) big, complicated rules systems that are heavy on memorization and light on deductive reasoning. Obvious examples include the National Football League, the Linux operating system, the underground music scene, the comic book scene, your neighborhood Bible-study group and other paper-and-dice RPGs.
Whenever a community framework of _any kind_ -- gaming, sporting, social, technical, whatever -- is based on a huge system of fiddly rules and trivia to memorize, it attracts MAVENS who derive satisfaction and status from their knowledge of the system.
Mavens create the pulse of a community. They set the beat. Some mavens generate new content. Others specialize in documentation. Some become critics and provide valuable reviews. Some become entrepreneurs within the economy and act as vendors, facilitators, go-betweens, fences or even thieves, depending on whats possible in the framework. Some mavens become hipsters and try to make the community seem more exclusive and prestigious by virtue of being insufferable (but ultimately valuable) dickheads.
A community only needs a small percentage of its members to be mavens in order to grow and thrive.
If you count and score peoples actions, and then stack-rank them in a set of high score lists, it sends the addictive pull soaring. Sometimes the stack ranking is even built right into the system. If youre a Freemason, then being a Master Mason is way better than being a lowly Apprentice. Oh sure, go ahead and laugh it up over their silly ranking system. Then go back to your day job and worry some more about your next promotion.
Explicit stack-ranking is such a critical ingredient that without it the addiction dish pretty much fails. Its a catalyst. Its yeast for the tasty Token Loaf.
But in token economies with _physical_ tokens -- not just counts of actions or connections, but distinct physical or virtual items that you can collect and accumulate -- stack ranking isnt enough by itself. You also need _DISPLAY CASES_. Because -- follow my reasoning carefully here -- what the fuck good is collecting things if you cant show off your collection?
Actually even without display cases collecting can still be fun, because it scatches that collectors itch, which has its roots in the fundamental pattern recognition activity our brains engage in to survive. People can always work around the social issue by talking about their collections in forums or whatever.
But letting people show off their collections makes it a whole different ball game. Communities will _always_ find a venue for showing off their collections, even if youre too stupid to provide one for them. But if you feature it directly in the system, it concentrates everyones focus within the system, making it inherently stickier. (In the sense of "measurable consecutive hours spent on the site.") If, on the other hand, you force people to wander off to eBay or a random forum to share and trade their collectibles, then youre letting your revenue stream walk out the door.
A display case in a game can be as simple as allowing you to look at someone elses inventory. Seriously, how hard is that, Gearbox? Players want this feature so badly that every day they risk losing their best items by dropping them and picking them up again just so others can see them flash by, you lovable dumb fuckers! How can you not know this by now?
A token display case can be _anything_ and _anywhere_, as long as it has the players name and hopefully pic attached to it somewhere.
Oh yeah. Pics. Fuck me, now theres a side-rant for you.
TL;DR: PERSONALIZATION IS ANOTHER _HIGHLY_ KEY INGREDIENT FOR ADDICTION. Everyone wants to make their avatar stand out as a unique reflection of their own personal bad taste. And everyone wants a fucking profile page. Christ, now were getting into shit thats so painfully obvious that it makes my eyes twitch, but most companies _still_ dont seem to get it.
In any case I wont talk too much about it today, except to observe that no matter how well you support custom avatars, you could be doing more, and it _will_ make people happier. Doesnt even matter what you do, as long as its more personalizable.
And even a little bit of personalization is infinitely better than nothing. Even the cookie-cutter new Kingdoms of Amatrope is roughly a 1.5 on a 1-10 scale here -- but at least its not a zero. Game designers have been offering personalization for at least _a thousand years_, but Borderlands is like a 0.2, god dammit. They let you change your shirt and hair color. Whoop.
But lets just move on; weve got bigger fish to fry.
THE BORDERLANDS RECIPE
OK, were finally ready to look at whats driving people to play Borderlands four years after the launch, and three and a half years after almost any other game would have disappeared into the annals.
Well take a look at what Gearbox did right, and then Ill bag in a mean but friendly way on Gearbox for being total jackasses, and then well have cake.
The Borderlands token economy is based on _items_, 90% of which are guns. Weapons in games are an especially powerful kind of token because theyre self-reinforcing: the better your weapons, the easier it becomes to collect more of them.
Borderlands has really clever system for generating randomized guns. The guns -- made by around a dozen manufacturers -- have a whole bunch of variable parameters: damage, range, accuracy, rate of fire, magazine size, reload speed, zoom, sway, recoil, bullet speed and "elemental" effects. Some rarer guns also have fancy custom effects -- stuff like increased critical-hit damage, ricocheting bullets, increased blast radius, multiple projectile effects, unusual bullet trajectories, you name it.
Even disallowing a bunch of overpowered or nonsense combinations, when you multiply out all these dimensions you have between 8 and 17 million possible guns, depending on whos counting. Thats a pretty good spread.
If all you want to do is finish the game and move on, then you dont need to think much about guns. You only need a handful at most -- one each for short-range, medium-range and long-range, and maybe a couple of elemental effects for enemies that require them. Guns are leveled, so every once in a while youll want to hit a BioShock-style vending machine and upgrade. But you can get through the game with nothing but a good submachine gun, a combat rifle and maybe a sniper rifle.
To avoid running out of ammo in the final vaginal assault, I suppose its a good idea to have one weapon in each of the games seven ammunition categories. But even thats not strictly necessary, because there are guns and class mods that regenerate ammo for you, plus a fair amount of ammo just lying around in most places. Borderlands is definitely not Survival Horror. Theres _plenty_ of ammo.
So a lot of people (including me, the first time around) just notice in a vague way that their guns seem to be getter better as the enemies are getting stronger, and thats about it.
During my second playthrough a year later, I was surprised to learn online that each gun has half a dozen distinguished components. A guns abilities derive from the assembly of its grip, body, barrel, scope and so on -- each of which has its own rarity and special effects. You can actually look at a gun from afar and figure out a ton about it.
Borderlands guns follow a normal distribution, with most guns being roughly average, but with occasional outliers that can be very powerful. Following Diablos convention, the guns are color-coded by rarity: white, green, blue, purple, yellow, orange and dark orange, the latter being the most powerful. You dont start encountering random orange weapons until late in the game, and you quickly learn to stop and examine them closely whenever you find one.
Aaaaaand... thats pretty much it. Thats the Borderlands recipe. At any rate, thats what they designed and set out to build. And its not too bad. Id say their recipe was blue, maybe bluish purple in the first release. Pretty potent.
But then, through a series of deliberate improvements and happy accidents, their token-economy recipe ripened into a deep, lush orange.
POWER-UP #1: PLAYTHROUGH 2.5
A little-advertised and little-understood feature of the initial release is that there are THREE very different playthroughs of the game, each with its own signature characteristics.
The first playthough is the one everyones familiar with. You start off level 1 without so much as a gun to your name, and by the time you wax the big beaver youre around level 35. Your best weapons are blue and purple, youve worked your way through about half the skill tree, and youre just starting to get comfortable with the system. And then -- boom, its over.
They dont even give you any money or anything. Just a mercifully brief verbal thank-you from the most gratuitously annoying "Guardian Angel" in history -- not just gaming history, but _all_ history, period. All she ever fucking says is lame variations of: "Now is the time for you to go do whatever it is you were planning to do next. Ta!" The real prize you get for finishing Borderlands is that she _finally_ shuts the hell up.
_
"Now is the time for you to mute the fucking game for a while."_
After that you can still go back and do all the side-quests you skipped, but its pointless because youre now vastly overpowered. Not only is there no challenge, but the loot enemies drop is the same level they are, so theres no benefit. So at that point most people call it a day and move on.
Gearbox got so carried away with setting up farming for their primary tokens that they started making people farm non-tokens too: absolutely essential shit that you need in order to participate in the token economy at all.
FARMING NON-TOKENS IS HELLA NOT FUN. _Forcing_ people to do it is flat-out fucking retarded. So the very first thing everyone does when they figure out the endgame is download WillowTree -- a nifty open-source unsanctioned warranty-voiding player-file editor -- and bump up their inventory slots. Some people bump it to 72, the maximum value obtainable in legitimate (albeit heavily farmed) gameplay. Others shrug and say Fuck It, Gearbox is being fucking stupid here, gimme 999 slots and let me deal with the gray hairs as I scroll around painfully.
Its shit like this that makes me think that Gearbox only _kind_ of gets it. But whatever; inventory and bank slots are fixable out of band with WillowTree, so its not the end of the world.
POWER-UP #3: PEARLESCENTS
Remember the Borderlands weapon color-coding scheme, where white means _shit_ and orange means _the shit_? Well, your inventory sorts them by color, with better colors being higher up and thus more easily accessible.
Check this out: in the original Borderlands release there was a bug where some weapons were generated with off-scale rarity, so the game didnt know what color to assign them and they defaulted to white. Nobody at the time knew they they werent orange. But there they were, right up at the top, leering at you with their big white grin like they knew somethin you didnt.
These weapons were the subject of intense debate on the forums, and in the information vacuum they soon became shrouded in mystique. Players began calling them "Pearlescents". They of course became highly collectible.
Gearbox paid attention, and hit on the absolutely brilliant and game-changing idea of FORMALLY SUPPORTING PEARLESCENTS IN THEIR THIRD DLC RELEASE. They fixed the glitch and simultaneously introduced a tiny category of cyan-colored super-legendary weapons, one per manufacturer, with fancy names and fancy effects. And yes, they were actually called Pearlescents.
These cyan weapons are ultra-rare. Insanely rare. Stupidly rare. _Irritatingly_ rare. Taaaaaantalizingly rare. A lot of players never get close enough to sniff one of them. They go to all the potential drop points, and they shoot all the right bad guys, again and again, and after weeks on end they still may not have spotted one.
If you believe the Big Dogs at the fashion houses, the art houses, the back-rooms at Harry Winstons, the auction blocks at Sothebys -- if you believe that rarity creates desire, then with this move Gearbox gave their player base a cyan-veined fucking priapism.
Borderlands gun collectors arent dicking around with purples and oranges, noooooes. Rare as some of them are, it would never have been enough to keep the game thriving for four-plus years. Borderlands gun collectors are after the Pearlescents (or "Pearls" for short).
Funny thing is, as weapons theyre not even that great, for the most part. The power ranges of the different colors have significant overlap, so only the top 15% or so oranges are better than the top purples. And only around the top 10% of all Pearlescents are better than the top oranges. So even when you find a Pearl you probably wont use it.
Doesnt matter. Theyre rare, and special, so people go looking for them.
POWER-UP #4: THE ARMORY BUG
Gearbox admits (again and again) that they fucked up the ending of the main game. Right from the opening scene they start promising you a treasure-filled Vault, and you chase it the entire game, only to find that "Vault" was just a euphemism for Hentai Boss Monster, or maybe some twisted metaphor for realizing you were abused as a child, or some shit like that.
They get it now. Theyre sorry.
And they made up for it. After putting out the obligatory and awesome zombie expansion pack ("The Zombie Island of Dr. Ned"), they released an epic DLC titled The Secret Armory of General Knoxx.
This DLC has a _very_ different style and feel from the main game: an open road through the middle of a dusty sunken sea, with off-ramps to dunes infested with towering spiders, windbitten bandit camps, thriving midget colonies, junkyard strip clubs, corporate military installations, and the most hilariously awesome gay prison in gaming history.
The plotline involves overthrowing a surprisingly well-written general named Knoxx whos been consigned to the planet and -- to his lasting chagrin -- reports directly to a five-year-old named Admiral Mikey. Knoxx is charming and funny and deadly, and the DLC taken as a whole is one of the best-realized and most memorable mini-worlds ever produced in a video game.
The *entire* Borderlands endgame takes place in this DLC. It is the grand finale, the steady-state home base of operations for gun collectors everywhere. Even though there was one more DLC afterwards ("Claptraps New Robot Revolution"), with a setting and story every bit as brilliant and distinctive as the previous two, it has ultimately failed to capture much long-term attention from endgamers because it has no Pearlescents. It _does_ have its own new token economies, such as collecting rare robot parts, but its missing power-ups 4 and 5, so its just not as much fun.
Looking again at the "Secret Armory" part of the Knoxx title, we see that Gearbox has once again promised a big vault full of loot -- but this time, they deliver. In a big way. The storyline winds up giving you three separate runs through Knoxxs armory, timed at two and a half minutes each, during which you can grab all the loot you can carry.
_
The Marcus Kindcaid "Super Sweep" looting spree_
Thats cool and all; its blatant penance for the first game, and with it all is forgiven. But lets face it: 150 seconds isnt a very long shopping spree, even if you get three of them. Knoxxs Armory is _huge_. Its a four-story warehouse with elevators and ramps and movable platforms and crate-mazes, with the loot-chests scattered everywhere. Its got around 120 chests in all, with 20 being the super-rare kind that have a zillion-to-one chance of generating Pearls, but its so vast that on any given run youll only get to a small fraction of the loot.
Which means the Armory, as nice as it is story-wise, isnt such a great bargain for collectors. Or it _wouldnt_ have been, except for the infamous "Armory Glitch".
It turns out -- and Im one of the people who stumbled on this bug completely by accident -- that its possible to fall through the floor at a particular spot on the way in, landing directly in the armory without arming the 2-minute timer, and then you can loot every single chest at your leisure.
_
After falling through the crack, its a 40-foot drop into the Armory_
Its the guiltiest gaming pleasure ever. Youre transformed into a kid who can sneak into the Chocolate Factory while the oompa loompas are sleeping. And you can do it again the next night, every night, forever.
The bug is actually present throughout the game. At pretty much any edge joining elevated flat surfaces in Borderlands you can wiggle around and fall through the crack. Most of the time its an incredible annoyance -- for instance youll fall into an elevator shaft whose sole purpose is spawning bad guys, and the only way out is to kill yourself with grenades or exit the game.
But the bug turns the Armory into a Farmery, and that, friends, is the super-MSG that makes the Borderlands end-game recipe the most successful of its kind since Ye Olde Diabloe back before you were born.
Strictly speaking I suppose "farming" is the wrong word for it. Its not really farming if the vegetables can kill you. I think when you have to slug your way into a vault through a bunch of bad guys, and each time you run a genuine risk of dying, they call it "grinding". So fine, its the Secret Grindery.
The "Armory glitch", as they call it on the forums, is the gateway drug that hooks you in and leads you inevitably to the Lobster Safari, up up up to the high mountaintop arena where youll spend the rest of your free time forever and ever. Youll need to make a hundred or so runs on the armory before youre well-equipped enough to face the Lobster. But during that few weeks youll be as happy as a heavily armed kid in a four-story military candy store.
POWER-UP #5: CRAWMERAX THE INVINCIBLE AND THE "LEDGE GLITCH"
So here we are at the climax. We made it! There should be a popcorn vendor here.
Crawmerax the Invincible is a whale-sized one-eyed purple people eater that Gearbox introduced in DLC3 _specifically_ for advanced multi-player co-op. Hes introduced in a late side-mission accurately titled "YOU. WILL. DIE." Youre not supposed to be able to kill him by yourself. Hes intended as a challenge for groups of heavily-armed, experienced Vault Hunters whove grown weary of existence and want to die like theyre playing Demons Souls.
Crawmeraxs location is heavily advertised with big road signs reading "Secret Final Boss Monster: 20 km", "Secret Final Boss Monster: exit now". When you arrive the earth shakes. Inside his lair is a cavernous "staging area" with some ammo/health vending machines, a few mutilated corpses, miscellaneous hastily dumped construction equipment, and an elevator ascending directly into the solid rock overhead. It looks as if the military has built just enough infrastructure to let all comers try to slay the beast, and so far no luck.
The second critical Borderlands bug is the "ledge glitch". Like the Armory falling-through-the-floor glitch, its a bug that makes it possible to farm Crawmerax... sort of. Its success rate is low enough that it can seem like hes farming YOU. The glitch involves running like mad to a sheer drop on the left, diving into a small nook just below the arena floor, squatting down, facing the corner like hes the Blair Witch, and hoping you survive the acid barf until he decides whether youre unreachable or youre his next ledge meal. It can go either way.
_
"Ill have the bisque."_
*IF* all goes as planned, and thats a pretty big if, then he starts roaring and displaying, but he wont attack you as long as you stay put. At that point its _still_ nontrival to kill him, because one of his total six vulnerable spots is on his back, facing away from you. There are whole strategy guides devoted to hitting that spot. I can tell you its no picnic.
But the payoff. Oh, the payoff.
Players call Craw the "Lobster Piñata" on account of the vast amount of high-quality loot he drops when he dies. Except he doesnt really "drop" the loot so much as _explode_ multicolored items in all directions, and for several loooong seconds, tokens simply rain from the sky.
If raiding the Armory is like refined highbrow shopping at Saks Fifth Avenue, then Crawmeraxs death is like a bomb going off in a Toys R Us. Its like Santas Sleigh plowing into the Hindenberg. Its absolutely spectacular to watch.
In multiplayer mode after the piñata explodes, everyone scrambles around like cockroaches in three distinct waves: first a mad dash to try to spot a Pearlescent, which _might_ happen every forty or fifty kills, then a light circular sprint to check out the orange weapons, and finally a smooth comb through every item to look for upgrades and rare finds.
If theres nothing good, many players start dropping their Pearlescents and then "pretending" to find them by picking them up again. This is incredibly risky, as the co-op mode is _extremely_ buggy when it comes to inventory manipulation, but people do it anyway because Gearbox didnt give them any other way to show off their prizes.
Then its back to the cave entrance, out-and-back to make him respawn (thank you Gearbox for that, btw), and the hunt starts all over again. Ive seen multiplayer runs on Crawmerax take anywhere from 20 seconds to over an _hour_ of nonstop chaos, depending on luck and everyones equipment. Heck, last night four of us tried six or eight times and just gave up. Hes that tough.
The "ledge glitch", which allows single players to have a chance of killing him, is a CRITICALLY IMPORTANT part of the addictive cycle. Many players are reluctant to jump into online co-op play because they feel underequipped and under-experienced. Killing Crawmerax over and over can gradually improve your gear and confidence to the point where youre ready to try the co-op version.
THE PATH OF LEAST RESISTANCE
Game developers always worry about players getting stuck in endgame ruts, going after the same boss monsters again and again for months or years. Players will eventually discover the easiest way to advance in the token economy, at which point all other routes become wasted effort.
The Diablo III team at Blizzard claims in interviews that theyre all angsty about this, that it keeps them up at night. Theyre thinking waaaay too fucking hard about it. They need to man up and launch. People seem to like grinding OK, and the alternative of making all paths equally difficult is an impossible problem. Even if you made the game self-tuning via dynamic feedback loops, people would be pissed because its nondeterministic, with stuff getting randomly nerfed or powered up without warning.
So just launch already.
Gearbox waited until DLC3 to introduce a grinder path, and its interesting that they have _two_ of them -- the Armory and Crawmerax. Whats even more interesting is that theyre approximately equal in terms of payoff over time. A legit player with decent equipment can raid the armory two, maybe three times an hour, and probably see an average of 8 oranges per run. The same player could ledge-glitch Crawmerax maybe 4 or 5 times an hour, and see maybe 4 oranges per run.
In terms of legendary weapons per hour, the two grinds are surprisingly close (~20/hour), so it comes down to a matter of personal preference. Crawmerax will get you killed more often, but he seems to have a higher drop-rate for Pearlescents. Most player probably graduate from Armory runs to Crawmerax runs to multiplayer Crawmerax runs, and are eventually (after months of upgrades) able to "solo" him alone without using the ledge glitch.
_
This dude had way too much free time. Im jealous._
Regardless of which path you prefer, the whole treadmill exists to try to collect rare guns. And were talking guns that you cant really show anyone else, except indirectly via screenshots, videos, or using the desperate gamble of dropping them in a live game. Players can usually tell which weapon youre wielding by its looks and firing pattern, but they cant see its stats.
In the end, the underground collection scene is a little surprising for existing at all. It was half intentional and half accidental.
I dont hear Gearbox talking about this stuff in interviews.
Well see.
EPILOGUE
Yeah, well, none of it prolly matters, since rumor has it theyre finally going to announce the Diablo III release date.
Borderlands 2, we hardly knew ye.
APPENDIX: DONT DO THIS
Hi-ho, since were on the subject of Dos and Donts for creating an optimal gaming experience, I feel obligated to highlight some of the bigger fuck-ups present in Borderlands. I do this in the sincere hope that theres still time to fix them in the sequel. Its probably still a year away from launch, as I write this, so theres hope.
"ITS LIKE WHITE CHRISTMAS": The Borderlands game world is a rich, thriving, multiculural melting-pot with thousands of white settlers and exactly one black guy. _One_. And hes from offworld. Jesus H. Teddy Fucking Roosevelt Christ on a sidecar, Gearbox -- thats *not* what we meant when we asked for a "token" economy. We were all horribly embarrassed to be members of the human race when Josh Whedons Firefly series pulled this stunt, featuring a fully Asianized future without any actual Asians in it. And we thought: "Gosh, well, at least _now_ nobodys ever gonna make _that_ particular douchewit mistake again." What. The. Fuck.
_
Firefly cast. Sigh._
HOT BUTTONS: Having only four equipped-weapon slots in a world where EVERYONE ELSE has figured out that you can squeeze in eight on a wheel -- thats pretty fucking disappointing.
NETWORKING: After spending years developing and polishing the single-player game, they apparently outsourced the multiplayer lobby and connectivity code directly to Sonys network-security team. Whats wrong with Borderlands networking? Well, um, lets see... how about "everything". Yes, that sounds accurate. It crashes all the fucking time, randomly eats your precious inventory items, fails to give you even the slightest insight into what any given party is doing until you actually join their game, randomly goes full amnesiac about which character you were using and logs you in as a level-1 unequipped n00b... the list goes on.
Once youre actually connected to a game and your party is firing on all cylinders, the experience is usually pretty smooth, as long as you _never_ change anything in your equipped inventory slots. But finding and connecting to a reasonable group of players is just a miserable shit sandwich.
MONEY S(T)INKS: Next time around they need a money sink. I know I ranted against too much immersive realism, but if youre going to make money THAT readily available, why bother with it at all? People have so much money in the endgame that its customary to leave the Crawmerax arena after hes dead by _killing yourself_, since it respawns you about a hundred feet closer to the exit than if you take the elevator teleporter. When players are choosing to pay a hundred million dollars to save ten seconds of running, its a good bet your game has a money problem.
INVISIBLE WALLET: Oh, and speaking of money, you cant tell how much you have, nor how much anything costs, because they only have seven digit cash displays. By the end of the game every price over $10M is displayed as 9999999 dollars, and your wallet has 9999999 dollars. You can only see how much money you have when you die, and it tells you you lost 150323855 dollars (the max you can lose at once, 7% of 2^31-1).
_
"So how much?"_
MIDGETS: You should be able to play as one. It would be utterly bad-ass. Havent you guys read _Game of Thrones?_
FAQ
_Q: Arent you being a little hard on Gearboxs vagina?_
Hey, its not _that_ little. Nobodys ever complained bef... oh, I misread that.
_Q: Doesnt modding ruin the economy?_
I mentioned that if Billy breaks into the teachers drawer and distributes all the Gold Stars then they become valueless. And I also mentioned that theres a player-file editor called WillowTree, one that happens to be capable of creating any weapon the game can generate -- and many combinations that the game does NOT generate because they are overpowered and/or nonsensical.
_So of course there are modders. And they show up with their modded weapons, and they throw them around in big piles, daring people to come on over to the Dark Side, to grow up and git yerself a reeeeeal_ gun.
Surprisingly, it doesnt seem to hurt anything. In fact in co-op Crawmerax missions its common for someone to throw out a pile of modded shields of invincibility, and for at least one player to wear one. Otherwise its too easy for everyone to die simultaneously, causing Craw to return to full strength. Even with a super-shield, you can easily be blasted off a cliff and die anyway, so its not a _total_ cheat. A high percentage of players have apparently decided that overpowered shields are part of the acceptable core infrastructure for otherwise legit gun-collecting. I _think_ this implies that Gearbox didnt make their top-end legit shields powerful enough.
_
Kroms Canyon. Just cuz._
Im pretty sure if Borderlands were MMO, with persistent highly-visible score lists and real money on the line (because for MMOs there is _always_ an exchange rate between game currency and real-world currency, whether the publisher likes it or not), then yeah: the modding would fuck everything up.
BUT THE BORDERLANDS TOKEN ECONOMY IS ONLY JUST BARELY ALIVE. Gearbox hasnt done a damn thing to increase its addictive pull since releasing The Secret Armory of General Knoxx, the DLC that introduced all three of the critical enhancements that cemented the economy. So its all basically honor-system. And the players are surprisingly honorable. You can tell immediately if -- and to what extent -- people are abusing the mods system. There are a lot of players out there who stay on the legit side to keep it sustainably challenging. Hell, some of them dont even use the ledge glitch. At least not when people are watching.
You can also tell that the modders are no longer having fun, because their behavior becomes bizarre -- the kind of shit you normally only see at the end of a Bethesda game, where once youve leveled up high enough you get completely bored and start jumping off cliffs naked to try to air-kill boss monsters with a single swing of a broomstick.
_Q: Are you going to bring Wyvern back?_
Patience. Im working on it.
_Special thanks to Andrew Wilson for proofreading this post and making me take out stuff Id regret._
_Oh, and I almost forgot -- Thank You, everyone at Gearbox, for creating such an incredible game!_I woke up this morning...ish... to discover that Hacker News had finally had enough of me being at Google, so they forced me into early retirement.
On Monday I was honored to be able to deliver a keynote talk at OSCON Data. In the talk, I announce at the end that I am quitting a project that I had very publicly signed up for, one that I am not passionate about and dont personally think is very important to the human race. Though others clearly do, and thats a legitimate viewpoint too.
But the power of suggestion can make you see and hear something entirely different. If, for instance, someone tells you that I gave the talk wearing a gorilla suit, then when you watch it, I will magically appear to be wearing a gorilla suit. Its actually a gray jacket over a black shirt, but you will perceive the jacket as the back-hair of a male silverback gorilla! And to be honest the talk could have benefited from the judicious application of a gorilla suit, so no harm there.
Similarly, if someone on Hacker News posts that "Steve Yegge quits Google in the middle of his speech" and links to the video, then you will watch the video, and when I say the word "project" at the end of my speech, a magical Power of Suggestion Voice-Over will interrupt -- in a firm manly voice totally unlike my own quacking sounds -- with "Gooooooogle". And then you will promptly sink into a 15-minute trance so that the voice-over can occur in the middle of my speech where Hacker News said it happened, instead of 96.7% of the way through the talk where it actually happened.
I am going to harness this amazing Power of Suggestion, right here, right now. Here goes.
You are going to come work at Google! You are going to study up, apply, interview, and yes, you are going to work there! And it will be the most awesome job youve ever had or ever will have!
I hope for your sake that this little experiment works, because Google is frigging awesome, and youll love it here. And theyll be happy to have you here. Its a match made in heaven, Im tellin ya. It might take you a couple tries to get in the door, because Googles interview process -- whats the word Im looking for here -- ah yes, their process sucks at letting in all the qualified people. Theyre trying to get better at it, but its not really Googles fault so much as the fault of interviewers who insist that youre not qualified to work there unless you are exactly like them.
Of course, there are interviewers like that wherever you go. The real problem is the classic interview process, which everyone uses and which Google hasnt innovated on, not really. Its like deciding whether to marry someone after four one-hour dates that all happen on the same day in a little room that looks kind of like a doctors office except that the examining table is on the wall.
The reason I havent been blogging lately is that working at Google is so awesome that I just dont feel like doing anything else. My project is awesome, the people are awesome, the work environment is over-the-top-crazy-awesome, the benefits are awesome, even the corporate mission is awesome. "Organize the worlds hardline goods in little brown boxes delivered straight to your doorstep" -- thats an awesome mission, yeah?
Wait, sorry, that was a flashback to the Navy or something. "Organize the worlds information" -- thats the one. Its a mission that is changing the course of human events. It is slowly forcing governments to be more open, forcing corporations to play more fairly, and helping all of us make better decisions and better use of our time.
In that vein, the part of my brain that makes Good Decisions was apparently broken a few weeks ago, when I allowed myself to be cajoled into working on something that I wasnt passionate about. I am an eternal optimist, and I figured I could teach myself to be passionate about it. And I tried! I spent a few weeks pretending that I was passionate about it -- thats how I got through my Physics classes in college with A grades, so I know its a mental trick that can sometimes work.
But then I wrote my OSCON Data speech, in which I basically advise everyone to start working on important problems instead of just chasing the money. Or at the very least, go ahead and chase the money in the short term, but while you are doing that, prepare yourself to help solve real problems.
And after writing the speech I realized Id completely failed to follow my own advice. Im getting old and I only have so many "big projects" left that I can actually participate in. So in my mind its a complete cop-out for me to take the easy path and work on a project that my company is excited about but I am not.
Now, as it happens, I am in fact working on a very cool project at Google. Its not important in the same sense that curing cancer or getting clean water to impoverished cities are important. But its a project that has the potential to revolutionize software development, and NOT through some new goddamn dependency-injection framework or web framework or other godawful embarrassing hacky workaround for a deficient programming language. No. It is a project that aims to turn source code -- ALL source code -- from plain text into Wikipedia. Ive been on it for three and a half years, and I came up with the idea, and the team running with the idea is fantastic. The work may not be directly important, but it is an enabler for important work, much like scaling infrastructure is an enabler.
So I am happy to continue working on that project for now. Yes, at Google. I may even blog it up at some point. But Im very serious about brushing up on my math and statistics, some of which I havent applied directly in 20 years, and start focusing on machine learning problems. Particulary, if I may be so fortunate, the problem of curing cancer. I may not be able to participate directly for a few years, as I need to keep working and paying the bills just like you. But Im studying hard -- I started up again a few days ago -- and Ive demonstrated to myself quite a few times that if I do anything daily for a few years I can get pretty good at it.
Anyway, Im late for work. Isnt that nice? I like the sound of it. It has a nice ring to it: "Im late... for my job."
So come work with me! Unless you are curing cancer, of course.
San Jose, CA (Reuters) -- Online auctions cartel eBay (NASDAQ: EBAY) and its collections and incarceration arm PayPal announced that on July 21, 2011, the two companies had jointly been awarded United States Patent No. 105960411 for their innovative 10-click “Buy it Now” purchasing pipeline.
The newly-patented buying system guides users through an intuitive, step-by-step process of clicking “Buy It Now”, entering your password, logging in because they signed your sorry ass out again, getting upsold shit you don’t want, continuing to your original destination, accepting the default quantity of 1 (otherwise known as “It”), committing to buy, clicking "Pay Now", entering a different password than your first one, clicking "Log In" again god dammit, declining to borrow money from eBay’s usury department, reviewing the goddamn purchase details since by now you’ve completely forgotten what the hell you were buying, and finally confirming the god damned payment already.
The 10-click checkout system, known colloquially as 10CLICKFU -- which many loyal users believe stands for “10 Clicks For You” -- was recently awarded top honors by the National Alliance of Reconstructive Hand Surgeons. 10CLICKFU incorporates a variable number of clicks ranging from eight to upwards of fifteen, but eBay’s patent stipulates that any purchasing system that lies to you at least nine times about the “Now” part of “Buy It Now” is covered by their invention.
The patent award came as a surprise to many analysts, since several of eBay’s related patent attempts had been rejected on the basis of prior art. In one well-publicized filing, eBay had tried to patent a purely decorative, non-operational “Keep me signed in” checkbox, but Sony’s PlayStation Network already had one just like it. And another eBay patent claim for excruciating page load times was rejected because the iPad App Store is still loading.
But eBay’s boldest and potentially furthest-reaching patent attempt was for “100% Inaccurate Button Text”. The invention claim was based on several of their UI elements, but rested primarily on the “Buy It Now” button, which eBay claims contains enough inaccuracies to render it "complete bullshit." Their patent was rejected by the US Patent Office review committee on the grounds that the Firefox browser’s “Do this automatically from now on” checkbox has been complete bullshit for over fifteen years. eBay says they will appeal the ruling because the checkbox is not technically a button.
eBay’s spokesperson Paula Smugworth announced that eBay will continue to innovate on ways to remind their users that monopolies can do whatever the hell they want. “Not that eBay is a monopoly,” she added. “But if we were a monopoly, then we could do whatever the hell we wanted. I’m just sayin’.”
eBay’s stock rose on the news, driven largely by anonymous shill bidders.
The worldwide Haskell community met up over beers today to celebrate their unprecedented discovery of an industry programmer who gives a shit about Haskell.
On Wednesday, researchers issued a press release revealing that 27-year-old Seth Briars of North Carolina, a Java programmer at Blackwater accounting firm Ross and Fordham, actually gives a shit about Haskell.
"Mr. Briars has followed every single one of our press releases for years," the press release stated. "Probably even this one."
Haskell researcher Dutch Van Der Linde explained how they had stumbled on the theoretical possibility of Briars and his persistent interest in Haskell. "We knew that there are precisely 38 people who give a shit about Haskell," said Van Der Linde, "because every Haskell-related reddit post gets exactly 38 upvotes. Its a pure, deterministic function of no arguments -- that is, the result is independent of what we actually announce. But there are only 37 of us on our mailing list, so we figured there was a lurker somewhere."
"That, or it was an off-by-1 error not detectable by our type system," Van Der Linde added. "But we dont, uh, like to dwell on, I mean with good unit testing practices we can, um... sorry, I need to get some water."
As Van Der Linde stumbled off in a coughing fit, his fellow researcher Bonnie MacFarlane outlined their basic dilemma: "Finding a person who gives a shit about Haskell is an inherently NP-complete computer science problem. Its similar in scope and complexity to the problem of trying to find a tenured academic who didnt have the bulk of his or her work done by uncredited graduate students. So even though we suspected Briars existed, we needed a strategy to smoke him out."
She explained the trap they set for Briars: "We crafted a fake satirical post lampooning Haskell as an unusable, overly complex turd -- a writing task that was emotionally difficult but conceptually trivial. Then we laced the post with deeper social subtext decrying the endemic superficiality and laziness of global industry programming culture, to make ourselves feel better. Finally, each of us upvoted the post, which was unexpectedly contentious because nobody could agree on what the reddit voting arrows actually mean."
"And then we waited to see who, if anyone, would give a shit," she said.
MacFarlane concluded, "Our elegant approach didnt work, so we hired a Perl hacker to go dig up the personal details on all 38 accounts that had ever upvoted a Haskell post, and the only one we didnt know was Seth Briars. So we reached out to him, and thankfully so far he hasnt threatened to sue us."
Briars says he is pleased to have been recognized for his apparently unique shit-giving about Haskell. "Ive been giving a shit about Haskell for a long as I can remember. I follow all their announcements and developments closely, just in case I ever get the urge to use the language for something someday."
"Its a beautiful, elegant language," Briars observed as he busied himself cleaning a fingernail. "Youd be hard-pressed to find a more expressive and composable core. And theyve made astounding advances over the years in performance, interoperability, extensibility, tooling and documentation."
"Im kind of surprised Im the only person on earth who gives a shit about it," Briars continued. "Id have thought there would be more people following the press releases closely and then not using Haskell. But they all just skip the press releases and go straight to the not using it part."
"People see words like _monads_ and _category theory_," Briars continued, swatting invisible flies around his head for emphasis, "and their Giving a Shit gene shuts down faster than a teabagger with a grade-school arithmetic book. Im really disappointed that more programmers dont get actively involved in reading endless threads about how to subvert Haskells type system to accomplish basic shit you can do in other languages. But I guess thats the lazy, ignorant, careless world we live in: the so-called real world."
Haskell researcher Javier Escuella remains hopeful that one day they may be able to double or even triple the number of industry programmers who give a shit about Haskell. "I believe the root cause of the popularity problem is Haskells lack of reasonable support for mutually recursive generic container types. If we can create a monadic composition-functor wrapper that is perceived as sufficiently sexy by hardened industry veterans, then I think we will see an uptick in giving a shit, possibly as much as a full extra person."
Haskell aficionado Harold MacDougal is not quite as sanguine as his colleague Escuella. "I doubt Haskell will ever be appreciated by the uneducated natives of this industry. As exciting as it is, the discovery of Briars should be considered an anomaly, and not as a sign that more people will ever give a shit. Programmers only seem to pay attention to things when there is humor involved."
"We do have an experimental humor monad," added MacDougal. "But it doesnt seem to be getting much adoption. Haskell fans just dont see the need for it."
-------------------------
MORE NEWS
Previous article: PERL COMMUNITY DEBATING ADDING MONADS The Perl lists are brimming with discussions about the value of adding monads to Perl. "We dont really know what they do, but it doesnt make sense _not_ to have something in Perl," said Perl hacker Landon Ricketts. Read more
Next article: MICROSOFT TO INTRODUCE MUTUALLY RECURSIVE ERROR MESSAGES Software giant Microsoft announced today the launch of their new REDRUM platform, an elegant system that allows Windows system error messages to shuffle blame around indefinitely by using continuation-passing. Read more
EYJAFJÖLL, ICELAND -- Java programmers around the globe are in a panic today over a Wikileaks press release issued at 8:15am GMT. Wikileaks announced that they will re-release the source code for thousands of Open Source Java projects, making all access modifiers public and all classes and members non-final.
Agile Java Developer Johnnie Garza of Irvine, CA condemns the move. "They have no right to do this. Open Source does _not_ mean the source is somehow open. Thats my code, not theirs. If I make something private, it means that no matter how desperately you need to call it, I should be able to prevent you from doing so, even long after Ive gone to the grave."
According to the Wikileaks press release, millions of Java source files have been run through a Perl script that removes all final keywords except those required for hacking around the 15-year-old Java languages "fucking embarrassing lack of closures."
Moreover, the Perl script gives every Java class at least one public constructor, and turns all fields without getters/setters into public fields. "The script yanks out all that @deprecated shit, too," claims the controversial announcement.
Longtime Java programmer Ronnie Lloyd of Austin, TX is offended by the thought of people instantiating his private classes. "Its just common sense," said Lloyd, who is 37. "If I buy you a house and put the title in your name, but I mark some of the doors Employees Only, then youre not allowed to open those doors, even though its your house. Because its really my house, even though I gave it to you to live in."
Pacing and frowning thoughtfully, Lloyd continued: "Even if I go away forever and you live there for 20 years and you know _exactly_ whats behind the doors -- heck, even if its a matter of life and death -- plain old common sense still dictates that youre never, _ever_ allowed to open them for any reason."
"Its for your own protection," Lloyd added.
Wesley Doyle, a Java web developer in Toronto, Canada is merely puzzled by the news. "Why do they think they need to do this? Why cant users of my Open Source Java library simply shake their fists and curse my family name with their dying breaths? That approach has been working well for all the rest of us. Who cares if I have a private helper function they need? What, is their copy/paste function broken?"
Wikileaks founder Julian Assange, who coined the term "Opened Source" to describe the jailbroken open-source Java code, fears he may be arrested by campus security at Oracle or possibly IBM. The Wikileaks founder said: "Today the Eclipse Foundation put out a private briefing calling me a non-thread-safe AbstractKeywordRemovalInitiatorFactory. What the fuck does that even mean? I fear for my safety around these nutjobs."
The removal of @deprecated annotations is an especially sore issue for many hardworking Java developers. "I worked hard to deprecate that code that I worked hard to create so I could deprecate some other code that I also worked hard on," said Kelly Bolton, the spokesperson for the League Of Java Programmers For Deprecating The Living Shit Out Of Everything.
"If people could keep using the older, more convenient APIs I made for them, then why the fuck would they use my newer, ridiculously complicated ones? It boggles the imagination," Bolton added.
The Eclipse CDT team was especially hard-hit by the removal of deprecation tags. Morris Baldwin, a part-time developer for the CDTs C++ parsing libraries says: "We have a policy of releasing entire Java packages in which every single class, interface and method is deprecated right out of the box, starting at version 1.0."
"We also take careful steps to ensure that its impossible to use our pre-deprecated code without running our gigantic fugly framework," the 22-year-old Baldwin added. "Adding public constructors and making stuff non-final would be a serious blow to both non-usability _and_ non-reusability."
The Agile Java community has denounced the Wikileaks move as a form of terrorism. "It was probably instigated by those Aspect-Oriented Programming extremists," speculates Agile Java designer Claudia Hewitt, age 29. "I always knew they wanted to use my code in ways I couldnt predict in advance," she added.
Many Java developers have vowed to fight back against the unwelcome opening of their open source. League of Agile Methodology Experts (LAME) spokesperson Billy Blackburn says that work has begun on a new, even more complicated Java build system that will refuse to link in Opened Source Java code. The new build system will be released as soon as several third-party Java library vendors can refactor their code to make certain classes more reusable. Blackburn declined to describe these refactorings, claiming it was "none of yalls business."
Guy Faulkner, a 51-year-old Python developer in Seattle, was amused by the Wikileaks announcement. "When Python developers release Open Source code, they are saying: Here, I worked hard on this. I hope you like it. Use it however you think best. Some stuff is documented as being subject to change in the future, but were all adults here so use your best judgment."
Faulkner shook his head sadly. "Whereas Java developers who release Open Source are code are saying: Here, I worked hard on this. I hope you like it. But use it exactly how I tell you to use it, because fuck you, its my code. Ill decide whos the goddamn grown-up around here."
"But why didnt they write that Perl script in Python?" Faulkner asked.
-------------------------
MORE NEWS
Previous article: SAN FRANCISCO AIRPORT ANNOUNCES THAT ALL RESTROOMS NEAR YOU ARE NOW DEPRECATED
The SFO port authority announced today that all airport restrooms located anywhere near you are now deprecated due to "inelegance". The newer, more elegantly designed restrooms are located a short 0.8 mile (1.29 km) walk from the International Terminal. Read more
Next article: ECLIPSE SITS ON MANS COUCH, BREAKS IT
New Hampshire programmer Freddie Cardenas, 17, describes the incident: "We invited Eclipse over for dinner and drinks. Eclipse sat down on our new couch and there was this loud crack and it broke in half. Those timbers had snapped like fuckin matchsticks. Then my mom started crying, and Eclipse started crying, and I ran and hid in my bedroom." Read more
Well! Ive sure had a nice relaxing blog-free year. No worries, no haters, no Nooglers wandering by my office and staring at me through the window as if they expect me to crap in my hand and hurl it at them. Not that I wasnt tempted.
Nope, its just been peace and quiet and reading and coding and practicing my guitar and stuff. Its been awesome.
And now that everyones completely forgotten who I am, or whatever exactly Id said that made them feel all butthurt inside -- as measured by my incoming email rate, which is finally near-zero -- I figure its probably safe to get back in the water.
Im not really sure what my plans are going forward, other than staying employed at Google until the day comes when I need one of their comfy, brightly-colored caskets. Other than that, my plans are flexible. Im feeling downright leisurely at the moment.
I realize now that I was trying way too hard to change the world via blogging, and it made me care maybe just a _little_ too much. This was bad for my mental and emotional health. Caring is fine. Lots of things are worth caring about. Very few of them merit sacrificing your health.
Fortunately during my ad-hoc sabbatical I was able to gain some new perspectives by distancing myself a bit from the constant storm going on in the tech world.
One nice perspective I gained is this: THERE IS NOTHING ON THIS EARTH THAT CAN MAKE EVERYONE HAPPY. Reddit is a huge, living, breathing demonstration of this, since essentially no reddit post ever goes above maybe 80% approval, and a "good" post seems to hover around 65%-70% liked.
That made me feel better about the haters. Haters abound. Theyre just a fact of life, part of the human condition. Theres no need to waste energy hating haters.
Another perspective I gained was that decorating your mansion with works of art you know nothing about is amazingly rewarding, as long as you can mix it up by leaping across rooftops and assassinating bad guys and hanging with your buddy Leonardo. I swear, if they ever make a movie about my life, the handsome and dashing actor who plays me, when asked on his deathbed which of lifes pleasures had given him the greatest happiness, will say something cheesy that makes the audience ooh and aww with appreciation, but itll be total Hollywood bullshit, because what I really will have said was "gaming".
Yet another perspective I gained is that I now actually agree with everyone who complained that my blog posts were too long. Reddit has ruined my attention span for online material. There seems to be no such thing as too frequent, but theres definitely such thing as too long. So Ill be better about that.
I used to have this pet theory that the length of my blogs is a big part of why theyve been noticed at all. I mean, look at this dude. If hed written only one or two crazy things, hed be just another nutjob, but by dint of almost superhuman persistence hes managed to get the _entire world_ to laugh at him.
I was sort of aiming for getting people to laugh _with_ me, but I used the same basic recipe as Time Cube Dude. And the formula seemed to be working, modulo the haters.
However, Dave Barry -- my Personal Childhood Hero (66% liked!) -- always wrote his columns in chunks of 800 words, even if it necessitated inserting filler words such as "booger" and "legislative session" into his articles about wine tasting or car engines or bat guano, or whatever it was that caught his fancy that week.
Overall it seems likely that post-length is less important than factors such as quality, consistency, passion, relevance, and legislative booger session.
So I was originally thinking of writing up to a maximum of 800 words today, and Im at about 500 now, but Ive been really successful at my Not Caring Too Much Initiative, so... later! Nice chatting with ya. Cheerio!
HAHAHA DISREGARD THAT...
Just to ensure this post isnt _entirely_ devoid of content Ill share something important that I learned last year.
Heres what I learned: after Carpal Tunnel Syndrome, the second most common hand ailment is known as Trigger Finger.
Its more formal name is _digital tenovaginitis stenosans_, which is ancient Latin for "electronic hand inflamed vagina without writing", which I believe is why most people prefer to call it Trigger Finger.
I have it, you know. Trigger Finger, I mean, not an inflamed vagina.
Although Trigger Finger is "idiopathic", a fancy word meaning that doctors dont have a fucking clue what causes it, it is widely known in musical circles as a musicians injury. It happens to musicians who overpractice, usually in preparation for a recital, performance or recording session.
I found all this out _after_ being diagnosed with it.
It is not idiopathic in my case. I have the benefit of hindsight, and I know exactly what caused it. It turns out that if you play a certain right-hand arpeggio on a classical guitar enough times -- where "certain arpeggio" here refers to Hector Villa-Lobos Etude No. 1, and "enough times" is approximately 650,000 times in a 5-month period[wtf?] -- you acquire Trigger Finger. Thats not _precisely_ what I was playing, but itll serve.
Trigger Finger is a painful, debilitating, demoralizing injury. I highly recommend not letting it happen to you. Your body will begin telling you when its time to ease up on the practice sessions. Listen to your body when it says that.
As for specifics, theres not much to tell. My hand started hurting. Then it hurt real bad for a month. Ibuprofen and cold/hot packs didnt help. It got steadily worse. Even quitting guitar altogether for another month didnt help. I could no longer use my right hand, and it was beginning to feel permanent. I wasnt even sure why it was happening. I was terrified and I began to despair.
My Google doctor was great. She referred me to a specialist -- a hand surgeon. I told her I didnt really want to see a hand... S-word. I could barely say it aloud. She reassured me that seeing a specialist didnt necessarily mean surgery. They might have other tricks up their sleeves. So I decided to brave it.
My first trip to the specialist only took about 15 minutes. She listened to my disoriented bleating, asked me a few questions, gently felt my hand here and there, and informed me that I had Trigger Finger. She said she was going to give me a cortisone shot. She was pulling out a giant needle as she told me this. It just sort of materialized from under the table, the way a knife appears in a bar fight. It was a very large needle. She explained calmly that the cortisone is a steroid that stays wherever you inject it. They use it on athletes to reduce inflammation from certain injuries.
Then she stuck the giant needle all the way into the base of my right middle finger and squeezed. Compared to the pain of my trigger finger, the injection felt like a mosquito bite.
She told me that Id start feeling better in a week, and in a month Id be pretty much all cleared up. If not, I should come back and see her for more treatment. And no, I wasnt going to lose my hand.
It was kind of weird, but on my way back to my car I think someone had been cutting onions in the elevator. A lot of onions. In the last fifteen minutes my whole life had been handed back to me with an almost casual lack of concern. I was overwhelmed with onions.
A month later I was back to see her. The cortisone had helped a lot. I gave it a 66% approval rating. She said she could give me another shot, or do surgery. This time Id done my homework. I elected for surgery. That was back in September. It was an interesting story in its own right, but the upshot is she did great. And then after that there was a lot of physical therapy.
I typed the word "September" in the previous paragraph three times before I got it right. The word doesnt even have letters that need my right middle finger. My right hand, which had shaped itself into an unusable, agonized claw between March and July, is still afraid to flex and extend my middle finger. Its up to, oh, a 95% approval rating now, which I believe is phenomenally successful. Who could ask more of a hand surgery? It could have been much worse. Much much.
But that last 5% is rough. The haters in my hand are a constant reminder of the old pain. When I type, or play piano or guitar, my right-hand fingers twist and curl in elaborate, incomprehensible dances to avoid a pain that is for the most part no longer there.
Yep, I think itll be easier to keep my blog posts shorter going forward.
Ironically, some good came out of the experience. Ive switched to a sustainable new guitar style and a new repertoire, one I enjoy greatly. And I now pay much more attention to economy of motion in my typing. And I spend more time finding pain-saving Emacs shortcuts. It makes me wonder what I might have achieved had I focused on it sooner.
Surgery notwithstanding, on the whole I still think it was a great year. The year was in fact more complicated and more painful than Ive let on here, but thats life for ya.
And now that Im rested up, I believe Im ready to start tech blogging again... in moderation, anyway. The rest and relaxation and research did wonders for me. I used to have lot of open, long-standing concerns about the future of programming and productivity, but my sabbatical last year finally brought me some clojure.
-------------------------
[1] Talk about caring too much. I may explain this 650k figure in a future blog post if I can ever get over my embarrassment.
Were getting close to the end of my blog. After todays entry, I only have three left to write. After that, Ill only blog anonymously or (more likely) not at all.
This is part three of five in my "Programmers View of the Universe" series. I struggled for a while with how best to introduce the ideas in this installment, and ultimately opted for a short story.
This is a science fiction short story. Its different from many other sci-fi stories in that it is set in the "near future", but it has realistic schedule estimates. So unlike 1984, 2001, The Singularity is Near and all the other sci-fi stories that grossly underestimated their project durations, this one is set 1000 years in the future. I.e., right around the corner.
The story is disrespectful to pretty much everyone in the world. It will create a fantastic shit storm. This is probably a good time to point out that I dont speak for my employer. _[Edit: Yay for fiction! Apparently marking something as fiction placates people. Nice to know.]_
The story is 18 pages (PDF from Google Docs print preview). Thats not unusual for my blog, but I went ahead and published it as a standalone document.
Id encourage you to enjoy it, but Im old and embittered enough to know better. You probably shouldnt even read it. Just wait for someone to summarize it for you.
Installments 4 and 5 will not be short stories; they will be regular old blog rants. In them I will further develop these ideas, and I will also attempt to clear up any gross misconceptions about the story, of which there are bound to be many.
My final blog-rant entry is the only one I care about anymore. Ive been working on it so hard that my fingers have started to fail. Its been tons of fun, aside from the chronic pain. Its about a neat programming language, and Emacs, and lots of other stuff. I cant wait!
Oh yeah. Heres the link to the story. Ive never done a read-only Google Docs link before, so its probably broken. Or editable. I dont know. Well see.
This week is going to suck. People are going to be mad. Maybe I should take a vacation and come back when the whining is finished. Can someone email me and let me know when its all blown over?
Over the holidays I read a neat book called Predictably Irrational: The Hidden Forces That Shape Our Decisions, by Dan Ariely. The book is a fascinating glimpse into several bizarre and unfortunate bugs in our mental software. These bugs cause us to behave in weird but highly predictable ways in a bunch of everyday situations.
For instance, one chapter explains why bringing an uglier version of yourself to a party is guaranteed to get you more attention than other people who are arguably better-looking than you are. I personally do this all the time, except that Im usually the ugly one. The same principle explains a ploy used by real-estate agents to get you to buy ugly houses.
Another chapter explains the bug that causes you to be a packrat, and shows why you desperately hold on to things you own, even if you know deep down that they would rate lower than pocket lint on eBay.
In any case, well, good book. Im going to harsh on it a teeny bit here, but its only one tiny part towards the end, one that actually has little to do with the rest of the research presented in the book. I still highly recommend it. Its only about a 4- or 5-hour read: beyond the reach of most social-network commenters, perhaps, but you can probably handle it just fine.
So: about that harshing. Dan Ariely, who seems like a pretty fascinating guy in his own right, independent of his nifty book, says something thats kinda naïve towards the end. It doesnt _seem_ naïve at all when you first read it. But naïve it is.
Towards the end of the book -- and I apologize here, since my copy is on loan to a friend at the moment, and you cant search inside the book on Amazon.com no-thanks to the books publisher, so I cant double check the exact details -- but towards the end, Dan works himself into a minor frenzy over what seems like a neat idea about credit cards.
CREDIT CARD BUCKETS
Dans idea is simple and appealing: lets partition credit limits into "buckets". People are always maxing out their credit cards, and it leads to all sorts of financial misery, since the rates are always by definition just epsilon short of legal usury, so most people can never, ever pay down the debt.
Dans idea is more or less as follows: you divide up your credit card available balance into "buckets", where each bucket represents a type of expense. You might, for instance, have a bucket for rent and utilities, a bucket for alimony, a bucket for chocolate and desserts, a bucket for sports and leisure activities, a bucket for dining out, a bucket for home improvement, a bucket for groceries, and a bucket for discretionary spending, or "misc".
Each bucket would have its own credit limit, and the sum of all the individual limits would be your credit limit. Lets assume you pay off your credit card entirely every month -- not typical, but it simplifies the explanation. Each month, youd have a certain amount of money to spend on each bucket, and you would not be allowed to spend more than the limit for that purchase type.
OK, so thats the way I understood the proposal. There were several pages devoted to it, as I recall, so I may have missed a few nuances here and there, but Im hoping Ive captured the essence of it.
As an aside, I read a short diatribe many years ago by a working mom whose kids were always asking her why they couldnt spend more money on entertainment purchases like video games. She was having trouble getting through to them, so one month she took her paycheck, went to a bank, and got it issued to her in 1-dollar bills. She took the bills home and piled them up on the table in front of her kids, who were amazed at the giant pile of money she had made. She then went through her budget with them, stacking the bills into piles by expense type: this many dollars for rent, this many for utilities, this many for groceries, this many for soccer uniforms, etc. At the end there were only a couple of dollars left, and the kids soberly realized that they needed to wait about 20 years and then start downloading games illegally online.
Anyway, Dans idea was kind of similar. In order to train consumers not to overspend in a given category, leading to overall overspending, they would be able to opt-in to a program that partitioned their credit, and presumably it would lead to much wiser, more deliberate spending.
It seemed logical enough to me! It sounds similar to calorie-counting: Ive found that explicitly keeping track of my calorie consumption each day does wonders for lowering my overall calorie intake. Along the same lines, I am 100% sure that if I had an explicit budget, rather than just a vague gut feel for how much Im spending, then I would spend less money each month. We dont really have a proverb for this concept, but we do for the opposite: "out of sight, out of mind". If your budget is in plain sight, well then...
Plus the idea of having _types_ in my credit-card accounting was all the more alluring since Im a programmer, and I "get" the idea of types. Types are great. Dan is effectively suggesting a strongly-typed approach to credit-card spending, so the programmer in me was all for it.
EVIL BANKS (AS IF THERES ANY OTHER KIND)
Unfortunately, this story has a sad, bitter ending. Normally I _would_ want to add: "like all overly-strong-typing scenarios", but that would just be mean. So Im not saying it!
Dan goes on to explain that banks are far too evil, or at any rate far too self-serving to implement his incredibly cool idea. He actually took the idea to at least one bank, meeting with their board of directors and presenting the idea. How cool is that?
Dan says the bank executives seemed to like his idea, and indicated that it might be a great way to get new customers. Credit cards are all pretty much the same (i.e., loan sharks), so they need to find ways to differentiate themselves. A nifty credit-bucketing program seemed like something marketing could run with. They all said theyd look into it and see about maybe implementing it.
And then... nothing. They never implemented the idea! Not even a little prototype of it. Nothing.
Dan theorized that profit margins, as always, are the culprit here. Even if banks could potentially sign up more customers on the promise of better spending control, theres a fundamental problem here, which is that credit cards make money for the banks _based on spending_. If consumers arent spending as much, the banks wont make as much profit!
Banks make money off credit cards in at least three ways: they charge the merchant a fee at point of sale, they charge you interest on the loan, and they charge you fees such as the overdraft fee when you inadvertently overspend your limit. All those ways require you to make purchases, and the last way actually requires you to overspend your limit -- exactly what Dans idea is trying to prevent!
I suppose a truly evil bank might look at buckets as an opportunity to screw you on fees for each individual bucket. But Dan seems to think that on the whole, the fear of decreased margins -- induced by the suddenly more rational consumer spending -- is what is preventing banks from implementing his idea.
At the time I was reading the book, I thought, well gosh, I _hate_ banks. In fact, I dont even _use_ a bank -- I now use an investment brokerage that has banking services on the side. You dont have to be rich to do this, and it saves you from ever having to walk into a bank again. And if you choose this route, then whenever you walk into a bank you will immediately be struck by what amazing ghettos they are: little brick buildings with little vaults holding your little dollars, little lines to talk to little tellers who provide you with little help... theyre awful. They stink. I detest banks; Ive found the whole notion loathsome for at least ten years before hating them became globally fashionable a few months ago.
So yeah. Dan had me. The banks are evil. Thats why they arent implementing his idea. Case closed.
THE LITTLE WINGED NAGGING PROGRAMMER ANGEL ON MY SHOULDER
So just like Memento Guy in L.A. Confidential, my mind wouldnt let the case close forever. (Why can I remember Rollo Tomasi but not the actors name? Oh wait, Guy Pearce. Him.)
For the next couple of weeks, Dans situation replayed in my head like a bad song. I myself have given presentations to boards of executives in the past, usually presentations that had come to naught, and I felt a certain empathy with him.
As it happens, I also served time in Amazon.coms Customer Service Tools group for four years, leading the group for the latter half of my stay, and I know a thing or two about credit-card processing. Not a whole lot, but definitely a thing or two.
And Im a programmer. Just like you. (You might not know it yet, but you are. Trust me.)
The programmer part of me started wondering: how would I implement Dans idea? What would it take to add "bucketization" to credit cards?
And the programmer part of me started to get a sinking feeling in the pit of his... uh, its... stomach. It got the chills. And a fever. At the same time. Why? Because _it_, by which I mean "a part of me that wishes I could forget it", has been on software projects like that before.
The little nagging voice in my head started enumerating all the things you would need to do, like counting so many sheep. First I imagined I worked at the bank, some poor schmuck of a programmer wearing a suit, working bankers hours and golfing every day at 3pm. So, you know, pros and cons. Then I imagined my boss coming in and saying: "Steve, we gotta implement Buckets. The board just approved it. Make it happen. Yesterday."
Aw, crap. OK, what to do. First, we need a spec. So, like, I ask my boss a few preliminary questions:
* Can customers control the buckets, or are they fixed?
* If fixed, how many are there? What are their names?
* Lets assume for the remaining questions that they are NOT fixed, since a predefined set of buckets would be "insanely stupid" and rejected by customers. So, how many buckets can a customer make? Min and max?
* Can customers give the buckets names? If not, do they have to use numbers?
* What characters can they use in the name? Whats the maximum length? If we need to truncate the name in a printed statement, how do we truncate it?
* Can a customer change their buckets mid-month?
* Can a customer change their buckets between months? What if their balance is nonzero? Can they transfer balance between buckets?
* Can a customer change the name of a bucket? Do names have to be unique?
* Exactly _how_ does a customer name a bucket? Online? Over the phone? By snail mail forms? Talking to bank teller? All of the above?
* Same question for all other configuration settings. How? Where?
* Do credit-card customer service reps have to know about the buckets? How much do they have to know? _(hint: everything)_ Is there training involved? _(hint: yes)_
* Do the customer-service tools have to be redesigned to take into account this bucketization?
* What about the banks customer self-service website?
* What about the phone interactive voice-response tree?
* What about the software that sends email updates to the customer?
* What about the software that generates printed billing statements? How exactly does it represent the buckets, the individual spending limits and balances, the carry-overs from month to month, the transfers, the charge-backs, the individual per-bucket fees?
* What about the help text on the website? What about the terms and conditions? What about the little marketing pamphlets? Should they try to explain all this shit, or just do some hand-waving?
* Can a customer insert a new bucket into the list? How are the credit limits of the remaining buckets re-allocated? What if adding a new bucket puts one or more of the older buckets over the limit? Do we charge fees? Do we tell the customer theyre about to be charged a fee right before they create the bucket? Is it, like, OK/Cancel? Do we send them a follow-up email telling them they just fucked themselves over? What exact wording do we use?
* Can a customer delete a bucket? What if theres money in it? What if its overdrawn? How do we represent the overdraft fee in the database? How do we show the deletion event in their bill?
* Can a customer merge or consolidate buckets?
* What if a customer has an emergency situation, plenty of limit in other buckets, and they really really need to charge to a couple of buckets, but they want to avoid an overdraft fee? What do they do? Are the buckets mandatory or discretionary?
* How the hell do we even tell if theyre buying "chocolate", anyway? The vendor doesnt tell us the purchase type. How do we know how to charge the right bucket? What if its ambiguous? What if the buckets overlap? Does the customer need a point-of-sale interface for deciding which bucket to put the charge in? Can they do "separate checks" and split the charge into several buckets?
* Where are you going? Answer me!
* WHAT THE EVER-LOVING *FUCK* ARE YOU PEOPLE SMOKING? HUH? HAVE YOU EVEN THOUGHT ABOUT THIS PROJECT FOR MORE THAN A MILLISECOND? THE SPEC FOR THIS PROJECT WILL BE 5,000 PAGES! IT WILL TAKE THOUSANDS OF MAN-YEARS TO IMPLEMENT, AND *NOBODY* WILL UNDERSTAND HOW IT WORKS OR HOW TO USE IT, EVEN IF WE SOMEHOW MANAGE TO LAUNCH IT! ITS FRIGGING IMPOSSIBLE! ITS INSANE! __YOU__ ARE INSANE! I QUIT! NO, WAIT, YOURE FIRED! ALL OF YOU! AAAAAAAAAUUUGH!
The little nagging white-robed behaloed programmer whispering in my ear was getting pretty goddamned irritating at this point. And it asked a LOT more questions than the ones in the list above, which is merely a representative sample. My stress level began approaching what I might call "Amazon levels", and I dont even work there anymore. Thank God.
But for all its downsides (e.g. as voiced by my brother Mike, who was in the Navy on an aircraft carrier in the Persian Gulf during the 1990s Gulf War, and later worked at Amazon, and declared after four years that Amazon was _way_ more stressful), Amazon did teach me a valuable lesson, namely that YOUR IDEA IS INSANE. It doesnt even matter what it is. Its frigging insane. You are a nut case. Because anything you try to do at Amazon these days involves touching a thousand systems, all of which are processing gazillions of transactions a second, and you want to completely redo the database schema, and _you dont even know the answers to these fucking questions, DO YOU?_
I suppose I should think of it as a valuable experience. If nothing else, I understand Complexity in a way most people will, mercifully, never have to.
Anyway, I hope Ive imparted the basic flavor of my thinking after having been totally bought into Dans idea. Heres how I (now) envision the days just after Dans meeting with the bank executives:
DAY 1: _(executives)_: Managers, wed like you to look into this incredible new idea from Dan Ariely. We think it could revolutionize consumer credit-card spending in a way that makes everyone love us and sign up for our services, dramatically increasing both our profit margin and our overall customer satisfaction. And its an incredibly simple idea!
DAY 2: _(managers)_: Programmers, project managers and marketers, wed like you to flesh out this idea from On High, and give us some time estimates. We all know we only have a budget of about 2 months for ideas from the Board this year, so lets try to make it fit. How long will it take?
DAY 3: _(grunts)_: A billion years. We quit. Fuck you.
DAY 4: _(managers)_: Executives, we think it will take about 3 years. Its surprisingly hard. We wouldnt be able to do anything else at our current staffing levels. Should we move forward?
DAY 5: _(executives)_: Two _years_? Good lord! For a project that _might_ increase our profit margins and customer satisfaction, but could also cause customers to be so confused that we have to triple our Customer Service headcount? We dont think that sounds... well, reasonable. Although it would be a very interesting experiment, its simply too expensive for us to attempt. Should we tell Dan? Well... it might be patentable, and we might be able to get around to it _someday_ if theres a sudden glut of programmer expertise, so... maybe wed better just sit on it for now. Whos up for golf?
In reality Im sure it went down a _little_ different from that. For instance, they may have had the Day 5 meeting late, and then gone to a strip club instead of a driving range.
But Im pretty sure that aside from the mundane details, thats exactly how it went down. Because that kind of shit happened at Amazon pretty much every week I was there, for almost seven years. (And astonishingly, we actually managed to launch at least half those crazy ideas, by burning through people like little tea lights. But thats another story. Plus, no bank can execute like Amazon can. Banks just dont have the culture for it. Bless em.)
LEGALIZATION
So. Im two glasses of wine into this whine. Im going to go get a third glass, mark the calories off in my spreadsheet, and then wrap up. If you dont know whats coming by now, then youre pretty stupid, but on the plus side youre an amazingly fast reader, so Ill go through the motions anyway.
It doesnt actually matter what your stance is on the legalization of marijuana, for purposes of this little essay. You could be radically opposed to it on religious, moral, or "parental" grounds. Or you could be so radically in favor that youve been laughing hysterically and rubbing your hands together incessantly ever since you started reading this post. If you know what I mean. Or you could be somewhere in between, moderate and yet open-minded.
It doesnt matter.
This blog is about _complexity_, the bugbear that haunts software developers, program managers, project managers, and all other individuals associated with trying to launch new software projects and services.
Dan Ariely would have made a great VP (that is, Vice President). If you think that legalizing marijuana is a black-and-white, lets just decide it and get the frigging thing legalized once-and-for-all issue, then you too have some VP blood in you.
VPs have what my brother Mike refers to as "Shits Easy Syndrome".
You know. As in, shits easy. If its easy to imagine, then its easy to implement. Programming is just turning imagination into reality. You can churn through shit as fast as the conscious mind can envision it. Any programmer who cant keep up is an underperformer who needs to be "topgraded" to make room for incredible new college hires who can make it happen, no matter what "it" happens to be, even if they have to work 27 hours a day, which of course they can because by virtue of being new college hires, they have no social lives and no spouses or significant others, and they probably smoke a lot of crack from being in the dorms so they can stay awake for weeks at a time.
Thats the kind of programmer we need at our venerable institution. And we are completely anti-slavery, for the record.
Shits Easy syndrome is, well, pretty easy to acquire. Heck, you dont even have to be a VP. Directors sometimes get it if they stay away from the code for too long.
As for the rest of us, well, we ought to know better. YOU, as a frequenter of reddit and a programmer (wannabe or actual, it doesnt matter), _you_ ought to know better.
Lets ask our little naggy angel: what would it take to legalize marijuana?
I dont know the answer, and Im certainly no expert, but Ive been on enough projects like this to know how to start asking the right questions.
WHAT EXACTLY DO YOU MEAN... "LEGALIZATION"?
So... one minor, teeny-weeny almost insigificant caveat before I continue: I have smoked marijuana (and inhaled it, deeply) on more occasions than I can count. And yet Im almost undoubtedly smarter than your kid that youre so goddamned worried about. I skipped three grades (3rd, 7th and 8th), entered high school at age 11 and graduated at age 14, took A.P. courses, had stellar SAT scores, was a U.S. Navy nuclear reactor operator, went to the University of Washington and earned a Computer Science degree, worked at major corporations like Amazon.com and Google for many years as a senior staff engineer and/or senior development manager, and now Im an internationally famous blogger.
I dont usually dwell on that, but today its relevant. Its relevant because Ive smoked a LOT of pot, and I dare you to prove that it has impaired me in any scientifically detectable way. We would debate, and you would lose; nevertheless I _double-dog dare you_.
So, well, sure... from _that_ perspective, yeah, Im in favor of legalization. The laws are stupid. Legalize it, already. For cryin out loud. Jeez.
However, from a _programmers_ perspective (and keep in mind that I was also an Engineering Project Lead at Geoworks for 3 years, a Technical Program Manager at Amazon for a year, a Senior Development Manager at Amazon for about 5 years, and now Im a plain-vanilla programmer with 3.5 years at Google, so Ive done it all), the idea gives me the chills. And a fever.
Because laws are pretty much like programs. You have to specify their behavior at almost (not quite, but almost) the same level of detail, using a language thats almost as crappy as our programming languages today -- English. Or whatever your native language is: it sucks too. If you dont believe me, ask a lawyer. Or try to write a technical spec in your native tongue that the programmers dont ultimately poke full of holes.
Aw, dont try. Dont even bother. Just trust me on this one. Todays natural languages are completely unsuitable for specificity, and "legalese", as much as we all love to ridicule it, is our collective best effort to permit being logical, specific, and unambiguous.
I have more respect for The Average Reddit Commenter than I have for, well, the average commenter in any other forum, period, assuming that "period" is stevey-legalese for "except for LTU, news.ycombinator and their ilk, mumble mumble."
But the Average Reddit Commenter has gone too far. Everyone these days, when debating the merits and demerits of marijuana legalization, seems to have completely overlooked the fact that its HARD. Its a project of vast, nearly unimaginable complexity.
Think about it. What kinds of laws do we have about alcohol and tobacco? Is it just one law each, saying "its legal" or "its illegal?" Of course not, and youre insulted that I asked such an obviously rhetorical question, yet intrigued by my line of reasoning. Admit it! How is marijuana similar to alcohol? How is it different? How is it similar and different to tobacco?
Lets let the little angel ask a few preliminary questions, just to see where it takes us, shall we?
* Is it legal to drink alcohol in a TV commercial? No? OK, what about marijuana, then? Can you smoke it in a commercial? Can you SHOW it? Can you talk about it? Can you show marijuana smoke at a party, without anyone actually being seen smoking it? Can you recommend its use to children under the age of 9? What exactly are the laws going to be around advertising and marijuana?
* Do we let everyone out of prison who was incarcerated for possession and/or sale of marijuana? If not, then what do we tell them when they start rioting? If so, what do we do with them? Do we subsidize halfway houses? Do we give them their pot back? How much pot, exactly, do they need to have possessed in order to effect their judicial reversal and subsequent amnesty? A bud? An ounce? A cargo ship full?
* Is it legal to sell, or just possess? If the latter, then how do we integrate the illegality of selling it into the advertising campaigns that tell us its legal to own it?
* If its legal to sell it, WHO can sell it? Who can they sell it to? Where can they sell it? Where can they purchase it? Are we simply going to relax all the border laws, all the policies, all the local, state and federal laws and statutes that govern how we prioritize policing it? All at once? Is there a grandfather clause? On what _exact_ date, GMT, does it become legal, and what happens to pending litigation at that time?
* Are we going to license it? Like state alcohol liquor licenses, of which there are a fixed number? What department does the licensing? How do you regulate it? Who inspects the premises looking for license violations, and how often? What, exactly, are they looking for?
* Is it OK to smoke marijuana at home? At work? In a restaurant? In a designated Pot Bar? On the street? Can you pull out a seventeen-foot-long water bong and take a big hit in the middle of a shopping mall, and ask everyone near you to take a hit with you, since its totally awesome skunkweed that you, like, cant get in the local vending machine? If its not OK, then why not?
* Can you drive when youre stoned? Whats the legal blood-THC level? Is it state-regulated or federal-regulated? For that matter, what is the jurisdiction for ALL marijuana-related laws? Can states override federal rulings? Provinces? Counties? Cities? Homeowners associations?
* What exactly is the Coast Guard supposed to do now? Can illegal drug smugglers just land and start selling on the docks? Are consumers supposed to buy their marijuana on the street? What happens to the existing supply-chain operations? How are they taxed? Who oversees it?
* Can you smoke marijuana on airplanes? Can airplanes offer it to their customers in-flight? Is it regulated in-flight more like tobacco (dont get the smoke in other peoples faces) or alcohol (imbibe as you will, as long as you dont "appear intoxicated"?) What about marijuana brownies? Are you allowed to eat it in areas where youre not allowed to smoke it?
* Can an airplane captain smoke pot? A ship captain? A train conductor? The driver of a car? An attendee at a Broadway musical? A politician in a legislative session? What is the comprehensive list of occupations, positions and scenarios in which smoking pot is legal? What about eating pot? What about holding it? What about holding a pot plant? What about the seeds?
* Speaking of the seeds, are there different laws governing distribution, sale and possession of seeds vs. plants vs. buds vs. joints? If so, why? If not, why not?
* What laws govern the transportation of marijuana in any form into or out of countries where it is still illegal? What policies are states able to enact? Is it OK under any circumstances for a person to go to jail over the possession or use of marijuana? If so, what are those circumstances?
* Are there any laws governing the use of marijuana by atheletes? U.S. military personnel? Government employees? Government contractors? U.S. ambassadors, in title or in spirit? What are our extradition laws? What do we do about citizens who are subject to the death penalty in countries like Singapore for the possession of sufficient quantities of what we now consider to be legal substances?
* What about derivatives? Are the laws the same for hashish? How do we tell the difference? What if someone engineers a super-powerful plant? How do the new laws extend to a potential spectrum of new drugs similar to THC?
* For driving and operating machinery, do we have legal definitions that are equivalent of blood-alcohol percentage, and if so, what are these definitions? How do we establish them? How do we figure out what is actually dangerous? How do we test for these levels? When they are established, do we we put up signs on all roadways? Do we update the Drivers Education materials? How do we communicate this change to the public?
* How does legalization impact our public health education programs? Do they have to immediately retract all campaigning, advertising and distributed literature that mentions marijuana? How does legalization interact with the "Say no to drugs" programs? Do we need extra education to differentiate between a drug that is now legal (but wasnt before) and drugs that are still illegal? Whats our story here? What about other drugs that are even less addictive and/or less intrusive than marijuana?
* Monsanto is eventually going to sue the living shit out of _someone_ for using genetically-engineered pot seeds. Can they sue individuals with a single plant in their windowsill? _(answer: yes)_ Will Oprah step in and help that beleaguered individual? _(answer: well see!)_
Im not an expert, and in fact Ive gone to extra-special effort to avoid all possibility of being accused of having researched this subject. I know NOTHING about it.
But the questions, theyre bugging me. How the hell do we implement all this? Sure, its "legal" in Amsterdam, or so they say. Ive never been there, and I suspect their laws are way too vague for the overly-litigious United States of America.
I hope its obvious that we cant say "its just like tobacco" (its not) or "its just like alcohol" (its not), or (God help us) "its just like doing alcohol and tobacco together, so take the intersection of their laws".
Marijuana, whether you like the idea of legalizing it or not, is a _project_. It requires an _implementation_, and the implementation is a lot like that of a software project. The US federal government is analogous to "The Company", and the states are analogous to "The Teams" that comprise the company. Some of them have free time; some do not. Some of them agree with the overall goal; some do not. And every single miniscule little detail has to be worked out, and written up, and voted on, and approved, and then specified, and implemented, and enforced.
And there will be bugs. And loopholes. And unexpected interactions. The best-laid plans will go awry.
People will die. Its a certainty. Some people are going to die as a direct consequence of legalization of marijuana. I dont like it, and you dont like it, and most of us would probably argue that it shouldnt hold up legislation or legalization indefinitely... but we have to take it into account. Because if its YOU who dies, smashed to death on the iceberg by your skunkweed-stoned ship captain, youre going to be REALLY pissed off. I guarantee it.
Shit is NOT easy. Remember that. Shit is NOT easy. If you think its easy, then you are being naïve. You are being a future VP. Dont be that way.
Try to think about how you would implement it. Yourself. If your boss came to you and said: "Make it happen. Yesterday." Have you ever legalized marijuana?
I havent. But I wouldnt want to be the people who decide how to legalize it. Their asses are majorly on the line. Even more than ours were at Amazon.
My advice: give it some time. Hell, give _Obama_ some time. Whether you still like him or not, hes not a frigging King, hes a President. He cant make stuff happen overnight by waving his magical sceptre. He just cant. I dont know what you were thinking, but "overnight" is a pipe dream, and "a few months" is _definitely_ "overnight", in presidential terms.
Moral of the story: Shit is _not_ easy. Stuff takes time. Months. Years. Decades. Its OK! Youll still be here. Count your calories. Exercise. And youll still be here to see it all happen.
Patience. Its a wonderful thing. I cant wait!
So Ive got all these fancy blog posts planned. More than planned, actually -- theyre well underway. But its also been a busy couple of months, so nothings really ready yet.
To make my schedule even worse, I kind of sort of got myself a little bit addicted to the writings of this one blogger. Normally I cant frigging stand blogs, including my own. Everyone always asks me what blogs I read, and I reply: "Um... do books count?" Which of course is greeted with blank stares, since nobody seems to read books anymore these days. Such a shame.
Anyway, recently I tried another fling with reddit. Im always getting addicted to reddit for a little while, and then coming off it cold-turkey for a month or two while I reassess whether I want to be spending my life reading pun threads and upmodded lolcat pictures and conspiracy theories.
Not to mention the comments, which can individually sometimes be quite cool, but in aggregate only make sense when you read them aloud with the voice of the Comic Book Guy from the Simpsons: "I upvoted you for the appropriate uses of its and their in the link title, but downvoted you because your link actually appeared on a little-known German social networking site several hours ago. I feel it is important that you understand that this is not a zero-vote of abstention, but rather a single upvote and a single downvote cancelling one another out." If you read the comments with CBGs voice, a lot of them make a whole lot more sense.
No matter how many times I quit, in the end I always wind up sneaking a peek at the reddit home page in a moment of weakness, and before long Im hooked again, following the adventures of memes about narwhals and how is babby formed and all this other stuff I have zero context for, but occasionally its hysterically funny. Im guessing this is more or less what heroin is like. Some days I dont even shower.
In any case, my latest reddit flirtation ultimately led to a secondary and more severe addiction to the writings of some dude who goes by the name "davesecretary". Yeah, him. Hes a fucking genius. I started reading his old re-posted stories, got hooked and basically blacked out and lost about three days where nobody knew where the hell I was. Eventually I made my way to his stories about his trip to China. At that point I became actively envious of someone for I think the second or maybe third time in my adult life. Im just not the jealous type normally, but damn that guy can _write_. He has a real gift.
So now Im flat-out jealous. Im not going to make a secret of it, either, and pretend hes started some trend and Im just a bigger light jumping on the bandwagon or some lame shit like that. No, he writes way better than I do, and Im just going to copy him blatantly, like Terry Brooks rewriting Lord of the Rings line-by-line badly as Shitstorm of Shannara.
I had actually already been thinking of publishing some true stories. In fact before I stumbled on davesecretarys tales, I had written down the story below about my brother Dave taking out the garbage, and I had been planning more. But Dave Secretarys stories were like the Great Pyramids next to the mud pueblo of my own ambition. Hes like a force of nature; stories just surround the guy like gnats, and he spews out the funniest, most interesting possible versions of them imaginable.
So while I work on my more technically ambitious blogs-in-progress, I thought Id kick off a series of light reading. Its all true stories. None of it is anywhere near as cool as trying to find a box bigger than Kyles goddamn cereal box, and nothing in them is anywhere near as funny as the line "Iodine, Dave." But at least these are my own stories, and theyre kinda fun to write down once you get going on them. I probably have about a hundred stories like the ones here. Ill start with nine random stories that managed to get themselves written first, in no particular order.
I hope you like at least one or two of them. If not, well, Ill read your complaints out loud as CBG, so well be even.
-------------------------
So this one time Im when in my early 20s, a fragile time during which Im as fat as a walrus, Im spending way too much time thinking about the name of the Star Wars character "Dash Rendar". Hes some random Star Wars dude, except hes not from one of the actual movies. Hes only in the aftermarket books or games or toys or whatever.
I have no idea where Id heard the name "Dash Rendar", but I can tell you I absolutely loathed it. Lucas names are always a little hard to believe, but this one was just too much. I guess when youre that fat its easy to get irritated by little things. At that point in my life I was genuinely annoyed that the famous one in the family hadnt been little Pete Rendar or Floyd Rendar or whoever else with a normal name. No. It just had to be frigging Dash.
Im telling you all this so you can understand my frame of mind for what happened next.
Im coming up the elevator from the underground garage because Im too obese to take the stairs, and Im in a surly mood on account of this stupidly-named character Dash Rendar. Im trying to understand basic human nature here: how could anyone, even a nine year old, suspend disbelief for such an idiotic name?
Im all alone in the elevator, so on the spur of the moment I decide to pretend to be a nine year old and see if it makes any difference. So I make this muscle pose and say in my most ominous nine-year-old voice: "I am DASH RENDAR!"
Right then the elevator doors open and fucking Dash Rendar walks in. I am not making this up. If you lined up ten thousand guys and asked a hundred people to point to the one whose name in real life was most probably Dash Rendar, theyd all point to this dude.
He of course witnessed my little announcement, pose and all, and all I could do was stand there slack-jawed, trying to think of a clever and succinct way to say: "I was just sort of trying to understand what kind of idiot would call himself Dash Rendar, and itjustsortofrequiredmetoawfuck." Dash was staring at me with a look like hed just stepped in dog shit, except when he lifted up his foot hed found me there instead. I was embarrassed enough to wish I would die spontaneously, but not quite enough for it to actually happen. So instead I just walked out of the elevator and never came back there again. Ever.
Afterwards I definitely established some new rules about what kinds of things Ill try in elevators.
-------------------------
My brother Dave used to work as a waiter at Applebees. This was back about 2 years after hed graduated high school as a varsity football player, after which hed settled into a life of comfort and no small quantity of pizza, and hed blimped up to about 260 pounds. But it happened so fast that he wasnt entirely in touch with how much he actually weighed.
One day before the restaurant opened they were having the usual employee meeting in the back room, where everyone stood in a circle while the manager ran them through the days specials and whatever else waiters need to know for the day. Dave was sweating and getting a little fidgety, so he found a chair and sat down.
The next thing that happened was a loud BANG, and everyone looked over at Dave, who was sitting on the floor with his mouth open in a big round O. He had completely flattened the chair. It had once been a strong industrial solid metal chair, but now all four of its legs were sticking in different directions like a baby deer crushed under a UPS truck.
Everyone was a little stunned, and the boss felt like he needed to say something to break the awkward silence, so he said delicately: "Gee, Dave, looks like it might be time to start cutting back a bit, eh?"
Dave said later (after losing like 100 pounds) that at the time he didnt know what the FUCK the boss was talking about, and all he could think was: This chair is DEFECTIVE!
-------------------------
So that reminds me of this time I was watching the Seattle Seahawks play at their (at the time) brand-new stadium. We had sweet seats on the 50 yard line, in the second row back, and we were watching a pretty awesome game between the Seahawks and I think it was the Bears or the Giants. I forget which, but it was definitely one of the two. Probably the Bears, so lets go with that.
Anyway, the row of seats in front of me was occupied by some Bears fans. There were about twelve of them, and every single one looked to weigh well over four hundred pounds. It was seriously the fattest row of people Ive ever seen. They were squeezed up against each other in this tangle of arms and legs because they couldnt fit in their assigned seats that had been designed for ordinary three-hundred pound fatass football fans.
Every time the Bears had a good play, they would all stand up and roar "YeaaaaAAAAH!!!!", and then theyd all sit down at the same time, on account of their arms being interlocked due to the aforementioned fatitude. They did this little act over and over for the first half of the game: leaping up, bellowing fiercely, then crashing down again as a unit.
Finally I think just after half-time the Bears executed an especially good defensive play, and they all stood up and screamed wildly as usual. But this time when they crashed their asses down, they ripped the bleacher right out of the fucking concrete.
There was this horrible thundering tearing noise like an earthquake, and the whole row of whales spilled forward onto their faces, with their twelve giant asses sticking up at the rest of us. It looked like a bomb had gone off. There were like twenty iron girders that had been ripped right out of the concrete. To say it was the most wonderful thing Id ever witnessed would be a gross understatement. I think even now, ten years later, it might still be my favorite whale ass sticking up scene in my whole memory.
To make it all even more sublimely beautiful, our beer splashed all over them in slow-motion because the cup-holders on the backs of their seats were thrown forward with the rest of the wreckage.
Anyway, after a few seconds of general shock and hilarity, the ushers ran over to see if the herd were all OK. They had finally managed to extricate themselves and wipe most of the beer off, and they all stood up and started screaming: "This bleacher is DEFECTIVE!"
-------------------------
Once my family all went to this Chinese restaurant in Seattles Chinatown. It was one of my favorites, and on this particular occasion I was there with my at-the-time girlfriend Stephanie. Stephanie just happened to be mainland Chinese, from Beijing. But unlike everyone else Ive known from Beijing, who by and large seem like pretty laid back people, Stephanie was unusually sensitive to race and culture issues, and she kept a keen eye out for any perceivable slights or offenses. (Everyone else said she was just plain nuts, but Im trying to be charitable here.)
Steph asks my brother Mike what hes ordering. He ponders the menu a while and answers: "Fly lice." Stephanie instantly blows up at him: "What do you mean FLY LICE!!!1!!?? Youre insulting our culture? We have 5000 years of Chinese culture and you Caucasians are always insulting us! I cant believe you made fun of it and called it fly lice! Thats very rude! Thats not how we pronounce it! I cant believe Im so offended!" and so on in that vein. Everyone else in the family, including me, suddenly becomes tremendously preoccupied with trying to figure out how to do origami with our chopstick wrappers.
Mike listens patiently, since lets just say this sort of outburst isnt wholly unfamiliar territory when Stephanies around, and after she finally simmers down a little he says gently: "Uh... ok. Sorry."
So the (Chinese) waitress comes around and is taking our orders, and when she gets to Mike, Stephanie starts hyperventilating a little in case shell have sudden need of some extra screaming oxygen. Mike says in his blandest and most American possible voice: "Id like to have the fried rice, please."
The waitress nods happily and says "FLY LICE" really loud and writes it down on her pad. Mike, always an excellent poker player, manages to keep his face pretty straight, but I can tell his eyes are sparkling just a little. Meanwhile Stephanies eyes have grown to the size of large saucers, and she hisses loudly: "She said FLY LICE!!!!!! Hee hee hee hee hee hee HEE HEE HEE HEE HEE!" Mike just shrugs, like, "hey man, I just want to not get yelled at anymore."
We dont call it fly lice now, though, since you just never know.
-------------------------
When I was growing up the holidays were always crazy because I had over 20 aunts and uncles, and seventy or eighty first cousins, and of course we had to do a massive holiday family party at my grandparents teeny house every year. Every family would bring a few friends, so there would be well in excess of 100 people running around in pure holiday chaos for a day.
We used to do this secret-santa gift exchange where everyone would put their name in a hat a few weeks in advance, and then wed all draw names, and that was the person you had to get a gift for. It wasnt split into adults and kids or anything like that.
With an extended family that big, its kinda hard to keep track of everyone, especially the new additions. It was also kinda hard to know what people actually wanted, and some of my uncles werent too big on preparation. Uncle Harold was pretty much the worst at it. We loved him to pieces, but he just couldnt get the hang of the holiday party or the secret santa exchange.
To give you the basic Uncle Harold flavor, one year he didnt show up because on the way to the party he decided to stop in a bar down the street, and he stayed there drinking bloody marys until midnight. Another year he was tasked with bringing the turkey, and he showed up at dinnertime with a frozen-solid turkey. And one year his secret santa target was Aunt Celie, who was infamously religious and pretty much nothing else, just religious, so Harold, totally at a loss, got her a picture of Jesus and taped fifty bucks to the back of it. All the adults were pissed off and all the kids thought it was pretty much the funniest goddamn thing in history. Either way, Uncle Harold just couldnt win at Christmas.
But this one year he really outdid himself. I think I was about 11 or 12 years old, just old enough that I was no longer running around screaming like the next generation of cousins. So I got a chance to sit back and watch the party unfold. Im sitting there watching Uncle Harold give his secret santa gift to one of my cousins, one of Aunt Dianas kids. For some reason Harold looks embarrassed.
My dad had 11 brothers and sisters, all married with like 6 or 8 kids each, and there was no way any of us could really keep track. Usually wed get in touch with an immediate family member and try to get some direction, but Harold, as usual, just winged it. I dont remember which cousin it was, but last time Harold had seen him, he was a baby, so Harold, thinking it through with his usual clarity, got him a jar of Gerber baby food.
Unfortunately it turns out its been like six years since Harold had last seen him. So my cousin opens up his present and I hear him saying to my aunt Diana: "Mom. Uncle Harold got me baby food." Diana was busy with two other crying kids, and shes not feeling particularly merciful, so she says: "Well go tell him thank you."
So my cousin walks over to the tree where Harold is sitting, and he says: "Thank you for the baby food, Uncle Harold."
-------------------------
This story is more grim than funny, but its a story. With sequels, no less!
In college I lived in a big four-bedroom apartment with 3 roommates for about a year. It wasnt air-conditioned and during summer it was hotter than hell, and we all kept our windows open most of the time.
I was just starting to drift off to sleep one night, laying on my back, when I heard this loud clicking noise. It sounded like snapping a ball-point pen out of its plastic cap, and the clicks were coming at roughly one per second in irregular bursts.
The clicking noise was coming from between my eyes. Not, like, a foot away, or an inch away, but from directly between my eyes, near the bridge of my nose.
I remember thinking: "Thats odd. Whatever could be making that loud clicking noise coming from between my eyes? Ive never heard anything like that before."
Part of me was falling asleep, and was thinking "mufbmlflfsleeep go sleep go sleep mblfjust dont worry abouuuuuut it", or something like that.
But there was this other little part of me, a tiny voice, that was busy thinking out loud to its tiny self: "It could be a spider."
My falling-asleep part woke up a little at that point, and pondered it for a few dozen clicks. Then it replied: "No. No, its not a spider, because that would just be TOO horrible to contemplate. So thats not what it is."
And the tiny voice was like, "Well you come up with a non-spider explanation then. You cant just ignore loud clicking noises coming from BETWEEN YOUR EYES, dude."
Falling-asleep voice: "Spiders dont make noise. So there."
Tiny voice: "Small ones dont. Medium-sized ones dont. But we dont really know what BIG spiders do, now, do we? I bet a really big one could make a noise just like that."
At this point, opinion-wise, Id say I was coming down solidly on the side of falling-asleep voice, but neither the clicking nor the tiny voice would shut up, so I rolled over onto my side so I could think about it from a different angle.
As I rolled over I felt something huge jump off my face. It landed on my arm and started dragging itself along with what felt like a dozen scrabbly legs, and of course I was like WAAAAAAAAAAAH!!!! and I jumped out of bed faster than any human being has ever done before, and turned on the light.
Falling-asleep voice and tiny voice were both, like, "Oh, shit!" Sitting there next to my pillow was the biggest fucking wolf spider Ive ever seen in my life. It had a good four inch legspan, and it was very specifically glaring at me. There was no mistaking its expression. It was really pissed off. At me.
So I do the only thing I know how to do in Large Spider situations, which is to scream for help. "AAAH!!! A FUCKING SPIDER CRAWLED ON MY FACE!" My roommates doors all started opening and they came out in their bath robes, all keenly interested to see the spider that had crawled on my face.
Jacob is first to arrive. He takes one look at this massive glaring spider and says: "Holy shit dude, that things as big as a horse!" and leaves as fast as he came in.
Dave shows up next. "Holy crap, that thing crawled on your face? You must be psychologically damaged!" Dave opts to stand and goggle at the giant spider, from a safe distance.
Then Eric comes in and says: "Oh, its just a hobo spider. Ill take care of it." He walks into the bathroom and grabs ONE square of toilet paper. Not a whole handful, no, just one teeny slice, like hes going to wipe the spiders butt with it or something, and he goes and grabs the spider with it. We can clearly see the spiders legs waving around outside the toilet paper, and Eric says: "Ill just put it outside your window."
I of course was having none of that crap. "How the hell do you think it got on my face in the first place? Flush it! Now now now! Flush!" Eric, clearly disappointed, flushed the thing, and Im certain that it now lives in the Seattle public sewer system, feeding on rats and stray dogs and plotting its revenge on me.
My actions that day clearly angered the Spider God, because over the next few years I had several more spider-on-face incidents, at least one of which was way more dramatic than this one. Its a bit traumatic to tell both stories at once, though, so Ill wait a bit on that one.
-------------------------
When my brother Dave was around 14, our family lived in a house in Southern California. It was kinda rainy at the time, which is sort of unusual for where we lived. On this particular California winter day it was Daves turn to take out the trash.
Our city-issue trash bin was out in the carport, this sort of concrete alley next to the house where our big Dodge van was parked. The trash bin was on the other side of the van, next to a six-foot brick wall separating us from the neighbor.
Dave grabs the trash bag from the kitchen and heads outside. He walks over to the trash bin, opens the lid and sees that its completely full. Theres a plastic garbage bag right at the top, and the lid just barely closes over it. Dang.
Its starting to rain pretty hard, and Dave just wants to get his bag into the can and get back inside, so he figures hell do the old Human Trash Compactor trick. He puts down his bag, climbs up on the brick wall, aims for the bag on top of the bin, and jumps.
Usually this works pretty well, right? Your plummeting body weight smushes the bag down just far enough to make room for another bag, and you hop out unharmed.
Unfortunately for Dave, it turns out that this time the bin was completely empty except for the bag on top. That one bag was full of trash, but it was so bulky that it hadnt fallen down into the bin.
So Dave jumps off the wall, plunges all the way into the bin, and the lid slams shut on top of him. His momentum makes the bin tip over, and it wedges itself solidly between the van and the wall, trapping Dave inside with the garbage.
Daves no more claustrophobic than the next person, but the whole thing caught him a little off guard. In his minds eye hed been envisioning some sort of heroic plunge that would compress the garbage about a foot or so, followed by a dashing bounce onto his feet on the driveway. Instead, in just under one second hed materialized sideways in a dark stinky fallen-over garbage can getting pounded by rain, and the lid wouldnt open.
So he did the first thing that came naturally to him, which was to panic and kick and scream and thrash and flail and try to claw his way out. Pretty much what you or I would have done.
However, he was really stuck in there pretty good, and nobody could hear him because of the rain, so it took about ten minutes of violent side-to-side heaving before he finally rolled the can out from between the van and the wall.
So were all sitting in our nice, comfy living room, watching TV. Dave has gone to take the garbage out. Ten or fifteen minutes later, after weve all totally forgotten about him, the front door bangs open and Dave barges in, wild-eyed and soaking wet and yelling at us at the top of his lungs. Hes completely covered in garbage: coffee grounds, smears of leftover food, pieces of dirty paper, part of a banana peel on his shoe, brown slime all over his head and arms and legs and clothes, and hes screaming: "DIDNT ANY OF YOU HEAR ME? I WAS STUCK IN THE GARBAGE CAN!"
-------------------------
Back when I started working at Amazon.com in 1998, the company was in this little building in downtown Seattle in kind of a bad part of town. I mean it wasnt terrible, but there were definitely some issues. There was a needle exchange across the street, which was cool and all, but a fair number of the drug users did their thing in the alley behind the building we were in. So you didnt really want to walk out the back door, or youd run the risk of stepping on some dude with a syringe dangling out of his arm.
The nearest place to eat was the Scaryaki joint across the street, next to the needle exchange. It was this teriyaki place that we all called the Scaryaki, since even though the food was pretty good almost nobody ever went there because it was scary. Usually wed walk a couple blocks past it, which took us to Pike Place Market, which is also scary in its own way but is much less overtly threatening. Plus theres more culinary variety.
One day my friend Jacob and I left the zon premises to eat at Matts in the Market, which was this really awesome hole-in-the-wall joint that was Zagat rated and nobody knew about it and it was delicious.
On the way back to work, were crossing the street in front of the Scaryaki, and I cant help but notice Jacobs doing an unusual amount of looking at my ass and my legs. Were both straight, and as a consequence we usually try to avoid staring at each others asses as a general rule, so Im a little annoyed. But hes definitely checking me out with waaaay too much interest.
I finally give him the Glare, and he says: "Whered you get those pants from?"
I shrug. "I dont know."
Jacob is now really keenly eying my pants, which are these dark blue cotton slacks Ive been wearing to work on and off for at least a few months now. He starts being more demanding. "No, really, whered you get those pants from?"
Im like, "Dude, I dont know! Stop looking at me!"
Jacob suddenly announces, in a really loud voice that everyone within a block or so can hear: "Youre wearing my pants!" A couple of people in the vicinity, including some druggies and some coworkers, perk up with some interest as the argument unfolds.
Me: "No Im not! What are you talking about?" __
Jacob: __ "Those are my pants! Whered you get them from? Those are mine!"
Me: "I dont know where I got them! I just had them! Theyre mine, OK? Leave me alone!" Im now running full bore for the doors, since I want to get back to work and out of this sudden surreal nightmare.
And all the while Im thinking to myself: "Where DID I get these pants? I never buy blue slacks. I found them in my closet one day, so they MUST be my pants. They fit pretty well, even though theyre a little long. So theyre mine! I must just not remember buying them!"
We burst into the lobby, where everyone in the frigging company is on their way back from lunch, and Jacob is running after me yelling: "I recognize those pants youre wearing! Those are MY PANTS!" There was chaos for a while. It was pretty messed up.
The funny thing is, it turned out they actually were his pants. He and our mutual friend Jeff and I had all gone out to El Gaucho earlier that year. Its the awesomeest steakhouse in Seattle, arguably in the whole country, and we all dressed up after work one day to eat there. I drove, and they threw their work clothes into the trunk of my car, which was almost completely filled with random crap. When they got their clothes out later, Jacob didnt see his pants or something, and I must have thrown them in my laundry when I finally cleaned my car, months later. One day I started wearing the pants without really thinking too deeply about their origins, and the rest was history.
He probably could have picked a better way to tell me, though.
-------------------------
I remember when I was 23 years old, my dad decided to have one of those Dad to Son talks. Hed clearly thought seriously about it, because he sat me down and gave me one of those This Is Important looks. He said: "You know how sometimes they lose your file?"
"Uh... what?"
"You know, like when you call up to make a dentist appointment, and then when you get there they have no record of the conversation? Or you set up an account with the cable company, or whatever, and they cant find it later?"
"Er... yeah, sure. That happens to everyone sometimes."
My dad is like, "Nope. It happens to us way more often. And when it happens, tell them to look under W."
"What? W?"
"Yep. Ask them to look under Wegge instead of Yegge. 9 times out of 10 theyll find it."
"WTF?"
"Yep. Its weird. But when you spell our name, Y-e-g-g-e, a lot of people write or type W instead of Y."
Im thinking, "You waited until Im 23 years old to tell me this?" But Im also thinking: "Damn, are people really that stupid? And if so, how the hell didnt I notice this myself?"
"Uh, thanks Dad. Ill keep an eye out!"
So I watch. And listen. And sure as shit, hes absolutely right. The percentage is pretty high, like maybe 10% to 20% of the time. Someone (in person or on the phone) will ask me to spell my name, and Ill say Y, and theyll enter W. A lot of the time Ill actually be in a position to watch them as they do it -- Ill be looking over the rental-car counter or whatever, and when youre looking at a keyboard upside-down from the side, you can see the W as they type it.
I have no idea how many years it took my dad to figure this out, but hes a pretty perceptive dude, and he was 43 when he told me. So were talking about half a lifetime of watching people fuck up, and eventually realizing theres a pattern to it. Bravo.
Nowadays Ill say "Yellow Echo Golf Golf Echo" if Im on the phone, since it slows them down just enough to think about what theyre typing. In person, when I can see what theyre typing, I still say "Y", because Im always dying to know if theyre one of Those People. If they get it wrong, Im like, "No, Y, not W", and they always say: "Gosh, I have no idea why I did that!"
Me neither. But I think its because when you pronounce the letter Y, it starts with a W-sound, as in "WHY"? Sometimes it even seems as if the slower and more deliberately I say Y, the more likely they are to get it wrong, because my lips started off with this big W sound.
Anyway, the best one ever was at a Jiffy Lube. The Lube Supervisor Dude was asking me for my personal information and writing it all down on a form on his clipboard. He apparently felt he was better qualified to write my personal data on the form than I was, and to be fair, he had pretty good handwriting.
He asked me to spell my name, and I said: "Y". He wrote "W". So far, so good. I really didnt want these fuckers to have my personal information just because they gave me an oil change, anyway.
I said "e", and he wrote "i". Wow, this was new.
I said "g, g" and he wrote "jj". Cool!
I started to get kind of excited to see if hed get every letter wrong. I said very carefully: "e", and he wrote "i", completing my last name as "Wijji". It was Jiffily Lubriciously Awesome. I told him "THATS EXACTLY RIGHT!", like hed just won on Who Wants to be a Millionaire, and he gave me this big shit-eating grin with several missing teeth. It was nice to make him feel happy.
But check it out: I told this story to my friend Gayle, whose last name has a bunch of doubled letters in it, including ending in "nn". She said that shed found empirically that in order to get people to write "nn" at the end of her name, she had to say as many as four or five Ns. Shes gotten into the habit of spelling it with "enn enn enn enn enn" at the end so theyll write two of them.
Go figure.
-------------------------
OK, last story for today.
Ive been a snowboarder for almost 20 years, but waaaay back in the old days I actually used to ski. I learned in my late teens, and I spent 2 or 3 seasons skiing, mostly using Navy rental equipment. I have all sorts of Navy stories to tell, but theyll have to wait for another day. Suffice it to say the Navy provided the rental equipment for this story.
This was back when I was at the US Navy S5G Nuclear Reactor Prototype training plant in the middle of Idaho. "S" was for "Submarine", 5 was the reactor model number, and "G" was for General Electric, who had made the reactor. Hmmm, maybe I should tell some of my reactor stories someday... theyre definitely interesting. Anyway, the reactor was at a Department of Energy site in the desert in the middle of Idaho, with a 60-mile bus ride from the nearest city, Idaho Falls. Ill leave it to your imagination as to why they located the experimental and training reactor facilities out in the middle of a desert, but your first three guesses are probably all more or less correct.
Anyway, the downside to the program was that we were in Idaho, where the fun thing for residents to do is follow potato trucks in their cars, trying to hit dislodged potatoes. Thats pretty much the pinnacle of entertainment in that fugging dustbin wasteland. The upside was that there were some really nice ski resorts nearby, so during winter we got to do a lot of skiing.
This one time we were either at Jackson Hole or Grand Targhee. Both of them are in Wyoming, but only about 2 hours from town, so we usually went to one of them. The way the Navy Nuke program works during the prototype training phase is that you work 7 days on, 1 day off, 7 days on, 2 days off, 7 days on, 4 days off. This is actually a pretty doable schedule, and the once-a-month 4-day weekend was always something you planned in advance, since thats a lot of days for screwing around.
A lot of my friends were skiers, and snowboarding was still pretty new back then. There was one dude, I think his name was Lundberg (everyone goes by last names in the Navy), who was learning to snowboard, and he pretty much spent all his time falling on his ass. So I decided to ski. The Navy pretty much takes care of all your needs, including ski rentals, so I went to the local Navy ski rental place and got hooked up with some rental skis, boots and poles.
Unfortunately this one time, which was probably no more than my seventh or eighth trip ever, the rental guy adjusted my bindings too loose. Bindings are the things on your skis that hold your boots in place, and you dont want them to be too tight or you could break your legs in a bad crash. You want them to be tight enough to hold during an aggressive turn, but loose enough to pop out in a crash. And mine were way too loose.
We ski for a few hours, and Ive warmed up a little, so I want to work on my turns instead of snowplowing down steep stuff like a sissy. So I follow my friends to a black diamond. When I was skiing I would actually follow my friends pretty often, which was completely idiotic because theyve been skiing since they were in the womb, and they look like frigging Olympians, and theyre always dragging me off to slopes I have no business being on. But for some reason I still did it.
This slope is a pretty steep black -- nothing for me today on my snowboard, but back then it was way out of my league. But it doesnt hurt that much to fall, so I went ahead and tried a real turn. Of course as luck would have it my binding popped out and I went flying straight down the slope on my face. It was so steep that I actually started picking up speed, and I wasnt experienced enough to know how to stop myself, so I just started sliding faster and faster downhill, face-first on my stomach.
I remember being vaguely aware of people stopping and looking at me and pointing as I accelerated by them, but after a hundred feet or so I had picked up enough speed that I couldnt really do anything except this emit gurgling screaming noise. The snow was bunching up in my face and going in my eyes and up my nose and down my throat, not to mention down my shirt and pants, but all this plowing wasnt really slowing me down, so I wound up sliding about three hundred yards on my face. I didnt have any really good ski gear at the time, though. I was actually wearing baggy jeans, if you can believe it. So when I near the bottom stopped I was caked in snow and ice.
Someone was nice enough to bring me my ski from way up the slope, and I hobbled into the lodge to grab a beer. I was sitting there and everyone was laughing and having a great time, but I was FREEZING. I went and sat by the big fire, but it wasnt helping. I started shivering badly. Finally I stood up and lifted up my shirt and about ten pounds of snow fell out, which sort of explained it.
My brilliant friends told me that I should go back up for another run before we left for the day. They said you stay warm on the slopes, which is more or less correct if youre actually skiing and not being a human snowplow. I really wanted to warm up so I went back up for one more run.
At the top I took a cat-track between slopes -- just a road, basically. The cat-track was running along the side of a steep roped-off drop-off with a bunch of trees. The cat track headed for the drop-off, then made a sharp turn to the left. When I got to the turn, I turned too hard, and my right binding snapped open and my ski shot out from under me, and of course I wiped out.
Im laying there on my back, thinking "man, Im just not sure about this whole skiing thing", and an older guy skis up next to me and asks me if Im OK. Im still laying on my back, with my right leg bent awkwardly under me, trying to catch my wind. I tell him I lost my ski, and that I think it went over the edge. I ask him if he can see it.
He skis over to the ledge. I lay there, sun shining on my face, snow on my back, freezing my balls off, just kind of trying to enjoy the moment as best I can while he assesses the situation. I look over and hes at the ledge, scanning the mountain. A bird flies by. Some time elapses -- probably no more than 30 seconds, but it seemed like eternity.
I finally ask the guy, "Hey, can you see my ski?" He looks at me, then back down the mountain. Then he takes a deep breath as he decides how best to tell me. Finally he speaks.
"Yep. Shes still on the run."
This is the second installment of a little series of discussions. Theyre not much more than that, just discussions. And I hope Im _inviting_ discussion rather than quenching it. But Ill be honest -- the goal of this series is to pound a stake through the heart of a certain way of thinking about the world that has become quite popular. If my series fails in that regard, I hope it may still provide some entertainment value.
Part one, The fish, was about a twisty line and a fishs examination of it. Today we move to a twisty plane.
EMBEDDED SYSTEMS
There are many kinds of computer programs, and many ways to categorize them. One of the broadest and most interesting program categories is _embedded systems_. These systems are the centerpiece of todays little chat.
Embedded systems are a bit tricky to define because they come in so many shapes and sizes. Loosely speaking, an embedded system is a little world of its own: an ecosystem with its own rules and its own behavior. So an embedded system need not even be a computer program: a fish tank is also a kind of embedded system.
We call them _embedded_ systems because they exist within the context of a _host system_. The host system provides the machinery that allows the embedded system to exist, and to do whatever it is that the embedded system likes to do.
For fish tanks, the host system is the tank itself, which you may purchase from a pet store. A tank has walls for holding the water in, filters and pumps for keeping the water clean, lights for keeping the fish and plants alive a little longer, and access holes for putting your hand through to clean the tank. Theres not much to it, really. The embedded system is everything inside: the water, the plants, the rocks, the fish, and the little treasure chest with bubbles coming out.
For computer games, another popular kind of embedded system, the host system is the computer that runs the game: a PC, a game console, a phone, anything that can make a game exist for a while so that you may play it.
Programmers have been building embedded systems for many decades: a whole lifetime now. It is a well-studied, well-understood, well-engineered subject. Gamers understand many of the core principles intuitively; programmers, even more so. But in order to apply all that knowledge _outside_ the domain of embedded systems, we will need some new names for the core ideas.
The most important name is the One-Way Wall. I do not have a better name for it. It is the most important concept in embedded systems. In lieu of a good name, I will explain it to you, and then the name will stand for the thing you now understand. Its the best I could come up with.
But first lets dive into an embedded system and see what this wall looks like from the inside.
MARIO KART
I will assume youve played Mario Kart, or youve at least watched someone else play it. Mario Kart is the ultimate racing game in terms of sacrificing realism for fun. It bears so little resemblance to reality that its a wonder it tickles our senses at all. The Karts skid around with impossible coefficients of friction, righting themselves from any wrong position, and generally making a mockery of the last four centuries of advances in physics.
Its really fun.
Mario Kart, like all games, has a boundary around the edge of the playing area. In Mario Kart you bump into it more often than in most other games, which is part of the reason I chose it to be our Eponymous Hero. If you want to win a race, you will need to become quite good at accelerating around corners, which means you will spend a fair amount of time bumping up against an invisible wall.
You know the wall Im talking about, right? Its invisible, so you can see right past it to the terrain beyond. But the wall is there, and you are not permitted to venture beyond it.
In slower-paced games, when you arrive at the invisible map boundary, you will sometimes be told by the game: "You cant go that way. Turn back!" And since that is your only option, you comply. These invisible boundaries are non-negotiable.
In other games, you may stop on contact with the boundary, or perhaps bounce off. But the boundary is always there.
Imagine Mario and Luigi chatting about the you-cant-go-that-way wall. Their conversation might go something like this:
MARIO: "Luigi, my brother!"
LUIGI: "Maaaarioooooo!"
MARIO: "Yes, Luigi. I am a here. Tell me my brother, why is it that every a time I a spin around the cactus in the third a bend of the Desert a Track, I get a stuck and have to start accelerating from nooooothing?"
LUIGI: "Whaaaaa?"
MARIO: "Brother, the Desert a Track! Its Number a Three! You know the big a bend, where you have to slow down? I am always a forgetting to slow, and I just a stop. Just a like that!"
LUIGI: "I dont know, Brother. That Bowser, he is always a squishing me right before I hit that turn, and I am a flat like a pan a cake for a looooong a time."
MARIO: "What about that a time two races ago, where Wario hit you and you a spun around and you a headed for the hill, and you got a stuck and wailed for me?"
LUIGI: "Ah, that Wario! I will get him next a time!"
MARIO: "Why did you not just turn around then?"
LUIGI: "My Kart, it did not a move, no matter how I wailed."
MARIO: "That is a what I am a speaking of, Brother. Your Kart moves in all other places, but if you head for the hills, it just a stops a suddenly!"
LUIGI: "I a _hate_ a stopping a suddenly."
MARIO: "Yes, Brother. So do I. But why can we not traverse that a part of the hill? What is on the other a side?"
LUIGI: "I think it is Track 4, Brother. They do not want you to wind up in the lake."
MARIO: "Whaaaa? Who is this a they? And why can we a not see the lake from a Track a 3? I think there is a nothing there, Brother. Just a more hills."
LUIGI: "No, my brother. I think it is a Track a 4. Or maybe Track a 2. There must be a something there."
MARIO: "Perhaps you are a right, my brother. We should get a back to a racing now. We can talk a more about a this after the next a race."
LUIGI: "Yes, brother. I will a get that Wario this a time!"
WELL-FORMED NONSENSICAL QUESTIONS
In our Highly Realistic Dialog, Mario is wondering about the Invisible Boundary at the edge of the track. He asks his brother the seemingly obvious question: "What is on the other side?"
As a gamer, if you pause to consider the question at all, the answer seems to be: "Nothing I care about." The invisible (or sometimes visible) Boundary seems just like any other wall. It is designed to keep you in where youre at, or out of where youre not, depending on your point of view.
But the gamers view is that the boundary _does_ have another side. You have no way to get there, but it exists. For maximum gameplay immersion a game universe needs to appear consistent. Thus, when you peer through the wall it appears that the other side is just more of the same.
To an embedded systems programmer, Marios question is complete nonsense, like a fish asking the temperature of the water on the other side of the glass. There _is_ no water on the other side, and for that matter a fish cant ask questions, so the question itself is based on nonsense premises.
From a perspective within the Mario Kart universe, the "other side" of the Invisible Boundary is... a kind of nothingness. The embedded world quite literally _ends_ at the boundary. The level designers usually try to make it appear as if the current world continues on, but this is an illusion. They discontinue the models polygons beyond the line of sight. Put another way, the water stops at the edge of the tank.
The nothingness beyond the Invisible Boundary of an embedded system is much deeper and more significant than simply not having objects there. In that nothingness there is no programmatic substrate upon which to _place_ objects. If you were to ask: "What lies between Marios head and Luigis head," the answer may well be "nothing", since no game objects may overlap the line between their heads at the particular time you ask the question. But that "nothing" is qualitatively different from the "nothing" on the other side of the Invisible Boundary. Between Mario and Luigi there is a _space_ - a place in their universe where objects and forces and logic apply, even if they are Mario Karts twisted versions of physics and logic. That universe ends abruptly at the surface of the boundary.
The question "Whats on the other side" is well-formed in a strictly mathematical sense. You could project a line from Mario to the nearest boundary, and ask the more precise question: "If Mario is at distance D from nearest boundary B, what overlaps the point P obtained by extending the Mario-boundary line L to a distance D beyond B?" ("Whaaaaa?")
The new formulation of the question is more well-formed, but it is every bit as semantically nonsensical.
WHATS REALLY ON THE OTHER SIDE
In the context of the Mario Kart universe, the system boundary truly only has on side, and Mario and Luigi are on that side. From their perspective, we cant meaningfully ask the question "Whats on the other side?" However, there _is_ a semantically significant interpretation of "the other side" of that invisible boundary. To get to this better answer we must leave Marios universe.
From the perspective of an embedded systems programmer, the entire Mario Kart universe is data structures laid out in the memory space of some machine. There is a program -- the game engine -- that _interprets_ the data in a way that makes it appear to be a universe that is similar to ours in various intuitively appealing ways, yet different from our universe in various fun ways.
It is very unlikely that the polygons and other level data are laid out in strictly contiguous order in the host machines memory space. It is more likely that they are spread out, with gaps, like dominoes spilled on a tile floor.
The question "Whats on the other side?", when viewed from the perspective of a systems programmer, might be phrased more precisely and meaningfully as follows: "What occupies the memory spaces adjacent to the memory used by the Track Three Desert Level?"
This is the same as Marios question, but we had to step outside the Mario Kart universe in order to ask the question in a way that made sense. The Mario Kart universe, like most game universes and in fact most embedded systems, is designed to appear _complete_. There is apparently "stuff" beyond the boundary; you just cant go that way.
When we step up into the host system and ask the analogous question about the "other side", both the question and the potential answers become much more complex: many orders of magnitude more complex, in fact. Fortunately, due to the Mario Kart system being so simple, increasing the complexity still gives us a vaguely accessible problem.
Lets try to cook up an answer to this new, more complex question regarding the contents of the program memory space on the "other side" of the invisible wall.
In terms of atomic program units (say, bits or bytes), the amount of memory used by a game like Mario Kart is actually high enough to defy our sense of scale. A game with hundreds of megabytes or gigabytes of runtime data has billions of discrete elements, which is too many for us to keep track of individually. One of the key jobs of a programmer is to organize their code so that the elements can be managed at human-CPU scale: up to seven "things" at a time. But this organization cant mask the fact that billions of individual elements exist, each with its own role to play in supporting the embedded system.
Hence, even for a game as "simple" as Mario Kart, were stuck imagining how it works internally. Even the programmers who built the game have only a dim and incomplete mental model of the game. Its like building a skyscraper, or a city: you build it a piece at a time, and its rarely fruitful to try to picture everything inside of it simultaneously.
Anything that goes wrong or is out of place in the program could take days to track down with state of the art tools. Thats just how it is. Were not able to comprehend large problems all at once; we can only attempt it in small increments.
Bearing this necessarily incomplete understanding in mind, what exactly _would_ we find in the machine memory between the elements of Mario and Luigis track mini-universe?
The answer is familiar to programmers and perhaps surprising to non-programmers. The answer is: _almost anything_. It could be elements of a different program, or random garbage not being interpreted by any program, or supplemental data for the Mario Kart universe, such as language translation strings. Or Luigi could be right: it could be Track 4, pre-cached for the next race. Perhaps not exactly the way hes imagining it, but... close.
MOVING BEYOND THE WALL
Our little visualization of the host systems memory raises another natural question: what would happen if you "zapped" Luigis Kart across the boundary?
This question isnt entirely as nonsensical as "whats on the other side?" With the proper programming tools, you might be able to observe changes in the machine memory as Luigis Kart moves, and these changes might follow an observable pattern that leads to a relevant prediction of sorts.
As just one possibility, Kart motion might be represented as shifting a Kart memory structure along the sequential indexes of an in-memory array. This arrangement is unlikely for performance reasons, but its certainly possible, and should serve to illustrate the point. If you were to find that moving Luigis Kart ten meters (in the scale of Luigis track universe) resulted in a structure motion of 1000 memory addresses, then you might make the reasonable prediction that in another ten meters his representation would move another 1000 addresses.
This might well put him beyond the end of the physical memory array. In most real-world scenarios this would be a bug, and would result in some sort of nasty response from the machine, such as a game crash. Or in a more user-friendly environment, the game engine might simply prevent his motion in that direction and move him back into his universe. This might put his Kart in an uncomfortable position, but it will at least be a _real_ position according to the logic of the Mario Kart universe. At least the Kart wont have disappeared.
However, it is _infinitesimally_ possible that Luigis Kart could be moved into another physical machine process in the host system -- say, another instance of Mario Kart running on a time-sharing operating system or virtual machine -- in such a way that Luigi and his cart physically disappear from the old universe (a process address space) and appear in the new universe (another process address space).
Even if this infinitely remote possibility were to occur, chances are high that the sudden appearance of Luigi and his Kart would wreak local or even global havoc in the new universe. He might get lucky and crush Bowser into a pan-a cake, but more likely he would wind up stuck in a stone wall or a hill, unable to move. Or even more likely, his Karts memory structure would be copied into a non-game area, such as the translation string section, and Luigis sudden mysterious appearance might manifest as nothing more than garbled speech from one of the other characters.
There are many possibilities, too many to imagine. The most important takeaway here is that no matter how _unlikely_ Luigis safe intra-universe migration may be, it is _possible_. With some external help from the host system, it would even be straightforward: perhaps no more than a few days work from a skilled programmer.
In real-world programs sharing adjacent address spaces, its improbable (but possible!) that migrating data from one process to another in a randomly chosen destination spot would have semantically meaningful results in the destination address space.
To put it in simpler terms: under just the right circumstances, Luigi could teleport to the other side, and wind up in a different world.
GHOSTS IN THE MACHINE
An embedded system is a little environment. My Betta in the previous installment of this little series lived in a very simple embedded system. Its most obvious difference from my own environment is that the tank was filled with water, while my room was filled with air. The differences from the host system can yield different behavior and different rules. In the fish tank, the rules are only a little different -- for example, most things are more buoyant inside the tank. In a virtual embedded system, the rules and behavior might be _completely_ different from those of the host system. It all depends on how the embedded system was designed to work.
Every embedded system has a containing surface, which might be called its frontier. The frontier is _one-sided_ in the sense that the rules of the embedded system may simply stop at the boundary: there is no "other side". In some embedded systems (such as a Euclidean space embedded in a non-Euclidean space), even the supposedly intuitive notion of extending a line across the boundary only has meaning on one side, the "inside" of the frontier.
If there is a formal mathematical term for this One-Sided Frontier idea, Ive yet to come across it, and Ive spent quite some time looking. If you have any suggestions, let me know. The closest I could find are the spaces that are undefined in noncontinuous functions, such as the Tangent function for a value of 90 or 270 degrees. If you ask a question
f(x) at these values of x, the answer is something like "You cant go that way."
So the other side of the frontier is undefined. This statement has a deep, almost philosophical meaning to programmers. It does NOT mean "nothing"; no, no no! It means _anything_. It is a modern-day Wand of Wonder, a Bag of Tricks, a Horn of Plenty, a Box of Chocolates. You never know what youre going to get.
If a computer programs code inadvertently reaches into undefined, then it has stepped across a mystical wall into another universe, and anything could happen. Its as if Gremlins have taken over the machine. Your PC speaker might beep erratically or continuously. Your video array might burst into random fireworks. Your printer might fire up and print out mysterious symbols. Your program may even continue to operate normally, but with the addition of unexplainable and unrepeatable phenomena.
I have seen all these things happen. All C/C++/assembly programmers have seen bugs like this -- program bugs with "crazy" behavior. The bugs are so spooky, and so hard to track down, that computers are now designed to be gatekeepers at the Wall, and when you try to step across they say STOP! Its much better to be blocked immediately than to let your program wander into undefined, where ghosts may take hold of your data in ways that may even cause your legal department to take notice. Stranger things have happened, as they say.
This computerized gatekeeping is commonly called "protected mode". The computer checks every addressing instruction, and any time a program tries to access the memory area outside its own, things halt immediately. When you see the message "segmentation fault", or a blue screen of death, or some other sign that a fatal, unrecoverable program error has occurred, it is almost always because someone or something in the embedded system tried to escape.
HOLES IN THE WALL
From the perspective of a computer game, the system frontier is relatively uninteresting. Its not much different from any other obstacle.
However, in other embedded systems these frontiers are almost mystical in nature. They provide endless fascination for computer scientists working in the domain of _programming languages_, which are yet another kind of embedded system.
To see how it works, lets consider the situation in Mario Kart, which is simpler.
In Mario Kart, most of the racers are computerized opponents, or AIs. A few of them, usually at least one, are controlled by people, using steering wheels or other physical controllers. Sometimes (e.g. in simulation mode) all the opponents may be computerized.
In order for people to interact with the embedded system, there must be some way to send information back and forth across the system frontier.
Coming from the Mario Kart side, we typically have video and sound signals. The embedded system generates these signals and sends them to your TV or device.
Coming from the people side, we typically have motion input: which way to move the Karts. The signals begin as your physical motions: buttons you press, or in newer controllers, the direction you tilt the controller. They are sent from the host system to the embedded system and they generally produce observable effects in the embedded system. Like magic!
Mario Kart is especially interesting because the external camera is physically present within the game. You can see it in the instant replays of your races: a little guy with a camera, floating behind you on a little cloud. The game designers have arranged things so that you can almost never see him while racing, because hes always flying behind your head, along your line of sight.
But hes there. And in fact that camera is _always_ there, in all video games. Its just that the Mario Kart designers have chosen to reify him as a cute little turtle guy with a camera, floating behind you on a cloud.
The external camera is a one-way hole in our one-way wall: it sends data _out_ of the embedded system. For all intents and purposes it also sends the sound data, since the sounds are usually scaled as a function of distance from the camera.
During a race, theres a lot of stuff going on in the embedded system, and theres a lot of stuff going on in the host system. But they constrain their communication via mostly-invisible channels, and these channels are restricted in the kinds of communication they may convey. Your controller can send motion inputs, but (at least today) it cant send video data. And (at least today) the characters in the game cant control your arms and legs, the way you can control theirs.
Hopefully now you should have a mental picture of this magical wall between an embedded system and the Great Undefined Beyond on the "other side" of its system frontier. The wall may have deliberate holes in it: channels, really. Information may flow across these channels in predefined ways. And the channels are almost always invisible to occupants of the embedded system.
REFLECTIONS
Ive spent a lot of effort telling you about a rather strange, twisty kind of surface: the frontier of an embedded system. This frontier exists for all embedded systems. It may have "holes" (data channels) in it. The number and nature of these channels is entirely up to the designers of the embedded system. Programmers sometimes call these channels "reflection" or "metaprogramming".
The holes in the frontier may or may not be detectable within the embedded system itself. They may only be detectable within the host system. This, again, is up to the designers. For most of the embedded systems I know of, the channels are "well defined", in the form of application programming interfaces offered to either the embedded system or the host system for communicating across the frontier.
But you need a channel for this kind of communication. Setting it up is usually not cheap. And setting up meta-reflection (in other words, being able to "see" the channel from within the embedded system or host system) is even more work.
So most of the time, channels through the embedded system frontier are completely invisible and undetectable from inside the embedded system. Theyre quite real, and information flows either one way or both ways, but they _cannot_ be detected from within the embedded system, and their behavior is intrinsically non-repeatable via experimentation.
When data comes across these invisible channels, stuff "just happens" in the embedded system, with no clear indicator nor explanation as to why.
In our discussion so far, I have intentionally blurred the distinction between the host system (such as a fish tank or a game console device) and the host systems host system (such as your bedroom or living room). But youve probably noticed by now that _all_ host systems are embedded in some larger system. This is just the way things work. The fish tank is in your bedroom, which is a system embedded in a house, which is a system embedded in a neighborhood, embedded in a county, a nation, a continent, a planet, a solar system, a galaxy, a universe.
Its perhaps not as clear in the case of fish tanks, but host systems often overlap and even cooperate. A city is composed of many interleaved subsystems. So is your body. Its not always a simple containment relationship. Systems are made of, and communicate with, other systems.
But one way or another, _all_ systems are embedded systems.
There is no reason to assume that our Universe works any differently. Our Universe is a system; that much should be self-evident. It has boundaries; astronomers and astrophysicists have recently even determined that the boundary appears to be a dodecahedron.
We are already painfully aware of the question "what happened before the Big Bang, if in fact the Big Bang occurred in the way all the evidence suggests", and its inherent nonsensicality. What happened before the beginning of Time? What lies beyond the end of the Universe?
Programmers already know intuitively the answer to these questions. The answer is: undefined is there. _Undefined_ is NOT the same as "nothing", no indeed. Its _anything_. And we cant go that way.
This has ramifications for the way we think about things today.
I believe I will have more to say about this soon. Right now I think I will go play Mario Kart: a game as fun as any I think Ive ever played.I finished Fallout 3 maybe six or eight weeks ago, and it was hands-down one of the best games Ive ever played. A game like that gets you in the mood for more gaming, so I thought to myself: "Hey, I should plop down $160 for Fable II!"
Actually thats not _exactly_ what I thought, but its what happened. I bought the game for $60, fired it up, got up to the part in the intro where a bird craps on your head (yes, this is how it starts), and it locked up hard. Reset the XBox, tried again, and this time got as far as some guy selling snake oil gadgets before it locked up again. Snake oil, indeed.
I tried playing for about an hour, with the game crashing every 3 to 5 minutes, and I finally went online to read about how it kills XBoxes and its the Game of Death and blah blah blah, all interesting but not especially helpful. Eventually I stumbled across discussions of the "install to hard drive" option. Nobody actually said _how_ to do it, so it took another hour of digging to deduce that you need to purchase a $100 wireless network adapter (or 100 feet of network cable, I guess). So I shut it down for the night, waited for the stores to open, forked over the $100, and installed the game to the hard drive. To Lionheads credit the game never crashed again, making it significantly more stable than Oblivion or Fallout 3.
I tried hard to like Fable 2. I didnt even need to like it $160. I would have settled for a $60 value. I vaguely remember liking Fable 1, although I cant remember anything about the game except for one neat scene where you had to escort two NPCs through a dark valley. One of the NPCs has been bitten by a Balverine (a werewolf), and the two argue the whole trip about whether hes going to turn. Its a funny conversation and the scene has a funny ending. Other than that, I just have vague recollections of shooting birds on the roof of some guild, and needing to get a 6-foot handlebar moustache for some side quest. The rest of it is basically a blank. But I had set some flag to the effect that "I liked it," and I wanted to like the sequel too.
Unfortunately, with a few noteworthy exceptions that Ill call out in the "Highlights" below, the game is entirely forgettable. Its already fading from memory as we speak. It wasnt as bad as some people make it out to be. Its playable for a couple of days, and it has its fun moments. But its not a very good game, and its definitely not a very memorable game. This is sad, considering the amount of effort that went into its development.
The no-spoiler synopsis of Fable 2 is that its a bad Zelda clone. You can smell the desperation; there are dozens and dozens of direct rip-offs from the Zelda franchise. Heck, I wouldnt have minded a half-decent Zelda clone; theyre some of the best games of all time. But Fable 2 misses the mark by a mile. The humor is juvenile bordering on imbecilic, the hints are hamfisted, the areas are small and cramped, the minigames are lackluster, the music is virtually nonexistent, and the story pacing is rushed and breathless. Its a cargo-cult copy of Zelda that winds up having no identifiable soul: forgettable across the board.
Ive given up every piece of Microsoft software and hardware I own except for the XBox, which I had been holding onto _just_ for Fable 2. Now that its come and gone like a bird crapping on my head, Im giving up. No more XBox or PC games for me. Ever.
Hence, Fable 2 cost me $160. I hope you got it for cheaper than that.
Anyway, heres a quick rundown of the lowlights and highlights of the game, as I see them. Enjoy!
LOWLIGHTS
1) HUMOR. Fable 2 tries hard to incorporate humor into the game -- _too_ hard. The writers use the trusty old "stopped clock" approach to humor, in which they inundate you with jokes, and 1 out of every 43,200 of them is funny. Amazingly, this perseverance leads to 3 or 4 genuinely funny ones, mostly near the entrance to the Crucible (arena). But by the time you get there, youve already tuned out all attempts at humor and have probably even tried killing yourself a few times. So they may fail to register.
2) THERESA. The game features an old lady who watches everything you do and talks at you constantly. This starts in the very beginning of the game and lasts until the very end, with no option to turn her off. Your character cant so much as take a crap without Theresa piping in with helpful advice on which hand to use. "That is ancient paper. Be cautious." She uses some magical form of communication system that only breaks down in the fog -- probably shortwave radio -- and theres no way to turn the fugging thing off.
I really hated Theresa.
3) EXPRESSIONS. You cant talk to people. Instead, the game gives you a series of increasingly repugnant forms of nonverbal communication. Initially youre limited to belching, farting, giving people the finger and making lewd pelvic thrusting motions, but as you rise in fame Theresa informs you that youve earned the right to use the "kiss my ass" expression. I am not making this up. I tried to avoid using expressions altogether, but the game forces you to do it once in a while. Made me want to take a shower.
4) RETINAL BLINDNESS. Fable 2 is nauseatingly saturated. They just dont know how to lay off the paint gun. There are a few OK-ish-ly tasteful areas, such as the big trees in Bower Lake, but most of the game is a frightully garish mix of lime greens, oranges, purples, reds, blues, and general oil-spill iridescence. It makes you color-blind fast, even if you didnt start that way. Finding anything onscreen is like trying to spot where someone threw up on a Matisse.
5) LINEARITY: the game is unrelentingly linear for the first hour or two (a _long_ time), after which it settles into, well, linearity. The gameplay occasionally approaches the smashing-through-lines-of-baddies feel of Gauntlet Legends, which I liked, but mostly it makes you feel like a rat running a big maze, following a neverending golden trail of cheese.
A major contributor to the linear feel, even after the game opens up, is the plethora of tiny little fences and obstacles that you cant hop. It makes it really hard to know where you can walk, and it feels like youre constantly bumping into things, because, well, you are. So the game is linear at all resolutions: high (the plotline), medium (most of the level designs) and low (the path designs). Linearity can cramp even the best of games -- Kingdom Hearts comes to mind. Its just a bad way to design things. And Fables linearity felt especially suffocating after just having finished Fallout 3, which is immense and wide-open.
6) CONTROLS. Its been a long time since I played a game whose controls were so accident-prone. Normally a games controls take some getting used to, and then its like driving a car. In Fable 2, even after days of play, Id still be trying to hop a fence and wind up shooting the front door off a mansion, blowing boards everywhere and scaring the shit out of the villagers.
Hell, even when I was trying to buy my final sword (this was a $50k sword Id been looking for all day), I tried to bring up the "buy sword" menu for the blacksmith, and I accidentally wound up casting a massive Inferno spell, causing him to literally run screaming across town. It was weeks of in-game time before I saw him again. God dammit. They really should have had different controls in safe zones.
7) ELEPHANTINE MAMMARY GLANDS. I dont know what planet these guys have been living on, but giant udders fell out of fashion at least two or three decades ago. Every single woman in Fable 2 had knockers significantly bigger than her head. It reminded me of my trip to Paris, where every statue of a woman is bare-breasted, presumably so that you can tell its a woman -- a practice which unfortunately suggests that theres really no other way to tell. Dumb French statue-making assholes.
I mean, the people in Fable 1 were ugly -- the main character worst of all. They all had this "Im a programmer who never gets outdoors" look, and I expected (and got) no better from Fable 2. But I was really disappointed that every female in the game was a fugging dairy farm. I mean, someone with some taste and maturity should have a talk with these asshats, and explain to them what women actually look like. Or they should pick up a frigging Victorias Secret catalog or watch a goddamn Target commercial or _something_. Jesus.
The milk jug thing... it was really just too much. I have zero respect for those jerk-offs at Lionhead.
8) ASS-KISSING. This was probably the most serious problem with the game. It was a disease in Fable 1 that went malignant in Fable 2. Whoever designed these games was apparently neglected as a child, because the gameplay revolves around gaining "renown". Lionheads hopelessly adolescent view of "renown" is that villagers should follow you around and say things like "yay!" and "hurrah!" Its even worse than Im making it sound.
They spent so much time coding this crap that they forgot to code _pushing_ into the game: you cant push people out of your way. So as soon as you wander into a dead-end alley youre fucked: a bunch of people will crowd in after you asking for autographs and offering you gifts and all this sickening bullshit.
To the games credit, and I count this as a highlight, if you pull out your six-barreled rifle, take the safety off, and aim right at their heads, it clears everyone out pretty fast. You can imagine how desperate I was by the time I tried that approach. But they coded it correctly, bless em.
9) TOO EASY. The game just wasnt hard, period. There were no hard fights. I never died. I dont even know what happens when you die in Fable 2. I used a couple of Resurrection Phials, but only because I had become so lazy in combat that I didnt care anymore. This was a serious flaw in the game: it essentially removed the element of fear, which was the only emotion (other than disgust) that the game had a chance of evoking.
10) DEMON DOORS. Oh man, oh man. These were probably the low point of the whole game. They made me want to puke. I would run past them as fast as I could so I didnt have to listen to their inane drivel. This was some of the worst game writing Ive ever seen. I just dont want to talk about it.
11) MISGUIDED INNOVATION. They really should have stuck with copying Zelda, and Kingdom Hearts, and Gauntlet: Legends, and all the other games they copied blindly, and badly. Because whenever they introduced something entirely new, it almost invariably sucked. Examples? OK. Sure. Since you asked, and all.
How about the "innovation" that when you eat nearly any food (and it only takes a few bites), you bloat up to the size of Orson Welles, and the only way to get rid of it is to _eat celery_. No amount of exercise will make any difference, but eating a few bites of celery makes it go away. Innovative!
Innovation: you can purchase almost any property in the game. Is this realistic? No. In reality, not everything would be for sale (and especially not posted on the front doors). Real-estate transactions wouldnt be instantaneous. You would need the owner present to buy something. Etc, etc. So given that this feature makes the game _less_ realistic, what purpose does it serve? Is it fun? No. Buying real estate is about as fun as attending insurance seminars, so I dont know what the hell they were thinking. It _could_ have been fun in the right setting, with suitable other human participants, in a Parker-Bros. Monopoly kind of way. Maybe. But slapping it on the side of an RPG and calling it innovation? It boggles the mind.
And what about the the busywork jobs (blacksmith, woodcutter, bartender) for making gold? Um, dudes -- busywork jobs only exist in MMORPGs to _limit per-player CPU usage_. Theyre NOT FUN. "Innovatively" bringing them into a single-player game was just flat-out brain damaged.
12) NONEXISTENT TARGET AUDIENCE. What age group is the target market for this game? If you enumerate the possibilities, you arrive at the inescapable conclusion that the game was either (a) created _by_ imbeciles, or it was (b) created _for_ imbeciles, or possibly (c) all of the above.
Its presumably not intended for kids, or you wouldnt be finding condoms in treasure chests, soliciting and obtaining sex from male and female prostitutes of all shapes and sizes, performing pelvic thrusts to solve quests, and so on.
Its not for adults either, or you wouldnt be bombarded with the constant barrage of scatological humor, beginning with the bird shitting on your head, continuing with warnings about "extending the fart command and messing it up", and going pretty much straight downhill from there.
Is it intended for teenagers, then? Poooossibly, but (a) that ignores their primary demographic, which is 30-year-olds, and (b) I dont know any teenagers that are _that_ stupid, nor so hard-up for attention that they need AI villagers to yell "hurray!" whenever you pass them, even if youre in a graveyard at midnight.
Dipshits. This game was designed by dipshits. The coding was great, the artwork was great, the sound effects were great; the details were for the most part rock-solid. But the creative direction was just inexcusably bad.
HIGHLIGHTS
OK, Ive been pretty tough on the game so far. Fable 2 did actually have a few genuine highlights worth calling out. You could even argue that these highlights make the game almost worth playing, in spite of all the crap you have to suffer through in order to get to them.
1) BANSHEES. Fable 2s banshees are, in a word, awesome. Ive been racking my brain trying to think of a VG monster as cool as these banshees in any game Ive ever played. Im coming up with a few ties, but nothing that beats them. The YouTube videos dont come close to doing them justice. Fable II is worth playing just to get to Wraithmarsh.
The only real problem with the banshees is that since none of the combat is challenging (see Lowlight #9), theyre nowhere near as scary as they could have been. But theyre amazingly stylish. Id call them an innovation, but Im inclined to believe Lionhead stole their basic design from some other game, given all the other copying theyve done. (The Fable 2 Trolls, for instance, are about as Zelda-clone-esque as you can get without inviting a lawsuit.)
2) LUCIENS SPEECH, where he addresses the recruits. Really great speech. Riveting and convincing. Amazing how Microsoft-run studios that are so consistently bad at humor are so good at creating convincingly evil speeches about taking over the world.
Actually the whole centerpiece drama in the tower was very nicely done. I have to give them credit for that part of the game: it was exceptional by any standard. It basically saved the game from being a total loss.
3) HAMMER. Shes cool. Great voice acting, surprisingly good scripting, neat character, lots of depth. One of the better-realized VG supporting characters Ive seen in many years.
4) THE DOG. Apparently there was a lot of hype about the dog. Or so I hear, after actually having played the game. Whatever the hype, the reality is that its a very believable dog. I especially liked how it would run ahead of you -- Ive seen pets that follow you, but the dog would often anticipate your direction and run ahead, kinda turned back towards you like "cmon! lets go!" I encountered no glitches with the dog; the coding was rock solid. Overall it was, well, very... doggy. And what more could you ask for in a dog, really?
As a tribute to the believability of the dog, Ill offer a minor spoiler. (Skip ahead if you dont want a spoiler!) At the end of the main storyline, you are granted one wish. Your choices are: (a) get all the people who died back, (b) get your dog back, or (c) get a bunch of money. What I really wanted was a sort of amalgam of the 3 choices: I wanted my money back for this dog of a game. But when push came to shove, I picked the dog. I kinda missed him.
5) ARCHITECTURE. Overall the architecture was really nice. The only somewhat dubious exception was Bowerstone, which looks almost exactly like Euro Disney. I kept expecting Tigger to come waltzing around, cursing in French under his breath, just like he did on my real-life trip to Euro Disney a few years back.
Other than the Euro Disney influence, which I could take or leave, I thought the architecture was nice throughout the game. I liked the waterfront town of Bloodstone. I liked the manors in Oakfield. I liked the gypsy wagons. I liked the vendor carts. I loved pretty much every creepy structure in Wraithmarsh. The overall look of the game was beautiful, once you got past the color-saturation problem, and the architecture was a huge contributor.
6) FIGHT MUSIC. Unlike in Fable 1, most of the music in Fable 2 is forgettable background/atmosphere music. They didnt get Danny Elfman this time around, and it shows. The theme for Bower Lake is nice as far as it goes, which is exactly 2 chords over and over and over. But its still OK. The rest of the music didnt leave any sort of impression on me at all, except for the fight music, which _almost_ made up for everything. It was very good. There were at least two fight themes and both of them were cool. If only the rest of the music had been... present. It was like it wasnt even there. It phoned in its performance.
Folks at Dorkhead studios: Zeldas music is one of the top five reasons for its success as a franchise. Same goes for Mario and Kingdom Hearts and Final Fantasy. Their music is always great, and its always in your face. The music isnt muttering or mumbling; its shouting. And they can get away with it _because_ its always great. Even when its bad or annoying, which is rare, the music still anchors each place and event in the game in your memory, in a way that _only_ music can. You guys really screwed the pooch on this one.
7) MIXED-TACTIC FIGHTING. They did a great job of setting things up so that you could use melee, ranged weapons and magic effectively in combat. It was refreshing to be able to switch styles in mid-fight: you could use your sword to kill everything near you, then start blasting everything ten feet or more distant with your rifle. Or you could clear a little space and cast a time-slowing spell, and then just start zinging around whaling on bad guys. The combat was never _hard_, but it was on the whole fairly _satisfying_.
The downside of ultra-convenient access to melee, range weapons and spells was that you could effortlessly use them all simultaneously while trying to buy vegetables from a produce stall in the main market. I really wish theyd made it just a teeny bit harder to cast spells in public areas.
8) WELL, DAMN. I CANT THINK OF A HIGHLIGHT #8. I thought of some more lowlights, though: long area loads, unresponsive controls during "scenes", only a handful of available spells, months of coding/design effort wasted on useless features like "groin shots" and tatoos...
Oh, and the lack of control over when quest scenes actually unfold -- they trigger from proximity to the relevant NPC rather than interacting with the NPC because, oh, thats right, _you cant interact with anyone_ except to fart on them or give them the finger. Oops! So youre always accidentally wandering into a dungeon that triggers some quest, and theres no way out except to back _entirely_ out of that phase of the quest, which may involve losing hours of your time, all because you walked through the wrong door. Damn that pissed me off.
And how the hell do you sleep in an Inn? I never managed to figure it out. Id wind up spending $10k for some hovel just to get a frigging bed to sleep in. It was amazingly bad UI design, if there even _is_ a way to do it. If not, then their helpful tutorial message lied to me at least a dozen times.
Argh. Well, this highlights section is going downhill in a hurry, so I think Ill end it here.
BETTER THAN A CRAP ON THE HEAD?
Maybe, maybe. But compared to Fallout 3, Fable 2 pretty much sucked. It had a couple of nice features, but they were drowned in an ocean of painfully adolescent design. Such a shame.
Ive tried to be fair here. I dont mean to discourage you from playing the game, since for all I know theres nothing better out there right now.
If you do decide to play it, I hope Ive set your expectations _very_ low. That way, well, who knows? You might actually have some fun with it.
But if you open even _one_ of those Demon Doors Ill lose all respect for you.
So! I have all these cool things I want to write about, but I broke my thumbnail. Can you tell thats a long story?
See, this summer I got excited about playing guitar again. I usually switch between all-guitar and all-piano every other year or so. This summer I dusted off the guitars and learned a bunch of pieces, and even composed one. I was prepping for -- among other things -- a multimedia blog entry. It was going to have a YouTube video, and a detailed discussion of a wacky yet powerful music programming language youve probably heard of but never used, and generally just be really cool.
And then it all came crashing down when I busted my thumbnail off. And I mean _off_ -- it broke off at least a quarter inch below where the nail and skin meet. Ick. I just accidentally jabbed my steering wheel, and that was that.
I remember reading an interview with some dude who said he had punched a shark in the nose. He said it was like punching a steering wheel. So now I know more or less what its like to punch a shark in the nose, I guess. Theres always an upside!
Anyway, that was going to be my magnum opus (literally: Op. 1) for the year, but it fell through for now. Ill have to revisit the idea next year. My thumbnails back, but its been at least 2 months since I touched my guitar, so Ill have to start over again.
Work has been extraordinarily busy, what with having to collect all these Nuka-Cola Quantum bottles and so on. Im sure you can imagine. So I havent had much time to blog lately.
But I do like to publish at least once a month, whether or not anyone actually cares. Its been about a month, or it feels that way anyway, and all I have to show for it is this box of Blamco Mac and Cheese.
So Im cheating this month.
You know how on Halloween how you walk around in your costume holding your little bag and you say "trick or treat", and every once in a while some asshole does a trick instead of dumping half a pound of candy into your bag? And then he has to explain to all the dumbfounded and unhappy kids that "Trick or Treat" means that a trick is perfectly legal according to the semantics of logical-OR, and the kids remember that a-hole for the rest of their childhoods and avoid his house next year?
Yeah.
So Im doing a trick this time. Hee. Its actually kind of fun when youre on the giving end.
My trick is this: in lieu of saying anything meaningful or contemporarily relevant, Im writing about something I did over a year ago. And there isnt much to say, so this really will be short.
EJACS
Around a year ago, I wrote a blog called Steveys Boring Status Update, mostly in response to wild rumors that Id been fired from Google. Not so. Not yet, anyway.
In that blog I mentioned I was working nights part-time (among other things) on a JavaScript interpreter for Emacs, written entirely in Emacs Lisp. I also said I didnt have a name for it. A commenter named Andrew Barry suggested that I should NOT call it Ejacs, and the name stuck.
Ejacs is a port of Narcissus. Narcissus is a JavaScript interpreter written in JavaScript, by Brendan Eich, who by pure unexpected coincidence is also the inventor of JavaScript. Narcissus is fairly small, so I thought it would be fun to port it to Emacs Lisp.
It turns out Narcissus is fundamentally incomplete. It cheats. Its that trick guy on Halloween. Narcissus has a working parser and evaluator, but for its runtime it calls JavaScript. So its kind of like saying youre building a car by starting from scratch, using absolutely nothing except for a working car.
This meant I wound up having to write my own Ecma-262 runtime, so that the evaluator would have something to chew on. In particular, the Ecma-262 runtime consists of all the built-in properties, functions and objects: Object, Function, Array, String, Math, Date, RegExp, Boolean, Infinity, NaN, parseInt, encodeURIComponent, etc. A whole bunch of stuff.
I basically did this by reading the ECMA-262 specification and translating their algorithms into Emacs-Lisp. That spec is lousy for learning JavaScript, but its absolutely indispensable if youre trying to _implement_ JavaScript.
I didnt know Emacs-Lisp all that well before I started, but boy howdy, I know it now.
Emacs actually has a pretty huge runtime of its own -- bigger than you would ever, ever expect given its mundane title of "text editor". Emacs has arbitrary-precision mathematics, deep Unicode support, rich Date and Calendar support, and an extensive, fairly complete operating system interface. So a lot of the porting time was just digging through Emacs documentation (also extensive) looking for the Emacs version of whatever it was I was porting. That was nice.
I had big plans for Ejacs. I was going to make it a full-featured, high-performance JavaScript interpreter, with all the Emacs internals surfaced as JavaScript native host objects, so you could write Emacs customizations using object-oriented programming. It was totally going to be the "meow" part of the cat.
And then I broke my thumbnail.
Actually, what happened was js2-mode.
JS2-MODE
After I got the interpreter working, I was at this crossroads. There were two big use cases: a JavaScript _editing_ mode, or a JavaScript _Emacs development_ mode. Both were going to be a lot of work.
It turns out you really want the editing mode first, if possible, so that when youre doing all your JavaScript programming you have a decent environment for it. So I picked the editing mode.
Unfortunately I found the Ejacs parser wasnt full-featured enough for my editing needs, since at the time I was working on my Rhino-based Rails clone and writing tons of JavaScript 1.7 code on the JVM.
I spent a little time trying to beef up the parser, then realized it would be a lot faster to just rewrite it by porting Mozilla Rhinos parser, which is (only) about 2500 lines of Java code. Ejacs is something like 12,000 lines of Emacs-Lisp code, all told, so that didnt seem like a big deal.
So I jumped in, only to find that while the parser is 2500 lines of code, the scanner is another 2000 lines of code, and theres another 500 or so lines of support code in other files. So I was really looking at porting 5000 lines of Java code.
Moreover, the parse tree Rhino builds is basically completely incompatible with the Ejacs parse tree. It was richer and more complex, and needed more complicated structures to represent it.
So after Id ported the Rhino parse tree, what I really had was a different code base. I went ahead and finished up the editing mode, or at least enough to make it barely workable (another 5000 lines of code), and launched it. It was a surprisingly big effort.
And it left poor Ejacs lying unused in the basement.
So today, faced with nothing to write about, I figured Id dust off Ejacs, launch it with lots of fanfare, and then youd hardly notice that I cheated you. Right?
Youre not coming to my house next year. I can tell already.
Anyway, heres the code: http://code.google.com/p/ejacs/
Theres a README and a Wiki and installation instructions and stuff. I cant remember how to put the code in SVN, and Im having trouble finding it on the code.google.com site. As soon as I figure it out Ill also make it available via SVN.
EMACS LISP VS. JAVASCRIPT
In the interests of having _something_ resembling original worthwhile content today, Ill do a little comparison of Emacs Lisp and JavaScript. I know a lot about both languages now, and a few folks mentioned that a comparison would be potentially interesting.
Especially since I think JavaScript is a better language.
So... the best way to compare programming languages is by analogy to cars. Lisp is a whole family of languages, and can be broken down approximately as follows:
* Scheme is an exotic sports car. Fast. Manual transmission. No radio.
* Emacs Lisp is a 1984 Subaru GL 4WD: "the car thats always in front of you."
* Common Lisp is Howls Moving Castle.
This succinct yet completely accurate synopsis shows that all Lisps have their attractions, and yet each also has a niche. You can choose a Lisp for the busy person, a Lisp for someone without much time, or a Lisp for the dedicated hobbyist, and youll find that no matter which one you choose, its missing the library you need.
Emacs Lisp can get the job done. No question. Its a car, and it moves. Its better than walking. But it pretty much combines the elegance of Common Lisp with the industrial strength of Scheme, without hitting either of them, if you catch my drift.
Anyway, heres the comparison. Heres why I think JavaScript is a better language than Emacs Lisp.
PROBLEM #1: MOMENTUM
A recurring theme is that Elisp and JavaScript both will both exhibit a particular problem, and there are specific near-term plans to fix it in JavaScript, but no long-term plans to fix it in Elisp.
Its easier to resign yourself to a workaround when you know its temporary. If you know the language is going to be enhanced, you can even design your code to accommodate the enhancements more easily when they appear.
People are working on improving JavaScript. Its not happening quite as fast as Id hoped earlier this year, but its still happening. As far as I know, Emacs Lisp is "finished" in the sense that no further evolution to the language is deemed necessary by the Emacs development team.
PROBLEM #2: NO ENCAPSULATION
Every symbol in Emacs-Lisp is in the global namespace. There is rudimentary support for hand-rolled namespaces using obarrays, but theres no equivalent to Common Lisps
in-package, making obarrays effectively useless as a tool for code isolation.
The only effective workaround for this problem is to prefix every symbol with the package name. This practice has become so entrenched in Emacs-Lisp programming that many packages (e.g. apropos and the elp elisp profiler) rely on the convention for proper operation.
The main adverse consequence of this problem in practice is program verbosity; it makes Emacs-Lisp more difficult to read and write than Common Lisp or Scheme. It can also have a non-negligible impact on performance, especially of interpreted code, as the prefix characters can approach 5% to 10% of total program size in some cases.
The problems run slightly deeper than simple verbosity. Without namespaces you have no real encapsulation facility: there is no convenient way to make a "package-private" variable or function. In practice theres little problem with program integrity; its hard for an external package to change a "private" variable inadvertently in the presence of symbol prefixes. However, it makes it annoyingly difficult for users of the package to discern the "important" top-level configuration variables and program entry points from the unimportant ones. Elisp attempts a few conventions here, but its a far cry from real encapsulation support.
JavaScript also lacks namespaces. Theyre being added in ES/Harmony, but in the meantime, browser JavaScript code typically uses the same name-prefixing practice as Emacs-Lisp.
However, JavaScript has lexical closures, which provide a mechanism for creating private names. One common encapsulation idiom in browser JavaScript is to wrap a code unit in an anonymous lambda, so that all the functions in the code unit become nested functions that close lexically over the top-level names in the anonymous lambda. This trick is nowhere near as effective in Emacs-Lisp, for several reasons:
* elisp is not lexically scoped and has no closures
* elisp nested defuns are still entered into the global namespace
* CLs `flet and `labels are only weakly supported, via macros, and they frequently confuse the debugger, indenter, and other tools.
Some elisp code (e.g. much of the code in cc-engine) prefers to work around the namespace problem by using enormous functions that can be thousands of lines long, since let-bound variables are slightly better encapsulated. Even this is broken by elisps dynamic scope:
(defun foo ()
(setq x 7))
(defun bar ()
(let ((x 6))
(foo)
x)) ; you would expect x to still be 6 here
(bar)
7 ; doh!
So let-bound variables in elisp can still be destroyed by your callee: a dangerous situation at best.
Emacs is basically one big program soup. Theres almost no encapsulation to speak of, and it hurts.
PROBLEM #3: NO DELEGATION
One of the big advantages to object-oriented programming is that there is both syntactic support and runtime support for automatic delegation to a "supertype". You can specialize a type and delegate some of the functionality to the base type. Call it virtual methods or prototype inheritance or whatever you like; most successful languages support some notion of automatic delegation.
Emacs Lisp is a lot like ANSI C: it gives you arrays, structs and functions. You dont get pointers, but you do get garbage collection and good support for linked lists, so its roughly a wash.
For any sufficiently large program, you need delegation. In Ejacs I wound up having to implement my own virtual method tables, because JavaScript objects inherit from Object (and in some cases, Function, which inherits from Object).
Writing your own virtual method dispatch is just not something you should have to do in 2008.
PROBLEM #4: PROPERTIES
I wrote about this at length in a recent blog post, The Universal Design Pattern. JavaScript is fundamentally a properties-based language, and its really nice to be able to just slap named properties on things when you need a place to store data.
Emacs Lisp only offers properties in the form of simple plists - linked lists where the odd entries are names and the even entries are values. Symbols have plists, and symbols operate a little bit like very lightweight Java classes (in that theyre in the global namespace), but that only gets you so far. If you want the full JavaScript implementation of the Properties Pattern, youll have to write a lot of code.
And so I did. Your implementation choice for object property lists has a huge impact on runtime performance. Emacs has hashtables, but theyre heavyweight: if you try to instantiate thousands of them it slows Emacs to a crawl. So theyre no good for the default Object property list. Emacs also has associative arrays (alists), but their performance is O(n), making them no good for objects with more than maybe 30 or 40 properties.
I wound up writing a hybrid model, where the storage starts with lightweight alists, and as you add properties to an object instance, it crosses a threshold (I set it to 50, which seemed to be about right from profiling), it copies the properties into a hashtable. This had a dramatic increase in performance, but it was a lot of work.
I experimented with using a splay tree. I implemented Sleater and Tarjans splay tree algorithm in elisp; Ejacs comes with a standalone splay-tree.el that you can use in your programs if you like. I was hoping that its LRU cache-like properties would help, but I never found a use case where it was faster than my alist/hashtable hybrid, so its not currently used for anything.
And then in the end, after I was done with my implementation, it was a _library_ (at least from the Emacs-Lisp side of the house). It wasnt an object system for Lisp. Its only really usable inside the JavaScript interpreter, where it has syntactic support.
You really want syntactic support. Sure, people have ported subsets of CLOS to Emacs Lisp, but Ive always found them a bit clunky. And even in CLOS its hard to implement the Properties Pattern. You dont get it by default. CLOS has lots of support for compile-time slots and virtual dispatch, but very little support for dynamic properties. Its not terribly hard to build in, but thats my point: for something that fundamental, you dont want to have to build it.
PROBLEM #5: NO POLYMORPHIC TOSTRING
One of the great strengths of JavaScript is the toSource extension. I dont know if they support it over in IE-land; I havent been a tourist there in a very long time. But in real versions of JavaScript, every object can serialize itself to source, which can then be evaled back to construct the original object.
This is even true for functions! A function in JavaScript can print its own source code. This is an amazingly powerful feature.
In Emacs Lisp, some objects have first-class print representations. Lists and vectors do, for instance:
(let ((my-list ()))
(push 1 my-list)
(push 2 my-list)
(push 3 my-list)
my-list)
(3 2 1)
(let ((v (make-vector 3 nil)))
(aset v 0 1)
(aset v 1 2)
(aset v 2 "three")
v)
[1 2 "three"]
But in Emacs Lisp, many built-in types (notably hashtables and functions) do NOT have a way to serialize back as source code. This is a serious omission.
Also, trying to print a sufficiently large tree made entirely of defstructs will crash Emacs, which caused me a lot of grief until I migrated my parse tree to use a mixture of defstructs and lists. Note that simply typing the name of a defstruct, or passing over it ephemerally in the debugger, will cause Emacs to try to print it, and crash. Fun.
The problem of polymorphic debug-printing (or text-serialization) is, I think, a byproduct of Emacs not being object-oriented. If you want a debug dump of a data structure, you write a function to do it. But Emacs provides a half-assed solution: it debug-prints lists very nicely, even detecting cycles and using the #-syntax for representing graph structures (as does SpiderMonkey/JavaScript). But it has no useful debugging representation for hashtables, functions, buffers or other built-in structures, and theres no way to install your own custom printer so that the debugger and other subsystems will use it.
So it sucks. Printing data structures in Emacs just sucks.
The situation in Ecma-262-compliant JavaScript really isnt that much better, although you can at least install your own toString on the built-ins. But any competent "server-side" JavaScript implementation (i.e. one designed for writing real apps, rather than securely scripting browser pages) has a way to define your own non-enumerable properties, so you can usually override the default behavior for things like toString and toSource.
And all else being equal, at least JavaScript functions print themselves.
EMACS ADVANTAGES: MACROS AND S-EXPRESSIONS
Pound for pound, Emacs Lisp seems roughly as expressive as JavaScript or Java for writing everyday code. It shouldnt be that way. Emacs Lisp ought to be more succinct because its Lisp, but its incredibly verbose because of the namespace problem, and its also verbose to the extent that you want to use the properties pattern without worrying about alist or hashtable performance.
Elisp does have a few places where it shines, though. One of them is the cl (Common Lisp emulation) package, which provides a whole bunch of goodies that make Elisp actually usable for real work. Defstruct and the loop macro are especially noteworthy standouts.
Some programmers are still operating under the (ancient? legacy?) assumption that the cl package is somehow deprecated or distasteful or something. Theyre just being silly; dont listen to them. Practicality should be the ONLY consideration.
The cl package wouldnt have been possible without macros. JavaScript has no macros, so even though it has better support for lambdas, closures, and even (in some versions) continuations, there are still copy/paste compression problems you cant solve in JavaScript.
Emacs Lisp has defmacro, which makes up for a LOT of its deficiencies. However, it really only has one flavor. Ideally, at the _very_ least, it should support reader macros. The Emacs documentation says they were left out because they felt it wasnt worth it. Who are they to make the call? Its the users who need them. Implementer convenience is a pretty lame metric for deciding whether to support a feature, especially after 20 years of people asking for it.
Elisp is s-expression based, which is a mixed bag. It has some advantages, no question. However, it fares poorly in two very common domains: object property access, and algebraic expressions.
JavaScript is NOT s-expression based (or it wouldnt be a successful language, many would argue), but it does offer some of the benefits of s-expressions. JSON is one such benefit. JavaScripts declarative object literals (or as a Lisp person would say, "syntax for hashes") and arrays provide a powerful mechanism for designing and walking your own declarative data structures.
JavaScript also has all the usual (which is to say, expected) support for algebraic operators. And unlike Java, JavaScript even got the precedence right, so its not full of redundant parentheses.
OVERALL COMPARISON
In the end, it comes down to personal choice. Ive now written at least 30,000 lines of serious code in both Emacs Lisp and JavaScript, which pales next to the 750,000 or so lines of Java Ive crapped out, and doesnt even compare to the amount of C, Python, assembly language or other stuff Ive written.
But 30,000 lines is a pretty good hunk of code for getting to know a language. Especially if youre writing an interpreter for one language in another language: you wind up knowing both better than you ever wanted to know them.
And I prefer JavaScript over Emacs Lisp.
That said, I suspect I would _probably_ prefer Clojure over Rhino, if I ever get a chance to sit down with the durn thing and use it, so its not so much "JavaScript vs. _Lisp_" as it is vs. Emacs Lisp.
I would love to see Emacs Lisp get reader macros, closures, some namespace support, and the ability to install your own print functions. This reasonably small set of features would be a huge step in usability.
However, for the nonce Im focusing on JavaScript. Ive found that JavaScript is a language that smart people like. Its weird, but I keep meeting really really smart people, folks who (unlike me) are actually intelligent, and they like JavaScript. Theyre always a little defensive about it, and almost a little embarrassed to admit it. But they think of it as an elegant, powerful, slightly flawed but quite enjoyable little language.
I tell ya: if youre a programming language, its a very good thing to have smart people liking you.
It doesnt make me smart, but I kinda like it too. Even though theres (still) a lot of hype these days about Java, and people tootling on about how Javas going to be the next big Web language... I just dont see it happening. There are too many smart people out there who like JavaScript.
So enjoy the interpreter. Ejacs is just a toy, but I think it also shows a kind of promise. Scripting Emacs using JavaScript (if anyone ever actually implements it) could be really interesting. It could open up the worlds most powerful, advanced editing environment to millions of people. Neat.
In the meantime, it doesnt actually do squat except interpret EcmaScript in a little isolated console, so dont get your hopes up.
Reminder -- heres the Ejacs URL: http://code.google.com/p/ejacs - enjoy!
And with that, Im off to find some Nuka-Cola Quantum. I just wish those bastards hadnt capped me at level 20.I write a column for computer programmers called "Steveys Blog Rants." Its basically a magazine column -- I publish to it about once a month. The average length of my articles is about 12 pages, although they can range anywhere from 4 to 40 pages, depending on how Im feeling. But for precedent, dont think blogs: think of Readers Digest. The blog format sets the wrong expectations.
Hence, some people complain that my articles are too long. Others complain that I have not given my arguments sufficient exposition, and that my articles are in fact too short on detail to warrant any credibility. This is a lose-lose situation for me, but I keep at it nonetheless because I enjoy writing. Even if nobody were to read my blog, the act of writing things down helps me think more clearly, and its engaging in the same way that solving a Sudoku puzzle is engaging.
You should try it yourself. All it takes is a little practice.
My blog topics vary widely, and sometimes I even venture outside the realm of programming. Programming is where Im most comfortable, and its also where people seem to ascribe to me some level of punditry: Im not necessarily _right_, but even my greatest detractors grudgingly admit that Im entitled to an opinion, by virtue of my having spent twenty years hacking day and night without any sign of wanting to give it up and turn into a pointy-haired manager.
Even though I love both programming and to a lesser extent writing about it, there are also lots of non-programming topics Id like to write about. Being a career programmer gives you an interesting perspective on issues not directly related to programming. You start to see parallels. So maybe Ill branch out some more and see how it goes.
THE PROGRAMMERS VIEW
The first thing you notice as a programmer is that it trains you -- forces you, really -- to think in a disciplined way about complex logic problems. It also gives you a big booster shot of confidence around problem-solving in general. Junior programmers tend to have very high opinions of themselves; I was no exception.
In time, though, programming eventually humbles you, because it shows you the limits of your reasoning ability in ways that few other activities can match. Eventually every programmer becomes entangled in a system that is overwhelming in its complexity. As we grow in our abilities as programmers we learn to tackle increasingly complex systems. But every human programmer has limits, and some systems are just too hard to grapple with.
When this happens, we usually dont blame ourselves, nor think any less of ourselves. Instead we claim that its someone elses fault, and it just needs a rewrite to help manage the complexity. In many cases this is even true.
Over time, our worldwide computer-programming community has discovered or invented better and better ways ways to organize programs and systems. Weve managed to increase their functionality while keeping the complexity under control.
But even with such controls in place, systems can occasionally get out of hand. And sometimes they even need to be abandoned altogether, like a dog thats gone rabid. No matter how much time and love youve put into such systems, theres no fixing them.
Abandoning a system is a time of grieving for the folks whove worked on it. Such systems are like family.
And theres a disturbing lesson at the tail end of such experiences. The scary thing is that its very easy, as a programmer standing at the precipice of complexity, to envision systems that are orders of magnitude more complex, millions of times more complex, even unimaginably more complex.
In the end, programming shows us how small we are.
THE FISHS VIEW
Long ago, I used to have a Siamese fighting fish, also known as a _Betta splendens_, or simply a "betta". You can buy these fish at almost any pet store. I kept my betta, who was a deep vibrant red, in a pretty little 15-gallon tank decorated with a resplendence of real freshwater plants. And for a while I think my betta was happy there.
Like many Americans, I went through a phase in which I kept and ultimately killed many, many tropical fish. I didnt kill them intentionally; I wanted them to live and thrive. But keeping them alive for long is a challenge when you dont live in the tropics. So they might live for a few months or maybe a year, but they would always die prematurely. It was sad, and eventually I could no longer bear it, so I stopped keeping them.
Of all my fish, my betta left the biggest impression on me. The betta is a remarkable fish in several ways. For one thing, bettas are physically beautiful, and when they are at full display, their fins expand, peacock-like, into a fluid rose shape that is undeniably dramatic.
Bettas are also remarkable because they fight. They do not fight to establish a pecking order, as other fish do; they fight to kill. The males display their fins and then fight whenever they see another male betta, or even their own reflection, so they have to be kept alone and away from mirrors.
But bettas, I think, are most remarkable for their intelligence. Of all of the hundreds of tropical fish I kept, only bettas displayed anything resembling intellectual curiosity.
This really makes bettas some of the saddest stories in the tropical fish industry. Like other hobbyist fishes, they are stolen from their natural habitat and shipped overseas, or at best farmed in unsavory conditions. But unlike most other fish, bettas are also dyed to enhance their color. They are generally housed in tiny fist-sized bowls because of their ability to breathe air when necessary. And they are bred to express their fighting genes, and are often made to fight by their owners. Whereas other fish are kidnapped and sold, bettas are _abused_.
But worst of all, I believe their high intelligence endows them with greater capacity for suffering than other fish species. They can suffer physically and emotionally, but as we will see shortly, they can also suffer intellectually.
So bettas are a sad story.
MY BETTA
Here is the specific sad story of my betta, the fish that left such an impression on me.
I had taken to lying on my bed and watching my betta for an hour or longer. The betta was the sole occupant of the tank in my bedroom. I had filled the tank with plants and copious natural light, so the effect was calming and serene. At times I almost envied the betta for the nice home Id made for him.
One day, after the betta had been in his new home for several weeks, I found him exploring. It was a most unusual exploration, and one that I will never forget.
For the first few weeks, the betta explored the way you would expect any reasonably intelligent fish to go about the task. For the first few days he swam around to every nook and cranny of the tank, to make sure he had the lay of the land. Then for a few more days he experimented with staying put in different locations to see how he liked their feel.
Just like people, most fish will soon find a spot or a path they like best, and theyll stay in that spot or on that path for the rest of their lives.
But my betta was different. After his initial explorations he became restless. Im no Fish Whisperer, but I could _tell_ that he was restless. You would have thought so too. The betta started spending most of his time looking out of the tank, examining my bedroom. And he was clearly looking at specific things in the bedroom, not just "out there" in general. He would periodically swim around looking mildly agitated. He was acting like he wanted out.
I did everything I could to placate him. I experimented with different fish foods. I changed the water weekly and monitored it carefully to keep its temperature and pH within acceptable ranges. I added more lights. I added more plants. I rearranged the plants. In desperation, I even added a little castle.
Every time I tried something new, it would pique his interest for a little while. But in time, and faster each time than before, he would revert to his state of restlessness.
Id never seen quite this behavior in other fish, so already he was demonstrating what seemed to be above average intelligence.
And then one day I found him engaged in an exploration that was altogether new.
He wasnt exploring the tank. Hed already investigated its topology for weeks. This time, he was exploring the _nature_ of the tank. Thats what caught my attention, and not just for that day, but for the rest of my life.
There was a twenty-inch vine in the tank that extended from the lower left back corner to the upper right front corner, along the diagonal of the main volume of the tank. The vine belonged to one of the many plants Id put in there in the hopes of making it feel more like the Mekong river basin and less like a plexiglass tank in Seattle, Washington.
The betta had his nose on the vine. He was floating just above it, twitching his fins slightly to stay in place, and he was keeping his eyes as close to the vine as possible while keeping it in focus: about half a centimeter to a centimeter. And he was traversing the vine.
With the tiniest of motions, he was propelling himself along the vine towards the lower back right corner, keeping it under close scrutiny at all times. This excursion, from the halfway point to the end of the vine, took him perhaps three minutes. He was taking his time.
When he got to the end of the vine, he remained rooted in place while he inspected the 3-inch-radius spherical volume at the end of the vine, which was truncated in three dimensions by the walls and floor of the tank. He spent about three or four minutes doing this inspection, evidently making sure the vine really did terminate in the corner, and did not escape the tank.
After he had thoroughly scrutinized everything in the betta-sized vicinity of the vines end, he turned back to the vine, nose pressed close, and began working his way along the vine in the other direction.
At this point I sat down to watch, because if he was doing what I thought he might be doing, then... I didnt know what to think. I wanted to see it for myself.
Over the next seven to ten minutes, he crept along the vine, never losing sight of it nor getting further than a centimeter from it, until he reached the upper-right front corner of the tank. He then proceeded to repeat his inspection of the volume at vines end, ensuring himself that the vine terminated in the tank rather than protruding beyond the wall.
But what if he had missed something?
Sure enough, he turned and looked down the length of the vine for a time. And then he put his nose back on the vine and began again his long descent to the other end.
He did this for five days.
By the second day my amazement had turned to concern, and by the third day I felt utterly helpless. Here was an intelligent prisoner, my captive, exploring the mechanics of his prison with a thoroughness that only the imprisoned can afford, looking for an escape with deathly tenacity.
But while purchasing my betta had been easy enough, returning him to his real home would be unthinkably difficult, and probably unsuccessful even if Id tried. Returning him to the fish store seemed like a dead end; he could easily wind up worse off than he was now.
So I concluded that there was nothing I could do. As he inspected the vine, I bit my nails, and timed passed in silence.
After the fifth day he gave up. And then he did something that I still dont understand, even though Ive heard about this kind of thing before, and even though I personally saw him do it: he died of unhappiness.
It only took him a few days. He refused his food, he stopped moving, and to all external appearances he had become ill. But I knew better.
THE LESSON
Whenever I find myself struggling against the tide of massive system complexity, I think of my betta. He had a big heart, a small brain, and a small range of sensory input. I watched him use them all as methodically as any programmer to reason his way through to a soundness proof of the inescapability of his prison.
We like to think of ourselves as being pretty smart. Admit it. We do. But in the grand scheme of things were intellectually little better off than that fish. We can easily find problems so complex that reasoning about them can take days or weeks of microscopic scrutiny, like my fish swimming along his vine.
And we can just as easily envision problems thousands or millions of times more complex: problems beyond the reasoning abilities of any person, any group of people, or even our entire species.
This has ramifications for the way we think about things today.
I believe I will have more to say about this soon. Right now I need to go mourn my fish, whose soul shone as brightly as that of anyone Ive known.
org.eclipse.jdt.core.dom. This package defines strongly-typed classes and interfaces that Eclipse uses for modeling the Java programming language itself.
If you click through any class inheriting from ASTNode youll see that it has a property list. ASTNodes javadoc comment says:
"Each AST node is capable of carrying an open-ended collection of client-defined properties. Newly created nodes have none.I like this example for several reasons. First, its a very simple use of the Properties pattern. It doesnt muck around with prototypes, serialization, metaprogramming or many of the other things Ill talk about in a little bit. So its a good introduction. Second, its placed smack in the middle of a very, very strongly-typed system, showing that the Properties pattern and conventional statically-typed classed-based modeling are by no means mutually exclusive, and can complement one another nicely. And third, their property system _itself_ is fairly strongly typed: they define a set of support classes such asgetPropertyandsetPropertyare used to access these properties."
StructuralPropertyDescriptor, SimplePropertyDescriptor, and ChildListPropertyDescriptor to help place some constraints on client property values. Im not a huge fan of this approach myself, since I feel it makes their API fairly heavyweight. But its a perfectly valid stylistic choice, and its useful for you to know that you can implement the pattern this way if you so choose.
JAVASCRIPT
At the other end of the "how far to go with it" spectrum we have the JavaScript programming language, which places the Prototype Principle and Properties Pattern at the very core of the language.
People love to lump dynamic languages together, and theyll often write off JavaScript as some sort of inferior version of Perl, Python or Ruby. I was guilty of this myself for over a decade.
But JavaScript is substantively different from most other dynamic languages (even Lisp), because it has made the Properties Pattern its central modeling mechanism. It borrowed this heritage largely from a language called Self, and some other modern languages (notably Io and one other language that Ill talk about below) have also chosen prototypes and properties over traditional classes.
In JavaScript, every user-interactible object in the system inherits from Object, which has a built-in property list. Prototype inheritance (think back to our example of the Emmitt Smith instance having been the prototype for the L.T. instance) is a first-class language mechanism, and JavaScript offers several kinds of syntactic support for accessing properties and declaring property lists (as "object literals").
JavaScript is often accurately described as the worlds most misunderstood programming language. Armed with our newfound knowledge, we can start to see JavaScript in a new light. To use JavaScript effectively, you need to gain experience with a whole new School of Modeling. If you simply try to use JavaScript as a substitute for (say) Java or Python, youll encounter tremendous friction.
Since most of us have precious little actual experience with property-based modeling, this is exactly what happens, and its no wonder JavaScript gets a bad rap.
Pushing It Even Further
In addition to how centrally you want to use the Properties pattern in your system, you can also decide how _recursive_ to make it: do your properties have explicit meta-properties? Do you have metaprogramming hooks? How much built-in reflection do you offer?
JavaScript offers a few metaprogramming hooks. One such hook is the recently-introduced __noSuchMethod__, which lets you intercept a failed attempt to invoke a nonexistent function-valued property on an object.
Unfortunately JavaScript does not offer as many hooks as Id like. For instance, there is no corresponding ___noSuchField___ hook, which limits the overall flexibility somewhat. And there are no standard mechanisms for property-change event notification, nor any reasonable way to provide such a mechanism. So JavaScript gets it mostly right, but it stops short, possibly for performance reasons, of offering a fully-extensible metaprogramming system such as those offered by SmallTalk and to some extent, Ruby.
The Pattern Takes Shape...
Before we move on to other uses of the Property Pattern, lets put JavaScript (and its central use of the pattern) into perspective, by comparing it to another successful language.
First: JavaScript is not my favorite language. Ive done a _lot_ of JavaScript programming over the past 2 years or so, both client-side and server-side, so Im as familiar with it as I am with any other language Ive used.
JavaScript in its current incarnation is not the best tool for many tasks. For instance, its not great for building APIs, and its not great for Unix scripting the way Perl and Ruby are. It has no library or package system, no namespaces, and is missing many other modern conveniences. If youre looking for a general-purpose language, JavaScript leaves you wanting.
But JavaScript _is_ the best tool for many other tasks. As just one example, JavaScript is an _outstanding_ language for writing unit tests -- both for itself, and also for testing code in other languages. Being able to use the Properties Pattern to treat every object (and class) as a bag of properties makes the creation of mock objects a dream come true. The syntactic support for object literals makes it even better. You dont need any of the silly frameworks you see coming from Java, C++ or even Python.
And JavaScript is one of the two best _scripting languages_ on the planet, in the most correct sense of the term "scripting language": namely, languages that were designed specifically to be embedded in larger host systems and then used to manipulate or "script" objects in the host system. This is what JavaScript was designed to do. Its reasonably small with some optional extensions, it has a reasonably tight informal specification, and it has a carefully crafted interface for surfacing host-system objects transparently in JavaScript.
In contrast, Perl, Python and Ruby are huge sprawls, all trying (like C++ and Java) to be the best language for every task. The only other mainstream language out there that competes with JavaScript for scripting arbitrary host systems is Lua, famous for being the scripting language of choice for the game industry.
And wouldnt you know it, Lua is _also_ a language that uses the Properties Pattern as its central design. Its central Table structure is remarkably similar to JavaScripts built-in Object, and Lua also uses prototypes rather than classes.
So the worlds two most successful scripting languages are prototype-based systems. Is this just a cosmic coincidence? Or is it possible that a suitably designed class-based language could have been just as successful?
Its hard to say. Ive used Jython as an embedded scripting language for a long time, and its worked pretty well. But Ive personally come to believe that the Properties Pattern is actually better suited for _extensibility_ than class-based modeling, and that prototype-based languages make better extension languages than class-based languages. Thats effectively whats happening with embedded scripting: the end-users are _growing_ and _extending_ the host system.
In fact I was convinced of it before I even knew JavaScript. Lets take a look at another interesting "Who uses it?" example: Wyvern.
WYVERN
My multiplayer game Wyvern takes the Properties Pattern quite far as well, although in some different directions than what weve discussed so far. I designed Wyvern long before Id heard of Self or Lua, and before Id learned anything about JavaScript. In retrospect its amazing how similar my design was to theirs.
Wyvern is implemented in Java, but the root GameObject class has a property list, much like JavaScripts Object base class. Wyvern has prototype inheritance, but since Id never heard of prototypes before, I called them _archetypes_. In Wyvern, any game object can be the archetype for any other game object, and property lookup and inheritance work more or less identically to the way they work in JavaScript.
I arrived at this design after scratching my head for _months_ (in late 1996) over how to build the ultimate extensible game. I wanted _all_ the game content to be created by players, and I came up with dozens upon dozens of detailed use cases, in all of which I wanted players to be able to extend the game functionality in surprising new ways. In the end I arrived at a set of interleaved design patterns, including a rich command system, a rich hooks/advice system, and several other subsystems Id love to document someday.
But the core data model was the Properties Pattern.
In some ways, Wyverns implementation is more full-featured than JavaScripts. Wyvern offers more metaprogramming facilities, such as vetoable property change notifications, which gives in-game objects tremendous flexibility in responding to their environment. Wyvern also supports both transient and persistent properties, a scheme Ill discuss below.
On other ways, Wyvern just made different decisions. One big one is that Wyverns property values are statically typed. The property _names_ are always strings, just like in JavaScript, but the values can be various leaf types (ints, longs, booleans, strings, etc.), or functions (a trick that wasnt easy in Java), or even archetypes.
But despite the differences, Wyverns core property-list infrastructure is a lot like that of JavaScript, Self and Lua. And its been a design Ive been fundamentally happy with for over ten years. Its met or exceeded all my original expectations for enabling end-user extensibility, particularly in its ability to let people extend the in-game behavior on the fly, without needing to reboot. This has proven extraordinarily powerful (and popular with the players.)
Where Wyvern clearly got the pattern wrong was in its lack of support for _syntax_. As soon as I decided to use the Properties pattern centrally in my game, I should have decided to use a programming language better suited for implementing the pattern: ideally, one that supports it from the ground up.
I eventually wound up using Python (actually, Jython) for a ton of my code, and it was far more succinct and flexible than anything I wrote in Java. But I was foolishly worried about performance, and as a result I wound up writing at least half the high-level game logic in Java and piling on hundreds of thousands of lines of getProperty and setProperty code. And now the system is hard to optimize; it would have been much easier if Id had a cleaner separation of game-engine infrastructure from "scripty" game code.
Even if Id done the whole game in Python, Id still have had to implement a prototype inheritance framework to enable any object to be able to serve as the prototype for any other object.
I realize I havent really explained _why_ prototype inheritance works so well, except for my brief mention of mock objects for unit testing. But to keep this article tractable, I had to delete several pages of detailed examples, such as "Chieftain Monsters" that could be programmatically constructed by adding a few new properties to any existing monster
When I told you this pattern was big enough for a book, I meant a _big_ book. Without the examples handy, all I can do is say that using JavaScript/Rhino (or Lua, once it became available on the JVM) might have made my life easier. Or heck, writing my own language might have been the best choice for a system that large and ambitious.
In any case, live and learn. Its a lot of code, but Wyvern is still a properties-based, prototype-based system, and it has amazing open-ended flexibility as a result.
Weve been through the two big examples now (Wyvern and JavaScript). Ill close this "Who Uses It" section with just a few more key examples.
LISP
Lisp features a small-scale example of the Properties Pattern: it has property lists for symbols. Symbols are first-class entities in Lisp. Theyre effectively the names in your current namespace, like Javas fully-qualified class names.
If Java classes all had property lists, it would still be a small-scale instantiation of the Properties pattern, but it would open up an awful lot of new design possibilities to Java programmers. Similarly, Lisp stops short of making _everything_ have a property list, but to the extent it offers property lists theyre exceptionally useful design tools.
Emacs Lisp actually makes heavy use of the Properties Pattern, inasmuch as essentially every one of its thousands of configuration settings is a property in the global namespace. It supports the notion of transient vs. persistent properties, and it offers a limited form of property inheritance via its buffer-local variables.
Unfortunately Emacs doesnt support any notion of prototypes, and in fact it doesnt have any object-orientation facilities at all. Sometimes I want to model things in Emacs using objects with flexible property lists, and at such times I find myself wishing I were using JavaScript. But even without prototypes, Emacs gains significant extensibility from its use of properties for data representation.
Keep in mind that there are, of course, big tradeoffs to make when youre deciding how much to use the Properties pattern; Ill discuss them in a bit. Im not criticizing _any_ of the systems or languages here for the choices theyve made; all of them have been improved by their use of this pattern, regardless of how far they decided to take it.
XML REVISITED
Earlier I described XML as a first-class modeling school. Now that we have more context, its possible to view XML as being an instantiation of the Properties Pattern, inasmuch as it uses the pattern as part of its fundamental structure.
XMLs view of the pattern is two-dimensional: properties can take the form of either attributes or elements, each kind having different syntactic and semantic restrictions. Many folks have criticized XML for the unnecessary complexity of this redundant pair of property subsystems, but in the end it doesnt really matter much, since two ways to model properties is still better than zero ways.
So far weve seen various policies around static checking: Eclipse (strong/mandatory), JavaScript/Lua (very little), and Wyvern (moderate).
XML offers what I think is the ideal policy, which is that it lets you decide for yourself. During your initial domain modeling (the "prototyping" phase -- a term now loaded with Even More Delicious Meaning), you can go with very weak typing, opting for nothing more than simple well-formedness checks. And for many problems, this is as much static checking as youll ever need, so its nice that you have this option.
As some of your models become more complex, you can choose to use a DTD for extra validation. And if you need a really heavy-duty constraint system, you can migrate up to a full-fledged XML Schema or Relax NG schema, depending on your needs.
XML has proven to be a very popular modeling tool for Java programmers in particular -- more so than for the dynamic language communities. This is because Java offers essentially zero support for the Properties Pattern. When Java programmers need access to the pattern, the easiest approach is currently to use XML.
The Java/XML combination has proven reasonably powerful, despite the lack of syntactic integration and numerous other impedance mismatches. Using XML is still often preferable to modeling things with Java classes. Even Eclipses AST property lists might have been better modeled using XML and a DOM: it would have been less work, and the interface would have been more familiar. And as for Apache Ant: JSON-style JavaScript objects for build files would have been exactly what they needed, but by the time theyd realized they needed a plug-in system, the damage was done.
As Mozilla Rhino becomes better documented, and as more Java programmers begin to appreciate the usefulness of JSON as a lightweight alternative to XML, JavaScript may begin to close the gap. Rhino provides Java with the Properties Pattern much more seamlessly than any XML solution. Ive already mentioned that its superior (even to XML) for unit tests and representing mock test data.
But it goes deeper than unit testing. Every sufficiently large Java program, anything beyond medium-sized, needs a scripting engine, whether the authors realize it or not. Programs often have to grow to the size of browsers or spreadsheets or word processors before the authors finally realize they need to offer scripting facilities, but in practice, even small programs can immediately benefit from scripting. And XML doesnt fit the bill. Its yet another example of programmers choosing a School of Modeling because they know it, rather than learning how to use the right tool for the job.
So it goes.
BIGTABLE
Last example: Googles Bigtable, which provides a massively scalable, high performance data store for many Google applications (some of which are described in the paper - click the link if youre curious.) This particular instantiation of the Properties Pattern is a multidimensional table structure, where the keys are simple strings, and the leaf values are opaque blobs.
Hardcore relational data modelers will sometimes claim that large systems will completely degenerate in the absence of strong schema constraints, and that such systems will also fail to perform adequately. Bigtable provides a nice counterexample to these claims.
That said, explicit schemas _are_ useful for many domains, and Ill talk more about how they relate to the Properties Pattern in a bit.
This would probably be a good time to mention Amazons Simple Storage Service, but I dont know anything about it. Ive heard they use name-value pairs.
In any case, I hope these examples (Eclipse AST classes, JavaScript, Wyvern game objects, Lisp symbols, XML and HTML, and Bigtable) have convinced you that the Properties pattern is ubiquitous, powerful, and multifaceted, and that it should be part of any programmer or designers lineup.
Lets look in more depth at how its implemented, its trade-offs, and other aspects of this flexible design strategy.
PROPERTIES PATTERN HIGH-LEVEL OVERVIEW
At a high level, every implementation of the Properties Pattern has the same core API. Its the core API for _any_ collection that maps names to values:
* GET(NAME)
* PUT(NAME, VALUE)
* HAS(NAME)
* REMOVE(NAME)
There are typically also ways to iterate over the properties, optionally with a filter of some sort.
So the simplest implementation of the Properties Pattern is a Map of some sort. The objects in your system are Maps, and their elements are Properties.
The next step in expressive power is to reserve a special property name to represent the (optional) parent link. You can call it "parent", or "class", or "prototype", or "mommy", or anything you like. If present, it points to another Map.
Now that you have a parent link, you can enhance the semantics of GET, PUT, HAS and REMOVE to follow the parent pointer if the specified property isnt in the objects list. This is largely straightforward, with a few catches that well discuss below. But you should be able to envision how youd do it without too much thought.
At this point you have a full-fledged Prototype Pattern implementation. All it took was a parent link!
From here the pattern can expand in many directions, and well cover a few of the interesting ones in the remainder of this article.
REPRESENTATIONS
There are two main low-level implementation considerations: how to represent property keys, and what data structure to use for storing the key/value pairs.
KEYS
The Properties pattern almost always uses String keys. Its possible to use arbitrary objects, but the pattern becomes more useful with string keys because it trivially enables prototype inheritance. (Its tricky to "inherit" a property whose key is some opaque blob - we usually think of inheritance as including a set of named fields from the parent.)
JavaScript permits you to use arbitrary objects as keys, but whats really happening under the covers is that theyre being cast to strings, and they lose their unique identity. This means JavaScript Object property lists cannot be used as be a general-purpose hashtable with arbitrary unique objects for keys.
Some systems permit both strings and numbers as keys. If your keys are positive integers, then your Map starts looking an awful lot like an Array. If you think about it, Arrays and Maps share the same underlying formalism (a surjection, not to put too fine a point on it), and in some languages, notably PHP, there isnt a user-visible difference between them.
JavaScript permits numeric keys, and allows you to specify them as either strings or numbers. If your object is of type Array, you can access the numeric keys via array-indexing syntax.
Quoting
JavaScript syntax is especially nice (compared to Ruby and Python) because it allows you to use _unquoted_ keys. For instance, you can say
var person = {
name: "Bob",
age: 20,
favorite_days: [thursday, sunday]
} and the symbols _name_, _age_ and _favorite_days_ are NOT treated as identifiers and resolved via the symbol table. Theyre treated exactly as if youd written:
var person = {
"name": "Bob",
"age": 20,
"favorite_days": [thursday, sunday]
} You also have to decide whether to require quoting _values_. It can go either way. For instance, XML requires attribute values to be quoted, but HTML does not (assuming the value has no whitespace in it).
Missing Keys
You will need to decide how to represent "property not present". In the simplest case, if the key isnt in the list, the property is not there (but see Inheritance further on).
If a property is frequently removed and re-added, it may make sense to leave the key in the list with a null value. In some systems, you may need null to be a valid property value, in which case youd need to use some other distinguished (and reserved) value for this micro-optimization to work.
DATA STRUCTURES
The simplest property-list implementation is a linked list. You can either have the alternating elements be the keys and values (Lisp does this), or you can have each element be a struct containing pointers to the key and value.
The linked list implementation is appropriate when:
* youre just using the pattern to allow user annotations on object instances
* you dont expect many such annotations on any given instance
* youre not incorporating inheritance, serialization or meta-properties into your use of the pattern
Logically a property list is an unordered set, not a sequential list, but when the set size is small enough a linked list can yield the best performance. The performance of a linked list is O(N), so for long property lists the performance can deteriorate rapidly.
The next most common implementation choice is a hashtable, which yields amortized constant-time find/insert/remove for a given list, albeit at the cost of more memory overhead and a higher fixed per-access cost (the cost of the hash function.)
In most systems, a hashtable imposes too much overhead when objects are expected to have only a handful of properties, up to perhaps two or three dozen. A common solution is to use a hybrid model, in which the property list begins life as a simple array or linked list, and when it crosses some predefined threshold (perhaps 40 to 50 items), the properties are moved into a hashtable.
Note that well often refer to property sets as "property lists" (or "plists" for short), because theyre so often implemented as lists. But its fairly unusual for the order to matter. In the rare cases when it matters, there are usually two possibilities: the names need to be kept in insertion order, or they need to be sorted.
If you need constant-time access and want to maintain the insertion order, you cant do better than a LinkedHashMap, a truly wonderful data structure. The only way it could possibly be more wonderful is if there were a concurrent version. But alas.
If you need to impose a sort order on property names, youll want to use an ordered-map implementation, typically an ordered binary tree such as a splay tree or red/black tree. A splay tree can be a good choice because of the low fixed overhead for insertion, lookup and deletion, but with the tradeoff that its theoretical worst-case performance is that of a linked list. A splay tree can be especially useful when properties are not always accessed uniformly: if a small subset M of an objects N properties are accessed most often, the amortized performance becomes O(log M), making it a bit like an LRU cache.
Note that you can get a poor-mans splay tree (at least, the LRU trick of bubbling recent entries to the front of the list) using a linked list by simply moving any queried element to the front of the list, a constant-time operation. Its surprising that more implementations dont take this simple step: an essentially free speedup over the lifetime of most property lists.
INHERITANCE
With prototype inheritance, each property list can have a parent list. When you look for a property on an object, first you check the objects "local" property list, and then you look in its parent list, on up the chain.
As I described in the Overview, the simplest approach for implementing inheritance is to set aside a name for the property pointing to the parent property list: "prototype", "parent", "class" and "archetype" are all common choices.
Its unusual (but possible) to have a multiple-inheritance strategy in the Properties pattern. In this case the parent link is a list rather than a single value, and its up to you to decide the rules for traversing the parent chains during lookups.
The algorithm for inherited property lookup is simple: _look in my list, and if the property isnt there, look in my parents list. If I have no parent, return null._ This can be accomplished recursively with less code, and but its usually wiser to do it iteratively, unless your language supports tail-recursion elimination. Property lookups can be the most expensive bottleneck in a Properties Pattern system, so thinking about their performance is (for once) almost never premature.
THE DELETION PROBLEM
If you delete a property from an object, you usually want subsequent checks for the property to return "not found". In non-inheritance versions of the pattern, to delete a property you simply remove its key and value from the data structure.
In the presence of inheritance the problem gets trickier, because a missing key does _not_ mean "not found" - it means "look in my parent to see if Ive inherited this property."
To make the problem clearer, assume you have a prototype list called Cat with a property named "friendly-to-dogs", whose value defaults to the boolean true. Lets say you have a specific cat instance named Morris, whose prototype is Cat:
var Cat = {
friendly_to_dogs: true
}
var Morris = {
prototype: Cat
} Lets say Morris has a nasty run-in with a dog, and now he hates all dogs, so we want to make a runtime update to his friendly-to-dogs property. Our first idea might be to delete the property, since a missing key or null value are often interpreted as false in a boolean context. (This is true even in class-based languages like C++ or Java, in which a hasFooBar function will return true if the internal fooBar field is non-null.)
However, Morris does not have a copy of "friendly-to-dogs" in his local list: he inherits it from Cat. So if your deleteProperty method does nothing but delete the property from the local list, he will continue to inherit "friendly-to-dogs", which will irk him (and you) endlessly until you figure out where the bug is.
You cant delete "friendly-to-dogs" from the Cat property list, or all of your cats will suddenly become dog-haters, and youll have outright war on your hands. (Note that in some settings this is _exactly_ what you want to do, illustrating the inherent universal trade-off between flexibility and safety.)
The solution for Morris is to have a special "NOT_PRESENT" property value that deleteProperty sets when you delete a property that would otherwise be inherited. This object should be a flyweight value so that you can check it with a pointer comparison.
So to account for deletion of inherited properties, we have to modify our property-lookup algorithm to look in the local list for (a) a missing key, (b) a null value, or (c) the NOT_PRESENT tag. If any of these apply, the property is considered not present on the object. [Note: the semantics of null values are up to the system designer. You dont have to make null values mean "not there." Either way is fine.]
READ/WRITE ASYMMETRY
One logical consequence of prototype inheritance as weve defined it is that reads and writes work differently. In particular, if you read an inherited property, it gets the value from an ancestor in the prototype chain. But if you _write_ an inherited property, it sets the value in the objects local list, not in the ancestor.
To illustrate, lets add a "night-vision" property to our Cat prototype. Its value is expected to be an integer representing how well the cat can see in the dark. Lets say that the default value is 5, but our hero Morris has been eating his carrots, so we want to set his "night-vision" property value to 7.
The setProperty code does not need to check the parent chain: it simply adds the key/value pair {"night-vision", 7} to Morriss local property list. If we set the property on Cat, then all cats would have Morriss super-vision, which isnt what we want.
This asymmetry is normal. Back in our L.T. / Emmitt Smith example, when we were adding properties to L.T., we didnt want to modify Emmitt! Its just how the pattern works: you override inherited values by adding local properties, even when the override is a deletion.
READ-ONLY PLISTS
Many implementations of the pattern offer "frozen" property lists. Sometimes (e.g. for debugging) its useful to flag an entire property list as read-only. Ruby supports this via the "freeze" method on the built-in root Object class. In any sufficiently large, robust implementation of the Properties pattern, you should include the option to freeze your property lists.
If you offer a "freeze" function, you should think about whether you want to offer a "thaw" as well. The decision depends on whether you want to offer programmers additional protection, or you just want to lock them up and throw away the key.
My personal view is that Java programmers tend to overuse the "freeze" function when they start with Ruby. For that matter, they tend to overuse "final" in Java. I mentioned before the trade-off between _flexibility_ and _safety_. When you use the Properties Pattern, youre consciously choosing flexibility over safety, and in many domains this is the right choice. In fact, safety can be viewed as a kind of optimization: something that should ideally be layered on, functioning behind the scenes rather than being interleaved with the user APIs and flexible data model.
A nice (and simple to implement) compromise on safety and flexibility is to offer a ReadOnly property attribute, as JavaScript does. There are certain properties (such as the parent pointer) that are less likely to need to change as the system evolves, so its probably OK to lock them down early on. Doing this on a property-by-property basis is much less draconian. Even better, you should consider making the ReadOnly property attribute non-inheritable, so that subtypes can choose their own policies without compromising the integrity of the supertypes.
Were more or less done with inheritance: its not very complicated. There are a few other inheritance-related design issues that Ill cover in upcoming sections.
PERFORMANCE
Performance is one of the biggest trade-offs of using the Properties Pattern. Many engineers are so concerned with performance (and its attendant paradoxes and fallacies) that they refuse to consider using the Properties pattern, regardless of the situation.
As it happens, the patterns performance can be improved and mitigated in several clever ways. I wont cover all of them here, but Ill touch on some of the classics and one or two new approaches.
INTERNING STRINGS
Make sure your string keys are interned. Most languages provide some facility for interning strings, since its such a huge performance win. Interning means replacing strings with a canonical copy of the string: a single, immutable shared instance. Then the lookup algorithm can use pointer equality rather than string contents comparison to check keys, so the fixed overhead is much lower.
The only downside of interning is that it doesnt help much when youre constructing a property name on the fly, since you still need to hash the string to intern it.
Thats not much of a downside, so as a rule, you should always intern your keys. A large percentage of property names in any system are accessed as string literals from the source code (or are read from a configuration file and can be interned all at once when the file is read), and interning works in these common cases.
COROLLARY: dont use case-insensitive keys. Its performance suicide. Case-insensitive string comparison is really slow, especially in a Unicode environment.
PERFECT HASHING
If you know all the properties in a given plist at compile-time (or at runtime early on in the life of the process), then you might consider using a "perfect hash function generator" to create an ideal hash function just for that list. Its almost certainly more work than its worth unless your profiler shows that the list is eating a significant percentage of your cycles. But such generators (e.g. gperf) do exist, and are tailor-made for this situation.
Perfect hashing doesnt conflict with the extensible-system nature of the Properties pattern. You may have a particular set of prototype objects (such as your built-in monsters, weapons, armor and so on) that are well-defined and that do not typically change during the course of a system session. Using a perfect hash function generator on them can speed up lookups, and then if any of them is modified at runtime, you just fall back to your normal hashing scheme for that property list.
COPY-ON-READ CACHING
If you have lots of memory, and your leaf objects are inheriting from prototype objects that are unlikely to change at runtime, you might try copy-on-read caching. In its simplest form, whenever you read a property from the parent prototype chain, you copy its value down to the objects local list.
The main downside to this approach is that if the prototype object from which you copied the property ever changes, your leaf objects will have the now-incorrect old value for the property.
Lets call copy-on-read caching "plundering" for this discussion, for brevity. If Morris caches his prototype Cats copy of the "favorite-food" property (value: "9Lives"), then Morris is the "plunderer" and Cat is the plundered object.
The most common workaround to the stale-cache problem is to keep a separate data structure mapping plundered objects to their plunderers. It should use weak references so as not to impede garbage collection. (If youre writing this in C++, then may God have mercy on your soul.) Whenever a plundered object changes, you need to go through the plunderers and remove their cached copy of the property, assuming it hasnt since then changed from the original inherited value.
Thats a lot of stuff to keep track of, so plundering is a strategy best used only in the direst of desperation. But if performance is your key issue, and nothing else works, then plundering may help.
REFACTORING TO FIELDS
Another performance speedup technique is to take your N most commonly used properties and turn them into instance variables. Note that this technique is only available in languages that differentiate between instance variables and properties, so it would work in Java but not in JavaScript.
This optimization may sound amazingly attractive, especially during your first round of performance optimizations. Be warned: this approach is fraught with pitfalls, so (as with nearly all performance optimizations) you should only use it if your profiler proves that the benefits will outweigh the costs.
The first cost is API incompatibility: suddenly instead of accessing all properties uniformly through a single getProperty/setProperty interface, you now have specific fields that have their own getters and setters, which could potentially have massive impact in your system (since, after all, these are the most commonly-accessed properties). And unless youre using Emacs, your refactoring editor probably isnt smart enough to do rename-method constrained on argument value.
You can mitigate the API problem by continuing to go through the get/setProperty interface, and have them check the argument to see if its one of the hardwired fields. This will result in an increasingly large switch-statement (or equivalent), so youre trading API code maintenance for API simplicity. It also slows down the field access considerably, which offsets the performance gain from using a field.
The next cost is system complexity: you have twice as many code paths through _every_ part of your system that deals with the Properties pattern. Does inheritance still work the same? What about serialization? Transient properties? Metaprogramming? What about your user interface for accessing and modifying property lists? You face a huge code-bloat problem when you split property access into two classes: plist properties and instance variables.
The next cost is accidentally getting it wrong: how do you know what the most-accessed properties are? You may do some runtime profiling and see that its one set, but over time the characteristics of your system might change in such a way that its an entirely different set. Realistically you will have to instrument your system to keep track, on a regular basis, of which properties are accessed at a rate beyond your tolerance threshold, so you can convert them to fields as well. This isnt a one-off optimization.
But all these costs pale in comparison to the big one, which is extensibility: instance variables are set in stone. It will be a vast amount of work to try to give them parity with your plist properties: change-list notifications, the ability to override them or remove them, and so on. Its likely that you will wind up sacrificing at least some flexibility for these fields.
So use this optimization with extreme caution.
REFRIGERATOR
The last performance optimization Ill mention is more about conserving memory than CPU. If youre worried about the overhead of a per-object property-list field, even if its usually null, then you can implement property lists using a separate, external data structure.
I dont know what its normally called, so Im calling it the Refrigerator, since youre basically putting yellow stickies all over it. The idea is that you dont need to pay for the overhead of property lists when very few of the objects in your system will ever have one. Instead of using a field in each class with a property list, you maintain a global hashtable whose keys are object instances, and whose values are property lists.
To fetch the property list of an object, you go look to see if its on the Refrigerator. The property list itself can follow any of the implementation schemes I discussed in Representations above.
I first heard this idea from Damien Conway in roughly 2001 at a talk he gave. He said he was considering using it for Perl 6, and I thought it was pretty clever. I dont remember what he called it, and I dont know if he wound up using it, but consider this idea to be his gift to you. Thanks, Damien!
REDACTED
Brendan Eich came up with astoundingly clever performance optimization for the Properties Pattern, which he told me about back in January. I was ready to publish this article, but I told him Id hold off until he blogged about his optimization. Every once in a while hed ping me and tell me "any day now."
Brendan, its _October_, dammit!
ROLLING YOUR OWN
Needless to say, Ive only scratched the surface on performance optimization of the Properties pattern. You can get arbitrarily fancy. The point Im trying to get across is that you shouldnt despair when you discover your system is unacceptably slow after designing it to use the Properties pattern. If this happens, dont panic and throw out your flexibility - go optimize! The game of optimization can be fun and rewarding in its own right.
Just dont do it before you need it!
TRANSIENT PROPERTIES
While implementing Wyvern, I discovered that making changes to a persistent property list is a wonderful recipe for creating catastrophes.
Lets say some player casts a Resist Magic spell, which boosts her "resist-magic" integer property value by, oh, 30 (thirty percent). Then, while the spell is active, the auto-saver kicks in (writing her enhanced "resist-magic" property value out to the data store along with the rest of her properties), and then the game crashes.
Voilà - the player now has permanent 30% magic resistance!
It doesnt have to be a game crash, either. Any random bug or exception condition (a database hiccup, a network glitch, cosmic rays) can induce permanence in what was intended to be a transient change to the plist. And when youre writing a game designed to be modified at runtime by dozens of programmers simultaneously, you learn quickly to expect random bugs and exception conditions.
The solution I came up with was transient properties. Each object has (logically speaking) _two_ property lists: one for persistent properties and one for transients. The only difference is that transient properties arent written out when serializing/saving the player (or monster, or what-have-you.)
Wyverns property-list system has typed values. I havent talked about Properties Pattern type systems yet, but in a nutshell my property values can be ints, longs, doubles, strings, booleans, archetypes (which is basically any other game object), or "bean" (JavaBean) properties.
My early experimentation yielded the interesting rule that non-numeric transient properties _override_ the persistent value, but numeric properties _combine_ with (add to) the persistent value.
A simple example should suffice. If you have a persistent boolean property "hates-trolls" (and who doesnt, really?), and you accidentally ingest a Potion of Troll Love, then the potion should set a transient value of {"hates-trolls", false} on your character. It overrides the persistent value. Theres no combining going on; it just replaces the original.
However, for our "resist-magic" int property, if you put on a ring of 30% magic resistance, it should (by default) _add_ to your current value, which may be a combination of innate resistance and resistances conferred from other magic items and spells.
This numbers-are-additive principle applied pretty uniformly across my entire code base and property corpus, so its built into the lookup rules for Wyverns property lists. getIntProperty("foo") must get both the transient and persistent (possibly inherited) values for "foo" and add them before returning the result.
I experimented with different approaches for representing transient properties. Originally I used a kind of hungarian-notation, prefixing transient property names with an @-character ("@foo") and keeping them in the same hashtable as the persistent properties. One advantage of "@" was that it was invalid character in XML attribute names, so it was originally _impossible_ for me to accidentally serialize a transient property.
Eventually I migrated to keeping them in separate (lazily created) tables. This made it easier to deal with interning names (not having to prepend "@" all the time) and generally made bookkeeping easier. I dont remember all the trade-offs involved with the decision anymore (it was about 7 years ago), so youll have to retread that road yourself if you decide to offer transient properties in your system.
THE DELETION PROBLEM (REMIX)
Transient properties introduce their own version of deletion problem. You cant just remove the property from the transient list, since the lookup algorithm will just look for it in the persistent list. And you dont want to remove it from the persistent list; that defeats the purpose of using transients.
The solution is similar to what I did for deleting inherited properties: you insert a placeholder into the transient list saying "NOT_PRESENT", and as long as that placeholder is in the list, its as if the object doesnt have the property.
Note that this implies the existence of two similar API calls: removeTransientProperty for deleting a transient property from the transient list, and transientlyRemoveProperty for temporarily hiding a property from the persistent list.
PERSISTENCE
Persisting property lists is a huge topic; Ill just touch on the basics.
For starters, XML and JSON (and for that matter, s-expressions) are all perfectly valid choices for serialization format. You can probably imagine how this works, so I wont beat it to death.
Text-based formats have big wins in readability, maintainability, and bootstrapping (you dont need to create special tools to read and write them).
For performance - both for network overhead and disk storage - you might want to consider designing a compressed binary format. One easy way to test whether this will be a win for you is to take the intermediate approach of gzipping your data to see how well it compresses, and whether it produces a discernable blip in performance.
Wyvern initially used a filesystem trie-like structure for storing its data, but as the number of distinct objects grew to several hundred thousand, I had to switch to a database.
You can use an RDBMS, but youre in for a world of hurt if you try to map the Properties pattern onto a relational schema. Its a tricky problem, and probably isnt one that you want to solve yourself.
I wound up using an RDBMS and just shoving the XML-serialized property list into a text/clob column, and denormalizing the twenty or thirty fields I needed for queries into their own columns. This gets me by, but isnt a happy solution.
What you really want is a hierarchical data store optimized for loose tree structures: in a word, an XML database. At the time I was designing Wyverns persistence strategy (1998-ish), XML databases were pure vaporware, and even after a few years they were still fairly exploratory and unstable.
Today things are different, and there are many interesting options for XML databases, ranging from 100% free (e.g. Berkeley DBs) through 100% expensive (e.g. Oracle XML).
You might also look into Object databases, but Ive never heard of anyone coming through that experience with anything but battle scars to show for it.
QUERY STRATEGIES
Querying goes hand-in-hand with persistence: once you have a bunch of objects in a data store, youll want to ask questions about them. Producing a High Score List is a good example: you want to compute a function of some set of properties across all the players in your database.
If youre just using the filesystem, youre stuck with grep or its moral equivalent on Windows, which is likely to be painfully slow. So dont do that.
If youre using an RDBMS, and youve serialized your property lists into a single row-per-object clob or text column, then you can use (My)SQLs LIKE and RLIKE operators, or their equivalents, to do free-text searches.
However, your property lists are likely to be hierarchical (e.g. player inventory is a bag, which has its own properties _and_ collection of objects its holding), and free-text search doesnt understand hierarchy. So this approach is really just a faster version of grep.
Querying is the biggest reason for using an XML database, since it gives you XPath and XQuery as expressive languages that work on XML data about as well (give or take) as SQL works on relational data.
Because you have the advantage of working in "these days" (2008+) as opposed to "those days" (1998), you now have the interesting option of using JavaScript/JSON and JQuery. I dont know much about it, but what little I do know seems promising.
One final approach, which may not scale very well unless you can find a way to parallelize it, is to simply load all the objects into an instance of your server, and use programmatic access to walk the objects and construct your query results manually. Although it requires some infrastructure to make it work (and to make it not crash your system, once you have enough objects), it has the major benefit of giving you a full programming language, which can be useful if youre doing a complex query and your XPath/XQuery skills arent up to par.
BACKFILLS
Data integrity, a.k.a. Safety, is one of the two biggest trade-offs (the other being performance) you make when you choose to use the Properties pattern. In the absence of a schema, your system is open to property-list corruption through bugs and user error (e.g. typos).
Interestingly, in big companies Ive worked at that have strong schema constraints, they _still_ always seem to run into data-integrity problems, so its not clear how much the schema is really helping here. A schema can certainly help with navigability and performance, but no schema can completely avert data corruption problems.
As soon as you notice youve got bad data, you need to do what many people in the industry term a "backfill": you have to run through all the existing data and fix the problem. Sometimes this is as simple as running a SQL update on a single column. And sometimes it involves writing a complex program that painstakingly computes the inverse of whatever bogus operation created the bad data in the first place.
And sometimes backfills require just winging it, since the lost data may be irrecoverable and you need to use heuristics to minimize the damage. No matter how you store your data and how careful you are about replicating it and backing it up, this kind of thing can happen at pretty much any scale.
The Properties pattern doesnt really introduce anything new to the backfill landscape; all the usual options apply. Youll just need to be mindful that user error (especially mis-typed property names) can make backfills a bit more commonplace, so you should plan to spend a fair amount of time developing a convenient backfill infrastructure for your system.
I should mention, embarrassing as it is, one other option, which I call "lazy backfill", and Ive used it extensively. Sometimes Ill notice a data-corruption issue that needs fixing but doesnt really justify a day of my time to fix all at once. So I have a small subsystem in place for player logins and map loads: I iterate through the property lists looking for properties that Ive flagged (hardwired in the code) as "bad data", and I call helper backfill functions on the fly to fix just that property list.
This is obviously a hack, and it also imposes some minor performance overhead (probably not detectable via profiling) on logins and map loads, but Ill be honest: its served me well, and Ive fixed at least 20% of my data-corruption problems this way.
TYPE SYSTEMS
Ive already touched on this a little here and there. Eclipses AST property lists use an interesting type system that provides a reasonable amount of metadata for each property, although (I think) it stops short of allowing properties to have their own property lists.
JavaScript properties have a small, fixed amount of metadata. Each property has a set of flags. The flags include ReadOnly (cant modify the value), Permanent (can modify the value but cant delete the key), DontEnum (key doesnt show up in iterators but can be read directly), and others depending on the implementation.
Wyvern has its own Java-like flavor of typed properties, largely because I implemented the system in Java long before the advent of auto-boxing, and I needed a convenient way of dealing with primitive types vs. object types. If I were to do it all over again, I probably wouldnt go that route. I _would_ want some sort of scheme for metaproperties (aka "property attributes") -- perhaps in a separate per-object metaproperty-list. But Id simplify the interface and get rid of all the primitive-typed versions of all my has/get/set inherited/persistent/transient property calls.
I wont go into any further detail about type systems, except to say that (a) you can use them to any degree you desire; theres nothing intrinsic to the Properties Pattern that precludes them, and (b) Duck Typing becomes fairly crucial to systems that are designed fully around the Properties Pattern, so if your language has any structural-typing support itll help.
TOOLKITS
Wyvern has a Map Editor that allows you to create and edit objects. Since all the game objects are property lists that use the prototype inheritance pattern, the conventional JavaBean approach doesnt work, since the JavaBeans API (which is more or less designed for this problem, except with instance fields) uses Java reflection, and my properties dont have individual getters and setters.
Wyvern wound up with something very similar to a JavaBeans property editor, except it knows how to read, write and display my GameObject property lists.
It wasnt a huge amount of work, but its something you should keep in mind as you decide whether to use the Properties pattern in your system. If you need a GUI for object edits, youll probably need to do some custom UI work.
PROBLEMS
Ive talked about the main problems imposed by the Properties pattern: performance, data integrity, and navigability/queryability. Theyre all trade-offs; youre sacrificing in these areas in order to achieve big wins in flexibility and open-ended future extensibility for users you may never meet. (You also get huge wins in unit-testability and prototyping speed, but I assume these benefits are obvious enough that I dont need to dwell on them.)
One other problem is reversability: its hard to back out of the Properties pattern once you commit to it. Once people start adding properties, especially if theyre using programmatic construction of key names, youll have trouble on your hands if you want to try to refactor the whole system to use instance fields and getters/setters. So before you use this pattern, you should put your system through a prototyping phase (ironically enough) to determine whether it will work out as it scales.
FURTHER READING
I wasnt able to find much, but here are some interesting articles and papers on the subject.
Dealing with Properties [Fowler 1997]
Prototype-based programming (Wikipedia)
Do-it-yourself Reflection [Peter Sommerlad, Marcel Rüedi] (PDF)
Prototype pattern (Wikipedia)
Self programming language (Wikipedia)
Refactoring to Adaptive Object Modeling: Property Pattern [Christopher G. Lasater]
NEW UPDATES
_Oct 20th 2008:_ The comments on the article are outstanding. People have pointed out lots of further reading, as well as well-known systems (Atom, RDF) that make extensive use of the pattern at their core. Thanks, folks! This is great stuff.
_Oct 20th 2008:_ Martin Fowler sent me a link to a 1997 paper he wrote on the topic. Its in the Further Reading section. Well worth a read. He mentions a few important considerations that I left out:
* THE (STATIC) DEPENDENCIES PROBLEM -- your compiler cant generally help you with finding dependencies on properties, or even tell you what property names are used in your system. He suggests using a registry of permissible property names as one solution. I strongly considered that approach for Wyvern, but wound up relying on a mix of dynamic tracing and static analysis to get me the dependency graph, and its been "accurate enough" for my needs. In particular, synthesizing property names on the fly happens about as (in)frequently as Reflection happens in Java. So for the most part the dependencies issue is as tractable as it is in Java: "tractable enough".
* SUBSTITUTING ACTIONS Fowler suggests that its difficult to replace an existing property access with an action. This is only true in the Java implementation (hence, true for my game). In languages like Python, Ruby and JavaScript 1.5 that support property-access syntax for getters and setters this is a non-issue.
Overall, Martins take on the pattern is "avoid it when possible", which is sound (if conservative) advice. My take is that everyones doing it anyway, so we should formalize it. Ive used it as the central data model for my 500,000-line multiplayer game for 10 years, and I assert that the benefits vastly outweigh the problems. I also witnessed the patterns use in Amazons Customer Service Tools database for some 5 years, and again, the benefits vastly outweighed the downsides.
You just have to know what youre getting into before you dive in, which is sort of the point of my article.
_Oct 20th 2008:_ Fellow Googler Joe Beda mentioned that IE4 originally supported arbitrary attributes on HTML elements, which dramatically extended the flexibility for web developers. Today, no browsers support it, though John Resig claims HTML 5 will fix this. In the meantime, developers use fake css classes and hidden elements; its a mess. I actually deleted a pretty large rant about this problem from the article. But yeah. Its a problem. When you dont provide the Properties Pattern to people, they find horrible workarounds, which is much worse than anything that can go wrong if you simply support it directly. (Joe mentioned that it posed serious technical problems with the cache, though, so I wouldnt assume its trivial to add the support back in to browsers today.)
FINAL THOUGHTS
I havent covered the whole landscape for this pattern. There are concurrency issues, access-control issues (e.g. in Wyvern, some properties, such as email address, can only be read by very high-level Wizards), documentation issues, and a host of other considerations.
Let me summarize what I think are the key takeaways.
First: this is a critically important pattern. I call it the "Universal" design pattern because it is (by far) the best known solution to the problem of designing open-ended systems, which in turn translates to long-lived systems.
You might not think youre building that kind of system. But if you want your system to grow, and have lots of users, and spread like wildfire, then you _are_ building exactly that kind of system. You just havent realized it yet.
Second: even though people rarely talk much about this pattern, its astoundingly widespread. It appears in strongly-typed systems like Eclipse, in programming and data-declarative languages, in end-user applications, in operating systems, and even in strongly typed network protocols, although I didnt talk about that use case today. (Nutshell: a team I know using CORBA got fed up and added an XML parameter to every CORBA API call, defeating the type system but permitting them to upgrade their interface without horking every existing client. Bravo!)
Third: it can perform well! Or at least, "well enough". The playing field for potential optimizations is nearly unbounded, and with enough effort you can reduce just about everything to constant time.
Finally, its surprisingly versatile. You can use it on a very small scale to augment one teeny component of an existing system, or you can go the whole hog and use it for everything, or just about anything in between. You can start small and grow into it as you become more comfortable with the pattern.
The Properties Pattern is _not_ "just" name/value pairs, although the name/value pair certainly lives at the heart of the pattern.
If you believe Hofstadter, the Properties Pattern (using the Prototype Principle) is an approach to modeling that complements class-based modeling: both are fundamental to the way our brains process symbolic information.
I suspect that if youve read this far, youll start seeing the Properties Pattern everywhere you look. Its embedded in many other popular patterns and systems, from Dependency Injection to LDAP to DNS. As legacy systems struggle to evolve to be more user configurable and programmable, youll see more and more of your favorite software systems (and, I hope, languages) incorporating this pattern to varying extents.
Again: I wasnt able to find much literature about this pattern. If you happen to know of books, papers or articles that expound on it in more detail, please link to them in the comments!
I hope you found this interesting and potentially useful. It was definitely more work than my typical posts, so if it doesnt go over well, Im happy to go back to the comedy routine. Well see.I havent been blogging much this summer. Mostly its because all my free time has been spent engaged in an important research project called "What Would Niko Bellic Do?" Ive been enrolled in a high-quality Management Scenario Simulator with the unconventional name "Grand Theft Auto IV", probably some sort of inside joke, and Ive been going through all its Developer Management training courses.
You know how these corporate training videos go, right? They set up some contrived scenario with actors youre supposed to identify with, and the actors have inane discussions about sexual harrassment or bribing government officials or stealing company equipment, and then youre asked to answer questions about whether it was OK for Bob to grab Sues ass in that particular edge-case scenario.
Seriously, I just took one of these courses at work. Youd think theyre a joke, but no, theyre considered Important Employee Training.
Well, this Niko Bellic course isnt much different, just more fun. I finished the final management training session a couple weeks ago. And by truly amazing coincidence, right after I finished the final training mission, my blog made it into the Top 100 Blogs for Development Managers (Q3 2008).
I personally thought that was a little weird, seeing as Ive never written explicitly about software development management. Unless of course you count that one time where I wrote about how you dont need them.
BUT, now that Ive finished all the Niko Bellic Advanced Management Training courses, Im officially a Certified Expert Dev Manager, capable of not only handling unforseen tricky management situations, but also teaching others about them! So now I think my blog being in that list is totally justified.
All the training in the simulator was top-notch. A lot of it was ground Id covered before in my days as an actual dev manager. I already knew how to react, for example, when my mafia boss told me hed been betrayed by my previous manager, and I had to take him out. Id already gone through stuff like that at Amazon dozens of times, so I breezed through those missions.
But even though a lot of the training was pretty obvious stuff, which you come to expect from these corporate training courses, I was pretty impressed by the final Management Training Mission, in which I had to make an Important Management Decision about an employee in my care. Im sure many of you have also completed this course.
In the missions leading up to the finale mission, the "Grand Theft Auto" training software gradually unfolds a background story in which you and this other employee used to work together but had a disagreement that never fully reached closure. You have some trouble arranging a meeting with the employee to discuss it. But as luck would have it, after doing some especially fine work for one of your internal customers, they manage to convince the employee to meet with you, by gagging and handcuffing him and stuffing him on a plane in St. Petersburg and dropping him off at the docks in Liberty City. I had to admit, the simulator really nailed the value of building strong relationships with internal customers.
You can imagine how hard it is to write an online corporate training seminar -- I mean, you have to get all this material across but keep it entertaining so people actually remember the lessons. The fine folks at the consulting firm "Rockstar" have done an admirable job of presenting the content. They cover all of the HR-sensitive topics an aspiring manager needs to know about: conflicts of interest, employee drug abuse, bribery, sexual harassment, stealing company property, I mean the list goes on and on. Its amazingly thorough. These guys clearly have real-world software industry experience.
And they managed to make it fun! They did this by using certain metaphorical devices, purely for dramatic effect. For instance, in the final mission, youre given a choice: you can either let the employee walk away, allowing bygones to be bygones, or you can execute him by shooting him in the head as he begs for his life. This playful enactment is obviously a metaphor for deciding whether to take the "next step" in dealing with a problem employee: whether or not to put him on a Performance Improvement Plan, or if thats already failed, whether or not to terminate employment. By generalizing it to a back-alley execution scenario, theyve given you a way to apply the lesson across many similar situations. Its actually quite brilliant, and Im sure other firms offering management training courses will take note.
As you know, in many management situations theres no "right answer": its a personal decision, and different managers will make different choices. Hence, the final mission, unlike all the other missions, has no "success" outcome: you just pick a course of action and you have to live with it, just like in real life. Regardless of whether you choose to forgive him for displaying a moment of weakness, showing that you understand hes human like the rest of us, or you pop a cap in his motherfucking partner-betraying ass, you goddamn mother fucker *blam* *blam* *blam* *blam* *BLAM* *click* *click* *click*, the other actors will empathize with you and feel your pain, since no tough decision comes without a few regrets.
As it happens, the simulation software is remarkably open-ended. When I did the mission, I decided to let the employee walk, but then I accidentally ran him over with an 18-wheeler as I was driving out of the parking lot, killing him instantly. Oopsie! I confess I may not have been paying strict attention to some of the simulators more obscure traffic rules by the end of the training sessions. Fortunately they werent grading me on little driving slip-ups.
The game chose to interpret my accidental crushing of the employee beneath the wheels of my semi as intentional termination. I like to think of it as modeling the "passive-aggressive" approach, in which you dont fire the employee personally; you have someone else do it, such as HR, or perhaps your boss. Some managers prefer the passive-aggressive approach.
To illustrate why its popular, Ill use an analogy from the restaurant industry. Have you ever noticed that at restaurants, your waiter doesnt bring your food? Other waiters always bring out your food, during which time your waiter is nowhere to be seen. This is so that if you become infuriated because you specifically ordered tartar sauce on the side, and after a 45-minute wait the chef seems to have emptied the entire bottle of tartar sauce on your fish sandwich in some sort of twisted artistico-culinary attempt to make it look like he threw up on it, then you _dont blame your waiter_. Instead, you unwittingly direct your anger at the person who brought your food, who makes sympathetic noises ("Gosh, Im so sorry - I cant believe they messed that up!") and runs away, never to be seen again. After its eventually resolved (by still other people bringing replacements out), your waiter finally rematerializes and apologizes for the kitchen screwup.
Studies have shown that this yields higher tipping on average. Thats why your waiter doesnt bring your food. Now you know.
The passive-aggressive employee-discipline approach is a bit unusual, but some managers feel its better to have the employee mad at "the company", since then the manager will then get better manager reviews from the employee. Ill write more about this technique in my upcoming "How to Be an Evil Manager" handbook. Should be quite popular! Maybe itll even bump me up the Top 100 list.
OBLIVIATING
Now that Ive finished those training courses, I have all these half-written blogs I need to finish up. Tons of cool stuff to talk about. Unfortunately theyre all stalled in various stages because I spent the summer playing video games and doing a lot of bike rides. Sometimes youve just gotta do that, you know.
GTA IV was a really good game, so after I finished it (which required some time before I got tired of experimenting with 5- and 6-star warrant levels), I started looking around for an equally good game. GTA IV was basically the best game Id played since Oblivion, which was two years ago. I dont play games that often (just a few each year), so I figured there _must_ be something good out there.
I looked and looked, and by golly, the best game out there was _still_ Oblivion. Not only that, but in the intervening 2 years its sprouted all sorts of mods and expansions like the "Knights of the Nine" and the "Shivering Isles". Neat.
So after lots of searching (and not finding) a good follow-up to GTA IV, I stumbled on a used copy of the Game of the Year edition. Since nothing else out there today looked anywhere near as good, I started playing Oblivion again.
Remember how a few months ago I finally turned off my last Windows box? (I switched 100% to using Mac clients and Linux servers: at home, at work, and on the road.) Well, technically to restart Oblivion where I left off, I was going to have to turn my XBox 360 back on, which felt like cheating. But the used Game of the Year edition I found was for the PS/3.
Decisions, decisions...
Actually it wasnt much of a decision at all. Im on a low-Microsoft Diet, meaning I avoid Windows whenever I possibly can. Given that Id gone for 3 or 4 months or so without interacting with a single Windows installation, I really didnt want to turn on my XBox. Plus its summer and I dont need a space heater yet.
So I went for it. Even though my previous character had finished the main quest and was waiting for her imperial dragon armor, I decided to start from scratch and play the whole thing over again on the PS/3, but this time continue on to the expansions after finishing the main quest.
I tell ya: if Oblivion had come out today rather than 2+ years ago, it would _still_ win the Game of the Year. Its just the best game, period.
To be honest, I kind of miss Morrowind (the previous game in the series). I find myself missing it even when Im playing Oblivion. I think it may have actually had more personality than Oblivion. But Im into the Shivering Isles expansion now, and its _way_ better than Oblivion on personality; Im enjoying it even more than the main game.
I still find it weird that they dont re-make classic games. Dont you? I mean, they re-make classic movies, so why not games? Sure, weve got game franchises like Zelda and Mario and Donkey Kong -- Nintendo is awesome at building character brands. But even Nintendo doesnt re-make great old games; they just carry forward characters and ideas and themes. But that just makes me miss the old games!
For instance, Zelda 64: Ocarina of Time -- my God, that was a fantastic game. One of the best ever. Ditto for Final Fantasy X. You could argue that was the best game ever, and I wouldnt try to pop a cap in your motherffffffffffff -- sorry, sorry, management training getting the better of me. Lets just say I wouldnt try to argue with you. It was a uniquely great game.
But both of those games, great as they were, could stand a make-over: updated graphics, updated game physics and mechanics, updated for newer platforms, etc. Theyd be playable and (presumably?) popular today, and it would surely be a lot easier than trying to design a new game from scratch, so studios could release these remakes in "dry spells" between flagship game releases for extra revenue.
It seems like itd be better than releasing crap games that they _know_ suck. Which is most of them.
I dont know. Maybe people just prefer new games, and thats that. Maybe Im different. But nearly 40 years as a gamer has convinced me that truly great games only come along every few years, and the rest are all just filler. Id rather go back and re-play a favorite game, and get that rush of nostalgia, than fill the void by playing crap.
Ultima IV -- now _there_ was a game. Once or twice a decade a truly extraordinary, groundbreaking game comes along, and I remember when Ultima IV was that game. I would _love_ to play it again, except this time without having to swap the City Disk with the Wilderness Disk and listen to my Commodore 64 crunch away for minutes on end.
One of the really key elements to the overall atmosphere of a game is the music, and most great games were great -- in large part, I think -- due to their music. Zelda 64, Final Fantasy X, Kingdom Hearts, Ultima IV, they all had great music. Sometimes the composer was limited by the sound system - in Ultima IVs case, _very_ limited. But I have all of Ultima IVs music on my iPhone, converted from MIDI files. Dead serious. I _love_ those tunes. They remind me of the old Quest for the Avatar days, back in the Navy when my 3 roommates and I manned the console 24 hours a day in shifts, trying to find mandrake root.
Someone please make an updated Ultima IV! I want IV back, dammit. The sequels never even came close.
Not likely, though. Maybe in a generation or two. Its usually at least a few decades before they remake a classic movie, so maybe Im just not waiting long enough.
If they remake a classic game, I think its really important to have the same music. This is where remakes (if you can call em that -- theyre usually either sequels or unrelated new games with similar elements) tend to screw things up. The music should be updated by a competent composer, but it should be the _original_ themes. The Mario franchise does a pretty good job of this, actually. Zelda, not so much. I havent heard the "Gerudo Valley" theme in any Zelda game since Z64, which came out ten years ago, and along with the title theme its one of the two best Zelda themes ever.
Remember Metroid? It had really cool music all around. I didnt play any of the Metroid franchise after that, until last year when I played Metroid: Prime on the Wii. The music _sucked_! Actually the game kinda sucked too. The controls were phenomenal, and made me want to play _all_ games like that, but the game was boring. Sigh.
Its also kind of important that remakes Not Suck. I guess franchise sequels arent really remakes, though, so apparently its OK if they suck. That seems to be the standard. There are exceptions. Mario Galaxys music was Top 5 of all time. But if you cant afford that Japanese guy, its OK to reuse old music! Im serious! I give you my explicit permission!
Anyway, Ill probably keep playing Oblivion until New Castle Wolfenstein comes out. I just dont see anything else all that compelling on the horizon. Just counting the weeks until Bethesdays next Elder Scrolls release, I guess. Morrowind to Oblivion was a 5-year wait, so Im gonna have to find _something_ to do for the next 3 years.
Excuses aside, I should really blog more often. I was pretty amazed at how much email my last entry received. To be perfectly on the up-and-up, I havent read the comments. Im sorry, but Ive been too busy trying to find a Daedric Katana to enchant. Priorities first!
But I checked my GMail account on Friday and it was completely swamped with dozens and dozens of new emails, many more than usual. People seemed to feel it made us buddies, which meant they could email me just to say "hi", and ask if I wanted to "hook up." I dont know if thats good or not, but I tried to answer all their mail.
Oh wait, that was Nikos account in the Corporate Email Training simulator. My bad.
Anyway, todays blog entry was just procrastination. Im supposed to be getting some work done, so Ill get back to it. Maybe this bandit will have a katana.
Some CEO emailed me the other day. I dont remember who it was; people mail me all the time about their blah blah yawn product service thingy, and on the rare occasions I bother to read mail from strangers, I dont usually remember anything about the email, even if I respond to it. I can remember broad categories of questions I get, but everything else is just a blur. Thats senility for ya.
But _this_ dude, or possibly dudette, asked me a really good question. One that made an impression. And by "really good", I mean it was one of the flat-out wrong-est business questions you could possibly ask. It was like asking: "How can I get the most out of my overseas child laborer pool?"
Theres no right answer to a question thats just _wrong_.
But his question, well, people ask it a lot. In fact there are whole books that give answers to this inherently-wrong question.
His question was: ONLY BUILD STUFF FOR YOURSELF
Thats the Golden Rule of Building Stuff. If youre planning to build something for someone else, _let someone else build it._
BUILDING STUFF FOR YOURSELF
You can look at _any_ phenomenally successful company, and its pretty obvious that their success was founded on building on something _they_ personally wanted. The extent that any company begins to deviate from this course is the extent to which their ship starts taking on water.
And the key leading indicator that theyre getting ready to head off course? You guessed it: its when they start talking about gathering business requirements.
Because, dude, face it: if its something _you_ want, then you already know what the requirements are. You dont need to "gather" them. You think about it all the time. You can list the requirements from memory. And usually its pretty simple.
For instance, a few years ago I announced to some friends: "I sure wish someone would make a product you can spray on dog poop to make it, you know, just dissolve away." My friends laughed loudly and informed me that this was (apparently) the premise of some Adam Sandler movie I hadnt seen.
Well, OK, sure, but... I mean, they kinda missed the point. I still want the product. Its requirements list is pretty simple. Heres the business requirements list:
a) It should dissolve dog poop.
Gosh. Can that really be the entire list? You bet! Sure, there are lots of implicit requirements: it shouldnt cost a fortune, it should be environmentally friendly, it shouldnt kill kittens, etc. But those kinds of requirements are true for _all_ products and services.
If I knew how to make this product, then Adam Sandler movie or no, Id probably try to make it. The target market is larger than just pet owners; anyone living near a park would probably own a bottle or two. I would use the stuff like its going out of style. Id attach it to my shoes so that every time I took a step it would spray the area in front of me, like a walking garden hose.
Building a product for yourself is intrinsically easier, since you dont have to gather requirements; you already know what you want. And you also know, for any given compromise anyone suggests, _whether it will ruin the product_. If someone says, "I have a product that dissolves dog poop, but it takes 18 hours", then youll know youve entered into "prolly not worth it" territory. You dont have to go ask the focus group. You just know.
THE MISTAKE OF IMAGINATION
Despite its obvious advantages, following the rule of building stuff for yourself is actually really hard to carry out in practice. Why? Oddly enough, its ego.
For one thing, people like to think theyre unique and special, and that their tastes arent necessarily widely shared by others. This is what drives fashion: the need to differentiate yourself from "the crowd", by identifying with some smaller, cooler crowd.
The reality is that for any given dimension of your personality, there are oodles of people just like you. If you want something, other people want it too. You define a market: a bunch of them, in fact.
You just have to be smart about _which_ of your needs you want to fulfill, since if its building yachts, well, its not exactly going to be mass-market, unless you find a way to build a mass-market yacht. Which would be pretty cool, incidentally. Ill buy one if you make it.
Its also really easy to fool yourself into thinking that this is a product or service you _would_ use, because, hey, you have a great imagination. When you lose your car keys, you can picture them in all sorts of places: the kitchen drawer, the coffee table, on top of the fridge, and when you picture them there, its just as vivid as a memory. So you wind up looking for them everywhere! Your powerful imagination is pretty much your worst enemy when it comes to deciding whether youd like something enough to use it yourself.
We all made the Mistake of Imagination on the OmniGo projects. "I could see myself using this product," wed tell ourselves, "if, that is, I were the kind of person who used this product, which I could sort of envision." Youll tell yourself almost anything to justify the work youre doing. Giving up in mid-project is a big loss of face for an individual, harder for whole teams, virtually impossible for most companies. The OmniGo had four companies involved, making it the hardest possible project to back out of, even though by the halfway point virtually everyone involved knew the product would kind of suck.
What we _really_ wanted, while we were building the OmniGo, was summed up by our brilliant product manager Jeff Peterson. We were having beers one day, and he said, "You know, this thing is just getting way too complicated! It doesnt need a 12C calculator emulator! It doesnt need a spreadsheet! It doesnt need a database application! I mean, come on! All it needs is a notepad, a simple calendar, an alarm clock, and maybe a pocket calculator, and it should fit in your front shirt pocket, and it should be a _phone_."
It was 1994 and he was describing the iPhone. And you know what? He was right! That _was_ what we wanted. But HP was driving the spec, and they werent building it for themselves. They were building the product _specifically_ for this imaginary group of high-power on-the-go consumer accountants who use 12C calculators and want a whole desktop suite crammed onto their 1994-era mobile device. And thats just who bought the thing: pretty much nobody. (They sold a few thousand units, which in mass-market terms is "pretty much nobody.")
TRIMMING THE REQUIREMENTS
Who was it who said that youre done writing not when theres nothing left to add, but when theres nothing left to take away? Was it St. Exupéry? I promised not to do any research for this rant, but I think it might have been him.
Ideally the product youre building for yourself should be _simple to describe_, so that other people can quickly evaluate whether they, too, want this thing. Its often called the "elevator pitch", because you should be able to describe the product in the time between when the cable snaps and the elevator hits the ground. "Dissolves dog poooooop!!! " It used to just be the time for an elevator ride, but those investors keep raising the bar.
You can almost always make a product better by trimming the requirements list. Were talking brutal triage: throwing out stuff thats really painful to lose, such as the ability to change the battery.
If youre lucky, you should be building a product that so excites everyone involved that _everyone_ has an opinion, and you wind up spending most of your time in triage.
When youre _trimming_ the business requirements, then youre exhibiting healthy project behavior. This contrasts directly with _gathering_ requirements, which has both the connotation that youre clueless about the product _and_ the connotation that youre inflating the requirements list in direct conflict with schedule, usability and fashion. Trimming: good. Gathering: bad.
Trimming the requirements list is a leading indicator that youre a smart company whos about to launch something major. An ideal requirements list looks something like this:
a) It should dissolve dog poop.
As a great real-life example, consider the the Flip camcorder, which kinda came out of nowhere and "stole" 13% of the camcorder market (although Id bet good money that it actually created new market share). Does it dissolve dog poop? Well, no, but its still pretty cool:
1) it costs $150 or less. (A lot less, actually.)
2) it has no cables or wires. Just one flip-up USB connector.
3) it has one big red button: RECORD, plus a teeny one for playback.
4) it doesnt take cartridges or cassettes or discs or cards or anything
5) it doesnt have any controls or settings or _anything_
6) it stores one hour of video and has roughly one hour of battery life
7) its about the size of a cell phone
8) it records videos that work well with YouTube
9) it comes in pretty colors
I mean, DAMN, those guys knew what they were doing. We always used to joke about a product so simple that it only had one button, which we pressed for you before it left the factory. Thats how simple a product needs to be in order to make the mass market. One button, pretty colors. They _nailed_ it. Talk about a missed startup opportunity. (Flip guys: if your equity plan is still reasonable and you want someone to make your desktop software not suck, and yes, it really sucks, please give me a call.)
You dont need an _original_ idea to be successful. You really dont. You just need to make something that people want. Even if someone else appears to be making something popular, its usually possible to improve on the idea and grab market share. And its painfully counterintuitive at times, but the best improvements often come from simplifying.
The easiest way to build a product that kicks ass is to start with someone elses great idea (camcorders, for instance), and _take stuff away_.
In any event, originality is overrated. Coming up with something completely original isnt just hard to do: its also hard to _sell_, because investors (and possibly customers) will need to be educated on what this new thing is and why people would want it. And when it comes to buying stuff, nobody likes to be educated. If the product isnt immediately obvious, investors and customers will pass it up.
Its easy to come up with new product ideas if you start with the understanding that everything sucks. There are no completely solved problems. Just because someone appears to be dominating a market with an "ideal" offering doesnt mean you cant take market share from them by building a better one. _Everything_ can stand improvement. Just think about what youd change if you were doing it for yourself, and everything should start falling into place.
If nothing else, building things for yourself is more fun, so youre successful regardless of what happens. But it also has great product-survival characteristics, because people cant bluff you into making something lame.
SOMETIMES YOU JUST CANT WIN
By way of dont-sue-me disclaimer, I should point out that building something for yourself doesnt _guarantee_ success. Even if you build a product that most of your target market really really wants, and you hit the right price point and release date and everything, your product can still fail catastrophically.
The example that leaps to mind is that company in the 1990s (can anyone remind me of the name? Ive forgotten) that built a mountain-bike seat extender. I ride mountain bikes, yes, on actual mountains, so this product made _immediate_ sense to me. I _really_ wanted one. Sounds like a winner, right?
The basic physics problem this company was solving is that a lower seat position gives you better balance, and a higher seat position gives you more power. Its a trade-off. You generally want more power when youre grinding uphill, and you want more balance when youre speeding downhill. But during a race you would have to give up precious seconds to adjust your seat between every uphill and downill transition; youd get your butt kicked. Even as a casual rider, adjusting your seat height all the time is annoying enough that most people just dont do it, resulting in some sacrifice of balance, power, or both.
So this brilliant, innovative company came out with a well-made product that lets you adjust your seat height "in flight", as it were, without slowing down, and without adding much (if any) weight to the bike. I dont remember exactly how it worked, but it was a reasonable implementation.
Interestingly, _it didnt matter how it worked_. It could have been actual magic: a product that read your mind and raised or lowered your seat by exactly the right amount, at exactly the right speed (you dont want it to rabbit-punch you in the nuts, for example -- remind me sometime to tell you about how I found that out), as frequently or infrequently as necessary to strike the _perfect_ trade-off of balance and power for any slope, at any time.
And it _still_ would have failed, even if 90% of the mountain biker population, like me, secretly coveted the product. It could have cost fifty cents, been available everywhere, and been installable by a four-year-old with one hand in under two seconds. It could have come pre-installed on all bike models, via a brilliant channel deal with all the main manufacturers (or retailers), and bikers would have ripped the thing off the bike faster than their kickstand (which is usually the first thing to go.)
So, uh, what happened, exactly? Wasnt I just ranting that building a product _for yourself_, one that you _know_ is the Right Thing for some well-defined audience consisting of people just like you in some dimension -- wasnt I ranting that this would always work?
Well, no. Its just the best way to pick projects. But they can still fail for surprising reasons.
In this case, the fundamental marketing force that the company failed to take into account was _fashion_. People dont often think of mountain biking (or programming, for that matter) as a fashion industry, but failing to understand the role of fashion is a recipe for random disasters.
What happened was this: while they were hyping the product -- by demoing it at trade shows, letting bike magazines check it out, and generally working their way towards getting retailer shelf space -- the pro bikers took one look at the thing and said: "Hey, looks pretty cool. Maybe Ill get one for my girlfriend. Or my fugging _grandma_. How much is it?"
At which point, everyone (me included) who had been ranting about standing in line to buy one when they came out, we, ah, we were very quick to point out that we were also excited to buy them _for our grandmothers_, whom we loved just as much as the pros love their grandmothers, thank you very much. In fact, our grandmothers are too macho to use this thing. Maybe well put one on our kids stroller! So there!
And thats how a great product, one that probably would have changed biking nearly as fundamentally as the derailleur, was doomed from birth because the trendsetters of Mountain Biking Fashion pronounced it a Product for Sissies, and that was that.
Heck, even derailleurs are falling out of fashion in some circles. Go figure. Someone took the time-tested "solved problem" of bicycles, removed something, and wound up creating a new market.
Fashion is generally hard to predict, but it usually means "sacrificing comfort or convenience for the sake of style". Take another look at the iPod: it has almost no features. It doesnt have an "off" button. Heck, you cant even change the battery. Not exactly convenient in many ways. But it sure has style!
Fashions not the only way your otherwise perfect product can fail, but its definitely one to keep an eye on.
LESS IS MORE... MORE OR LESS
OK, fine, I havent exactly been following my own advice about minimalism. But I _have_ been writing my blog the way _I_ like it, havent I? So even if nobody reads it, Im still having fun. If youre going to create something that has a nonzero chance of failure -- and believe you me, its nonzero -- then you might as well have fun doing it, right?
Anyway, there you have it: the slightly expanded version of the email I sent that CEO guy. Gathering business requirements is hokum. Hooey. Horseshit. Call it what you want, but its a sign of organizational (or individual) cluelessness. If you dont already know _exactly_ what to build, then youre in the wrong business. At the very least, you should hire someone who does know. Dont gather business requirements: hire domain experts.
If you cant think of _anything_ in your companys "space" that you personally would use, then you should think seriously about (a) changing your companys direction, or (b) finding another company. This is true no matter what level youre at. You should be working on something you love, or failing that, at least working on something that you know really well.
"But... but..." I hear you saying. I hear you! You sort of get what Im saying, but you have all these reservations and objections and questions and stuff.
Well, thats what the comments section is for. Im sure you can think of some _other_ explanation for why Warren Buffett is the richest person in the world. Lets hear it!
"Hello, blah blah, Im the CEO or C-something-O or whatever of a company that blah blah BLAH, and I read your blog about Agile from almost 2 years ago, which kinda resonated with me in a scary way, inasmuch as I realized perhaps belatedly that I was being duped, BUT I was sort of wondering: HOW DO YOU GO ABOUT GATHERING BUSINESS REQUIREMENTS, SO YOU KNOW WHAT TO DELIVER TO THE CUSTOMER? Signed, blah blah blaaaaaaaaaah."_(Note: not a verbatim transcript. But close enough.)_ And my answer was: BUSINESS REQUIREMENTS ARE BULLSHIT! Well... actually it was: "_gathering_ business requirements is bullshit", plus a bunch of accompanying explanation. But the shorter version sure is catchy, isnt it? The rest of this little diatribe expands a bit on my reply to this guy. (I actually sent him an email with much of this material in it. When I _do_ reply to strangers, I still try do a thorough job of it.) Before we dig into the meat and potatoes of todays meal, lets talk about my credentials, and how it is that I am qualified to offer an opinion on this topic. MY CREDENTIALS If youre a business person stumbling on my blog for the first time, welcome! Allow me to introduce myself: _I dont matter._ Not one teeny bit. You shouldnt care about my credentials! All that matters is the message, which is hopefully so self-evident that if you hadnt already realized it, youll be kicking yourself. Its OK, though. Kick away! In fact, give yourself an extra kick for me. Kicking yourself is a great way to remember a new lesson down in your bones. When you get actual bones involved, the painful throbbing for the next few days provides just enough of a reminder that youll never forget again. In any case, I make no claim to any credentials _whatsoever_, and this advice is all straight from my butt. Do what you like with it. The advice, that is. With my credentials out of the way, lets see about this guys question. By the way, Im specifically addressing this rant towards you _only_ if you (or your team, or company) are building your own product or service offering, or at least defining it. If youre asking consultants to define the product for you, well, youre on your own. Good luck. And if youre a consultant, well, you dont get much choice about what to build, so youre also (mostly) on your own. Although Id advise you to try to find work that fits the strategy I outline here, as youll have more fun with it and do a better job overall. GATHERING REQUIREMENTS 101 Its the first thing everyone wants to do! Its the first thing they teach you in Project Management Kindergarten: the _very first thing you should do_ on a new project is look both ways before crossing the street to your building. Assuming you parked across the street. And the next thing is: start gathering business requirements! Ah, the good old days of Project Management 101. Life seemed so easy back then. The requirements-gathering process they teach you typically involves finding some "potential customers" for your product, and interviewing them in a nonscientific way to try to figure out what they want out of your proposed product. Or service. Or whatever. It doesnt really matter. Potential customers can be hard to come by, especially since youre building something that nobody will ever, EVER want. Well, thats getting ahead of myself. Lets just agree that when youre setting out to Gather Business Requirements, your potential customers usually arent already sitting in the room with you. Sometimes its a lot of effort to go find real customers and bring them in and get them to agree to be grilled for hours. Hence, requirements-gathering sometimes takes the form of "role play", where you get some poor saps in Accounts Receivable to pretend theyre Potential Customers for your product, and you interview them instead. You might be laughing about those poor Role Playing schmucks, but in reality it doesnt much matter _who_ you interview, because by the time you get to this phase in the project, youve already screwed it up beyond all hope of repair. That process of looking around for customers? Its a plot device that novelists like to call _foreshadowing_. If you dont have any readily available customers now, then they sure as hell wont be readily available when your product launches. Allow me to illustrate with a Case Study. AN ILLUSTRATIVE CASE STUDY See? Credentials or no, I sure can talk the lingo, eh? I also like my incisive use of the popular business term "schmuck" in the previous section. Its a term that can apply to people in all phases of project catastrophes, not just Requirements Gathering. For our Case Study, I will not do any research and the product will be entirely fictional. This is the approach used by most real companies. Lets say that our Ideas Department has decided that we should get into Personal Nose Hair Grooming devices, because theres clearly a huge unmet need for nose hair grooming, as evidenced by Karl, our accountant, who has a thatch of nose hair thats _almost_ long and thick enough to be a mustache, if only he would groom the thing a little. And weve seen lots of people like Karl. Clearly a nose-hair grooming kit would be the ideal addition to any mans personal grooming lineup, which typically consists of spitting into the sink and thinking he should get the mirror fixed someday. So we need to gather some Business Requirements. Unfortunately nobody wants to talk directly to Karl, because, well, nose hair can sometimes be a mildly sensitive issue in some cultures. Plus nobody wants to make eye contact with him. So instead we hire some people to go out and do focus-group studies on the subset of people who are comfortable talking publicly about their nose-hair grooming problem, notwithstanding the fact that everyone in this tiny category is obviously too crazy to trust with important business questions. The focus groups find a few nut-jobs who say: "Yeah, Id _love_ a nose-hair grooming appliance! Plucking your nose hair is painful, and trimming it involves jamming a small whirly Osterizer thing all the way up to your brain. Maybe if I just combed it into a mustache, nobody would notice!" And behold, we have the makings of some Business Requirements. The product is a complete failure, of course. The project postmortem reveals that the user studies and focus groups failed to realize that only certain men, namely "men with significant others", ever even _notice_ their nose hair, even when said hair becomes long enough to interfere with tying their shoelaces. The significant other must forcibly remove the nose hairs with garden shears at least twice before the man gets the hint and starts attending to the problem himself. Not to mention the fact that a nose-hair mustache would be even more obvious and comical than a comb-over. Well, almost. In the end, it doesnt matter how great a Nose Hair Groomer we build, because we failed at the business-requirements gathering phase: nobody actually wants this product. The people who _might_ want it dont know they have a nose-hair issue, and the ones who do know just trim it. The thing that might surprise you is that this fictitious (and yet eerily familiar) Case Study isnt just an illustration of how gathering business requirements can go afoul. Were not talking about a _failure mode_ for Gathering Business Requirements (GBR). Were talking about something more fundamental: the GBR phase of a project is a leading indicator that the project will fail. Put another way: GBR is a virtual guarantee that a project is building the wrong thing, so no amount of brilliant execution can save it. FROM BUSINESS REQUIREMENTS TO BAD IDEA I learned a lot about the Fine Art of Building Shit that Nobody Wants back at Geoworks, when in 1993-1994 I was the Geoworks-side Engineering Project Lead for the HP OmniGo 100 and 110 palmtop organizer products. Both of them launched successfully, and nobody wanted either of them. People spend a lot of time looking at why startups fail, and why projects fail. Requirements gathering is a different beast, though: its a _product_ failure. It happens during the project lifecycle, usually pretty early on, but its the first step towards product failure, even if the _project_ is a complete success. Self-professed experts will tell you that requirements gathering is the most critical part of the project, because if you get it wrong, then all the rest of your work goes towards building the wrong thing. This is sooooort of true, in a skewed way, but its not the complete picture. The problem with this view is that requirements gathering basically never works. How many times have you seen a focus group gather requirements from customers, then the product team builds the product, and you show it to your customers and they sing: "Joy! This is exactly what we wanted! You understood me perfectly! Ill buy 500 of them immediately!" And the sun shines and the grass greens and birds chirp and end-credit music plays. That _never_ happens. What really happens is this: the focus group asks a bunch of questions; the customers have no frigging clue what they want, and they say contradictory things and change the subject all the time, and the focus group argues a lot about what the customers really meant. Then the product team says "we cant build this, not on our budget", and a negotiation process happens during which the product mutates in various unpleasant ways. Then, assuming the project doesnt fail, they show a demo to the original customers, who say: "This is utterly lame. Yuck!" Heck, even if you build exactly what the customer asked for, theyll say: "uh, yeah, I asked for that, but now that I see it, I clearly wanted something else." So the experts give you all sorts of ways to try to get at the Right Thing, the thing a real customer would actually buy, without actually building it. You do mockups and prototypes and all sorts of bluffery that the potential customer _cant actually use_, and they have to imagine whether theyd like it. Its easy enough to measure _usability_ this way, but virtually impossible to measure _quality_, since there are all sorts of intangibles that cant be expressed in the prototype. I mean, you can try -- we sure tried on the OmniGo products -- but in this phase nobody "imagines" that the thing will weigh too much, or be too slow, or will go through batteries like machine-gun rounds, or that its boot time will be 2 minutes, or any number of other things that ultimately affect its sales. And even if you do a world-class job of prototyping and getting at the "real" desired feature set, it still goes through a series of iterations with the engineers and product team, who arent the actual customers and have little interest in building what the customer _really_ wants (because theyre not being measured or evaluated on that -- theyre evaluated on delivering what everyone ultimately _agrees_ to deliver). During each iteration they push back on things the customers asked for. As the proposal degrades, the customers like it less and less, but they want to be helpful, so eventually they say, "Yeah, I guess I could use that." (Which means: I wouldnt take one of these even if they were giving it away.) Dont get me wrong: Im not saying that usability teams cant do a good job. Its just that when the project implementation team and the target customer arent exactly the same group of people, then there are inevitably _negotiations_ and _compromises_ that water an idea down about two levels of quality: great becomes mediocre, and ideas that start as "pretty good" come out "just plain bad." So Im tellin ya: gathering requirements is the Wrong Way To Do It. At _best_, it results in mediocre offerings. To be wildly successful you need a completely different approach. THE INVESTING ANALOGY Warren Buffett and Peter Lynch, both famous and successful investors, say pretty much the same thing about investing. Peter Lynchs mantra sums it up: "invest in what you know." If you actually take the time to read Lynchs books (which I have), youll see that this pithy mantra is a placeholder for something a bit more subtle: you should invest in what you know _and like_. You should invest in companies that make products or services that you are _personally_ excited about buying or using _right now_. When you invest with this strategy, youre taking advantage of your local knowledge, which tends to be more accurate than complicated quantitative packets put together by analysts. And your local knowledge is _definitely_ more accurate than the reports produced by the companies, who want to paint themselves in the nicest light. Warren took a lot of heat in the 1990s for not investing in the tech sector. But hey, he didnt feel comfortable with tech, so he didnt invest in it. One way to look at this is: "ha ha, what a dinosaur, he sure missed out, and now hes, uh, only the richest person in the world by a small margin." But another, more accurate way to look at it is this: _hes the richest person in the world, you asshole._ When he gives you investment advice, TAKE IT! And Warrens advice is: dont invest in stuff you dont understand! _Even if it seems like a sure thing._ Thats the hard part. Sometimes it _looks_ like a surefire winner for some large group of people that doesnt actually include you personally at this particular moment. But its a really large group! Lets say, for instance, that you hear that Subway (the sandwich franchise) is going to do an IPO. Theyve been privately held all these years and now theyre going public. Should you invest? Well, lets see... the decision now isnt quite as cut-and-dried as it was in their rapid-expansion phase, so, um, let me see, with current economic conditions, I expect their sales to, uh... let me see if I can find their earnings statement and maybe some analyst reports... NO! No, No, NO!!! Bad investor! Thats the kind of thinking that loses your money. The _only_ question you should be asking yourself is: how many Subway sandwiches have I eaten in the past six months? If the number is nontrivial -- say, at least six of them -- and the rate of sandwiches per month that you eat is either constant or increasing, _then_ you can think about looking at their financials. If you dont eat their sandwiches, then youd better have a LOT of friends who eat them every day, or youre breaking the cardinal rule of Buffett and Lynch. The investing analogy is an important one, because if youre a company or team planning to build something, then youre planning an investment. Its not exactly the same as buying stock, but it amounts to the same thing: youre betting your time, resources and money -- all of which boil down to money (or equivalently time, depending on which one is in shorter supply.) So when translated into project selection, Buffetts and Lynchs advice becomes: _only build what you know._ The longer, more accurate of the version of the investing rule -- only invest in what you know _and_ are excited about using yourself right now -- has a simpler formulation for products and businesses. That formulation goes like this:
_Disclaimer: I do not speak for Google! These are my own views and opinions, and are not endorsed in any way by my employer, nor anyone else, for that matter._
Everyone knows and quotes Joels old chestnut, "SMART, and GETS THINGS DONE." It was a blog, then a book, and now its an aphorism.
People quote Joels Proverb all the time because it gives us all such a nice snuggly feeling. Why? Because to be in this industry at all, even a bottom-feeder, you _have_ to be smart. You were probably the top kid in your elementary school class. People probably picked on you. You were the geek back when geeks werent popular. But now "smart" is fashionable.
Thats right! All those loser kneebiter jocks in high school who played varsity and got all the girls and sported their big, winning, shark-white smiles as they barely jee-parlor-fran-saysed their way through the classes you coasted through: where are they now? A bunch of sorry-ass bank managers and failed insurance salesmen and suit-wearing stiffs at big law firms working for billion-dollar conglomerates where their daddies got them VP jobs where they just have to show up and putt against the other VPs on little astroturf greens in the hallways!
Thats right, los... er, well, some of them are doing OK, I guess. "But theyre not as rich as Bill Gates!" Thats the other big tautology-cum-aphorism we geeks like to pull out when were feeling insecure. Notwithstanding that Bill had a rich daddy and made his money through exactly the kind of corporate shenanigans that Big Meat is using on us today, with those selfsame jocks. We like to think the more important point is that Bill, Jobs, Bezos and the other tech billionaires (all fierce, business-savvy people) were geeks, and are thus evidence of the revenge of the nerds. And hey, tech did make a lot of money in the 80s and 90s. So "smart" is sort of in fashion now.
Unfortunately, "smart" is a generic enough concept that pretty much everyone in the world thinks theyre smart. This is due to the Nobel-prize-winning Dunning-Kruger Effect, which says, in effect, that people dont know when theyre not smart. _(Note: people have pointed out that it was an Ig-Nobel. Me == Dumbass. Fortunately, "me == dumbass" was more or less the point of the article, so Ill let it stand.)_
So looking for SMART is a bit problematic, since we arent smart enough to distinguish it from B.S. The best we can do is find people who we _think_ are smart because theyre a bit like us.
Er, what about GETS STUFF DONE, then? Well, the other qualification you need to be in this industry is that you have to have produced _something_. Anything at all. As soon as you write your first working program, youve gotten something done. And at that point you probably think of yourself as being a pretty hot market commodity, again because of the Dunning-Kruger Effect.
So, like, what kind of people is this SMART, AND GETS THINGS DONE adage actually hiring?
Ive said it before: I thought I was a top-5% (or so) programmer for the first fifteen years I was coding. (1987-2002). Every time I learned something new, I thought "Gosh, I was really dumb for not knowing that, but _now_ Im definitely a superstar." It took me fifteen frigging years before I realized that there might in fact still be "one or two things" left to learn, at which point I started looking outward and finally discovered how absolutely bambi-esquely thumperly incompetently clueless I really am. Fifteen years!
But hell, lets be honest here: I still think Im smart. We all do. Sure, Ive managed to figure out, at least partly, just how un-smart I am, but Ive got the whole "Im smart" thing so hardwired from when I was a kid surrounded by "dolts" who couldnt memorize things as fast as I could, or predict the teachers punch line way in advance, or whatever questionable heuristic I was using for measuring my own smartness, that its hard not to think that way. And of course the "dolts" were way better than me at just about everything else. And they think theyre pretty smart too! Definitely above average, anyway.
SQUAAAAAWK
So we all think were smart for different reasons. Mine was memorization. Smart, eh? In reality I was just a giant, uncomprehending parrot. I got my first big nasty surprise when I was in the Navy Nuclear Power School program in Orlando, Florida, and I was setting historical records for the highest scores on their exams. The courses and exams had been carefully designed over some 30 years to maximize and then test "literal retention" of the material. They gave you all the material in outline form, and made you write it in your notebook, and your test answers were graded on edit-distance from the original notes. (Im not making this up or exaggerating in the slightest.) They had set up the ultimate parrot game, and I happily accepted. I memorized the entire notebooks word-for-word, and aced their tests.
They treated me like some sort of movie star -- that is, until the Radar final lab exam in electronics school, in which we had to troubleshoot an actual working (well, technically, not-working) radar system. I failed spectacularly: Id arguably set another historical record, because I had _no idea_ what to do. I just stood there hemming and hawing and pooing myself for three hours. I hadnt understood a single thing Id memorized. Hey man, I was just playing their game! But I lost. I mean, I still made it through just fine, but I lost the celebrity privileges in a big way.
Having a good memory is a serious impediment to understanding. It lets you cheat your way through life. Ive never learned to read sheet music to anywhere near the level I can play (for both guitar and piano.) I have large-ish repertoires and, at least for guitar, good technique from lots of lessons, but since I could memorize the sheet music in one sitting, I never learned how to read it faster than about a measure a minute. (Its not a photographic memory - I have to work a little to commit it to memory. But it was a lot less work than learning to read the music.) And as a result, my repertoire is only a thousandth what it could be if I knew how to read.
My memory (and, you know, overall laziness) has made me musically illiterate.
But when you combine the Dunning-Kruger effect (which affects me just as much as it does you) with having one or two things Ive been good at in the past, its all too easy to fall into the trap of thinking of myself as "smart", even if I know better now. All you have to do, to be "smart", is have a little competency at something, anything at all, just enough to be dangerous, and then the Dunning-Kruger Effect makes you think youre Gods gift to that field, discipline, or what have you.
This is why everyone loves SMART, AND GETS THINGS DONE, which Joel always writes in boldface. We love it because right from the start it has the yummy baked-in assumption that _you_ are smart, and that _you_ get things done. And it also tacitly assumes that you know how to identify other people with the same qualities!
But... you dont.
It sucks, but the Dunning-Kruger Effect is frighteningly universal. Its a devils pitchfork two horrible, boldfaced PRONGS. First:
INCOMPETENT PEOPLE GROSSLY OVERESTIMATE THEIR OWN COMPETENCE
We already talked about that, right? Youre a good programmer! Heck, youre a great programmer! Youre "smart", so anything you dont know you can go look up if you need it! Right?
Welcome to incompetence.
The second prong is a bit longer, and has a barbed poison tip:
INCOMPETENT PEOPLE FAIL TO RECOGNIZE GENUINE SKILL IN OTHERS
Both prongs are intrinsically funny when youre watching them in action in _someone else_, and theyre incomprehensible when anyone tries to poke you with them. Not necessarily offensive, mind you: you _might_ get offended if someone tries to imply that youre not as competent as you feel you are, but its more likely (per the D-K effect) that youll simply not believe them. Not even close. Youre smart! Theyre just wrong! Gosh!
The second prong, that of the ability to recognize true competence, has major ramifications when we conduct interviews. Thats what Joel was writing about in SMART AND GETS THINGS DONE, you know: conducting technical interviews.
How do you hire someone whos smarter than you? How do you _tell_ if someones smarter than you?
This is a problem Ive thought about, over nearly twenty years of interviewing, and it appears that the answer is: you cant. You just have to get lucky.
Its easy for a candidate to _sound_ smart. Anyone can use big jargon-y words and talk the talk like nobodys business, but Im living, breathing proof that articulacy doesnt connote any other form of intelligence. Heck, the Markov-chain synopses of my blogs that people post in quasi-jest tend to look like I wrote them.
All too often I find myself on interview loops where the candidate knows a seemingly astounding amount about coding, computer science, software engineering, and general industry topics, but we find out at the last minute that they cant code Hello, World in any language.
This is, of course, one of the failure-patterns that Joels GET THINGS DONE clause is designed to catch. But interviews are conducted under pretty artificial conditions, and as a result they wind up being most effective at hiring people who are good at interviewing. This is a special breed of parrot, in a way. Interviewing isnt a particularly good predictor of performance, any more than your rank in a coding competition is a predictor of real-world performance. In fact, somewhat depressingly, theres almost no correlation whatsoever.
IF INTERVIEWS SUCK, THEN WHY DO WE STILL DO THEM THIS WAY?
Lots of reasons. One is just history: everyone else does it that way. Companies tend to hire pretty similar HR departments, and HR tends to guide companies towards doing it the way everyone else does it. Same goes for technical management, which is all too often HR-driven as the company grows.
Another is that interviewing is already a fairly time-intrusive function for the interviewers, and it tends to be miserable for the interviewees. Trying to make the process somehow more rigorous or accurate just exacerbates these side effects.
Another is the "rite of passage" phenomenon, wherein engineers feel that if they had to go through the gauntlet, then everyone else should, too.
So for the most part, everyone does the same non-working variety of interviews, and hopes for the best.
As far as identifying good people goes, the best solution Ive ever seen was at Geoworks, where you were required to do a six-month internship before you could get hired full-time. This seems to be the norm in non-tech departments at most tech companies. They often substitute "contractor" for "intern", but it works out roughly the same. Geoworks is the only company Ive seen stick to their guns and make it mandatory for engineers.
However, Im convinced that it only worked because Geoworks seeded the engineering staff with great people. The founding engineers set up a truly beautiful software engineering environment, with lots of focus on tools, mentoring, continuing education, "anarchy projects" to let off steam and encourage innovation, and a host of other goodnesses.
Ive been a contractor at companies that had no good engineers at all, literally none whatsoever. A mandatory six-month internship at such companies would only serve to _lower_ their average bar, since anybody competent would leave after the six months was up. This doesnt contradict the D/K Effect. Its easy to spot lackluster, soulless engineering organizations, and doing so doesnt imply that youre especially smart.
THE "EXTENDED INTERVIEW"
Anyway, Ive often wondered where the Geoworks founders found such great engineers. The short answer is: "Berkeley", but Im really looking for something deeper than that; namely: how did they _know_?
Along similar lines, Ive long felt that Amazons success was due in no small part to Bezos having seeded his tech staff with great engineers. World-class great. I dont know where or how he found them, since, again, how do you hire someone whos smarter than you? Hes a brilliant guy, but his original choices (ex-Lucid folks, by and large) seem a stroke of blind luck thats hard to attribute to mere genius. Ill probably never know how it happened. Wish I did!
They werent too big on software engineering, though, or more likely they all felt that time-to-market trumped everything else (and were correct, at least in their case), so Amazon is successful but lacks a high-quality software engineering culture. Its gotten much better over the years, of course, but its a far cry from Geoworks. Its largely up to individual teams at Amazon to decide how much engineering to sprinkle into their coding. Theres no central group (or distributed peer group) who can tell any team how to build their software. This is effectively mandated from the top, in fact.
But time-to-market is a pretty powerful business force. Maybe thats the missing link? Lucid was founded by Dick Gabriel, the "worse is better" guy, and Amazon took the "worse is better" idea (internally) to untold new extremes.
Dunno! But it sure worked for them.
Google has a process similar to the Geoworks 6-month internship idea. Geoworkss internship was a form of "extended interview", since it was obvious even in the 1980s that the classic interview format isnt a very good predictor of performance. At Google, you dont have to do an internship. However, unlike at most other companies, youre not "slotted" into a real position on the tech ladder until youve been on the job at least six months. During that time you need to prove that you can function at the level you were hired for, and if its wrong in either direction your level is adjusted at slotting time.
The "extended interview" (in any form) is the _only_ solution Ive ever seen to the horrible dilemma, HOW DO YOU HIRE SOMEONE SMARTER THAN YOU? Or even the simpler problem, How do you identify someone whos _actually_ SMART, AND GETS THINGS DONE? Interviews alone just dont cut it.
Let me say it more directly, for those of you who havent taken this personally yet: you cant do what Joel is asking you to do. Youre not qualified. The SMART AND GETS THINGS DONE approach to interviewing will only get you copies of yourself, and the work of Dunning and Kruger implies that if you hire someone better than you are, then its entirely accidental.
Dont get me wrong: you should still try! Dont throw the bath and baby away. SMART AND GETS THINGS DONE is a good weeder function to filter out some of the common "epic fail" types.
But realize that this approach has a cost: it will also filter out some people who are just as good as you, if not better, or even _way_ better, along dimensions that are entirely invisible to you due to Dunning-Kruger forces mostly beyond your control.
So theres this related interviewing/hiring heuristic that I think may better approximate the kinds of people you really want to hire: DONE, AND GETS THINGS SMART.
Ill take Joels cue and write it in BOLD so you know its important. Its not CONDESCENDING or anything. Really. Lets all recite it together, to make it catchy and stuff. Maybe we should have a little ditty for it, so it sticks in our heads annoyingly, forever. I think the Happy Birthday song will do nicely:
HAP-PY BIRTH-DAY TO YOOOOOOOU,
DONE-AND GEH-ETS THINGS SMAAAART
HMMM HMM HMMMMM HMMMM HMMMM WHOEEEEEEVER
DONE AND GEH-EEEEEETS THIIIIIINGS SMAAAAAAAART *clap* *clap*
There! Youll remember it every time anyone has a birthday.
DONE, AND GETS THINGS SMART
All gentle faux-condescension aside (or as the classroom jocks would have read, "fox condesomething"), Joels SMART, AND GETS THINGS DONE heuristic seems really obvious to everyone. It has this magical "were all smart in this together" appeal. But sadly, for the reasons Ive outlined, almost everyone interprets it to mean "CARBON COPY, OF MYSELF". Great guidance gone astray!
The DONE, AND GETS THINGS SMART approach is also a way of finding great people, but it recognizes that the Dunning-Kruger Effect requires some countermeasures. Its modeled on the early successes Ive witnessed at Geoworks, Amazon, and Google, all of whom had one thing in common: they hired BRILLIANT SEED ENGINEERS. This boldface is really addictive when you get started on it!
They all managed to continue hiring great people afterwards, but the seed engineers were the most important. Im hoping that this is intuitively obvious in much the same way that wanting smart people who get things done is intuitively obvious.
I think you really, _really_ want great seed engineers, and that this is a different class of engineer from the "pretty darn good" engineer we typically hire using Joels oft-misinterpreted advice.
Seed engineers. Its key. You can apply the "I want ideal seed engineers" rule recursively to an organization, a department, a project, a team, and even your officemates. Were not just talking about startups.
Let me ask you a brutally honest question: since you began interviewing, how many of the engineers youve voted thumbs-up on (i.e. "hire!"), are engineers youd personally hire to work with you in your first startup company? Lets say this is a hypothetical company youre going to found someday when you have just a little more financial freedom and a great idea.
I posit that most of you, willing to admit it or not, have a lower bar for your current company than you would for your own personal startup company.
The people youd want to be in your startup are _not_ of the SMART AND GETS THINGS DONE variety.
For your startup (or, applying the recursion, for your new project at your current company), you dont want someone whos "smart". Youre not looking for "eager to learn", "picks things up quickly", "proven track record of ramping up fast".
No! Screw that. You want someone whos superhumanly godlike. Someone who can teach _you_ a bunch of stuff. Someone you admire and wish you could emulate, not someone who you think will admire and emulate you.
You want someone who, when you give them a project to research, will come in on Monday and say: "Im DONE, and by the way I improved the existing infrastructure while I was at it."
Someone who always seems to be finishing stuff so fast it makes your head spin. Thats what my DONE clause means. It means theyre frigging done all the time.
I met my first DONE, AND GETS THINGS SMART engineers back at Geoworks. This was looong before I had any sort of a clue that I suck as an engineer, but these folks (Andrew Wilson and Chris Thomas, if you really must know) were _weird_. They never seemed to be working that hard, but they were not only 10x as productive as their peers, they also managed technical feats that were quite frankly too scary for anyone else. They could (as just one trait) dive in and learn new languages and make fixes to tools that the rest of us assumed were, I dunno, stuff youd normally pay a vendor to fix. They dove into the hairiest depths of every code base they encountered and didnt just add features and make fixes; they waved some sort of magic wand and improved the system while they were in there: they would GET THINGS SMART. Make the systems smarter, that is. Sort of like getting your act together, but _they_d do it for _your_ code.
Ive met many more such engineers along the way. Theyre out there. Theyre better than you. They were better twenty years ago than I am today or ever will be. Maybe its natural ability. Maybe its luck in education or upbringing. Maybe they have a secret recipe for improving rapidly and learning utter fearlessness. I dont know. But Ive met em, and they arent "smart". Theyre abso-flutely fugging brilliant.
You cant interview these people. For starters, theyre not interested; these are the people that companies hold on to as long as humanly or companyly possible. The kinds of people that companies file lawsuits over when theyre recruited away.
You can only find DONE, AND GETS THINGS SMART people in two ways, and one of them I still dont understand very well.
The first way is to get real lucky and have one as a coworker or classmate. You work with them for a few years and come to realize theyre just cut from a finer cloth than you and your other unwashed cohorts. You may be friends with some of them, which helps with the recruiting a little, but not necessarily. The important thing is that you _recognize_ them, even if you dont know what makes them tick.
This is the one great hope we programmers have for fighting the Dunning-Kruger Effect, the one hope we have for getting something better than the average "just like me" Solid Plugger Joe Nobody you pick up with the SMART, AND GETS THINGS DONE approach. Our only "out" is that working side-by-side with someone will show us clearly when they vastly outclass us.
Your devious little mind will come up with all sorts of rationalizations for why theyre so damn good, so you can continue to think of yourself as SMART, AND GETS THINGS DONE material. You may conclude that theyre just a genetic anomaly, and its no fair even trying to compare yourself to someone who obviously has an unfair gift from the heavens. Or you may tell yourself that theyre just a domain expert in various domains that you dont "need" right now. Or you may simply choose not to think about it too much. Good Old Compartmentalization to the rescue!
But working with them directly _will_ show you when theyre better. Its the only way. Youll gradually realize that your math deficiencies arent just something that you might need to beef up on if you ever "need to"; youll see that virtually every problem space has a mathematical modeling component that you were blissfully unaware of until DONE, AND GETS THINGS SMART gal points it out to you and says, "Theres an infinitely smarter approach, which by the way I implemented over the weekend." You stare slack-jawed for a little while, and then she says: "Heres a ball. Why dont you go bounce it?"
These people arent just pure gold; theyre golden-egg-laying geese. They are the ones you want to bring with you to your own startup. Not the SMART, AND GETS THINGS DONE, JUST AS SOON AS I READ UP ON THE SUBJECT, ON THE COMPANYS DIME riff-raff like you and me. No. Theyre your seed engineers: the ones who will make or break your company with both their initial technical output and the engineering-culture decisions they put into place -- decisions that will largely determine how the company works for the next twenty years.
Ive been debating whether to say this, since itll smack vaguely of obsequiousness, but Ive realized that one of the Google seed engineers (exactly one) is almost singlehandedly responsible for the amazing quality of Googles engineering culture. And I mean both in the sense of having established it, and also in the sense of keeping the wheel spinning. I wont name the person, and for the record he almost certainly loathes me, for reasons that are my own damn fault. But Id hire him in a heartbeat: more evidence, I think, that the DONE, AND GETS THINGS SMART folks arent necessarily your friends. Theyre just people youre lucky enough to have worked with.
At first its entirely non-obvious whos responsible for Googles culture of engineering discipline: the design docs, audited code reviews, early design reviews, readability reviews, resisting introduction of new languages, unit testing and code coverage, profiling and performance testing, etc. You know. The whole gamut of processes and tools that quality engineering organizations use to ensure that code is open, readable, documented, and generally non-shoddy work.
But if you keep an eye on the emails that go out to Googles engineering staff, over time a pattern emerges: theres one superheroic dude whos keeping us all in line.
How do you interview for such a person? You cant! Everyone will tell you theyre Gods Gift to engineering quality. Everyone knows how to give it impressive lip service. Heck, there are lots of people who take it way too far and try to gridlock the organization in their over-enthusiasm, when what you really want is a balanced and pragmatic approach. Id argue that its virtually impossible to detect these "soft skills" in a classic interview setting, except to the extent that youre hiring your own clone, which according to our thesis here, is NOT what you want. You want DONE, AND GETS THINGS SMART: done faster than you, and made _your_ system smarter!
Im guessing that Googles founders worked with this person in school, enough to recognize his valuable talents. Hence they used Identification Approach #1: get lucky in who you work with.
Incidentally, they hired plenty of other brilliant seed engineers who were equally responsible for Googles great technical infrastructure. Im just using this one guy as an illustrative example. But you really want the DONE, AND GETS THINGS SMART people on every team. If you could mix in one DONE, AND GETS THINGS SMART person with every five to ten SMART, AND GETS THINGS DONE people, then youd be in good shape, since the latter, being "smart", can hopefully learn a lot from the former.
But when youre starting a company, or an organization, or a big project, the need for DONE, AND GETS THINGS SMART seed engineers is _desperate_. Its dire, in the sense that if you dont get the right seed people in place, youre dooming your organization to mediocrity, if you manage to succeed at all.
And its direst when youre in a startup, because you cant pillage people from elsewhere in your organization who you know are good. And because DONE, AND GETS THINGS SMART people are worth their weight in refined plutonium, theyre probably reasonably happy in their current position.
So lets assume youre looking at the vast ocean of programmers, all of whom are self-professed superstars whove gotten lots of "stuff" done, and you want to identify not the superstars, but the super-_heroes_.
How do you do it? Well, Brian Dougherty of Geoworks did it somehow. Jeff Bezos did it somehow. Larry and Sergey did it somehow. Im willing to bet good money that _every_ successful tech company out there had some freakishly good seed engineers. But a lot of company heads who make these decisions arent necessarily industry programmers, and they still manage to find some world-class people in all the noise. So there must be a second way to identify DONE, AND GETS THINGS SMART.
I think Identification Approach #2, and this is the one I dont understand very well, is that you "ask around". You know. You manually perform the graph build-and-traversal done by the Facebook "Smartest Friend" plug-in, where you ask everyone to name the best engineer they know, and continue doing that until it converges.
I think this might work, assuming you have lots of connections initially (lots of roots for your graph), so you dont get stuck in some slummy local minimum.
Ive seen companies go to university professors and ask them who their brightest students are; thats a variant of this approach, although it usually only turns up _future_ DONE, AND GETS THINGS SMART engineers. Every once in a very rare while youll get a recent college grad in this category, but I think more often they tend to be experienced enough to make Gandalf feel young again.
Because technical brilliance, seemingly superhuman productivity, and near-militaristic adherence to software discipline arent enough. They also need leadership skills. They dont have to be _great_ leaders; in fact in a pinch, just being bossy might work for a while, as long as theyre bossing people in the right directions. But they need to have the ability to guide the organization (or new team, or whatever) in uniformly excellent directions, which requires _some_ leadership, even if its bad or amateurish leadership.
As much as I suspect Approach #2 may work, I think Approach #1 is probably more reliable. Take a closer look at your coworkers who are doing things that you "could learn if you ever need it". Read up on the old Dunning-Kruger Effect. I recommend it with irony dialed at 11, since personally I have yet to read more than the Wikipedia article and a few other articles here and there. Ill read it if I "need to". Psh.
DONE, AND GETS THINGS SMART
Not superstars: superheroes! People who are freakishly good at what they do. People who finish things so fast that they seem to have paranormal assistance. People who can take in any new system or design for all intents instantaneously, with no "ramp-up", and who can immediately bring insights to bear that are quite simply beyond your rustic abilities.
Those are the folks you want. Im not going to tell you: "Dont settle for less." Far from it. You still want to hire the SMART, AND GETS THINGS DONE folks. But those folks have a long way to grow, and they probably have absolutely no idea just how far it is. So you want some DONE, AND GETS THINGS SMART people to guide them.
And now, to play us out...
Dooooone and Ge-ets Things Smart,
Done and Ge-ets Things Smart,
Done and Geeee-eeeets Thiiings Smaaaa-aaaart,
Done and Ge-ets Things Smart!
I will once again plagiarize myself by transcribing a talk I gave.
First: be warned! I offer this gesture of respect to you — yes, you! — when I say that this is at least 20 minutes of reading. This is long even for me. If youre surfing reddit, gobbling up little information snacks, then its best to think of this entry as being more like a big downer cow. Unless youre _really_ hungry, you should wait for it to be sliced into little bite-sized prion patties before consuming it.
If you do read it, youll see the CJD analogy is surprisingly apt. I ramble even more than usual, and lose my train of thought, and the slides might as well be scenes from a David Lynch movie for all the relation they have to my actual talk.
But once again I find myself astonished at how much I agree with myself, by and large. Funny how that works. And I made a few decent jokes here and there. So Im transcribing it.
If youre impatient, and I wouldnt blame you a bit, the best part is probably "Static Typings Paper Tigers". That might be worth reading. As for the rest... *shrug* If youre really starved for content, you might find some of it entertaining.
THE SETTING
I gave this talk at the Google I/O Conference in San Francisco a few weeks ago. My talk was boringly titled "Server-Side JavaScript on the Java Virtual Machine", and there were initially only about 40 or 50 people in the room (out of a 2500-person conference) when I started the talk.
Even though I personally thought the talk was pretty boring, people kept trickling in, and I estimate there were about 400 people stuffed in the room by the end. It was standing-room only, and people were spilling out into the hall. The conclusion? The other talks must have been _really_ boring.
After my talk it became pretty clear to me that it should have been titled "Rhinos and Tigers", so thats its new name. Ive tried to make it flow well by splitting it into arbitrary sub-sections, whose titles arent really part of the talk. But otherwise its pretty much a word-for-word transcription, except with most of the umms and aaaahs elided. Ive included the subset of the slides that seemed relevant; you can find the rest at the Google I/O site.
So enjoy! Or not. Ive given you plenty of warnings. Youll see!
RHINOS AND TIGERS
Hello! Im Steve Yegge. I work at Google. Ive been there about three years, and its real nice.
Im going to be talking about server-side scripting in general, and talking a lot about Mozilla Rhino and the technology behind it. Im going to try to get through it in maybe 20-25 minutes, maybe 30 if I start ranting, at which point Ill open it up for questions. I kind of want you guys to help drive how this goes.
Make sense? _(Ed: Well, it made sense at the time. Sigh.)_
All right, cool. Lets get started.
Sooo... Im going to be talking about Server-Side JavaScript on the Java Virtual Machine.
Yes. Weve got this big animal. Rhino.
So lets see... who here has used a JVM language before? Oooh! My god, lots of you, almost all of you. Great!
Well Im going to be talking about Rhino in particular. Ill be making reference to other JVM languages. I want to kind of help you see how this fits into the landscape, why you might use it, why you might not use it.
But for those of you who havent used a JVM language, the Java Virtual Machine is sort of like .NET: you can run multiple languages on it. You can write an interpreter in Java, or you can compile your language down to Java bytecode. Or you can compile it down to your own bytecode; there are different ways to do it.
But typically these languages are sort of drop-in replacements for Java. Which means you can implement classes, you can implement interfaces, you can subclass things. It gives you an alternate syntax and semantic layer on top of the libraries, and on top of the virtual machine.
Ill assume that this makes sense... well, actually, I wont!
FOO CHAOS
Theres this dude named Walter Bright, who wrote the D programming language, among many other things. _(Raise hand)_ Has anyone heard of Walter? Hes this really smart dude. He wrote Symantec Cafe, and the game Empire [and Zortech C++].
He told me the other day, [talking about] one of my blog rants, that he didnt agree with the point that Id made that virtual machines are "obvious"__. You know? I mean, of course you use a virtual machine!
But hes a compiler dude, and he says theyre a sham, theyre a farce, "I dont get it!" And so I explained it [my viewpoint] to him, and he went: Ohhhhhhh.
Virtual machines are great for language interoperability. If everybody in the world used his language, then yeah, you probably wouldnt need a virtual machine. Youd probably still want one eventually, because of the just-in-time compilers, and all the runtime information they can get.
But by and large, we dont all use D. In fact, we probably dont all use the same five languages in this room. And so the VM, whether its the CLR, or the Java VM, or Parrot, or whatever... it provides a way for us to interoperate.
Now Ill tell ya — I was at Foo Camp last summer. Ive been wanting to tell this story... Im telling you guys; its the coolest story. And its relevant here. Heh. Very relevant.
So I was in this tent... you know what Foo Camp is? Its OReillys, you know, Friends Of OReilly invite thing that they do each summer. Its coming up in a couple of weeks. And people give presentations; people show up and just wander into your [presentation] tent, and wander back out if they dont like it.
So I was in this discussion at the very end of the last day, where the Apple LLVM guy Chris [Lattner] was hosting a talk on dynamic languages running on different VMs. And there was the Smalltalk Squeak guy there, and there was Charles Nutter for JRuby and representing the JVM. John Lam was there for IronRuby and CLR, and there were the Parrot people. I cant even remember them all, but the whole room was _packed_ with the VM implementors of the VMs today, and people who are implementing languages on top of them.
This was a _smart_ group of people, and well-informed. And you know, I was like a fly on the wall, thinking man, look at all [these brains].
And Chris, well, he let everybody go around the room and talk about why their VM was the best. And they were all right! Thats the weird thing: every single one of them was right. Their VM was the best for what they wanted their VM to do.
Like, Smalltalk [Squeak]s VM was the best in the sense of being the purest, and it was the cleanest. Whereas the Java one was the best because, you know, it has Java! Everybodys was the best. Parrots was the best because it was vaporware. Ha! Ha, ha ha. Sorry guys.
So! He [Chris] asked this really innocent question. He goes, "You know, I dont really know much about this stuff..."
Which is bad, you know. When somebody says that to you at Foo Camp, it means theyre settin you up.
He says, "So how do these languages talk to each other?"
And the room just _erupted_! It was chaos. All these people are like, "Oh, its easy!" And the rest of them are like "No, its hard!" And theyre arguing, and arguing, and arguing. They argued for an _hour_.
And then they stood up, still arguing, and they kept talking about it, heading into the dinner tent. And they sat down, going at it for like three hours.
It was _chaos._
Because some people were saying, "Well, you know, if Rubys gonna call Python, well, uh, you just call, right? You just share the same stack, right?"
And the others are like, "Well, what about different calling conventions? What if they support optional vs. non-optional arguments? What if they support default arguments? What about the threading model? What about the semantics of, you know, the
So! Yeah, theres a lot of stuff I could talk about. I gave a practice of this talk to Mike Loukides, an editor at OReilly, and it completely changed what I wanted to talk about.
I do want to talk about Rhinos technology; I want you to come away understanding it. But more importantly, I want you guys to understand where this fits in this Google conference. And where it fits in _your_ plans, going forward.
See, its really interesting. We all know, or at least most of us I think agree, that server-side computing is finally starting to make inroads onto the desktop. Fat clients arent so much the norm anymore. Youve got applications like Google Maps, GMail, Google Docs, those kinds of apps, that are doing "desktop quality" applications in the browser, with the server handling your storage.
Thats kind of one of the messages of this conference. Everybodys doing it, right? Its not just Google. And it makes a certain amount of sense, so Im not going to go into the reasons why youd do that. Im assuming its sort of a given. _(Editors Note: youd be amazed at how many emails I get from people who maintain its a fad of some sort, one thats going away, which is why I bother to make this disclaimer.)_
The interesting thing is this: all applications... who was it who said "All apps will eventually grow to the point where they can read mail, and if they dont, theyll be replaced by ones that can"? _(audience: "JWZ")_ JWZ? Jamie Zawinski. Yeah. Its a variant of somebody elses quote [Greg Kuperbergs], but...
So its true, right? Apps grow. If you like an app, youre gonna want to start doing more and more stuff in it. If you like it a _lot_, like I like Emacs, heh... you know, you like your editor. Everybody here is a programmer, right? You all use development environments? Do you ever find it kind of annoying that you have to switch from your IDE to your browser? Why isnt the IDE the browser too? Why arent these unified?
I mean, lets face it: I only run two apps. Unless I need to run, like, OmniGraffle or the Gimp, or something to do a document, or Keynote here to do the presentation — I just switched to Macs, so Im learning all these names, but, this PowerPoint thing — most of the time, when Im developing, Im running shells, and Im running Emacs, and Im running a browser. Thats it! So you kind of wish theyd be the same.
Well, once they get big enough, your IDE and Emacs and the browser have this thing in common, which is that they are _scriptable_!
Thats the magic point at which your application becomes sort of alive. Right? Because people can change it, if it doesnt work the way they like it.
GreaseMonkey! Perfect example. You dont like our web page that we give you? Write a GreaseMonkey script and change it all around, right? Thats cool! Scripting is really important.
I mean, Emacs, it stands for "Editor Macros", and it started off as a really thin engine, and the whole editor was written in scripts. And now its huge. It has a million lines or so of Emacs-Lisp code floating around.
So its weird... you go through this transformation, where your scripting languages are originally for, well, scripting. And it eventually grows into application level/scale development. OK?
Now we all see this happening in clients. Excel, for instance, is scriptable. And the reason that Excel is so powerful, I mean the reason that you can go to the bookstore and get a book thats this thick on Excel, and scientific computing people use it, whatever, is that it has a very very powerful scripting engine.
In fact, all of Microsoft Office has it. Right? You can fire up Ruby or Python or Perl, and you can actually control, though the COM interface, you can actually tell IE to open a document and scroll to a certain point in it. Or you can open up Microsoft Word and actually... I mean, if you want to do the work, you could actually get to where youre typing into your Perl console and its showing up over in Word.
Server-side computing has to get there. Its _gonna_ get there.
But how many server-side apps are user scriptable today? Precious few. Google has a couple, you know, like our JotSpot acquisition, which is [scriptable] in Rhino...
So were talking about something thats kind of new. I mean, we can all see it coming! But its still kind of new, the idea, right? Why?
Because this is scary hard, from a security perspective. Heh. Youre going to run code on _my_ servers? Uh... OK...
I mean, Yahoo! Store, you know, Paul Grahams Viaweb that went on to become Yahoo! Store. People have done it, right?
I wrote a game that was really cool. Scriptable! I mean, high school kids were teaching themselves to program so they could write areas and monsters and spells and quests for this game that I wrote, which was written in Java and scriptable in Jython.
Its a big deal! I mean, people want to be able to write these apps.
However, I had to live with the fact that I didnt personally have enough bandwidth to come up with a decent security model to keep them from... its a trust-based security model in my game. They write a script, they could erase my hard disk, right? So Ive got to be very careful, and recognize that I can only let certain people that I trust do it. And that Ive got to be prepared for really big disasters.
Because also theres denial-of-service. Its inadvertent: oh, their script is taking up all the bandwidth [or CPU or memory] on my server, and everybody else in the game is paralyzed. Right? I mean, how do you deal with it?
Youve got to deal with user [i.e., programmer] throttling: memory usage, the database or datastore usage, like Amazons computing cloud, you know, they have a lot of this stuff in place. But usually its pretty coarse-grained when it gets started, right? You get a box, and a certain amount of disk storage, and you get the whole CPU, because how are you gonna allocate CPUs out to people when the languages themselves that are being used for scripting dont support that? _(Editors Note: obviously you can just use process scheduling, but Im talking more about multithreaded processes like my game, or Second Life, where many users may be scripting within the same processes. It makes things harder.)_
Were getting there; its happening. But its new. And its hard. Because you dont want people to be able to go and get access to your companys proprietary code or resources and wreak havoc. You just want to host their computing.
So when you decide youre going to take your server-side application, with its beautiful Ajax app talking to the server, and now you want open it up: to add extensibility for your end users -- theyre not just scripting the client; theres scripting happening on the server thats theirs -- you have to make a decision!
Namely, what language are you going to give them?
We have... see, unfortunately its hard for me to talk about Google products, because all I know are their internal code names, and not their launch names. I can never remember them. But we have... something like that. Heh. Called... Prometheus, I think? Uh, wha... what is it?
_(audience member: Google App Engine)_ Ahem, the Google App Engine, of course! Yes. The Google App Engine. Ahem. Yes. _(me: embarrassed)_
And I think its... Python. Right now. But of course they want to open it up, because, its like, you dont really want to force people to switch editors, unless you want a real fight on your hands. You kinda dont really want to force people to switch languages either, right? People want to use whatever theyre comfortable with.
So again, you wind up with this hosted environment, where youre supporting multiple languages; which one do you pick [first]? They picked Python. You can pick [anything], but youve got these problems to deal with. And Im going to argue today that Rhino is actually a really good choice for this particular problem space.
OK, weve got people pooling up in the back here. Is it time to invite them in? Come on in, sit down, theres space! All right, cool. Yeah. Welcome!
So yeah. Thats what Im talking about today. Do you guys understand the perspective now, the context? Im talking about server-side scripting, that either you do yourself inside your company, because you feel like youve got some logic that needs to be kind of "glue" logic — "scripting" — or, more importantly, youre opening it up to your users. Which means you need to sandbox it, and you need to meter it and throttle it.
ADVANTAGES OF SCRIPTING ON THE JVM
All right. Yeah. So this is a JVM language. A JVM language can share the Java libraries, and the Java Virtual Machine. Its really cool, right? And really powerful.
Right off the bat, these JVM implementations of other languages, like JRuby vs. Ruby, Jython vs. Python, right? They get all these free benefits, that may not necessarily exist in the C runtimes for these languages.
Example? Java has a really good garbage collector these days. A generational garbage collector thats becoming an incremental [and/or concurrent] generational garbage collector... I mean, its good! Whereas for a lot of these [C-based] languages, they use mark-and-sweep, reference-counting...
Another one is native threads. Its veeery nice to have native threads, and also have well-defined semantics for the threads and how they interact with the memory model. I mean, a lot of these [non-JVM] languages are like, "Well, we have threads, but you probably... dont want to use them." Because youre kind of on your own.
So what happens is people use process-switching; its the share-nothing model. And thats great too, for certain situations. Provided youve got good engineering library support around it, like the java.util concurrency libraries. They can help you design something without having to do a formal proof around it to get it to work.
That helps a lot in multicore. It helps! JavaScript has no [language-level] threads, because Brendan Eich says "over his dead body". I totally understand where hes coming from on this, right? Theres certainly the "promise" of better concurrency out there. Erlang, you know, and maybe STM...
But hey man, today? I mean, right now? You want to write something with high throughput, and youve got a lot of I/O to do, and its parallelizable? And you want to get a lot of throughput on one box, because its got multiple cores?
_(shrugging)_ Well, threads get the job done. So if youve got it in your so-called "scripting language", its a big win.
Weve got garbage collection, threads... and asynchronous I/O, right? When Java first came out there was the whole "one thread per socket" model [actually, two], which meant that you couldnt write a webserver that could handle ten thousand concurrent requests. It didnt matter how much memory or CPU your box had. Anyone here ever tried to fire up 10,000 threads on one box?
Yeah... yeah. What happens is, the scheduler and task-switching resources for managing the threads swamp your machine. So eventually Java wrote a wrapper around the Unix or Windows or whatever native interfaces so you could get super-high throughput.
So all of the sudden, by sticking something on the JVM... Sure, you initially get a bit of a hit in performance. When these people first port a language to the Java Virtual Machine, its usually about twice as slow, right? BUT, its got async I/O, and its got [native] threads, and its got better non-pausing (by and large) garbage collection. And from there, they can make it smarter.
But theyve also got the JIT. I dont know, I mean, did anybody here... I gave a talk on dynamic languages recently at Stanford, but I dont want to rehash that if you guys already know about that.
Basically I argued in that talk — successfully at Stanford, so I think that was... something — that for just-in-time compilers, its becoming pretty clear, they have a lot better access, a lot better data at runtime about how youre actually using your program right now than the compiler ever had.
So the JITs can do all kinds of inlining and optimizations that you just cant do in a compiler. And what that means is that everybody running on this VM gets to benefit from the JIT.
So there are lots of advantages to these JVM languages. Or .NET, if you happen to be using Microsoft stuff. Its a good model.
BUT WHY RHINO?
So why Rhiiiiino? Why JavaScript?
_(loudly)_ Who here thinks JavaScript is kind of icky? Come on, come on, be honest. Yeah, there we go, a couple of people. Yeah.
Yeahhhh... and you know what? It is! Right? Because, well, because of vendor implementation issues. Thats one [reason]. Also, Brendan was kind of forced to rush it out the door. You guys know... back at Netscape, when they built JavaScript, it was called, um, LiveScript?
And Brendan was building Scheme for the browser. Scheme!
Everyone in here who knows Scheme, raise your hand. _(Many people, at least fifty, raise their hands.)_
Holy... smokes! A lot more than I wouldve guessed. Wow.
OK, well, as it happens, you guys are not "representative". _(laughter)_
And so, Netscape kinda looked at it, and said: "Yeah, well, we did say Scheme, but, uh, theres this Java thing, this giant marketing thing, and we want to be involved with it." And some back-door deals happened, and then they came to Brendan and said: "Make it look like Java."
So now its Scheme with Java syntax, right? So he had to pull a lot of all-nighters for a couple of weeks, and he came up with JavaScript.
So, you know, its got some flaws. Some of which make you want to go scrape your teeth on the sidewalk because its less painful. So its true, but what language doesnt have some flaws?
The interesting thing about Rhino, which is an implementation of JavaScript in Java, is that theres only one language. You dont have to worry about vendor-implementation or cross-platform problems because... its just Rhino. So right out of the starting gate, thats a win.
Plus, Rhino gives you the ability to work around some of the problems. A classic one is the problem in JavaScript where you cant define non-enumerable properties. Right? You know how you can go
Oh yeah. So unit testing. I mean, like, all the other stuff on this slide is like "blah blah blah", but then Unit Testing [in Rhino] — this was a real surprise to me.
I write a lot of Java code day to day, [out of the] probably five different languages I code in regularly. And unit testing was always a chore. Unit testing is a chore.
I mean, come ON. Unit testings a chore, right? _(raise hand)_ Who here thinks unit tests are just a poor mans static type system? Eh? _(some laughter)_ Yeah!
Well, not really, since you have to write unit tests for them [the static languages] too. _(more laughter)_
You need to write unit tests, and unfortunately in Java its VERY PAINFUL. Im speaking into the mic now, so that everybody can hear. Unit testing in Java is painful!
Its _so_ painful that people, the Java... community, the Java world, has evolved _around_ it. OK? Theyve said: "Dont use constructors. Constructors are baaaaad."
_(pointed pause)_ I mean... what!? _(laughter)_
I mean, like, if you program in Ruby, say, you know that you can actually change the way the metaclass produces objects. So if you want it to be a singleton, or you want it to meet certain... you want it to be a Mock object, you just replace the
OK. Down on the bottom weve got some code here. Actually on the top, too. So I do a
And oh, yeah... Im going to be talking shortly here about Rhino on Rails, which is this thing that I did... its not Rhino on Rails, actually. Its actually, _I_ called it "Rhinos not Ruby". Because I got kinda burned at Google for using Ruby. Yeah. Uh, for good reasons, good reasons. But they were like: "No."
And so of _course_ I called it "Rhinos not Ruby": RnR. Because people know JavaScript; theyre kinda comfortable with JavaScript, so they were OK with it. So I had to port Rails; it was kind of a lot of work, but, you know, well it works! Were using it here internally; its nice. I mean, I actually know its nice, because six months went by and I didnt look at it for those six months. And for this recent project, I picked it up, and I was, like, is this gonna be gross?
But actually, its really pretty nice. Because youve got all the Java libraries, and all the integration with Google stuff. Its cool. Ill try to open-source it this year, if I forget to say that later on.
Anyway, I was writing unit tests for this thing, and... uh... _(I completely blow my stack. Who am I? Where am I?)_
Have I lost where I am on the slides? _(Duh.)_
Ive diverged from the slides. Ill come back to RnR shortly. Basically, I got unit-testing religion. Thats the end of that sort of stack.
If you can do it easily, and you dont have to rewrite your application to be unit-testable? Man. Thats a big difference.
So why would you _not_ use Rhino for server-side scripting?
Well, its not super-duper fast right now. Its on the order of about twice as slow as Java, depending on what youre doing. So if it really has to be super, super fast — use C++! Right? Naaaah, you can use Java. Like I was saying the other day [at Stanford], its widely admitted inside of Google — theres this whole discussion, is Java as fast as C++, right? And Google Java developers definitely admit that its as fast as C++. The C++ people, well... yeah. _(sigh)_
Lets see... if youre writing a library, then Rhinos actually not so good right now. There is no standard library interface for scripting languages. We havent got there yet. Its all, its all related to what I told you about before, you know the calling interop. A lot of these [languages] have their own package systems: their own
So unfortunately we have thirteen minutes left. Im sorry. So lets really quickly go through some of the really cool things about Rhino, the technology here.
You can JavaScript from Java, and Java from JavaScript. Guess which ones easier?
Obviously calling Java from JavaScript is easier, because Javas really cumbersome. It doesnt have anything to help you, so you have to do basically what I was talking about with Swing earlier.
This is what you need to do to create a JavaScript object called
Uh... this is the actual code that I was referring to earlier, where you can define non-enumerable properties. I called it
Runtime delegation: this is one of the reasons unit testing is really easy. You guys know about Smalltalk, uh, method-missing? [
OK, so let me tell you a little bit about embedded XML. Its kind of interesting, kinda neat. This is supported in Firefox, in some browsers. Its a spec that Adobe and some other people, BEA, put together.
And its cool! Because you can say stuff like
All right. So this is Rhino. Now you know. After I explain this diagram, youll know what you need to know about Rhino to talk to other people about it.
You start with some JavaScript code. It goes to a parser. That turns it into a tree. A syntax tree.
Rhinos parser today, currently, immediately begins the next step of rewriting it as code generation. Right away, as its parsing. Now this is a problem. Because if it takes an if-statement or a switch-statement or a for-loop, and it generates sort of assembly-language like jumps to targets? And generates labels, you know, converts it into sort of three-address code, thats eventually going to actually become three-address code: assembly or bytecode.
Then it kind of sucks if youre trying to use the parse tree for your IDE. To, like, syntax highlight, or pretty-print, or show errors and warnings, or whatever. Unfortunately a lot of languages — most languages — do this because theyre written by compiler guys and compiler gals. And they dont see the value. But unfortunately were all doing more and more processing of code. Language tools, right? Frontend stuff.
So I rewrote Rhinos parser recently, and Im currently fixing the code generator. And Im gonna get it out into the Rhino mainstream in a couple of weeks here. Because my project at Google is doing a lot of code processing. And its a faithful representation. So if your big beef about Rhino is that theres no Visitor over the AST: Im fixing that.
And then there are two paths here: you see one on the left that goes code generator to bytecode. And theres the bytecode, or pseudo-bytecode, for the JavaScript code up there. And then it goes to an interpreter. The interpreter is a big loop. Bytecode is this [roughly] postorder traversal of the tree, that flattens it in a way that allows you to push onto a stack to evaluate the operands.
Its all actually very important; you should go read up on how it works if youre not familiar with it, or if youve forgotten since you first learned it.
And the interpreter is actually pretty fast, because its a loop. Theres not a lot of calling out. I mean, there are some calls out into the runtime, but mostly its this loop: push, pop, push, pop. So the JIT picks it up and can optimize it pretty well.
The reason that theres two code paths here, the reason that they wrote the interpreter — they originally had just a classfile compiler — was that compiling to a classfile is this batch/static operation, where you want it [the resulting bytecode] to be fast. You want to do the standard, classic compiler optimizations. You want to generate the control-flow graph, you want to eliminate dead code, you want to do good register allocation.
In JavaScripts case, its often possible not to generate a closure. You can actually use Java instance variables and Java local variables instead of these heavier-weight JavaScript [activation objects]. Because at the logical level, JavaScript doesnt really even have a stack. It has object allocations on the heap; those are your Function invocations. Sloooow. Right? Because [in comparison] the Java stack translates to the C stack.
So the fact that the compiler can go through and optimize away a lot of this JavaScript dynamic stuff thats provable youre not gonna need, well, thats nice! But it takes time. The interpreter is a path that allows you to dynamically develop: load code in and see how its gonna work right now, at the unfortunate expense of the Rhino people having to maintain these two code paths. But, you know, theres a lot shared between them.
And thats it! The script runtime implements all the JavaScript, you know, Ecma spec stuff:
So RnR, I already talked about it. It doesnt have a database backend yet, because were using Googles internal store, like Bigtables. Which is why I havent open-sourced the thing yet.
Its weird: somebody told me the other day, they sent me mail and said: "I think youre todays Paul Graham". And he meant this in the most negative connotation possible. "Youre todays Paul Graham, and RnR is the next Arc."
I was like, "What!?" And he said, well, hes a server-side JavaScript guy. I mean, there arent that many, right? Most of us are thinking client-side. But hes a server-side JavaScript guy. And he goes to people and says, why arent you using server-side JavaScript? And they say: "Were waiting for Steve Yegge to release RnR."
And Im... this is news to me! Im working on... stuff, you know. Work. And this [open-sourcing RnR] is part-time and everything.
This year, now that we know people are interested in it, we will release it. _(At least well try for this year - Ed.)_
Its just a little weird, right? Because Sun hired the JRuby guys, and theyre doing JRuby on Rails, and its eventually going to be part of the Java Development Kit. Its gonna be, you know: its Suns lightweight answer to EJB and all those giant frameworks. You want to build something quickly and use the Rails model, well, run Rails on the JVM!
So I thought: if JRuby on Rails had been (a) ready when we started using it [i.e. writing RnR], and (b) Google would let me use Ruby, then I would have used that! So RnR was like a transitional thing.
But... again, you know, I think that there are other people in situations where you really prefer to use JavaScript. So yeah, I guess Ill... open-source it. Were working on it.
This is the last slide, by the way; I know you guys are tired. Were doing a lot of work on it. Im working on it personally, I mean working on Rhino. Because I think its all right. Im used to JavaScript now, and I think its a good implementation. Its a good compiler, so its good practice for me, to learn how compilers work.
Weve got a debugger, but were making the debugger better. Weve got a sandboxing model, but that could definitely be wrapped up and made available to you folks.
Wed like to open-source our JSCompiler: the thing that compresses JavaScript for Google Maps and GMail and stuff. I know there are some open-source ones out there. We dont think that its competitively in our best interest to keep the thing internal. Itd be better to get it out there so you all benefit from it, and so you can all hack on it, right? Were working on open-sourcing our JSCompiler and other stuff.
So thats it! I wanted to cover Rhino, but I also wanted to leave time for questions. And Ive left you _(looking at big LED clock in the back)_ one minute and sixteen seconds for questions. Sorry about that.
So really quickly, if there are any burning questions, Ill repeat the question and try to answer it. Otherwise feel free to come up afterwards and chat.
Q&A
Q: WHY WONT GOOGLE LET YOU USE RUBY?
Yeah, thats a good question. Um... uh... I kinda wrote that up in a blog. Isnt that stupid? "Read my blog!"
The short answer is: it imposes a tax on the systems people, which means that its not completely self-contained within the team thats using it. And for that reason, primarily, its really not a good idea right now.
Any other burning questions?
Q: DO THREADS SUCK?
Well, you know... theyre... you know... Yeah. But I mean, what other options do you have? I mean, you have multiprocessing/share-nothing, which is heavyweight and it requires more work.
So I use threads. Id prefer something better, but theyre what weve got today.
Q: THERE ARE SOME GUYS THAT I WORK WITH, AND ONE OF THEIR COMMENTS ON JAVASCRIPT LATELY, SINCE IVE BEEN WANTING TO USE RHINO BECAUSE I LOVE JAVASCRIPT... WHAT THEY BROUGHT UP IS THAT JAVASCRIPT IS BECOMING A LOT LIKE PYTHON, AND THAT MAY OR MAY NOT BE SUCH A GREAT THING. I WANTED TO KNOW WHAT YOU HAVE TO SAY ABOUT THAT.
Ah. OK. Well, yeah, it already has borrowed some stuff from Python in JavaScript 1.7: 
Hello! Im Steve Yegge. I work at Google. Ive been there about three years, and its real nice.
Im going to be talking about server-side scripting in general, and talking a lot about Mozilla Rhino and the technology behind it. Im going to try to get through it in maybe 20-25 minutes, maybe 30 if I start ranting, at which point Ill open it up for questions. I kind of want you guys to help drive how this goes.
Make sense? _(Ed: Well, it made sense at the time. Sigh.)_
All right, cool. Lets get started.
Sooo... Im going to be talking about Server-Side JavaScript on the Java Virtual Machine.
Yes. Weve got this big animal. Rhino.
So lets see... who here has used a JVM language before? Oooh! My god, lots of you, almost all of you. Great!
Well Im going to be talking about Rhino in particular. Ill be making reference to other JVM languages. I want to kind of help you see how this fits into the landscape, why you might use it, why you might not use it.
But for those of you who havent used a JVM language, the Java Virtual Machine is sort of like .NET: you can run multiple languages on it. You can write an interpreter in Java, or you can compile your language down to Java bytecode. Or you can compile it down to your own bytecode; there are different ways to do it.
But typically these languages are sort of drop-in replacements for Java. Which means you can implement classes, you can implement interfaces, you can subclass things. It gives you an alternate syntax and semantic layer on top of the libraries, and on top of the virtual machine.
Ill assume that this makes sense... well, actually, I wont!
FOO CHAOS
Theres this dude named Walter Bright, who wrote the D programming language, among many other things. _(Raise hand)_ Has anyone heard of Walter? Hes this really smart dude. He wrote Symantec Cafe, and the game Empire [and Zortech C++].
He told me the other day, [talking about] one of my blog rants, that he didnt agree with the point that Id made that virtual machines are "obvious"__. You know? I mean, of course you use a virtual machine!
But hes a compiler dude, and he says theyre a sham, theyre a farce, "I dont get it!" And so I explained it [my viewpoint] to him, and he went: Ohhhhhhh.
Virtual machines are great for language interoperability. If everybody in the world used his language, then yeah, you probably wouldnt need a virtual machine. Youd probably still want one eventually, because of the just-in-time compilers, and all the runtime information they can get.
But by and large, we dont all use D. In fact, we probably dont all use the same five languages in this room. And so the VM, whether its the CLR, or the Java VM, or Parrot, or whatever... it provides a way for us to interoperate.
Now Ill tell ya — I was at Foo Camp last summer. Ive been wanting to tell this story... Im telling you guys; its the coolest story. And its relevant here. Heh. Very relevant.
So I was in this tent... you know what Foo Camp is? Its OReillys, you know, Friends Of OReilly invite thing that they do each summer. Its coming up in a couple of weeks. And people give presentations; people show up and just wander into your [presentation] tent, and wander back out if they dont like it.
So I was in this discussion at the very end of the last day, where the Apple LLVM guy Chris [Lattner] was hosting a talk on dynamic languages running on different VMs. And there was the Smalltalk Squeak guy there, and there was Charles Nutter for JRuby and representing the JVM. John Lam was there for IronRuby and CLR, and there were the Parrot people. I cant even remember them all, but the whole room was _packed_ with the VM implementors of the VMs today, and people who are implementing languages on top of them.
This was a _smart_ group of people, and well-informed. And you know, I was like a fly on the wall, thinking man, look at all [these brains].
And Chris, well, he let everybody go around the room and talk about why their VM was the best. And they were all right! Thats the weird thing: every single one of them was right. Their VM was the best for what they wanted their VM to do.
Like, Smalltalk [Squeak]s VM was the best in the sense of being the purest, and it was the cleanest. Whereas the Java one was the best because, you know, it has Java! Everybodys was the best. Parrots was the best because it was vaporware. Ha! Ha, ha ha. Sorry guys.
So! He [Chris] asked this really innocent question. He goes, "You know, I dont really know much about this stuff..."
Which is bad, you know. When somebody says that to you at Foo Camp, it means theyre settin you up.
He says, "So how do these languages talk to each other?"
And the room just _erupted_! It was chaos. All these people are like, "Oh, its easy!" And the rest of them are like "No, its hard!" And theyre arguing, and arguing, and arguing. They argued for an _hour_.
And then they stood up, still arguing, and they kept talking about it, heading into the dinner tent. And they sat down, going at it for like three hours.
It was _chaos._
Because some people were saying, "Well, you know, if Rubys gonna call Python, well, uh, you just call, right? You just share the same stack, right?"
And the others are like, "Well, what about different calling conventions? What if they support optional vs. non-optional arguments? What if they support default arguments? What about the threading model? What about the semantics of, you know, the this pointer? What about all this _stuff?_"
And theyre like _(waving hands)_ "Ooooh, well gloss over it, gloss over it, smooth it over." And the reply is: "You _cant_. This is fundamental. These languages work differently!"
And oh my god, it was really interesting. And it was also very clear that its ten years of research and implementation practice before they get this right. Before youll be able to have a stack of calls, where youre calling from library to function, library to function in different languages.
So today, VMs are good for interoperability, but youve gotta use a bridge. Whether its JRuby, or Jython, or Rhino, they provide a set of APIs — you know about javax.script, right? Its this new thing introduced to the JDK, where they try to formalize just a couple of APIs around how you call the scripting language from your language... you know, its a sort of "reverse foreign-function interface". And then how they call each other, maybe.
But its all done through... kind of like serialization. You marshal up your parameters as an array of arguments, and its a heavyweight call that goes over [to the script runtime] and comes back, and its like... its a pain! You dont want that. But today, thats kind of what were stuck with.
At least we have that, though, right? I mean, try having Ruby call Python today, and they have different FFIs. You can do it, but you definitely want the VMs help.
So, Walter, thats why you need VMs.
POWER TO YOUR USERS
So! Yeah, theres a lot of stuff I could talk about. I gave a practice of this talk to Mike Loukides, an editor at OReilly, and it completely changed what I wanted to talk about.
I do want to talk about Rhinos technology; I want you to come away understanding it. But more importantly, I want you guys to understand where this fits in this Google conference. And where it fits in _your_ plans, going forward.
See, its really interesting. We all know, or at least most of us I think agree, that server-side computing is finally starting to make inroads onto the desktop. Fat clients arent so much the norm anymore. Youve got applications like Google Maps, GMail, Google Docs, those kinds of apps, that are doing "desktop quality" applications in the browser, with the server handling your storage.
Thats kind of one of the messages of this conference. Everybodys doing it, right? Its not just Google. And it makes a certain amount of sense, so Im not going to go into the reasons why youd do that. Im assuming its sort of a given. _(Editors Note: youd be amazed at how many emails I get from people who maintain its a fad of some sort, one thats going away, which is why I bother to make this disclaimer.)_
The interesting thing is this: all applications... who was it who said "All apps will eventually grow to the point where they can read mail, and if they dont, theyll be replaced by ones that can"? _(audience: "JWZ")_ JWZ? Jamie Zawinski. Yeah. Its a variant of somebody elses quote [Greg Kuperbergs], but...
So its true, right? Apps grow. If you like an app, youre gonna want to start doing more and more stuff in it. If you like it a _lot_, like I like Emacs, heh... you know, you like your editor. Everybody here is a programmer, right? You all use development environments? Do you ever find it kind of annoying that you have to switch from your IDE to your browser? Why isnt the IDE the browser too? Why arent these unified?
I mean, lets face it: I only run two apps. Unless I need to run, like, OmniGraffle or the Gimp, or something to do a document, or Keynote here to do the presentation — I just switched to Macs, so Im learning all these names, but, this PowerPoint thing — most of the time, when Im developing, Im running shells, and Im running Emacs, and Im running a browser. Thats it! So you kind of wish theyd be the same.
Well, once they get big enough, your IDE and Emacs and the browser have this thing in common, which is that they are _scriptable_!
Thats the magic point at which your application becomes sort of alive. Right? Because people can change it, if it doesnt work the way they like it.
GreaseMonkey! Perfect example. You dont like our web page that we give you? Write a GreaseMonkey script and change it all around, right? Thats cool! Scripting is really important.
I mean, Emacs, it stands for "Editor Macros", and it started off as a really thin engine, and the whole editor was written in scripts. And now its huge. It has a million lines or so of Emacs-Lisp code floating around.
So its weird... you go through this transformation, where your scripting languages are originally for, well, scripting. And it eventually grows into application level/scale development. OK?
Now we all see this happening in clients. Excel, for instance, is scriptable. And the reason that Excel is so powerful, I mean the reason that you can go to the bookstore and get a book thats this thick on Excel, and scientific computing people use it, whatever, is that it has a very very powerful scripting engine.
In fact, all of Microsoft Office has it. Right? You can fire up Ruby or Python or Perl, and you can actually control, though the COM interface, you can actually tell IE to open a document and scroll to a certain point in it. Or you can open up Microsoft Word and actually... I mean, if you want to do the work, you could actually get to where youre typing into your Perl console and its showing up over in Word.
Server-side computing has to get there. Its _gonna_ get there.
But how many server-side apps are user scriptable today? Precious few. Google has a couple, you know, like our JotSpot acquisition, which is [scriptable] in Rhino...
So were talking about something thats kind of new. I mean, we can all see it coming! But its still kind of new, the idea, right? Why?
Because this is scary hard, from a security perspective. Heh. Youre going to run code on _my_ servers? Uh... OK...
I mean, Yahoo! Store, you know, Paul Grahams Viaweb that went on to become Yahoo! Store. People have done it, right?
I wrote a game that was really cool. Scriptable! I mean, high school kids were teaching themselves to program so they could write areas and monsters and spells and quests for this game that I wrote, which was written in Java and scriptable in Jython.
Its a big deal! I mean, people want to be able to write these apps.
However, I had to live with the fact that I didnt personally have enough bandwidth to come up with a decent security model to keep them from... its a trust-based security model in my game. They write a script, they could erase my hard disk, right? So Ive got to be very careful, and recognize that I can only let certain people that I trust do it. And that Ive got to be prepared for really big disasters.
Because also theres denial-of-service. Its inadvertent: oh, their script is taking up all the bandwidth [or CPU or memory] on my server, and everybody else in the game is paralyzed. Right? I mean, how do you deal with it?
Youve got to deal with user [i.e., programmer] throttling: memory usage, the database or datastore usage, like Amazons computing cloud, you know, they have a lot of this stuff in place. But usually its pretty coarse-grained when it gets started, right? You get a box, and a certain amount of disk storage, and you get the whole CPU, because how are you gonna allocate CPUs out to people when the languages themselves that are being used for scripting dont support that? _(Editors Note: obviously you can just use process scheduling, but Im talking more about multithreaded processes like my game, or Second Life, where many users may be scripting within the same processes. It makes things harder.)_
Were getting there; its happening. But its new. And its hard. Because you dont want people to be able to go and get access to your companys proprietary code or resources and wreak havoc. You just want to host their computing.
So when you decide youre going to take your server-side application, with its beautiful Ajax app talking to the server, and now you want open it up: to add extensibility for your end users -- theyre not just scripting the client; theres scripting happening on the server thats theirs -- you have to make a decision!
Namely, what language are you going to give them?
We have... see, unfortunately its hard for me to talk about Google products, because all I know are their internal code names, and not their launch names. I can never remember them. But we have... something like that. Heh. Called... Prometheus, I think? Uh, wha... what is it?
_(audience member: Google App Engine)_ Ahem, the Google App Engine, of course! Yes. The Google App Engine. Ahem. Yes. _(me: embarrassed)_
And I think its... Python. Right now. But of course they want to open it up, because, its like, you dont really want to force people to switch editors, unless you want a real fight on your hands. You kinda dont really want to force people to switch languages either, right? People want to use whatever theyre comfortable with.
So again, you wind up with this hosted environment, where youre supporting multiple languages; which one do you pick [first]? They picked Python. You can pick [anything], but youve got these problems to deal with. And Im going to argue today that Rhino is actually a really good choice for this particular problem space.
OK, weve got people pooling up in the back here. Is it time to invite them in? Come on in, sit down, theres space! All right, cool. Yeah. Welcome!
So yeah. Thats what Im talking about today. Do you guys understand the perspective now, the context? Im talking about server-side scripting, that either you do yourself inside your company, because you feel like youve got some logic that needs to be kind of "glue" logic — "scripting" — or, more importantly, youre opening it up to your users. Which means you need to sandbox it, and you need to meter it and throttle it.
ADVANTAGES OF SCRIPTING ON THE JVM
All right. Yeah. So this is a JVM language. A JVM language can share the Java libraries, and the Java Virtual Machine. Its really cool, right? And really powerful.
Right off the bat, these JVM implementations of other languages, like JRuby vs. Ruby, Jython vs. Python, right? They get all these free benefits, that may not necessarily exist in the C runtimes for these languages.
Example? Java has a really good garbage collector these days. A generational garbage collector thats becoming an incremental [and/or concurrent] generational garbage collector... I mean, its good! Whereas for a lot of these [C-based] languages, they use mark-and-sweep, reference-counting...
Another one is native threads. Its veeery nice to have native threads, and also have well-defined semantics for the threads and how they interact with the memory model. I mean, a lot of these [non-JVM] languages are like, "Well, we have threads, but you probably... dont want to use them." Because youre kind of on your own.
So what happens is people use process-switching; its the share-nothing model. And thats great too, for certain situations. Provided youve got good engineering library support around it, like the java.util concurrency libraries. They can help you design something without having to do a formal proof around it to get it to work.
That helps a lot in multicore. It helps! JavaScript has no [language-level] threads, because Brendan Eich says "over his dead body". I totally understand where hes coming from on this, right? Theres certainly the "promise" of better concurrency out there. Erlang, you know, and maybe STM...
But hey man, today? I mean, right now? You want to write something with high throughput, and youve got a lot of I/O to do, and its parallelizable? And you want to get a lot of throughput on one box, because its got multiple cores?
_(shrugging)_ Well, threads get the job done. So if youve got it in your so-called "scripting language", its a big win.
Weve got garbage collection, threads... and asynchronous I/O, right? When Java first came out there was the whole "one thread per socket" model [actually, two], which meant that you couldnt write a webserver that could handle ten thousand concurrent requests. It didnt matter how much memory or CPU your box had. Anyone here ever tried to fire up 10,000 threads on one box?
Yeah... yeah. What happens is, the scheduler and task-switching resources for managing the threads swamp your machine. So eventually Java wrote a wrapper around the Unix or Windows or whatever native interfaces so you could get super-high throughput.
So all of the sudden, by sticking something on the JVM... Sure, you initially get a bit of a hit in performance. When these people first port a language to the Java Virtual Machine, its usually about twice as slow, right? BUT, its got async I/O, and its got [native] threads, and its got better non-pausing (by and large) garbage collection. And from there, they can make it smarter.
But theyve also got the JIT. I dont know, I mean, did anybody here... I gave a talk on dynamic languages recently at Stanford, but I dont want to rehash that if you guys already know about that.
Basically I argued in that talk — successfully at Stanford, so I think that was... something — that for just-in-time compilers, its becoming pretty clear, they have a lot better access, a lot better data at runtime about how youre actually using your program right now than the compiler ever had.
So the JITs can do all kinds of inlining and optimizations that you just cant do in a compiler. And what that means is that everybody running on this VM gets to benefit from the JIT.
So there are lots of advantages to these JVM languages. Or .NET, if you happen to be using Microsoft stuff. Its a good model.
BUT WHY RHINO?
So why Rhiiiiino? Why JavaScript?
_(loudly)_ Who here thinks JavaScript is kind of icky? Come on, come on, be honest. Yeah, there we go, a couple of people. Yeah.
Yeahhhh... and you know what? It is! Right? Because, well, because of vendor implementation issues. Thats one [reason]. Also, Brendan was kind of forced to rush it out the door. You guys know... back at Netscape, when they built JavaScript, it was called, um, LiveScript?
And Brendan was building Scheme for the browser. Scheme!
Everyone in here who knows Scheme, raise your hand. _(Many people, at least fifty, raise their hands.)_
Holy... smokes! A lot more than I wouldve guessed. Wow.
OK, well, as it happens, you guys are not "representative". _(laughter)_
And so, Netscape kinda looked at it, and said: "Yeah, well, we did say Scheme, but, uh, theres this Java thing, this giant marketing thing, and we want to be involved with it." And some back-door deals happened, and then they came to Brendan and said: "Make it look like Java."
So now its Scheme with Java syntax, right? So he had to pull a lot of all-nighters for a couple of weeks, and he came up with JavaScript.
So, you know, its got some flaws. Some of which make you want to go scrape your teeth on the sidewalk because its less painful. So its true, but what language doesnt have some flaws?
The interesting thing about Rhino, which is an implementation of JavaScript in Java, is that theres only one language. You dont have to worry about vendor-implementation or cross-platform problems because... its just Rhino. So right out of the starting gate, thats a win.
Plus, Rhino gives you the ability to work around some of the problems. A classic one is the problem in JavaScript where you cant define non-enumerable properties. Right? You know how you can go for (i in foo) ..., and itll enumerate the keys of your object as if it were a hashtable.
Nice feature, right? And you can add properties to objects; you can go to Object.prototype, which is the root object of the whole system, and add your own function(s) there. But what happens is, youve added a function thats now enumerable in everybodys arrays, and everybodys data structures, so their for..in loops break.
Which means that fundamentally you cant install a library thats completely seamless and emulates Ruby or Python or some other really expressive set of library functions. Like, you want your map and collect, and your String.reverse, and... you know what I mean?
You cant do it in browser JavaScript, so people wind up going different routes. They either do what Prototype does, and just install it, and youre screwed if you use for..in, but you dont use for..in, right?
Or they use functions. They dont use object-oriented programming. And you know, functional programming is great and everything, but OOP, as weve discovered in programming in the large, is a nice organizational tool. Putting the function or method together with the thing that its operating on is a nice way of organizing things.
So its kind of unfortunate when you have to use functions, because if you have to say, you know, HTMLElement.getChildren.whatever, it gets inverted with functions: whatever(getChildren(HTMLElement)). You have to call from the innermost one to the outermost... its "backwards", right?
Rhino winds up getting around that problem completely. We did, anyway, internally. Because its written in Java. So you can call the Rhino interface. You can call Parser, or the interpreter, or the runtime; you can do whatever you want.
So I wrote this little defineProperty function, thats like five lines of code. It calls into the script runtime Java class that implements JavaScript objects, which has a non-enumerable defineProperty.
JavaScript has non-enumerable properties; it just doesnt let you add your own. Its just a language flaw, right?
That [defineBuiltin function] enabled us, in the project Im going to be talking about a little bit later here, to implement all of Ruby and Pythons runtime — all the functions we liked — in [server-side] JavaScript, in a non-intrusive way. We were also able to implement a class system, and all this other stuff.
So Rhino is _not_ browser JavaScript.
Man, weve got more people pooling up at the entrances. You guys are welcome to come in, squeeze in and sit down... come on in... welcome. Theres still space. Especially up here kinda in the front, in the middle, where nobody wants to sit. But trust me, its better there.
So yeah. Rhinos history: its like ten years old. Or more? More than ten years, maybe. It started inside Netscape, side by side with SpiderMonkey. A lot of people have been hacking on it. Rhinos pretty robust.
RHINO AT THE SHOOTOUT
I have a question for ya. I did this "JVM shootout" like three and a half years ago. I was kind of tired of using Java for scripting, and I wanted to look at all the JVM languages. So I did this game. You know about the game Sokoban? I would have done Sudoku if the craze had hit then. Its a little dude who pushes these blocks around these mazes?
Well, I reimplemented this thing, which is about, you know, six or seven hundred lines of Java code. It had a [user] interface, and a little search algorithm that had him chase your mouse. It was just big enough of an application that I could reimplement it in like 10 different languages, and actually compare how it was speed-wise, how to use them [the languages], how well they interoperated with Java... it was an actual apples-to-apples comparison.
Most of them really, really, REALLY stank. It was baaaad. I mean, there are like 250 JVM languages out there, but most of them are just complete toys. But there were ten or so that were actually pretty good. You could do anything in them, and they had decent performance, and they were good, right?
And it [the shootout] kind of petered out, because it started looking like Rhino-JavaScript was going to win. I had this sort of solution selection matrix of criteria where... it was kind of a heuristic function where I weighted all these terms, right? Just to kind of get a feel for which one [was best].
And I wanted JRuby to win. You should never go into these comparisons wanting one of them to win, because, you know, youre either going to bias it or youre gonna be disappointed. JRuby at the time was really slow. Its much faster now, and everything, but at the time, it was so new.
Jython was good, but it wasnt fast enough, and the author of Jython had gone off to greener pastures.
Rhino! Rhino had good tools, and it had good performance, and it was... JavaScript! Eeeeww!
So I never even really... I published it, but I didnt leave it up. Im actually going to bring it back up again soon; Im going to update it and do a couple of new languages. Because I find this an eternally fascinating question, which is: what is the next big language, especially on the JVM, going to be?
DOMAIN-SPECIFIC LANGUAGES
Java will always have a place. But I think there are domains, like Java Swing, you know? The Java GUI stuff? Javas really not very good for that. Weve kind of figured out that Object-Oriented Programming doesnt work that well for UIs. You want it to be declarative. HTML showed that you want a dialog, with a title bar, and a body, and it _nests_, to match the [UI] tree.
That works really well. Its succinct. Even in HTML its succinct, right? Whereas with a [Java-style] object-oriented language, youve got to say, you know, createContainer(), addChild(), addChild(), addChild(), til the cows come home. And it doesnt _look_ anything like... you cant pattern-match it and say "ah yes! this looks just like my UI!"
So people write these wrappers around Swing. Like theres Apache Jelly, which wound up with this XML framework to do Swing programming, that was 30% less verbose than Java.
What are the odds that XMLs going to wind up being less verbose than _anything?_ _(loud laughter)_ Really! I mean, I was shocked. 30% less verbose. And I looked at it, too. They werent cheating. I mean, they did the best Swing implementation in Java that they could, but Jelly was better.
So there are domains for which Java is just not appropriate. But you still maybe want to use a VM for all the reasons that I outlined earlier.
So yeah! Theres room for these other languages. But which one? All of them? Are they going to solve the problem I brought up from Foo Camp earlier? To where it doesnt matter which language youre using; they can call each other, and [mix however you like?]
I mean, hows your editor going to feel about that? Hows your team member going to feel about it? A lot of people dont like learning new languages.
Who here doesnt like learning new languages? Come on, be honest... _(A few people raise hands)_ Yeah! New languages. No fun!
Its actually kind of... you should try it. _(laughter)_ You know? It is. Its a good idea.
Theres this dude — has everyone heard of The Little Schemer? The Little Schemer, The Seasoned Schemer, The Reasoned Schemer? Cool books, right? Teach you functional programming in this really bizarre format that hooks you in.
Dan Friedman, the guy who [was] one of the collaborators on those books -- I was reading an article he wrote. Early in his career he realized that languages are fundamental to computer science and to engineering; theyre really important. And he wanted to be able to learn a new language every quarter.
And after he did that for a while, he said, you know what? I want to learn a new language every _week_. OK? And you can actually get to the point where you can do this. Now it probably takes 2-3 months before youre actually as comfortable with the new language as you were with your favorite old one. This happened to me with JavaScript; I was freaking out for the first couple of months, thinking "this is _never_ gonna work".
But eventually you get over the hump, and youre like _(relieved sigh)_ "aaaah, yes." Right? You learn how to work around the problems. Its just like learning Java or whatever your first language happened to be. Youve got to learn your way around all these problems, and youve gotta learn how things work, and how the libraries are laid out. And all of the sudden it becomes like breathing.
So Dan Friedman, after he said he was learning a language a week, I thought, "wow, thats pretty macho." But he said, nah, that wasnt good enough: he wanted to be able to _implement_ a language a week. And he got pretty serious about it. Spent years researching how to do this effectively. _(Well, now Im just speculating - Ed.)_
This is where Id love to see engineers today. Knowing languages will make you a better programmer. It will! It will even if youre not using them. Just write a little application in it, and it opens your mind. [Each new one] opens your mind. And now suddenly you know the superset of all the languages out there. So its not scary to learn new ones.
And you can also recognize situations where a _language_ is actually the right tool for the job. Not a library, not a framework, not some new object or some new interface. A language!
Case in point? Regular expressions. _(raise hand)_ Who likes to write their own giant deterministic finite automata to do string matching? Heh. Its weird — nobody raised their hand.
Who likes to do lots and lots of manual DOM manipulations, as opposed to, say, XPath? XPath is a language, right? DOM manipulations, you know... it depends, but usually, no: not if you can get away with using a language for it.
I could talk for hours about that, so Im not going to. But, you know... its good to learn new languages. So Im gonna teach you JavaScript today. Im gonna dive in. So lets go!
THE RIGHT WAY TO DO UNIT TESTING
Oh yeah. So unit testing. I mean, like, all the other stuff on this slide is like "blah blah blah", but then Unit Testing [in Rhino] — this was a real surprise to me.
I write a lot of Java code day to day, [out of the] probably five different languages I code in regularly. And unit testing was always a chore. Unit testing is a chore.
I mean, come ON. Unit testings a chore, right? _(raise hand)_ Who here thinks unit tests are just a poor mans static type system? Eh? _(some laughter)_ Yeah!
Well, not really, since you have to write unit tests for them [the static languages] too. _(more laughter)_
You need to write unit tests, and unfortunately in Java its VERY PAINFUL. Im speaking into the mic now, so that everybody can hear. Unit testing in Java is painful!
Its _so_ painful that people, the Java... community, the Java world, has evolved _around_ it. OK? Theyve said: "Dont use constructors. Constructors are baaaaad."
_(pointed pause)_ I mean... what!? _(laughter)_
I mean, like, if you program in Ruby, say, you know that you can actually change the way the metaclass produces objects. So if you want it to be a singleton, or you want it to meet certain... you want it to be a Mock object, you just replace the new function on the fly, and youve got yourself a Mock constructor, right?
But Javas set in stone! So they use factories, which leads to this proliferation of classes. And you have to use a "Dependency Injection Framework" to decouple things, right?
And its just, like, _(panting)_... We were doing business logic early on in Java. When Java came out, it was like: "Ooooh, its a lot easier to write C code", basically, or C++. Rather than focusing on your memory allocation strategy, or which of your companys six conflicting string classes youre gonna use, you get to focus on "business logic".
But unfortunately that only took you so far. Now youre focusing on Mock object frameworks. It [Java] only took you a little farther.
Now I _swear_, man, doing Mock objects in Rhino is so easy! Its easier, even, than in JRuby or in Jython, because JavaScript has JSON. I didnt even know, like, the name of it when I started doing this, right? But JSON... Ive gotta show you guys this. This is so cool. _(flipping through slides)_
Yeah, tools, blah blah blah. Well come back to it. Oh, its way down in here. Urrggh. Come on... heres one!
OK. Down on the bottom weve got some code here. Actually on the top, too. So I do a new Thread with a new Runnable, and, uh... it sure looks a lot like Java code, huh? This is one advantage of JavaScript, actually. Java...Script, right? Ten years later its finally becoming the scripting language for Java?
So that syntax _(with an obj literal following new Runnable())_ is a little weird, but theres another one here that says:
js> obj = {run: function() { print(hi) }}
So Ive declared an object literal, using "JSON style". Now JSON doesnt let you do — does JSON let you do functions? Probably not, right? But I mean, fundamentally youre doing this declarative property-value list, right?
And so what Ive got is this anonymous thing that has a named "run" property whose value is a function! That prints "hi". And now I can create a new Thread, with a new Runnable that wraps it, and what effectively Ive done is Ive used that thing as the Runnable interface [implementation], which expects a function called "run" that takes no arguments and does whatever the threads supposed to do.
This is how you do mock objects!
I have this huuuge legacy system, right? With hundreds of static methods. Static methods are also bad these days, right? Noooo static methods. Cuz theyre not mockable. Right? Java has changed Java. Because Javas not unit-testable. So now you cant just go to the store and [buy a book and] learn Java. You have to learn all these... fashions. You have to learn whats in vogue.
Subclassing! _Not_ in vogue right now. You talk about subclassing, and people are like "NNnnnnnooooo, you should use manual delegation even though its really hard and cumbersome and awkward."
And youre like, "but I just want to change this one method, and plus its built into the language, and theres syntax for it, and its kind of well-understood..." And they just say "NO!"
Its out of favor. For similar reasons. Oh my god...
And Im telling ya: the reason unit testing is easy, is, fundamentally, the way you develop in a dynamic language is _different_ from the way you develop in a static language: C++, Java... OCaml, Scala, whatever. Your favorite static language.
To a large extent, especially in C++ and Java, the way you develop is:
* you write the code
* you compile the code
* you load the code
* you run the code
* it doesnt work
* you back up
* you change it again
So its this batch cycle, right? 1950s. Submit your punch cards, please.
In a dynamic language — and this is clearest when youre writing in Emacs Lisp [because of the *scratch* buffer] — but its somewhat clear when youre developing in a console, in Python or Ruby, Perl, Lua, whatever, you write an expression, and you give it some mock data.
Youre writing a function, youre building it on the fly. And when it works [for that mock data], youre like, "Oh yeah, it works!" You dont run it through a compiler. You copy it and paste it into your unit test suite. Thats one unit test, right? And you copy it into your code, ok, this is your function.
So youre actually proving to yourself that the thing works by construction. Proooof by construction.
Obviously you still need your unit tests _(er, I meant integration tests – Ed.)_, because theres going to be higher-order semantics, you know, calling conventions between classes and methods...
Whew! This room is really filling up. Um, is there anything we can do to help here, guys in the back? _(Tech guy says something inaudible in the video)_ Yeah, please! Therere more seats here. I just want to... I dont want to get to where people cant even make it into the room.
Yeah, so unit testing. I know you guys all hate unit testing. So did I. Or you say, "I looooove unit testing," but then, you know, your test coverage is still sitting at like 80%.
Im telling you, man, this is a huge, huge thing. It changes the way you do your development.
RHINOS NOT RUBY
And oh, yeah... Im going to be talking shortly here about Rhino on Rails, which is this thing that I did... its not Rhino on Rails, actually. Its actually, _I_ called it "Rhinos not Ruby". Because I got kinda burned at Google for using Ruby. Yeah. Uh, for good reasons, good reasons. But they were like: "No."
And so of _course_ I called it "Rhinos not Ruby": RnR. Because people know JavaScript; theyre kinda comfortable with JavaScript, so they were OK with it. So I had to port Rails; it was kind of a lot of work, but, you know, well it works! Were using it here internally; its nice. I mean, I actually know its nice, because six months went by and I didnt look at it for those six months. And for this recent project, I picked it up, and I was, like, is this gonna be gross?
But actually, its really pretty nice. Because youve got all the Java libraries, and all the integration with Google stuff. Its cool. Ill try to open-source it this year, if I forget to say that later on.
Anyway, I was writing unit tests for this thing, and... uh... _(I completely blow my stack. Who am I? Where am I?)_
Have I lost where I am on the slides? _(Duh.)_
Ive diverged from the slides. Ill come back to RnR shortly. Basically, I got unit-testing religion. Thats the end of that sort of stack.
If you can do it easily, and you dont have to rewrite your application to be unit-testable? Man. Thats a big difference.
So why would you _not_ use Rhino for server-side scripting?
Well, its not super-duper fast right now. Its on the order of about twice as slow as Java, depending on what youre doing. So if it really has to be super, super fast — use C++! Right? Naaaah, you can use Java. Like I was saying the other day [at Stanford], its widely admitted inside of Google — theres this whole discussion, is Java as fast as C++, right? And Google Java developers definitely admit that its as fast as C++. The C++ people, well... yeah. _(sigh)_
Lets see... if youre writing a library, then Rhinos actually not so good right now. There is no standard library interface for scripting languages. We havent got there yet. Its all, its all related to what I told you about before, you know the calling interop. A lot of these [languages] have their own package systems: their own import, their own require, right? So if youre gonna write a library, you should probably still write it in Java. Maybe.
If youre doing a framework, where youre defining how things are called: whether were calling you, or youre calling us, then its OK.
And if you really _hate_ JavaScript, then thats, you know, thats fine... But keep in mind, again, that you may be providing something for your end-users. If you go out to a high school and you survey people, and you ask, "So what language are you learning? How are you teaching yourself programming?" Its a sure bet that a lot of them are doing Ajax stuff. Right? A lot of them are doing JavaScript.
If you want to make your end-users happy, and you want to immediately capture a very big user base, then no matter how you detest JavaScript (and again, Rhino-JavaScripts really not as bad as browser JavaScript, its much better), your users probably will prefer it. Its something to keep in mind.
All right.
STATIC TYPINGS PAPER TIGERS
And then weve got Scala. Ive gotta mention Scala. Who here knows... youve heard of Scala? Yeah? _(a few hands go up)_ Mmmmm, yeah, getting there... looks like some people, OK.
Scala is a very strongly typed language for the JVM. Its from researchers in Switzerland; theyre professors. Its from sort of the same school of thought that static typing has evolved with over the last fifteen years in academia: Haskell, SML, Caml, these sorts of H-M functional languages.
And Scalas interesting because it actually takes a functional static type system and it layers... it merges it with Javas object-oriented type system, to produce.... Frankensteins Monster.
Ive got the language spec here in my backpack. Oh, my god... I mean, like, because its getting a little bit of momentum, right? So I figure Ive got to speak from a position of sort of knowledge, not ignorance, when Im dissing it. (Heh heh.)
And so _before_, I was like: "Oh yeah, Scala! Strongly typed. Could be very cool, very expressive!"
The... the the the... the language spec... oh, my god. Ive gotta blog about this. Its, like, ninety percent [about the type system]. Its the biggest type system youve _ever_ seen in your life, by 5x. Not by an order of magnitude, but man! There are type types, and type type types; theres complexity...
They have this concept called complexity complexity Meaning its not just complexity; its not just complexity-complexity: its _parameterized_ complexity-complexity. _(mild laughter)_ OK? Whoo! I mean, this thing has types on its types on its types. Its _gnarly_.
Ive got this Ph.D. languages intern whose a big Haskell fan, and [surprisingly] a big Scheme fan, and an ML fan. [But especially Haskell.] He knows functional programming, he knows type systems. I mean, hes an expert.
He looked at Scala yesterday, and he told me: "Im finding this rather intimidating."
Im like, "THAT sounds like its gonna take off!" _(loud laughter)_ Oh yeah!
But the funny thing about Scala, the really interesting thing — you guys are the first to hear my amazing insight on this, OK? — is: its put the Java people in a dilemma. Theres a reeeeeeeal problem.
The problem is, the Java people say, "Well, dynamic languages, you know, suck, because they dont have static types." Which is kind of circular, right? But what they mean, is they say: No good tools, no good performance. But even if you say, look, the tools and performance can get as good, they say, "Well, static types can help you write safer code!"
Its... you guys know about those talismans? The ones, where, "Whats it for?" "To keep tigers away"? _(some chuckling)_ Yeah? And you know, people are like, "How do you know it keeps tigers away?" And your reply is: _(sneering)_ "Do you see any tigers around here!?" _(minor laughter)_
So this is what... OK, so for a long time, for many years... and you know, Ive written more Java code than most Java programmers _ever_ will. _(Editors note: nearly 1M lines in production. Ouch.)_ So trust me. I tried. OK? Im not just coming in and saying "I dont want to learn Java." No. I know Java as well as the next person.
But I come to them and say, lets do proof by – say, argument by example! You know, an existence proof. IMDB is written in Perl, right? Yahoo! – many of their properties are written in PHP. A lot of Microsoft stuffs written in VB, right? ASP .NET? Amazon.coms portal site is Perl/Mason.
A lot of companies out there are building big, scalable systems – and I mean scalable in the sense of throughput and transactions, stuff like that, but also scalable in terms of human engineering — in these dynamic languages with no static types. [Using nothing more than good engineering principles.]
So... isnt that a demonstration that you dont need the static types to keep those tigers away?
And theyre like: "Well! But! What if... what if a tiger came?" _(laughter)_ Right? "People need shotguns in their house in case a bear comes through the door, right?" The Simpsons made fun of that. _(laughing continues)_
Yeah. So, you know, for a long time, I was like: "Yeah, yeah, yeah. OK. So tigers could come. Fine."
Scala, now, is the tiger thats going to kill Java. Because their [type-talisman] argument now has become a paradox, similar to the Paul Graham Blub Paradox thing, right? Because theyre like, "Well, we need static typing in order to engineer good systems. Its simply not possible otherwise."
The Scala people come in and they go: "Your type system _suuuuuucks_. Its not sound. Its not safe. Its not complete. Youre casting your way around it. It doesnt actually prevent this large class of bugs. How many times have you written catch (NullPointerException x) ... in Java? Our type system doesnt allow [things like] that."
Our type system does what _you_ said _your_ type system was doing.
So, therefore, you should be using it! ∴
And the Java people look at it and go: "Wehellll... _(cough cough)_... I mean, yeah, I mean... _(*ahem*)_" _(running finger under collar, as if sweating profusely)_ They say, "Welllll... you know... its awfully... cummmmmbersome... I..."
"We can actually get around the problems in practice that you guys say your type system is solving through Good Engineering Practices."
_(laughter begins to grow)_
HA!!! _(I point accusingly at the audience, and theres more laughter)_
Yeah.
So Scala is creating a real problem for [Javas] static typing camp now. Because their last little bastion of why theyre using it, the whole tigers argument, theyre like, "Ah, well... we... we keep shotguns in our house." [This is what theyve been reduced to.]
OK? Yeeeeahhhh...
So back to dynamic languages!
But my point was — from a previous slide actually — its very interesting. See, I wrote this Rails port, and it wasnt... I never got it to where it was quite as succinct as Rails, because JavaScript has curly braces and a little bit of extra syntactic cruft. But it was close!
And then we used this framework to build this web app internally. It was for external consumption. Its kind of a long story that I wont go into. But we had like 20 engineers on this thing, for close to a year. And we had a huge application. I mean in terms of user functionality: Ajax-enabled pages, server-side persistence stuff... it was a big app.
And it was, like 40,000 lines of code, including the templates and the client-side JavaScript. The whole thing! OK? I mean, you add in unit tests, you know, you add in everything, including build files and stuff, and this thing was up to like, maybe 55,000 lines of code.
_Thousand_.
I mean, Java programmers would be saying, "We havent hit 55 million yet. _(Looking at feet)_ But, well... were gonna." _(laughter)_
And its like, I tell em that _(shaking head)_, I tell em that, and theyre like: "Well." _(avoiding eye contact)_
Thats what they say. "Well."
And thats, you know, thats pretty much it. _(laughter)_
BEHIND THE RHINO
So unfortunately we have thirteen minutes left. Im sorry. So lets really quickly go through some of the really cool things about Rhino, the technology here.
You can JavaScript from Java, and Java from JavaScript. Guess which ones easier?
Obviously calling Java from JavaScript is easier, because Javas really cumbersome. It doesnt have anything to help you, so you have to do basically what I was talking about with Swing earlier. JavaScriptObject j = new JavaScriptObject() You know. JavaScriptObject, Context.enter! Youve got all this _stuff_ on the Java side. cuz its Java.
But, uh... but it works! And you can do both directions. Heres an example of a Java program to bootstrap... actually I believe this is completely standalone; it works out of the box. Its a Rhino Demo:
This is what you need to do to create a JavaScript object called foo that has a function called a. A property called "a", sorry, whose value is "hello".
So what you do is you call Context.initStandardObjects(), which sets up the JavaScript runtime. You only have to do it once. And then you call newObject to create a new JavaScript object. And then you call evaluateString to evaluate it in the context of this object.
Its one example of how you do it, but its not too hard. You can call back and forth.
So that means that anything that was written in JavaScript that you feel, oh Gosh this really needs to be componentized, you need to stick it down in a Java library for whatever reason: you can do it! You can migrate code back and forth between the JavaScript layer and the Java layer. This is true for all JVM languages, I think.
Uh... this is the actual code that I was referring to earlier, where you can define non-enumerable properties. I called it defineBuiltin. Theres some closure stuff going on here... I dont want to bore you guys. _(Editors note: Function.bind() based on Douglas Crockfords original)_
Runtime delegation: this is one of the reasons unit testing is really easy. You guys know about Smalltalk, uh, method-missing? [doesNotUnderstand actually] Its method_missing in Ruby. Its the... if you call an object — I think Objective C probably has something like this too.
You call an object, and you say: "Im calling foo", and it says: "I dont have a foo". Right?
Normally, what happens when you do this? In Java it goes *BARF*. As it should, probably... _unless_ what you really wanted to do was delegate to some other object. Right? "Design Patterns". Say you want to write a Bridge that says: "Oh! Youre calling foo, but you dont want to call it on me. You want to go to the game server, call it there, marshal it, send it back. Well pretend its a remote method call."
Right? Theres a lot of stuff youve got to go through in Java to do stuff like this. In JavaScript — as you all know, if youre using dynamic languages...
Man, weve got a huge pool of people in the back. Its getting pretty rough. But were almost out of time! Fortunately. Heh.
OK, so let me tell you a little bit about embedded XML. Its kind of interesting, kinda neat. This is supported in Firefox, in some browsers. Its a spec that Adobe and some other people, BEA, put together.
And its cool! Because you can say stuff like
var company =
Now of course theres this weird, big religious debate going on, between JSON advocates and XML advocates. Its weird! Theyre, like, locking horns.
When I was a kid — when I was a kid, jeez... When I was _twenty_, it feels like when I was a kid — I used to have tropical fish. And my brothers and I noticed two things about tropical fish.
One is that they die, because were not in the tropics. _(some laughter)_ Sad.
And the other is that if you put a bunch of different species of tropical fish in a tank together, they ignore each other... except for the ones that are the same [or nearly the same] species. They bite each other. Thats what they do. Fish bite each other. They have a pecking order, right?
JSON and XML are muscling in on each others space, and there are bristles, OK, and its so silly! Its silly. The whole thing, right? I mean, XML is better if you have more text and fewer tags. And JSON is better if you have more tags and less text. Argh! I mean, come on, its that easy. But you know, theres a big debate about it.
Nevertheless, sometimes XML is appropriate [in JavaScript], especially if youre loading it from a file or whatever. These literals are interesting. And so it provides new operators and new syntax for actually going in and... its kind of like XPath. Except its JavaScript-y.
And I tell you: it is the worst documented thing on the planet! Its horrible, man, working with E4X initially. But... eventually you can figure it out. And I have a document that hopefully Ill be ready to release pretty soon, that actually covers it pretty well. And Adobe has some good documentation for it.
And then eventually it clicks, like learning any other language. This is a minilanguage. And you go: "Ha, I get it! I get it. Its not as crazy and dumb as I thought. It actually works."
Its kind of a neat feature. You guys know other languages that embed XML? Scala does. I dont see all of you using that, but C# 3, I think, does XML? Coming [soon]? _(Editors note: its apparently been deprioritized by the C# team, although VB 9 has them.)_
Anyway, its kind of an interesting approach.
INSIDE THE RHINO
All right. So this is Rhino. Now you know. After I explain this diagram, youll know what you need to know about Rhino to talk to other people about it.
You start with some JavaScript code. It goes to a parser. That turns it into a tree. A syntax tree.
Rhinos parser today, currently, immediately begins the next step of rewriting it as code generation. Right away, as its parsing. Now this is a problem. Because if it takes an if-statement or a switch-statement or a for-loop, and it generates sort of assembly-language like jumps to targets? And generates labels, you know, converts it into sort of three-address code, thats eventually going to actually become three-address code: assembly or bytecode.
Then it kind of sucks if youre trying to use the parse tree for your IDE. To, like, syntax highlight, or pretty-print, or show errors and warnings, or whatever. Unfortunately a lot of languages — most languages — do this because theyre written by compiler guys and compiler gals. And they dont see the value. But unfortunately were all doing more and more processing of code. Language tools, right? Frontend stuff.
So I rewrote Rhinos parser recently, and Im currently fixing the code generator. And Im gonna get it out into the Rhino mainstream in a couple of weeks here. Because my project at Google is doing a lot of code processing. And its a faithful representation. So if your big beef about Rhino is that theres no Visitor over the AST: Im fixing that.
And then there are two paths here: you see one on the left that goes code generator to bytecode. And theres the bytecode, or pseudo-bytecode, for the JavaScript code up there. And then it goes to an interpreter. The interpreter is a big loop. Bytecode is this [roughly] postorder traversal of the tree, that flattens it in a way that allows you to push onto a stack to evaluate the operands.
Its all actually very important; you should go read up on how it works if youre not familiar with it, or if youve forgotten since you first learned it.
And the interpreter is actually pretty fast, because its a loop. Theres not a lot of calling out. I mean, there are some calls out into the runtime, but mostly its this loop: push, pop, push, pop. So the JIT picks it up and can optimize it pretty well.
The reason that theres two code paths here, the reason that they wrote the interpreter — they originally had just a classfile compiler — was that compiling to a classfile is this batch/static operation, where you want it [the resulting bytecode] to be fast. You want to do the standard, classic compiler optimizations. You want to generate the control-flow graph, you want to eliminate dead code, you want to do good register allocation.
In JavaScripts case, its often possible not to generate a closure. You can actually use Java instance variables and Java local variables instead of these heavier-weight JavaScript [activation objects]. Because at the logical level, JavaScript doesnt really even have a stack. It has object allocations on the heap; those are your Function invocations. Sloooow. Right? Because [in comparison] the Java stack translates to the C stack.
So the fact that the compiler can go through and optimize away a lot of this JavaScript dynamic stuff thats provable youre not gonna need, well, thats nice! But it takes time. The interpreter is a path that allows you to dynamically develop: load code in and see how its gonna work right now, at the unfortunate expense of the Rhino people having to maintain these two code paths. But, you know, theres a lot shared between them.
And thats it! The script runtime implements all the JavaScript, you know, Ecma spec stuff: Array, String, Boolean, Date, the Math functions. And a lot of it just delegates down to Java where possible.
Pretty clean! Pretty standard. Its a pretty standard compiler, interpreter and runtime. Youre gonna see this if you dig into your favorite JVM language. Youll probably see something similar to this. This is actually more mature than a lot of them. A lot of them start off by interpreting the parse tree, and its slow from the method calls.
So this is why Rhinos fast. Now it could be a lot faster, and were working on it. Hopefully, you know, these things can be as fast as Java, in the same way that Java made the claim that it can be "as fast as C++". And for long-running applications, thats usually true. Especially with parallelism, right? Threads. And especially if the JIT has a chance to look at the actual code paths being used and compile them down into machine code _specific_ for that code path, as a fall-through.
Obviously for benchmarks, where they fire something up and run a loop a thousand times, or whatever? C++ is faster because the JIT hasnt had any time to kick in and evaluate whats going on. But for long-running services — which is what were all writing, yeah? At this Ajax conference — the JIT will kick in. And your Rhino code now will get very close to where Javas performance is. _(Provided youre not doing number-crunching - Ed.)_ So dont worry about that so much.
So RnR, I already talked about it. It doesnt have a database backend yet, because were using Googles internal store, like Bigtables. Which is why I havent open-sourced the thing yet.
Its weird: somebody told me the other day, they sent me mail and said: "I think youre todays Paul Graham". And he meant this in the most negative connotation possible. "Youre todays Paul Graham, and RnR is the next Arc."
I was like, "What!?" And he said, well, hes a server-side JavaScript guy. I mean, there arent that many, right? Most of us are thinking client-side. But hes a server-side JavaScript guy. And he goes to people and says, why arent you using server-side JavaScript? And they say: "Were waiting for Steve Yegge to release RnR."
And Im... this is news to me! Im working on... stuff, you know. Work. And this [open-sourcing RnR] is part-time and everything.
This year, now that we know people are interested in it, we will release it. _(At least well try for this year - Ed.)_
Its just a little weird, right? Because Sun hired the JRuby guys, and theyre doing JRuby on Rails, and its eventually going to be part of the Java Development Kit. Its gonna be, you know: its Suns lightweight answer to EJB and all those giant frameworks. You want to build something quickly and use the Rails model, well, run Rails on the JVM!
So I thought: if JRuby on Rails had been (a) ready when we started using it [i.e. writing RnR], and (b) Google would let me use Ruby, then I would have used that! So RnR was like a transitional thing.
But... again, you know, I think that there are other people in situations where you really prefer to use JavaScript. So yeah, I guess Ill... open-source it. Were working on it.
This is the last slide, by the way; I know you guys are tired. Were doing a lot of work on it. Im working on it personally, I mean working on Rhino. Because I think its all right. Im used to JavaScript now, and I think its a good implementation. Its a good compiler, so its good practice for me, to learn how compilers work.
Weve got a debugger, but were making the debugger better. Weve got a sandboxing model, but that could definitely be wrapped up and made available to you folks.
Wed like to open-source our JSCompiler: the thing that compresses JavaScript for Google Maps and GMail and stuff. I know there are some open-source ones out there. We dont think that its competitively in our best interest to keep the thing internal. Itd be better to get it out there so you all benefit from it, and so you can all hack on it, right? Were working on open-sourcing our JSCompiler and other stuff.
So thats it! I wanted to cover Rhino, but I also wanted to leave time for questions. And Ive left you _(looking at big LED clock in the back)_ one minute and sixteen seconds for questions. Sorry about that.
So really quickly, if there are any burning questions, Ill repeat the question and try to answer it. Otherwise feel free to come up afterwards and chat.
Q&A
Q: WHY WONT GOOGLE LET YOU USE RUBY?
Yeah, thats a good question. Um... uh... I kinda wrote that up in a blog. Isnt that stupid? "Read my blog!"
The short answer is: it imposes a tax on the systems people, which means that its not completely self-contained within the team thats using it. And for that reason, primarily, its really not a good idea right now.
Any other burning questions?
Q: DO THREADS SUCK?
Well, you know... theyre... you know... Yeah. But I mean, what other options do you have? I mean, you have multiprocessing/share-nothing, which is heavyweight and it requires more work.
So I use threads. Id prefer something better, but theyre what weve got today.
Q: THERE ARE SOME GUYS THAT I WORK WITH, AND ONE OF THEIR COMMENTS ON JAVASCRIPT LATELY, SINCE IVE BEEN WANTING TO USE RHINO BECAUSE I LOVE JAVASCRIPT... WHAT THEY BROUGHT UP IS THAT JAVASCRIPT IS BECOMING A LOT LIKE PYTHON, AND THAT MAY OR MAY NOT BE SUCH A GREAT THING. I WANTED TO KNOW WHAT YOU HAVE TO SAY ABOUT THAT.
Ah. OK. Well, yeah, it already has borrowed some stuff from Python in JavaScript 1.7: yield, Array comprehensions, destructuring assignment. And these are good features. Theyre good. Theyre not going to change it to be syntactically whitespace sensitive, right?
I dont know. The guys working on it have really taken off in completely different directions from Python. Theyre looking at an optional static type system, so in that sense its maybe more like Groovy. And theyre looking at maybe fixing some of the bugs.
But I dont know how thats going to evolve yet. Because theres obviously a lot of people who have skin in the game, a lot of people interested in affecting the way the spec evolves. So its all kind of up in the air right now.
All right, so were really out of time. Id like to thank you for coming. And please come up afterwards. Thanks!

Some guys at Stanford invited me to speak at their EE Computer Systems Colloquium last week. Pretty cool, eh? It was quite an honor. I wound up giving a talk on dynamic languages: the tools, the performance, the history, the religion, everything. It was a lot of fun, and it went over surprisingly well, all things considered.
Theyve uploaded the video of my talk, but since its a full hour, I figured Id transcribe it for those of you who want to just skim it.
This is the first time Ive transcribed a talk. Its tricky to decide how faithful to be to my spoken wording and phrasing. Ive opted to try to make it very faithful, with only minor smoothing.
Unfortunately I wound up using continuation-passing style for many of my arguments: Id occasionally get started on some train of thought, get sidetracked, and return to it two or three times in the talk before I finally completed it. However, Ive left my rambling as-is, modulo a few editors notes, additions and corrections in [brackets].
I didnt transcribe Andys introduction, as it seems immodest to do so. It was funny, though.
Technical corrections are welcome. Im sure I misspoke, oversimplified, over-generalized and even got a few things flat-out wrong. I think the overall message will survive any technical errors on my part.
THE TALK...
Thank you everybody! So the sound guys told me that because of a sound glitch in the recording, my normally deep and manly voice, that you can all hear, is going to come through the recording as this sort of whiny, high-pitched geek, but I assure you thats not what I actually sound like.
So Im going to be talking about dynamic languages. I assume that youre all dynamic language interest... that youve got an interest, because theres a dude down the hall talking about Scala, which is you know, this very strongly typed JVM language (a bunch of you get up and walk over there – exactly.) So you know, presumably all the people who are like really fanatical about strong typing, who would potentially maybe get a little offended about some of the sort of colorful comments I might inadvertently make during this talk — which, by the way, are my own opinions and not Googles — well, well assume theyre all over there.
All right. I assume you all looked through the slides already, so I dont need to spend a whole lot of time with them. Ill go into major rant-mode here at the end. My goal is... for you guys to come away with, sort of a couple of new pictures in your mind, thinking about how languages have evolved over the last 20 years, where theyre going, what we can do to fix them, that kind of thing.
Does anyone here know how to use a Mac? Its showing me this weird, uh... thing... OK. All right. Here goes.
So!
Popular opinion of dynamic languages: slooooow! Theyre always talking about how Python is really slow, right? Python is, what, like 10x to 100x slower? And they have bad tools.
And also theres this sort of, kind of difficult-to-refute one, that says at millions of lines of code, theyre maintenance nightmares, right? Because they dont have static types. That one, uh, unfortunately were not going to be able to talk much about, because not many people have millions-of-lines code bases for us to look at -- because dynamic languages wind up with small code bases. But Ill talk a little bit about it.
So first of all, one of my compatriots here, whos an actual smart person, like probably everybody in this room, youre all waaay smarter than me — I got invited here for the booger jokes, all right? – hes a languages guy, and he said: "You know, you cant talk about dynamic languages without precisely defining what you mean."
So Im going to _precisely_ define it. Dynamic languages are, by definition... Perl, Python, Ruby, JavaScript, Lua, Tcl... all right? [_(laughter)_] Its the working set of languages that people dismiss today as "dynamic languages." Ill also include Smalltalk, Lisp, Self, Prolog, some of our stars, you know, from the 70s and 80s that, uh, well theyll come up here today too.
Im deliberately not going down the path of "well, some static languages have dynamic features, and some dynamic languages have static types", because first of all its this neverending pit of, you know, argument, and second of all, as youre going to see, its completely irrelevant to my talk. The two... sort of qualities that people associate with "dynamic": one would be sort of... runtime features, starting with eval, and the other would be the lack of type tags, the lack of required type tags, or even just escapes in your type system. These things work together to produce the tools problems and the performance problems, ok? And Ill talk about them, and how theyre going to be fixed.
All right!
I just talked about that [slide].
So! Uh... yeah, thats right, Im at Stanford! Forgot about that. So Ive been interviewing for about 20 years, at a whole bunch of companies, and yeah, Stan– every school has this sort of profile, right? You know, the candidates come out with these ideals that their profs have instilled in them. And Stanford has a really interesting one, by and large: that their undergrads and their grad students come out, and they believe that C and C++ are the fabric with which God wove the Universe. OK? And they truly [think]: what is it with all these other languages?
Whereas like MIT and Berkeley, they come out, and theyre like "languages, languages, languages!" and youre like, uh, dude, you actually have to _use_ C and C++, and theyre like "oh." So its funny, the kinds of profiles that come out. But this one [first slide bullet point], I mean, its kind of a funny thing to say, because the guys a Ph.D., and hes just discovered Turings thesis. Of _course_ all you need is C or C++. All you need is a Turing machine, right? You know?
What were talking about here is fundamentally a very personal, a very political, sort of a, its almost a fashion statement about who you are, what kind of language you pick. So, you know... unfortunately we could talk, I mean Ive got 2 hours of ranting in me about this topic, but Im gonna have to, like, kinda like narrow it down to... were gonna talk about dynamic languages because people are out there today using them. Theyre getting stuff done, and it works. All right? And they really do have performance and tools issues.
But theyre getting resolved in really interesting ways. And Im hoping that those of you who are either going out into the industry to start making big things happen, OR, youre researchers, who are going to be publishing the next decades worth of papers on programming language design, will take some interesting directional lessons out of this talk. Well see.
All right. So why are dynamic languages slow? Uh, we all know theyre slow because... theyre dynamic! Because, ah, the dynamic features defeat the compiler. Compilers are this really well understood, you know, really really thoroughly researched... everybody knows THIS [_brandish the Dragon Book_], right?
Compilers! The Dragon Book! From your school! OK? Its a great book. Although interestingly, heh, its funny: if you implement everything in this book, what you wind up with is a really naïve compiler. Its really advanced a long way since... [the book was written] and they know that.
Dynamic languages are slow because all the tricks that compilers can do to try to guess how to generate efficient machine code get completely thrown out the window. Heres one example. C is really fast, because among other things, the compiler can inline function calls. Its gonna use some heuristics, so it doesnt get too much code bloat, but if it sees a function call, it can inline it: it patches it right in, ok, because it knows the address at link time.
C++ — youve got your virtual method dispatch, which is what C++ you know, sort of evangelists, thats the first thing they go after, like in an interview, "tell me how a virtual method table works!" Right? Out of all the features in C++, they care a lot about that one, because its the one they have to pay for at run time, and it drives them nuts! It drives them nuts because the compiler doesnt know, at run time, the receivers type.
If you call
OK. So obviously theres a bunch of ways you can speed up a dynamic language. The number one thing you can do, is you can write a better program. The algorithm, you know, is gonna trump any of the stuff youre doing at the VM – you can optimize the hell out of Bubble Sort, but...
Native threads would be really nice. Perl, Python, Ruby, JavaScript, Lua... _none_ of them has a usable concurrency option right now. None of them. I mean, they kinda have them, but theyre like, Buyer Beware! Dont ever use this on a machine with more than one processor. Or more than one thread. And then youre OK. Its just, you know...
So actually, this is funny, because, all right, show of hands here. Weve all heard this for fifteen years now – is it true? Is Java as fast as C++? Who says yes? All right... weve got a small number of hands... so I assume the rest of you are like, dont know, or it doesnt matter, or "No."
[_Audience member: "We read your slides."_] You read my slides. OK. I dont know... I cant remember what I put in my slides.
But its interesting because C++ is obviously faster for, you know, the short-running [programs], but Java cheated very recently. With multicore! This is actually becoming a huge thorn in the side of all the C++ programmers, including my colleagues at Google, whove written vast amounts of C++ code that doesnt take advantage of multicore. And so the extent to which the cores, you know, the processors become parallel, C++ is gonna fall behind.
Now obviously threads dont scale that well either, right? So the Java people have got a leg up for a while, because you can use ten threads or a hundred threads, but youre not going to use a million threads! Its not going to be Erlang on you all of the sudden. So obviously a better concurrency option – and thats a huge rats nest that Im not going to go into right now – but its gonna be the right way to go.
But for now, Java programs are getting amazing throughput because they can parallelize and they can take advantage of it. They cheated! Right? But threads aside, the JVM has gotten really really fast, and at Google its now widely admitted on the Java side that Javas just as fast as C++. [_(laughter)_]
So! Its interesting, because every once in a while, a C++ programmer, you know, they _flip_: they go over to the Dark Side. Ive seen it happen to some of the most brilliant C++ hackers, I mean theyre computer scientists, but theyre also C++ to the core. And all of a sudden theyre stuck with some, you know, lame JavaScript they had to do as an adjunct to this beautiful backend system they wrote. And they futz around with it for a while, and then all of a sudden this sort of light bulb goes off, and theyre like "Hey, whats up with this? This is way more productive, you know, and it doesnt seem to be as slow as Id sort of envisioned it to be."
And then they maybe do some build scripting in Python, and then all of a sudden they come over to my desk and they ask: "Hey! Can any of these be _fast_?" Ha, ha, ha! I mean, these are the same people that, you know, a year ago Id talk to them and Id say "why not use... _anything_ but C++? Why not use D? Why not use Objective-C? Why not use _anything_ but C++? Right?
Because we all know that C++ has some very serious problems, that organizations, you know, put hundreds of staff years into fixing. Portability across compiler upgrades, across platforms, I mean the list goes on and on and on. C++ is like an evolutionary sort of dead-end. But, you know, its fast, right?
And so you ask them, why not use, like, D? Or Objective-C. And they say, "well, what if theres a garbage collection pause?"
Oooh! [_I mock shudder_] You know, garbage collection – first of all, generational garbage collectors dont have pauses anymore, but second of all, theyre kind of missing the point that theyre still running on an operating system that has to do things like process scheduling and memory management. There _are_ pauses. Its not as if youre running DOS! I hope. OK?
And so, you know, their whole argument is based on these fallacious, you know, sort of almost pseudo-religious... and often its the case that theyre actually based on things that used to be true, but theyre not really true anymore, and were gonna get to some of the interesting ones here.
But mostly what were going to be talking about today is the compilers themselves. Because theyre getting really, really smart.
All right, so first of all Ive gotta give a nod to these languages... which nobody uses. OK? Common Lisp has a bunch of really high-quality compilers. And when they say they achieve, you know, "C-like speed", youve gotta understand, you know, I mean, theres more to it than just "does this benchmark match this benchmark?"
Everybody knows its an ROI [calculation]. Its a tradeoff where youre saying: is it _sufficiently_ fast now that the extra hardware cost for it being 10 or 20 percent slower (or even 2x slower), you know, is outweighed by the productivity gains we get from having dynamic features and expressive languages. Thats of course the rational approach that everyone takes, right?
No! Lisp has all those parentheses. Of course nobodys gonna look at it. I mean, its ridiculous how people think about these things.
But with that said, these were actually very good languages. And let me tell you something thats NOT in the slides, for all those of you who read them in advance, OK? This is my probably completely wrong... its certainly over-generalized, but its a partly true take on what happened to languages and language research and language implementations over the last, say 30 years.
There was a period where they were kind of neck and neck, dynamic and static, you know, there were Fortran and Lisp, you know, and then there was a period where dynamic languages really flourished. They really took off. I mean, Im talking about the research papers, you can look: theres paper after paper, proofs...
And implementations! StrongTalk was really interesting. They added a static type system, an optional static type system on top of Smalltalk that sped it up like 20x, or maybe it was 12x. But, you know, this is a prototype compiler that never even made it into production. Youve gotta understand that when a researcher does a prototype, right, that comes within, you know, fifty percent of the speed gains you can achieve from a production compiler... because they havent done a tenth, a hundredth of the optimizations that you _could_ do if you were in the industry cranking interns through the problem, right?
I mean HotSpots VM, its got like ten years of Suns implementation into not one, but two compilers in HotSpot, which is a problem theyre trying to address. So were talking about, you know, a 12x gain really translates to something a lot larger than that when you put it into practice.
In case I forget to mention it, all these compiler optimizations Im talking about, I do mean _all_ of them, are composable. Which is really important. Its not like you have to choose this way or you have to choose that way. Theyre composable, which means they actually reinforce each other. So God only knows how fast these things can get.
This is the only interesting... this is actually the only, I would say, probably _original_, sort of compelling thought for this talk today. I really – I started to believe this about a week ago. All right? Because its an urban legend [that they change every decade]. You know how theres Moores Law, and there are all these conjectures in our industry that involve, you know, how things work. And one of them is that languages get replaced every ten years.
Because thats what was happening up until like 1995. But the barriers to adoption are really high. One that I didnt put on the slide here, I mean obviously theres the marketing, you know, and theres the open-source code base, and there are legacy code bases.
Theres also, there are also a lot more programmers, I mean many more, orders of magnitude more, around the world today than there were in 1995. Remember, the dot-com boom made everybody go: "Oooh, I wanna be in Computer Science, right? Or I just wanna learn Python and go hack." OK? Either way. (The Python hackers probably made a lot more money.)
But what we wound up with was a bunch of entry-level programmers all around the world who know _one_ language, whichever one it is, and they dont want to switch. Switching languages: the second one is your hardest. Because the first one was hard, and you think the second ones going to be that bad, and that you wasted the entire investment you put into learning the first one.
So, by and large, programmers – you know, the rank-and-file – they pretty much pick a language and they stay with it for their entire career. And that is why weve got this situation where now, this... See, theres plenty of great languages out there today. OK?
I mean obviously you can start with Squeak, sort of the latest Smalltalk fork, and its beautiful. Or you can talk about various Lisp implementations out there that are smokin fast, or theyre smokin good. Or in one or two cases, both.
But also theres, like, the Boo language, the io language, theres the Scala language, you know, I mean theres Nice, and Pizza, have you guys heard about these ones? I mean theres a bunch of good languages out there, right? Some of them are really good dynamically typed languages. Some of them are, you know, strongly [statically] typed. And some are hybrids, which I personally really like.
And nobodys using _any_ of them!
Now, I mean, Scala might have a chance. Theres a guy giving a talk right down the hall about it, the inventor of – one of the inventors of Scala. And I think its a great language and I wish him all the success in the world. Because it would be nice to have, you know, it would be nice to have that as an alternative to Java.
But when youre out in the industry, you _cant_. You get lynched for trying to use a language that the other engineers dont know. Trust me. Ive tried it. I dont know how many of you guys here have actually been out in the industry, but I was talking about this with my intern. I was, and I think you [_(point to audience member)_] said this in the beginning: this is 80% politics and 20% technology, right? You know.
And [my intern] is, like, "well I understand the argument" and Im like "No, no, no! Youve never been in a company where theres an engineer with a Computer Science degree and ten years of experience, an architect, whos in your face _screaming_ at you, with spittle flying on you, because you suggested using, you know... D. Or Haskell. Or Lisp, or Erlang, or take your pick."
In fact, Ill tell you a funny story. So this... at Google, when I first got there, I was all idealistic. Im like, wow, well Google hires all these great computer scientists, and so they must all be completely language-agnostic, and ha, ha, little do I know... So Im up there, and Im like, weve got this product, this totally research-y prototype type thing, we dont know. We want to put some quick-turnaround kind of work into it.
But Google is really good at building infrastructure for _scaling_. And I mean scaling to, you know, how many gazillion transactions per second or queries per second, you know, whatever. They scale like nobodys business, but their "Hello, World" takes three days to get through. At least it did when I first got to Google. They were _not_ built for rapid prototyping, OK?
So that means when you try to do what Eric Schmidt talks about and try to generate luck, by having a whole bunch of initiatives, some of which will get lucky, right? Everybodys stuck trying to scale it from the ground up. And that was unacceptable to me, so I tried to... I made the famously, horribly, career-shatteringly bad mistake of trying to use Ruby at Google, for this project.
And I became, very quickly, I mean almost overnight, the Most Hated Person At Google. And, uh, and Id have arguments with people about it, and theyd be like Nooooooo, WHAT IF... And ultimately, you know, ultimately they actually convinced me that they were right, in the sense that there actually _were_ a few things. There were some taxes that I was imposing on the systems people, where they were gonna have to have some maintenance issues that they wouldnt have [otherwise had]. Those reasons I thought were good ones.
But when I was going through this debate, I actually talked to our VP Alan Eustace, who came up to a visit to Kirkland. And I was like, "Alan!" (after his talk) "Lets say, hypothetically, weve got this team who are really smart people..."
And I point to my friend Barry [pretending its him], and Im like: "Lets say they want to do something in a programming language thats not one of the supported Google languages. You know, like what if they wanted to use, you know, Haskell?"
What I really wanted to do at the time was use Lisp, actually, but I didnt say it. And [Alan] goes, "Well!" He says, "Well... how would _you_ feel if there was a team out there who said they were gonna use... LISP!" [_(laughter)_]
Hed pulled his ace out of his [sleeve], and brandished it at me, and I went: "thats what I wanted to use." And he goes, "Oh." [_(turning away quickly)_] And that was the end of the conversation. [_(laughter)_]
But you know, ultimately, and it comes up all the time, I mean weve got a bunch of famous Lisp people, and (obviously) famous Python people, and you know, famous language people inside of Google, and of course theyd like to do some experimentation. But, you know, Googles all about getting stuff done.
So that brings us full circle back to the point of this topic, which is: the languages we have today, sorted by popularity at this instant, are probably going to stay about that popular for the next ten years.
Sad, isnt it? Very, very sad. But thats the way it is.
So how do we fix them?
How – how am I doing for time? Probably done, huh? Fifteen minutes? [_(audience member: no, more than that)_] OK, good.
So! Im gonna talk a little bit about tools, because one interesting thing I noticed when I was putting this thing together, right, was that the ways you solve tools problems for dynamic languages are very similar to the way you solve perf problems. OK? And Im not going to try to keep you guessing or anything. Ill tell you what the sort of... kernel of the idea is here.
Its that... the notion of "static" versus "dynamic", where you kind of have to do all these optimizations and all these computations statically, on a language, is very old-fashioned. OK? And increasingly its becoming obvious to _everybody_, you know, even the C++ crowd, that you get a lot better information at run-time. *Much* better information.
In particular, let me come back to my inlining example. Java inlines polymorphic methods! Now the simplest way to do it was actually invented here at Stanford by Googler Urs Hoelzle, whos, you know, like VP and Fellow there, and its called, its now called Polymorphic Inline Caching. He called it, uh, type-feedback compilation, I believe is what he called it. Great paper. And it scared everybody, apparently. The rumors on the mailing lists were that people were terrified of it, I mean it seems too hard. And if you look at it now, youre like, dang, that was a good idea.
All it is, I mean, I told you the compiler doesnt know the receiver type, right? But the thing is, in computing, I mean, heuristics work pretty well. The whole 80/20 rule and the Power Law apply pretty much unilaterally across the board. So you can make assumptions like: the first time through a loop, if a particular variable is a specific instance of a type, then its probably going to be [the same type] on the remaining iterations of the loop. OK?
So what he [Urs] does, is he has these counters at hot spots in the code, in the VM. And they come in and they check the types of the arguments [or operands]. And they say, all right, it looks like a bunch of them appear to be class B, where we thought it might be class A.
So what were gonna do is generate this fall-through code that says, all right, if its a B – so they have to put the guard instruction in there; it has to be _correct_: it has to handle the case where theyre wrong, OK? But they can make the guard instruction very, very fast, effectively one instruction, depending on how you do it. You can compare the address of the intended method, or you can maybe do a type-tag comparison. There are different ways to do it, but its fast, and more importantly, if its _right_, which it is 80-90% of the time, it falls through [_i.e., inlines the method for that type - Ed._], which means you maintain your processor pipeline and all that stuff.
So it means they have _predicted_ the type of the receiver. Theyve successfully inlined that. I mean, you can do a whole bunch of branching, and they actually found out through some experimentation that you only need to do 2 to 4 of these, right, before the gain completely tails off. So you dont have to generate too much of this. And theyve expanded on this idea now, for the last ten years.
Getting back to my point about whats happening [over the past 30 years], there was an AI winter. You all remember the AI winter, right? Where, like, investors were pumping millions of dollars into Smalltalk and Lisp companies who were promising theyd cure world hunger, cure cancer, and everything?
And unfortunately they were using determinism!
Theyre using heuristics, OK, but you know... before I came to Google, you know, I was really fascinated by something Peter Norvig was saying. He was saying that they dont do natural language processing deterministically any more. You know, like maybe, conceivably, speculating here, Microsoft Words grammar checker does it, where youd have a Chomsky grammar, right? And youre actually going in and youre doing something like a compiler does, trying to derive the sentence structure. And you know, whatever your output is, whether its translation or grammar checking or whatever...
None of that worked! It all became way too computationally expensive, plus the languages kept changing, and the idioms and all that. Instead, [Peter was saying] they do it all probablistically.
Now historically, every time you came along, and you just obsoleted a decade of research by saying, "Well, were just gonna kind of wing it, probabilistically" — and you know, Peter Norvig was saying they get these big data sets of documents that have been translated, in a whole bunch of different languages, and they run a bunch of machine learning over it, and they can actually match your sentence in there to one with a high probability of it being this translation.
And its usually right! It certainly works a lot better than deterministic methods, and its computationally a lot cheaper.
OK, so whenever you do that, it makes people MAD.
Their first instinct is to say "nuh-UUUUUUH!!!!" Right? Im serious! Im serious. It happened when John von Neumann [and others] introduced Monte Carlo methods. Everyone was like "arrgggggh", but eventually they come around to it. They go "yeah, I guess youre right; Ill go back and hit the math books again."
Its happening in programming languages _today_. I mean, as we speak. I mean, theres a paper Im gonna tell you about, from October, and its basically coming along and... its not _really_ machine learning, but youre gonna see its the same kind of [data-driven] thing, right? Its this "winging it" approach thats actually much cheaper to compute. And it has much better results, because the runtime has all the information.
So let me just finish the tools really quick.
And Im not talking to you guys; Im talking to the people in the screen [i.e. watching the recording] – all these conversations Ive had with people who say: "No type tags means no information!" I mean, effectively thats what theyre saying.
I mean...
function foo(a, b) { return a + b; }
var bar = 17.6;
var x = {a: "hi", b: "there"}; Whats
OK, so... then you get into dynamic languages. This [code] is all JavaScript. This is actually something Im working on right now. Im trying to build this JavaScript code graph, and you actually have to know all these tricks. And of course its undecidable, right, I mean this is, you know, somebody could be defining a function at the console, and Im not gonna be able to find that.
So at some point youve gotta kind of draw the line. What you do is, you look at your corpus, your code base, and see what are the common idioms that people are using. In JavaScript, youve got a couple of big standard libraries that everybody seems to be including these days, and they all have their slightly different ways of doing function definitions. Some of them use Object literals; some of them use the horrible
Yeah, right here! And I say,
Another big point that people miss is that the Java IDEs, you know, that are supposedly always right? Theyre wrong. If you miss _one time_, youre wrong. Right? In Java Reflection, obviously, the IDE has no information about whats going on in that string, by definition. Its a string: its quoted; its opaque.
And so they always wave their hands and say "Ohhhhh, you cant do Rename Method!"
Even though Rename Method came from the Smalltalk environment, of course, right? And you say, "It came from the Smalltalk environment, so yes, you can do Rename Method in dynamic languages."
And they say "NO! Because itll miss sometimes!"
To which, I say to you people in the screen, youd be _astonished_ at how often the Java IDEs miss. They miss every single instance of a method name that shows up in an XML configuration file, in a reflection layer, in a database persistence layer where youre matching column names to fields in your classes. Every time youve deployed some code to some people out in the field...
Rename Method only works in a small set of cases. These Refactoring tools that, really, theyre acting are like the Holy Grail, you can do ALL of that in dynamic languages. Thats the proof, right? [_I.e., static langs miss as often as dynamic - Ed._]
Its not even a very interesting topic, except that I just run across it all the time. Because you ask people, "hey, you say that youre ten times as productive in Python as in your other language... why arent you using Python?"
Slow? Admittedly, well, well get to that.
And tools. Admittedly. But I think whats happened here is Java has kind of shown the new crop of programmers what Smalltalk showed us back in the 80s, which is that IDEs can work and they can be beautiful.
And _more importantly_ – and this isnt in the slides either, for those of you who cheated – they _have_ to be tied to the runtime. They complain, you know, the Java people are like "Well you have to have all the code loaded into the IDE. Thats not scalable, its not flexible, they cant simulate the program just to be able to get it correct."
And yet: any sufficiently large Java or C++ system has health checks, monitoring, it opens sockets with listeners so you can ping it programmatically; you can get, you know, debuggers, you can get remote debuggers attached to it; its got logging, its got profiling... its got this long list of things that you need because the static type system failed.
OK... Why did we have the static type system in the first place?
Let me tell you guys a story that, even if you know all this stuff, is still going to shock you. I credit Bob Jervis for sharing this with me (the guy who wrote Turbo C.)
So javac, the Java compiler: what does it do? Well, it generates bytecode, does some optimizations presumably, and maybe tells you some errors. And then you ship it off to the JVM. And what happens to that bytecode? First thing that happens is they build a tree out of it, because the bytecode verifier has to go in and make sure youre not doing anything [illegal]. And of course you cant do it from a stream of bytes: it has to build a usable representation. So it effectively rebuilds the source code that you went to all that effort to put into bytecode.
But thats not the end of it, because maybe javac did some optimizations, using the old Dragon Book. Maybe it did some constant propagation, maybe it did some loop unrolling, whatever.
The next thing that happens in the JVM is the JIT undoes all the optimizations! Why? So it can do _better_ ones because it has runtime information.
So it undoes all the work that javac did, except maybe tell you that you had a parse error.
And the weird thing is, Java keeps piling... Im getting into rant-mode here, I can tell. Were never going to make it to the end of these slides. Java keeps piling syntax on, you know, but its not making the language more expressive. What theyre doing is theyre adding red tape and bureacracy for stuff you could do back in Java 1.0.
In Java 1.0, when you pulled a String out of a Hashtable you had to cast it as a String, which was really stupid because you said
String foo = (String) hash.get(...) You know, its like... if you had to pick a syntax [for casting], you should at least pick one that specifies what you think its supposed to be, not what its becoming - _obviously_ becoming - on the left side, right?
And everybody was like, "I dont like casting! I dont like casting!" So what did they do? What they _could_ have done is they could have said, "All right, you dont have to cast anymore. We know what kind of variable youre trying to put it in. Well cast it, and [maybe] youll get a
OK. So! Moving right back along to our _simple_ dynamic languages, the lesson is: its not actually harder to build these tools [for dynamic languages]. Its different. And nobodys done the work yet, although people are starting to. And actually IntelliJ is a company with this IDEA [IDE], and they... my friends show off the JavaScript tool, you know, and its like, man! They should do one for Python, and they should do one for every single dynamic language out there, because they kick butt at it. Im sure they did all this stuff and more than Im talking about here.
All right. Now we can talk about perf. This is the Crown Jewels of the talk. Yeah. So... unfortunately I have to make the disclaimer that everybody thinks about performance wrong, except for you guys cuz you all know, right? But seriously, I mean, you know, you understand, I started out of school... _*sigh*_
OK: I went to the University of Washington and [then] I got hired by this company called Geoworks, doing assembly-language programming, and I did it for _five years_. To us, the Geoworkers, we wrote a whole operating system, the libraries, drivers, apps, you know: a desktop operating system in assembly. 8086 assembly! It wasnt even good assembly! We had four registers! [Plus the] si [register] if you counted, you know, if you counted 386, right? It was _horrible_.
I mean, actually we kind of liked it. It was Object-Oriented Assembly. Its amazing what you can talk yourself into liking, which is the real irony of all this. And to us, C++ was the ultimate in Roman decadence. I mean, it was equivalent to going and vomiting so you could eat more. They had IF! We had jump CX zero! Right? They had "Objects". Well we did too, but I mean they had syntax for it, right? I mean it was all just such weeniness. And we knew that we could outperform any compiler out there because at the time, we could!
So what happened? Well, they went bankrupt. Why? Now Im probably disagreeing – I know for a fact that Im disagreeing with every Geoworker out there. Im the only one that holds this belief. But its because we wrote fifteen million lines of 8086 assembly language. We had really good tools, world class tools: trust me, you need em. But at some point, man...
The problem is, picture an ant walking across your garage floor, trying to make a straight line of it. It aint gonna make a straight line. And you know this because you have _perspective_. You can see the ant walking around, going hee hee hee, look at him locally optimize for that rock, and now hes going off this way, right?
This is what we were, when we were writing this giant assembly-language system. Because what happened was, Microsoft eventually released a platform for mobile devices that was much faster than ours. OK? And I started going in with my debugger, going, what? What is up with this? This rendering is just really slow, its like sluggish, you know. And I went in and found out that some title bar was getting rendered 140 times every time you refreshed the screen. It wasnt just the title bar. Everything was getting called multiple times.
Because we couldnt see how the system worked anymore!
Small systems are not only _easier_ to optimize, theyre _possible_ to optimize. And I mean globally optimize.
So when we talk about performance, its all crap. The most important thing is that you have a small system. And then the performance will just fall out of it naturally.
That said, all else being equal, lets just pretend that Java can make small systems. Heh, thats a real stretch, I know. Lets talk about actual optimization.
And by the way, here are some real examples, sort of like the Geoworks one, where a slower language wound up with a faster system. Its not just me. Ive seen it all over the place. Do you know why this one happened? Why was the Ruby on Rails faster than Struts? This started one of the internets largest flamewars since Richard Stallman dissed Tcl back in the 80s, you know. You guys remember that? [_(laughter)_]
I mean, the Java people went _nuts_, I mean really really nuts, I mean like angry Orcs, they were just like AAAaaaaauuuugh, they did NOT want to hear it. OK? It was because they were serializing everything to and from XML because Java cant do declarations. Thats why. Thats the reason. I mean, stupid reasons, but performance comes from some strange places.
That said, OK, disclaimers out of the way...
Yeah yeah, people are using them.
Um, yeah. So JavaScript. JavaScript has been really interesting to me lately, because JavaScript actually does care about performance. Theyre the first of the modern dynamic languages where performance has become an issue not just for the industry at large, but also increasingly for academia.
Why JavaScript? Well, it was Ajax. See, what happened was... Lemme tell ya how it was supposed to be. JavaScript was going away. It doesnt matter whether you were Sun or Microsoft or anybody, right? JavaScript was going away, and it was gonna get replaced with... heh. Whatever your favorite language was.
I mean, it wasnt actually the same for everybody. It might have been C#, it might have been Java, it might have been some new language, but it was going to be a _modern_ language. A fast language. It was gonna be a scalable language, in the sense of large-scale engineering. Building desktop apps. Thats the way it was gonna be.
The way its _really_ gonna be, is JavaScript is gonna become one of the smokin-est fast languages out there. And I mean _smokin_ fast.
Now its not the only one thats making this claim. Theres actually a lot of other... you guys know about PyPy? Python in Python? Those crack fiends say they can get C-like performance. Come on... COME ON! They... I mean, seriously! Thats what they say.
Heres the deal, right? Theyre saying it because theyre throwing all the old assumptions out. They can get this performance by using these techniques here, fundamentally. But if nobody believes them, then even when they achieve this performance its not gonna matter because still nobodys gonna believe them, so all of this stuff were talking about is a little bit moot.
Nevertheless, Im going to tell you about some of the stuff that I know about thats going on in JavaScript.
So type inference. You _can_ do type inference. Except that its lame, because it doesnt handle weird dynamic features like upconverting integers to Doubles when they overflow. Which JavaScript does, interestingly enough, which is I guess better behavior than... I mean, it still overflows eventually, right?
We overflowed a
So... oh yeah, I already talked about Polymorphic Inline Caches. Great! I already talked about a lot of this stuff.
This ones really cool. This is a trick that somebody came up with, that you can actually – theres a paper on it, where you can actually figure out the actual types of any data object in any dynamic language: figure it out the first time through by using this double virtual method lookup. Theyve boxed these things. And then you just expect it to be the same the rest of the time through [the loop], and so all this stuff about having a type-tag saying this is an
This is the really cool one. This is the really, really cool one. Trace trees. This is a paper that came out in October. This is the one, actually... Ill be honest with you, I actually have two optimizations that couldnt go into this talk that are even cooler than this because they havent published yet. And I didnt want to let the cat out of the bag before they published. So this is actually just the tip of the iceberg.
But trace trees, its a really simple idea. What you do is your runtime, your VM, you know, its interpreting instructions and can count them. Well, it can also record them! So any time it hits, basically, a branch backwards, which usually means its going to the beginning of a loop, which usually means its going to be a hot spot, especially if youre putting a counter there... Obviously [in] the inner loops, the hot spots will get the highest counts, and they get triggered at a certain level.
It turns on a recorder. Thats all it does. It starts recording instructions. It doesnt care about loop boundaries. It doesnt care about methods. It doesnt care about modules. It just cares about "What are you executing?"
And it records these tree – well actually, traces, until they get back to that point. And it uses some heuristics to throw stuff away if it goes too long or whatever. But it records right through methods. And instead of setting up the activation, it just inlines it as it goes. Inline, inline, inline, right? So theyre big traces, but theyre known to be hot spots.
And even here in the Dragon Book, Aho, Sethi and Ullman, they say, you know, one of the most important things a compiler can do is try to identify what the hot spots are going to be so it can make them efficient. Because who cares if youre optimizing the function that gets executed once at startup, right?
So these traces wind up being trees, because what can happen is, they branch any time an operand is a different type. Thats how they handle the overflow to Double: therell be a branch. They wind up with these trees. Theyve still got a few little technical issues like, for example, growing exponentially on the Game of Life. Theres a blog about it, um... Im sorry, Ive completely forgotten his name [_Andreas Gal_], but I will blog this. And the guy thats doing these trace trees, he got feedback saying that theyve got exponential growth.
So they came up with this novel way of folding the trace trees, right, so there are code paths that are almost identical and they can share, right?
Its all the same kind of stuff they were doing with _these_ [Dragon Book] data structures back when they were building static compilers. We are at the very beginning of this research! What has happened is, weve gone from Dynamic [to] AI Winter... dynamic research stopped, and anybody who was doing it was sort of anathema in the whole academic [community]... worldwide across all the universities. There were a couple of holdouts. [Dan Friedman and] Matthias Felleisen, right, the Little Schemer guys, right? Holding out hope.
And everybody else went and chased static. And theyve been doing it like crazy. And theyve, in my opinion, reached the theoretical bounds of what they can deliver, and it has FAILED. These static type systems, theyre WRONG. Wrong in the sense that when you try to do something, and they say: No, category theory doesnt allow that, because its not elegant... Hey man: whos wrong? The person whos trying to write the program, or the type system?
And some of the type errors you see in these Hindley-Milner type [systems], or any type system, like "expected (int * int * int)", you know, a tuple, and "but got (int * int * int)", you know [_(clapping my hands to my head)_] its pretty bad, right? I mean, theyve, I think theyve failed. Which is why theyre not getting adopted.
Now of course thats really controversial. There are probably a bunch of type-systems researchers here who are really mad, but...
Whats happening is: as of this Ajax revolution, the industry shifted to trying to optimize JavaScript. And that has triggered what is going to be a landslide of research in optimizing dynamic languages.
So these tricks Im telling you about, theyre just the beginning of it. And if we come out of this talk with one thing, its that its cool to optimize dynamic languages again! "Cool" in the sense of getting venture funding, right? You know, and research grants... "Cool" in the sense of making meaningful differences to all those people writing Super Mario clones in JavaScript.
You know. Its _cool_.
And so I encourage you, if youre a language-savvy kind of person, to jump in and try to help. Me, Im probably going to be doing grunt implementations, since Im not that smart.
And I dont even need to talk about this [last optimization — Escape Analysis], since you already knew it.
All right! So thats it. Thats my talk. CPUs... you get all the good information about how a program is running _at run time_. And this has huge implications for the tools and for the performance. Its going to change the way we work. Its eventually – God, I hope sooner rather than later – going to obsolete C++ finally.
Its going to be a lot of work, right?
And then, when we finish, nobodys going to use it. [_(laughter)_] Because, you know. Because thats just how people are.
Thats my talk! Thanks. [_(applause)_]
Questions? No questions? I think were out of time, right? [_(audience: no, we have half an hour)_]
Q: WHATS YOUR DEFINITION OF MARKETING?
Hey man, Im doing it _right now_. [_(laughter)_]
I am! In a sense, right? I mean, like, Perl was a marketing success, right? But it didnt have Sun or Microsoft or somebody hyping it. It had, you know, the guy in the cube next to you saying "Hey, check out this Perl. I know youre using Awk, but Perls, like, weirder!"
The marketing can happen in any way that gets this message across, this meme out to everybody, in the Richard Dawkins sense. Thats marketing. And it starts from just telling people: hey, its out there.
Q: DO YOU SEE ANY OF THIS STUFF STARTING TO MOVE INTO MICROPROCESSORS OR INSTRUCTIONS?
Ah! I knew somebody was going to ask that. So unfortunately, the JITs that are doing all these cool code optimizations could potentially be running into these weird impedance mismatches with microprocessors that are doing their own sets of optimizations. I know _nothing_ about this except that its... probably gonna happen. And, uh, God I hope they talk to each other. [_Editors note: after the talk, I heard that trace trees started life in hardware, at HP._]
Q: YOU COULD IMAGINE CMS (?) PULLING ALL THESE STUNTS AND LOOKING AT STUFF AND SAYING, "OH, I KNOW THAT THIS IS JUST MACHINE LANGUAGE... OH, LOOK! THATS AN INT, AND..."
Yes. I do know... that theres a compiler now that compiles [machine code] into microcode, a JIT, you know, I was reading about it.
Q: SO ONE PROBLEM WITH PERFORMANCE IS THAT ITS NOT JUST FAST PERFORMANCE VS. SLOW PERFORMANCE. WHAT THEYRE HAVING A LOT OF TROUBLE WITH IS THAT A FUNCTION ONE TIME TAKES A MILLISECOND OR A MICROSECOND, AND ANOTHER TIME IT TAKES 300 OR 500 OR 1000 TIMES LONGER. [_PART OF QUESTION MUTED_] ANY THOUGHTS ON HOW TO IMPROVE THE PERFORMANCE PREDICTABILITY OF DYNAMIC LANGUAGES?
Yeah... _*sigh*_. Well, I think for the forseeable future, I mean honestly having talked to several of the VM implementers, theyre not making any claims that JavaScripts going to be as fast as C any time soon. Not for the forseeable future. Its going to be very fast, right, but its not going to be quite... theyre not going to make the crazy promises that Sun did.
Which means that these dynamic speedups are primarily going to be useful in long-running distributed processes, for which a little glitch now and then isnt going to matter in the grand scheme of the computation. Or, theyre going to be, you know, the harder one is in clients, where youve got a browser app, and youre hoping that the glitch youre talking about isnt on the order of hundreds of milliseconds.
Generational garbage collectors is the best answer Ive got for that, because it reduces the pauses, and frankly, the garbage collectors for all the [new] dynamic languages today are crap. Theyre mark-and-sweep, or theyre reference counted. Theyve got to fix that. Right out of the starting gate, thats gonna nullify 80 percent of that argument.
For the rest, I dont know. Its up to you guys.
Q: YOU ALSO HAVE TO LOOK AT YOUR STORAGE MANAGEMENT; YOU HAVE TO UNDERSTAND YOUR STORAGE IN YOUR PROGRAM, AND HAVE SOME SORT OF BETTER CONTROL OVER THAT...
Thats right. As you point out, its domain-specific. If your bottleneck is your database, all bets are off.
Q: YOU SEEM TO BE EQUATING DYNAMIC LANGUAGE WITH DYNAMIC ENCODING, THAT YOU HAVE "DYNAMIC LANGUAGE EQUALS JIT"."
For this talk, yes. [_(laughter)_]
Q: ...BUT THE SAME THING CAN BE DONE FOR STATIC LANGUAGES.
Yeah, absolutely!
Q: ...AND AS SOON AS THE MARKETING STARTS GETTING SOME MARKET PENETRATION, THE C++ PEOPLE WILL JUST SIMPLY COME AROUND AND SAY, "YOU CAN HAVE MAINTAINABILITY _AND_ PERFORMANCE".
Yep! They can, actually. Thats what theyll say. And Ill say: "All right. Ill give you a thousand dollars when youre done." OK? Because C++ have actually shot themselves in their own foot. By adding so many performance hacks into the language, and also actual features into the language for performance, like pointer manipulation, the language itself is large enough that its very difficult. Its much more difficult to envision doing a JIT that can handle pointers properly, for example, right? You can do it! Its just a _lot_ more work than it is for these simple languages. [_In retrospect, Im not so sure about this claim. Trace trees may not care about pointers, so maybe it wouldnt be that hard? Of course, theyd have to move to a JIT first, requiring an initial slowdown, so itd probably never happen. -Ed._]
So theyre winding up... theyre winding up in a situation where theyre gonna have to weigh it carefully, and say, "OK: when all is said and done, is my language actually gonna be faster than these other languages that have gone through this process already?" Because now were on a more level playing field.
Especially as its getting increasingly difficult to predict exactly what the hardware architecture is going to be, and those mismatches tend to have a huge impact on what the JIT actually can do. I mean, hardwares getting really out there now, and the compiler writers are still trying to figure out what to do about it. I mean, even the stuff theyre doing in the JITs today might not apply tomorrow.
So I realize its a weak answer, but Im saying, you know, its a hard proposition for me to imagine them doing. Theyll try! Maybe theyll succeed.
Q: THE OTHER SIDE OF DYNAMIC LANGUAGES IS SECOND-ORDER SYSTEMS: THE ABILITY TO DO AN EVAL. AND THE DIFFICULTY WITH THAT IS INTELLECTUAL TRACTABILITY. MOST PEOPLE USE SECOND-ORDER LANGUAGES TO WRITE FIRST-ORDER PROGRAMS. IS THERE ANY REAL REASON TO EVEN HAVE A SECOND-ORDER LANGUAGE FOR WRITING COBOL?
Can they hear these questions in the audio? Because this is a really good point.
So this is, I mean, I dont know the answer to this. This is a hard question, OK?
Java has kind of gotten there without even having eval. Theyve tiered themselves into sort of second-order people who know how to manipulate the type-theory stuff, you know, right? People go off to them with batch requests: "Please write me a type expression that meets my needs". And it comes back. So were already in sort of the same situation we were in with hygienic Scheme macros, you know, or with any sort of macro system, or any eval system. Which is that really only a few people can be trusted to do it well, and everybody else kind of has to... right?
So I dont know, maybe its just a cultural... maybe its already solved, and we just have to live with the fact that some programming languages are going to have dark corners that only a few people know. Its unfortunate. Its ugly.
[_My PhD languages intern, Roshan James, replies to the questioner: Your usage of the phrase second-order, where does that come from? A comment as to what youre telling us, which is that some sort of phased evaluation, specific to Scheme at least, were saying... some would say the complexity of writing Scheme macros is roughly on the order of writing a complex higher-order procedure. Its not much harder. A well thought out macro system is not a hard thing to use._]
...says the Ph.D. languages intern! Hes representing the majority viewpoint, of course. [_(laughter)_] Ill tell you what: Ill let you two guys duke it out after the talk, because I want to make sure we get through anybody elses questions.
Q: YOURE ASSUMING YOU HAVE A GARBAGE-COLLECTED ENVIRONMENT. WHAT DO YOU THINK ABOUT REFERENCE COUNTING, WITH APPROPRIATE OPTIMIZATION?
Ummmm... No. [_(laughter)_] I mean, come on. Garbage collection... you guys know that, like, its faster in Java to allocate an object than it is in C++? Theyve got it down to, like, three instructions on some devices, is that right? And the way the generational garbage collector works is that 90% of the objects get reused. Plus theres fine-grained interleaving with the way the memory model of the operating system works, to make sure theyre dealing with issues with, whaddaya call it, holes in the heap, where you cant allocate, I mean theres a whole bunch of stuff going on. [_This was me doing my best moron impersonation. Sigh. -Ed._]
So, like, it works. So why throw the extra burden on the programmer? Even [in] C++, by the way, Stroustroup wants to add garbage collection!
Q: IF YOU BELIEVE YOUR OTHER ARGUMENTS, YOU CAN DO THE REFERENCE COUNTING OR LOCAL POOLING, AND POINT OUT WHEN ITS ACTUALLY WRONG.
Right. The philosophical answer to you guys is: compilers will eventually get smart enough to deal with these problems better than a programmer can. This has happened time and again. [For instance] compilers generate better assembly code [than programmers do].
All the "tricks" that you learned to optimize your Java code, like marking everything final, so that the compiler can inline it – the VM does that for you now! And it puts [in] some
foo.bar(), foo could be some class that C++ knows about, or it could be some class that got loaded in afterwards. And so it winds up -- this polymorphism winds up meaning the compiler can compile both the caller and the callee, but it cant compile them together. So you get all the overhead of a function call. Plus, you know, the method lookup. Which is more than just the instructions involved. Youre also blowing your instruction cache, and youre messing with all these, potentially, code optimizations that could be happening if it were one basic-block fall-through.
All right. Please – feel free to stop me or ask questions if I say something thats unclear. I know, just looking around the room, that most of you probably know this stuff better than I do.
So! The last [bullet point] is really interesting. Because nobody has tried, for this latest crop of languages, to optimize them. Theyre _scripting languages_, right? They were actually designed to either script some host environment like a browser, or to script Unix. I mean the goal was to perform these sort of I/O-bound computations; there was no point in making them fast. Except when people started trying to build larger and larger systems with them: thats when speed really started becoming an issue.
foo? Its a _function_. How did I know that? [_(laughter)_] Whats bar? Whats x? You know, its a composite type. Its an Object. It has two fields that are strings. Call it a record, call it a tuple, call it whatever you want: we know what it is.
The syntax of a language, unless its Scheme, gives you a lot of clues about the semantics, right? Thats actually the one place, maybe, where lots of syntax actually wins out [over Scheme]. I just thought of that. Huh.
with statement, you know, that JavaScript people hate.
But your compiler can figure all these out. And I was actually going through this Dragon Book, because they can even handle aliasing, right? Your IDE for JavaScript, if I say "var x = some object", and you know...
Did I handle this here [in the slides]?
foo is an object, x is foo, and I have an alias now. The algorithm for doing this is right here in the Dragon Book. Its data-flow analysis. Now they use it for compiler optimization to do, you know, live variable analysis, register allocation, dead-code elimination, you know, the list kind of goes on. Its a very useful technique. You build this big code graph of basic blocks...
So its actually one of the few static-analysis thats actually carrying over in this new dynamic world where we have all this extra information. But you can actually use it in JavaScript to figure out function declarations that didnt actually get declared until way later in the code.
ClassCastException."
Instead, they introduced generics, right, which is this huge, massive, category-theoretic type system that they brought in, where you have to under[stand] – to actually use it you have to know the difference between covariant and contravariant return [and argument] types, and you have to understand why every single mathematical... [_I tail off in strangled frustration..._]
And then what happens on mailing lists is users say: "So Im trying to do X." And they say: "WELL, for the following category-theoretic reasons ...theres no way to do it." And they go: "Oh! Oh. Then Im gonna go use JavaScript, then." Right?
I mean, its like, what the hell did this type system do for Java? It introduced inertia and complexity to everybody whos writing tools, to everybody whos writing compilers, to everybody whos writing runtimes, and to everybody whos writing code. And it didnt make the language more expressive.
So whats happening? Java 7 is happening. And I encourage you all to go look at _that_ train wreck, because oh my God. Oh, God. I didnt sleep last night. Im all wired right now because I looked at Java 7 last night. And it was a _mistake_. [_(laughter)_] Ohhh...
long at Google once. Nobody thought that was possible, but it actually happened. Ill tell you about that later if you want to know.
int – which might not actually be technically correct, if youre going to overflow into a Double, right? Or maybe youre using an int but what youre really using is a bytes worth of it, you know. The runtime can actually figure things out around bounds that are undecidable at compile time.
So thats a cool one.
ClassLoader hooks to see if you load a class that makes it non-final, and then [if the assumption is invalidated later] it undoes the optimization and pulls the inlining out.
Thats how smart the VM is right now. OK? You only need a few compiler writers to go out there and obsolete _all_ the tricks that you learned. All the memory-pooling tricks...
Hell, you guys remember StringBuffer and StringBuilder in Java? They introduced StringBuilder recently, which is an unsynchronized version, so they didnt have to have a lock? Guess what? Java 6 optimizes those locks away. Any time _you_ can see that the lock isnt needed, they can see it. [_Editors Note: See "Biased locking" in the linked perf whitepaper. Its an astoundingly cool example of the predictive-heuristic class of techniques Ive talked about today._]
So now all these tricks, all this common wisdom that programmers share with each other, saying "I heard that this hashtable is 1.75 times faster than blah, so therefore you should...", all the micro-optimizations theyre doing – are going to become obsolete! Because compilers are smart enough to deal with that.
Q: YOU DIDNT MENTION APL, WHICH IS A VERY NICE DYNAMIC LANGUAGE...
I didnt mention APL!? Oh my. Well, Im sorry. [_(laughter)_]
Q: THE THING IS, WELL – SEVERAL OF THE APL SYSTEMS, THEY INCORPORATED MOST OF THE LIST OF PROGRAM TRANSFORMATIONS YOURE TALKING ABOUT. AND THEY DID IT SEVERAL DECADES AGO.
Yeah, so... so you couldve said Lisp. You couldve said Smalltalk. "We did it before!" And that was kind of, that was one of the important points of the talk, right? It _has_ been done before. But Im gonna stand by my – especially with APL – Im going to stand by my assertion that the language popularity ranking is going to stay pretty consistent. I dont see APL moving 50 slots up on the [list].
Im sorry, actually. Well not for that case. [_(laughter)_] But Im sorry that in general, the languages that got optimized really well, and were really elegant, arguably more so than the languages today, you know, in a lot of ways, but theyre not being used.
I tried! I mean, I tried. But I couldnt get anybody to use them. I got lynched, time and again.
Q: SO WHATS THE LIGHT AT THE END OF THE TUNNEL FOR MULTITHREADING?
Oh, God! You guys want to be here for another 2 hours? [_(laughter)_] I read the scariest article that Ive read in the last 2 years: an interview with, I guess his name was Cliff Click, which I think is a cool name. Hes like the HotSpot -server VM dude, and somebody, Kirk Pepperdine was interviewing him on The Server Side. I just found this randomly.
And they started getting down into the threading, you know, the Java memory model and how it doesnt work well with the actual memory models, the hardware, and he started going through, again and again, problems that every Java programmer – like, nobody knows when the hell to use volatile, and so all of their reads are unsynchronized and theyre getting stale copies...
And he went through – went through problems to which he does not know the answer. I mean, to where I came away going Oh My God, threads are irreparably busted. I dont know what to do about it. I really dont know.
I do know that I did write a half a million lines of Java code for this game, this multi-threaded game I wrote. And a lot of weird stuff would happen. Youd get NullPointerExceptions in situations where, you know, you thought you had gone through and done a more or less rigorous proof that it shouldnt have happened, right?
And so you throw in an "if null", right? And Ive got "if null"s all over. Ive got error recovery threaded through this half-million line code base. Its contributing to the half million lines, I tell ya. But its a very robust system.
You _can_ actually engineer these things, as long as you engineer them with the certain knowledge that youre using threads wrong, and theyre going to bite you. And even if youre using them right, the implementation probably got it wrong somewhere.
Its really scary, man. I dont... I cant talk about it anymore. Ill start crying.
Q: THESE GREAT THINGS THAT IDES HAVE, WHATS GONNA CHANGE THERE, LIKE WHATS GONNA REALLY HELP?
Well, I think the biggest thing about IDEs is... first of all, dynamic languages will catch up, in terms of sort of having feature parity. The other thing is that IDEs are increasingly going to tie themselves to the running program. Right? Because theyre already kind of doing it, but its kind of half-assed, and its because they still have this notion of static vs. dynamic, compile-time vs. run-time, and these are... really, its a continuum. It really is. You know, I mean, because you can invoke the compiler at run time.
Q: IS IT ALLOWED AT GOOGLE TO USE LISP AND OTHER LANGUAGES?
No. No, its not OK. At Google you can use C++, Java, Python, JavaScript... I actually found a legal loophole and used server-side JavaScript for a project. Or some of our proprietary internal languages.
Thats for production stuff. Thats for stuff that armies of people are going to have to maintain. It has to be high-availability, etc. I actually wrote a long blog about this that Ill point you to that actually... Like, I actually came around to their way of seeing it. I did. Painful as it was. But theyre right.
Q: [_QUESTION IS HARD TO HEAR_]
[_me paraphrasing_] Are we going to have something Lisp Machines didnt?
Q: YES.
Well... no. [_(loud laughter)_]
I say that in all seriousness, actually, even though it sounds funny. I, you know, I _live_ in Emacs. And Emacs is the worlds last Lisp Machine. All the rest of them are at garage sales. But Emacs _is_ a Lisp Machine. It may not be the best Lisp, but it is one.
And you know, T.V Raman, you know, research scientist at Google, who, he doesnt have the use of his sight... hes a completely fully productive programmer, more so than I am, because Emacs is his window to the world. Its his remote control. EmacsSpeak is his thesis. Its amazing to watch him work.
Emacs, as a Lisp Machine, is capable of doing _anything_ that these other things can. The problem is, nobody wants to learn Lisp.
Q: AND IT DOESNT HAVE CLOSURES.
And it doesnt have closures, although you can fake them with macros.
Im actually having lunch with an [ex-]Emacs maintainer tomorrow. Were going to talk about how to introduce concurrency, a better rendering engine, and maybe some Emacs Lisp language improvements. You know, even Emacs has to evolve.
But the general answer to your question is No. Lisp Machines pretty much had it nailed, as far as Im concerned. [_(shrugging)_] Object-oriented programming, maybe? Scripting? I dunno.
Q: MANY YEARS AGO, I MADE THE GREAT TRANSITION TO A FULLY TYPE-CHECKED SYSTEM. AND IT WAS WONDERFUL. AND I REMEMBER THAT IN THE BEGINNING I DIDNT UNDERSTAND IT, AND I JUST DID WHAT I HAD TO DO. AND ONE DARK NIGHT, THE COMPILER GAVE ME THIS ERROR MESSAGE, AND IT WAS RIGHT, AND I THOUGHT "OH WOW, THANK YOU!" ID SUDDENLY FIGURED IT OUT.
Yes! "Thank you." Yes.
Although its very interesting that it took a long time before it actually told you something useful. I remember my first experience with a C++ compiler was, it would tell me "blublblbuh!!", except it wouldnt stop there. It would vomit for screen after screen because it was Cfront, right?
And the weird thing is, I realized early in my career that I would actually rather have a runtime error than a compile error. [_(some laughs)_] Because at that time... now this is way contrary to popular opinion. Everybody wants early error detection. Oh God, not a runtime error, right? But the debugger gives you this ability to start poking and prodding, especially in a more dynamic language, where you can start simulating things, you can back it up... Youve got your time-machine debuggers like the OCaml one, that can actually save the states and back up.
Youve got amazing tools at your disposal. Youve got your print, your console printer, youve got your logging, right? [_Ummm... and eval. Oops. -Ed._] Youve got all these assertions available. Whereas if the compiler gives you an error that says "expected expression angle-bracket", you dont have a "compiler-debugger" that you can shell into, where youre trying to, like – you could fire up a debugger on the compiler, but I dont recommend it.
So, you know, in some sense, your runtime errors are actually kind of nicer. When I started with Perl, which was pretty cryptic, you know, and I totally see where youre coming from, because every once in a while the compiler catches an error. But the argument that Im making is NOT that compilers dont occasionally help you catch errors. The argument that Im making is that youre gonna catch the errors one way or the other. Especially if youve got unit tests, or QA or whatever.
And the problem is that the type systems, in programming in the large, wind up getting in your way... way too often. Because the larger the system gets, the more desperate the need becomes for these dynamic features, to try to factor out patterns that werent evident when the code base was smaller. And the type system just winds up getting in your way again and again.
Yeah, sure, it catches a few trivial errors, but what happens is, when you go from Java to JavaScript or Python, you switch into a different mode of programming, where you look a lot more carefully at your code. And I would argue that a compiler can actually get you into a mode where you just submit this batch job to your compiler, and it comes back and says "Oh, no, you forgot a semicolon", and youre like, "Yeah, yeah, yeah." And youre not even really thinking about it anymore.
Which, unfortunately, means youre not thinking very carefully about the algorithms either. I would argue that you actually craft better code as a dynamic language programmer in part because youre _forced_ to. But it winds up being a good thing.
But again, I – this is all very minority opinion; its certainly not majority opinion at Google. All right? So this is just my own personal bias.
Q: [_QUESTION TOO HARD TO HEAR OVER AUDIO, SOMETHING ABOUT IS IT POSSIBLE FOR THE COMPILER AT LEAST TO OFFER SOME HELP_]
You know, thats an interesting question. Why do compiler errors have to be _errors_? Why couldnt you have a compiler that just goes and gives you some advice? Actually, this is what IDEs are excelling at today. Right? At _warnings_. Its like, "ah, I see what youre doing here, and you dont really need to. You probably shouldnt."
Its weird, because Eclipses compiler is probably a lot better than javac. Javac doesnt need to be good for the reasons I described earlier, right? It all gets torn down by the JIT. But Eclipses compiler needs to give you that exact help. The programmer help, the day-to-day help, I missed a semicolon, I missed this, right? And Eclipse and IntelliJ, these editors, their compilers are very very good at error recovery, which in a static batch compiler usually just needs to be: BLAP, got an error!
OK? So to an extent I think we _are_ getting tools that come along and act like the little paper-clip in Microsoft Office. You know. Maybe not quite like that.
Q: THE ONLY THING I WORRY ABOUT IS THAT THERES A CHUNK OF CODE THAT YOU REALLY WANT TO WORK SOMETIMES, BUT THE ERROR-RECOVERY PARTS ARE HARD TO TEST.
Thats the part you _have_ to do at runtime, right? Well, I mean, when you get into concurrency youre just screwed, but if youre talking about situations where its very difficult to... I mean, its computationally impossible to figure out whether all paths through a code graph are taken. I mean, its NP-complete, you cant do this, right? But the VM can tell you which code paths got taken, and if it doesnt [get taken], you can change your unit test to force those code paths to go through, at which point youve now exercised all of your code. Right? Thats kind of the way, you know, they do it these days.
And I would say its a pain in the butt, but I mean... its a pain in the butt because... a static type-systems researcher will tell you that unit tests are a poor mans type system. The compiler ought to be able to predict these errors and tell you the errors, way in advance of you ever running the program. And for the type systems theyve constructed, this is actually true, by and large, modulo assertion errors and all these weird new runtime errors they actually have to, heh, inject into your system, because of type-system problems.
But by and large, I think what happens is unless the type system actually delivers on its promise, of always being right and always allowing you to model any lambda-calculus computation your little heart desires, OK? Unless it can do that, its gonna get in your way at some point.
Now again, this is all personal choice, personal preference. I think, you know, static compilers and error checkers, they have a lot of value and theyre going to be around for a long time. But dynamic languages could get a lot better about it.
Im not trying to refute your point. Im just saying that... there are tradeoffs, when you go to a dynamic language. I have come around... Ive gone from dynamic to static and back to dynamic again, so Ive done the whole gamut. And Ive decided that for very large systems, I prefer the dynamic ones, in spite of trade-offs like the one you bring up.
Q: ACTUALLY YOU SAID, FOR HYBRID SYSTEMS, WHERE PIECES OF IT ARE DYNAMIC AND PIECES ARE STATIC...
I think some of it has been pretty... theres a great paper from Adobe about it, right? Evolutionary programming? Talks about how you prototype your system up, and then you want to lock it down for production, so you find the spots where there are API boundaries that people are going to be using. You start putting in contracts, in the way of type signatures and type assertions.
Why not build that functionality into the language? Then you dont have to build a prototype and re-implement it in C++. Thats the thinking, anyway. It seems like a great idea to me, but it hasnt caught on too much yet. Well see. [_Editors note: The main hybrid examples Im aware of are StrongTalk, Common Lisp, Groovy and EcmaScript Edition 4. Any others?_]
All right, well, now were definitely over time, and Im sure some of you guys want to go. So thank you very much!
-------------------------
At this point I got mobbed by a bunch of grad students, profs, and various interested people with other questions. They dumped an absolutely astounding amount of information on me, too – papers to chase down, articles to follow up on, people to meet, books to read. There was lots of enthusiasm. Glad they liked it!
"Were going to get lynched, arent we?" -- _Phouchg_
And you thought Id given up on controversial blogs. Hah!
PREAMBLE
This must be said: Jamie Zawinski is a hero. A living legend. A major powerhouse programmer who, among his many other accomplishments, wrote the original Netscape Navigator and the original XEmacs. A guy who can use the term "downward funargs" and then glare at you just daring you to ask him to explain it, you cretin. A dude with arguably the best cat-picture blog ever created.
Ive never met him, but Ive been in awe of his work since 1993-ish, a time when I was still wearing programming diapers and needing them changed about every 3 hours.
Lets see... that would be 15 years ago. Ive been slaving away to become a better programmer for fifteen years, and Im still not as good -- nowhere _near_ as good, mind you -- as he was then. I still marvel at his work, and his shocking writing style, when Im grubbing around in the guts of the Emacs-Lisp byte-compiler.
It makes you wonder how many of him there are out there. You know, programmers at that level. He cant be the only one. What do you suppose theyre all working on? Or do they all eventually make 25th level and opt for divine ascension?
In any case, Im sad that I have to write the obit on one of his greater achievements. Sorry, man. Keep up the cat blog.
FORKING XEMACS
I have to include a teeny history lesson. Bear with me. Its short.
XEmacs was a fork of the GNU Emacs codebase, created about 17 years ago by a famous-ish startup called Lucid Inc., which, alas, went Tango Uniform circa 1994. As far as I know, their two big software legacies still extant are a Lisp environment now sold by LispWorks, and XEmacs.
Id also count among their legacies an absolutely outstanding collection of software essays called Patterns of Software, by Lucids founder, Richard P. Gabriel. I go back and re-read them every year or so. Theyre that good.
Back when XEmacs was forked, there were some fireworks. Nothing we havent seen many times before or since. Software as usual. But there was a Great Schism. Nowadays its more like competing football teams. Tempers have cooled. At least, I think they have.
As for the whole sordid history of the FSF-Emacs/XEmacs schism, you can read about it online. Im sure it was a difficult decision to make. There are pros and cons to forking a huge open-source project. But I think it was the right decision at the time, just as decomissioning it is the right decision today, seventeen years later.
XEmacs dragged Emacs kicking and screaming into the modern era. Among many other things, XEmacs introduced GUI widgets, inline images, colors in terminal sessions, variable-size fonts, and internationalization. It also brought a host of technical innovations under the hood. And XEmacs has always shipped with a great many more packages than GNU Emacs, making it more of a turnkey solution for new users.
XEmacs was clearly an important force helping to motivate the evolution of GNU Emacs during the mid- to late-1990s. GNU Emacs was always playing catch-up, and the dev team led by RMS (an even more legendary hacker-hero) complained that XEmacs wasnt really playing on a level field. The observation was correct, since XEmacs was using a Bazaar-style development model, and could move faster as a direct consequence.
A lot of people were switching over to XEmacs by the mid-1990s: the fancy widgets and pretty colors attracted GNU Emacs users like moths to a bug-zapper.
Problem was, it could actually zap you.
THE DOWNSIDE OF THE BAZAAR
I personally tried to use XEmacs many times over a period of many years. I was jealous of its features.
However, I never managed to use XEmacs for very long, because it crashed a lot. I tried it on every platform I used between ~1996 and 2001, including HP/UX, SunOS, Solaris, Ultrix, Linux, Windows NT and Windows XP. XEmacs would never run for more than about a day under moderate use without crashing.
Ive argued previously that one of the most important survival traits of a software system is that it should never reboot. Emacs and XEmacs are at the leading edge of living software systems, but XEmacs has never been able to take advantage of this property because even though it can live virtually forever, its always tripping and falling down manholes.
Clumsy XEmacs. Clumsy!
I assume its propensity for inopportune heart attacks is a function of several things, including (a) old-school development without unit tests, (b) the need to port it to a gazillion platforms, including many that nobody actually uses, (c) a culture of rapid addition of new features. There are probably other factors as well.
Im just speculating though. All I know is that its always been very, very crashy. It doesnt actually matter what the reasons are, since theres no excuse for it.
Interestingly, most XEmacs users Ive talked to say they dont notice the crashing. Im sure this is because its all relative. XEmacs doesnt crash any more often than Firefox, for instance. Probably less often. When Firefox crashes I make a joke about it and restart it, because the crashing rarely has an impact. It even restores your state properly most of the time, so its just a minor blip, an almost trivial inconvenience, so long as whatever text field you happen to be editing has an auto-save feature. And most of the good ones do.
XEmacs may crash even less than Eclipse and IntelliJ. Crashing editors usually arent a big problem. Programmers all learn the hard way to save their buffers frequently. For me, saving is like punctuation; I save whenever my typing pauses, out of reflex. Doesnt matter whether its Emacs or NeoOffice or GMail or... do I use any other apps? Oh yeah, or the Gimp. When I pause, I save, and if youre a programmer I bet you do too. So occasional crashes may seem OK.
Another reason the crashes arent called out more often is that most Emacs and XEmacs users are at best casual users. They open up an {X}Emacs session whenever they need to edit a file, and close it when theyre done. Its just Notepad with colors and multi-level Undo.
If your average session length is shorter than the editors MTBF, then yeah, youre not going to notice much crashing.
In contrast, your more... ah, _seasoned_ (read: fanatical) Emacs users gradually come to live in it. Anything you cant do from within Emacs is an annoyance. Its like having to drive to a government building downtown to take care of some random paperwork they should have been offering as an online service a decade ago. You can live with it, but youre annoyed.
Even Firefox, the other big place I live, really wants to be Emacs. Tabs dont scale. Tabbed browsing was revolutionary in the same way adding more tellers to a bank was revolutionary: its, like, 4x better. w00t. Emacs offers the unique ability to manage its open buffers in another first-class buffer, as a _list_. Imagine what your filesystem life would be like if the only view of a directory was one tab per file. Go to your pictures directory and watch it start vomiting tabs out like it tried to swallow a box of chiclets. Fun!
I feel sad when I see Eclipse users with fifty open tabs, an army of helpful termites eating away at their screen real-estate and their time.
I have a feeling Ive veered off course somewhere... where was I? Oh yeah. Crashing.
So XEmacs has never been a particularly good tool for serious Emacs users because even though its written in C, it crashes like a mature C++ application. You know the drill: major faceplants, all the fugging time.
Your ability to become an inhabitant of Emacs is gated by how stable it is. GNU Emacs has always been famously stable. Sure, the releases happen less frequently than presidential inaugurations. Sure, for a long time it always lacked some XEmacs feature or other. But its really, really stable. Its MTBF is measurable in weeks (or even months, depending on what youre doing with it) as opposed to hours or days.
Emacs, like Firefox, can be configured to back up your state periodically, so that in theory it can recover after a crash. Thats part of the problem: you didnt actually have to configure Firefox to get that behavior. It does it automatically. And to be honest, Ive never had much luck with the Emacs save-state facilities. Im a pretty competent elisp hacker these days, but the
desktop.el has never worked reliably for me. I could probably get it to work, but Ive always found it easier to write specialized startup "scripts" (lisp functions) that load up particular favorite configurations.
If I cant get desktop-save working, Id guess that fewer than 1/10th of 1 percent of Emacs users use that feature. So crashes blow everything away.
If the state isnt being auto-saved, the next best thing is for it not to crash.
XEmacs never got that right.
DONT GET ME WRONG...
I just realized Im going to get screamed at by people who think Im just an XEmacs-hater slash GNU-fanboy.
Make no mistake: Im a fan of XEmacs. I think it was a great (or at least, necessary) idea in 1991. I think the execution, aside from the stability issue, was top-notch. I think it had a good architecture, by and large, at least within the rather severe constraints imposed by Emacs Lisp. I think it spurred competition in a healthy way.
I think the XEmacs development team, over the years, has consisted of engineers who are ALL better than I am, with no exceptions. And I even like certain aspects of the interface better, even today now that GNU Emacs has caught and surpassed XEmacs in features. For instance, I like the XEmacs "apropos" system better.
If youre going to scream at me for irrational reasons, it really ought to be for the _right_ irrational reasons. Legitimate dumb reasons for screaming at me include: youre lazy and dont want to learn anything new; you invested a lot of time in XEmacs and dont see why you should be forced to switch; you are a very slow reader, causing you to skip three out of every five words I write, resulting in your receipt of a random approximation of my blog content, with a high error bar; youre still mad about my OS X blog. All good bad reasons.
Heck, you could even scream for rational reasons. Perhaps you have a philosophical beef with the FSF or GPL3. Perhaps XEmacs still has some vestiges of feature support that do not yet exist in GNU Emacs, and you truly cant live without them. I would think youre being a teeny bit uptight, but I would respect your opinion.
Whatever you do, just dont yell at me for thinking Im dissing XEmacs or taking some sort of religious stance. Far from it. I just want a unified Emacs-o-cratic party.
XEMACS VS. GNU EMACS TODAY
GNU Emacs pulled into the lead in, oh... Id say somewhere around maybe 2002? 2003? I wasnt really keeping track, but one day I noticed Emacs had caught up.
Even today I maintain XEmacs/FSF-Emacs compatibility for my elisp files - some 50k lines of stuff Ive written and maybe 400k lines of stuff Ive pilfered from EmacsWiki, friends, and other sources. I still fire up XEmacs whenever I need to help someone get un-stuck, or to figure out whether some package Ive written can be coerced to run, possibly in restricted-feature mode, under XEmacs.
For years I chose stability over features. And then one day GNU Emacs had... well, everything. Toolbars, widgets, inline images, variable fonts, internationalization, drag-and-drop in and out of the OS clipboard (even on Windows), multi-tty, and a long laundry-list of stuff Id written off as XEmacs-only.
And it was still stable. Go figure.
I dont have the full feature-compatibility list. Does it even exist? You know, those tables that have little red Xs if the Evil Competitor product is missing some feature your product offers, and little green checkmarks, and so on. We ought to make one of those. It would be useful to know what (if any) XEmacs features are preventing the last holdouts from migrating to FSF Emacs.
But for the past five years or so, just about every time an XEmacs user on a mailing list has mentioned a feature thats keeping them from switching, its been solved.
If GNU Emacs isnt a perfect superset of XEmacs yet, Im sure we could get it there if we had the big unified-platform carrot dangling in front of us. And I bet its pretty close already.
Features and stability aside, XEmacs is looking pretty shabby in the performance department. Its font-lock support has never been very fast, and a few years back GNU Emacs took a giant leap forward. XEmacs can take 4 or 5 seconds or longer to fontify a medium-sized source file. Sure, it shows that big progress bar in the middle of the screen, so you know its not dead, but when youre used to it being almost instantaneous, coming back to XEmacs is a real shocker.
And XEmacs has bugs. Man, it has a lot of bugs. I cant begin to tell you how many times Ive had to work around some horrible XEmacs problem. It has bugs (e.g. in its fontification engine and cc-engine) that have been open for years, and they can be really painful to work around. Ive had to take entire mode definitions and if-xemacs them, using an ancient version of the mode for XEmacs because nothing even remotely recent will run.
You may not notice the bugs, but as elisp developers, we feel the pain keenly.
FUNDAMENTAL INCOMPATIBILITIES
As if issues with stability, performance and bugs werent enough, XEmacs has _yet another_ problem, which is that its APIs for dealing with UI elements (widgets and input events, but also including things like text properties, overlays, backgrounds and other in-buffer markup) are basically completely different from their GNU-Emacs counterparts. The two Emacsen share a great deal of common infrastructure at the Lisp level: they have mostly compatible APIs for dealing with files, buffers, windows, subprocesses, errors and signals, streams, timers, hooks and other primitives.
But their APIs range from mildly to completely different for keyboard and mouse handling, menus, scrollbars, foreground and background highlighting, dialogs, images, fonts, and just about everything else that interfaces with the window system.
The GUI and display code for any given package can be a significant fraction of the total effort, and it essentially has to be rewritten from scratch when porting from GNU Emacs to XEmacs or vice-versa. Unsurprisingly, many package authors just dont do it. The most famous example I can think of is James Clarks nxml-mode, which claims itll never support XEmacs. I found that pretty shocking, since I thought it was basic Emacs etiquette to try to support XEmacs, and here James was cutting all ties, all public about it and everything. Wow.
But I totally understand, since I really dont want to rewrite all the display logic for my stuff either.
Ill be the first to admit: the API discrepancies are not XEmacss fault. I cant see how they could be, given that for nearly all these features, XEmacs had them first.
For a developer trying to release a productivity package, it doesnt really matter whose fault it is. You target the platform that will have the most users. I dont know what XEmacss market share is these days, but Id be very surprised if its more than 30%. Thats a big number, but when youre an elisp hacker creating an open-source project in your limited spare time, that number can start looking awfully small. Teeny, even.
XEMACS SHOULD DROP OUT OF THE RACE
At this point its becoming painful to watch. GNU Emacs is getting all the superdelegates. That warmonger VIM is sitting back and laughing at us. But XEmacs just wont quit!
Im sure there are a few old-timers out there who still care about the bad blood that originally existed between the two projects. To everyone else its ancient history. As far as I can tell, there has been an atmosphere of polite (if subdued) cooperation between the two projects. Each of them has incorporated some compatibility fixes for the other, although its still mostly up to package authors to do the heavy lifting of ensuring compatibility, especially for display code.
I havent seen any XEmacs/GNU-Emacs flamewars in a long time, either. Were all just *Emacs users, keeping our community alive in the face of monster IDEs that vomit tabs, consume gigabytes of RAM, and attract robotic users who will probably never understand the critical importance of customizing and writing ones own tools.
When the Coke/Pepsi discussion comes up these days, its usually an XEmacs user asking, in all seriousness, whether they should transition to GNU Emacs, and if so, would someone volunteer to help migrate their files and emulate their favorite behaviors.
Yes, someone will volunteer. I promise.
THE DUBIOUS FUTURE OF EMACS
Ive got good news and bad news.
The good news is: Emacs is a revolutionary, almost indescribably QWAN-infused software system. Non-Emacs users and casual users simply cant appreciate how rich and rewarding it is, because they have nothing else to compare it to. There are other scriptable applications and systems out there -- AppleScript, Firefox, things like that. Theyre fun and useful. But Emacs is _self-hosting_: writing things in it makes the environment itself more powerful. Its a feedback loop: a recursive, self-reinforcing, multiplicative effect that happens because youre enhancing the environment youre using to create enhancements.
When you write Emacs extensions, sometimes youre automating drudgery (always a good thing), sometimes youre writing new utilities or apps, and sometimes youre customizing the behavior of existing utilities. This isnt too much different from any well-designed scriptable environment. But unlike in other environments, sometimes youre improving your editing tools and/or your programming tools for Emacs itself. This notion of self-hosting software is something Ive been wanting to blog more about, someday when I understand it better.
Eclipse and similar environments _want_ to be self-hosting, but theyre not, because Java is not self-hosting. In spite of Javas smattering of dynamic facilities, Java remains as fundamentally incapable of self-hosting as C++. Self-hosting only works if the code can "fold" on itself and become more powerful while making itself smaller and cleaner. Im not really talking about macros here, even though thats probably the first thing you thought of. Im thinking more along the lines of implementing JITs and supercompilers in the hosted runtime, rather than in the C++ or Java "hardware" substrate, which is where everyone puts them today.
I suspect (without proof) that in self-hosted environments, you can eventually cross a threshold where your performance gains from features implemented in the hosted environment outpace the gains from features in the substrate, because of this self-reinforcing effect: if code can make _itself_ faster and smarter, then it will be faster and smarter at making itself faster and smarter. In C++ and Java, making this jump to the self-reinforcing level is essentially intractable because, ironically, they have so many features (or feature omissions) for the sake of performance that they get in their own way.
To be sure, Emacs, the current crop of popular scripting languages, and other modestly self-hosting environments are all pretty far from achieving self-reinforcing _performance_. But Emacs has achieved it for _productivity_ - at least, for the relatively small percentage of Emacs users who learn enough elisp to take advantage of it. There are just enough of us doing it to generate a steady supply of new elisp hackers, and the general-purpose artifacts we produce are usually enough to keep the current crop of casual users happy.
THE BAD NEWS: THE COMPETITION ISNT THE IDES
Ive argued that Emacs is in a special self-reinforcing software category. For productivity gains, that category can _only_ be occupied by editors, by definition, and Emacs is currently way ahead of any competition in most respects. So most Emacs users have felt safe in the assumption that IDEs arent going to replace Emacs.
Unfortunately, Emacs isnt immunized against obsolescence. It still needs to evolve, and evolve fast, if its going to stay relevant. The same could be said of any piece of software, so this shouldnt be news. But its particularly true for Emacs, because increasing numbers of programmers are being lured by the false productivity promises of IDEs.
They really are false promises: writing an Eclipse or IntelliJ (or God help you, Visual Studio) plugin is a monumental effort, so almost nobody does it. This means theres no community of building and customizing your own tools, which has long been the hallmark of great programmers. Moreover, the effort to create a plugin is high enough that people only do it for really significant applications, whereas in Emacs a "plugin" can be any size at all, from a single line of code up through enormous systems and frameworks.
Emacs has the same learning-curve benefit that HTML had: you can start simple and gradually work your way up, with no sudden step-functions in complexity. The IDEs start you off with monumental API guides, tutorials, boilerplate generators, and full-fledged manuals, at which point your brain switches off and you go over to see whats new on reddit. ("PLEASE UPMOD THIS PIC ITS FUNNY!")
And lets not even get into the Million Refactorings yet. Its a blog Ive been working on for years, and may never finish, but at some point Id like to try to show IDE users, probably through dozens or even hundreds of hands-on examples Ive been collecting, that "refactoring" is an infinite spectrum of symbol manipulation, and they have, um, twelve of them. Maybe its thirteen. Thirteen out of infinity - its a start!
Programmers are being lured to IDEs, but the current crop of IDEs lacks the necessary elements to achieve self-hosting. So the only damage to Emacs (and to programmers in general) is that the bar is gradually going down: programmers are no longer being taught to create their own tools.
IDEs are draining users away, but its not the classic fat-client IDEs that are ultimately going to kill Emacs. Its the browsers. They have all the power of a fat-client platform and all the flexibility of a dynamic system. I said earlier that Firefox wants to be Emacs. It should be obvious that Emacs also wants to be Firefox. Each has what the other lacks, and together theyre pretty damn close to the ultimate software package.
If Emacs cant find a way to evolve into (or merge with) Firefox, then Firefox or some other extensible browser is going to eclipse Emacs. Its just a matter of time. This wouldnt be a bad thing, per se, but theres a good chance it would be done poorly, take forever, and wind up being less satisfying than if Emacs were to sprout browser-like facilities.
EMACS AS A CLR
So Emacs needs to light a fire and hurry up and get a better rendering engine. Port to XUL, maybe? I dont know, but its currently too limited in the application domains it can tackle. I realize this is a very hard problem to solve, but it needs to happen, or at some point a rendering engine will emerge with just enough editing power to drain the life from Emacs.
Emacs also needs to take a page from the JVM/CLR/Parrot efforts and treat itself as a VM (thats what it is, for all intents) and start offering first-class support for other languages. Its not that theres anything wrong with Lisp; the problem is _X programmers_. They only want to use X, so you have to offer a wide range of options for X. Emacs could be written in any language at all, take your pick, and it wouldnt be good enough.
RMS had this idea a long, long time ago (when he was making the rather controversial point that Tcl isnt a valid option for X), and it eventually led to Guile, which led more or less nowhere. Not surprising; its a phenomenally difficult challenge. There are really only two VMs out there that have achieved even modest success with hosting multiple languages: the CLR and the JVM. CLRs winning that race, although its happening in a dimension (Windows-land) that most of us dont inhabit. Parrot is... trying really hard. Actually, I should probably mention LLVM, which (like Parrot) was designed from the ground up for multi-language support, but took a lighter-weight approach. So lets call it four.
In any case, its a small very group of VMs, and they still havent quite figured out how to do it: how to get the languages to interoperate, how to get languages other than the first to perform decently, and so on.
This is clearly one of the hardest technical challenges facing our industry for the next 10 years, but its also one of the most obviously necessary. And Emacs is going to have to play that game. Im not talking about hacked-together process bridges like PyMacs or el4r, either -- I mean first-class support and all that it entails.
Ive mentioned the rendering engine and the multi-language support; the last major hurdle is concurrency. I dont know the answer here, either, but it needs an answer. Threads may be too difficult to support with the current architecture, but there are other options, and someone needs to start thinking hard about them. Editing is becoming a complicated business -- too complicated for hand-rolling state machines.
COMPETE OR DIE
So Emacs has some very serious changes ahead.
Lets face it: were not going to see real change unless ALL the Emacs developers out there - todays crop of JWZs - band together to make it happen. But today were divided. Two groups of brilliant C hackers working on separate, forked code bases? Thats bad. Two groups of maniacal elisp hackers working on incompatible packages, or at best wasting time trying to achieve compatibility? Also bad.
Developers are starting to wake up and realize that the best "mainstream" extensible platform (which excludes Emacs, on account of the Lisp) is Firefox or any other non-dead browser (which excludes IE). Dynamic typing wins again, as it always will. Dynamic typing, property-based modeling and non-strict text protocols won the day for the web, and have resisted all incursions from heavyweight static replacements. And somehow the web keeps growing, against all the predictions and lamentations of the static camp, and it still works. And now the browsers are starting to sprout desktop-quality apps and productivity tools. It wont be long, I think, before the best Java development environment on the planet is written in JavaScript.
Emacs has to compete or die. If Firefox ever "tips" and achieves even a tenth of the out-of-the-box editing power of Emacs, not just for a specific application but for all web pages, widgets, text fields and system resources, Emacs is going to be toast. I may be the last rat on the ship, but Im sure not going down with it; even _I_ will abandon Emacs if Firefox becomes a minimally acceptable extensible programmers editor. This is a higher bar than you probably think, but it could happen.
We no longer need XEmacs to spur healthy competition. The competition is coming in hard from entirely new sources. What we need now is unity.
THEN WHY NOT UNIFY BEHIND XEMACS?
I threw this in just in case you blew through the article, which Id find perfectly understandable. To summarize, Ive argued that XEmacs has a much lower market share, poorer performance, more bugs, much lower stability, and at this point probably fewer features than GNU Emacs. When you add it all up, its the weaker candidate by a large margin.
Hence theres only one reasonable strategy: Hill, er, I mean XEmacs has to drop out of the race.
Im really sorry about this. Im a close personal friend of XEmacs, but I just cant endorse it anymore. I used to be a laissez-faire kinda guy, as long as you were using _some_ flavor of Emacs. But at this point, if youre using XEmacs youre actively damaging not only your long-term productivity, but mine as well. So Id like to ask you to think long and hard about switching. Soon.
If youre a local Emacs-Lisp guru, please offer your services to XEmacs users who would like to switch over. The more pain-free the migration is, the faster it will happen.
If youre a graphic artist, consider making a nice, tasteful "Euthanize XEmacs!" logo. Not that message, precisely, but something along those lines. Make sure its tasteful. Perhaps "XEmacs is dead - long live XEmacs"? Yes, I think that would do nicely.
If you happen to know someone on the XEmacs development team, send them some chocolates, or movie tickets, or something. A thank-you, even. We should honor their service. But those guys are the most qualified on the planet to step in and start helping drive GNU Emacs forward, assuming the FSF team will have them. Emacs is in very bad shape indeed if they will not.
If youre a local system administrator, consider sudo rm -rf xemacs. Sorry, I mean consider politely asking your emacs-users mailing list if they might be willing to set a timeline for deprecating XEmacs, thereby starting the most massive flamewar in your companys history. Cant hurt!
If youre seeing red, and you skipped most of this article so you could comment on how amazingly lame this idea is, I recommend taking a little walk and getting some fresh air first.
If youre RMS, thank you for making Emacs and all that other stuff, and for keeping it free. Please be nice to those who wish to help. Youre scary to the point of unapproachability, just cuz youre you.
XEmacs team, JWZ, and XEmacs package authors: thank you all for helping drive progress in the greatest piece of software of all time. I can only hope that someday I may have chops like that.
Now how about we turn this into the most famous reverse-fork in history?I recently switched to using OS X full-time for all my client-side computing. Still using Linux on the backends, of course, at home and at work, but I now use Macs for my client machines.
Im not a Mac fanboy. Im sort of a wannabe Mac fanboy, but Im not familiar enough with the OS yet (either as a user or as a programmer) to really rave about it. I will say this: it was kinda fun turning off that last Windows box for the last time.
My main reason for switching was that Im getting old and the fonts look nicer. Pretty stupid reason, isnt it? I thought so too. But getting old kinda sneaks up on you. Ive gone from preferring six-point font when I was twelve to 20-point font now that Im 40. So at least for me, my ideal font point size appears to be
(age/2).
That sucks.
One day I noticed that I could actually read the screen when I was browsing in the Apple store, and I did some experimentation and found that yes, I can actually read normal-persons fonts on the Mac. And they look kinda nice, too - the antialiasing engine seems to be smarter (and faster) than the ones Ive seen on Windows and Linux.
So there ya go. Fonts. And now I have to learn all this new stuff, like what all those weird little symbols mean on the keys, and how to use the Finder, and what a "DMG file" is, and other stuff. But the screen looks _soooo_ nice, so its worth it. How do they do it? Its not just the fonts. OS X windows look whiter and cleaner than their Windows/Linux cousins running on the same display with similar video hardware. Its a mystery to me, but its kinda cool.
ALMOST PAIN-FREE MIGRATION
Ive been using a work-issued MacBook Pro laptop for the past year, and that helped a lot with the transition, since when youre on the road trying to get some work done, you have no choice but to figure out how to do basic OS tasks. So that was a nice, slow, reasonably pain-free way to teach myself the basic skills you need.
I only bring this up because I know a _lot_ of programmers (myself included) whove tried Macs repeatedly and run away scared. If you stick with it a little longer, its not too bad! Particularly with their latest OS X release (Leopard), its gotten a lot easier to do basic configuration for people accustomed to Linux.
For starters, it comes with a good X11 implementation, and there is a MacPorts project that ports all your favorite Unix stuff. And theyre not lame half-broken ports like the ones you have to live with in Cygwin. For instance, in a Bash shell running inside Emacs you can ssh into a Linux box and not get a bunch of greebly control characters.
And OS X is Unix, based on FreeBSD, so all your normal favorite Unix stuff pretty much works the same, or at least as much as you can expect across different flavors of Unix.
The only real reason I was using Windows at all, to be honest, was for hardware-device compatibility and for multimedia. The Mac has drivers for everything I cared about (my router, my printer, my camera, etc.) and beats Windows hands-down on any sort of multimedia, so it was becoming clear that Windows wasnt buying me much anymore.
I suppose I could do a blow-by-blow guide for how a Unix-and-Windows user can configure their Mac for maximum happiness. If anyones interested, anyway. Not today, though. The bottom line is that pretty much everything you dont like is configurable... with one ugly exception. And I dont mean "Mary Ann on Gilligans Island Ugly", either. I mean Ugly ugly.
THE BIG FOCUS ISSUE
Ever since the Dawn of Time (Jan 1, 1970), people have been bitching about the lack of focus-follows-mouse on Mac computers. They started complaining about it fourteen years before the first Mac was even released, thats how bad it was.
Every time they bring it up on Mac forums, the Mac users with non-Unix backgrounds ask "whats that?" And then a bunch of wrong answers start flying around, with a few right answers interspersed but drowned out in the noise.
So let me tell you what it is first, in case youre not from a Unix background. Focus-follows-mouse means that when you move the mouse cursor, the window under the cursor gets the keyboard focus. But saying that confuses Mac people who all assume that "focused" is synonymous with "foreground", because thats the way it works on the Mac.
The confusion stems from the fact that focus-follows-mouse comes in not one, but TWO, yes thats right, _two_ yummy flavors.
FLAVOR #1: AUTOFOCUS -- in this flavor, reminiscent perhaps of a sweet juicy mandarin orange, the window under the mouse gets the keyboard focus but _does not come to the front_. This allows you to interact with a partially-obscured window. Its especially useful when you have a terminal or shell window open, and its running a background process that you want to observe... you guessed it, in the background! You leave a little bit of the bottom and/or side of the window uncovered so you can keep an eye on the output.
Real-life use case: lets say youre a programmer who writes in C++. You will, of course, spend most of your working day playing Solitaire and reading reddit, because C++ is too goddamned stupid to do anything but gigantic, slow batch compiles of the entire dependency universe. So you have at least four windows open at any given time: your editor, your compile shell, your browser, and your Solitaire game. Youve spent a lot of time adjusting your window configuration to be "just right", and unless you have a 30-inch screen (for instance, because you work for Google), your windows overlap.
Watching your compile status is like checking your rear-view mirror; you do it every 7 seconds or so, even though you know the compiler will take a minimum of 15 minutes. Its like a slow-motion train wreck that you just cant tear your eyes from, even while playing Solitaire and reading reddit. And every once in a while youll need to enter a command (e.g. "make", after youve fixed the umpteenth compiler warning about doing a perfectly valid type conversion). The last thing you want is to have to click the window to bring it to the front just so you can type "make", because then youll need to go futz around with your window configuration again to get the window to go to back to whatever Z-location it used to be in the window stack.
I know it doesnt sound like a big effort, but programmers are really, _really_ lazy, and they like to minimize motion. Theyd use feeder tubes if the Health Department would let them.
So in the autofocus flavor, its important that the window that gets the focus does _not_ automatically come to the front.
FLAVOR #2: AUTORAISE -- in this pungent flavor, somewhat evocative of a slightly overripe Durian fruit left in the tropical sun for about nine hours, moving the mouse into a new window automatically brings that window to the front. In the especially horrible default configuration, it comes to the front instantly, so the act of moving your mouse across the screen makes it look like that old "rectangles" screen saver, and your window configuration is utterly obliterated in under a second.
Many programmers feel that autofocus is delicate butterfly and autoraise is a big, stinky buffalo. Thats just how they feel about it. No accounting for taste. I, for one, think of autoraise as a big, stinky, _deceased_ buffalo carcass that someone thoughtfully dragged into my living room while I was on vacation, probably towards the beginning of the vacation, and then they turned up my thermostat to 110°F, closed the windows and tossed a Durian fruit at the wall for good measure.
But maybe its just me.
So one of the most annoying aspects of the whole "how do I get focus-follows-mouse behavior on my Mac" debate is that everyone assumes you mean autoraise. There are a number of packages out there, most of them commercial, that offer autoraise as a feature, and Mac users point you to these products and then get all smugly about how theyve solved your problem and how Macs still rule the universe, when in fact the problem is still festering away.
Its no wonder people still use Linux as their UI. That one feature alone keeps hordes of programmers from switching. (And yes, you can get the behavior on Windows using their TweakUI power tools, so some programmers use Windows as a Linux shell with a decent media player.)
SUPER-STEVEY TO THE RESCUE (WELL, ALMOST)
Given that I switched quite recently to the Mac, Im still reeling from the lack of focus-follows-mouse behavior. To help you put yourself in my shoes, imagine that your latest operating system upgrade (whatever OS you happen to be running) includes a new mandatory feature wherein each time you click on a window to focus it, a loud alarm goes off ("BLONK! BLONK! BLONK! ...") and you have to open the System menu and select "silence window alarm" to shut it up.
Thats what not having autofocus is like to people whove been using it for the past 10 to 30 years (in my case, 20 years). BLONK! BLONK! BLONK! Im serious. Its that bad. Not exaggerating even a tiny bit.
Im sure you could eventually get used to this behavior, and even find yourself arguing on newsgroups that you rather like the blonk blonk sound, since it reminds you that youve recently chosen to switch to another application or to another window within the current application, plus its really not that big a deal because you can just open the System menu and turn it off.
Its amazing how so many people choose to rationalize stuff theyre forced to live with. Why not just admit it sucks? Sometimes stuff sucks! Cmon, admit it! Jeez!
But even if you _eventually_ managed to rationalize it, youd be pretty fugging pissed off the first, oh, ten thousand or so times it happened to you after the upgrade.
So the other day, after the 100th or so BLONK! BLONK! BLONK! alarm when I innocently tried to type into a different window, I found myself quietly contemplating the pros and cons of getting an assault rifle, heading down to the Apple campus, and making my opinions known to all until the SWAT team took me out. And then I thought: "hey, as attractive as that option sounds right now, I have a better idea: why not fix it myself? I make occasional claims to being a programmer, right? How blonking hard can it be? Ill budget one evening for it."
I actually wound up spending 2 evenings on it, since although coming up to speed on the Apple tools and APIs was almost trivial, this particular issue turned out to be thorny in a variety of unexpected ways.
But I did get it working, for some definition of "working", and now Im in a position to settle the debate for the forseeable future, which I estimate to be about the next five to seven years in this case.
THE DEFINITIVE ANSWER
_Short version:_ you can _almost_ get it working, but not quite, on account of an arguable bug in one of the Carbon (that is, Mac C++) APIs. What I got working was a system-wide autofocus mode that unfortunately only re-routes _unmodified_ keys to the window under the cursor. You can fake the Shift modifier by translating the key code manually, but the Control, Command and Alt/Option keys never make it through to background applications. (They do get delivered to the foreground app if your mouse is there, so the bug only happens for background apps.)
So if your use case is limited to, say, typing commands into a command shell, and you dont need to use the control, alt or command keys, then you can have working autofocus! In fact, you can even make it so that only your terminal windows (or any application list of your choice) get the autofocus behavior, and all other applications get the normal must-click-to-focus behavior. So even my "failed" attempt might yet hold some small utility for us ancient Unix hackers. Ill play with it for a while and see.
_Long Version:_ As close as I came in my little 2-day effort, I now believe at this point that its unlikely that autofocus will ever be available on Macs, sad as that news will be for thousands of would-be Mac users. And not just any users: theyre _programmers_, all potentially capable of learning to write Mac applications and collectively enhancing the value of Apples platform. So its kind of a big opportunity cost for Apple. But there are both technical issues and design issues that make it a serious problem to support autofocus on Macs.
Its probably not _impossible_, but the cost is high enough that when their OS engineers think about tackling it, theyll probably decide its not worth the effort, since the company seems to fail to appreciate just how big a stumbling block the lack of autofocus-sans-autoraise really is for so many competent Unix programmers out there.
So, Apple OS engineers, Im not saying youre not smart. Im just saying youre not smart _enough_. ;-)
Just kidding, of course, and Ill dispense with the child psychology. Heres why I think theyre not going to fix it. The rest of this blog entry consists of boring technical details, so if youre getting antsy, please feel completely free to skip to the very end.
HOW IT WORKS
First, one caveat: Im not a Mac programmer. I dont even play one on TV. I just downloaded Xcode (their development toolkit) for the first time three days ago. Ive never written any Mac programs before this one, not even an AppleScript script, and I only started looking at their APIs a couple days ago. So I might be wrong about some or all of this.
The first problem you encounter is that theres no public Mac API for getting any sort of usable handle to a running application so you can interact with it programmatically. This is apparently for security reasons. I wont harp on this decision, although it does seem odd to deny sophisticated (read: sudo-enabled) users the choice of loading privileged apps into their system. Any application can run amok with your filesystem, personal data and network connection, so it seems odd that youd arbitrarily choose not to let them also run amok with your other running apps.
In any case, theres a loophole. Apple, out of sheer generosity, goodwill, and the kindness of their heart o hearts, and also partly because United States Federal Law requires it, but mostly out of sheer generosity, goodwill and the kindness of their hearts, has provided a set of "Accessibility APIs" that give you a certain federally mandated level of remote control over running applications in the system.
OS X actually has two more or less discrete sets of APIs: one for C/C++ (Carbon), and one for Objective-C (Cocoa). Cocoa incidentally also happens to be the API you use for Python and Ruby scripting on the Mac; I took a detour for a few hours and learned the basics of RubyCocoa, and was quite pleased at how well it worked.
One of the reasons I took the RubyCocoa detour was that the subset of the Accessibility APIs I needed for implementing autofocus is fairly cumbersome to explore using C++ and Xcode. I made an executive decision to spend (and potentially waste) some time seeing if I could make faster progress using one of the scripting APIs, because I was encountering bugs and/or unexpected behavior that called for some exploratory programming.
Carbon offers an abstraction called an AXUIElementRef, which is a proxy object representing a UI element (e.g. an application, a window, or widget) in any running app on the system. This subsystem is designed and implemented entirely using the Properties Pattern, which, as it happens, Ill be blogging about at length in the very near future. Normally this pattern is quite flexible, and I can fully understand their reasons for using it here: it gave them legal compliance with an absolute minimum of effort.
But the Properties pattern is healthiest in a dynamic environment that lets you poke around reflectively to get the names of properties, fetch their values, traverse parent links, and so on. Carbon provides APIs for manipulating all these UI-element properties with C++, but it really is cumbersome: lots of casting, lots of wrappers, lots of recompilation every time you want to try just one more thing. Call me spoiled, but I only budgeted a day for this feature!
So I learned a bit of RubyCocoa, and it appears - as far as I can tell - that the relevant Accessibility APIs are _only_ available through Carbon, and not through Cocoa, which means if you want to use them, you cant use Objective-C, Ruby or Python. Or at least I couldnt find a way. If Im wrong, someone _please_ correct me, since Id really like an experimentation platform that handles Carbon APIs that have no Cocoa equivalents.
REALLY GRUBBY DETAILS
I told you itd be boring! What are you still doing here?
OK, whatever. Youre a glutton for punishment, I tell ya.
There are three basic components to the focus-follows-mouse solution:
* Create an "event tap" to filter all system keypresses.
* Find the frontmost window under the mouse
* Redirect the keypresses to that window.
Thats all there is to it. This is the solution I envisioned before Id even downloaded Xcode, and unsurprisingly, it appears to be the only reasonable way to accomplish the task in OS X. I mean, how else would you do it?
The event-tap API is straightforward, with just one teeny, minor exception almost not worth mentioning, which is that it doesnt work. It compiles, runs, and fails silently. This took me several hours to figure out. It turns out that event taps are considered to be part of the Accessibility APIs, and for security reasons, your process either has to be running as root, or you have to enable "assistive technologies" in the Universal Access section of System Preferences. I stumbled across this in some random newsgroup after a LOT of searching. In retrospect it was kinda there in the API documentation, but they didnt make it super clear.
Whew! There went several hours down the drain, but now I had a C program that fired up and listened for keypresses, printing them to stdout. The event-tap API gives you the option of swallowing the keypresses (or changing the event, or even returning a new event to replace the old one), so its plenty flexible enough for our needs.
Next, I needed to find the window under the cursor, which first meant finding the global cursor position. This also turned out to be surprisingly non-obvious. The best solution I found, from someones blog, was to create a NULL event and then get its mouse coordinates. So intuitive! Just like Mom used to do it!
Sigh.
Once you have your mouse coordinates, you use a "hit test" to find the window at that screen position. Its one of two Carbon-only APIs you need: you create a proxy for the system-wide UI object with AXUIElementCreateSystemWide, and pass it to the global hit-test function AXUIElementCopyElementAtPosition to get the UI element under the mouse.
Then it gets a little ugly, though not terribly so. These AXUIElementRef objects have all their information in property lists. This would be trivial to navigate in RubyCocoa, but the AXUI API set doesnt seem to exist in Cocoa -- specifically the parts that deal with "any running application" rather than "_your_ application".
So you grub around in the object and its parent chain, clunkily printing stuff in C++ and releasing reference-counts, until you find an ancestor with a "role" attribute of "application". Thats the element you need for delivering keyboard events.
We can deliver the keyboard event to the unfocused window, through this poorly-documented API call:
AXError AXUIElementPostKeyboardEvent(AXUIElementRef application,
CGCharCode keyChar,
CGKeyCode virtualKey,
Boolean keyDown);
All good so far. Excluding the time spent figuring out the access-control problem with event taps, and the time spent playing with RubyCocoa, Im only about five hours into the whole endeavor.
Oh yeah, and the time spent dicking with Xcode trying to figure out how to add a library build target to the executable. Ive done it two or three times now and _still_ cant remember how I did it.
The AXUIElementCopyElementAtPosition function points you to its non-Accessible cousin CGPostKeyboardEvent, which is _also_ more or less undocumented. Can you tell they really dont want you to use this stuff? This cousin function has a teeny bit of explanation in its header file CGRemoteOperation.h, which Xcode provides no easy way of locating via search. You can look for it in Spotlight, hoping youll get lucky and it wont hang like it did for me just now. Or you can do what I did and just use Google Desktop search to pop to it instantly.
The explanation in the header file says:
/*
* Synthesize keyboard events. Based on the values entered,
* the appropriate key down, key up, and flags changed events are generated.
* If keyChar is NUL (0), an appropriate value will be guessed at, based on the
* default keymapping.
*
* All keystrokes needed to generate a character must be entered, including
* SHIFT, CONTROL, OPTION, and COMMAND keys. For example, to produce a Z,
* the SHIFT key must be down, the z key must go down, and then the SHIFT
* and z key must be released:
* CGPostKeyboardEvent( (CGCharCode)0, (CGKeyCode)56, true ); // shift down
* CGPostKeyboardEvent( (CGCharCode)Z, (CGKeyCode)6, true ); // z down
* CGPostKeyboardEvent( (CGCharCode)Z, (CGKeyCode)6, false ); // z up
* CGPostKeyboardEvent( (CGCharCode)0, (CGKeyCode)56, false ); // shift up
*/
Thats it. Thats what they give you. Open questions about the explanation include (a) why are they passing capital-Z if they already reported that the shift key was down, (b) if theres a guesser when you pass NULL, why do you need to pass Z, (c) how do you get the char code for a given key code, and is it the same on all keyboards, and (d) why didnt they include a "THE FIRST PARAGRAH IS A LIE" disclaimer around the first paragraph?
Open questions be damned. We fearlessly press on, and just pass "whatever" and see what happens. Specifically, I always pass NULL for the char code, and pass the key code I got from the event tap callback as-is.
And it worked! Sort of! I start my little app (which has no UI), move the mouse into a window from a non-foreground application, and I can type into it!
Except I can only type unmodified keys. Its completely ignoring my keyboard event posts for Shift, Control, Alt and Command. Thats the lie part. They said theyd generate flags changed events. They lied.
After some more painful C++ experimentation, I find that the call does NOT ignore modifier keys when posting to (a) the focused application or (b) the system-wide application, which just posts to the focused app. The call only drops modifier keys on the floor for non-focused apps.
Theres a big ol thread about this exact problem from six years ago on the Apple accessibility-dev mailing list. Six years! I read every last word of the thread.
The first takeaway is that Apple OS engineers dont want you to do stuff that they dont want you to do, and they specifically define "stuff they dont want you to do" as "stuff they dont think you want to do." This is actually endemic to Apple forums in general. Whenever someone says "I want focus follows mouse behavior!", some people inevitably reply that "you really dont want to do this". Its that whole "we designed it the _right_ way for everyone" mentality that turns off so many would-be Mac users.
For what its worth, the Apple engineers really were trying to be helpful in this guys situation, and I know how hard it can be to respond to a mailing list in the capacity of "developer representing the company". But it took them a long time to understand his needs, because (and Im speculating here) they implemented the Accessibility APIs only because their Mom told them to, and they dont truly appreciate at a deep level what it means to have a disability, and how important it is for many people to be able to choose a different UI paradigm.
And yes, I am taking the arguably un-PC position that having your fingers hardwired to focus-follows-mouse from 20 years of use can be viewed as a minor disability of sorts. Ill be the first to admit that its not the kind of "real" disability the government probably had in mind when they mandated the Accessibility APIs.
But I _did_ switch to the Mac because my eyes are slowly beginning to fail. Ironic that I should be forced to trade one disability for another.
THE UNDERLYING ISSUE
OK, lets assume for the moment that Apple really does have our best interests at heart, and that they can get over the painful notion that their usability test findings may not actually apply to 100% of all users 100% of the time.
Even if they wanted to help fix it -- and in this case, all theyd need to do is NOT drop the modifier keys on the floor when you call AXUIElementPostKeyboardEvent -- there are some deep-rooted architectural issues at play here, and I finally "got it" while reading that thread.
The problem is that Macs, always and forever, have put the menu bar of the focused application at the top of the screen. The menu bars of unfocused applications are hidden and are not in any way user-interactible.
As you might expect, this UI assumption has been baked into the Mac APIs from the very beginning. Programmers will take advantage of any axiom they can in order to get things working, so over time this has turned from a UI design assumption into an architectural "feature".
In particular, when an app is in the background, its menu structure may not be intact, and the app may be in a state that assumes it will not be receiving any keyboard input. One concrete example mentioned in the thread was that when an app is in the background, the child menu items do not have parent links (although the parents still have pointers to the children.)
This has serious ramifications for focus-follows-mouse. There are certain built-in hotkeys that can activate menu entries, and apps are also free to define their own. If you try to activate a menu in a background application, it _could_ in theory wind up crashing the app, if the app is assuming an intact menu structure and is traversing bottom-up rather than top-down.
You could attempt a knee-jerk solution by allowing Control and Alt through, but deny the Command modifier, since thats the most common menu-activation modifier (I think). But theres another class of applications (Emacs included) that dynamically generates at least part of its menu structure based on the data content. For instance, the Emacs Imenu package generates a list of jump targets from a source-code buffer. Even typing a new function definition could still trigger a rebuild of the IMenu, which (for all anyone knows) could crash Emacs.
You could of course ask app developers to fix this on an application-by-application basis, but there are generations of legacy apps that can never be fixed. The only way to _guarantee_ that pressing a key could never crash an application would be to fix the OS X user-interface architecture to normalize application behavior for foreground and background operation. This could be hard. It could expose its own set of difficult or even intractable problems for legacy apps. Or maybe its really easy. I dont know, since I cant see their code. But I suspect its not easy.
And Apple has no real motivation to fix it, because their UI was designed for "everyone". People who would use focus-follows-mouse are presumably a tiny minority, so even if theyre mostly programmers the cost/benefit likely isnt there.
It is, of course, completely fixable if Apple _really_ wanted to fix it. Ive heard many war stories over the years from Microsoft folks whove had to put compatibility hacks into OS releases, some quite extensive, in order to support popular applications that relied on undocumented OS behavior that suddenly broke. Imagine those poor guys that had to implement perfect DOS/Windows emulation on NT, for example. I suspect that by comparison, fixing focus-follows-mouse would be relatively straightforward.
But I predict it wont happen in the next 5 to 7 years, unless the government suddenly decides that this API is required for properly assistive technologies.
Its interesting that you can get _so close_ using the existing APIs: I have true focus-follows-mouse behavior implemented for non-modified keys. Sure, the window doesnt actually focus, so some applications dont even show a cursor. But if youre willing to live with occasional glitches, the feature works great.
In any event, whats weirdest about all this is that the API lets you send non-modifier keys to the background app, because as I pointed out, its still possible for vanilla keys to crash applications! If the state of the app is materially different when its unfocused, and the app isnt expecting keyboard input when unfocused, then it could crash. Dropping the modifier keys on the floor may reduce the probability of Badness, but it certainly doesnt eliminate the possibility.
Was it an accident that they let any keys through at all? That would surprise me greatly, since the OS engineers seem determined to close undocumented behavior loopholes. But if they had a reason (perhaps a legal reason) to permit unmodified keys through, what was it about the reason that lets them drop the modifier keys?
I wish I knew.
OPEN LETTER TO THE APPLE WISH FAIRY
If I could wave a magic wand, Id ask for them to fix the API to pass the modifier keys along to the app, and just put a note in the docs that Bad Things could happen, so Buyer Beware.
They already do this for those exact APIs anyway. CGPostKeyboardEvents documentation says: "This function is not recommended for general use because of undocumented special cases and undesirable side effects." And this is for the API that only talks to the focused foreground app! The AXUI version is obviously double-buyer-beware, and apps cant even use it without prompting for the superuser password, unless the user has purposely enabled assistive technologies.
There would be bugs, yes. Some applications would have to push out new releases to properly support focus-follows-mouse, and some legacy apps would never be fixed. But you could disable the behavior on an app-by-app basis, or just take a "Doctor, it hurts when I do this" approach.
Unfortunately I dont have a magic wand; all I have is my distinctly nonmagical blog, which Im using to soothe myself via copious whining.
EPILOGUE
This whole thing has been an interesting lesson in how the government can actually force companies to open up their locked-down systems. The whole "youll have it _our_ way, and like it" mentality is crumbling with these assistive technologies. I hope the feds mandate opening things up even further, since were only partway there so far.
In the interim, Im sure Ill eventually get used to life without autofocus. BLONK! My 30-inch screen helps, as does Spaces, since its easier to give windows their own non-overlapping real estate.
I might even give autoraise another try. Some developers have implemented it in a horrible way, by generating a mouse click when you move the mouse into a window, which often results in activating UI objects unintentionally just by moving the mouse. Ouch. But there might be some commercial implementations that do it "right", or I can just hack my autofocus app to do it that way. Combined with an autoraise delay and minimizing my overlapping windows, it might just work. Well see.
And everything else so far about the Mac has seemed pretty nice, or when its been not nice, at least its been fixable with a little configuration effort. I liked the OS X APIs on the whole, and the RubyCocoa thing is pretty sweet. I might even wind up writing some native clients -- something I thought Id never do again, given how awful my Windows native-client experiences have been over the years.
So Ill keep using my Macs. Theyre all just plumbing for Emacs, anyway. And now my plumbing has nicer fonts.
IF YOU SKIPPED STRAIGHT TO THE END...
...you didnt miss much. See you next time!Ive written a new JavaScript editing mode for GNU Emacs, and released it on code.google.com.
This is part of a larger project, in progress, to permit writing Emacs extensions in JavaScript instead of Emacs-Lisp. Lest ye judge: hey, some people swing that way. The larger project is well underway, but probably wont be out until late summer or early fall.
My new editing mode is called js2-mode, because eventually I plan to support JavaScript 2, also known as ECMAScript Edition 4. Currently, however, it only supports up through JavaScript 1.7, so the name is something of a misnomer for now.
Its almost ten thousand lines of elisp (just for the editing mode), which is more than Id expected. So I figured Id tell you a little about what it does, why I made certain choices, and whats coming up next. Even if youre not a JavaScript user, you might find the technical discussion mildly interesting.
FEATURES
In no particular order, heres what inside a string literal, it will autoamtically turn it into a multi-line string concatenation.
You can also hit Alt-q (fill-paragraph) inside a comment or a string to see hopefully useful things happen. Let me know if it doesnt do what you expect.
SYNTAX ERRORS
The mode highlights syntax errors in red. This can be annoying as you type, but Im told (by Eclipse/IntelliJ users) that you get used to it.
You can control this behavior via a customization variable.
STRICT WARNINGS
JavaScript defines a whole bunch of strict-mode warnings: things like "dont have a trailing comma in an Array or Object literal", or "your variable name conflicts with one of the function parameters". Ive implemented some of them, with more to come. They get underlined in orange.
I actually found some bugs in live code Id written with this feature. Pretty cool!
JSDOC HIGHLIGHTING
Theres a program similar to javadoc called "jsdoc" that lets you do documentation comments for your JavaScript functions and other declarations. It defines a similar set of
js2-mode currently supports.
M-X CUSTOMIZE
All the user-configurable variables are defined as Custom variables for use with M-x customize. This means you can type
M-x customize-group RET js2-mode RET
to see a list of all the configuration options.
All the colors used for syntax highlighting are defined in the same js2-mode customization group, for convenience.
Many people complain that Emacss Customize feature is lame. I thought so too, for a long time, and Im certainly not claiming its as good as a "real" UI. But I now appreciate that it gets the job done admirably for a text editor: there are no dependencies on GUI widgets, and you can use the Customize package over an ssh or telnet session. Thats at least _kind_ of cool.
ACCURATE SYNTAX HIGHLIGHTING
This mode includes a recursive-descent parser that I ported from Mozilla Rhino. That means its always right. It doesnt use heuristics or guesswork; its exactly the same parser used by JavaScript engines. If its ever wrong, then its a bug in my code, and its fixable.
The amount of syntax highlighting is configured by a variable called js2-highlight-level. It ranges from 0 to 3, with the default set at 2. Zero (or nil/NaN) means no highlighting. level 1 does basic syntax highlighting: keywords and declarations. Level 2 adds highlighting for Ecma-262 builtins (and SpiderMonkey extensions) such as Infinity, __proto__ and decodeURIComponent. Level 3 adds highlighting for all built-in functions and properties for all native JavaScript objects (Function, Date, Array and so on.)
The highlighting faces are my own choices, because I felt it was important for me to foist my personal style choices on the general public. Actually, thats only partly true: its also because I like the color schemes employed by Eclipse and IntelliJ better than the default Emacs color scheme. So comments are green (not red!), keywords are blue, strings are soft blue, var decls are sea green, and so forth.
Fortunately for those (hopefully few) among you who love blood-red comments, you can add this to your .emacs file to make js2-mode honor your font-lock settings:
(setq js2-use-font-lock-faces t)
Or, alternately, M-x customize-variable RET js2-use-font-lock-faces RET and set the value to t, which is Lisp for "true". This particular variable requires an Emacs restart to take effect.
I just added a TODO item for myself: define Eclipse and IntelliJ color schemes that you can choose from. Should be pretty easy.
ASYNCHRONOUS HIGHLIGHTING
Unlike most other Emacs modes (but like nXML mode), js2-mode does not use font-lock-mode, which is the standard Emacs infrastructure for doing syntax-coloring in buffers.
Although font-lock is quite fast and fairly flexible, it still uses heuristics to figure out what to highlight, and they can occasionally be wrong. If youve ever opened up prototype.js in Emacs and seen the second half of the file turned string-blue on account of being confused over a regular expression literal with a quote in it, you know what Im talking about.
James Clarks nXML-mode does its own syntax coloring without using font-lock. It can do this because James wrote his own, fully-compliant, validating XML parser, so adding colors was a snap. I thought this was pretty macho, and since I also happen to have a full parser, I blatantly copied his idea.
For the OOD-loving and API-minded among you, the "beautiful" way to do syntax coloring would have been to finish parsing, _then_ walk the AST using a Visitor interface, applying the coloring in a second pass. I tried it, and it was, as they say, "butt slow". In fact (perhaps not surprisingly) walking the AST takes exactly as long as parsing, so it was twice as slow as doing it inline.
So I bit the bullet and moved my syntax-coloring to happen inline with parsing. Fortunately it only introduced about 30 lines of code to the 4000-line parser/scanner, because most of the coloring happens in the scanner, at the token level. Go figure.
Unfortunately, my parser is asynchronous. It "sort of" happens on another thread, although whats really happening is that it waits for Emacs to become idle and parses until you hit a key or use the mouse. I wanted it to be synchronous, boy howdy I did, but it just wasnt quite fast enough. It can parse about 5000 lines a second, give or take, but for any file longer than 1000 lines or so, the parsing was happening _every time you typed a key_ (thats what synchronous means, obviously), and the 0.2+ second delay became painfully noticeable.
I had two options: incremental parsing, or asynchrous parsing. Clearly, since Im a badass programmer who cant recognize my own incompetence, I chose to do incremental parsing. I mentioned this plan a few months ago to Brendan Eich, who said: "Let me know how the incremental parsing goes." Brendan is an amazingly polite guy, so at the time I didnt realize this was a code-phrase for: "Let me know when you give up on it, loser."
The basic idea behind incremental parsing (at least, my version of it) was that I already have these little functions that know how to parse functions, statements, try-statements, for-statements, expressions, plus-expressions, and so on down the line. Thats how a recursive-descent parser works. So I figured Id use heuristics to back up to some coarse level of granularity -- say, the current enclosing function - and parse exactly one function. Then Id splice the generated syntax tree fragment into my main AST, and go through all the functions siblings and update their start-positions.
Seems easy enough, right? Especially since I wasnt doing full-blown incremental parsing: I was just doing it at the function level. Well, its not easy. Its "nontrivial", a word they use in academia whenever theyre talking about the Halting Problem or problems of equivalent decidability. Actually its quite doable, but its a huge amount of work that I finally gave up on after a couple of weeks of effort. There are just too many edge-cases to worry about. And I had this nagging fear that even if I got it working, it would totally break down if you had a 5,000 line function, so I was kinda wasting my time anyway.
So, without telling Brendan (and dont you dare mention it to him), I switched to asynchronous parsing. Actually, first I went around to my Eclipse- and IntelliJ-using friends, and I forced them to give me live demonstrations of Java editing on large files. This is why I have so few friends. It turns out that Eclipse and IntelliJ both use asynchronous parsing as well, which made me feel better about the basic approach.
Asynchronous parsing is pretty simple in principle: when the user is typing, dont do anything. Just let em type. When they stop, start a timer for, say, a 200 to 500 millisecond delay, and when the timer expires, start parsing. Every once in a while, see if they typed anything. If so, stop parsing and let them type.
The main downside of this approach is that for some programmers, the 500-ms timer fires between every keystroke, so the file never actually finishes parsing. (Yes, that was a mean joke. I have a blog on this subject coming up; Im declaring war on people too lazy to learn to type.)
Actually, now that I think about it, I did mention my change of heart (and asynchronous approach) to Brendan a week or two ago, and he jumped immediately to the smart-guy conclusion: I need continuations. Fortunately Id thought of this, albeit not in the 200 milliseconds it took him to arrive at that conclusion (over wine, no less!), so I was able to retort: "um, yeah... its in my to-do list. Right now I hack it."
And hack it I do! I rely on the fact that my parser does 5000 lines a second, so if the parse gets interrupted, at _some point_ even the fastest, most dedicated typist will have to pause for a second, and Ill finish the parse (which in turn finishes the highlighting and error/warning reporting - see below).
Unfortunately (as Brendan instantaneously concluded), this means that if the parse gets 99.9% complete, and you hit the up-arrow, it abandons the entire parse (and parse tree built so far), starting from scratch again when Emacs goes idle. So if you open a big file (like prototype.js) and start navigating around it, you may not see any results until you stop typing or scrolling.
The proper fix will be to record where Im at, and pick up where I left off when I restart the parse. Thats what nxml-mode does, but Im forced to concede that James Clark is way cooler than I am. If you have a multi-threaded system, then its trivial, and if your system supports continuations, its also trivial. But Emacs has neither of these.
Instead, I pause every 100 statements or so (this is a lame heuristic, I agree) and check for user input. Since Im pausing at the top level of the parser, in the loop where it consumes whole statements, I really dont need to store that much information to fake a continuation, so this problem is eminently fixable.
But I had to release this thing eventually, which meant drawing the line somewhere. So for now it has asynchronous full-restart parsing. This means that as you edit the file, just like in Eclipse and IntelliJ, it can take a second or two for the parser to catch up with you after you pause.
It doesnt (or shouldnt) interfere with your editing, though, so hopefully this isnt a big issue.
MISSING HIGHLIGHTING
Its on my js2-mode TODO list to highlight E4X literals. E4X is a JavaScript language extension (an official Ecma standard, in fact) that allows you to embed XML literals in your JavaScript code and provides various XML operators and functions that let you do DOM-style manipulations and XPath-style queries, but with JavaScript-style syntax and semantics.
I parse these properly, but dont highlight them yet. The Rhino parser just parses them as strings, so to get more accuracy Ill need to make my own little XML parser. It must (I think) be my own little parser because E4X permits embedding arbitrary JavaScript expressions in curly-braces as a form of templating. This complicates the XML parsing because you can find one or more {javascript-expr} expressions in the middle of any XML element name, attribute name, attribute value, text node, or just about anywhere else that doesnt cross a quote or angle-bracket boundary.
Ill get around to it eventually.
INDENTATION
I would have been publishing this article at _least_ a month ago if it werent for indentation. No, six weeks, mininum.
See, I thought that since I had gone to all the thousands of lines of effort to produce a strongly-typed AST (abstract syntax tree) for JavaScript, it should therefore be really easy to do indentation. The AST tells me exactly what the syntax is at any given point in the buffer, so how hard could it be?
It turns out to be, oh, about fifty times harder than incremental parsing. Surprise!
Just to give you a feel for the size of the problem, the package cc-engine (including its cc-* helper packages) bundled with GNU Emacs 22 is approximately 27,000 lines of lisp code, and its _all_ dedicated to indentation. Theres a teeny tiny smattering of maybe 500 lines dedicated to filling, and sure, it supports several C-like languages, but lets face it: 25k lines for indenting? 27k lines of _Lisp_ code? (Meaning it would be, like, five times that much Java?)
What the hell is so hard about indentation?
For starters, in order to provide user-configurable indentation for every possible syntactic context, you need to name all the syntactic contexts. cc-engine defines about 70 syntactic positions in a data structure called c-offsets-alist. This is a map of {context-name : indent-level}, where indent-level can be a number (a multiple of the variable c-basic-offset), or a symbol specifying some multiple of c-basic-offset, or even a function to call to figure out how to indent.
Its pretty darn flexible. And people still complain about it! Apparently 70 different syntactic contexts isnt enough to let you specify your indentation exactly the way you like it.
Anyhoo, most existing JavaScript editing modes for Emacs use cc-engine and try to coerce it into indenting JavaScript properly. This usually meets with lackluster results, since JavaScript is gradually drifting further and further from C. So is Java, but someone actually bothers to try to keep cc-engine up to date for Java.
Heres the deal: the cc-engine code for interpreting that c-offsets-alist data structure (with all the indentation configuration options) is pretty small. Most of the code goes to parsing and trying to figure out the current syntactic context.
You can probably guess what I tried to do. I wanted to let people customize their js2-mode indentation much the same way they can customize their c-mode or java-mode indentation, using c-offsets-alist. So I figured Id use the exact same configuration data structure, and use my parse tree to replace c-guess-basic-syntax (and the 25k lines of lisp code for implementing it!)
_(time passes...)_
Approximately one month later, I threw in the towel. I renamed my js2-indent.el to doomed-indent.el, and my js2-indent-test.el unit-test file to doomed-indent-test.el, and gave up on this approach for the forseeable future. 1500 lines of painfully crafted lisp code down the drain.
Ugh. Sure, it was only a few hours a week, but it still felt like a lot of work. And it was a lot of calendar time.
Amazingly, surprisingly, counterintuitively, the indentation problem is _almost totally orthogonal_ to parsing and syntax validation. Id never have guessed it. But for indentation you care about totally different things that dont matter at all to parsers. Say you have a JavaScript argument list: its just (blah, blah, blah): a paren-delimited, comma-separated, possibly empty list of identifiers. Parsing that is pretty easy. But for indentation purposes, that list is rife with possibility! You might want to indent it like this:
(blah,
blah, blah)
or this:
(
blah,
blah,
blah)
or this:
(blah,
blah,
blah)
or this:
(
blah,
blah, blah)
Lets face it: you could be a total lunatic, and _Emacs has to make you happy_. So instead of simply parsing a plain argument list, you need to determine and capture the (a) the fact that it looks like an argument list, (b) the position and indentation of the open-paren, (c) whether the cursor is before or after the open-paren, (d) whether the arg list is nonempty, (e) whether the cursor is before the first list element, (f) whether the cursor is on the line containing the closing paren, (g) whether there are any block or single-line comments interspersed between _any_ of the list elements or parentheses, (h) whether the AAAAUGH, I cant _stand it_ anymore!
The problem is, this explosion of "one case to arbitrarily many cases" occurs for every single grammatical construct in your language. So if you have 70-ish such constructs (as JavaScript does - Java has almost double that, because of the type system), and each one expands to 5 to 10 possible indentation situations, well, youve got an awful lot of edge cases to deal with.
Worse, having a rich AST doesnt help you much. You can figure out that its an argument list, and possibly where the cursor is in the list, but you still have to grope around in the buffer looking for other contextual cues that matter for indentation but which the parser threw away. So each syntactic case in the 700-odd scenarios I had to handle expanded to anywhere from 2 to 10 lines of lisp code.
I was about 1500 lines into my doomed-indent.el (plus unit tests), and maybe (optimistically) 35% finished, when it occurred to me: "is there a better way?"
KARL ANGALSDKJFADSLKFJ TO THE RESCUE
I remembered that there are several javascript editing modes out there already, and none of them does a very good job (or I wouldnt be working on js2-mode). But one of them, "javascript.el", I remembered as being pretty good at indentation. It wasnt perfect, and Id had to write some custom hacks for it here and there, but it was actually pretty decent. How did it work?
I went and looked at it. Its written by a guy named, according to the comment header, Karl Landström. Id always assumed that this was just some my-font-doesnt-support-Unicode gobbledygook, and that his name was actually something more reasonable like Karl Landstr\301^HB^P\302\301!\204^0^@. But upon closer inspection, I think he may be a fan of the artist formerly known as the artist formerly known as Prince, aka "Prince", because the "ö" in his name shows up pretty consistently across platforms and fonts. So it may be intentional. Perhaps his parents were ardent mathematicians.
In any case, Karl Landstrlaksjdflaksjd is an amazingly clever guy, because his indenter, which beats the pants off all the JavaScript modes based on cc-engine, is only about 200 lines of elisp. 200!? How does he do it?
Well, in a nutshell, he makes the inspired assumption that indentation is almost always a function of brace/paren/curly nesting level, and he uses a little-known built-in Emacs function called parse-partial-sexp, written in C, which tells you the current nesting level of not only braces, parens and curlies, but also of c-style block comments, and whether youre inside a single- or double-quoted string. How useful! Good thing JavaScript uses C-like syntax, or that function would have been far less relevant.
The rest of his code handles cases where you have a JavaScript keyword such as if, while or finally (a "possibly braceless keyword"), where you can optionally leave off the curly-brace, and it should still indent one basic step for the nested statement.
The results are actually pretty darn good, and assuming youre reasonably flexible about where you position your parens and curly-braces, you can exert at least some control over the indentation. (E.g. you can move a curly down to its own line and manually indent it, and subsequent lines will indent from that curly.)
Go Karl!
Unfortunately, its not perfect (no solution so elegant could ever be, at least for a language based on the inelegant syntax of C), so I was faced with a dilemma: should I pile hack upon hack until it becomes the new cc-engine? Or is there another way?
Well, Ive always been vaguely admiring of python-modes Emacs indentation, which chooses among various likely indentation points when you press TAB repeatedly. Why not use that approach for JavaScript?
So thats what I wound up doing. I put a _few_ tweaks into Karls original indenter code to handle JavaScript 1.7 situations such as array comprehensions, and then wrote a "bounce indenter" that cycles among N precomputed indentation points.
For any given line, there are some obvious possible indentation points:
- whatever position Karls guesser wants to use
- the beginning of the line
- after the = if the previous line is an assignment
- same indentation as the previous line
- first preceding line with less indentation than the preceding line
I wrote a function that computes all these positions, based on heuristic parsing (NOT on my AST, which might not even be available yet if the parse is taking a while), and the TAB key cycles among them.
This moved the accuracy, at least for my own JavaScript editing, from 90% accurate with Karls mode up to 99.x% accurate, assuming youre willing to hit TAB multiple times for certain syntactic contexts.
There are still plenty of user-defined situations (e.g. parts of Googles internal JavaScript style guide) that my guesser doesnt compute. You dont want to compute _every_ possible indentation point, or the TAB key degenerates into the space key modulo the line length, so at some point Ill add a customization hook that lets you write a function to help decide the right indentation.
Anyway, where was I. Oh yeah. Indentation is a real pain in the b-hind. Im glad to be (mostly) done with it. At least hopefully you now understand why my mode isnt configurable the way other C-like modes are, and you sympathize with me. Next time I have time to write ten thousand lines of indentation-related guessing, Ill fix it.
Meanwhile, if you find points where it doesnt do what you want, let me know (or post them on the Wiki), and Ill either hack them in or write that customization hook.
OTHER STUFF
I didnt expect to spend so long on just syntax highlighting and indentation. Its just the beginning! Unfortunately Im out of patience, and Im guessing you are too. So heres a short list of other features.
CODE FOLDING
I support hiding function bodies and /*...*/ block comments as {...}. Its in the menu. Turn on menu-bar-mode, or right-click in the buffer, to invoke these functions.
At some point Ill generalize it to hiding any curly-brace construct, the way Eclipse does. This was just an experiment to see how easy it would be. (Answer: pretty easy! Emacs has good built-in support for this kind of thing.)
COMMENT AND STRING FILLING
One neat trick I stole from Eclipse: if you hit @whatever tags. We use it at Google, albeit with limited success because its a Perl program that core-dumps on most of our JavaScript code base. My mode highlights the various tags in jsdoc comments, if you happen to use them.
Its possible that youll notice it highlighting curly-brace constructs inside jsdoc comments, such as:
@return {SomeType} my return value
Googler Bob Jervis has written a type-inferencing engine for our JSCompiler, in his 20% time, that uses the type-tags weve defined in an enhanced version of jsdoc comments. Its still pretty new, and were planning to open-source it and integrate it with Mozilla Rhino at some point, but since its 20% time, theres no telling when itll be released. But hopefully thatll explain the bizarre highlighting you might sometimes see.
IF THIS ISNT GOOD ENOUGH FOR YOU...
Well, you have three options.
First, you can whine about it. If you whine in the appropriate places, such as the Wiki, then Ill eventually notice and try to fix whatever it is thats bothering you.
Next, you can offer to help. I havent uploaded the original source code, but I can certainly start doing so. (The file js2-.el is generated from a little build script I wrote, to make installation easier.) If youre a good Emacs-Lisp programmer, and you want to help make this mode better, let me know and we can get you hooked up!
Finally, if you can afford it (or if your company can afford it), consider using IntellIJ IDEA. Yes, its commercial, but if you spend even 30 seconds on their site it becomes apparent that "commercial" means "better". Their JavaScript support is way better than mine, and is as far as I can tell the gold standard for JavaScript editing today.
Eventually I hope to be able to reach feature parity with IntelliJ, and its certainly possible, but itll be some work. In the meantime, if you cant wait, give them a try!
WRAP-UP
At this point I have to go to the bathroom so bad that I dont care what other features Ive added. You can look at the Wiki!
If you habitually (or even occasionally) use GNU Emacs to edit JavaScript, please give this mode a try! Its probably got a fair number of bugs and usability issues, since its brand-new, but itll improve more quickly if you play guinea pig for a while.
Feel free to email me directly with comments, suggestions, or bug reports, or you can go to the Wiki and add your comments there.
Enjoy!I only play a handful of games a year, so I havent written a game review since my post on Oblivion a year ago. But since then Ive played a few unusually good games, including one awesome PS/3 title last weekend, so I figured its time for another review.
First, heres my gamer profile, so you can decide now whether reading any further is worth your time. Im a console guy. I dont play games on a PC because I want the whole big-screen, comfy-couch immersion experience. Some of my favorite games in the past include Final Fantasy X, the Zelda franchise, the Mario franchise, Morrowind and Oblivion, Fable, Doom and Doom II (but not Quake or Unreal), and Donkey Kong 64. So there ya go. Thats the kind of game I like. Adventure and RPG games, mostly, and FPS types only if they have sufficient atmosphere.
Heres the memorable stuff Ive played lately.
PORTAL
Everyones been buzzing about this Xbox-360 game: they all say youve gotta play it, really cool gameplay, blah blah blah, and to be honest it sounded boring as hell. Plus its only available through a 3-game title called The Orange Box, and the other games on it dont interest me.
A friend of mine finally bought me the game, came over, and sat there and glared at me until I played it. Thats one way to do it!
Summary: it rocks. Great gameplay, new game dynamics, genuinely funny storyline, and an unexpectedly cool ending song thats easy to play, if you happen to be a guitarist.
I stayed up all night and finished the game in one sitting, so its not very long at all. Only takes about 90 minutes the second time through, maybe. I recommend playing it through a second time with the developer commentary turned on, especially if youre a programmer or designer, since it was all really interesting stuff.
The plot summary is pretty simple. You have to take a series of increasingly complex tests using a "portal gun" that can shoot linked orange and blue portals onto horizontal surfaces. Jumping through either portal shoots you out the other one, maintaining your momentum, which makes for some really cool suicide-jump situations and a whole lot of different puzzle types. A female computer voice messes with you throughout the tests. At the end of the last test, things go south, and you have to escape into the guts of the test facility, crawling through ducts and messy industrial back-rooms, to find the computer and destroy it. Its really the second half of the game thats the fun part; you have to use everything you learned during the testing phase and solve some tricky puzzles while not getting shot, falling into toxic waste, or dying in any number of other exciting ways.
The game is more or less flawless in its execution: it was smooth, well-designed, bug-free, and it stayed interesting without ever getting frustrating. It was a little masterpiece. Cant wait for the sequel.
MARIO GALAXY
I asked my friend Andrew Wilson if he has a Wii yet, and he retorted: "NO, because Im a _grown-up_!" Funny stuff. Im assuming his wife wont let him buy one.
If you dont have a Wii, well, youre missing out. Its far from perfect - in fact the device itself is pretty weak, and is missing HD output, bluetooth and a number of other pretty important things. But its controllers really blow conventional controllers away. Ive played one FPS on the Wii (Metroid Prime 3), and although the game itself was IMO a bit borderline, the controls were amazing. Going back to the XBox 360 or Playstation controllers feels totally rinky-dink, like Ive traveled back to the 1980s and Im using an Atari Joystick.
Plus its actually true that non-gamers like the Wii, and the multiplayer party games wind up being a lot more fun than they are on other consoles. You can work up quite a sweat trying to beat the crap out of people in boxing matches or outrun them in olympic games. Fun stuff.
Anyway, Id been kind of reluctant to play Super Mario Galaxy. Every time I saw someone playing it in a game store, theyd be running around teeny tiny planets like the ones in that book The Little Prince, and it looked like it was going to be absolutely nauseating. And I mean literally nauseating, in the motion-sickness sense. It just didnt look anywhere near as fun as previous classic Mario titles (Mario 64 comes to mind - I just replayed that on my Nintendo DS and loved it just as much as ten years ago.)
But I figured Id give it a try, and boy, was I wrong about it. Mario Galaxy is arguably the best Mario title since the original Super Mario Bros. arcade game. Nintendo just nailed it, across the board: well-balanced gameplay with lots of variety, superb level design, gorgeous sets, and probably the best music ever from a Mario franchise game - check out this orchestra recording the Gusty Garden Galaxy Theme, for instance. Its just one of several equally awesome pieces written for the game. (I also particularly liked the Good Egg Galaxy theme, the "Teresa Waltz" in the Ghostly Galaxy, the Bowser Battle music, the Buoy Base theme, and the Disney-esque Observatory waltz. But almost all the music was worth listening to, and I cant wait until the soundtrack is available.)
The game has oodles of atmosphere. You can practically _feel_ the wind in the Gusty Garden Galaxy. You can almost smell the honey in the Honeyhive Galaxy. In the Beach Bowl you feel the sun on your back and smell the surf. Its really weird, actually - Ive never played a game before that made it feel like all five senses were in play the way this game does.
I experienced none of the motion-sickness or discomfort I had been dreading. You get used to the gravity surprisingly fast, and before long it doesnt matter what combination of camera angle and orientation you happen to have active - you just keep on running, jumping, and smashing things, upside-down and backwards. Its the most truly three-dimensional game environment Ive ever played.
My favorite part of the game, which I played over and over just for the effect, was probably the first Battlerock level. The music is reminiscent of Mars from Holsts The Planets, and builds up steadily while you make your way towards this huge rock in the distance. As you get closer you realize its sort of like the Death Star, and its shooting at you like mad as the music builds to a climax. Great level!
One of the big turn-offs for me in the Mario and Zelda franchises has been overly-tough boss battles. I think Twilight Princess and Mario Galaxy have both finally dialed it right: the boss fights can be challenging but are achievable without having to take a two-hour time-out from the plotline just to practice some crazy move, as has been the case in so many past titles.
Anyway, the game made a fanboy out of me, so, you know, take all this with a grain of salt. You might like it, you might not. I thought it was great.
ZELDA TWILIGHT PRINCESS
I played Twilight Princess last year - I cant remember exactly when, but it feels like a long time ago. Great game, though; if you havent played it and you like the genre, you need to get your hands on a GameCube or a Wii. I played TP on my old GameCube, since it was before I owned a Wii, and I was _highly_ skeptical of the Wii controls for that kind of game, at the time. I couldnt imagine holding my arm out for ten minutes trying to keep my bow steady. (This was long before I learned that the Wii controls often work best when your wrists and elbows are on the couch and youre just flicking the Wiimotes around like laser pointers.)
I originally wasnt going to write a long review of this game, since its not exactly making news headlines anymore. I did think it was just as good as Ocarina of Time and Wind Waker, but not any better. In fact, I think I might have enjoyed Wind Waker a little more than Twilight Princess. That said, TP was one of the best games of the past ten years, and had some truly groundbreaking gameplay and atmospheric elements.
The more I think about it, the more Im remembering how much fun I had with it. So it gets a spot in my most-recommended list for the past year.
A few scenes still stand out after nearly a year. Flying the big Nazgul-mount-ish monster upriver while getting shot at from the banks played like a scene out of Peter Jacksons Lord of the Rings. The minigame in the canoe coming back downriver was loads of fun. The burning bridge and fall into Lake Hylia were really cool. Searching for bugs for that wacked-out little girl was... weird, but fun. The horseback riding and combat were both completely new and exciting. And I just about died laughing when I wrecked my first wild boar.
The game has an eerie and heartrending central climax, "Midnas Desperate Hour." Youre stuck in wolf form, carrying your dying friend Midna to the castle to try to save her while some really sad music plays, and it just goes from bad to worse. You start off having to run through town in broad daylight as a wolf for the first time, and the townspeople are all running from you screaming in horror, which doesnt make you feel any better about the situation. Then you have to make your way across the most bleak, windswept rooftops imaginable, buffeted by this sort of ash-storm, and then, well, lets just say theyre setting you up to stab your heart out when you get to the castle. Its pretty agonizing. The mood does eventually pick up, though, and eventually you do beat the bad guy and save the day.
If you havent played it, you should give it a try. In some sense its just another Zelda game with all the expected elements, including lonely temples filled with puzzles, little towns filled with bizarre characters, the usual power-ups and mini-games, and the classic heart-container scheme. But it gets major points for having a dramatic sweep and story-arc not really present in previous Zelda titles. Its also got much more achievable boss battles, so you can finish the game without ever getting insanely frustrated.
Now Im making myself want to go back and play it again, so Ill shut up about it.
UNCHARTED: DRAKES FORTUNE
I just played Drakes Fortune last weekend. Its the reason I wrote this blog entry, actually. In my humble and totally biased fanboy opinion, you should run out and buy it _right now_, along with a PS/3 and a bigger TV.
I saw a commercial for the game last year and thought: "Hey, that almost looks like a game that could induce me to buy a Playstation 3. Almost." But I was worried about the whole Blu-Ray vs. HD-DVD thing, and PS/3s used to be kind of steep (they were, what, like six hundred bucks initially?), and I just kinda dragged my feet on the whole issue forever, as I suspect most of you have been doing.
Then Blu-Ray won the war a couple weeks ago, all rather suddenly and unexpectedly, and I thought, hey, this might be a good time to replace my ancient TV and get myself a PS/3, so I can finally drag myself out of the non-HD dark ages. So I went big-screen shopping, and unlike for every other TV Ive bought in the past, I decided not to cheap out this time. I wound up buying a Panasonic 58" 1080p plasma HDTV, which after lots of research and staring at screens in stores, narrowly beat out the Sony Bravia XBR4 52" 1080p LCD HDTV. Theyre both great products, but the plasma still beats the LCD for gaming.
The 58-inch plasma is jaw-droppingly awesome. I replaced my Comcast box with one of their HD-DVRs, and it works great but has almost no space for recording shows, so at some point Ill probably have to pick up an HD Tivo. Its too bad, really, since I have the lifetime-membership Tivo, and Ill almost certainly have to switch to paying a monthly fee if I upgrade to the HD version, since I missed the promotion window when they came out. But you really do need the extra space, so the Comcast one is going to have to go. In the meantime, the HD channels look great, and they only cost me about ten bucks more than the package I already had, so it was well worth it.
Let me tell you: nothing, _nothing_ Ive seen so far on my new screen, not Blu-Ray discs, not HD channels, not Oblivion on the XBox 360, nothing can compare to the visual feast provided by Drakes Fortune. Its reportedly only 720p, but it _looks_ like its 1080p and then some. I took a bunch of pictures of it while I was playing, cheesy ones on my iphone that Im embarrassed to upload, but it had so many stop-and-stare areas that I just kept taking pics.
Anyway, visuals aside, the game itself is sort of a cross between Tomb Raider (the original, which was pretty fun) and Gears of War, which Ive never played, but people say Drakes Fortune has similar combat mechanics. And its a bit like the recent horror movie The Descent towards the end. Yeah. I didnt see _that_ element coming, but it was pretty darn cool. And scary.
The game is short and linear: an interactive movie, essentially. I took my time with it and still finished in about 13 hours. There are some puzzles, but not too many, and none of them are even remotely difficult to figure out. Which is good, I suppose; in my old age I dont like to spend too long on any one puzzle, so Im pretty quick to go look online for hints if I get stuck. Didnt have to do that for this game, which helped increase the immersiveness.
The game starts off a little slow, to be honest, and after the first hour I was starting to worry that it would suck. You have to stick it out until you parachute onto the second island, after which it gradually picks up the pace until it feels like its a flat-out race to the finish.
Most of the game, especially the latter half, revolves around the combat, nearly all of which is against modern-day pirates. You have to get good at ducking behind walls and fast aiming to have a hope of surviving (although trust me on this one, if a non-FPS gamer like me can master it in a few hours, you can too). I found the combat to be loads of fun, and Im getting ready to play the game through again just for that. You run across a good selection of weapons, including a nifty Dragon Sniper rifle, and you have to make tough decisions about what to carry with you since youve got limited inventory space: one handgun and one rifle or shotgun, basically, plus a little ammo.
The voice acting is really good, much better than it was in Oblivion, although Im sure it was easier since the game is shorter and there are fewer characters. The lines are a little cheesy at first, but either the writing got better as the game progressed, or I just got used to it. Either way, by mid-game it felt pretty believable. In fact I laughed a lot at the characters reactions. The main character, uh, _mumble_ Drake, I forget his first name, says _exactly_ what I would have said, the first time a grenade lands next to him. Its not repeatable here. The swearing/profanity in the game isnt over-the-top, but they use it effectively now and then for humor value.
The AIs are great, and for a lot of the game youve got at least one friend following you around and helping you fight stuff or solve puzzles. They almost feel like real people, and theyre nowhere near the nuisances NPC helpers have been in hundreds of games in the past.
The generator-room sequence is just plain scary. Ive played scarier games, sure - Fatal Frame comes to mind - but this one really had my pulse going towards the end of the game. For an hour, at least, until I finally made it out of there. Im lucky I didnt keel over.
There are a few fast-paced movie-like combat sequences thrown in. Theres one especially memorable scene where youre riding shotgun in a Jeep, and you have to shoot at bad guys coming after you while youre racing through the jungle. Cool stuff. The whole feel of the game is very Indiana Jones, actually. Naughty Dog did a great job with it across the board.
And the visuals - well, they simply defy any possibility of describing them here. Steaming tropical jungles, crumbling Spanish fortresses, foamy waves crashing against the rocks a hundred feet below you, realistic character motion and rendering - the game is sumptuous, no better word for it.
You really have to play this game. You only need to set aside one day for it, so you might as well check it out!
SOME FINAL THOUGHTS
I pretty much bought my XBox 360 for Oblivion, although it wound up being my DVD player for a couple years after my Sony player died. It didnt matter, though, since Oblivion was good enough to warrant the purchase of an XBox 360. And I did find one or two other 360 titles that were fun (e.g. Kameo: Elements of Power, a fun but short romp from ex-Rareware, my ex-favorite game studio.) Im hanging on to my 360 specifically so I can play Fable 2 this holiday season. The original Fable was lots of fun, and had a cool werewolf scene Ill remember to my dying day. I hope the sequel is every bit as good.
But Im clearly someone who doesnt need much incentive to buy a game console.
I pretty much bought my PS/3 for Drakes Fortune, hoping it would be worth it, although its also nice to have a Blu-Ray player. Id have to say Drakes Fortune doesnt singlehandedly justify the purchase of a PS/3 for gaming purposes, but it _does_ illustrate the potential of the platform, and I think there are tons of great titles to come. For a while most of the good games are likely to be ports that are also available on the 360, but they say the PS/3 has a lot more parallel-processing power, so as game developers learn to take advantage of it Im guessing the PS/3 eventually gets the edge.
Its still early-adopter material, though, so if you already own a Blu-Ray player, a PS/3 may not be worth it this year. Drakes Fortune will still be a fun game next year, and will be a lot cheaper than its current $60 price tag.
And with that, its a wrap! Looking forward to seeing your game suggestions in the comments.






























