Planet HashPHP

chip: Announcing Nodul.es: CPAN for Node.js

Paul Querna — Thu, 02 Sep 2010 17:39:37 +0000

Last weekend, our team named “Ponies for Orphans” participated in the Node Knockout competition. The team included 3 of my co-workers from Cloudkick, Russell, Tomaz, Logan, and myself. In 48 hours, we had to build a project based on Node.js.

We were brainstorming ideas before the competition, thinking about all the cool things we could do; We even planned out some multiplayer game ideas. We quickly figured out that none of us had done anything extensive with Canvas or SVG, and the existing 3rd party libraries aren’t very comprehensive, with the possible exception of Processing.js. We also felt that we wanted something that would continue to be used after the competition. We refocused our ideas on projects that would work well with our team composition of being backend programers, and eventually settled on Nodul.es:

Nodul.es: CPAN for Node.js

Nodul.es is a web based view of the NPM package repository for Node.js. Our goal was simple, implement what we liked about CPAN for Perl and Python’s PyPi in 48 hours of coding.

Currently you can browse by:

Let’s look at an example of a module page; Tim Smart’s node-compress module is a good example. We pull out metadata from both the NPM repository, the latest commit from Github, and find all modules that have a dependency upon it.

Internals of Nodul.es

Nodul.es is built around Node.js, using its asynchronous abilities extensively.

We split the system into 3 main components:

Indexer: Indexes the raw data about packages from the NPM Registry. This is just a raw JSON dump from NPM’s CouchDB backend.
Source Downloader: Downloads the latest releases of all NPM modules, and extracts them so we can get extra metadata out about the module.
Webapp: The simple part, pulls data out of our datastore, and displays html pages to end users.

All of these services interact MongoDB, which provides data storage for all of the indexed data, and ways to get it back out for webpages.

We also used several external dependencies in building Nodul.es:

async – For flow control of asynchronous operations.
clutch – For URL routing inside the webapp.
Mu – For HTML Templating in the webapp.
paperboy – For static file serving (ie, CSS/javascript) in the media subdirectory.
prettify – For code highlighting, for a feature not released!
sprintf – For string formatting, in the logs, nice logs are good.

What’s next for Nodul.es

We built Nodul.es in 48 hours, and until the voting is over, we aren’t allowed to change it. But we have a ton of features partially completed that we had to pull because we didn’t want to ship broken and incomplete features, they include:

Source Browser: We want to provide a similar source browsing experience to CPAN in this respect, letting you quickly see how someone is doing something. We already have most of the infrastructure for this, because we have downloaded the source tarballs.
Sitemaps: We are adding Sitemaps, so that all search engines can find the modules easily. Currently finding modules is an odd combination of using command line tools or getting lucky with a web search.
More Github integration: The vast majority of Node.js modules are hosted on Github, so we want to do things like show module development activity, and use that to provide sorts on things like Category pages.
Your ideas: Nodul.es is open source. We want to make it the best module browser for any language out there. Submit Ideas, submit pull requests, lets get going!

Ashley: My Top 5 Artists (Week Ending 2010-8-29)

Wed, 01 Sep 2010 03:26:57 +0000

My Top 5 Artists (Week Ending 2010-8-29):

Imported from Last.fm Tumblr by JoeLaz

Bjorn: "The BBC told us that the floods would cover a third of England"

Mon, 30 Aug 2010 21:40:56 +0000

“The BBC told us that the floods would cover a third of England”

- Oxfam America Blog » Blog Archive » A crisis unlike any other

Laney: OpenPGP key transition

Iain — Sun, 29 Aug 2010 23:24:31 +0000

I recently decided, in advance of any future events I may attend, to join the bandwagon and transition away from my old 1024D key (20BFCDC7) to a lovely new 4096R key (1C5041D4). This post is my transition announcement. You can find the transition document here, and below. Both keys will still be valid for the time being, but I shall not be renewing the old one after it expires on 2010-11-26, and it will be revoked shortly thereafter. As far as possible, I'd appreciate all correspondence to use the new key.

If, after reading the transition document, you feel comfortable signing my key (and have signed the old one), then I'd appreciate your signature via email.

(format of transition document cribbed from Matt Zimmerman's transition, thanks!)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1,SHA256

,----[ OpenPGP key transition ]
| Time-stamp: <2010-08-30 00:08:28 laney>
`----

I've recently set up a new GPG key, and will be transitioning away from
my old one. I have done this in order to migrate to a larger RSA key and
stronger hash functions, and NOT due to any known key compromise.  The
old key will continue to be valid for some time, but future
correspondence should use the new one wherever possible.

This message is signed by both keys to certify the transition.

The old key was:

pub   1024D/20BFCDC7 2007-11-27
Key fingerprint = 2B1E 742E B9CA C441 EA0B  4CBA 3F2D 129C 20BF CDC7
uid                  Iain Lane <iain@orangesquash.org.uk>
uid                  Iain Lane <laney@ubuntu.com>
uid                  Iain Lane <ial@cs.nott.ac.uk>
uid                  Iain Lane <psxil@nottingham.ac.uk>
sub   2048g/5D64CB7D 2007-11-27 [expires: 2010-11-26]

and the new key is:

pub   4096R/1C5041D4 2010-08-29 [expires: 2013-08-28]
Key fingerprint = 3D0E FB95 E7B5 237F 16E8  2258 E352 D5C5 1C50 41D4
uid                  Iain Lane <iain@orangesquash.org.uk>
uid                  Iain Lane <psxil@nottingham.ac.uk>
uid                  Iain Lane <laney@ubuntu.com>
uid                  Iain Lane <ial@cs.nott.ac.uk>
sub   4096R/8FB24134 2010-08-29 [expires: 2013-08-28]

To fetch my new key from a public keyserver, you can run:

gpg --keyserver pgp.mit.edu --recv-keys 1C5041D4

and verify its fingerprint matches the one above:

gpg --fingerprint 1C5041D4

If you already know my old key, you can now verify that the new key is
signed by the old one:

gpg --check-sigs 1C5041D4

If you have previously signed my old key, and you're satisfied that
you've got the correct new key, then I'd appreciate it if you would sign
my new key as well:

gpg --sign-key 1C5041D4

Then I would appreciate it if you would mail me the signed key. A lot of
people like to use caff(1) to sign keys.

Thanks,
Iain Lane
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAkx66HUACgkQPy0SnCC/zccWmwCfXgHVQkIAwjwgSosANYdaCcTd
LoUAn1Emf+obkXmW40qjs7PSDlMv2rN8iQIcBAEBCAAGBQJMeuh1AAoJEONS1cUc
UEHUqTsQALTBe1wAtZ/8StNo7GwMgnFUTmfmgyGG5sX6I3d8N2vUOhQ3uKYMLrxE
5JrtpG06DyTjU4woVDoCkUGtatIVhkaY7Q/U04l/t6w1wC+lfP+EGos117JYtzc4
rFDttJHlo2CzIz695Gk5T+4JRax1VBF8PEQgVqvwMHKG5C+L87QtqDgLFZ1vElQK
qIFIokU7Gfo3tNXQB9uwQlgN4m46nfL/j8FcqWYNOBD5SiB0A78zuDljHKHtGiEi
AYFRjHZA2mha3xePCqOc7uC8dO9Lyqn9RK9bbcFgbW+G3M8o7ZQd/0a5SQifBMp6
8IyFdRuPHIdAF9YcBMaW1cf+ioEL9OIHTMmq3vmGVJIRjwzTtbl/CXaBCAdMvJeh
SlPnm256/wTUkc2np9erJQ9JwfRB84neT3XVd3HE7AdTuMd9E3BTY2Lqav3YI8kI
rVMPD5Yi0tXMfz6dUSU9hKg0ttiklkY9whtSf4sTKvsyw+ZM5apFFK/a0aXaitUA
A0xP7SSc4g5vXd3YBm5JvO4DEFcmuk+lRLSxUy4g7rWWB3hdwwI6jgb9+b7TTHBC
vW4eH4wG7JH4/rIlIwLndQqkbzkY+FMzHfWvMf/xwKCjeHfKF4iw96kUHahu/arR
d2vzHUdiWpQwhsb15EkNcS5De5vE/vjgHHehkmuZwmhjyIzQNNyC
=woWY
-----END PGP SIGNATURE-----

Ashley: scienceygoodness: Etsy find of the day: Because in the words of...

Wed, 25 Aug 2010 18:11:11 +0000

scienceygoodness:

Etsy find of the day: Because in the words of the description itself, “So that while children are coloring, they are also exposed to the names of chemicals that will make those colors! So instead of thinking “I want green” they will think “I want Barium Nitrate Ba(NO3)2…”

Ashley: Me + coat = yay. Linky to M&S model wearing one...

Wed, 25 Aug 2010 17:48:16 +0000

Me + coat = yay. Linky to M&S model wearing one here… http://www.marksandspencer.com/Long-Sleeve-Funnel-Belted-Anorak/dp/B003VFAM04

It’s also water resistant (for a coat to actually be waterproof you usually need to go to an outdoors store, or a sports shop), so I finally own a practical coat again…

Apologies for the mobile phone shot, this was originally a quick snap in store to assess rolphus’ opinion from home!

Bjorn: "It’s no accident that in 21st century America, torture has been mainstreamed, climate denial has..."

Tue, 24 Aug 2010 21:21:33 +0000

“It’s no accident that in 21st century America, torture has been mainstreamed, climate denial has taken firm hold, book burning, racial dog whistles and brazen religious intolerance are part of our discourse and par for the course. This is how the right plays the game, using Limbaugh, Hannity, Fox, Drudge, blogs, chain emails, talk radio, etc. to shamelessly and defiantly drag the conversation as far right as possible.”

- The glaringly simple formula for rightwing dominance of our national debate : Peter Daou

Bjorn: "There is a simple formula for rightwing dominance of our national debate, even when Democrats are in..."

Tue, 24 Aug 2010 21:20:59 +0000

“There is a simple formula for rightwing dominance of our national debate, even when Democrats are in charge: move the conversation as extreme right as possible, then compromise toward the far right.”

- The glaringly simple formula for rightwing dominance of our national debate : Peter Daou

Ashley: My Top 5 Artists (Week Ending 2010-8-22)

Tue, 24 Aug 2010 14:56:37 +0000

My Top 5 Artists (Week Ending 2010-8-22):

Imported from Last.fm Tumblr by JoeLaz

chip: Writing Node.js Native Extensions

Paul Querna — Mon, 23 Aug 2010 17:34:01 +0000

Have a big blog post over on the Cloudkick Blog about Writing Node.js Native Extensions.

Bjorn: "Successful dieters lose weight while they are sleeping;"

Sat, 21 Aug 2010 22:18:01 +0000

“Successful dieters lose weight while they are sleeping;”

- Breakfast: The Most Important Meal of An Athlete’s Day | Active.com

Ashley: "This sort of thing is what makes me so uneasy about trusting Google with anything. It’s the same..."

Sat, 21 Aug 2010 22:11:08 +0000

“

This sort of thing is what makes me so uneasy about trusting Google with anything. It’s the same story: Google is “open” with the products that don’t make them money and closed with those that do, using “open” as a marketing buzzword against Apple and hoping nobody notices how incredibly closed and secretive most of their products and operations really are.

iOS is far more “closed” than Android, but at least Apple doesn’t try to bullshit me about it. They put it right out there. “We control everything because we think it’s better that way. If you don’t like it, there’s the door.”

And since they’re honest with me, I trust them more.

”

- Marco Arment

Exactly.

I think I missed this when Marco reblogged me back in July, but he nailed Google’s bullshit on the head (thanks to Till Matthis Massen for bringing it to my attention). And with Google’s full-frontal assault on net neutrality, I’m experimenting with cutting my ties.

I’ve deleted my Google Apps account and one Google account, switched to Fever for reading news, and Skype for my work proxy phone number (it works great now that Skype can accept calls in the background on iOS 4 and doesn’t require minutes for phone calls, unlike Google Voice). I just need to drop Feedburner from 1FPS and Finer Things, and I can delete my personal Google account altogether.

(via chartier)

Ashley: buddhabrot: fuckyeahcrystals:whisperingwillow:fuckyeahmineralsan...

Sat, 21 Aug 2010 22:09:42 +0000

buddhabrot:

fuckyeahcrystals:whisperingwillow:fuckyeahmineralsandfossils:

Elbaite, Na(Al_1,5Li_1,5)Al₆(OH)₃(OH)(BO₃)₃Si₆O₁₈

Locality: Madagascar

Ashley: lemdi: thetardis: chrisdwoo: shakingtambourine: kaletzibarec:...

Sat, 21 Aug 2010 22:06:39 +0000

lemdi:

thetardis:

chrisdwoo:

shakingtambourine:

kaletzibarec:

TRUTH!

dunnosource, graphjam.com

THIS.

Only missing cracks, boxes, and rooms I can’t see.

Bjorn: It’s so strange, after some weight loss, to recognize a set of pants on your shelf that at...

Sat, 21 Aug 2010 17:32:21 +0000

It’s so strange, after some weight loss, to recognize a set of pants on your shelf that at first glance leave you with an ominous feeling since for quite some time they haven’t fit you. But then they do, baggy even, yet that lingering feeling that these light faded jeans haven’t fit me in quite some time doesn’t go away, even as I’m wearing them.

You’d think I should feel good about this, but after years of gaining weight again and again, each time to a new horrible new record, I don’t. Weight wise, I’m just about where I was when I started my doomed diet two years ago: 257 lb. I’ve lost 25 lb. in the last four months and I’ve only yet made it to the point where two years ago when 257 lb. was such a horror to me that it spurred me to go on a drastic diet.

In other news, I completed two weeks of the couch to 5k plan but am going to take a week off now because my shins are in too much pain. I will start over in a week. Hopefully a combination of weight loss and stronger bones at the end of the week will lead me to finish the program.

I’m just trying to be careful, moderate, not get hurt, not get too hungry.

Bjorn: BBC News - Last US combat brigade exits Iraq

Thu, 19 Aug 2010 04:50:00 +0000

BBC News - Last US combat brigade exits Iraq:

The war is ending. 4,415 US troops lost their lives in Iraq. I wish it was 4,414. I wish it was 0. Rest in peace Sergeant Major Ellis. I will never forget the example he set for excellence. I have never before or since met anyone so dedicated to his work, his Marines, and to the Corps. This man was a leader. He gave his life, both figuratively and literally, to what he believed in. He was also a pretty good guy, who cared about what his Marines thought of him. He was a father who left behind a young daughter. I know his death was not in vain, that his service was not without purpose, but I hope the mission he served becomes a success, that a free, friendly and prosperous Iraq will become a reality and bring honor to his sacrifice.

ELLIS, JOSEPH J
SGTMAJ US MARINE CORPS
DATE OF BIRTH: 09/08/1966
DATE OF DEATH: 02/07/2007
BURIED AT: SECTION 60 SITE 8536
ARLINGTON NATIONAL CEMETERY

Bjorn: "Just 61% of the adult population, age 20 or over, has any kind of job right now."

Sun, 15 Aug 2010 22:07:27 +0000

“Just 61% of the adult population, age 20 or over, has any kind of job right now.”

- ROI: Is a Crash Coming? Ten Reasons to Be Cautious - WSJ.com

SIR-Millar: Run… and a move to Cambridge

Andy Millar — Fri, 13 Aug 2010 10:30:10 +0000

I suppose things have been a little quiet recently; but I can blame that entirely on having to commute 2 hours a day to London for work… Yes, I’ve (temporarily) moved to Cambridge.

In other news, I went for a short run yesterday morning, only about 2 miles, but here’s the route:

chip: Async TLS

Paul Querna — Wed, 11 Aug 2010 09:17:27 +0000

We started discussing TLS in Node.js at the meetup in Palo Alto tonight.

Lets imagine you wanted to implement SSL/TLS in an Asynchronous framework, like node.js.

For the sake of discussion, I will be using OpenSSL as an example. At least as far as I know, these issues also apply equally to GnuTLS or NSS. I would be happy to be wrong!

The Goal

The goal is to provide both a TLS Client and Server API, allowing high level code to determine many of the common behavoirs you need to hook to provide a powerful TLS Platform. This includes basics like verification of certificates chains, but should also include: SSL Session Caching, OCSP stapling, SNI Validation, SPDY Protocol hinting, and more.

The Problem

OpenSSL can decouple IO operations from sockets, using the BIO abstraction. This means your process can handle the actual socket, and its buffers, which is good for Node.js, and for most other asynchronous systems that don’t want to block for SSL to do work.

While the IO operations has a good abstraction in OpenSSL, many common operations, rely upon a callback.

For example, lets consider the OpenSSL SSL Session Cache API:

SSL_CTX_sess_set_new_cb(ctx,    ssl_callback_NewSessionCacheEntry);
SSL_CTX_sess_set_get_cb(ctx,    ssl_callback_GetSessionCacheEntry);
SSL_CTX_sess_set_remove_cb(ctx, ssl_callback_DelSessionCacheEntry);

It is a basic caching API, you have 3 functions for caching an SSL Session object, Add new, Reading existing, and deletion.

If you examine the function signature for the get function, it returns an SSL_SESSION object directly, meaning when you return from the function you must either have the correct session, or return NULL to indicate a cache miss:

SSL_SESSION *ssl_callback_GetSessionCacheEntry(SSL *ssl,
                                               unsigned char *id,
                                               int idlen, int *do_copy)
{
  /* Your SSL Session cache goes here! */
  return NULL;
}

The difficulty for async systems here, is that they most likely want to now perform file IO, network IO, or potentially other operations that go outside the current C stack in order to fetch the Session.

In Node.js’ case, this means you cannot provide a callback as users expect it to work in Node — they expect to be able to make an async callback, and then notify the caller when they have found the data.

In an ideal world, the Node.js api would look something like the following:

var sslctx = crypto.createContext{key: privateKey, cert: certificate,
session_cache_get: function(session_id, result_callback) {
  memcached.get(session_id, function(data, err) {
    result_callback(data, err);
  })
}});
var server = http.createServer(..);
server.setSecure(sslctx);
server.listen(8443);

We started talking through the ideas. How could you accomplish this API for TLS in Node?

This cannot work with the standard OpenSSL callbacks, because of how Node.js works, after the initial cache get call returned undefined, we would unwind up the C-stack, and we have no way to notify OpenSSL later on that we got a Session Cache from memcached.

Possible Hacks

There are a few more hackish ways to solve this, they include:

Using Co-routines from C. Something like libtask could be used to jump out of the OpenSSL stack, back down to Node.js, and it could resume again once we go the response for the session.
Running every SSL Context inside a dedicated thread. When a callback is invoked, dispatch a message to the main thread, where Node.js will notify the waiting thread once it has an answer. I think this is actually one of the easier solutions, but it kills the promise of an Evented framework like Node.js, and not having a 1:1 client to thread mapping.

The Rabbit Hole

Hey guys, what if we just implemented the a TLS Protocol parser?

It wasn’t a new idea. But then we started talking it through the idea of implementing a TLS protocol parser, but still using OpenSSL for all of the actual cryptography, it seemed to make more and more sense. This would let an http-parser style API be used for TLS, which as far as any of us know, has not been done. The parser could be written in C (or javascript, but thats irrelevant), the TLS record protocol itself isn’t too complex, it consistents of a few fixed width fields, a few optional fields, but most of the complexity comes from the implementation of all the cryptography, which none of us have an interest in replacing.

I am scared. Reimplementing SSL or TLS just seems wrong.

But on the other hand, most SSL implementations are tightly coupled to their cryptographic libraries, GnuTLS perhaps being the least so, but these libraries we still designed before many evented style programing paradigms became popular. It seems like there is a niche to be filled by a liberally licensed, TLS record protocol parser library, which provided stubs to use OpenSSL (or another) backend for the actual cryptography, but basing everything on callbacks to user code.

Is this insane?

Bjorn: Monkey Saves Puppy In Nanjing Explosion

Tue, 10 Aug 2010 16:50:47 +0000

Monkey Saves Puppy In Nanjing Explosion

Bjorn: Op-Ed Columnist - The Marriage Ideal - NYTimes.com

Tue, 10 Aug 2010 02:52:08 +0000

Op-Ed Columnist - The Marriage Ideal - NYTimes.com:

It’s an interesting article and I agree with some of the quotes I’ve posted, but the conclusion that a marriage of individuals that share the same sex isn’t equal to that of a woman and a man is downright shameful. If I had read the article to its conclusion before I started quoting it I probably would not have posted anything about it at all.

Bjorn: "a culture in which weddings are optional celebrations of romantic love, only tangentially connected..."

Tue, 10 Aug 2010 02:44:47 +0000

“a culture in which weddings are optional celebrations of romantic love, only tangentially connected to procreation, has no business discriminating against the love of homosexuals.”

- Op-Ed Columnist - The Marriage Ideal - NYTimes.com

Bjorn: "The lifelong commitment of a gay couple is more impressive than the serial monogamy of straights."

Tue, 10 Aug 2010 02:44:12 +0000

“The lifelong commitment of a gay couple is more impressive than the serial monogamy of straights.”

- Op-Ed Columnist - The Marriage Ideal - NYTimes.com

Ashley: Living in the future. Note this is a train.

Sun, 08 Aug 2010 08:39:14 +0000

Living in the future. Note this is a train.

Ashley: My Top 5 Artists (Week Ending 2010-8-1)

Tue, 03 Aug 2010 08:20:37 +0000

My Top 5 Artists (Week Ending 2010-8-1):

Imported from Last.fm Tumblr by JoeLaz

Lunar_Lamp: Restrict Domains Postfix Sends To

Lunar_Lamp — Mon, 02 Aug 2010 18:24:12 +0000

Another entry tagged "Er, I can't believe I didn't know that" with a side note in "bleedin' obvious".

To arbitrarily restrict which domains a postfix instance will send to (when receiving email via SMTP) you can simply add the following lines to your main.cf:

relay_domains = example.com,lunarlamp.co.uk,google.com
smtpd_recipient_restrictions = reject_unauth_destination

I promise to post some interesting stuff soon, rather than these banal little snippets!

Bjorn: landscapelifescape: Cardwell, Queensland, Australia pacific...

Sun, 01 Aug 2010 08:00:49 +0000

landscapelifescape:

Cardwell, Queensland, Australia

pacific morning (by paul (dex))

Bjorn: via images.4chan.org

Sun, 01 Aug 2010 04:21:00 +0000

via images.4chan.org

Bjorn: "You should ease into your running program gradually. In fact, the beginners’ program we..."

Sun, 01 Aug 2010 03:58:55 +0000

“You should ease into your running program gradually. In fact, the beginners’ program we outline here is less of a running regimen than a walking and jogging program. The idea is to transform you from couch potato to runner, getting you running three miles (or 5K) on a regular basis in just two months.”

- Cool Running :: The Couch-to-5K ® Running Plan

Going to do this! I want to run again! This looks perfect!

Laney: Tumblin'

Iain — Sat, 31 Jul 2010 20:43:05 +0000

Greets $world. I recently decided to keep this blog for technical/development posts only, as it's syndicated on Planet Ubuntu{,-uk} and I would feel somewhat self-conscious posting my usual drivel to such an esteemed planet. Perhaps that's why I rarely post here (not that I really worry about my post frequency any more).

More fun/trivial/banal stuff will be posted over at my Tumblr blog. I'd very much welcome your follows/subscriptions, as it feels a bit lonely over there right now.

Ashley: Sooperdave is awesome, and I’ve posted his stuff before....

Sat, 31 Jul 2010 18:49:49 +0000

Sooperdave is awesome, and I’ve posted his stuff before. Click on t’image to go to his site.

Ashley: Charlie Brooker in kitchenware. By Cakie of b3ta.com. Their...

Sat, 31 Jul 2010 18:45:00 +0000

Charlie Brooker in kitchenware. By Cakie of b3ta.com. Their profile is here. This guy does some awesome things with cross-stitch, so clicking is worth it.

Ashley: Yay for eclectech at b3ta.com! Click on the image to go to his...

Sat, 31 Jul 2010 18:40:30 +0000

Yay for eclectech at b3ta.com! Click on the image to go to his site

Ashley: Online Food Shopping... just wtf do we do it?

Sat, 31 Jul 2010 16:43:00 +0000

The gearbox died on the mister’s car in Denmark. Hilarity ensues, especially when the gearbox of the *tow truck* also died when taking the car away…

We’re back home and safe and holidayed out for a bit, but we do need to eat. I’ve done a stop-gap shop at the mini-Tesco outside our flat but we do really need a proper food shop. So I’m trying to shop for food without seeing it. This is baffling me.

What on *earth* do we eat? I can pin down the basics (milk, bread, eggs) and a few meals we tend to make (beef/chicken stir fry, nombanara, pie and gravy) but I’m just stumped. I’m sitting with half my cookbooks round me wondering why this is so easy when you can actually see the food.
Rolphus has a dairy allergy. Apart from waitrose.com there is *no* easy way of selecting “no cow juice yo” for sections of the food store. Surely we should be taking advantage of the magic database and allow the user to automatically eliminate certain items?
How on earth do you know everything isn’t going to turn up with a use-by date of tomorrow? In the supermarket itself we’re usually fairly canny with this and go to the back to pick the items to make sure we can get 7 days out of the shop… there’s no way of doing this online. And most of these vendors have a “minimum order amount”, so ordering every few days isn’t possible for a couple.
Why do most of sites require my email address *to* *see* *the* *prices*? Tesco already have this signed up as uselessbastards@leymoo.com in some petty attempt to make myself feel better. I figured I’d have a look, trying to be fair and all. Not bothering with Asda though. Surely people only shop there when they have no choice? It’s not as if it’s cheaper, and the food is awful, and their idea of making something “premium” is adding a ton of sugar and cream and not bothering the check the taste.
I still feel nervous ordering something such as a beef shoulder, steak or lamb without knowing the quality of the batch. Maybe that’s the scabby housing-estate part of me, or something.

Has anyone managed to get over this? Or does everyone just order and hope?

Bjorn: RSA Animate - Drive: The surprising truth about what motivates...

Fri, 30 Jul 2010 21:03:11 +0000

RSA Animate - Drive: The surprising truth about what motivates us (via theRSAorg)

Bjorn: landscapelifescape: Brooklyn Bridge, New York City, NY,...

Fri, 30 Jul 2010 07:38:56 +0000

landscapelifescape:

Brooklyn Bridge, New York City, NY, USA

East Pier (by Timothy Schenck)

Bjorn: "You said it’s been tough for me, but the truth is it’s not tough for me,” Obama..."

Thu, 29 Jul 2010 23:25:58 +0000

““You said it’s been tough for me, but the truth is it’s not tough for me,” Obama said. “I don’t spend a lot of time worrying about me. I spend a lot of time worrying about them.””

- President Obama calls African-Americans a ‘mongrel people’ - TheHill.com

Bjorn: "people who have something to do, even something pointless, are happier than people who sit idly"

Thu, 29 Jul 2010 15:25:13 +0000

“people who have something to do, even something pointless, are happier than people who sit idly”

- To make one happy, make one busy

Bjorn: "What happened is that I saw a movement. I stopped. I was talking to an empty chair, but out of my..."

Thu, 29 Jul 2010 04:25:45 +0000

“What happened is that I saw a movement. I stopped. I was talking to an empty chair, but out of my peripheral vision I saw something move. I look at Ildefonso and he had just become rigid! He actually sat up in his chair and became rigid. His hands were flat on the table and his eyes were wide. His facial expression was different from any I’d seen. It was just wide with amazement!
And then he started-it was the most emotional moment with another human being, I think, in my life so that even now, after all these years, I’m choking up [pauses]-he started pointing to everything in the room, and this is amazing to me! I’ve thought about this for years. It’s not having language that separates us from other animals, it’s because we love it! All of a sudden, this twenty-seven-year-old man-who, of course, had seen a wall and a door and a window before-started pointing to everything. He pointed to the table. He wanted me to sign table. He wanted the symbol. He wanted the name for table. And he wanted the symbol, the sign, for window.
The amazing thing is that the look on his face was as if he had never seen a window before. The window became a different thing with a symbol attached to it. [emphasis added, GD] But it’s not just a symbol. It’s a shared symbol. He can say “window” to someone else tomorrow who he hasn’t even met yet! And they will know what a window is. There’s something magical that happens between humans and symbols and the sharing of symbols.
That was his first “Aha!” He just went crazy for a few seconds, pointing to everything in the room and signing whatever I signed. Then he collapsed and started crying, and I don’t mean just a few tears. He cradled his head in his arms on the table and the table was shaking loudly from his sobbing. Of course, I don’t know what was in his head, but I’m just guessing he saw what he had missed for twenty-seven years.”

- Life without language « Neuroanthropology

Bjorn: "Police officers know for sure that nobody will hold them accountable if a crime is not solved, but..."

Wed, 28 Jul 2010 05:02:56 +0000

““Police officers know for sure that nobody will hold them accountable if a crime is not solved, but they will be held accountable if they allow a demonstration to occur,” Mr. Makarov said. “They will not be held accountable for putting an innocent person in jail or beating one on the street, but they will be if somebody takes a stand against the authorities.””

- Videos Rouse Russian Anger Toward Police - NYTimes.com

Ashley: Home and tired…

Sun, 25 Jul 2010 19:04:15 +0000

Home and tired…

Ashley: Currently at Kloopy’s watching the F1 Forum.

Sun, 25 Jul 2010 15:24:23 +0000

Currently at Kloopy’s watching the F1 Forum.

Ashley: Hamburg hotel didn’t have wi-fi in rooms. So here’s...

Fri, 23 Jul 2010 17:00:07 +0000

Hamburg hotel didn’t have wi-fi in rooms. So here’s my current location…

Bjorn: All Summer - Kid Cudi, Vampire Weekend and Best Coast (Download...

Tue, 20 Jul 2010 21:49:55 +0000

All Summer - Kid Cudi, Vampire Weekend and Best Coast (Download Link) (via xSsj3Sasukex)

Bjorn: NIGHTWAVES – REMIXES (Electro/Dreamwave/Synthpop – US)

Tue, 20 Jul 2010 20:05:21 +0000

NIGHTWAVES – REMIXES (Electro/Dreamwave/Synthpop – US)

Bjorn: SUNBIRDS – RIVER RUN (Alternative/Pop/Rock – UK)

Tue, 20 Jul 2010 19:42:02 +0000

SUNBIRDS – RIVER RUN (Alternative/Pop/Rock – UK)

Bjorn: Jonathan Boulet - A Community Service Announcement (via...

Tue, 20 Jul 2010 19:32:04 +0000

Jonathan Boulet - A Community Service Announcement (via modularpeople)

Bjorn: saturday nite with… LYKKE LI

Tue, 20 Jul 2010 19:29:38 +0000

saturday nite with… LYKKE LI:

For the past two years or so, Swedish songstress Lykke Li has been a mainstay of every electro music blog around the world. Ever since the release of her debut album, Youth Novels, Li’s sweet and sour compositions have been a constant source of inspiration for producers and DJs, with new remixes popping up every second day…

Bjorn: lubita77: #diadoamigo...

Tue, 20 Jul 2010 16:57:19 +0000

lubita77:

#diadoamigo #emo helpmeunderstand:

saaoliveiraa:

(via maybeilikethat)

Ashley: My Top 5 Artists (Week Ending 2010-7-18)

Tue, 20 Jul 2010 02:56:45 +0000

My Top 5 Artists (Week Ending 2010-7-18):

Imported from Last.fm Tumblr by JoeLaz

Bjorn: Sleeping Fallacies

Mon, 19 Jul 2010 03:41:56 +0000

You know in the Disney version of the story of Sleeping Beauty, the “evil” witch makes just one person fall asleep forever, but the “good” fairy god mothers put the entire kindgom to sleep (so they wouldn’t learn that their princess had fallen asleep). More than just entire kindgom, also the king and at least some from the neighboring kingdom of Prince Phillip. Something doesn’t add up. I suppose intentions matter. The evil witch Maleficent’s intentions were jealousy and the fairy god mothers were to protect people from pain and sadness but…I think they would have gotten over it in a few days, or somewhat later. There’s also the part about the King ordering the entire village to destroy every spindle to protect the princess. That’s just plain tyranny right there. You couldn’t do that in Walt Disney’s favorite country, the United States, since we have protections against seizure of private property. But yet when you tell a story painted in pretty colors and apply the appropriate melodies at various parts to enhance the story it just all goes down like bitter medicine on a spoonful of sugar. Maleficent is really the only interesting character in that entire story and I would have loved it if she had, in her beautiful dragon form, eaten up the entire tyrannical nobility so that the plebs might have had some freedom, although I doubt she would have been so kind.. Yet the number of people harmed by her compared to the number harmed by the “good” fairy god mothers seems out of sync with the labels applied to each. That’s all.

Ashley: :D

Sun, 18 Jul 2010 12:32:53 +0000

Bjorn: joshsternberg: Breathtaking photos of bioluminescence (via...

Sat, 17 Jul 2010 16:29:35 +0000

joshsternberg:

Breathtaking photos of bioluminescence (via @stevesilberman)

Ashley: I believe I was 8 or 9 here… eep.

Sat, 17 Jul 2010 01:19:37 +0000

I believe I was 8 or 9 here… eep.

Ashley: My Top 5 Artists (Week Ending 2010-7-11)

Mon, 12 Jul 2010 23:44:25 +0000

My Top 5 Artists (Week Ending 2010-7-11):

Imported from Last.fm Tumblr by JoeLaz

the_angry_angel: Android & Samsung Galaxy S Fun

the_angry_angel — Mon, 12 Jul 2010 18:02:10 +0000

I've recently acquired a new phone to replace my aging iPhone 3G: The Samsung Galaxy S (or the GT-i9000 everywhere outside of the UK [apparently]). This is my first dip into the Android world, and I've got mixed feelings about it.

In the past I had Windows Mobile devices for work, around the 5 series, and in many ways Android reminds me very much of that experience. Now hear me out before you start tracking me down and attempting to kill me: Applications vary wildly on quality and the operating system gives you a reasonable amount of flexibility at the expensive of battery life and in some instances a complexity of use.

After 2 years of almost exclusive iPhone use I'm starting to remember what the appeal was of the original iPhone, despite the somewhat draconian strangle hold Apple has over the platform. From time to time I yearn for a dictator to swoop down and with an ironfist make every useful application I've installed follow some sort of coherent usability and style guidelines. I long for someone to clear the piles of crap from the Application Marketplace, or at least create some clever system where they don't bubble to the surface.

Despite this I do still love the Android platform, but it just feels like it's not quite finished - yet. The default mail application needs some love (or better yet, to merge the changes from k9), the browser needs a little tweaking, and so on and so on. I'm not yet regretting the move to Android, and I'm not sure that I will for sometime to come, if at all. The fact that I now have a mass storage device in my pocket at all times, which is also a hackable unix box underneath is very comforting.

As for the Samsung Galaxy S, it's a great bit of hardware on paper. In real life I feel it does need more RAM, and the battery really does need to be better. With some faffing it's acceptable. This is in part down to some of the applications that are bundled, and in part down to the massive screen and GPS, but even with just bluetooth on in the car it really does drain more quickly than I've become accustomed to. If anything it's another reminder of the Windows Mobile days of yore. The build quality is one place where I really feel like I've been spoiled by Apple. I've become very used to aluminium laptops^[1] and phones that feel solid. On a favourable day I would describe the Galaxy S, as "classic phone" or "classic Samsung", and on an unfavourable day as "cheap". It's a bit of a let down considering the outstanding specs on the rest of the phone.

The one thing that is the "killer app" for this phone is the inclusion of Swype as standard. Whilst not exclusive to the Galaxy S, or even Android, I can tell you that this; Swype should become the defacto standard for input. The fact that the iPhone is inflexible enough to allow modifications to this extent places the nail in Apple's coffin as a mobile phone supplier for me.

If you've got any phone with Swype included in the default ROM I highly recommend that you enable it and have a play. If you're unlucky enough not to have a compatible phone trawl the nearest city until you find someone with one so you can understand why I feel it's so great. It may sound completely nuts, or like a gimmick, but it's really not.

[1] Having had accidents with laptops, ranging from dropping off tables to having rackmount servers and IBM thinkcentres landing on them, only my previous Apple Macbook Pro survived to live another day. To me this makes the case for decent build quality.

chip: Overclocking mod_ssl

Paul Querna — Sat, 10 Jul 2010 19:51:11 +0000

At Velocity, I saw Adam Langley give a great presentation entitled Overclocking SSL. Last week Adam posted a distilled version of the Overclocking SSL presentation on his blog.

He covers many topics for improving SSL performance. Unfortunately, his recommendations are decidedly focused on how Google runs their servers, and not a practical guide to how to improve your performance with a more standard Apache 2 and mod_ssl setup. Since I don’t work at Google, but I like my web servers to be fast, I decided to try as many of his recommendations as possible with mod_ssl.

Disclaimer I am not a cryptanalyst. Be paranoid when you are messing with SSL, small mistakes can invalidate your entire security framework. Ask your local cryptanalyst about these changes!

Basic Configuration: Certificate Key Size

Google uses a 1024bit RSA key for their encrypted websites. However, Certificate Authorities are no longer issuing new 1024 bit keys, because the CAB Forum has required them to be phased out at all levels. It is believed these small keys are insecure, so for pratical purposes this means you will want a 2048bit key. Make sure you do not use a 4096 bit key, the key operations are about 5 times slower — make sure you have a 2048bit key, it strikes the balance of speed and security.

The Certificate key sizes doesn’t just affect how many CPU cycles that are used for the calculations, the public versions of the keys are sent to the client when it connects. I go into more detail about TCP round trips bellow, but if your certificate is a 4096 bit key, it means your clients need to download double the data to even get started.

Basic Configuration: Picking Ciphers

The SSLCipherSuite directive controls the ciphers that mod_ssl will negotiate with clients. The string parameter is complicated — it is a combination of aliases of ‘HIGH’, “LOW”, old names, specific names, etc. To see what OpenSSL actually enables, you’ll want to use the `openssl ciphers` command.

This is what you get for the default configuration of mod_ssl:

$ openssl ciphers 'ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP'

DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:AES256-SHA:DHE-RSA-AES128-SHA:
DHE-DSS-AES128-SHA:AES128-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:
DES-CBC3-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:SEED-SHA:RC4-SHA:RC4-MD5:
EDH-RSA-DES-CBC-SHA:EDH-DSS-DES-CBC-SHA:DES-CBC-SHA:DES-CBC3-MD5:
RC2-CBC-MD5:RC4-MD5:DES-CBC-MD5:EXP-EDH-RSA-DES-CBC-SHA:
EXP-EDH-DSS-DES-CBC-SHA:EXP-DES-CBC-SHA:EXP-RC2-CBC-MD5:EXP-RC4-MD5:
EXP-RC2-CBC-MD5:EXP-RC4-MD5

The exact list will depend upon your version of OpenSSL, but on most modern operating systems, the first cipher that will be attempted to be used is AES-256. AES-256 is without a doubt a more secure selection, but it isn’t what Google is using. They are using the older RC4 (aka ARC4) cipher, with SHA1 hashing. There have been many different attacks on RC4, many due to bad implementations, but as long as it is used correctly, it is still secure enough. The selection of a cipher is still a judgement call for your product, but RC4 is approximately 3x faster than AES-256 on most machines right now.

In Apache, lets configure it to try to use RC4 w/ SHA1 hashing:

SSLCipherSuite RC4-SHA:AES128-SHA:ALL:!ADH:!EXP:!LOW:!MD5:!SSLV2:!NULL
SSLHonorCipherOrder on

The SSLHonorCipherOrder directive is used to force the server’s cipher choice on to the client.

And lets run the Cipher Suite string through `openssl ciphers` you can see the exact configurations that are being allowed:

$ openssl ciphers 'RC4-SHA:AES128-SHA:ALL:!ADH:!EXP:!LOW:!MD5:!SSLV2:!NULL'

RC4-SHA:AES128-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:SEED-SHA:
DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:AES256-SHA:DHE-RSA-AES128-SHA:
DHE-DSS-AES128-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:DES-CBC3-SHA

This will use RC4, and fall back to AES-128, before going to other stronger ciphers, but over the defaults, it is significantly faster.

SSL Session Cache and Resumption

mod_ssl’s supports a plugable backend for storing client sessions with the SSLSessionCache directive. The two most commonly used are the shm and dbm on a single machine. The shm backend is faster than dbm, and should be used in almost all cases.

However, as Adam noted, most people have more than one machine doing SSL Termination. This means a distributed SSL session cache is needed. I wrote the patch for mod_ssl to support a memcached SSL Session cache 3 years ago. This patch wasn’t backported, so you’ll need to use Apache 2.3.x, which is currently in Alpha. To configure it, just pass a list of memcached nodes:

SSLSessionCache memcache:10.0.0.1,10.0.0.2,10.0.0.3

Reducing Round Trips

The best tool to measure this is Wireshark, so you can see both the volume of data, and the round trips. The easy way to test with this is using the `openssl s_client` command. This command lets you easily create SSL connections, and tune various things on both the client and server.

Here is a truncated example of using s_client against encrypted.google.com:

$ openssl s_client -debug -tls1 -host encrypted.google.com -port 443
..... data dumps .....
---
SSL handshake has read 1893 bytes and written 285 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA
....

The interesting parts you can see here are both the negotiated ciphers, and the total bytes written by each side to establish the connection. The majority of the data sent by the server is from the size of the server certificate.

As Adam discussed in depth, because many certificates have a chain, and most are at least 2048 bits long, it is very easy for a new TCP connection to overflow your initial TCP window. Your goal is to make sure you are sending the correct chain, but not sending too much or irrelevant certificates. Here is a example of www.cloudkick.com, which uses the GoDaddy CA, and an intermediate certificate:

$ openssl s_client -tls1 -host www.cloudkick.com -port 443 -debug

---
Certificate chain
 0 s:/O=*.cloudkick.com
      /OU=Domain Control Validated
      /CN=*.cloudkick.com
   i:/C=US
     /ST=Arizona
     /L=Scottsdale
     /O=GoDaddy.com, Inc.
     /OU=http://certificates.godaddy.com/repository
     /CN=Go Daddy Secure Certification Authority
     /serialNumber=07969287
 1 s:/C=US
       /ST=Arizona
       /L=Scottsdale
       /O=GoDaddy.com, Inc.
       /OU=http://certificates.godaddy.com/repository
       /CN=Go Daddy Secure Certification Authority
       /serialNumber=07969287
   i:/C=US
     /O=The Go Daddy Group, Inc.
     /OU=Go Daddy Class 2 Certification Authority
---
...............
---
SSL handshake has read 2974 bytes and written 422 bytes
---
...............

In this case, the server sent the both the certificate for *.cloudkick.com, and the Go Daddy intermediate certificate. Try as we might, the server in this case had to send 2974 bytes to get started, over 1000 bytes more than what encrypted.google.com needed. This is just a reality of using a chain certificate, and using 2048 bit keys. Just make sure you aren’t sending extra certificates, and to keep your data bellow 4kb to prevent an ACK being needed in the small windows as TCP connections are being started.

OCSP Stapling

One of the biggest problems with the existing SSL infrastructure is that validating the status of a certificate is hard and slow. OCSP Stapling doesn’t make it easier to understand, but it does at least make it faster. OCSP stapling support was originally funded from a grant by Mozilla. It has been added to Apache httpd 2.3, so you’ll need to download that alpha release in order to use it.

OCSP Stapling takes the Certificate’s Authorities OCSP response and bundles it in the initial response to the client. This OCSP response is a cryptographic signature verifying your certificate is still valid for X days. This means the client doesn’t need to resolve another DNS name, and hit another service just to validate your certificate.

In Apache 2.3 and above, the configuration to enable OCSP Stapling is quite simple; Just put these directives in your global scope:

SSLUseStapling on
SSLStaplingCache "shmcb:logs/stapling_cache(128000)"

You can test OCSP stapling using the `openssl s_client` command again and the -status parameter:

$ openssl s_client -host encrypted.google.com -port 443 -tls1  -tlsextdebug  -status
....
OCSP response: no response sent
....

Even Google hasn’t enabled OCSP stapling yet!

If OCSP stapling was enabled, you would see something like this as the output:

OCSP response:
======================================
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.",
                           OU = http://certs.godaddy.com/repository/,
                           CN = Go Daddy Validation Authority
    Produced At: Jul 10 17:18:44 2010 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: 70292276537F1ABC8FD53C9484E914CB762A052A
      Issuer Key Hash: FDAC6132936C45D6E2EE855F9ABAE7769968CCE7
      Serial Number: 047C0A27B3C295
    Cert Status: good
    This Update: Jul 10 14:15:00 2010 GMT
    Next Update: Jul 10 23:18:44 2010 GMT

Here my server provided a signature from Go Daddy, saying that my certificate was valid for at least another 5 hours.

False Start, Snap Start and Next Protocol Extensions

Google has proposed a series of extensions and modifications to the TLS protocol in order to reduce round trips, both at the initial negotiation, and when to start sending client data.

TLS False Start is mostly a client change, but even if you wanted to implement the proposed server false start, it really depends upon OpenSSL updates to support it. The only recommendation here is to not use ancient versions of OpenSSL — which is important anyways because of the SSL Renegotiation attacks discovered last year.

The Snap Start proposal will need server support, but currently no released version of OpenSSL supports it yet.

Next Protocol Negotiation Extension lets the client tell the server that it is gong to change protocols once the SSL negotiation finishes. Conceptually to me this is similar to Server Name Indication, where the client is leaking application logic to the SSL layer. This will make upgrades to the SPDY protocol faster, but again there is not a released version of OpenSSL with support yet.

The missing patches

Adam mentions a patch reducing OpenSSL’s default buffer allocations from 50kb to 5kb, and suggests the Tor project has a similar patch. I have been unable to find it.
I was unable to find any patches for the Next Protocol Negotiation Extension.

Closing

Hopefully your mod_ssl site is faster after all of this, but if you have any recommendations or ideas to improve it further, please let me know!

Ashley: What's your best face?

Fri, 09 Jul 2010 19:30:00 +0000

Or in other words, hot or not done right. The geeks at okcupid got bored and made this. You submit some pictures of yourself, choose 100 people yourself to donate to the pool of data and wait around 12 hours. You get an email telling you what the world thinks is your best photo ^_^

I did it, and got the following results

http://www.okcupid.com/mybestface?rid=5413135142001983386

Amusing bits:

Conservatives like most the picture *with the iron in*. Women -> ironing? :p
No-one likes what I made for topherchris. *cwy*
The world appears to agree with rolphus, who loves the pic that was voted the best
Christians like dresses.
No-one likes hippies.
Vegetarian males aged 23-30 like wireless headphones.

Depressing bits

A lot of the time it looks like I was voted just because I was white. I understand people have preferences and stuff but this just seemed horrible.

Bear in mind giving them your data puts you on the “join okcupid” mailing list, but you can turn that off after the first email.

Also, have the blog post on the okcupid blog that started this all off:

http://blog.okcupid.com/index.php/2010/01/20/the-4-big-myths-of-profile-pictures/

Ashley: Happy Friday world!

Fri, 09 Jul 2010 17:04:06 +0000

Must get aircon /o\

chip: The Illusion of Stability

Paul Querna — Fri, 09 Jul 2010 08:16:38 +0000

Back at the May 2010 Board meeting of the Apache Software Foundation, there was a discussion about releases. It got me thinking about how my own use of many open source projects has changed.

The Past – Long Cycles, few releases, software ships on physical media

Myst pushed the limits by being shipped on a CD-ROM instead of floppy disks, and Riven followed with pushing the adoption of DVDs by shipping on 1 DVD, or 5 CDs. Physical media kept accelerating to a point, now days most software can be downloaded. Even for game consoles, previously one of the last barriers for things like patches, games like Call of Duty: Modern Warfare 2 have a half dozen post-gold master patches, pushed down to internet connected consoles.

If you look at the development of software over the last 20 years, one of biggest changes for many products is the shift in distribution, lots of people talk about Software as a Service, but really you need to just look at software on desktops — products like Google Chrome automatically apply updates without bothering the user, and most products ship with an auto-update mechanism at a minimum.

But the fundamental difference in this is a shift in the software development and release models, that the software distribution systems have finally caught up to.

Release Cycles

I view Ubuntu as one of the first large projects to recognize this shift and embrace it. Many large-scale projects in the mid-2000s had massive problems tracking dependencies, and synchronizing anything resembling a stable final product was a challenge. Ubuntu’s model of picking a date, and shipping whatever was stable at that point shifted the responsibility model for stability. Other projects have done this before Ubuntu, but Ubuntu has stuck to it and exposed so many more people to the model. Every 6 months, Ubuntu drew a line in the sand, and whatever was stable before that date, became the next Ubuntu release.

This meant, you didn’t just wait for the new release of GNOME or KDE and then try to stabilize everything; You certainly hoped for dependencies to add new features before your dates, but if they missed it, they would go into the next release. Compare this to the traditional Linux distribution: multiple rounds of betas to squash out all the integration pain of bringing together thousands of dependencies into a stable final product.

But I don’t make a Linux distribution!

Most software projects I’ve worked on have had large dependencies on Open Source Software. Not thousands of projects like a Linux distribution. Some only had a few dependencies. Others had a few dozen projects that they were directly built on top of. In the past, you took a recent stable release and built packages of it or pulled it into a vendor branch. There was an expectation that releases were… Stable and would get maintenance patches for serious bugs.

At Bloglines, the product was built on top of 30+ open source projects, from BerkeleyDB, to libcurl, to Clearsilver. We tried to take the stable releases, and knit it all together into something that worked. For the most part we were successful. However, we patched lots of projects, some of them were patches we pushed upstream, others were Bloglines specific modifications, but we thought it was okay, we were taking stable version of Foo, and appling a few patches. We knew upgrading to the next version of Foo might be painful, but there normally was documentation explaining what changed.

The End of releases

But what the Apache Board meeting got me thinking about, was our dependencies at Cloudkick. We use a ton of Python at Cloudkick, a few projects like Twisted, and Python itself, we generally use a stable release, and its fine and dandy. But for many of our more esoteric dependencies like txAMQP, libcloud, scribe, a few Django applications, oauth, sales force libraries, etc, we are using snapshots, mostly from someone’s GitHub repository.

I am grateful for the projects we build on, and we try to contribute back to them whenever we can, but it is no longer taking a stable release and making a few local modifications — we are lucky if a project has releases at all, let alone stable releases!

I don’t think its the fault of things like GitHub, they have download areas, and some projects use them, but the majority don’t. They give you a git url, and its up to you to pick a ‘stable’ point in time, and hope for the best.

What did those releases provide anyways?

On some levels, I miss stable releases. It made me feel good, it let me judge at face value, some programer I probably will never meet in person felt good enough about some code, to call it ‘stable’. But the reality was, any code I’ve pushed hard, I’ve found bugs, and then I patched those bugs or added new features.

Code that wasn’t pushed hard, it probably didn’t matter if someone else thought it was stable, it was good or bad, it worked or it didn’t.

Those releases from someone else provided me with the illusion of a stable product. Something I can build upon; But the truth when I look hard at the projects I thought were stable, we ended up patching some of those the most!

This newer age paradigms for software releases mean you can move unbeilievability quickly, bringing together diverse peices of software and building communities around new software faster than ever. I look at the Node.JS modules page, and I’m blown away. There are many projects that probally should be added to that list, many removed, but the sheer number of projects, most of them only a few months old, exploding in popularity, its all enabled because the expectations for an open source product have changed.

You no longer wait for the once a year stable release of any dependency, you grab the snapshot from GitHub, find the author on IRC or twitter if there is a problem, patch it locally, submit a pull request, and then keep on building your product.

This model does bring other problems, many of them core to traditional thought at the Apache Software Foundation. Code pulled from SVN trunk aren’t vetted in the same way at the ASF. Many of the younger ASF projects have had more trouble making releases, and this is difficult for the slow moving foundation to always understand why releases are not a higher priority. I believe this lack of stable products definitely has hurt the Ruby/Rails community in the last few years too.

We should embrace this change. We are all developing software at breakneck speeds. Software has always been unstable, nowadays we are just honest enough to admit it. What we need is better tooling, not just a distributed version control, but more on the deployment and packaging side for most web applications. The tools have not caught up to changes in development and dependency philosophy, when most sites are still deployed with some variation of a hacked together shell script.

———————–

Thanks to Geoff for giving feedback on this post.