Planet HashPHP

Ashley: gamefreaksnz: 8-Bit Flower Bouquet The 8-Bit Flower Bouquet is...

Sun, 05 Feb 2012 10:15:06 +0000

8-Bit Flower Bouquet

The 8-Bit Flower Bouquet is guaranteed to never wilt and is the perfect gift for the old-skool girl gamer in your life.

USD$14.99

Bjorn: Bike riding. (Taken with picplz at Albany Tot Park in Albany,...

Sat, 04 Feb 2012 19:15:54 +0000

Bike riding. (Taken with picplz at Albany Tot Park in Albany, CA.)

Ashley: mabelmoments: Spot on: The Obliteration Room by Yayoi Kusama –...

Thu, 02 Feb 2012 10:15:05 +0000

mabelmoments:

Spot on: The Obliteration Room by Yayoi Kusama – in pictures

The Japanese artist Yayoi Kusama transformed a completely white room, including furniture, into a spectacle featuring her signature dots, helped by children who visited the exhibition over two weeks and placed brightly coloured stickers throughout the installation at the Gallery of Modern Art, Brisbane. The interactive children’s project is part of Kusama’s Look Now, See Forever exhibition. Kusama will also have a major exhibition at London’s Tate Modern from 9 February.

Ashley: thedailywhat: Optical Illusion of the Day: An aerial photo of...

Mon, 30 Jan 2012 10:15:06 +0000

thedailywhat:

Optical Illusion of the Day: An aerial photo of an expanding-contracting toll road creates a real-life-Inception-esque illusion.

[22words.]

Bjorn: Pizza time. (Taken with picplz at California Pizza Kitchen in...

Sun, 29 Jan 2012 23:30:14 +0000

Pizza time. (Taken with picplz at California Pizza Kitchen in Emeryville, CA.)

Bjorn: Enjoying rides. (Taken with picplz at Rides at the Oakland Zoo...

Sun, 29 Jan 2012 21:17:11 +0000

Enjoying rides. (Taken with picplz at Rides at the Oakland Zoo in Oakland, CA.)

Ashley: unknownskywalker: Siriusly twinkling If you live nearly...

Sun, 29 Jan 2012 14:30:05 +0000

unknownskywalker:

Siriusly twinkling

If you live nearly anywhere on Earth and look to the southeast shortly after sunset, you’ll see the figure of Orion. Follow the three belt stars to the east, and you’ll see a bright star: Sirius, the brightest star in the night sky. If it’s near the horizon, you may see it twinkling madly: flickering, dancing, perhaps even changing color.

This gave astronomer David Lynch an idea: take a time exposure of Sirius with a camera and telephoto, and purposely wiggle the mount. He tried it on January 4, 2012, and the result is the picture shown above.

As the vibrating camera caused the star to trail around, the changing colors got recorded along the track. The changing brightness of Sirius can be seen as well, as parts of the loop-de-loop fade and intensify.

The reason stars twinkle is because of our atmosphere: little blobs of air are constantly in motion. These air parcels act like lenses, and as light passes through them, the path of the ray gets bent a little bit. That’s what causes the dancing motion, the actual twinkling.

Ashley: thecured: A worker checks in a special room where the Parma...

Sun, 29 Jan 2012 10:15:05 +0000

thecured:

A worker checks in a special room where the Parma hams are hung to dry in Langhirano near Parma. Prosciutto di Parma can only be produced in a very restricted area of 29 sq km (11.2 sq mile) around the town of Parma in the region of Emilia Romagna, just north of Tuscany. Around 10 million hams are sold every year, of which about 2 million are exported, mainly to France, the United States and Germany, which each consume about 400,000 a year. (REUTERS, file photo from 2009)

Bjorn: "Google’s staffing director responded that the employee who contacted the Apple engineer..."

Sat, 28 Jan 2012 05:48:24 +0000

“Google’s staffing director responded that the employee who contacted the Apple engineer “will be terminated within the hour.””

- Steve Jobs told Google to stop poaching workers | Reuters

Bjorn: McSweeney’s Internet Tendency: FAQ: The “Snake Fight” Portion Of Your Thesis Defense.

Sat, 28 Jan 2012 03:12:04 +0000

McSweeney’s Internet Tendency: FAQ: The “Snake Fight” Portion Of Your Thesis Defense.

Ashley: nickdouglas: tomoatmeal: Official Lyrics to “Indianapolis...

Fri, 27 Jan 2012 14:30:05 +0000

nickdouglas:

tomoatmeal:

Official Lyrics to “Indianapolis Jones”

Dun da-dun dun, dun da-dunnnnnn

Dun da-dun dun! Dun da-DUN DUN DUN!

Dun da-dun dun! Dun da-DUNNNNNN!

Dun da-da dun!

da-da dun!

da-da dun!

da-daaa dun da-dun!

(*Repeat until he’s done whipping people)

Done.

Ashley: Model railway bar :D

Fri, 27 Jan 2012 10:15:05 +0000

Model railway bar :D

Ashley: Terry Gilliam shows some animation fun

Thu, 26 Jan 2012 18:45:05 +0000

Terry Gilliam shows some animation fun

Ashley: Film Poster Patterns

Thu, 26 Jan 2012 14:30:06 +0000

Film Poster Patterns

Ashley: Classic Photographs in Lego

Thu, 26 Jan 2012 10:15:05 +0000

Classic Photographs in Lego

Ashley: britticisms: Three-dimensional works by Daniel Arsham I like...

Wed, 25 Jan 2012 18:45:05 +0000

britticisms:

Three-dimensional works by Daniel Arsham

I like how the presence and use of texture creates a visually surreal quality to the works.

Ashley: kari-shma: Books suspended above an entrance to the Istanbul...

Wed, 25 Jan 2012 14:30:05 +0000

kari-shma:

Books suspended above an entrance to the Istanbul modern art museum by Hanif Shoaei | Source

Ashley: aboltoutoftheblue: Muppet balloon animals! O.O LOVE!!!

Wed, 25 Jan 2012 10:15:06 +0000

aboltoutoftheblue:

Muppet balloon animals!

O.O LOVE!!!

Ashley: Photo

Tue, 24 Jan 2012 18:45:05 +0000

Ashley: inevitablefragments: Stuart Williams - Luminous Earth...

Tue, 24 Jan 2012 14:30:06 +0000

inevitablefragments:

Stuart Williams - Luminous Earth Grid (1993)

1,680 fluorescent lamps over 10 acres

Ashley: jankujanai: cool

Tue, 24 Jan 2012 10:15:06 +0000

jankujanai:

cool

Ashley: fuckyeahfluiddynamics: A drop of sugar syrup falls into a pool...

Mon, 23 Jan 2012 18:46:06 +0000

fuckyeahfluiddynamics:

A drop of sugar syrup falls into a pool of methylated spirits, producing a Worthington jet and several ejected droplets. Although surface tension holds the jet in a smooth shape, the refractive index of the spirits reveals the turbulent mixing within the jet. (Photo credit: Rebecca Ing)

Ashley: Photo

Mon, 23 Jan 2012 14:30:05 +0000

Ashley: stephandeprouw: Pretty.

Mon, 23 Jan 2012 10:15:05 +0000

stephandeprouw:

Pretty.

Bjorn: Gabrielle Giffords to resign from Congress [Video] - latimes.com

Sun, 22 Jan 2012 20:33:45 +0000

Gabrielle Giffords to resign from Congress [Video] - latimes.com:

Ashley: blech: british rail design cover by smallritual on Flickr: The...

Sun, 22 Jan 2012 18:47:05 +0000

blech:

british rail design cover by smallritual on Flickr:

The [Danish] design of this book looks like 2006, in British terms, rather than 1986. the 1980s were not a good period for swiss-style modernism in Britain. The book celebrates the British Rail corporate identity at a time when it seemed outdated in Britain - a case of ‘you don’t know what you’ve got ‘til it’s gone’. The British have never been very good at sticking to rational design systems - they get distracted by romanticism and nostalgia.

(Edited for capitalisation.)

Ashley: I ran out of tumblr...

Sun, 22 Jan 2012 14:55:08 +0000

Off to Civ5!

Ashley: blech: station timetable by smallritual on Flickr. Steve...

Sun, 22 Jan 2012 14:30:06 +0000

blech:

station timetable by smallritual on Flickr.

Steve Collins (who goes by smallritual online) has been posting scans of the nearly impossible to find Danish Design Council book on British Rail’s design and identity. As he writes of the pocket timetables, “In a sense, the invisibility of this kind of design is the point.” Certainly I look at that now as almost a work of art, whereas for most of my life it was just background.

Ashley: nebraska-admiral: What do you call a cow with two legs? Lean beef. What do you call a cow with no...

Sun, 22 Jan 2012 10:15:05 +0000

nebraska-admiral:

What do you call a cow with two legs?

Lean beef.

What do you call a cow with no legs?

Ground beef.

Bjorn: Bed time story time. (Taken with picplz.)

Sun, 22 Jan 2012 05:39:26 +0000

Bed time story time. (Taken with picplz.)

Bjorn: Japan's immigration control: Gulag for gaijin | The Economist

Sun, 22 Jan 2012 00:37:55 +0000

Japan's immigration control: Gulag for gaijin | The Economist

Ashley: Lego action movie :D

Sat, 21 Jan 2012 22:58:51 +0000

Lego action movie :D

Bjorn: "Those who count on quote ‘Hollywood’ for support need to understand that this industry..."

Sat, 21 Jan 2012 19:00:59 +0000

““Those who count on quote ‘Hollywood’ for support need to understand that this industry is watching very carefully who’s going to stand up for them when their job is at stake. Don’t ask me to write a check for you when you think your job is at risk and then don’t pay any attention to me when my job is at stake,””

- MPAA Directly & Publicly Threatens Politicians Who Aren’t Corrupt Enough To Stay Bought | Techdirt

Ashley: The pitch drop experiment is a long-term experiment which...

Sat, 21 Jan 2012 18:46:06 +0000

The pitch drop experiment is a long-term experiment which measures the flow of a piece of pitch over many years. Pitch is the name for any of a number of highly viscous liquids which appear solid, most commonly bitumen. At room temperature, tar pitch flows at a very slow rate, taking several years to form a single drop.

The most famous version of the experiment was started in 1927 by Professor Thomas Parnell of the University of Queensland in Brisbane, Australia, to demonstrate to students that some substances that appear to be solid are in fact very-high-viscosity fluids. Parnell poured a heated sample of pitch into a sealed funnel and allowed it to settle for three years. In 1930, the seal at the neck of the funnel was cut, allowing the pitch to start flowing. Large droplets form and fall over the period of about a decade. The eighth drop fell on 28 November 2000, allowing experimenters to calculate that the pitch has a viscosity approximately 230 billion times that of water.

This is recorded in the Guinness Book of Records as the world’s longest continuously running laboratory experiment, and it is expected that there is enough pitch in the funnel to allow it to continue for at least another hundred years. To date, no one has ever actually witnessed a drop fall.

The image above features The University of Queensland pitch drop experiment with its current custodian, Professor John Mainstone (taken in 1990, two years into the eighth drop).

Credit: John Mainstone and The University of Queensland

Ashley: Photo

Sat, 21 Jan 2012 14:30:05 +0000

Ashley: inevitablefragments: Sir Jacob Epstein - Torso in Metal from...

Sat, 21 Jan 2012 10:15:05 +0000

inevitablefragments:

Sir Jacob Epstein - Torso in Metal from ‘The Rock Drill’ (1913-14)

bronze

Bjorn: The problem(s) with religion.

Tue, 17 Jan 2012 21:02:00 +0000

The problem(s) with religion.

Ashley: I score around a 7…

Sun, 15 Jan 2012 23:00:58 +0000

I score around a 7…

Ashley: chimpgoods: Chocolate Bar Bed Toppings Click here for more...

Sun, 15 Jan 2012 18:47:05 +0000

chimpgoods:

Chocolate Bar Bed Toppings

Click here for more stuff!

Bjorn: Richard Dawkins celebrates a victory over creationists

Sun, 15 Jan 2012 17:46:51 +0000

Richard Dawkins celebrates a victory over creationists

Ashley: "Dance act Underworld have been chose to mastermind the music for next year’s Olympics opening..."

Sat, 14 Jan 2012 18:49:05 +0000

“Dance act Underworld have been chose to mastermind the music for next year’s Olympics opening ceremony. The duo - Rick Smith and Karl Hyde - have been appointed the event’s music directors, to work alongside Oscar-winning film-maker Danny Boyle.”

- Keep your intricately choreographed routines, China, Underworld is going to rave London’s face off. (via bg5000)

OH MY GOD YES.

(via apoplecticskeptic)

Bjorn: "It would be like a knife fight in a phone booth,"

Fri, 13 Jan 2012 04:26:43 +0000

““It would be like a knife fight in a phone booth,”

- U.S. Warns Top Iran Leader Not to Shut Strait of Hormuz - NYTimes.com

Bjorn: "The greatest achievement of Western civilization is the tradition of critical discussion."

Tue, 10 Jan 2012 06:58:29 +0000

“The greatest achievement of Western civilization is the tradition of critical discussion.”

- (5) History: What is the greatest achievement of Western civilization? - Quora

Ashley: thedrunkenmoogle: The Portal Two (Portal 2...

Fri, 06 Jan 2012 10:15:06 +0000

thedrunkenmoogle:

The Portal Two (Portal 2 Cocktails)

Ingredients:
Blue Curacao
Vodka
Lemonade
Cointreau
Rum
Orangina
Small tumblers

Directions: “This drink is, of course, designed to resemble the two coloured portals from the excellent sci-fi puzzle game. For the blue version, get a small tumbler and pour in 10ml of Blue Curacao, 10ml of vodka and top up with lemonade. For orange you’ll need another tumbler, this time filled with 10ml of Cointreau, 10ml of rum and Orangina. If you fancy, you can jazz the glasses up with coloured sugar rims. Simply pour some sugar into a sandwich bag with the relevant food colouring, shake them up, pour the resulting mix into a dish and dip your tumbler in. “You can mix the portals together if you like,” says James. “It’s a very orangey flavour, which we thought tied in with The Orange Box, the compilation in which the original Portal appeared.”

The finest beverage breakthroughs from the Aperture alcohol research labs. Drink them in the name of science. You monster.

Drink created by James Dance of Loading for an article in The Guardian. Check out the article for the rest of the drinks and check out Loading for a great gaming bar and cafe.

Bjorn: Still love this video. A repost from last year I guess. :P

Fri, 06 Jan 2012 06:07:08 +0000

Still love this video. A repost from last year I guess. :P

chip: Rackspace Open Sources Dreadnot, a Continuous Deployment tool

Paul Querna — Thu, 05 Jan 2012 21:55:40 +0000

Today we open sourced Dreadnot, our take on a Continuous Deployment tool. Details are posted over on the Rackspace Cloud Blog

Source is up on github.com/racker/dreadnot.

Theory: Sabbatical Day 41

Thu, 05 Jan 2012 17:16:26 +0000

It's been a long time since the last update. So long I can't track what number this should be, so I am calling it day 41.

So what's been going on since last time. Mainly going on holiday to switzerland, Christmas, and dropping out of the course I was studying. And a few things that are actually useful.

I've turned off new customer acquisition (principally) adwords for Steady Service as I have a list of things that I need to address, and I don't believe it makes sense to carry on paying money to attract people when there are things I know the service really needs. I will need to avoid falling into the trap of making a large number of changes, turning things back on, and then not having any data.
I've signed up to play with the Rackspace Cloud Monitoring Beta. This should allow automatic monitoring of all the sites I host. Unfortunately it has a restful rather than a declarative API, so I'll need to do some work to take puppet configurations and turn them into changes to be performed against rackspace.
I've been doing some interesting reading about more computer science topics, including the zipper, derivatives of data types and the lambda calculus.
I'm due to go back to work as normal next week, so I've been starting to figure out what is happening there.

Bjorn: Internet Access Is Not a Human Right - Vint Cerf

Thu, 05 Jan 2012 16:59:49 +0000

Internet Access Is Not a Human Right - Vint Cerf:

Very good opinion piece by the father of the internet.

Bjorn: "This will be the first time any space agency has attempted the complex set of maneuvers required to..."

Tue, 03 Jan 2012 05:40:55 +0000

“This will be the first time any space agency has attempted the complex set of maneuvers required to place two robotic spacecraft into the same precise orbit around a planetary body other than Earth so that they can fly in formation.”

- So cool. Solar System Exploration: Gravity Recovery and Interior Laboratory: Welcome

Bjorn: Locomotion Smoke and Mirrors | Robot Invader

Tue, 03 Jan 2012 03:32:00 +0000

Locomotion Smoke and Mirrors | Robot Invader:

Pretty cool blog post about how Chris at Robot Invader solved handling the player’s intent for his hit game Wind-up Kight.

Bjorn: "Palm and H.P. employees involved in WebOS say that there was little hope for the software from the..."

Mon, 02 Jan 2012 05:39:16 +0000

“Palm and H.P. employees involved in WebOS say that there was little hope for the software from the beginning, because the way it was built was so deeply flawed.”

- H.P.’s TouchPad, Some Say, Was Built on Flawed Software - NYTimes.com

Bjorn: Our new house. (Taken with picplz.)

Mon, 02 Jan 2012 01:35:24 +0000

Our new house. (Taken with picplz.)

Ashley: Have some train maps.

Sun, 01 Jan 2012 14:30:06 +0000

Have some train maps.

Ashley: Photo

Sun, 01 Jan 2012 10:15:05 +0000

Ashley: woof.

Sat, 31 Dec 2011 19:13:24 +0000

100460:

roses are grey.
violets are grey.
i am dog.

chip: 2011 Timecards

Paul Querna — Sat, 31 Dec 2011 13:14:15 +0000

Work Project:

Hobby Project:

2012 Goal

Finish the hobby project.

Created using Dustin’s git-timecard.

Ashley: joost5: whitewhine: Stephen Hawkins: Basically a useless...

Fri, 30 Dec 2011 20:34:10 +0000

joost5:

whitewhine:

Stephen Hawkins: Basically a useless retard.

Sometimes I feel we are fighting a losing battle against the Derp.

Bjorn: "Remember that end users just do what they are told (by advertising and RSPs [salespeople])."

Tue, 27 Dec 2011 16:54:04 +0000

“Remember that end users just do what they are told (by advertising and RSPs [salespeople]).”

- Former Microsoft Exec Scopes Windows Phone’s Failure | News & Opinion | PCMag.com

chip: Write Logs for Machines, use JSON

Paul Querna — Mon, 26 Dec 2011 23:15:40 +0000

Logging for Humans

A printf style format string is the de facto method of logging for almost all software written in the last 20 years. This style of logging crosses almost all programing language boundaries. Many libraries build upon this, adding log levels and various transports, but they are still centered around a formated string.

I believe the widespread use of format strings in logging is based on two presumptions:

The first level consumer of a log message is a human.
The programer knows what information is needed to debug an issue.

I believe these presumptions are no longer correct in server side software.

An example of the problem

An example is this classic error message inside the Apache HTTP Server. The following code is called any time a client hits a URL that doesn’t exist on the file system:

ap_log_rerror(APLOG_MARK, APLOG_INFO, 0, r,
  "File does not exist: %s", r->filename);

This would generate a log message like the following in your error.log:

[Mon Dec 26 09:14:46 2011] [info] [client 50.57.61.4]
  File does not exist: /var/www/no-such-file

This is fine for human consumption, and for decades people have been writing Perl scripts to munge it into fields for a computer to understand too. However, the first time you add a field, for example the HTTP User-Agent header, it would break most of those perl scripts. This is one example of where building a log format that is optimized for computer consumption starts to make sense.

Another problem is when you are writing these format string log messages, you don’t always know what information people will need to debug the issue. Since you are targeting them for human consumption you try to reduce the information overload, and you make a few guesses, like the path to the file, or the source IP address, but this process is error prone. From my experience in the Apache HTTP server this would mean opening GDB to trace what is happening. Once you figure out what information is relevant, you modify the log message to improve the output for future users with the relevant information.

What if we logged everything into JSON?

If we produced a JSON object which contained the same message, it might look something like this:

{
    "timestamp": 1324830675.076,
    "status": "404",
    "short_message": "File does not exist: /var/www/no-such-file",
    "host": "ord1.product.api0",
    "facility": "httpd",
    "errno": "ENOENT",
    "remote_host": "50.57.61.4",
    "remote_port": "40100",
    "path": "/var/www/no-such-file",
    "uri": "/no-such-file",
    "level": 4,
    "headers": {
        "user-agent": "BadAgent/1.0",
        "connection": "close",
        "accept": "*/*"
    },
    "method": "GET",
    "unique_id": ".rh-g2Tm.h-ord1.product.api0.r-axAIO3bO.c-9210.ts-1324830675.v-24e946e"
}

This example gives a much richer picture of information about the error. We now have data like the User-Agent in an easily consumable form, we could much more easily figure out that BadAgent/1.0 is the cause of our 404s. Other information like the source server and a mod_unique_id hash can be used to correlate multiple log entries across the lifetime of an request.

This information is also expandable. As the knowledge of what our product needs to log increases, it is easy to add more data, and we can safely do this without breaking our System Admins precious Perl scripts.

Why now?

This idea is not new, it has just never been so easily accessible. Windows has had “Event Logs” for a decade, but in the more recent versions it uses XML. The emergence of JSON as a relatively compact serialization format that can be generated and parsed from almost any programming languages means it makes a great light weight interchange format.

Paralleling the big data explosion, is a growth in machine and infrastructure size. This means logging and the ability to spot errors in a distributed system has become even more valuable.

Logging objects instead of a format string enables you to more easily index and trace operations across hundreds of different machines and different software systems. With traditional format strings it is too fail deadly for the programmer to not log all the necessary information for a later operator to trace an operation.

Generating JSON with Log Magic

Log Magic is a small and fast logging library for Node.js that I wrote early on for our needs at Rackspace. It only has a few features, and it is only about 300 lines of code.

Log Magic has the concept of a local logger instance, which is used by a single module for logging. A logger instance automatically populates information like the the facility in a log entry. Here is an example of creating a logger instance for a module named 'myapp.api.handler and using it:

var log = require('logmagic').local('myapp.api.handler');

exports.badApiHandler = function(req, res) {
  log.dbg("Something is wrong", {request: req});
  res.end();
};

The second feature that Log Magic provides is what I call a “Log Rewriter”. This enables the programmer to just consistently pass in the request object, and we will take care of picking out the fields we really want to log. In this example, we ensure the logged object always has an accountId and txnId fields set:

var logmagic = require('logmagic');
logmagic.addRewriter(function(modulename, level, msg, extra) {
  if (extra.request) {
    if (extra.request.account) {
      extra.accountId = extra.request.account.getKey();
    }
    else {
      /* unauthenticated user */
      extra.accountId = null;
    }
    extra.txnId = extra.request.txnId;
    delete extra.request;
  }
  return extra;
});

The final feature of Log Magic is dynamic routes and sinks. For the purposes of this article, we are mostly interested in the graylog2-stderr, which outputs a GELF JSON format message to stderr:

var logmagic = require('logmagic');
logmagic.route('__root__', logmagic['DEBUG'], "graylog2-stderr");

With this configuration, if we ran that log.dbg example from above, we would get a message like the following:

{
    "version": "1.0",
    "host": "product-api0",
    "timestamp": 1324936418.221,
    "short_message": "Something is wrong",
    "full_message": null,
    "level": 7,
    "facility": "myapp.api.handler",
    "_accountId": "ac42",
    "_txnId": ".rh-3dT5.h-product-api0.r-pVDF7IRM.c-0.ts-1324936588828.v-062c3d0"
}

Other implementations

There are many other libraries that are starting to emerge that can output logs in a JSON or GELF format:

winston: (Node.js) A more complete (or complex?) logging module compared to Log Magic, but the prolific crew at Nodejitsu have done a great job.
graypy: (Python) A graylog2 logger that interacts with the standard Python logging module.
gelf4j (Java) We use a modified version of this library that logs to stderr instead of using UDP.

The Transaction Id

One field we added very early on to our system was what we called the “Transaction Id” or txnId for short. In retrospect, we could of picked a better name, but this is essentially a unique identifier that follows a request across all our of services. When a User hits our API we generate a new txnId and attach it to our request object. Any requests to a backend service also include the txnId. This means you can clearly see how a web request is tied to multiple backend service requests, or what frontend request caused a specific Cassandra query.

We also send the txnId to our user’s in our 500 error messages and the X-Response-Id header, so if a user reports an issue, we can quickly see all of the related log entries.

While we treat the txnId as an opaque string, we do encode a few pieces of information into it. By putting the current time and the origin machine into the txnId, even if we can’t figure out what went wrong from searching for the txnId, we have a place to start deeper debugging.

Transporting Logs

Since our product spans multiple data centers, and we don’t trust our LAN networking, our primary goal is that all log entries hit disk on their origin machine first. Some people have been using UDP or HTTP for their first level logging, and I believe this is a mistake. I believe having a disk default that consistently works is critical in a logging system. Once our messages have been logged locally, we stream them to an aggregator which then back hauls the log entries to various collection and aggregation tools.

Since all of our services run under runit, our programs simply log their JSON to stderr, and svlogd takes care of getting the data into a local file. Then we use a custom tool written in Node.js that is like running a tail -F on the log file, sending this data to a local Scribe instance. The Scribe instance is then responsible for transporting the logs to our log analyzing services.

For locally examining the log files generated by svlogd, we also made a tool called gelf-chainsaw. Since JSON strings cannot contain a newline, the log format is easy to parse, you just split up the file by \n, and try to JSON.parse each line. This is useful for our systems engineers when they are on a single machine, trying to debug an issue.

Collecting, Indexing, Searching

Once the logs crossing machines, there are many options to process those logs. Some examples that can all accept JSON as their input format:

Perl Scripts (Hah! Did you think Perl will ever go away?)
Graylog2 (open source)
LogStash (open source)
Loggly (SaaS)
Splunk (Proprietary Software, can do JSON with an extra tool)

For Rackspace Cloud Monitoring we are currently using Graylog2 with a patch to support Scribe as a transport written by @wirehead.

Bellow is an example of searching for specific txnId in our system in Graylog2:

While this example is simple, we have some situations where a single txnId spans multiple services, and the ability to trace all of them transparently is critical in a distributed system.

Conclusion

Write your logs for machines to process. Build tooling around those logs to transform them into something that is consumable by a human. Humans cannot process information in the massive flows that are created by concurrent and distributed systems. This means you should store the data from these systems in a format that enables innovative and creative ways for it to be processed. Right now, the best way to do that is to log in JSON. Stop logging with format strings.

Ashley: Photo

Mon, 26 Dec 2011 18:11:13 +0000