Thursday, September 8, 2011

Data: Links Followed to Govt Sites

Very kewl.

via bitly blog on 7/26/11

Do you know exactly what you were reading 10 years ago this very moment? 10 years from now, you may. The data we create is a living archive of our interests and intentions, and we’re creating more of it every day — a staggering 1.2 zettabytes last year alone. At bitly, we are always looking to learn from our data, and we are thrilled that a particularly interesting set of bitly data is being made publicly available for hacking and analysis.

This Friday evening, bitly will host the New York branch of a nationwide, 4-city open data hackathon. The team has made the realtime clickstream data for all URLs open to the public, providing an unprecedented window into the way that we engage with government content. We are extremely excited to see what talented hackers and coders can do with such a rich and fascinating data set.

In advance of the hackathon, the Measured Voice team has put together a microsite at that ranks the most popular URLs per day by click count (an average of about 56,000 total daily clicks over the last few months). The diversity of content at the top of the list is fascinating; a NASA Tweetup announcement alongside information about the recent FAA furloughs  and a strongly worded FDA warning to Diamond Foods about walnut packaging.

This basic ordered list provides a compelling look at the popularity of specific government URLs – and it is only scratching the surface of the underlying data. Where in the country are most people looking up information about FAA furloughs? (And, using a complementary API like SimpleGeo’s Context, what do we know about the demographics of those regions?) Which social networks drive the most traffic to NASA’s website? What US government content is most frequently accessed outside the US? There’s a wealth of insight encoded in the data just waiting to be discovered. 

Indeed, working with data isn’t just a matter of crunching numbers and writing code; it also means knowing what you’re looking for and knowing what you’re looking at. Through the right lens, the data from the project could answer questions that we hadn’t even thought to ask – and some of the data that answers those questions may very well be yours.

If you’d like to explore this data with us, then come to our NYC office this Friday (or visit the other hackathons in San Francisco, San Diego and Washington D.C.) and build something new!