Reading time: 3 – 5 minutes

Being back in school and taking statistics and finance classes again has reminded me how much I love data. I am not one of those crazy guys who loves data in an unhealthy way and thinks that more of it will solve of all of the problems in the world mind you.  Nor do I believe it can always be trusted. The world is too uncertain and it is often hard to use data to predict the “once every 10,000 years” events that often do the most to shape the world we live in (read The Black Swan by Nassim Taleb for more on that).

What is interesting to me as of late is that we really have reached the point of having free and ubiquitous data.  In fact, we probably have too much data. I would argue that the financial crisis happened partly due to the fact that too much data was available for humans to effectively analyze and understand. I mean, let’s get real, it isn’t like the data wasn’t there.  If you looked at the right data and crunched the numbers you could have seen, as some did, that we were in for a world of hurt.

So now that we are at the point where data is essentially a commodity of sorts we have to figure out better, faster and cheaper ways of analyzing and visualizing data.  The better we get at this the better decisions we’ll make and the more crises we can avert.

FlowingData posted a quote from Google’s Chief Economist Hal Varian that sums things up nicely:

I keep saying the sexy job in the next ten years will be statisticians. People think I’m joking, but who would’ve guessed that computer engineers would’ve been the sexy job of the 1990s?

The ability to take data—to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it—that’s going to be a hugely important skill in the next decades, not only at the professional level but even at the educational level for elementary school kids, for high school kids, for college kids. Because now we really do have essentially free and ubiquitous data. So the complimentary scarce factor is the ability to understand that data and extract value from it.

I think statisticians are part of it, but it’s just a part. You also want to be able to visualize the data, communicate the data, and utilize it effectively. But I do think those skills—of being able to access, understand, and communicate the insights you get from data analysis—are going to be extremely important. Managers need to be able to access and understand the data themselves.

Hal Varian, The McKinsey Quarterly, January 2009

Since I have been thinking more about data and specifically how the financial crisis could have possibly been averted if we could have had a better handle on the mountain of data crammed into arcane forms like 10-Ks and 10-Qs, I have been reading up on XBRL.

XBRL, or Extensible Business Reporting Language, is an open standard for business reporting that I am very excited about.  It could do for financial data what feeds and xml did for content.  When all of the data is structured we can then start to set up smart software to analyze the large amounts of data that exist.  That’s a big deal.

The good news is that XBRL is slowly becoming mandated for public companies in the U.S. an abroad so we’re not far off from being able to analyze and visualize financial quickly and easily.

XBRL, as I tweeted earlier, is the intersection of some of my favorite things: software, structured data and finance.  I am planning on digging into XBRL much more and will probably write about my findings as I, well, find them. Stay tuned.