Pirates of the Web: Curse of the Black Box Data Repository

Many of us experience great quests in life, some within ourselves, some out in the world, sometimes seeking some buried treasure like Captain Jack Sparrow. My great quest goes back five long years, during which time I have waged battles and sailed into uncharted waters each time I was called to action. Now, as the quest cries out once more to be taken up I greet it as an old friend. I am speaking, of course, about getting useable data off the U.S. Bureau of Labor Statistics website.

For years I was a cost analyst, projecting budget needs for new government acquisitions. A critical part of this task is building up good labor rates. The best public labor rate statistics are held by the U.S. bureau of the same name. BLS’s website is the most tantalizing and impenetrable data repository outside of that Veteran’s Affairs Office in a Pennsylvania mountain, which still processes pensions on paper, for some reason (seriously). Like that office, it is also a literal goldmine.

Part of the problem is that the website was made many years before the days of broad cloud-storage adoption in the halcyon era of 2008-2009, or three years BDC (Before Debt Ceiling, a date measurement widely used by me to refer to the debt ceiling crisis in 2011 after which Congress decided working an entire 7 hours a day, 10 months a year was, like, harrrddd and stopped legislating effectively for the next five years).

As such, the data is extremely compartmentalized and much of it is pre-summarized to save on space despite the fact that the size of the tables is laughably small by today’s standards. Similarly, the layout is a maze of hyperlinks between sub-categories and parallel tables, and the internal search bar is a joke. The result is that, much like the paradoxical Isle of Tortuga in Pirates of the Carribean, tables cannot be found by anyone that does not already know exactly what they’re looking for. The root directory, unlike Johnny Depp’s compass in Pirates, does not much aid in the quest of a casual HTML adventurer, despite the fact that many of the navigation links refer directly to it.

Wouldn’t it be stupendous if they opened up a static mirror to the raw data that allows us to hit them with some SQL, or even download everything directly for munging (munging is of course the greatest buzzword developed since synergy). I know that will probably not happen any time before self-driving cars, but it is nice to have a familiar starting point in my annual letter to Santa.


So why do I keep fighting this war? Because when the data is useful it’s incredibly useful and exceptionally thorough. In the unlikely chance anyone from BLS is reading this, thank you—I mean it. But friends, I’m also happy to say I’ve won a small battle today. If anyone out there in the world is looking for a cleansed table version of the BLS National Wage Data, Consumer Price Index, or January 2018 Employment Report, look no further—see below for a downloadable Excel file. I will leave you with one final reference to the first in the 3-hour Disney World advertisement series that is the Pirates franchise – It is without question the worst data sharing website you have ever heard of, but you have heard of it. Happy questing. 

Click here to download Excel file

Comments

Post a Comment

Popular posts from this blog

What does it take to win at Kaggle? An Introduction to Data Strategy

Image
For our latest project, we were set to work on a completed  Kaggle competition  and tasked with analyzing the appearance of West Nile Virus (WNV) in Chicago to see if we could identify where the virus would start. Having learned my lesson from my experience with Ames housing prices , and being given a completed Kaggle leaderboard to start with, I wanted to know what constituted a 'good' score, and how hard it would be to get there. Hoping for a nice clean graph, I took a look at the score submissions over time: This is obviously a mess. If you look closely, there are some red, yellow, and green data points in there. These are respectively scores from groups with: The lowest 10 scores on the board Scores closest to the average Scores in the top 10 overall These are isolated here: Rolling over will isolate the groups. There seems to be a general elbow curve up and to the right, with groups slowly getting closer to a max score ...
Read more

Creating a Dynamic Calendar

Image
Overview This is a sample Excel workbook which takes dates and events in a list, color codes them, and exports them into a natural and printable calendar format. It is designed to take events with durations and dependencies (as in a Gant Chart), and to update them quickly. The downloadable link is here . Sample Project The example in this workbook is a project with 5 sub-projects. Each project is about 1 week long, with a review scheduled for one day after completion and associated documentation due within 3 days of completion. Sample Input Results for Month of May Background and Discussion Some time ago, I was tasked with helping to manage an Agile development. In addition to the Sprints, I was tasked with keeping track of developing internal documentation, filing approvals for release, and developing detailed updates to management, including long-term and financial planning. I had about 150 sub-tasks associated with this project, and they...
Read more

Implementing Latin Hypercube Sampling in Excel

Image
Implementing LHS for Cost Analysis What is a Cost Model and Why Do We Use LHS? A model consists of many random variables in combination. Many or all of these random variables will fall on a distribution, which means that there is a range of likely possibilities (say 1-10) called outcomes. Some outcomes can be more likely than others, based on these underlying distributions (like a normal distribution). Generally, there is at least one more parameter, usually standard deviation, that tells us about how wide the spread is, or how quickly results get less likely as we move away from the middle. Let’s look at a simple example. Consider the model: unit cost times quantity equals total cost. Suppose unit cost and quantity are both random variables with normal distributions. The average unit cost is $10 with standard deviation $2, and the average quantity is 75 dollars with standard deviation 10. In this simple model with this particular distribution, the math works out that we ...
Read more