How Much Should We Trust Those Jobs Reports?
Buzzfeed’s Willie Herrmann points out that the Bureau of Labor Statistics has been doing a lot of revising of its employment reports lately:
A full 50% of the time, the initial perception of the jobs numbers would have been incorrect: Reported jobs growth exceeded economists’ expectations, but then the revised numbers actually fell short of them, or vice versa. In other words, if you want to know what today’s jobs report means for the economy, reading the headlines is no more useful than flipping a coin.
Just take a look at the chart:
Months in red are the months in which the initially reported number either fell short of or exceeded economists’ expectations but then the revised either exceeded or fell short of those expectations. Leaving the expectations issue aside, though, it seems pretty clear that we probably should be more skeptical about those initial Jobs Reports.
Well, we should be skeptical unless it’s bad news. Then we should rush to embrace the bad news as proof that we should immediately return to the gold standard and begin digging bunkers and hoarding guns.
Month to month horserace reporting is entertaining but not really useful. Use of trend analysis, such as a moving average, is more accurate.
Why don’t you show comparable data for the Bush & Clinton years? Are the revision rates higher? Lower? Roughly equal? If you are not willing to do that, this is a joke, like pretty much everything you say about the economy.
Oh, and BTW, all the revised figures, except for Jul 11, are better than the initial. So much for Obama and the Commerce Dept cooking the books.
The BLS hasn’t been revising a lot of its reports lately – it revises all of its reports for 12 months after initial release as a part of its methodology. That’s why the Buzzfeed analysis stops at March 2012 – it is the most recent month to have its revision period closed out.
Its also important to note – which the Buzzfeed article does not – that of the twelve months that it chose to highight – the ending number was higher than the beginning number in 11 out of 12 months. Why is that important? It is important because it reveals that almost all of the time, the initial job growth report is ultimately revised *upward.* This demonstrates a) independence on the part of the BLS, since – as you note – only the initial numbers get full press coverage and were the BLS politicized, there would be pressure to overinflate initial growth numbers, and b) the BLS’s model is built with an emphasis on conservatism, with a propensity to underreport its initial findings.
@michael reynolds:
I’ve got to admit that when I saw the jobs data this morning I did spend a moment or two wondering if Doug would even cover it. It did not fit his drum-beat.
Well, I did not see his endorsement of the long term averages coming! I’ve been pointing to those same long term charts for a year now. They never seemed to matter when the news of the week was bad.
Imagine, a come to Jesus moment .. driven by a good jobs report.
If it’s a jobs report under a Republican Administration, it doesn’t matter because the White House is considered the God-given property of the GOP.
If it’s a jobs report under a Democratic Administration, especially under a Kenyan Muslim Usurper Administration, it’s always a Harbinger of Doom and you should place all your investments in gold, can goods and shotguns.
The difference between the initial reports and the revised reports is +900,000 jobs.
I’m no Pollyanna with regard to our current economic situation, but that seems like something worth noting.
Also:
That’s from the Buzzfeed article. It’s not great chart design, but that explanation makes more sense.
@Tom Strong:
Not great design?! It’s absolutely aweful design — especially the choice of red to mark the off months. Regardless of author intention, for most westerners, that immediately conjurers up debt (i.e. in the red). That’s only magnified by the fact that this is about financial numbers.
And the analysis isn’t all that much better. The issue isn’t that it’s wrong 50% of the time, but that the formula that it uses is clearly being conservative. Now admittedly, being too conservative is problematic as well (and the administration would have profited from having the numbers right when they came out).
But suggesting that it’s a coin flip as to whether the number will be accurate misses the larger point.
“How Much Should We Trust Those Jobs Reports?”
More than Dick Morris, Karl Rove, Steve Doucy and Charles Krauthammer on their best days ever. Certainly more than most Cardinals attending the Conclave. Far less than evolution and general relativity.
From the buzzfeed article:
This is really the height of bad journalism.
I quickly cobbled together a different chart featuring the same data (with and added delta and % off column). Check out how this chart tells a different story:
http://www.mattbernius.com/images/jobsPerceptions.PNG
> 100% of the time from April 11 to March 12 the job numbers were wrong.
> 50% of the time a negative jobs report was actually positive.
> 8% of the time a positive report was actually negative.
> 92% of the time the job predictions were revised UP (by an average of ~200% not counting the major Nov 11 outlier)!
So the idea that this is a coin toss — at least for this period — is completely false.
Now, as I said above you can debate if the initial calculations are too conservative — leaving off one major outlier, the final job numbers were, on average, ~163% higher than the prediction. But given the choice between constantly revising down, this doesn’t seem a bad thing.
More importantly, based on this period, no one should ever trust the initial job numbers.
http://www.washingtonpost.com/blogs/ezra-klein/files/2012/10/revisions.jpg
This is a chart showing revisions going back to the 80s. They fluctuate widely in both magnitude and direction. The point isn’t that the numbers are likely to be revised up – they’re not, and the sample is too small to show that. The point is that month-to-month, horserace analysis is overblown and not very useful.
If we’re prepared to treat as gospel public opinion polls which are based upon phone calls with around 1,000 people and sampling extrapolations, I think we should be well prepared to trust surveys based upon 60,000 phone interviews (BLS household report) and 400,000 payroll reports (BLS establishment survey), along with standard sampling techniques and long-standing seasonality adjustment factors created by experts.
Would you stake your life on these BLS reports? Obviously not. But they’re pretty f’n good approximations. Always have been. Certainly they’re more than well within the ballpark of accurate.
The jobs report is nothing but a survey. It has a margin of error of something crazy like plus/minus 100,000 jobs in the first place. Even without revision, slight month to month changes aren’t the best place to focus. You need a good deal more context than month X vs. month Y
anyone who reads the BLS technical notes knows this…the margin of error on the establishment survey is 100,000, and it’s over 400,000 on the household survey…blame the reporters, not the BLS..
Greetings:
Another aspect that never seems to draw media attention is what statisticians refer to as “statistical significance”. Any measure includes some degree of inaccuracy or insignificance, so there are any number of mathematical tests which show whether a change in a measure is of any real statistical significance. When a measure of 7% or 8% varies by a tenth of a percent or so, that may very well not be of any mathematical significance or of extremely little.
Since the Obama administration has figured out how to “game” the unemployment and other economic numbers, as is unburdened by any real sense of personal honor, these reports should be viewed as as good for them as they felt they could get away with. Which, I suppose, is better than nothing, and certainly better than those “rates” of inflation that no longer seem to come up in economic discussions at all.
Forward !!!
@11B40:
Those darn Chicago guys with their cookin’ the books and all…
So let me guess: The reality is that no-one has a job?
Perhaps you could show proof (from a reputable news source of couse) that the Obama administration is doing anything with the numbers that has not been done before.
The memory of conservatives screeching about how the polls that showed Obama heading for a big win last November were rigged and wildly wrong is fresh in everyones minds.
Conservatives have yet to show that they can successfully add 1 and 1 and get the correct answer…
@anjin-san:
On the other hand, we should be somewhat honored that Jack Welch deigns to post comments here.
Eleven?
I think the lesson here is that we should be skeptical unless the jobs reports are issued by a Republican Administration.
And bravo to you for holding this administration accountable! However, it is amusing how people like you gave previous administrations a pass on things like the deceitful campaign to get us into Iraq or the Iran-Contra Affair…I guess personal honor is only important sometimes…
@al-Ameda:
Yup. If Romney had won the election, the stock market was at all-time highs & we had this jobs report, the GOP would be bleating from every rooftop about the Romney Recovery, the media would be nodding along and they’d be poo-pooing any on the left who didn’t buy in to their frame as, “talking down the economy.”
Oh, and that BuzzFeed article is amazing. What they did with that chart (taking data that was almost entirely positive, subtracting from it expectations and then using the red to turn it into a 50/50 muddle) and the headline (“The Jobs Report Is Wrong”) has got to be one of the most blatant examples of Drudge-baiting I’ve ever seen. I’d be shocked if he didn’t bite. Doug certainly couldn’t resist.
@Jeremy R:
Romney would probably have announced the jobs report from the deck of the USS Lincoln, the backdrop being a huge banner proclaiming “Socialism is Dead!”
@al-Ameda:
Or maybe something recycled from the campaign:
“We Built It“
If I want an accurate picture of the jobs situation all I have to do is check the walking track right down the road at 10:00am during the week. Right now it is holding steady after increases since 2008.
And are you tired of these daily stock market record reports? Like that has something to do with the economic factors that affect the middle class: food prices, wages, gas prices, and the hot dog prices at the local convenience stores.
@michael reynolds:
Nice snark, but dumb. Too much inaccuracy in the craft over the short term.
The only think we know is that the over the long term the numbers have been miserable, with the unemployment rate fueled only by people dropping out of participation. Now THAT’s a fact.