With a valid measure I would expect a roughly even distribution over time between underestimates and overestimates. For a valid measure worth considering I'd also expect the stat to be released later when revisions are less likely because more actual data has been collected
This is a valid hypothesis. It’s wrong, and I’ll explain why. (It’s a bad and invalid thing to conclude.)
If measurement errors were iid, you’d be correct. But they’re not. They’re well documented for not being so. Earlier survey results are biased by directional response bias inasmuch as the employers with the lease changes respond first. So the earliest releases tend to match whatever was going on before. Then the employers who had to do paperwork respond. And then, finally, someone gets around to calling the folks who never got back. Some of them aren’t around anymore.
So yeah, the directional tendency in revisions is well documented. And for a long time, the early releases were appreciated. But maybe American statistical and media literacy is such that only final releases should be released, which would mean we’d always be working with data 6 months to a year out of date.
At some point a lack of decision to take compensating action becomes faking the numbers.
There have been revisions since the forever, and this is because they depend in part of surveys, and if companies (and the people with-in them) don't bother responding in a timely or accurate manner then that's going to throw the sampling off.
> CES estimates are considered preliminary when first published each month because not all respondents report their payroll data by the initial release of employment, hours, and earnings. BLS continues to collect payroll data and revises estimates twice before the annual benchmark update (see benchmark revisions section below).
* https://www.bls.gov/opub/hom/ces/presentation.htm#revisions
Post-COVID surveying seems to have become more difficult (and BLS budget stagnation/cuts haven't helped). This has been a known issue for a while; see Odd Lots episode "Some of America's Most Important Economic Data Is Decaying":
> Gathering official economic data is a huge process in the best of times. But a bunch of different things have now combined to make that process even harder. People aren't responding to surveys like they used to. Survey responses have also become a lot more divided along political lines. And at the same time, the Trump administration wants to cut back on government spending, and the worry is that fewer official resources will make tracking the US economy even harder for statistical departments that were already stretched. Bill Beach was commissioner of labor statistics and head of the US Bureau of Labor Statistics during Trump's first presidency and also during President Biden's. On this episode, we talk to him about the importance of official data and why the rails for economic data are deteriorating so quickly.
Honestly, your comment made me mildly angry. That said, can you say why you believe parent's comment is misleading?
Admittedly, if I could find a single instance of someone willing to vouch or share insight on having filled out a BLS survey, that'd cure a healthy chunk of skepticism. There's still be the other distortions in the data to account for, but I'd at least have an instance proving that yeah, there is somebody filling out these surveys and it isn't just something they say they do to make their magic unemployment number sound legit.
Note, I'm in a massive sceptical shit phase at the moment. Last decade has burned my optimism hard. So when it comes to my ability to assume benevolent intent right now, there's a heavy bias against doing it, and a heavier bias in the direction of "what would be the easiest way to keep the System limping along?" The answer to that is "say you do one thing, in reality do another, and as long as no one comes lookin', it's gold." The finance industry runs on Trust moreso than anything else, and there ain't much to be said for Trusting anything you can't verify these days. Not from other humans.
I’ve never met a single chicken farmer. Does that mean I should be sceptical about them existing? Like, what sort of metric is this for truth finding?
> to assume benevolent intent
No need. Markets move on these data. The rich and powerful bet their money on what they say.
Unless you have introduced yourself with this question to thousands of people, this is a totally meaningless statement. It says more about your social circle, your grasp of descriptive statistics, and the weird online stew you are soaking your brain in than it says about the CPS.
So my main argument ( and frankly the only argument that should matter ) is that is a bad fit for the goal of estimating values ( even though we do know its failure modes ). Is that not enough?
<< do other countries
No, it doesn't mean I am wrong.
<< That's what people are asking you to do.
No. What I am being asked to do is: "Show me a better way, but I only accept a better way that is already utilized by someone else". Not a recipe for a thoughtful exchange of ideas.