So the effect could be tiny.
Worse, the report says it excludes observations outside the 0.5th and 99.5th percentiles of the 2021–22 fall-to-spring score-change distribution. I'm not sure how this is defensible. Either you trust your measurement instrument or you don't.
If you're using a difference-in-differences design, excluding observations can change what you're measuring, particularly if the exclusions differ between treatment and control groups or over time. Without trimming those observations, the estimated effect drops from 0.0087 SD to 0.0058 SD (and is no longer statistically significant).
The authors preregistered their study, but there's nothing in the report indicating that the trimming of 'extreme' values was part of that preregistration...
My message to them:
Dear Dr. Burland and Professor Kane,
I was reading your November 2026 report, Phone-Free Schools and Achievement Growth, and saw that the study was preregistered on the Open Science Framework. I haven’t been able to locate the registration.
Could you please share a link to the preregistration or let me know where I can find it? I’m particularly interested in checking whether the exclusion of extreme fall-to-spring score changes was specified in the original analysis plan.
Thank you for your time and for making the research available.
Best regards
Their reply: Rahim: That part of the study was not pre-registered. We have used the NWEA data from earlier years, but had not encountered such extreme outliers. The magnitude of gains or losses we dropped were clearly errors—multiple standard deviation gains in a single year, also inconsistent with the midyear measurement. I’m ccing our colleague, Samantha Cullum, who can provide the link to the pre-registration. Tom KaneWow! I love this.
I love that you bothered to email them with a thought out question.
I love that they bothered to respond with a thought out answer and they took you quite seriously.
I love that you bothered to tell us about the exchange.
Thank you so much for all of this. This is what we should all strive to do with our time and as a community.
- Trust the anecdotes and this effect can't be fully captured by the study
- Trust the study and assume the parents are projecting their desire for banning phones to be a good thing.
I'm leaning towards #2, since I also have a strong desire to ban phones around my kids (they're 3 and 1(x2) so yes, around them not for them). I'd personally be really prone to confirmation bias myself around phone bans.
-Jeff Bezos, paraphrased
So, yes, I almost agree with the statement "People have been thinking crime has been skyrocketing." IMO, we're never necessarily far from a situation in which widespread impunity could cause crime to escalate rapidly.