If you want to refute a study which presents numbers and statistics, you have to provide it in the form of numbers and statistics.
replyI don't, actually. I can simply point out that its data is out of date, and then the numbers themselves are called into question without other numbers to compare them to. And that's what I did: I shared two arguments in two comments that converge on the shared conclusion that the paper's data is not up to date (or, more precisely, cannot be up to date).
reply> I can simply point out that its data is out of date
replyYou can, but it doesn't make it correct or, evidently, yield a logical argument. Do you notice that your logic is circular? The date doesn't matter if the passage of time has not affected the outcome and that's basically what I questioned, which you refuted without any evidence. So the question still stands. Are there any comparable stats you have access to that change the picture?
You're trying to get me to make a claim stronger than what I'm making. The claim I'm making is that capabilities have moved since the paper's data (countless benchmarks support this), and it's not possible to measure the end-to-end effect of that inside industrial applications with complete access, adoption, and internal connectivity at the same pace. Therefore, the study is fundamentally out of date and conclusions about the present or future cannot be drawn from it unless you assume that technological progress cannot have affected the relevant outcomes, which is a strong assumption you'd have to back up separately. Note that the paper's model ties end-to-end outcomes to generated code quality and correctness which is precisely one of the variables that continues to move.
replyWhat the study actually shows is that organizations that had very little time to absorb older generations of a disruptive emerging technology experienced an indeterminate change in output (at the 12 month mark, anywhere from down 13% to up 25% -- the paper can't say where reality falls). It certainly doesn't establish any ceiling.