upvote
This looks like measuring what is easy to do, rather than what really matters.

Even that's hard. There aren't enough signals to attribute changes directly to AI, so these apps seem to correlate the signal that the user was interacting with AI to the changes they made e.g "Bob used AI at that time, and they opened a PR at a similar time, so Bob probably used AI to make that PR."

Until the tooling for gathering data on AI usage improves the data will be fairly interesting because correlations often point to something related, but won't be a source of truth.

reply
Yeah all of these along with raw token usage are metrics that were being used around December to February by people who just didn't have anything to go off yet.

Skill/Hook usage rates, budget spend, auto-approval rate, focus area heatmaps, MTTR, MTTD are all there now

reply
I'd be interested to overlay, I don't know, customer satisfaction or anything that can show the follow-on effect of all this output. Linear won't have that information.

My guess is some will jump up (where the team has managed to make themselves move effective and responsive) and many will plummet (doesn't need explaining).

Then there might be something to look at.

reply
I'd be interested to overlay, I don't know, customer satisfaction or anything that can show the follow-on effect of all this output. Linear won't have that information.

Very few businesses can accurately attribute customer value to the work they do, especially once they're passed start-up scale. A mature company makes lots of small changes and they're rarely measurable.

reply
[dead]
reply