This sounds exactly like UX usability testing. Sit down with someone and watch them work through whatever process you’ve created.

It’s normal to compensate them for their time.

Normally though you don’t modify it after each participant. But for something very niche like following a README (as opposed to an e-commerce flow targeted to the general population) it might be fine, if less rigorous.