Not sure if your serious.
You picked the simplest service that isn't expensive at all to run. Have you tried with an RDS cluster for example, how does that work?
Does your integration test create a new cluster to use? That's slow to set up.
Does it use an existing one? Now you might have failures due to noisy neighbors and your cost will not be small.
And that's just RDS, as soon as it's not serverless, costs aren't trivial anymore.
Again, while local cloud emulator is great for cost sensitive customers, at some point, you should test Dev against almost exact environment you will be running Prod on. If not, you are asking for eventual outage in Prod when some slightly different AWS thing bites you.