IMO memory size is the most important since you can always wait longer for it to compute while a blazing fast but dumb model is much less useful.
I went with a 64GB M2 Max a few years back even though it stretched my budget and I'm glad I did. I'm not sure what I'd pick in this case. Tough call.
It depends on what model(s) you want to run.