I've thought about how to fix this too. I'm "locked in" to OpenAI because of what it knows about me. Literally it's told me things like "blah blah but you shouldn't do that because you take X medication which has Y in it" and I'm like "...really? can you reference that?" and it gives me some old thread where I chatted about it and unbeknownst to me I completely forgot.
In other words it's legit helpful as my "second brain", but I also don't like that I'm locked in and can't bring all this shared memory/context over.