Back in the XVII/XIX century, a similar problem existed regarding life - the problem of "what makes living things tick". The assumption at that time was that while we can understand the biological processes around life, we will never understand the so-called "vital force", which causes things to live - life itself. I know it sounds weird now, but back in the day the mental models were different. Phenomenas like "water boils" and "organisms self-replicate" were treated as completely different domains of reality, without an overarching uniform scientific model.
It turned out that after around 100 years, we can figure out the chemical/physical processes and the need for the term "vital force" became redundant.
While this is certainly not an argument proving that the Hard Problem is not in fact hard, it is an interesting idea to think about. Perhaps its all a matter of developing better, higher-resolution neurological models which will at some point give us the tools to decompose qualia.
Even if we were to e.g. identify some field that seemed to coincide with entities reporting a subjective experience, we wouldn't have a way of determining if they truly do, or just act as if they do, nor is it clear such entities would be able to report the difference.
As it is, we struggle to quantify even much more basic differences in experience that we can introspect. E.g. I have aphantasia - I don't see things in my minds eye - and I regularly come across people who insists both that can't be true, and that it can't be true that others see things. And some of the people I've spoken to who insist aphantasia isn't real clearly has it based on digging into their thinking about it.
Even at that level we rely on trusting people's claims about their introspection - we don't know, we assume based on testimony.
Anyway, further reading: https://en.wikipedia.org/wiki/Vitalism I was in fact looking this up recently for fiction-writing purposes.
What we could do is break down other facets of what we talk about under the umbrella of consciousness, and find measurable subsets.
E.g. a lot of people insists LLMs can't reason either. Coming up with a testable definition of what that means might be doable.
Overall also just separating the subjective experience from the rest leaves us in a position where proving the possibility of AGI "just" rests on whether or not human brains exceed the Turing computable.
If they don't, then subjective experience or not is irrelevant for the question of reasoning and intelligence, as in that case a subjective experience either can't affect the computation or must itself be at least possible to fully simulate by any Turing complete system.
The problem of subjective experience then would largely be down to faith and feelings but would also be entirely orthogonal to the rest.
Billions * billions of pipes and valves can result in emergent behavior that appears conscious while at the same time the sound of a single independent water pipe can moan and sound like human speech or otherwise lifelike and evoke human emotions.
I think LLMs are doing both of these things and often people are more impressed by the independent fixtures (the moan) rather than the emergent behavior. Both the sound and the emergent behavior can be built on purpose or on accident.
I think it helps to look at this through an Information Theory lens. What information is coming into the system (the human or the machine)? What information goes out of the system which is novel? How much of this can be attributed to attempting to parse random noise aka. `Random_Imagination_Engine` vs something else? The number of inventors who come up with a breakthrough idea after mis-hearing someone is surprisingly high.
If we make the distinction between phenomenal consciousness from access consciousness we can see that LLMs clearly can make decisions based on input (A-Consciousness) but they probably don't have raw feelings and sensations (P-Consciousness).
And yes, turing equivalence is turing equivalence, I don't see why a system of pipes can't make an AI.
We can't prove it, but given the total absence of evidence of anything exceeding the Turing computable, as a hypothesis it is a reasonable one that would require truly extraordinary evidence to rise above "magical thinking".
Now, that is also far from proving they are "conscious".