There was never a process where Arab or Muslim conquerors completely replaced the native population, they just added to it. Conversion and Arabization gradually transformed the existing population’s identity. If descent from conquerors means colonizer, then virtually no population on earth is non-colonial. Arabs in Palestine, Normans in England, Turks in Anatolia, Romans everywhere…
Colonizer status is typically attached to an event and structure, not inherited indefinitely.