You can try this command. But I doubt it will work as intended if you have ever upgraded Ubuntu versions.

  comm -23 <(apt-mark showmanual | sort -u) <(gzip -dc /var/log/installer/initial-status.gz | sed -n 's/^Package: //p' | sort -u)

https://askubuntu.com/a/492343/1056703

NixOS seems ideal for this.