There's plenty of tutorials but you can request a Google Takeout of your photos and then point immich-go at it. I can confirm that this works really well.
Agree - it's annoying. rclone to the rescue. Requires choosing Google Drive as the destination and then downloading with rclone to your computer. It's the closest to sane I've found.