Find an easy to understand product, like a step above a todo list app. Share a spec and have them code it in their own IDE using LLM. Focus on speed. Then have them extended it, adding some features. Then ask them to show you where certain functionality exists in the code and to explain the architecture of how it is built. Ask them where it is weak and could be improved.
That's it, that's the interview. SWEs are getting quite good at the prompting, but fast navigation and understanding of the code is the key.
Anyone that can effectively translate a spec to code with minimal handholding shows product awareness. Anyone who can produce a working app can prompt AI reasonably. Anyone who can revise their design on the fly to support new features is someone with flexible thinking and some humility. Anyone who can navigate and explain what AI built is someone that understands how to build a mental map of a codebase. Anyone who can do all of this in an interview is someone who can handle pressure, work quickly and communicate.
You are welcome to keep trotting out linked list traversals and sliding window problems if you like, but leetcode is now (and maybe always has been) about culture fit more than an ability to build software.
For interviews where candidates are using AI I think you can supercharge this approach, but I do not think there is value in turning it into a code navigation puzzle as there is little value in that can be evaluated in the ~45 min you have for coding in a 1 hr interview slot. Ultimately in many software engineering jobs these days we need to understand how to build quality systems without looking into the underlying system except as necessary. I am not sure how to best evaluate this judgement in an AI assisted interview
Example mock code review (or quasi pair code review):
- look at 400-600 lines of code that are problem domain specific
- If onsite, we leave the room for an agreed time period where they can review the code on their own
- The code is purposefully sprinkled with layers of defects, basic language issues, footgun issues, test quality/coverage, styling, and architectural design
- Just let them talk through their findings, then pick their brains to understand how deeply they understand their own claims
- Stronger candidates will go from the basics up to design and more meta topics. Weaker ones usually only touch the surface