Lip reading is one of the other modalities we tried as well beyond IMU, EMG and RF! We worked a lot on fine tuning Auto-AVSR, one of the most commonly used open source lipreading models. It's very enticing to work on, since it gets to about 17% word error rate, which looks good enough that you would think working on it further would get it down to product quality, but we found there's an information-theoretic limit on the modality. Speech has 40 phonemes, but only about 12 to 15 visemes (externally visually separable phonetic states). For example P and B look very similar from a camera. This causes the scaling laws on lipreading to flatten out at an error that's too high to be product ready but just low enough to make for a good demo. Our original plan was actually to ship a lipreading software to see if people would want this behavior, but we realized you couldn't make lipreading work!
replyNow that is insightful! The demo definitely fooled me and even the first repo requires you to train the model.
replyLip reading AI would be pretty big since privacy would probably go out the window.
but can't a decision model or some other model deduce "P" or "B" based on the context (i.e., previous words said)?
replyreminds me of tesla fsd v. waymo. may be a market for both? i.e., camera-only or camera + specific hardware.
Not well enough! You can get to 15-20%ish WER but going beyond that doesn't seem feasible with the info available from a camera, even with a great language model
replyWhy not a pencil in the ear or something you can put in a shirt pocket? Or even earbuds?
replyAny deaf/HoH person who has tried to do lipreading could have told you all of this too.
replyGood luck on the project!