Yup we worked with IMUs as well when we started this. They measure aggregated face motion, so cannot distinguish between fine phonetic classes, only broad words. The open source demo got to a 5 word classifier, I believe. We tried a lot to make IMUs work too, in various form factors, but the signal doesn't contain enough information for open ended decoding beyond classifying between a few words. We would have loved to be able to just push software!
reply> There was this really interesting one where it reads your lips
replyThis is what I thought the original post was about. The glasses you have to wear is where that lost me.
If we could get lip-reading via your computer/phone camera down perfectly, that's the real win. Making people wear your goofy frames is where you lose a massive portion of people. Nobody wants to look like Zuckerberg lol
Lip reading is one of the other modalities we tried as well beyond IMU, EMG and RF! We worked a lot on fine tuning Auto-AVSR, one of the most commonly used open source lipreading models. It's very enticing to work on, since it gets to about 17% word error rate, which looks good enough that you would think working on it further would get it down to product quality, but we found there's an information-theoretic limit on the modality. Speech has 40 phonemes, but only about 12 to 15 visemes (externally visually separable phonetic states). For example P and B look very similar from a camera. This causes the scaling laws on lipreading to flatten out at an error that's too high to be product ready but just low enough to make for a good demo. Our original plan was actually to ship a lipreading software to see if people would want this behavior, but we realized you couldn't make lipreading work!
replyNow that is insightful! The demo definitely fooled me and even the first repo requires you to train the model.
replyLip reading AI would be pretty big since privacy would probably go out the window.
but can't a decision model or some other model deduce "P" or "B" based on the context (i.e., previous words said)?
replyreminds me of tesla fsd v. waymo. may be a market for both? i.e., camera-only or camera + specific hardware.
Not well enough! You can get to 15-20%ish WER but going beyond that doesn't seem feasible with the info available from a camera, even with a great language model
replyWhy not a pencil in the ear or something you can put in a shirt pocket? Or even earbuds?
replyAny deaf/HoH person who has tried to do lipreading could have told you all of this too.
replyGood luck on the project!