Researchers at Carnegie Mellon University in Pittsburgh, PA Has created a camera system Which ‘can see the sound vibration with such precision and detail that it can reconstruct the music of a single instrument in a band or orchestra.’
Fancy system, developed in the school of computer science Institute of Robotics (RI), using a pair of cameras and a laser to ‘detect high-speed, low-amplitude surface vibrations.’ Vibrations are then used to reconstruct sound and capture isolated audio without interference or microphone. Even high-direction microphones struggle to eliminate nearby noise and deal with ambient noise. Furthermore, traditional mics cannot eliminate the effects of acoustics when capturing audio.
‘We’ve invented a new way of looking at sound,’ he said Mark ShininA post-doctoral research associate Illumination and imaging laboratory (ILIM) at the Institute of Robotics. ‘It’s a new kind of camera system, a new imaging device, capable of seeing something invisible to the naked eye.’
The research team has successfully demonstrated their new system. The group “captured isolated audio of individual guitars playing at the same time and separate speakers playing different music at the same time.”
CMU’s camera system is not the first of its kind. MIT researchers first created some visual microphones in 2014. CMU’s systems improve on previous work in many ways, including practicality and cost. ‘We’ve made the optical microphone much more practical and usable,’ he said Srinivas Narasimhan, A professor at RI and head of ILIM. “We have improved the quality by reducing costs.” The CMU method uses ordinary cameras, which are much less expensive than the high-speed cameras used in previous studies.
The system analyzes the difference of the ‘spectral pattern’ from the image captured with a rotating shutter and a global shutter. An algorithm then works to calculate the difference in the spectral pattern from two different video streams. These differences are then converted into vibrations to reconstruct the original word. CMU writes, ‘A speckle pattern refers to how coherent light behaves in space after being reflected from a rough surface. The team created a speckle pattern by aiming a laser at the surface of a vibrating object like the body of a guitar. The pattern of those spots changes as the surface vibrates. A rolling shutter captures an image by quickly scanning it, usually from top to bottom, stacking one row of pixels on top of another to create the image. A global shutter captures an image in a single instance. ‘
 |
| ‘Mark Shinin (left) and Dorian Chan (right) were part of a CMU research team that developed a camera system that could detect sound vibrations with such precision that it could capture isolated audio of individual guitars playing at the same time.’ Credit: Carnegie Mellon University |
Research paper, ‘Dual-shutter optical vibration sensing, Has been honored as one of the ‘Best Papers’ at the recent 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) in New Orleans, LA. If you miss it, NVIDIA has also presented research on an AI tool that converts a series of 2D images into a 3D model at a CVPR conference.
Practical uses of optical vibration-sensing cameras include allowing sound engineers to observe individual instruments without listening to other instruments while mixing, monitoring the vibrations of industrial equipment to check problems, conducting mechanical health surveys, and more. To learn more about research, see CMU imaging Website
