Researchers construct 3D scenes using reflections from eyes

The big picture: The Internet has long demonstrated the potential hazard involving unintentional reflections in photos. In 2015, researchers at MIT developed an algorithm to remove distracting (and possibly embarrassing) reflections from images. Now there’s a new threat on the horizon that takes basic reflections to a whole new level. Researchers from the University of Maryland have developed a system that essentially turns the human eye into a mirror. Using a camera to capture a sequence of eye images along with recent advancements in 3D vision and graphics, the team was able to reconstruct a 3D scene of what an individual was looking at. A key element of the technique is the fact that humans are not stationary. Head movement – even if only slightly – naturally creates different perspectives of the same scene and helps to recreate it virtually. Earlier work in the field involved using a moving camera to capture images. The researchers used cornea pose optimization and iris texture decomposition while training their neural network to deal with the complexity of iris textures and overcome low resolution reflections. The researchers were able to reconstruct 3D scenes using data captured from human eyes of varying colors as well as synthetic eyeballs. Sample recreations have been shared online and, well, they are rather haunting. Notably, their results were obtained in a laboratory environment with ideal lighting conditions and deliberate head movement. Results from synthetic eyes turned out better. Rebuilding a scene using organic photos would likely be much more of a challenge although in the right setting, it could still be possible. As the researchers correctly highlight, the human eye is an underappreciated source of information about the world around us. It is not a stretch to imagine technology like this being used in criminal cases lacking more conventional evidence (and it would not be the first time that reflections have been used in legal situations). The team’s paper, “Seeing the World through Your Eyes,” has been published online should you want to learn more.