A new camera that mimics the vision of the mantis shrimp helps driverless cars see better. The new camera improves dynamic range and adds polarization information. It improves the ability of driverless cars to identify hazards in situations where visibility is bad. In other words, the new device could help self-driving vehicles operate more safely.
The visual system of the mantis shrimp is among nature’s most complex. According to Wikipedia:
“Mantis shrimp are thought to have the most complex eyes in the animal kingdom and have the most complex visual system ever discovered.”
The new camera helps driverless cars see better by detecting polarization. It also features a dynamic range that is approximately ten thousand times greater than current commercial cameras.
New camera sees better in challenging conditions
Dynamic range is a measure of the darkest and brightest areas a camera can capture simultaneously. In challenging conditions, the new camera can see much better than other cameras. It can see better, for example, when a car moves from a dark tunnel into bright sunlight. Foggy or hazy conditions are also examples of situations where visibility is poor.
Researchers from the University of Illinois at Urbana-Champaign and Washington University in St. Louis describe the new camera in the journal Optica (citation below).
Team leader, Viktor Gruev, said:
“In a recent crash involving a self-driving car, the car failed to detect a semi-truck because its color and light intensity blended with that of the sky in the background.”
“Our camera can solve this problem because its high dynamic range makes it easier to detect objects that are similar to the background and the polarization of a truck is different than that of the sky.”
Gruev is an Associate Professor of Electrical and Computer Engineering at the University of Illinois at Urbana-Champaign.
New camera – other applications
The new camera could be used for other applications, the researchers say. They are exploring its usage in detecting cancerous cells because their light polarizations differ from those of healthy tissue. The researchers are also looking into using it to improve ocean exploration.
First author, Missael Garcia, said:
“We are beginning to reach the limit of what traditional imaging sensors can accomplish. Our new bioinspired camera shows that nature has a lot of interesting solutions that we can take advantage of for designing next-generation sensors.”
Garcia is a Postdoctoral Researcher at the University of Illinois Urbana-Champaign.
New camera mimics shrimp vision
Mantis shrimp (stomatopods), a grouping that includes hundreds of species across the world, have a logarithmic response to light intensity. This makes them sensitive to a very high range of light intensities. They can, therefore, perceive extremely bright and dark elements at the same time, i.e., within the same scene.
The researchers wanted to achieve a similarly high dynamic range for their new camera. They tweaked the way the device’s photodiodes convert light into an electrical current. Traditionally, the camera would operate the photodiodes in reverse bias mode. In this case, however, they used forward bias mode.
The electrical current output, therefore, changed from being linearly proportional to the light input to having a shrimp vision-like logarithmic response.
Mimicking what mantis shrimp does
The mantis shrimp integrates polarized light detection in its photoreceptors. The scientists mimicked this for their new camera by depositing nanomaterials directly on the imaging clip’s surface. Specifically, the chip that contained the forward-biased photodiodes.
Prof. Gruev said:
“These nanomaterials essentially act as polarization filters at the pixel level to detect polarization in the same way that the mantis shrimp sees polarization.”
We can use traditional imaging sensor fabrication processes to make the sensors. However, they are not optimized for making photodiodes that function in a forward bias. To compensate, the scientists developed extra processing steps to clean up the images. The additional processing steps also improved the signal-to-noise ratio.
New camera field test
Initially, the researchers tested the camera under various polarization conditions, light intensities, and colors in the lab. They then took it into the field to determine how well it could operate. Specifically, how well it operated in bright conditions and in the shadows.
Co-author Tyler Davis said:
“We used the camera under different driving lighting conditions such as tunnels or foggy conditions. The camera handled these challenging imaging conditions without any problems.”
Davis is an undergraduate student at the University of Illinois at Urbana-Champaign’s Department of Electrical and Computer Engineering.
Nan Cui, a graduate student in Electrical Engineering at Washington University in St. Louis, was also a member of the research team.
Working with an airbag manufacturer
The research team is currently working with a manufacturer of airbags. They want to determine whether the new camera’s polarization imaging capability and high dynamic range can help detect objects better.
Hopefully, the new technology can help deploy the airbag a few milliseconds earlier than is currently possible. Also, they want to see whether the new camera can help avert collisions.
Ocean exploration
The research team has also received funding to use their imaging system to make small GoPro-like cameras for ocean exploration.
Prof. Gruev said:
“We are coming full circle by taking the camera, which was inspired by mantis shrimp, to different tropical oceans to learn more about how this shrimp behaves in its natural habitat.”
“They live in shallow waters and bury themselves under corals or in little burrows. This creates a challenging high dynamic range imaging situation because there’s a lot of light in the water but dim conditions inside the holes.”
Safety is paramount for makers and researchers of self-driving cars. Earlier this year, Uber shut down its self-driving operation in Arizona. Uber made the decision after a self-driving Volvo ran a female pedestrian over as she was crossing the road. The woman died.
Citation
M. Garcia, T. Davis, S. Blair, N. Cui, V. Gruev, “Bioinspired polarization imager with high dynamic range,” Optica, 5, 10, 1240-1246 (2018). DOI: https://doi.org/10.1364/OPTICA.5.001240.