This may have been posted before, I'm not sure how old the article is.
Northrop Grumman is talking about the advantages of neuromorphic cameras, especially when coupled with machine learning.
Interesting how at the end they talk about a potential use case being detecting and defeating a bullet. This sounds a lot like what the Australian RAAF / Project Jericho MANTIS camera could be used for.
https://www.northropgrumman.com/what-we-do/digital-transformation/neuromorphic-cameras-provide-a-vision-of-the-future/By Scott Gourley
From enhanced battlefield protection systems to maintaining aerial drone delivery fleets, neuromorphic cameras have the potential to enhance many future defense, commercial and industrial tasks. When coupled with machine learning, this technology may soon set the stage for dramatic enhancements in how systems operate and how we perceive and understand surrounding environments.
Like the Human Eye
“Cameras we use today have an array of pixels: 1024 by 768,” explained Isidoros Doxas, an AI Systems Architect at Northrop Grumman. “And each pixel essentially measures the amount of light or number of photons falling on it. That number is called the flux. Now, if you display the same numbers on a screen, you will see the same image that fell on your camera.”
By contrast, neuromorphic cameras only report changes in flux. If the rate of photons falling on a pixel doesn’t change, they report nothing.
“If a constant 1,000 photons per second is falling on a pixel, it basically says, ‘I’m good, nothing happened.’ But, if at some point there are now 1,100 photons per second falling on the pixel, it will report that change in flux,” Doxas said.
“Surprisingly, this is exactly how the human eye works,” he added. “You may think that your eye reports the image that you see. But it doesn’t. All that stuff is in your head. All the eye reports are little blips saying ‘up’ or ‘down.’ The image we perceive is built by our brains.”
"You need a lot of energy to send a number from pixel to computer or from eye to brain. And you don’t want to spend all that energy. In fact, that’s why people started thinking about building neuromorphic cameras. They require much less energy because the pixels just report changes and not actual values.”
Advantages of Neuromorphic Cameras
Doxas identified several advantages in neuromorphic imaging, beginning with reduced power requirements.
“You need a lot of energy to send a number from pixel to computer or from eye to brain,” he said. “And you don’t want to spend all that energy. In fact, that’s why people started thinking about building neuromorphic cameras. They require much less energy because the pixels just report changes and not actual values.”
He continued, “Another important advantage is speed. If you have a million pixels, and you have to send the computer a thousand frames per second, that’s one billion numbers per second. However, usually nothing changes in a scene from one millisecond to the next, so you don’t need to report that entire image.”
Doxas likened the process to compression methods for video entertainment, noting that a 4K-resolution movie represents 8 million pixels, times three colors, times 30 frames per second.
“That’s over a gigabyte per second,” he said. “Yet, you can watch that over an internet connection that’s only a few megabits per second. That’s because little changes from one frame to the next. They leverage that fact and compress frames in the same way. The difference here is that neuromorphic cameras do the compression.”
Decompression is accomplished by computers, where reporting speeds accelerating from one to tens of thousands of frames per second allow for millisecond reaction times. This paves the way for a huge range of different applications — from active combat systems detecting and defeating a bullet, to self-driving cars interpreting dangerous situations almost instantly.
Doxas said that Northrop Grumman has been involved with the technology for more than a decade, highlighting a collaboration with Johns Hopkins University that resulted in the recent design of a readout integrated circuit as well as “the brains that go behind that circuitry.”
Future efforts will include increasing the number of pixels, further lowering power and increasing resolution.
Coupling with Machine Learning
Optimizing the new camera technology involves the application of machine learning methods that can work directly with the photon plus and minus signals. With self-driving cars, for example, machine learning can construct an image of a cat or dog with just a few pixel pluses or minuses, resulting in much quicker decisions compared with images built by convolutional neural network sensors.
“In the same vein, Northrop Grumman can use this system for non-invasive diagnostics for high-speed parts. This technology will dramatically change power requirements and time to decision across any number of applications,” Doxas concluded.
(20min delay)