TU press release puts SCIoI’s event cameras in the spotlight
With a time resolution of microseconds, six thousand times the contrast of conventional cameras, and potentially much smaller file sizes for the images, event cameras promise a revolution in photography. Unlike conventional cameras, they do not record entire images at once, but instead register only the changes in brightness (the “events”) for each light-sensitive pixel separately. Because these devices mimic the visual systems of animals, they are often called “silicon retinas” and find application in the field of robotics. At Science of Intelligence, researchers use these cameras to evaluate images and possibly create new eyes for robot, allowing them to navigate their environment with accuracy.
Last week, the TU’s press office published a very informative article about these cameras, including interviews and quotes by our PI Guillermo Gallego and doctoral researcher Friedhelm Hamann. The article, which is written in German, can be found here, but we’ve selected and translated some of its most interesting parts.
Event cameras: a whole new way to take pictures
In today’s digital photography, the basic method by which images are captured has not changed since the first exposure of a photographic plate in 1826 by Frenchman Joseph Niépce. Light is allowed to fall on a light-sensitive sensor for a certain time and the result is then read out: either by “developing” film material or precisely by simultaneously reading out all the electrical signals of the light-sensitive pixels of a digital camera. “The process for optical images that has evolved in nature, on the other hand, works quite differently,” says Prof. Dr. Guillermo Gallego of Faculty IV “Electrical Engineering and Computer Science” at TU Berlin and a senior scientist in the Science of Intelligence Cluster of Excellence.
Inspired by the human eye
The “cones” and “rods” in the retina of the human eye, which are responsible for color vision and sensitive brightness differentiation, respectively, send out electrical nerve impulses right after light incidence. These signals are then pre-processed and the resulting signal is finally transmitted to the brain. “Event cameras work in a similar way. Each individual, light-sensitive pixel contributes independently in time to the overall image. There is no exposure time after which all the pixels are read out at once, as with normal cameras,” explains Gallego. “Unlike the eye, however, the pixels do not trigger a signal every time light falls on them, but only when the brightness changes. This has the advantage that the amount of data to be processed is much smaller in principle.”
No exposure time means significantly higher dynamic range of the event camera
Gallego demonstrates one of the major advantages of event cameras when comparing them to a normal video camera pointed at his desk outside his office window. On the normal video image, only the desk itself is properly exposed, while the window is brightly overlit and the space under the desk is completely dark. With the event camera, things are completely different: the camera not only provides a view of the tangle of cables under the desk, but also shows the real view out the window with the buildings opposite. Why? Because the video camera takes pictures 30 times a second, with very short exposure times. The exposure time is automatically set so that it can correctly reproduce most of the image – but this is no longer possible with the extreme deviations in brightness at the window and under the table. “But the event camera doesn’t even have to decide on an exposure time that lumps all the image elements together, so to speak. On the other hand, after the camera is switched on, each pixel reports in each case whether and how much the incident light quantity has changed,” says Gallego.
The range of brightness that a camera can still display is called “dynamic range” and is usually expressed in the logarithmic unit decibel (dB). While a professional digital camera has a dynamic range of about 45 dB, event cameras cover a range of 120 dB. Due to the logarithmic scale, this corresponds to an improvement of 6,000 times.
Slow-motion recordings with unimagined high resolution
Due to the elimination of exposure time and the fast response time of pixels in the range of microseconds, high-speed recordings are also possible with event cameras, which can be used for slow-motion movies with extreme resolution. The typical measure of high-speed recording is the frame rate in frames per second; a typical high-speed camera records about 10,000 frames per second. Since event cameras do not expose whole frames, their performance must be converted to a “virtual” frame rate for comparison. This would be around 200,000 frames per second – which impressively illustrates the potential of the technology.
Hardware and software are still in their infancy
“Event cameras were originally developed by neuroscientists to establish a model of human vision,” says Friedhelm Hamann, a doctoral student at Science of Intelligence. It was only later that researchers came up with the idea of using them to drive photographic innovations as well. “That’s why conventional digital photography has a head start of several decades, both on the hardware side and on the algorithms for image analysis.” In his doctoral thesis, Hamann is primarily interested in the latter. Intelligent algorithms are particularly important when information needs to be extracted from camera data, for example for the orientation of robots in space or autonomous vehicles.”
