Silicon Motion Cameras

 admin

(Redirected from Event Camera)

An event camera, also known as a neuromorphic camera, silicon retina or dynamic vision sensor, is an imaging sensor that responds to local changes in brightness. Event cameras do not capture images using a shutter as conventional cameras do. Instead, each pixel inside an event camera operates independently and asynchronously, reporting changes in brightness as they occur, and staying silent otherwise. Modern event cameras have microsecond temporal resolution, 120 dB dynamic range, and less under/overexposure and motion blur[1][2] than frame cameras.

Functional description[edit]

Event cameras contain pixels that independently respond to changes in brightness as they occur.[1] Each pixel stores a reference brightness level, and continuously compares it to the current level of brightness. If the difference in brightness exceeds a preset threshold, that pixel resets its reference level and generates an event; a discrete packet of information containing the pixel address and timestamp. Events may also contain the polarity (increase or decrease) of a brightness change, or an instantaneous measurement of the current level of illumination.[3] Thus, event cameras output an asynchronous stream of events triggered by changes in scene illumination.

Silicon
Comparison of the data produced by an event camera and a conventional camera.
Typical characteristics of image sensors
SensorDynamic

range (dB)

Equivalent

framerate* (fps)

Spatial

resolution (MP)

Power

consumption (mW)

Human eye30–40200-300-10[4]
High-end DSLR camera (Nikon D850)44.6[5]1202–8-
Ultrahigh-speed camera (Phantom v2640)[6]6412,5000.3–4-
Event camera[7]1201,000,0000.1–0.230

Here's a list of lsusb entries with cameras that have successfully been reused: ID 090c:37a9 Silicon Motion, Inc. Taiwan (formerly Feiya Technology Corp.) Lenovo EasyCamera, YUYV 4:2:2, 640x480 20fps, up to 1280x1024 7.5fps (likely interpolated) ID 064e:a219 Suyin Corp. 1.3M WebCam (notebook emachines E730, Acer sub-brand). SM76x Driver Download Windows. OS Description Version Release Date Download; Windows 7, 8, 8.1 and 10 32-bit and 64-bit Driver: V2.2 R04: Feb 5, 2021.

*Indicates temporal resolution since human eyes and event cameras do not output frames.

Types[edit]

While all event cameras respond to local changes in brightness, there are a few variants. Temporal contrast sensors (like the pioneering DVS[1] (Dynamic Vision Sensor) or the sDVS[8] (sensitive-DVS)) produce events that indicate polarity (increase or decrease in brightness), while temporal image sensors[3] indicate the instantaneous intensity with each event. The DAVIS[9] (Dynamic and Active-pixel Vision Sensor) contains a global shutter active pixel sensor (APS) in addition to the dynamic vision sensor (DVS) that shares the same photosensor array. Thus, it has the ability to produce image frames alongside events. Many event cameras additionally carry an inertial measurement unit (IMU).

Event cameras
NameEvent outputImage framesColorIMUManufacturerCommercially available
DVS128[1]PolarityNoNoNoInivationNo
sDVS128[8]PolarityNoNoNoCSICNo
DAVIS240[9]PolarityYesNoYesInivationYes
DAVIS346[10]PolarityYesNoYesInivationYes
SEES[11]PolarityYesNoYesInsightnessYes
Samsung DVS[12]PolarityNoNoYesSamsungNo
Onboard[3]PolarityNoNoYesPropheseeYes
Celex[13]IntensityYesNoYesCelePixelYes

Algorithms[edit]

Image Reconstruction[edit]

A pedestrian runs in front of car headlights at night. Left: image taken with a conventional camera exhibits severe motion blur and underexposure. Right: image reconstructed by combining the left image with events from an event camera.[14]

Image reconstruction from events has the potential to create images and video with high dynamic range, high temporal resolution and minimal motion blur. Image reconstruction can be achieved using temporal smoothing, e.g. high-pass or complementary filter.[14] Alternative methods include optimization[15] and gradient estimation[16] followed by Poisson integration.

Spatial Convolutions[edit]

Silicon Motion Cameras Camera

Camera

Silicon Motion Camera

Silicon

The concept of Spatial Event-Driven Convolution was initially postulated in 1999[17] (before the DVS invention), but later generalized during EU project CAVIAR[18] (during which the DVS was invented) by projecting event-by-event an arbitrary convolution kernel around the event coordinate in an array of integrate-and-fire pixels.[19] Extension to multi-kernel event-driven convolutions[20] allows for event-driven deep convolutional neural networks.[21]


References[edit]

  1. ^ abcdLichtsteiner, P.; Posch, C.; Delbruck, T. (February 2008). 'A 128$times$128 120 dB 15μs Latency Asynchronous Temporal Contrast Vision Sensor'(PDF). IEEE Journal of Solid-State Circuits. 43 (2): 566–576. Bibcode:2008IJSSC..43..566L. doi:10.1109/JSSC.2007.914337. ISSN0018-9200.
  2. ^Longinotti, Luca. 'Product Specifications'. iniVation. Retrieved 2019-04-21.
  3. ^ abcPosch, C.; Matolin, D.; Wohlgenannt, R. (January 2011). 'A QVGA 143 dB Dynamic Range Frame-Free PWM Image Sensor With Lossless Pixel-Level Video Compression and Time-Domain CDS'. IEEE Journal of Solid-State Circuits. 46 (1): 259–275. Bibcode:2011IJSSC..46..259P. doi:10.1109/JSSC.2010.2085952. ISSN0018-9200.
  4. ^Skorka, Orit (2011-07-01). 'Toward a digital camera to rival the human eye'. Journal of Electronic Imaging. 20 (3): 033009–033009–18. Bibcode:2011JEI....20c3009S. doi:10.1117/1.3611015. ISSN1017-9909.
  5. ^DxO. 'Nikon D850 : Tests and Reviews | DxOMark'. www.dxomark.com. Retrieved 2019-04-22.
  6. ^'Phantom v2640'. www.phantomhighspeed.com. Retrieved 2019-04-22.
  7. ^Longinotti, Luca. 'Product Specifications'. iniVation. Retrieved 2019-04-22.
  8. ^ abSerrano-Gotarredona, T.; Linares-Barranco, B. (March 2013). 'A 128x128 1.5% Contrast Sensitivity 0.9% FPN 3μs Latency 4mW Asynchronous Frame-Free Dynamic Vision Sensor Using Transimpedance Amplifiers'(PDF). IEEE Journal of Solid-State Circuits. 48 (3): 827–838. Bibcode:2013IJSSC..48..827S. doi:10.1109/JSSC.2012.2230553. ISSN0018-9200.
  9. ^ abBrandli, C.; Berner, R.; Yang, M.; Liu, S.; Delbruck, T. (October 2014). 'A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor'. IEEE Journal of Solid-State Circuits. 49 (10): 2333–2341. Bibcode:2014IJSSC..49.2333B. doi:10.1109/JSSC.2014.2342715. ISSN0018-9200.
  10. ^Taverni, Gemma; Paul Moeys, Diederik; Li, Chenghan; Cavaco, Celso; Motsnyi, Vasyl; San Segundo Bello, David; Delbruck, Tobi (May 2018). 'Front and Back Illuminated Dynamic and Active Pixel Vision Sensors Comparison'(PDF). IEEE Transactions on Circuits and Systems II: Express Briefs. 65 (5): 677–681. doi:10.1109/TCSII.2018.2824899. ISSN1549-7747.
  11. ^'Insightness – Sight for your device'. Retrieved 2019-04-22.
  12. ^Son, Bongki; Suh, Yunjae; Kim, Sungho; Jung, Heejae; Kim, Jun-Seok; Shin, Changwoo; Park, Keunju; Lee, Kyoobin; Park, Jinman (February 2017). '4.1 A 640×480 dynamic vision sensor with a 9µm pixel and 300Meps address-event representation'. 2017 IEEE International Solid-State Circuits Conference (ISSCC). San Francisco, CA, USA: IEEE: 66–67. doi:10.1109/ISSCC.2017.7870263. ISBN9781509037582.
  13. ^Chen, Shoushun; Tang, Wei; Zhang, Xiangyu; Culurciello, Eugenio (December 2012). 'A 64 $times$ 64 Pixels UWB Wireless Temporal-Difference Digital Image Sensor'. IEEE Transactions on Very Large Scale Integration (VLSI) Systems. 20 (12): 2232–2240. doi:10.1109/TVLSI.2011.2172470. ISSN1063-8210.
  14. ^ abScheerlinck, Cedric; Barnes, Nick; Mahony, Robert (2019). 'Continuous-Time Intensity Estimation Using Event Cameras'. Computer Vision – ACCV 2018. Lecture Notes in Computer Science. Springer International Publishing. 11365: 308–324. arXiv:1811.00386. doi:10.1007/978-3-030-20873-8_20. ISBN9783030208738.
  15. ^Pan, Liyuan; Scheerlinck, Cedric; Yu, Xin; Hartley, Richard; Liu, Miaomiao; Dai, Yuchao (June 2019). 'Bringing a Blurry Frame Alive at High Frame-Rate With an Event Camera'. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA: IEEE: 6813–6822. arXiv:1811.10180. doi:10.1109/CVPR.2019.00698. ISBN978-1-7281-3293-8.
  16. ^Scheerlinck, Cedric; Barnes, Nick; Mahony, Robert (April 2019). 'Asynchronous Spatial Image Convolutions for Event Cameras'. IEEE Robotics and Automation Letters. 4 (2): 816–822. arXiv:1812.00438. doi:10.1109/LRA.2019.2893427. ISSN2377-3766.
  17. ^Serrano-Gotarredona, T.; Andreou, A.; Linares-Barranco, B. (Sep 1999). 'AER Image Filtering Architecture for Vision Processing Systems'. IEEE Trans. Circuits and Systems (Part I): Fundamental Theory and Applications. 46 (9): 1064–1071. doi:10.1109/81.788808. ISSN1057-7122.
  18. ^Serrano-Gotarredona, R.; et, al (Sep 2009). 'CAVIAR: A 45k-Neuron, 5M-Synapse, 12G-connects/sec AER Hardware Sensory-Processing-Learning-Actuating System for High Speed Visual Object Recognition and Tracking'. IEEE Trans. on Neural Networks. 20 (9): 1417–1438. doi:10.1109/TNN.2009.2023653. hdl:10261/86527. ISSN1045-9227.
  19. ^Serrano-Gotarredona, R.; Serrano-Gotarredona, T.; Acosta-Jimenez, A.; Linares-Barranco, B. (Dec 2006). 'A Neuromorphic Cortical-Layer Microchip for Spike-Based Event Processing Vision Systems'. IEEE Trans. Circuits and Systems (Part I): Regular Papers. 53 (12): 2548–2566. doi:10.1109/TCSI.2006.883843. hdl:10261/7823. ISSN1549-8328.
  20. ^Camuñas-Mesa, L.; et, al (Feb 2012). 'An Event-Driven Multi-Kernel Convolution Processor Module for Event-Driven Vision Sensors'. IEEE Journal of Solid-State Circuits. 47 (2): 504–517. doi:10.1109/JSSC.2011.2167409. ISSN0018-9200.
  21. ^Pérez-Carrasco, J.A.; Zhao, B.; Serrano, C.; Acha, B.; Serrano-Gotarredona, T.; Chen, S.; Linares-Barranco, B. (November 2013). 'Mapping from Frame-Driven to Frame-Free Event-Driven Vision Systems by Low-Rate Rate-Coding and Coincidence Processing. Application to Feed-Forward ConvNets'. IEEE Trans. on Pattern Analysis and Machine Intelligence. 35 (11): 2706–2719. doi:10.1109/TPAMI.2013.71. ISSN0162-8828.
Retrieved from 'https://en.wikipedia.org/w/index.php?title=Event_camera&oldid=997366129'