2020 BRN Discussion, page-16910

Yesterday I posted the above thinking that the significance of theresearch I had located would capture the attention of other posters.Unfortunately it was largely overlooked and while normally I would just move ongiven the market uncertainty that exists at this time I feel compelled to spellout its significance. So the following pointsshould be noted and then the following extracts from this article should beread:

1. Five senior researchers for Valeo are making out an argument in 2019 for the use of aspiking convolutional neural network chip in LiDAR and various sensorsincluding voice recognition for autonomous driving.

2. They reference in this article SpiNNaker and BRAINCHIP.

3. Though they reference SpiNNaker it is research based and BRAINCHIP’sADE was in their hands because they were and are early access programcustomers.

4. The CEO has recently stated that AKD1000 is finding a sweet spot inLiDAR sensors.

Keeping all this in mind if you read the following article you willhave no doubt that Valeo is well advanced in the work required to implementAKD1000 technology in 3D point cloud LiDAR sensors for autonomous drivingassistance.

The statements by the CEOthat AKD1000 has been successfully validated and the advantages these fiveengineers from Valeo state that a chip like AKD1000 would bring to autonomousdriving sensors makes the known relationship clearly a “no brainer” given thatthe only company in the world with a commercial off the shelf SCNN validatedchip is BRAINCHIP:

Exploring Deep Spiking Neural Networks for Automated DrivingApplications

Sambit Mohapatra1 , Heinrich Gotzig1 , Senthil Yogamani2 , StefanMilz3 and Raoul Zollner ¨ 4

1Valeo Bietigheim, Germany 2Valeo Vision Systems, Ireland 3ValeoKronach, Germany 4Heilbronn University, Germany{sambit.mohapatra,heinrich.gotzig,senthil.yogamani,stefan.milz}@valeo.com, [email protected]

Keywords:Visual Perception, Efficient Networks, Automated Driving.

Abstract:

Neural networks have become the standard model for various computervision tasks in automated driving including semantic segmentation, movingobject detection, depth estimation, visual odometry, etc. The main flavors ofneural networks which are used commonly are convolutional (CNN) and recurrent(RNN). In spite of rapid progress in embedded processors, power consumption andcost is still a bottleneck. Spiking Neural Networks (SNNs) are graduallyprogressing to achieve low-power event-driven hardware architecture which has apotential for high efficiency. In this paper, we explore the role of deepspiking neural networks (SNN) for automated driving applications. We provide anoverview of progress on SNN and argue how it can be a good fit for automateddriving applications…..

CNNs can be implemented both in software and in hardware and due totheir frame based information processing, the hardware resources can bemultiplexed. Thus, higher memory bandwidth and faster data transfer are key forreal-time performance. Unlike CNNs, SNNs process events instead of frames,hence hardware needs to be always available as event generation is notpredictive. Though it may seem to be a limitation, this means, the network istightly coupled to the hardware and can produce faster response than anequivalent CNN. To improve the efficiency of a SNN architecture, a modular andre-configurable hardware is more suitable. (Farabet et al., 2012).

Given the potential benefits of SNNs, a general question arises onwhether CNNs can be adapted to SNNs? Infact, adapting pre-trained CNNs toequivalent SNNs is easier and produces better results that building a SNN withSTDP and unsupervised or supervised learning. Such adaptations have some key benefits: 1) Aspiking convolution operator, analogous to the convolution operator in CNNswould operate much faster due to event based processing, while producingsimilar results as traditional CNN. 2) Since events are asynchronous, eachconvolution operator, supported by its linked modules can operate independentof others, if it has an event for processing. This eliminates the need for aglobal synchronization among the operators. Such an asynchronous convolutionoperator may be then implemented as a standard block in hardware forreusability. 3) Since information is processed on a per-event basis, power isalso consumed on a per-event basis. Since sensors typically produce a lot ofredundant and sparse data, this could bring a significant reduction in powerconsumption and computational load.

Finally SNNs can be queried for results anytime after the firstspikes are produced at the output since Table 1: Comparison of ANN and SNN invarious computer vision datasets (Rueckauer et al., 2017). Data set ANN errorrate (%) SNN error rate (%) MNIST [12] 0.86 0.86 CIFAR-10 11.13 11.18 ImageNet23.88 25.4 information processing is not frame based (Rueckauer et al., 2017).Several implementations of deep SNNs on neuromorphic hardware such as SpiNNakerand BrainChip have demonstrated sensor applications that support thispotential of SNNs. Some evidence to support the strong possibilities inresearch of SNN based networks for object detection is presented in Table 1. Itis based on an implementation by (Rueckauer et al., 2017). It presents acomparison of classification error rates for CNNs and SNN implementation onstate of the art data sets (Cao et al., 2015). 3 SNNs in Automated Driving 3.1Use cases in Automated Driving Event Driven Computing: Automated driving has awide variety of scenarios. At high level, the main scenarios are parking,highway driving and urban driving (Heimberger et al., 2017). The scene dynamicsand understanding is typically different for these scenarios and a customizedmodel is generally used for these scenarios. There are also various scenariosbased on weather condition like rainy, day or night, foggy, etc. Thecombination of various environmental condition is exponential and difficult tohave a customized model for each scenario. At the same time, transfer functioncan be shared across these different scenarios and event triggered mechanismcan be used to adapt the regions used. This can be accomplished loosely usingshared encoder and gating mechanisms within CNN. However, SNN naturallycaptures event triggered model.

There is a class of cameras called event based cameras which encodeinformation at the sensor level. Recently, deep learning algorithms weredemonstrated on event based camera data (Maqueda et al., 2018). Point Cloud:Light Detection and Ranging (LiDAR) sensors have recently gained prominence asstate of the art sensors in sensing the environment. They produce a 3Drepresentation of the objects in the field of view as distances of points fromthe source. This col- lection of points over a 3D space is called a 3D PointCloud. Though cameras have been used for a long time and they provide a moredirect representation of the surrounding, LiDARs have gained ground because ofsome critical advantages such as long range, robustness to ambient lightconditions and accurate localization of objects in 3D space. They producesparse data and hence suitable for SNNs. 3.2 Opportunities SNNs have showngreat potential to either aid or replace CNNs in real-time tasks such as objectdetection, posture recognition etc. (Hu et al., 2016). Large SNN architecturescan be implemented on neuromorphic spiking platforms such as TrueNorth(Benjamin et al., 2014). and SpiNNaker (Furber et al., 2014). The TrueNorth hasdemonstrated to consume as low as couple hundred mW power while packing amillion neurons in it (Sawada et al., 2016). Driven by the strong motivation toreduce power consumption of integrated circuits, implementations of spikingmodels have shown to consume in the order of nJ or even pJ (Azghadi et al.,2014) for signal transmission and processing (Indiveri et al., 2006).

Some neuromorphic designs also feature on-chip learning (Indiveriand Fusi, 2007). Spiking applications and spike based learning is also suitedto dynamic applications like speech recognition systems. In such systems,training is not sufficient at manufacture as it has to adapt to dynamicconditions such as accents.

Other similar sensors are event based Dynamic Vision Sensor (DVS) (Lichtsteiner et al., 2008)(Lenero-Bardallo et al., 2010). Some of the applications especially in theobject detection and perception based tasks that are of direct relevance to theautomotive industry as mentioned briefly below.

Object classification on the CIFAR-10 dataset: (Cao et al., 2015)designed a Spiking equivalent model of a CNN for object detection on theCIFAR10 data set. The CNN was trained on the dataset and the trained model wasthen converted into spiking with each individual block such as convolution, maxpooling, ReLU, being replaced by spiking equivalents. Their transformed modelachieves an error rate of 22.57%. CIFAR-10 is a collection of 60,000 labeledimages of 10 classes of objects (Cao et al., 2015) The network architecture isillustrated in Figure 4. 2. Human action recognition: (Zhao et al., 2015)constructed a network to recognize human actions and posture and successfullytested it. The network was trained on an event-based dataset of small videosequences with simple human actions like sitting, walking or bending. Theyachieved a detection accuracy of 99.48%. This work is an indication of howSNNs may be applied to such event based inference tasks.

We summarize the key benefits of SNN for automated driving:

• Event driven mechanism which brings adaptation for differentscenarios. • Low power consumption when realized as neuromorphic hardware. •Simpler learning algorithm which leads to possibility of on-chip learning for longerterm adaptation. • Ability to integrate directly to analog signals leading totightly integrated system. • Lower latency in algorithm pipeline which isimportant for high speed braking and maneuvering. 4 Conclusion Spiking NeuralNetworks (SNN) are biologically inspired where the neuronal activity is sparseand event driven in order to optimize power consumption. In this paper, weprovide an overview of SNN and compare it with CNN and argue how it can beuseful in automated driving systems. Overall power consumption over the drivingcycle is a critical constraint which has to be efficiently used especially forelectric vehicles. Event driven architectures for various scenarios inautomated driving can also have accuracy advantages.