Hi ui,
This patent is the one which explains rank coding and its advanyages over rate coding. The US published application was US20210027152.
[005] ... A typical deep convolutional neural network may need to perform in excess of 3 billion multiply-accumulate functions to classify a single object in an image. The processing nodes used in general-purpose computers are usually not fast enough to perform the billions of operations required for classification within a reasonable time span. Arrays of specialized multiply-accumulate devices, graphics processing units (GPU), vector processors, analog multipliers and Digital signal processors have been used to increase the throughput and reduce the latency of deep convolutional neural networks. All these devices have in common that they operate computational cores in parallel and process sequences of data rapid succession, thus being able to process large data sets in a short time. However, with great computational power comes high power consumption. A typical graphics processing unit may consume as much as 200 to 300 watts.
[006] ... Previous attempts to use spiking neural networks in classification tasks have failed because of erroneous assumptions and subsequent inefficient spike rate approximation of conventional convolutional neural networks and architectures. In spike rate coding methods, the values that are transmitted between neurons in a conventional convolutional neural network are instead approximated as spike trains, whereby the number of spikes represent a floating-point or integer value which means that no accuracy gains or sparsity benefits may be expected. Such rate-coded systems are also significantly slower than temporal-coded systems, since it takes time to process sufficient spikes to transmit a number in a rate-coded system. The present invention avoids those mistakes and returns excellent results on complex data sets and frame-based images
[0049] ... event-based convolution is implemented in a Spiking Neural Network (SNN) using event-based rank-coding rather than a perceptron, which has advantages in speed and considerably lower power consumption. Rank coding differs from rate-coding of spike events in that values are encoded in the order of spikes transmitted. In rate coding the repetition rate of spikes transmitted expresses a real number. CNNs process color images which are defined as: imageWidth × imageHeight × channelNumber. A color image generally has 3 channels (Red, Green, and Blue). CNNs often have many layers, so the output of a convolutional layer is the input to the next convolutional layer. Descriptions are provided as to how convolutions take place in perceptron-based CNN before discussing the event-based convolution methods implemented in the present invention and show that the result of convolution in a CNN and event-based convolution in the present invention return the same results. Convolution
[0053] ... Temporal Spike rank coding, conventional perceptron neuron Multiply Accumulate operations (MAC) and conventional binary coding of characters in a computer system. In the present invention spike rank coding is used to perform the functions of a convolutional neural network.
[0054] Rate coding is shown in the top panel 310. The spikes received by two input synapses 320, 330 of a multitude of synapses are shown over one complete integration period labeled t1, in a plurality of integration periods. The first synapse received 25 spikes, while the second synapse received 27 spikes during this integration period. The sum of all synapses is 52, which is the simulated membrane potential of the neuron. Subsequently a non-linear function such as Tanh(n) or a linear rectifier (ReLU) function is applied to the simulated output value. The resulting output value is transmitted as a series of spikes to one or more synapses in the next neural layer. The integration time is long to allow sufficient spikes to occur to receive a value. In the lower left panel 330, rank coding is illustrated. The spikes received by four of a multitude of synapses is shown for six integration periods labeled t0 to t5. The integration time is short, and repeating spikes within each integration period are ignored. The integrated value for the first integration period is three, and four in the subsequent four integration periods. The last integration period has the value 2. These values are the simulated membrane potential of the neuron. If the simulated membrane potential reaches or exceeds a threshold value, a spike is transmitted to one or more synapses in the next neural layer. In the middle right hand panel 350, the integration method of a section of a neuron of a plurality of neurons in a conventional perceptron is shown. A collection of weight values labeled W11 to W95 are multiplied with input values I0 to I9. The resulting values are accumulated to form the neuron simulated membrane potential. Subsequently a non-linear function such as Tanh(n) or a linear rectifier (ReLU) function is applied to the simulated output value. The resulting output value is transmitted as an integer or floating-point value to one or more synapses in the next neural layer. In the lower right-hand panel 370, a conventional binary coding method is shown for reference. Binary coding schemes are widely used in conventional computer systems to encode characters and numbers. Boolean algebra is applied to compute with binary coded numbers and characters. T
1. A system, comprising:a spike converter configured to generate spikes from the digital input data;
an inbound filter configured to select relevant spikes from the generated spikes, wherein the relevant spikes are relevant to a neuron;
a memory configured to store kernels in inverted format, and further configured to store weights indexed by channel;
a packet collection module configured to collect the relevant spikes until a predetermined number of relevant spikes have been collected in a packet in memory, and to organize the collected relevant spikes by channel and spatial coordinates in the packet; and
a convolution neural processor configured to perform row-by-row strides in the memory, the convolution neural processor using a scratchpad memory within the memory,
wherein the inbound filter removes events outside a scope of the convolution neural processor.
(20min delay)