A couple of weeks back @BarrelSitter found and posted a new...

A couple of weeks back @BarrelSitter found and posted a new patentapplication by Brainchip that has now been published and is waiting approval.Barrel, myself and uiux discussed its significance and today @homeales put up areference to the same patent having missed the original posting and discussion.

This patent application is very significant for a number ofreasons but in particular because it articulates in the embolden and underlinedparagraph below just why AKIDA technology is not the same as anything elsebeing done by anyone else including Intel.

The following total extracted paragraphs give some life to the multipleideas being exposed for the first time to the world but there is much more tobe found but it gets very heavy going for someone such as myself who is no morethan self-taught regarding this neuromorphic science.

Since putting this post together @homeales has posted a link tothe first examination of the patent which found it to have sufficient noveltyto proceed giving greater weight to the assertion in the emboldened underlinedparagraph:

My opinion only DYOR, FF. AKIDA Ballista:

“As is known in the art, deep neural networks (DNNs) have become apopular choice for tasks such as image classification, face recognition, andthe like. However, the typical implementation has resulted in the need formassive computations on von Neumann architectures, which use substantial energyand require large areas. Consequently, efforts have seen a shift away fromconventional digital hardware to implement neural algorithms, including varioustypes of special purpose hardware. However, prominent among the challenges ofspecial purpose hardware is the need for scalability, efficiency and powerconsumption.

In deriving an innovative architecture in response to thesechallenges, the inventors identified at least three contributions: (a) eventbased operation, (b) design of the of the hardware, which controls how much informationis pushed through, and thereby controls the accuracy and power consumption ofthe process, and (c) design of the actual procedure and the precision/bitnumbers for the weights and activations in the neural network so that lessmemory space is used.

The choice of the number of bits is one important consideration inthe design of the layers in neural network architecture. The default number ofbits commonly used in neural networks is 32 bits, with multipliers being 32 bitby 32 bit. Lesser bits may be used, with a resulting smaller power consumptionand chip area required, but with a loss of accuracy. Thus, a 1 bit×1 bitmultiplier would represent the lowest limit on bit choice. Initial experimentshave shown that a 92% accuracy is achievable using ternary weights. Experimentsby the inventors have also shown that a 4×4 bit multiplier provides a sweetspot in terms of relatively low reduction in accuracy (compared with thetraditional 32 bit×32 bit approach), while not significantly increasing the powerconsumption and chip area requirement over that required for a 1 bit×1 bitsolution. The sweet spot choice results in a reduction by a factor of 3 to 4 inthe number of overall parameters used. In various embodiments, the number ofbits used may be different on different layers, i.e., in these embodiments, alayer by layer choice of the number of bits to use (1×1, 2×2, 4×4 etc.) may bemade.

In order to make advancements in a hardware-only implementation ofa spiking neural network, the inventors sought to reduce the operation count.In various embodiments, the approach is event-based, and the spikes areencoded, rather than every single convolution stride. Spikes occur only whenthere are activations, and since events are processed only when spikes occur(rather than upon every convolution stride), there are far fewer events. Sincethere are fewer events, this results in fewer operations. In a typicalembodiment of the present disclosure, the operation count may be reduced by afactor of 3 or 4 by adopting the innovative approach.

In a further innovative advancement, the inventors sought to use aparticular convolution type that requires fewer parameters (e.g., weights).Since these parameters require storage in order to perform the necessary neuralcomputations, a reduction in the requirement of physical memory entriesresults. Although the calculations involved in such a separable convolution aremore involved, the additional calculations are manageable and lead to asubstantial reduction in the number of parameters, and associated physicalmemory entries. Such a reduction in the number of physical memory entriesenables a much larger neural computational capacity to fit within the given RAMarea of a particular chip.

Conventional neural network implementations typically requiresubstantial memory (e.g., SRAM) to save the potential value at each neuron.With the large number of neurons in a typical neural network, the majority ofavailable memory (e.g., SRAM) is used to save potentials associated with the neurons.Consequently, the total number of neurons available in a hardwareimplementation of an SNN neural network is ultimately limited by the availablememory (e.g., SRAM). In order to make advancements in a hardware-onlyimplementation of a spiking neural network, the inventors sought to use memorydifferently by saving the activation levels, rather than the potential. Themagnitude of activation levels requires typically 12 to 20 bits to capturetheir value, while a spike may be captured in only 1, 2, or 4 bits. Thisresulting six-fold reduction in bit requirements substantially increases theeffective storage capability.

Conventional implementations of neural networks perform thecomputations in the network by following its structure, one layer at a time.This approach produces a large amount of intermediate data that are steadilyoutput to memory as the computation progresses. Upon completion of a layer, theintermediate data are sent back to the same computational hardware, and theprocess repeats until computation in all layers has concluded. The amount ofintermediate data that must be transported between the computational hardwareand memory increases with increasing neural network size. In an additionaladvancement, the inventors have employed fusion of the computation acrossmultiple layers. Rather than processing each layer to completion beforeproceeding to the next layer, the inventors have restructured the computationsuch that multiple layers are computed together, which avoids the need to storeor retrieve the intermediate data from the memory. In short, fusing takes theoperations of two or more layers and fuses them into a single operation.”

“The approach of the inventors is unique becauseof the type of research done between the two research communities ofneuromorphic engineering and machine learning.

Neuromorphic engineering researchers commonlyutilize unsupervised learning algorithms along with event-based systems.However, they very rarely utilize event-based convolution.

Machine learning researchers regularly utilizeboth supervised and unsupervised learning algorithms along with convolutionoperations. However, they very rarely venture into algorithms that areevent-based.

The uniqueness of the present approach stems fromthe fact that the system has bridged the gap between two different researchcommunities.”

Claims

1. A system, comprising:

a spike converter configured togenerate spikes from the digital input data;

an inbound filter configured to selectrelevant spikes from the generated spikes, wherein the relevant spikes arerelevant to a neuron;

a memory configured to store kernels ininverted format, and further configured to store weights indexed by channel;

a packet collection module configuredto collect the relevant spikes until a predetermined number of relevant spikeshave been collected in a packet in memory, and to organize the collectedrelevant spikes by channel and spatial coordinates in the packet; and

a convolution neural processorconfigured to perform row-by-row strides in the memory, the convolution neuralprocessor using a scratchpad memory within the memory,

wherein the inbound filter removesevents outside a scope of the convolution neural processor.

2. The system of claim 1, further including apooling layer wherein row pooling and column pooling are performed separately.

3. The system of claim 1, wherein the systemperforms a learning algorithm that limits learning to one neuron per filter.

4. The system of claim 1, wherein the systemperforms a learning algorithm that limits learning to one neuron per layer.

5. The system of claim 1, wherein the systemperforms a learning algorithm that includes a winner-take-all algorithm.

6. The system of claim 1, wherein a ratio ofneurons to memory bits is in the range 0.10 to 0.81.

7. The system of claim 1, wherein a ratio of energyused per synaptic operation (SO) is less than 7 pJ/SO.

8. The system of claim 1, wherein a ratio of energyused per synaptic operation (SO) is less than 3.5 pJ/SO.

9. The system of claim 1, wherein the memory isSRAM.

10. The system of claim 1, wherein the digitalinput data is digital image data.

11. A method, comprising:

receiving digital input data;

generating, by a spike converter,spikes from the digital input data;

selecting, by an inbound filter,relevant spikes from the generated spikes;

storing kernels in a memory in invertedformat, the storing further including storing weights indexed by channel;

collecting the relevant spikes until apredetermined number of relevant spikes have been collected in a packet inmemory;

organizing the collected relevantspikes by channel and spatial coordinates in the packet; and

performing, using a convolution neuralprocessor and a scratchpad memory, a convolution in hardware using row-by-rowstrides in the memory.

12. The method of claim 11, further includingpooling in a pooling layer wherein row pooling and column pooling are performedseparately.

13. The method of claim 11, further includingperforming a learning algorithm that limits learning to one neuron per filter.

14. The method of claim 11, further includingperforming a learning algorithm that limits learning to one neuron per layer.

15. The method of claim 11, further includingperforming a learning algorithm that includes a winner-take-all algorithm.

16. The method of claim 11, wherein a ratio ofneurons to memory bits is in the range 0.10 to 0.81.

17. The method of claim 11, wherein a ratio ofenergy used per synaptic operation (SO) is less than 7 pJ/SO.

18. The method of claim 11, wherein a ratio ofenergy used per synaptic operation (SO) is less than 3.5 pJ/SO.

19. The method of claim 11, wherein the memory isSRAM.

20. The method of claim 11, wherein the digitalinput data is digital image data.

Patent History

Publication number: 20210027152
Type: Application
Filed: Jul 24, 2020
Publication Date: Jan 28, 2021
Applicant: BrainChip, Inc. (Aliso Viejo, CA)
Inventors: Peter AJ VAN DER MADE (Nedlands), Anil S. MANKAR (Aliso Viejo, CA), Kristofor D. CARLSON (Aliso Viejo, CA), Marco CHENG (Aliso Viejo, CA)
Application Number: 16/938,254"

https://patentscope.wipo.int/search/docs2/pct/WO2021016544/pdf/62CNkCGY6W7Z0tBg4c1xshs-nxzc-9knrqHM1VwNIcyDjYjnEzzNmQPqz01qd1-7japyqOMycrsJqX7e3OmeE1KydyyCJ1kGgUuMxLcUt-vgVwlpAeCjBsHkwn82Hhlj?docId=id00000058200788