Figure 1. A) Receptive field of a simple cell in V1 (DeAngelis, et al, 1995), B) A **or Model of the receptive field of simple cell. C) Learned receptive fields of a CNN (Brachmann & Redies,2016)
The path from biological neurons to artificial neural networks has witnessed several innovations, with each step bringing us closer to more efficient and capable AI systems. Initially, fixed, non-learnable kernels like the **or filter above mimicked the visual processing of the human brain Figure 1A, focusing on specific features within visual stimuli Figure 1B. These early models laid the groundwork for more complex representations.
With the advent of deep learning, learnable kernels became prevalent, allowing AI models to adapt their filters based on the data they processed (Figure 1C). However, this flexibility comes with increased computational demands and a surge in the number of parameters. TENN addresses these challenges by revisiting the foundation of kernel representation. It uses a structured approach to reduce parameter explosion and discretization issues, thereby maintaining computational efficiency while retaining flexibility.
(20min delay)