2020 BRN Discussion, page-24504

http://venturebeat.com/2021/01/12/google-trained-a-trillion-parameter-ai-language-model/
http://www.technologyreview.com/2020/12/04/1013294/google-ai-ethics-research-paper-forced-out-timnit-gebru/
http://www.nature.com/articles/s43587-020-00020-4

"Dept of the near future
Google recently trained a new language model with 1.6 trillion parameters. The more parameters, the more sophisticated a model can appear. OpenAI’s GPT-3, about which I’ve written, had 175bn parameters and good craft some technically impressive outputs. Now that firms have figured out that increasing the complexity of these models drops jaws, there is obviously some competition going on. Microsoft (which pulled out a 17bn model last year that was briefly the world’s biggest) is also in the running. So long as bragging rights matter, and given powers-of-ten, this means that we shouldn’t be surprised to see a 10 trillion parameter model within a year, assuming the teams can find a sufficiently large training corpus. Of course, none of these obviates the limitations of such an approach. Indeed, the “Stochastic Parrots” paper by Gebru, Bender, et al outlines many of the risks of such large models, in particular, the way they will rigorously encode whatever problematic material is fed into them and they become rather complicated to analyse, debug and eliminate the source of such issues. (Elsewhere, I found this overview of deep learning in longevity research interesting.)"