2020 BRN Discussion, page-28436

http://www.nextplatform.com/2021/02/11/the-billion-dollar-ai-problem-that-just-keeps-scaling/
http://venturebeat.com/2020/06/01/ai-machine-learning-openai-gpt-3-size-isnt-everything/
http://www.taiwannews.com.tw/en/news/3805032
"

The models that drive AI systems are the factories of the future. Nvidia’s Bryan Cantazaro reckons that by 2025 the most sophisticated language models will cost $1bn to train. Bear in mind that on a price-performance basis, compute in 2025 will be five to 20 times cheaper than today. So in terms of today’s compute, that would be $5 to 20bn dollars. OpenAI’s GPT-3 was rumoured to cost $12m to train last summer, although Sam Altman, OpenAI’s boss, told me it was much more. Interesting commentary from James Wang, reflecting on his forecasts for training scale:

James Wang @draecomino

The original forecast we did for AI training landed at $1 billion to train the largest model by 2025. But it seemed so crazy that we plotted a range and landed at $100m.Nvidia's Bryan Catanzaro @ctnzr's just said he could see $1B models by 2025.Lesson: follow your conviction.

February 17th 2021

43 Retweets237 Likes

The amazing thing is that we can start to compare the cost of training single AI models with the cost of building the physical fabs that make chips. TSMC’s state-of-the-art 3nm fab will run to around $20bn when it is completed in two years. A fab like this may be competitive for 5-7 years, which means it’ll need to churn out $7-8m worth of chips every day before it pays back.

"