BRN
Click for more information
7.02% 26.5¢ brainchip holdings ltd

Generative AI is on Mobile and it’s Powered by ArmJanuary 17,...

  1. TBALL
    127 Posts.
    lightbulb Created with Sketch. 69
    clock Created with Sketch.
    05/02/24
    09:08
    Post #: 72234352

    Generative AI is on Mobile and it’s Powered by Arm

    January 17, 2024
    Exciting new developments that demonstrate the advanced AI capabilities of the Arm CPU.

    Generative AI, which includes today’s well-known, highly publicized large language models (LLMs), has arrived at the edge on mobile. This means that AI generative inferences, from generating images and videos to understanding words in context, are starting to be processed entirely on the mobile device, rather than being sent to the Cloud and back.

    Arm is the to enable AI to run everywhere and when it comes to generative AI on mobile, there are some exciting, new developments that demonstrate this in action, from the latest AI-enabled flagship smartphones to LLMs being directly processed on the Arm CPU.

    New AI-powered smartphones

    High performance AI-enabled smartphones are now on the market, which are built on Arm’s v9 CPU and GPU technologies. These include the new , Samsung Galaxy S24, and the Google Pixel 8.

    The combination of performance and efficiency provided by these flagship mobile devices are delivering unprecedented opportunities for AI innovation. In fact, Arm’s own CPU and GPU performance improvements have doubled AI processing capabilities every two years during the past decade.

    This trend will only advance in the future with more AI performance, technologies, and features on our robust consumer technology roadmap. This will be supported by the rise of AI inference at the edge, the process of using a trained model like LLMs to power AI-based applications, with CPUs being best placed to serve this need as more AI support and specialized instructions continue to be added.

    It all starts on the CPU….

    In most cases, the use of AI on our favorite mobile devices starts on the CPU, with some good examples being face, hand and body tracking, advanced camera effects and filters, and segmentation across the many social applications. The CPU will handle such AI workloads in their entirety or be supported by accelerators, including GPUs or NPUs. Arm technology is crucial to enabling these AI workloads, as our CPU designs are pervasive across the SoCs in today’s smartphones used by billions of people worldwide.

    This has led to 70 percent of AI in today’s third-party applications running on Arm CPUs, including the latest social, health and camera-based applications and many more. Alongside the pervasiveness of the designs, the flexibility and AI capabilities of the Arm CPU makes it the best technology for mobile developers to target for their applications’ AI workloads.

    In terms of flexibility, Arm CPUs can run a wide variety of neural networks in many different data formats. Looking ahead, future Arm CPUs will include more AI capabilities in the instruction set for the benefit of Arm’s industry-leading ecosystem, like the . These help the world’s developers deliver improved performance, innovative features and scalability for their AI-based applications.

    The combination of leading hardware and software ecosystem support means Arm has a performant compute platform that is enabling the rise of generative AI at the edge, which could include gaming advancements, image enhancements, language translation, text generation and virtual assistants. We will be demonstrating some examples of these next-gen AI workloads and more at Mobile World Congress 2024.

    LLM on mobile on the Arm compute platform

    We have produced a virtual assistant demo that utilizes Meta’s LLAMA2-7B LLM on mobile via a chat-based application. The generative AI workloads take place entirely at the edge on the mobile device on the Arm CPUs, with no involvement from accelerators. The impressive performance is enabled through a combination of existing CPU instructions for AI, alongside dedicated software optimizations for LLMs through the ubiquitous Arm compute platform that includes the Arm AI software libraries.

    As you can see from the video above, there is a very impressive time-to-first token response performance and a text generation rate of just under 10 tokens per second that is faster than the average human reading speed. This is made possible by highly optimized CPU routinesin the software library developed by the Arm engineering team that improves time-to-first token by 50 percent and text generation by 20 percent, compared to the native implementation in the LLAMA2-7B LLM.

    The Arm CPU also provides the AI developer community with opportunities to experiment with their own techniques to provide further software optimizations that make LLMs smaller, more efficient and faster.

    Enabling more efficient, smaller LLMs means more AI processing can take place at the edge. The user benefits from quicker, more responsive AI-based experiences, as well as greater privacy through user data being processed locally on the mobile device. Meanwhile, for the mobile ecosystem, there are lower costs and greater scalability options to enable AI deployment across billions of mobile devices.

    Find out more information about this demo from the Arm engineers that developed it in .

    Driving generative AI on mobile

    As the most ubiquitous mobile compute platform and leader in efficient compute, Arm has a responsibility to enable the most efficient and highest-performing generative AI at the edge. We are already demonstrating the impressive performance of LLMs that are running entirely on our leading CPU technologies. However, this is just the start.

    Through a combination of smaller, more efficient LLMs, improved performance on mobile devices built on Arm CPUs and innovative software optimizations from our industry-leading ecosystem, generative AI on mobile will continue to proliferate.

    Arm is foundational to AI and we will enable AI everywhere, for every developer, with the Arm CPU at the heart of future generative AI innovation on mobile.

    By , Vice President of Product Management, Client Line of Business, Arm
    https://newsroom.arm.com/generative-ai-on-mobile?utm_source=linkedin&utm_medium=social-organic&utm_content=blog&utm_campaign=mk04_client_na_2023


    Last edited by TBALL: 05/02/24
    BRN Price at posting: 16.5¢ Sentiment: Buy Disclosure: Held
    Likes Created with Sketch. 11  Upvote
    lightbulb Created with Sketch. 1  Great analysis
    Stroke 3 Created with Sketch. Comments Created with Sketch. Reply
 

Featured News

ADX 4DMedical partners with West Los Angeles VA Medical Center for Veteran lung imaging study
right-arrow Created with Sketch.
PMT Patriot sniffs pegmatite grades of 3.58% among the snows of Quebec
right-arrow Created with Sketch.
SBM AI helps lead the way for St Barbara to new FY25-27 production targets
right-arrow Created with Sketch.
AL3 AML3D to investigate copper-nickel alloy printing for US Defence
right-arrow Created with Sketch.
WSR Westar uncovers historic ore in WA's Murchison grading 40g/t
right-arrow Created with Sketch.
RCE Recce Pharma picks up Chinese patent for anti-infective drugs
right-arrow Created with Sketch.

Top Stories

watchlist Created with Sketch. Add BRN (ASX) to my watchlist
(20min delay)
Last
26.5¢ 		Change
-0.020(7.02%) 		Mkt cap ! $491.8M
Open High Low Value Volume
28.0¢ 28.5¢ 26.0¢ $1.810M 6.683M

Buyers (Bids)

No. Vol. Price($)
3 79990 26.5¢
 

Sellers (Offers)

Price($) Vol. No.
27.0¢ 169000 6
View Market Depth
Last trade - 16.10pm 09/05/2024 (20 minute delay) ?
Last
26.5¢ 		  Change
-0.020 ( 6.46 %)
Open High Low Volume
28.5¢ 28.5¢ 26.0¢ 4661260
Last updated 15.59pm 09/05/2024 ?
BRN (ASX) Chart
arrow-down-2 Created with Sketch. arrow-down-2 Created with Sketch.