On-device AI: From point applications to agentic experiences

What stage has on-device AI reached?

After three to four years of development, on-device AI is reaching an important inflection point.

From an application perspective, on-device AI did not begin with large models. Imaging was actually one of the earlier areas where AI was deployed on devices. In recent years, smartphone photography has continued to improve, driven in part by on-device AI technology. Today, on-device AI has expanded further into multimodal applications such as simple question answering, chatbots, image generation and editing, text processing, speech transcription, and real-time translation. The conditions for deploying many of these capabilities on-device are already relatively mature.

However, many on-device AI applications still focus on individual, single-point functions. Automatic speech recognition (ASR), text summarization, image processing, and simple chatbots, for example, mostly solve one specific task. If devices are expected in the future to further perform system-level, planning-oriented tasks such as intent understanding, task orchestration, and operating target apps in real time, stronger on-device model capabilities will be needed.

This means that on-device AI is gradually evolving from solving individual tasks toward more sophisticated agentic experiences.

Why is on-device AI needed? What is the relationship between the device and the cloud?

Qualcomm does not emphasize an on-device-only approach, but rather device-cloud collaboration, or the "continuum of compute." One important concept within this framework is "on-device first."

Traditional AI services that rely on the cloud consume cloud tokens, GPU resources, and cloud scheduling resources for every inference, while also depending on network connectivity. Deploying suitable AI capabilities on-device can reduce reliance on the network, provide more timely and stable responses, and leverage the advantages of devices in terms of security, privacy, and personalization.

An intuitive way to understand this is to think of the cloud as more of a "general-purpose knowledge base," while the device holds more "specialized knowledge" specific to the individual user.

The device can take into account who the user is, where they are, what they have done in the past, and their current environment to determine which services they may need next. This is also the foundation of "proactive AI services."

Therefore, the device is not an isolated system. The cloud can handle tasks that require greater intelligence and computing power, while the device can offer advantages in real-time performance, stability, privacy, and personalization. The two work in collaboration with each other.

Why has MoE become an important technical direction for on-device AI?

MoE stands for Mixture of Experts, meaning a Mixture-of-Experts model. To understand MoE, you can start by comparing it with a traditional Dense model.

Take a 3-billion-parameter Dense model as an example. For every inference, all 3 billion parameters must be run through before the final output can be produced.

MoE works differently. It divides the model's parameters into multiple different "expert" modules, each of which is responsible for a portion of the parameters and may specialize in different tasks. When the model processes a specific task, it only needs to invoke the expert modules relevant to that task.

For example, an MoE model contains 20 expert modules, but when performing a task, it may need to invoke only 2 or 3 of them instead of running all 20 expert modules simultaneously.

A simple “chef” analogy: a Dense model is like having only one chef at home who makes whatever dish you want. He may know many cuisines, but is not necessarily best at every one. MoE is like having multiple chefs at the same time, each specializing in a different cuisine—some in Shandong cuisine, some in Beijing cuisine, and some in Huaiyang cuisine. When you want Huaiyang cuisine, only the chef specializing in it needs to work, while the others can rest.

In other words, although the model has many “expert” modules, only some of them need to participate in the computation when it actually executes a task. This mechanism reduces the number of parameters that actually need to run for each inference and improves the model’s runtime efficiency.

Why is MoE particularly important for on-device AI?

MoE has already been applied in cloud-based large models. As on-device AI continues to develop, an increasing number of on-device models are also adopting MoE solutions.

This technology is especially important for devices because smartphones, PCs, and other devices are constrained by factors such as memory and power consumption.

The current mainstream approach typically allows only models with 2 billion to 3 billion parameters to run on-device. Models of this size can already perform individual tasks such as ASR, text summarization, image processing, and simple conversations quite well. However, if the goal is to complete complex system-level or planning tasks, the models’ intelligence is still subject to certain limitations.

At this stage, models with 2 billion, 3 billion, or 4 billion parameters are not wholly unable to perform these complex tasks, but their level of completion and accuracy still leaves room for improvement, potentially reaching 60%, 70%, or 80%. With MoE, there is an opportunity to further improve the intelligence of on-device models and the level of completion and accuracy for complex tasks.

This is also an important significance of MoE for on-device AI: under the memory and power constraints of devices, it provides a new technical path for deploying and running models with higher levels of intelligence.

As the relevant technologies continue to advance, capabilities currently demonstrated live—such as intent understanding, task orchestration, and operating target apps in real time—may in the future be applied more widely across devices such as smartphones, cars, and PCs.

What did Qualcomm’s on-device MoE agent demonstration at the China International Import Expo show?

Because an MoE model contains multiple expert modules, its overall size is relatively large. To run it truly on-device, two highly practical issues must be addressed in addition to the model’s own intelligence: how to predict in advance which expert modules will be needed next based on the current task, and, after making the prediction, how to quickly load those expert modules from storage so they can participate in the computation in time.

During large-model inference, DDR bandwidth is often an important bottleneck. If the expert modules needed at the next step can be identified in advance and prefetched to the appropriate location, waiting time can be reduced and the model’s runtime efficiency improved.

Qualcomm’s on-device MoE agent demonstration at the ninth China International Import Expo explored precisely these issues and was jointly completed by Qualcomm, Infris AI, Longsys, and StepFun. To enable efficient operation of on-device MoE models, the partners collaboratively optimized different aspects spanning the model, inference, storage, and chip platform: StepFun focused on enhancing the model’s intelligence; Infris AI used its inference solution to predict and prefetch the required expert modules; Longsys optimized the efficiency of loading expert modules from Universal Flash Storage (UFS) into memory so that they could be activated more quickly; and Qualcomm, focusing on the chip platform and software stack, worked with its partners to optimize expert-module routing, expert-module prediction, cache prediction, and other areas, thereby improving the overall runtime efficiency of MoE models on-device. Thus, this demonstration showed not just a model, but a systematic optimization effort spanning the model, inference, storage, and chip platform. It also reflected the shift in on-device AI from isolated technological innovation toward coordinated industry collaboration.

Why does on-device AI increasingly require ecosystem collaboration?

As AI and large models continue to develop, the focus of Qualcomm’s collaboration with upstream and downstream industry partners is also changing.The goal is not simply to “move” models onto end devices, but to make them run better on-device and further integrate these capabilities into a wide range of devices, including smartphones, PCs, glasses, and automobiles.

In the on-device AI ecosystem, different partners play different roles: model providers supply the “intelligence.” Qualcomm works with model providers to optimize models through model quantization and a combination of chip technologies and the software stack, making models smaller and inference faster, and advancing collaboration from “model + chip” to “model for chip” or “chip for model.” Device manufacturers control users’ “entry points” and possess device, operating system, and agent capabilities. At the same time, devices carry a wealth of personal and personalized data, so chip capabilities and model intelligence need to be integrated into AI experiences that users can perceive. Developers bring AI capabilities to more apps and use cases, incorporating the capabilities and permissions of chips, models, and operating systems into specific applications, while using a rich app ecosystem to address the “long-tail effect” of AI applications. Qualcomm provides the foundation for on-device AI capabilities, continuously enhancing AI processing capabilities such as NPUs and collaborating with model providers, device manufacturers, and developers through the Snapdragon platform, full-stack AI software, and development tools to drive the deployment of AI capabilities across more devices and applications.

The MoE agent Demo at this CIIE is also a concrete embodiment of this collaborative ecosystem model: from models, inference solutions, and storage to chip platforms, different partners address challenges at different stages, working together to enable more complex AI capabilities to run on-device.

This article was provided by Qualcomm. QbitAI is authorized to repost it, and all views belong to the original author.