Recently, during the 2024 Mobile World Congress (MWC 2024), HuiLi Technology and MediaTek once again collaborated to introduce innovative applications of generative AI at the edge. Leveraging the next-generation AI processor integrated into the MediaTek Dimensity 9300 and HuiLi Technology's LoRA fusion technology, users can now generate videos in real-time with different animation styles while recording images on edge devices. This marks the industry's first application of real-time video generation at the edge based on LoRA fusion technology, further pushing the boundaries of edge-side generative AI applications.
Real-time Fun Video Generation with Edge Processing and LoRA Fusion Technology
As a key partner of MediaTek in the field of generative AI technology, HuiLi Technology combines MediaTek's generative AI model edge-side "skill expansion" technology, NeuroPilot Fusion, to continuously expand AI applications and functionalities on the base model. It has successfully achieved real-time generation capabilities of various stylized videos on mobile devices, paving the way for a series of new AI-driven mobile applications.
During the training of stylized base models and LoRA style models, HuiLi significantly reduces the running steps of diffusion models through consistent distillation algorithms. Leveraging MediaTek's NeuroPilot framework, it further reduces the single-step time consumption of diffusion models without classifier-guided distillation. While ensuring output quality, HuiLi has achieved near 1 frame/s real-time stylization generation effect on mobile devices, making the generation experience more natural and smooth.
In live demonstrations, users can accurately identify human images during the shooting process and transform them into various styles of fun videos. In addition, the model can also stabilize and delicately transform backgrounds and items held by people in the scene into backgrounds and props that match the artistic style, making the overall video effect more natural and coordinated. For example, when a user shoots a video holding a disc-shaped item, in an oil painting style video, the disc is recognized and recreated as a palette, while in a cyberpunk style, the disc is defined and displayed as a shield.
In the past, switching between different artistic styles of LoRA in mobile applications required replacing the entire model, which made it difficult to achieve real-time switching and loading during actual video shooting. When the application contained multiple styles of LoRA, it occupied a large amount of memory, leading to gigabyte-level memory requirements for the installation package. Now, leveraging the NeuroPilot framework's LoRA fusion feature, Hualing has compressed the size of autonomously trained LoRA models to the 10MB level. Different LoRA styles can be used in conjunction with a single base model, allowing users to switch freely in a very short time. This results in faster processing speeds, fully meeting the personalized experience needs of edge AI users.
Activating the heat of generative AI creation, seizing the AI self-media era
In recent years, generative AI has been continuously heating up in the content creation arena, with content creators and consumers alike eagerly calling for more innovative and cutting-edge application experiences. Hualing's application achievements in LoRA fusion on the edge have opened up a more imaginative creative space in the era of "everyone is a self-media."
In content social platforms such as Douyin and Xiaohongshu, the previous mobile shooting and creation gameplay mainly focused on beauty filters and adding accessories. The new video generation gameplay brought by edge LoRA fusion provides users with more new options for content inspiration and shooting styles, greatly improving efficiency in creation. Based on real-time video generation capabilities, users can freely choose from various painting styles such as cyberpunk, watercolor, oil painting, ink painting, and cartoon during real-time shooting. After the shooting ends, the video is generated immediately with rich elements, high quality, and efficiency, providing an excellent user experience.
Moreover, in scenarios such as cultural tourism, the application prospects of real-time video generation through mobile shooting are even broader. During the process of creators' check-ins, they only need to record a scene on-site, and they can create works in different styles through this feature, allowing users to traverse various painting styles' dimensional worlds freely, bringing more creative immersive experiences to users.
With the development of edge generative AI, the potential of the mobile AI arena is fully emerging. The application upgrade of technologies such as LoRA fusion will further tap into the application potential of AI on edge devices such as mobile phones and empower participants in various arenas to accelerate their entry and explore greater participation space. As pioneers and builders in the era of large models, Hualing Technology will continue to increase technological empowerment to bring more cutting-edge AI application experiences to partners and users.
