The global open-source large model ecosystem has seen a disruptive breakthrough at the architectural level. Google officially released the new unified multimodal model Gemma412B on June 3rd. The biggest innovation of this model is completely eliminating the "encoder" component, which has been essential in traditional multimodal models, achieving a qualitative leap in local deployment and inference efficiency on consumer-grade hardware.

In traditional multimodal architectures, models typically rely on separate visual and audio encoders to convert image and sound signals into dimensions compatible with text tokens, which无形中 increases the model's size and computational complexity. However, Gemma412B takes a different approach, using a lightweight embedding layer to directly process visual input, completing the conversion with just a single matrix multiplication, position embedding, and normalization operation; simultaneously, audio signals are also directly projected into the dimension space of text tokens. This streamlined "encoder-free" design not only significantly reduces computational steps but also makes the entire model extremely lightweight.

image.png

Thanks to the optimized architecture, this high-performance model with 12 billion parameters is perfectly compressed within the operational threshold of consumer-grade hardware. Developers or general users can directly deploy and run it smoothly on high-end laptops with just 16GB of VRAM or unified memory. This means users don't need to rely on expensive cloud computing power to handle complex visual and audio tasks offline.

In terms of actual performance, the multi-step reasoning and agent workflow (Agent) capabilities of Gemma412B have approached those of Google's larger 26B MoE model. To further extract performance, the model also features Multi-Token Prediction (MTP) technology, which can predict multiple tokens simultaneously, significantly accelerating the inference response speed on the edge side.

Currently, Gemma412B has been officially open-sourced under the friendly Apache 2.0 license, and the model weights have also been released. The new model has received full support from the mainstream development ecosystem, seamlessly supporting multiple inference frameworks such as Ollama, LM Studio, MLX, SGLang, and vLLM. Google's own AI Edge Gallery has also provided an edge-side deployment package immediately. For enterprise production environments, developers can also perform large-scale cluster deployments through Google Cloud's relevant tools. With the cumulative downloads of the Gemma4 series models exceeding 150 million times, this new architecture will undoubtedly trigger a new wave of technological excitement in the open-source developer community.