WebLLM: high-performance in-browser LLM inference engine
热度趋势
趋势数据积累中
百分比基于当前可用热度信号，而非评论数或独立用户人数。
官方发布涉及Mistral 模型访问、订阅权益规则，适合跟踪产品开放节奏和用户影响。
WebLLM是一个高性能的浏览器内LLM推理引擎，支持多种Mistral模型，包括Mistral-7B-v0.3、Hermes-2-Pro-Mistral-7B、NeuralHermes-2.5-Mistral-7B和OpenHermes-2.5-Mistral-7B。它为ServiceWorker提供了API支持，允许开发者将生成过程集成到Service Worker中。此功能有助于优化离线体验，并避免在每次页面访问时重新加载模型，从而提高Web应用程序的效率。
WebLLM
High-Performance In-Browser LLM Inference Engine.
Documentation | Blogpost | Paper | Examples
Overview
WebLLM is a high-performance in-browser LLM inference engine that brings language model inference directly onto web browsers with hardware acceleration. Everything runs inside the browser with no server support and is accelerated with WebGPU.
WebLLM is fully compatible with OpenAI API. That is, you can use the same OpenAI API on any open source models locally, with functionalities including streaming, JSON-mode, function-calling (WIP), etc.
We can bring a lot of fun opportunities to build AI assistants for everyone and enable privacy while enjoying GPU acceleration.
You can use WebLLM as a base npm package and build your own web application on top of it by following the examples below. This project is a companion project of MLC LLM , which enables universal deployment of LLM across hardware environments.
Key Features
- In-Browser Inference: WebLLM is a high-performance, in-browser language model inference engine that leverages WebGPU for hardware acceleration, enabling powerful LLM operations directly within web browsers without server-side processing.
- Full OpenAI API Compatibility : Seamlessly integrate your app with WebLLM using OpenAI API with functionalities such as streaming, JSON-mode, logit-level control, seeding, and more.
- Structured JSON Generation: WebLLM supports state-of-the-art JSON mode structured generation, implemented in the WebAssembly portion of the model library for optimal performance. Check WebLLM JSON Playground on HuggingFace to try generating JSON output with custom JSON schema.
- Extensive Model Support : WebLLM natively supports a range of models including Llama 3, Phi 3, Gemma, Mistral, Qwen(通义千问), and many others, making it versatile for various AI tasks. For the complete supported model list, check MLC Models .
- Custom Model Integration : Easily integrate and deploy custom models in MLC format, allowing you to adapt WebLLM to specific needs and scenarios, enhancing flexibility in model deployment.
- Plug-and-Play Integration: Easily integrate WebLLM into your projects using package managers like NPM and Yarn, or directly via CDN, complete with comprehensive examples and a modular design for connecting with UI components.
- Streaming & Real-Time Interactions: Supports streaming chat completions, allowing real-time output generation which enhances interactive applications like chatbots and virtual assistants.
- Web Worker & Service Worker Support: Optimize UI performance and manage the lifecycle of models efficiently by offloading computations to separate worker threads or service workers.
- Chrome Extension Support: Extend the functionality of web browsers through custom Chrome extensions using WebLLM, with examples available for building both basic and advanced extensions.
Built-in Models
Check the complete list of available models on MLC Models . WebLLM supports a subset of these available models and the list can be accessed at prebuiltAppConfig.model_list .
Here are the primary families of models currently supported:
- Llama: Llama 3, Llama 2, Hermes-2-Pro-Llama-3
- Phi: Phi 3, Phi 2, Phi 1.5
- Gemma: Gemma-2B