Experimenting with the proposed Cross-Origin Storage API in Transformers.js

Back to Articles

(This is a guest post by Developer Relations Engineer Thomas Steiner from the Chrome team at Google.)

Transformers.js provides Web developers with a simple way to use the power of transformers in their Web apps through task-specific pipelines. To run inference in the browser, developers create an instance of pipeline()

and specify a task they want to use the pipeline for. As a concrete example, the following snippet shows how to set up an automatic speech recognition (ASR) pipeline.

import { pipeline } from 'https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.2.0' ;

const asr = await pipeline ( 'automatic-speech-recognition' , 'Xenova/whisper-tiny.en' , { device : 'webgpu' }, ); const result = await asr ( 'jfk.wav' ); console . log (result);

The cache challenge

You will notice in the source code that I specified Xenova/whisper-tiny.en

as the model, which is a very decent choice for common English automatic speech recognition tasks. In fact, it's even the default model according to the Transformers.js default model resolution , as per the linked excerpt .

Model resources

When you run this example in the browser , Transformers.js automatically takes care of downloading and caching the relevant model resources and Wasm files. The following screenshot shows the Chrome DevTools Cache storage section after visiting the app. When you reload the page, the resources are served from the Cache API , and the model returns results almost instantly.

However, Xenova/whisper-tiny.en

being a popular model (and, as mentioned before, even being the ASR default model in Transformers.js), you can well imagine that more than just one app that you visit would use it. To simulate this situation, here's the same example app from before, but served from a different origin . When you visit this different origin app, rather than being usable almost instantly, the browser instead has to download and cache all the model resources again, even if they're byte-by-byte the same as before. Even in this toy example, this adds up to 177 MB of duplicate download and storage, as you can examine in the Storage section of the Chrome DevTools Application panel . You can imagine that this quickly adds up.

Wasm runtime resources

But it gets worse. Let's add a second pipeline to the toy example: sentiment analysis. Sentiment analysis by default uses the Xenova/distilbert-base-uncased-finetuned-sst-2-english

model. By not specifying the model, Transformers.js' default model resolution automatically picks it for you.

const classifier = await pipeline ( 'sentiment-analysis' ); const sentiment = await classifier (result. text ); console . log (sentiment);

Two entirely different AI models, but they depend on the same 4,733 kB ort-wasm-simd-threaded.asyncify.wasm

WebAssembly (Wasm) runtime file <a href="https://onnxruntime.ai/docs/api/js/int