Transformers Model Integration Contracts

On July 30, 2026, Transformers enabled PyTorch's scaled dot-product attention (SDPA) for T5. As a side effect, UMT5's decoder could attend to future tokens. During training, it could peek at the answer it was supposed to predict. Nothing crashed, and the bug stayed in main for almost two months.

When we later replayed those commits, a small UMT5 model running on CPU flagged it on every one of them. This post is about that model and the tests built around it.

Why bugs reach main

Transformers supports over 750 model types, and its test suite is big: 88,000+ CPU tests on main, plus 200,000+ GPU tests in a daily run. Model tests are only part of it: we also test pipelines, training, distributed execution, quantization and attention kernels.

The integration tests load real checkpoints and exercise the model's API on real hardware. Running all of that on every PR would mean keeping a lot of GPUs busy all day long, which isn't practical. By the time a scheduled run finds a regression, it may already be in main.

Our approach is to shrink the model instead. We build a fixture: a small model made from the real checkpoint's configuration, with fewer and narrower layers but every kind of layer it has, and its tokenizer and processor files unchanged. It is published once, then reused. We run "fast integration tests" against it on a cheap CPU runner. In our CPU study , the experimental UMT5 fixture used 66 MB of weights; the source checkpoint is 1,227 MB.

This also covers very large models: some don't fit on our standard GPU runners, but their fixtures run on CPU.

Model integrators already build tiny test models (Transformers' tiny-random-* checkpoints). We wanted a common recipe, and checks that go beyond loading the model and running a forward pass. The UMT5 bug would have passed a forward pass just fine.

Fixtures won't replace full integration tests, but on PRs they give us a chance to catch bugs before they reach main.

Introducing Model Integration Contracts

transformers-mic is a proposed standard for packaging these tests. A Model Integration Contract (MIC) has three parts:

- a fixture: the shrunk model described above, published once and pinned

- a usage path: a runnable Markdown example of something people do with the model, such as generating text or running a training step

- baselines: the outputs recorded when running that example, reviewed by a human

The contract holds if running the usage path on the fixture still produces the baselines.

A contract covers an architecture and a usage path, not a single checkpoint: compatible checkpoints can share a fixture, while each keeps its own expected results. We publish them in huggingface-mic .

For example, the SmolLM contract generates from prompts such as "The capital of France is" and compares its token IDs with a recorded answer. After loading the model and tokenizer, the generation code is, simplified to one prompt:

inputs = tokenizer( "The capital of France is" , return_tensors= "pt" ) outputs = model.generate(**inputs, max_new_tokens= 4 , do_sample= False ) new_tokens = outputs[:, inputs[ "input_ids" ].shape[ 1 ]:] print (tokenizer.batch_decode(new_tokens, skip_special_tokens= True ))

It also compares training loss, checks that gradients are finite and verifies that an optimizer step changes the weights. See the full usage.md .

A fleet dataset on the Hub pins each MIC to a frozen revision, along with the package versions used to record its baselines. Each CI run records the fleet revision it used, so we can reproduce it later.

On a PR, we pick MICs based on the files the patch changes.

The UMT5 regression

We replayed frozen MICs over five months of Transformers commits. The small UMT5 fixture flagged the regression from its introduction on July 30, 2026 until its fix on September 28 . Label log-probabilities differed from the reviewed baseline by 1.22 and training loss by about 0.20.

One detail mattered: the contract used SDPA, the default attention path. Running only the eager attention variant missed the bug. A fixture only helps if it goes through the same code as the real checkpoint.