How Hugging Face Inference Endpoints, Jobs, and Buckets Power Search on Papers with Code

3 months ago, we started a revival of Papers with Code (see also the announcement tweet ). Its goal is to make open AI research accessible and digestible, so that people can easily find the artifacts related to a paper, find state-of-the-art (SOTA) across the various domains of AI, share interesting research and build on top of each other's work. In other words, its goal is to power the wave of research that leads to the next Transformer .

Of course, making AI research accessible requires a powerful search engine, so that humans and agents can quickly find relevant and related work, either through the website or the pwc search CLI command , which agents can use via the Skill .

It's important to note that searching for research is not quite the same as searching for regular text. A useful paper search engine should find an exact title or arXiv identifier, but it should also understand a query such as “small language models for code generation” even when those words do not appear together in a paper. It needs to recognize that “the original BERT paper” is a navigational request, tolerate an incomplete title or typos, and still respond quickly when a model service is cold or temporarily unavailable.

Search results on Papers with Code for the query DINO.

For Papers with Code , we built this as a hybrid search system. This is also based on our prior experience at ML6 , where we developed RAG -based systems for clients. It turned out that hybrid search typically outperforms keyword- and vector-based search systems, as it combines the best of both worlds (see also this blog for more info). Keyword search finds exact mentions, whereas vector search finds more fuzzy, semantically similar terms. Note that rerankers (also called cross-encoders) can further improve the results, although they also come with additional overhead and latency.

Hybrid retrieval outperforms keyword- and vector-only search. Figure from Microsoft, Azure AI Search: Outperforming vector search with hybrid retrieval and reranking (2023).

Papers with Code relies on a PostgreSQL database, hence its full-text search capabilities provide a fast lexical baseline. For dense embeddings, pgvector is used to add semantic recall, and the reciprocal rank fusion (RRF) algorithm combines the two. Three Hugging Face services are used for the dense embeddings:

- Hugging Face Jobs gives us burstable GPU compute for embedding the paper corpus.

- Hugging Face Storage Buckets provides the durable handoff between our database, experiments, and Jobs.

- Hugging Face Inference Endpoints serves low-latency embeddings for live queries and incremental updates.

Today, the system maintains embeddings for more than 110,000 current papers sourced from arXiv and Daily Papers . This post explains the architecture, the design decisions behind it, and the lessons we learned while taking it to production.

TL;DR

We deliberately split search into an offline corpus build and an online search service:

Architecture of the offline corpus build and online hybrid search pipeline.

The expensive, throughput-oriented work runs as Jobs. Durable artifacts live in a Bucket. Only the small query-embedding step sits on the request path, behind a protected Inference Endpoint, to power the online search. If that endpoint is cold, busy, or unhealthy, search immediately falls back to full-text retrieval. This separation makes the system both powerful and fast.

Start with a strict embedding contract

Embedding pipelines often fail in subtle ways: a model revision changes, query and document prompts are mixed up, vectors are truncated differently, or an updated abstract no longer matches its stored vector.

We avoid this by treating the embedding format as a versioned API. Every paper is encoded as:

normalized title + "\n\n" + normalized abstract

For each vector generation, we record:

- the model repository and exact revision;

- the output dimension;

- the input-format version;

- whether the input is a query or a document;