I made a way to migrate between embedding models without re-embedding your entire corpus [R]

A developer has created a method to migrate between embedding models without the need to re-embed an entire corpus. This innovation addresses the significant backfilling cost associated with upgrading models, such as moving from model A to model B, which would typically require re-embedding all documents. For instance, re-embedding 1 billion vectors with a model like qwen embed 8b at 106 docs/second could take approximately 108 days on an H100. The developer is seeking community feedback on this new workflow.

This method offers a way to migrate between embedding models without the typical re-embedding cost, unlike previous approaches that required extensive re-embedding of entire corpora.