Best practices when running a benchmark on online models [D]

A developer is seeking best practices for benchmarking online models, particularly for low-resource languages, while preventing data leakage. The concern is that input data used for predictions might be used for training by API providers like Google and OpenAI, even with paid accounts. The developer is questioning the trustworthiness of these providers' assurances regarding data usage and is looking for established methods to evaluate online models without compromising input data.

This post highlights the specific challenge of data leakage when benchmarking online models, unlike local models where this is not an issue.