返回
RCreddit.com
21
·9小时前·开发者社区 · RSS

You changed one thing. Why is your whole AI pipeline rebuilding again?

查看原文
GitHub模型发布订阅权益开源代码

热度趋势

趋势数据积累中

百分比基于当前可用热度信号,而非评论数或独立用户人数。

推荐理由

GitHub 相关模型动态已经出现,适合跟踪能力变化、生态影响和后续可用性。

AI 摘要

“aimake”的开发者推出了一款针对AI/ML管道的增量构建系统,其功能类似于AI应用的“make”。该系统旨在解决AI管道在仅修改一处后便需整体重建的问题。通过“aimake plan”,用户可以查看哪些部分需要重建;使用“aimake build”,则只运行过时的步骤;而“aimake explain”则能帮助用户理解某个步骤为何需要重建。开发者表示,如果用户认为这个想法有用,希望能在GitHub上获得关注。

I built aimake — an incremental build system for AI/ML pipelines, basically make for AI apps.

GitHub: https://github.com/arjun988/aimake

The idea is simple:

Dataset → Preprocess → Embeddings → Index → Prompt → Eval → Report

aimake builds a dependency graph, fingerprints inputs by content rather than timestamps, and only rebuilds steps whose inputs actually changed.

So if you change your prompt:

Before: Dataset ✓ Preprocess ✓ Embeddings ✓ Index ✓ Prompt ✗ changed Eval ✗ Report ✗ After: 2 rebuilt · 5 reused aimake plan # see what would rebuild aimake build # only stale steps run aimake explain # see why a step needs rebuilding

It's not Airflow (orchestration) and it's not DVC (data versioning alone).

It's make for AI pipelines: dependency graph + content fingerprints + incremental builds + caching.

Shipped so far

- Content-hash fingerprints instead of mtime-based caching

- Incremental + parallel builds

- plan / build / explain CLI

- Experiment comparison + hyperparameter search

- S3 cache + Hugging Face / DVC / Docker / Ollama / W&B plugins

You can try it with:

pip install aimake

The main reason I built this is that AI pipelines are expensive to rerun. Changing a prompt shouldn't mean recomputing your dataset, embeddings, vector index, etc.

If you build RAG , evaluation or any AI pipelines, I'd love feedback: what's the most painful step you wish was cached?

And if you think the idea is useful, a ⭐ on GitHub would really help.

You changed one thing. Why is your whole AI pipeline rebuilding again? · BuzzRadr