Back
RCreddit.com
21
·9 hr ago·Dev community · RSS

You changed one thing. Why is your whole AI pipeline rebuilding again?

View original
GitHubPlans & limitsOpen source

Heat trend

Collecting trend data

The percentage is based on available heat signal, not comment count or independent people.

AI summary

The developer of "aimake" introduces it as an incremental build system for AI/ML pipelines, akin to "make" for AI applications. It addresses the issue of entire AI pipelines rebuilding after a single change. With "aimake," users can see what would rebuild using "aimake plan," run only stale steps with "aimake build," and understand why a step needs rebuilding via "aimake explain." The developer seeks GitHub stars if the idea is deemed useful.

I built aimake — an incremental build system for AI/ML pipelines, basically make for AI apps.

GitHub: https://github.com/arjun988/aimake

The idea is simple:

Dataset → Preprocess → Embeddings → Index → Prompt → Eval → Report

aimake builds a dependency graph, fingerprints inputs by content rather than timestamps, and only rebuilds steps whose inputs actually changed.

So if you change your prompt:

Before: Dataset ✓ Preprocess ✓ Embeddings ✓ Index ✓ Prompt ✗ changed Eval ✗ Report ✗ After: 2 rebuilt · 5 reused aimake plan # see what would rebuild aimake build # only stale steps run aimake explain # see why a step needs rebuilding

It's not Airflow (orchestration) and it's not DVC (data versioning alone).

It's make for AI pipelines: dependency graph + content fingerprints + incremental builds + caching.

Shipped so far

- Content-hash fingerprints instead of mtime-based caching

- Incremental + parallel builds

- plan / build / explain CLI

- Experiment comparison + hyperparameter search

- S3 cache + Hugging Face / DVC / Docker / Ollama / W&B plugins

You can try it with:

pip install aimake

The main reason I built this is that AI pipelines are expensive to rerun. Changing a prompt shouldn't mean recomputing your dataset, embeddings, vector index, etc.

If you build RAG , evaluation or any AI pipelines, I'd love feedback: what's the most painful step you wish was cached?

And if you think the idea is useful, a ⭐ on GitHub would really help.

You changed one thing. Why is your whole AI pipeline rebuilding again? · BuzzRadr