How can I provide a large amount of context to an LLM?
A platform developer seeks optimal methods for providing large amounts of structured context to an LLM, specifically for tasks like DAG generation requiring extensive node referenc…
I'm building a platform where an LLM has to reference a large number of existing nodes. For example, when generating a DAG, it needs to know about many previously defined nodes and correctly reference them while constructing the graph.
I'm trying to figure out the best way to provide this large amount of context while optimizing for latency, cost, and reasoning quality.
Is context caching a good solution when most of the context remains the same across requests? Alternatively, would a Retrieval-Augmented Generation (RAG) setup with a vector database be a better choice? My concern is that the model may need to reference a large number of nodes, not just retrieve a handful of semantically similar ones.
How do people handle situations where an LLM needs access to a very large amount of structured context? I would really appreciate any information, guidance, recommendations, experiences, or resources. Thank you so much!