80.5%fewer prompt tokens over a ten-turn run with 100 tools
108msaverage tool selection time for 100 tools
69.7%BFCL accuracy versus 69.3% with all tools exposed
The problem
Agents that expose a large tool catalogue send every tool definition to the model on every turn. That inflates prompt tokens, adds latency, and makes tool choice less reliable as the catalogue grows.
Approach
- Kept the full tool registry available for execution while selecting a smaller relevant subset per model request.
- Used local hybrid retrieval instead of a second LLM call: MiniLM dense embeddings for semantic match and SPLADE sparse embeddings for exact terms.
- Combined both rankings with Reciprocal Rank Fusion and kept the index synchronised when tool definitions change.
- Measured selection quality against the full-catalogue baseline and against an LLM-based tool selector.
Stack
- LangChain
- Python
- zvec
- SPLADE
- MiniLM
- RRF
What these numbers do and do not prove
These are benchmark results reported by the project on its own evaluation setup. They are reproducible from the repository benchmark scripts, but they are not an independent third-party measurement.