Use MCP servers in AnythingLLM with Toolport
Toolport is a free, open source local MCP gateway. Add your servers to Toolport once, flip the AnythingLLM toggle, and AnythingLLM gets all of them through a single gateway entry, with lazy discovery loading a handful of meta-tools instead of every server's full tool list.
Setup
- Download Toolport for Windows, macOS or Linux, and add your MCP servers to it (or import the ones your other AI agents already have; secrets go to your OS keychain).
- Open the Clients tab and toggle AnythingLLM on. Toolport writes its gateway entry into AnythingLLM's config, next to anything already there.
- Restart or reload AnythingLLM. Your full server set is available through Toolport, and any server you add later shows up without touching AnythingLLM again.
Where AnythingLLM keeps its MCP config
- Windows
%APPDATA%\anythingllm-desktop\storage\plugins\anythingllm_mcp_servers.json - macOS
~/Library/Application Support/anythingllm-desktop/storage/plugins/anythingllm_mcp_servers.json - Linux
~/.config/anythingllm-desktop/storage/plugins/anythingllm_mcp_servers.json
JSON with a top-level mcpServers object in AnythingLLM's plugin storage.
Running local models?
Lazy discovery asks the model to search for tools before calling them, and small local models (7B class) often aren't up to that flow. With a capable model it works well; with a smaller one, have Toolport expose the full tool catalog instead, which any model with basic tool calling can use. Either way your servers, keys and traffic stay on your machine, which is the point of running local in the first place.
Why a gateway instead of AnythingLLM's own server list?
- One setup, every agent. The same servers work in AnythingLLM and every other AI agent on your machine; add a server once and they all get it.
- Up to 91% fewer tool tokens. Instead of every server dumping its tool list into context, the agent loads a handful of meta-tools and searches on demand (measured).
- Secrets in the OS keychain. API keys are injected at runtime and never sit in AnythingLLM's config file.
- Watched by default. Rug-pull and tool-poisoning detection on every tool, plus per-tool governance and a destructive-tool kill switch. How the security works.