Use MCP servers in AnythingLLM with Toolport
Toolport is a free, open source local MCP gateway. Add your servers to Toolport once, flip the AnythingLLM toggle, and AnythingLLM gets all of them through a single gateway entry. Toolport picks a full tool list or search for each client, and you can change it in Clients.
Setup
- Download Toolport for Windows, macOS or Linux, and add your MCP servers to it (or import the ones your other AI agents already have; secrets go to your OS keychain).
- Open the Clients tab and toggle AnythingLLM on. Toolport writes its gateway entry into AnythingLLM's config, next to anything already there.
- Restart or reload AnythingLLM. Your full server set is available through Toolport, and any server you add later shows up without touching AnythingLLM again.
Where AnythingLLM keeps its MCP config
Windows %APPDATA%\anythingllm-desktop\storage\plugins\anythingllm_mcp_servers.json macOS ~/Library/Application Support/anythingllm-desktop/storage/plugins/anythingllm_mcp_servers.json Linux ~/.config/anythingllm-desktop/storage/plugins/anythingllm_mcp_servers.json
JSON with a top-level mcpServers object in AnythingLLM's plugin storage.
Running local models?
Lazy discovery asks the model to search for tools before calling them, and smaller local models often struggle with that multi-step flow. With a capable model it works well; with a smaller one, have Toolport expose the full tool catalog instead, which any model with basic tool calling can use. Either way Toolport, your keys and local-server traffic stay on your machine; remote MCP servers and hosted model APIs still receive the requests you send them.
Why a gateway instead of AnythingLLM's own server list?
- One setup, every agent. The same servers work in AnythingLLM and every other AI agent on your machine; add a server once and they all get it.
- Secrets in the OS keychain. API keys are injected at runtime and never sit in AnythingLLM's config file.
- Watched by default. Rug-pull and tool-poisoning detection on every tool, plus per-tool governance and a destructive-tool kill switch. How the security works.
Free on this machine. Pro puts it on every machine. Team gives it to everyone.