Files All the Way Down — Empirical Findings
Empirical results from “Files All the Way Down: A Design Space Analysis of Filesystem-Native LLM Tool Integration.”
Live experiments on Claude Code (Linux x86_64). All reproducible via bash research/run-all.sh.
Token consumption is a direct proxy for inference cost and latency. We ran identical tasks against LiveFoldersFS and MCP and recorded per-turn token usage via the Anthropic API. LF-native beats MCP by 6–10% on every task, with 100% task completion across all runs.
folder.yaml via anthropic_tools.json — achieves 6–10% lower token
consumption than MCP across an 8-task benchmark, with 100% task completion on all tasks and all backends.
| Backend | How it works | Result |
|---|---|---|
livefolders (bash) |
Original cat/echo invocation, system prompt from system_prompt.md |
2.5–5× more expensive than MCP (up to 8× on near-timeout runs). Not production-suitable. |
livefolders-manifest |
Same filesystem; system prompt is the concise per-endpoint listing synthesized by the VFS | +2–8% vs MCP (consistently more expensive, but large improvement over bash) |
livefolders-native ★ |
Tool schemas loaded from anthropic_tools.json; model calls structured Anthropic tools |
−6–10% vs MCP on every task |
livefolders-unified |
anthropic_tools.json schemas collapsed into one tool per namespace with an action enum |
−5–9% vs MCP (2–5% worse than native due to reduced per-action guidance) |
mcp (baseline) |
Standard MCP server over stdio; tool schemas via list_tools |
Baseline (0%) |
Model: claude-haiku-4-5 (maximises sensitivity to schema overhead). 5 runs per task, results averaged.
All 5 backends call the same upstream API (mockapi.io, 50 users). Traces logged to Weights & Biases Weave.
| Task | Description | LF-native | LF-unified | MCP | LF-native vs MCP |
|---|---|---|---|---|---|
count_users |
Count total users | 1,893 | 1,915 | 2,094 | −9.6% |
count_composed |
Count via pipe-composed endpoint (server-side) | 1,892 | 1,914 | 2,093 | −9.6% |
single_user |
Look up a user by name | 1,962 | 2,235 | 2,163 | −9.3% |
list_users |
List all users with name and ID | 2,917 | 2,915 | 3,176 | −8.2% |
cross_reference |
Find user's account creation date | 3,190 | 3,231 | 3,471 | −8.1% |
count_and_find |
Count users AND find a specific user's ID | 2,102 | 2,148 | 2,282 | −7.9% |
filter_and_count |
Find users with 'a' in name; count them | 2,582 | 2,622 | 2,794 | −7.6% |
list_compact |
List users as compact id:name pairs (pipe) | 2,786 | 2,805 | 2,975 | −6.4% |
5-run averages. With only 5 runs per task, effects below ~3% should be treated as indicative rather than statistically robust.
hidden: true in folder.yaml) are excluded from anthropic_tools.json,
while MCP exposes all registered tools unconditionally. For the users API (6 visible, 3 hidden endpoints),
the LF schema is ~15% smaller than the equivalent MCP list_tools response.
filter_and_count, adding a server-side filter endpoint dropped
output tokens from ~963 to ~386 — the model receives a pre-aggregated answer rather than 50 raw users.
cross_reference task benefits from a get endpoint (fetch-by-ID) that returns
exactly the needed fields (name, id, createdAt, avatar)
rather than requiring the model to scan a full listing in context.
livefolders backend
(plain cat/echo via system prompt) is 155–400% more expensive than MCP on tasks that complete,
with runs approaching the 10-turn safety cap pushing toward 8× the token cost.
Use anthropic_tools.json (livefolders-native) for production deployments.
anthropic_tools.json Works
LiveFoldersFS synthesizes a virtual anthropic_tools.json file at read time from folder.yaml.
Each invokable endpoint becomes an Anthropic-format native tool definition that can be injected directly
into the tools=[] API parameter — enabling structured tool calling without shell access.
|
folder.yaml (source)
name: users
description: User management tools.
files:
- name: count
type: read_invoke
description: Count total users
handler: curl -s .../users | jq length
- name: search
type: write_invoke
description: Search users by name
input:
type: string
min_length: 1
handler: ...
- name: _internal_stage
type: read_invoke
hidden: true ← excluded from tools
handler: ...
|
anthropic_tools.json (synthesized)
[
{
"name": "users__count",
"description": "Count total users",
"input_schema": {
"type": "object",
"properties": {}
}
},
{
"name": "users__search",
"description": "Search users by name",
"input_schema": {
"type": "object",
"properties": {
"input": {
"type": "string",
"minLength": 1
}
},
"required": ["input"]
}
}
]
|
anthropic_tools.json automatically — the model only sees the
endpoints it should call, reducing schema tokens. The file is never written to disk; it is synthesized fresh on every read
from the current folder.yaml.
All seven systems evaluated across ten criteria using three evidence tiers: T1 = live experiments · T2 = structured assessment (code + install) · T3 = paper-only. Rating: ✓ strong · ~ partial · ✗ weak · — N/A.
| Criterion | LiveFolders T1 |
MCP T1 |
ToolFS T2 |
AgentFS T2 |
llm9p T2 |
InferNode T3 |
Quine T3 |
|---|---|---|---|---|---|---|---|
| 01 Setup complexity | ✓ | ~ | ~ | ~ | ~ | ✗ | ~ |
| 02 LLM compatibility | ~ | ~ | ~ | ~ | ~ | ~ | ~ |
| 03 Discoverability | ✓ | ~ | ~ | — | — | ~ | ~ |
| 04 I/O expressiveness | ~ | ~ | ~ | ~ | ~ | ~ | ✓ |
| 05 Security | ✓ | ✓ | ~ ⚠ | ~ | ✗ ⚠ | ✓ | ~ |
| 06 Stateful tools | ✓ | ~ | ✓ | ~ | ~ | ~ | ~ |
| 07 Composability | ✓ | ~ | ~ | — | ~ | ✓ | ✓ |
| 08 Observability | ✓ | ✓ | ~ | ✓ | ✗ | ~ | ~ |
| 09 Hot-reload | ✓ | ✗ | ✗ | — | ~ | ~ | ~ |
| 10 Publishing | ✓ | ~ | ~ | — | — | ✗ | ~ |
| ✓ count | 8 | 3 | 1 | 1 | 0 | 3 | 2 |
⚠ ToolFS criterion 05: WASM sandbox was an unimplemented stub (InMemorySandbox) at the time of T2 code inspection (early 2025); subsequent releases may differ.
llm9p criterion 05: unauthenticated TCP port accessible to any network process.
LiveFoldersFS is os-coupled and posix-invocation: the FUSE kernel layer is the sole entry point, and every tool invocation reduces to a standard file write followed by a file read — no protocol, no SDK.
src/fs/vfs.rsImplements the FUSE Filesystem trait. Maintains three in-memory maps keyed by inode:
write_buf (accumulates write bytes), result_buf (handler output),
trace_buf (companion .log content). The release() event
triggers handler dispatch for write_invoke endpoints; read_invoke fires on read().
src/registry/Maps names to Arc<dyn Tool> implementations. Built-in tools registered at startup;
external tools loaded from subdirectories under tools_dir, each described by a
folder.yaml manifest parsed by src/manifest.rs.
src/fs/how_to_gen.rs & schema_gen.rsSynthesize how_to.md and schema.json at read time from the manifest.
Never written to disk. anthropic_tools.json is also synthesized here,
excluding hidden endpoints automatically.
|
src/sandbox/Applies PR_SET_NO_NEW_PRIVS, Landlock LSM filesystem access control, and seccomp-BPF
network filtering before each handler process starts. Transparent to handler authors —
shell scripts run unchanged inside the sandbox.
Hot-reload via inotify
An inotify-based watcher monitors tools_dir for changes and hot-reloads the registry
without remounting. Updated handlers are visible within ~1 second, with no agent reconnection.
Daemon layer
Forks after mount options are validated, redirecting stderr to a log file and writing a PID file for the stop subcommand. livefolders mount returns to the prompt
immediately while the daemon runs in background.
|
Coupling depth determines ergonomics ceiling.
OS-coupled systems inherit decades of POSIX tooling. The token benchmark adds a quantitative dimension:
when LiveFoldersFS exposes tools as Anthropic-native schemas via anthropic_tools.json,
it achieves 6–10% lower token consumption than MCP. The advantage compounds on aggregate tasks
because server-side composition moves work from model context to tool execution —
structurally unavailable to protocol-decoupled systems without equivalent server-side aggregation.
Tradeoff: os-coupled systems require shell and filesystem access; MCP operates over HTTP and
can serve agents anywhere.
Invocation interface determines safety floor.
RPC systems (MCP) enforce input schemas unconditionally before any handler runs.
POSIX systems historically concentrated security on handler authors.
LiveFoldersFS narrows this gap with opt-in per-endpoint validation (v0.7.0) plus VFS-layer sandboxing
via Landlock + seccomp-BPF (v0.11.0). Remaining gap: MCP validation is unconditional;
LiveFoldersFS requires explicit opt-in per endpoint. Worst finding in the survey:
llm9p's unauthenticated TCP port is accessible to any network process.
Publishing model is underexplored.
LiveFoldersFS is the only surveyed system treating tool distribution as a first-class design concern.
livefolders install github.com/you/repo requires no registry, no package upload, no config editing.
All other systems require manual configuration or informal community registries.
As the ecosystem matures, install friction will determine whether tool ecosystems grow organically
or remain fragmented.
claude-haiku-4-5) and a single upstream API (mockapi.io, 50 users). Token margins may shift for larger models, denser APIs, or different tool-call-to-output ratios.bash research/run-all.sh from the repository root.