.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
If you've ever seen "session_search permanently disabled: database is locked" in your Hermes logs, you've hit the issue tracked as #3139 on the GitHub repo. The short version: SQLite doesn't love it when two processes both try to write to the same database file aggressively and Hermes's CLI plus its gateway both want to update the session DB any time you talk to the agent. Most of the time SQLite's WAL (write-ahead log) mode handles this fine. Sometimes it doesn't and the agent disables session search for the rest of that process's lifetime to avoid corruption.
This guide covers what's happening, the WAL settings that prevent the issue from recurring and the recovery steps if your state.db has already gotten into a wedged state.
What the error means
SQLite operates in two journaling modes for write durability. The default mode (rollback journal) lets one writer hold an exclusive lock on the entire database while writing. Other writers block. Other readers also block, sometimes. WAL mode (write-ahead log) lets readers proceed without blocking writers and vice versa, which is much friendlier to multi-process access. Hermes uses WAL mode.
WAL mode handles concurrent reads-with-one-writer cleanly. What it doesn't handle is two processes both trying to write at the same time on a slow disk or under unusual filesystem conditions. The second writer gets "database is locked" until the first one finishes, with a configurable timeout. Past the timeout, the second writer gives up and the application has to handle the failure.
Hermes's failure handler, in current versions, is to disable session_search for the rest of that process's lifetime. The reasoning is that if the database is locked, the agent shouldn't keep retrying the same query and risk making it worse. The user-visible effect: the agent stops being able to search past conversations until you restart it.
When the issue fires
Three situations correlate strongly with this error in real installs.
First, running the interactive CLI (hermes in another shell) at the same time as the gateway service. Both processes write to state.db; under load, the second one to start an active write contends with the first.
Second, running the gateway under Docker with the database file mounted from the host filesystem. The 9p / virtiofs layer some hypervisors use for host-to-container file mounts has slower fsync performance than native ext4 and SQLite's lock-hold times go up enough that contention starts mattering.
Third, running on a VPS with very slow disk (cheap shared storage on legacy providers), where individual SQLite operations take long enough that contention is more likely.
If you've never seen the error, you probably won't. If you've seen it once, you'll likely see it again, because the conditions that trigger it tend to recur.
The configuration that prevents it
Three SQLite pragmas tune Hermes's tolerance for contention. They go in ~/.hermes/config.yaml under the state_db section:
state_db:
journal_mode: WAL
synchronous: NORMAL
busy_timeout_ms: 10000
wal_autocheckpoint: 1000
cache_size_kb: 65536
mmap_size_mb: 256What each does:
journal_mode: WAL is the default; calling it out so you can confirm it's set. If yours says anything else, set it explicitly to WAL.
synchronous: NORMAL trades a tiny bit of durability for much better write throughput under contention. NORMAL ensures the WAL is flushed to disk on commit but doesn't force the main DB file's checkpoint to be durable until a checkpoint runs. The risk is losing the last few seconds of writes on a sudden power failure; for an agent's session DB this is acceptable. The default in some Hermes versions is FULL, which is safer but slower.
busy_timeout_ms: 10000 tells SQLite to wait up to 10 seconds for a contended lock before giving up. The default in older Hermes versions is much shorter (1 second), which under load is too aggressive; bumping to 10 seconds lets transient contention resolve naturally instead of triggering the failure handler.
wal_autocheckpoint: 1000 runs a WAL checkpoint after 1000 pages have been written to the WAL. Frequent checkpoints keep the WAL file from growing unboundedly, which improves long-running write performance. Default is 1000; setting it explicitly so you know what you have.
cache_size_kb: 65536 gives SQLite a 64 MB page cache. The default is 2 MB, which is fine for tiny databases but starves a state.db that's accumulated months of history. Bumping to 64 MB significantly improves common query patterns.
mmap_size_mb: 256 enables memory-mapped I/O for the first 256 MB of the database. SQLite memory-maps reads, which is faster than syscall-per-block. For a state.db under 256 MB this means most reads hit RAM directly.
Apply by editing the config and restarting the gateway:
vim ~/.hermes/config.yaml
sudo systemctl restart hermesThe settings take effect on the next process start; existing connections aren't reconfigured mid-flight.
Recovery if state.db is currently wedged
If you're seeing the error right now and the agent isn't recovering on its own, three steps usually unstick it.
Stop everything that talks to state.db:
sudo systemctl stop hermes
# Also exit any interactive hermes session you have open elsewhereWait 5 seconds for any in-flight transactions to time out. Then check for stale lock files:
ls -la ~/.hermes/state.db*You should see state.db plus state.db-wal plus state.db-shm. If you see additional files like state.db-journal (a rollback-journal artefact) it suggests something tried to fall back from WAL mode at some point; safe to remove if no Hermes process is running:
rm -f ~/.hermes/state.db-journalTest that the DB opens cleanly:
sqlite3 ~/.hermes/state.db "PRAGMA integrity_check;"If integrity_check returns "ok", the database is healthy. If it returns errors, you have actual corruption (rare, but it happens after improper shutdowns); restore from your latest backup per the backup guide.
Restart Hermes:
sudo systemctl start hermes
journalctl -u hermes -fThe agent should start cleanly. Send a test message; verify session search works again with /search <some-old-keyword> in the interactive CLI.
If the issue recurs after fixing settings
The settings above resolve the most common cases. If you keep hitting the error, the underlying problem is likely environmental. Two patterns I've seen.
State.db on a slow filesystem. If your ~/.hermes/ is on a network mount, an old spinning disk or a heavily-shared VPS storage tier, every fsync takes long enough that even short transactions hold locks longer than they should. The fix is to move state.db to faster storage. NVMe is ideal; even cheap local SSD is much better than network storage.
State.db on a Docker bind mount with virtiofs or 9p. Switch to a Docker named volume instead of a bind mount; named volumes use the host's native filesystem and don't have the virtualisation overhead.
volumes:
- hermes-data:/root/.hermes # Named volume: fast
# NOT: - ./data:/root/.hermes # Bind mount: slowerIf you really need the bind mount (say you want to inspect files from the host), keep the database itself on a named volume and bind-mount only the specific subdirectories you care about. The Docker Compose article covers the pattern.
Fast-path checks before deep debugging
If you're hitting the error sporadically, run through these before changing config.
df ~/.hermes shows free disk space on the partition holding state.db. If it's above 95 percent full, SQLite slows down dramatically and contention becomes more likely. Free up space.
ls -la ~/.hermes/state.db ~/.hermes/state.db-wal shows the file sizes. If the WAL file is huge (over 100 MB on a personal install), the auto-checkpoint isn't running. Run a manual checkpoint:
sqlite3 ~/.hermes/state.db "PRAGMA wal_checkpoint(TRUNCATE);"Or simpler, restart Hermes; the next clean shutdown checkpoints automatically.
iostat -x 5 5 shows disk I/O stats over 25 seconds. If %util is consistently above 80, your disk is saturated by something. Identify the culprit (could be Hermes, could be something else on the box) and either move it off or upgrade to faster storage.
Long-term: separate read and write paths
For a heavy-traffic Hermes install, the cleanest pattern is to give the gateway sole write access to state.db and have any other tooling (manual queries, monitoring scripts) connect read-only.
SQLite's read-only mode is pragma-set on connection. For sqlite3 CLI:
sqlite3 -readonly ~/.hermes/state.db "SELECT COUNT(*) FROM messages"For Python applications:
conn = sqlite3.connect("file:state.db?mode=ro", uri=True)Read-only connections never contend with writers. If your monitoring or analytics scripts use this mode, they can query state.db continuously without ever causing a busy-timeout for the gateway's writes.
The OpenClaw equivalent
OpenClaw uses a similar SQLite-backed memory store and can hit similar issues. The OpenClaw troubleshooting guide covers the OpenClaw-specific recovery steps; the underlying SQLite tuning is the same.
The 1-click route
The LumaDock Hermes Agent VPS template uses NVMe local storage by default, which is fast enough that the database-locked error is rare under normal usage. The settings above are baked into the template's default config, so you get them without the find-and-fix exercise.
