visor_abci_state.json errors on a Hyperliquid node

visor_abci_state.json is a small file that points the Hyperliquid node at its saved state. If it is missing on a first start, the error is normal and the node downloads state from a peer. After a crash it means the pointer was lost: restore it from a backup and the node loads local state in minutes instead of downloading it again.

The short version#

• First start: the error is expected. Wait 10 to 30 minutes. Do not restart.
• After a crash: the pointer is gone, so the visor goes to peers. Restore a recent copy before you start the node.
• Prevention: copy the pointer every minute while it is valid.

1. The error#

The supervisor logs one of these, as in hyperliquid-dex/node issue #132:

could not load VisorAbciState after retrying: async_sleep_retry retries exhausted: AsyncSleepRetry::retry desc=[load_visor_abci_state] n_tries=2 failed: missing file: /root/hl/hyperliquid_data/visor_abci_state.json

error loading /root/hl/hyperliquid_data/visor_abci_state.json: missing file: /root/hl/hyperliquid_data/visor_abci_state.json

The node side says the same thing:

hl-node @@ using local AbciState: missing

2. What the file is#

The visor loads state from data/periodic_abci_states/<date>/<height>.rmp on a restart. The pointer tells it that local state exists. This is the file on our node on 9 November 2026:

{
  "hardfork_version": 111,
  "initial_height": 1212283000,
  "height": 1214802259,
  "scheduled_freeze_height": null,
  "consensus_time": "2026-11-09T09:49:15.285818437",
  "wall_clock_time": "2026-11-09T09:49:15.733512198"
}

• The pointer is about 230 bytes. The saved state is about 2 GB: 1,976,882,534 bytes at height 1214780000 on the same node.
• The node writes a new .rmp every 10000 blocks. On our node that was about every 12 minutes.
• The gap between wall_clock_time and consensus_time is a quick lag check: 0.46 seconds in this sample.

ls -l ~/hl/hyperliquid_data/visor_abci_state.json
ls ~/hl/data/periodic_abci_states/$(date -u +%Y%m%d)/ | tail -3

3. On a first start#

A new node has no state on disk, so the file cannot exist. The visor retries, gives up, and the node asks a peer. In issue #132 the log shows the download starting a few seconds later:

hl-node @@ gossip_server_connect_to_peer connected to abci stream from <ip>:4001

Leave it alone for 10 to 30 minutes. Each restart starts the download again. See first sync in the setup guide.

4. After a crash#

An unclean death of the node process can take the pointer with it, even though a state snapshot at most about 12 minutes old is on disk. The visor then ignores local state and downloads about 1.4 GB from peers. We lost 75 minutes to this once and had three cold starts in one afternoon on 15 October 2026.

The fix is to put a recent copy back before the visor starts. Order matters. On our node the pointer went missing after systemctl stop had already returned, because the old processes were still exiting. So: stop, wait until they are gone, restore, start.

DATA=/root/hl/hyperliquid_data
BAK="$DATA/visor_abci_state.json.guard-backup"

systemctl stop hlnode
# wait until the supervisor and the node have exited, then:

AGE=$(( $(date +%s) - $(stat -c %Y "$BAK") ))
RMP=$(find /root/hl/data/periodic_abci_states -name '*.rmp' -newermt '-2 hours' | sort | tail -1)
# only when the pointer is missing or invalid, never over a good one
if ! python3 -m json.tool "$DATA/visor_abci_state.json" >/dev/null 2>&1 \
  && [ "$AGE" -le 1800 ] && [ -n "$RMP" ]; then
  cp "$BAK" "$DATA/visor_abci_state.json"
fi

systemctl start hlnode

• The 30 minute limit is on the backup file. A backup that old means the job that writes it stopped.
• A restore needs a recent .rmp on disk. Without one the visor has nothing local to load.
• The visor re-reads a missing pointer every 2 seconds for about 6 seconds before it goes to peers. Restore a second time inside that window if a process deleted it again.

On our node a restored pointer gave a local load and current fills in about 3 minutes, with no peer download. The log line to look for is linked abci state desc=periodic.

5. When the peer download fails#

If there is no pointer and no usable local state, the node has to download. These lines are the download failing, and each is a different peer:

could not read abci state from <ip>: early eof
abci_stream from <ip> timed out: lu::timeout deadline has elapsed
Peer full

• Open TCP 4001 and 4002 to the internet. Peers deprioritize nodes they cannot reach.
• Refresh the root peer list and keep peers close to you. The transfer is about 1.4 GB, and reports in #143 and #181 show a 60 second deadline on it, so a far peer cannot finish in time.
• Do not restart while it retries. The node moves to other peers on its own.
• After a network upgrade many nodes download at once and Peer full is common. Issue #179 shows a node stuck for days this way.

6. Keep restarts local#

Copy the pointer once a minute, only while it is valid and its height has not gone backwards:

#!/usr/bin/env bash
# run from cron every minute
set -eu
DATA=/root/hl/hyperliquid_data
PTR="$DATA/visor_abci_state.json"
BAK="$PTR.guard-backup"
height() { python3 -c "import json,sys; print(json.load(open(sys.argv[1]))['height'])" "$1" 2>/dev/null || echo ""; }

H=$(height "$PTR")
[ -n "$H" ] || exit 0          # absent or invalid: nothing to back up
PREV=$(height "$BAK")
if [ -z "$PREV" ] || [ "$H" -ge "$PREV" ]; then
  cp "$PTR" "$BAK.tmp" 2>/dev/null || { rm -f "$BAK.tmp"; exit 0; }
  mv "$BAK.tmp" "$BAK"
fi

• Keep the copy next to the file, so it survives a cleanup job that only touches data/.
• A freshly started node briefly has no pointer: the visor consumes it at boot and writes a new one after its first save. The exit 0 keeps the old backup during that window.
• Do not let disk cleanup delete the snapshot the pointer needs. hyperliquid-node-kit's cleanup skips it.

Questions#

Related guides#

• How to run a Hyperliquid non-validator node
• Hyperliquid node falling behind: fix a slow or far gossip peer
• Hyperliquid node crash after the October 2026 hardfork
• Hyperliquid node disk filling up: prune EVM checkpoints

References#

• hyperliquid-dex/node issue #132. The VisorAbciState log lines on a first start.
• hyperliquid-dex/node issue #143. State transfer of 829 MB against a 60 second timeout.
• hyperliquid-dex/node issue #181. Asks to drop the fixed 60 second deadline.
• hyperliquid-dex/node issue #179. Stale local state and Peer full on every download.
• hyperliquid-dex/node issue #187. A full state reload of about 1.37 GB after a short lag.
• ironflowsh/hyperliquid-node-kit. Open-source scripts, including a cleanup that keeps the state the pointer needs.