Hyperliquid node falling behind: fix a slow or far gossip peer

A Hyperliquid non-validator that falls behind while the server sits idle usually has a bad stream peer: too far away or too slow to keep up. The node picks the peer it streams blocks from out of its peer list, so a list full of far peers can hold it behind for hours. Ours sat 72 seconds behind the chain for 8 hours on 7 November 2026, because of a peer more than 200 ms away.

The short version#

• Check: compare the last block_time in your fills file with the clock.
• Cause: a slow or distant stream peer. It is rarely CPU.
• Fix: keep near peers in override_gossip_config.json and only a few far ones as fallback.
• Cost: the list is read at start, so changing it needs a restart, and a restart can leave a gap to fill from the archive.

1. Measure the lag#

The process can be running and still be far behind. Every line the node writes has a block_time, so read the last one:

f=$(ls -d ~/hl/data/node_fills_by_block/hourly/*/* | sort -V | tail -1)
tail -n 1 "$f" | grep -o '"block_time":"[^"]*"'
date -u +%Y-%m-%dT%H:%M:%S

The state pointer gives a second reading: the difference between consensus_time and wall_clock_time in hyperliquid_data/visor_abci_state.json. A healthy node shows well under a second. Our alert for a stale node fires at 30 seconds behind for 2 minutes.

2. Why the peer matters#

The node reads blocks from one upstream peer at a time. From the public report in issue #156, its health check looks at whether blocks keep arriving, not at how fast. A peer that delivers about 2 blocks a second, when the chain produces about 12, passes that check indefinitely. In that report the node stayed on it for 48 minutes and fell further behind every minute.

Distance makes it worse. On our node in Europe a peer over 200 ms away, chosen from a list of mostly far root peers, kept the node in a drop and refetch loop 72 seconds behind for 8 hours. CPU was not the limit: an idle server and a growing lag point at the peer.

3. What it looks like#

Lines reported in #156 and #187 when the stream peer goes bad:

client_block_reader error, ending task: lu::timeout nv_stream_msg: deadline has elapsed
forward_client_blocks no more blocks from reader, reconnecting to a new peer
missing client blocks before received client block, bootstrapping from peer
Peer is missing client blocks from round ...

Two more to watch. received old client block, is peer behind? means the peer itself lags the chain. And if the node falls about 90 seconds behind it can panic with peers are still missing client blocks, restarting to fetch a new abci state, which throws away its state and starts a full download (issue #187).

A slow peer can also show nothing in the logs except applied block lines arriving too slowly. Count them against the clock.

4. Fix the peer list#

Start from Hyperliquid's published root peers and keep the ones close to you. The peer refresh script in our open-source node kit does this: it fetches the list, measures the TCP connect time to port 4001 from your server, keeps the fastest 16 and writes the file. It runs once a day and does not restart the node.

curl -s -X POST https://api.hyperliquid.xyz/info \
  -H 'Content-Type: application/json' -d '{"type":"gossipRootIps"}'

• Keep try_new_peers at true so the node can find others if a listed peer dies.
• Our rule for a node in Europe: keep every peer within 40 ms, plus the best few far peers as fallback. In Tokyo there is no cap.
• Measure freshness, not throughput. Ranking peers by catch-up speed sent us to peers that lag the chain tip during US hours.
• Open TCP 4001 and 4002 so peers can reach you.

5. Restarting and the data gap#

The peer list loads at start, and there is no way to force a new stream peer on a running node. A restart is the only lever, so use it deliberately:

• Update the peer file first, then let the next natural restart pick it up when you can wait.
• Before a forced restart, make sure the state pointer is backed up, so the node loads local state. See the visor_abci_state.json guide.
• Expect a hole in your output files. In issue #156 a restart caught up in about 6 minutes and left a 28 minute gap. Fill it from the S3 archive.

Questions#

Related guides#

• How to run a Hyperliquid non-validator node
• visor_abci_state.json errors on a Hyperliquid node
• Hyperliquid node crash after the October 2026 hardfork
• Hyperliquid historical data: API, S3 archive and Python

References#

• hyperliquid-dex/node issue #156. A node pinned to a degraded stream peer, and the cost of a restart.
• hyperliquid-dex/node issue #187. A node about 90 seconds behind panics and reloads state.
• hyperliquid-node-kit: refresh-peers.sh. Peer refresh script used in section 4.
• ironflowsh/hyperliquid-node-kit. Open-source install, cleanup, watchdog and peer refresh.