Monitoring Disk and Memory Usage on Flow nodes

Node Operations
1

Dear Node Operators,

We want to address an ongoing issue regarding disk and memory usage for Flow nodes. As highlighted in our previous communication, it is critical to monitor the disk and memory usage for all node types (Consensus, Collection, Verification, Execution, and Access nodes). We want to provide further guidance on mitigating the impact of the issue.

Root Cause

The root cause of this issue lies in the compaction process run by the Badger database used in the Flow node software. This process has been known to cause:

  1. Spikes in memory usage: These spikes can lead to the node going Out of Memory (OOM).
  2. Spikes in disk usage: These spikes can cause nodes to reach their disk capacity limits.

We are actively working on a long-term solution to migrate to the Pebble database, which is expected to resolve these issues. However, this transition will take time to complete.

Immediate Steps for Node Operators

In the meantime, we request all node operators to follow these recommendations:

  1. Monitor Disk and Memory Usage:
  • Regularly monitor both disk and memory usage across all node types.
  1. Adjust Memory Limits:
  • If you observe intermittent memory spikes approaching the maximum memory capacity of your node, adjust the memory limit for Go using the GOMEMLIMIT environment variable. The recommended setting is 70% of the total memory. So for example for a consensus node running with 16GB of memory, the limit can be set to 12 GB as GOMEMLIMIT=12GiB. Please adjust this value as per your specific node configuration.
  1. Provision Additional Disk Space:
  • If you notice disk usage spikes nearing the node’s maximum disk capacity, ensure additional disk space is provisioned to prevent any interruptions.

Conclusion

We deeply appreciate your ongoing commitment and efforts in running Flow nodes. Your cooperation is crucial in maintaining the stability and performance of the network.

Should you have any questions or need further assistance, please don’t hesitate to reach out on Discord.

Thank you for your understanding and support.

Best regards,
The Flow Team

Issues related to this:

1 Like