Please monitor your node system resources

Node Operations
1

Hello Flow Node operators,

As you must have noticed, Flow now undergoes far fewer network upgrades (also known as spork) that incur network downtime. For reference, here is a table that lists the total number of network upgrades by year,

Year Total number of network upgrades
2020 11
2021 11
2022 5
2023 4
2024 1 (Crescendo launch planned for Q3)

This year (2024), there will only be one network upgrade for the Flow Crescendo launch.

Note: Height Coordinated Upgrades (HCU), which do not incur any network downtime, will continue to occur approximately once every month to ensure the latest node software changes are deployed to testnet and mainnet.

Fewer network upgrades have translated to a network SLA number of 99.9% for 2023 and more than 99.9% YTD (see dashboard). Moreover, it also translated to lesser time commitment from all node operators who have to take action during each network upgrade.

However, this also means that this is the longest the nodes have run without deleting any data - a step which is done during a network upgrade. Many nodes have now accumulated chain data since the last network upgrade in November last year.

Hence, to ensure the smooth operation of the network till the next network upgrade, we request you do the following (for all node types):

  1. Monitor all system resources - CPU, memory and disk on an ongoing basis and set alerts when they cross a threshold of 90%.

  2. Provision the nodes with enough memory and disk if you see the threshold being crossed. Please see this forum post for an open issue that was identified recently.

  3. Reclaim disk space if required by setting up the node for dynamic bootstrapping, see this page for instructions. Reclaiming disk space should help reduce memory usage and the potential of the node going out-of-memory (OOM error).

    Note: Dynamic bootrsapping on an access node restricts the chain data that the node can serve to the height at which the access node is boostrapped (see limitations of dynamic bootstraping an access node)

Please reach out on the #flow-validators discord channel if you have any questions.

Thank you,
Flow Foundation

3 Likes