Execution node upgrade to v0.31.15 and Managing Disk Space Usage

Node Operations
1

This post provides step-by-step instructions for Execution node operators to upgrade their Execution Nodes to version v0.31.15, which introduces a new feature to slow down disk space usage growth by storing the chunk data pack in a separate directory and allowing it to be pruned as and when needed.

No action is needed for any other node type except for Execution node

Overview of the New Feature

The latest Execution Node software (docker tag: v0.31.15) introduces a significant enhancement that enables the separation and pruning of the chunk data pack. The execution node provides the verification node with chunk data packs to verify execution. However, the chunk data pack has been observed to consume a very high percentage of the execution node’s disk space, sometimes as high as 85 to 90%.

The new feature allows for safely removing the chunk data pack stored locally on the Execution node disk, resulting in efficient disk space management.

Upgrade Process

  1. Backup: Before upgrading, ensure you have backed up any critical data and configurations related to your Execution Node.

  2. Stop the node.
    systemctl stop flow-execution if using systemctl or just stop the docker instance.

  3. Update Execution node version to docker tag v0.31.15 (or v0.31.15-without_netgo)

  4. Add the new flag chunk-data-pack-dir

  • Locate the command line or configuration file used to start the Execution Node and add the following flag:
    --chunk-data-pack-dir=/data/chunk_data_pack
    This flag specifies the directory where the chunk data pack will be stored. Adjust the directory path to match your system’s file structure if necessary.
    Note: the new flag is mandatory, otherwise the execution node will enter a crashloop.
  1. Start your execution node container
    systemctl restart flow-execution if using systemctl or start the docker container.

The Execution node will now exclusively use the new chunk data pack directory specified by the flag to store chunk data.

Pruning Chunk Data Pack
If the execution node is close to running out of disk space, you can now reclaim space by pruning the chunk data by following these steps:

a. Stop the Execution Node:
b. Remove the Chunk Data Pack Directory:
Run the following command to remove the chunk data pack directory:
rm -rf /var/flow/data/chunk_data_pack
c. Start the Execution Node
Upon restart, the Execution node the chunk data pack directory will be automatically recreated.

Note: Always exercise caution when performing system operations, and make sure you have a backup of important data before making any changes.

The chunk data pack accounts for 85% of the disk usage on an execution node. Hence, deleting the chunk data pack directory in the future will help reduce 85% of the growth in disk usage.
For e.g. lets say the current disk usage is 100TB. Afte after switching to this new version, the disk usage will still start at 100TB, since none of the existing data space will be reclaimed. However, as the disk usage increases, at say 1TB per day, then after one week, the node would have used up an additional 7TB. If you now prune the chunk data pack directory as described above, you should be able to recover 5.95TB (85% x 7TB) data. In other words, the size of /var/flow/data/chunk_data_pack is the disk space that will be reclaimed.

Pruning the data folder does introduce downtime for the node hence do not execute it very frequently but judiciously only when the disk usage is close to being full.

Thank you,
Flow Team