Flow mainnet experienced a few issues in the past few days which led to network being unavailable several times intermittently.
Following are the issues that were encountered:
-
There was a new field Execution ID, that was added in the Execution result which an Execution nodes produces. The value of the field was not being calculated consistently by all the execution nodes for transactions which threw an error. This resulted in blocks not being sealed. We have resolved this by setting the Execution ID to be a fixed value for now. This is the commit in the code.
-
Memory usage on all node types have increased. The increased memory usage caused Execution nodes to restart. We have, for now, increased the memory on the Execution nodes and are also in the process of investigating the root cause.
-
A step was missed after the last spork. The step switched network addresses from the old spork to the new spork. When the epoch transition occurred yesterday, the access nodes couldn’t talk to the execution node. Hence if you were using one of our public facing access node, you would have seen errors for some of the API calls. We have resolved that issue by making DNS changes
thank you,
Flow Team