The Ethereum network experienced a technical issue May 11 and 12, which was attributed to an "exceptional scenario."
The beacon chain encountered two separate instances on the aforementioned dates where finality could not be attained for three and eight epochs. For around 25 minutes on Thursday, the Ethereum mainnet was unable to fully confirm transactions; blocks were proposed but not finalized.
We can start putting this loss of finality issue behind us, @Teku_ConsenSys and @prylabs have deployed fixes that will prevent the attestation flooding. This is one step on our diversity & decentralization journey, let's learn from it and move forward with greater purpose. pic.twitter.com/cSRgPTWeuy— superphiz.eth 🦇🔊🛡️ (@superphiz) May 13, 2023
The same issue, in which the ETH network could not finalize blocks for more than an hour, was reported by Wu Blockchain on Friday. Wu added that, as a result, dYdX momentarily paused deposits.
In both scenarios, the client teams noted that although the ETH network was unable to finalize, users were able to transact on the network thanks to its design.
Due to client diversity, not all client implementations were impacted by the exceptional scenario, which made this conceivable. Once every client had caught up, the network was finalized once more.
The client teams provided a new update on the incident, saying that while the full cause is still being investigated, the seeming cause was linked to heavy load on some of the consensus layer clients, which in turn was caused by an exceptional scenario.
To forestall future occurrences, Prysm Ethereum Client, a core implementation of the Ethereum protocol, and Teku, a full Ethereum 2.0 client, have both published updates that implement optimizations to stop beacon nodes from experiencing high resource usage during these exceptional scenarios.
Prysm patch for mainnet incidents is out. 🚨— terence.eth (@terencechain) May 13, 2023
Recommend upgrading if your node is under heavy stress! 🙏 https://t.co/rFUOwJYNka
Prysm announced the v4.0.3-hotfix, which contains the necessary optimization to prevent the beacon node from experiencing high resource usage during turbulent times, as seen on May 11 and May 12.
Teku, on the other hand, released Teku v23.5.0, which filters old attestations that were observed to cause problems on the mainnet.