With blockchains promising to become "world computers" upon which anyone can create projects ranging from mere toys to critical infrastructure, uptime is crucial, and a five-hour-long outage is devastating.
One "coordinated restart" apparently wasn't enough, because a second one followed later that day. Developers reportedly didn't know why the blockchain suddenly began to slow, though it followed shortly after validators began adopting a new version of Solana code, pointing to a possible culprit in the new release. The new version had reportedly operated for six months on the testnet before it began to be deployed.
Other theories were also considered, as reported by CoinDesk: "One leading theory was that a 'fat block' gunked up the blockchain's mechanics."
The outage is reminiscent of the ones that plagued the network through 2022, leading some to question whether it could be suitable for replacing critical infrastructure.
A validator operator reported that "It appears a misconfigured node caused an unrecoverable partition in the network." It's a bit startling that, in a supposedly decentralized network, one single node can bring the entire network offline.
OptiFi has promised to return user deposits and settle all positions. In a post-mortem, they wrote that they had learned that "Every deployment needs a rigorous process and single point failure can be avoided. Please don't rush like what we did, especially for defi projects". They further outlined a "peer-surveillance approach" in which three people would be required to deploy any changes together. They also asked the Solana team to implement a two-step confirmation for such a potentially destructive command.
However, Solana has been plagued with stability issues, and on June 1 it was taken offline by its developers for what CryptoWhale says was the eighth time this year. This occurred only days after an incident in which the Solana blockchain clock drifted significantly behind real-world time.
Blockchain timekeeping is also selling point of Solana, which talks up its "proof of history" algorithm in a blog post where Solana Labs co-founder Anatoly Yakovenko says, "our clocks never drift".
This is hardly the first instability the network has demonstrated, much to the chagrin of its users. Transaction flooding is an issue on Solana in part because of the low transaction fees compared to networks like Bitcoin and Ethereum, which have relatively high gas fees that would make flooding extremely expensive.