In Celer, we prioritize security with a comprehensive approach to protocol, code, and defense-in-depth operational security. A coding vulnerability in the Proof-of-Stake (PoS) security mode of Celer’s State Guardian Network (SGN) was brought to our attention by fellow builders at Jump Crypto. This issue has been addressed and all SGN validators have upgraded their software. No user funds were affected.
Was there any impact?
There was no impact on the Celer ecosystem. Specifically:
- No funds were lost.
- No malicious cross-chain messages were passed.
- The vulnerability was not publicly accessible, and no funds were at immediate risk at the time of discovery.
How could this weakness have been used?
- To take advantage of this weakness, someone with malicious intent would have had to compromise one validator in the Celer SGN.
- The weakness itself would allow the compromised validator to bypass the quorum stake check of the SGN and therefore compromise the PoS security mode of Celer. The result would have been that a malicious cross-chain message could have been generated at the SGN’s PoS consensus layer.
- Thankfully, Celer’s comprehensive approach to security with multiple lines of defense meant that, even in the worst-case scenario, any potential loss would have been bounded. This would not have been a “one transaction drains all funds” type of risk.
How would Celer’s comprehensive security approach be effective at defending against such an attack if it had happened?
- Celer is built based on the “defense-in-depth” principle. Instead of assuming all of our code and software is flawless, we assume that every piece of software can have bugs and build multiple lines of defensive measures to cross-examine the system state and actively protect funds in case of any issues. In this particular case, the following defenses were in place as defense mechanisms:
- Defense 1: Inconsistent event monitoring alerts and emergency shutdowns
- When an SGN validator detects that it cannot find the proposed message by another validator on the designated source chain, it will output an error log message. If both Celer-operated validators, as part of a security council, generate this error log, this error will trigger security sentinel software to pause all contracts.
- From the time a malicious message is proposed to its finalization, there is about a 1-minute delay. By the time any single malicious message with all validator signatures is generated, most of the contracts would have already been paused.
- Defense 2: Rate-limiting and app guardians
- Even if some malicious messages slipped past the first line of defense, rigorous rate limiting, circuit breakers, and transaction-delay thresholds are built into the smart contracts. This defense explicitly assumes that the consensus layer can fail. The blackhat would have had to break down the transactions into smaller sizes and would not have been able to drain all of the protocol-locked funds in a short period of time.
- In addition, if any malicious transaction hits the blockchain, app guardians and security sentinel software will see a violation of transaction invariant (funds going out without matching funds going in) and pause all contracts.
- Defense 3: Optimistic-delay-like security model
- In the case of cross-chain messaging use cases, many of them utilize the optimistic-delay-like security model with the two-phase commit process in addition to the SGN’s PoS security. Therefore, even if the SGN consensus was compromised, malicious messages would have been stopped as long as there was still one honest app guardian live.
- In addition to in-protocol defenses, we have partnered with active security operation providers to enhance comprehensive on-chain and off-chain monitoring and anomaly detection.
- Defense 1: Inconsistent event monitoring alerts and emergency shutdowns
How Celer DAO is continuously improving protocol security practices and mechanisms:
- Prioritizing a yearly refresh of security audits with additional security audit partners (expected to span between Q2 and Q3 2023)
- Expanding existing bug bounty programs to include the SGN codebase (expected Q2 2023)
- Continuously investing in defense-in-depth practices.
Finally, we thank Jump Crypto again for the responsible disclosure of this issue. Though the discovery is not covered by the existing bug bounty programs, we plan to raise a community proposal to grant the Jump Crypto team a retrospective bounty reward once we include the SGN codebase in the bug bounty programs in the coming months. Onward and forward.