On the evening of September 22, a user sent an ATOM transfer.
After a night passed, the transaction still remained "awaiting confirmation."
The private key was not lost, and the wallet showed no signing anomalies. When checked again the next day, multiple public RPCs all showed that Cosmos Hub had stopped at block height 33,086,740.
With no new blocks being produced, naturally there was nowhere that could package this transaction.
Only about a day later, after Cosmos Hub resumed producing blocks, did this ATOM transfer, which had been in a waiting state all along, finally succeed.
For ordinary users, this may be the most intuitive lesson for understanding blockchain consensus.
We are used to saying that "no central institution can shut down a public chain," but reality is clearly much more complex. A sufficiently decentralized blockchain indeed usually does not have that "shutdown button" in a server room, but it can still stop.
This pause of Cosmos Hub happened to fully expose this set of mechanisms, usually hidden at the underlying layer, to ordinary users.
1. Why did Cosmos suddenly "stop producing blocks"?
First, it is necessary to clarify a question that is easy to confuse: what was directly attacked this time was not Cosmos Hub.
The incident first occurred on Neutron.
On September 22, a Neutron governance proposal called "AIATO: AI Agent Takeover" was passed. The attacker exploited a loophole in chain-level governance permissions and used privileged instructions natively provided by the wasmd framework to change the contract administrators of applications such as Astroport and Drop to addresses controlled by the attacker.
This is not what we usually understand as a "code vulnerability" or "protocol flaw."
It can be simply understood as: the applications themselves have their own "door locks," but Neutron's chain-level governance also holds a higher-privilege "master key," and when the attacker controls the governance outcome, it is equivalent to obtaining this key, allowing them to reassign administrators, migrate contracts, and further transfer the assets within.
What truly dragged Cosmos Hub in was the cross-chain fund transfer that followed.
Cosmos Labs' postmortem shows that before Neutron stopped operating, the attacker had already moved some assets to multiple networks, among which about 1.7 million ATOM were transferred into Cosmos Hub and began to be exchanged through cross-chain liquidity.
In other words, Cosmos Hub itself was not directly attacked, and ordinary Hub users' funds were not directly stolen because of the Neutron vulnerability.
But the ATOM obtained from the attack had already entered the Hub, and in order to prevent the remaining ATOM from continuing to flow out, some Cosmos Hub validators began to stop running their nodes.
By around 19:18 on September 22 (SGT), the validators that had stopped running already represented more than one-third of total voting power, so Cosmos Hub could no longer continue forming new blocks and finally stopped at 33,086,740.
This step is very critical.
It means that Cosmos Hub does not have a "Pause" button that some company can directly click, nor did it first go through an on-chain governance vote. What truly stopped the network was that enough validators no longer participated in forming consensus.
But what is more noteworthy is actually the recovery process afterward.
About 4 hours after the chain stopped, validators received a complete recovery plan: execute a one-time state modification at the chain-stopping height, transferring the remaining ATOM in the attacker's address to a multisig address jointly managed by community validators.
Cosmos Labs then produced the Gaia v28.3.0 patch based on the plan already agreed upon by validators, tested it, and distributed it to validators.
This version of Gaia would execute a one-time state change at the specified recovery height, transferring 1,227,121 ATOM from the attacker's address to a 4-of-6 multisig address composed of six parties: Nansen, Keplr, Enigma, Silknodes, Kiln, and Polkachu.
By the early morning of September 23, validators confirmed to have installed v28.3.0 already exceeded 67% of total voting power, so at 12:00 UTC that day, Cosmos Hub coordinated a restart. About 6 minutes later, this one-time state modification was executed at block height 33,086,741, and the network resumed normal block production.
In the final analysis, throughout the entire process from Cosmos stopping block production to resuming operation, validators first caused the network to lose Liveness, and then more than two-thirds of voting power accepted a new set of state transition rules, ultimately making this set of rules the canonical state after recovery.
At this point, a seemingly simple question arises: since it is a decentralized public chain, why can more than one-third of validation power stop it, while restoring the network requires enough validators to jointly accept and run the same software?
The answer is actually hidden in the word "consensus."
2. So-called consensus was never "never stops"
One of the things most easily misunderstood about blockchain is equating "decentralization" with "never goes down."
In fact, what the consensus mechanism truly solves is how, without a central bookkeeper, many nodes can reach agreement on transaction order and ledger state.
However, different public chains do not implement this in the same way.
For example, Bitcoin's most classic mechanism is PoW, proof of work—miners compete to produce blocks using computing power. When two valid branches appear on the network for a short time, nodes choose one of them to continue building on according to cumulative work.
So Bitcoin does not have a clear moment when "after 67% voting, this block is forever Finalized." It is closer to a kind of probabilistic finality: the more subsequent blocks there are, the more computing power cost is required to reorganize and remove earlier transactions.
This is also why people used to say that a Bitcoin transaction is best waited for 6 block confirmations. After all, even with higher computing power, one cannot simply bypass the consensus rules that nodes are executing.
Of course, this does not mean that Bitcoin's state is "absolutely impossible to modify" under any circumstances. Theoretically, if the entire ecosystem accepts a new client and new consensus rules, through a Hard Fork, it is likewise possible to make state changes that were invalid under the old rules become valid.
But here lies the problem: who has the ability to make enough miners, Full Nodes, trading platforms, wallets, and users jointly accept such a new set of rules?
Almost none.
The development team cannot decide the consensus rules for the entire Bitcoin network on its own, and it is also difficult for miners and trading platforms, because the consensus threshold that needs to be crossed is very high. When Binance was hacked for 7,000 BTC, some people suggested that CZ contact large miners to operate, but in the end nothing came of it.
Ethereum provides another very classic example.
After switching to PoS, Ethereum now uses the Gasper consensus composed of Casper FFG and LMD-GHOST together. To put it simply, one part of the mechanism is responsible for determining "which chain should currently be followed," while the other part is responsible for giving blocks true Finality.
Only when validators representing at least two-thirds of staked ETH agree on the corresponding checkpoint can a block move further toward finalization; conversely, if more than one-third of the stake does not participate in correct voting for a long time, the network may temporarily be unable to form Finality. However, Ethereum also designed an inactivity leak, which gradually reduces the effective weight of offline validators when finalizing cannot be achieved for a long time, giving the network a chance to eventually restore Finality.
To truly change this outcome, it is likewise necessary to change the protocol rules and clients.
Just as in the 2016 The DAO incident, the Ethereum community ultimately carried out a Hard Fork, executing at block 1,920,000 a special state modification that the Ethereum Foundation at the time directly called an irregular state change, transferring the relevant ETH into a recovery contract.
However, some miners and community members who refused to upgrade and continued to maintain the original state eventually formed Ethereum Classic (ETC), leading to the well-known ETH and ETC fork, showing that not everyone accepted this set of rules.
Cosmos Hub is different again. It uses CometBFT, which is closer to typical BFT consensus.
It can be understood as a more typical BFT consensus, meaning that for a block to truly be committed, it needs to obtain a Commit from more than two-thirds of the voting power.
Its advantage is that Finality is very clear. Once a block has been committed after voting by sufficient verification power, there is no need to continue waiting for more and more blocks as with PoW, exchanging probability for a sense of security.
But its other side is also very direct: if one-third or more of the voting power no longer provides the votes needed to form a Commit, then no matter how hard the remaining validators try, they cannot muster more than two-thirds.
At this point, the safest choice for the network is exactly the "pause in block production" seen this time, so from the perspective of distributed systems, this brief halt of Cosmos Hub is actually not mysterious.
In a word, after a group of validators holding sufficient voting power stops participating, the consensus protocol, according to its own rules, would rather lose availability than continue confirming new blocks without sufficient consensus.
Behind this actually correspond two concepts in distributed systems that are often confused by ordinary users:
- Safety: different nodes must not simultaneously confirm two conflicting final states;
- Liveness: whether the network can still continue running forward and process new transactions;
For BFT systems, when there are insufficient nodes participating in consensus, pausing is sometimes precisely the price paid to maintain Safety. To put it bluntly, this decentralized ledger would rather stop there first than let the remaining people each keep their own records.
Looking back from this angle, one will find that many seemingly completely different incidents in the history of public blockchains actually revolve around the same thing:
When distributed nodes can no longer form a consensus on the "correct state," what should the network do?
III. From Bitcoin to Solana, where is the real risk boundary of public blockchains?
This is not the first time Cosmos has put this question on the table.
As early as 2013, Bitcoin experienced a very classic chain fork incident.
At that time, Bitcoin 0.8 switched its underlying database from Berkeley DB to LevelDB. Subsequently, a block containing a large number of transaction inputs appeared. New version nodes could process it normally, but some old version nodes, due to the lock count limit of Berkeley DB, judged this block as invalid.
Thus a very awkward scene appeared: everyone was running Bitcoin, but the old and new clients began to give different answers to "whether this block is legal or not."
The network therefore split into two chains, and the new version 0.8 side once had about 60% of the hash rate, and could not rely on normal hash rate competition to quickly converge on its own.
In the end, large mining pools coordinated to switch back to the old version, regained more hash rate on the side of the old rules, and only then did the network converge again. Bitcoin later specifically reviewed this incident with BIP 50.
By 2016, Ethereum's The DAO incident pushed the problem one step further.
As mentioned above regarding The DAO incident, the Ethereum community ultimately carried out a Hard Fork, executing at block 1,920,000 a special state modification explicitly called an irregular state change by the Ethereum Foundation, transferring the relevant ETH into a recovery contract.
But not everyone agreed with this handling. Some miners and community members who refused to accept the state modification continued to maintain the original rules, which led to the long-existing Ethereum Classic (ETC).
This DAO Fork was also a classic event, equivalent to telling everyone that when extreme events occur, besides code consensus there is also social consensus. If sufficiently consistent opinions cannot be formed, a chain really can split into two.
Solana in 2021 demonstrated another completely different failure path.
In September of that year, a large number of bot transactions flooded the network, causing validator nodes to run out of memory and many nodes to crash. Eventually the entire network could not form a consensus on the current state and stopped confirming new blocks for about 17 hours, after which validators jointly coordinated to restore the network.
Putting these incidents together, one will find that they are not the same thing:
- The problem with Bitcoin in 2013 was that different clients began to enforce different validity rules;
- The problem with Solana in 2021 was that a large number of validator nodes could no longer continue to participate normally in consensus, and the network lost Liveness;
- What Ethereum DAO faced was closer to the question of whether a community should actively modify state through new protocol rules;
- And this time Cosmos Hub has another layer of particularity: the network first actively lost Liveness through validator coordination to prevent attacked assets from continuing to move; afterward, a sufficiently high proportion of voting power jointly accepted the new software and recovery state, allowing the network to form consensus again;
So, rather than simply reducing these events to "so blockchains can actually shut down" or "decentralization is all fake," it is better to admit a more real fact:
The consensus mechanism has never been a machine that cannot break. What it truly provides is actually a set of decentralized rules, such as who decides the correct chain when disagreements arise; how many participants are needed for a state to obtain finality; whether the network chooses to continue running or stop when failures occur; and under extreme circumstances, what kind of collective action can change the rules of operation going forward.
This also leaves this Cosmos incident with a question more worth pondering for ordinary users than "whether the chain should be stopped."
Final Thoughts
We often say, not your keys, not your coins.
This statement of course still holds true, it's just that it emphasizes asset control—as long as the private key is in your own hands, wallets, trading platforms, or other third parties cannot sign a transfer on your behalf.
The premise is that the blockchain you are on must be capable of processing that signature at any time.
On the day the Cosmos Hub stopped producing blocks, users still held their own private keys, and the assets did not disappear into thin air because of it, it's just that even if you correctly signed a transaction, there was no new block to accept it.
The recovery process further illustrates that if enough consensus participants accept a new set of state rules, the on-chain state of specific accounts may also change without being signed by the private key of the original address.
This does not invalidate "Not your keys, not your coins," but it reminds us that private key sovereignty and underlying consensus power have never been the same thing.
And for wallets, the same is true.
Wallets can ensure that private keys and signing rights are in users' own hands, can identify chain-level anomalies as quickly as possible, accurately display transaction status, establish RPC and node redundancy, and reconfirm the final result of transactions after the network recovers.
But wallets cannot restore consensus for a public chain, nor can they guarantee that the underlying network will never be interrupted, much less guarantee that the rules and state on the chain will never undergo changes at the consensus level.
So, what a mature decentralized system truly needs to pursue may never have been "nothing can ever be changed"; on the contrary, it should clarify these imperfect boundaries as much as possible: Who can pause consensus? How much weight is needed? Under what circumstances is emergency intervention allowed?
Because true decentralization cannot make the system never encounter accidents; the key is that even if an accident really happens, we can still know who, based on what rules, and with how much consensus, decided how this ledger should be recorded next.
