Yesterday, the bitcoin exchange MtGox – riddled by problems – issued a press release saying the bitcoin protocol was to blame for its ongoing problems. That statement, which caused the markets to nosedive temporarily, is outright false. The problem is, and was, bad code hygiene in the MtGox exchange itself. Here are the details.
Yesterday, when MtGox blamed “transaction malleability” as the cause of MtGox’ problems, implying that the problems at MtGox affected all exchanges and everything bitcoin, that was a sign of a very elastic relationship with facts. It’s true that transaction malleability was a factor, but not nearly in the way that MtGox implied. (We’ll be returning to what the “malleability” is.)
Here’s the real problem: MtGox is running its own homebuilt bitcoin software, and has not cared to update and upgrade that software along with the developments of the bitcoin protocol. Recently, after a very long grace period, the bitcoin protocol tightened slightly in order to disallow unnecessary information in transaction records, and did this to fix the malleability problem that MtGox blamed.
So the problem of malleability remained at MtGox, while having been fixed in the rest of the world. This – the discrepancy itself – was the root cause of the problem, because it meant that MtGox started issuing invalid transaction records for bitcoin withdrawals. Obviously, they were rejected by the bitcoin network.
Let me explain in a bit more detail.
When you write an amount of money, say twenty-three thousand four hundred and twenty-two dollars and fifty-four cents, you typically write that as $23,422.54. But it would also be valid to write it as $0,023,422.54. Or $0,000,023,422.54. This fact – that one number can be written in many ways, all valid – is the malleability. (For the sake of completeness, it wasn’t the amount, but another number in the transaction record that was concerned.)
This was tightened in the bitcoin protocol to only allow the shortest version of writing a number, $23,422.54, in this specific code change, which happened a whole year ago.
This change was ignored by MtGox, if I may speculate, probably because “it kept working anyway”. That is, until bitcoin 0.8, when the core developers decided to enforce this change across the protocol, having had the tightening published for over a year. The moment bitcoin 0.8+ gained majority deployment on the network, such invalid transactions started getting rejected.
In other words, MtGox’ lack of code hygiene and lack of very basic IT release processes led to the MtGox code getting out of sync with the bitcoin protocol itself. It kept writing numbers in a way that wasn’t always the shortest possible way in some of its transaction records, and therefore, the inevitable happened: those transaction records were rejected by the bitcoin network.
As a complete side note, this situation is well described by a saying in Sweden that we use to honor our neighboring Finns and their gung-ho attitude toward life, the universe, and everything. The saying is supposed to be pronounced slowly with a slight sauna-induced slur and a strong Finnish accent, like such:
Now, let’s return to MtGox’ press release. There, they state that skilled hackers had the ability to rewrite bitcoin withdrawals with the speed of lightning before they reached the bitcoin network, implying that hackers changed valid transactions enroute. This, skilled hacking, was the cause of all their problems, they claimed. But that’s not what happened at all. MtGox were creating invalid transaction records for some small but significant portion of their bitcoin withdrawals.
What this means is that MtGox wasn’t the subject of some skilled hacking related to transaction malleability. Instead, bad code hygiene was causing MtGox to broadcast invalid transactions, which could trivially be corrected and re-broadcast, causing all these problems downstream.
This, in turn, leads to all the described problems with double-spending, internal databases of account records getting out of sync with the blockchain records, et cetera. Once somebody has corrected one of MtGox’ malformed transactions and re-broadcast it, MtGox would still consider it unsuccessful, making things go out of sync.
So, is this hard to do the right way? No. I can say that authoritatively – I spent seven years as a CTO-for-hire putting exactly these kinds of hygiene, accountability, trackability, and predictability processes into place at startups with growth pains, saving more than one startup from the blame-game death spiral. MtGox is dying from the lack of a very basic leadership and management toolkit.
Oh, and that Swedish saying about the Finns in the audio clip above? The one that references how the protocol strictness tightened but MtGox went gung-ho ahead anyway? It means “The road turned, but Pekka didn’t”.
The author is personally affected by the MtGox malfunction, having a five-figure dollar amount in stuck unprocessed bitcoin withdrawals.
As a final note, I can’t help feeling a bit of immature glee at all the doomsday sayers that screamed crash! all over the media, who seemed just waiting to pounce on the opportunity to declare Bitcoin dead. Uhm, yeah. It turns out that over the whole day of February 10, the Bitcoin price fell a total of 19 US cents. As of this writing (01:30 UTC on Feb 11), it’s up a bit (705) from where it opened yesterday (688).
Sources: this post by TheComputerScientist, this post by nullc (Greg Maxwell), and a few other sources whose identity I’ll protect.
Amazing article Rick. So the evil nafarious hackers that were stealing from gox were actually just taking advantage of Gox leaving their cash register open to anyone smart enough to decode the transactions.
No doubt this has been happening for months and gox have been routinely re-crediting accounts which have been affected, and slowly but surely, more and more of these transactions have been fixed and rebroadcast. Until it got to the point where their hot wallet ran dry and they finally decided to investigate.
Classic gox.
I don’t think there were any hackers (crackers really) involved. Gox just said it’s theoretically possible for someone to do that. They just forgot to mention that while the bitcoin protocol plugged that hole a year ago, they didn’t, and are therefore no longer running bitcoin because of incompatibility with the rest of the network.
As I understand the situation after reading a bit about it, it is not possible to change the destination addresses or amounts of the transactions, only the transaction id.
I checked some of the transactions Mtgox issued for some of my failed withdrawal attempts, and one was malformed (too large, or too many inputs, don’t remember exactly which) and one was rejected because at least one of the inputs was already spent. This indicates two things, that their custom bitcoin client needs updating to create proper transactions and their accounting needs to track transactions success by looking at the output addresses to avoid double spending their own coins. Most bitcoin clients already do this from what I hear so it is not an issue them.
What I am concerned about, is whether anyone managed to take advantage of Mtgox’s ineptitude by rewriting their failed withdrawal transaction and then contacting support and claiming the transaction didn’t go through. If Mtgox support only checks the transaction id (hash is a more appropriate name) they would see it was not successful and resend it perhaps with new inputs that are not yet spent. This way thieves could double their money and the only thing Mtgox would see is that increasingly more transactions are failing.
Rick, I like your site, lets have a fika sometime!
It wasn’t leaving the cash register open. It was Gox paying with a check with too many 0s, and someone erasing the 0s so that the bank would accept it. Gox then would only look for the check identical to the one they sent out, and never saw it, and there are some reports of people intentionally or unintentionally seeing a withdrawal attempt failed, ask for another withdrawal, and Gox honoring it.
Those changing the transactions could very well have just been trying to get their rightfully owed money back (they couldn’t change the amount of the recipient), although they might have been able to convince Mt. Gox’s customer no-service that they never got a payment, and sent an additional check.
It was failure after failure on Gox’s side, yet the blame is on Bitcoin. Hopefully this puts the nail in their coffin.
Maybe the price of bitcoin didn’t fall much yesterday, but since Gox’s statement went out it dropped by 1/3. If that’s not a market crash then I don’t know what is. Of course it will grow back in a couple of months; it’s just another crisis of which we’ve seen a couple before. The problem is that bitcoin is also built on trust and Gox is undermining that trust by their lack of professionalism, cutting the branch on which they themselves are sitting in the process. They should be nominated for some kind of corporate Darwin award.
A couple of months?
http://bitcoincharts.com/charts/bitstampUSD#rg2ztgSzm1g10zm2g25zv
OK, well it depends on which exchange you’re looking at. I’m mostly interested in Mt.Gox since that’s where I have an account (though that might change as soon as they resume transactions). Mt.Gox was at $1000 before the scandal (around Feb 1) and dropped to $600, while BitStamp dropped from $800 to $700. Reactions to the yesterday’s statement were also different, with Gox dropping a bit and staying down and BitStamp dropping and quickly returning. So you see, this whole mess with Mt.Gox affected the overall market prices. The statement was just the cherry at the top of the cake.
You cannot talk about the Bitcoin price on Gox as it is no longer exchanging real dollars for real bitcoins. It has become decoupled from the external economy and is being delisted from many reporting tools. I wish you good luck in getting your money back.
It only crashed because stupid ( or uninformed) people choose to sell and the smart ones bought it all up. Most stayed out of it and didnt loose ( or gain) anything.
[…] Rick Falkvinge has provided a good summary of the […]
“The road turned, but Pekka didn’t” – I love it (wouldn’t have guessed nordic humor could be so close to southerner)
In the aftermath of a transit accident a policeman inquires a standing observer:
Officer: So can you tell me how did this happen?
Southerner: Well, herm… can you see that wall right there?
Officer: I do.
Southerner: He didn’t!
“The road turned, but Pekka didn’t” – I love it (wouldn’t have guessed nordic humor could be so close to southerner)
In the aftermath of a transit accident a policeman inquires a standing observer:
Officer: So can you tell me how did this happen?
Southerner: Well, herm, can you see that wall right there?…
Officer: I do.
Southerner: He didn’t!
[…] by ente_ [link] [7 […]
[…] READ MORE […]
[…] by coinwatcher [link] […]
As someone with BTC stuck in MtGox I’m wondering:
How much work is it to correct the Mt Gox code?
Would it be just a correction that addresses the issue or would the whole thing have to be re-written?
Are we talking days? Weeks?
Anybody have an idea?
More good news:
http://www.deepdotweb.com/2014/02/13/silk-road-2-hacked-bitcoins-stolen-unknown-amount/
Silk Road 2 Hacked, All Bitcoins Stolen – $2.7 Million
[…] beschreibt eine Änderung im Bitcoin-Protokoll, die vor mehr als einem Jahr eingeführt wurde, aber […]
[…] the vulnerability may not have been the cause of the breach, according to analysis of the MtGox blunder by Pirate Party founder Rick Falkvinge. The technology expert said it was […]
[…] the vulnerability may not have been the cause of the breach, according to analysis of the MtGox blunder by Pirate Party founder Rick Falkvinge. The technology expert said it was […]
This will have serious repercussions. Bitcoin already showed the signs of a typical asset bubble, but now the punch will be even bigger, if the gigantic Bitcoin exchange falls.
https://bitcoinfoundation.org/blog/?p=422
[…] a disadvantage might not have been a means of a breach, according to analysis of a MtGox fumble by Pirate Party owner Rick Falkvinge. The record consultant pronounced it was […]
[…] weten hoe het echt zit met de Transaction Malleability bug kan waarschijnlijk het best terecht bij deze uitleg van Rick […]
[…] Given the recent problems related to the transaction malleability aspect of the protocol, it is easy to predict that the […]
More good news – looks like that gox website is down.
I’m so glad that their clients lost their coins.
This is very very bad news. Bitcoin killed? 🙁
http://www.scribd.com/doc/209050732/MtGox-Situation-Crisis-Strategy-Draft
[…] to halt withdrawals due to “transaction malleability”, as I’ve written about before. It’s not anywhere near the level of mockery they deserve for cratering a community and […]
[…] as such. This is immediately identitied as technical bullshit by a number of heavy names, plus myself. Gox promises an update on February […]
The situation is serious, we have had people commit suicide as they have lost very significant sums of money. But its not just Gox that it is the problem, it is the unwillingness to hear any criticism or bad news.
Crypto code is hard. Crypto protocol design is hard. It is totally unlike other problems in that bugs are not just accidents, someone is trying to cause them to occur.
Deployment of payment systems is the hardest type of cryptography. Banks do not depend on cryptography alone they have audit systems that are continuously checking to detect and report discrepancies and there are further checking systems to see that those systems work. And these have totally different code bases from the systems they are auditing.
Doing the job right is expensive. Far more expensive than Gox could reasonably afford. They lost money every day they were operating. I see no reason to believe any of the other exchanges is different.
One issue that I don’t see much notice of is the lack of transparency in the bitcoin markets. They can post a price, but how much BitCoin are they willing to buy at that price? Spreads can be huge, $20 is quite common. There is no indication of the order size for the Bid/Ask.
The drop in the BitCoin price after Gox went under is suspiciously small. 6% of the float went missing. That should have started a run. Yet we are to believe that the price quickly returned to the price prior. Is this really correct? How much Bitcoin can I cash out today at the advertised price? Gox was limiting cash withdrawals to $10,000 a month. I have a hard time accepting the $7 billion market cap as real unless I can cash out at least $7 million a day without any restrictions at all.
PHP you hit the nail on several important issues. There sure are several really suspicious things about several of the exchanges and the inflated value of Bitcoin.