Revelations of the mismanagement at the now-bankrupt Japanese bitcoin exchange keep surfacing. When laying the puzzle as pieces keep coming, it becomes obvious that security at the billion-dollar vault was practically nonexistent. This adds to previous insights of economic and/or fraudulent mismanagement.
An interesting blog post from Mark Karpeles resurfaced recently. Mr. Karpeles was the CEO of the now-imploded Japanese bitcoin exchange MtGox, nicknamed “Empty Gox” for its previously-rumored insolvency. The blog post reveals a stunning ignorance of the concept of security, going beyond nonexistent security and into daredevil-reckless territory.
Jacob Appelbaum, the world-class security researcher and one of the spokespeople for the anonymity service Tor that has saved many activist lives worldwide, tweeted sarcastically about the article:
I think this is perhaps the most amusing technical blog post I've read in ages: //t.co/uJwtz5xtrU
— Jacob Appelbaum (@ioerror) February 28, 2014
The article in question (gone from the server, but saved by the Internet Archive) was about how Karpeles had decided to write his own security mechanisms for remote access to his core servers. This goes against every grain, every practice, every professionalism of good security that exists. Security is hard and needs thousands of eyes to find the small but important bugs – just last week, a bug in Apple’s iOS was discovered where an attacker could have impersonated any target. And that was from Apple.
Any person who calls themselves a professional in the IT field will end the conversation with anybody, no matter what title, who boasts that they have created their own security. You just don’t do it. It’s beyond reckless. It’s practically a guarantee that you will get broken into tracelessly.
It gets worse. Karpeles didn’t just write his own remote-access security (“SSH server”). He did so in the programming language PHP, which is a dangerously unsafe language intended for low-security applications like displaying web pages. It basically has no error checking or safety nets of any kind. So not only did Karpeles think it was a good idea to do something that almost guaranteed MtGox to get hacked, he did so using one of the worst possible tools imaginable. It wasn’t enough to shoot himself in the foot and reload, he had to pick a bazooka to do it.
(UPDATE: As some have pointed out, this is no definite proof said home-cooked SSH server written in PHP was used as production code on Gox. This observation is correct. However, the primary observation here is the reckless disregard for security. This is further accented by three more observations: first, in the comments in the article, Karpeles states that he intends to wrap this PHP SSH code into a production library, and second – quote in same comment field – “RSA re-implemented in pure PHP is not a bad thing.” The third observation is that a commenter named Nanashi pointed out the PHP SSH server as “the least secure implementation ever” in 2010, and while we still don’t know if it was run at Gox in production for remote server access, it’s a rather striking coincidence that the same name – Nanashi – was behind an the enormous database leak from Gox’ internal databases described later in this article.)
This is not professional behavior. This is completely-over-the-top amateurish, from somebody who a) doesn’t understand security at all and b) is so convinced of their own perfection that they dismiss every criticism. People are even pointing out flaws in his implementation in his own comment field, and he just dismisses it, despite the fact that these flaws would be enough for an adversary to assume remote control of his core servers – “ownage”, as it is called.
Traditional SSH servers too secure for you? Write one in PHP! //t.co/v2in7x6NJd
— Kyle Steely (@modalexii) February 28, 2014
When you read these facts, if you understand security, your hairs stand on your arms, you are pushing away from the screen in balking disbelief, and your eyes are going wide. This guy had taken on safekeeping of a billion dollars for his clients?
Let’s be clear: To anybody who is the slightest aware of good security practices, this article from the principal architect of the bitcoin exchange is not some government-issue red flare going off in the corner of your eye. This is a goddamn Betelgeuse going off.
To put it in non-technical terms, this is roughly somebody who claims they are qualified to be a heart surgeon because they have read the back cover of “Anatomy for Dummies”. Not just that, but they actually open a heart surgery practice.
It’s like asking for a hardened veteran infantry officer to lead a batallion into battle, and having a random guy who has read military comic books show up for the task.
It’s like asking if somebody can build a complex skyscraper, and somebody shows up with a grin from ear to ear explaining that they already found everything they need for the job in trash bins on the way to the meeting.
Somebody who was utterly not qualified to go near any kind of security job had built a vault for a billion dollars using a completely unsafe webpage scripting language. And people were using it, trusting him with their money, more or less because he said he was honest in the Terms of Service.
It gets worse. The forensic site MtGoxProtest has an interesting inside view of security practices at the company that stored bitcoin, US Dollars, and euros for its clients to a value of about a billion dollars (at peak bitcoin value).
If the product was so thorougly shoddy in terms of security, some very skilled staff at some very skilled companies are able to mitigate that by rigorous processes and consistent pride in their work. What about Gox? How did they relate to security in their daily work?
They didn’t give a shit.
Security alarms would go off, somebody would notice something totally alarming, and they would basically just ignore it.
On the surface, security looked decent. Clients would log in using two-factor authentication, not relying on a hackable password alone. Clients could separate withdrawal security from authentication security, adding a second security layer when they wanted to get money out of their account.
(This is disregarding all jokes that you couldn’t actually get any money out of the vault, because it was empty – hence “Empty Gox”.)
But it happened much too frequently that client accounts were emptied anyway overnight, and provably so by somebody else than the account holder. This should have set of major alarm bells at the Gox offices; somebody was apparently and obviously able to circumvent their security layers and access the servers directly. Mere suspicion of that is cause for a total shutdown until forensics have cleared out what the hell happened.
So what happened?
They didn’t give a shit. They blamed the customers and went about their daily business.
Coupled with the above article from Karpeles – who wrote much of the initial Gox codebase – about how Gox would violate every security practice in existence and then invent some more just so they too could be violated, it becomes clear that the strict login procedures were just for show. Gox was leaking like a bloody sieve, and Karpeles was too incompetent and too proud to understand the magnitude of the disaster in the making.
According to the insiders’ information, security researchers would regularly submit alarming reports of gaping security holes that would just as routinely be completely ignored. And then security researchers did what they do when companies ignore them, which is publish their findings. So now there was not only a billion-dollar vault with security holes the size of the Empire State Building, there were also published research papers on where they were and how they worked.
And Gox? They continued to not give a shit.
It gets worse. They treated business processes the exact same way as they did security processes: “It seems to run well and we don’t really care”. I have hinted in my previous posts that I’ve got stuff that would be jawdropping; I’m not sure it is anymore, it’s just in line with the total mismanagement – no, fraudulent operations – that has been going on. They basically didn’t know anything about contract or finance business, either.
My specific case was that I had been offered X bitcoin for Y US Dollars on the exchange webpage, clicked “buy”, and even got a separate confirmation box: “Do you want to buy X bitcoin for Y US dollars?”. As I clicked “Yes” to that, that’s entering into a legally binding contract. But I wasn’t delivered X bitcoin – I was delivered X-56 bitcoin for the Y USD, which is a rather large difference. As I pointed this out to support, that they had an unfulfilled obligation of 56 bitcoin to me, they explained patiently how the quote price was calculated from a technical standpoint, why I had been charged a much higher price than quoted, and implied that the technology and interface were working just as designed.
Listen here Karpeles, I don’t care in the slightest why you think the price should be higher than quoted – if you quote a price and I accept the offer, you deliver on it, and you deliver exactly what was offered. If you can’t do so, that’s your problem, not mine.
They didn’t understand these very basics of running a business. They didn’t understand the concept of offers, accepts, and contracts. Or they just didn’t care. This 56-coin claim of mine was one of the open items in support threads when Gox folded, and I was totally prepared to bring that to court. Now it’s rolled up in my overall bankruptcy claim instead.
It gets worse.
Yesterday, a leak was posted with internal accounting data at MtGox. It contained every customer balance, their last login timestamp, withdrawal limits for every customer, and a lot of other client data. Whoever orchestrated that leak had access to the internalmost servers at Empty Gox.
That is such total ownage of somebody’s poor security, there’s nothing left to say. It’s confirmation of everything from the original article – that this is a person who didn’t understand the most basic security practices.
Gox appears to have been run on the kind of security that only an idiot would have on their luggage.