Briefly profitable alt-coin mining on Amazon through better code

Over the past three weeks, I've been running crypto-coin mining software on between 20 and 60 Amazon EC2 GPU instances at a (very small) profit, with the goal of understanding the currencies, exchanges, and algorithms in more detail.  This is post #1 of 2 discussing scrypt coin mining on Amazon.  In the follow-up post, I'll go into the algorithmic and engineering details of building a better CUDA-based miner.

A few weeks ago, Gün and colleagues wrote a paper describing a new collusion attack against Bitcoin.  I hadn't thought much about BTC before that, but he got me curious.  As I explored, I discovered Litecoin,  an alternative to Bitcoin that is designed to reduce the comparative advantage of using custom ASICs (or GPUs) for mining it relative to using a conventional CPU.  Its future is even less certain that BTC, of course, as a later comer, but the technically interesting bits lie in its proof-of-work hash function:  Scrypt.  Scrypt is designed to be "memory-hard" in addition to being computationally hard.

Aside:  "Proof-of-work".  Nearly all of the modern cryptocurrencies operate by allowing "mining" by finding a partial collision with a hash function.  The output of mining is that many different keys end up signing the history of transactions for the currency, ensuring that no one key controls the currency as long as no one entity (or colluding group) does not posses sufficient computational power to dominate "mining."  The incentive to "mine" is that successful mining operations give the miner a few coins as a payment.  A way to view this is that mining spends power - literally, electrical power - to burn computation.  The currencies are automatically regulated so that only a certain number of coins can be minted per week:  If more miners join the pool, then the next week, the "difficulty" of mining (the length of the partial collision) will be increased. Economically, then, one might expect the price of entry and power of mining required to equalize with the value of the coins minted, and, indeed, that seems to roughly happen.

Back to scrypt.  My research group does a lot of work these days on memory efficiency, so I wondered if this would be an interesting function to look at for applying some of our techniques.  I was able to get a 5% speedup on the CPU implementation of scrypt mining ("cpuminer") by adding some selective prefetching, but nothing too interesting.  And then I turned to GPUs.

There's a persistent belief that NVidia GPUs are much worse for crypto-currency mining than AMD GPUs.  And, indeed, if you look at the table of Litecoin mining hardware performance, you'll see that this trend seems true.  The AMD Radeon HD7990, for example, can mine about 1300 khash/s at about 389 watts (numbers culled randomly from the innertubes, may be off).  That's good - 3.3 kh/s/w.  Comparing against NVidia's similarly high-end GTX 780 at 320 kh/s at a 250W TDP, and probably a bit less in practice.  Let's call it 1.5 kh/s/w, or under half of what the AMD cards get.

Now, you might ask, why did I mention that NVidia Grid K2?  Because that's what Amazon's G2 GPU instances offer.  Until last week, the spot market price of these instances was $0.075 per hour:

And the graph shows what happens when the litecoin price exceeds the cost of mining it at the Amazon spot market price:  Lots of people move in and start mining on Amazon.  Including me.  This is the Litecoin-to-USD graph for roughly the same time period, from

I've had a horrible case of bronchitis for the last month (feeling a lot better now, phew).  But not being able to work out, or even stay at work as long as I wanted, etc., left me with spare time on my hands sitting in bed with my laptop.  So it turned out that, by coincidence, just before the spike in litecoin prices, I'd spent the previous week and a half writing a better scrypt miner for NVidia-based cards using the Kepler architecture.  Which meant that, as of this writing, my code was about 30% or 40% faster than any of the public code for scrypt hashing on Nvidia cards.  The core improvements:  It went from about 150 kh/sec to about 220 kh/s, and its CPU use dropped from 30% to 0.1%.

With the GPU hitting 215-220 kh/s and the CPU at 35 kh/s (remember, it was idle), I figured that I could conservatively get 230 kh/s.  Assuming a 10% overhead in shifting the money back from LTC -> USD, and taking a conservative view of the likely sale price (there's a lag between mining them and selling them), then mining on Amazon would produce about $50-$75/month/instance.

I suspect I started running my GPU miners on Amazon before most other people.  For about 4 hours, I was running at $0.075/hr.  Then the spot market price jumped.  And jumped.  And jumped again.  And then it went through the roof, as you can see.   For a while, G2 instances in Northern Virginia were going for about $0.3 / hour.  In order for that to be profitable, someone would have to be getting 350 kh/s out of their instance.  It's possible.  I'm not a parallel implementation of crypto wizard, and someone may have figured out a better trick.  But I'd be a little surprised.  I think a more likely explanation is:
  • Some people are mining using EC2 credits they purchased at a discount.  Amazon gives away $100 AWS EC2 credits to get people hooked.  It seems likely that some people are trying to exploit this.
  • Some people are mining using their employer's amazon bills.  (!)
  • Some people are being silly and losing money, hoping that the coins will rise.
  • Legitimate spot market users who're trying to get work done despite all of this alt-coin mining.
The latter blurs a bit:  During all of this, I've been periodically checking out instances at higher prices in order to run benchmarks, for example.  So there could be a lot of people dropping in to test the waters and then fleeing.

For longer than I expected, the advantage of faster code let me continue making a profit on Amazon even after other miners had driven the spot market price nearly to equilibrium.  I managed it by switching to some of the other alternative scrypt currencies -- Worldcoin became my fallback, with an automated setup to transfer the coins to the Vircurex exchange and sell them for Bitcoins.  But at this point, the game appears pretty over, at least until the market becomes rational.  It was tons of fun for a few weeks. :)

For the curious, the processing pipeline looked like:

  1. Mine coins on Amazon through a mining pool
  2. Mining pool auto-deposit (Worldcoin/Litecoin) to the Vircurex exchange
  3. Convert WDC to BTC at Vircurex
  4. Transfer BTC to Coinbase (US-based Bitcoin startup)
  5. Transfer USD to bank
  6. Pay Amazon bill in USD.  Wince.

I experimented with services like Multipool and Hashcows that try to automatically switch to the most profitable coin, but found them dissatisfactory.  Multipool leaves you with a lot of little puddles of silly alt-currencies to exchange.  Hashcows hasn't quite had the uptime I want.  Mining worldcoin was a nice compromise - it stayed more profitable than Litecoin for most of my experiment.

When the profit margin shrunk, the game became more stressful - part of the reason I'm out now.  With the increase in scrypt coin prices, the load on the mining pools increased, and most began experiencing frequent outages.  An outage is annoying when you're paying Amazon by the hour.  I had to hack the cpuminer code to add a backup server, etc., etc.  Writing the initial CUDA miner code was fun.  This wasn't - it started to feel like work, with way less learning-per-unit-time.

And then the spot market went stupid:

Yes - the spot market prices are measured in dollars per hour, whereas the non-spot prices for those instances are only $0.65 / hour.  Somewhere, there are a large number of people voluntarily giving Amazon 10x more money than they need to.

It's clear who the real winner is in this game:  My gross revenue was about $1000, and I paid Amazon $500 of that. But earning $500 by keeping distracted while sick at home with month-long bronchitis was pretty cheap entertainment.  Otherwise I'd have probably sat there clicking refresh on Hacker News all day and letting my brain drool out.

Next post will go into more technical details about how to make a better scrypt miner.


Popular posts from this blog

Reflecting on CS Graduate Admissions

Chili Crisp Showdown: Laoganma and Flybyjing

Two examples from the computer science review and publication process