What good is a recommendation that’s based on data if your algorithms don’t have access to ALL of the data that’s needed to generate the most relevant recommendations? More importantly, as a startup with limited time and resources, how can you open your platform to a potentially limitless firehose of inbound data and survive long enough to make a dime?

These are two serious questions we struggled to answer while scaling our recommendations as a service API. It’s easier and more cost efficient than ever to build a big data startup thanks to services like AWS, Heroku and Google’s Cloud Platform. That simplicity, however, still comes at a price. It only takes a day or two for us to burn through our allocated AWS free tier credits before our bank account starts shrinking.

It would be easy for us to charge for inbound API requests/data storage to solve our problem but doing so wouldn’t be fair to our customers because our value is in the recommendations we serve, not the resources required to generate them.

If you had been eavesdropping on our conversations while we worked to answer these questions you would have heard that:

• It didn’t make sense to tie our customers to specific data structures given the fact that every customer’s data is different. We built our data set methods with NoSQL in mind relying entirely on Amazon’s DynamoDB. A big benefit of using DynamoDB is that it gives us the ability to control our read/write throughput, in real-time, according to our needs.
• Simply dialing up the read/write throughput in DynamoDB to accept all inbound requests didn’t make sense because doing so would have been too costly. DynamoDB gets really expensive the more write throughput you provision. It’s difficult for us to predict how much data might come in at any given point until a customer has been on board for a while, so relying solely on write throughput would be a mistake.
• Queuing inbound data in SQS makes sense and is lightening fast. The only problem with queuing is that we still have to have the workers to process everything that’s sitting in the queue in near real-time.
• Heroku’s dynos are expensive (in startup terms) and had recently come under fire for routing issues which caused some customer’s monthly bills to skyrocket. It made sense for us to use their web dynos to serve our API but using their worker dynos to process the queues, not so much.
• Firing up an Amazon EC2 instance or three to manage the workers isn’t something we’re interested in doing mainly due to the extra administrative needs EC2 creates, not to mention the added costs of running instances 24/7.

We were so close! If it wasn’t for the damn workers we’d be set!

I had been tinkering with IronWorker and a few other Heroku worker add-ons for a week or two but wasn’t really feeling any of the offerings. The fundamentals of what we were looking for were there but all still seemed wildly inefficient. Tasks that should have taken less than twenty seconds to execute were taking minutes. Each add-on had its quirks and some of them were deal breakers. We’d come so far and had realized that there was a distinct possibility that we’d have to implement our own worker service.

Then fortune smiled. The folks at Iron.io sent me a “Tips & Tricks” email on the same day I was getting ready to start coding our worker service (no joke). In that email was a link to a blog post by one of Iron.io’s Co-Founders, Travis Reeder, entitled How to Reduce a 9 Hour Job to a 9 Minute Job. Were they reading my mind?

It turns out that we weren’t the only ones with such struggles as the first line of the post so poignantly states: “A common problem developers have is how to run through a large number of tasks in a short amount of time.” PREACH ON! I kept reading.

I immediately felt better (and a little silly) after reading the post. Why hadn’t I attempted to dig deeper into Iron.io’s blog before I threw in the towel? Or hit them up on HipChat (which I subsequently did, and they were awesome). The answer was there the whole time!

The post got me thinking about breaking work into smaller pieces to take advantage of concurrent task execution. I imagined that we could spawn additional tasks based on the total number of items in a queue and let recursion take its course until the workers burned through all of the records in each queue. The end result was a collection of workers that run on a schedule and are entirely self-contained within IronWorker. Their workflow goes a little something like:

• The parent worker runs every minute and decides on the fly which queues need to be processed
• The parent worker spawns a child worker for each queue (up to n queues) that pulls 100 messages from each queue for processing. It alternates queues based on the number of messages in each queue to ensure that all queues have an equal chance of having their records processed during a parent cycle.
  (Note: the number of messages we pull from each queue during each cycle required manual tuning based on our needs; 100 works for us, for now. n can be anything from 1 to the maximum number of concurrent tasks you have available in your pricing plan.)
• Each child worker does its thing and empties its allocated messages from the queue in under 15 seconds.

This took us from having the ability to process 1,000-2,000 records/minute to 120,000/minute (2,000/second) based on our current pricing plan. Bumping ourselves up to the next plan would put us somewhere around 300,000/minute which clocks in at 5,000/second. And we didn’t bump write throughput up a single tick. Increase the write throughput and things get crazy fast!

And the best part? Everything I’ve mentioned above took less than a day to implement and costs less than $500/month (AWS costs included). FO REALZ.

Limitless scalability is real and the team at Iron.io is making it happen. IronWorker is rock solid and our stack wouldn’t be the same without it.

Tags: Amazon, AWS, Google Cloud Platform, Heroku, Iron.io, IronWorker, scalability, What The Stack, worker