Cloudbursting - Hybrid Application Hosting
I get to meet with lots of developers and system architects as part of my job. Talking to them about cloud computing and about the Amazon Web Services is both challenging and rewarding. Cloud computing as a concept is still relatively new. When I explain what it is and what it enables, I can almost literally see the light bulbs lighting up in people's heads as they understand cloud computing and our services, and what they can do with them.
A typical audience contains a nice mix of wild-eyed enthusiasts and more conservative skeptics. The enthusiasts are ready to jump in to cloud computing with both feet, and start to make plans to move corporate assets and processes to the cloud as soon as possible. The conservative folks can appreciate the benefits of cloud computing, but would prefer to take a more careful and measured approach. When the enthusiasts and the skeptics are part of the same organization, they argue back and forth and often come up with an interesting hybrid approach.
The details vary, but a pattern is starting to emerge. The conservative side advocates keeping core business processes inside of the firewall. The enthusiasts want to run on the cloud. They argue back and forth for a while, and eventually settle on a really nice hybrid solution. In a nutshell, they plan to run the steady state business processing on existing systems, and then use the cloud for periodic or overflow processing.
After watching (sometimes in real time in the course of a meeting) this negotiation and ultimate compromise take place time and time again in the last few months, I decided to invent a new word to describe what they are doing. I could have come up with some kind of lifeless and forgettable acronym, but that's not my style. I proposed cloudbursting in a meeting a month or two ago and everyone seemed to like it.
So, here we go. Cloudbursting is an application hosting model which combines existing corporate infrastructure with new, cloud-based infrastructure to create a powerful, highly scalable application hosting environment.
Earlier this week my colleague Deepak Singh pointed me to a blog post written by Thomas Brox Røst. In the post, Thomas talks about how he combined traditional hosting with an EC2-powered, batch mode page regeneration system. His site (Eventseer) contains over 600,000 highly interconnected pages. As traffic and content grew, serving up the pages dynamically became prohibitively expensive. Renerating all of the pages on a single server would have taken an unacceptably long 7 days, and even longer as the site became more complex. Instead, Thomas used a cloudbursting model, regenerating the pages on an array of 25 Amazon EC2 instances in just 5 hours (or, as he notes, "roughly the cost of a pint of beer in Norway."). There's some more information about his approach on the High Scalability blog. Thomas has also written about running Django on EC2 using EBS.
I'd be interesting in hearing about more approaches to creating applications which cloudburst.
-- Jeff;
