Instant Big Data and other solutions

by Maarten Ectors on 3 September 2014

For each new technology that becomes trendy you can buy a book with an animal on the cover. The first chapters will guide you through installing, integrating and scaling the new technology. This was fine when software was just a couple of apt-gets away. With horizontally scalable solutions, micro-services, containers, PaaS platforms, Big Data solutions, etc. software now consists of many elements that can each be scaled independently and distributed over many (virtual) servers. Add autoscaling to the mix to make things even more complex. Additionally the number of software solutions is almost exponentially growing. Each week there is yet another key-value store, NoSQL or NewSQL solution. The Docker community releases new tools every day. OpenStack has new sub-projects frequently.  CloudFoundry adds buildpacks continuously.

The result: Solution Architects don’t have time to try out each new solution, let alone use them for anything useful.

Juju, Charms and Bundles

Canonical and the Ubuntu community have been working already for quite some time on Juju, the award-winning open source cloud orchestrator. Juju allows anybody to instantly deploy, integrate and scale any software on any public cloud, private cloud or bare-metal server [via MaaS]. In the last year we have seen many new charms for everything from Big Data, NoSQL, Docker, OpenStack, CloudFoundry, etc. We have also seen these charms bundled hence anybody can quickly deploy and integrate them instantly.

Chuck Butler, Amir Sanjar and others did a wonderful job on making the latest Hadoop Yarn, Hive, ElasticSearch, Spark, etc. available as charms. See their video on how a Hortonworks bundle can be deployed in minutes.

Taking it a step further: Instant Solutions

Deploying a charm or bundle is saving time but in most cases a solution architect will still need to read the book and find out how to use the software that is encapsulated inside the charm. This is why we want to take it to the next level: instant solutions.

Example 1: Instant Big Data Stream Processing with Instant Storm

Storm was originally created by Nathan Marz when he was at Twitter to analyse billions of tweets in real-time. Storm is now part of the Apache Foundation and can be used for any real-time analytics, data loading, data processing, etc. You create topologies of bolts and spouts to process a continuous stream of messages in real-time.

Storm is a non-trivial scale-out solution with many moving parts, e.g. nimbus, zookeeper, worker, supervisor, etc. Until now there was only one solution to quickly get a storm cluster set-up and it was tailored towards AWS.

Via a Juju charm bundle anybody can not set-up Storm in minutes via one drag-and-drop, two clicks or one command, i.e. juju quickstart bundle:~a.rosales/storm/5/instantstorm. The Storm bundle can be deployed in minutes on any public cloud, private cloud or bare-metal server via MAAS.

However a Storm charm bundle would still oblige everybody to read the chapter in the book about how to deploy storm topologies onto a cluster. To solve this the StormDeployer was created. The StormDeployer is a subordinate charm that takes as input a URL. This URL should point to a yaml text file that describes where to download storm topologies, how to package them and how to deploy them. A developer can now create storm topologies and share them via a single URL. You can either execute one line, i.e.juju set stormdeployer “deploy=http://somedomain/somefile.storm” or update the deploy field in the service settings of the StormDeployer in the Juju GUI. Minutes later the topologies should be running on your Storm cluster. Anybody can now deploy the instant storm bundle and run topologies in minutes.

The Instant Storm Solution allows anybody to share Storm topologies that solve a concrete problem. Users will not have to be experts in how to deploy a Storm cluster or how to upload a Storm topology. They just deploy a Storm bundle and add the topologies they want deploy via a single URL. Find out more about Instant Storm on this blog post.

Example 2: Instant Scalable & Parallel Video Transcoding

If you want to setup a large cluster of video transcoders and have your video transcoded automatically by a cluster of machine then there is an instant solution for you. The author, Dustin Kirkland, provides all the technical details on how the solution works in this blog post. For the rest of us, it is enough to know that you need to pass a URL to your video in the service settings of the Transcode charm for it to be transcoded on 8 machines in parallel. If you want to have a cluster of a 100 transcoding servers, you just scale up the transcode charm, i.e. juju add-unit -n 100 transcode or via two clicks in the Juju GUI.

Instant Solutions for Concrete Problems

Anybody can create instant solutions. You just need to make sure all software components are available as a charm. Create a bundle of all the charms you need. Make a mechanism for people to use your instant solution without having to learn anything about how the solution or the charms work.

The possibilities are limitless. Imagine setting up an eCommerce shop integrated with back-office ERP in minutes and just having to decide on the design and upload your product inventory. Your company could have a communication solution that lets you pick between many alternative voice mail systems, conference bridges, telecom apps, etc. and an easy GUI to search between all your past calls. You could even have a Startup-out-of-the-Box with website, wiki, CRM, continuous deployment, backups, etc. all included. Commercial vendors can show customers the value they bring in minutes instead of weeks or months. System integrators can offer instant solutions in minutes and provide support and customisation as an add-on.

Are you a developer that wants to surprise the world by solving problems in minutes? Are you a software vendor that is unhappy showing slides about your potential solutions? Are you a system integrator that is bored of installing the same software over and over again following a waterfall approach? Why don’t you take a look at Juju and solve some of the world’s toughest problems with instant solutions?