How to Get Started with Apache Mesos, Marathon, and ZooKeeper
One of the larger problems companies have when deciding to Dockerize their applications is that they often continue to think of managing their compute, memory, and CPU resources in a traditional Hypervisor-centric model.
The benefits of Docker in production can be more fully realized when the speed of infrastructure provisioning matches or exceeds the speed of application development, application containerization, and application release. In short, coupling the rapid application spin-up time of a Docker container or cluster of Docker services should be easily abstracted through a similarly rapid and simple datacenter interaction.
Imagine if you could program against your entire datacenter as if it were a single pool of resources…
…Enter Apache Mesos, a distributed systems kernel with built-in support for Docker containers, an HTTP API, a Web UI, true high availability, and fault tolerance that is designed natively for the cloud.
Following along with the Advanced Mesos Tutorial provided by Mesos, let’s experiment with a simple Vagrant VM running Mesos.
First Things First
To begin, you’ll want to Install Vagrant and VirtualBox. For ease of starting, you may download my simple GitHub starter repo. Vagrant will use the ansible_local provider to install Ansible on the guest VM and configure it with the necessary packages to run Mesos (PR’s accepted for those Ansibulls out there with better ideas for the playbook). Once you’ve cloned the repository, you can CD into it and run: $ vagrant up
Per the Mesos instructions, our Vagrantfile and Ansible playbook.yml will execute tasks which perform the following:
- Install a simple Centos-7.1 Vagrant VM
- Configure your /etc/hosts file to have the correct hosts
- Add the Mesosphere RPM repository
- Install the Mesos and Marathon RPMs
Remember, you can use vagrant halt, vagrant destroy, and vagrant up to rebuild the VM at will.
Continuing on, the Ansible playbook installs Zookeeper, which Apache Mesos uses as the master election service.
- Install the necessary ZooKeeper RPMs & install it
- Initialize the zookeeper-server to use ID of 1
- Start ZooKeeper
Next, our Ansible playbook will start the Mesos Master and Slave processes as well as starting Marathon.
Let’s run $ vagrant ssh to get into our Vagrant VM and test out a few things. $ vagrant ssh $ sudo netstat -nlp | grep mesos
Now, you should be able to access the Mesos user interface with your browser at 192.168.33.10:5050
You can test the mesos-execute command via the following:
$ export MASTER=$(mesos-resolve
cat /etc/mesos/zk 2>/dev/null)
$ mesos help
Marathon is a production-proven Apache Mesos framework for container orchestration, providing a REST API for starting, stopping, and scaling applications. Written in Scala, Marathon can run in highly-available mode by running multiple copies. The state of running tasks gets stored in the Mesos state abstraction.
Marathon is a meta framework, meaning you can start other Mesos frameworks such as Chronos or Storm with it to ensure they survive machine failures. It can launch anything that can be launched in a standard shell. In fact, you can even start another Marathon instances via Marathon.
Mesos can launch anything that can be launched in a standard shell.
Analogous to running $ sudo service marathon start, we’ll use Ansible’s service module to enable and start marathon. If you’ve ran $ vagrant up, this task has already been completed for you.
Visit the Marathon GUI by clicking 192.168.33.10:8080/
Returning to the Vagrant VM CLI, you can test installing an application that will be run by Marathon.
$ python -m SimpleHTTPServer
Now visit 192.168.33.10:8000 and you should see:
Feel free to CTRL+Z this process and $ exit out of the Vagrant VM.
Now, let’s start using the Marathon GUI located at: 192.168.33.10:8080/ui/#/apps
Click on Create Application:
Here, we’re presented with a window in which we can name and configure an application to run in Marathon. In this case, we’ll use the same simple Python webserver:
For name: mypythonserver
For command: python -m SimpleHTTPServer $PORT
The $PORT ENV variable will dynamically assign this to a port.
Click Create Application and we’re off. We now see the status go from deploying to running:
We can discover details about this instance running and find the port to access it from a web browser:
In my case, the random port assigned is 31377, which I can access in a browser:
We can now test the scaling ability with Marathon. I’ve set my instance configurations for mypythonserver to be 0.1 for CPU and 32 for Memory. Click on Scale Application, select 4 as the number of instances to scale to, and watch Marathon go to work.
This is just the tip of the iceberg for using Mesos to automate your datacenter. We’ll move onto more complex deployments in the next installment. Until then, happy hacking!