Mesosphere

Mesosphere is a software solution that provides ways of managing server infrastructures and basically expands upon the cluster-management capabilities of Apache Mesos. Mesosphere has also launched the DCOS (data center operating system), used to manage data centers by spanning all the machines and treating them as a single computer, providing a highly scalable and elastic way of deploying apps on top of it. DCOS can be installed on any public cloud or your own private data center, ranging from AWS, GCE, and Microsoft Azure to VMware. Marathon is the framework for Mesos and is designed to launch and run applications; it serves as a replacement for the init system. Marathon provides various features such as high availability, application health check, and service discovery, which help you run applications in Mesos clustered environments.

This session describes how to bring up a single-node Mesos cluster.

Docker containers

Mesos can run and manage Docker containers using the Marathon framework.

In this exercise, we will use CentOS 7 to deploy a Mesos cluster:

  1. Install Mesosphere and Marathon using the following command:
    # sudo rpm -Uvh http://repos.mesosphere.com/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
# sudo yum -y install mesos marathon

    Apache Mesos uses Zookeeper to operate. Zookeeper acts as the master election service in the Mesosphere architecture and stores states for the Mesos nodes.

  2. Install Zookeeper and the Zookeeper server package by pointing to the RPM repository for Zookeeper, as follows:
    # sudo rpm -Uvh http://archive.cloudera.com/cdh4/one-click-install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm
# sudo yum -y install zookeeper zookeeper-server
  3. Validate Zookeeper by stopping and restarting it:
    # sudo service zookeeper-server stop
# sudo service zookeeper-server start

    Mesos uses a simple architecture to give you intelligent task distribution across a cluster of machines without worrying about where they are scheduled.

  4. Configure Apache Mesos by starting the mesos-master and mesos-slave processes as follows:
    # sudo service mesos-master start
# sudo service mesos-slave start
  5. Mesos will be running on port 5050. As shown in the following screenshot, you can access the Mesos interface with your machine's IP address, here, http://192.168.10.10:5050:
    Docker containers
  6. Test Mesos using the mesos-execute command:
    # export MASTER=$(mesos-resolve `cat /etc/mesos/zk` 2>/dev/null)
# mesos help
# mesos-execute --master=$MASTER --name="cluster-test" --command="sleep 40"
  7. With the mesos-execute command running, enter Ctrl + Z to suspend the command. You can see how it appears in the web UI and command line:
    # hit ctrl-z
# mesos ps --master=$MASTER

    The Mesosphere stack uses Marathon to manage processes and services. It serves as a replacement for the traditional init system. It simplifies the running of applications in a clustered environment. The following figure shows the Mesosphere Master slave topology with Marathon:

    Docker containers

    Marathon can be used to start other Mesos frameworks; as it is designed for long-running applications, it will ensure that the applications it has launched will continue running even if the slave nodes they are running on fail.

  8. Start the Marathon service using the following command:
    # sudo service marathon start

    You can view the Marathon GUI at http://192.168.10.10:8080.

Deploying a web app using Docker

In this exercise, we will install a simple Outyet web application:

  1. Install Docker using the following commands:
    # sudo yum install -y golang git device-mapper-event-libs docker
# sudo chkconfig docker on
# sudo service docker start
# export GOPATH=~/go
# go get github.com/golang/example/outyet
# cd $GOPATH/src/github.com/golang/example/outyet
# sudo docker build -t outyet.
  2. The following command tests the Docker file before adding it to Marathon:
    # sudo docker run --publish 6060:8080 --name test --rm outyet
  3. Go to http://192.168.10.10:6060/ on your browser in order to confirm it works. Once it does, you can hit CTRL + C to exit the Outyet Docker.
  4. Create a Marathon application using Marathon Docker support, as follows:
    # vi /home/user/outyet.json
{
   "id": "outyet",
   "cpus": 0.2,
   "mem": 20.0,
   "instances": 1,
   "constraints": [["hostname", "UNIQUE", ""]],
   "container": {
      "type": "DOCKER",
      "docker": {
         "image": "outyet",
         "network": "BRIDGE",
         "portMappings": [ { "containerPort": 8080, "hostPort": 0, "servicePort": 0, "protocol": "tcp" }
        ]
     }
   }
}

# echo 'docker,mesos' | sudo tee /etc/mesos-slave/containerizers
# sudo service mesos-slave restart
  5. Containers are configured and managed better with Marathon Docker, as follows:
    # curl -X POST http://192.168.10.10:8080/v2/apps -d /home/user/outyet.json -H "Content-type: application/json"
  6. You can check all your applications on the Marathon GUI at http://192.168.10.10:8080, as shown in the following screenshot:
    Deploying a web app using Docker

Deploying Mesos on AWS using DCOS

In this final section, we will be deploying the latest launch of DCOS by Mesosphere on AWS in order to manage and deploy Docker services in our data center:

  1. Create an AWS key pair in the region where the cluster is required to be deployed by going to the navigation pane and choosing Key Pairs under NETWORK & SECURITY:
    Deploying Mesos on AWS using DCOS
  2. After being created, the key can be viewed as follows and the generated key pair (.pem) file should be stored in a secure location for future use:
    Deploying Mesos on AWS using DCOS
  3. The DCOS cluster can be created by selecting the 1 Master template on the official Mesosphere site:
    Deploying Mesos on AWS using DCOS

    It can also be done by providing the link for the Amazon S3 template URL in the stack deployment:

    Deploying Mesos on AWS using DCOS
  4. Click on the Next button. Fill in the details such as Stack name and KeyName, generated in the previous step:
    Deploying Mesos on AWS using DCOS
  5. Review the details before clicking on the Create button:
    Deploying Mesos on AWS using DCOS
  6. After 5 to 10 minutes, the Mesos stack will be deployed and the Mesos UI can be accessed at the URL shown in the following screenshot:
    Deploying Mesos on AWS using DCOS
  7. Now, we will be installing the DCOS CLI on a Linux machine with Python (2.7 or 3.4) and pip preinstalled, using the following commands:
    $ sudo pip install virtualenv
$ mkdir dcos
$ cd dcos
$ curl -O https://downloads.mesosphere.io/dcos-cli/install.sh
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3654  100  3654    0     0   3631      0  0:00:01  0:00:01 --:--:--  3635
$ ls
install.sh
$ bash install.sh . http://mesos-dco-elasticl-17lqe4oh09r07-1358461817.us-west-1.elb.amazonaws.com
Installing DCOS CLI from PyPI...
New python executable in /home/vkohli/dcos/bin/python
Installing setuptools, pip, wheel...done.
[core.reporting]: set to 'True'
[core.dcos_url]: set to 'http://mesos-dco-elasticl-17lqe4oh09r07-1358461817.us-west-1.elb.amazonaws.com'
[core.ssl_verify]: set to 'false'
[core.timeout]: set to '5'
[package.cache]: set to '/home/vkohli/.dcos/cache'
[package.sources]: set to '[u'https://github.com/mesosphere/universe/archive/version-1.x.zip']'
Go to the following link in your browser:
https://accounts.mesosphere.com/oauth/authorize?scope=&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&client_id=6a552732-ab9b-410d-9b7d-d8c6523b09a1&access_type=offline
Enter verification code: Skipping authentication.
Enter email address: Skipping email input.
Updating source [https://github.com/mesosphere/universe/archive/version-1.x.zip]
Modify your bash profile to add DCOS to your PATH? [yes/no]  yes
Finished installing and configuring DCOS CLI.
Run this command to set up your environment and to get started:
source ~/.bashrc && dcos help

    The DCOS help file can be listed as follows:

    $ source ~/.bashrc && dcos help
Command line utility for the Mesosphere Datacenter Operating System (DCOS). The Mesosphere DCOS is a distributed operating system built around Apache Mesos. This utility provides tools for easy management of a DCOS installation.
Available DCOS commands:

        config       Get and set DCOS CLI configuration properties
        help         Display command line usage information
        marathon     Deploy and manage applications on the DCOS
        node         Manage DCOS nodes
        package      Install and manage DCOS packages
        service      Manage DCOS services
        task         Manage DCOS tasks
  8. Now, we will deploy a Spark application on top of the Mesos cluster using the DCOS package after updating it. Get a detailed command description with dcos <command> --help:
    $ dcos config show package.sources
[
  "https://github.com/mesosphere/universe/archive/version-1.x.zip"
]
$ dcos package update
Updating source [https://github.com/mesosphere/universe/archive/version-1.x.zip]

$ dcos package search
NAME       VERSION            FRAMEWORK     SOURCE             DESCRIPTION
arangodb   0.2.1                True     https://github.com/mesosphere/universe/archive/version-1.x.zip   A distributed free and open-source database with a flexible data model for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.
cassandra  0.2.0-1               True     https://github.com/mesosphere/universe/archive/version-1.x.zip  Apache Cassandra running on Apache Mesos.
chronos    2.4.0                 True     https://github.com/mesosphere/universe/archive/version-1.x.zip  A fault tolerant job scheduler for Mesos which handles dependencies and ISO8601 based schedules.
hdfs       0.1.7                 True     https://github.com/mesosphere/universe/archive/version-1.x.zip  Hadoop Distributed File System (HDFS), Highly Available.
kafka      0.9.2.0               True     https://github.com/mesosphere/universe/archive/version-1.x.zip  Apache Kafka running on top of Apache Mesos.
marathon   0.11.1                True     https://github.com/mesosphere/universe/archive/version-1.x.zip  A cluster-wide init and control system for services in cgroups or Docker containers.
spark      1.5.0-multi-roles-v2  True     https://github.com/mesosphere/universe/archive/version-1.x.zip  Spark is a fast and general cluster computing system for Big Data.
  9. The Spark package can be installed as follows:
    $ dcos package install spark
Note that the Apache Spark DCOS Service is beta and there may be bugs, incomplete features, incorrect documentation or other discrepancies.
We recommend a minimum of two nodes with at least 2 CPU and 2GB of RAM available for the Spark Service and running a Spark job.
Note: The Spark CLI may take up to 5min to download depending on your connection.
Continue installing? [yes/no] yes
Installing Marathon app for package [spark] version [1.5.0-multi-roles-v2]
Installing CLI subcommand for package [spark] version [1.5.0-multi-roles-v2]
  10. After deployment, it can be seen in the DCOS UI under the Services tab, as shown in the following screenshot:
    Deploying Mesos on AWS using DCOS
  11. In order to deploy a dummy Docker application on the preceding Marathon cluster, we can use the JSON file to define the container image, command to execute, and ports to be exposed after deployment:
    $ nano definition.json
{
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "superguenter/demo-app"
    }
  },
  "cmd":  "python -m SimpleHTTPServer $PORT",
  "id": "demo",
  "cpus": 0.01,
  "mem": 256,
  "ports": [3000]
}
  12. The app can be added to Marathon and listed as follows:
    $ dcos marathon app add definition.json
$ dcos marathon app list
ID       MEM    CPUS  TASKS  HEALTH  DEPLOYMENT  CONTAINER  CMD
/demo   256.0   0.01   1/1    ---       ---        DOCKER   python -m SimpleHTTPServer $PORT
/spark  1024.0  1.0    1/1    1/1       ---        DOCKER   mv /mnt/mesos/sandbox/log4j.properties conf/log4j.properties && ./bin/spark-class org.apache.spark.deploy.mesos.MesosClusterDispatcher --port $PORT0 --webui-port $PORT1 --master mesos://zk://master.mesos:2181/mesos --zk master.mesos:2181 --host $HOST --name spark
  13. Three instances of the preceding Docker app can be started as follows:
    $ dcos marathon app update --force demo instances=3
Created deployment 28171707-83c2-43f7-afa1-5b66336e36d7
$ dcos marathon deployment list
APP    ACTION  PROGRESS  ID
/demo  scale     0/1     28171707-83c2-43f7-afa1-5b66336e36d7
  14. The deployed application can be seen in the DCOS UI by clicking on the Tasks tab under Services:
    Deploying Mesos on AWS using DCOS
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset