Experimenting with OpenStack* Sahara* on Docker* Containers

1. Install Docker at first.
   Option 1 - Automatically Install from repo
   # sudo yum install docker
   Option 2 - Manually Install the latest Docker
   # wget https://get.docker.com/builds/Linux/x86_64/docker-latest -O docker
   # chmod +x docker
   # sudo ./docker -d &
   For more information, please refer to https://docs.docker.com/installation/binaries/

   For RHEL6, you will need RHEL 6.5 or higher, with a RHEL 6 kernel version 2.6.32-431 or higher as this has specific kernel fixes to allow Docker to work. For more details, just refer to link above. In order for Nova to communicate with Docker over its local socket, add nova to the docker group and restart the compute service to pick up the change:
   # usermod -G docker nova
   # service openstack-nova-computerestart

2. Install Nova Docker Driver:
   # pip install -e git+https://github.com/stackforge/nova-docker#egg=novadocker
   Install the required modules:
   # cd src/novadocker/
   # python setup.py install

3. Nova Configuration
   Nova must to be configured to use nova docker driver. Edit “/etc/nova/nova.conf” to configure below options:
   [DEFAULT]
   compute_driver = novadocker.virt.docker.DockerDriver
   Create the directory /etc/nova/rootwrap.d(consistent with the “filters_path” in the file /etc/nova/rootwrap.conf), inside the directory create a file “docker.filters” with the following content:
   [Filters]
   # nova/virt/docker/driver.py:'ln', '-sf', '/var/run/netns/.*'
   ln: CommandFilter, /bin/ln, root

4. Glance Configuration
   Configure the options below in glance conf.
   [DEFAULT]
   container_formats = ami,ari,aki,bare,ovf,docker

How to use Docker in OpenStack

Below is an example in how to use the Docker image in OpenStack. You can create a custom Docker image and upload the image by using below commands.

1. Search a docker image available in Docker public registry
   # docker search $image_name

2. Pull the image
   # docker pull $tags/$image_name

3. Save the image and register it in Glance
   # docker save #tags/#image_name | glance image-create --is-public=True --container-format=docker --disk-format=raw --name $tags/$image_name

4. Boot the instance using Docker image
   # nova boot --image “samalba/hipache” --flavor m1.tiny test

5. Check the instance is booted
   # nova list

6. Check the instance in Docker
   # docker ps

BUILD A CUSTOM DOCKER IMAGE

You may want to build your own Docker image for OpenStack. Docker can build an image automatically by reading the instructions from a Dockerfile. A Dockerfile is a text document that contains all the commands you would like to execute in the image. By calling “docker build” command, you can build your own image.

How to Build Docker Image

# sudo docker build -t $tags/$image_name /path/to/dockerfiledir

Note: Docker will re-use the intermediate images, accelerating the building of an image significantly. Please make sure all the required commands are at the top of the Dockerfile. For those changeable commands, please put them at the bottom of the Dockerfile. This method can help in building the image by using cache to run the usual commands.

For a Dockerfile usage, please refer to http://docs.docker.com/reference/builder/

For a complete Dockerfile example, please check the reference.

TROUBLE SHOOTING

1. The number of vCPU is always “1” for every computing node.
   Solution:
   There is a different concept between Docker and Hypervisor. Please update the source code below in the nova docker driver.
   stats = {
       'vcpus' : 1,  # Change the number to apply the number of vCPU
       'vcpus_used': 0,
       ...
   }

2. When Remotely running the command “ls .ssh/authorized_keys” -fails during starting the instances.
   This command is for cloud-init to generating authorized_keys in the instance. But Docker cannot support this feature. For this situation, please MARK the command in “/usr/lib/python2.7/site-packages/sahara/service/engine.py”

3. Remote login failed by using private key.
   Modify _ssh.connect(host, username=username, password=”xxx”, sock=proxy) in “/usr/lib/python2.7/site-packages/sahara/utils/ssh_remote.py”
   Please make sure the password is also set for Docker image.

4. When you remotely run the command “sudo hostname” it fails.
   Docker cannot support modifying the hostname before Docker v1.2. Please upgrade Docker to v1.2 or later.
   Workaround: There is a workaround to customize the host file in Docker manually. Please refer to: http://jasonincode.com/customizing-hosts-file-in-docker/#.VFl1DPmUdZu

5. When you remotely run the command “sudo mv etc-host /etc/hosts” it fails.
   Docker cannot support modifying /etc/hosts before DOcker v1.2. Please upgrade Docker to v1.2 or later.

6. After upgrade to Docker v1.2, “sudo mv etc-host /etc/hosts” responds the device is busy.

   Please use “sudo cp etc-hosts /etc/hosts” to replace the command “sudo mv etc-hosts /etc/hosts” in sahara/service/engine.py of Sahara source code. Another way to do is to set the hosts manually when all the instances launched.

7. My instances cannot reach each other.
   Please make sure all the settings are fine in /etc/hosts and proxy variables: http_proxy, https_proxy, and no_proxy

8. Could not open session when run the command “service cloudera-scm-server-db”
   By default Docker cannot allow the authorization to create the Database using the command. To fix, add the “Privileged” parameter in nova docker driver and set it to True to solve the issue.
   Workaround: A workaround is to modify “/etc/security/limits.d/xxx.conf” after the instance launched and set the value from “hard”/”-” to “soft” to avoid the issue.

9. How to check if CM can be reachable?
   Use the command:
   # curl -X GET -u "admin:admin" -l http://$cm_host:7180/api/v7/tools/echo?message=hello

10. CM response “Connection refused”.
    Please check the firewall has been passed for Docker.
    # iptables -t nat -A OUTPUT -j DOCKER

11. The log response ApiException:{}(error 500)
    Add extra time in sahara/plugins/cdh/deploy.py of Sahara source code, default is 300 secs for timeout. Another root cause could be proxy issue. Please make sure you have correct settings for http_proxy, https_proxy, and no_proxy in the environment.

12. There are several ports that cannot be accessed when starting the Cluster.
    Please expose all the necessary ports in Dockerfile.
    Or you can add “Publish-all-ports” parameter in client.py of nova docker driver source code, please make sure set it to true.

13. There is no storage space in Data Node.
    By default, Docker uses a 10GB Root Disk and a reserved space also needs to be set for non-DFS usage in CM.
    Please change the parameter when you launch Docker binary. For more information, please refer to https://github.com/snitm/docker/blob/master/daemon/graphdriver/devmapper/README.md
    Another way to do is to set reserved space to smaller in CM HDFS Configuration.

14. My instance cannot access files for swift package in Sahara.

    Please make sure your instances can reach the internet at first.

    Or you can set up your own site (like ftp or nfs) to get the necessary files. Please also remember to change the swift package url in node group templates of Sahara.

15. There is no cloudera-scm-agent running on the host.
    Sometimes cloudera-scm-agent may not started automatically or get an error when starting. Please restart the service manually by using the command: “sudo service cloudera-scm-agent restart”

16. Docker cannot connect to proxy.
    Please use HTTP_PROXY when you launch Docker process in the background, for example: “sudo HTTP_PROXY=xxx ./docker -d &”. Then you create the Docker image by using this process with HTTP_PROXY.

17. Container cannot start in other Computing Nodes except the Controller?
    Docker Image must be copied to individual Computing Nodes manually. Please also don’t forget to register the image to Docker register and using “docker images” to confirm the image is existing in the Docker.