Lately I’ve been spending a lot of time working on my recently built swarm consisting of two Odroid-C1’s, an Odroid-XU4, and a NanoPC-T1.
I was using Shipyard as a management interface and everything worked OK, but it didn’t have any automated failover or high-availability solution. If one of the devices went down, the containers running on that device would die and wait for the device to come back up to restart. You could scale the containers, but that doesn’t work for all servers (like IRC and Murmur). I wanted to tinker and see if I could get around that limitation.
I had another issue that my persistent storage for the containers were directly connected to the Odroid-XU4 for USB 3 support, and consisted of a 3x3TB BTRFS Raid 1 resulting in about 4.5TB usable space. The problem with this is it makes the XU4 a single point of failure for the entire swarm, if it goes down, the containers lose access to their data.
I began working on solutions to resolve both of these issues to try and build an (almost) completely self-contained and truly HA/Fault-Tolerant Arm environment for my servers. When I started, Docker 1.12-rc1 or 2 had just come out and I hadn’t heard about it yet, so almost all of the documents online when looking for HA Docker Clusters resulted in the two big boys right now: Marathon on Mesos (Apache), and Kubernetes (Google).
Of course, there are tons of installation guide’s for these, but many of them were for older versions and I am adding a complicated layer of running them on ARM devices instead of typical hardware. However, there was install guides for Arm, for both of them, and ultimately decided to give Kubernetes a shot.
After a few days of work I did have Kubernetes up and running, but I had various problems with it. If a node was rebooted, when it came back up and joined the swarm the DNS wasn’t working right, and then had an issue where I rebooted all the devices (including master) and when the master came back up, before reconnecting to the other members it already decided it was the only node available and tried starting *all* of my pods on itself, resulting in the device it was on hanging and running of out CPU/Memory. Lesson learned there.
I cleaned up all my pod’s, and was about to try again when I started coming across articles that docker was baking in this capability to the new 1.12 release. Since I was running Arch Linux Arm at the time on all my devices, I proceeded to go through and compile my own binaries for the new Docker 1.12 on the devices (Note: No longer necessary. 1.12 is in the repo’s). I had to manually add swap space for the C1’s to complete the build, and the XU4 completed on it’s own. I have since found dphys-swapfile which is specifically for this, looks like it was added for Raspberry Pi’s but works on any device if it’s in the repositories.
The first issue I came across using Docker 1.12 on the Odroid’s was kernel modules were missing from the default Arch Linux packages for this device. So I grabbed the sources, and used the Pi Cross Compiler to add missing modules, like vxlan. Unfortunately, even after all the drivers were added to the kernel, it is still a 3.10.x based. The swarm seems to work, they all create and distribute the containers as necessary, but the overlay network is unable to route traffic between the devices. The only errors I can find in the logs relate to this github report, but he’s running an entirely different environment. There also seems to be an issue promoting multiple managers.
$ docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 79byib1ccr3az9p9t4xdzrten c1backup Ready Active 9upjydbkp4gkmm5kpfcbo4i0t c1primary Ready Active c1ff633ccxwlv2z9sew9kuc43 * piprimary Ready Active Leader chb1ddqz9bsxfuciv5kfizq3v pibackup Ready Active cpujpidfsjdsw4w72ceiyrj5a odroidxu4 Ready Active
So, I decided to buy myself a couple Rpi3’s to get up and running and start comparing between them to see what’s missing. So now I’m running 2xRpi 3, 2xOdroid-C1 and 1xOdroid-XU4 in my little swarm. I have temporarily benched the NanoPC-T1, as I am having even more kernel problems with that one than the Odroid’s. I reinstalled all devices using HypriotOS, as they have roots for both the C1 and the XU4. I am using v0.2.4 and v0.2.1 respectively for those devices. Unfortunately, even with HypriotOS and finding these docker builds for debian ARM, I still had to add additional kernel modules, and I still had the same problem. So then I started down the rabbit hole of trying to compile my own, newer, 4.x kernels for the Odroid’s. Supposedly, there is experimental “mainline” support for both the C1 and XU4 in 4.4+ kernels.
Unfortunately, at this point in time, I’ve only gotten the XU4 to actually boot into a 4.6 kernel. The steps at the linked page basically worked for the XU4. The C1’s took several days of effort before I finally got them to even boot and output to the USB Serial, but they hang saying “waiting on root device.” This is not uncommon it seems. This problem is, as-yet, unresolved. It’s quite unfortunate these third party vendors are making better, more powerful devices than the Rpi but have proprietary components making mainline kernel support difficult.
The Odroid-XU4 is running kernel 4.6, but still doesn’t work perfect. It is better than the C1’s, the Pi’s are able to route traffic to and from the XU4 OK, and the DNS for the various services resolve properly on any container that launches on the Pi’s or the XU4, but the XU4 doesn’t seem to route properly to the Pi’s. I’m hopeful that this problem can be fixed.
To combat the storage problem, I’ve moved my data to GlusterFS. I purchased some stackable USB enclosures and now have 1x4TB, 3x3TB drives in a replicated configuration plugged into 4 of the 5 hosts. Gives me about 6TB usable, with 2 copies of all data in case any single server goes offline.
So, that’s the current status of my Hybrid Arm-based Docker Swarm. All 5 nodes are connected to the cluster, but any containers that launch on either Odroid-C1 cannot communicate with other containers in the swarm, but containers running on either Raspberry Pi, or the XU4, work (mostly) as expected. I am currently trying to figure out Label’s and Constraints with docker as I can then make sure any containers that require communication between each other fall on devices with that connectivity.
Once I can figure out how to fix the Odroid-C1’s or get a working 4.x kernel, I’ll post the config and updates, hopefully a guide. It is extremely close, there’s just something small, I’m sure, missing to make these work 100%. Altogether, I’m extremely close to having a fully fault tolerant, scalable, Arm cluster environment.
$ docker service ls ID NAME REPLICAS IMAGE COMMAND 0d15xk8nx6lx haproxy 1/1 haproxy:v2 /haproxy-start 3b5y481sdxj0 mail 1/1 mail:v3 /usr/local/bin/start-mailserver.sh 3ueqqlueg3vn inspircd 1/1 irc:v3 /inspircd/bin/inspircd --runasroot --nofork 6pf43r1rbjri znc 1/1 znc:v3 /entrypoint.sh 7yhxglcud90j mariadb 1/1 mariadb:v1 /usr/bin/mysqld_safe 8ztaod3lqz0x murmur 1/1 murmur:v2 /start.sh 9t1143fa2t7x blog 1/1 blog:v3 /entrypoint.sh apache2-foreground bgvkt4ynfjwl web 1/1 nginx:v3 /start.sh f3m4gg0kf3gf anope 1/1 anope:v1 /opt/services/bin/services -n`