Raspberry Pi Cluster

Building a Raspberry Pi Cluster

Credit

Scott Hanselman – The original article that I found for support.

Alex Ellis – Who created the original post that Scott referenced.

Shopping List

Here is a Amazon list showing all of the items that I purchased.  I am also including the static list in case some of the items get discontinued.  I built a 6 node cluster because some of the items were easy to support.  I also found 4 was an easy number to support.

Amazon List

  • Must Haves
    • Raspberry Pi 3
    • GeauxRobot 6-layer Dog Bone
    • GL.iNet GL-MT300N Travel Router
    • D-Link 8-port Unmanaged Switch GO-SW-8G
    • Cat 7 Ethernet Cable 1 foot – 6 color pack
    • USB 2.0 Male to Micro USB 1 foot
    • Anker 60W 10-port USB Wall Charger
    • Samsung 32GB Micro SD card (I did 128 GB because it was on sale for $2 more)
  • Nice To Haves
    • Raspberry Pi 7″ Touch Screen
    • Case for official Raspberry Pi 7″ Touch Screen
    • iPazzPort Wireless Mini Keyboard with Touchpad

The total cost for everything before taxes and shipping was $679.  This obviously is not a cheap hacking project.

Installation Steps (Each Node)

  1. Download Raspbian Stretch – I tried the lite version but could not get the SSH to connect.  The current version at the time of this writing was 11-29-2017 filed under 12-01-2017.
  2. Download Etcher – This will help you flash your SD cards with the Raspbian you just downloaded.
  3. Add a new file to the boot drive – This allows SSH to be enabled on boot.  See release notes from 11-25-2016 regarding this.
    1. Change into SD card drive. Mine was /Volumes/boot/.
    2. Add the empty file – touch ssh.  The file must be name ssh with no extension.
  4. Insert micro SD card into Raspberry Pi.
  5. SSH into Raspberry Pi – You might need to run an nmap scan or visit your router’s admin page to see the ip address of Raspberry Pi if it doesn’t respond to the default machine name.
    1. ssh pi@raspberrypi.
    2. Password: raspberry.
  6. Configure Raspberry Pi.
    1. sudo raspi-config .
    2. Change default password.
      1. Select 1 Change User Password.
    3. Rename machine.
      1. Arrow down and select 2 Network Options.
      2. Select N1 Hostname.
    4. Press esc until you exit Raspberry Pi Configuration.
  7. sudo reboot to set new changes.
  8. Install Docker.
    1. curl -sSL get.docker.com | sh && sudo usermod pi -aG docker.
  9. Disable swap.
    1. sudo dphys-swapfile swapoff && sudo dphys-swapfile uninstall && sudo update-rc.d dphys-swapfile remove.
  10. Edit /boot/cmdline.txt.
    1. sudo nano /boot/cmdline.txt.
    2. Add cgroup_enable=cpuset cgroup_memory=1 to end of the line.
    3. Ctrl-X.
    4. Shift-Y.
  11. Reboot for these settings to take effect – sudo reboot.
  12. Install Kubernetes.
    1. curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt-get update -q && sudo apt-get install -qy kubeadm.

All done.  These steps need to be done for each node.

Creating Supervisor Node

  1. Set your master node with a static IP.
  2. Initialize the master node – sudo kubeadm init --apiserver-advertise-address=192.168.1.201.
  3. IMPORTANT! – Copy the join script that is printed at the end of the previous command.  You will need this later when creating the worker nodes.
  4. Copy and take ownership of admin.conf.
    1. mkdir -p $HOME/.kube.
    2. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config.
    3. sudo chown $(id -u):$(id -g) $HOME/.kube/config.

Creating Worker Node

  1. Run the join script that you copied from the Supervisor Node above.

Setup Networking

  1. Install Weave.
    1. kubectl apply -f https://git.io/weave-kube-1.6.

Check System

  1. kubectl get pods --namespace kube-system.  You should have 1/1, 2/2, and 3/3 for your services.  Everything should also be Running.
  2. kubectl get nodes. All nodes should be in the Ready state.

Deploy Your First Container

  1. nano function.yml
  2. Paste YAML.
    apiVersion: v1
    kind: Service
    metadata:
      name: markdownrender
      labels:
        app: markdownrender
    spec:
      type: NodePort
      ports:
        - port: 8080
          protocol: TCP
          targetPort: 8080
          nodePort: 31118
      selector:
        app: markdownrender
    ---
    apiVersion: apps/v1beta1 # for versions before 1.6.0 use extensions/v1beta1
    kind: Deployment
    metadata:
      name: markdownrender
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            app: markdownrender
        spec:
          containers:
          - name: markdownrender
            image: functions/markdownrender:latest-armhf
            imagePullPolicy: Always
            ports:
            - containerPort: 8080
              protocol: TCP
    
  3. Ctrl-X.
  4. Shift-Y.
  5. kubectl create -f function.yml.
  6. curl -4 http://localhost:31118 -d "# test".

Complete! You now have a running Raspberry Pi Cluster running Kubernetes.

Kubernetes Dashboard

  1. Install kubectl on your local machine – https://kubernetes.io/docs/tasks/tools/install-kubectl
  2. scp your admin.config over to your local machine – scp pi@cluster-supervisor:/home/pi/.kube/config ~/.kube/config
  3. Proxy into your cluster supervisor –kubectl proxy, if everything worked then going to http://127.0.0.1:8001/version will present you with some good information.
  4. Now that your proxy is working you can install the dashboard – kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/alternative/kubernetes-dashboard-arm.yaml
  5. Add the rolebindings – kubectl create clusterrolebinding add-on-cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
  6. Find the kubernetes-dashboard IP – kubectl get svc --namespace kube-system
  7. Create a SSH tunnel to the dashboard UI – ssh -L 8080:10.111.182.181:80 pi@cluster-supervisor.
  8. Check to see if everything worked – http://127.0.0.1:8080.

Refactors and merging projects of Angular 4 into Angular 1.5

Last week I had to sit through a meeting where we were trying to figure out how to merge an Angular 4 app into an Angular 1.5 app.  First off, why?  The original intent was to merge the features of each project into one to create a new product.

Our defense was that we were using the latest, at the time, Angular library and that we had just finished a complete refactor and re-architecture of our entire project that took 2 months.  However, we lost the argument because the other project already had alpha/beta/production users.  What they had failed to do was a refactor, code review, or unit tests for the last year.  They have been working off of prototype code from the very beginning.

After the meeting I asked to have a static code analysis done of both code bases with HP Fortify and SonarQube.  The results were without question in our favor.  Both HP Fortify found 3 errors in our code which turned out to be false positives and SonarQube produced 0 issues, code smells, duplications, etc.  For the other project there were 130 issues with HP Fortify and 29 bugs, 3 vulnerabilities, 231 code smells, and 33% duplication with SonarQube.

Even after the static code analysis was done 4 developers took a look at the code base and it was even uglier than they thought.  Most of them saying they would be embarrassed to have the code base in their portfolio.  The code was hacky and only one developer was working on it for an entire year with no code reviews or merge requests.

Here is the rest of the story.  The project that came up with 0 errors was prototyped with minimal architecting effort, but got to a point after our MVP demo that it needed to be refactored in order to produce a sound and maintainable project.  What the static code analysis was not able to identify was the bad architecting that we originally did.  SonarQube found only 9 issues before our refactor.  I didn’t expect a static code analyzer to find out architecting mistakes, but the amount of difference between our code architecture before and after the refactor was massively different.  That is why we need some code reviews done to ensure that our findings for providing the accurate information to make a decision on our code base merge.

The obvious decision would be to merge our Angular 1.5 app into our Angular 4 app, but there is a business case that might not allow for that timeline, so we have prove our case that our technical solution is a better way forward for the business.

Top Down Control, But Not a Dictator

BLUF: Military leadership is not a ritualistic environment that lines up 100 yards from each other taking musket shots to see who is left standing.

A lot of outsiders of the military think that we live by some sort of dictatorship and do as we are told and don’t ask questions or apply thought to actions.  We are not T-800s just running around with a single mission.

In all leadership education you will find there are leaders and there are followers.  There are times when you need to be a good follower and there are times when you need to be a leader.  Combined Arms Group, once known as Delta Force, is such a lethal force because of the ability for the statement “When in charge, take charge.” to be so impactful.  This group of soldiers allows for individuals to be proactive and accomplish a mission without a command given.  If you see something wrong and you want to fix it, then you have the ability in that environment to take command and ask for tasks to be completed in order to accomplish the final solution.  This can come from the bottom of the food chain.

Here is the difference between a flat organization and a hierarchical organization with great leaders.  A flat organization only seems flat because everyone has the opportunity to make a difference, but there is still an understanding of who is charge of the entire organization.  In a flat organization if you have an individual trying to influence from a lower position then there is a greater opportunity for the said individual to topple the leadership and inflict problems into the organization.

However, you can still have a flat organization where you empower individuals to make their greatest contributions and positively affect the organization.  As a leader you still need to assert your leadership from a position or influence and not authoritative power.  A definition of leadership is influencing individuals to do what you want without threats or maltreatment.

A top down hierarchy is the classic organizational style that will set the stage to accomplish the above leadership style.  It is each individual level of leadership that affects whether you have a flat organization or a dictatorship.  If you can empower individuals at every level to provide an impactful achievement and proactive attitude, then you can provide a culture and organization that can accomplish anything.

The military has always taught a top down leadership style, but you will never see a coup d’etat from the United States military because we do not let those individuals prosper to a level of influence.  The United States military provided me with the best leadership development skills to learn and understand that the lowest person can be so impactful.

Any leader that is not providing an environment to let you do your best efforts is only leading you down a path to failure and career depression.  A leadership should provide you with the ability to learn, advance and even stress out a bit because they are putting you in uncomfortable positions.  As I have said in a previous post, being a leader that provides everything your teammates need and a bit of guidance, then you will be better of for it.

Concept of the Operation (CONOP)

We trying to get your idea sold to upper management or leadership it could come as a big challenge to demonstrate and explain exactly what you want to do or how you are going to achieve.  A concept of the operation, CONOP, is the best hybrid approach that I have found to answer questions quickly and tell someone the “5 W’s”.

A CONOP is a one page document that will answer the overall concept of a problem/solution set at a 10,000′ view.  There is no fine grain detail or planning that takes place in this document.  It is almost like a elevator pitch mixed with words and visual cues to answer the initial questions in order for someone to want to get to the next level; next level being a detailed plan for execution.

In a CONOP you need to answer the “5 W’s” both written and visual to capture both ways that someone understands information.  You should have a diagram or image of what is going to occur in the most important part of the solution.  A mission statement that has the words to answer the “5 W’s” and then extra information that would help identify a potential plan of action or answer some devil’s advocate questioning.

The above image is an example of a CONOP layout with possible information sections. “Main Idea” would be where your diagram or image goes to visual depict your idea.  The other sections are not necessary, but I would highly suggest a minimum of the “Mission Statement”, “Key Tasks”, and “Timeline” sections.

Somehow you need to be concise enough to get your idea in entirety on one page for review by management and leadership to OK further development of the idea.  This document will help capture that idea in order to tell your story with trying to aggressively defend it.

1/3, 2/3 Rule

BLUF: Give your team the time they deserve to accomplish their planning before execution.

During any operation in the military there is a rule of thumb that should be applied to any mission planning set.  The “1/3, 2/3 Rule”.  This rule states that you should only use 1/3 of the planning time for your planning and give 2/3 of the planning time to your subordinates.

For example, if you need to move a logistics package from Tampa, FL to Atlanta, GA and it needs to happen in 3 days,  then you should take 1 day (1/3) of the time to plan out the logistical movement at your level and then give your team 2 days (2/3) to plan out their part at their level.

If you think about this, then you are allowing the most planning time given to your teammates rather than being selfish and hoarding all of that time and everyone else’s time to create my plan and then throwing it at everyone else hours before the execution has to happen and expect a great execution to occur.  Also, if you wait until the end to give your teammates your plan, then your team is sitting around doing nothing and guessing about what actions they need to take to make this a successful execution.  They can only guess if you actually told them that something is coming down for them to get ready for.

This planning technique highlights a few leadership characteristics that I have discussed in previous posts.

  1. Don’t be selfish, be self servant.  Allow your team the most time to plan for mission execution.
  2. Provide a WARNO (Warning Order).  This will answer the 5 W’s for your team to prepare for without knowing the intricate details of the plan.
  3. OODA loop – Don’t let this 1/3, 2/3 Rule confuse to be a hard and fast rule about you do your 1/3 planning first and then they do their 2/3 planning.  This is still an iterative fashion that you should continue to feed information to your team as your receive it.