Automating Arc-Enabled microk8s with flux setup on ubuntu edge devices.

Kubernetes is popular for running many workloads on cloud and on-premise, but there are lots of use cases emerging for running your solutions with Kubernetes on edge devices to enjoy the benefits like low latency, lesser network disruptions, automation, cloud-native way of developing, scaling, observability.

As more and more enterprises continue to adopt Kubernetes on edge, it is important to offer DevOps/Platform teams a familiar tooling option to take advantage of benefits offered by Kubernetes in edge computing architectures.

Flux is becoming more relevant in all the gitops scenarios where the deployments are automated from the git repository itself. A deployment in the Edge use case can span up to thousands of devices and flux proves to be a super tool in helping reconcile the latest git state on the edge devices thus providing a touch free deployment mechanism for deploying applications on edge devices.

In this article I’ll explain how you can provision an ubuntu edge device with required components, enable the device with Azure-Arc for manageability from Azure cloud and automate the deployment of your solutions using flux. Azure-Arc will help you manage your distributed edge clusters from one central place. Here is how the overall solution will look like:

Let’s consider a scenario where you have multiple work sites, and each site will have its own edge device to run your solution. We will use this repo as a sample git config repository to deploy a simple busybox application on one or multiple sites.

Pre-Requisites

Here is the list of pre-requisites that you need to have before provisioning the device:

1. Azure Subscription(If you don’t have one you can create a free azure subscription)
2. A service principal with contributor access to your subscription. (Instructions for creating a service principal)

Note:- In the pre-requisite #2 above, note down the values for Application (Client) Id, Teanant Id and Client Secret as you will need those in next section.

3. An ubuntu edge device with latest Az-Cli installed on it.

4. Clone this sample git ops repository and you will need to setup your own private repository from the cloned repo as the script is written to work with private repositories considering the need of enterprises.

Folder structure of sample git-ops repository:

Clusters directory contains the configurations for multiple sites e.g in this case site-1, site-2 and site-n are the microk8s cluster configurations running on 3 different edge devices. Each site folder can have its corresponding yaml configurations for its apps and flux configurations for the continuous deployment of your apps. In the sample repo there is just a demo-app inside site-1 but you can deploy more apps to your cluster by just adding the yaml configurations for your new app.

Note:- Once you add the new app to your cluster site folder it will get automatically deployed to your cluster.

5. Create a github PAT token with repo access. Copy this token as we will need it to configure this in configuration so that flux can access your repo.

Running the provisioning script:

1. Login to your ubuntu edge device.
2. Clone the arc-enabled-microk8s-flux-setup repository.
3. cd arc-enabled-microk8s-flux-setup #cd into the cloned repository.
4. cp .env-template .env #create .env file from template.
5. vim .env #Update the configurations required by the provisioning script.

SITE_NAME=’site-1’              #Name of git config folder containing your site resources. In this demo it should be set to site-1.
AZ_TEANANT_ID=’’          #Azure teanant id.
AZ_SP_ID=’’               #Azure service principal id.
AZ_SP_SECRET=’’           #Azure service principal secret.
AZ_ARC_RESOURCEGROUP=’’   #Azure resource group for your arc-cluster
AZ_ARC_RESOURCEGROUP_LOCATION=’’ #Azure location for your resources
GITOPS_REPO=’’            #Name of the git repository, just the name without the "https://github.com/"
GITOPS_PAT=’’             #Your PAT token
GITOPS_BRANCH=’’          #Your git branch
#Note: GITOPS_* variables will be used by flux to deploy your application.

6. Run the ./setup.sh

Here is what the script does:

Installs microk8s

sudo systemctl start snapd.socket
# Install & set up microk8s
sudo snap install microk8s --classic
# sleep to avoid timing issues
sleep 10
# Check microk8s status
sudo microk8s status --wait-ready

Installs Kubectl

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
# Set up config for kubectl
sudo rm -rf ~/.kube
mkdir ~/.kube
sudo microk8s config > ~/.kube/config

Enables microk8s dns

sudo microk8s enable dns
sleep 5
kubectl wait --for=condition=containersReady pod -l k8s-app=kube-dns -n kube-system

Installs Flux

curl -s https://fluxcd.io/install.sh | sudo bash
. <(flux completion bash)

Setup Flux

rm -rf $HOME/$GITOPS_REPO
git clone https://$GITOPS_PAT@github.com/$GITOPS_REPO $HOME/$GITOPS_REPO
cd $HOME/$GITOPS_REPO
git checkout $GITOPS_BRANCH
kubectl apply -f "clusters/$SITE_NAME/flux-system/flux-system/controller.yaml"
sleep 3
flux create secret git gitops -n flux-system \
--url "https://github.com/$GITOPS_REPO" \
--password "$GITOPS_PAT" \
--username gitops
kubectl apply -k "clusters/$SITE_NAME/flux-system/flux-system"
sleep 5
flux reconcile source git gitops
# Switching Back to Home Directory
cd $HOME

Azure Login

az login --service-principal -u $AZ_SP_ID -p $AZ_SP_SECRET --tenant $AZ_TEANANT_ID

Create Resource Group

if [ $(az group exists --name $AZ_ARC_RESOURCEGROUP) == false ]; then
    az group create --name $AZ_ARC_RESOURCEGROUP --location $AZ_ARC_RESOURCEGROUP_LOCATION --output table    
fi

Arc-Setup

az extension add --name connectedk8s
az extension add -n k8s-extension
az provider register --namespace Microsoft.Kubernetes
az provider register --namespace Microsoft.KubernetesConfiguration
az provider register --namespace Microsoft.ExtendedLocation
az connectedk8s connect --name $SITE_NAME --resource-group $AZ_ARC_RESOURCEGROUP

Generate token to connect from Azure-Arc

kubectl create serviceaccount admin-user
kubectl create clusterrolebinding admin-user-binding --clusterrole cluster-admin --serviceaccount default:admin-user
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: admin-user
  annotations:
    kubernetes.io/service-account.name: admin-user
type: kubernetes.io/service-account-token
EOF
TOKEN=$(kubectl get secret admin-user -o jsonpath='{$.data.token}' | base64 -d | sed $'s/$/\\\n/g')

script.sh takes around 10–15 minutes to execute and once it’s done, you can login to your azure portal and navigate to your resource group and you will see a Kubernetes -Azure Arc cluster with the name of your site created.

Kubernetes Azure-Arc Cluster

At the end of the script output, you will see a token was generated which you can use in the portal to access your cluster resources.

Token generated for Kubernetes azure arc cluster

Busybox app deployed and running in your cluster from site-1>>demo-app directory.

Resources

1. Azure Arc-enabled Kubernetes.
2. For accessing KeyVault secrets from arc-enabled microk8s kubernetes cluster you can use Azure Key Vault Secrets Provider extension.
3. Creating azure service principals
4. Connect an existing Kubernetes cluster to Azure Arc
5. Use GitOps with Flux v2 in Azure Arc-enabled Kubernetes.