Testing custom Github Actions relying on KinD

It has been a while since I’ve written an article here. Last few months have been busy.

I’m still working on my “Elixir and Kubernetes : A love story” series, but recently, I have been prioritizing automating the deployment of Kubirds on our infrastructure.

🚀 Deploying, the GitOps way

We have a few repositories to do this:

• cloud-infra : containing our Terraform manifests to create the Kubernetes cluster and DNS records
• k8s-cluster-state : containing our klifter project to deploy our Helm charts to the cluster
• charts : containing our Helm repository

Everything is automated with Github Actions.

klifter and klander are homemade tools to provide Continuous Deployment and Reconciliation of our Kubernetes cluster. But, using them within a Github workflow was quite tricky.

This is why we developed custom Github Actions to simplify this:

- uses: datapio/actions/klifter@main
  with:
    path: path/to/klifter-project
    kubeconfig: path/to/kubeconfig.yml
- uses: datapio/actions/klander@main
  with:
    kubeconfig: path/to/kubeconfig.yml

🧪 Testing our Github Actions

The next tricky part was testing those actions.

We started by creating a Kubernetes cluster with the engineerd/setup-kind Github Action, which creates the cluster the KinD (short for “Kubernetes in Docker”). It edits the current user’s kubeconfig automatically.

The problem:

The kubeconfig points to 127.0.0.1:<random port>, which in turns route trafic to the API Server within the Docker container deployed by KinD.

Because our custom Github Action is based on a Docker container as well, we can’t just use the kubeconfig as-is. Inside the Docker container, there is no API Server on 127.0.0.1.

The solution:

We need to edit the kubeconfig in order to point to the Github’s Docker host instead. The following script allows us to get the IP address of this host:

Then, in our workflow, before running the klifter or klander actions, we modify the kubeconfig :

Now, our actions can use the new kubeconfig to connect to the KinD cluster.

The next problem:

We are now getting a Connection Refused error when running kubectl commands.

This is because the KinD cluster’s API Server is configured to listen on 127.0.0.1, not the Docker host IP address.

The solution:

Fortunately, KinD can take a configuration file as input before creating the cluster, allowing us to configure the IP address of the API Server:

We save this configuration to .github/config/kind.yml.in and then generate the final configuration using our get-docker-host.sh script:

Conclusion

Using this method, we can now create a new empty Kubernetes cluster for our test suite, and run Docker images (either with docker run, or as a custom Github Action) interacting with this cluster.

To see a complete example, you can see the workflow we use here.