Testing custom Github Actions relying on KinD
It has been a while since I’ve written an article here. Last few months have been busy.
I’m still working on my “Elixir and Kubernetes : A love story” series, but recently, I have been prioritizing automating the deployment of Kubirds on our infrastructure.
🚀 Deploying, the GitOps way
We have a few repositories to do this:
cloud-infra
: containing our Terraform manifests to create the Kubernetes cluster and DNS recordsk8s-cluster-state
: containing our klifter project to deploy our Helm charts to the clustercharts
: containing our Helm repository
Everything is automated with Github Actions.
klifter and klander are homemade tools to provide Continuous Deployment and Reconciliation of our Kubernetes cluster. But, using them within a Github workflow was quite tricky.
This is why we developed custom Github Actions to simplify this:
- uses: datapio/actions/klifter@main
with:
path: path/to/klifter-project
kubeconfig: path/to/kubeconfig.yml- uses: datapio/actions/klander@main
with:
kubeconfig: path/to/kubeconfig.yml
🧪 Testing our Github Actions
The next tricky part was testing those actions.
We started by creating a Kubernetes cluster with the engineerd/setup-kind Github Action, which creates the cluster the KinD (short for “Kubernetes in Docker”). It edits the current user’s kubeconfig
automatically.
The problem:
The kubeconfig
points to 127.0.0.1:<random port>
, which in turns route trafic to the API Server within the Docker container deployed by KinD.
Because our custom Github Action is based on a Docker container as well, we can’t just use the kubeconfig
as-is. Inside the Docker container, there is no API Server on 127.0.0.1
.
The solution:
We need to edit the kubeconfig
in order to point to the Github’s Docker host instead. The following script allows us to get the IP address of this host:
Then, in our workflow, before running the klifter or klander actions, we modify the kubeconfig
:
Now, our actions can use the new kubeconfig
to connect to the KinD cluster.
The next problem:
We are now getting a Connection Refused
error when running kubectl
commands.
This is because the KinD cluster’s API Server is configured to listen on 127.0.0.1
, not the Docker host IP address.
The solution:
Fortunately, KinD can take a configuration file as input before creating the cluster, allowing us to configure the IP address of the API Server:
We save this configuration to .github/config/kind.yml.in
and then generate the final configuration using our get-docker-host.sh
script:
Conclusion
Using this method, we can now create a new empty Kubernetes cluster for our test suite, and run Docker images (either with docker run
, or as a custom Github Action) interacting with this cluster.
To see a complete example, you can see the workflow we use here.