Problem
The current docker setup works for small systems like cygnus or pyxis. For everything else, it is at a point where it isn't nicely maintainable.
Outline is set up by having manually converted their docker-compose.yml into single docker_container resources. During updates we have only ever updated the image tags and never checked if there were architectural changes to the docker-compose.yml that would need to be reflected in the docker_container resources.
CodeZero and SigNoz are set up using the docker_compose resource. This has its own problems with needing to vendor the compose files into the repository and merging it with overrides. While those points are not that problematic, the .env handling is. We vendor the default .env and supply a second override.env which is generated by Terraform. To not have constant drift, the override env file is generated by a provisioner and replaced on any change. This however requires that the pipeline creates an empty override env file at the same location before terraform runs because the docker_compose resource will raise an error if it is missing. The docker_compose resource also doesn't detect drift between the currently running containers and the desired state. It only performs changes if the configuration changes.
For CodeZero we are vendoring the docker-compose.yml and .env with vendir. This works with a single environment, but once we setup additional environments, it will be difficult to have multiple environments on different versions.
Planned solution
Migrate to k8s clusters. Plan is to use k3s as lightweight single-node clusters which we can extend with more nodes when needed.
FluxCD will be used as reconciliation engine. Instead of full GitOps, Terraform will manage the Flux CRDs as kubernetes_manifest resources.
Problem
The current docker setup works for small systems like cygnus or pyxis. For everything else, it is at a point where it isn't nicely maintainable.
Outline is set up by having manually converted their
docker-compose.ymlinto singledocker_containerresources. During updates we have only ever updated the image tags and never checked if there were architectural changes to thedocker-compose.ymlthat would need to be reflected in thedocker_containerresources.CodeZero and SigNoz are set up using the
docker_composeresource. This has its own problems with needing to vendor the compose files into the repository and merging it with overrides. While those points are not that problematic, the.envhandling is. We vendor the default.envand supply a secondoverride.envwhich is generated by Terraform. To not have constant drift, the override env file is generated by a provisioner and replaced on any change. This however requires that the pipeline creates an empty override env file at the same location before terraform runs because thedocker_composeresource will raise an error if it is missing. Thedocker_composeresource also doesn't detect drift between the currently running containers and the desired state. It only performs changes if the configuration changes.For CodeZero we are vendoring the
docker-compose.ymland.envwith vendir. This works with a single environment, but once we setup additional environments, it will be difficult to have multiple environments on different versions.Planned solution
Migrate to k8s clusters. Plan is to use k3s as lightweight single-node clusters which we can extend with more nodes when needed.
FluxCD will be used as reconciliation engine. Instead of full GitOps, Terraform will manage the Flux CRDs as
kubernetes_manifestresources.