After my last blogpost about MAC, I felt bad for preaching water and drinking wine: I set up this blog in 2021 on an Ubuntu 20.04 LTS (🤮) with an nginx from the official Ubuntu repositories, and then didn’t do much to it except for the obligatory updates. What was already a bit backworldish in 2021 is a bit embarrassing in 2023. Since I’m planning to host more on the internet than this blog, I really wanted to look into containers. Of course, I’ve already fired up docker containers and host a few services at home in LXCs on my Proxmox server, but I really don’t know much about the subject.
I wanted to do things right and secure from the beginning, so I decided to use podman instead of docker. While I was learning more about docker and trying a few things, I noticed that the docker daemon always runs as root. You can say that this might be a bit hair-splitting and sensitive, but I don’t like it. And since there is an alternative with podman, whose USP is to be daemonless, the decision was not hard for me. To fully enjoy the security benefits of podman, it was clear to me that I should choose a distribution with good SELinux support.
As a passionate distrohopper I am of course happy to try a new distribution. Actually, my default choice would have been Debian 12, because by now all my other servers and LXCs are running this OS and I feel comfortable there. But for this job I think AlmaLinux is the best choice. Since the best distributions for a great out-of-the-box SELinux support are imho RHEL, Fedora and Gentoo, I wanted to use the most stable alternative. Since I don’t own any RHEL licenses I had to decide between AlmaLinux and Rocky Linux. Both are bug-by-bug-compatible with RHEL, so it doesn’t really matter. But as the Rocky Enterprise Software Foundation (RESF) is privately owned for-profit by one single person, I decided to go with AlmaLinux.
Because RHEL or AlmaLinux and podman are completely new territory for me, I decided to document my setup (more for myself). But maybe it helps someone, who knows?
Initial setup
After spinning up a fresh AlmaLinux 9.2 image in the Hetzner cloud, it’s time to install some packages. But first we have to enable – which was new to me – Extra Packages for Enterprise Linux (EPEL) to get all the packages we (or I) want:
dnf install epel-release
dnf config-manager --set-enabled crb
dnf update
Now that we have enabled EPEL it’s time to install a bunch of packages:
dnf install vim policycoreutils-python-utils util-linux-user dnf-automatic
I also installed some other packages like tmux, fish or net-utils, but ymmv. We’ll need policycoreutils-python-utils to run semanage, util-linux-user to do crazy things like chsh and dnf-automatic to update our server automatically. For that we need to enable a systemd timer:
systemctl enable --now dnf-automatic-install.timer
Now it’s time to add a user and grant them sudo rights. Interestingly adduser on Alma is different from adduser on Debian, so we have to set the password afterwards and not interactively. It still creates the home directory of the new user, so it is not an useradd alias:
adduser jerry
usermod -aG wheel jerry
passwd jerry
Configuring sshd
Now it’s time to ssh-copy-id your public key for the newly created user. After that I like to change the sshd_config. Alma has pretty good defaults (e.g. ‘PasswordAuthentication no’), but I highly recommed to set ‘PermitRootLogin no’ aswell. I also disabled all the other things I will never use, like X11Forwarding and limited the allowed users with ‘AllowUsers jerry’ to my sudo user. In addition I am one of those persons who like to change the ssh port to a non-standard port to avoid cluttered log files. On an SELinux enabled system you’ll have to inform SELinux about that change:
semanage port -a -t ssh_port_t -p tcp 1337
Setting SELinux to ’enforcing’
SELinux enabled? Yeah, kind-of. It’s set to permissive by default. So quickly edit /etc/selinux/config with the text editor of choice, set ‘SELINUX=enforcing’ and reboot.
Changing the primary IPv6 on Hetzner cloud
The official Hetzner docs did not really help me, because they suggest to edit /etc/NetworkManager/system-connections/cloud-init-eth0.nmconnection which does not exist on my system, the whole system-connections directory was empty. Luckily I was able to change IP via nmcli:
sudo nmcli connection edit "System eth0"
nmcli> set ipv6.addresses 2001:db8::fefe/64
nmcli> save persistent
sudo systemctl restart NetworkManager.service
Podman setup
To run podman completely rootless we’ll need the following packages:
sudo dnf install podman slirp4netns fuse-overlayfs
Lets go completely paranoid and create an unprivileged user which is not in the wheel group. We’ll also enable lingering for that user, so they will be able to run processes without being logged in:
sudo adduser pottmeister
sudo loginctl enable-linger pottmeister
The newly created user doesn’t even need a password, we can su into their shell:
sudo su - pottmeister
Since it’s an unprivileged user, we have to adjust the unpriviliged ports – it feels wrong, but it’s the current best practice and seems to be okay on a SELinux enabled single user server behind a firewall:
sudo sysctl net.ipv4.ip_unprivileged_port_start=80
To download images from Docker Hub we need to login first, afterwards we can safely pull images:
podman login docker.io
podman pull docker.io/caddy
To use caddy efficently, we need some volumes for the config, the certificate data and the webroot of course. I searched for best practices, but did not stumple upon anything. I thought about putting everything into /var, I also have seen /var/lib/containers/exported_volumes/ in the wild. If there is a standard, please hmu, I want to know. For simplicity I decided to use the unprivileged user’s home directory, because its already there and owned by them. After creating the respective directories and uploading my static hugo site to the webroot and writing a very small caddyfile:
outpost.bz {
root * /var/www
encode zstd gzip
file_server
}
I was able to spin up my blog like this:
podman run --name caddy-prod --rm -p 80:80 -p 443:443 -v /home/pottmeister/podman/caddy/:/etc/caddy:Z -v /home/pottmeister/podman/webroot:/var/www:Z -v /home/pottmeister/podman/caddy_data:/data:Z -v /home/pottmeister/podman/caddy_config:/config:Z docker.io/caddy
What does this command do? It creates a pod (or container) named caddy-prod and sets it to be removed after it stopped with the –rm flag. It binds the container’s port 80 and 443 to the respective ports on the host OS (the first value is the host’s port) and the -v flag maps directories of the host OS to the container. The :Z suffix tells SELinux to label those directories accordingly so the container is allowed to access those volumes. The last argument docker.io/caddy specifies the container image to be used. So we could easily run this command in tmux now, detach and sit back. But that’s stupid, when systemd is available and we also don’t want to log into the unprivileged user all the time.
Writing a systemd unit file
To easily start and stop the container I decided to write a very simple systemd unit file, so I created it as /etc/systemd/system/podman-caddy-prod.service This is how it looks like:
[Unit]
Description=Running the rootless caddy container with podman
After=network.target
[Service]
User=pottmeister
ExecStart=/usr/bin/podman run --name caddy-prod --rm -p 80:80 -p 443:443 -v /home/pottmeister/podman/caddy/:/etc/caddy:Z -v /home/pottmeister/podman/webroot:/var/www:Z -v /home/pottmeister/podman/caddy_data:/data:Z -v /home/pottmeister/podman/caddy_config:/config:Z docker.io/caddy
Restart=always
[Install]
WantedBy=multi-user.target
Now we can enable it and start and stop the service like we wish:
sudo systemctl enable podman-caddy-prod
Final thoughts
If I did anything wrong or things could be achieved easier and nicer, I appreciate feedback. Please write me an email to any address of this domain, I’m always happy if I can learn something!