Fixing /var/lib/docker/containers too large issue

Today, some staging deployments failed and I found out one of the servers has run out of disk space, where /var/lib/docker/containers had grown unexpectedly large because of large log files.

Finding the largest folders:

1# start with /
2sudo du -h -x --max-depth=1 / | sort -hr
3
4# then go into largest one one by one
5sudo du -h -x --max-depth=1 /var | sort -hr
6sudo du -h -x --max-depth=1 /var/lib | sort -hr
7sudo du -h -x --max-depth=1 /var/lib/docker | sort -hr

Then, I found /var/lib/docker/containers was the culprit.

This server ran multiple docker containers, none of which were particularly disk-intensive. GPT suggested this could mostly be related to large log file size. To check:

1sudo du -sh /var/lib/docker/containers/*/*.log

Sure enought, there was a huge 140GB log file from clickhouse.

To truncate all log files,

1sudo truncate -s 0 /var/lib/docker/containers/*/*.log

To limit the log file size, you can add a /etc/docker/daemon.json file with the following container to use JSON logging with a limit.

1{
2  "log-driver": "json-file",
3  "log-opts": {
4    "max-size": "10m",
5    "max-file": "3"
6  }
7}

Make sure to restart docker (sudo systemctl restart docker) to apply changes.