On a fine day, the server ran out of hard drives because of … Docker!

Tram Ho

The project is in closed beta, preparing to invite users to try it out. Suddenly one day … received a message from Ms. BrSE: “Honey, please check and help the server to see if there is a problem, the API cannot call”. Quickly SSH onto the server, I immediately got the message no space left on device . But why run out of hard drives so fast, source code has more than 800MB each. After a while of searching, we found the culprit … Docker.

. overview

The main components that take up memory in Docker:

  • Images : Each Docker images takes up from a few tens of megabytes to gigabytes.
  • Containers : Data of applications contained in containers.
  • Local Volumes : The container’s data is mounted on the actual hard drive.
  • Build Cache : Available from Docker version 18.09

To test these parameters, we execute the command:

Hmmm. The total capacity is less than 1GB, so where does the hard drive capacity go?

If your computer becomes full of images, Volumes or Containers, simply delete the unnecessary ones

Images

Delete the Docker images that are not in use

Volumes

Similar to volumes

Build Cache

Also build cache

2. Find the cause

To find out the cause of full hard drive, we go back to the most basic way is to list out the size of the folders to see which item is taking up the most hard drive.

On Linux, we run the command, the system will list us directories by capacity from low to high:

Immediately capture the 14GB folder (partition / on the server 15GB)

So that’s because of Docker. But what do these folders save?

  • /var/lib/docker/containers : Stores logs of containers (saved as JSON by default)

By default, Docker will not limit the size of the log files in the container, if the server runs for a while, the log file size will gradually increase, possibly to tens or hundreds of GB.

3. Solution

Limit log file size of Docker container, clear log file periodically.

Clear log file

After the truncate is finished:

Periodically clear log files with Logrotate

Logrotate is a system utility that manages automatic log deletion periodically. We will configure with Logrotate to limit the size as well as periodically clear the Docker log file.

Modify the content of the docker-logs file

References

https://medium.com/better-programming/docker-tips-clean-up-your-local-machine-35f370a01a78

https://blog.birkhoff.me/devops-truncate-docker-container-logs-periodically-to-free-up-server-disk-space/

Share the news now

Source : Viblo