Troubleshooting disk space issues
Depending on your server and disk size, you may run out of disk space - the following steps will help you troubleshoot such issues.
- How much disk space is your server using?
- Is your /tmp mount full?
- Which files are taking most disk space?
- Are your log files using the most disk space?
- Do you need more disk space?
Use the following command to check the current disk space used by your server:
The output of this command may look like the following:
$ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 20G 8.3G 11G 45% / none 4.0K 0 4.0K 0% /sys/fs/cgroup udev 3.7G 12K 3.7G 1% /dev tmpfs 748M 424K 748M 1% /run none 5.0M 0 5.0M 0% /run/lock none 3.7G 0 3.7G 0% /run/shm none 100M 0 100M 0% /run/user /dev/xvdb 30G 45M 28G 1% /mnt
The first filesystem is the main one - it is mounted at
/. As you can see, this filesystem is currently at 45% usage. You may want to address this issue if this percentage is higher.
If your server is on AWS and has a 7.8 GB primary filesystem, you are likely running on an old server (created with the default AWS settings). Any new AWS servers are created with a minimum of a 20 GB disk. To upgrade, see the Do you need more disk space? point below.
Some process require writing to your /tmp mount, which can become full depending on your processes. This is done as a protection against low disk space, whereby some daemons automatically shadow the /tmp directory with a RAM disk if the root partition runs out of disk space.
To identify this, simply run the
df -h command from above, and the output would look something like this:
$ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 20G 6.4G 13G 34% / /dev/xvdb 15G 39M 15G 1% /mnt overflow 1.0M 1.0M 0 100% /tmp
To remove /tmp mount, simply run:
sudo umount -l /tmp/
You can also permanently disable this feature by running:
echo 'MINTMPKB=0' > /etc/default/mountoverflowtmp
You can use the following command to determine which files and folders are using the most disk space:
sudo ncdu /
This will scan through your disk and return a detailed breakdown of disk usage by folders and files on your server. You can use the arrow keys to move between folders to determine exactly where the most disk space is being used, and if this is actually necessary. This will help guide you as to where to remove unnecessary files.
The result of the previous investigation may indicate that your logs are taking most of the available disk space. If so, you can either change the level of logging that your application outputs or increase the speed of log rotation on the server. Log rotation moves, compresses and deletes old log files according to a schedule you can determine.
The log rotation settings can be customized in
/etc/cloud66/logrotate/*.conf. If your logs aren't to blame for your low disk space, move onto the next step.
Depending on the size of the server and disk as well as the requirements of your application, you may need to use a bigger server, or if possible attach a larger disk. Some cloud providers allow you to change the disk size from their cloud dashboard. You can also scale up a new server and specify a
root_disk_size value in your manifest file. Once the scale up is complete, you can remove the old server.