Running out of inodes

Our application can no longer deploy to our server - it looks like we’re running out of inodes:

ubuntu@ip-10-234-144-175:/$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda1 524288 519043 5245 99% /

I suspect this could be a build up of log files or something but I’m not sure where to start looking as I don’t know exactly how you configure the servers.

I’ll continue to have a look to see if I can find where the source of the inode usage. Any help or pointers you can provide would be much appreciated.

Sorry to hear that you’re having issues. I’d suggest that you install ncdu, which is a great tool to troubleshoot disk space issues (sudo apt-get install -y ncdu). It launches in the folder it’s run from, so I’d suggest running it from /. It’ll take a bit of time to scan your system, but then you can traverse your file system to find the folders that are using the most disk space.

Please let us know if that helps.

I can see that the issue is in the bundlers gems directory - any advice on cleaning this up?

If you have a lot of gems that are being pulled in from git repos, each gem folder has many file handles in them (this is because git is pretty verbose when it comes to file descriptors).

As to the solution, to tidy up older gem folders, the simplest way for this is to simply run “bundle clean” in your stack path. This will remove any gems from the bundler that are no-longer referenced by your Gemfile.lock.

You can use this deploy hook to clean them up:

production:
    after_rails:
      - snippet: cloud66/gemclean
        target: rails
        execute: true
        apply_during: deploy_only
        run_on: all_servers

Please keep in mind that this will have consequences if you decide to rollback your deployment at some point:

1) Those gems (or branches/refs) may no longer exist, which will mean that your deployment will fail because of missing dependencies 2) Rollbacks will take longer, because the gems need to be fetched again (may not be negligible)

We don’t run this in our default deployment scripts is that this problem only affects a very small subset of users, so it’s not necessary in most cases. You could always run this as a scheduled command ever X days/weeks/months if that is better.

When i do sudo apt-get install -y ncdu I get the following error:

Reading package lists… Done Building dependency tree Reading state information… Done You might want to run ‘apt-get -f install’ to correct these: The following packages have unmet dependencies: linux-headers-virtual : Depends: linux-headers-3.2.0-75-virtual but it is not going to be installed E: Unmet dependencies. Try ‘apt-get -f install’ with no packages (or specify a solution).

You must be logged in to answer this question