Workaround for subdirectory limit on ext3 filesystem

Just keep in mind that this is a temporary workaround for a problem that should be addressed in a more robust fashion. So on the ext3 filesystem there is a subdirectory limit of 32,000. The problem is that when you reach this limit and you try to add another subdirectory you get a programmer type error: “Too many links”. If you aren’t at the command line and calling mkdir directly this can be very cryptic bubbling up in your programs stack trace. What it means is that there are too many hard links(directories or files) in the directory.

It is a real show stopper, especially if it is the directory to which upload all over your website’s user generated content. Here is a quick workaround until you can solve the problem more gracefully. The limit is for hard links and not soft links (symbolic links). So, what you want to do is:

  1. Create an overflow directory.
  2. Move all of your current subdirectories from the current directory to the overflow directory.
  3. Create a symlink for each subdirectory in the current directory pointing the directory in the overflow directory.

Of course you won’t want to do this by hand, so you might want to whip up a script to do this. Mine was only a few lines of ruby (also note that it was made easier by the fact that the directories were sequential numbers):

  1.     require ‘fileutils’
  2.    old_dir = "/village/data_file/content"
  3.    new_dir = "/village/data_file/overflow"
  4.  
  5.    (0..41671).each do |index|
  6.      dir = File.join(old_dir, index.to_s)
  7.      if File.exist?(dir)
  8.        FileUtils.mv(dir, new_dir)
  9.        FileUtils.ln_sf(File.join(new_dir, index.to_s), dir)
  10.      end
  11.    end

One Response to “Workaround for subdirectory limit on ext3 filesystem”

  1. thegioraproject.com » Blog Archive » Image attachments in grails using ImageMagick Says:

    […] Workaround for subdirectory limit on ext3 filesystem […]

Leave a Reply