Workaround for subdirectory limit on ext3 filesystem

Just keep in mind that this is a temporary workaround for a problem that should be addressed in a more robust fashion. So on the ext3 filesystem there is a subdirectory limit of 32,000. The problem is that when you reach this limit and you try to add another subdirectory you get a programmer type error: “Too many links”. If you aren’t at the command line and calling mkdir directly this can be very cryptic bubbling up in your programs stack trace. What it means is that there are too many hard links(directories or files) in the directory.

It is a real show stopper, especially if it is the directory to which upload all over your website’s user generated content. Here is a quick workaround until you can solve the problem more gracefully. The limit is for hard links and not soft links (symbolic links). So, what you want to do is:

  1. Create an overflow directory.
  2. Move all of your current subdirectories from the current directory to the overflow directory.
  3. Create a symlink for each subdirectory in the current directory pointing the directory in the overflow directory.

Of course you won’t want to do this by hand, so you might want to whip up a script to do this. Mine was only a few lines of ruby (also note that it was made easier by the fact that the directories were sequential numbers):

  1.     require ‘fileutils’
  2.    old_dir = "/village/data_file/content"
  3.    new_dir = "/village/data_file/overflow"
  4.  
  5.    (0..41671).each do |index|
  6.      dir = File.join(old_dir, index.to_s)
  7.      if File.exist?(dir)
  8.        FileUtils.mv(dir, new_dir)
  9.        FileUtils.ln_sf(File.join(new_dir, index.to_s), dir)
  10.      end
  11.    end
March 2nd, 2008 by Kevin Burke linux1 Comment

Piping grep to rm

I discovered a neat trick to deleting folders/files based on a pattern. Let’s say you have a directory structure like this:

/test
/test/aaaa
/test/bbbb
/test/cccc
/test/ddd
/test/eee
/test/fff

You’ve decided that any folder with a’s, b’s, or c’s has to go. You definitely don’t want to delete them all by hand; especially if there are a couple thousand of them. The first thing you need to do is find the directories that you want to remove:

  1. ls | grep "[a-c]"

Which produces the following:
aaaa
bbbb
cccc

Great! Now what? All you need to do is pipe that to rm using xargs:

  1. ls | grep "[a-c]" | xargs rm -rf

All files that match the given pattern will be deleted. But, be careful that the pattern only matches the files you would like to delete. Run ls | grep first before you pipe anything to rm, unless you like trying to restore deleted files.

Technorati :

February 28th, 2008 by Kevin Burke linux1 Comment