Hibernate events plugin v0.3

Nothing new in this release, but it does now work with JDK 5 and 1.4.2! And they all rejoiced….yea

The zip: http://thegioraproject.com/files/grails/plugins/grails-hibernate-events-0.3.zip

March 27th, 2008 by Kevin Burke Uncategorized8 Comments

Image attachments in grails using ImageMagick

I’ll admit it, I used ImageMagick for my image manipulation instead of Java and I’ll show you how I did it.

I tried using the image tools plugin. It’s terrific. It is easy to install and simple to use. I couldn’t ask for anything more. The only problem is that the default interpolation algorithm (nearest neighbor) doesn’t create the greatest images when scaling. This is a JAI problem. So I did some trial and error and found that bicubic interpolation is much better; not great, but better. It was slow. This is a JAI problem. It got even slower when I tried a multi pass approach to scaling the image. It was just too slow for the image quality that it was producing. It seemed I had a problem with JAI.

On a past RoR project I used a plugin that facilitated file uploads on models. It had a built-in thumbnailing feature that used ImageMagick for image processing. I had a decent experience with it, so I decided to give it a shot.

Pros: It is noticeably faster , the image quality is superb, and it supports modifying and saving gifs straight up. You can even scale a jpeg and save it as a gif if you so desired. Note that I wasn’t using mediaLib which speeds up processing, though quality is actually the biggest sticking point.

Con: I’m interfacing ImageMagick using Groovy’s String.execute() method (a wrapper for ProcessBuilder). Not that this is an ImageMagick problem, as there is a JMagick library that uses JNI to interop. However, judging from some reports on the intartubes, there haven’t been too many success stories.

If your still interested here’s how it went down:

1. Installed ImageMagick

Downloaded and installed ImageMagick

2. Installed Hibernate Events Plugin

  1. grails install-plugin http://thegioraproject.com/files/grails/plugins/grails-hibernate-events-0.2.zip

3. Created a generic Attachment Domain class

  1. import grails.util.GrailsUtil
  2. import org.springframework.web.multipart.MultipartFile
  3. import org.codehaus.groovy.grails.commons.ConfigurationHolder
  4.  
  5. class Attachment {
  6.     //Allows us to have spring bind the file automatically. <input name="attachment.file" type="file" />
  7.     MultipartFile file
  8.     //Save file metadata to the database
  9.     String contentType
  10.     String fileName
  11.     Long size    
  12.  
  13.     //Don’t try to save the MultipartFile.  Hibernate will barf all over you.
  14.     static transients = ["file"]
  15.  
  16.     //Location for saving files i.e. /home/me/app_files/attachments/
  17.     def baseDir = { ConfigurationHolder.config.paperclip.files.dir[GrailsUtil.environment] }
  18.  
  19.     //Use Hibernate Events Plugin to save the file after it’s metadata has been saved.
  20.     def afterInsert = {
  21.         writeFile()
  22.     }
  23.  
  24.     //Create a unique directory to store the file based on the saved id. Produces a two deep
  25.     //directory to stave off the 32000 hard link limit on ext3 filesystem.  So the path might look
  26.     //like 0000/0001/filename.extension
  27.     //see: http://thegioraproject.com/2008/03/02/workaround-for-subdirectory-limit-on-ext3-filesystem/
  28.     def directory = {
  29.         def stringId = id.toString()
  30.         def fullId = stringId.padLeft(8, "0")
  31.         [fullId[0..3], fullId[4..7]].join("/")
  32.     }
  33.  
  34.     //Joins the directory and filename to create a path.  Useful for creating links on views: can call
  35.     //attachment.path() to produce href.
  36.     def path = {
  37.         [directory(), fileName].join("/")
  38.     }
  39.  
  40.     //Location on disk to write the file to.
  41.     def absolutePath = {
  42.         [baseDir(), path()].join("/")
  43.     }
  44.  
  45.     //Writes the file to the disk making sure all parent directories are present
  46.     protected void writeFile() {
  47.         def destination = new File(absolutePath())
  48.         destination.mkdirs()
  49.         file.transferTo(destination)
  50.     }
  51.  
  52.     //When Spring binds the file to the object we set the metadata for the file on the object.
  53.     //Eliminates some setter code.
  54.     public void setFile(MultipartFile upload) {
  55.         file = upload
  56.         contentType = upload.contentType
  57.         fileName = upload.originalFilename
  58.         size = upload.size
  59.     }
  60. }

4. Created an Image Domain class

  1. import grails.util.GrailsUtil
  2. import org.codehaus.groovy.grails.commons.ConfigurationHolder
  3.  
  4. class Image extends Attachment {
  5.     //Root directory of the ImageMagick install determined by the environment.  So if we’re using
  6.     //development it will get the config variable paperclip.magick.dir.development from Config.groovy
  7.     def magickDir = { ConfigurationHolder.config.paperclip.magick.dir[GrailsUtil.environment] }    
  8.  
  9.     //After saving make sure we save the original by calling writeFile() on Attachment,
  10.     //then create the thumbnails
  11.     def afterInsert = {
  12.         writeFile()
  13.         createThumbnails()
  14.     }
  15.  
  16.     //Location on disk to write the thumbnail to
  17.     def absoluteThumbnail = {type ->
  18.         [baseDir(), thumbnail(type)].join("/")
  19.     }
  20.  
  21.     //Kind of like path on Attachment.  Allows us to call attachment.thumbnail("300") on the view to
  22.     //produce an img src.
  23.     def thumbnail = {type ->
  24.         def tokens = fileName.tokenize(".")
  25.         def name = [tokens[0..-2].join("."), type, tokens[-1]].join(".")
  26.         [directory(), name].join("/")
  27.     }
  28.  
  29.     //The meat of it all.  It gets a little lost because there isn’t a lot of code.  So we loop over
  30.     //an array to produce two thumbnails.  One will be 100×100, the other will be 300×300.  However
  31.     //ImageMagick will keep the images aspect ratio using the geometry we supplied.  This method will
  32.     //create a string like "/ImageMagick/convert /path/image.jpg -thumbnail 100×100 /path/image.100.jpg"
  33.     //and then execute it as if you were on the command line.  The ImageMagick command will resize and
  34.     //save the image to the supplied dir.  The execute() call returns a process object which we call
  35.     //waitFor on to make sure the processing finishes before we move on.  We can also get the exit code
  36.     //from the process to handle exceptions.
  37.     //More info ongeometry flags and resizing here: http://www.imagemagick.org/Usage/resize/#resize.
  38.     //It is very flexible.
  39.     protected void createThumbnails() {
  40.         [300, 100].each {
  41.             def process = "${magickDir()}/convert ${absolutePath()} -thumbnail ${it}x${it} ${absoluteThumbnail(it)}".execute()
  42.             process.waitFor()
  43.         }
  44.     }
  45. }

Hopefully the comments explain how everything works together. If you have questions, comments or criticism, please let me know. And yes, I feel a little dirty calling another program in a process, but when it works this well, getting a little dirty feels pretty good.

Image comparison (my youngest, Ariana):

ImageMagick
ImageMagick 100px Thumb

ImageTool (JAI)
ImageTool(JAI) 100px Thumb

ImageMagick
arianamagick300.JPG

ImageTool (JAI)
arianaimagetool300.jpg

March 26th, 2008 by Kevin Burke grails6 Comments

Hibernate events plugin v0.2

What, another release already? Yep, we’re releasing another version of the grails hibernate events plugin.  Well, obviously I wasn’t paying attention in grails/hibernate class and it turns out that while it requires a hibernate.cfg.xml to participate in the creation of the SessionFactory, you can still programmatically add event listeners to the SessionFactory after its been created. Big ups to Burt Beckwith for the tip (who, judging by the grails mailing list alone, has every api under the sun memorized!). So here is the hibernate events plugin sans hibernate.cfg.xml:

http://thegioraproject.com/files/grails/plugins/grails-hibernate-events-0.2.zip

And again, any feedback is much appreciated.

March 24th, 2008 by Kevin Burke grails2 Comments

Hibernate events plugin

On a current grails project I had a need to act on my domain classes using hibernate’s persistence lifecycle. However, it turns out that grails only provides three lifecycle methods: beforeInsert, beforeUpdate, beforeDelete. Four if you count onLoad. I was really interested in having an afterInsert method so that I could save a file to the filesystem after it’s metadata had been saved to the database. So I used the hibernate event system to round out the lifecycle methods with seven more hooks:

  • afterInsert
  • afterUpdate
  • afterDelete
  • beforeLoad
  • afterLoad
  • beforeSave
  • afterSave

So if a domain model contains a closure with any of these names they will be called during their turn in the lifecycle. I think they are mostly self explanatory except for before/afterSave. The purpose for these methods is to be called when the persistence call is either an insert or an update. So if you don’t care if the object is being saved for the first time or being updated you can use this method.

Unfortunately the plugin uses a hibernate.cfg.xml, because it is the only way you can participate in the creation of the SessionFactory.

I haven’t decided if I will release the plugin yet. Maybe if there is interest.

Update: Ok, so some interest was expressed. Here it is: http://thegioraproject.com/files/grails/plugins/grails-hibernate-events-0.1.zip. Feedback is always welcome.

Update 2: I’ve added a doc page to the grails wiki.  Not really much is needed, but it could clear things up a bit.  http://docs.codehaus.org/display/GRAILS/Hibernate+Events+Plugin

March 22nd, 2008 by Kevin Burke grails8 Comments

Have God watch over your processes; or How to keep your app running when your not watching

My latest application at work is a Ruby on Rails(RoR) app, that we just deployed recently. Being a RoR app it hasn’t been so pleasant in production. I’m sure that it is due to my lack of experience in deploying RoR apps, but it has been difficult in any case. It is deployed to a cluster of 3 mongrels behind an apache proxy/load balancer. Just this morning the app became unresponsive as one of the mongrel instances was hung and had the cpu pegged. Not hard to fix, killed the processes and restarted the cluster. More of a nuisance really.

The problem, though, is that if this keeps happening (it most likely will), it would be more than just a nuisance. We needed to find a way to monitor the app to make sure it was always running. Luckily there is a product out there that does more than just monitor your app. Luckily I found God ;). From the God website:

“God is an easy to configure, easy to extend monitoring framework written in Ruby.

Keeping your server processes and tasks running should be a simple part of your deployment process. God aims to be the simplest, most powerful monitoring application available.”

It doesn’t lie. It is one of the best pieces of software that I have used in a little while now. I was up and running in about 30 minutes. Most of that time was spent checking out the different options and whatnot. So now if one of the mongrels fails for some reason, God will start it back up. If one of the mongrels is using too much memory, God will restart it. If one of the mongrels is pegging the processor, God will restart it. You can even watch the logs in real time for each individual mongrel instance, and control God from the command line. There is also a great tutorial on loading God on system startup.

If you’re interested make sure you read through the home page for God to see all the great things it is capable of.

Technorati :

March 4th, 2008 by Kevin Burke rails1 Comment

Workaround for subdirectory limit on ext3 filesystem

Just keep in mind that this is a temporary workaround for a problem that should be addressed in a more robust fashion. So on the ext3 filesystem there is a subdirectory limit of 32,000. The problem is that when you reach this limit and you try to add another subdirectory you get a programmer type error: “Too many links”. If you aren’t at the command line and calling mkdir directly this can be very cryptic bubbling up in your programs stack trace. What it means is that there are too many hard links(directories or files) in the directory.

It is a real show stopper, especially if it is the directory to which upload all over your website’s user generated content. Here is a quick workaround until you can solve the problem more gracefully. The limit is for hard links and not soft links (symbolic links). So, what you want to do is:

  1. Create an overflow directory.
  2. Move all of your current subdirectories from the current directory to the overflow directory.
  3. Create a symlink for each subdirectory in the current directory pointing the directory in the overflow directory.

Of course you won’t want to do this by hand, so you might want to whip up a script to do this. Mine was only a few lines of ruby (also note that it was made easier by the fact that the directories were sequential numbers):

  1.     require ‘fileutils’
  2.    old_dir = "/village/data_file/content"
  3.    new_dir = "/village/data_file/overflow"
  4.  
  5.    (0..41671).each do |index|
  6.      dir = File.join(old_dir, index.to_s)
  7.      if File.exist?(dir)
  8.        FileUtils.mv(dir, new_dir)
  9.        FileUtils.ln_sf(File.join(new_dir, index.to_s), dir)
  10.      end
  11.    end
March 2nd, 2008 by Kevin Burke linux1 Comment