Custom implicit variables in GSP’s

I really should start every post with “groovy is a really swell language”. You can do some pretty cool things with it.

Anyway, I had a desire to have an variable implicitly available in all of my GSP’s so that I didn’t have to wrap certain blocks in a tag that made the variable available. There were a couple of reasons for this. The first is that if an exception was thrown in the block of code, the stack trace would indicate that it was coming from my custom tag. More importantly however it was a real pain in the tuckus to have to litter my GSP’s with a tag that simply introduced a variable into the scope. After several attempts to hijack the page scope before the page was rendered I settled on the following:

  1. GroovyPage.metaClass.getCurrentUser = {-> ctx.authenzedManager.currentUser() }

As it turns out every page extends GroovyPage which is just a groovy script. So you can add methods using it’s metaClass to add all kinds of functionality. This might be a good way for plugins to add sort of helper methods, since, unlike tags, it will actually return a value and not write it to out. Anyway, by adding getCurrenUser I can now call ${currentUser.whatever} from my GSP. It makes me happy.

Rock over London. Rock on Chicago.

Hibernate events plugin v0.3

Nothing new in this release, but it does now work with JDK 5 and 1.4.2! And they all rejoiced….yea

The zip: http://thegioraproject.com/files/grails/plugins/grails-hibernate-events-0.3.zip

Image attachments in grails using ImageMagick

I’ll admit it, I used ImageMagick for my image manipulation instead of Java and I’ll show you how I did it.

I tried using the image tools plugin. It’s terrific. It is easy to install and simple to use. I couldn’t ask for anything more. The only problem is that the default interpolation algorithm (nearest neighbor) doesn’t create the greatest images when scaling. This is a JAI problem. So I did some trial and error and found that bicubic interpolation is much better; not great, but better. It was slow. This is a JAI problem. It got even slower when I tried a multi pass approach to scaling the image. It was just too slow for the image quality that it was producing. It seemed I had a problem with JAI.

On a past RoR project I used a plugin that facilitated file uploads on models. It had a built-in thumbnailing feature that used ImageMagick for image processing. I had a decent experience with it, so I decided to give it a shot.

Pros: It is noticeably faster , the image quality is superb, and it supports modifying and saving gifs straight up. You can even scale a jpeg and save it as a gif if you so desired. Note that I wasn’t using mediaLib which speeds up processing, though quality is actually the biggest sticking point.

Con: I’m interfacing ImageMagick using Groovy’s String.execute() method (a wrapper for ProcessBuilder). Not that this is an ImageMagick problem, as there is a JMagick library that uses JNI to interop. However, judging from some reports on the intartubes, there haven’t been too many success stories.

If your still interested here’s how it went down:

1. Installed ImageMagick

Downloaded and installed ImageMagick

2. Installed Hibernate Events Plugin

  1. grails install-plugin http://thegioraproject.com/files/grails/plugins/grails-hibernate-events-0.2.zip

3. Created a generic Attachment Domain class

  1. import grails.util.GrailsUtil
  2. import org.springframework.web.multipart.MultipartFile
  3. import org.codehaus.groovy.grails.commons.ConfigurationHolder
  4.  
  5. class Attachment {
  6.     //Allows us to have spring bind the file automatically. <input name="attachment.file" type="file" />
  7.     MultipartFile file
  8.     //Save file metadata to the database
  9.     String contentType
  10.     String fileName
  11.     Long size    
  12.  
  13.     //Don’t try to save the MultipartFile.  Hibernate will barf all over you.
  14.     static transients = ["file"]
  15.  
  16.     //Location for saving files i.e. /home/me/app_files/attachments/
  17.     def baseDir = { ConfigurationHolder.config.paperclip.files.dir[GrailsUtil.environment] }
  18.  
  19.     //Use Hibernate Events Plugin to save the file after it’s metadata has been saved.
  20.     def afterInsert = {
  21.         writeFile()
  22.     }
  23.  
  24.     //Create a unique directory to store the file based on the saved id. Produces a two deep
  25.     //directory to stave off the 32000 hard link limit on ext3 filesystem.  So the path might look
  26.     //like 0000/0001/filename.extension
  27.     //see: http://thegioraproject.com/2008/03/02/workaround-for-subdirectory-limit-on-ext3-filesystem/
  28.     def directory = {
  29.         def stringId = id.toString()
  30.         def fullId = stringId.padLeft(8, "0")
  31.         [fullId[0..3], fullId[4..7]].join("/")
  32.     }
  33.  
  34.     //Joins the directory and filename to create a path.  Useful for creating links on views: can call
  35.     //attachment.path() to produce href.
  36.     def path = {
  37.         [directory(), fileName].join("/")
  38.     }
  39.  
  40.     //Location on disk to write the file to.
  41.     def absolutePath = {
  42.         [baseDir(), path()].join("/")
  43.     }
  44.  
  45.     //Writes the file to the disk making sure all parent directories are present
  46.     protected void writeFile() {
  47.         def destination = new File(absolutePath())
  48.         destination.mkdirs()
  49.         file.transferTo(destination)
  50.     }
  51.  
  52.     //When Spring binds the file to the object we set the metadata for the file on the object.
  53.     //Eliminates some setter code.
  54.     public void setFile(MultipartFile upload) {
  55.         file = upload
  56.         contentType = upload.contentType
  57.         fileName = upload.originalFilename
  58.         size = upload.size
  59.     }
  60. }

4. Created an Image Domain class

  1. import grails.util.GrailsUtil
  2. import org.codehaus.groovy.grails.commons.ConfigurationHolder
  3.  
  4. class Image extends Attachment {
  5.     //Root directory of the ImageMagick install determined by the environment.  So if we’re using
  6.     //development it will get the config variable paperclip.magick.dir.development from Config.groovy
  7.     def magickDir = { ConfigurationHolder.config.paperclip.magick.dir[GrailsUtil.environment] }    
  8.  
  9.     //After saving make sure we save the original by calling writeFile() on Attachment,
  10.     //then create the thumbnails
  11.     def afterInsert = {
  12.         writeFile()
  13.         createThumbnails()
  14.     }
  15.  
  16.     //Location on disk to write the thumbnail to
  17.     def absoluteThumbnail = {type ->
  18.         [baseDir(), thumbnail(type)].join("/")
  19.     }
  20.  
  21.     //Kind of like path on Attachment.  Allows us to call attachment.thumbnail("300") on the view to
  22.     //produce an img src.
  23.     def thumbnail = {type ->
  24.         def tokens = fileName.tokenize(".")
  25.         def name = [tokens[0..-2].join("."), type, tokens[-1]].join(".")
  26.         [directory(), name].join("/")
  27.     }
  28.  
  29.     //The meat of it all.  It gets a little lost because there isn’t a lot of code.  So we loop over
  30.     //an array to produce two thumbnails.  One will be 100×100, the other will be 300×300.  However
  31.     //ImageMagick will keep the images aspect ratio using the geometry we supplied.  This method will
  32.     //create a string like "/ImageMagick/convert /path/image.jpg -thumbnail 100×100 /path/image.100.jpg"
  33.     //and then execute it as if you were on the command line.  The ImageMagick command will resize and
  34.     //save the image to the supplied dir.  The execute() call returns a process object which we call
  35.     //waitFor on to make sure the processing finishes before we move on.  We can also get the exit code
  36.     //from the process to handle exceptions.
  37.     //More info ongeometry flags and resizing here: http://www.imagemagick.org/Usage/resize/#resize.
  38.     //It is very flexible.
  39.     protected void createThumbnails() {
  40.         [300, 100].each {
  41.             def process = "${magickDir()}/convert ${absolutePath()} -thumbnail ${it}x${it} ${absoluteThumbnail(it)}".execute()
  42.             process.waitFor()
  43.         }
  44.     }
  45. }

Hopefully the comments explain how everything works together. If you have questions, comments or criticism, please let me know. And yes, I feel a little dirty calling another program in a process, but when it works this well, getting a little dirty feels pretty good.

Image comparison (my youngest, Ariana):

ImageMagick
ImageMagick 100px Thumb

ImageTool (JAI)
ImageTool(JAI) 100px Thumb

ImageMagick
arianamagick300.JPG

ImageTool (JAI)
arianaimagetool300.jpg

Hibernate events plugin v0.2

What, another release already? Yep, we’re releasing another version of the grails hibernate events plugin.  Well, obviously I wasn’t paying attention in grails/hibernate class and it turns out that while it requires a hibernate.cfg.xml to participate in the creation of the SessionFactory, you can still programmatically add event listeners to the SessionFactory after its been created. Big ups to Burt Beckwith for the tip (who, judging by the grails mailing list alone, has every api under the sun memorized!). So here is the hibernate events plugin sans hibernate.cfg.xml:

http://thegioraproject.com/files/grails/plugins/grails-hibernate-events-0.2.zip

And again, any feedback is much appreciated.

Hibernate events plugin

On a current grails project I had a need to act on my domain classes using hibernate’s persistence lifecycle. However, it turns out that grails only provides three lifecycle methods: beforeInsert, beforeUpdate, beforeDelete. Four if you count onLoad. I was really interested in having an afterInsert method so that I could save a file to the filesystem after it’s metadata had been saved to the database. So I used the hibernate event system to round out the lifecycle methods with seven more hooks:

  • afterInsert
  • afterUpdate
  • afterDelete
  • beforeLoad
  • afterLoad
  • beforeSave
  • afterSave

So if a domain model contains a closure with any of these names they will be called during their turn in the lifecycle. I think they are mostly self explanatory except for before/afterSave. The purpose for these methods is to be called when the persistence call is either an insert or an update. So if you don’t care if the object is being saved for the first time or being updated you can use this method.

Unfortunately the plugin uses a hibernate.cfg.xml, because it is the only way you can participate in the creation of the SessionFactory.

I haven’t decided if I will release the plugin yet. Maybe if there is interest.

Update: Ok, so some interest was expressed. Here it is: http://thegioraproject.com/files/grails/plugins/grails-hibernate-events-0.1.zip. Feedback is always welcome.

Update 2: I’ve added a doc page to the grails wiki.  Not really much is needed, but it could clear things up a bit.  http://docs.codehaus.org/display/GRAILS/Hibernate+Events+Plugin

Have God watch over your processes; or How to keep your app running when your not watching

My latest application at work is a Ruby on Rails(RoR) app, that we just deployed recently. Being a RoR app it hasn’t been so pleasant in production. I’m sure that it is due to my lack of experience in deploying RoR apps, but it has been difficult in any case. It is deployed to a cluster of 3 mongrels behind an apache proxy/load balancer. Just this morning the app became unresponsive as one of the mongrel instances was hung and had the cpu pegged. Not hard to fix, killed the processes and restarted the cluster. More of a nuisance really.

The problem, though, is that if this keeps happening (it most likely will), it would be more than just a nuisance. We needed to find a way to monitor the app to make sure it was always running. Luckily there is a product out there that does more than just monitor your app. Luckily I found God ;). From the God website:

“God is an easy to configure, easy to extend monitoring framework written in Ruby.

Keeping your server processes and tasks running should be a simple part of your deployment process. God aims to be the simplest, most powerful monitoring application available.”

It doesn’t lie. It is one of the best pieces of software that I have used in a little while now. I was up and running in about 30 minutes. Most of that time was spent checking out the different options and whatnot. So now if one of the mongrels fails for some reason, God will start it back up. If one of the mongrels is using too much memory, God will restart it. If one of the mongrels is pegging the processor, God will restart it. You can even watch the logs in real time for each individual mongrel instance, and control God from the command line. There is also a great tutorial on loading God on system startup.

If you’re interested make sure you read through the home page for God to see all the great things it is capable of.

Technorati :

Workaround for subdirectory limit on ext3 filesystem

Just keep in mind that this is a temporary workaround for a problem that should be addressed in a more robust fashion. So on the ext3 filesystem there is a subdirectory limit of 32,000. The problem is that when you reach this limit and you try to add another subdirectory you get a programmer type error: “Too many links”. If you aren’t at the command line and calling mkdir directly this can be very cryptic bubbling up in your programs stack trace. What it means is that there are too many hard links(directories or files) in the directory.

It is a real show stopper, especially if it is the directory to which upload all over your website’s user generated content. Here is a quick workaround until you can solve the problem more gracefully. The limit is for hard links and not soft links (symbolic links). So, what you want to do is:

  1. Create an overflow directory.
  2. Move all of your current subdirectories from the current directory to the overflow directory.
  3. Create a symlink for each subdirectory in the current directory pointing the directory in the overflow directory.

Of course you won’t want to do this by hand, so you might want to whip up a script to do this. Mine was only a few lines of ruby (also note that it was made easier by the fact that the directories were sequential numbers):

  1.     require ‘fileutils’
  2.    old_dir = "/village/data_file/content"
  3.    new_dir = "/village/data_file/overflow"
  4.  
  5.    (0..41671).each do |index|
  6.      dir = File.join(old_dir, index.to_s)
  7.      if File.exist?(dir)
  8.        FileUtils.mv(dir, new_dir)
  9.        FileUtils.ln_sf(File.join(new_dir, index.to_s), dir)
  10.      end
  11.    end

Piping grep to rm

I discovered a neat trick to deleting folders/files based on a pattern. Let’s say you have a directory structure like this:

/test
/test/aaaa
/test/bbbb
/test/cccc
/test/ddd
/test/eee
/test/fff

You’ve decided that any folder with a’s, b’s, or c’s has to go. You definitely don’t want to delete them all by hand; especially if there are a couple thousand of them. The first thing you need to do is find the directories that you want to remove:

  1. ls | grep "[a-c]"

Which produces the following:
aaaa
bbbb
cccc

Great! Now what? All you need to do is pipe that to rm using xargs:

  1. ls | grep "[a-c]" | xargs rm -rf

All files that match the given pattern will be deleted. But, be careful that the pattern only matches the files you would like to delete. Run ls | grep first before you pipe anything to rm, unless you like trying to restore deleted files.

Technorati :

Dead simple dependencies with grails + ivy

It is unbelievably easy to integrate Ivy as a dependency manager for your grails application. Just a couple of steps really (Note: this is for grails >= 1.0 as < 1.0 had ivy integrated):

  1. Install the Ivy plugin:
    1. grails install-plugin ivy

    This will create an ivy.xml and an ivyconf.xml, and place them in the project root. I modified my ivy.xml thusly:

    1. <ivy-module version="1.0">
    2.    <info organisation="codehaus" module="grails"/>
    3.    <dependencies>
    4.        <dependency org="mysql" name="mysql-connector-java" rev="5.0.5"/>
    5.    </dependencies>
    6. </ivy-module>
  2. You can then run the following command and ivy will place your dependencies in the lib folder of your project root:
    1. grails get-dependencies

That’s it! Now I just need to figure out why they don’t really use this as a selling point for grails. The only thing I can figure is that there are so many great features it would get lost in the list anyway.