Today I had some knowledge delivered straight to the top of my dome, by Darcy. The issue was I had two models that I wanted to join and then apply scopes to. I wanted to retrieve Story objects that were joined to HourEntry objects, that were from this week. However when I tried to use the Time scope on the query it was attempting to call those methods on Story.
ree-1.8.7-2011.02 :132 > Story.joins(:hour_entries).this_week
NoMethodError: undefined method `this_week' for #
The problem was that the .this_week call was being sent to Story rather than to HourEntry. Here’s where Darcy chimed in with the scoped.merge call. It went a little something like this :
Story.scoped.merge HourEntry.this_week
This successfully merged the scope defined by Story with the scope defined by the HourEntry.this_week call and I get all the stories that have HourEntry objects that are returned by the .this_week scope.
This then returns all the stories you’ve worked on this week. Add a bit of grouping/distinct and it’s done and done. The additional nice part about this is composing these scopes will allow me to do a lot more in the future, very quickly.
There was a question that was asked on our IRC channel today about how to get all the associations for an Activerecord model. I’m assuming it was to do some debugging or something, in any case I did a bit of digging around in the Rails docs and it turns out the answer isn’t that hard.
Doing something like :
Model.reflect_on_all_associations
Will give you all the associations for that model, it’s pretty dirty though and so an easy way to tidy it up is :
RVM is best known as a tool to help developers upgrade their applications to newer versions of Ruby and Rails (3 specifically). That said, for ruby developers, it has many features which help to make their workflow far simpler. Whilst some have shown with effort and rigorous manual process you can achieve much of what RVM offers, the following is a look into some less often discussed areas of RVM that make it my most favoured tool when it comes to development.
Gemsets and the Gemset Hierarchy
Whilst Bundler solves most of the problems related to isolating gems for me personally, on most projects I develop for there are things that make rvm’s implementation of gemsets (especially when used along with bundler) invaluable to me as a ruby developer.
Isolating Gems
A major strength of RVM is the ability to isolate gems at the system level. This keeps them safely partitioned from any other gems unrelated to my project. The way bundler isolates gems is for the most part at run time – It changes the way rubygems works to solve it’s problem and either installs all gems to a user specified path or to your gem directory.
If you are using the usr specified path option, you must to remember that your custom rubygems bin directory is not contained in your PATH. This means that you must either manually add it to the PATH, or your need to manually prefix each command with bundle exec, or alternatively you can generate the binaries into your applications bin directory and each command with ./bin/
If you take the opposite approach and install your gems into your gem home, you then need to be aware of the fact that each next install will overwrite any previous binaries – the canonical example being Rails 2 and Rails 3. The way that rubygems generates binaries lets you run gem-binary-name _version_ something to call the binary loaded from a specified gem version. Unfortunately, in Rails 3, the rails binary was moved from the rails gem into the railties gem which just so happens to break this feature of rubygems. You can try this for yourself on a new rvm gemset, install rails 2.3.8 and rails 3.0.0 and then try rails _2.3.8_ -v. If you are using the simple solution of installing to gem home, even though you are using bundler it will still overwrite the binary.
One common suggestion is to just create a wrapper binary manually (e.g. rails2) or always prefix your commands with bundle exec. In contrast RVM’s isolation works by explicitly setting rubygems home and path to directories isolated under RVM. RVM takes this even further using gemsets (think subdirectories) to handle more fine grain isolation. For example, I personally use 1 gemset per application, thus avoiding the issues described above as for each application I will have only a single version of rails installed. Even better, switching is shell-local and only persists for a single shell instance. This means that I can have two different applications with two completely different versions of Rails running at the same time in different terminal sessions without issue.
As an added bonus, because RVM isolates at the rubygems system level (using environment variables), your application doesn’t actually need to know or care about conflicts from gems that the application does not use.
Combining RVM and Bundler yields a great deal of flexibility and consisency. This affords you the ‘best of both worlds’, system level isolation with the ability to install gems from git repositories, and in application gem loading based on application environment.
The second feature I use often is the global gemset. Each ruby interpreter installation comes with two gemsets out of the box – the default (blank) gemset (e.g. ree-1.8.7-2010.02) and the global gemset (e.g. ree-1.8.7-2010.02@global). When you use any gemset, rvm not only sets the gem home to a uniq directory for the given gemset (ensuring gems are installed in the correct place) but it also sets GEM_PATH.
Much like PATH, GEM_PATH is a colon separated list of directories that rubygems uses to look up gems when requiring them. By installing a gem to the global gemset for any ruby, e.g. ree-1.8.7-2010.02@global, it will automatically be made available (including it’s binaries) inside both the default gemset and any user-defined gemsets. E.g.,
ree-1.8.7-2010.02
ree-1.8.7-2010.02@bighelpmob
ree-1.8.7-2010.02@tedxperth
Will all have access to gems installed in ree-1.8.7-2010.02@global.
This comes in super handy for things like awesome_print, wirble, bundler, git-up, homesick – Namely, gems you want available in every environment.
Unfortunately, bundler currently does not support multiple items in BUNDLE_PATH. We have hope it will be added in the future but for now gems you want shared between apps using bundler, you’ll get some duplication.
Default Global Gemset Contents
As an added bonus along side the global gemsets, rvm provides a way to declare a file which tells it what gems to install initially. To do this, it uses a ruby string-based directory hierarchy (see the link below) to look for a global.gems file that rvm then imports – For example, my ~/.rvm/gemsets/global.gems (that is imported into the global gemset of all ruby interpreter installs) contains the following simple list:
If you are writing gems, one of the most important things you can do as a Ruby developer these days is to ensure they work on all of the major implementations. At the moment, for most gem developers you should generally be testing your code against:
REE / 1.8.7
1.9.2
Rubinius
Jruby
And optionally, MagLev and MacRuby. RVM provides tools making it simple to run code and commands against multiple rubies. This includes but is not limited to running Rake tasks and tests / specs. The idea being to make it as frictionless as possible to test your gem against all of the above interpreters. As an example, you can run:
rvm rake test
Which will run rake test against all of your installed ruby interpreters (using default gemset for each). You may need to do some manual setup (e.g. installing test gem dependencies) but rvm tries to make it all as simple as possible and more importantly, heavily integrated into your normal work flow whilst staying out of the way.
Along side rake, rvm also provides wrappers for many common binaries (e.g. ruby, spec and the like) plus the --json and --yaml flags which make it simple to get a summarised view of the program results. Using rvm exec, you can even perform set operations against any arbitrary command.
There is a whole lot more set operations can do and I suggest anyone using or considering using rvm read the set’s section on the rvm site. Go forth and have sets!
Tools for easing dependency-related pains
One of the most commonly encountered problems (since for most people, the ruby interpreter you’d normally use is distributed as a binary install) are related to external dependencies – Namely, things like readline, iconv and openssl.
In an attempt to make life easier (and in particular, work around common problems like a neutered libedit instead of readline on OSX), rvm provides the rvm package set of commands that make it easy to install sandboxed versions of these in the ~/.rvm/usr directory.
As an example, to work around the libedit vs. readline issue on OSX, you can build your ruby against readline 6.0 by running:
And, as an added bonus, to automatically patch older versions of ree and 1.8.7 to respond to Control-C immediately (for example, in irb) instead of when you press enter, you can replace the rvm install command above with:
Say, for example, a new version of 1.9.2 comes out in the next few weeks and you want to automatically update your rubies to match it correctly. As of approximately rvm 1.0.0, we added an rvm upgrade command that automatically handles doing the following:
Installing the new ruby
Moving across gemsets
Updating wrapper scripts and aliases
The only thing you typically have to update are references in your .rvmrc’s and your passenger configuration. But, with a little forethought (e.g. creating an alias using rvm alias create), it’s entirely possible to make it so the entire upgrade is automatic.
I have personally used this feature to move from the 1.9.2 release candidate to the final patchlevel 0 release – I used rvm upgrade to simplify the process of updating each time it was bumped.
This is currently undocumented on the rvm site but documentation will be added – for the moment, in your rvm install, run rvm help upgrade and/or rvm upgrade help.
Location Matters
One of the underlying decisions related to rvm’s architecture is where it installs your rubies – In most cases, rvm will install your rubies into ~/.rvm/rubies and gems into ~/.rvm/gems. This means that you should almost never have to use sudo for commands (there are a few exceptions, most notably passenger which relies on root permission in order to bind to port 80).
This approach is simple on the scale of things but also brings a lot of freedom – for one, it means you are free to experiment with a Ruby / RVM setup, wiping it away when you’re done with a simple rm -rf ~/.rvm && mv .rvm-original .rvm.
.rvmrc files
All of this is fine and dandy, but when you still have to manually switch ruby interpreters and rubies each time you do something, it can become mildly annoying rather fast. For this exact reason RVM offers project-specific .rvmrc files – If you haven’t encountered them before, project .rvmrc’s are files that are automatically sourced into the current shell when you change (cd) into a directory containing one. These files can contain anything that is valid shell script (and to prevent RVM automatically running code you don’t want, it defaults to asking you to trust it the first time it is run) making things like project bootstrapping and configuration simple and mostly automatic after a quick setup in the project root directory:
rvm --rvmrc --create <ruby>@<gemset>
In my own projects, I use these to automatically install the Ruby if not present, create and use a project-specific gemset and then do the minimum possible required to bootstrap the new environment – in most cases, this is simply a matter of installing a few gems (e.g. for me, I install bundler via rvm gemset import).
In some cases, I even automatically run bundle install (with 1.0, if you have the gems already installed, this is very fast) hence the first time I cd into a project it essentially bootstraps the entire environment for me automatically.
Ultimately, this approach means that the Ruby interpreter and gemset switching are completely transparent to me – No matter which project I’m using, ruby and irb are available and point to the correct executable for the current project I am in. When integrated with some code I wrote for ruby summer of code, it’s even possible to have passenger automatically use this .rvmrc file to automatically change your application’s gemset at run time.
I hope that you have found at least one new or useful thing about RVM. RVM has been invaluable to me as a developer on multiple open source projects over the last several months. It makes it simple for me to do things that are otherwise harder or less flexible and generally stays out of my way.
We have a test suite here that now is rapidly approaching 2 hours using a single core. Let me just repeat that. A developer realistically would have to leave their machine testing overnight to see if the suite is working. That’s really not good enough.
Specjour has been a bit of a turn-key miracle worker with our RSpec suite, however lately we’ve started to require some custom database setup that we do in a seeds.rb file as well as some custom bundler install parameters as most of our devs don’t have MySQL installed. Both of our needs were being nicely stomped on by Specjour so I thought it was time to look elsewhere.
I took a trip down Hydra lane and while getting to the point of having a working, local, dual runner system was a piece of cake, getting something working remotely via SSH took me hours of pain. Debugging the remote SSH workers was a nightmare and I spent a couple of hours running through code before deciding it was probably better to update our existing solution rather than tooling up a brand new one.
Back to the Specjour code. Specjour includes a rails directory inside which is an init.rb which Rails will run at initialisation (it’s part of what Rails does) but the Specjour initialiser will always just run the default database setup task no matter what initialiser you’ve got setup. We had a specjour initialiser that runs if ENV['PREPARE_DB'] was populated, which it is by Specjour, the problem was that the Specjour initialiser ran in the Rails after_initialization hook and therefore stomped all over our database setup.
The first step was just to have our initialiser write to another ENV element and then to have the Specjour after_initialize handler respect this. This isn’t too hard to implement as the after_initialize handler is just a block that is attached and so inside of this block you just need to check that ENV element. In my case I created a new ENV['DB_PREPPED'] element when my database setup had completed and then when the after_initialize block runs it checks for ENV['DB_PREPPED'] and will do nothing if that’s been set to true.
Easy. I now had Specjour respecting our database setup task.
The next step was to try and test this outside of a Rails application, not only that but to test the operation of a block (anonymous function?). To do this I setup a stub on a mock Rails class and let it capture the after_initialize block and then I ran a number of specs against this block.
module Specjour
module DbScrub
end
end
DO_NOT_REQUIRE = true
describe "Rails Initialiser" do
before :all do
ENV['PREPARE_DB'] = "true"
stub(Specjour::DbScrub).scrub
class Rails
class << self; attr_accessor :configuration; end
class << self; attr_accessor :test_block; end
end
config = Object.new
stub(config).after_initialize { |args|
object = Object.new
Rails.test_block = args
object
}
Rails.configuration = config
require 'rails/init'
end
... tests ...
This code essentially mocks up Rails.configuration and then stubs the after_initialize method. This stub then places the block that after_initialize yields to into Rails.test_block. When I require 'rails/init' it sequentially processes the file (as with all Ruby) and the stub will capture the block. After this is a bunch of tests I run an whether the Specjour::DbScrub.scrub method is called or not, so it’s nothing special.
I felt like at this stage I had fairly well tested the main aspects of the database setup.
The next issue was with how bundler was being handled. We have a situation where we would like to install sometimes without some gems. Some of the gems we use and have written use applications we’d rather not maintain in development and get tested in our staging and production environments. We generally will run a bundle install in development without the production or metrics groups so I wanted to have the ability to pass through a custom bundler command. That’s pretty easy now with my gem. Inside .specjour/bundler.yml there is a command property. I think this is more complex than what’s required, but I can foresee us needing a number of custom rake tasks and shell scripts so this bundler.yml should have probably started life as a settings/commands/something_generic.yml
To test this part of my changes was pretty simple. I basically just stubbed the system calls to bundler to give certain return values and checked to make sure the correct program flow happened.
describe ".bundle_install" do
let :manager do
stub.instance_of(Specjour::Manager).project_path { "/tmp" }
stub(Dir).chdir(anything) { |args|
args.last.call # This yields to the block for Dir.chdir()
}
manager = Specjour::Manager.new
stub(manager).project_path { "blah" }
mock(manager).system('bundle lock')
manager
end
it "should perform a bundle lock" do
stub(manager).system('bundle check > /dev/null') { true }
manager.bundle_install
end
it "should check if there are gems required" do
mock(manager).system('bundle check > /dev/null') { true }
manager.bundle_install
end
context "when gems are required" do
before :each do
# Not a before :all as it needs to hook into the let hook above
stub(manager).system('bundle check > /dev/null') { false }
end
context "and there is a bundler YAML file" do
before :each do
config_file = ".specjour/bundler.yml"
mock(File).exists?(config_file) { true }
mock(File).read(config_file) { "" }
mock(YAML).load(anything) {
{ 'command' => "do it" }
}
end
it "should get the bundle command from the YAML file" do
mock(manager).system('do it > /dev/null')
manager.bundle_install
end
end
context "and there is no bundler YAML file" do
before :each do
mock(File).exists?(".specjour/bundler.yml") { false }
end
it "should perform a bundle install" do
mock(manager).system('bundle install > /dev/null')
manager.bundle_install
end
end
end
end
You can see that I stubbed our the Dir.chdir block to just yield directly to the call, otherwise it’ll throw an exception. Then I stubbed and mocked out the Kernel.system calls as necessary. Kernel methods are generally included into Ruby objects so you don’t stub Kernel, you stub the object that has the Kernel methods. Most of the testing is pretty basic, but I’d be keen to hear if I’m doing anything incorrectly!
This was my first major venture into adding functionality to a public project and it was good fun. I think it made me do a little better work than I might normally, it’s a great motivation to potentially have peers look at how you do things.
After bundling it all up and testing it here with over a dozen developers and even more machines I’m pretty happy with how it functions. I’ve made a pull request back to the original gem creator and hopefully he’ll like what I’ve done. In the meantime if you want to check it out then my Specjour is available on Git Hub.
We’re currently deciding on hardware to build a bit of a testing cluster on which we’ll run whatever the current best remote testing package we can find. At the moment, for us, that’s ended up being Specjour. We ended up pitting one of our i7 iMacs against a $400 Acer box we bought. We installed Ubuntu’s REE 1.8.7 on the Acer machine and RVM and REE 1.8.7 on the iMac.
i7 iMac – Quad Core Hyperthreading
real 11m14.131s
user 0m0.776s
sys 0m0.348s
Acer – E5200 Dual Core E5200
real 35m56.658s
user 0m0.870s
sys 0m2.500s
It seems like the little Acer box offers slightly better value for money in this case. I’d like to see how we’d do with some virtualisation sitting on top this, but I think the included memory will be quite limiting in this regard. I wonder whether the processor resources are actually being fully utilised.