RVM is best known as a tool to help developers upgrade their applications to newer versions of Ruby and Rails (3 specifically). That said, for ruby developers, it has many features which help to make their workflow far simpler. Whilst some have shown with effort and rigorous manual process you can achieve much of what RVM offers, the following is a look into some less often discussed areas of RVM that make it my most favoured tool when it comes to development.
Gemsets and the Gemset Hierarchy
Whilst Bundler solves most of the problems related to isolating gems for me personally, on most projects I develop for there are things that make rvm’s implementation of gemsets (especially when used along with bundler) invaluable to me as a ruby developer.
Isolating Gems
A major strength of RVM is the ability to isolate gems at the system level. This keeps them safely partitioned from any other gems unrelated to my project. The way bundler isolates gems is for the most part at run time – It changes the way rubygems works to solve it’s problem and either installs all gems to a user specified path or to your gem directory.
If you are using the usr specified path option, you must to remember that your custom rubygems bin directory is not contained in your PATH. This means that you must either manually add it to the PATH, or your need to manually prefix each command with bundle exec, or alternatively you can generate the binaries into your applications bin directory and each command with ./bin/
If you take the opposite approach and install your gems into your gem home, you then need to be aware of the fact that each next install will overwrite any previous binaries – the canonical example being Rails 2 and Rails 3. The way that rubygems generates binaries lets you run gem-binary-name _version_ something to call the binary loaded from a specified gem version. Unfortunately, in Rails 3, the rails binary was moved from the rails gem into the railties gem which just so happens to break this feature of rubygems. You can try this for yourself on a new rvm gemset, install rails 2.3.8 and rails 3.0.0 and then try rails _2.3.8_ -v. If you are using the simple solution of installing to gem home, even though you are using bundler it will still overwrite the binary.
One common suggestion is to just create a wrapper binary manually (e.g. rails2) or always prefix your commands with bundle exec. In contrast RVM’s isolation works by explicitly setting rubygems home and path to directories isolated under RVM. RVM takes this even further using gemsets (think subdirectories) to handle more fine grain isolation. For example, I personally use 1 gemset per application, thus avoiding the issues described above as for each application I will have only a single version of rails installed. Even better, switching is shell-local and only persists for a single shell instance. This means that I can have two different applications with two completely different versions of Rails running at the same time in different terminal sessions without issue.
As an added bonus, because RVM isolates at the rubygems system level (using environment variables), your application doesn’t actually need to know or care about conflicts from gems that the application does not use.
Combining RVM and Bundler yields a great deal of flexibility and consisency. This affords you the ‘best of both worlds’, system level isolation with the ability to install gems from git repositories, and in application gem loading based on application environment.
The second feature I use often is the global gemset. Each ruby interpreter installation comes with two gemsets out of the box – the default (blank) gemset (e.g. ree-1.8.7-2010.02) and the global gemset (e.g. ree-1.8.7-2010.02@global). When you use any gemset, rvm not only sets the gem home to a uniq directory for the given gemset (ensuring gems are installed in the correct place) but it also sets GEM_PATH.
Much like PATH, GEM_PATH is a colon separated list of directories that rubygems uses to look up gems when requiring them. By installing a gem to the global gemset for any ruby, e.g. ree-1.8.7-2010.02@global, it will automatically be made available (including it’s binaries) inside both the default gemset and any user-defined gemsets. E.g.,
ree-1.8.7-2010.02
ree-1.8.7-2010.02@bighelpmob
ree-1.8.7-2010.02@tedxperth
Will all have access to gems installed in ree-1.8.7-2010.02@global.
This comes in super handy for things like awesome_print, wirble, bundler, git-up, homesick – Namely, gems you want available in every environment.
Unfortunately, bundler currently does not support multiple items in BUNDLE_PATH. We have hope it will be added in the future but for now gems you want shared between apps using bundler, you’ll get some duplication.
Default Global Gemset Contents
As an added bonus along side the global gemsets, rvm provides a way to declare a file which tells it what gems to install initially. To do this, it uses a ruby string-based directory hierarchy (see the link below) to look for a global.gems file that rvm then imports – For example, my ~/.rvm/gemsets/global.gems (that is imported into the global gemset of all ruby interpreter installs) contains the following simple list:
If you are writing gems, one of the most important things you can do as a Ruby developer these days is to ensure they work on all of the major implementations. At the moment, for most gem developers you should generally be testing your code against:
REE / 1.8.7
1.9.2
Rubinius
Jruby
And optionally, MagLev and MacRuby. RVM provides tools making it simple to run code and commands against multiple rubies. This includes but is not limited to running Rake tasks and tests / specs. The idea being to make it as frictionless as possible to test your gem against all of the above interpreters. As an example, you can run:
rvm rake test
Which will run rake test against all of your installed ruby interpreters (using default gemset for each). You may need to do some manual setup (e.g. installing test gem dependencies) but rvm tries to make it all as simple as possible and more importantly, heavily integrated into your normal work flow whilst staying out of the way.
Along side rake, rvm also provides wrappers for many common binaries (e.g. ruby, spec and the like) plus the --json and --yaml flags which make it simple to get a summarised view of the program results. Using rvm exec, you can even perform set operations against any arbitrary command.
There is a whole lot more set operations can do and I suggest anyone using or considering using rvm read the set’s section on the rvm site. Go forth and have sets!
Tools for easing dependency-related pains
One of the most commonly encountered problems (since for most people, the ruby interpreter you’d normally use is distributed as a binary install) are related to external dependencies – Namely, things like readline, iconv and openssl.
In an attempt to make life easier (and in particular, work around common problems like a neutered libedit instead of readline on OSX), rvm provides the rvm package set of commands that make it easy to install sandboxed versions of these in the ~/.rvm/usr directory.
As an example, to work around the libedit vs. readline issue on OSX, you can build your ruby against readline 6.0 by running:
And, as an added bonus, to automatically patch older versions of ree and 1.8.7 to respond to Control-C immediately (for example, in irb) instead of when you press enter, you can replace the rvm install command above with:
Say, for example, a new version of 1.9.2 comes out in the next few weeks and you want to automatically update your rubies to match it correctly. As of approximately rvm 1.0.0, we added an rvm upgrade command that automatically handles doing the following:
Installing the new ruby
Moving across gemsets
Updating wrapper scripts and aliases
The only thing you typically have to update are references in your .rvmrc’s and your passenger configuration. But, with a little forethought (e.g. creating an alias using rvm alias create), it’s entirely possible to make it so the entire upgrade is automatic.
I have personally used this feature to move from the 1.9.2 release candidate to the final patchlevel 0 release – I used rvm upgrade to simplify the process of updating each time it was bumped.
This is currently undocumented on the rvm site but documentation will be added – for the moment, in your rvm install, run rvm help upgrade and/or rvm upgrade help.
Location Matters
One of the underlying decisions related to rvm’s architecture is where it installs your rubies – In most cases, rvm will install your rubies into ~/.rvm/rubies and gems into ~/.rvm/gems. This means that you should almost never have to use sudo for commands (there are a few exceptions, most notably passenger which relies on root permission in order to bind to port 80).
This approach is simple on the scale of things but also brings a lot of freedom – for one, it means you are free to experiment with a Ruby / RVM setup, wiping it away when you’re done with a simple rm -rf ~/.rvm && mv .rvm-original .rvm.
.rvmrc files
All of this is fine and dandy, but when you still have to manually switch ruby interpreters and rubies each time you do something, it can become mildly annoying rather fast. For this exact reason RVM offers project-specific .rvmrc files – If you haven’t encountered them before, project .rvmrc’s are files that are automatically sourced into the current shell when you change (cd) into a directory containing one. These files can contain anything that is valid shell script (and to prevent RVM automatically running code you don’t want, it defaults to asking you to trust it the first time it is run) making things like project bootstrapping and configuration simple and mostly automatic after a quick setup in the project root directory:
rvm --rvmrc --create <ruby>@<gemset>
In my own projects, I use these to automatically install the Ruby if not present, create and use a project-specific gemset and then do the minimum possible required to bootstrap the new environment – in most cases, this is simply a matter of installing a few gems (e.g. for me, I install bundler via rvm gemset import).
In some cases, I even automatically run bundle install (with 1.0, if you have the gems already installed, this is very fast) hence the first time I cd into a project it essentially bootstraps the entire environment for me automatically.
Ultimately, this approach means that the Ruby interpreter and gemset switching are completely transparent to me – No matter which project I’m using, ruby and irb are available and point to the correct executable for the current project I am in. When integrated with some code I wrote for ruby summer of code, it’s even possible to have passenger automatically use this .rvmrc file to automatically change your application’s gemset at run time.
I hope that you have found at least one new or useful thing about RVM. RVM has been invaluable to me as a developer on multiple open source projects over the last several months. It makes it simple for me to do things that are otherwise harder or less flexible and generally stays out of my way.
We have a test suite here that now is rapidly approaching 2 hours using a single core. Let me just repeat that. A developer realistically would have to leave their machine testing overnight to see if the suite is working. That’s really not good enough.
Specjour has been a bit of a turn-key miracle worker with our RSpec suite, however lately we’ve started to require some custom database setup that we do in a seeds.rb file as well as some custom bundler install parameters as most of our devs don’t have MySQL installed. Both of our needs were being nicely stomped on by Specjour so I thought it was time to look elsewhere.
I took a trip down Hydra lane and while getting to the point of having a working, local, dual runner system was a piece of cake, getting something working remotely via SSH took me hours of pain. Debugging the remote SSH workers was a nightmare and I spent a couple of hours running through code before deciding it was probably better to update our existing solution rather than tooling up a brand new one.
Back to the Specjour code. Specjour includes a rails directory inside which is an init.rb which Rails will run at initialisation (it’s part of what Rails does) but the Specjour initialiser will always just run the default database setup task no matter what initialiser you’ve got setup. We had a specjour initialiser that runs if ENV['PREPARE_DB'] was populated, which it is by Specjour, the problem was that the Specjour initialiser ran in the Rails after_initialization hook and therefore stomped all over our database setup.
The first step was just to have our initialiser write to another ENV element and then to have the Specjour after_initialize handler respect this. This isn’t too hard to implement as the after_initialize handler is just a block that is attached and so inside of this block you just need to check that ENV element. In my case I created a new ENV['DB_PREPPED'] element when my database setup had completed and then when the after_initialize block runs it checks for ENV['DB_PREPPED'] and will do nothing if that’s been set to true.
Easy. I now had Specjour respecting our database setup task.
The next step was to try and test this outside of a Rails application, not only that but to test the operation of a block (anonymous function?). To do this I setup a stub on a mock Rails class and let it capture the after_initialize block and then I ran a number of specs against this block.
module Specjour
module DbScrub
end
end
DO_NOT_REQUIRE = true
describe "Rails Initialiser" do
before :all do
ENV['PREPARE_DB'] = "true"
stub(Specjour::DbScrub).scrub
class Rails
class << self; attr_accessor :configuration; end
class << self; attr_accessor :test_block; end
end
config = Object.new
stub(config).after_initialize { |args|
object = Object.new
Rails.test_block = args
object
}
Rails.configuration = config
require 'rails/init'
end
... tests ...
This code essentially mocks up Rails.configuration and then stubs the after_initialize method. This stub then places the block that after_initialize yields to into Rails.test_block. When I require 'rails/init' it sequentially processes the file (as with all Ruby) and the stub will capture the block. After this is a bunch of tests I run an whether the Specjour::DbScrub.scrub method is called or not, so it’s nothing special.
I felt like at this stage I had fairly well tested the main aspects of the database setup.
The next issue was with how bundler was being handled. We have a situation where we would like to install sometimes without some gems. Some of the gems we use and have written use applications we’d rather not maintain in development and get tested in our staging and production environments. We generally will run a bundle install in development without the production or metrics groups so I wanted to have the ability to pass through a custom bundler command. That’s pretty easy now with my gem. Inside .specjour/bundler.yml there is a command property. I think this is more complex than what’s required, but I can foresee us needing a number of custom rake tasks and shell scripts so this bundler.yml should have probably started life as a settings/commands/something_generic.yml
To test this part of my changes was pretty simple. I basically just stubbed the system calls to bundler to give certain return values and checked to make sure the correct program flow happened.
describe ".bundle_install" do
let :manager do
stub.instance_of(Specjour::Manager).project_path { "/tmp" }
stub(Dir).chdir(anything) { |args|
args.last.call # This yields to the block for Dir.chdir()
}
manager = Specjour::Manager.new
stub(manager).project_path { "blah" }
mock(manager).system('bundle lock')
manager
end
it "should perform a bundle lock" do
stub(manager).system('bundle check > /dev/null') { true }
manager.bundle_install
end
it "should check if there are gems required" do
mock(manager).system('bundle check > /dev/null') { true }
manager.bundle_install
end
context "when gems are required" do
before :each do
# Not a before :all as it needs to hook into the let hook above
stub(manager).system('bundle check > /dev/null') { false }
end
context "and there is a bundler YAML file" do
before :each do
config_file = ".specjour/bundler.yml"
mock(File).exists?(config_file) { true }
mock(File).read(config_file) { "" }
mock(YAML).load(anything) {
{ 'command' => "do it" }
}
end
it "should get the bundle command from the YAML file" do
mock(manager).system('do it > /dev/null')
manager.bundle_install
end
end
context "and there is no bundler YAML file" do
before :each do
mock(File).exists?(".specjour/bundler.yml") { false }
end
it "should perform a bundle install" do
mock(manager).system('bundle install > /dev/null')
manager.bundle_install
end
end
end
end
You can see that I stubbed our the Dir.chdir block to just yield directly to the call, otherwise it’ll throw an exception. Then I stubbed and mocked out the Kernel.system calls as necessary. Kernel methods are generally included into Ruby objects so you don’t stub Kernel, you stub the object that has the Kernel methods. Most of the testing is pretty basic, but I’d be keen to hear if I’m doing anything incorrectly!
This was my first major venture into adding functionality to a public project and it was good fun. I think it made me do a little better work than I might normally, it’s a great motivation to potentially have peers look at how you do things.
After bundling it all up and testing it here with over a dozen developers and even more machines I’m pretty happy with how it functions. I’ve made a pull request back to the original gem creator and hopefully he’ll like what I’ve done. In the meantime if you want to check it out then my Specjour is available on Git Hub.
Today I was fiddling with some code to get particular types of payments that are due on particular days and I ran across a couple of things I don’t want to solve again.
Firstly, the problem of being able to have default arguments to a block in ruby. It’s solved nicely in Ruby 1.9 but we’re using 1.8.x on our boxes at the moment. The work around is incredibly simple though and goes something like this :
That’s all there is to it really. You could go the whole hog with hashed attributes and so on but I think it starts to get a bit smelly if your anonymous functions are taking more than one argument.
The other issue I had was whether a named scope can include a join, and it can.
The coolest thing about all of this though (and I feel I’m very late to the party here) is that I now get to do things like :
Payment.credit_card.due_on(Date.today)
# or
Payment.credit_card.due_on
# or
Payment.due_on(Date.today + 1.month)
In the above I had just used the default argument to the due_on named scope to be today’s date.
I mean, I’d seen a bunch of tutorials on named scopes and how they worked but hadn’t found a use case in my work. I think it’s finally twigged for me though and will be making use of them a bit more in the future.
Just to cut to the chase, Specjour will easily allow you to run your tests in parallel on your local machine, or in a distributed way on your network. It cut our test suite from over 11 minutes to under 2.5 minutes. We went from one Macbook Pro to a Macbook Pro plus an iMac. Specjour is gooood, but continuing …
We were running into issues with our testing suite running past 15 minutes to run. It caused a number of issues, firstly it disuaded people from running the suite for small changes they thought would be okay, secondly people tended to slow down or not work at all when the suite was running. Given we were building tests into an existing code base and coverage was running generally under 30% this was already really worrying.
We’ve been concentrating primarily on functional testing with RSpec so I went looking for solutions for trying to parallelise RSpec specifically. I tried parallel-tests but I couldn’t get it to work. Then I heard about Specjour on a Hashrocket video so I decided to give that a try. It worked pretty well out of the box, I just had to plug up a wayward plugin, update the rsync config file for the project and change one of the files in the gem to call a different rake task.
Specjour essentially uses Bonjour to find workers on the network and dispatch tests to them and it uses rsync to synchronise the project you’re testing to the remote server. Once the sync has happened then runs the rake task to setup the databases, loads the environment and starts accepting the tests that are dispatched to it.
The instructions will work for the majority of people, I only had some hiccups due to our project. Essentially I gem installed specjour which gave me a specjour executable which I ran to create a dRb server that communicates via Bonjour. Then in my project I added specjour to my Gemfile (all managed via Bundler!) and ran rake specjour.
For me, I ended up having a few hundred failing tests, all to do with the Ruby version of a stack overflow (stack level too deep). It turned out it was caused by a plugin calling alias method chain twice so that the original method ended up being aliased to the replacement method and so there was an infinite loop. Once I applied a fix then we were only down to a few failing tests to do with seed data.
I changed the db_scrub.rb file to perform a different rake task for us as we do require some seed data and then we were on our way.
Previously on my Macbook Pro the suite ran in just over 11 minutes, after turning on a four core iMac and my Macbook Pro I had six workers ready to go and it took about 2.5 minutes to run our tests. I’d really like to crank up the other few iMacs and the 10 or so Macbooks and see what happens then.
I think the only changes we’ll require is to be able to specify which task is run to create the databases, other than that it works perfectly.
In closing I’d say I tried a number of distributed and parallel methods for running our tests and I like Specjour the best. Apparently it will run Cucumber tests as well, I’m hoping to move on to that soon enough.
I used this post today as a guide on how to get another iMac up and running so I thought it was probably a good point to chuck this up here, if only for our own reference.
We are starting with a 27″ iMac and a base Snow Leopard install.
The first step was to install Xcode from the Snow Leopard disc, you’ll find it in the Optional Installs part of the disc.
Ruby and RubyGems comes with Snow Leopard by default so we’ll use them. You’ll likely need to update the RubyGems system :
sudo gem update --system
Now we can install Rails :
sudo gem install rails
This will install the latest stable version of Rails. We need a specific version for some of our applications (until we’ve tested it under the newer version) and you can do this by adding the -v switch :
sudo gem install rails -v=2.3.4
After this we can switch on Apache and install passenger, the module that runs Rails on Apache. So turn on Web Sharing in the Sharing panel of your System Preferences.
This will compile the Apache module and give you some text to paste into your apache conf file to load it. I keep this in a separate config file in /etc/apache2/other/passenger.conf. It should look something like :
Next download the Passenger Preference Pane so you can control your Rails sites from the OS X Preferences area. You can get it from http://www.fngtps.com/passenger-preference-pane. You should now be able to run Rails websites on your local machine very easily.
I like to use a Pomodoro timer when pairing and the one I like the best for OS X is just called Pomodoro Desktop. It will keep people focussed and also can serve to keep the control swapping in a fairly regular way. You can find out about how Pomodoro works here.
We use Github for our repository and exchange is done via SSH, so you’ll need to generate an SSH key and put that into your Github account.
ssh-keygen
cat ~/.ssh/id_rsa.pub
Copy the output and put that into your public SSH keys in Git Hub.
When pairing with Git the ability to have both parties responsible for the work they’re doing is appreciated sometimes. Especially twelve months from now when you can’t remember why you did something. Not that you don’t comment your code, or always write beautifully understandable code. There is a nice gem called hitch. It comes out of Hashrocket and handles the ‘hitching’ and ‘unhitching’ of partners on a machine. We have a pairing station so generally we just want to change the pair that’s in operation, so it works well. You’ll need to be inside a git repository to do the initial setup. I cloned one of our projects and then you issue :
hitch -m
Which will prompt you for your main email address, this is the email address that all devs in your team receive, or a group email of some sort.
Now to hitch two devs you’d do :
hitch dev1 dev2
This will prompt you for names of the dev1 and dev2 users so that they’ll be displayed in your commit. It’ll also associate your commits to the email address devs+dev1+dev2@company.com if you setup devs@company.com as the email address when you issued the hitch -m command. If you setup gravatars for all your devs+user1+user2@company.com email addresses then you’ll get a nice picture in Github too.
I think that’s about it really. We obviously just install gems as required and make customisations on a per developer need. We generally use Textmate for editing so this means installing a theme (we use the Railscast theme) and the relevant bundles for Rspec, Haml and Sass.