Figuring out a branching strategy

To be honest, before I started working at Frogmetrics, I didn't know how project branches were managed. I was working at a research lab, and most of what we did was prototyping. After we demonstrated that it worked, we threw it over a wall, and then it was someone else's problem (poor them). A lot of the time, I was doing work on my own, so there was no (perceived) need for source control. The only time we used source control was when I was working on the New Horizon spacecraft. And even then, someone else managed the branches. Because it was SVN, we all mainly worked off trunk.

Until recently, I never contributed to an open source project either. Therefore, I really had no idea when to branch. So when we started working on the analytics, we really had no idea what a good branching strategy would be.

Googling didn't help, because either everyone else doesn't call it "branching strategy", or everyone already knew how to do it. I eventually figured something out though.

Last week, after talking to AJ of Scoopler about git, he ended up asking about branching strategy. It became apparent that branching strategy wasn't an obvious thing, so I decided to write something here. This is obviously not the only way to do it, so if you have other suggestions, by all means, comment.

At first, we didn't know what we were doing. We knew that we wanted to have a branch that had the same code on the production server, and another branch where we're working on the 'next' version. So we had a branching strategy that looked like this:

As you can see, we branched every time we deployed weekly. This gave us the option of doing bug fixes on the every deploy version, while keeping a working branch. However, this was a terrible way to do things. This branching strategy required you to keep merging back bug fixes that you had made earlier. In addition, we were using bug tracking software to track all the issues from week to week, which results in much ticket shuffling and overhead.

Now, we're doing this. Locally, we still branch for every feature that we're working on. And when there's a major set of features that need to be implemented by more than one developer, we create a remote branch for it that we push and pull to/from.

Thus, we're treating master as the golden copy of the code base. It is always deploy-able, passes all tests, and is perfect code as we know it. This allows us still the advantage of doing bug fixes and deploying independent of what features are currently on deck or in the hole, and yet we don't have to do merges every time are about to do a new version. We simply (and somewhat arbitrarily) tag versions as we go along, and only merge feature branches back. When we can, we rebase the branches to keep the history clean. In addition, we try to make our commits atomic and about one thing, rather than one feature set. That way, it makes it very helpful to remove a piece of code, cherry-pick a changeset to another branch, or find an offending commit that broke something.

So far, it's worked pretty well, but it might evolve as we go on.

Well, hope that helps. This post wasn't as fun to write, but it was something I hadn't see too much of out on the web, so I figured I'd contribute. Fun times.

Unable to freeze rails due to problem in rake task

I think the current version of stable Rails is 1.2.3. For those of you using this version, rather than Edge Rails, there's a little gotcha in the rake tasks.

Since I'm on a shared host, it's good practice to freeze your version of rails into the vendor's directory. You do this by using a rake task, per "rake rails:freeze:gems" But before you do that, if you're using SVN, you'll want to use "svn delete" to remove the vendors/rails directory. None of the rake tasks use SVN delete. They all use "rm -rf", which in my experience makes SVN freak out if the .svn directory is gone.

However, even with that done, freezing a new version of gems was failing.

It was looking for rails version 1.4.0, and not being able to install it. And even worse, when you try to run rake again, it said it couldn't find it!

Well, the latter was simple. A failed freeze leaves a blank vendor/rails directory, and if you look in 'config/boot.rb', it says:

if File.directory?("#{RAILS_ROOT}/vendor/rails")
require "#{RAILS_ROOT}/vendor/rails/railties/lib/initializer"
else
require 'rubygems'
...blah blah blah..
So make sure you remove vendor/rails.

The latter took a little bit of work digging around the rake tasks, and though it wasn't hard, I wasted about an hour. It ends up that the culprit is that the default rake task uses Gem.cache.search('rails'), which returns all gems with the name 'rails' in it.

I have a couple gems installed with the word 'rails' in it.

rails (1.2.3, 1.2.0, 1.1.6)
rails_analyzer_tools (1.4.0)
railsbench (0.9.2)

So it took the latest one, which was 1.4.0, and tried to install rails 1.4.0, which doesn't exist!

To hot fix it, the railities/lib/tasks/framework.rake file, under the freeze namespace and gems task, change "Gem.cache.search" to "Gem.cache.find_name"

That way, it only finds 'rails', and not all the other games with 'rails' in the name of the gem. This problem is solved in edge Rails, so no need to submit a patch. Tip!

Mobtropolis Public Release

I've been working on Mobtropolis for about 10-12 weeks now. It's was finally released last week Tuesday. It's something that makes people expand their world by helping them discover and share local adventures around them. The easiest way to think about it is as a dynamic large-scale photo scavenger hunt or a photo-dare site that helps you expand your world--hopefully for the better.

Behavior is hard to change, so it's framed slightly in terms of a game. The basic mechanics should be familiar to those that frequent social news sites. Anyone can submit scenes or vote them up. The higher something's voted, the higher its visibility to others. Anyone can do a scene and take a photo as proof. They can then send it in via their camera phone, or upload it from their digital camera when they get back to desktop. Their friends who voted for a scene will then get an email with the photo attached.

It's been oddly thrilling to get photos of people doing scenes that you submitted.

There's still a lot of work to be done on it. Eventually, I hope to marry the virtual and the real in a tighter loop and better integration with mobile devices. However I'm putting it out in according to startup mantras of "Release early, then iterate like crazy". So check it out, and if you'd be so kind, give me some feedback, good or bad, so I can make it better.

http://www.mobtropolis.com

Enjoy!

Email attachments not downloading on Textdrive servers

This was one of those things that had me reeling in pain a month ago. I couldn't figure out what exactly was wrong...and I've run into this problem twice! The symptom was ActionMailer, which in turn is using Rails TMail extensions, didn't seem to be downloading attachment correctly on the server. It worked correctly on my development machine, but when I run it on the production server, it just didn't want to work. All attachments would be two bytes. I had suspected a host of other things, but I'll spare you the stupidity. It's a long chain from mail attachment to webpage, and it took a lot of work to narrow it down to Rail's TMail extensions.

It ends up that this is only an issue if base64.so is on the user's system. According to this two year old post on Textdrive, ActionMailer's TMail will choose between two implementations of base64, a C one or a Ruby one, based on the existence of the library base64.so

The problem is, the ruby version assumes that a string will return, and the C version assumes an array will return. And because of that, calling first() on a string only gives the first byte or two, as opposed to calling first on an array, which gives the first element (a string)

I hotfixed it, because I couldn't get a plugin to work and override the classes. I didn't spend too much time with it, so if anyone else wants to show me how to do the plugin for this simple fix, I'd be happy to learn.

In base64.rb in the rails directory (I froze my rails in my vendors directory) I changed the code to:

module TMail

  module Base64
    def rb_decode( str, strict = false )
      str.unpack('m').join
    end    
  end

end

And subsequently, in unquoter.rb

module Unquoter

  class << self
    def unquote_base64_and_convert_to(text, to, from)
      convert_to(Base64.decode(text), to, from)
    end    
  end

end

That should fix your problems for now. I was using rails 1.2.0 and it still hasn't been fixed. I know someone else submitted a patch for it already (Rails bug #7861). Tip!

Deploying backgroundrb

Often times you don't get to control what's on the server. Slave or daemon gem required by backgroundrb might not be on your server. To get around that, I froze the gem in the vendors directory. (ie. unpack the gem)

Then in the backgroundrb script under your script directory, add the following right before require 'backgroundrb_server'

# Load gems
if BACKGROUNDRB_STANDALONE == false
  rails_root = BACKGROUNDRB_ROOT
  gem_path = "#{rails_root}/vendor/gems/slave-1.2.1"
  lib_path = "#{gem_path}/lib"
  init_path = File.join(gem_path, "init.rb")
  $LOAD_PATH << lib_path
  
  eval(IO.read(init_path), binding, init_path)
end

It'll shoot up some warning, but you can ignore those, or write your own silent_warnings() method.

"MySQL server has gone away" on textdrive

Debugging is always hard, because you have to understand what's going on. In this day and age of leaky abstractions, there's just always more and more to know. This is why I think concepts are important. If you know concepts, you can more readily figure out details.

So I had a quizzing error last weekend that I was scratching my head over for about two days. This was mainly because I was getting the errors from backgroundrb. It would just hang with no exceptions reported. As it turns out, there's a bug in Backgroundrb 0.2.1, the latest version.

Thanks to Mathais on the Backgroundrb mailing list, my problem was exactly as he describes. I monkey patched it and the backgroundrb server log started spewing errors out. It was about MySQL servers going away.

After reading about why MySQL goes away at all, I figured out that one needs to check MySQL's interactive_timeout setting. The database will drop the connection to it from the client (in this case, the web app), if there has been no activity for at least that amount of time. By default, it is set to four hours. On the server I put the app on, however, it is set to 10 minutes.

There were a couple solutions to this. One could be, as I posted before, to retry the connection. The other is a setting that I found in Rails.

ActiveRecord::Base.verification_timeout = 570

I put this in the environment.rb file under config to keep the connection to the database alive. I set the timeout to be under the interactive_timeout, so that Rails will keep telling the MySQL server that it's still around.

I don't know if this is exactly a good idea, since on a shared server, that means everyone will be holding on to connections they're not using. I'm not sure what the performance implications are for a long standing connection is, if there is any. But for now, it seems like it's working.

Avoiding the SUDO police with Capistrano

When deploying on a shared host, often times, you won't be able to sudo anything. I was originally thinking that I had to override the cleanup task in cappy, but a quick look in google found: Avoiding the SUDO police with Capistrano. You can simply "set :use_sudo, false" in your deploy.rb. Tip!

Capistrano tasks for BackgrounDRb — Bryan’s Bytes

Capistrano tasks for BackgrounDRb — Bryan’s Bytes

Here's a good little snippet I found for running BackgroundRb through Capistrano. Not much commentary from me, other than hurray~ I had wondered about why it wasn't working. I didn't know nohup existed as a command--I've always used the trailing '&'. Goes to show you that a little background goes a long way.

Capistrano and Mongrel are easy to use, but deployment is still hard

This summary is not available. Please click here to view the post.