(Aha!) Part of the reason why great hackers are 10 times as productive

I knew in college that some dudes were faster than I was in terms of programming. Since peer programming wasn't exactly encouraged in college, and at work I did mostly prototyping work, I never really knew how fast other programmers worked.

So when I read Paul Graham (and Joel's) claim that great hackers are at least ten times as productive as average programmers (too lazy to cite right now), I was kinda shocked. Surely, ten times is an order of magnitude! Something that takes an average programmer a 40 hour week to do the great hacker can do in a 4 hour afternoon?

I wondered about that, since there are times when I get stuck on something, then I just start stabbing around randomly out of frustration. I had assumed that great hackers were faster only because they had either the experience or insight to side-step whatever I was doing wrong.

But lately, I've been re-reading everyone's essays that write about programming productivity. And one thing that caught my eye the second time around was when Paul Graham was talking about bottom up programming and how he didn't really believe in objects, but rather, he bent the language to his will. He was building new blocks for himself so he could think about the problem at a higher level of abstraction.

This is basic stuff! I mean, that's the whole point of higher-level programming. When you refactor methods out, you're creating a vernacular so that you can express the problem in terms of the problem domain, not in terms of general computing. This is much like if you want to drive a car, you'd want to be able to step on the gas, rather than time the firings of the pistons. And if you want to control traffic in a city, you'd rather tell all cars to go to a destination, rather than stepping on the gas and turning the steering wheel for each one.

But taken into the light of bending a language to your will, it makes it more clear for me as to how great hackers are ten times as productive. Great hackers are productive not only because they know what problems to sidestep and can problem solve systematically and quickly, but they also build a set of tools for the problem domain as they go along. They are very good pattern recognizers and will be able to generalize a particular pattern of code, so that they can use it again. But not only that, great hackers will create an implicit understanding attached to the abstraction, ie. what we might call common sense.

A case in point. Before Ruby, I'd used for loops over and over again, never really thinking that I could abstract a for loop. It wasn't until they were taken away in Ruby did I realize that map, inject, and their cousins are all abstractions of the for loop. When I see "map" I know that it performs a transformation on every element. But I also know that the array I get back will be the same size, that each element operation doesn't affect other elements, among other things. These are implicitly stated, and they allow for shorter code.

When that happens, you can simply read "map", and get all the connotations it comes with, and hence it comes with meaning. It also becomes easier to remember, since it's a generalized concept that you can apply in different places in the code. The more times you use it, the easier it is to remember, instead of having specialized cases of the same kind of code where the behavior is different in different parts of the code.

A great hacker will take the initial time upfront to create this generalized code, and will save in the long run being able to use it. Done over and over again, it all adds up. So it's not that for any given problem, a great hacker will be done in 4 hours what it takes an average programmer 40 hours, but that over time, a great hacker will invest the time to create a tools and vocabulary that lets him express things easier. That leads to substantial savings in time over the long haul.

I hesitated writing about it, as it's nothing I (nor you) haven't heard before. But I noticed that until recently, I almost never lifted my level abstraction beyond what the library gave me. I would always be programming at the level of the framework, not at the level of the domain. It wasn't until I started writing plugins for rails extracted from my own work and reading the Paul Graham article that a light went off for me. It was easier to plug things like act_as_votable together, rather than to still mess around with associations (at the level of the framework). I still believe you should know how things work underneath the hood, but afterwards, but all means, go up to the level of abstraction appropriate for the problem domain.

DSLs (Domain specific languages) are really just tool-making and vernacular creation taken to the level of bending the language itself. It's especially powerful if you can add implicit meaning to the vernacular in your DSL. It's not only a way of giving your client power in their expression, but it's also a refactoring tool so that you can better express your problem in the language of the problem domain. Instead of only adding methods to your vernacular, you can change how the language works. It was with this in mind that I did a talk on DSLs this past weekend at the local Ruby meetup. First part is on Dwemthy's Array, and the second is using pattern matching to parse logo. Both seemed pretty neat when I first read about it. Enjoy!

---

DSL in Ruby through metaprogramming and pattern matching

Communicating your intent in Ruby

I've been using Ruby most everyday for about two years now. While I'm no expert, I know enough to be fairly productive in it. And beyond liking the succinctness and power that you often hear other people talk about, it's made me a better programmer. But there's an aspect of Ruby that worries me somewhat.

To start, programming is recognized rightfully as a means to build something from pure thought. But it's also a form of communication, to other programmers that will touch your code later, and to yourself when you look at it months from now. We're at a point that other than embedded and spacecraft programming, we have the luxury of using programming languages that focus ease for the programmer, rather than for the ease of the machine. Fundamentally, that's the philosophy that Ruby takes.

And while Ruby's nice in a lot of ways, I'm not sure about how it communicates an object's interface. When you're allowed to modify objects and classes on the fly, how do you communicate interfaces between modules you mixin and methods/modules you add? By interface, I mean, how do you use this class so that it does what it's suppose to? Normally, it's pretty obvious--you look at the names of the methods declared in the code. A well-written class has public methods exposed, or you look at its ancestor's public methods. You might need some documentation to figure out how to call them in the right order, but generally, you have some idea just by looking at the method signatures.

However, when you throw mixins and metaprogramming in the mix, it becomes less easy to tell just from looking at the method signatures in the code--the structure of the code. You have to specifically read the code, or you have to rely on someone who knew intent to document it in detail.

An example communicating interfaces for mixins: the module Enumerable contains a lot of the Collections related methods. The cool thing is that if you wanted these functions in your own class, all you have to do is define each() in your class, mixin the Enumerable module, and you get all of these "for free". However, outside of documentation explicitly stating it, it's not as immediately obvious in method signatures that this is what you have to do in order to use it. It's only after scanning through the entire code that you notice each() being used for all the methods.

Of course, Ruby contains enough metaprogramming power to protect yourself against this. one can do something like this:

class MethodNeededError < RuntimeError
  def initialize(method_symbol, klass)
    super "Method #{method_symbol.to_s} needs to be in client class #{klass.inspect}"
  end
end

module Enumerable
  def self.included(mod)
    raise MethodNeededError.new(:each, mod) unless mod.method_defined?(:each)
  end
end

This only works if you put the include after you define each(). That's just asking for trouble when the order of your definitions in your class matter.

A fair number of people are writing mini-DSLs in ruby using metaprogramming tricks. One of the common ones is the use method_missing to define or execute methods on the fly. ActiveRecord's dynamic finds are implemented this way. The advantage of communication of interface here in the structure of the code is obvious. Unless it was documented well, you can't tell just by looking at the method signatures.

Why do I harp on interface signatures? I mean, in the instance of requiring each(), it works by just letting it fail in the enumerated methods, since it'll complain about each itself. In the instance of method_missing, just read the regex in the body. While these are true, none of these allow for rdoc to generate proper documentation. The whole point of documentation is to show you the interface--how to use that piece of code. I'm just afraid that given Ruby's philosophy of being able to write clear, powerful, and succinct code, it might fall short when people start using these metaprogramming tricks like alias_method_chain and method_missing more and more. Maybe rdoc needs to be more powerful and read code bodies for regex in method_missing?. It already documents yields in code bodies, but that seems awfully specific.

I'm not a exactly a fan of dictating interfaces like in Java. When you're first coding something up, you're sketching, so things are bound to change. Having plumbing like interface declaration gets in the way, imo. However, when something's a bit more nailed down, it'd be nice to be able to communicate to other programmers your intent without them having to read code bodies all the time.

In the end, I side on flexibility. However, I kinda wish Ruby had some type of pattern matching for methods so I didn't have to read method_missing all the time. But then again, that would be messy in all but the simplest schemes. Can you imagine a class that responded to email addresses as method calls? I guess I'd have to file this one under "bad ideas"

Don't reopen ActiveRecord in another file

The power of Ruby lies partially in how one can reopen classes to redefine them. Besides namespace clashes, this is usually a good way to extend and refine classes to your own uses. However, last night, I got bitten in the ass trying to refactor a couple classes. In Rails, you're allowed to extend associations by adding a class the association call.

class User < ActiveRecord::Base
  has_many :stories, :through => :entries, :source => :story, 
    :extend => StoryAssociationExtensions
end

where StoryAssociationExtensions is a class module holding methods, like expired() that I can perform on the challenges association, so I can do stuff like

  @user = User.find(:first)
  @user.stories.expired  # gives all expired stories

So when refactoring and cleaning up, I renamed StoryAssociationExtensions to AssociationExtensions and reopened up Story class and put it in there. I just wanted to clean up the namespace, and put the association extensions somewhere that made semantic sense. Naturally, I thought putting association extensions for a class belongs in a class. Well, it doesn't work. And don't do it. Hopefully, I'm saving you some pain.

class Story < ActiveRecord::Base
  module AssociationExtensions
    def expired
      self.select { |c| c.expired? }
    end
  end
end

Well, this works if you've reopened the class within the same model file, story.rb in this case. However, if you reopen the class in another file elsewhere, your model definition won't get loaded properly, which leads to associations and methods you defined not to exist.

So imagine my bewilderment when associations didn't work on only certain ActiveRecord Models. In addition, they worked on the unit tests and script/console, but didn't work when the server was running. All that at 3am in the morning. :(

Good thing for source control, so I could revert (but I have to say, svn isn't as easy to use as it could be).

I ended up creating a directory in model called collection_associations and putting the associations in there under a module CollectionAssociations namespace. Not exactly the best arrangement but it'll do for now.

I'm still not sure why ActiveRecord::Base instances don't like being reopened, but I'm guessing it has something to do with only getting loaded once. If anyone has an explanation, I'd like to read about it.

free warning!

State change observer for ActiveRecord

When I started writing some code recently, I noticed that my controllers were getting fat. There was much to do, but there was a bunch of stuff in there that didn't have anything to do with actually carrying out the action--things like sending notifications. ActiveRecord already has observers to take action on certain callbacks. However, what I needed was to take actions on certain state transitions. Not seeing any immediate solutions in the Rails API, I decided to test myself and try writing one. I was bored too. So while I'm not sure if it was worth the time writing it, it certainly was kinda interesting. Here's what I came up with:

Just as a contrived example, let's say we are modeling the transmission of a car. It has three modes: "park", "reverse", "drive". We want to send a notification when a user tries to change it from "reverse" to "drive", but not when he tries to change it from "park" to "drive". If it didn't matter, and we just wanted to send notifications when the state changed to drive, we'd just use the observers that came with ActiveRecord. But since we do care where the state transition came from, here's what I came up with:

class CreateCarTransmission < ActiveRecord::Migration
  def self.up
    create_table :car_transmission do |t|
      t.column :engine_id, :integer, :null => false
      t.column :mode, :string, :null => false, :default => "park"
    end
  end

  def self.down
    drop_table :car_transmission
  end
end

class CarTransmission < ActiveRecord::Base
  include StateTransition::Observable
  state_observable CarTransmissionNotifier, :state_name => :mode
end

So then for my notifier I have:

class CarTransmissionNotifier < StateTransition::Observer
  def mode_from_drive_to_reverse(transmission)
    # send out mail and flash lights about how this is bad.
  end
end

And that's it. Whenever in the controller, I change the state from "reverse" to "drive", lights will flash and emails will be sent out condemning the action, and my controllers stay small and lean.

class CarController < ApplicationController
  def dismantle
    @car = Car.find(params[:id])
    @car.update_attribute :mode, "reverse"
    @car.update_attribute :mode, "drive"
  end
end

So where's the magic? It took a bit of digging around. There were two major things I had to do. I had to insert observers during initialization and I had to override setting of attributes to include an update to notify observers.

ActiveRecord doesn't exactly allow you to override the constructor. I don't think I tried too hard to mess around with it. Looking on the web, I happened upon has_many :through again, where he has some good tips that helped me through Rail's rough edges. While I didn't exactly follow his advice, I did find out about the call back, :after_initialize. It must be something new, because I don't see it in the 2nd edition of the Rails book, and the current official API doesn't list it. Other Rails API manuals seem to be more comprehensive, like RailsBrain and Rails Manual.

Then overridding attributes has always been a bit of a mystery. I found a listing of the attribute update semantics, which was helpful to figure out what I was looking for, but it was false, in that you can't use the first one (article.attributes[:attr_name]=value) to set an attribute. Looking in the Rails code for 1.2.3, it shows that attributes is a read_only hash. But it's right that you should override the second one (article.attr_name=value), since update_attribute() and update_attributes() depends on it.

Again, it ends up that the function I was looking for wasn't found in the official API as a method, other than a short mention in the description of ActiveRecord under Overriding Attributes, which makes it harder to find. Ends up that we can use write_attribute().

So that's pretty much it. Using some standard meta-programming like how plugins do it, you wrap it up, and it's pretty simple:

require 'observer'

module StateTransition
  module Observable
    class StateNameNotFoundError < RuntimeError
      def message
        "option :state_name needs to be set to the name of an attribute"
      end
    end

    def self.included(mod)
      mod.extend(ClassMethods)
    end

    module ClassMethods

      def state_observable(observer_class, options)
        raise StateNameNotFoundError.new if options[:state_name].nil?
        state_name = options[:state_name].to_s
        
        include Object::Observable

        define_method(:after_initialize) do 
          add_observer(observer_class.new)
        end

        define_method("#{state_name}=") do |new_state|
          old_state = read_attribute(state_name)
          if old_state != new_state
            write_attribute(state_name, new_state) # TODO yield the update method
            changed
            notify_observers(self, state_name, old_state, new_state)
          end
        end
      end

    end
    
  end

  class Observer
    def update(observable, state_name, old_state, new_state)
      send("#{state_name}_from_#{old_state}_to_#{new_state}", observable)
    rescue NoMethodError => e
      # ignore any methods not found here
    end  
  end

end

I had a difficult time figuring out how to define methods for an instance of a class. The only thing I came up with was to use define_method, or to include a module with instance methods in them. instance_eval() didn't work. The meta programming for ruby gets rather confusing when you're doing it inside a method--it seems hard to keep track of which context you're in.

So if you can make a use of this, great. If you think it's worth moving it into a plugin, let me know that too. If you know of a better way, by all means, let me know. tip!

Where do you put the rules of Monopoly?

It's apparently a favorite interview question. I have to admit, the first response in my head wasn't a great one, which was "everywhere".

What are game rules? When browsing the rules of popular games like Monopoly and Scrabble, they seem to follow a similar format:

• The initial conditions of the game (the setup)
  
• Then given a condition,
• the set of allowable actions for the player to do
  
• the effects of the condition
  

I embarked on somewhat of the same problem, though on a much smaller scale. I wanted to implement a point system that gave points to those that participated in a Rails app. My first reaction was to make a Rule abstract class, and have different rule classes that subclass it. Something like this (variable names have been changed to protect the innocent):

class SceneRules
  def initialize(model)
    @scene = model
    @user = @scene.submitting_user
  end

  def on_submit
    @user.points += 1
  end
end

class SceneshotRules
  def initialize(model)
    @sceneshot = model
    @user = @sceneshot.sceneshot_uploader
  end
 
  def on_submit
    @user.points += 10
    @scene.submitting_user.points += 5
  end

end

Then I'd be able to call it from the controllers. However, on second thought, it's rather ugly, since I'd be updating the karma everywhere in the controllers. If I understand cross-cutting concerns correctly, scoring karma would be a good example of one. I suppose it's a good candidate for aspect orientated programming, so I scrapped the code above.

Logging is often cited as the poster-child problem to solve with AOP. Logging needs to be done everywhere in the code, but it really has nothing to do with the responsibilities of the class that it's performed in. So you have the same code doing the same thing, duplicated everywhere because there's no one place to put it to make things easy to change.

By the same token, game rules and scoring are of the same nature. And because game rules involve lots of different objects at once, and scoring is interspersed throughout, I think that makes it a good candidate for AOP. However, Ruby has no such direct support for AOP. Instead, the closest thing we have are observers, before/after/around filters (in Rails), and some meta-programming.

I wanted something that allowed me to list out rules like games like Monopoly and Scrabble. I'd have a setup, and some conditions and their effects. Scoring is simplified here because the only time you can score is when one of the models is created or changes state. This is a good fit to the observers and the filters available in Rails.

class ScoringRules < ActiveRecord::Observer
  observe Sceneshot, Scene
  include Rules   

  setup { :scene => Proc.new { |sceneshot| sceneshot.scene } },
        { :sceneshot_uploader     => Proc.new { |sceneshot| sceneshot.sceneshot_uploader },
          :scene_submitting_user => Proc.new { |sceneshot| sceneshot.scene.submitting_user } }

  rule :after_create_sceneshot do |board, players|
    players[:scene_submitting_user].score += 5
  end

  # put more rules here

  def after_create(model)
    rule_dispatch(:after_create, model)
  end

end

Here, the Rules module is what encapsulates the setup, rule, and rule_dispatch calls. I needed setup so that I can access different "game elements" (the board and the players) to update the scoring. It basically stores the setup as a list of lambdas that it can execute at a later time when the rule needs to be executed. Now, when a model is created, we ask the rule dispatcher to figure out which rules execute based on the rules we've named, and then execute the attached block. The block is passed a hash of different game pieces that it needs to update the score and the game conditions. That's it.

I thought it was an interesting way about it and probably warranted some criticism. Is there any particular disadvantage of doing it this way? And if you can think of a way to not have to explicitly state the model relationships in the setup, that'd be nice. half-tip!

Syntactic sugar for dealing with empty containers

In any web application, we're often just reading a collection of rows from the database and displaying it in the browser. Often times, we'll have code that looks like this:

<% unless @friends.empty? -%>
  <% @friends.each do |friend| -%>
    <li><%= h friend.username %></li>
  <% end -%>
<% else -%>
  No friends yet
<% end -%>

I don't know why, but this kinda gets to me, and doesn't look all that neat. I probably attribute it to having to upgrade and maintain a piece of C server code that was nested 8 or 9 layers deep all in one huge main(). It might be counter-productive, but I tried to see if I could do better.

<% if @friends.each do |friend| -%>
  <li><%= h friend.username %></li>
<% end.empty? -%>
  No friends yet
<% end -%>

Well, this is kinda nice in a way that it's only one hierarchy deep. When I look at it, one section is for what to display when there are elements in the list, and one is for when there isn't. I suppose your mileage may vary. However, I didn't like the "if" in front. It obscures the intent of displaying the list. So, in the pursuit of more counter-productivity and perhaps in the spirit of pseudo-altering the language, I tried this out:

<% @friends.each do |friend| -%>
  <li><%= h friend.username %></li>
<% end.empty do -%>
  No friends yet
<% end -%>

Well, that worked. I kinda like it. Since the message was so simple, I had wanted empty() to take a message, and just display it, but because it's a "%" and not a "%=", the message won't get displayed, so I had to do it in a block. In a way, it's almost like being able to write my own "else" statement. If I had used curly braces instead of "do/end", it might look pretty close. Here's the code for empty:

class Array
  def empty(message = "")
    if self.empty?
      return block_given? ? (yield message) : message
    end
  end
end

Like it? Hate it? Tip!

Is preloading child tables always a good idea?

Optimization isn't something you should do too early on, but I think a little house cleaning every so often to make sure your pages aren't ridiculously slow is healthy. With any optimization task, you'd want to benchmark the results and see if there's an actual gain. The very basic tool for benchmarking is the ordinary script/performance/benchmark. The easiest to find analysis tools is the rails_analyzer gem. The last time I used rails analyzer, it wasn't that easy to use. The command line arguments seemed arcane. But its bench tool, which can benchmark controllers as opposed to just object models, is fairly easy to use.

Using the bookmarking example from before, let's say you have something like:

class SceneController < ApplicationController
  def list
    @books = Book.find_books
  end
end

class Book < ActiveRecord::Base
  def self.find_books
    find(:all, :include => [:bookmarks], 
               :conditions => ["books.created_on > ?", 6.month.ago])
  end

  def bookmarked_by?(user)
    self.bookmarks.select { |bm| bm.owner_id == user.id }.empty? ? false : true
  end
end

In the listing of books, one would display whether it's actually bookmarked by a user or not. Normally, without the :include, the listing would make repeated queries to the DB every time it displayed a book list element, since it will use bookmarked_by?(user_id) to determine if a user bookmarked the book. So instead of just 1 query, it would make n + 1 queries.

Preloading child tables isn't necessarily wise all the time. It really depends on what you intend to do with the data after you fetch it. As the Agile rails book warns, preloading all that data will take time. If you look at your log files, you'll see that it's a significant amount.

If you're only going to load a limited number of these book list elements on a single page at a time, it actually might make sense to forgo preloading of child tables, and just use a find() instead of a select.

class SceneController < ApplicationController
  def list
    @books = Book.find_books
  end
end

class Book < ActiveRecord::Base
  def self.find_books
    find(:all, :conditions => ["created_on > ?", 6.month.ago], 
         :limit => 20, :order => "created_on desc")
  end

  def bookmarked_by?(user)
    Bookmark.find(:first, 
                  :conditions => ["book_id = ? and owner_id = ?", id, user.id]) ? true : false
  end
end

And if you're going to display counts of arrays, but all means, use counter caching. It's easy to do (as long as you follow instructions!), for most situations.

Intuitively, if you want to display over a certain n number of book list elements, it makes more sense to use :include and select it. However, I wanted to point out that when you make decisions like this, you'll always want to measure the load times, because you earn what you measure.

Also, use the right number of runs. Too short number of a number of times you run a function, the more variation you'll have in your benchmarks. Let's say that you get two numbers for two different methods.

$ bench -u http://localhost:3000/method1 -r 50 -c 5
50....45....40....35....30....25....20....15....10....5....
Total time: 240.383527755737
Average time: 4.80767055511475

$ bench -u http://localhost:3000/method2 -r 50 -c 5
50....45....40....35....30....25....20....15....10....5....
Total time: 156.147093772888
Average time: 3.12294187545776

So it's obvious that method2 is better right? Well, not necessarily. While benchmarks only show averages, you'll need to pay attention to standard deviations. The bigger the standard deviation, the more runs you'll need to figure out the average load time, and the number of decimal points you can trust. That way, you can figure out whether the difference in load times is statistically significant or not.

That way, you can ascertain whether the optimization you made were worth the trouble or not. tip!

Javascript scoping and this

Javascript has been a surprising language, mostly because the difference between my experience with it now, and when I touched it in high school. Going along with the theme of seeing everything as the proverbial functional nail, there's been a bunch of things to get use to, such as how scoping work works in javascript.

Javascript is a lexically scoped language (scope determined when function is defined, not when it's executed). This much, I think I get.

However, unlike Java and C++, "instance methods" of Objects doesn't include attributes in its scope. One has to explicitly refer to it. So let's say I have something like the code below. Just something simple, where one wants to process each element in a list. Of course, one can do this with a for loop and it wouldn't be an issue, but since playing with Ruby, I've found that iterators were more clear when writing code. It's probably bad design, but for the sake of argument, let's say that tile's draw() method takes the Renderer object as an argument.

function Renderer() {
  this.tiles = new Array();
}
Renderer.prototype = {
  draw: function() {
    this.tiles.each(function(tile) {
      tile.draw(this);
    });
  }
};

This will not work, because apparently, "this" inside the anonymous function of each() doesn't refer to renderer, but (I think) to the anonymous function. So draw() will usually complain something about how the renderer doesn't have the right properties.

This is a bit odd, since this would work in Ruby. The only way I've found around this is just to throw "this" into a variable, then refer to that variable.

function Renderer() {
  this.tiles = new Array();
}
Renderer.prototype = {
  draw: function() {
    var my_renderer = this;
    this.tiles.each(function(tile) {
      tile.draw(my_renderer);
    });
  }
};

According to javascript's scoping rules, this works. But to me, it looks rather ugly. Anyone know of an easier way to get around it?

Using anonymous functions inside functions

I was reading an article on Javascript and its use of first class functions. I'd quote it if I could find it again, but basically, it extolled the virtues of being able to create anonymous functions, blocks, and closures easily in Javascript. One of the things that it claimed was that creating local functions only used within a method makes the code much more readable. I was skeptical, but as I was refactoring this function, I thought I'd give it a shot even though I was coding in Ruby. Ruby also allows for the creation of first class functions.

It's just a helper function that generates a bookmark button--however, what gets generated depends on whether a user is logged in and whether a user had already bookmarked the page.

  def bookmark_button(page, user, indicator_id)
    if page.bookmarked_by? session[:user]
      "Bookmarked!"
    elsif page.done_by? session[:user]
      link_to_remote image_tag('heart_add.png', :height => 16, :width => 16) + " Bookmark again!", 
                               :url => { :controller => :bookmark_list, :action => :add, :page_id => page.id  }, 
                               :loading => "Element.toggle('#{indicator_id}')", 
                               :complete => "Element.toggle('#{indicator_id}')"
    else
      link_to_remote image_tag('heart_add.png', :height => 16, :width => 16) + " Bookmark!", 
                               :url => { :controller => :bookmark_list, :action => :add, :page_id => page.id  }, 
                               :loading => "Element.toggle('#{indicator_id}')", 
                               :complete => "Element.toggle('#{indicator_id}')"
    end
  end

Well, I thought it was pretty straightforward to read, though half of it was duplicated code. It wasn't very DRY. So instead of fudging around with another way to structure the control flow, I tried my hand at using blocks. I ended up with this:

  def bookmark_button(page, user, indicator_id)
    bookmark_link = Proc.new { |label| 
      link_to_remote "#{image_tag('heart_add.png', :height => 16, :width => 16)} #{label}", 
                     :url => { :controller => :bookmark_list, 
                               :action => :add, 
                               :page_id => page.id  }, 
                     :loading => "Element.toggle('#{indicator_id}')", 
                     :complete => "Element.toggle('#{indicator_id}')"
    }

    if page.bookmarked_by? session[:user]
      "Doing this"
    elsif page.done_by? session[:user]
      bookmark_link.call("Bookmark again!")
    else
      bookmark_link.call("Bookmark!")
    end
  end

As you can see, I put the link_to_remote() code into a block, since the only difference between the two is the label. In addition, it is possible for the block to use variables that are in this method, but outside the block, even if the execution is outside the scope of this function, if the block is ever passed outside of bookmark_button. I know in javascript, this causes memory leaks in IE. I haven't heard that it's a problem in Ruby yet.

But in this case, I'm just creating a function inside a method to make things more readable and cleaner. I think for the most part, it worked pretty well. As long as it's not overdone (notice I didn't create a proc for the labels), it should make for more readable code, and you don't pollute the class namespace with methods that are only used in this one method. I should get use to this. tip!

Nerd time, issue 7

Hi all,

There's more interesting things lately than usual. Here's another set of stuff to discover. I've put the more easily digestible stuff up top. This time, it's a lot geekier. The lower you go, the nerdier it gets.

Humanized Endo--CLI + GUI
The Humanized interface combines GUI and CLI, which really reminds me of emac's interface...but tons prettier. You can eval text as commands on the page in emacs, as well as executing commands. They basically want to do away with the desktop metaphor. The second link is a presentation. The guy presenting, Asa Raskin, is Jef Raskin's son...Jef is the guy that started the Macintosh before Steve Jobs took over.
http://www.humanized.com/
http://www.humanized.com/weblog/2007/05/18/die_desktop_die/

Multi-touch interfaces
More demo eye candy from Jeff Han.
http://www.thelastminuteblog.com/2007/03/19/new-jeff-han-video-multi-touch-ui/

How google earth works.
It explains some of the MIP-mapping techniques that GE uses, to get good filtering characteristics on its texture maps, so that things look crisp and clear, even at sharp angles.
http://www.realityprime.com/articles/how-google-earth-really-works

Distributed version control.
The second link is a talk given by Linus talking up distributed version control and his own version of it called Git. He also spends time ragging on SVN and how much it sucks. Ian's the only other person I know that's been using distributed version control with darcs. I've tried it, and it's not too bad. It took some time to understand some of the implications of DVC.
http://ianclatworthy.wordpress.com/2007/06/21/version-control-the-future-is-adaptive/
www.youtube.com/watch?v=4XpnKHJAok8

Haskell Faster than C on Great Language shootout benchmark.
I didn't read into detail, but what I gleamed is that lazy evaluation has its advantages. Read into it what you will. Overall, if haskell has to do work, it is slower than C. But if it can 'cheat', it will be faster in some cases.
http://neilmitchell.blogspot.com/2007/07/making-haskell-faster-than-c.html
http://www.haskell.org//pipermail/haskell/2006-June/018127.html

More on functional style programming.
I've been using more functional style programming lately. I like being able to chain things together, though sometimes, it doesn't make it necessarily easier to read. That's still dependent on the coder. Functional style programming has its advantages, but it's not made obvious here.
http://gensym.org/2007/4/7/enumerate-map-filter-accumulate

Lock-free hash tables.
This is kinda neat, actually. It's a talk on a concurrent hash table algorithm, where it doesn't use any locks (but it does use fencing during table resizes), and scales to 4000 processors. What I found neat is that the table resizing can be stacked, so that if you have 700 threads writing to a hash table all at once, it'll exponentially resize as it's reading and writing, where the reading and writing threads do some of the work copying table entries from the old table to the new table during the resize. More than one resize can be happening at the same time too.
http://video.google.com/videoplay?docid=2139967204534450862

Surface computing and building your own hardware

Microsoft Announces Surface Computer

Microsoft recently announced their surface computing platform. It's where you get to manipulate objects with your hand on a screen. I think after Minority Report, everyone wanted something where you could manipulate virtual objects. Since there, there's been a realization of that. But few of us had the imagination/drive/ability to actually do something about it.

I had seen simple demonstrations of this type of interface at malls, where a projector and a camera would use occlusion to calculate interactions with the objects. But it was kinda like having a stub to manipulate objects--you couldn't pick them up and manipulate it. Microsoft's surface computer seems to have done away with that, and added a sense of interaction between real objects and the virtual ones in the surface of the desktop--so one can load photos, simply by dragging the photo 'into' the camera.

As for multitouch sensing aspect of surface computing, it's not the first. The idea has been around since the 80's, if not earlier. However, the first demonstration that permeated the web was Jeff Han's demo at TED. Multitouch-sensors weren't available commerically, so you'd have to be able to build your own. According to Jeff in the talk, he said it was low cost and scalable. It makes me suspect that many EEs could have built it. But we didn't.

"People that love software want to build their own hardware." - Alan Kay

I use to think that this quote was only applicable in the days when software was much closer in abstraction to hardware; when people were writing in assembler and C. Nowadays, the only people that seem to do that are embedded programmers, and having done embedded programming for sensor networks, I can say it's not half as fun as web or application programming. Having to manage memory, or build your own malloc wasn't fun, to say the least. It was kinda having to time the spark plugs in your engine to go, instead of just pushing on the gas pedal.

However, I've taken a new view to the quote. When I think about all software, they all process information in some way. The input has to come from somewhere, and the output has to go somewhere to realize the bits in some form. However, the inputs are limited by what humans are willing to enter, and more importantly in this post, what kinds of hardware that will collect this data.

I can't wait until clothes keep track of themselves and match themselves. Technically, it's possible now to write the software, but one would have to enter the information by hand so that the computer can do the tracking and matching. But if there was hardware for clothes to serve this information, then it expands the space of information for software to operate on.

In this light, I can see where the quote is applicable. To expand the reach of software to access information that is only currently available in the physical world, you'll have to be willing to build hardware.

Innovation is force fed; someone get the lube!

In an earlier post, I had talked about what users know and what you know, when it comes to listening to your users. That said, when it comes to building new products, either in another line, or something to replace your old product, you should go back to not listening to your users--at least on the first draft. The act of creation is effectively the effort of one (or the few). At least when it comes to first drafts, too many cooks do spoil the broth. That might be a bit Ayn Randian, but the only thing I've ever heard of where design by committee was successful was the Space Shuttle and the Lunar Lander. (If there's more examples, please enlighten me.)

When you're building a product you're essentially forcing your world view onto others. You're basically saying, "I find this to be a pain. And this is not the world as it should be. As a builder, I can correct it after mouthing off for a while." And this is usually why people don't warm up to innovative ideas readily--someone is shoving their world view in your face. And unless you're someone that has been looking for a solution to the same problem when it's introduced to you, you won't be receptive to it. Even innovative people suffer from this affliction of shortsightedness.

“Don’t worry about people stealing an idea. If it’s original, you will have to ram it down their throats.” – Howard Aiken

Because innovative products can be so jarring, they should soften the blow a bit--or as others like to call it lowering the barriers. This is where influences from design, gaming, and etiquette can help.

Beyond the current trend of sleek lines, horn-rimmed glasses and black turtle necks of designers, design isn't just about putting a gradient background on your web app, or painting things in pastel colors. Hackers making a product should understand that design is the study of how to best solve communication and usability problems with limiting constraints. What information would the user need to know right this second, and how should you convey it to make it as easy to understand as possible? And from the answers to those questions will emerge a form that is also pleasing to the eye.

Gaming is an avenue more familiar to hackers than design is. However, games are often seen as mere trifles of play reserved for kids--though this is changing. If you've played enough video games and thought about WHY they're fun, will help also, because to bring out the essence of fun in what's normally perceived as tedium will give your product an edge. In the lecture about the ESP game by Luis von Ahn, he laments the fact that there's millions of cycles of human computation wasted. There was 9 billion hours played of solitaire last year (est.). Considering that the Empire State Building took 7 million hours and Panama Canal took 10 million hours, that's a lot of wasted hours. We should be able to put those cycles to good use by making people play games to solve problems that computers can't yet solve. So a symbiosis of humans and computers can be considered a large distributed computer to solve hard problems, such as object recognition in images. You might have played it.

In other web apps, the idea of a collection is a powerful mechanism of play. Social networking sites play on the idea of collecting friends, much in the same way that in Pokemon, you "gotta catch them all!". In others, the idea of a scoreboard is a powerful motivator, as seen on Digg and Reddit.

And last of all, the idea of etiquette seems far removed from being applicable to innovative products. However, no matter how much technology people surround themselves with, we are still social beings and will have social tendencies. Because of that, we expect certain behaviors and interactions between ourselves and our machines. We get mad and frustrated at computers and devices because they're usually not very polite. They stop responding when they're busy doing something, but don't tell you what they're doing. They don't remember what you told them last time and asks us over and over again. And when they don't know how to ask for help when something goes wrong, since the error messages are unintelligible to most users. These are all hallmarks of an annoying person, and were it a real person, I'd have kick them to the curb.

The iPod, and in general, Apple products, are known for their politeness. When I first got an iPod, it was the 5th generation. I was surprised that it stopped the music, if the ear buds got unplugged, and that it turned itself off, after it's been paused for a while. Basically, it knew what was going on, and reacted to it in a fashion that makes sense to its owner. That sounds like the promise of Agent based software hyped so long ago. Maybe it should make a slow come-back.

The sad thing is, computer apps and devices have been annoying us for so long, that we have kinda gotten use to it. I think as research on classifiers become more readily available to programmers as being embedded in the language, and the rising influence of designers in applications, we should see a trend towards more polite products. If you can make a product that is polite, it'll go a long way in gathering fans.

In the end, you want people to use what you build if it has value. And users want to GET THINGS DONE, so they can move on with their lives. All products should solve problems, there's no doubt that it's essential. All other points are moot if your product is useless. But given that it does solve a problem, if it is also beautiful, fun, and polite, it will go a long way in lowering barriers so that we can all have pearls Before Breakfast.

Reconnecting to database server in Rails

I've had more posts up my sleeve, though I haven't had time to actually polish them up. I should make my blog posts go back to its roots, where I just said anything as a first draft. That way, you'll get more stuff. So as usual, I happened across my travels through Rails-land and saw something that I don't think gets seen too often...since I couldn't find it on the first page of Google. It was an error like this:

>> user = Account.find(1)
ActiveRecord::StatementInvalid: Mysql::Error: MySQL server has gone away: 
SELECT * FROM accounts WHERE (accounts.id = 1) from /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.0/lib/active_record/
connection_adapters/abstract_adapter.rb:128:in `log'
...blah blah blah...

Since connections are expensive (in terms of time) to make, web frameworks, and anyone making raw connections to the database, will use the same connection for multiple SQL queries, and close the connection when you're done.

Usually you won't see this in Rails, because it does a pretty good job of maintaining the connection, either per session, or per user action in the controller. However, when you have a background process running using something like BackgrounDrb, if there is no activity between the background worker and the database for a couple hours, the database is going to close the connection, and the worker will still think the connection is valid. In other words, ActiveRecord::Base.connected? will return true.

Here is also where I found a use for 'else' in blocks as mentioned by Jamis Buck. When the connection goes out cold, we can't really tell that its' because it's been sitting there too long. It will raise an ActiveRecord::StatementInvalid, which is the same thing raised when you have a bug during development. As a simple fix, I just wanted something to try reconnecting to the database once, just in case it was only because the connection was cold.

class SomeBackgroundWorkerClass
def initialize
  @already_retried = false
end

def some_database_operation
  begin
    Account.find(1)
    # or some other database operations here...
  rescue ActiveRecord::StatementInvalid
    ActiveRecord::Base.connection.reconnect!
    unless @already_retried
      @already_retried = true
      retry 
    end
    raise
  else
    @already_retried = false
  end
end
end

So, that way, as long as it succeeds every other time, it'll keep on going. Tip!

Updating just the join table

Having a model that has a has_and_belongs_to_many relationships with another model affords you the convenience of a bunch of added on methods that get created when you define the relationship. These are all pretty nice. But I found that I had to forgo these methods for a more crude method.

Let's say you have two models, taken from the Rails book: Article and User.

class Article < ActiveRecord::Base 
  has_and_belongs_to_many :users
end

class User < ActiveRecord::Base 
  has_and_belongs_to_many :articles 
end

In order to create a new article and associate it to a user right away, you can use create!:

user = User.find(session[:user].id)
user.articles.create!(:title => "The Art of FizzBuzz")

But sometimes, an article might be linked to other models as well. Let's say that there's a Shelf model, and an Article habtm Shelves too. Then, you'd have to pull something like:

user = User.find(session[:user].id)
shelf = Shelf.find(params[:shelf_id])
article = user.articles.create!(:title => "Go and foobar yourself")
shelf.articles << article

Now, that last line is tricky. It's adding the new article to the articles of a shelf. Technically, it should just be inserting ids in the join model. However, that's not the case. It will ask shelf to load all its articles first, and then update the join table. Now, if you're going to manipulate articles of that shelf later on in the controller method, I think this would be the way to go.

However, if you're importing articles from the net, that might not work so well. In that case you just needed to add the association in articles to shelves in the join table. The current implementation of <<, concat, and push seems to enforce an explicit query for it at least once.

Therefore, if "shelf" has a lot of articles, then you'll experience a large slowdown in importing your articles--for every new article, you're asking the database to return a list of all current articles on that shelf. Database caches common queries, but in this case, it doesn't help, since you're importing a new article every time, which can belong to different shelves. But the time you come back to the same shelve, it may have been cleared from the cache already.

This is very much like Joel's story about Shlemiel the Painter. It's not that <<, concat, push is implemented poorly, but that it's used for a different scenario with different assumptions--that you're going to be doing other things to the collection within the scope of the controller method.

The only solution I've come up with is an ugly one. I created a model out of the join table, and added a method called link. It finds the associated link, and if it doesn't find one, it creates it.

class ArticlesShelves < ActiveRecord::Base
  def self.link(article, shelf)
    find_by_article_id_and_shelf_id(article.id, self.id) || 
      create!(:article_id => article.id, :shelf_id => shelf.id)
  end 
end

This has lowered the importing of articles from a minute and a half for each article belonging to a shelf with lots of articles, to about 0.5 second for each article on a low powered machine. I personally don't like this solution, since it introduces a very specialized model object with only one purpose, rather than a cohesive set of responsibilities.

While it is possible to push the method "link" to both Article and Shelf, I'm not sure exactly how to query for just the join table if the active record counterpart ArticlesShelves does not exist, other than using find_by_sql(). But even then, how do you execute an "insert" SQL query?

If you've got a better solution, let's hear it. :)

Google Maps of the World of Hello World

Google maps now allows the ability to create your own maps. The title link is a map of the major programming languages in use in the world. So it's like lots of little hellos around the world. Cute. Looks like Africa, India, Australia, and South America have a lot of catch up to do. They don't do all the languages in the world, but Japan would also have quite a few if it listed all the variants in the Lisp family.

It's also noticeable that the coasts dominate with programming languages. UIUC's gotta step up.

But most significantly, making your own maps has been a long time coming, and I would have originally thought they'd leave mappr.comFrappr alone in this field. But I think it makes strategic sense for them, especially if they make it easy to post maps that people create.

A default behavior for failure or nil

I read "The Rails Way", mainly because Jamis Buck writes there. Three days ago, they had actually posted one thing that I had address just a few days prior--that is, how to prevent users from looking at other users' data.

I have to admit, I was rather tickled by my solution. It's nice when you figure things out. But alas, after reading Koz's post on association proxies, I have to admit, I like his solution better. At least I made it to anti-pattern #3.

So what did he do? He simply used the find in the association, and let things throw an exception otherwise.

def show
 @todo_list = current_user.todo_lists.find(params[:id])
rescue ActiveRecord::RecordNotFound => e
 flash[:warning] = "Stop playing around with your urls"
 redirect_to '/'
end

It also takes care of the case where you have say, many items that belong to a todo_list. You can load it by using :condition in the association find. The only reason I can think of not to use it is if it happens to be a slow solution. But no use optimizing if you have no measurements and hard numbers.

Koz's solution let the default behavior of the method take care of things. This is a way of thinking that I need to start using more of.

I was use to nil being a failure state, something that you checked, and if it happened, everything's gotta stop--like the examples I often saw in C:

status = CallSomeMethod(with, some, parameters);
if status == NULL {
  return ERROR_CODE
  // or throw some exception here if you're using C++ or Java
}

Often times, these things got cumbersome, because NULL (or even an empty array or hash) was not considered to be a valid input for many functions, and would stop computation by returning error codes and throwing exceptions in C++.

One of the things that I found nice about the code from Ruby gurus was that it was conventional to do (and subsequently, I saw in Perl):

setting = params[:setting] || "default"

Instead of:

setting = params[:setting].nil? ? "default" : params[:setting]

Or

setting ||= "default"

Instead of:

setting = setting.nil? ? "default" : setting

It was because the operator || took nil by default, and had an appropriate behavior for dealing with nil, and that has made all the difference in being able to chain functions together. In addition, having default failure behaviors require you not have to write error checking code all over the place either.

I'm not sure what this is called (if anyone knows, enlighten me), but a completeness and liberal in what you accept for your input and strict in what you output also applies here to methods and functions. I think it makes code more readable, because it is not peppered with common sense error checking code.

each vs. inject vs. map

Despite working in Matlab for a fair amount of time, where you have to think in terms of matrix operations, it's still hard to shake the "loop it through" kind of thinking when dealing with collections of things from a C heritage. So say I have posts with comments, and I want to get all the comments of all posts.

all_comments = []
posts.each { |post| all_comments += post.comments }

This was the way that I use to do it using a more C-like thinking. I really never liked doing it this way because you have that floating initialization with all_comments, and it can get separated from the actual loop when you have all sorts of stuff doing on. Then I found inject:

posts.inject([]) { |all_comments, post| all_comments + post.comments }

This code does the same thing, pretty much with the initialization in the loop itself. I liked it a lot better. However, map has its uses:

posts.map { |post| post.comments }.flatten

I'm not sure which one is faster on my machine, but the last one has an appeal in that a "map" operation tells me that this piece of code can be done in parallel. Given the way it's written, semantically it means that every piece in the collection can be independently calculated from each other, and then put together at the end (with flatten)--regardless of how it's actually implemented right now. Not that this isn't true for the first two code pieces, however, "inject" and "each" does not immediately imply that it can be parallelized.

I know that compilers nowadays are pretty sophisticated, with pipelining and all. But having a programmer use "map" could only be a help to the compiler figure out which part of the code parallelizes, no?

Update: I found another post talking about closures in Haskell, since the java people are resisting closures in Java. It gives a better argument over why a for loop is no good.

Convention over configuration is a culture over reinvention

Most programs nowadays are written with some code that the developer writes that are built on top libraries. We string together and manipulate the libraries in interesting ways in order build an application. No one goes and rewrites a graphics engine or a networking library anymore (unless you're doing something special or unique in that arena). So besides debugging, for every new type of application a developer is doing, they are mostly spending time figuring out how to use libraries.

I've found that some libraries were really hard to figure out how to use. Other libraries were a lot easier to adopt. Why? I've found that it was mainly due a culture that was common across the libraries, and often times, this is more apparent in some languages than in others.

When I started using Rails, I was struck by how some things weren't dictated by the syntax of the language, but merely a convention set by the framework developer. And as long as you followed the convention, things just worked. This not only simplified the complexity of rails, because it didn't have to handle all sorts of cases, but it also made it simpler to find your way around the framework once you knew its 'culture'.

"Convention over configuration", is what the creator of Rails calls it, and after thinking about it for a little bit, it makes sense. In Rails you don't have to make the mapping of class names to database table names, or their id attribute. There is already a convention, a culture, for doing that. Culture of a framework allows you to make assumptions that lets you say more with less.

It is the same with any human language. Idioms such as, "famous last words", "a rose by any other name", and "rolling a rock uphill" have a lot of meaning behind the words that are the result of references that most other people know--culture. So with just a couple words, you can convey quite a bit. The catch is that the other person has to understand that culture in order for you to convey much with little.

I think programming language--libraries especially--would benefit from a culture, so that programmers can more easily hop from library to library. Culture also has the advantage of being malleable over time. This way programmers can shift their conventions over time if the old ones aren't working.