Foxy Fixtures and polymorphic tables

Well, I'm behind on everything, which means a bunch of interesting blog posts are queued up. But this one seemed short enough to warrant a small post.

I've always hated fixtures for the same reason that other people hate them, but nonetheless, I've bit the bullet to use them. Along comes Rails 2.0's foxy fixtures, and it becomes a little easier.

What it doesn't detail, however, is how to use your newly foxy fixtures for polymorphic models. If I have a vote model that I can use to vote on any type of table, with the old fixtures, I'd have:

my_vote:
  id: 1
  account_id: 1
  votable_id: 3
  votable_type: "Scene"

Normally, you just get rid of the foreign keys since it now checks the belongs_to associations of each model, and you can just use the label names. Same goes with the primary key id. It'll be autogenerated based on a hash of the fixture label.

my_vote:
  account: my_account
  votable: eat_hotdog
  votable_type: "Scene"

Note that you're using the association names, and NOT the foreign key name, so you don't use "_id" anymore (that bit me in the ass for a little bit).

However, you'll find that with polymorphic models, you won't be able to do that. Searching around the good 'ole web lead me to find that Foxy Fixtures originally came from a plugin called Rathole, and at the very end of the README, it states:

Also, sometimes (like when porting older join table fixtures) you'll need to be able to get ahold of Rathole's identifier for a given label. ERB to the rescue:

Go John Barnette! That way, you can simply do this in your fixtures as a fall-back:

my_vote:
  account: my_account
  votable_id: <%= Fixtures.identify(:eat_hotdog) %>
  votable_type: "Scene"

Tip!

Testing MIME response types

I feel like I might have covered this before, but I was looking for a way to test respond_to. I had found this post on how to test it, but after looking at it for a while, I found myself rewriting it. Mainly, I took out parts that convert the Mime types, and inserted Rail's own Mime type objects. You can use it like this:

request_mime(:fbml) do 
  get :list
  assert_response_mime(:fbml)
end

request_mime("text/xml") do 
  get :list
  assert_response_mime("text/xml")
end

Just include it in your test_helper.rb file in test/

class Test::Unit::TestCase
  include Threecglabs::MimeTestHelpers
end

Here's "mime_test_helpers.rb". Just throw it in lib/

module Threecglabs
  module MimeTestHelpers

    def self.included(mod)
      mod.class_eval do
        include MimeRequest
        include MimeAssertions
      end
    end

    module MimeRequest
      # changes the mime type of the request within the block
      #
      #   request_mime(:fbml) do 
      #     get :list
      #     assert_response_mime(:fbml)
      #   end
      def request_mime(mime_type_name_or_extension)
        if mime_type_name_or_extension.kind_of?(String)
          mime_type = Mime::Type.lookup(mime_type_name_or_extension)
        elsif mime_type_name_or_extension.kind_of?(Symbol)
          mime_type = Mime::Type.lookup_by_extension(mime_type_name_or_extension.to_s)
        else
          raise ArgumentError.new("mime type must be string or symbol")
        end
        old_mime_type = @request.accepts
        @request.accept = mime_type.to_s
        yield
        @request.accept = old_mime_type
      end      
    end

    # These are assertions to test respond_to, whether they return a specific mime type
    # as a response to a request
    module MimeAssertions

      # Helps out with response testing, by letting to assert that the most recently-made 
      # request responded with a particular MIME extension, like :html, :fbml, :xml, etc.
      def assert_response_mime(expected_mime_type_ext)
        expected_mime_type = Mime::Type.lookup_by_extension(expected_mime_type_ext.to_s)

        # Mime::Type.parse doesn't parse accept parameters correctly, therefore
        # we account for having multiple types in the accept
        response_mime_types = @response.headers['type'].split(/,\s*/).map do |accept_type|
          mime_type_name = accept_type.split(/;\s*/).first
          Mime::Type.parse(mime_type_name)
        end
        assert_block("Responded with #{response_mime_types.map(&:to_s).inspect} when expecting #{expected_mime_type}") {
          response_mime_types.any? { |response_mime_type| expected_mime_type == response_mime_type }
        }
      end

    end
  end
end

Snippet!

irb console tricks

irb Mix Tape — err.the_blog

Not a big entry, but I think worth posting because it pays to learn your tools. I've got no insights of my own to add. Someone else's tip!

Render_to_string only counts if failed

render_to_string followed by render issues - Ruby Forum

I was puzzled by this yesterday, and good thing I went to do something else instead of tearing my hair out. Came back fresh today and found this. Apparently, render_to_string normally doesn't count as a render--given that it succeeds. If it throws an exception for some reason, then it'll get count as a double render if you rendered elsewhere. Small tip!

Energy over Space

Annotation (Harper's Magazine): Keyword: Evil

It's an annotated blueprint of one of Google's datacenters. While it's all good and well to point these criticisms out, that doesn't mean nothing's being done about it. Google itself has announced that it was creating a renewable energy R&D group. In addition, it's also in the semiconductor's interest to build cooler chips. Hot chips won't sell as well now that datacenters are concerned with gigaflop per watt.

I have a gut feeling that if the next president is in tune with what's going on in tech, he/she'd challenge Americans to rise to the occasion to innovate through the energy problems, just as JFK challenged Americans to fly to the moon half a century ago. I remember in the 90's as a kid, there was a stint of environmentalism. Captain Planet. Save the Walrus tshirts. And then it kinda faded away until now, a whole decade later. I'm hoping that it's not a fad, like it was in the 90's. There are plenty of interesting problems in this space. And I hope we figure them out before we're able to easily colonize other planets. If space tech supercedes energy tech, then it would end up to be more economical to just ditch a planet once we dirtied it up for another one. And pity the natives that happened to be living there. Humans would be the tyrants looking to subjugate other hospitable planets, and not the victims, like so many science fiction stories would have us believe. It would look like empirialism all over again. So I hope we figure out this renewable energy stuff before we figure out how to easily colonize outer space.

MIME responder filter for Rails

I didn't think I had to do this, but I ended up writing a filter that acts like a switch statement for different MIME types. Let me explain. Normally, in Rails, you can respond to different requests for different content with something like this:

class PostController < ActionController::Base
  def list
    @posts = Post.find(:all)
    respond_to do |format|
      format.html
      format.fbml
      format.xml { render :xml => @people.to_xml }
    end
  end
end

When you have something like this, the browser (or whatever client) can ask for different MIME types. Here, we can return html to a browser, xml to maybe a data importer, and fbml to facebook.

I spent last week integrating Mobtropolis with facebook. Mobtropolis doesn't require a facebook account to use it, so like other websites, it has its own authentication mechanism, something like:

class PostController < ActionController::Base
  before_filter :website_authenticate_filter, :except => [:index, :list]
end

When I started using facebooker library, it already came with an authentication before_filter. That means we have two authentication filters, one native, and one for facebook. Mobtropolis users don't have to be in facebook to use it, and facebookers don't have to sign up again in mobtropolis to use it.

However, since before_filters are executed in succession, it leads to a case where the facebook authentication would be called if html was requested, and vice versa. The alternative was to take apart both authentication filters, and create a monolithic filter to handle the two different cases. Instead, I did this:

class PostController < ActionController::Base
   before_respond_to_filter :except => [ :index, :list ] do |format|
    format.html :website_authentication_filter
    format.fbml :facebook_authentication_filter
  end
end

That way, I didn't have to mix together the guts of each authentication filter, and it solved the problem of the wrong authentication filter being run. You can also use it like:

class PostController < ActionController::Base
  before_responds_to_filter :only => :home do |format|
    format.html do |controller|
      return if controller.logged_in?
      controller.send(:redirect_to, :controller => :home)
    end
    format.fbml :ensure_application_is_installed_by_facebook_user
  end   
end

By the way, I tried to alter the filter_chain as a request came in. Filter chains are copied and passed around the filters, so you can't write a filter that alters the filter chain. So don't waste your time crawling around in the guts of Rails to do this like I did. It's just as well, as that'd be a nightmare to maintain.

It does have some weaknesses though. You can only assign the filters to the same set of :except and :only options in the filters.

It ended up the code for this sort of magic was fairly easy. I'm not sure if there's an easier way to do what I wanted, but I'll see if Rails core people would find it useful (or not). In the meanwhile, for those of you Rubyists that have written plugins before that want to play with it. As with the usual mumbo jumbo, it's provided as is, I'm not maintaining it, and do whatever you want with it:

module Threecglabs
  module Filters

    # MimeResponderFilter 
    module MimeResponderFilter

      def self.included(mod)
        mod.extend(ClassMethods)
      end
  
      # Filters can respond to different mime types, so that you can use 
      # different filters depending on which mime type is being requested
      #
      #   before_responds_to_filter :except => [:login, :signup, :forgot, :invite_request, :profile] do |format|
      #     format.html :authentication_filter
      #     format.fbml :ensure_application_is_installed_by_facebook_user
      #   end
      #
      # This way, one can take the appropriate actions in setting up authentication 
      # from different mime types, and still separate the implemenation of the different
      # kinds of implementations
      #
      # The formats also take blocks, like regular filters
      #
      #   before_responds_to_filter :only => :home do |format|
      #     format.html do |controller|
      #       return if controller.logged_in?
      #       controller.send(:redirect_to, :controller => :home)
      #     end
      #     format.fbml :ensure_application_is_installed_by_facebook_user
      #   end
      #
      # NOTE: an :all format defaults to :html, therefore, a format.html is required
      module ClassMethods        
        def before_respond_to_filter(options = {}, &block)
          before_filter MimeResponderFilter.new(&block), options
        end
        
        private
        # This is a call that implements a MIME responder filter
        class MimeResponderFilter#:nodoc:
          attr_reader :filters
          
          def initialize(&block)
            @filters = {}
            block.call(self)
          end
          
          def filter(controller)
            filter = @filters[controller.request.format.to_sym] || @filters[:html]            
            if filter.kind_of?(Proc)
              filter.call(controller)
            else
              controller.send!(filter)
            end
          end
          
          # implements the "format.#{mime_type}" part of the filter
          def method_missing(mime_type, method_name = nil, &block)
            if block_given?
              @filters[mime_type.to_sym] = block
            else
              @filters[mime_type.to_sym] = method_name.to_sym
            end
          end
        end
      end

    end
  end
end

Snippet!

Erlang Advocacy and the class of problems it solves

I spend more time than I should reading hacker.news. Granted, I sometimes feel like it's the US Weekly or Maxim of the tech world, when stories like "5 ways you know you're failing at your start up" and the general hubbub over Facebook back in June or the recent Microsoft and Yahoo buyout. However, the quality of the comments there is generally high, and on occasion, I'll start replying to a comment, and before I know it, two hours have passed, and it's better off as a blog post.

The challenge of the Erlang advocate is not to convince me, over and over, that Erlang wins its class; the challenge is to convince me that Erlang's class of problem is so important to my life that I should study Erlang rather than vascular surgery, or television repair, or other obscure technical skills that I don't know that much about.

The follow-up question then would be how many existing problems can be converted to the type that Erlang can solve well? And how many problems previously impractical are now practical to solve? So if it ends up to be a bigger class than you thought, you may well be limiting the number problems that you can solve easily and practically out of the ones that will be important in the future.

I like Erlang (aside from some syntax ugliness), so I'll give it a shot. I think it's important because it allows a program to easier to scale out rather than scale up. If it was running an algorithm that are parallelize-able, then you can just technically add more cheap processors to it for a speed up, rather than designing a faster processor. We'd want speedups in this way because CPUs are becoming multi-cores, and to take advantage of them one will have to write some type of parallel program, since it's proven difficult to fully automate parallelizing serial programs. In addition, with bandwidth pipes getting bigger and the internet more and more pervasive, it is possible that you can leverage other computers you don't own (but given permission) for computational or storage resources in the future most of whom don't belong to a single entity. (think more SETI@home than Amazon)

In addition, parallelized systems (not just erlang) can be more fault tolerant and can fail more gracefully (or hobble along, if you'd like to call it that). Sensor networks are one example. Instead of a single radar to detect the environment, you throw a bunch of sensors out there, and they network themselves and report what they see/hear back to you. If some of them gets destroyed that's ok, because the system's still functioning with less sensors.

A swarm of UVAs doing surveillance in an area is another example. If you have a single computer commanding all the UVAs, it's actually quite hard, because you don't want them to crash into each other, so that's N^2 comparisons (less if you do oct-trees, probably). And if a target comes into the area, it becomes a non-trivial allocation problem: how do you decide which UVA to assign to monitoring it, and when do you switch them out when their fuel runs low? It ends up that doing it in with an actor model, where each UVA decides what to do at any given moment (local interactions) might not be optimal, but it's redundant and fault-tolerant.

Biological systems work this way as well. Gene expression is actually a network of genes being turned on and off by proteins expressed by other genes being turned on and off recursively. If one gene can't activate another, it might not always be detrimental, because there might be other pathways to express that gene. Since I'm not as familiar with the intricacies of biological systems, I'll refrain from saying anymore.

I was surprised when I was watching a video of Alan Kay talking about OOP that the OOP C++/Java I learned in college wasn't what he had in mind. Rather, he meant OOP to be more like biological systems and more process orientated. Encapsulation was only meant so that objects (analogous to actors in erlang) would have to pass messages to each other (method calls).

So what concrete examples of problems fall into this class Erlang is good at? The obvious ones are the embarrassingly parallelized algorithms, like genetic algorithms, neural networks, 3d rendering, and if I'm not mistaken, FFTs. Indexing web pages is another. But then again there are other algorithms that are inherently serial like protocol handshaking or newton's method.

I don't know what the ratio is between embarrassingly parallel and Serial problems are, but my gut is that with the advent of multi-cores and availability of the internet, I think there will be plenty of parallel problems to go around.

Ruby and Haxe language writers are both implementing the actor model like Erlang, if that's any indication of how important they think it is. While I don't think Erlang will be the 100 year language, the ideas by which it's a poster child will reverberate in the descendant languages for a long time to come.

How to find multiple file types using linux's "find"

I've always found *nix's "find" and "grep" rather hard to use. Not only are there different flavors of regular expressions to use, but mainly different syntax. For find, the directory you're searching for comes first. For grep, it comes last. To find the negation of something, you'd use "-not" and for grep it's "-v". pain.

Anyway, been trying to learn my tools better, and I found out how to grep and replace expressions across multiple files through emacs. Since rails uses all sorts of file extensions, naturally, I wanted to grep for find different files types. I had thought the -name options took regexs (it doesn't), so I had tried it in regex (no go)...only to find that it's something like this:

find . -name "*.rb" -o -name "*.rhtml"

the -o is the equivalent of a boolean "or". small tip...

2D barcodes rebirth

Google just announced 2Dbarcodes for print ads, which prompted Joel to talk about the :CueCat, which was a large failure in the late 1990's. Wired Magazine sent you these barcode readers shaped like cats with this one issue. The idea was that you could scan barcodes in print ads, and given the URL encoded in the barcode, it'll take you there in your browser. I can hear wails of people going, "now why would I want to do that?" Of course, that's from a lack of imagination.

Now, ideas are brilliant or stupid only in the right context. Joel is correct about his assessment of the :CueCat. However, there are instances where it works and is in widespread use.

Japanese print ads have 2D barcodes on them. Even blogs and webpages have 2D barcodes, so you can access information (URL or otherwise) from your phone.

There are a couple things different there. Most Japanese peoples' access to the internet has been through their cell phones, rather than through their computers. Landlines are much more expensive than having a cell phone. In addition, the majority of Japanese are in urban areas where they use public transportation. That gives them a lot of down time to play with their phones. The print ads in the train have the 2D bar codes on them, so people can check out the ad while they're riding the train. Given that unless you have a full keypad (real or virtual), it's still harder typing in a phone, than using a 2D barcode.

That said, I don't think Americans (Don't know about europe) will find as much use for 2D barcodes for print ads, as we drive everywhere. The tech world has changed a lot since the :Cuecat was around, so I think most of the right players are in place for 2D barcodes. I don't think having 2Dbarcodes in newspapers makes sense. It makes much more sense for print ads, you can see on the street in strictly urban areas, like New York City.

If I'm walking around town, I'd find it very handy to be able to check out how many tickets are left for a show and being able to buy tickets for the show from the URL in a 2D barcode on the print ad for a show across town. It would also be useful if sewn onto tags of pillows, clothing, other products, etc, so that would provide product information, or at least a URL that has the manual, or specs. That way, if some piece were broken, it'd be an easy way to order replacement parts.

I think it's a tiny, tiny step into tying the real world with the virtual, and part of the move to form a Clickable Earth.

Making sure extra tasks get run when installing plugins with Piston

When I graduated college, I was a EE major that could code. However, in hindsight, I don't think I was as good because I had no idea how to put together moderate sized systems or above. One of the things that I like about Rails is that they have a system for plugins. I was too uncouth to consider it before. Once you extract a plugin out, you essentially have to treat it as a separate library or package. This forces you to encapsulate, because it's a bit of a pain changing plugins. I know you can extract code out to DLLs or Gems, but I never did it because it seemed like I didn't need it--plus it was extra steps.

I eventually would want a programming language that lets me extract out libraries inside the language, and make its public interface RESTful, and auto-extern its path in SVN and whatever the equivalent is in distributed version control like git/darcs.

Recently, I've started using Piston. It's a ruby plugin manager that's essentially a wrapper around svn:externals. It's been pretty easy to use. In addition, my plugins won't get stale. However, when you install a plugin with piston, you only import the files. It doesn't actually execute the install.rb file, like /script/plugin install would. I'll try to see if I can submit a patch to piston later, but for now, simply run "install.rb" in the root of the plugin, and it should do the install for you. small tip!

Interfacing and distributing code as a language feature

Recently, I've been looking to send email from erlang. Despite all the cool things it does, its INets library doesn't have an SMTP client in it. Trapexit happened to be down this week, so I ended up hunting around on the web (next time, I should just ask on newsgroups).

I found ErlMail as part of an Erlang Software Framework suite, and it had a simple SMTP client in it. I was glad I didn't have to write one myself, but I found it didn't do authentication. it was written cleanly enough that I was able to patch in authentication without much of a problem, and submitted it to the maintainer.

The short experience made me pretty thankful that SMTP protocol is in plain text, rather than bit-packed. Since much of our general purpose languages are pretty good at manipulating text, it's comparatively easy to interface with it and write a wrapper for it.

However, it's a pity that for every new language that comes out, a new wrapper must be written for an SMTP client. Same goes for REST/SOAP interfaces for any number of APIs that we see out there, from Google maps to facebook apps.

Part of the problem is the mismatch in syntax of the protocol/interface and the language the programmer is working in. Simply looking at REST interfaces, there's no programming language that makes native method calls like /post?title=32?body=f1rst%20post. Not that would make sense to do so since there are other RPC (remote procedure call) protocols too.

The only way I can see out of it is for a language(or library) dynamically generate code that maps a native call syntax to a RPC syntax. For every new type of RPC interface (REST, SOAP, JSON, etc.), we'd have a single file that describes the mapping. Then, when a new service or platform comes out with a REST interface, you don't need to write a new wrapper for that interface.

However, lots of languages have different method invocation capabilities. Some need types to arguments, others can take first class functions or blocks, and still others can take an unknown number of arguments. I don't think it would be easy, but it'd be nice to have so we stop wasting our time writing interface wrappers. Maybe Lisp was on to something when it said code is data and data is code.

XMPP for machines

Jive Talks: XMPP (a.k.a. Jabber) is the future for cloud services

I found this great because it gave me a way to think about something familiar in a different way. I'm use to thinking about XMPP as just an IM and presence protocol, used by applications that let humans to communicate with other humans. But I didn't take it one step further and think of it as a messaging service between machines, mostly because I was under the impression that polling problem was solved (by the likes of Comet).

If this is possible, then by the same token, one should be able to run a "IMsite" over XMPP, analogous to a "web site" over HTTP. It's just that there currently is no "browser" for XMPP. If there were, you can technically send DOM updates or javascript (or whatever the browser can interpret) over XMPP. I imagine one should be able to take the mozilla engine and tack XMPP instead of HTTP in front (probably easier said than done).

That way we should be able to build browser apps that need near-real-time updates. The obvious one is chat. In fact, most of our XMPP clients are specialized to do that. Other applications are collaboration software, like a shared whiteboard (if sending SVG over XMPP would not be a bandwidth hog). Video lectures with auto advancing slides might be another one. Fleet tracking might be another. MMORPGs would also be easier to write on such a platform. It'd be interesting to see where this goes.

Update: Looks like people already tacked XMPP onto Mozilla

Setting time in your fixtures

In your rails fixtures, you should be able to set the time dynamically. If you set your database to default to UTC, make sure you use:

post:
  id: 1
  body: "This is my witty post"
  created_on: <%= 3.days.ago.utc.xmlschema %>

Unless you want to wreck havoc in your test cases. tip!

Getting inline-block working across browsers

In mobtropolis, there's a gallery of pictures that I have to show. While, normally, it's not too hard showing them as inline elements, if you just have pictures, it gets a bit tougher if you have a bunch of stuff that you have to inline correctly. Sites like facebook solve the problem by using a fixed layout, so you know the width of the area you can work in, and can thus use a table.

When you have a stretchable layout, however, that doesn't work. And what you need to do is use CSS, "display: inline-block" The problem is, it doesn't have consistent support across browsers. Only Opera and Safari use "display: inline-block" and "display: inline-table" correctly. IE6 and Firefox both use "display: inline" only and don't recognize inline-table and inline-block.

So thank goodness for design blogs out there: Align List Items Horizontally with CSS . His solution is a bit of a work around, but it prevents me from writing any javascript when I didn't need to.

.ib-fix li { display:-moz-inline-box; -moz-box-orient:vertical;
             display:inline-block; vertical-align:top; word-wrap:break-word; }
* html .ib-fix li { display:inline; }
* + html .ib-fix li { display:inline; }
.ib-fix li > * { display:table; table-layout:fixed; overflow:hidden; }

I should add that you need to also add this to get it to work in IE6, and get rid of the weird padding on the left:

ul.ib-fix
{ 
  list-style: none inside none;
  padding: 0px;
}

Yay. small tip! As a bonus, I also found out about ie7.js, a javascript library that fixes incompatibilities in ie7 and ie6.

The possibility of a reshapable keyboard

I have a bunch of hard drives that failed. I suspect that it's the controller card that is broken, but either way, I can't use it. What to do with old hard drives? That's when I started looking around, and it ends up that there are rare-earth magnets inside. (as well as a voice actuator that you can hook up to an amp to get hard drive speakers.)

And thus, I found a long article on rare earth magnets. Magnets have long fascinated people, as it makes all sorts of things possible, like speakers, hard drives, motors, and generators. But I didn't know about magnetic braking and that you can buy ferrofluids (also described in the article).


Ferrofluids are liquids that responds to a magnetic field. When you put a magnetic field near it, it responds by getting spiky. The stronger the field, the more dense the spikes. I was able to play with some in a enclosed sac once. It's kinda weird. You can actually feel resistance in the liquid when you put a magnet by it, like something's in the liquid.

While the optimus keyboard lets you re-display the keys in any way you wish, I've always wanted a keyboard that I can reshape. I'd rather have the keyboard actually be a membrane stretched over a flat rectangular plate. And depending on the application, the membrane would be able to take on different shapes. So instead of having keys when I'm looking at a map, the "keyboard" would be in the shape of the terrain I'm manipulating. Then I can pan and tilt. If I'm flying a plane, I'd rather have a joystick I can manipulate. I suppose you can make a rudimentary one with ferrofluids in an enclosed membrane. Not only can you reshape the liquid with controlled electromagnetic fields, but you should also be able to detect human interaction with the membrane by how it changes the magnetic field.

If a reshapable keyboard were to exist, you can also hook up two together through the internet. That way, you can interact with other people through touch, and not just text. If I put my hand on the reshapable keyboard and push down, the connected keyboard at the other end should have an imprint of my hand, pushing up out of the membrane. I'd also be able to augment my interactions so that my hand can appear to be holding something that it might not really be on my end. An inane thing would be to play paper, rock, scissors, where instead of the hand gestures, you'd actually see a sheet of paper, a chunk of rock, or pair of scissors rise out of the reshapable keyboard. A more useful application might be to keep family members or loved ones in touch--literally.

And if the membrane were embedded with OLEDs then it can be possible to add color to the membrane, so the interface would be something you can directly manipulate.

When I dreamt this up, I was thinking of gaming applications or remote surgery. Imagine the kind of fun and good you can do with the technology! However, after a bit of thought, I think a more likely scenario is that geeks adopt it for that, and then the porn industry makes it widespread. Just wait and see.

Mobtropolis bruhaha: an interview

Chicago, unlike many homogenized cities in the US, has character. And Chicagoans are pretty proud of the uniqueness of their city, from the losing streak of their Cubs to how you should never put mustard on a hot dog. As far back as I can remember, Chicago TV stations would have shows that cover the various local unique flavors of Chicago and places you can visit.

Thatcher Kamin, producer at the local TV station, channel 26 WCIU does a show called the Chicago Insider. The show not only covers venues, but also local Chicago people doing different things, mostly artists, musicians, and the occasional business. Thatcher found me and wanted to do an interview about Mobtropolis, and as far as I know, I'm the only programmer on there.

Since the powers that be at the TV station don't quite understand the web yet (through no fault of Thatcher), they didn't provide an embeddable video, nor a link to the video itself. Thus, I'm forced to just provide you with instructions.

Simply visit www.wciu.com, and click on "play" at the little flash thing up top.

It was an experience getting interviewed, and I can now understand why actors don't like watching their own films. You notice all sorts of stuff you never knew about yourself before--like twitching your eyebrows. I can't watch it more than once myself. There was a lot of stuff cut out, and I had no idea what was going to make it and what wasn't. All in all, it was fun, but I can see I need to work on my elevator pitch much more. Well, if anything, I hope you're entertained by the funky beats in the background, and give Mobtropolis a shot if you were amused by the interview.

Nerd time Issue 14

I use to work at a research lab, and while they do know what's going on with military tech, they don't really get news of consumer web technologies. I read up on it alot while I was there, and I still do now. I started a little mailing list after I left to keep them updated, and that's what became Nerd time. Here is the latest for your enjoyment.

Hey all,

Happy 2008! It's been over a month since the last nerd time. At first, it was because not much was happening. I've found that Silicon Valley(especially the software/web side of things) is much like Hollywood in the sense that there are fads and fashions that people make ruckus about and then ditch after a month. And it's like sensationalism news when nothing noteworthy is happening, but people still find lots of trifle things to make noise about. So even though we attempt to filter it all through our favorite feeds and social news sites, it's still a chore.

I met up with Ray the first time since College, and he said he feels out of touch with the tech world, even though he's a chip designer. All he has to do is read nerd time! You get my own biased, hand-picked version of what's noteworthy.

Mike and Nick, check out Johnny Lee and Buglabs, being hardware freaks that you are.

As usual, the easy stuff is up top. If you want to stretch your brain, go further down.

Johnny Lee's wonderful world of HCI
Wiiremote VR tracking, Wii multitouch, Automatic registration of projection screens. It's this researcher's venture into HCI. more melding of computing into everyday lives. The videos are more self explanatory. If you skip everything else, I suggest you don't skip these.
http://www.youtube.com/watch?v=Jd3-eiid-Uw
http://www.youtube.com/watch?v=5s5EvhHy7eQ
http://www.cs.cmu.edu/~johnny/projects/thesis/
http://www.youtube.com/watch?v=s59PObuJGWY

Facebook, Google, and Plaxo join DataPortability Workgroup
I strongly believe that a users's data belongs to the user, and they should be able to take it around with them. I was pretty excited about Google's announcement about OpenSocial platform around October or so. I haven't personally taken a look at it myself, but according to pips here and there, it's been said it's incomplete and there was a case of a security hole. But this announcement of fb joining a data portability workgroup is good news, as making your data portable with you across applications will go a long way in making using computers a lot easier.
http://www.techcrunch.com/2008/01/08/this-day-will-be-remembered-facebook-google-and-plaxo-join-the-dataportability-workgroup/

EarthClassMail
This is a startup that you redirect all your snail mail to, and it opens and scans it in for you. You basically can read snail mail like you do email. You can even choose to recycle junk mail (good for the earth), and shred important documents. Neat idea, though of course, it all has to rely on trust of the company. As the social-political climate of the US over foreign energy dependence gains awareness with the push for 'green', we might see more companies like this that tries to move things into digital to save on green.
http://www.techcrunch.com/2008/01/08/133-million-for-startup-that-wants-to-kill-snail-mail/

Google's philanthropic arm going to develop clean energy cheaper than coal.
Just because something is digital doesn't necessarily mean it has a small environmental footprint. The servers it's running on might be burning coal. I think it's rather big, actually. It remains to be seen whether Google will democratize energy production, to shake up a very traditional energy industry, or simply join as another player. Regardless, this venture makes sense for them, as they run gigantic server farms.
http://www.techcrunch.com/2007/11/27/google-takes-on-global-warming/

Prediction Markets at GooglePlex study organization information flow
Google uses internal prediction markets to gather data on what's going on. Prediction markets are basically using the mechanics of stock markets to predict things like, "is google stock over valued?", "Will the U of I win the Rose bowl?", "Will Comet win mindshare with developers?" This particular article talks about mapping information flow in organizations. I think as the world moves faster and faster, mid-sized companies are exploring more flexible organization structures to be able to respond quicker to changing markets. You'll probably see more studies into company organization in the future.
http://googleblog.blogspot.com/2008/01/flow-of-information-at-googleplex.html

PathIntelligence
This is interesting because it's another indication of how real-life is merging with digital. Any website monitors its traffic to know which pages are more popular than others. It uses that as implicit feedback to what customers like. This does the same for physical storefronts. Using bluetooth on customer's cell phones to track foot traffic, they can see which racks and displays are selling more. As more and more sensors enter into the world, we'll be able to do analysis and better understand our lives. I won't talk about privacy concerns, cuz that's just a can of worms.
http://www.techcrunch.com/2007/12/14/path-intelligence-monitors-foot-traffic-in-retail-stores-by-pinging-peoples-phones/

Amazon SimpleDB and computation on tap
I wish I brought Amazon stock. The more I find out about Jeff Bezos, the more I understand that it's not really a bookstore that he built, but a philosophy. Anyway, Amazon, in this past year has been releasing "computation on tap" They basically sell distributed computing and storage services to developers and startups. Lots of people have taken advantage of it, for good reasons you can read elsewhere. In the past month, they released simple db, which is really just a fast index db. With computing on tap, it enables small teams to do large things. If computing and storage is a commodity, I suppose the value becomes what you can build with it.
http://www.amazon.com/gp/browse.html?node=342335011

WiTricity wireless power
Wireless power has been in development for some time now. I had mentioned it in one of my earlier nerd times. This one, although just experimentally demo'd, is rather neat. We probably won't see wireless power more ubiquitous until maybe 5 years later. I don't quite understand it myself (should have paid attention in PHYS 112), but apparently, it's not radiation like we're familiar with. Instead, it uses magnetic resonance, so that only two electromagnetic devices with the same resonance will transfer energy efficiently, everything else, like humans, won't.
http://web.mit.edu/newsoffice/2007/wireless-0607.html
http://www.technologyreview.com/Infotech/19889/?a=f

Buglabs
This is a startup making mobile devices that you can hack. They plug together like lego pieces and run linux. People can prototype and experiment with different mobile form factors and fulfill their niche needs.
http://video.google.com/googleplayer.swf?docId=-1932063822649530376&hl=en
http://video.google.com/googleplayer.swf?docId=-3090880151267528595&hl=en
http://video.google.com/videoplay?docid=-3950989589304402454&hl=en
http://www.buglabs.net/products

Quagmire 2D programming language
This is a 2D virtual machine, and you can watch it as it uses its memory visually on a 2D surface. What's interesting is that the latter examples looks like chaos, which reminds me of the pictures you see in "A New Kind of Science" In it Wolfram claims that basic computers nowadays just have simple loop patterns rather than the complex ones you'd see in class 3 CAs. Problem is, we don't know yet exactly what class 3 CAs compute better, and how to harness it, other than CA120 is turing complete.
http://www.pawfal.org/Art/Quagmire/

Comet - AJAX's cousin
Comet is using AJAX to push information to browsers, rather than having browser poll the server. The term was coined about two years ago, in reference to the bathroom cleaner. So you can get stuff like chat working in browsers, or like a stock market feed.
http://simonwillison.net/2007/Dec/5/comet/

Sunspider - javascript benchmarking
Javascript is getting faster and faster with work being donw on the underlying engines underneath popular browsers. I've seen "lemmings" implemented completely in javascript. You'll continue to see this sort of thing. However, flash, silverlight, and flex are still far ahead of the curve when it comes to interactivity of the web.
http://webkit.org/perf/sunspider-0.9/sunspider.html
http://www.elizium.nu/scripts/lemmings/

Version control in emacs
The new emacs 22 has svn version control built it! *high fives Ian*
http://www.credmp.org/index.php/2007/12/08/emacs-hidden-gems-version-control/

Tumblelogs
Tumble logs are the half cousin of twitter and blogs. They represent a flow of conciousness digital media tidbits from the poster's daily life. We might start to see more of this if it catches on. However, other than personal expression, I don't exactly see a future for it. But then again, I was wrong about Friendster when it first came out.
http://tumblr.com
http://www.robbyonrails.com/articles/2007/12/19/putting-tumblr-to-work-for-you

Next gen ANNs
A talk on next generation Artificial Neural Networks. I haven't watched it yet, but will this weekend. It promises to be pretty exciting.
http://www.youtube.com/watch?v=AyzOUbkUf3M

Ycombinator in Ruby
I finally figured out what ycombinators are. If I had to take a stab at it, it's a way to implement recursion when you don't have named recursions, loops, iterators, etc, and all you have are functions and a couple substitution rules. It's pretty neat, and will stretch your brain a bit to understand it. Traditionally, you only do this in lambda calculus, but having the code in Ruby made it easier to understand.
http://www.eecs.harvard.edu/~cduan/technical/ruby/ycombinator.shtml

DRYing up your views with a TableBuilder

In the last two months, when I was adding more features to mobtropolis, I found it painful to try and lay things out all over again from scratch. As a result, it sucked to see ugly layouts on the new pages juxtaposed with all the styling I had done before. It wasn't until a week ago that I said to myself, "Stop the madness!" and started refactoring my views--something I never thought of doing much of until now. When you don't, the barrage of angle brackets blows out of proportion and it starts to look pretty damn fugly with complex views.

What I should be able to do is take common mini-layouts in my views and make them available as helpers so that I can think in terms of bigger chunks to piece together pages, rather than in divs and spans. In addition, it makes your interface more consistent for your users.

Some good resources were presentations from the 2007 RailsConf, like V is for Vexing and Keeping your views DRY. While a lot of view DRYing talks about form builders, I didn't see any on table builders, so I decided to take a stab at it. Personally, I don't like to overuse tables for layouts. But as certain elements in my page layouts have been repeated, I refactored them into first helpers, and then when I did more than one, I extracted it out into a simple table builder. This is how you'd use it:

For example, I have a mini-layout where I show simple stats:


Here's how I used a simple table builder to display the above:

<% score_card do |sc| -%>
  <% sc.placard("Votes", "", num_voters, "") -%>
  <% sc.placard("Sceneshots", "", num_sceneshots, "")-%>
  <% sc.placard("Participants", "", num_participants, "") -%>
  <% sc.placard("Challenges","", num_challenges, "") -%>
<%  end -%>

And I find that I started using the same sort of thing in other places, like in a user's profile:

<% score_card do |sc| -%>
  <% sc.placard("Submitted Scenes", "", num_scenes, "") -%>
  <% sc.placard("Submitted Sceneshots", "", num_sceneshots, "") -%>
<% end -%>

I cut out some details so you can see that it's just a block that gets passed a ScoreCard object, from which you call placard to add another score to the score_card. It sure beats writing <table> and <td> over and over again.

To declare the helper, we create a helper with the structure of the table inside the declaration of a ScoreCard object. We have a ScoreCard object to hold the contents of the placards. When they're called in the block above in the template, they get stored in the ScoreCard object, and not written out to erb immediately. That way, we can place them wherever in the table we please, by making the call to card.display(:placards):

module ScorecardHelper
 def score_card(html_options = {}, &block)
    options = { :class => :scorecard, :width => "100%" }.merge(html_options)
    ScoreCard.new(block) do |xm, card|
      xm.table(options) do
        xm.tr(:valign => :top) do
          xm << card.display(:placards)
        end
      end
    end
  end
end

So then what's ScoreCard look like? Pretty simple. It has a call to each cell that can be filled in the mini-layout. It's kinda analogous to how form_for passes in a form object, on which you can call text_field, etc.

require 'lib/table_builder'

# Used to create a scorecard helper
class ScoreCard < TableBuilder
  cells :placards

  # a placard is placed into scorecards
  def placard(text, text_id, score, widget)
    xm = Builder::XmlMarkup.new
    @placards[:html] += xm.td do
      xm.span(:style => "font-size: 1.2em") { xm << "#{text}" }
      xm.em(:id => text_id, :class => "primary_number") { xm << "#{score}" }
      xm << "#{widget}"
    end
  end

end

Notice that there's a call to cells() to initialize the type of cell, and a method of the same name that builds the html for that cell. If you have other types of cells, you simply put it in the list of cells, and then create a method for it that is called in the template. By convention, you'd stick the html of the cell contents in "@#{name_of_cell}"[:html], and if you wanted, pass in the html_options, and stick it in "@#{name_of_cell}"[:options]. Then, you can access those in the helper wherever you want.

Let's try another one. I have a mini_layout with a picture, and some caption underneath it, like a polaroid.

<% polaroid_card do |p_card| -%>
  <% p_card.picture do -%>
    <%= sceneshot_for scene -%>
  <% end -%>
  <% p_card.caption do -%>
    <%= pluralize(scene.num_of_sceneshots, "sceneshot") %>
  <% end -%>
<% end -%>

The associated helper and PolaroidCard object are pretty simple:

module PolaroidCardHelper
  # a polaroid card is used to show a picture with a caption below it.
  def polaroid_card(html_options = {}, &block)
    options = { :class => :polaroidcard, :style => "display: inline;" }
    PolaroidCard.new(block) do |xm, card|
      xm.table(options) do 
        xm.tr { xm.td(card.html_options(:picture)) {
            xm << card.display(:picture)
          }}
        xm.tr { xm.td(card.html_options(:caption).merge(:class => "caption")) {
            xm << card.display(:caption)
          }}
      end
    end
  end
end

require 'lib/table_builder'

class PolaroidCard < TableBuilder
  cells :picture, :caption

  def picture(html_options = {}, &block)
    @picture[:html] = capture(&block)
    @picture[:options] = html_options
  end

  def caption(html_options = {}, &block)
    @caption[:html] = capture(&block)
    @caption[:options] = html_options
  end
end

I've tried to pull all the plumbing out into TableBuilder (dropped it into lib/), and only leave the flexibility of creating the table structure in the helper, and the format of the cell in the object. It ends up TableBuilder isn't too complex either. It uses some metaprogramming to create some instance variables. I know it pollutes the object instance variable namespace, but I wanted to be able to say @caption[:html], rather than @cells[:caption][:html].

class TableBuilder < ActionView::Base
  class << self
    # used in the child class declaration to name and initialize cells
    def cells(*names)
      define_method(:initialize_cells) do 
        @cell_names = names.map { |n| "@#{n}".to_sym }
        @cell_names.each do |name| 
          if instance_variable_defined?(name)
            raise Exception.new("name clash with ActionView::Base instance variables") 
          end
          instance_variable_set(name, { :html => "", :options => {} })
        end
      end
    end
  end

  def initialize(decor_block, &table_block)
    super
    initialize_cells
    decor_block.call(self)
    html = table_block.call(Builder::XmlMarkup.new, self)
    concat(html, decor_block.binding)    
  end

  def display(cell_name)
    instance_variable_get("@#{cell_name}")[:html]
  end

  def html_options(cell_name)
    instance_variable_get("@#{cell_name}")[:options]
  end

end

I've found have these helpers cleans up my views significantly, though I have to admit, it's not exactly easiest to use yet. In addition, I'm not exactly thrilled about having TableBuilder inherit from ActionView::Base, but it was the only way I could figure out to get the call to concat() to work. In any case, the point is to show you that refactoring your views into helpers is a good idea, and even something like a table builder goes a long way, even if you don't do it the way I did it. Lemme know if this helps or hinders you. snippet!

Painite teaser

It seemed like "What the heck is a Ycombinator" struck a chord with people. In retrospect, I probably should have done less rambling. So after a silence of a week, I'll get to the point. Here's a little teaser of what I've been working on over the holidays.

require 'painite'

p_es = Prob::EventSpace.new do |es| 
  es << Prob::RandVar.new("cancer") do |pdf|
    pdf["cancer = :sick"] = 0.01
    pdf["cancer = :healthy"] = 0.99
  end
  es << Prob::RandVar.new("test | cancer") do |pdf|
    pdf["test = :pos | cancer = :sick"] = 0.8
    pdf["test = :neg | cancer = :sick"] = 0.2
    pdf["test = :pos | cancer = :healthy"] = 0.096
    pdf["test = :neg | cancer = :healthy"] = 0.904
  end
end

p_es["cancer = :sick | test = :pos"] # => 0.8 * 0.01 / (0.8 * 0.01 + 0.096 * 0.99) 

It's still not quite complete, as it's harder than I expected to generalize it. The interface is still a little bit in flux, but I hope to nail it down soon.

Nice little irb console tidbits

Slash7 with Amy Hoy - Secrets of the Rails Console Ninjas

Not a big deal, but always nice to be more familiar with tools you use all the time. I knew about "reload!", but had no idea about "_", and the especially YMLicious "y @bob"

referring tip!

What the heck is a Ycombinator

I woke up at 4am this morning for no reason and decided to go through Ycombinator in Ruby. Given that I read Hacker News daily, it's a shame I didn't know what a Ycombinator was. I thought the article was pretty clear, at least compared to Raganwald's article. As an aside, from purely a learning experience, it was kinda fun and eye opening. At least I can see why geeks think it's neat.

I had written a reply to a comment on hacker news about it, and it turned out to be really long, so I just extracted it out here. I was suppose to start some coding, but I got wrapped up in writing up the comment. My comment ended up to be pretty long, but it was shorter than it could have been. You guys get to suffer through the long version. Haha. But I'm not going to go into details. You can read that in the linked articles. Instead, I'll paint some broad strokes.

  Y = λf·(λx·f (x x)) (λx·f (x x))

or in Ruby, copied from the aforementioned article:

y = proc { |generator|
    proc { |x|
        proc { |*args|
            generator.call(x.call(x)).call(*args)
        }
    }.call(proc { |x|
        proc { |*args|
            generator.call(x.call(x)).call(*args)
        }
    })
}

or if you prefer elegance, Tom's solution in response to Raganwald

def y(&f)
    lambda { |x| x[x] } [ 
        lambda { |yf| lambda { |*args| f[yf[yf]][*args] } } ]
end

I found lambda calculus above hard to read. However, if you go through the code in Y Combinator in Ruby, you'll find it's not too bad. I find that this lecture is also pretty good, as it takes you step by step, with a little bit of humor as well.

If I had to take a stab at it, A Ycombo is a way to implement recursion mechanism when the language doesn't provide named recursion, loops, or iterators, and all you get are first-class functions and a few substitution rules.

Functions are just mappings from one set of things to another set of things--ie. give it a number, and it'll point to another number. Ycombo relies on a property of functions that sometimes, when you put something into a function, you get the exact same thing out, i.e. something gets mapped to itself, like f(x) = x^2, where f(1) = 1 is an example of this property. They call this the fixed point of a function.

The thing about functions is that they don't just have to map numbers to numbers. They can map functions to other functions. A derivative is an example of a function that takes one function as an input, and spits another function back out. like d(x^2) = 2x. Where is the fixed-point of a derivative? One of them is when you put e^x into it. d(e^x) = e^x. I'm sure there are more.

This is important because if you can find the point in which a function can return a function unchanged, you can use that to call the function again, which is what we call recursion. And all the trickiness you see in ycombinator that you see is mainly a result of functional programming not keeping state, so you have to pass everything you need into a function. So if you have to do recursion, you have to have a mechanism pass the function itself, so you can call it again. And this mechanism kinda bends back onto itself, and that's why you see a part of the ycombinator repeated twice in the code above, and in the lambda calculus.

It seems pretty inane to use ycombo given that modern high level languages provide named recursions, and if anything, for loops and iterators with closures. But what if you don't? How do you process lists/arrays without loops or named recursion? Generally, you'd have to make your own recursion mechanism.

When would you ever have languages that don't have those things? Probably not often. But Forth comes to mind. I've never used Forth, but from what little I know about it, the language starts off with some basic primitives and that's it. No loops, no if statements. How do you get anything done? Well, you build your own control structures. People use Forth because it's possible to build your own very small compiler from the ground up written in Forth itself, and still understand the entire thing. I suspect it's used in embedded programming because of that reason.

So you'd pretty much use it when you want to have a powerful system from first principles. I'm guessing it's possible to build your own computer by implementing only functions and substitution rules in hardware, and then you can derive everything else, numbers, pairings, and even recursions in software. That way, you keep your hardware costs down, while retaining the power of a Turing machine.

Speculating here...but another thing that might be interesting is that ycombinator might be a way to compress code. Software size should be reducible if the code is compressible. And recursion can be thought of as a compressing code, and expanding it when the recursion is run. I wonder if there's other ways to reduce software bloat with the use of a ycombinator besides recursion?

(Aha!) Part of the reason why great hackers are 10 times as productive

I knew in college that some dudes were faster than I was in terms of programming. Since peer programming wasn't exactly encouraged in college, and at work I did mostly prototyping work, I never really knew how fast other programmers worked.

So when I read Paul Graham (and Joel's) claim that great hackers are at least ten times as productive as average programmers (too lazy to cite right now), I was kinda shocked. Surely, ten times is an order of magnitude! Something that takes an average programmer a 40 hour week to do the great hacker can do in a 4 hour afternoon?

I wondered about that, since there are times when I get stuck on something, then I just start stabbing around randomly out of frustration. I had assumed that great hackers were faster only because they had either the experience or insight to side-step whatever I was doing wrong.

But lately, I've been re-reading everyone's essays that write about programming productivity. And one thing that caught my eye the second time around was when Paul Graham was talking about bottom up programming and how he didn't really believe in objects, but rather, he bent the language to his will. He was building new blocks for himself so he could think about the problem at a higher level of abstraction.

This is basic stuff! I mean, that's the whole point of higher-level programming. When you refactor methods out, you're creating a vernacular so that you can express the problem in terms of the problem domain, not in terms of general computing. This is much like if you want to drive a car, you'd want to be able to step on the gas, rather than time the firings of the pistons. And if you want to control traffic in a city, you'd rather tell all cars to go to a destination, rather than stepping on the gas and turning the steering wheel for each one.

But taken into the light of bending a language to your will, it makes it more clear for me as to how great hackers are ten times as productive. Great hackers are productive not only because they know what problems to sidestep and can problem solve systematically and quickly, but they also build a set of tools for the problem domain as they go along. They are very good pattern recognizers and will be able to generalize a particular pattern of code, so that they can use it again. But not only that, great hackers will create an implicit understanding attached to the abstraction, ie. what we might call common sense.

A case in point. Before Ruby, I'd used for loops over and over again, never really thinking that I could abstract a for loop. It wasn't until they were taken away in Ruby did I realize that map, inject, and their cousins are all abstractions of the for loop. When I see "map" I know that it performs a transformation on every element. But I also know that the array I get back will be the same size, that each element operation doesn't affect other elements, among other things. These are implicitly stated, and they allow for shorter code.

When that happens, you can simply read "map", and get all the connotations it comes with, and hence it comes with meaning. It also becomes easier to remember, since it's a generalized concept that you can apply in different places in the code. The more times you use it, the easier it is to remember, instead of having specialized cases of the same kind of code where the behavior is different in different parts of the code.

A great hacker will take the initial time upfront to create this generalized code, and will save in the long run being able to use it. Done over and over again, it all adds up. So it's not that for any given problem, a great hacker will be done in 4 hours what it takes an average programmer 40 hours, but that over time, a great hacker will invest the time to create a tools and vocabulary that lets him express things easier. That leads to substantial savings in time over the long haul.

I hesitated writing about it, as it's nothing I (nor you) haven't heard before. But I noticed that until recently, I almost never lifted my level abstraction beyond what the library gave me. I would always be programming at the level of the framework, not at the level of the domain. It wasn't until I started writing plugins for rails extracted from my own work and reading the Paul Graham article that a light went off for me. It was easier to plug things like act_as_votable together, rather than to still mess around with associations (at the level of the framework). I still believe you should know how things work underneath the hood, but afterwards, but all means, go up to the level of abstraction appropriate for the problem domain.

DSLs (Domain specific languages) are really just tool-making and vernacular creation taken to the level of bending the language itself. It's especially powerful if you can add implicit meaning to the vernacular in your DSL. It's not only a way of giving your client power in their expression, but it's also a refactoring tool so that you can better express your problem in the language of the problem domain. Instead of only adding methods to your vernacular, you can change how the language works. It was with this in mind that I did a talk on DSLs this past weekend at the local Ruby meetup. First part is on Dwemthy's Array, and the second is using pattern matching to parse logo. Both seemed pretty neat when I first read about it. Enjoy!

---

DSL in Ruby through metaprogramming and pattern matching

Getting rid of emacs 23 splash screen

Pretty Emacs Reloaded

The latest emacs 23 is pretty neat, but it noticeably has that splash screen in the beginning. It's generally good practice for an app not to present new users with just a blank screen. There's should be something there to say, 'hey, these are the next steps'.

That said, I've found it annoying, probably because I'm use to it not being there. So I've found that you can turn it off through this comment.

(setq inhibit-splash-screen t)

Just put that in your .emacs file in your home directory, and you're all set. tip~

Generating rdoc for gems that refuse to generate docs

I recently upgraded to capistrano 2.1, and it's woefully lacking in documentation. Jamis Buck had already picked a documentation coordinator about a month ago, but nothing seemed to be happening since.

So it was time to go dumpster diving in cappy code. I at least wanted to see what the standard variables were. To my surprise, there were some docs in code, but I couldn't generate it with

gem rdoc capistrano

For those of us that had never made a gem, all you have to do to force it to is to edit the associated specification for the gem (/usr/lib/ruby/gems/1.8/specifications/), and add

s.has_rdoc = true

Maybe if I dig enough stuff out of it, I'll pose some prelim documentation for cappy 2.1 and donate it to the documentation effort.

As an aside and musing, ideally, the code itself would be documentation. However, just because you're reading code, doesn't mean that you can get the overall picture of how to use it. Even if you filtered out the details, and saw just the class and method declarations, that still wouldn't be enough since you can't see how things fit together. I don't know what a good solution would be. The simplest API is often complex enough to be nearly useless without good documentation.

Small tip!

The user owns this and that

In most any community-based website, your users are creating objects. And sometimes, you'd only like to display them when they're owned by the user. Usually, it's easy enough to do the check based on the association in the template views.

  <% if @story.account == session[:member] %>
    <%= render :partial => "story"
  <% end %>

However, for the sake of perhaps over-reaching, but something more readable, we try:

class Story < ActiveRecord::Base
  belongs_to :account

  def owned_by?(user)
    self.account == user
  end
end

  <% if @story.owned_by?(session[:member]) %>
    <%= render :partial => "story"
  <% end %>

But this gets to suck, because you have duplicate code in different classes that are just slightly different due to association names. One way to solve it is to put it in a module, and include it in different classes. After all, that's what mixins are for.

module Ownership
  def owned_by?(user)
    self.account == user
  end
end

class Story < ActiveRecord::Base
  include Ownership
  belongs_to :account
  # blah blah blah
end

class Post < ActiveRecord::Base
  include Ownership
  belongs_to :account
end

Or alternatively, you can put it in the account, but in reverse. But now, you have to search through the associations of the owned ActiveRecord object.

class Account < ActiveRecord::Base
  def owns?(ar_object)
    ar_object.class.reflect_on_all_associations(:belongs_to).detect { |assoc|
      ar_object.send(assoc.name) == self
    } ? true : false
  end
end

I find the tertiary operator to be kinda ugly there, but it doesn't make sense to return the reflection object. Regardless, this lets you do:

  <% if session[:member].owns?(@story) %>
    <%= render :partial => "story"
  <% end %>

However, this doesn't work for self-associated AR objects, or objects that have two or more belongs_to to the same account. It relies on a unique belongs_to association for every object belonging to account. I'm not sure yet, which way's the better way to go, and in the end it probably doesn't matter as much, but I do like being able to say user.owns?(anything) for any object without really thinking about what the associations are. half-tip.

A simple distributed crawler in Ruby

A week ago, I took a break from Mobtropolis, and...of all things ended up writing a simple distributed crawler in Ruby. I hesitated posting it at first, since crawlers are conceptually pretty simple. But eh, what the heck.

This was just an exercise to do some DRb and Hpricot, so don't use this for your production work, whatever it may be. An actual crawler is far more robust than what I wrote. And don't keep it running hammering at stuff, since it'll get you banned.

First, this is how you use it:

WebCrawler.start("http://en.wikipedia.org/") do |doc|
  puts "#{doc.search("title").inner_html}"
end

And that's it. It returns documents in an XPath traversable form, courtesy of Hpricot.

A web crawler is a program that simply downloads pages, takes notes of what links there are on that page, and puts those links on its queue of links to crawl. Then it takes the next link off its queue and downloads that page and does the same thing. Rise and Repeat.

First, we create a class method named start that creates an instance of a webcrawler and then starts it. Of course, we could have done without this helper method, but it makes it easier to call.

module Crawler
  class WebCrawler
    class << self
      def start(url)
        crawler = WebCrawler.new
        crawler.start(url) do |doc|
          yield doc
        end
        return crawler
      end
    end
end

So next, we define the initialization method.

module Crawler
  class WebCrawler
    def initialize
      puts "Starting WebCrawler..."
      begin
        DRb.start_service "druby://localhost:9999"
        puts "Initializing first crawler"
        puts "Starting RingServer..."
        Rinda::RingServer.new(Rinda::TupleSpace.new)
        
        puts "Starting URL work queue"
        @work_provider = Rinda::RingProvider.new(:urls_to_crawl, Rinda::TupleSpace.new, "Queue of URLs to crawl")
        @work_provider.provide
        
        puts "Starting URL visited tuple"
        @visited_provider = Rinda::RingProvider.new(:urls_status, Hash.new, "Tuplespace of URLs visited")
        @visited_provider.provide
      rescue Errno::EADDRINUSE => e
        puts "Initialize other crawlers"
        DRb.start_service
      end
      puts "Looking for RingServer..."
      @ring_server = Rinda::RingFinger.primary
      
      @urls_to_crawl = @ring_server.read([:name, :urls_to_crawl, nil, nil])[2]
      @urls_status = @ring_server.read([:name, :urls_status, nil, nil])[2]
      @delay = 1
    end
  end
end

This bears a little explaining. The first webcrawler you start will create a DRb server if it doesn't already exist and do the setup. Then, every subsequent webcrawler it'll connect to the server and start picking URLs off the work queue.

So when you start a DRb server, you call start_server with a URI, then you start a RingServer. What a RingServer provides is a way from subsequent clients to find services provided by the server or other clients.

Next, we register a URL work queue and a URLs visited hash as services. The URL work queue is a TupleSpace. If you haven't heard of TupleSpace, the easiest way to think of it is as like a bulletin board. Clients post items on there, and other clients can take them out. This is what we'll use as a work queue of URLs to crawl.

The URLs visited is a Hash so we can check which URLs we've already visited. Ideally, we'd use the URL work queue, but DRb seems to only provide blocking calls for reading/taking from the TupleSpace. That doesn't make sense, but I couldn't find a call that day. Lemme know if I'm wrong.

module Crawler
  class WebCrawler
    def start(start_url)
      @urls_to_crawl.write([:url, URI(start_url)])
      crawl do |doc|
        yield doc
      end
    end

    private

    def crawl
      loop do
        url = @urls_to_crawl.take([:url, nil])[1]
        @urls_status[url.to_s] = true
        
        doc = download_resource(url) do |file|
          Hpricot(file)
        end or next
        yield doc

        time_begin = Time.now
        add_new_urls(extract_urls(doc, url))
        puts "Elapsed: #{Time.now - time_begin}"
      end
    end
  end
end

Here is the guts of the crawler. It loops forever taking a url off the work queue using take(). It looks for a pattern in the TupleSpace, and finds the first one that matches. Then, we mark it as 'visited' in @urls_status. Then, we download the resource at the url and use Hpricot to parse it into a document then yield it. If we can't download it for whatever reason, then we grab the next URL. Lastly, extract all the urls in the document and add it to the work queue TupleSpace. Then we do it again.

The private methods download_resource(), extract_urls(), and add_new_urls() are merely details, and I won't go over it. But if you want to check it out, you can download the entire file. There are weaknesses to it that I haven't solved, of course. If the first client goes down, everyone goes down. Also, there's no central place to put the processing done by the clients. But like lazy textbook writers, I'll say I'll leave that as an exercise for the readers. snippet!

---

webcrawler.rb

Communicating your intent in Ruby

I've been using Ruby most everyday for about two years now. While I'm no expert, I know enough to be fairly productive in it. And beyond liking the succinctness and power that you often hear other people talk about, it's made me a better programmer. But there's an aspect of Ruby that worries me somewhat.

To start, programming is recognized rightfully as a means to build something from pure thought. But it's also a form of communication, to other programmers that will touch your code later, and to yourself when you look at it months from now. We're at a point that other than embedded and spacecraft programming, we have the luxury of using programming languages that focus ease for the programmer, rather than for the ease of the machine. Fundamentally, that's the philosophy that Ruby takes.

And while Ruby's nice in a lot of ways, I'm not sure about how it communicates an object's interface. When you're allowed to modify objects and classes on the fly, how do you communicate interfaces between modules you mixin and methods/modules you add? By interface, I mean, how do you use this class so that it does what it's suppose to? Normally, it's pretty obvious--you look at the names of the methods declared in the code. A well-written class has public methods exposed, or you look at its ancestor's public methods. You might need some documentation to figure out how to call them in the right order, but generally, you have some idea just by looking at the method signatures.

However, when you throw mixins and metaprogramming in the mix, it becomes less easy to tell just from looking at the method signatures in the code--the structure of the code. You have to specifically read the code, or you have to rely on someone who knew intent to document it in detail.

An example communicating interfaces for mixins: the module Enumerable contains a lot of the Collections related methods. The cool thing is that if you wanted these functions in your own class, all you have to do is define each() in your class, mixin the Enumerable module, and you get all of these "for free". However, outside of documentation explicitly stating it, it's not as immediately obvious in method signatures that this is what you have to do in order to use it. It's only after scanning through the entire code that you notice each() being used for all the methods.

Of course, Ruby contains enough metaprogramming power to protect yourself against this. one can do something like this:

class MethodNeededError < RuntimeError
  def initialize(method_symbol, klass)
    super "Method #{method_symbol.to_s} needs to be in client class #{klass.inspect}"
  end
end

module Enumerable
  def self.included(mod)
    raise MethodNeededError.new(:each, mod) unless mod.method_defined?(:each)
  end
end

This only works if you put the include after you define each(). That's just asking for trouble when the order of your definitions in your class matter.

A fair number of people are writing mini-DSLs in ruby using metaprogramming tricks. One of the common ones is the use method_missing to define or execute methods on the fly. ActiveRecord's dynamic finds are implemented this way. The advantage of communication of interface here in the structure of the code is obvious. Unless it was documented well, you can't tell just by looking at the method signatures.

Why do I harp on interface signatures? I mean, in the instance of requiring each(), it works by just letting it fail in the enumerated methods, since it'll complain about each itself. In the instance of method_missing, just read the regex in the body. While these are true, none of these allow for rdoc to generate proper documentation. The whole point of documentation is to show you the interface--how to use that piece of code. I'm just afraid that given Ruby's philosophy of being able to write clear, powerful, and succinct code, it might fall short when people start using these metaprogramming tricks like alias_method_chain and method_missing more and more. Maybe rdoc needs to be more powerful and read code bodies for regex in method_missing?. It already documents yields in code bodies, but that seems awfully specific.

I'm not a exactly a fan of dictating interfaces like in Java. When you're first coding something up, you're sketching, so things are bound to change. Having plumbing like interface declaration gets in the way, imo. However, when something's a bit more nailed down, it'd be nice to be able to communicate to other programmers your intent without them having to read code bodies all the time.

In the end, I side on flexibility. However, I kinda wish Ruby had some type of pattern matching for methods so I didn't have to read method_missing all the time. But then again, that would be messy in all but the simplest schemes. Can you imagine a class that responded to email addresses as method calls? I guess I'd have to file this one under "bad ideas"

Don't reopen ActiveRecord in another file

The power of Ruby lies partially in how one can reopen classes to redefine them. Besides namespace clashes, this is usually a good way to extend and refine classes to your own uses. However, last night, I got bitten in the ass trying to refactor a couple classes. In Rails, you're allowed to extend associations by adding a class the association call.

class User < ActiveRecord::Base
  has_many :stories, :through => :entries, :source => :story, 
    :extend => StoryAssociationExtensions
end

where StoryAssociationExtensions is a class module holding methods, like expired() that I can perform on the challenges association, so I can do stuff like

  @user = User.find(:first)
  @user.stories.expired  # gives all expired stories

So when refactoring and cleaning up, I renamed StoryAssociationExtensions to AssociationExtensions and reopened up Story class and put it in there. I just wanted to clean up the namespace, and put the association extensions somewhere that made semantic sense. Naturally, I thought putting association extensions for a class belongs in a class. Well, it doesn't work. And don't do it. Hopefully, I'm saving you some pain.

class Story < ActiveRecord::Base
  module AssociationExtensions
    def expired
      self.select { |c| c.expired? }
    end
  end
end

Well, this works if you've reopened the class within the same model file, story.rb in this case. However, if you reopen the class in another file elsewhere, your model definition won't get loaded properly, which leads to associations and methods you defined not to exist.

So imagine my bewilderment when associations didn't work on only certain ActiveRecord Models. In addition, they worked on the unit tests and script/console, but didn't work when the server was running. All that at 3am in the morning. :(

Good thing for source control, so I could revert (but I have to say, svn isn't as easy to use as it could be).

I ended up creating a directory in model called collection_associations and putting the associations in there under a module CollectionAssociations namespace. Not exactly the best arrangement but it'll do for now.

I'm still not sure why ActiveRecord::Base instances don't like being reopened, but I'm guessing it has something to do with only getting loaded once. If anyone has an explanation, I'd like to read about it.

free warning!

Book Review: GIS for Web Developers

I'm involved in running the local Ruby Meetup in Chicago, and one of the co-organizers got O'Reilly to send us some book for free. In return, I suppose the deal was to give them some exposure for their books, so I got to review this book.

I had actually ordered the book before I found out about the free copy, so now I have two. So it's on amazon now under wil822 seller if anyone wants to buy the first copy off of me.
With that, here we go:

---

GIS stands for Geographic Information Systems and these technologies existed previously, but were often in the domain of military, scientific, or government applications. Geospatial datasets were either public, but hard to decrypt. Or they were easy to view, but expensive. However, with the advent of Google Earth, NASA World Wind, yahoo maps, and google maps, GIS has come into mainstream awareness and attention.

“Earth materializes, rotating majestically in front of his face. Hiro reaches out and grabs it. He twists it around so he's looking at Oregon. Tells it to get rid of the clouds, and it does, giving him a crystalline view of the mountains and the seashore.” - SnowCrash by Neal Stephenson

Even with the explosion of what's available to us geospatially on the web, it would be naïve to think that the development of GIS has plateaued. There are several complementary technologies that are starting to mature that will develop it even further, such as open mobile phone, ubiquitous wireless connectivity, and open social networks. These are but some of the few reasons to look at GIS as a programmer, as it opens up more possibilities for innovation, creativity, and making useful things.

A word of warning to start: GIS for web developers is not a book telling you how to use Google or Yahoo maps APIs to embed maps in your web application. If you don't need much more than to plop down points on a map, then there's no need to know most of what's in GIS for web developers.

However, if you need to suck down geospatial data from different sources to better provide services for your users, then this book might have something for you. GIS has a host of terminology and difficulties that get abstracted away for you in a consumer application like Google or Yahoo maps. GIS for web developers starts with basic GIS terminology, takes you through spatial databases, washes you in OGC web services, and finally plops you down OGC clients. In addition, it focuses on free (as in free speech, not just free beer) open source solutions.

Like a tour guide for the novice, it starts by introducing various basics, and even tells you where to look for different datasets, how to interpret them, and the difficulties that might arise from it. To start, geospatial datasets aren't exactly widely publicized. They're available, but you'd have to know where to look. For example, the US Census Bureau provides a geocoding database called TIGER, and they also provide geospatial data on the demographics of each county in the United States. Many US government agencies provide geographic data, but you'd have to know where to look.

Even if you know where to look, the file formats of geospatial data are many and varied. The book preps you on various popular file formats and translators to help you along when you encounter this in your searches for geospatial data. In addition, it goes into difficulties that might arise even if you've got the file format straight. For geospatial data, there are many different ways to project a 3D sphere into a 2D plane. Because of that, you have to make sure all your data is in the same projection, or else they won't line up correctly. Throw in the fact that the earth isn't really a sphere, you'll be glad that you had this book to figure out how to make sure everything lines up.

Next, it takes you on a short tour of spatial databases. Up until now, I would have thought that querying, “All points of interest within 20 miles of here” would be something implemented at the application level. However, GIS for web developers shows us this is not the case. Apparently, PostgreSQL already has spatial capabilities built in, and one can have table attributes that indicate location and make queries based on distance.

The rest of the book is devoted to getting your feet wet about servers and client standards by the Open Geospatial Consortium. It shows you how to setup an OGC standard geospatial server implementation called GeoServer, so you can serve up geospatial data in a RESTful way. Then a brief romp through using OpenLayers to embed a map widget, before tying together the odds and ends at the end.

This book is a good start to exploring GIS, but it only scrapes the surface. It suggests the possibilities towards the end about going beyond putting push pins on the earth. There's still a lot of room for possibility to explore. A word on editing, however. On page 60 in chapter 4 on rasters, a whole paragraph was repeated. It was as if someone cut and pasted and hit paste one time too many. That should have been caught.

Overall, it's a good book in terms of its information content, given that you're not just looking to embed Google maps in your web application. It takes you through the terminology of GIS, database concerns, and then through standard OGC web services and clients. By the tech book standards, this one is brief, but that's intended, as this is an introduction. The writing itself isn't spectacular, but it's clear and not a drudgery to read through. If you can stand the occasional corny joke and you're looking to suck down geospatial datasets, then this is the book for you.