Adaptive polling as an alternative to HTTP streaming

I've been fairly interested in how the HTTP protocol works lately. For a long time, I didn't think much of it. It sat on top of the TCP/IP layers, and there wasn't much I need to do with it. It did what it was suppose to do: let clients fetch pages from servers upon request.

But then I started reading about REST (about a year after the hubbub), and in general about why stateless connections are desirable (it's scaleable). This lead me down equally saturated road of AJAX and eventually some joke about Comet. What was coined as "Comet" was really a play-on-words for another cleaning product applied to another old web technology--namely persistent HTTP connections.

Traditionally, HTTP doesn't allow servers to push data to clients. With the way the web is architectured, most clients are behind firewalls and routers, so the server has no way of knowing which machine to push it to, unless it was talked to first. In other words, only clients can initiate data requests. This isn't enough sometimes, as servers might need to push data to clients, such as live stock ticker feeds in your web browser without page reloading.

The trick to persistent HTTP connections was to get clients to initiate the XHR connection to the server first, and for the server to not immediately reply to the request. The server will hold off on replying (leaving an open connection from the client) until there's actually a message to be sent back to the client (i.e. when there's a new stock update). And that way, it'll look like a server-push. And then the client initiates another connection all over again after a certain wait.

This is the way that LivePage and JotSpot Live implements their responsive apps. However, the concern for most people is that it doesn't scale--at least not when they tried it circa 1998. A server having thousands of open connections to clients will probably buckle, although Twister might have already solved this problem, but I haven't looked into it much yet.

Another concern of mine is that the Ajaxian pattern of HTTP streaming can also require the client and the server to hold state. This is because a server does not know what version of the last set of updates it has received. Therefore, the client sends the server what version it has had (state), and the server will only reply if it has a newer version. This seems to violate the REST architecture. While I only have the original 2000 thesis to say this is not scalable, it seems to make servers a bit more complex.

So why not use polling? Usually, it's because too much polling is wasted bandwidth. And not enough polling, you have stale data. So depending on the nature of the data that you're trying to stream, polling may or may not be a solution. However, it is stateless, and it should scale better, as long as polling isn't overdone.

That lead me to wonder if there was adaptive polling. Why not have clients try and predict their polling frequency based on past observations of their past polling to optimize their polling success. Polling success is defined as every time they poll, they get 1) new data and 2) freshest data.

It ends up that it's a very similar problem in two other fields (and I'm sure many others): web caching and sensor networks. In web caching, you want to cache web pages, so that you can show clients results faster if the page hasn't changed. How do you know the page has changed, and when to throw away the cached copy and obtain a fresh one? In sensor networks, each connection is expensive in terms of energy consumption. How do you know when a node has fresh data, and how often should you poll to obtain polling success? In this case, a master node is analogous to the client and a slave node is analogous to the server.

There's an additional issue to consider. One wouldn't want all the clients hammer the server all at once for a poll. That would make it seem like a flash mob to the server at periodic intervals. It would be best if the clients can spread out their requests, so that the traffic to the server is more constant. That way, the server wouldn't be overloaded. How do you coordinate the polling times of thousands of clients? Wouldn't that create more traffic on the network for the clients to ask each other? I'm guessing no, because the delay in response time from the server would indicate how busy it was at this moment. Using that as a type of "pheromone" from other clients (indicator left by other clients), a client should be able to adjust its offset time for its next polling request.

Splatting in case statements

RedHanded � Wonder of the When-Be-Splat

I always feel like I'm playing catch up to Rubyists.

BOARD_MEMBERS = ['Jan', 'Julie', 'Archie', 'Stewick']
HISTORIANS = ['Braith', 'Dewey', 'Eduardo']

case name
when *BOARD_MEMBERS
"You're on the board! A congratulations is in order."
when *HISTORIANS
"You are busy chronicling every deft play."
end

That's pretty damn cool. The thing about new languages is that when you're learning to write with it, you'll write it in the style of the old language that you're use to. C programmers will write C++ as if it were C. Java programmers will write Python as if it were Java. Therefore, you might think that there isn't much to be gained from the new language other than some syntactic sugar sprinkled here and there.

As least for me, being open to other constructs like blocks, closures written more like functional programming has lead to more succinct and readable code.

a = [1, 2, 3]
Hash[*a.collect { |v|
[v, v*2]
}.flatten]

I would have done this with a for loop before, and that's probably less readable. But I have to admit, succinct code only has meaning if you know the vocab.

FireBug for all other things

When working with RJS templates, it can be a pain, especially if you roll your own javascript in there. There's almost no way to debug it, so you have to be very very careful, or use your brain-the-compiler.

But aside from that, try out Firebug. It's a pretty need in-browser javascript debugger for Firefox.

Symbol conversation in MMORPGs

Blue Rabbit�s Climate Chaos - Adventure Games - GamersHood - Online Games Paradise

This was something that was shown to me by Alison. I just tried it out, just to see what was fun about it. Didn't play much, but I was struck by the fact that this game decided to employ pictograms instead of words for conversation.

Now, I don't know why Blue Rabbit employed this mode of conversation. Perhaps it's because the target audience is young children.

However! I think this would be key to building a more dynamic MMORPGs. I haven't played World of Warcraft, so I don't know if quests are static. But I remember in Everquest, the quests were the same, time after time. Oh sure, there might be grace periods where it wouldn't be there, but for the most part, the same person would have his daughter kidnapped time after time.

Instead of having static quests, I think it would be better to have dynamic quests. It gives a better sense of realism to the world that the gamer is playing in, if the NPCs(non player characters) had different needs at different times.

In the Sims, each NPC is an agent with goals and needs. And it basically interacts with its environment to fulfill those goals and needs as time progresses. But never do any of the characters ask another Sim to fulfill those needs for him. Sure, they have conversations with each other to fulfill the direct need for being social. But they never ask the messy roommate to clean up his mess. They always get irritated and clean it up themselves, or rely on the player to make someone else clean it up.

With a simplified vocabulary of pictogram language, an NPC would be able to express what he desires. And that would be up to the player in the quest to fulfill it. These goals, like in the Sims would change as the environment and needs change.

file_column is easy to use

HowToUseFileColumn in Ruby on Rails

File_column really is a cinch to use. But not without knowing that you needed:

add_column :entry, :image, :string

in the migration. And here I was reading through file_column code. Things are always clearer in hindsight. But it did teach me a few tricks here and there, about how to add dynamic methods to objects.

Annologger update: Commenting is available!

It's got no pictures of stars, but it's simple. Commenting is up for annologger!

Human verification CAPTCHAs will be done tomorrow, so that you don't get comment spam. In the meantime, get your friends, your readers, your fans, to comment, comment, comment away.

Annologger Update: By popular demand, Annolog Badges available

I'm happy to announce that you can now get annologger badges for your blog or website! What's a badge you say? It's basically a code snippet generated for you that you can cut and paste into any webpage, blog or otherwise. That way, you can floss your events on your blog now. :)

You can get your own annolog at http://www.annologger.com. Under the goodies section, you can create your own annolog badge.

It took longer than I had anticipated, due to not ever working with rjs templates before. I'll write a tutorial up later. On to comments for your annolog!

---

Wilhem has built Annologger, a tool that lets people worship your dentist appointments.

Late to the RESTful party

Apparently, I'm the last fool to really read about it. I only first heard about REST maybe 2 months ago by a long post by one of the rails guys.

Lately (as in the last 6 months), there's been a resurgence in figuring out the HTTP protocol. It's suppose to be RESTful. Mainly, the idea that network architecture are seen as a collection of resources identified uniquely by a URI. And that the whole network application is simply the user in a large state machine, where traversing the different resources equate to state transitions. This has implications of server and client design to be simpler.

Each HTTP request also has a method associated with it. The methods in HTTP most commonly used are GET and POST. In the early days of the web (ie when we were in college), I saw that forms submitted by GET or POST, and for a long time, I had no idea what the difference was. GET is intended to "read" but make no state changes in the server, and POST is intented to make state changes. So doing form submissions with GET is not only semantically wrong, but insecure, since it puts form contents in the url.

In addition to GET and POST, there are others, (I never knew). And the bunch of them map well to CRUD(Create, read, update, delete) operations. And using the native HTTP methods, you can take advantage of things already built into HTTP, like caching (for scalibility) without having to build it yourself.

Here's a simple intro , and I think one of the articles that spawned the discussion. This is the original disseration on RESTful architecture, if you want to read it.

Wilhem has built Annologger, a tool that lets people worship your dentist appointments.

Annologger Update: now with microformats and iCal

It's been long overdue, but I've added features that I've wanted for some time--mainly microformats and iCalendar. iCalendar isn't fully featured yet, meaning if you subscribed to it using your outlook calendar, you cannot update your annolog from your outlook calendar. That said, I'm working on it, and trying to keep the code base organized and healthy, so that it's not a mess that can easily be broken. It's like cleaning up your kitchen as you cook.

So while it might not look like much, there have been some minor bug fixes and major refactoring to get these features in. Thanks for the feedback so far, and keep them coming. :)

---

Wilhem has built Annologger, a tool that lets people worship your dentist appointments.

Time can be needlessly complicated

International standard date and time notation

To think that of all things I could have picked to build, it was something centered around time. Having worked on a sunrise/sunset and moonrise/moonset calculator, you'd think that I would have learned my lesson.

Time is difficult because there's always exceptions to the rule, and some specifics are unclear on first thought.

Does midnight belong to the beginning of the day, or the end? Is the end date and time of an all day event inclusive or exclusive? It ends up that luckily, there were plenty of smart people that thought about all of this decades before I came about, but they like to write in a boring prose, in specs like ISO8601.

By the way, midnight belongs to the beginning of the day, and all day events have exclusive end times. However, that's not what people mean when they say they're going to iceland from october 18th to october 22nd. When people say that, the date is inclusive. Aye.

And do you store all times in UTC? Depends on the application. Throw in time zones, there's even more confusion, not to mention taking day light savings into account. And if you really want to get nitpicky, there's always leap years and leap seconds to think about.

Wilhem has built Annologger, a tool that lets people worship your dentist appointments.

Enumerable still...

Still playing around with Enumerables. It was almost not worth mentioning, but it's a short post. I was looking for a short way to read in a file's contents all into memory all at once. Normally you wouldn't do this, because it eats up memory if you do it this way. But my files were short.

File.open('README', 'r') do |file|
  file.inject { |contents, line| contents << line }
end

This will open up a file README and return the entire contents as a string. It's pretty cool, since I don't have to muck around with temporary variables much...and it's readable...well, if 'inject' makes sense to you. Also cool is that, like Java container classes, as long as you implement 'each' in your Ruby class, you get all the ones in Enumerable for free. You just have to include it.

---

Wilhem has built Annologger, a tool that lets people worship your dentist appointments.

Screencast of Annologger

Screeniac � Annologger.com

I'm usually pretty busy so I don't search often on the web for what people have been saying about Annologger. But for the first thing that I put out, it's both a bit exciting and apprehensive to hear about it.

I had a really hard time choosing a name for it. To this day, I'm still not quite satisfied with the name, but apparently, people get what it is. So that's good news.

And secondly, it seems easy enough to use, so people seem to get it. I've had a lot more japanese users lately, (presumably from this review and others), and they seemed to put more stuff on there than 'test'.

But all in all, I need to put in the other features that I've been dying to get done, so people get see what its potential is.

Time to get back to work.

Singleton classes not seemingly the same as Singleton pattern

So I was looking at someone else's code today, and I saw this:

module Formats
class BasicNestedFormat
  class << self
    def foo
      ...foo code...
    end
  end
end
end

Huh? What did "<< self" mean? I guess I didn't read my Ruby book closely enough. It's apparently a "singleton class", which doesn't seem to be exactly a "singleton pattern".

A "singleton class" is a sole metaclass that 'holds' methods for a single object. It would have been called a 'metaclass', except that it's not the class of a class, but a class of a class that only exists for that one object.

Keep in mind that this is the case because you can add methods to objects in ruby at runtime. Since everything in Ruby is an object(even the classes), it has to go somewhere, and in an anonymous metaclass is where it goes.

One can almost think of them as 'static' methods in Java. However, because objects can be added methods at runtime, they don't necessarily belong in the class, but to the metaclass of an object that only belongs to that object.

They're not like 'static' classes of Smalltalk or Java. You can call self (like this in Java) in Ruby's 'static' methods.

It's a curious construct because you can use it to specify 'static' methods all in one go. It's the equivalent of the following:

module Formats
class BasicNestedFormat
  def self.foo
      ...foo code...
  end
end
end

Apparently, you can also use it to do prototype factories. There's more to it than meets the eye. You can find better explainations here and here.

Ruby is weird.

---

Wilhem has built Annologger, a tool that lets people worship your dentist appointments.

The need to write code, the need to ease reading

Lately, I've been doing more thinking, reading, prototyping, and moving, than I've actually been writing code. One of the things that got me thinking enough to write this post was the classic syndrome of software developers and engineers to be gripped by the "Not invented here" syndrome.

Some would say that's because engineers and developers like to create. That's part of it too, but I think it's also partially because we only get better if we write things ourselves. The only way to learn about something fully is to actually do it. The problem is, a lot of things that we would learn by doing, has already been done for a while now. Beyond linked lists and parsers, most anything you can think of has already been done. Not that we can necessarily do it better, but doing it is half the fun, and it's the only way you can get better.

You won't become a better programmer by passively studying other people's code. Similarly, you don't magically become a better writer by reading a lot of books. You become a better writer by.. wait for it.. writing. - Jeff Atwood

So it's a balancing act between using other people's code and reusing code to be more productive, and writing your own to learn and reduce dependencies.

Another aspect that compells people to write things themselves is that

It's harder to read code than to write it. - Joel S.

I think it's because code, like math, like poetry, is dense. There are meanings and implications that aren't literal and aren't at the surface. There are implications for what gets written at every line, and relies on a culture and context behind it to fully understand it. This is why it's hard to read math equations, Alexander Pope(first one that came to mind), and much less code.
But that doesn't mean that there can't be language constructs to faciliate the ease of reading code. Python does it as a language choice that restricts whitespace. Maybe the computer doesn't care about whitespace, but people surely do.

I have recently found that iterators with descriptive names actually help in reading code. Instead of a generic for or while loop, it kinda helps to have collect, and inject.

It would be nice if code was self-documenting, meaning that you would be able to tell what the code was doing from what the method names and variables were, rather than the comments around it. It would be nice if method signatures were all you needed to know what to put into a method (and no, static typing doesn't help completely here). I wonder if language constructs could help out with it, or would we always need to rely on the stylistic tastes and discipline of the individual programmer to write readable code and documentation?

---

Wilhem has built Annologger, a tool that lets people worship your dentist appointments.

Looking for cumulation in Ruby?

Module: Enumerable

This is pretty basic, but I only recently discovered it. I have a task that I do often in code. Usually, there is some list of things that I'd need to go through, and tack it on to another list if it meets some condition.

funny_posts = []
posts.each do |post|
 if post.is_funny?
   funny_posts << post
  end
end

The same problem appears when I try to summate all the things that fit some condition. I dislike having the initialization there in the beginning. Is there a better/prettier way? I would have thought collect() would be it, but it is mapping one value for another. It's almost like a function in math--a certain input gives you a certain output.

So it was finally that I saw inject(), and it seems to be what I want.

funny_posts = posts.inject([]) do |funny_posts_thus_far, post|
 if post.is_funny?
   funny_posts_thus_far << post
  end
end

I think that'll work. The difference doesn't look like much, but somehow it bothers me when I have that floating assignment before the loop. It's easy during maintainance that someone moves 'funny_posts=[]' far away from the loop, when semantically, it's a part of how the loop will work correctly.

inject() also applied to cases where you're trying to find the maximum in a list.

# find the longest word
  longest = %w{ cat sheep bear }.inject do |memo,word|
     memo.length > word.length ? memo : word
  end
  longest                                         #=> "sheep"

I guess it's the difference between functional programming and procedural...using return of value instead of relying on side effects. I use to write off functional programming as something that was antiquited, but now, I'm finding gems here and there in functional programming. It makes me wonder how procedural took off so rapidly. Perhaps programmers think easier in procedural programming?

Update: I guess I should read Enumerable more closely. The example I had with collecting funny posts is done with a method called partition() in Enumerables.

The names of things

In a number of fairy tales and old legends, the name of a fairy, a monster, or a god was pivotal to the story. 'To know someone's name is to control them', it is said. I actually never gave it much thought growing up. How silly. What's in a name? A Rose by any other name would smell just as sweet.

But when it comes to ideas and concepts, a name is pretty important. Naming a variable, class, or method correctly means that there's some convention the reader of the code can go by to be able to infer how to use it. It's a type of documentation.

So when it comes to finding names for things, I have a hard time. How are 'agreements' and 'disputes' related? Are agreements and disputes the same type of thing? If so, what is the name of that thing? Or are they not the same type, but merely the same thing in different states? An agreement is the alignment of opinion between two parties where a dispute is the disalignment of opinion between two parties?

How about the person that 'seconds a motion'? What do you call that person? What do you call the person that brought up the motion in the first place?

When I think about it, public method signatures should be well designed, so that, like well designed tools, it should be obvious how to use it. However, 'obvious' comes with a background and context, and even culture.

Update:
According to wikipedia, A person that makes the motion is a mover. The person that seconds a motion could be called a supporter. And according to my lawyer and english major friends, agreements and disputes are apples and oranges, apparently. agreements and disagreements are more on the same type.

Designing iPod vulnerability into it makes it cuddly

Everyone often raves about apple design. How sleek they look. How cool they look. However, there are sometimes practical design aspects that people complain about. Namely, I remember the first generation Nanos would have a faceplate that scratched easily.

However, I wonder if there is a side to the design that hadn't been considered. For a device like the iPod, it had the requirement of storing large amounts of data, but flash devices weren't that big in size yet. Therefore, hard drives were the only choice. However, we all know that electronics and especially hard drives are sensitive to shock. It would lose its performance and its ability to store data if it was knocked around all the time.

Whether it is intentional or a consequence of making it look sleek, the exterior of the iPod, and perhaps its vulnerability, leads people towards behavior that make them take care of it. They buy protective accessories for it, and I'm sure they throw it around less than their phones.

It would be brilliant if that was part of the requirements and spec: to get people to take care of their iPod by making it both sleek and vulnerable. It's much the same way (in function, not in emotion) that a certain instinct gets triggered when we see cute fuzzy things, except Apple managed it without fur or a large forehead.

---

Wilhem has built Annologger, a tool that lets people worship your dentist appointments.

Waiting for camera manufacturers

Jeffrey Veen

You know I'm not writing code since I've been posting a lot, and also reading feeds. But one last post...

"But simplicity isn't just interface improvements, but acknowledging the right tool for the job. One of the things that has always impressed me about the iPod, for example, is that the devices have no capacity for editing metadata, deleting or moving songs, or any of the other mundane tasks of maintaining your music library. Instead, designers at Apple moved all those tasks to iTunes, exploiting your computer's keyboard, mouse, and screen real estate. Flickr takes the same approach. They could have waited for camera manufacturers to add GPS chips or asked cameraphone users to thumb in their location. They chose, however, to exploit the fact that metadata can be added asynchronously without much penalty."

spoke to me. I have been thinking about this, and was wondering if geo metadata was in pictures. My initial thought was, camera vendors really need to move beyond just taking pictures. Cameras should send photos to a repository automatically over wifi, or cameras should record not only when but also where you took a photo.

Sometimes, you can't, because you don't have any control over those products. Therefore, you make due with whatever interface is available to tie things together first, and then slowly make it better.

Simple yet deep

Point and shoot software - Signal vs. Noise (by 37signals)

I think 37signals has it right with making things simple, and emphasizing simplicity, at least on a first use basis. But I still maintain something that Guy Kawasaki said in his book, art of the start...

I'm paraphrasing, but basically simple for new users, but deep for veteran users. There's something to be said about keep mastering a product over time.

New users, no matter how tech savvy, like simple to use. But there's something to be said for the constant exploration of a device (or domain for that matter). I liken it to driving stick after learning to drive automatic.

It's often hard to build the two conflicting things in a product...control vs simplicity. But I think it's important to think of deepness as well.

Too little focus in the information we get

My Whole Life in Happy Little Folders, by Jeffrey Veen

Jeffrey Veen was complaining that it seemed like all he did was un-bold things in RSS feeds. There's just so much info that you get overwhelmed. This was one of the primary reasons I unsubscribed to ridiculously prolific blogs like smartmobs, and now signal vs noise has hell of a lot more noise now. I still subscribe due to the few gems that get dropped occasionally. But once I find others that are more quality and less quantity, I'm ready to drop them off my feed reader.

But other than complaining about blogs, this was the primary reason that I moved from bloglines to reader google. I found that having the number of unread items just felt like I was being a slacker by not reading it. Google reader is much more unforgiving. News you don't read just become that--news you don't read, and it just scrolls on by as the days past by.

It's not something foresight would have seen, unless you've actually used a feed reader before. But I'm glad that google got that right with their feed reader.

I think there's always phases to technology, as others pointed out. But beyond the adoption curve, there is the phase where the users are over-saturated, and need some type of filter. This is often overlooked in my opinion.