Segmentation of social news sites

Giles Bowkett: Summon Monsters? Open The Door? Heal? Or Die?

I have to admit, I almost stopped reading after the first couple paragraphs justifying himself being jerk-ish, but he does have a healthy dose of good points towards the middle.

The underlying assumption of 'wisdom of the crowds' is that people make independent decisions, and they have the same amount of time to do it. Neither are true in social news sites. The former being untrue because you can up vote stories that are already on the front page. That just makes it into a positive feedback system that blows up and amplifies small signals. It's ok when the community is small, but as it gets larger, the more likely that noise will make it.

The second point I didn't think about until Giles pointed out explicitly--that since votes come for free, that people that spend their time on the sites are the ones that influence it the most.

Combine the two effects, you have a recipe for amplification of noise. The problem is, you need the amplification mechanisms like up voting on the front page in place when the site is small to grow it, and then when it reaches a certain size, the mechanics of social sites need to change (to what, none of us exactly figured it out yet) to protect the users from themselves.

I'm venturing to guess personalization and fuzzy segmentation to be one solution. As Paul Buchheit mentioned earlier about how twitterers hardly get any spam, it's because if anyone's saying stuff you don't want to hear, you can just unfollow them. Twitter works in this regard because there is a built-in small world network with a relatively low transmission rate between nodes (as opposed to facebook which has a small world network, but high transmission rates of information between nodes...which results in lots of unwanted invitations to bite zombies and vampires). Social sites like Digg, reddit, and hacker news, don't really have a network. It's just one single "place", where what happens on it affects everyone, and small perturbations get amplified.

However, I don't think such a strategy would work well in the beginning. The very thing that helps a small community in the beginning hurts a larger community, and the very thing that would protect a larger community from itself would stunt the growth of a new smaller one.

I think this would be an interesting topic and ripe for research. It actually reminds me of ant colonies, where younger ant colonies will act like teenagers, taking more risks, focus on growing, and experimenting. Older ant colonies are more about taking less risks, maintaining the brood, and surviving. There's some sort of decentralized mechanism that kicks in for ant colonies to do that, or maybe once they reach a certain size. I think looking into the literature for that might yield some clues into how to design community sites so that they can grow in the beginning, and not implode when they get bigger.

Nerd time Issue 14

I use to work at a research lab, and while they do know what's going on with military tech, they don't really get news of consumer web technologies. I read up on it alot while I was there, and I still do now. I started a little mailing list after I left to keep them updated, and that's what became Nerd time. Here is the latest for your enjoyment.

Hey all,

Happy 2008! It's been over a month since the last nerd time. At first, it was because not much was happening. I've found that Silicon Valley(especially the software/web side of things) is much like Hollywood in the sense that there are fads and fashions that people make ruckus about and then ditch after a month. And it's like sensationalism news when nothing noteworthy is happening, but people still find lots of trifle things to make noise about. So even though we attempt to filter it all through our favorite feeds and social news sites, it's still a chore.

I met up with Ray the first time since College, and he said he feels out of touch with the tech world, even though he's a chip designer. All he has to do is read nerd time! You get my own biased, hand-picked version of what's noteworthy.

Mike and Nick, check out Johnny Lee and Buglabs, being hardware freaks that you are.

As usual, the easy stuff is up top. If you want to stretch your brain, go further down.

Johnny Lee's wonderful world of HCI
Wiiremote VR tracking, Wii multitouch, Automatic registration of projection screens. It's this researcher's venture into HCI. more melding of computing into everyday lives. The videos are more self explanatory. If you skip everything else, I suggest you don't skip these.
http://www.youtube.com/watch?v=Jd3-eiid-Uw
http://www.youtube.com/watch?v=5s5EvhHy7eQ
http://www.cs.cmu.edu/~johnny/projects/thesis/
http://www.youtube.com/watch?v=s59PObuJGWY

Facebook, Google, and Plaxo join DataPortability Workgroup
I strongly believe that a users's data belongs to the user, and they should be able to take it around with them. I was pretty excited about Google's announcement about OpenSocial platform around October or so. I haven't personally taken a look at it myself, but according to pips here and there, it's been said it's incomplete and there was a case of a security hole. But this announcement of fb joining a data portability workgroup is good news, as making your data portable with you across applications will go a long way in making using computers a lot easier.
http://www.techcrunch.com/2008/01/08/this-day-will-be-remembered-facebook-google-and-plaxo-join-the-dataportability-workgroup/

EarthClassMail
This is a startup that you redirect all your snail mail to, and it opens and scans it in for you. You basically can read snail mail like you do email. You can even choose to recycle junk mail (good for the earth), and shred important documents. Neat idea, though of course, it all has to rely on trust of the company. As the social-political climate of the US over foreign energy dependence gains awareness with the push for 'green', we might see more companies like this that tries to move things into digital to save on green.
http://www.techcrunch.com/2008/01/08/133-million-for-startup-that-wants-to-kill-snail-mail/

Google's philanthropic arm going to develop clean energy cheaper than coal.
Just because something is digital doesn't necessarily mean it has a small environmental footprint. The servers it's running on might be burning coal. I think it's rather big, actually. It remains to be seen whether Google will democratize energy production, to shake up a very traditional energy industry, or simply join as another player. Regardless, this venture makes sense for them, as they run gigantic server farms.
http://www.techcrunch.com/2007/11/27/google-takes-on-global-warming/

Prediction Markets at GooglePlex study organization information flow
Google uses internal prediction markets to gather data on what's going on. Prediction markets are basically using the mechanics of stock markets to predict things like, "is google stock over valued?", "Will the U of I win the Rose bowl?", "Will Comet win mindshare with developers?" This particular article talks about mapping information flow in organizations. I think as the world moves faster and faster, mid-sized companies are exploring more flexible organization structures to be able to respond quicker to changing markets. You'll probably see more studies into company organization in the future.
http://googleblog.blogspot.com/2008/01/flow-of-information-at-googleplex.html

PathIntelligence
This is interesting because it's another indication of how real-life is merging with digital. Any website monitors its traffic to know which pages are more popular than others. It uses that as implicit feedback to what customers like. This does the same for physical storefronts. Using bluetooth on customer's cell phones to track foot traffic, they can see which racks and displays are selling more. As more and more sensors enter into the world, we'll be able to do analysis and better understand our lives. I won't talk about privacy concerns, cuz that's just a can of worms.
http://www.techcrunch.com/2007/12/14/path-intelligence-monitors-foot-traffic-in-retail-stores-by-pinging-peoples-phones/

Amazon SimpleDB and computation on tap
I wish I brought Amazon stock. The more I find out about Jeff Bezos, the more I understand that it's not really a bookstore that he built, but a philosophy. Anyway, Amazon, in this past year has been releasing "computation on tap" They basically sell distributed computing and storage services to developers and startups. Lots of people have taken advantage of it, for good reasons you can read elsewhere. In the past month, they released simple db, which is really just a fast index db. With computing on tap, it enables small teams to do large things. If computing and storage is a commodity, I suppose the value becomes what you can build with it.
http://www.amazon.com/gp/browse.html?node=342335011

WiTricity wireless power
Wireless power has been in development for some time now. I had mentioned it in one of my earlier nerd times. This one, although just experimentally demo'd, is rather neat. We probably won't see wireless power more ubiquitous until maybe 5 years later. I don't quite understand it myself (should have paid attention in PHYS 112), but apparently, it's not radiation like we're familiar with. Instead, it uses magnetic resonance, so that only two electromagnetic devices with the same resonance will transfer energy efficiently, everything else, like humans, won't.
http://web.mit.edu/newsoffice/2007/wireless-0607.html
http://www.technologyreview.com/Infotech/19889/?a=f

Buglabs
This is a startup making mobile devices that you can hack. They plug together like lego pieces and run linux. People can prototype and experiment with different mobile form factors and fulfill their niche needs.
http://video.google.com/googleplayer.swf?docId=-1932063822649530376&hl=en
http://video.google.com/googleplayer.swf?docId=-3090880151267528595&hl=en
http://video.google.com/videoplay?docid=-3950989589304402454&hl=en
http://www.buglabs.net/products

Quagmire 2D programming language
This is a 2D virtual machine, and you can watch it as it uses its memory visually on a 2D surface. What's interesting is that the latter examples looks like chaos, which reminds me of the pictures you see in "A New Kind of Science" In it Wolfram claims that basic computers nowadays just have simple loop patterns rather than the complex ones you'd see in class 3 CAs. Problem is, we don't know yet exactly what class 3 CAs compute better, and how to harness it, other than CA120 is turing complete.
http://www.pawfal.org/Art/Quagmire/

Comet - AJAX's cousin
Comet is using AJAX to push information to browsers, rather than having browser poll the server. The term was coined about two years ago, in reference to the bathroom cleaner. So you can get stuff like chat working in browsers, or like a stock market feed.
http://simonwillison.net/2007/Dec/5/comet/

Sunspider - javascript benchmarking
Javascript is getting faster and faster with work being donw on the underlying engines underneath popular browsers. I've seen "lemmings" implemented completely in javascript. You'll continue to see this sort of thing. However, flash, silverlight, and flex are still far ahead of the curve when it comes to interactivity of the web.
http://webkit.org/perf/sunspider-0.9/sunspider.html
http://www.elizium.nu/scripts/lemmings/

Version control in emacs
The new emacs 22 has svn version control built it! *high fives Ian*
http://www.credmp.org/index.php/2007/12/08/emacs-hidden-gems-version-control/

Tumblelogs
Tumble logs are the half cousin of twitter and blogs. They represent a flow of conciousness digital media tidbits from the poster's daily life. We might start to see more of this if it catches on. However, other than personal expression, I don't exactly see a future for it. But then again, I was wrong about Friendster when it first came out.
http://tumblr.com
http://www.robbyonrails.com/articles/2007/12/19/putting-tumblr-to-work-for-you

Next gen ANNs
A talk on next generation Artificial Neural Networks. I haven't watched it yet, but will this weekend. It promises to be pretty exciting.
http://www.youtube.com/watch?v=AyzOUbkUf3M

Ycombinator in Ruby
I finally figured out what ycombinators are. If I had to take a stab at it, it's a way to implement recursion when you don't have named recursions, loops, iterators, etc, and all you have are functions and a couple substitution rules. It's pretty neat, and will stretch your brain a bit to understand it. Traditionally, you only do this in lambda calculus, but having the code in Ruby made it easier to understand.
http://www.eecs.harvard.edu/~cduan/technical/ruby/ycombinator.shtml

A simple distributed crawler in Ruby

A week ago, I took a break from Mobtropolis, and...of all things ended up writing a simple distributed crawler in Ruby. I hesitated posting it at first, since crawlers are conceptually pretty simple. But eh, what the heck.

This was just an exercise to do some DRb and Hpricot, so don't use this for your production work, whatever it may be. An actual crawler is far more robust than what I wrote. And don't keep it running hammering at stuff, since it'll get you banned.

First, this is how you use it:

WebCrawler.start("http://en.wikipedia.org/") do |doc|
  puts "#{doc.search("title").inner_html}"
end

And that's it. It returns documents in an XPath traversable form, courtesy of Hpricot.

A web crawler is a program that simply downloads pages, takes notes of what links there are on that page, and puts those links on its queue of links to crawl. Then it takes the next link off its queue and downloads that page and does the same thing. Rise and Repeat.

First, we create a class method named start that creates an instance of a webcrawler and then starts it. Of course, we could have done without this helper method, but it makes it easier to call.

module Crawler
  class WebCrawler
    class << self
      def start(url)
        crawler = WebCrawler.new
        crawler.start(url) do |doc|
          yield doc
        end
        return crawler
      end
    end
end

So next, we define the initialization method.

module Crawler
  class WebCrawler
    def initialize
      puts "Starting WebCrawler..."
      begin
        DRb.start_service "druby://localhost:9999"
        puts "Initializing first crawler"
        puts "Starting RingServer..."
        Rinda::RingServer.new(Rinda::TupleSpace.new)
        
        puts "Starting URL work queue"
        @work_provider = Rinda::RingProvider.new(:urls_to_crawl, Rinda::TupleSpace.new, "Queue of URLs to crawl")
        @work_provider.provide
        
        puts "Starting URL visited tuple"
        @visited_provider = Rinda::RingProvider.new(:urls_status, Hash.new, "Tuplespace of URLs visited")
        @visited_provider.provide
      rescue Errno::EADDRINUSE => e
        puts "Initialize other crawlers"
        DRb.start_service
      end
      puts "Looking for RingServer..."
      @ring_server = Rinda::RingFinger.primary
      
      @urls_to_crawl = @ring_server.read([:name, :urls_to_crawl, nil, nil])[2]
      @urls_status = @ring_server.read([:name, :urls_status, nil, nil])[2]
      @delay = 1
    end
  end
end

This bears a little explaining. The first webcrawler you start will create a DRb server if it doesn't already exist and do the setup. Then, every subsequent webcrawler it'll connect to the server and start picking URLs off the work queue.

So when you start a DRb server, you call start_server with a URI, then you start a RingServer. What a RingServer provides is a way from subsequent clients to find services provided by the server or other clients.

Next, we register a URL work queue and a URLs visited hash as services. The URL work queue is a TupleSpace. If you haven't heard of TupleSpace, the easiest way to think of it is as like a bulletin board. Clients post items on there, and other clients can take them out. This is what we'll use as a work queue of URLs to crawl.

The URLs visited is a Hash so we can check which URLs we've already visited. Ideally, we'd use the URL work queue, but DRb seems to only provide blocking calls for reading/taking from the TupleSpace. That doesn't make sense, but I couldn't find a call that day. Lemme know if I'm wrong.

module Crawler
  class WebCrawler
    def start(start_url)
      @urls_to_crawl.write([:url, URI(start_url)])
      crawl do |doc|
        yield doc
      end
    end

    private

    def crawl
      loop do
        url = @urls_to_crawl.take([:url, nil])[1]
        @urls_status[url.to_s] = true
        
        doc = download_resource(url) do |file|
          Hpricot(file)
        end or next
        yield doc

        time_begin = Time.now
        add_new_urls(extract_urls(doc, url))
        puts "Elapsed: #{Time.now - time_begin}"
      end
    end
  end
end

Here is the guts of the crawler. It loops forever taking a url off the work queue using take(). It looks for a pattern in the TupleSpace, and finds the first one that matches. Then, we mark it as 'visited' in @urls_status. Then, we download the resource at the url and use Hpricot to parse it into a document then yield it. If we can't download it for whatever reason, then we grab the next URL. Lastly, extract all the urls in the document and add it to the work queue TupleSpace. Then we do it again.

The private methods download_resource(), extract_urls(), and add_new_urls() are merely details, and I won't go over it. But if you want to check it out, you can download the entire file. There are weaknesses to it that I haven't solved, of course. If the first client goes down, everyone goes down. Also, there's no central place to put the processing done by the clients. But like lazy textbook writers, I'll say I'll leave that as an exercise for the readers. snippet!

---

webcrawler.rb

Nabaztag and pet appliances

A couple weeks ago, I happened upon this strange internet rabbit. Nowadays, there are lots of electronic pets that react to people, so that wasn't anything new. But there were some aspects of Nabaztag that were intriguing and was worth some musing. Nabaztag is a kind of appliance/toy that's connected to the web. It's a little hard to describe at first, but this how it works does a good job of giving examples.

Generally, I see it as a simplified interface for the web embodied in a pet avatar. If you've ever watched any anime, you'll be familiar with pet sidekicks, usually for comic relief or raising the cuteness factor. If you imagine a sidekick through whom you can channel to communicate/interact with others, or to receive news, that'd probably be on the spot to what a Nabaztag does.

But why do I think it's worth posting?

When the internet was conceived, there were many users on a few computers. This has changed significantly. There are now a few users on many computers. Computers are not only mainframes, but first desktop, then laptops, and now budding, mobile devices. Eventually, there will be many more devices retrofitted for the web, such as refrigerators, stoves, and clocks. But that doesn't mean that there won't be communications appliances made specifically for the web.

Nabaztag seems to be, one of the early steps in making communications appliances in a form that people bond with. This can work one of two ways.

1) Just on the news today was a piece on how toddlers treated a QURIO as one of their own and bonded with it.



I can see something QURIO-like that will do just enough to fool us in the right ways for us to bond with it, like the toddlers have, and make it part of the family--like a home robot. Not really a robot to do heavy labor, but more like a companion/pet that can give you the weather and channels your friends to you.

2) Alternatively, we can have electronic pets that don't fool us, but rather, they are representatives and reflections of ourselves and we use them to interact with our friends' pets (also representatives of themselves). We already do this to some extend through MMORPGs as well as the many Sim Games. However, the difference is that we play those avatars. Here, the pets are recognized to be separate from ourselves, but they are our delegate. Just as people socialize through their dogs at dog parks, I think people will start to socialized based on their physical electronic pets.

If the pets learn its owner's habits, when it meets other people's pets, one might be able to trade information/gossip, or judge how well they'd get along with each other based on how well their pets got along with each other.

So why check the weather through a robot rabbit than through the browser on your computer? Sometimes, it's a lot faster through the rabbit, since it has lights and indicators you can check at a glance. Presumably, that's why the Ambient Orbs have been making money. Information becomes a part of a person's environment, rather than something that's queried.

But an even more compelling reason is that it becomes another dimension in interacting with other humans, and in self-expression. The pet becomes an extension of self that one uses to interact with others. And with the web, the interaction doesn't have to be physical. One of the interesting demonstrations on Nabaztag is that two rabbits can get married, and thus would imitate each other's ears. If a user on one end can control the ears of your friend's pet, you can communicate tactile touch, and your pets would be channeling you to your friend. Extended to a gel-like tactile substance that changes shape like a piezo-electric, it would make it even more real.

And while I don't know for sure whether any of this will happen, it seems like an exciting area to explore.

Mobtropolis Public Release

I've been working on Mobtropolis for about 10-12 weeks now. It's was finally released last week Tuesday. It's something that makes people expand their world by helping them discover and share local adventures around them. The easiest way to think about it is as a dynamic large-scale photo scavenger hunt or a photo-dare site that helps you expand your world--hopefully for the better.

Behavior is hard to change, so it's framed slightly in terms of a game. The basic mechanics should be familiar to those that frequent social news sites. Anyone can submit scenes or vote them up. The higher something's voted, the higher its visibility to others. Anyone can do a scene and take a photo as proof. They can then send it in via their camera phone, or upload it from their digital camera when they get back to desktop. Their friends who voted for a scene will then get an email with the photo attached.

It's been oddly thrilling to get photos of people doing scenes that you submitted.

There's still a lot of work to be done on it. Eventually, I hope to marry the virtual and the real in a tighter loop and better integration with mobile devices. However I'm putting it out in according to startup mantras of "Release early, then iterate like crazy". So check it out, and if you'd be so kind, give me some feedback, good or bad, so I can make it better.

http://www.mobtropolis.com

Enjoy!

Twittering as a platform

Amazon is posting their deals on twitter. I'm not quite sure that people would want deal ads on their cell phones all the time...

I'm kind of amazed, as are other people in the naysayer category have been, that Twitter had taken off as it has. At its basic form, it's just passing back and forth messages, a problem seemingly solved by email decades ago. However, twitter obvious is not a question of the underlying technology, but rather, how it is presented to and used by people. It's gotten people use to the idea of instant self-expression, no matter how inane--for better or worse. I would have chalked it up for sensors to monitor and log ourselves, but twitter demonstrated that people will report or say anything if there's an audience. Perhaps trolls have already paved the way in this regard.

That said, I think it's easy to write Twitter off as a fad, since the world's largest collection of quips doesn't quite seem to make the world a better place. My guess is that there's probably value in Twitter, but only when it's married with other sorts of data or text processing. Just off the top of my head, geospatial data and emotion detection algorithm on twitter data could generate a heat map of how people are feeling place to place, or time to time. I imagine advertisers would find this information valuable, since they can set up targeted advertising when people are statistically most vulnerable to impulse buying at a certain time or place.

If twitter can manage an API or platform to support this sort of thing, they'll be around for a while, I think. If not, well, at least we'd have the largest collection of quips for the archaeologists of the 22nd century.

"Web 3.0" and "Killer App" sound like "Crystal Ball" to me

Ahh, web 3.0.

Indeed, as nanobeeper's asks and puts into perspective, What's up with the web 2.0 angst?, there doesn't seem to be a need to get bent all out of shape over the term. And yet, I usually don't use the term myself and am pretty reluctant to, for fear of being someone-who-doesn't-know-what-they're-talking-about, like the braying butthole in Jeffery Zeldman's famous post. It's what happens when marketers get out of control, and generally, it applies when someone that knows just enough to be dangerous. Fanboys of Japan is a good example. If you meet someone that LOVES Japan, they've either only watched anime (or been to Japan once or twice), or they've lived there for at least a decade. Usually the former.

But what I want to post here today isn't want I think is or isn't web 3.0, but more about the usage of the term. Why do people use it?

Killer app 3.0

It's an interesting parallel that ever since Visicalc came out and the term "killer app" was termed, people since then has been talking about the "killer app" on this platform or that. "The killer app of the web is..." "The killer app of the mobile phones is..." It's certainly reminds me of the way people talked about web 2.0. "Web 2.0 is...." "Web 3.0 is..."

The similarity between talking about web 3.0 and talking about killer apps is that when people talk about them, they're using those terms to try to communicate what they see, predict, or would like to be the future. Technologists are, if anything, always looking for the Next Big Thing. We're use to change, and in fact, we thrive on it. We're all interested in the future of change because if we're right about it, that kind of information is an advantage over whatever our goals are. But as we all know, predicting the future is well, inaccurate at best.

I'd trade intelligence for hindsight

Often times, we have limited scope, experience, and knowledge. That certainly will affect what we think to be in the realm of the possible and what will be in the realm of the impossible. If you look back on quotes about technology predictions, some of them might stun you at how stupid they are. But then again, you have the gift of hindsight. Keep in mind what technology was available at the time for them to relate to the new tech, as well as the fact that first iterations of any product sucks--as Guy Kawasaki so famously points out. (If you want to read more, they're from wikipedia)

"Heavier-than-air flying machines are impossible." -- Lord Kelvin, British mathematician and physicist, president of the British Royal Society, 1895

"Who the hell wants to hear actors talk? The music — that's the big plus about this." Warner Bros. was investing in sound technology though Henry Warner was more excited about the potential of scoring over dialogue. [3]

"Caterpillar landships are idiotic and useless. Those officers and men are wasting their time and are not pulling their proper weight in the war." -- Fourth Lord of the British Admiralty, 1915

"The wireless music box has no imaginable commercial value. Who would pay for a message sent to no one in particular?" -- Associates of David Sarnoff responding to the latter's call for investment in the radio in 1921.

"While theoretically and technically television may be feasible, commercially and financially it is an impossibility, a development of which we need waste little time dreaming." -- Lee DeForest, American radio pioneer and inventor of the vacuum tube, 1926

The last two quotes are notable. Lee DeForest, who had enough foresight and innovation to see that a radio had value, couldn't see beyond that to see how a television would have value just five years later. We all have limited breadth and imagination, but some people are worse than others. It would do you well to ignore those people. Sometimes you can recognize them if they counter with "Why would I ever do [insert whatever idea you just told them]"

And even if we had perfect scope, perfect breadth, it would still be hard. Predicting the future is computationally intensive.

As for myself, I didn't immediately see the value of social networks apps until Facebook showed up, even though I read research papers on social networks. And currently, I don't really get Twitter and Scribd, but the fact that people are using it, well, there's value somewhere in there.

So what's the chorus in all the noise?

So where does that leave us with Web 3.0? If you look at it as people merely trying to say what their predictions about the future of the web, it doesn't conjure up as much anger, because you know they may very well be wrong. But collectively, what everyone predicts to be web 3.0 will have some value because part of it might be a self-fulfilling prophecy. If we all say it's true, you can be sure that some of us will work to make it true.

Based on that flash in the pan, I was curious. What was the collective consensus on what web 3.0 is? I looked in two places. Wikipedia and del.icio.us. Just from eyeballing it, it seems to be that people are in consensus, at least about the semantic web. This would be the type of thing that Inkling Markets would be good for. I created a market for it, if you're so inclined to buy stock on web 3.0.

So take what any individual says to be the future with open mind and a grain of salt, but really pay attention to where the global trend is moving. As Joe Kraus says, you want to see what the trend is, take it out of geek land, and ride that wave.