Uranium Ore to G-Strings: Recommendations Gone Bad

Amazon's recommendation engine, I believe, relies heavily on collaborative filtering. The theory behind collaborative filtering reads something like: I will tend to like things that people like me like, where "people like me" is defined to be: people that perform similar actions, like buy the books I buy, or rate movies they way I do. Now, take a moment to go the Amazon page for the Uranium Ore product located at:

http://www.amazon.com/Uranium-Ore/dp/B000796XXM

and check out the "Customers that bought this item also bought ..." section. So after laughing for about 5 minutes, I started to think of why this might happen.

Here's a theory I had: there exists a group of users that like to spoof Amazon's engine for laughs. So they go out and provide a bunch of recommendations on goofy products, like those appearing as a recommendation for Uranium Ore. Then, they comb Amazon for products that don't have a lot of purchase transactions. Since Amazon's recommendation engine relies more heavily on purchase data to determine similar users to then pivot into recommendations, in the absence of this data, I reasoned, it probably relies more heavily on user provided ratings. So, without having to spend any money, the spoofers can simply rate the product well, and voila, out pop the G-String recommendations.

Well I don't think this explanation cuts it. I went through and followed recommendations made by a bunch of users w/ comments like: This is the best strawberry jam I've ever tasted. It works well with toast AND English muffins, and these users certainly did not make recommendations of the G-String kind. So, what gives?

Thanksgiving at Fox Farm in Plain

We've been having a fun Thanksgiving up here at Fox Farm in Plain, Washington. Fox farm is an old, well, fox farm, as in raising foxes for furs. The fox raising has long since stopped, and now its mostly a piece of land owned by a playful family that rents out a little cabin. The land is about 15 acres set in a beautiful valley near Lake Wenatchee in the Cascade mountains. The kids have been having a blast; the farm has chickens, goats and horses among other critters. The father of the family works up at Stevens Pass and collects all kinds of contraptions on the property, like old ski lifts, World War I vehicles, really old snow CATs, and more. Its pretty interesting just wandering the farm and running across things. We've also been enjoying having a nice campfire at night. Even in 20 degree weather, there's nothing like a campfire to stay cozy.

Nayan's Birthday Party

We had a blast the other day celebrating Nayan's 6th birthday. We went down to Sunset Bowl in Ballard. The kids had a hoot. The older kids were seeking order and civility in the bowling process. The younger kids just wanted to grab a ball and throw it down the lane. The younger kids eventually won out, so we settled for pure chaos with the names of random party goers appearing on the who's-to-bowl-next bowling monitors. After some initial discomfort with the anarchy, everyone had a great time, including the birthday boy.

Slurpies at the Science Center

We were at the science center the other day and I had a dying urge for a slurpy. Here's a quick clip of the kids savoring theirs. My appologies for the gunfire audio, but Blogger's video upload is a bit buggy.

The Rise of Khalsa - Punjabi Animation

Its been 14 years since I had my original epiphany on how the web would change the world, and I must say, I'm continually surprised at how rapidly things can change on the Internet. It was a little over a year ago when I wrote about the difficulties of finding kids content in Hindi. Today I did a YouTube search for Hindi cartoons and found a bunch of good content. I thought certainly there couldn't be much Punjabi content, but then my Punjabi cartoons query revealed a bunch as well.

So while searching, I ran across this animation called The Rise of Khalsa in Punjabi. Someone posted it on YouTube in 7 parts. You can also purchase the DVD HERE. The YouTube link to the first part is HERE. I have so far watched the first 3 parts, and it seems to be fairly well done. The video depicts Banda Singh Bahadur's rise and conquest of Punjab by the Khalsa army starting in the early 1700s; the Sikh victories contributed to the ultimate fall of the Mughal raj.

There are some violent images, so be aware if watching it with your kids. Unfortunately, there are no English subtitles in the YouTube video.

Lord of the Flies: Hard Lessons for a 5 Year Old

I posted HERE about Nayan's decision to wear a patka and be a proper Sikh. Well it seems he's changed his mind. For 4 weeks Nayan was perpetually taunted and called "weirdo", "freak" and repeatedly ordered to "take it off" by other kids. He was very surprised. He honestly expected other kids to be as excited as he was in his religious adherence. What particularly frustrated him, was that whenever a child asked him "why he had that thing on his head" - and he tried to explain it, they ran off laughing.

We had no idea he was being harassed until last week. He was afraid to say anything since the school authorities had lectured the kids not to "tattle." Ameen went in to speak with his teacher upon finding out. The teacher, to her credit, took action to separate some of the harassing kids. But I believe it was all too late. These things can be pretty viral in their propagation. The damage was already done.

Nayan learned a hard first life lesson: fitting in is really important, and in the all white school of North Beach Elementary, he already stands out. With all the pierced, tattooed, pink haired "open mindedness" of left wing Seattle - one would think this wouldn't happen. Well it seems many humans, even as young as 5, simply are obsessed with something as harmless as a head scarf. It saddened me to see him so excited with his new haircut and eagerness to be accepted, but alas, at least his heart was in the right place. He was very logical about it, he said, "Papa, perhaps I can try again when I'm older, and better at explaining Sikhism."

Saturday Morning Soccer in Ballard

We don't yet have a mini van, but we do spend Saturday mornings at Nayan's soccer matches. I can see why this is THE world sport; the game seems to come very naturally to kids, and unlike T-ball, they really get into it. A quick video of the kids playing afterward is shown below (the gunfire audio is courtesy of Google's new video uploading software, the original is HERE):

Seth Falcon joins Hypertext Solutions

Some folks have been asking what type of individuals we are looking for here at Hypertext Solutions. In general, we are looking for star contributors in multiple areas including but not limited to the areas of: text analysis, natural language processing, distributed programming, information retrieval, and data mining. I thought it might be helpful to chat with one of our new hires and ask him why he joined us.

Seth Falcon began working for Hypertext Solutions last week. In addition to being a key individual contributor to the well known open source Bioconductor project, Seth helped lead the overall development efforts for the project and was responsible for orchestrating the efforts of over a hundred contributors scattered world wide. Seth is a known contributor to R, a widely used statistical programming language. Seth brings with him a wealth of statistical and programming knowledge which will help us as we develop beyond our core areas of unstructured and structured text handling.

I asked Seth a couple questions when he first started last week; I have shared these below:

Seth, you obviously were working on very intriguing and influential work. You could have gone anywhere, both to large companies as well as myriad startups. What was it about Hypertext Solutions that really stood out for you as the place to join?

There were a number of factors that made Hypertext Solutions the most intriguing choice for me.  First of all, a small start-up was appealing because I want the excitement of building The Next Big Thing  along with the responsibility of helping to evolve the development culture.  Another factor is the problem domain.  The search space is a rapidly growing domain rife with interesting problems.  So not only did Hypertext provide the start-up environment I was looking for, but it gave me an opportunity to work with cutting edge technologies in a domain that I find interesting and challenging.

So, how do you like it so far?

A few days before I started, I received a "book bomb" from the company (don't worry, just a bunch of books, absolutely no explosives) consisting of a number of technical and fiction titles.  I thought this was a great way to start communicating the company's vision to new hires.    Today is day three.  I continue to be impressed by the technologies that we are working on and am enjoying getting to know my colleagues. I'm really excited about ramping up and starting to contribute to our products.

More about Seth can be found on his blog at: User Primary.

First Day of School

The other day was Nayan's first day of kindergarten. This was a bitter sweet moment for me. While I'm overjoyed Nayan is growing up to be a mature, articulate, and thoughtful young boy, its sad, in a way, to know he's growing up. I recall distinctly when I was 17, waving goodbye to all of my family and friends, as I boarded an airplane to head off to Chicago and my new college life; my family and friends were so emotional -- sad, and crying. In me, there was no sadness; I had to pinch myself to stop from smiling; I was thrilled at arriving in my new life and leaving my small town world behind. Without a doubt, when that day comes with Nayan waving goodbye to me, I'll be a real emotional mess.

Despite the teachers rounding us parents up and taking us to the "cry" room - no one was outwardly sad, most of the moms were just grinning looking forward to their precious hours of freedom. There were, however, many practical concerns. Nayan has never been away from home for such a long duration. Simply getting him to eat his lunch is a whirlwind of a challenge. Well he learned quickly. After 3 days of stubbornly refusing to eat anything but the cookies in his lunch, he ultimately caught on.

I was also concerned about Nayan getting picked on for his patka. Nayan has recently decided he will be a proper Sikh and never cut his hair. As others, Ameen and I were quite surprised by his decision, and powerful will. Nayan has virtually no role models here in the states who have kept their hair, but his dearest cousins in India all have their hair kept. Ameen, after many failed trips to the barber, ultimately relented. So it turns out no one teased him, as Ameen was certain would be the case. Our neighborhood is really full of such open minded and caring people. Thank God we don't live in France, where Nayan would have been ejected from the school system for simply following his faith.

KDD 2007 Videos Available

The KDD folks have made select talks and interviews from KDD 2007 available online. If you were unfortunate, like me, and unable to attend, you can view the videos HERE.

I recommend Calculating Latent Demand in the Long Tail by Chris Anderson of Wired Magazine.

The Rock Star Treasure Hunt

I've been interviewing a lot of people lately. Trying to find truly great employees is such an interesting process. I find it difficult to really put my finger on what predicts a future epic employee. I've interviewed a large number of people over many years. I've also hired a good number, and then gone on to see how they performed.

Over the years I have constructed a mathematical model to try to help me gauge different attributes of engineers and ultimately score them. Its not a terribly fancy model, but I do have a number of attributes I rank a candidate on after an interview; these scores are then combined to give me a final candidate score. In many ways, this is a ridiculous thing to do -- try and distill a human being down to a simple number between 0 and 1. What I find, is that I am constantly playing with the model to see which attributes really matter and in what relative weight to one another. I often go back and forth with numbers until I can align them w/ my intuition; I realize this is a silly game, but I find the process helps me think through the many issues and realize ultimately what is important. So I put the scores of the absolute best people I have worked with, and the worst, along with the scores I think I would have given them after interviewing them; this helps give me a little perspective. Some really key lessons stand out for me.

1. Programming problems and puzzles only take you so far. I like to ask a lot of tough questions, but the one lesson that sticks out in my head, is of the hundreds of people I have interviewed over the years, the one person who stood out as the uncontested best at answering my technical questions, turned out to be the absolute worst employee. As a candidate, I recall being utterly impressed by this persons answers. I kept drilling him from all angles, and then said, ok, these answers are simply great and I have to hire the candidate. I did, and then I realized what a mistake it was. It turns out, this person was a complete Bartelby the Scrivener, cleverly refusing to do any work. He simply would not do anything, other than reinstall various operating systems on his machine, and answer other people's questions. It turns out, he was really quite good at answering peoples questions, but a "mere" oracle was not what we needed.

2. Sort of the opposite of my first lesson, is that someone who my instincts tell me is worth hiring, but I may still be hesitant on, may still be great. I barely hired at least one of my greatest all time hires.

3. Motivation is a hard thing to gauge, but an essential part of any great contributor. People that have something to prove, can be incredibly motivated. For example, I once hired someone who had never programmed anything more than some toy applications based on a reading of "Java in 21 days." I knew I was gambling, but I was just so impressed with how much this person knew about something he knew nothing about only 3 weeks prior. Obviously he had many other great technical attributes, a strong math and physics background, etc. In any event, he was able to rewrite 6 months of work from a 15 year veteran in 5 long days and have the system really shining within a few short weeks. A few months later he was the star programmer amongst a strong team of 10.

Okay, I'll end this post, as I'm not sure what my conclusions are, other than, its not a straightforward thing to hire the best of the best.

Out of the dark - Hypertext Solutions buys InFact

Well for those of you who have wondered what my top secret quest for the past few months has been about, you can now, finally, read all about it HERE. It has been an extremely interesting time in my professional career leading the effort to make this deal happen. I have indeed learned a great deal about many things non-technical. I have met an amazingly talented bunch of people who have helped guide me through this amorphous and somewhat enigmatic process.

More than anything, I'm simply thrilled to join Hypertext Solutions (note: the name is just a placeholder, we're still in stealth mode) where I can focus on taking the work that has consumed the last 6 years of my life out to massive success. I'm so happy to be back in the start up world, and especially, a well funded one with a powerful and disruptive strategy.

Finally, for those of you who are interested in perfecting and building upon the world's best NLP based text analysis and search platform, send me your resume! I'd love to talk to you.

And now, here is a snippet from the press release:

SEATTLE--(BUSINESS WIRE)--Insightful Corporation (NASDAQ: IFUL), announced today the sale of its InFact search technology and associated intellectual property rights to Hypertext Solutions Inc. for $3.65 million in cash. Hypertext is a Seattle-based startup company focused on making information on the web more “intelligent.” The transaction closed today.

“Our move to sell the InFact technology enables us to focus on accelerating the growth of our S-PLUS product line and our Data Analysis business, which grew 14% during the first half of 2007 as compared to the first half of 2006,” said Jeff Coombs, President and CEO of Insightful Corporation. “We believe we are well positioned to drive additional growth in the predictive analytics space, and are currently investing heavily to deliver an increasingly enterprise-scalable platform and a series of high-value solutions targeted at the life science and financial services industries.”

Google Scalability Conference Talks

Greg Linden has an excellent summary of the Google Scalability Conference with a number of links to the talks. I've seen a few, and highly recommend the first keynote titled MapReduce, BigTable, and Other Distributed System Abstractions for Handling Large Datasets by Jeff Dean. Greg's full post is available HERE. Kyle Cordes also does a good job of summarizing YouTube's scaling experiences HERE. You can see the full YouTube scaling talk HERE.

Human Computation by Luis Von Ahn

I absolutely love StumpleUpon's video recommendation plugin. I now get great video's to watch highly tailored to my taste. Here is a lecture I ran into by Luis Von Ahn, a CS professor at CMU, lecturing on his approach to developing clever and addictive games to leverage human computational power to solve important problems. For example, one of the games allows him to build a database of metatags for images on the web. He argues he can metatag effectively all image content in Google's image search with his user population in a very short period of time. This lecture left me thinking social networking sites are significantly under utilizing their user populations.

How not to find great employees

If you're in a startup and are trying to attract great employees, whatever you do, do NOT post a job ad that looks like this:

CompanyX is a tiny technology startup in Seattle. We are working on some amazingly cool technology, and we face the ongoing challenge of matching people with other people (contextually and behaviorally) and scaling a complex Web application.

We're looking for scary smart people who are willing and excited to jump into a startup, work hard, have fun, make their mark, and work for part equity and part cash. We want to make a lot of money, but you should know this is not your Mommy's station wagon rolling down the freeway at 60mph. This is a pimped out Hummer on extremely rough terrain with life-threatening hazards around every corner. And we're going too fast. If that actually sounds FUN to you, then give us a yell and tell us exactly why we should have a conversation.

Trail Critters and Other Biking War Stories

So after non stop mountain biking for the past week up here at Whistler, I can say with confidence, bears rule the land. A few days back, I was flying through the woods on a cross country ride; I was descending some tight technical single track and ready to shoot out onto a gravel road when I noticed a rather large black bear mulling around in the trail exit onto the road. I stopped as fast as I could, smiled and waved at the bear, and proceeded to back out slowly. He watched me until I was almost out of sight. I then turned my back to him and zipped away -- heart racing of course.

Yesterday, there was a big fella grazing just a hundred yards or so at the top of the lift. I also met a French guy who told me all about his best friends grizzly encounter that had recently happened. Apparently his friend and another friend were mountain biking not too far from here, and a grizzly attacked the other friend. The Frenchman's friend charged the bear to save the other guy, and smacked the bear with his bike. The bear was pushing toward him while he was holding it back with his bike. He then tried to break the bears nose by punching it - to no avail of course. The bear then swiped him with his paw, and apparently ripped deep into his jaw bone. He then passed out, and awoke over half an hour later. Apparently the grizzly left him after determining he was dead, or unworthy. He escaped with only 256 stitches.

This is what riders up here do whenever talk of the furry ones comes up - we all have our war stories. Today, there is a section right above lower Whistler downhill, where you turn a tight corner and launch in the air and down fast into a long open descent. As I was rounding the corner just a moment before committing, I noticed a huge black bear grazing only a few feet from the trail at the bottom. Fortunately I slammed on my brakes and was able to back track out. Another instant and I would have had to race by him praying all the way that I wouldn't blow out a tire or that he wouldn't give chase.

Another rider up here had a mountain lion incident on Tuesday by the mini-disc golf course - he got confronted, and the lion kept pushing toward him. Finally he hopped on his bike and tried to out run the lion. Fortunately for him, he was a downhill racer so he was pretty quick. Apparently the lion chased him for quite a while. The lion story brought me back to my own lion encounter, but I'll save that for another time.

Once this sort of stuff creeps into your mind, its very hard to get rid of the thoughts. The past few days, I've been scanning for critters like mad. Its actually an annoying distraction when there's enough obstacles immediately in your path to pay attention to. Today I was dropping off a steep ladder when I got completely spooked by what turned out to be a life threatening squirrel. Ridiculously, I did end up with quite a crash - fortunately for my ego, it was a tough and terribly slippery spot. Well at least I got to test out my helmet and armor. In any event, I've had enough furry encounters in the past week to last me the year.

The Continued Adventures of Mateo: Fidel Oaxaca

My good friend Matt, now known as Mateo, dropped his life as a Seattle software geek, packed up his trailer, and drove down to the lovely beaches of Zipolite, Oaxaca. He has set up a new life down there. Occasionally I post some of his letters, with his permission of course, to remind us all that there is a world out there away from mortgages and 9 to 5 gigs. Here are some snippets from the latest:

Hey man. Thanks so much for getting back, I really appreciate it. [...] I'm now working with two other Mexican guys starting yet another Mexican non-profit to apply for world bank money to do a deforestation project. This is more Manolo's baby, he worked at the world bank for a number of years and his program was canceled by Wolfowitz. Needless to say he is quite pleased that W got the boot. Anyway he has retained a ton of contacts; I'm helping him prepare English applications and he is helping me find international money for things I'm interested in. The coolest thing, is learning from him.

In Mexico you can't do anything without the support of a local heavyweight. The local heavyweight can be a business leader, a member of an old family, a particularly good fisherman, or even somebody who tells good jokes. It has nothing to do with money or explicit power, it has to do with their personal power, their charm, their charisma, who knows what... Some people have this power [and] people follow them[;] they just know how to talk to all different kinds of people. With them you have a project. Without them you don't, they just take your money and nod. Usually if you go to a small town there are one or two guys who are heavyweights and you will quickly learn who. But one in a hundred, you can take the guy anywhere and plop him down and hell if he can't get everyone behind him in 30 minutes... no matter if they have never seen or heard of him.

Manolo is one of those. He looks like Fidel Castro at 50 with [a] gray beard, rides an old horse everywhere, and is shitfaced by 10:30 every day, but he's an incredible community leader. We went for a business meeting last week in the early morning sober hours. We were all in our bathing suits. We drank cheap Nescafe and he smoked cigars that he rolled from a single tobacco leaf at the table. He gave me the names of about 30 different international organizations that fund the work I'm interested in with tips for writing proposals and the names of his personal contacts. Then Filemon came by to borrow his gun to kill his cat, which had developed a [terribly] incurable disease. Manolo produced a small firearm and dispatched the poor cat, after which he announced he needed to take a ceremonial bath to cleanse himself of the bad vibes from killing an innocent animal. But no matter; we would resume the meeting with Alexis and I at the table and him out on the patio naked in a tub of water under the [bougainvillea] and next to his horse, who lives with him in his house. His house is the most amazing place I've seen, a sprawling decaying concrete estate with 180 degree views of the coast that he built himself over the last 20 years.

[...]

Nothing new here really. Things have quieted down a lot, we are waiting for the rain to start. Best to Ameen and the kids. take care, hasta la proxima,

Matt

Book Review: Come Back to Afghanistan by Said Hyder Akbar

I just completed an amazing book called Come Back to Afghanistan by Said Hyder Akbar, a 17 year old teenager from California. For anyone interested in the current state of Afghanistan, I highly recommend this book.

Books like this, i.e. personal accounts of a foreign and politically relevant land, are typically written by seasoned news reporter type folks, or other adults who already have a solid world view. Witnessing Hyder, a youthful teen, struggle to make sense of this world, often through colorful pop culture references, is so unique. In addition, I've always been fascinated by the recent history of Afghanistan. I was similarly glued to Soldiers of God, by Robert Kaplan, a detailed account of his experiences with the Soviet fighting Mujahideen. Perhaps it is the vivid descriptions of these rugged mountain war zones, places like the Khyber Pass or the Hindu Kush, that get to me. I'm also always interested in understanding the forces of tribalism, especially in South Asia.

Anyway, back to the book, not sure exactly how it happened, but somehow Hyder was given a microphone and recorder by Susan Burton who ultimately produced a show for This American Life, the NPR radio show hosted by Ira Glass. You can listen to the complete This American Life episode HERE. This show kicked off much of what appears in more vivid detail in the book. You can also buy the book on Amazon HERE.

Here's a snippet from a Publisher's Weekly review on Come Back to Afghanistan:

Akbar's refreshingly unsentimental reminiscences of visiting his father's homeland as a teen make for an intriguing portrait of Afghanistan at a time of significant transition. On 9/11, Akbar, who was born in Peshawar in 1984 but grew up in the U.S., was living near Oakland, Calif., where his father ran a clothing store. After the attack, the elder Akbar, a descendant of an Afghan political family, returned to his country to take a job as President Hamid Karzai's chief spokesman and, later, as governor of Kunar, a rural province. The author visited his father for three successive summers, and the result is this account, a closeup view of the creation of the country's post-Taliban democratic government, told from a perspective that's impressively both insider and objective. Akbar reports on chats with cabinet ministers and warlords, and sketches the lay of the land, visiting both sumptuous Kabul palaces as well as bombed-out villages. His youth and curiosity send him on some dangerous adventures (he retraces a mountain route between Afghanistan and Pakistan used by fleeing members of al-Qaeda and the Taliban), and that youthful flavor also infuses the writing with a hip stream-of-consciousness that is by turns funny, insightful and, occasionally, breathtaking.

All in all, I give this book 4 out of 4 stars.

Big Whistler Air

Here's a great video of riders getting some big air. By the way, that reminds me, there's a popular trail up here called Dirt Merchant - there's a lovely spot where you launch over a creek that flows between the launch and landing. Its such a great feeling - a middle aged guy like me can feel like Evil Kenievel for a wonderful few moments. But I digress, back to these far more talented kids, enjoy:

Whistler Fun and Lots of Sun

We got up to Whistler yesterday morning and have been having a blast ever since. Yesterday I had a great time riding a bunch of cross country trails all around the Lost Lake area. There are some really fantastic trails that I've never found before. Lots of great wooden planks and roller coaster fun on Rhinoceros Furniture - my these Canadians sure know how to make some unbelievable trails. Poor Maya didn't make it out of the house today; alas her paws are really sore from our heavy riding yesterday; I really want to find some good running shoes for her.

Today was a hoot doing some downhill riding. I ran into an old friend randomly who I haven't seen in 15 years. We had a blast riding together. And the weather has been just stunning - perfectly sunny and around 75 degrees.

Google Keeps Tweaking Its Search Engine

A recent NY Times article that is lighting up the search blogosphere called Google Keeps Tweaking Its Search Engine by Saul Hansell provides some insights on how the Google ranking system works. For anyone that has ever spent some time tweaking ranking metrics, this is an article worth checking out. Here are some snippets:

“Someone brings a query that is broken to Amit, and he treasures it and cherishes it and tries to figure out how to fix the algorithm,” says Matt Cutts, one of Mr. Singhal’s officemates and the head of Google’s efforts to fight Web spam, the term for advertising-filled pages that somehow keep maneuvering to the top of search listings.

Some complaints involve simple flaws that need to be fixed right away. Recently, a search for “French Revolution” returned too many sites about the recent French presidential election campaign — in which candidates opined on various policy revolutions — rather than the ouster of King Louis XVI. A search-engine tweak gave more weight to pages with phrases like “French Revolution” rather than pages that simply had both words.

At other times, complaints highlight more complex problems.

...

Mr. Singhal introduced the freshness problem, explaining that simply changing formulas to display more new pages results in lower-quality searches much of the time. He then unveiled his team’s solution: a mathematical model that tries to determine when users want new information and when they don’t. (And yes, like all Google initiatives, it had a name: QDF, for “query deserves freshness.”)

...

THE QDF solution revolves around determining whether a topic is “hot.” If news sites or blog posts are actively writing about a topic, the model figures that it is one for which users are more likely to want current information. The model also examines Google’s own stream of billions of search queries, which Mr. Singhal believes is an even better monitor of global enthusiasm about a particular subject.

As an example, he points out what happens when cities suffer power failures. “When there is a blackout in New York, the first articles appear in 15 minutes; we get queries in two seconds,” he says.

Mr. Singhal says he tested QDF for a simple application: deciding whether to include a few news headlines among regular results when people do searches for topics with high QDF scores. Although Google already has a different system for including headlines on some search pages, QDF offered more sophisticated results, putting the headlines at the top of the page for some queries, and putting them in the middle or at the bottom for others.

Clinton tackles the Fox smirk

I recall this being in the news a while back, but I missed it then for some reason. Anyway, its entertaining to watch Clinton go for the jugular.

The Camp Cody Reunion Tour

Nayan, Maya and I had a blast camping out in the tree house and sauna up at Camp Cody this weekend. The weather was a bit schizophrenic, but we still had much fun. Uncle Timothy now has a row boat that Nayan was able to row in at the Camp Cody pond. Since the water was freezing, he was a little tentative about following papa on his morning dives in, but the row boat was perfect. We also got some great mountain biking in. Nayan had a blast on the wide interurban trail, but got a little spooked on the single track. I think next year will be a big turning point as far as father son mountain biking goes.

Other CC updates: with the arrival of the pond, has come some very loud frogs. All night long the endless "ributting" had all three of us, the dog included, combing through the night to figure out where the culprit was hiding. It turns out there are many many frogs. Alas, we fell asleep.

Some juggler friends of Islando were performing at the Bellingham Juggler's Festival. The opening act was just terrific. It was mad Bruce Lee style action fighting, followed by slow motion facial contortions, followed by more fighting, and then, what else, lots of juggling! What could be a better combo then Kung fu and juggling for a 5 year old?

No Nayan hasn't gotten a haircut, this photo is from last year. It was a bit rain-i-er this trip.

Here's a view of the sauna from the tree house. We placed one of the trapeze mats (our friends Islando and Uquoia run their own circus and practice under the tree house) on the benches inside and "camped." We slept with the door open, so Maya was off running frog searching sniffing missions all night. It wasn't the deepest sleep I've had, but pleasant nonetheless.

Beautiful Ballard Karma

It has been raining off and on all day. Ameen and Ananya are in England this week, so I have taken the week off to spend some time with Nayan. We decided to go see the new Shrek movie. We were standing in line in the cold and drizzle. There was a huge line up. A woman came out and said, "Are you here to see Spiderman?" I said "No, we're here to see Shrek. She said, "You know it's sold out, but today is your lucky day." She gave us two tickets. I reached for my wallet and tried to give her some money, but she insisted our movie was her treat. Apparently it was her son's birthday party and there were a couple of no shows. I absolutely love Ballard. The rest of the world may be getting darker and more gruff, but beautiful Ballard is still full of wonderful people like this woman who value simple goodness.

Linus on Microsoft's Patent Infringement Announcement

I ran across an interesting article on ZDNet called Experts say Microsoft's patent quest won't go far. In the article, the author describes how many companies instruct their engineers to not perform any active searches. This is definitely true. I have spent many hours with my patent lawyer, and this is an emphatic point she makes to me repeatedly. Here is Linus's response:

"There are several reasons why engineers should not read other people's patents, only their own. And it's not a 'hide your head in the sand' issue, it's a very practical issue of it being a waste of time," Torvalds said.

For one thing, developing technology without looking at patents lets a person honestly say they developed that technology independently, which helps show that the patent in question doesn't meet the requirement of a technology not being obvious, he said. And engineers aren't likely to comprehend patents in the first place: "Unless you have a patent attorney at your side, patent language usually makes no sense."

He derided Microsoft for spreading FUD (fear, uncertainty and doubt) rather than tackling the issue forthrightly. "If Microsoft were to actually tell people what patents they claim we violate, we could either laugh in their face and show prior art, or just show them to be obvious, or we could do things differently," he said.

Goodness Gracious Me: Christmas Special Part 1

And another great episode of Goodness Gracious Me from the BBC. Enjoy.

Goodness Gracious Me - Typical Indian Parents

Here is another great clip from the BBC's Goodness Gracious Me episode Typical Asian Parents. What Indian child can't relate to this on some level? This episode reminds me of a week I spent in the 7th grade trying to convince my parents to let me play the bassoon. My parents were convinced this would lead to me being one of those sleepy haired guys wearing Scottish outfits playing for weddings in Punjab. Ultimately they relented, which is a good thing, since I would say a good chunk of my work ethic and ability to focus came from my early music instructors.

Startup Ideas - Not Even a Dime a Dozen

I've long had a habit of coming up with random ideas for startup companies. I have a few friends who do the same thing, and often we do so together over coffee or a beer. One of my friends told me about a blog with a startup idea a day. I did a search, but accidentally found this essay by Paul Graham called Ideas for Startups instead. I love that I can still serendipitously encounter wonderful things in the web. In any event, here's a great passage from Graham's essay:

I think people believe that coming up with ideas for startups is very hard-- that it must be very hard-- and so they don't try do to it. They assume ideas are like miracles: they either pop into your head or they don't.

I also have a theory about why people think this. They overvalue ideas. They think creating a startup is just a matter of implementing some fabulous initial idea. And since a successful startup is worth millions of dollars, a good idea is therefore a million dollar idea.

If coming up with an idea for a startup equals coming up with a million dollar idea, then of course it's going to seem hard. Too hard to bother trying. Our instincts tell us something so valuable would not be just lying around for anyone to discover.

Actually, startup ideas are not million dollar ideas, and here's an experiment you can try to prove it: just try to sell one. Nothing evolves faster than markets. The fact that there's no market for startup ideas suggests there's no demand. Which means, in the narrow sense of the word, that startup ideas are worthless.

This is so great. In the back of every entrepreneurs mind is this fear that if they openly discuss their latest great idea, it will instantly get stolen by someone. I now feel motivated to write up my last 30 startup ideas. We'll see if I actually do.

Movie Review: Chameli Starring Kareena Kapoor and Rahul Bose

One of my favorite movies of all time is Chameli, starring Kareena Kapoor and Rahul Bose, directed by Sudhir Mishra, and released in 2003. For those of you who are into the occassional "aaaart pilm" - I highly recommend it and give it 4 stars. If you have never watched a Bollywood movie, and can only watch one, it would have to be this or Sholay.

There are actually very few song scenes in the movie. Here is a snippet from the film when Kareena Kapoor "sings" Bhaage Re Mann.



And here is a snippet from AishFan, an IMDB user:

This movie is definitely a breakthrough in Bollywood cinema in terms of the story and the approach to telling it. The entire movie takes place in just one night. It is interesting to see how a family values, extremely wealthy man spends a night with a prostitute. Beginning with his prejudice, he slowly grows to accept, protect, and, one can even say, love her. It took an entire movie summed up in an extremely brief "climax" to perfectly convince and portray the theme that 'It doesn't matter what the backgrounds, gender, etc. are; there should just be love.'

Microhoo - Microsoft buying Yahoo

Well after many years of speculation, it is now all over the web. Among other places, according to the Financial Times article Microsoft eyes takeover of Yahoo, Microsoft is looking to acquire Yahoo. An outright merger doesn't seem like a great idea for either entity. The two companies have very different cultures and technology preferences. Not to mention, massive redundancy in many if not most areas. I have a hard time believing the internal Microsoft bureaucracies and vested interests won't end up "Hotmail-izing" the platforms at massive and unnecessary cost. As a result of this, and the significant cultural differences, I have a hard time imagining many of the great Yahoo folks would stick around. To me this smells like the AOL take over of Netscape. AOL simply wanted the traffic and name, and the great Netscape technologists fled by the droves. Yahoo has suffered few defections to Google due in part to its great work environment and people, I wonder how long that could hold up in a merger.

It seems to me, a smarter thing to do would be to leave the two companies separate, and create an alliance on the advertising side which is where they are both really suffering. Perhaps it makes sense to merge the Panama and MSN equivalent folks into a new organization where they would focus on providing a service comparable in quality to Ad Sense and Ad Words. This would allow them to pool their respect advertising networks as well.

On a related note, perhaps someone can explain this to me, but for every site I run, and of the handful of folks I have spoken to about this, we are all in agreement that almost all, in my case >99% , of our search traffic comes from Google. I should expect about 70%. Why is this? It is true that Google attracts a different type of user than the other engines, but I still can't understand the massive discrepancy. In addition, one of my advertising friends (this is someone who runs online ad campaigns for a living) says his company has virtually abandoned taking out ads for their campaigns on anyone other than Google. When I asked him why, his answer is they simply can't get the quality of click throughs for the same cost as they get on Google.

Here's an interesting snippet from a Forbes article titled Why Yahoo! Can't Fix Microsoft:

So why wouldn't buying Yahoo! help Microsoft? For starters, Microsoft is too slow and too cautious to keep up. While Microsoft howled after Google snapped up online advertising specialist DoubleClick last month for $3.1 billion, that's just the latest opportunity Microsoft has missed. In 2005, Google beat out Microsoft to buy a stake in AOL. And while Microsoft carped after the DoubleClick deal, Yahoo! rolled up its sleeves and snapped up the 80% of Right Media it didn't already own.

Of course, Microsoft often moves slow for a very good reason: It doesn't want to cannibalize its cash cows. While Google can launch online spreadsheet and word processing applications, Microsoft can't respond without damaging its powerful Office business. Elsewhere, it's more of the same. Everywhere Microsoft makes money, the Web poses almost intractable dilemmas.

Those kinds of dilemmas would make integrating Yahoo! into Microsoft's corporate culture a nightmare. And Yahoo!'s insiders have too much faith in Panama, the newly installed program designed to boost ad revenues, to give up.

The Simpsons - India Outsourcing

Here's a hilarious snippet from an episode of the Simpsons where Homer's nuclear power plant gets outsourced to India.

Lotion: Hair Gel Revisited

In my early morning haze, I have been known to do some odd things. My wife, on occasion, cracks up seeing the milk on top of the refrigerator, and a box of cereal inside. I've even shown up to work with my shirt inside out (usually happens because I turn my shirt inside out when I paint, and sometimes forget to switch it back). Today, my absentmindedness has reached a new extreme. After getting out of the shower, and finishing my shave, I often put lotion on my face. Well today I was helping Nayan, my son, get ready at the same time. I looked in the mirror, and realized all the lotion in my hand was now being rubbed into my hair. Proof, I suppose, that multi-tasking, at least when I'm doing the tasking, is overrated.

Photos from Ananya's Second Birthday Party

I know its a bit late, but I finally am getting around to posting some photos from Ananya's second birthday party. Its simply amazing how fast they grow. She's now very chatty and putting together great "Punjenglish" sentence like: "Sco Dhoro Papa" - translation: "Let's go run Papa."

UK's Soft Diplomacy with Iran Succeeds

Interesting article in the Christian Science Monitor called Lesson's Learned: Iran's release of British Prisoners regarding Britain's success securing the release of its prisoners from Iran. Here's a snippet:

"This is vindication for the British diplomats, who came under a lot of criticism," says Ali Ansari, an Iran expert at London's Chatham house think tank. "Diplomacy worked. People should reflect on this: There are diplomatic options when we deal with Iran. It's a very salutary lesson."

And here's a great quote on the counter strategy by Israel:

"What is to be gained from following the Israeli example of last summer [when it invaded Lebanon after its soldier was captured]?" asks Rosemary Hollis, a Middle East expert also at Chatham House, "You can't use a sledgehammer to crack a nut. They still haven't got their service personnel back, and they smashed up half of Lebanon."

How to beat Google

Interesting post and comments at skrenta.com on how to beat Google. Here are some snippets:

Our entire industry is scared witless by Google's dominance in search and advertising. Microsoft and Yahoo have been unsuccessful at staunching the bleeding of their search market share. VCs parrot the Google PR FUD machine that you need giant datacenters next to hydroelectric dams to compete. They spout nonsense about how startups should just use Alexa's crawl and put some ajax on top of it. Ye gods.

Grow a spine people! You have a giant growing market with just one dominant competitor, not even any real #2. You're going to do clean-tech energy saving software to shut off lightbulbs in high-rises instead? Pfft. Get a stick and try to knock G's crown off.

...

A conventional attack against Google's search product will fail. They are unassailable in their core domain. If you merely duplicate Google's search engine, you will have nothing. A copy of their product with your brand has no pull against the original product with their brand.

Duplicating Google's engine is uninteresting anyway. The design and approach were begun a decade ago. You can do better now.
...

Forget interface innovation. The editorial value of search is in the index, not the interface. That's why google's minimalist interface is so appealing. Interface features only get in the way.

Forget about asking users to do anything besides typing two words into a box.

Screen Projected Translations of Kirtan

Our Gurdwara, or place of worship, recently started a wonderful pilot project where translations of the ceremony are projected onto a screen in roman script, English, and Punjabi. As someone who was born and raised in the states, I can attest to the struggle a western native born goes through to try to understand our services.

Our Giani's, or priests, rarely if ever offer any translation into English, the primary and often only language of the young generation of Sikhs. In addition, most of our prayers and shabads, or hymns, are in Gurmukhi, which is a much older form of Punjabi. After years of struggling to become more fluent in Punjabi, I was frustrated to realize I still had far to go in understanding our holy texts in Gurmukhi. In my estimation, differences between Gurmukhi and Punjabi are something akin to the differences between the early modern English of Shakespear's era, perhaps even the Old English of Beowolf, and modern day English.

In any event, the projection idea is a fantastic new development. The way it works is a volunteer runs a laptop which projects onto a screen during the services. The volunteer listens to the service, and figures out which shabad or prayer is being recited. This person then pulls up a corresponding power point containing the translations and clicks through the slides as appropriate. If a prayer or shabad is not available in a prepared power point, the person goes to www.sikhitothemax.com and performs a search to get to the appropriate text. The system works amazingly well without being disruptive to the hymns and ragas; the serenity of the prayers in their native Gurmukhi is preserved, the Giani's can remain spontaneous in their service, and the Sangat can gain more insight into the texts. The one drawback is that the volunteer needs to be trained and spend some effort, but my guess is this will become easier over time as the kinks in the system are worked out. We're currently raising funds to purchase the full projection system, and it looks likely to happen.

My guess is other immigrant communities of different faiths have similar problems where the ceremonies are performed in a mother language often foreign to the western native generation. I really hope this idea catches on as it would be unfortunate if another generation feels alienated from its faith due to a basic communication disconnect.

Spoofs: Dil Luteya and Bulla Ki Jana

Here's a hilarious spoof on Dil Luteya and Rabbi's Bulla Ki Jana. If you don't speak Punjabi, my appologies; unfortunately there is no translation and this sort of thing doesn't translate well anyway.

Rabbi Shergill's Bulla Ki Jana Video

For those of you have never heard Rabbi, a lovely Sikh singer from Punjab, I highly recommend his debut album Rabbi. I had never seen a video of his; I really like this video for Bulla Ji Jana. Some nice street and life scenes from all over India.

Rabbi's lyrics remind me of the Sufi poet Rumi for their blend of mysticism and love. Here's the roman characters and english translation for Tere Bin courtesy of Wikipedia:

tere bin / besides you

sanu sohnia / my love

koi hor nahio labhna / i shan't find another

jo dave / who'll give

ruh nu sakun / peace to my soul

chukke jo nakhra mera / and indulge me

ve main sare ghumm ke vekhia / i have gone and seen it all

amrika , roos, malaysia / america, russia, malaysiana

kittey vi koi fark si / there wasn't any difference

har kise di koi shart si / they all had some condition

koi mangda mera si sama / some asked for my time

koi hunda surat te fida / some were fascinated with my face

koi mangda meri si vafa / some demanded my fidelity

na koi mangda merian bala / none wanted my demons

tere bin / besides you

hor na kise / no one else

mangni merian bala / wanted my demons

tere bin / besides you

hor na kise / no one else

karni dhup vich chhan / shall shade me in the sun

jiven rukia / (the) way you paused

si tun zara / slightly

nahion bhulna / i shan't forget

main sari umar / all my life

jiven akhia si akhan chura / you said, looking away

"rovenga sanu yad kar" / "you shall weep in my memory"

hasia si main hasa ajeeb / i laughed a strange laugh

(par) tu nahi si hasia / but you didn't

dil vich tera jo raaz si / you had a secret in your heart

mainu tu kyon ni dasia / why didn't you tell me

tere bin / besides you

sanu eh raz / none shall tell this

kise hor nahion dasna / secret to me

tere bin / besides you

peerh da ilaaj / what druid

kis vaid kolon labhna / has the cure to my ills

milia si ajj mainu / i found today

tera ik patra / a note of yours

likhia si jis 'te / on which you had scribbeled

tun shayr varey shah da / a varis shah couplet

park ke si osnu / upon reading which

hanjnu ik duliya / a teardrop fell

akhan 'ch band si / what was locked in the eye

seh raaz ajj khulia / was revealed today

ki tere bin / that other than you

eh mere hanjnu / these tears of mine

kise hor / won't be kissed by

nahio chumna / none else

ki tere bin / that other than you

eh mere hanjhu / these tears of mine

mitti vich rulnha / will wither in the dust

Children's Video on Guru Nanak Dev Ji

I ran across this lovely claymation video on the life of Guru Nanak Dev Ji, the founder and first prophet of the Sikh faith. It is in English, and well suited for children. Click on the links below to view the video:

Life of Guru Nanak Dev Ji - Part 1
Life of Guru Nanak Dev Ji - Part 2

12 Days of Christmas for the Brown Folk

And here's another little ditty that's a little late for Christmas, but entertaining nonetheless.

Indian Thriller Video

This video is hilarious. It's an unintended spoof of Michael Jackson's thriller. If you're looking to waste a little time, check it out.

Nature's Prozac: A Day On the Arm at Mount Baker

What an epic day of snowboarding I had yesterday. It has been dumping up at Baker, and I decided to take a vacation day from work and head up. I spent the previous night up in Bellingham at Camp Cody soaking in the hot tub staring into the forested hillside. It was only 2 inches of fresh, so we weren't all that excited to jump out of bed at 6.30AM. It turned out to be a great decision; the snow was uncharacteristically light - normal in Colorado or Utah, but not here in the chunky Cascades. Anyway, we hiked out the arm, and got some epic turns on virtually uncut powder. Ahh, this was a reminder of why I used to spend so many days of my life at the mountain. Since my main exercise lately has been toddler lifting, my body is soar and heavy, but of course, in a good way.

Book Review: The Glass Palace by Amitav Ghosh

I just finished Ghosh's amazing book: The Glass Palace. I spent all day and night Saturday until I could finally finish the last few hundred pages. It is a saga fit for the big screen, and perhaps one day, it will wind up there. I had little idea what World War II looked like from Burma, and Malaysia's standpoint.

This book provides an intimate portrait of life during this historic time, weaving the lives of many families spanning multiple generations; Ghosh provides lush descriptions of the palaces of the Burmese monarchy, the jungles of Burma and Malaysia, and many urban locales in India, Burma, Malaysia, and other places. I had only a foggy idea so many Indians were involved in the conquering of Burma by the Brits, and the subsequent occupation and economic construction to follow. My own grandfather went to Burma to strike it rich back in the 40s. After reading this book I'm intrigued to find out more about my grandfather's experiences there. I was also fascinated by Ghosh's portrait of the confused state of mind of the Indian soldiers forced to fight on behalf of their own colonizers; they were often sympathetic to the Japanese conquests; one group jumps sides to fight against the British alongside the Japanese. I had read about this in Kushwant Singh's Sikh History series, but Ghosh's portrayal really brought it to life for me. In any event, here's a summary of the book from Publisher's Weekly:

Ghosh's epic novel of Burma and Malaya over a span of 115 years is the kind of "sweep of history" that readers can appreciate and even love despite its demands. There is almost too much here for one book, as over the years the lives and deaths of principal characters go flying by. Yet Ghosh (The Calcutta Chromosome; Shadow Lines) is a beguiling and endlessly resourceful storyteller, and he boasts one of the most arresting openings in recent fiction: in the marketplace of Mandalay, only the 11-year-old Indian boy Rajkumar recognizes the booming sounds beyond the curve of the river as English cannon fire. The year is 1885, and the British have used a trade dispute to justify the invasion and seizure of Burma's capital. As a crowd of looters pours into the fabled Glass Palace, the dazzling throne room of the nine-roofed golden spire that was the great hti of Burma's kings, Rajkumar catches sight of Dolly, then only 10, nursemaid to the Second Princess. Rajkumar carries the memory of their brief meeting through the years to come, while he rises to fame and riches in the teak trade and Dolly travels into exile to India with King Thebaw, Burma's last king; Queen Supayalat; and their three daughters. The story of the exiled king and his family in Ratnagiri, a sleepy port town south of Bombay, is worth a novel in itself, and the first two of the story's seven parts, which relate that history and Rajkumar's rise to wealth in Burma's teak forests, are marvelously told. Inspired by tales handed down to him by his father and uncle, Ghosh vividly brings to life the history of Burma and Malaya over a century of momentous change in this teeming, multigenerational saga.

All in all, I give this book 4 out of 4 stars. If you are interested, you can purchase the book at Amazon by clicking HERE

Images from Iran

Unlike most folks, I'm not convinced we are about to invade Iran, but if we are, this is what we will be destroying: http://www.lucasgray.com/video/peacetrain.html

Some Art and Poetry Posts

I just posted a bunch of stuff on my art blog for anyone interested. The site is: deep-art.blogspot.com

Goodness Gracious Me: Bhangra Man Comes to America

Courtesy of Youtube, Goodness Gracious Me episodes have made it to the web. My wife and I have been searching for these ever since we saw a bunch of episodes on an Asiana flight on route to Delhi. Basically, GGM is like an Indian In Living Color put on by the BBC. Hopefully I'm wrong, but my guess is Google has yet to find out these are copyrighted material, so enjoy them while you can. While much of the humor requires a desi vantage, much still makes it through, at least I think. Here are some fun episodes if you are web video enabled:

Bharat Homes
Chunky Lafunga
Indian wedding

and my personal favorite: Bhangra Man

Ms. Dewey: Heir to Clippie

I've never thought of search as entertainment, but this site is hilarious. If you've ever had any doubt what the future of search was all about, surely Ms. Dewey will enlighten you. Some fun searches to try: art, namibia, and my personal favorite, gangster. Check it out: www.msdewey.com. Flash is required.

After being thoroughly entertained for half an hour, I realized this was a bit too polished for a Saturday Night Live skit, which is what I thought the site in essence was. After reading a bit more on the web, bizarrely, I found out this site is not a joke. I learn this was a serious attempt at search UI. Here's a quote from a Microsoft spokesperson courtesy of David Utter:

"This is not an advertising campaign. This [was] really just an experiment for exploring different ways to introduce people to search and Live Search specifically. We are not promoting the site but simply putting it out on the Web for discovery."

This is too much. How on earth could someone take this project seriously as a potential search interface? I think Microsoft should back peddle quickly and claim it was a brilliant online advertising tactic. More fuel for the Live.com branding fiasco fire.

A Day of Snow Play at Alpental

Here are some photos from the other day when we were all playing up at the mountain. It was Ananya's first day in the snow. She was initially very nervous about the white stuff, but eventually was jumping on the sled and screaming "more" repeatedly in Punjabi.

Vertical Searchers Unite

I ran across a post by Greg Linden discussing a couple research papers on search. Especially interesting is the Bender paper on peer to peer search. Basically the paper discusses a strategy on how to create "specialized" peers with topic based crawls of the web. While I think this is problematic in a true peer to peer system, due to the unpredictable quality of peers (or predictably bad depending on your level of cynicism) and their limited bandwidth among other things, it did make me think of a related idea. What if some of the myriad "vertical search" providers proliferating on the web could follow a standard for communicating the contents of their verticals along the lines of thinking in the Bender paper. Its not peer to peer, but to some extent it addresses the monopolistic concerns of the big 4, and it does address the dependability of a search node problem. In other words, create a meta search infrastructure, with built in economic accommodations of some sort, that vertical search providers could plug into; unlike the meta crawlers of yore that focus on meta crawling general search engines (e.g. metacrawler, dogpile, ...), they metacrawl niche vertical engines constrained by topic. If done right, this might lower the barrier of entry for small players to make an impact in broader search.

Saturday Snow Play

Here are some photos from this Saturday. These are all taken on a walk down to the beach. Ananya made it all the way down to the beach, and most of the way back. Nayan, for the first time ever, went all the way down and back with out whining.