Vannevar Bush was the most effective and influential science advisor of the last century. In 1945 he presciently warned of getting lost in the torrent of information that humanity is generating.
He also pointed to a solution: a memory storage and retrieval system he named the memex. Vannevar Bush's dream is embodied in today's web — and, fortunately, search engines were developed right along with it! Starting with the now-forgotten Archie in 1990 to Google and Yahoo today, search engines allow us to make effective use of the mountain of information that is the web.
What makes modern search engines so useful? How do they lead us to what we wish to find? They start by sending out spiders — programs that methodically crawl the web and store the information they find. Search engines work on these huge collections of words that are listed along with the webpages on which they were found. But how do unthinking machine know what is important to us humans?
