Ok, now this is interesting.
http://websearch.alexa.com/welcome.html
Alexa (ie. Amazon) are offering developers access to their search index. Essentially that means that if you wanted to, say, create a video search engine, you could run your scripts against Alexa’s index and do what you need to the data. Actually, I can see this far more likely to appeal to market researchers – if Company X wants to know everything on the Internet about their competitor, Company Y – or even how others refer to X – this is the perfect place to run it. The difference between this and, say, running a Google search is that it’s your search algorithms, not Google’s. You can get to the raw data here.
I have to say (how geeky is this) that the documentation and API at http://pages.alexa.com/awsp/docs/WebHelp/AWSP_User_Guide.htm is far more interesting. It provides an insight into how a large search engine organises their files and indexes. I suspect there’s a lot to learn from this.