How does Yandex do its trick?

This site is best viewed in a modern browser with JavaScript enabled.

complete

I want to programatically get the contents of a Yandex.com search result.

The proglem is that a search page url does not change when you do a search on yandex.com and advance to see more pages. It must be done somehow by javascript. Any ideas?

Sup3rkirby

Hmm, I'm not sure if this is exactly what you want but:

http://www.yandex.com/yandsearch?text=text+or+content+to+search+for

As you are typing there are dynamic results displayed in a small list below the textbox, which is done through AJAX. But the actual search page URL is as posted above.

complete

That just gives you one page.

If you scroll down to the bottom of that page, you will see a "Search More" button. If you click that button, the URL http://www.yandex.com/yandsearch?text=text+or+content+to+search+for does not change in the address bar of the browser and yet the content displayed changes. So how do you programatically get the second page?

Sup3rkirby

The 'Show More' button for Yandex uses AJAX. AJAX is a useful set of functions and methods that allows a page to send and receive data to other files, scripts and servers without needing to change pages or reload anything.

Because of how AJAX works, you can't call on files/scripts from a domain other than your own, thus you wouldn't be able to use any of the AJAX calls Yandex uses unless you somehow could put your files on the same domain. And since the script to fetch more results is a server-side script it is not possible to view the source and see exactly what it does to fetch these results that it sends back to the page.

For the sake of knowledge I will say cross-domain AJAX is technically possible, but it requires a host to set special headers on their page that allow cross-domain scripts to call to it and access data or send request. This isn't very common currently so you can't ever expect to be able to use it.

In the end, all of that doesn't really matter but I did decide to explain since the original topic did ask how it was done. You obviously wouldn't be able to use the same AJAX calls but you can simply add the 'p' parameter to your search URL to change pages. It isn't shown in the normal web searches since the page uses AJAX to load them, but I noticed it on a button in their image search. It seems to work universally on the site.

http://www.yandex.com/yandsearch?p=1&text=text+or+content+to+search+for
http://www.yandex.com/yandsearch?p=3&text=text+or+content+to+search+for