I have a large site with many different pages on different subjects. I have excellent search position rankings for different pages on different search engines but it seems when each engine spiders or indexes my site it finds different pages. I am currently not running any refresh or robot tags, should I be? How do I get the search engines to visit my site on a regular basis and to find all of the pages.
I have paid for submissions to inktomi, msn, altavista and so on and they all say that they refresh on a regular basis. How do you submit again without spending a fortune? It appears that most of them reindex on a weekly or every couple of weeks basis. How do you get them to reindex without breaking out the credit card again.