Web Spider Help!

[kyle_kirst]

So i have a question, I am building a website, in which i want to display update prices and information from hundreds of online vendors(kind of alike nextag.com/mysimon.com, but on a much smaller level). To do this, i want to deploy a shopping webbot to find, index, and update prices on a daily basis for multiple different produts. The back end database will be MySQL. Would it be best to program the spider in PHP, or another language. Or should i outsource this project and pay a developer to do this? Thxs for your thoughts, Kyle

[TheRave]

PHP was written for generating web pages, it wasn't really designed for searching/browsing/crawling webpages.

A spider is more a like an automated browser. Its like Internet Explorer but instead of a human operating it just automatically follows links, instead of a human reading it the information is processed and read into a database. So its basically an executable, which means you should be looking at C++, Java, VB(yuck!) or similar.

My suggestion would be to outsource this as it can be fairly complex.

Obviously once all the data from the spider is in the database you can then access and display this data using PHP for all your users to see.