mdoigny
08-16-2005, 04:01 PM
Hello!
After reading my server logs for some days now, it seems that my site is not getting indexed like it should.
This is what i found in the logs:
The home page is accessed by google more than once a day, but only the main index "GET /"
girafebot (never heard of it) downloads the images in the home page, but nothing else
i provide a sitemap (xml) for google, that is downloaded more than once a day by googlebot
pages with adsense are downloaded by mediapartners-google whenever a visitor downloads the corresponding page (to provide targeted ads)
I provide a robots.txt file containingUser-agent: *
Disalow:
and a meta tag <META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">
I try to provide anything that could make a spider happy; the right meta tags, plain href tags, enough content, ...
The pages are on a home server using non-standard ports. The ip itself is quite stable (the same for months).
Do i need to provide a base href containing the port number???
The home page (the only one that get's indexed) is located at
http://afr_upd.verfaillie.be:20800
What could be wrong with it (spider-wise?)
thanks
After reading my server logs for some days now, it seems that my site is not getting indexed like it should.
This is what i found in the logs:
The home page is accessed by google more than once a day, but only the main index "GET /"
girafebot (never heard of it) downloads the images in the home page, but nothing else
i provide a sitemap (xml) for google, that is downloaded more than once a day by googlebot
pages with adsense are downloaded by mediapartners-google whenever a visitor downloads the corresponding page (to provide targeted ads)
I provide a robots.txt file containingUser-agent: *
Disalow:
and a meta tag <META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">
I try to provide anything that could make a spider happy; the right meta tags, plain href tags, enough content, ...
The pages are on a home server using non-standard ports. The ip itself is quite stable (the same for months).
Do i need to provide a base href containing the port number???
The home page (the only one that get's indexed) is located at
http://afr_upd.verfaillie.be:20800
What could be wrong with it (spider-wise?)
thanks