www.webdeveloper.com
Results 1 to 10 of 10

Thread: Why are Search Engines not crawling to my site?

  1. #1
    Join Date
    Mar 2005
    Posts
    8

    Why are Search Engines not crawling to my site?

    I have a site map, all my links work. but for some reason the Search engines don't seem to crawl past my first page.

    Here is my search logs...

    4 different robots* Hits Bandwidth Last visit
    Inktomi Slurp 7+2 23.68 KB 27 Mar 2005 - 12:51
    Unknown robot (identified by hit on 'robots.txt') 0+6 732 Bytes 30 Mar 2005 - 07:00
    Googlebot 2 5.79 KB 02 Mar 2005 - 20:16
    AskJeeves 1+1 3.40 KB 30 Mar 2005 - 23:59

    www.clarus-solutions.com

    My questions are....
    1. Is php affecting the crawl? If so, what can I do to get the robots to crawl php.
    2. I have a robots.txt file, is there anything else i should have
    3. Even though they have crawled my first page, I still am not showing up on any search results. Why?
    4. Is there any other method I should consider to move up the ranks? (besides paid advertising)

    Thanks!

    Robert Banta
    http://clarus-solutions.com
    rbanta@clarus-solutions.com

  2. #2
    Join Date
    Mar 2005
    Posts
    78
    php won't affect the bots
    Your meta tag is way off
    And it resides above the html tag
    meta tags should be placed in the head

    Code:
    <META 
    content="web design, graphics design, IT, instruction, Microsoft, hosting, freelance, sql, html, php, asp, database, mysql, banner, flash, xml, wml, smart phone, pda, wireless, javascript, michigan, troy, sterling heights, printer, publisher, designer, 3D, animation, web page design, proofer, data storage, video storage, hard disk, mouse, graphic tablet, firewire" 
    name=keywords>
    <html>
    <head>
    Keywords should be limited to ten words
    Those words must be inside your <body>

    the proper meta syntax is
    Code:
    <meta name="keywords" content="web design, graphics design, IT, instruction, Microsoft, hosting, freelance, sql, html, php, asp, database, mysql, banner, flash, xml, wml, smart phone, pda, wireless, javascript, michigan, troy, sterling heights, printer, publisher, designer, 3D, animation, web page design, proofer, data storage, video storage, hard disk, mouse, graphic tablet, firewire" />
    You should have a robots meta tag such as
    Code:
    <meta name="robots" content="index,follow" />
    Oddly googlebot likes link outs If you have a page on your site
    with listings that link out to other sites It actually boosts your ratings.

  3. #3
    Join Date
    Mar 2005
    Posts
    8
    Thanks, I 'll make the changes! No idea why the bot is not crawling to my site map tho?

  4. #4
    Join Date
    Jun 2004
    Location
    midlands uk
    Posts
    274
    <meta name="robots" content="index,follow" />
    Most people do not need the meta robots tag on their web pages. Having it on a web page does not help the page get indexed in any way. In fact, the tag is mainly used as a way to keep your web pages out of the search engines. Nor does it even work well for this, since most major search engines do not support it.

    It tells search engines to index the page and follow links to inside pages, so that they can be index. But search engines will do this even without the code, to the same degree that they index any web site. So don't waste your time with it.


    Oddly googlebot likes link outs If you have a page on your site
    with listings that link out to other sites It actually boosts your ratings.
    googlebot likes links in and out of a site only if the sites linking to are relevant
    to the content of your site.
    i.e. if its a web design site linking to web design related pages helps ranking

    andy
    Last edited by andyshep; 04-01-2005 at 05:21 PM.

  5. #5
    Join Date
    Feb 2005
    Location
    The dark place
    Posts
    91
    You might want to look at using a robots.txt file

  6. #6
    Join Date
    Apr 2005
    Location
    Hastings, Sussex, UK.
    Posts
    244
    Link check (link:clarus-solutions.com) - 0
    Site check (site:clarus-solutions.com) - 1 (Home Page)
    Character check ("+clarus-solutions.com") - 8

    Google has a nasty habit of only scanning the first page of a new domain it finds. Make sure you get some inbounds links and just to make sure Google finds them, run the address of the page your links reside into Googles AddURL page. Unfortunately, you have to give Google a reason to deep crawl you site. A check also found Yahoo has done the same, home page only. MSN has even found you yet. I'd work on the links.

  7. #7
    Join Date
    Apr 2005
    Posts
    16
    A remote possibility is that your pages are banned??? Try looking that up by going to google and typing this in "yoursite.com" no http, etc..
    Google does NOT only scan the first page of a domain it finds.
    If that were true, google would have 0 catalogue entries.. google does not discriminate pages by domain.
    "websites" are determined by linking structure, not by ip/url domain etc.
    --
    Are you making sure that your links make sense?
    for example this is a worthy link
    <a href="/about/index.html" title="about me">About Me</a> Obviously index.html should have appropriate meta and titles etc.. otherwise google has nothing to record.
    a crappy link is
    <a href="/about/inde.html">Read more</a> and then there's nothing in that page for title, etc.

  8. #8
    Join Date
    Apr 2005
    Location
    Hastings, Sussex, UK.
    Posts
    244
    Quote Originally Posted by leocharre
    Google does NOT only scan the first page of a domain it finds.
    So the fact that only the first page of his site has been indexed doesn't point to that fact? Being banned does not mean your home page appears in Google. Anchor text and title attributes do not contribute to the decision Google makes when indexing a new site. They only assist in the ranking process but what good is that if the majority of your pages aren't listed?

    Work on backlinks instead of tweaking internal links on the page as you'll get more immediate benefit. Statistical analysis will force you to amend page copy to suit your market anyway.

  9. #9
    Join Date
    Mar 2005
    Posts
    8

    Re

    Statistical analysis will force you to amend page copy to suit your market anyway.

    What do you mean?

  10. #10
    Join Date
    Apr 2005
    Location
    Hastings, Sussex, UK.
    Posts
    244
    Quote Originally Posted by clarussolutions
    What do you mean?
    Keyword optimisation is a market driven task and will always evolve. You need stats to inform you which keywords people are typing into which search engines so you can amend the primary keyphrases accordingly.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles