dcsimg
www.webdeveloper.com
Results 1 to 5 of 5

Thread: [RESOLVED] htaccess Block User Agent

  1. #1
    Join Date
    Dec 2008
    Posts
    148

    resolved [RESOLVED] htaccess Block User Agent

    Hi, I have the below mod rewrite to block unwanted bot. But somehow it is also blocking google and bing iPhone bots. I need help to make the rule more restricted so it doesn't catch user agent that I don't want to block. NOTE: It doesn't block the regular googlebot and bingbot just the iPhone version. Any help will be appreciated. Thx. Complete list of bot is in attachment (too many characters)

    Attachment 16613

    Code:
    RewriteEngine on
     RewriteCond %{HTTP_USER_AGENT} A6\-Indexer [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} abby [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} Abrave [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} Accelobot [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} Access [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} Accoona [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} AcoonBot [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} AdvBot [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} adobeair [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} AdsBot\-Google [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} Agobot [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} AhrefsBot [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} aipbot [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} akula [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} alexa [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} AllSubmitter [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} Apexoo [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} AportWorm [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} AraBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ZyBorg [NC]
    RewriteRule !^403\.shtml$ - [F]

  2. #2
    Join Date
    Mar 2012
    Posts
    2,767
    I note that the full list contains:

    Code:
     RewriteCond %{HTTP_USER_AGENT} Googlebot\-Image [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} Googlebot\-News [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} Googlebot\-Video [NC,OR]
     ...
     RewriteCond %{HTTP_USER_AGENT} msnbot\-media [NC,OR]
    Could these be affecting the bots? It also contains:
    Code:
     RewriteCond %{HTTP_USER_AGENT} facebookexternalhit [NC,OR]
     RewriteCond %{HTTP_USER_AGENT} facebookplatform [NC,OR]
    Just a thought...

  3. #3
    Join Date
    Dec 2008
    Posts
    148
    I don't think it is the first block issue because the regular google bot and bing bot was not block when they came at the same time. Why will the facebook bot affect the bots? I can try removing that and see what happen. I have the same bots in my list before and it all works. I mainly need someone to look at the rules I have applied (ie. the [NC,OR] or the RewriteRule !^403\.shtml$ - [F]. Is there a way to make it more restricted so it only blocks those specific bots. thx.

  4. #4
    Join Date
    Oct 2013
    Posts
    976
    Why are you banning bots from Google and Bing with .htaccess? They obey robots.txt; use that instead.
    Last edited by Kevin2; 03-11-2015 at 01:51 PM. Reason: cApItAlIzAtIoN

  5. #5
    Join Date
    Dec 2008
    Posts
    148
    I found my problem it was because there is web in the blocklist.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles