Page 1 of 2 12 LastLast
Results 1 to 10 of 17
  1. #1
    james
    Guest

    Default Spiders from HELL

    Sorry for the brown-outs recently but, as usual, it was the fault of badly tuned spiders. I know, I know - the same ol' pressf1 technical excuse. But we've blocked them from access so things should speed up.

  2. #2
    stu140103
    Guest

    Default Re: Spiders from HELL

    Good work james

  3. #3
    Hun Hunter Biggles's Avatar
    Join Date
    May 1988
    Location
    In the air
    Posts
    2,732

    Default Re: Spiders from HELL

    It may take a while for the blocking to kick in - spiders don't necessarily check robots.txt each visit.

  4. #4
    -=JM=-
    Guest

    Default Re: Spiders from HELL

    Thanks for letting us know people. It's always good to know why it is that it has been down.

  5. #5
    Chris
    Guest

    Default Re: Spiders from HELL

    Is it not possible to block the IP that the bad naughty spiders use????

    .Chris

  6. #6
    stu140103
    Guest

    Default Re: Spiders from HELL

    > Is it not possible to block the IP that the bad
    > naughty spiders use????

    Yeath, but different search engine use different bots & It will take a wee while to block them all, & some times search engine get new bots on new IP address etc…

  7. #7
    cicero
    Guest

    Default Re: Spiders from HELL

    Thought I had found an entomology forum for a minute.

  8. #8
    oldskool biatch
    Guest

    Default Re: Spiders from HELL

    > Yeath, but different search engine use different bots

    why do you spell Yes as Yeath?? do you lithp??

  9. #9
    Chris
    Guest

    Default Re: Spiders from HELL

    > Yeath, but different search engine use different bots
    > & It will take a wee while to block them all, & some
    > times search engine get new bots on new IP address
    > etc…

    Surely blocking the IPs of KNOWN naughty bad spiders will lead to a reduction of this problem, in much the same way that blocking known spammer domains, such as AOL MSN etc has halved the spam in my inbox.

    .Chris

  10. #10
    james
    Guest

    Default Re: Spiders from HELL

    Both the robots.txt has been changed and the specific "rogue" spider has been blocked at our firewall.

Similar Threads

  1. Replies: 89
    Last Post: 06-04-2007, 04:53 PM
  2. Yahoo! Slurp Spiders?
    By SurferJoe46 in forum PC World Chat
    Replies: 25
    Last Post: 18-11-2006, 05:37 AM
  3. Bots & Spiders
    By agent in forum PC World Chat
    Replies: 4
    Last Post: 12-01-2005, 01:51 PM
  4. OT: Camel Spiders... Are these real?!?!?
    By hamstar in forum PressF1
    Replies: 10
    Last Post: 19-04-2004, 08:46 AM
  5. Card game Spiders
    By in forum PressF1
    Replies: 1
    Last Post: 13-02-2001, 08:38 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •