PDA

View Full Version : Webcrawler



PCT Joe
26-01-2010, 03:44 PM
Hello all, I currently have disconnected the internet and am at the local libary, I'm wondering if anyone knows of a webcrawler that catalogs pages

for instance i just downloaded all of playtech.co.nz by ctrl + s'ing the whole way. is there anything that could do this automaticaly so i can still access the pricing at home and update it automatically.

Thanks,
Joe

Sweep
26-01-2010, 03:46 PM
There was one around called Webreaper as I remember it.

Still around too:-

http://www.webreaper.net/

Erayd
26-01-2010, 04:20 PM
Wget can do it (--mirror). I suggest you ask the site operator for permission first though - mirroring an entire site without so much as a by-your-leave is rather rude (and sometimes illegal, depending on your location).

bevy121
26-01-2010, 05:04 PM
yea - plus there's one or two others (http://www.google.co.nz/#hl=en&safe=off&ei=ZWleS5TWLIvk7AO0wOXBAg&sa=X&oi=spell&resnum=0&ct=result&cd=1&ved=0CAYQBSgA&q=website+downloader&spell=1&fp=d21ac5318dc167b) that do it as well lol

annie
26-01-2010, 05:37 PM
I like HTTRACK. But I usually get permission first, or download just a very small subset of a site.

I also usually babysit it, and abort if it's downloading more than a few meg.

PCT Joe
27-01-2010, 11:29 AM
Thanks, but im still having problems as they download and name it category.php and since all are the same well... yeah...

anyways ill try get in contact with playtech.