PDA

View Full Version : Automatic Webpage Capture



bot
16-05-2011, 06:15 PM
Is there software that will, say, capture a webpage or an element of a webpage every 10 minutes or so?

kahawai chaser
16-05-2011, 07:53 PM
What do you want to capture? Screen Grabs or do you want notification of any changes/updates in content? If content you can use something like Google alerts which emails changes in content. I know there are more advanced similar software for more regular updates, with more filters, etc, but require signing up. Unless some advanced scripting can be used. I used something similar a while back - but Google will block or warn of automated requests. Then can't search for a while.

I used to use Versionista which tracks changes, and I think sent out alerts. But requires online sign up.

bot
16-05-2011, 09:08 PM
What do you want to capture? Screen Grabs or do you want notification of any changes/updates in content? If content you can use something like Google alerts which emails changes in content. I know there are more advanced similar software for more regular updates, with more filters, etc, but require signing up. Unless some advanced scripting can be used. I used something similar a while back - but Google will block or warn of automated requests. Then can't search for a while.

I used to use Versionista which tracks changes, and I think sent out alerts. But requires online sign up.

I want a webpage to be refreshed and saved every 10 minutes. The reason for this is that I can't stay up that late!

PS it's for statistical reasons

wainuitech
16-05-2011, 09:54 PM
Theres a few suggestions and links to programs Listed here (http://www.elitetrader.com/vb/printthread.php?threadid=218439) - no idea if they work or are any good --- Give some a try and let everyone know :p:D

Edited: Try this in IrfanView ( if you want to use that) open IrfanView, options/Screen capture - looks like theres a timer option Shown here (http://www.thebest3d.com/iview/Irfanview-tutorial-how-to-capture-timelapse-and-make-movie.html) as well as how to turn it into a movie if you want.

Erayd
16-05-2011, 09:57 PM
Wget plus cron is ideal for this task.

Depending on what you need and how long you need it for, I may be happy to run a job on one of my servers and send you the result - PM me if you're interested.

bot
16-05-2011, 10:51 PM
Theres a few suggestions and links to programs Listed here (http://www.elitetrader.com/vb/printthread.php?threadid=218439) - no idea if they work or are any good --- Give some a try and let everyone know :p:D

Edited: Try this in IrfanView ( if you want to use that) open IrfanView, options/Screen capture - looks like theres a timer option Shown here (http://www.thebest3d.com/iview/Irfanview-tutorial-how-to-capture-timelapse-and-make-movie.html) as well as how to turn it into a movie if you want.

I'll give IrfanView a crack - the page seems to refresh every minute.

I'd prefer something that actually captured data - as I want to copy and paste it into a document.

Perhaps if there was something that automatically went "Ctrl-S" in Chrome, waited, say, 10 secs, then went "Enter". Something like a macro program would do the trick.

Although Chrome would have to be in focus...

BTW, thanks for all you guys help so far :)

sal
16-05-2011, 10:58 PM
@bot: if you're not familiar with wget, you should take Erayd up on his offer because it's exactly what you're looking for, a snapshot every X minutes (source and images) and not just a screenshot image.

Erayd
16-05-2011, 11:57 PM
@bot: if you're not familiar with wget, you should take Erayd up on his offer because it's exactly what you're looking for, a snapshot every X minutes (source and images) and not just a screenshot image.Welcome back Sal :D - long time, no see!

sal
16-05-2011, 11:59 PM
It's probably a bit late, but here's a AutoHotkey (www.autohotkey.com/download/) script that automates the job of saving and refreshing a webpage every 10 mins:


Var = 1
Loop {
Sleep, 100000
Send ^s
Sleep, 100000
Send Saved_webpage_v%Var%{Enter}
Sleep, 400000
Send ^r
Var += 1
}

Here's a couple I compiled earlier:

Save & refresh page every 1 minute -> SarpeOM.exe (http://b.spare.co.nz/get/SarpeOM.exe) (267KB)
Save & refresh page every 10 minutes -> SarpeTM.exe (http://b.spare.co.nz/get/SarpeTM.exe) (267KB)

Those are clean and legit executables, they only do what it says on the box (see code above).

First go to the webpage you want to save and do one run through "Ctrl+s" so it's saving to the right folder, run the ".exe" (it'll run from the tray) and then give focus to the relevant browser tab, and sorted.

bot
17-05-2011, 12:12 AM
It's probably a bit late, but here's a AutoHotkey (www.autohotkey.com/download/) script that automates the job of saving and refreshing a webpage every 10 mins:


Var = 1
Loop {
Sleep, 100000
Send ^s
Sleep, 100000
Send Saved_webpage_v%Var%{Enter}
Sleep, 400000
Send ^r
Var += 1
}

Here's a couple I compiled earlier:

Save & refresh page every 1 minute -> SarpeOM.exe (http://b.spare.co.nz/get/SarpeOM.exe) (267KB)
Save & refresh page every 10 minutes -> SarpeTM.exe (http://b.spare.co.nz/get/SarpeTM.exe) (267KB)

Those are clean and legit executables, they only do what it says on the box (see code above).

First go to the webpage you want to save and do one run through "Ctrl+s" so it's saving to the right folder, run the ".exe" (it'll run from the tray) and then give focus to the relevant browser tab, and sorted.

Thanks so much, this is just what I needed :)

:cookie:

sal
17-05-2011, 12:22 AM
Welcome back sal :D - long time, no see!
Hey man, I've been around...a whole lotta lurking.