Save scrape log feature
I want be able to save x lines to a log that gets overwritten or appended to. Sometimes my Net connection will die out unexpectedly in a long scrape (35000 pages), and it'd be nice to restart the scrape from the point where it died. Right now I'm writing out my own "log", but it'd just be easier. :)
Save scrape log feature
It sounds like you may be wanting to automate the restarting of screen-scraper, reinstantiating the last session that was running, and use variables that were collected just prior to you loosing your Internet connection when your connection comes back up. At certain points in the looping of your session you could interject a script that would utilize the method wasErrorOnRequest to check for an error response. When you received an error response you could then tell screen-scraper to pause for the time it usually takes for your connection to come back up again.
http://community.screen-scraper.com/API/wasErrorOnRequest
When your connection comes back up you will have lost your session with the Web server you're connecting to. So, if your screen-scraper session requires you to maintain a session with the Web server you will need to restart your screen-scraper session at the place you established your Web session.
Please let me know if you have additional questions.
-Scott
Save scrape log feature
I use Live HTTP Headers for most sites now.
Thank you Scott!