Managing multiple scraping seasons
This is a suggestion for a gui that would allow you to start and manage multiple scraping seasons simultaneously.
It should have the following features:
- Display which scraping seasons that are currently running, not started yet, completed successfully and failed.
- Able to browse the log and display the results of each season.
- Save (and restore) which scraping seasons in the manager.
- Save current state of all scrapings (so you can continue later or do edits or retry failed scrapings).
I know you can do a lot of things with scripts, but I however would like like a better graphical solution when running a lot of scraping seasons (10 or more) at the same time.
Managing multiple scraping seasons
Eon,
Thank you for your feedback. The Web interface is definitively still under development and your feedback is much appreciated.
Thanks,
Scott
Managing multiple scraping seasons
I've been testing the web interface, and it quite frankly almost exactly what I was looking for.
Selecting and running both single and multiple scraping seasons works fine, and the ability to review the logs for each scraping season afterwards is just perfect.
The schedule function is a very good idea, however I can't get it to work. Setting it up is no problem, but it just don't seem to run when the time comes.
Also the error function doesn't seem to be working. It always states the season have completed without error when it failed one or more patterns, and even when it was unable to retrive the site at all.
That was what I was able to test at this time. My trial have now expired, so I'm back to using the basic edition for the time being.
Managing multiple scraping seasons
Eon,
That's good news. We're still looking into the systray icon issue.
Tell us what you think of the web interface. Any suggestions are appreciated.
Thanks,
Scott
Managing multiple scraping seasons
Got it to work now, thanks.
I'm using vista, but I get no systray icon as you mention (have expanded the icons, it's not there), however this isn't important at all, just want to let you know.
Managing multiple scraping seasons
Eon,
Upgrading to any of the newest Alphas is good.
A couple of questions...
- Are you using Vista? If so, you'll need to use your current IP address in place of localhost. If you connect via a router it would the IP address the DHCP server assigned to you.
- When you try to browse http://localhost:8779/web.htm do you get your browser's standard "Page can not display" message or is it a message from Apache Tomcat?
- If you get your browser's normal "Page can not display" message then I've got a few further questions...
- Did you start the screen-scraper server?
- If so you'll want to confirm that the server is running by one of the following.
-- You see the screen-scraper logo (or sometimes a red/orange rocket icon) in the system tray (lower-right corner of your screen).
-- Click on your Start menu and choose Run. In the text box type: services.msc. This opens the Services console window. In the right-hand pane in the list of services you should find one named screen-scraper. Under the Status column it should say Started.
Please let us know what you find.
Thanks,
Scott
Managing multiple scraping seasons
I didn't get this to work. I'm using the professional trial version and I downloaded the latest alpha 3.0.42a (it's not the version you mention, could this be the problem?). However I can't connect to http://localhost:8779/web.htm (yes, I'm using the standard settings and I checked that it's set to port 8779). All firewall software disabled also.
Managing multiple scraping seasons
Eon,
We read you're mind...or you're currently reading ours ;)
Look no further than the screen-scraper Web interface. It does much of what you described. It's not fully released because it's still very much in development and doesn't look too pretty. But, it's available for the professional edition starting at 3.0.14a.
If you're not currently running the professional edition I recommend that you download the 30 day trial, install it (if you're running basic, as well, you'll need to follow some instructions [url=http://screen-scraper.com/support/faq/faq.php#MultipleInstances]here[/url] here), update the latest alpha ([url=http://screen-scraper.com/support/faq/faq.php#NoUpdates]help here[/url]), then type one of the following URLs into you're Web browser to voila!
Only one screen-scraper installed on your computer:
http://localhost:8779/web.htm
More than one instance of screen-scraper installed and you followed [url=http://screen-scraper.com/support/faq/faq.php#NoUpdates]instructions[/url] for installation on the FAQ:
http://localhost:8458/web.htm
And if you'd like to peruse the guts of it they can be found here:
~#ss pro edition install dir#~\resource\lws\webapps\ROOT\
See our brief [url=http://blog.screen-scraper.com/2007/04/03/version-3014a-of-screen-scraper-available/]blog entry[/url] on its inception.
Any feedback is appreciated.
Hope you enjoy,
Scott