Alpha Documentation
 |
Alpha Documentation |
We're constantly updating screen-scraper, but we only periodically make public/stable releases. The alpha releases often contain useful features and even bug fixes, though, so you might consider upgrading (click here for help on doing that). Because these releases haven't been tested as thoroughly as our public releases, though, use them at your own risk. You'll want to be sure to back up your work periodically just in case something breaks in an alpha release. All of that said, internally we all run the very latest alpha version, so we're careful to make sure things work. Occasionally a relatively major bug will sneak in, but we're always quick to fix it once it's noticed.
If you do decide to use the latest alpha version we'd very much appreciate feedback and bug reports. The best way to report these would probably be via our support form.
Also, because this is alpha documentation the docs may be a little scattered and inconsistent. Once we do a public release we'll clean them up, but you likely won't find them to be nearly as neat and tidy as the others.
CSV Writer
CSV Writer
We've created a slick new CSV writer for screen-scraper! You can find docs on it here.
Comparing scrapeable file requests with proxy requests
|
|
Comparing scrapeable file requests with proxy requests |
At times in developing a scraping session a particular scrapeable file may not be giving you the results you're expecting. Even if you generated it from a proxy session parameters or cookies may be different enough that the response from the server is very different or even an error than what you were anticipating. Generally in cases like this the best approach is to compare the request produced by the scrapeable file in the running scraping session with the request produced by your browser in the proxy session. That is, ideally your scraping session mimics as closely as possible what your web browser does.
To facilitate this, you can programatically compare the last request from a scrapeable file with a corresponding request from a proxy transaction. To do this, run your scraping session so that the scrapeable file of interest contains a request under the "Last Request" tab. Now click the "Compare with proxy transaction..." button. If this scrapeable file was generated from a proxy transaction screen-scraper will remember that and ask if you want to compare it with that particular transaction. If not, after you click the button it will prompt you to click on the transaction you'd like to compare it to. Do this by clicking on the proxy session containing the transactions, go to the "Progress" tab, then click on the transaction in the "HTTP Transactions" table.
At this point a window will appear that allows you to compare various elements from the two requests. Differences that may be significant are highlighted. Here you can determine if something like a POST parameter is different, or perhaps a cookie is missing that was set in the browser via JavaScript.
REST API
|
|
REST API |
screen-scraper's REST API was released in version 4.5.18a, and is documented in this blog posting.
Updating screen-scraper in a GUI-less environment
|
|
Updating screen-scraper in a GUI-less environment |
Often times screen-scraper will be running on a server that has no graphical interface. Updating to the latest version in such an environment previously required multiple steps, but can now be done with a simple Python script.
Any Unix-based computer worth its salt will already have python installed. To use the updater, open a terminal and navigate to the screen-scraper install directory. Ensure that screen-scraper is not currently running (via "./server status"). After that, issue this command to update to the latest version:
python ss_updater.py
If you want to force screen-scraper to upgrade to the latest unstable version, use this command:
python ss_updater.py -u
logVariables
| session.logVariables() (professional and enterprise editions only) |
When the session.logVariables() method is called the current session variables will be output to the log. Also, if screen-scraper is running in server mode and session.breakpoint() is called the session.logVariables() method is invoked instead.
|
