Troubleshooting

I'm unable to start screen-scraper in server mode. How can I troubleshoot this?

If you're having trouble starting screen-scraper in server mode or running scraping sessions in server mode run the following command in a batch or shell script as an alternate way to start the server.
jre/bin/java -Xmx128M -jar screen-scraper.jar --start-server --interactive
Running the server in this way does two things to help you troubleshoot your scraping sessions and the program itself.

  1. Bypasses the wrapper.exe program, running the screen-scraper.jar file directly.
  2. Writes out to the console/command window any messages sent to standard out.

You also have two commands you can use within the console/command window:

  1. status - Indicates the number of sessions running as "clients currently connected".
  2. quit - Stops the server.

Is HTML Tidy permanently turned on for basic edition 4.x?

Unfortunately, yes. This is a bug that slipped past our testing prior to the release of version 4.0. Because we do not offer alpha release ("unstable") upgrades in basic edition we are unable to resolve this issue until the next public release, version 5.0. We do not have set schedules for our public releases and can not say when the next release will be.

Why does screen-scraper run slowly on Microsoft® Windows Server 2003?

Though we have not done extensive testing on Microsoft® Windows Server 2003 we have had reports of unusually slow performance. We attribute this to the implementation of Sun's JavaTM code with extra security restrictions. One possible solution would be to install screen-scraper to run under Windows 2000 compatibility mode. Instructions on how to set the compatibility mode during installation can be found here: http://support.microsoft.com/kb/324265.

I receive an error when trying to run screen-scraper from the terminal in Linux. What is an alternate way to run screen-scraper?

As an alternative to executing the screen-scraper binary in Linux you may need to execute a shell script containing the following code. This shell script works only in launching the screen-scraper workbench. To work with screen-scraper in server mode use start_server.sh and stop_server.sh. Execute this shell script from the same location where screen-scraper was installed.

jre/bin/java -Xmx128m -jar screen-scraper.jar

When in server mode one of screen-scraper's processes consumes a lot of virtual memory. How do I stop this from happening?

In your "resourceconfwrapper.conf" file remove this line:

wrapper.java.additional.1=-Xss5M

That's a legacy parameter that is no longer needed by screen-scraper, but may still exist in your instance.

When I export my scraping session I get a zero-byte file. What am I doing wrong?

This is most likely because the character set you're currently using is set to something screen-scraper's file exporter can't deal with. We're working on a fix for this, but in the meantime try changing the "Default character set" in the "Settings" dialog box to "UTF-8".

When I try to install screen-scraper I get a message that reads, "Please select another location to extract the installer to." What am I doing wrong?

This is an issue related to the installer software we use (InstallAnywhere). To remedy the problem, try the following:

  1. Ensure that you have enough hard drive space to contain the decompressed files. You should have at least three times the size of the installer you're using.
  2. Temporarily disable any anti-virus or firewall software.
  3. Ensure that the user account under which you're installing has write access to any temporary folders on the computer. The simplest approach would be to use an account that has administrative access.
  4. Re-download the installer. In some cases the installer file can become corrupted, which can cause this error. Adobe has a good FAQ on dealing with this: http://www.adobe.com/cfusion/knowledgebase/index.cfm?id=f8582407.

When I try to connect to the screen-scraper server from my application it refuses connections. How do I fix this?

First check to ensure that the screen-scraper server is running. Details on doing that can be found here.

This may also be occurring because the IP address of the machine that is connecting to screen-scraper isn't listed in screen-scraper's list of allowed hosts. You can correct this in one of two ways:

  • If the machine running screen-scraper can launch the workbench (e.g., it's running Windows or Linux with Xwindows), you can adjust the security settings by opening the "Settings" window (click the wrench icon), clicking on the "Servers" icon, then entering the IP address (or a portion of the IP address) of the machine you want to allow to connect to screen-scraper in the box labeled "Hosts to allow to connect".
  • If the computer running screen-scraper can't launch the workbench (e.g., it's running Linux without Xwindows installled), you can adjust the security settings by altering the "resource/conf/screen-scraper.properties" file. Add the IP address (or a portion of the IP address) of the machine you want to allow to connect to the "IPAddressesToAllow" property (it's comma-delimited).

After making either of the changes mentioned above, you'll need to restart screen-scraper.

If that still doesn't help, check to ensure that you're trying to connect to screen-scraper using the port on which screen-scraper is listening. The default for the screen-scraper server is 8777, and the default for the SOAP server is 8779. These port numbers can both be altered via the "Settings" dialog box in the workbench (click the wrench icon), under the "Servers" section.

screen-scraper is telling me that it can't bind to certain ports. How do I fix this?

This error is caused by two possible scenarios.

Cause: Port Blocked. In order for screen-scraper to function properly it will need to open a series of local ports on your computer. There are occasions when these ports may be blocked by other software running on your machine, such as firewalls. If screen-scraper is telling you it can't bind to specific ports, you'll either need to free those particular ports up on your machine, or select different ports for screen-scraper to use. To free up the ports you may need to configure a firewall so that it allows for the ports to be bound. You may also need to quit another application that's using the same port (which could even be another instance of screen-scraper running on the same machine). If you'd like to configure screen-scraper to use different ports, see this FAQ.

Cause: Crash. You might also get this error message if the screen-scraper workbench or server crashed, but the database process remains alive. If after the port number in the message it shows "(for the database)", this may be the cause. To remedy this, you'll need to kill the database process manually, then start screen-scraper again. The process to kill will be called "java" on Linux and Mac OS X, and "java.exe" on Windows. If you're running Linux, you likely already know how to kill a process. To kill a process in Windows open the "Windows Task Manager" (hit Ctrl-Shift Escape), click on the "Processes" tab, then kill any "java.exe" processes you know you don't need.

screen-scraper is telling me that my database is corrupted. How do I remedy this?

On rare occasions the main screen-scraper database can become corrupted. This might happen if your computer crashes while screen-scraper is running, for example. Fortunately, as of version 2.8 screen-scraper will automatically back up your database periodically. Even if your database has become corrupted, it's likely you haven't lost much work.

In the directory where screen-scraper is installed (e.g., "C:Program Filesscreen-scraper professional edition"), you'll find the following directory path: "resourcedbbackup". This "backup" folder should contain a series of folders with dates and times, each of which will contain a backup of your database. You'll use these to restore your database, by following the steps below:

  1. Ensure that screen-scraper is not running. This would include the workbench, server, and any command line instances you might have running.
  2. Kill any java.exe, javaw.exe, java, or javaw processes running on your machine that might correspond to screen-scraper. It's possible that the screen-scraper database process could still be alive, which would lock the database files. You can kill processes on Windows using the Windows Taks Manager.
  3. Delete your existing database files. These are located in "resourcedb", and all begin with "ss".
  4. Copy the database files from the most recent backup folder into the "resourcedb" folder. For example, you might copy all of the files beginning with "ss" from "resourcedbbackupSeptember 8, 2006 09.23.31 AM" into "resourcedb".
  5. Try launching screen-scraper. If everything is normal, you're done. If you get the same "Database Corrupted" message, go back to step 1.
Syndicate content