Running screen-scraper as a Server

In order to invoke screen-scraper from something like a Visual Basic application or a PHP script screen-scraper needs to be run in server mode. When running screen-scraper in server mode it acts, in many respects, like a database server would. Interacting with screen-scraper via one of the "remote scraping sessions" is more or less analogous to querying a database via a database driver. Objects like scrapeable files and scripts set up in screen-scraper provide access to information much like tables and columns in a database would. Issuing a SQL query to a database is analogous to setting session variables and telling screen-scraper to initiate a named scraping session.

If you're running Windows NT, 2000, XP, or 2003, screen-scraper will run as a service. This allows it to be run in a background process, and doesn't require you to be logged in to the machine on which it's running. screen-scraper gets registered as a service upon installation, and may be run as a server using either the "Start server" and "Stop server" links from the "Start" menu, or using the "Services" control panel applet. When the server is running an icon will appear in the system tray, except in Windows Vista (see the next paragraph for a bit more on this). You can right-click this icon to stop the server.

On Windows Vista, the screen-scraper service can be started, stopped, and monitored via the "Services" control panel applet (under "Administrative Tools). The server can also be started and stopped via the "Start server" and "Stop server" shortcuts found under the "Start" menu. When the server is running, however, the system tray icon will not appear, as it does in other versions of Windows. In order to determine whether or not the server is running on Vista, you can use the following:

• The "Services" control panel applet
• Via a DOS prompt (open a DOS prompt, cd to the directory where screen-scraper is installed, and type "server /status")
• If you're using the Enterprise Edition, you can open a web browser and type in the URL for the web interface.

Under Unix/Linux or Mac OS X the server is controlled via the "server" script, which operates much like a typical Unix daemon. The server will run in a background process, allowing you to start and stop the server remotely, or log out of your session after starting it. You can issue the following commands to the "server" script:

• server start
• server stop
• server status

When running as a server any messages screen-scraper produces will be written to the "error.log" file, found in the "log" directory.

Each time you run a scraping session externally screen-scraper will generate a log file corresponding to that scraping session in the "log" folder found inside the folder where you installed screen-scraper. This can be invaluable for debugging, so you'll want to take a look at it if you run into trouble. You can turn server logging off by unchecking the "Generate log files" check box under the "Servers" section of the "Settings" dialog box. Within a scraping session script, the name of the log file can be found in the session variable "SS_LOG_FILE_NAME".

Once the server is running it will be listening for connections. The default port the server will listen on is 8778, which can be changed by altering the "Port" parameter found under the "Servers" section of the "Settings" dialog box. At this point clients can open sockets to the server and issue commands using a special protocol. Not to worry, you don't need to use the protocol directly; we take care of that part for you.

While the server is running scraping sessions and scripts can be imported into screen-scraper without stopping it. See the documentation on importing and exporting objects for more information.

IMPORTANT NOTE: While screen-scraper is running as a server it will accept connections from any other machine unless you specify otherwise. It is important to consider the security of your machine when running any service of this type. screen-scraper allows you to specify the IP addresses of the machine(s) you wish to allow to connect to it from the "Settings" window. After opening screen-scraper select "Settings" from the "Options" menu, then click on the "Server" tab. In the box labelled "IP addresses to allow to connect" enter a comma-delimited list of IP addresses that screen-scraper should accept connections from. Note that you can also specify just the beginning portions of IP addresses as well. For example, if you enter 111.22.333 screen-scraper would accept connections from 111.22.333.1, 111.22.333.2, 111.22.333.3, etc. Please note also that if nothing is entered into the text box screen-scraper will accept connections from any IP address. If you need to alter this setting in a GUI-less environment, you can close screen-scraper and edit the "resource/conf/screen-scraper.properties" file instead. The setting to change is "IPAddressesToAllow". When you start screen-scraper back up, it will make use of the new setting.

Troubleshooting Server Mode. If you're having trouble starting screen-scraper in server mode or running scraping sessions in server mode please see our FAQ on how to go about troubleshooting this issue.