Interacting with Screen-Scraper Externally

Invoking screen-scraper from the command line

If you've decided to use the basic edition of screen-scraper your only option for invoking screen-scraper externally is from the command line (invoking screen-scraper from the command line is also available in the professional and enterprise editions). You can find full documentation and examples on doing that at our Invoking screen-scraper from the command line documentation page. If you don't need to invoke screen-scraper from the command line you can skip to the Invoking screen-scraper from an external application section.

In order to invoke screen-scraper from the command line, you'll want to create a batch file (in Windows) or a shell script (in Linux or Mac OS X) to invoke the scraping session. If you're using Windows open a text editor (e.g., Notepad) and enter the following:

Save the batch file (call it "hello_world.bat") in the folder where screen-scraper is installed (e.g., C:\Program Files\screen-scraper professional edition\). Vista users, you will need to save your batch file to a location such as your Documents folder or your Desktop. Then, within Windows Explorer, manually transfer the file to the directory where screen-scraper is installed.

Within screen-scraper, you'll want to disable the "Initialize scraping session" script; otherwise, the value we pass in from the command line would get overwritten once that script is executed. Disable the script by clicking on the "Hello World" scraping session, then on the "Scripts" tab, then un-checking the "Enabled?" check box for the script.

You can then run the batch file by opening a DOS prompt, changing to the folder containing the batch file, then invoking it. You should see the text from screen-scraper's log appear in the DOS window. If you're running Linux or Mac OS X, you'll need to close the workbench before invoking your shell script.

Invoking screen-scraper from an external application

Note that the rest of this tutorial only applies to the professional and enterprise editions of screen-scraper.

Oftentimes you'll want to use a language or platform external to screen-scraper to scrape data. screen-scraper can be controlled externally using Java, PHP, Ruby, Python, .NET, ColdFusion, any COM-friendly language (such as Active Server Pages or Visual Basic), or any language that supports SOAP. In this next part of the tutorial we'll give examples in PHP, Java, ColdFusion, and Active Server Pages.

In order to interact with screen-scraper externally it needs to be running as a server. When running as a server screen-scraper acts much like a database server does. That is, it listens for requests from external sources, services those requests, and sends back responses. For example, when you issue a SQL statement to a database from an ASP script your script is opening up a socket to the database, sending the request over it, then receiving the database's response back over the socket. Once this transaction has been completed the socket will be closed, but the database will continue to listen for other requests. screen-scraper works in a similar way.

At this point we'd recommend reading over the documentation page that discusses running screen-scraper as a server, and gives details on how to start and stop it according to the platform you're running on. Follow the link below, then return back to this page when you're finished:

Running screen-scraper as a server

Before we start writing code to interact with screen-scraper externally we need to configure a few things. Depending on the language you'd like to program in, please follow one of the links below, which will give you an overview of interacting with screen-scraper using that language and guide you through any configuration that needs to take place. Once you're finished return back to this page.

Invoking screen-scraper from ColdFusion

Invoking screen-scraper from a COM-based application

Invoking screen-scraper from Java

Invoking screen-scraper from PHP

Each time you run a scraping session externally screen-scraper will generate a log file corresponding to that scraping session in the "log" folder found inside the folder where you installed screen-scraper. This can be invaluable for debugging, so you'll want to take a look at it if you run into trouble. You can turn server logging off by unchecking the "Generate log files" check box under the "Servers" section of the "Settings" dialog box.

If you haven't already, within screen-scraper, you'll want to disable the "Initialize scraping session" script; otherwise, the value we pass in from our external application would get overwritten once that script is executed. Disable the script by clicking on the "Hello World" scraping session, then on the "Scripts" tab, then un-checking the "Enabled?" check box for the script.

OK, we're now ready to write some code. Follow one of the links below.