Screen-Scraping Overview

In many ways working with screen-scraper is like working with a database, such as mySQL or SQL Server. With databases, you'll generally use an interface (often a graphical user interface) to create objects such as tables, columns, and indexes. Once you've set up the database you'll often write programming code to populate it with data as well as to pull information from it. Likewise with screen-scraper you'll use its graphical user interface to create objects needed to extract information from web sites. Once you've set up these objects you'll write programming code to interact with screen-scraper and make use of the data it extracts.

Extracting information from web sites using screen-scraper typically involves four main steps:

1. Use the proxy server to determine the exact files that need to be requested in order to get the information you're after.
2. Create a scraping session with scrapeable files that define the sequence of pages screen-scraper will request.
3. Generate extractor patterns to define the exact information you need screen-scraper to grab from each page.
4. Write small scripts or programming code to invoke screen-scraper and/or work with the data it extracts. If you don't do much programming, don't worry. Generally the scripts you'll need to write to work with screen-scraper are small and simple, and you can usually just modify the example scripts we provide.

We'll now walk through each of these steps in detail.