Proxy Server Setup

What is a Proxy Server?

An HTTP proxy server is basically just a program that sits in between a web browser and a web server, passing information between each. The screen-scraper proxy server records and allows you to view all requests that your web browser sends, and the corresponding responses that web servers send in return. This is done to facilitate the recreation of these steps when your scrapes gather information.

Start screen-scraper

To begin you will first need to open up screen-scraper:

On Unix/Linux or Mac OS X use the screen-scraper link that was created when you installed screen-scraper.

If you're running Windows, this is done by selecting the appropriate link from the Start menu.

Once screen-scraper has loaded you'll see the objects tree on the left (it likely only contains the Root Folder currently). Let's create our first object.

Creating a Proxy Session

A proxy session is a screen-scraper object that allows you to interact with the proxy server and its records. You'll generally use one each time you create a new scrape.

Create a proxy session now by clicking the (Add a new proxy session) icon or by selecting New Proxy Session from the File menu (you could also right-click on the Root Folder for a similar menu). Once you have added the proxy session, you screen should look similar to the following screenshot.

Configuring the Proxy Session

Give the proxy session the name Hello World by typing it into the Name field. The Port field determines the port number that your web browser will use when communicating with screen-scraper's proxy server. Leave the port at 8777.

Configure Browser to Use Proxy Session

Next, we need to set up your web browser so that it will use screen-scraper as a proxy server.

If you have multiple web browsers installed on your computer we recommend using one of them to continue through the tutorial and the other to interact with the proxy server. For example, if you have Internet Explorer and Firefox installed you may want to view the tutorial pages using Firefox and use Internet Explorer with the proxy server.

For the sake of this tutorial, we will use Internet Explorer as our browser that interacts with the proxy server. If you chose a different web browser, information on how to configure it can be found in our documentation: Firefox or Opera.

For the sake of this tutorial we will be using Internet Explorer as the browser that is interacting with the proxy server. How to configure Internet Explorer is described below.

1. Open Internet Explorer.
2. Click Internet Options in the Tools menu.
   
3. Go to the Connections tab.
4. Click on LAN Settings.
5. Click on the checkbox beginning with Use a proxy server for....
6. Click on the Advanced... button.
   
7. In the HTTP and Secure fields type localhost under the Proxy address to use column, and 8777 under Port
   

   If you have changed your proxy server settings to use a port other than 8777 then type your selected port in place of 8777.

8. Hit the OK button a few times till you get back to your web browser.

If you're using a dial-up connection the setup will differ slightly. Instead of the LAN Settings button you'll want to find your dial-up connection under the Dial-up and Virtual Private Network settings dialog box, then configure it via the Settings button.

Depending on your operating system, instead of localhost you may need to use either 127.0.0.1 or the IP address of the machine. If you have trouble connecting to screen-scraper's proxy with your web browser, please see this FAQ.

At this point your browser is set up such that any time you click on a link or submit a form the request will first go to screen-scraper, where it will be recorded, and then get sent to the web server it was intended for. The web server will respond back to screen-scraper, which will record the response, then send it along to your web browser.

If you're running Mac OS X, and are using screen-scraper Professional or Enterprise Edition, there's one more step you'll need to take: In screen-scraper, click the (settings) icon to bring up the Servers button in the panel on the left, then remove any text contained in the Hosts to allow to connect text box. Because of the way Mac OS X handles IP addresses, we do this so that screen-scraper will accept connections from your web browser.

Starting the Proxy Server

At this point we can get the proxy server running. Do this now in screen-scraper by clicking on the Start Proxy Server button in your proxy session. After this, click on the Progress tab, which will display all of the requests and responses recorded by the proxy server.

You're now ready to have screen-scraper record a few pages for you.