Proxy Server Overview

Purpose

screen-scraper's proxy server allows you to view HTTP requests and responses as they pass between your web browser and remote servers. In scraping files from web sites there are a few more details than you typically worry about when surfing, such as HTTP headers and POST data. The proxy server makes all of these details visible to you.

Description

When running, the proxy server listens on a specified port for incoming HTTP requests from your web browser. Upon receiving a request the proxy server records it, then sends it along to the server it was intended for. When that server responds the response is sent first to the proxy server, which, once again, makes a record of it, then sends it along to your web browser.

Viewing HTTPS requests

Often one of the headaches of scraping information from sites that use HTTPS is that it's not always easy to tell what's getting passed back and forth in the way of cookies, POST data, etc. Even if you put a proxy server in the way that lets you view the requests and responses, the information is encrypted as it's leaving your browser and as it's leaving the web server that responds to the request. screen-scraper gets around this problem by using it's own temporary certificate to ecrypt traffic from itself to the browser and then encrypting each request before sending it up to the server. The result of this is that your browser will issue a warning about the certificate that screen-scraper returned. You can safely accept the certificate and be assured that all your traffic is encrypted.

Running the proxy in server mode

screen-scraper has the ability to act as a proxy while in server mode. Combined with the ability to execute scripts, this new functionality opens up many new possibilities for how you use screen-scraper, including setting up blacklists, application integration and many more. To learn how to configure screen-scraper for this see the settings documentation and look for "Default proxy session to use when running in server mode" .

From here:

• Using the proxy server