Proxy Server

Description

When running, the proxy server listens on a specified port for incoming HTTP requests from your web browser. Upon receiving a request from your browser the proxy server records it, then sends it along to the server for which it was intended. When that server responds it is received by the proxy server, which, once again, makes a record of it, then sends it along to your web browser.

Purpose

screen-scraper's proxy server allows you to view HTTP requests and responses as they pass between your web browser and remote servers. In scraping files from web sites there are a few more details than you typically worry about when surfing, such as HTTP headers and POST data. The proxy server makes all of these details visible to you.

Viewing HTTPS requests

Often one of the headaches of scraping information from sites that use HTTPS is that it's not always easy to tell what's getting passed back and forth in the way of cookies, POST data, etc. Even if you put a proxy server in the way that lets you view the requests and responses, the information is encrypted as it's leaving your browser and as it's leaving the web server that responds to the request. screen-scraper gets around this problem by using it's own temporary certificate to encrypt traffic from itself to the browser and then encrypting each request before sending it up to the server. The result of this is that your browser will issue a warning about the certificate that screen-scraper returned. You can safely accept the certificate and be assured that all your traffic is encrypted.

We've also used other proxy software, such as Charles proxy, for handling SSL sites. They have additional features to allow the browser to trust the certificates so you don't see a warning in the browser. We've also added an import proxy session feature so you can import a JSON Session File from Charles and use it in screen-scraper to build a scraping session.

Running the proxy in server mode

This feature is only available to Professional and Enterprise editions of screen-scraper.

screen-scraper has the ability to act as a proxy while in server mode. Combined with the ability to execute scripts, this functionality opens up many possibilities for how you use screen-scraper. More information about how to go about using screen-scraper in this capacity is available on our using scripts with the proxy server page.