Using Scripts with the Proxy Server

Overview

screen-scraper has the ability to run custom made scripts while the proxy server is running. This allows you to harness the full power of the scripting environment like you can in scraping sessions. It is recomended that you read using scripts before continuing since many of the concepts apply to invoking scripts in the proxy server environment.

Using the scripts

Scripts are added to a proxy session by selecting proxy session in the tree view then selecting the "Scripts" and clicking on the "Add Script" button. You will notice that a script will then be added to the scripts table. You will need to click on the script name and select the script that you want to run. The options "Sequence", "When to Run" and "Enabled" function similarly to other places in screen-scraper where scripts can be invoked. In the proxy server environment the "When to Run" options specify when in the proxy cycle the script will be invoked. Depending on when you decide to run your script certain built in objects will be in scope that are unique to the proxy environment.

Built-in objects

screen-scraper offers a few objects that you can work with in a script in the proxy environment. See the "Variable scope" section (following this one) for more details.

• proxySession. This variable allows for interaction with the currently running proxy session. It has the following methods:
  • getVariable( String identifier ). Retrieves the value of a saved proxy session variable designated by identifier.
    example: cityCode = proxySession.getVariable( "CITY_CODE" );
  • setVariable( String identifier, Object value ). Designates that value should
    be saved for the duration of the proxy session, and can be accessed using the getVariable
    method using identifier.
    example: proxySession.setVariable( "CITY_CODE", dataSet.get( 0, "CITY_CODE" ) );
  • log( String message ). Causes message to be writen to the "Log" panel for the
    currently running proxy session.
    example: proxySession.log( "Inserting request parameters into the database." );
• request. This variable allows for interaction with the currently received HTTP request.
  It has the following methods:
  • addPOSTParameter( String key, String value ). Adds a POST parameter to the request.
    example: request.addPOSTParameter( "selectedState" , "Alaska");
  • removePOSTParameter( String key ). Removes a POST parameter from the request designated by key.
    example: request.removePOSTParameter( "selectedState" );
  • getURLAsString(). Returns the requested URL.
    example: url = request.getURLAsString();
  • setRequestLine(String requestMethod, String url, String httpVersion).
    Sets the complete request line for the HTTP request. The url string must be a valid uri.
    example: request.setRequestLine( "GET" , "http://somesite.com/somepage.html", "HTTP/1.1");
  • addHTTPHeader(String key, String value).
    Adds an HTTP header to the request.
    example: request.setHTTPHeader( "Cookie" , "someCookieValue");
  • removeHTTPHeader(String key, String value).
    Removes an HTTP header from the request. A key and value must be supplied since requests can have headers with duplicate keys.
    example: request.removeHTTPHeader( "Cookie" , "someCookieValue");
• response. This variable allows for interaction with the currently received HTTP response. It has the following methods:
  • getStatusLine(). Gets the status line returned from the server.
    example: statusLine = response.getStatusLine();
  • setStatusLine( String statusLine ). Sets the status line of the response.
    example: response.setStatusLine( "HTTP/1.1 200 OK" );
  • addHTTPHeader(String key, String value).
    Adds an HTTP header to the request.
    example: response.setHTTPHeader( "Set-Cookie" , "someCookieValue");
  • removeHTTPHeader(String key, String value).
    Removes an HTTP header from the request. A key and value must be supplied since requests can have headers with duplicate keys.
    example: response.removeHTTPHeader( "Set-Cookie" , "someCookieValue");
  • getContentAsString().
    Gets the content of the response.
    example: content = response.removeHTTPHeader( "Cookie" , "someCookieValue");
  • setContentAsString(String content).
    Sets the content of the response.
    example: response.setContentAsString( "<html><head ... </html>");

Variable scope

Depending on when a script gets run different variables may be in scope. The table that follows specifies what variables will be in scope depending on when a given script is run.

Debugging scripts

One of the best ways to fix errors is to simply watch the proxy session log (under the "Log" tab) and the "error.log" file (located in the "log" directory where screen-scraper was installed) for script errors. When a problem arises in executing a script screen-scraper will output a series of error-related statements to the logs. Often a good approach in debugging is to build your script bit by bit, running it frequently to ensure that it runs without errors as you add each piece.

Related pages:

• Scripting in Interpreted Java
• Scripting in JavaScript
• Scripting in JScript
• Scripting in Perl
• Scripting in Python
• Scripting in VBScript