Recording Search Results

As in the first tutorial, we'll be recording a browser session using the proxy server. Remember that a proxy session holds all of the HTTP requests and responses from your browser for the period of time you run it.

Create a new proxy session now either by clicking the "New Proxy Session" button (looks like a globe) or by selecting "New Proxy Session" from the "File" menu. When the proxy session appears type in "Shopping Site" in the "Name" field. In your web browser go to this URL: http://www.screen-scraper.com/shop/ (remember that you may want to use one browser with the proxy server and one to view the tutorials).

At this point start up the proxy server by clicking the "Start Proxy Server" button, then configure your web browser as you did in the first tutorial (if you need help try this page). In screen-scraper, ensure that the "Don't log binary files" checkbox is checked. Now click on the "Progress" tab so that you can see the pages appear as they get recorded.

We'll be doing a search in the shopping web site for the term "dvd" in the various products. Do this by typing "dvd" (without the quotes) into the search box located in the upper-right corner of the home page, then click the "Search" button. You'll see screen-scraper work for a bit, then, once it finishes, you should just see one row in the "HTTP Transactions" table. We'll want to traverse all of the search results, so, in your web browser, click the "Next >>" link. screen-scraper will work again for a bit while it records the next search results page. Later on we'll be scraping the details pages, so let's record one of those now. Click on the "Speed" link to view details on this DVD. These are the only pages we're interested in at this point, so go ahead and stop the proxy session by clicking the "Stop Proxy Server" button on the "General" tab. You'll also want to re-configure your web browser so that it's no longer using screen-scraper as a proxy server.