Where to Go From Here

The approach we outline in this tutorial works great for relatively small sets of data. When we extract records from the shopping site we're probably not going to extract more than 25 or so. When screen-scraper extracts the data it is saved in memory (remember we checked the "Automatically save the data set generated by this extractor pattern in a session variable" check box for the "DETAILS" extractor pattern, which is what causes this to happen), so it works fine because there aren't that many products.

So what happens when we want to extract and save large numbers of records? The simple answer is that you need to save them out as they're extracted rather than having screen-scraper keep them in memory. Usually this means either inserting the scraped records into a database or writing them out to a text file. We'll soon have a tutorial up that gives an example of saving records to a database. For now, take a look at this FAQ. We also provide an example in Tutorial 2 that illustrates how to write the data out to a file. Just remember that if you're writing the data out to a file you'll want to uncheck the box labeled "Automatically save the data set generated by this extractor pattern in a session variable" for the extractor pattern that pulls out the data you want to save. If it's checked it will cause screen-scraper to store all of the data in memory, which could cause it to run out of memory while it's running.