Previous value lingering when empty value should be returned

Hi!

I'm having a problem getting the correct data into my database.

Setup: I have a setup very much like the tutorial setup. I scrape one result page and then many detail pages.

Problem: Each detail page has one guaranteed value (PARTNO) and some optional ones. When I scrape a page with all fields set I get the result i expect. When i scrape another detail page afterwards without all the fields set the previous set value is returned instead of an empty value.

example:
Page 1:
------------------------------------------
Partno: 111111
Info: Info for partno 111111
------------------------------------------
returned to db: partno=111111&info=Info+for+partno+111111

Page2:
------------------------------------------
Partno: 222222
Info:
------------------------------------------
returned to db: partno=222222&info=Info+for+partno+111111

Page3:
------------------------------------------
Partno: 333333
Info:
------------------------------------------
returned to db: partno=333333&info=Info+for+partno+111111

Page4:
------------------------------------------
Partno: 444444
Info: Info for partno 444444
------------------------------------------
returned to db: partno=444444&info=Info+for+partno+444444

What is wrong here? How can I avoid this happening?

Previous value lingering when empty value should be returned

oddball,

You are not alone. This is one of those aspects of screen-scraper that isn't obvious at first but once you see how it works you should understand why it works this way.

The issue you're having is related to "[url=http://www.screen-scraper.com/support/docs/using_scripts_in_proxy_server.php#Variable_Scope]variable scope[/url]". Because the previous values are persisting I'm assuming you're setting those values as session variables. Note the term, "session variable." When you set screen-screen running you're starting what we call a "scraping session". When you set session variables they're "in scope" during the entire session (or, until screen-scraper finishes scraping).

So, when you're looping through a dataSet of extracted values you'll need to explicitly null out any values that you don't want to persist into the next iteration of the loop.

You can do this using...

session.setVariable("PARTNO", null);

...either at the end of a loop iteration or just before another loop starts. The later is recommended because you're more certain the setVariable methods will be called when you need them.

A highly recommended alternative to using session variables in this way is to instead use sub-extractor patterns and the DATARECORD token. I'm working on a more detailed explanation of what this involves so stay tuned.

In the mean time you can reference the DATARECORD token and sub-extractor patterns at the following link.

[url]http://www.screen-scraper.com/support/docs/using_extractor_patterns.php[/url]

Thanks,
Scott