Home » Forums » screen-scraper » screen-scraper public support

Running extractor pattern from script

I'm not sure this is in the tutorials (I worked through them a couple of weeks ago and this is the first instance I've come across this).

Using Screen-Scraper's scripting engine, how does one call/implement an extractor pattern?

Thanks!

‹ Proxy issues? Sub-Extractor Pattern bug ›
TomRod on 06/30/2007 at 1:09 pm

Running extractor pattern from script

TomRod,

The lengthy URL you're having trouble with is working on our end. Without knowing a bit more about your scraping session I can't say what might be of issue. You're welcome to email me your scraping session and I'll take a closer look.

My address is my first name and last initial at out domain.

Thanks,
Scott W.

 swilsonmc on 07/06/2007 at 3:04 pm

Running extractor pattern from script

This is probably the last question on the subject:

Does screenscraper have a limit to the length of the URL that is inputted? My motivation for this question is the following:

An example:

http://customer1.barchart.com/cgi-bin/mri/vsnchart.htx?page=chart&showcc=yes&sym=EDH09&cc=DX&mon=&year=&data=A&date=090307&size=b&den=med&jav=adv&expm=0&sly=N&ch1=011&arga=&argb=&argc=&ov1=&argd=&arge=&argf=&ch2=&argg=&argh=&argi=&ov2=&argj=&argk=&argl=&code=vsn&org=com&crea=Y

When I try to run this via a script inputted-variable using ~#TOKEN#~, the script is not inputting this as the HTML Target, and my scraping session breaks. However, with smaller HTML targets. such as

http://customer1.barchart.com/cgi-bin/mri/webplt.exe?sym=EDH82&data=A&vol=Y&jav=adv&grid=Y&late=y&org=com&perm=9&code=XVSN

my scraping sessions runs perfectly.

 TomRod on 07/05/2007 at 3:31 pm

Running extractor pattern from script

TomRod,

Sorry I didn't notice this before but your using an invalid method.

scrapeableFile.scrape("Rip");

needs to be...

session.scrapeFile("Rip");

http://www.screen-scraper.com/support/docs/api_documentation.php#scrapeFile

That should fix the errors you were having.

-Scott

 swilsonmc on 07/03/2007 at 4:14 pm

Running extractor pattern from script

So, I'm curious if this code is valid:

session.setVariable( "Name_Session_Var", "http://www.economagic.com/em-cgi/data.exe/libor/day-ca9m" );
session.setVariable( "NameTXT", "LIBORca9M" );
scrapeableFile.scrape("Rip");

Here Rip is the name of the scrapable File contained within a scraping session. I'm having this file called by the script. However, I get the following error when I run this script:

LiborSwap: An error occurred while processing the script: MetaLiborSwap
LiborSwap: The error message was: Attempt to invoke method: scrape() on undefined variable or class name: scrapeableFile : at Line: 4.
Processing scripts after scraping session has ended.

Now, the entire script is built from snippets of the above. The errors from earlier (when the file was writing to null.txt) are solved.[/code]

 TomRod on 07/03/2007 at 3:18 pm

Running extractor pattern from script

TomRod,

You've also exposed a common issue (at least for us internally). It is important to clear the value of session variables that are needed for each new sequence in a loop especially when there is not always a value extracted each time a pattern is applied. If you don't clear the value when an extractor fails to match the value will persist the entire session until it is replaced when either an extractor pattern extracts a value that replaces the old value or it is nulled out when you manually call...

session.setVariable("myVariable", null)

But, no, you do not need to clear the value of a variable in order for the value to be replaced when a value of an extractor pattern token matches. So, the issue you're having must be related to the variable being altered some how between the metascript and the rip script.

You can try this code to check if the variable is still in the session scope.

if ( session.getVariable("NAMEtxt") == null )
{
    session.log("Is Null");
{
else
}
    session.log("Not null.  It is:" + session.getVariable("NAMEtxt"));
}

Is there anything happening between the two scripts that could be the culprit?

-Scott

 swilsonmc on 07/03/2007 at 1:44 pm

Running extractor pattern from script

Using the modifications you gave, it writes to the first NAMEtxt of the first instantiation for all the instances.

I'm not sure how to clear a variable from the scraping session (so that I can reuse the same variable name over and over and change the variables in the background). Granted, that exposes my naivite to Java :-) Does screen-scraper need the variables cleared before a variable can be reassigned?

 TomRod on 07/03/2007 at 12:43 pm

Running extractor pattern from script

TomRod,

Everything looks ok that you've shared. In this case what I would do to help identify where the disconnect is happening would be to write to the log the places that you're expecting the "missing" variable to be. If anything is occurring between these two instances you should do something similar for them. So this idea may seem unnecessary but this practice often helps root out the problem. It's like dropping breadcrumbs. The use of the double asterisk help reveal if an expected variable comes across null or empty.

On your metascript:

session.setVariable( "NameTXT", "LIBORUSOV" );
//Write out the variable just to make sure it took
session.log("Metascript: NameTXT**" + session.getVariable("NameTXT") + "**");

On your Rip script:

//Write out the variable to see if carried over from when it was set in the meta script
//and before it is used in the FileWriter method
session.log("Rip: NameTXT**" + session.getVariable("NameTXT") + "**");
out = new FileWriter( "C:\\RawData\\" + session.getVariable("NameTXT") + ".txt",  true );

Let us know how this goes,

-Scott

 swilsonmc on 07/03/2007 at 10:12 am

Running extractor pattern from script

Scott,
Thanks for the swift reply. Yes, I created the directory before I ran the script.
What other information do you need?

Also, the data is being written to a file titled null.txt. My best guess is that the second snippet of code is for some reason not recognizing the variable set by the metascript (example is in the first snippet of code).

Thanks!

 TomRod on 07/02/2007 at 6:15 pm

Running extractor pattern from script

TomRod,

Could you verify that you have the following directory created within Windows Explorer before you start your scraping session?

If that does not fix te error you're getting then I'll need more info.

Please let us know what you find.

Thanks,
Scott

 swilsonmc on 07/02/2007 at 4:52 pm

Running extractor pattern from script

Ok, I've worked out a few things.

Here is the best explanation of the problem:

I have a metescript running many identical instances of code that look like this:

session.setVariable( "Name_Session_Var", "http://www.economagic.com/em-cgi/data.exe/libor/day-ussnon" );
session.setVariable( "NameTXT", "LIBORUSOV" );
scrapableFile.scrape("Rip");

the file Rip runs an extractor pattern that runs the following script:

session.log("Writing data to a file.");
out = new FileWriter("C:\\RawData\\" + session.getVariable("NameTXT") + ".txt", true);
out.write( "\r\n" );
out.write( session.getVariable( "Date" ) + "\t");
out.write( session.getVariable( "Rate" ) + "\t");

out.close();

Now, the problem I'm having is that this is being written to "null.txt" in the folder specified. As the first snippet says, it should be written to LIBORUSOV. Any advice? I appreciate the help!

 TomRod on 07/02/2007 at 3:37 pm

Running extractor pattern from script

TomRod,

I don't believe we cover the method for handling that in any of the tutorials. However, you can find it in our API documentation here.

[url]http://www.screen-scraper.com/support/docs/api_documentation.php#extractData[/url]

-Scott

 swilsonmc on 07/02/2007 at 10:16 am