Scrape Profiler
If a scrape is taking a long time, using the scrape profiler can help you see which scrapeable files and/or scripts are using all the time, so you could optimize their runtimes.
Another reason to consider using the scrape profiler is that there is a function to breakpoint when you overwrite a session variable, so similar to a breakpoint on variable change. Using this you can determine when a session variable is being overwritten when you don't expect it to be.
import com.screenscraper.profiling.session.ScrapeProfiler;
// This should be done in the very first script to run, right at the beginning (preferably in it’s own script)
ScrapeProfiler profiler = ScrapeProfiler.profileSession(session, false);
// Note that if you are trying to watch for a large stack of scripts or
// just want to see the state of things at any given point, you
// can call profiler.generateHtmlReport() at any point, which will
// return an HTML string you can then write to a file and view
// Once the scrape completes, it will generate an HTML file in your
// output directory, named "[scrapeName] - Profiling Data yyyy-MM-dd_HH_mm_ss_zz.html"
// (where the time values are the time the profiler was setup)
// which will contain data about each script, scrapeable file, extractor, etc...
// that shows execution times. These will be more accurate if the scrape
// ran for a while, as it tries to exclude the internal execution times
// meaning the percent time in each area won't add up to 100%
// Also you can have a breakpoint popup whenever a session variable
// is changed. Note this only works if the mapping is changed, not
// if something is changed in the variable itself. For example,
// if "Foo" is a map, and I call session.getVariable("Foo").put("x", "y")
// that won't trigger a breakpoint. However calling
// session.setVariable("Foo", "Something else") will trigger the
// breakpoint
profiler.setBreakpointOnSessionVariableChange("Foo");
// This should be done in the very first script to run, right at the beginning (preferably in it’s own script)
ScrapeProfiler profiler = ScrapeProfiler.profileSession(session, false);
// Note that if you are trying to watch for a large stack of scripts or
// just want to see the state of things at any given point, you
// can call profiler.generateHtmlReport() at any point, which will
// return an HTML string you can then write to a file and view
// Once the scrape completes, it will generate an HTML file in your
// output directory, named "[scrapeName] - Profiling Data yyyy-MM-dd_HH_mm_ss_zz.html"
// (where the time values are the time the profiler was setup)
// which will contain data about each script, scrapeable file, extractor, etc...
// that shows execution times. These will be more accurate if the scrape
// ran for a while, as it tries to exclude the internal execution times
// meaning the percent time in each area won't add up to 100%
// Also you can have a breakpoint popup whenever a session variable
// is changed. Note this only works if the mapping is changed, not
// if something is changed in the variable itself. For example,
// if "Foo" is a map, and I call session.getVariable("Foo").put("x", "y")
// that won't trigger a breakpoint. However calling
// session.setVariable("Foo", "Something else") will trigger the
// breakpoint
profiler.setBreakpointOnSessionVariableChange("Foo");
samt on 03/02/2016 at 4:10 pm
- Printer-friendly version
- Login or register to post comments