Debugging

Overview

There are times when you need to debug what is going on in your scrapes. The following can help with tracking down various issues.

Scrape Profiler

If a scrape is taking a long time, using the scrape profiler can help you see which scrapeable files and/or scripts are using all the time, so you could optimize their runtimes.

Another reason to consider using the scrape profiler is that there is a function to breakpoint when you overwrite a session variable, so similar to a breakpoint on variable change. Using this you can determine when a session variable is being overwritten when you don't expect it to be.

    import com.screenscraper.profiling.session.ScrapeProfiler;

    // This should be done in the very first script to run, right at the beginning (preferably in it’s own script)
    ScrapeProfiler profiler = ScrapeProfiler.profileSession(session, false);

    // Note that if you are trying to watch for a large stack of scripts or
    // just want to see the state of things at any given point, you
    // can call profiler.generateHtmlReport() at any point, which will
    // return an HTML string you can then write to a file and view

    // Once the scrape completes, it will generate an HTML file in your
    // output directory, named "[scrapeName] - Profiling Data yyyy-MM-dd_HH_mm_ss_zz.html"
    // (where the time values are the time the profiler was setup)
    // which will contain data about each script, scrapeable file, extractor, etc...
    // that shows execution times.  These will be more accurate if the scrape
    // ran for a while, as it tries to exclude the internal execution times
    // meaning the percent time in each area won't add up to 100%


    // Also you can have a breakpoint popup whenever a session variable
    // is changed.  Note this only works if the mapping is changed, not
    // if something is changed in the variable itself.  For example,
    // if "Foo" is a map, and I call session.getVariable("Foo").put("x", "y")
    // that won't trigger a breakpoint.  However calling
    // session.setVariable("Foo", "Something else") will trigger the
    // breakpoint
    profiler.setBreakpointOnSessionVariableChange("Foo");