Alpha Change Log

Overview

Alpha versions are used to fix minor bugs and feature enhancement testing before they are added to stable versions. As such anything that is in the alpha version is prone to change and instability as they are being improved. This log will follow the changes as they are made for your convenience.

View Release Notes for public versions.

Alpha Version Logs

6.0.1a

  • Added sutil.getRandomUserAgent and sutil.getRandomReferer.
  • Added IDE style completions. Two new properties are needed for this to work:
    ShowVariableCompletionsAt=2 (this is the number of characters that must be typed before a completion list appears) and GenericCompletions=true (this sets a flag indicating that generic completions should be used).
  • Added session.getCurrentStack (a basic method to get the stack).
  • Added scrapeableFile.applyXPathExpression and sutil.applyXPathExpression.

6.0.2a

  • Added dataSet.size, which is equivalent to dataSet.getNumDataRecords.
  • Now nulling session variables for appropriate extractor pattern tokens after each extractor pattern match instead of after the pattern has been applied.
  • Fixed a bug where the HTTP connection pool was getting shut down prematurely.

6.0.3a

  • Fixed a bug related to the previous change to null session variables.

6.0.4a

  • Fixed a bug such that a scrapeable session ID is now being generated even for scraping sessions that will run in the future.
  • Fixed a bug where nodes in the tree weren't being highlighted correctly.
  • Scrapeable files can now be added via a URL.
  • If the DatabasePort and WebServerShutdownPort properties are omitted from the screen-scraper.properties file they'll now be automatically set to the value of an open port.
  • The ProxyPort will now only be tested and used when screen-scraper is running in server mode if the AllowProxyScripting is set to true.
  • Added a "Load Response from Clipboard" button to the scrapeable file panel.
  • Updated BeanShell to the latest version, disabling unstable Windows scripting in the process (e.g., VBScript).

6.0.5a

  • sutil.makeGETRequest and sutil.makeHEADRequest now use proxy settings from the corresponding scraping session.
  • Temporarily rolled back to the previous version of BeanShell because of a bug.

6.0.6a

  • Upgraded Bean Shell to the latest version.
  • Searches within a proxy session now include notes.
  • Fixed an issue that would cause the workbench to freeze when the breakpiont window was up.
  • Now using global proxy settings if no session proxy settings are found.
  • Improved cookie handling in the proxy server.
  • Fixed a bug that would cause a proxy session to not be completely saved.
  • Added sutil.makeGETRequestNoSessionProxy.

6.0.7a

  • Fixed a bug that would cause the proxy to misbehave when filtering out less-useful transactions.

6.0.8a

  • Now decoding parameters when adding a scrapeable file from a URL.
  • The request entity for a scrapeable file can now be set in the workbench.
  • Fixed a bug where scraping session nodes in the tree were getting collapsed incorrectly.
  • Updated web server to use Jetty.

6.0.9a

  • Fixed a bug related to generating scrapeable files from a proxy transaction.
  • Fixed a bug related to adding jar files from the ext folder.

6.0.10a

  • Fixed a bug related to redirects within sutil.makeGETRequest.
  • Fixed a problem with the scraping server not starting up.

6.0.11a

  • Fixed an issue with the web server on Windows.
  • If request entity text box is blanked now setting the value to null.
  • Errors will no longer be thrown if a scraping session has already been stopped.

6.0.12a

  • Added sutil.makeGETRequestUseSessionProxy. The sutil.makeGETRequest method will now use no proxy.
  • Fixed an issue related to loading external jar files when running in server mode.

6.0.13a

  • Fixed a bug related to automatic anonymization.

6.0.14a

  • Extractor patterns invoked manually can now be tested on a sub-set of the HTML page.
  • Added scrapeableFile.setForcePOST.
  • Upgraded internal GWT libraries.
  • Prettied up the web UI.

6.0.15a

  • Added machine-readable values to REST interface output.
  • Now propertly handling en-dash characters in URL's.

6.0.16a

  • Can now handle HTTP responses that send two status lines.
  • In-line documentation in the script editor improved. Inside screen-scraper's doc folder if a javadoc folder is found containing api documentation it will be made available within the script editor.

6.0.17a

  • Re-enabled SOAP interface.
  • Now writing an error message to the log when a scraping session import via the SOAP interface fails.

6.0.18a

  • Added a global find feature.
  • Added RetryPolicy.runOnAllAttemptsFailed()
  • Fixed a bug in RetryPolicy related to scraping files recursively.

6.0.19a

  • The scrapeableFile.addHTTPHeader method is now available in Professional Edition.

6.0.20a

  • Widened the proxy text boxes.
  • Increased the height of the sub-extractor text panel.
  • Fixed "When to run" combo box to select the correct value when clicked.
  • Fixed a bug related to editing long HTTP parameters.
  • Added sutil.stripHTML.
  • Responses from ss web server now being compressed.

6.0.21a

  • Automatic internal DB backups can be set with ShouldBackUpInternalDB property.

6.0.22a

  • Save button now becoming active when a long HTTP parameter is updated.
  • Fixed a bug related to history navigation buttons.
  • Fixed a bug related to removing completed scraping sessions via the web UI.
  • Added session.clearProxySettings() method.

6.0.23a

  • Fixed a bug related to editing extractor pattern tokens that have the same identifier.
  • Added mouseover row highlights to web UI.
  • Impoved stability in multi-threaded scrapes.

6.0.24a

  • Updated the date picker for the web UI.
  • Now diplaying server time in "Settings" dialog box in web UI.
  • Added disk space usage to web UI.
  • Added String[] scrapeableFile.getRedirectURLs().
  • Added proxy filters.
  • Comparing scrapeable file requests and proxy requests now takes into account raw request entities.
  • Script error line numbers are now hyperlinked.

6.0.25a

  • Fixed a naming issue when copying scrapeable files.
  • Fixed a bug related to HTML in error messages.
  • Fixed a bug related to comparing HTTP requests in the workbench.
  • Fixed a bug in rendering session variables in the web UI.

6.0.26a

  • Fixed disk usage indicator in web UI.
  • Fixed a bug in how POST params are rendered when comparing HTTP requests.
  • Fixed a bug related to proxy pools.

6.0.27a

  • Fixed an internal issue related to tracking running scraping sessions.
  • Now aborting a running scraping session if unable to find a valid proxy while using the proxy pool.

6.0.28a

  • Minor fix to the DataManager.
  • Error messages are no longer hyperlinked when not running the workbench.

6.0.29a

  • Fixed an issue where one script producing an error would interrupte a series of scripts.
  • Fixed an SSL issue when running on AIX.

6.0.30a

  • Fixed an issue where scraping some SSL sites would generate an error.

6.0.31a

  • Fixed a bug related to a recent change to how SSL is initialized. Added the "Use only SSL version 3" checkbox under the "Advanced" tab for a scraping session.

6.0.32a

  • Fixed a couple of bugs related to a fix in the previous build.

6.0.33a

  • One more bug fix related to the recent SSL changes.

6.0.34a

  • Altering external proxy settings for a proxy session now take effects when restarting the proxy session.

6.0.35a

  • sutil.sendMail now supports alternate content types.

6.0.36a

  • Updated password fields to obscure text.
  • Update HttpClient and NTLM authentication.

6.0.37a

  • Code folding in scripts and last response
  • Syntax highlighting in last response
  • DataManager updates
  • Added a runOnAllAttemptsFailed() method to RetryPolicy
  • Added convenience methods: isRunningInWorkbench(), isRunningFromCommandLine(), isRunningInServer() to session
  • Improved handling of NTLM proxies

6.0.38a

  • Fixed a thread blocking issue when invoking a RunnableScrapingSession.

6.0.39a

  • Fixed an issue related to reusing HTTP connections.

6.0.40a

  • Fixed a naming issue related to generating multiple scrapeable files from proxy transactions.
  • Fixed an issue where imported scripts weren't being properly associated with corresponding objects.
  • Added scrapeableFile.getResolvedURL()
  • Updated the ss_updater.py file to use the REST interface.
  • Fixed a concurrency issue related to running the same scraping session multiple times.

6.0.41a

  • Use of $ in the regular expression field for an extractor pattern token is now allowed.
  • Fixed a bug where invoking a scrapeable file manually was causing the tree in the workbench to malfunction.

6.0.42a

  • Upgraded HttpClient to version 4.3.

6.0.43a

  • Downgraded back to HttpClient 4.2.

6.0.44a

  • Includes experimental code for parsing mailing addresses.

6.0.45a

  • do_lazy_scrape can now be passed as a parameter when running a scraping session via the REST interface.
  • Added finalize_scrapeable_session action to the REST interface.

6.0.46a

  • Updated a few URL's for remote services.

6.0.47a

  • Fixed a bug related to running a scraping session via the REST interface.

6.0.48a

  • Fixed an issue resolving relative URL's beginning with ?.
  • Improved an issue related to connections remaining open when using external proxy servers.

6.0.49a

  • Fixed a bug where lazy scrapes would halt prematurely when running from the command line.

6.0.50a

  • Fixed a bug caused by extractor pattern token names containing numbers.
  • Fixed two bugs related to searching in the log and script pane.

6.0.51a

  • Fixed a minor memory leak.

6.0.52a

  • Fixed two bugs related to finding in text areas.

6.0.53a

  • Fixed a threading issue with anonymous proxies.
  • Added ConvertHTMLEntitiesByDefault and TrimWhiteSpaceByDefault to screen-scraper.properties.

6.0.54a

  • The KeyManagerFactory algorithm to be used can now be set via the KeyManagerFactory property in the screen-scraper.properties file.