setMultiThreadWrite
void sqlDataManager.setMultiThreadWrite ( int numThreads ) (professional and enterprise editions only)
Description
Set number of threads that the data manager can have open at once. When set higher than one, the scraping session can continue to run and download pages while the database is being written. This can decrease the time required to run a scrape, but also makes debugging harder as there is no guarantee about the order in which data will be written. It is recommended to leave this setting alone while developing a scrape. Also, the flush method will always return true if more than one thread is being used to write to the database, even if the write failed.
Parameters
- numThreads The number of threads that the data manager can start and use to write data, as an integer.
Return Values
Returns void.
Change Log
| Version | Description |
|---|---|
| 5.0 | Available for professional and enterprise editions. |
Examples
Set Thread Count
// Import classes
import com.screenscraper.datamanager.*;
import com.screenscraper.datamanager.sql.*;
import org.apache.commons.dbcp.BasicDataSource;
// Set Variables
host = "127.0.0.1:3306";
database = "mydb";
username = "user";
password = "pwrd";
parameters = "autoReconnect=true&useCompression=true";
// Build the BasicDataSource for the database connection
BasicDataSource ds = new BasicDataSource();
ds.setDriverClassName( "com.mysql.jdbc.Driver" );
ds.setUsername( username );
ds.setPassword( password );
ds.setUrl( "jdbc:mysql://" + host + "/" + database + "?" + parameters );
ds.setMaxActive( 100 );
// Get MySQL datamanager
dm = new SqlDataManager( ds, session );
// Set number of threads that can be opened
// when interacting with the database
dm.setMultiThreadWrite(10);
// Build Schemas For all Tables
dm.buildSchemas();
import com.screenscraper.datamanager.*;
import com.screenscraper.datamanager.sql.*;
import org.apache.commons.dbcp.BasicDataSource;
// Set Variables
host = "127.0.0.1:3306";
database = "mydb";
username = "user";
password = "pwrd";
parameters = "autoReconnect=true&useCompression=true";
// Build the BasicDataSource for the database connection
BasicDataSource ds = new BasicDataSource();
ds.setDriverClassName( "com.mysql.jdbc.Driver" );
ds.setUsername( username );
ds.setPassword( password );
ds.setUrl( "jdbc:mysql://" + host + "/" + database + "?" + parameters );
ds.setMaxActive( 100 );
// Get MySQL datamanager
dm = new SqlDataManager( ds, session );
// Set number of threads that can be opened
// when interacting with the database
dm.setMultiThreadWrite(10);
// Build Schemas For all Tables
dm.buildSchemas();
scraper on 07/16/2010 at 5:22 pm
- Printer-friendly version
- Login or register to post comments