Bad proxy servers are not eliminated
While testing the proxy server pool, I am getting these messages:
Using proxy: 200.112.213.157:53281 - with credentials
Reporting the current proxy server as bad.
The current proxy server was previously reported as bad.
Reporting the current proxy server as bad.
The current proxy server was previously reported as bad.
BAZAR_FirmenDetails: An input/output error occurred while connecting to 'http://www.bazar.at/haring-group'. The message was Connect to 200.112.213.157:53281 [/200.112.213.157] failed: Connection timed out: connect.
BAZAR_FirmenDetails: There was an error on the last request. Now making attempt number 1 of 10 requests.
BAZAR_FirmenDetails: Requesting URL: http://www.bazar.at/haring-group
Using proxy: 201.183.231.11:3128 - with credentials
It seems to me that in contrast to the statement in the API documentation:
// As a scraping session runs, screen-scraper will filter out
// proxies that become non-responsive. If the number of proxies
// gets down to a specified level, screen-scraper can repopulate
// itself. Thats what this method call controls.
the "bad" servers are not being filtered!
- Is there some flag that toggels this behaviour or am I just wrong in reading the description.
- Can the condition "proxy server is bad" be detected from the script?
best regards
Christian Pieler
no answer
In the more than seven years of using the SS forum, I always got pretty quick and useful answers to my posts. Now it has been more than a week since I sent the above request with no answer. What has changed?
best regards
Christian Pieler
Sorry. It just slipped past
Sorry. It just slipped past me somehow.
A proxy marked as failed, should be removed from the current pool. If you read in the proxies from a file, however, there is nothing editing the file. I usually make a scrape to read in a list of proxies, test each, and write the good ones to a new file for use on a scrape. If your proxies are in a database you can log good vs bad, and the date/time tested so you can track how many you lose.