Proxy won't work for this login. Help!
OK I have successfully used proxy before to login to a site and scrape but I am running into a problem logging into Upromise.com . You should be able to make a dummy account easily to try it out.
When I use proxy to generate my login script, they have an extra parameter callled "loginKey=-6922524592987362011" that changes each time you login.
While the username and password stay the same this seems to be a long random string that always changes. It seems it cannot be predetermined, so I am not able to login without the correct value.
I have tried a few things that didnt work:
1) deleted the parameter and ran my scrape.
2) login via proxy, then run the rest of my scrape for info afterwards (hoping that the login via proxy would have generated the cookie I needed to continue... but it didn't)
Also I knew for sure it wasn't logged in because I went to the "Last Response" log and viewed it in HTML and saw the "error login" you would normally see in a regular browser. This happened for both and the few other things I tried that I can't remember (4 hours LOL)
There was only one other way I could think of but I don't know how to do. If there was a way I could get to the login page (easy part) and have screenscraper actually fill in the username and password on the page and then have screenscraper manually hit submit. But I don't know if this can be done, or how to do it. (I only know how to manipulate the URLs to change pages, not hit buttons or links.) I know how to scrape for links, so I looked at the properties of the submit button to try to emulate them by URL to no avail.
Please help...
I know it is possible (but maybe not with screenscraper) because I have used another product (HTTrack Website copier) that can copy whole websites but it cannot scrape for specific info. They have an engine that clicks every link on a site (like a spider) and then copies all the pages and files. You can put a website on a disk if it fits. Somehow with their engine they are able to click on links and submit buttons (well atleast for login I think.) I use these 2 products in conjunction for a seperate project.
Thanks,
Joe
Proxy won't work for this login. Help!
Hi Joe,
Given that you're unable to use our basic edition because of a bug in our software that we don't anticipate fixing in the near future, we'd be willing to offer you a significant discount on the professional edition. Please try the professional edition with the scraping session I sent you, and, if you think you'd like to continue using it, email me back and we can discuss a reduced cost for a pro license.
Best,
Todd
OK thanks Todd
Well I suppose I could use the pro edition trial for this. I would buy it, but $500 is a bit much for an individual who doesn't have any plans to make money off it.
Yeah I will ask for the scrape session though. thanks.
Joe
Proxy won't work for this login. Help!
Hi Joe,
Well, I have some good news and some bad news. The bad news is that this appears to be a rarely occurring bug that would be somewhat time consuming to fix. As such, we likely won't be addressing it at this time.
The good news is that the site can still be scraped using Internet Explorer as the HTTP client. This feature is only available in the professional edition of screen-scraper, so you'll need to use that. If you'd like to drop us a support request via our support form (accessible here http//www.screen-scraper.com/support/all.php ) I'd be happy to send you the initial scraping session that demonstrates logging in to the site.
Best,
Todd Wilson
Proxy won't work for this login. Help!
Hi Joe,
Thanks for the posting. There seems to be an unusual issue with this site related to its secure certificate. The library we use to handle the HTTP transactions doesn't seem to be handling this correctly, so we'll need to work on a fix for it. We'll post back a reply once we have a fix for you to try. Thanks for your patience in the meantime.
Kind regards,
Todd Wilson