newbie -- stuck trying to login
I'm stuck! I followed the shopping tutorial on signing in to the shopping site web page, but when I try to do this for a different website, nothing!
Here's what I dd:
1. I ran a proxy session where I signed in to the website.
2. I found the http transaction that had stuff in the 'post data panel'
3. I createda scrapable screen from this http transaction
4. Sure enough, the paramters tab show the post item/value pairs
5. I then ran this script:
=========================
// Generate a new "NYS DMV" scraping session.
runnableScrapingSession = new com.screenscraper.scraper.RunnableScrapingSession( "NYS DMV" );
// Tell the scraping session to scrape.
runnableScrapingSession.scrape();
=========================
6. The log file shows:
=================
Starting scraper.
Running scraping session: NYS DMV
Processing scripts before scraping session begins.
Scraping file: "File from NYS DMV"
File from NYS DMV: Processing scripts before a file is scraped.
File from NYS DMV: Preliminary URL: https://search.dmv.state.ny.us/winjahtml/default.asp
File from NYS DMV: POST data: panel_id=0&touched_objects=5007_s%3B5007_s%3B5007
%3B5008_s%3B5008_s%3B5008%3B5009%3B&DefaultButtonId=5009
&close_session=&HostKey=&FirstTab=5007&id_5007=xxxxxxx
&id_5008=xxxx&id_5002=%3CIMG+height%3D56+src%3D%22images%2FAADJ-KEY.GIF
%22+width%3D196%3E&id_5009=&id_5013=%3CDIV+style%3D%22TEXT-ALIGN%3A+left
%22%3Ewww.NYSDMV.com%2FDial-In%3C%2FDIV%3E
File from NYS DMV: Resolved URL: https://search.dmv.state.ny.us/winjahtml/default.asp
File from NYS DMV: Sending request.
File from NYS DMV: Processing scripts after a file is scraped.
Processing scripts after scraping session has ended.
Scraping session finished.
================
7. BUT Nothing happens!
I goto my login scrapable screen and click 'last response', but I only see the input form, not filled in. (it has a new date/time, so I know I'm actually getting something back) I was expected the form that welcomes me to the website, after successfully logging on, or the form that tells me my login failed.
I'd be very grateful for any advice on how to make this work!
TIA,
Brian
newbie -- stuck trying to login
Brian,
Secured sites like this can sometimes be extra tricky. In your proxy session were there any other transactions beyond the one you are using? Anything else being recorded by the proxy? You can ignore any of the transactions ending in .js entries for now.
Sometimes what happens is you'll submit your credentials to a site in the form of a post, the site evaluates your credentials and then sends you in either of two directions - into the site or back to the log in. Now, how they do this redirecting can vary and I'm thinking this may be where the issue is.
screen-scraper should follow 300 HTTP responses without any needed intervention. However, if the site is redirecting you using a different more unusual approach screen-scraper may need more assistance.
When do a test using invalid credentials it does not do any redirecting so I'm unable to tell if there is an extra step to anticipate. But with valid credentials you might see entries in the proxy transaction log that reveal more than I can see.
Is there any more info you can share without giving away your own access, of course?
Thanks,
Scott