Sending POST request on Investing.com returns HTTP 200 OK instead of what was requested
Using Windows 7, SS Pro 7.0.
Trying to scrape historical bond data from Investing.com with beginning and ending dates. Changing the date range requires sending an HTTP POST request with the new dates. I am having trouble sending the correct info for the POST request. I have used the "compare last request" feature. The URL and the POST data (parameters) are the same, but the Cookies and Headers are always different. Is this normal?
Investing.com server sends '200 OK' message and redirects to Investing.com's home page.
What is missing?
This is the raw request:
POST http://www.investing.com/instruments/HistoricalDataAjax HTTP/1.1
X-Requested-With: XMLHttpRequest
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Host: www.investing.com
Pragma: no-cache
Content-Length: 102
Accept-Encoding: gzip
Accept-Language: en-US
Cookie: PHPSESSID=muunhtrvokabhqhqllj3q5e2h7; GED_PLAYLIST_ACTIVITY=W3sidSI6InNRWkciLCJ0c2wiOjE0NjY3NzY4NDAsIm52IjowLCJ1cHQiOjE0NjY3NzY3NDIsImx0IjoxNDY2Nzc2NzQyfSx7InUiOiJrWk8yIiwidHNsIjoxNDY2Nzc2MzEwLCJudiI6MCwidXB0IjoxNDY2Nzc2MjUwLCJsdCI6MTQ2Njc3NjI1MH1d; SideBlockUser=a%3A2%3A%7Bs%3A10%3A%22stack_size%22%3Ba%3A1%3A%7Bs%3A11%3A%22last_quotes%22%3Bi%3A8%3B%7Ds%3A6%3A%22stacks%22%3Ba%3A1%3A%7Bs%3A11%3A%22last_quotes%22%3Ba%3A5%3A%7Bi%3A0%3Ba%3A3%3A%7Bs%3A7%3A%22pair_ID%22%3Bs%3A5%3A%2223697%22%3Bs%3A10%3A%22pair_title%22%3Bs%3A0%3A%22%22%3Bs%3A9%3A%22pair_link%22%3Bs%3A36%3A%22%2Frates-bonds%2Fu.s.-1-month-bond-yield%22%3B%7Di%3A1%3Ba%3A3%3A%7Bs%3A7%3A%22pair_ID%22%3Bs%3A5%3A%2223698%22%3Bs%3A10%3A%22pair_title%22%3Bs%3A0%3A%22%22%3Bs%3A9%3A%22pair_link%22%3Bs%3A36%3A%22%2Frates-bonds%2Fu.s.-3-month-bond-yield%22%3B%7Di%3A2%3Ba%3A3%3A%7Bs%3A7%3A%22pair_ID%22%3Bs%3A5%3A%2223699%22%3Bs%3A10%3A%22pair_title%22%3Bs%3A0%3A%22%22%3Bs%3A9%3A%22pair_link%22%3Bs%3A36%3A%22%2Frates-bonds%2Fu.s.-6-month-bond-yield%22%3B%7Di%3A3%3Ba%3A3%3A%7Bs%3A7%3A%22pair_ID%22%3Bs%3A5%3A%2223700%22%3Bs%3A10%3A%22pair_title%22%3Bs%3A0%3A%22%22%3Bs%3A9%3A%22pair_link%22%3Bs%3A35%3A%22%2Frates-bonds%2Fu.s.-1-year-bond-yield%22%3B%7Di%3A4%3Ba%3A3%3A%7Bs%3A7%3A%22pair_ID%22%3Bs%3A5%3A%2223701%22%3Bs%3A10%3A%22pair_title%22%3Bs%3A0%3A%22%22%3Bs%3A9%3A%22pair_link%22%3Bs%3A35%3A%22%2Frates-bonds%2Fu.s.-2-year-bond-yield%22%3B%7D%7D%7D%7D; fpros_popup=up; gtmFired=OK; position=4; userid=8BEF44CD-0055-2DCF-CDE2-596C2A803A7F; investing_logon=a%3A3%3A%7Bs%3A4%3A%22hash%22%3Bs%3A32%3A%22e46116e414ee18a62e72983c889b18a2%22%3Bs%3A3%3A%22uid%22%3Bs%3A9%3A%22200669260%22%3Bs%3A4%3A%22type%22%3Bs%3A5%3A%22false%22%3B%7D; geoC=US; show_big_billboard1=true; optimizelyEndUserId=oeu1466775976417r0.9478009957876945; optimizelySegments=%7B%224225444387%22%3A%22ie%22%2C%224226973206%22%3A%22direct%22%2C%224232593061%22%3A%22false%22%2C%225010352657%22%3A%22none%22%7D; optimizelyBuckets=%7B%226306990713%22%3A%220%22%7D; _ga=GA1.2.678420822.1466775977; __gads=ID=e14b1242718f1e04:T=1466775975:S=ALNI_MbpeLwsfAgH2JO_ISqw6YBn_3pVyw; __qca=P0-1097947770-1466775977290; mlUserID=uj3J8QA0fa6O; _gat_allSitesTracker=1; _gat=1; optimizelyPendingLogEvents=%5B%22n%3Dengagement%26u%3Doeu1466775976417r0.9478009957876945%26wxhr%3Dtrue%26time%3D1466791510.822%26f%3D6306990713%26g%3D4227733615%22%5D
Accept: text/plain, */*; q=0.01
Referer: http://www.investing.com/rates-bonds/u.s.-1-month-bond-yield-historical-data
Proxy-Connection: Keep-Alive
action=historical_data&curr_id=23697&st_date=01%2F01%2F2000&end_date=06%2F24%2F2016&interval_sec=Daily
Problem solved
In my original post I stated that I have used the 'Compare Last Request' feature, but thanks for the suggestion.
I solved the problem using Excel, of all things. I realize this is a screen-scraper forum, but here is the VBA code, in case this helps anyone:
Dim objHTTP As Object
URL = "http://www.investing.com/instruments/HistoricalDataAjax"
Set objHTTP = CreateObject("WinHttp.WinHttpRequest.5.1")
objHTTP.Open "POST", URL, False
objHTTP.setRequestHeader "X-Requested-With", "XMLHttpRequest"
objHTTP.setRequestHeader "Content-type", "application/x-www-form-urlencoded"
objHTTP.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
objHTTP.send ("action=historical_data&curr_id=" + YieldCode + "&st_date=01/01/2000&end_date=06/21/2016&interval_sec=Daily")
the correct HTTP response is returned here: objHTTP.responseText
Setting the following Header Requests were NOT necessary.
All this code was commented out:
'objHTTP.setRequestHeader "Pragma", "no-cache"
'objHTTP.setRequestHeader "Referer", "http://www.investing.com/rates-bonds/u.s.-1-month-bond-yield-historical-data"
'objHTTP.setRequestHeader "Proxy-Connection", "Keep -Alive"
'objHTTP.setRequestHeader "Content-Length", "102"
'objHTTP.setRequestHeader "Host", "www.investing.com"
'objHTTP.setRequestHeader "Cookie", CookieText
Generally that means the
Generally that means the request was good. If you go to the scrapeableFile > last request tab there is a button to compare to a proxy transaction. If you use that, then point it to the corresponding request in the proxy session you can see all the differences between your request and what was captured. Then make your request as identical as you can.