I'm using Delphi 10.2 and am getting a 400 Bad Request exception using TIdHTTP.Get(). I can enter the HTTP call string into any browser and it works correctly, but not in Indy.
My request string is:
accept text/html,*/*
accept charset UTF-8
basicauthentication true
contentType Text/xml
UserAgent Mozilla/4.0 (compatible; MyApp)Mozilla/4.0 (compatible; MyApp)
Thanks for your responses and suggestions. I discovered what was causing the problem. It appears that my URL had some embedded spaces in it that was triggering the bad request messages.
What was throwing me off is that pasting that same URL into a browser was working correctly but that same URL with indy was failing. After correcting the issue with the embedded spaces it is working as expected.
Related
I'm stuck with this problem for two days now. The jersey verion is 2.6.
I have enabled the logging of the request by adding a loggingfilter. Which works well.
My rerquest is logged using the right charset:
Content-Disposition: form-data; name="verskomm"
kleine Änderung
Wenn I want to consume this request using #FormDataParam I get
kleine ?nderung
Where can I set the charset which should be used to decode these parts of the request?
I have no influence on the request, as it is made by a third party program.
I have a program that uses an XMLHTTPRequest to gather contents from another web page.
Problem is, that web page has cloaking custom errors set-up (ie. /thisurl doesn't literally exist as a file on their web server, it is being generated by the custom 404 error file.), so its not returning the page it shows in the browser, instead its showing its default 404 error response from that custom error page, in my HTTPRequest response.
By using this website http://web-sniffer.net/ I have narrowed down what the problem may be, but I don't know how to fix it.
Web-sniffer has 3 different versions to submit the request:
HTTP version: HTTP/1.1
HTTP/1.0 (with Host header)
HTTP/1.0 (without Host header)`
When I use HTTP/1.1 or HTTP/1.0 (with Host header) I get the correct response (html) from the page. But when I use HTTP/1.0 (without Host header) it does not return the content, instead it returns a 404 error script (showing the custom error page).
So I have concluded that the problem may be due to the Host header not being present in the request.
But I am using MSXML2.XMLHTTP.3.0 and haven't been able to read the page using HTTP/1.1 or HTTP/1.0 (with Host header). The code looks like this:
Set objXML = Server.CreateObject("MSXML2.XMLHTTP.3.0")
objXML.Open "GET", URL, False
objXML.setRequestHeader "Host", MyDomain '< Doesnt work with or w/out this line
objXML.Send
Even after adding a Host header to the request, I still get the template of the 404 error returned by that custom error script in my response, the same as HTTP/1.0 (without Host Header) option on that web-sniffer site. This should be returning 200 OK like it does on the first two options on web-sniffer, and like in a web browser.
So I guess my question is, what is that website (web-sniffer.net) able to get the proper response with their first two HTTP version options, so I can emulate this in my app. I want to get the right page, but its only returning the 404 error from their 404 error template.
In response to an answerer, I have provided screen shots from 2 seperate cUrl requests below, one from each one of my servers.
I executed the same cURL command, same url (that points to a site on the main host), which is cURL -v -I www.site.com/cloakedfile . But looks like its not working on the main server, where it needs to be. It can't be a self-residing issue, because from secondary to secondary it works fine, these are both identical applications/sites, just different ip's/host names. It appears to be an internal issue, that may not be about the application side of things.
I dont have any idea bout MSXML2.XMLHTTP.3.0. But from you problem statement i understand that the issues is certainly due to some HTTP header field that is wrongly set or missed out in your request.
By default HTTP 1.1 clients set Host header. For example if you are connecting to google.com then the request will look like this
GET / HTTP/1.1
Host: google.com
The "Host" header should have the domain name of the server in which the requested resource is residing. Severs that has virtual hosting will get confused if "Host:" header is not present. This is what happens with groups.yahoo.com if you havent specified Host header
$ nc groups.yahoo.com 80
GET / HTTP/1.1
HTTP/1.1 400 Host Header Required
Date: Fri, 06 Dec 2013 05:40:26 GMT
Connection: close
Via: http/1.1 r08.ycpi.inc.yahoo.net (ApacheTrafficServer/4.0.2 [c s f ])
Server: ATS/4.0.2
Cache-Control: no-store
Content-Type: text/html; charset=utf-8
Content-Language: en
Content-Length: 447
And this should be the same issue you are facing with. And also make sure that you are sending the domain name of the server from which you are trying to fetch the resource. And the Host header should have a colon ":" to delimit the value like "Host: www.example.com".
I have a url. When I try to access it programmatically, the backend server fails (I don't run the server):
import requests
r = requests.get('http://www.courts.wa.gov/index.cfm?fa=controller.managefiles&filePath=Opinions&fileName=875146.pdf')
r.status_code # 200
print r.content
When I look at the content, it's an error page, though the status code is 200. If you click the link, it'll work in your browser -- you'll get a PDF -- which is what I expect in r.content. So it works in my browser, but fails in Requests.
To diagnose, I'm trying to eliminate differences between my browser and Requests library. So far I've:
Disabled Javascript
Disabled (and deleted) cookies
Set the User-Agent to be the same in each
But I can't get the thing to work properly in Requests or fail in my browser due to disabling something. Can somebody with a better idea of browser-magic help me diagnose and solve this?
Does the request work in Chrome? If so, you can open the web inspector and right-click the request to copy it as a curl command. Then you'll have access to all the headers, params, and request body, which you can play around with to see which are triggering the failure you're seeing with the requests library.
You're probably running into a server that discriminates based on User-Agent. This works:
import requests
S = requests.Session()
S.headers.update({'User-Agent': 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)'})
r = S.get('http://www.courts.wa.gov/index.cfm?fa=controller.managefiles&filePath=Opinions&fileName=875146.pdf')
with open('dl.pdf', 'wb') as f:
f.write(r.content)
I use RestSharp in my Windows Phone 7.1 project.
My problem is RestSharp always cache response data.
Example:
At the first time I send request, it returns data correctly. After some delete operations, I send that request again, but response seems the same as the first time, nothing's changed.
If I stop debugging and press F5 to start again, it works perfectly as expected.
I also tried request.AddParameter("cache-control", "no-cache", ParameterType.HttpHeader); and got no luck.
How can I fix this problem?
I have the same issue so just add header that specify not to cache response data
client is my RestClient with base url and than add default header Cache-Control with value no-cache.
client.AddDefaultHeader("Cache-Control", "no-cache")
I found solution in Rico Suter comment, thanks! I will mark this as accepted anwser
its a hack but try something like url = originalUrl + "&nocache=" + DateTime.Now.Ticks
The "Cache-Control" header should do the trick!
I think HTTP Headers are case-insensitive, but the server may not agree with me there! You should try using Cache-Control instead of cache-control...
Also, I would also add the Pragma header with no-cache value to the request (some old servers don't use the "Cache-Control" header, but they will sure recognize this one)!
And I would try to use Fiddler to debug the comms and check that the headers are really being sent to the server as expected!
Another solution can be to set the "If-Modified-Since" header with value of DateTime.Now:
client.AddDefaultParameter("If-Modified-Since", DateTime.Now, ParameterType.HttpHeader);
One of the request parameters in an http request made by the client contains Japanese characters. If I make this request in Firefox and look at the parameter as soon as it reaches the server by debugging in Eclipse, the characters look fine. If I do the same request using IE 8, the characters get garbled when I look at them at the same point in the server code (they are fine in both browsers, though). I have examined the POST requests made by both browsers, and they both pass the same sequence of characters, which is:
%2C%E3%81%9D%E3%81%AE%E4%BB%96
I am therefore thinking that this has to do with the encoding. If I look at the HTTP headers of the request, I notice the following differences. In IE:
Content-Type: application/x-www-form-urlencoded
Accept: */*
In Firefox:
Content-Type application/x-www-form-urlencoded; charset=UTF-8
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
I'm thinking that the IE 8 header doesn't state the UTF-8 encoding explicitly, even though it's specified in the meta tag of the HTML document. I am not sure if this is the problem. I would appreciate any help, and please do let me know if you need more information.
Make sure the page that contains the form has UTF-8 as charset. In IE's case, the best thing to make sure of this is by sending a HTTP header ('Content-Type: text/html; charset=utf-8') and adding a meta http-equiv tag with the content type/charset to your html (I've seen this actually matter, even when the appropriate header was sent).
Second, your form can also specify the content type:
<form enctype="application/x-www-form-urlencoded; charset=utf-8>