golang http client returning wrong content type - go

I have a very simple Go program that performs HTTP HEAD on a URL, and prints the content-type of the response:
package main
import (
"fmt"
"net/http"
)
func main() {
resp, _ := http.Head("https://jira.softwareplant.com/servicedesk/customer/portal/1/")
fmt.Println(resp.Header.Get("Content-Type"))
}
When I run it, it returns the following:
$ go run url.go
application/octet-stream;charset=UTF-8
However, when I do the same using curl, it returns a different content type (both in the original response, and after the redirect):
$ curl -I -L https://jira.softwareplant.com/servicedesk/customer/portal/1/
HTTP/1.1 302
Date: Thu, 11 Jun 2020 18:07:26 GMT
Content-Type: text/html;charset=UTF-8
Connection: keep-alive
X-AREQUESTID: 1207x5410258x1
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Content-Security-Policy: frame-ancestors 'self'
X-ASEN: SEN-L15483924
Set-Cookie: atlassian.xsrf.token=BWV3-4JDO-FP3E-CBA1_b0942d30c14d689f92051e7b2d8467e0a0ce2129_lout; Path=/; Secure
Set-Cookie: JSESSIONID=8FE57CA54FEC626F0521327DCBA1D3DB; Path=/; Secure; HttpOnly
X-ASESSIONID: 18hzbge
X-AUSERNAME: anonymous
Location: /plugins/servlet/desk/portal/1/
HTTP/1.1 302
Date: Thu, 11 Jun 2020 18:07:26 GMT
Content-Type: text/html;charset=UTF-8
Connection: keep-alive
X-AREQUESTID: 1207x5410259x1
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Content-Security-Policy: frame-ancestors 'self'
X-ASEN: SEN-L15483924
Set-Cookie: atlassian.xsrf.token=BWV3-4JDO-FP3E-CBA1_fcd3f481d084e039075ebbce34039870d7cd044d_lout; Path=/; Secure
Set-Cookie: JSESSIONID=7B895577760D8E31F02B818FA8C0E1B2; Path=/; Secure; HttpOnly
X-ASESSIONID: 289ito
X-AUSERNAME: anonymous
Location: /servicedesk/customer/portal/1//user/login?destination=portal%2F1/
HTTP/1.1 200
Date: Thu, 11 Jun 2020 18:07:26 GMT
Content-Type: text/html;charset=UTF-8
Connection: keep-alive
X-AREQUESTID: 1207x5410260x1
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Content-Security-Policy: frame-ancestors 'self'
X-ASEN: SEN-L15483924
Set-Cookie: atlassian.xsrf.token=BWV3-4JDO-FP3E-CBA1_dc371dbc76497b29f1fa939a65dc6dd5b3488e7f_lout; Path=/; Secure
Set-Cookie: JSESSIONID=E17E2F0C4B7CA5C9ADDC4BE468A5D459; Path=/; Secure; HttpOnly
X-ASESSIONID: 1byquf1
X-AUSERNAME: anonymous
Cache-Control: no-cache, no-store, no-transform
Am I doing something wrong? Is there a way in Go to get the correct content-type for such URLs?
I am using Golang 1.14.4 on Ubuntu. The above URL is not the only one that has this issue.

If you change the Accept header sent by Go, you will get Content-Type: text/html;charset=UTF-8:
package main
import (
"fmt"
"net/http"
)
func main() {
client := &http.Client{}
req, _ := http.NewRequest("HEAD", "https://jira.softwareplant.com/servicedesk/customer/portal/1/", nil)
req.Header.Set("Accept", "*/*")
resp, _ := client.Do(req)
fmt.Println(resp.Header.Get("Content-Type"))
}

Related

Why Spring Boot application does not work with relative quality factor of HTTP?

Lets suppose I have a Spring Boot application:
dependencies {
implementation("org.springframework.boot:spring-boot-starter-webflux:2.4.0")
}
with a simple RestController:
#RestController
class TestController {
#PostMapping("/test")
suspend fun test(#RequestBody request: Map<String, String>) {
throw RuntimeException("test")
}
}
When I use httpie client to make requests, then the result looks like:
➜ ~ http post :8080/test param=value --verbose
POST /test HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 18
Content-Type: application/json
Host: localhost:8080
User-Agent: HTTPie/2.3.0
{
"param": "value"
}
HTTP/1.1 500 Internal Server Error
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Length: 309
Content-Type: text/html;charset=UTF-8
Expires: 0
Pragma: no-cache
Referrer-Policy: no-referrer
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1 ; mode=block
<html><body><h1>Whitelabel Error Page</h1><p>This application has no configured error view, so you are seeing this as a fallback.</p><div id='created'>Wed Dec 09 22:24:52 MSK 2020</div><div>[906bb33e-5] There was an unexpected error (type=Internal Server Error, status=500).</div><div>test</div></body></html>
When I use cURL:
➜ ~ curl -XPOST localhost:8080/test -d '{"param": "value"}' -H "Content-type: application/json" -v
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> POST /test HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
> Content-type: application/json
> Content-Length: 18
>
* upload completely sent off: 18 out of 18 bytes
< HTTP/1.1 500 Internal Server Error
< Content-Type: application/json
< Content-Length: 170
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-XSS-Protection: 1 ; mode=block
< Referrer-Policy: no-referrer
<
* Connection #0 to host localhost left intact
{"timestamp":1607541989698,"path":"/test","status":500,"error":"Internal Server Error","message":"test","requestId":"7f249f68-9","exception":"java.lang.RuntimeException"}* Closing connection 0
The difference in header Accept. Httpie uses Accept: application/json, */*;q=0.5 header with relative quality factor 0.5 and, despite the fact that I requested JSON if possible (and it's possible), the application returns HTML representation.
Is it how Spring Boot should work and I do something wrong?
UPD: Everything works fine with Tomcat (spring-boot-starter-web). After some debugging I found that, when Tomcat used, errors are handled by org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController and for Netty it is org.springframework.boot.autoconfigure.web.reactive.error.DefaultErrorWebExceptionHandler. It looks like the source of misbehavior is here: when MediaType.ALL is removed from acceptedMediaTypes .

format of web api respons is not correct

request is present "accept: application/xml; charset=utf-8" but response content is json
Request:
Connection: Keep-Alive
Accept: application/xml; charset=utf-8
Content-Type: application/xml; charset=utf-8
Response:
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Mon, 07 Jan 2019 17:11:14 GMT
Content-Length: 31
{"cod":1,"msg":"","data":"100"}
software is developerd with .net framework 4.6.1
I noticed that if I return an anonymous class this will be serialized in json and never in xml.
to solve the problem, just return a non-anomalous calsse and it will be serialized correctly.

Laravel/Nginx - Customize Header request

I have simple laravel project. I recive data from GET url. Datas is sending from small device and i need set small/short header response for him. It's very important for optimization.
In laravel i don't use view, only Controller function.
Below is my full response:
Cache-Control: no-cache, private
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/html; charset=UTF-8
Date: Wed, 03 Oct 2018 06:58:24 GMT
Server: nginx/1.14.0 (Ubuntu)
Set-Cookie: XSRF-TOKEN=eyJpdiI6IkpZNEllbGt0YmtsVll6ajdCSVkrNHc9PSIsInZhbHVlIjoiOFhOakxDNUVka0tIRFwvTHdidXRwdjRHMWlFeEl5QzJIcUQ4VTJXTmkzXC9VZm5rRzI3ZjI3S2N2SEZ5aE1jVEluIiwibWFjIjoiNDkwMDU2MmI3NzdkMjA2MmVkNmUyMWZkMWNhN2YzNDA3ZDZlOGMwYzIyMDU1N2Q5ZjdlYzcxODg0NDJjZTkzZiJ9; expires=Wed, 03-Oct-2018 08:58:24 GMT; Max-Age=7200; path=/
Set-Cookie: laravel_session=eyJpdiI6ImpYZG5ZNk1yWnNQb1Y1RlRSOWQ3aUE9PSIsInZhbHVlIjoieW8ySFwvQWsySG1pSEFabGZRNkR5U0t5TUhSQzZsN3lFbG1zT29TSFdhdnFVQTczVWxSWXZsZnVPR216a2N3cE8iLCJtYWMiOiIyYmI3M2Q2YWUxMDAzOTYwNjRlOWMwYWVlZGRmMWI5NWY4N2EyNGQwZTVmODRkMThhOTQ0MGIzYzVlNGI0MWNjIn0%3D; expires=Wed, 03-Oct-2018 08:58:24 GMT; Max-Age=7200; path=/; httponly
Transfer-Encoding: chunked
What is the best solution? I use laravel 5.7, first i would like remove "Set-Cookie:" from one view only .

"Missing grant type" despite "Content-Type: application/x-www-form-urlencoded"

Similar to this question I am sending the following POST to the server:
content-type: application/x-www-form-urlencoded
authorization: Basic dGVzdGp3dGNsaWVudGlkOlhZN2ttem9OemwxMDA=
accept: */*
With the payload:
{"username"="test", "password": "pw", "email": "test#example.com"}
However, I'm still getting
{
"error": "invalid_request",
"error_description": "Missing grant type"
}
as response from the server. Any idea why this is not working?
Note that the request is working if I use curl:
$ curl testjwtclientid:XY7kmzoNzl100#localhost:8080/oauth/token -d grant_type=password -d username=john.doe -d password=pw -v
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
* Server auth using Basic with user 'testjwtclientid'
> POST /oauth/token HTTP/1.1
> Host: localhost:8080
> Authorization: Basic dGVzdGp3dGNsaWVudGlkOlhZN2ttem9OemwxMDA=
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Length: 49
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 49 out of 49 bytes
< HTTP/1.1 200
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Cache-Control: no-store
< Pragma: no-cache
< Content-Type: application/json;charset=UTF-8
< Transfer-Encoding: chunked
< Date: Thu, 02 Nov 2017 19:21:29 GMT
<
* Connection #0 to host localhost left intact
{"access_token":"<an-ugly-long-token>","token_type":"bearer","expires_in":43199,"scope":"read write","jti":"80e2b6af-d999-4fb6-a4cd-5e6ab9c3fcaa"}
Your payload should be something like grant_type=password&username=john.doe&password=pw, whereas you are passing it as JSON. You can check Spring security OAuth2 accept JSON for JSON payload

Spring Session 1.0.0.M1 cookie domain attribute

I have integrated Spring Session with Redis into my SpringBoot app. It seems that all works well except the cookie domain attribute. I just found how to set the cookie session domain attribute for in tomcat i.e. method "setSessionCookieDomain", but that does not work. for example.
I had configured domain attribute in the context of the tomcat for example the domain attribute of the cookie.
#Bean
public TomcatContextCustomizer tomcatContextCustomizer() {
System.out.println("TOMCATCONTEXTCUSTOMIZER INITILIZED");
return new TomcatContextCustomizer() {
#Override
public void customize(Context context) {
context.addServletContainerInitializer(new WsSci(), null);
context.setUseHttpOnly(true);
context.setPath("/");
context.setSessionCookiePath("/");
context.setSessionCookieDomain(".127.0.0.5");
// context.setSessionCookieDomain(".localhost");
// context.setSessionCookieDomain(".test.blabla.com");
}
};
}
When i open the https trace in wireshark and click follow ssl stream, here is what i get. All other attributes are listed except the domain. So my question is how do I set the domain attribute correctly in Spring Session 1.0.0.M1, Does spring session somehow override the tomcat context?
GET / HTTP/1.1
Host: 127.0.0.5:8888
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:34.0) Gecko/20100101 Firefox/34.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Lang: keep-alive
: keep-alive
: keep-alive
: keep-alive
: keep-alive
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
Strict-Transport-Security: max-age=31536000 ; includeSubDomains
X-Frame-Options: DENY
X-Content-Security-Policy: script-src 'self'; object-src 'self'
Content-Security-Policy: script-src 'self'; object-src 'self'
X-WebKit-CSP: default-src 'self'
X-Application-Context: application:Production
Set-Cookie: SESSION=5d0a738f-f011-4e43-a1ee-d691b8eba94c; Path=/; Secure; HttpOnly
Content-Type: text/html;charset=UTF-8
Conten10:01:27 GMT
10:01:27 GMT
10:01:27 GMT
10:01:27 GMT
10:01:27 GMT
<!DOCTYPE html>
Thanks for bringing this up. Spring Session should allow configuring the domain via the SessionCookieConfig but doesn't. I have created gh-87 to address this.

Resources