Getting comma separated ips from Http header - spring

I have a Spring boot app running on Tomcat. I have to resolve each ip to its Geolocation : city , province and Country . However,sometimes I receive ip address as a comma separated String instead of a single ip address. For example , 1.39.27.224, 8.37.225.221 .
The code to extract ip from a Http request that I am using :
public static String getIp(final HttpServletRequest request) {
PreConditions.checkNull(request, "request cannot be null");
String ip = request.getHeader("X-FORWARDED-FOR");
if (!StringUtils.hasText(ip)) {
ip = request.getRemoteAddr();
}
return ip;
}

The X-Forwarded-For can be used to identify the originating IP address of a client connecting to a web server through an HTTP proxy or load balancer.
The general format of this field is
X-Forwarded-For: client, proxy1, proxy2
In above example you can see that the request is passed through proxy1 and proxy2.
In your case you should parse this comma separated string and read the first value which is client's IP address.
Warning - It is easy to forge an X-Forwarded-For field so you might get wrong information.
Please take a look at https://en.wikipedia.org/wiki/X-Forwarded-For to read more about this.

Here is what I use in my servlet (running on Jetty behind HAProxy) -
I just try to get the first IP address in the X-Forwarded-For header:
Pattern FIRST_IP_ADDRESS = Pattern.compile("^(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})");
public static String parseXff(HttpServletRequest httpReq) {
String xff = httpReq.getHeader("X-Forwarded-For");
if (xff != null) {
Matcher matcher = FIRST_IP_ADDRESS.matcher(xff);
if (matcher.find()) {
return matcher.group(1);
}
}
// return localhost when servlet is accessed directly, without HAProxy
return "127.0.0.1";
}

Related

index_not_found_exception - Elasticsearch

In image #1, as you can see, I am getting a valid ES response on firing a GET request. However, if I try doing the same things through the NGINX reverse proxy that I have created and hit myip/elasticsearch, it returns me the error (image #2). Can someone help me with this?
server {
listen 80;
server_name myip;
location /elasticsearch/ {
proxy_pass http://127.0.0.1:9200;
}
location /kibana/ {
proxy_pass http://127.0.0.1:5601;
}
}
The right way is to specify both of those slashes. Slash after 127.0.0.1:9000 is essential, without it your request /elasticsearch/some/route would be passed as-is while with that slash it would be passed as /some/route. In nginx terms it means that you specified an URI after the backend name. That is, an URI prefix specified in a location directive (/elasticsearch/) stripped from an original URI (we having some/route at this stage) and an URI specified after the backend name (/) prepended to it resulting in / + some/route = /some/route. You can specify any path in a proxy_pass directive, for example, with proxy_pass http://127.0.0.1:9200/prefix/ that request would be passed to the backend as /prefix/some/route. Now if you understand all being said, you can see that specifying location /elasticsearch { ... } instead of location /elasticsearch/ { ... } would give you //some/route instead of /some/route. I'm not sure it is exactly the cause of your problem however configurations like
location /elasticsearch/ {
proxy_pass http://127.0.0.1:9200/;
}
are more correct.
Now may I ask you what you get with exactly this configuration in response to curl -i http://localhost:9200/ and curl -i http://localhost/? I want to see all the headers (of cause except those containing private information).
The problem is the path. Nginx is passing it unmodified.
Add a slash at the proxy_pass urls.
server {
listen 80;
server_name myip;
location /elasticsearch/ {
proxy_pass http://127.0.0.1:9200/;
}
location /kibana/ {
proxy_pass http://127.0.0.1:5601/;
}
}
From the documentation:
Note that in the first example above, the address of the proxied server is followed by a URI, /link/. If the URI is specified along with the address, it replaces the part of the request URI that matches the location parameter. For example, here the request with the /some/path/page.html URI will be proxied to http://www.example.com/link/page.html. If the address is specified without a URI, or it is not possible to determine the part of URI to be replaced, the full request URI is passed (possibly, modified).

Service Fabric https endpoint with kestrel and reverse proxy

I've been trying to setup Https on a stateless API endpoint following the instructions on the microsoft documentations and diverse post/blogs I could find. It works fine locally, but I'm struggling to make it work after deploying it on my dev server getting
Browser : HTTP ERROR 504
Vm event viewer : HandlerAsyncOperation EndProcessReverseProxyRequest failed with FABRIC_E_TIMEOUT
SF event table : Error while processing request: request url = https://mydomain:19081/appname/servicename/api/healthcheck/ping, verb = GET, remote (client) address = xxx, request processing start time = 2018-03-13T14:50:17.1396031Z, forward url = https://0.0.0.0:44338/api/healthcheck/ping, number of successful resolve attempts = 48, error = 2147949567, message = , phase = ResolveServicePartition
in code I have in the instancelistener
.UseKestrel(options =>
{
options.Listen(IPAddress.Any, 44338, listenOptions =>
{
listenOptions.UseHttps(GetCertificate());
});
})
servicemanifest
<Endpoint Protocol="https" Name="SslServiceEndpoint" Type="Input" Port="44338" />
startup
services.AddMvc(options =>
{
options.SslPort = 44338;
options.Filters.Add(new RequireHttpsAttribute());
});
+
var options = new RewriteOptions().AddRedirectToHttps(StatusCodes.Status301MovedPermanently, 44338);
app.UseRewriter(options);
here is what I got in azure (deployed through ARM template)
Health probes
NAME PROTOCOL PORT USED BY
AppPortProbe TCP 44338 AppPortLBRule
FabricGatewayProbe TCP 19000 LBRule
FabricHttpGatewayProbe TCP 19080 LBHttpRule
SFReverseProxyProbe TCP 19081 LBSFReverseProxyRule
Load balancing rules
NAME LOAD BALANCING RULE BACKEND POOL HEALTH PROBE
AppPortLBRule AppPortLBRule (TCP/44338) LoadBalancerBEAddressPool AppPortProbe
LBHttpRule LBHttpRule (TCP/19080) LoadBalancerBEAddressPool FabricHttpGatewayProbe
LBRule LBRule (TCP/19000) LoadBalancerBEAddressPool FabricGatewayProbe
LBSFReverseProxyRule LBSFReverseProxyRule (TCP/19081) LoadBalancerBEAddressPool SFReverseProxyProbe
I have a Cluster certificate, ReverseProxy Certificate, and auth to the api through azure ad and in ARM
"fabricSettings": [
{
"parameters": [
{
"name": "ClusterProtectionLevel",
"value": "[parameters('clusterProtectionLevel')]"
}
],
"name": "Security"
},
{
"name": "ApplicationGateway/Http",
"parameters": [
{
"name": "ApplicationCertificateValidationPolicy",
"value": "None"
}
]
}
],
Not sure what else could be relevant, if you have any ideas/suggestions, those are really welcome
Edit : code for GetCertificate()
private X509Certificate2 GetCertificate()
{
var certificateBundle = Task.Run(async () => await GetKeyVaultClient()
.GetCertificateAsync(Environment.GetEnvironmentVariable("KeyVaultCertifIdentifier")));
var certificate = new X509Certificate2();
certificate.Import(certificateBundle.Result.Cer);
return certificate;
}
private KeyVaultClient GetKeyVaultClient() => new KeyVaultClient(async (authority, resource, scope) =>
{
var context = new AuthenticationContext(authority, TokenCache.DefaultShared);
var clientCred = new ClientCredential(Environment.GetEnvironmentVariable("KeyVaultClientId"),
Environment.GetEnvironmentVariable("KeyVaultSecret"));
var authResult = await context.AcquireTokenAsync(resource, clientCred);
return authResult.AccessToken;
});
Digging into your code I've realized that there is nothing wrong with it except one thing. I mean, as you use Kestrel, you don't need to set up anything extra in the AppManifest as those things are for Http.Sys implementation. You don't even need to have an endpoint in the ServiceManifest(although recommended) as all these things are about URL reservation for the service account and SSL binding configuration, neither of which is required with Kestrel.
What you do need to do is to use IPAddress.IPv6Any while you configure SSL. Aside the fact that it turns out to be the recommended way which allows you to accept both IPv4 and IPV6 connections, it also does a 'correct' endpoint registration in the SF. See, when you use IPAddress.Any, you'll get the SF setting up an endpoint like https://0.0.0.0:44338, and that's how the reverse proxy will try to reach the service which obviously wouldn't work. 0.0.0.0 doesn't correspond to any particular ip, it's just the way to say 'any IPv4 address at all'. While when you use IPAddress.IPv6Any, you'll get a correct endpoint mapped to the vm ip address that could be resolved from within the vnet. You could see that stuff by yourself in the SF Explorer if you go down to the endpoint section in the service instance blade.

How to get remote client's IPV4 address from request object

I am using go-gin as server and rendering an html using the code like the following
func dashboardHandler(c *gin.Context) {
c.HTML(200, "dashboard", gin.H{
"title": "Dashboard"
})
Along with title I want to pass the remote client's IPV4 address as well. I tried using the following code to get the IP address but for localhost it gives me ::1:56797 as output. My server is running on localhost:8080
ip, port, err := net.SplitHostPort(c.Request.RemoteAddr)
fmt.Println(ip + ":" + port)
if err != nil {
fmt.Println(err.Error())
}
I followed Correct way of getting Client's IP Addresses from http.Request (Golang) for reference. Is there any way I get the IPV4 address from the request?
you can use this function to get the ip and user agent, but it will give a bracket character if you are trying from localhost but if you try from somewhere else it will work.
func GetIPAndUserAgent(r *http.Request) (ip string, user_agent string) {
ip = r.Header.Get("X-Forwarded-For")
if ip == "" {
ip = strings.Split(r.RemoteAddr, ":")[0]
}
user_agent = r.UserAgent()
return ip, user_agent
}

Build Url using ServletUriComponentsBuilder without port number in Spring

I am using ServletUriComponentsBuilder in my service class to build some urls but the problem is that it includes port number also where the servlet container is running, this is a problem when I am deploying my app on production behind a proxy server which is supposed to run only on port 80
Code that I am using is:
String sUri = ServletUriComponentsBuilder.fromCurrentContextPath().path("/student/edit/" + st.getId()).build().toUriString();
While c:url that I am using in JSP is working perfectly fine, it do not include port number. Is there any way by which ServletUriComponentsBuilder also start detects whether it needs to include port number or not.
Means if the application start on port 8080 then it can include port number but when app is accessed from port 80 then do not include?
Whats happening: If my tomcat is running on port 8080 while I have proxy server in place which serves request on port 80, but urls built by ServletUriComponentsBuilder still appends port 8080 after host, I need it to be 80
Take a look at ServletUriComponentsBuilder#fromRequest:
String scheme = request.getScheme();
int port = request.getServerPort();
String host = request.getServerName();
String header = request.getHeader("X-Forwarded-Host");
if (StringUtils.hasText(header)) {
String[] hosts = StringUtils.commaDelimitedListToStringArray(header);
String hostToUse = hosts[0];
if (hostToUse.contains(":")) {
String[] hostAndPort = StringUtils.split(hostToUse, ":");
host = hostAndPort[0];
port = Integer.parseInt(hostAndPort[1]);
}
else {
host = hostToUse;
}
}
....
Especially the line
String header = request.getHeader("X-Forwarded-Host");
will do the trick. All you have to do is set X-Forwarded-Host in your proxy server and start using ServletUriComponentsBuilder#fromRequest instead of ServletUriComponentsBuilder#fromCurrentContextPath. Your url should contain your public proxy hostname and no port.
This is a bug from this method:
public static ServletUriComponentsBuilder fromRequest(HttpServletRequest request) {
String scheme = request.getScheme();
int port = request.getServerPort();
String host = request.getServerName();
String header = request.getHeader("X-Forwarded-Host");
if (StringUtils.hasText(header)) {
String[] hosts = StringUtils.commaDelimitedListToStringArray(header);
String hostToUse = hosts[0];
if (hostToUse.contains(":")) {
String[] hostAndPort = StringUtils.split(hostToUse, ":");
host = hostAndPort[0];
port = Integer.parseInt(hostAndPort[1]);
}
else {
host = hostToUse;
}
}
ServletUriComponentsBuilder builder = new ServletUriComponentsBuilder();
builder.scheme(scheme);
builder.host(host);
if ((scheme.equals("http") && port != 80) || (scheme.equals("https") && port != 443)) {
builder.port(port);
}
builder.pathFromRequest(request);
builder.query(request.getQueryString());
return builder;
}
If X-Forwarded-Host is filled, and there is no port it's because we are on the port 80. But the else
else {
host = hostToUse;
}
So this case is a mixed case where the host is readed from the X-Forwarded-Host value but the port is read directly from the request (and it's the request the apache use to call tomcat).
We are working this issue here and did'nt find any alternative way to write a new UriComponentsBuilder (ok... maybe juste extends!)

Https redirect and login cookies on Heroku with Play Framework

I have a Play! framework Heroku project that has three deployments. One for running my dev machine, one for beta on Heroku, and one for production on Heroku. Their http and https urls are as follows:
DEV BETA PRODUCTION
HTTP URL | http://localhost:9000 http://domain-beta.herokuapps.com http://www.domain.com
HTTPS URL | https://localhost:9443 https://domain-beta.herokuapps.com https://secure.domain.com
HTTPS Type | My cert Piggyback (using Heroku's cert) Hostname-based SSL (using my cert)
I also have a class HttpsRequired that has methods for requiring HTTPS, and for redirecting back to HTTP (thanks to this post for the help).
public class HttpsRequired extends Controller {
/** Called before every request to ensure that HTTPS is used. */
#Before
public static void redirectToHttps() {
//if it's not secure, but Heroku has already done the SSL processing then it might actually be secure after all
if (!request.secure && request.headers.get("x-forwarded-proto") != null) {
request.secure = request.headers.get("x-forwarded-proto").values.contains("https");
}
//redirect if it's not secure
if (!request.secure) {
String url = redirectHostHttps() + request.url;
System.out.println("Redirecting to secure: " + url);
redirect(url);
}
}
/** Renames the host to be https://, handles both Heroku and local testing. */
#Util
public static String redirectHostHttps() {
if (Play.id.equals("dev")) {
String[] pieces = request.host.split(":");
String httpsPort = (String) Play.configuration.get("https.port");
return "https://" + pieces[0] + ":" + httpsPort;
} else {
if (request.host.endsWith("domain.com")) {
return "https://secure.domain.com";
} else {
return "https://" + request.host;
}
}
}
/** Renames the host to be https://, handles both Heroku and local testing. */
#Util
public static String redirectHostNotHttps() {
if (Play.id.equals("dev")) {
String[] pieces = request.host.split(":");
String httpPort = (String) Play.configuration.get("http.port");
return "http://" + pieces[0] + ":" + httpPort;
} else {
if (request.host.endsWith("domain.com")) {
return "http://www.domain.com";
} else {
return "http://" + request.host;
}
}
}
}
I modified Secure.login() to call HttpsRequired.redirectToHttps() before it runs, to ensure that all passwords are submitted encrypted. Then, in my Security.onAuthenticated(), I redirect to the homepage on standard HTTP.
This works great on my dev and beta deployments, but in production all of my HTTP requests are redirected to the HTTPS login page. I can still use the whole site in HTTPS, but I want regular HTTP to work too.
All of my pages are protected as members-only and require users to login, using the #With(Secure.class) annotation. I'm thinking that it must be related to the fact that the login happens at secure.domain.com instead of www.domain.com, and that they somehow generate different cookies.
Is there a way to change the login cookie created at secure.domain.com to make it work at www.domain.com?
Check out the documentation for the setting for default cookie domain.
http://www.playframework.org/documentation/1.2.4/configuration#application.defaultCookieDomain
It explains how you can set a cookie to work across all subdomains.
application.defaultCookieDomain
Enables session/cookie sharing between subdomains. For example, to
make cookies valid for all domains ending with ‘.example.com’, e.g.
foo.example.com and bar.example.com:
application.defaultCookieDomain=.example.com

Resources