X-Forwarded-For header doesn't contain client IP

[jeroenvisser101]

With a standard Rails app deployed, the remote address is taken from the X-Forwarded-For header in case the request originates from a internal IP (internal IP addresses are trusted proxies by default).

However, when deployed on Deis, both REMOTE_ADDR and X-Forwarded-For contain internal IP addresses:

HTTP_X_FORWARDED_FOR: 10.132.0.2 # Not sure who owns this IP
REMOTE_ADDR: 10.0.0.5 # The IP of the deis-router pod

None of which are the actual client IP. Is there a way to make sure the client IP is available from the application? Or is this a bug?

(my deis cluster is deployed on GKE)

[krancour]

@jeroenvisser101 thanks for reporting this. Most of this particular functionality was tested with AWS ELBs and it worked very nicely there. I've gotten more familiar with GCP today to get a handle on where some of the issues may be lurking and I found a variety of things. There's a lot going on here, so bear with me... feel free to ask questions...

1. Because the router's service is of Type: LoadBalancer, and because you're running on GKE, the Kubernetes "cloud provider" for GKE provisions a load balancer for you in GCP. This load balancer is a TCP load balancer; not an HTTP load balancer. The TCP load balancer does not support PROXY protocol. If it did, you could get the router to learn the real client IP from PROXY protocol by enabling that on the router as well. Like I said... the LB on the Google end seems not to support this...
2. So if you care about the real client IP, you're, sadly, left needing to disregard this automatically provisioned TCP load balancer and provision your own HTTP load balancer instead. Have you done that already? An HTTP load balancer does set / append to the X-Forwarded-For header.
3. Assuming no. 2 above is configured correctly, the inbound request to the router actually has multiple, comma-delimited values in the X-Forwarded-For header. The first is the real, end-client's IP. The second is (oddly) the public IP of the load balancer. Remember, this list reflects forwarded for. So this means there's one other hop between the load balancer and the router. I'm not 100% clear on what that is, but it's a private IP in the 10.x.x.x range and I'm chalking it up to some aspect of GCP networking that I don't understand.

Lastly, and not really an issue, per se. Keep in mind that the router.deis.io/nginx.proxyRealIpCidr annotation maps to a set_real_ip_from directive in the generated Nginx configuration and this tells Nginx's real IP module what proximate clients, identified by IP/CIDR range, are trustworthy in terms of accurately the end client's IP. The default value for this is 10.0.0.0/8, which is generally fine, but not in this case because the load balancer's IP should also be in the list. (fwiw, this set_real_ip_from directive is what prevents you and me from setting X-Forwarded-For headers ourselves to masquerade as someone else.)

So where does this lead me...

If you followed everything above, there is one little fix the router needs. I need to make the router.deis.io/nginx.proxyRealIpCidr annotation accept multiple, comma-delimited values to indicate multiple IP ranges that are considered trustworthy in regard to setting the PROXY protocol or X-Forwarded-For headers... this would het translated into multiple set_real_ip_from directives in the generated Nginx config. In conjunction with the real_ip_recursive on; directive, this would allow the router to backtrack all the way to the first (leftmost) IP in the (incoming) X-Forwarded-For header, which then is assumed to be the "real IP," and is thus the IP set in the (outbound) request to your application container.

When all is said and done, all I can say is that it will be possible to get what you want. It's won't necessarily be easy. You will have to define the load balancer yourself as I described above and you will also need to specify a value for the router.deis.io/nginx.proxyRealIpCidr annotation on the router's RC.

Going to use what's left of the afternoon to try and bang this out.

[jeroenvisser101]

@krancour thanks for your very elaborate explanation – I appreciate you taking the time for this.

Because the router's service is of Type: LoadBalancer, and because you're running on GKE, the Kubernetes "cloud provider" for GKE provisions a load balancer for you in GCP. This load balancer is a TCP load balancer; not an HTTP load balancer.

Right, however, GCP supports both TCP and HTTP(S) loadbalancing, also with k8s, and it has support for host-based routing (thus in theory not needing deis-router, if I'm understanding correctly). Would that be a viable alternative to set up a deis cluster? Say you have to replace deis-router with a GCP-specific router, say deis-gcp-router, which instead of deploying a Nginx proxy, will deploy a very detailed HTTP-loadbalancer configuration.

I'm not sure how this will work when you start a new deploy regarding HA, but I assume Google has thought of this and allows for changing the HTTP LB while serving traffic. (plus, you reference the service, so I guess it's all done automagically?)

It's also important to note that, if you deploy a basic Rails application on a GKE cluster, it would allow IP spoofing by default, which I think is a serious issue, and should at least be documented.

[krancour]

Ok. There's as much going on in your follow up as in my post. ;) Let me try and respond to each question as best I can:

Right, however, GCP supports both TCP and HTTP(S) loadbalancing

Yes. It absolutely does. The only "problem" is that when k8s creates a load balancer on your behalf, it's a TCP load balancer. fwiw, it does the same thing on AWS. If there's an option to compel k8s to create an HTTP load balancer instead, I'm unfamiliar with it, but would be eager to learn about it if it's something I have simply overlooked. Ultimately, this puts users in a position of having to define their own HTTP load balancer. On AWS, by contrast, it's a little easier to manager because you just modify the listeners on the ELB.

and it has support for host-based routing (thus in theory not needing deis-router, if I'm understanding correctly). Would that be a viable alternative to set up a deis cluster?

I'm not familiar with every nook and cranny of GCP networking. It's possible you can get away without using a deis-router. Even if possible, I'm not sure you'd truly want to not use it. The router does a lot more than just directing traffic. It's really your web server as well. Having done plenty of Rails in a past life, you usually want to deploy your Rails app behind a proper web server like Nginx so that non-application specific concerns like HTTPS termination or gzip compression can be handled centrally and/or by something more performant than Ruby. If you ditch the deis router, your application becomes responsible for those things.

I'm not sure how this will work when you start a new deploy regarding HA

I'm not sure either. I've heard from @rimusz that he's had success with Deis on ubernetes-light, which I think is an option within GKE now. He may not have attempted any advanced load balancer configuration. I'm not sure. I'd let him comment on what he did.

but I assume Google has thought of this and allows for changing the HTTP LB while serving traffic

Don't assume that. :-p

It's also important to note that, if you deploy a basic Rails application on a GKE cluster, it would allow IP spoofing by default

Another reason the deis router is working in your favor. Configured properly (after my fix) it will prevent this and your application need not worry about it.

EDIT: Misread the above. Do you know how it's preventing that?

[jeroenvisser101]

@krancour I believe Ingress-type services introduced in v1.1 would resolve this. And I think it'd be the correct spec, since it's actual HTTP traffic, and it'd also use the correct loadbalancer on GCP (and it should do the same on AWS, and any other for that matter).

[krancour]

@jeroenvisser101, the team actually made a deliberate decision to not use ingress resources because they are still in beta as of 1.2-- which is the minimum version of k8s we are requiring for our components.

We very much do wish we could use them, but the rationale for not doing so was as follows: The possibility that breaking changes may occur to that API might be considered remote by some, but it's not guaranteed. (Personally, I think it's in enough flux that I wouldn't take it for granted at all.) If we depend on ingress resources, any such breaking changes on the k8s end would require router and controller to detect the underlying version of k8s and handle ingress differently for different versions. We didn't want that.

The approach we use currently, with the "routable services" is actually inspired by the ingress type. They are "poor man's ingresses," if you will, which makes the router a "poor man's ingress controller." Its architecture is actually inspired by the nginx-alpha ingress controller available in the official kubernetes/contrib repository, although it's received far more polish and has had all the features of the v1.x router ported forward.

and it'd also use the correct loadbalancer on GCP

Also for as little as we've used ingress resources up to this point, (we've played enough just to be familiar), it's news to me that doing so would somehow create HTTP load balancers instead of TCP load balancers. I'm going to experiment with that to find out for sure, but something about that doesn't ring true. Even taking Deis Workflow and/or Router out of the picture, my understanding of the ingress type is that an "ingress controller" is meant to direct traffic within the cluster based on the rules encapsulated by those resources, however, the only load balancer external to the cluster (e.g. an ELB or GCP load balancer) would be the one for the ingress controller itself. Again... I'm curious enough to experiment with that and see. Ultimately, however, what we do want is a single external-to-the-cluster load balancer through which all traffic into the cluster flows, and ideally, we actually do want that to be a TCP load balancer (GCP) or use TCP-based listeners (ELB). The first reason for this is that it allows HTTPS termination to be configured 100% through Deis and has the SSL termination happening within the cluster instead of outside it. The second reason is that load balancers aren't free (at least not on every cloud provider). In AWS, for instance, I believe they cost about the same per month as an m1.small. For a cluster that hosts many different applications, there can be significant cost savings to users by funneling all inbound traffic through one load balancer.

[jeroenvisser101]

@krancour Sorry for the late reply. I understand, but it's blocking us from using it for our software (we need geolocation based on user's IP).

it's news to me that doing so would somehow create HTTP load balancers instead of TCP load balancers.

It seems to create a L7 loadbalancer on GCE, and since Ingress options support TLS termination, they should be HTTP(s) loadbalancers, but the exact implementation of a Ingress resource differs per provider.

my understanding of the ingress type is that an "ingress controller" is meant to direct traffic within the cluster based on the rules encapsulated by those resources, however, the only load balancer external to the cluster (e.g. an ELB or GCP load balancer) would be the one for the ingress controller itself.

I'm not sure about this, but my understanding is that a ingress resource is like a loadbalancer configuration object. The provider defines the implementation, but they should work similarly.

The second reason is that load balancers aren't free (at least not on every cloud provider). In AWS, for instance, I believe they cost about the same per month as an m1.small. For a cluster that hosts many different applications, there can be significant cost savings to users by funneling all inbound traffic through one load balancer.

I think this would be a great argument to choose for making it configurable. I think that if you application doesn't require IP's to be logged/checked/worked with, you could use the default just fine.

The first reason for this is that it allows HTTPS termination to be configured 100% through Deis and has the SSL termination happening within the cluster instead of outside it.

I believe that, since Ingress can be configured through the kube api, the first part would still be true for ingress resources. The second part would not be the case when using a ingress object, but I'm not sure what the advantage would be of this?

[jeroenvisser101]

In addition to that, I was wondering what router.deis.io/nginx.useProxyProtocol is and if it could help in this scenario.

[krancour]

@jeroenvisser101, I've dug into this quite far. What I've found, unfortunately, is that GKE isn't really an ideal place for anyone wishing to leverage these sort of network options. It would take pages and pages to communicate all I know at this point, but I'll try to hit the highlights here.

When you create a service of type LoadBalancer (like the router's service), a TCP load balancer gets created in GCP. Unlike an ELB, this does not support PROXY protocol. Ordinarily, using PROXY protocol would be an option for the router to learn the end user's real IP address, but it's not applicable here.

This leaves your only option (on GKE) being to manually create an HTTP/S load balancer and connect it up to the router's node ports. When you do this, the HTTP/S router will correctly set the X-Forwarded-For HTTP header. In order for the router to trust the contents of that header, the load balancer's IP needs to be added to the router RC's router.deis.io/nginx.proxyRealIpCidrs annotation. e.g.:

annotations:
  router.deis.io/nginx.proxyRealIpCidrs: 10.0.0.0/8,1.2.3.4/32

A very unfortunate side effect of this arrangement is that since all the traffic passing through the HTTP/S load balancer is handled at L7, HTTPS (if in use) needs to be terminated at the load balancer and not at the router. So that's another thing you'd need to configure manually. It gets worse...

Each listener on your load balancer can support a single SSL certificate. If all the domains you use for the apps in your cluster are subdomains of one single domain, you can install a wildcard cert for that domain and you'll be fine. If however, you have custom / "vanity" domains you wish to use for some applications, each of those is going to end up needing its own load balancer.

Terminating SSL at the load balancer is not without its drawbacks. In some regulated industries (e.g. healthcare) that is most likely to be a no no.

Unfortunately, there's no getting around any of what I've described above. All I can say is that in AWS, ELBs don't have these same limitations. (Over there, the only thing you need to do manually is enable PROXY protocol on the ELB after it's created. And I acutally have a PR open to Kubernetes that will make even this one step unnecessasry in the future-- PROXY protocol can be enabled on an ELB by setting an annotation-- which the router's svc will do.)

We have utilized informal escalation channels on our end to request that GCP add support for PROXY protocol, but I have no information regarding how likely that is to happen or what the timeframe looks like. Adding support for PROXY protocol would fix all of this.

Also, just to put closure on the ingress stuff...

Ingresses are just resources. What happens in response to those resource definitions is the responsibility of the ingress controller-- implementation and behavior may vary widely. The ingress controller built into a GKE cluster will create an L7 load balancer as you indicated, however:

1. There's no production worthy ingress controller implementation available anywhere besides GKE. Our router is the closest thing there is-- with the only difference from a proper ingress controller being that we use annotations on services instead of actual ingress resources. As I indicated, we aren't straying from that decision at this time as ingress remains in beta right now and we expect to be GA before k8s 1.3.
2. If you could make use of Ingress somehow, it doesn't let you off the hook for most of the difficulties I described with the HTTP/S load balancers in GCP. If you want, however, you want to experiment with this, there should be nothing stopping you from manually defining ingress resources (adding them to a fork for the workflow chart even).

I hope this all helps a little.

[jeroenvisser101]

@krancour thanks, I really appreciate the time you're putting in this. I understand most of the difficulties. I'll reserve some time next week to experiment with things and see if it suits our needs. I'll update this if I've found anything useful. I'll also be attending AWS meetup next week to see if that might be an option, and I've reached out to Google Cloud to see if they've got any input on this.

Thanks again for all the explanation. (Do you have a link to that PR so I can subscribe to it?)