Last modified March 1, 2017

Advanced Ingress Configuration

The NGINX-based Ingress Controller running inside your cluster has additional configuration options and features that can be customized. The functionality is split into three categories:

Per-Service Options

Aggregating Ingresses

You can aggregate several Ingress rules into a single Ingress definition like following:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: <ingress-name>
spec:
  rules:
  - host: <yourchoice1>.<cluster-id>.k8s.gigantic.io
    http:
      paths:
      - path: /
        backend:
          serviceName: <service1-name>
          servicePort: <service1-port>
  - host: <yourchoice2>.<cluster-id>.k8s.gigantic.io
    http:
      paths:
      - path: /
        backend:
          serviceName: <service2-name>
          servicePort: <service2-port>

Note: If you are using TLS you also need each of the hosts in the tls section (see below) of the yaml.

Path Based Fanout

You can route an Ingress to different Services based on the path:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: <ingress-name>
spec:
  rules:
  - host: <yourchoice>.<cluster-id>.k8s.gigantic.io
    http:
      paths:
      - path: /foo
        backend:
          serviceName: <service1-name>
          servicePort: <service1-port>
      - path: /bar
        backend:
          serviceName: <service2-name>
          servicePort: <service2-port>

Note: Your applications need to be capable of running on a non-root path either by default or by setting the base path in their configuration.

TLS

If your cluster has TLS enabled, you can terminate TLS either in your application itself by enabling SSL passthrough or let the Ingress Controller terminate for you.

SSL Passthrough

For SSL passthrough you need to set an annotation and enable TLS for the host:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: <ingress-name>
  annotations:
    ingress.kubernetes.io/ssl-passthrough: "true"
spec:
   tls:
   - hosts:
     - <yourchoice>.<cluster-id>.k8s.gigantic.io
  rules:
  - host: <yourchoice>.<cluster-id>.k8s.gigantic.io
    http:
      paths:
      - path: /
        backend:
          serviceName: <service-name>
          servicePort: <service-port>

Note: SSL passthrough cannot work with path based routing based on the nature of SSL.

Terminating TLS in Ingress Controller

For terminating TLS in the Ingress Controller you need to first create a TLS secret containing your certificate and private key in the same namespace as the Ingress object:

apiVersion: v1
kind: Secret
type: kubernetes.io/tls
metadata:
  name: mytlssecret
data:
  tls.crt: <base64 encoded cert>
  tls.key: <base64 encoded key>

Note: the data keys must be named tls.crt and tls.key!

Referencing this secret in an Ingress will tell the Ingress Controller to secure the channel from the client to the loadbalancer using TLS:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: <ingress-name>
spec:
   tls:
   - hosts:
     - <yourchoice>.<cluster-id>.k8s.gigantic.io
     secretName: mytlssecret
  rules:
  - host: <yourchoice>.<cluster-id>.k8s.gigantic.io
    http:
      paths:
      - path: /
        backend:
          serviceName: <service-name>
          servicePort: <service-port>

Authentication

The Ingress Controller includes support for adding authentication to an Ingress rule. You have the choice between basic or digest http authentication types.

First, you need to create a file called auth containing your usernames and passwords (one per line). You can do this either by using the htpasswd command line tool (like in the following example) or an online htpasswd generator.

$ htpasswd -c auth foo1
New password: <bar>
New password:
Re-type new password:
Adding password for user foo1

You can add users to the same file with:

$ htpasswd auth foo2
New password: <bar>
New password:
Re-type new password:
Adding password for user foo2

Next, we create a secret containing our auth file:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: myauthsecret
data:
  auth: <base64 encoded auth>

Last, we create the Ingress with the according annotations:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: <ingress-name>
  annotations:
    # type of authentication [basic|digest]
    ingress.kubernetes.io/auth-type: basic
    # name of the secret that contains the user/password definitions
    ingress.kubernetes.io/auth-secret: myauthsecret
    # message to display with an appropiate context why the authentication is required
    ingress.kubernetes.io/auth-realm: "Authentication Required - foo"
spec:
  rules:
  - host: <yourchoice>.<cluster-id>.k8s.gigantic.io
    http:
      paths:
      - path: /
        backend:
          serviceName: <service-name>
          servicePort: <service-port>

External Authentication

To use an existing service that provides authentication the Ingress rule can be annotated with ingress.kubernetes.io/auth-url to indicate the URL where the HTTP request should be sent. Additionally it is possible to set ingress.kubernetes.io/auth-method to either GET (default) or POST and ingress.kubernetes.io/auth-send-body to true or false (default).

This functionality is based on the auth_request module, which expects a 2xx response code from the external service if the access is allowed and 401 or 403 if denied.

CORS

To enable Cross-Origin Resource Sharing (CORS) in an Ingress rule add the annotation ingress.kubernetes.io/enable-cors: "true".

Rewrite

In some scenarios the exposed URL in the backend service differs from the specified path in the Ingress rule. Without a rewrite any request will return 404. To circumvent this you can set the annotation ingress.kubernetes.io/rewrite-target to the path expected by the service.

This can for example be used together with path based routing, when the application expects to be on /:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: <ingress-name>
  annotations:
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: <yourchoice>.<cluster-id>.k8s.gigantic.io
    http:
      paths:
      - path: /foo
        backend:
          serviceName: <service-name>
          servicePort: <service1port>

If the application contains relative links it is possible to add an additional annotation ingress.kubernetes.io/add-base-url that will prepend a base tag in the header of the returned HTML from the backend.

Rate limiting

The annotations ingress.kubernetes.io/limit-connections and ingress.kubernetes.io/limit-rps define a limit on the connections that can be opened by a single client IP address. This can be used to mitigate DDoS Attacks.

ingress.kubernetes.io/limit-connections: number of concurrent connections allowed from a single IP address.

ingress.kubernetes.io/limit-rps: number of connections that may be accepted from a given IP each second.

If you specify both annotations in a single Ingress rule, limit-rps takes precedence.

Secure backends

By default NGINX uses http to reach the services. Adding the annotation ingress.kubernetes.io/secure-backends: "true" in the Ingress rule changes the protocol to https.

Server-side HTTPS enforcement through redirect

By default the controller redirects (301) to HTTPS if TLS is enabled for that Ingress. If you want to disable that behaviour, you can use the ingress.kubernetes.io/ssl-redirect: "false" annotation.

Whitelist source range

You can specify the allowed client IP source ranges through the ingress.kubernetes.io/whitelist-source-range annotation. The value is a comma separated list of CIDRs, e.g. 10.0.0.0/24,172.10.0.1.

Note: Adding an annotation to an Ingress rule overrides any global restrictions set in the NGINX Ingress Controller.

Custom max body size

A 413 error will be returned to the client when the size in a request exceeds the maximum allowed size of the client request body. This size can be configured by the parameter client_max_body_size and is set to 1m (1 Megabyte) by default.

To configure this setting globally for all Ingress rules, the proxy-body-size value may be set in the NGINX ConfigMap.

To use custom values in a specific Ingress add following annotation:

ingress.kubernetes.io/proxy-body-size: 8m

Session Affinity

The annotation ingress.kubernetes.io/affinity enables and sets the affinity type in all Upstreams of an Ingress. This way, a request will always be directed to the same upstream server.

If you use the cookie type you can also specify the name of the cookie that will be used to route the requests with the annotation ingress.kubernetes.io/session-cookie-name. The default is to create a cookie named route.

The annotation ingress.kubernetes.io/session-cookie-hash defines which algorithm will be used to hash the used upstream. Default value is md5 and possible values are md5, sha1 and index.

The index option is not hashed, an in-memory index is used instead, it’s quicker and the overhead is shorter. Warning: The matching against the upstream servers list is inconsistent. So, at reload, if upstreams servers have changed, index values are not guaranted to correspond to the same server as before! Use with caution and only if you need to!

This feature is implemented by the third party module nginx-sticky-module-ng. The workflow used to define which upstream server will be used is explained in the module documentation (PDF).

Global Options

Your Ingress Controller can be further customized using a ConfigMap named ingress-controller-config-custom located in your kube-system namespace.

The official documentation of the NGINX Ingress Controller contains a list of allowed parameters as well as the default configuration options.

Custom NGINX Template

There’s a default nginx.tmpl mounted to the right path in your Ingress Controller. You can find it in your kube-system namespace as a ConfigMap called ingress-controller-config-template. You can edit the template to configure advanced options that are available in NGINX, but cannot be set via Ingress YAML or Custom ConfigMap like described above.

Note: Editing the template can result in a broken Ingress Controller, so handle with care. Furthermore, the template is tied to the Go code of the Ingress Controller. Do not change names in the variable $cfg.

For more information about the template syntax please refer to the upstream NGINX Controller Documentation

Further reading