Last modified September 28, 2018
Advanced Ingress Configuration
The NGINX-based Ingress Controller running inside your cluster has additional configuration options and features that can be customized. The functionality is split into two categories:
- Per-Service options in each Ingress’ YAML definition either directly or via Annotations.
- Global options that influence all Ingresses of a cluster via a ConfigMap.
You can aggregate several Ingress rules into a single Ingress definition like following:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: <ingress-name> spec: rules: - host: <yourchoice1>.<cluster-id>.k8s.gigantic.io http: paths: - path: / backend: serviceName: <service1-name> servicePort: <service1-port> - host: <yourchoice2>.<cluster-id>.k8s.gigantic.io http: paths: - path: / backend: serviceName: <service2-name> servicePort: <service2-port>
Note: If you are using TLS you also need each of the hosts in the
tls section (see below) of the yaml.
Path Based Fanout
You can route an Ingress to different Services based on the path:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: <ingress-name> spec: rules: - host: <yourchoice>.<cluster-id>.k8s.gigantic.io http: paths: - path: /foo backend: serviceName: <service1-name> servicePort: <service1-port> - path: /bar backend: serviceName: <service2-name> servicePort: <service2-port>
Note: Your applications need to be capable of running on a non-root path either by default or by setting the base path in their configuration.
If your cluster has TLS enabled, you can terminate TLS either in your application itself by enabling SSL passthrough or let the Ingress Controller terminate for you.
Warning: This feature was disabled by default in Nginx ingress controller managed by Giant Swarm. Reason is a potential crash of internal TCP proxier. We recommend to terminate TLS in ingress controller instead.
For SSL passthrough you need to set an annotation and enable TLS for the host:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: <ingress-name> annotations: nginx.ingress.kubernetes.io/ssl-passthrough: "true" spec: tls: - hosts: - <yourchoice>.<cluster-id>.k8s.gigantic.io rules: - host: <yourchoice>.<cluster-id>.k8s.gigantic.io http: paths: - path: / backend: serviceName: <service-name> servicePort: <service-port>
Note: SSL passthrough cannot work with path based routing based on the nature of SSL.
Terminating TLS in Ingress Controller
For terminating TLS in the Ingress Controller you need to first create a TLS secret containing your certificate and private key in the same namespace as the Ingress object:
apiVersion: v1 kind: Secret type: kubernetes.io/tls metadata: name: mytlssecret data: tls.crt: <base64 encoded cert> tls.key: <base64 encoded key>
Note: the data keys must be named
Referencing this secret in an Ingress will tell the Ingress Controller to secure the channel from the client to the loadbalancer using TLS:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: <ingress-name> spec: tls: - hosts: - <yourchoice>.<cluster-id>.k8s.gigantic.io secretName: mytlssecret rules: - host: <yourchoice>.<cluster-id>.k8s.gigantic.io http: paths: - path: / backend: serviceName: <service-name> servicePort: <service-port>
Warning: When enabling
TLS with the NGINX Ingress Controller, some more configuration settings become important. Notably
HSTS will be enabled by default with a duration of six month for your specified domain. Once a browser retrieved these
HSTS instructions it will refuse to read any unencrypted resource from that domain and un-setting
HSTS on your server will not have any affect on that browser for half a year. So you might want to disable this at first to avoid unwanted surprises. Please contact our support team to find out details on how to disable HSTS in your cluster.
Tip: If you want to use Let’s Encrypt certificates with your domains you can automate their creation and renewal with the help of cert-manager. After configuring cert-manager there is only an annotation with your Ingresses needed and your web page will be secured by a valid
The Ingress Controller includes support for adding authentication to an Ingress rule. You have the choice between basic or digest http authentication types.
First, you need to create a file called
auth containing your usernames and passwords (one per line). You can do this either by using the
htpasswd command line tool (like in the following example) or an online htpasswd generator.
$ htpasswd -c auth foo1 New password: <bar> New password: Re-type new password: Adding password for user foo1
You can add users to the same file with:
$ htpasswd auth foo2 New password: <bar> New password: Re-type new password: Adding password for user foo2
Next, we create a secret containing our
apiVersion: v1 kind: Secret type: Opaque metadata: name: myauthsecret data: auth: <base64 encoded auth>
Last, we create the Ingress with the according annotations:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: <ingress-name> annotations: # type of authentication [basic|digest] nginx.ingress.kubernetes.io/auth-type: basic # name of the secret that contains the user/password definitions nginx.ingress.kubernetes.io/auth-secret: myauthsecret # message to display with an appropiate context why the authentication is required nginx.ingress.kubernetes.io/auth-realm: "Authentication Required - foo" spec: rules: - host: <yourchoice>.<cluster-id>.k8s.gigantic.io http: paths: - path: / backend: serviceName: <service-name> servicePort: <service-port>
To use an existing service that provides authentication the Ingress rule can be annotated with
nginx.ingress.kubernetes.io/auth-url to indicate the URL where the HTTP request should be sent. Additionally it is possible to set
nginx.ingress.kubernetes.io/auth-method to specify the HTTP method to use (GET or POST).
This functionality is based on the auth_request module, which expects a
2xx response code from the external service if the access is allowed and
403 if denied.
To enable Cross-Origin Resource Sharing (CORS) in an Ingress rule add the annotation
In some scenarios the exposed URL in the backend service differs from the specified path in the Ingress rule. Without a rewrite any request will return 404. To circumvent this you can set the annotation
ingress.kubernetes.io/rewrite-target to the path expected by the service.
This can for example be used together with path based routing, when the application expects to be on
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: <ingress-name> annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - host: <yourchoice>.<cluster-id>.k8s.gigantic.io http: paths: - path: /foo backend: serviceName: <service-name> servicePort: <service1port>
If the application contains relative links it is possible to add an additional annotation
ingress.kubernetes.io/add-base-url that will prepend a
base tag in the header of the returned HTML from the backend.
ingress.kubernetes.io/limit-rps define a limit on the connections that can be opened by a single client IP address. This can be used to mitigate DDoS Attacks.
nginx.ingress.kubernetes.io/limit-connections: number of concurrent connections allowed from a single IP address.
nginx.ingress.kubernetes.io/limit-rps: number of connections that may be accepted from a given IP each second.
If you specify both annotations in a single Ingress rule,
limit-rps takes precedence.
By default NGINX uses
http to reach the services. Adding the annotation
nginx.ingress.kubernetes.io/secure-backends: "true" in the Ingress rule changes the protocol to
Server-side HTTPS enforcement through redirect
By default the controller redirects (301) to
HTTPS if TLS is enabled for that Ingress. If you want to disable that behaviour, you can use the
nginx.ingress.kubernetes.io/ssl-redirect: "false" annotation.
Whitelist source range
You can specify the allowed client IP source ranges through the
nginx.ingress.kubernetes.io/whitelist-source-range annotation. The value is a comma separated list of CIDRs, e.g.
Note: Adding an annotation to an Ingress rule overrides any global restrictions set in the NGINX Ingress Controller.
Custom max body size
A 413 error will be returned to the client when the size in a request exceeds the maximum allowed size of the client request body. This size can be configured by the parameter
client_max_body_size and is set to
1m (1 Megabyte) by default.
To configure this setting globally for all Ingress rules, the
proxy-body-size value may be set in the NGINX ConfigMap.
To use custom values in a specific Ingress add following annotation:
nginx.ingress.kubernetes.io/affinity enables and sets the affinity type in all upstreams of an Ingress. This way, a request will always be directed to the same upstream server.
If you use the
cookie type you can also specify the name of the cookie that will be used to route the requests with the annotation
nginx.ingress.kubernetes.io/session-cookie-name. The default is to create a cookie named
nginx.ingress.kubernetes.io/session-cookie-hash defines which algorithm will be used to hash the used upstream. Default value is
md5 and possible values are
index option is not hashed, an in-memory index is used instead, it’s quicker and the overhead is shorter. Warning: The matching against the upstream servers list is inconsistent. So, at reload, if upstreams servers have changed, index values are not guaranted to correspond to the same server as before! Use with caution and only if you need to!
The NGINX Ingress Controller creates an NGINX configuration file. You can directly pass chunks of configuration, so-called configuration snippets, into any ingress manifest. These snippets will be added to the NGINX configuration.
Here is an example adding an
Expires header to every response:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: myingress namespace: mynamespace annotations: nginx.ingress.kubernetes.io/configuration-snippet: | expires 24h; spec: rules: - host: host.example.com http: paths: - backend: serviceName: http-svc servicePort: 80 path: /
Make sure to use the exact annotation scheme
nginx.ingress.kubernetes.io/configuration-snippet in the
metadata section of the manifest.
Check out the ingress-nginx repository for more information.
Global (per cluster) options
Your Giant Swarm installation comes with some global defaults for Ingress Controller configuration. However, you can override these defaults by setting your per cluster configuration in form of a ConfigMap named
nginx-ingress-controller-user-values located in your
Note: This feature is only available in more recent cluster versions. To check if your cluster version supports customization of the ConfigMap, you can check if the above-mentioned ConfigMap is present.
Warning: Please do not edit any of the other nginx ingress related ConfigMaps. Only the user ConfigMap is safe to edit.
$ kubectl -n kube-system get cm nginx-ingress-controller-user-values NAME DATA AGE nginx-ingress-controller-user-values 0 11m
The official documentation of the NGINX Ingress Controller contains an overview of the configuration options and their defaults.
However, we currently only support a subset of those options, which we default as follows. If you need any other upstream documented option added to this list, please contact support.
data: disable-access-log: "false" enable-vts-status: "true" error-log-level: "error" hsts: "false" large-client-header-buffers: "4 8K" log-format-upstream: "$status $body_bytes_sent $http_referer" server-name-hash-bucket-size: "1024" server-name-hash-max-size: "1024" server-tokens: "false" worker-processes: "4" enable-underscores-in-headers: "" proxy-buffers-size: "" proxy-buffers: "" vts-default-filter-key: ""
Warning: We also allow setting
use-proxy-protocol: "true"/"false". This setting always applies globally for the
nginx-ingress-controller. All applications providing services behind ingresses need to understand this protocol or they will fail. Furthermore, the load balancer in front of the ingress controller also needs to be set up correctly. So currently, customizing setting only makes sense on bare metal installations and will require a matching configuration on the load balancers.
On cluster creation the ConfigMap is empty and above defaults will be applied to the final Ingress Controller deployment. To override any of the above values, you just need to add the respective line in the data field of the user ConfigMap.
When you want to have the default server on the nginx controller support TLS you need to provide a certificate. This is configured using the flag
--default-ssl-certificate. Now you can provide this value in the
user-value configmap to force the component to be restarted with the provided certificate. The value of the property should be the namespace and secret name which holds the certificate content.
data: default-ssl-certificate: "default/foo-tls"
Custom annotation prefix
By default we use the standard annotation prefix
nginx.ingress.kubernetes.io in the ingress controller. In case the customer needs to have a specific one this can be done via the ‘user-values’ configmap. This is recommended when there is more than one ingress controller. So in the ingress resource the prefix can be used to distinguish between controllers.
data: annotations-prefix: "custom.prefix.io"