Self managed Kubernetes with Nginx Ingress

In this tutorial we see how to use Nginx-Ingress on a single Kubernetes node. This tutorial is working on bare-metal and cloud instances. We will use Ubuntu as base but it should be similar on other systems.

Goals:

• Working Nginx Ingress
• Certificates generation with Let’s Encrypt
• Get the real visitor’s IP

Requirements:

• An updated Ubuntu system
• Kubernetes up and running on the private interface
• A private Network Interface
• Helm3 binary

For the private Network Interface, you can just add another IP to your primary interface. This network will be used for the Load Balancer.

We will use the 172.16.1.0/24 network and reserve 172.16.1.250 for the Load Balancer.

The Reverse Proxy

First we will need to install and configure Nginx as reverse Proxy. It will listen on our Public Interface and redirect the traffic to the Load Balancer.

curl -fsSL https://nginx.org/keys/nginx_signing.key | sudo apt-key add -
echo "deb http://nginx.org/packages/ubuntu `lsb_release -cs` nginx" \
    | sudo tee /etc/apt/sources.list.d/nginx.list
sudo apt update
sudo apt install nginx

Let’s configure the Reverse Proxy, in /etc/nginx/nginx.conf

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
  worker_connections  1024;
}

http {
  default_type  application/octet-stream;

  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';

  access_log  off;

  sendfile    on;
  tcp_nopush  on;
  tcp_nodelay on;

  keepalive_timeout  65;

  gzip  on;
  gzip_min_length 10240;
  gzip_proxied expired no-cache no-store private auth;
  gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/json application/xml;
  gzip_disable msie6;

  server_tokens off;
}

stream {
  server {
    listen 80;
    proxy_protocol on;
    proxy_pass 172.16.1.250:80;
  }
  server {
    listen 443;
    proxy_protocol on;
    proxy_pass 172.16.1.250:443;
  }
}

Don’t forget to restart Nginx.

service nginx restart

Configure the Load Balancer

We will do it easily with Metal-lb. Create the metallb-config.yml file (the default pool part is not required).

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 172.16.1.251/32
    - name: nginx-ingress
      protocol: layer2
      addresses:
      - 172.16.1.250/32

Let’s configure everything

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.9.3/manifests/metallb.yaml
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
kubectl apply -f metallb-config.yml

Install the Nginx Ingress

The Nginx Ingress service will receive the connection and redirect it to the wanted service. Create the configuration file, nginx-config.yml.

apiVersion: v1
kind: Namespace
metadata:
  name: nginx-ingress
  labels:
    name: nginx-ingress
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-ingress-controller
  namespace: nginx-ingress
  labels:
    app.kubernetes.io/name: nginx-ingress
    app.kubernetes.io/part-of: nginx-ingress
data:
  proxy-protocol: "True"
  use-proxy-protocol: "True"
  real-ip-header: "proxy_protocol"

We will install the Ingress controller

kubectl apply -f nginx-config.yml
helm repo add stable https://kubernetes-charts.storage.googleapis.com
helm install nginx-ingress stable/nginx-ingress --namespace nginx-ingress --set controller.service.externalTrafficPolicy=Local --set controller.service.annotations."metallb\.universe\.tf/address-pool"=nginx-ingress

Certificate management

For this part we will use cert-manager. Create the configuration file, cert-manager.yml.

---
global:
  rbac:
    create: true
  leaderElection:
    namespace: cert-manager
installCRDs: true
ingressShim:
  defaultIssuerName: letsencrypt-prod
  defaultIssuerKind: ClusterIssuer

Create cluster-issuer.yml

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-stage
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: youremail@domain.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-stage
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class: nginx
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: youremail@domain.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class: nginx

Let’s create Everything

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager --namespace cert-manager --values cert-manager.yml
kubectl apply -f cluster-issuer.yml

Conclusion

We now have Kubernetes with Load Balancer compatibility, Ingress compatibility and automatic certificate deployment. You can create your Ingress objects using the following configuration.

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    app.kubernetes.io/name: nginx-ingress
    app.kubernetes.io/part-of: nginx-ingress
    ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
  - hosts:
    - yourhost.com
    secretName: tls-secret
  rules:
  - host: "yourhost.com"
    http:
      paths:
      - path: /
        backend:
          serviceName: yourservice
          servicePort: 80