Today I wanted to get nginx set up on my 3 node microk8s cluster with cert-bot. If you plan on exposing anything to the public, you'll usually reach for a webserver like apache, nginx. There are more modern webservers such as Caddy or Traefik, but I chose to go with nginx.

If you plan on exposing anything to the public, this is a must. I'm on the latest stable channel of microk8s, __version 1.21__. 

Before we get started, there are a couple things we need to do first.

### NAT Port Forward Rule
You need to open up port 443 and 80 and forard them to the IP of one of your K8 nodes.

Example:
__<ip.of.node:80> - <ip.of.node:443>__

I use Ubiquiti networking gear. If you use the same, then you can go into Settings > Routing & Firewall > Port Forwarding and set this up.
![unifi-portforward](//images.contentful.com/soglt180o4mt/7jxhtcSDicMyV1lGVIA6Wz/631e6d5911daa3cadd451e3d7f5a992d/Screenshot_2021-07-31_at_17.50.47.png)

### DNS Setup
I use cloudflare to manage all my DNS. I’ve set up an A record that points to my public IPv4 address. Make sure to disable Cloudflare proxy (at least for now) so that we can verify that cert-manager did in fact create the certificates.

![cloudflare-dns](//images.contentful.com/soglt180o4mt/5kSulQL6LTjad1VMzVMVzg/d00001d785e9fdb2b62830a245e348f9/Screenshot_2021-07-31_at_17.30.51.png)

Now that thats out of the way, we can start to get our hands dirty with some Kubernetes deployments!

### Nginx deployment

First, enable DNS and ingress for your cluster and then create the deployment yaml file
```
sudo microk8s enable dns ingress
```

__nginx-service.yaml__
```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webserver-depl
spec:
  selector:
    matchLabels:
      app: webserver-app
  template:
    metadata:
      labels:
        app: webserver-app
    spec:
      containers:
        - name: webserver-app
          image: nginx:1.8
---
apiVersion: v1
kind: Service
metadata:
  name: webserver-svc
spec:
  selector:
    app: webserver-app
  ports:
  - name: webserver-app
    protocol: TCP
    port: 80
    targetPort: 80
```

Then apply it with:
```
sudo microk8s kubectl apply -f
```

#### ingress deployment

An ingress in k8s exposes both HTTP and HTTPS routes from outside the cluster, to services within. Be sure to read more about kubernetes [ingress objects](https://kubernetes.io/docs/concepts/services-networking/ingress/).

__ingress.yaml__
```
apiVersion: networking.k8s.io/v1 
kind: Ingress
metadata:
  name: ingress
spec:
  rules:
  # replace with your domain
  - host: "example.com"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: webserver-svc
            port:
              number: 80
```

Apply it
```
sudo microk8s kubectl apply -f ingress.yaml
```

You should be able to see the default "welcome to nginx" splash screen when you visit your domain now. You'll get an SSL warning when visiting, but don't worry, we'll get to that!

![nginx-splash](//images.contentful.com/soglt180o4mt/2IYAq9pHBFQfYvuCG1ojJ0/f39fd73115a55874eb25fea938dfe7ae/Screenshot_2021-07-31_at_18.03.57.png)

### cert-manager setup

Now that nginx is routing traffic, we can set up cert-manager.

```
sudo microk8s kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml
```

We need to create two deployments for our certificates; one for staging, and one for production.

__letsencrypt-staging.yaml__
```
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
#change to your email
    email: email@example.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - http01:
        ingress:
          class: public
```

__letsencrypt-staging.yaml__
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
#change to your email
    email: email@example.com
    privateKeySecretRef:
       name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: public

Apply both:
```
sudo microk8s kubectl apply -f letsencrypt-staging.yaml

sudo microk8s kubectl apply -f letsencrypt-prod.yaml
```

Once thats done, you need to update the ingress deployment to use the staging certificate.

__ingress.yaml__
```
apiVersion: networking.k8s.io/v1 
kind: Ingress
metadata:
  name: ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-staging"
spec:
  tls:
  - hosts:
    - example.com
    secretName: tls-secret
  rules:
  - host: "example.com"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: webserver-svc
            port:
              number: 80
```

Apply it:
```
sudo microk8s kubectl apply -f ingress.yaml
```

We can check on the status of our certificate. You'll know it worked if `Ready=True`. This will take a couple minutes so give it time. 

If it never shows "True", then get more info on it by running: `sudo microk8s kubectl describe certificate tls-secret`

Finally, swap out the staging certificate for the production certficiate.

__ingress.yaml__
```
apiVersion: networking.k8s.io/v1 
kind: Ingress
metadata:
  name: ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - example.com
    secretName: tls-secret
  rules:
  - host: "example.com"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: webserver-svc
            port:
              number: 80
```

Apply the updates:

```
sudo microk8s kubectl apply -f ingress.yaml`
```

Time to cross our fingers, we want `Ready=True` just like our staging certificate.

```
sudo microk8s kubectl get certificate
```

If all is well, you should now have a valid certificate when you visit your site!


Test Image

Microk8s Nginx Ingress & Certbot Setup

This post is part of a homelab series I'll be rolling out through the next few weeks. I have two custom servers I've built for my homelab, one of which has a Nvidia P2000 that I would like to use inside a Ubuntu VM.

In order to use a PCIe device inside your VM, you need do a little magic inside your Proxmox host.

---

### Host Config (Proxmox)

1. To start, we need to enable **IOMMU** on the Promxox host. With your editor of choice, open up `/etc/default/grub`, and comment out the `#GRUB_CMDLINE_LINUX_DEFAULT="quiet"`, replace with `GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"`. If you are using an intel cpu, then use `intel_iommu=on` instead.

```
# /etc/default/grub
...
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on pcie_acs_override=downstream,multifunction video=efifb:eek:ff"
GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/pve-1 boot=zfs"
...
```
Save the file and then run `update-grub`

Verify that IOMMU is enabled, reboot then run:
`dmesg | grep -e DMAR -e IOMMU`

2. Add the following lines to `/etc/modules`, save then exit.

```
# /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

```

3. This next step assumes you have created a VM inside proxmox already. If you haven't done so yet, go ahead. Default bios is fine for linux, if you are setting up a windows virtual machine, set the bios to `ovmf`. Once finished, shut down the vm and edit it's conf file. I'm adding `q35` as the machine type, and some cpu configurations.

```
# /etc/pve/qemu-server/100.conf
# Replace 100.conf with the ID of your VM.

...
cpu: host,hidden=1,flags=+pcid
machine: q35
...
```

4. This next step is little bit of a pain, but basically it allows us to properly identify our GPU so that we can add it in the VM GUI config area.

`lspci -v` to print out a list of hardware. It will be lengthy, but identify your GPU. Somewhere in that list, you should find your GPU. Here's what my output looks like:

```
...
2d:00.0 VGA compatible controller: NVIDIA Corporation GP106GL (rev a1) (prog-if 00 [VGA controller])
        Subsystem: NVIDIA Corporation GP106GL [Quadro P2000]
        Flags: bus master, fast devsel, latency 0, IRQ 148
        Memory at fb000000 (32-bit, non-prefetchable) [size=16M]
        Memory at d0000000 (64-bit, prefetchable) [size=256M]
        Memory at e0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at f000 [size=128]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] #19
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau
...
```

Now that we've found it, lets get more info on it by running `lspci -n -s 2d:00`

```
2d:00.0 0300: 10de:1c30 (rev a1)
2d:00.1 0403: 10de:10f1 (rev a1)
```

Great, so now we just want to add the pci ID's to the vfio.conf file. The ID's in my case are `10de:1c30` and `10de:10f1`.

Now let's just add it to the vfio.conf file:
`echo "options vfio-pci ids=10de:1c30,10de:10f1" > /etc/modprobe.d/vfio.conf` 

5. We need to add some lines to the proxmox's blacklist conf file.

```
# /etc/modprobe.d/pve-blacklist.conf

blacklist nvidiafb
blacklist nvidia
blacklist radeon
blacklist nouveau
```

Save then exit. This would also be a good time to restart your proxmox host.

6. Now we can finally add our GPU to our virtual machine. Under the hardware settings area, add a new PCI device:
![proxmox-1](//images.ctfassets.net/soglt180o4mt/6rPoHbQCiMD84RTB8lJq5U/45ca6285874c18be638251d25a0494f7/proxmox-1.png)

A list will appear, find and select your GPU from the list:
![proxmox-2](//images.ctfassets.net/soglt180o4mt/55WXiX60jOlVo4B9WTvC7J/fd13526194fa01ddda6084cc0b39da58/proxmox-2.png)

Be sure to enable All Functions, Rom-Bar, and PCI-Express:
![proxmox-3](//images.ctfassets.net/soglt180o4mt/27apBYP83OsVIJoQSJxXa2/647adf1d824b3e8d984d94d1558ddb60/proxmox-3.png)
If you Also turn on Primary GPU, then you won't be able to console into the virtual machine through the proxmox host (you can still ssh into it!). Click 'ok' and then start up your VM!

The last step is to ssh into the virtual machine, and check that our GPU is being recognized.

Run the following command. If all things went well, then you should be able to identify your GPU in the list:

`lshw -c video`

```
 *-display
       description: VGA compatible controller
       product: GP106GL [Quadro P2000]
       vendor: NVIDIA Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress vga_controller bus_master cap_list rom

```

If you see your graphics card in the output, congrats! The next step would be installing drivers, but I'll save that for another post.

Cheers!

---

### References
- [Official Promox](https://pve.proxmox.com/wiki/PCI(e)_Passthrough)
- [manjaro.site](https://manjaro.site/how-to-enable-gpu-passthrough-on-proxmox-6-2/)
- [Beginners Guide to GPU passthrough](https://www.reddit.com/r/homelab/comments/b5xpua/the_ultimate_beginners_guide_to_gpu_passthrough/)


Subscribe to the newsletter