Cluster Admin Docs

Current cluster: k0.hswaw.net

Persistent Storage (waw3)

HDDs on dcr01s2{2,4}. 40TB total capacity for now. Use this.

The following storage classes use this cluster:

waw-hdd-yolo-3 - 1 replica
waw-hdd-redundant-3 - 2 replicas
waw-hdd-redundant-3-object - 2 replicas, object store

Rados Gateway (S3) is available at https://object.ceph-waw3.hswaw.net/. To create a user, ask an admin.

PersistentVolumes currently bound to PVCs get automatically backed up (hourly for the next 48 hours, then once every 4 weeks, then once every month for a year).

Administration

Provisioning nodes

bring up a new node with nixos, the configuration doesn’t matter and will be nuked anyway
add machine to cluster/machines and ops/machines.nix
generate certs with bazel run //cluster/clustercfg gencerts
deploy using ops (see ops/README.md)

Applying kubecfg state

First, decrypt/sync all secrets:

secretstore sync cluster/secrets/

Then, run kubecfg. There’s multiple top-level ‘view’ files that you can run, all located in //cluster/kube. All of them use k0.libsonnet as the master state of Kubernetes configuration, just expose subsets of it to work around the fact that kubecfg gets somewhat slow with a lot of resources.

k0.jsonnet: everything that is defined for k0 in //cluster/kube/....
k0-core.jsonnet: definitions that re in common across all clusters (networking, registry, etc), without Rook.
k0-registry.jsonnet: just the docker registry on k0 (useful when changing ACLs).
k0-ceph.jsonnet: everything ceph/rook related on k0.

When in doubt, run k0.jsonnet. There’s no harm in doing it, it might just be slow. Running individual files without realizing that whatever change you implemented also influenced something that was rendered in another file can cause to production inconsistencies.

Feel free to add more view files for typical administrative tasks.

Ceph - Debugging

We run Ceph via Rook. The Rook operator is running in the ceph-rook-system namespace. To debug Ceph issues, start by looking at its logs.

A dashboard is available at https://ceph-waw2.hswaw.net/ and https://ceph-waw3.hswaw.net, to get the admin password run:

kubectl -n ceph-waw2 get secret rook-ceph-dashboard-password -o yaml | grep "password:" | awk '{print $2}' | base64 --decode ; echo
kubectl -n ceph-waw3 get secret rook-ceph-dashboard-password -o yaml | grep "password:" | awk '{print $2}' | base64 --decode ; echo

Ceph - Backups

Kubernetes PVs backed in Ceph RBDs get backed up using Benji. An hourly cronjob runs in every Ceph cluster. You can also manually trigger a run by doing:

kubectl -n ceph-waw2 create job --from=cronjob/ceph-waw2-benji ceph-waw2-benji-manual-$(date +%s)
kubectl -n ceph-waw3 create job --from=cronjob/ceph-waw3-benji ceph-waw3-benji-manual-$(date +%s)

Ceph ObjectStorage pools (RADOSGW) are not backed up yet!

Ceph - Object Storage

To create an object store user consult rook.io manual (https://rook.io/docs/rook/v0.9/ceph-object-store-user-crd.html). User authentication secret is generated in ceph cluster namespace (ceph-waw{2,3}), thus may need to be manually copied into application namespace. (see app/registry/prod.jsonnet comment)

tools/rook-s3cmd-config can be used to generate test configuration file for s3cmd. Remember to append :default-placement to your region name (ie. waw-hdd-redundant-3-object:default-placement)

Page Info