hackdoc:// HSCloud Docs
Home Cluster DC Codelabs

Page Info

Kubernetes cookbook

Bite-sized tips, examples, and templates for all common deployment needs

Basic template

prod.json starter pack, applicable to most simple deployments:

local kube = import "../../../kube/hscloud.libsonnet";

{
    local top = self,
    local cfg = self.cfg,

    cfg:: {
        name: 'app_name',
        namespace: 'namespace',
        domains: ['app_name.cloud.of-a.cat'],
        image: 'registry.hswaw.net/foo/bar:2137',
    },

    secretRefs:: {
        secret_name: { secretKeyRef: { name: cfg.name, key: 'secret_name' } },
    },

    local ns = kube.Namespace(cfg.namespace),

    deployment: ns.Contain(kube.Deployment(cfg.name)) {
        spec+: {
            template+: {
                spec+: {
                    containers_: {
                        default: kube.Container("default") {
                            image: cfg.image,
                            ports_: {
                                http: { containerPort: 5000 },
                            },
                            env_: {
                                APP_NAME_SECRET: top.secretRefs.secret_name,
                            },
                        },
                    },
                },
            },
        },
    },

    service: ns.Contain(kube.Service(cfg.name)) {
        target:: top.deployment,
    },

    ingress: ns.Contain(kube.TLSIngress(cfg.name)) {
        hosts:: [cfg.domain],
        target:: top.service,
    },
}

Kubernetes basics

How to use jsonnet

You can do it!

kube CLI tips

# See (almost) all resources:
kubectl -n NAMESPACE get all,ing,pvc,cm,secret

# View logs
kubectl -n NAMESPACE logs deploy/NAME
# or (nicer):
stern -n NAMESPACE NAME

# Shell access
kubectl -n NAMESPACE exec deploy/NAME -it -- sh

# Disable deployment
kubectl -n NAMESPACE scale deploy/NAME --replicas=0

# Restart deployment
kubectl -n NAMESPACE rollout restart deploy/NAME

# Cool kubernetes CLI dashboard
k9s

# see jsonnet result/diff
kubecfg diff path/to/prod.jsonnet
kubecfg show path/to/prod.jsonnet
jsonnet path/to/prod.jsonnet | jq

# copy resource with changes (e.g. rename, move to another namespace)
kubectl -n OLD_NAMESPACE get secret/FOO -o json | jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' | jq '.metadata.name="NEW_NAME"' | kubectl apply -n NEW_NAMESPACE -f -

ConfigMap

  • Use ConfigMap to inject config files into the container
  • You can also use to to inject other minor modifications to the container image (e.g. to avoid maintaining a Dockerfile)
  • Tip: use jsonnet’s importstr "./foo.bar", std.manifestJsonEx(...), std.manifestYamlDoc(...), std.manifestIni(...) as appropriate.

Search for kube.ConfigMap for examples.

Ingress

  • Use kube.TLSIngress (//kube/hscloud.libsonnet) for a simplified manifest (with SSL/TLS already configured)
  • See DNS for setting up a domain name
  • Use extraPaths:: to point specific paths at services other than target
  • Use metadata.annotations for non-HTTP backends, redirects, and other nginx configuration (see documentation). Only some annotations are allowed by default (ask ops for details).

Cron jobs

Search for kube.CronJob for examples.

Reliability tips

  • Consider adding livenessProbe or readinessProbe to container (see Kubernetes docs) to auto-restart on failures
  • Consider adding resources.limits to container (see Kubernetes docs) to prevent app from abusing resources
  • If app proves unreliable despite readinessProbe, admit defeat and use //kube/restarter.libsonnet to force regular restarts

Secrets

Use plain Kubernetes secrets.

# create secrets
kubectl -n NAMESPACE create secret generic NAME --from-literal=random_secret=$(pwgen 32 1) --from-literal=other_secret=secret_value

# view secret
kubectl -n NAMESPACE get secret/NAME -o jsonpath='{.data.random_secret}' | base64 -d && echo

# edit secret manually
# add *base64-encoded* values to .data.new_secret
kubectl -n NAMESPACE edit secret NAME

Usage in Jsonnet:

{
    secretRefs:: {
        some_secret: { secretKeyRef: { name: cfg.name, key: "some_secret" } },
    },

    deployment: ns.Contain(kube.Deployment(cfg.name)) {
        spec+: {
            template+: {
                spec+: {
                    containers_: {
                        default: kube.Container("default") {
                            // ...
                            env_: {
                                // Reference secrets in env
                                APP_SOME_SECRET: top.secretRefs.some_secret

                                // You can use secret env in other envs like so
                                // Quirk: referenced secret must be alphabetically before this
                                USED_SECRET: 'username:$(APP_SOME_SECRET)'
                            },
                        },
                    },
                },
            },
        },
    },
}

Alternative: Use secretstore (TODO, after planned upgrades)

Resource access & common tasks

Namespace

Use personal-FOO namespace (auto-provisioned) for experimentation.

For new production services, go to //cluster/kube/k0.libsonnet and adjust k0.admins.namespaces. Ask cluster ops to apply.

Cluster ops

kubecfg update cluster/kube/k0-admins.jsonnet

Docker Container Registry

  • Use registry.hswaw.net to authenticate
  • registry.hswaw.net/USERNAME/... is your personal container namespace
  • Please use unique tags (e.g. $(date '+%Y%m%d%H%M%S')) for deployments. Specifying :latest is a bad practice, and overriding existing tag will cause confusion (due to container image cache).
  • On ARM Macs: use --platform linux/amd64
  • Note: previously, registry.k0.hswaw.net domain was used. It continues to work for backwards compatibility, but registry.hswaw.net is preferred going forward.

Example:

docker build --tag registry.hswaw.net/palid/walne-generator:1.0-alpha
docker push registry.hswaw.net/palid/walne-generator:1.0-alpha

Persistent storage (Block storage)

Use PersistentVolumeClaim in waw-hdd-redundant-3 storage class.

Example:

{
    cfg:: {
         // ...
        storageClassName: 'waw-hdd-redundant-3',
    },

    deployment: ns.Contain(kube.Deployment(cfg.name)) {
        spec+: {
            template+: {
                containers_: {
                    default: kube.Container('default') {
                        // ...
                        volumeMounts_: {
                            data: { mountPath: '/var/data' },
                        },
                    },
                },
                volumes_: {
                    data: top.data.volume,
                },
            },
        },
    },

    data: ns.Contain(kube.PersistentVolumeClaim(cfg.name)) {
        // Hard to change later, make it large enough
        storage:: '10Gi',
        storageClass:: cfg.storageClassName,
    },
}

S3 / Object storage

  • Add user object in //cluster/kube/k0.libsonnet:k0.ceph.clients
  • Ask hscloud ops to update

DNS

  • *.cloud.of-a.cat:
    • simply create a app_name.cloud.of-a.cat Ingress
  • *.*.hscloud.ovh:
    • only for use in your personal-USER namespace
    • simply create an app_name.USER.hscloud.ovh Ingress
  • Bring your own domain:
    • create your own DNS CNAME record pointing at ingress.k0.hswaw.net
    • (optional) adjust admitomatic (see below)
  • *.hackerspace.pl, *.hswaw.net:
    • ask cluster ops to create/update relevant entries
    • adjust admitomatic (see below)
    • NOTE: please wait 15 minutes before creating Ingress, otherwise TLS certificate creation may get stuck for a while due to cached stale DNS records

Adjusting admitomatic

Go to //cluster/kube/k0.libsonnet and adjust k0.admitomatic.cfg.proto.allow_domain if you want your domain to be secured against hijacking by other cluster users (*.hackerspace.pl is one such domain). Ask cluster ops to apply.

Cluster ops

hackerspace.pl DNS: TODO

admitomatic update:

kubecfg update cluster/kube/k0-admitomatic.jsonnet
kubectl -n admitomatic rollout restart daemonset.apps/admitomatic

CockroachDB

  • Add user object in //cluster/kube/k0.libsonnet:k0.cockroach.clients
  • Ask hscloud ops to update

Postgres

Use //kube/postgres.libsonnet to create local deployment in app namespace. See template below.

Access: kubectl -n NAMESPACE exec deployment/NAME-postgres -it -- psql -U NAME NAME

Alternative: Consider using CockroachDB

Alternative: Ask hscloud ops for a database on blessed high-performance SSD node

Template:

{
    cfg:: {
        // ...
        storageClassName: 'waw-hdd-redundant-3',
    },

    secretRefs:: {
        postgres: { secretKeyRef: { name: cfg.name, key: 'postgres_password' } },
    },


    deployment: ns.Contain(kube.Deployment(cfg.name)) {
        spec+: {
            template+: {
                spec+: {
                    containers_: {
                        default: kube.Container("default") {
                            // ...
                            env_: {
                                // use references to avoid duplication mistakes
                                DB_NAME: top.postgres.cfg.database,
                                DB_USER: top.postgres.cfg.username,
                                DB_PASSWORD: top.postgres.cfg.password,
                                DB_HOST: top.postgres.svc.host,
                                DB_PORT: top.postgres.svc.port,

                                // alternatively (single env):
                                // Quirk: Must be alphabetically before SQLALCHEMY_DATABASE_URI
                                POSTGRES_PASSWORD: top.postgres.cfg.password,
                                SQLALCHEMY_DATABASE_URI: 'postgresql+psycopg2://%s:%s@%s:%s/%s' % [
                                    top.postgres.cfg.username,
                                    '$(POSTGRES_PASSWORD)',
                                    top.postgres.svc.host,
                                    top.postgres.svc.port,
                                    top.postgres.cfg.database,
                                ],
                            },
                        },
                    },
                },
            },
        },
    },

    postgres: ns.Contain(postgres) {
        cfg+: {
            prefix: cfg.name + '-',
            appName: cfg.name,
            storageClassName: cfg.storageClassName,
            version: '17.2', // Prefer using latest version when deploying https://hub.docker.com/_/postgres

            database: 'APP_NAME',
            username: 'APP_NAME',
            password: top.secretRefs.postgres,
        }
    },
}

Redis

Use //kube/redis.libsonnet to create local deployment in app namespace

SSO (OAuth2/OpenID Connect for HSWAW members authentication)

  • Register your own app at sso.hackerspace.pl
  • In application code:
    • Use whatever OAuth library is most convenient to you
    • Alternative: Use flask-spaceauth for Flask apps
    • Alternative: Use OAuth2-Proxy to keep all authentication on deployment/hscloud level. See //hswaw/paperless/paperless.libsonnet for an example. (TODO: Extract as a reusable jsonnet lib)

LDAP

Only use for very specific cases, when user/group listing or LDAP modification is required - otherwise use SSO.

Ask cluster ops to create an LDAP service account.

Examples: ldapweb, walne, codehosting.

Cluster ops

  • Create cn=…,ou=Services,dc=hackerspace,dc=pl in LDAP
  • Add relevant ACL in /etc/openldap/slapd.conf on boston-packets.hackerspace.pl

Beyondspace

Access to *.waw.hackerspace.pl (LAN devices) services from WAN/hscloud.

Ask cluster ops to set up. Then you can call internal devices from hscloud via basic auth (https://service:secret@foo.waw.hackerspace.pl/...).

Examples: printservant

Cluster ops

  • Add specific internal domain to beyondspace (//hswaw/machines/customs.hackerspace.pl/beyondspace.nix)
  • Create a service authentication token on customs

Mailing (SMTP/IMAP)

Ask ops to create local mailing user account, then use SMTP/IMAP.

Examples: ldapweb, paperless

Cluster ops

  • Create local service user on boston-packets.hackerspace.pl: useradd -rm SERVICE; passwd SERVICE
  • Create mailbox on boston-packets.hackerspace.pl: mkdir /var/spool/mail/SERVICE; chown SERVICE:mail /var/spool/mail/SERVICE
  • Optionally: add aliases (and/or incoming exec hook) in /etc/mail/aliases