Kubernetes
1. Kubernetes cheat sheet
2. Useful kubernetes troubleshooting
3. Check application logs
What is Kubernetes?
Kubernetes (k8s), is an open source platform pioneered by Google, which started as a simple container orchestration tool but has grown into a cloud native platform.
Container orchestration is about managing the lifecycle of containers, particularly in large, dynamic environments. It automates:
- deployment
- networking
- scaling
- availability of containerised workloads and services
What is k8s by Ubuntu
Learn k8s basics
Explain k8s in plain english
What is k8s by RedHat
What is k8s by Google
Kubernetes cheat sheet
Installing kubectl on Linux
# https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/
$ sudo snap install kubectl --classic
$ kubectl version --client
Installing kubectl on mac
# https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/
$ brew install kubectl
$ kubectl version --client
Useful Kubernetes troubleshooting
schooner (Kubernetes Custom Resource Definition or CRD at our company) -> deployment -> replicaset -> pods -> containers -> services, accounts, etc.
- If the pod is not up, there might be useful error info in describe replicaset or deployment
- Also useful to describe pods to find dependent container logs that may show error preventing pod from starting up
- eg
vault-static-secrets,digital-party-db-cluster-init - the earlier containers might fail so nothing shows up in artefact container (eg application-review-service)
- eg
// if you've changed AWS envs, this is useful to set your kube config to current env
$ aws eks --region ap-southeast-2 update-kubeconfig --name tap
// get all namespaces
$ kubectl get ns
$ kubectl -n digital get replicaset
// Useful for checking errors in replicaset
$ kubectl -n digital describe replicaset application-review-service
// Can use full name
$ kubectl -n digital describe replicaset application-review-service-xdfae
$ kubectl -n digital get deployments
// Useful for checking errors in deployments
$ kubectl -n digital describe deployments application-review-service
// schooler = CRD (custom resource definitions)
$ kubectl -n digital get schooner
$ kubectl -n digital describe schooner
$ kubectl -n digital describe schooner application-review-service
$ kubectl -n digital get pod
// Useful to find container logs that may show the error preventing pod from starting up
$ kubectl -n digital describe pod application-review-service
$ kubectl -n digital describe pod external-data-lookup
// Useful to find the version of the application
$ kubectl -n digital describe pod application-review-service-5d9875d667-8dhjb | grep gh
// Eg where party gateway logs were blank due to not being able to start up
$ kubectl -n party describe pod party-gateway-76d57d97b6-wrh2t
// Eg where application-review-service was failing due to party-db-cluster issue
$ kubectl -n digital logs -f application-review-service-5d9875d667-8dhjb -c digital-party-db-cluster-init
// Eg where external-data-lookup was failing as it could not login to vault using role, invalid role name
$ kubectl -n digital logs -f external-data-lookup-7bdc9b6ff6-78qll -c vault-static-secrets
$ kubectl -n digital logs -f -lapp=external-data-lookup -c vault-static-secrets
// 4 ways to restart pods: https://komodor.com/learn/kubectl-restart-pod/
// shutdown pod (or application-review-avoka-integration)
$ kubectl scale deploy application-review-service --replicas=0
// restart pod
$ kubectl scale deploy application-review-service --replicas=2
// delete pod
$ kubectl -n digital delete pod <pod name>
$ kubectl -n digital get jobs
// Delete job (need admin access)
$ kubectl -n digital delete job <job-name>
// Login to pod
// works for artefacts that use clouddb
$ kubectl exec -it -n digital mvsi-onboarding-adapter-585845d96f-9tgwk -c vault-agent -- /bin/sh
// works for artefacts connecting to old RDS (not clouddb)
$ kubectl exec -it -n mimo alipay-transaction-receiver-8674f7d55-4x9sp -c istio-proxy -- /bin/sh
// not work anymore
$ kubectl -n digital exec -it external-data-lookup-6f74f7c874-8cgdc /bin/sh
// check network connectivity from pod
// nc -zv <url> <pod>
$ nc -zv customer-zero-3-04-a.cluster-c2cim00ia1fp.ap-southeast-2.rds.amazonaws.com 3306
// Port forward to container to hit end point
// kubectl -n digital port-forward <pod> 8480:8480
$ kubectl -n digital port-forward internal-user-authentication-7d5b448dcc-b1b2h 8480:8480
// On another terminal, curl to hit endpoint = results in health json endpoint
$ curl localhost:8480/internal-user-authentication/health
// Check Prometheus alerting of artefact
$ kubectl -n digital describe PrometheusRule auth0-tyro-customer-stream-subscribe
Check application logs
An easier way to move around k8s is to install k9s
// kubectl -n digital logs -f -lapp=<pod> -c <artefact-container>
$ kubectl -n digital logs -f -lapp=existing-customer-apply-bff -c existing-customer-apply-bff
$ kubectl -n digital logs -f -lapp=application-review-service -c application-review-service
// ARS database schema migration
$ kubectl -n digital logs -f -lapp=application-review-service-schema-migration -c application-review-service-schema-migration
// Check specific log
// kubectl -n digital logs -f <pod> -c <artefact-container>
$ kubectl -n digital logs -f application-review-service-db499b44f-l7wvw -c application-review-service
// Check log of pod
$ kubectl -n mimo logs -f trigger-partner-commission-summarizer-13688597633-912w6
// When trouble shooting errors
// 1. Describe pod and make list of all the containers
$ kubectl -n digital describe pod application-review-service
// 2. Then check the specific log of various containers to find the error
// eg digital-party-db-cluster-init, vault-static-secrets
// not just artefact container (eg application-review-service)
$ kubectl -n digital logs -f application-review-service-5d9875d667-8dhjb -c digital-party-db-cluster-init
$ kubectl -n digital logs -f -lapp=external-data-lookup -c vault-static-secrets
// Check banking product-workflow-manager log when calling new Banking ToM endpoint
$ kubectl -n banking core logs -f -lapp=product-workflow-manager -c product-workflow-manager