Helm[60] is the package manager for Kubernetes that enables you to distribute and install services in Kubernetes. Helm packages are called charts. A chart defines all resources needed to run a service in a Kubernetes cluster—for example, its deployments, services, persistent volume claims, and so on. Charts on Kubernetes are like Debian packages on Debian or Homebrew formulas on macOS. As a service developer, you’ll want to build and share a Helm chart for your service to make it easier for people to run your service. (And if you’re dogfooding your own service, you’ll get the same benefit.)
A release is a instance of running a chart. Each time you install a chart into Kubernetes, Helm creates a release. In the Debian package and Homebrew formula examples, releases are like processes.
And finally, repositories are where you share charts to and install charts from; they’re like Debian sources and Homebrew taps.
To install Helm, run this command:
| | $ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 \ |
| | | bash |
Before we write our own Helm chart, let’s take Helm for a spin and install an existing chart. Bitnami[61] maintains a repository of charts for popular applications. Let’s add a Bitnami repository and install the Nginx chart, which is a web and proxy server:
| | $ helm repo add bitnami https://charts.bitnami.com/bitnami |
| | $ helm install my-nginx bitnami/nginx |
We can see the releases by running $ helm list:
| | $ helm list |
| | NAME NAMESPACE REVISION UPDATED STATUS... |
| | my-nginx default 1 2020... deployed... |
Let’s request Nginx to confirm that it’s really running:
| | $ POD_NAME=$(kubectl get pod \ |
| | --selector=app.kubernetes.io/name=nginx \ |
| | --template '{{index .items 0 "metadata" "name" }}') |
| | $ SERVICE_IP=$(kubectl get svc \ |
| | --namespace default my-nginx --template "{{ .spec.clusterIP }}") |
| | $ kubectl exec $POD_NAME curl $SERVICE_IP |
| | % Total % Received % Xferd Average Speed Time Time Time Current |
| | Dload Upload Total Spent Left Speed |
| | 100 612 100 612 0 0 597k 0 --:--:-- --:--:-- --:--:-- 597k |
| | <!DOCTYPE html> |
| | <html> |
| | <head> |
| | <title>Welcome to nginx!</title> |
| | <style> |
| | body { |
| | width: 35em; |
| | margin: 0 auto; |
| | font-family: Tahoma, Verdana, Arial, sans-serif; |
| | } |
| | </style> |
| | </head> |
| | <body> |
| | <h1>Welcome to nginx!</h1> |
| | <p>If you see this page, the nginx web server is successfully installed and |
| | working. Further configuration is required.</p> |
| | |
| | <p>For online documentation and support please refer to |
| | <a href="http://nginx.org/">nginx.org</a>.<br/> |
| | Commercial support is available at |
| | <a href="http://nginx.com/">nginx.com</a>.</p> |
| | |
| | <p><em>Thank you for using nginx.</em></p> |
| | </body> |
| | </html> |
We could use the same technique for deploying Nginx in a production environment, aside from setting some configuration parameters to fit our use case. Helm made it easy to install and configure an Nginx cluster, and we can manage other services the same way.
Uninstall the Nginx release by running the following:
| | $ helm uninstall my-nginx |
| | release "my-nginx" uninstalled |
Now, let’s build our own chart.
In this section, we’ll build a Helm chart for our service and use it to install a cluster in our Kind cluster.
Create your Helm chart by running these commands:
| | $ mkdir deploy && cd deploy |
| | $ helm create proglog |
Helm created a new chart in a new proglog directory that’s bootstrapped with an example that shows you what a Helm chart looks like—to write your own or to tweak for your own services. The proglog directory contains these directories and files:
| | . |
| | └── proglog |
| | ├── charts |
| | ├── Chart.yaml |
| | ├── templates |
| | │ ├── deployment.yaml |
| | │ ├── _helpers.tpl |
| | │ ├── ingress.yaml |
| | │ ├── NOTES.txt |
| | │ ├── serviceaccount.yaml |
| | │ ├── service.yaml |
| | │ └── tests |
| | │ └── test-connection.yaml |
| | └── values.yaml |
| | |
| | 4 directories, 9 files |
The Chart.yaml file describes your chart. You can access the data in this file in your templates. The charts directory may contain subcharts, though I’ve never needed subcharts.
The values.yaml contains your chart’s default values. Users can override these values when they install or upgrade your chart (for example, the port your service listens on, your service’s resource requirements, log level, and so on).
The templates directory contains template files that you render with your values to generate valid Kubernetes manifest files. Kubernetes applies the rendered manifest files to install the resources needed for your service. You write your Helm templates using the Go template language.
You can render the templates locally without applying the resources in your Kubernetes cluster by running $ helm template. This is useful when you’re developing your templates or if you want to apply your changes in a two-step plan-then-apply process because you can see the rendered resources that Kubernetes will apply.
To check out the resources Helm would create with the example chart, run this command:
| | $ helm template proglog |
You’ll see the following:
| | --- |
| | # Source: proglog/templates/serviceaccount.yaml |
| | apiVersion: v1 |
| | kind: ServiceAccount |
| | metadata: |
| | name: RELEASE-NAME-proglog |
| | labels: |
| | |
| | helm.sh/chart: proglog-0.1.0 |
| | app.kubernetes.io/name: proglog |
| | app.kubernetes.io/instance: RELEASE-NAME |
| | app.kubernetes.io/version: "1.16.0" |
| | app.kubernetes.io/managed-by: Helm |
| | --- |
| | # Source: proglog/templates/service.yaml |
| | rest |
We don’t need the example templates, so remove them by running this command:
| | $ rm proglog/templates/**/*.yaml proglog/templates/NOTES.txt |
Generally, Helm charts include a template file for each resource type. Our service will require two resource types: a StatefulSet and a Service, so we’ll have a statefulset.yaml file and a service.yaml file. Let’s begin with the StatefulSet.
You use StatefulSets to manage stateful applications in Kubernetes, like our service that persists a log. You need a StatefulSet for any service that requires one or more of the following:
Stable, unique network identifiers—each node in our service requires unique node names as identifiers.
Stable, persistent storage—our service expects the data its written to persist across restarts.
Ordered, graceful deployment and scaling—our service needs initial node to bootstrap the cluster and join subsequent nodes to its cluster.
Ordered, automated rolling updates—we always want our cluster to have a leader, and when we roll the leader we want to give the cluster enough time to elect a new leader before rolling the next node.
And by “stable,” I mean persisted across scheduling changes like restarts and scaling.
If your service isn’t stateful and doesn’t require these features, then you should use a Deployment instead of a StatefulSet. One example is an API service that persists to a relational database, like Postgres. You’d run the API service with a Deployment because it’s stateless, and you’d run Postgres with a StatefulSet.
Create a deploy/proglog/templates/statefulset.yaml file with this code:
| | apiVersion: apps/v1 |
| | kind: StatefulSet |
| | metadata: |
| | name: {{ include "proglog.fullname" . }} |
| | namespace: {{ .Release.Namespace }} |
| | labels: {{ include "proglog.labels" . | nindent 4 }} |
| | spec: |
| | selector: |
| | matchLabels: {{ include "proglog.selectorLabels" . | nindent 6 }} |
| | serviceName: {{ include "proglog.fullname" . }} |
| | replicas: {{ .Values.replicas }} |
| | template: |
| | metadata: |
| | name: {{ include "proglog.fullname" . }} |
| | labels: {{ include "proglog.labels" . | nindent 8 }} |
| | spec: |
| | # initContainers... |
| | # containers... |
| | volumeClaimTemplates: |
| | - metadata: |
| | name: datadir |
| | spec: |
| | accessModes: [ "ReadWriteOnce" ] |
| | resources: |
| | requests: |
| | storage: {{ .Values.storage }} |
I have omitted the spec’s initContainers and containers fields to make the snippet smaller (we will fill those in next). The only thing of note here is that our StatefulSet has a datadir PersistentVolumeClaim—the claim requests storage for our cluster. Based on our configuration, Kubernetes could fulfill the claim with a local disk, a disk provided by your cloud platform, and so on. Kubernetes takes care of obtaining and binding the storage to your containers.
Now, replace initContainers... in the previous snippet with this code:
| | initContainers: |
| | - name: {{ include "proglog.fullname" . }}-config-init |
| | image: busybox |
| | imagePullPolicy: IfNotPresent |
| | command: |
| | - /bin/sh |
| | - -c |
| | - |- |
| | ID=$(echo $HOSTNAME | rev | cut -d- -f1 | rev) |
| | cat > /var/run/proglog/config.yaml <<EOD |
| | data-dir: /var/run/proglog/data |
| | rpc-port: {{.Values.rpcPort}} |
| | # Make sure the following three key-values are on one line each in |
| | # your code. I split them across multiple lines to fit them in |
| | # for the book. |
| | bind-addr: \ |
| | "$HOSTNAME.proglog.{{.Release.Namespace}}.\svc.cluster.local:\ |
| | {{.Values.serfPort}}" |
| | bootstrap: $([ $ID = 0 ] && echo true || echo false) |
| | $([ $ID != 0 ] && echo 'start-join-addrs: \ |
| | "proglog-0.proglog.{{.Release.Namespace}}.svc.cluster.local:\ |
| | {{.Values.serfPort}}"') |
| | EOD |
| | volumeMounts: |
| | - name: datadir |
| | mountPath: /var/run/proglog |
Init containers run to completion before the StatefulSet’s app containers listed in the containers field. Our config init container sets up our service’s configuration file. We configure the first server to bootstrap the Raft cluster. And we configure the subsequent servers to join the cluster. We mount the datadir volume into the container so we can write to the same configuration file our app container will read from later.
Replace containers... in the previous snippet with this:
| | containers: |
| | - name: {{ include "proglog.fullname" . }} |
| | image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" |
| | ports: |
| | - containerPort: {{ .Values.rpcPort }} |
| | name: rpc |
| | - containerPort: {{ .Values.serfPort }} |
| | name: serf |
| | args: |
| | - --config-file=/var/run/proglog/config.yaml |
| | # probes... |
| | volumeMounts: |
| | - name: datadir |
| | mountPath: /var/run/proglog |
These containers define our StatefulSet’s app containers; we need one for our service. We mount the volume to the container for reading the configuration file and persisting the log. We use a flag to tell our service where to find its configuration file.
Kubernetes uses probes to know whether it needs to act on a container to improve your service’s reliability. With a service, usually the probe requests a health check endpoint that responds with the health of the service.
There are three types of probes:
Liveness probes signal that the container is alive, otherwise Kubernetes will restart the container. Kubernetes calls the liveness probe throughout the container’s lifetime.
Readiness probes check that the container is ready to accept traffic, otherwise Kubernetes will remove the pod from the service load balancers. Kubernetes calls the readiness probe throughout the container’s lifetime.
Startup probes signal when the container application has started and Kubernetes can begin probing for liveness and readiness. Distributed services often need to go through service discovery and join in consensus with the cluster before they’re initialized. If we had a liveness probe that failed before our service finished initializing, our service would continually restart. After startup, Kubernetes doesn’t call this probe again.
These probes should help improve your service’s reliability, but they can cause incidents if they’re not carefully implemented (like the example of the liveness probe that restarts the container before it’s finished initializing). The systems dedicated to improving the reliability of the service can cause more incidents than the service by itself.
You have three ways of running probes:
The first two are lightweight because they don’t require any extra binaries in your image. However, a command can be more precise and necessary if you use your own protocol.
gRPC services conventionally use a grpc_health_probe command that expects your server to satisfy the gRPC health checking protocol.[62] Our server needs to export a service defined as:
| | syntax = "proto3"; |
| | |
| | package grpc.health.v1; |
| | |
| | message HealthCheckRequest { |
| | string service = 1; |
| | } |
| | |
| | message HealthCheckResponse { |
| | enum ServingStatus { |
| | UNKNOWN = 0; |
| | SERVING = 1; |
| | NOT_SERVING = 2; |
| | } |
| | ServingStatus status = 1; |
| | } |
| | |
| | service Health { |
| | rpc Check(HealthCheckRequest) returns (HealthCheckResponse); |
| | |
| | rpc Watch(HealthCheckRequest) returns (stream HealthCheckResponse); |
| | } |
Let’s update our server to export the health check service.
Open internal/server/server.go and add the highlighted imports:
| | import ( |
| | "context" |
| | "time" |
| | |
| | api "github.com/travisjeffery/proglog/api/v1" |
| | |
| | grpc_middleware "github.com/grpc-ecosystem/go-grpc-middleware" |
| | grpc_auth "github.com/grpc-ecosystem/go-grpc-middleware/auth" |
| | grpc_zap "github.com/grpc-ecosystem/go-grpc-middleware/logging/zap" |
| | grpc_ctxtags "github.com/grpc-ecosystem/go-grpc-middleware/tags" |
| | |
| | "go.opencensus.io/plugin/ocgrpc" |
| | "go.opencensus.io/stats/view" |
| | "go.opencensus.io/trace" |
| | "go.uber.org/zap" |
| | "go.uber.org/zap/zapcore" |
| | |
| | "google.golang.org/grpc" |
| | "google.golang.org/grpc/codes" |
| | "google.golang.org/grpc/credentials" |
| | "google.golang.org/grpc/peer" |
| | "google.golang.org/grpc/status" |
| | |
| » | "google.golang.org/grpc/health" |
| » | healthpb "google.golang.org/grpc/health/grpc_health_v1" |
| | ) |
Then, update the NewGRPCServer function to include the highlighted lines in this snippet:
| | func NewGRPCServer(config *Config, grpcOpts ...grpc.ServerOption) ( |
| | *grpc.Server, |
| | error, |
| | ) { |
| | logger := zap.L().Named("server") |
| | zapOpts := []grpc_zap.Option{ |
| | grpc_zap.WithDurationField( |
| | func(duration time.Duration) zapcore.Field { |
| | return zap.Int64( |
| | "grpc.time_ns", |
| | duration.Nanoseconds(), |
| | ) |
| | }, |
| | ), |
| | } |
| | |
| | trace.ApplyConfig(trace.Config{ |
| | DefaultSampler: trace.AlwaysSample(), |
| | }) |
| | err := view.Register(ocgrpc.DefaultServerViews...) |
| | if err != nil { |
| | return nil, err |
| | } |
| | |
| | grpcOpts = append(grpcOpts, |
| | grpc.StreamInterceptor( |
| | grpc_middleware.ChainStreamServer( |
| | grpc_ctxtags.StreamServerInterceptor(), |
| | grpc_zap.StreamServerInterceptor( |
| | logger, zapOpts..., |
| | ), |
| | grpc_auth.StreamServerInterceptor( |
| | authenticate, |
| | ), |
| | )), grpc.UnaryInterceptor( |
| | grpc_middleware.ChainUnaryServer( |
| | grpc_ctxtags.UnaryServerInterceptor(), |
| | grpc_zap.UnaryServerInterceptor( |
| | logger, zapOpts..., |
| | ), |
| | grpc_auth.UnaryServerInterceptor( |
| | authenticate, |
| | ), |
| | )), |
| | grpc.StatsHandler(&ocgrpc.ServerHandler{}), |
| | ) |
| | gsrv := grpc.NewServer(grpcOpts...) |
| | |
| » | hsrv := health.NewServer() |
| » | hsrv.SetServingStatus("", healthpb.HealthCheckResponse_SERVING) |
| » | healthpb.RegisterHealthServer(gsrv, hsrv) |
| | |
| | srv, err := newgrpcServer(config) |
| | if err != nil { |
| | return nil, err |
| | } |
| | api.RegisterLogServer(gsrv, srv) |
| | return gsrv, nil |
| | } |
These lines create a service that supports the health check protocol. We set its serving status as serving so that the probe knows the service is alive and ready to accept connections. Then we register the service with our server so that gRPC can call this service’s endpoints.
Replace probes... in deploy/proglog/templates/statefulset.yaml with this snippet to tell Kubernetes how to probe our service:
| | readinessProbe: |
| | exec: |
| | command: ["/bin/grpc_health_probe", "-addr=:{{ .Values.rpcPort }}"] |
| | initialDelaySeconds: 10 |
| | livenessProbe: |
| | exec: |
| | command: ["/bin/grpc_health_probe", "-addr=:{{ .Values.rpcPort }}"] |
| | initialDelaySeconds: 10 |
Then add these highlighted lines to your Dockerfile to install the grpc_health_probe executable in your image:
| | FROM golang:1.14-alpine AS build |
| | WORKDIR /go/src/proglog |
| | COPY . . |
| | RUN CGO_ENABLED=0 go build -o /go/bin/proglog ./cmd/proglog |
| » | RUN GRPC_HEALTH_PROBE_VERSION=v0.3.2 && \ |
| » | wget -qO/go/bin/grpc_health_probe \ |
| » | https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/\ |
| » | ${GRPC_HEALTH_PROBE_VERSION}/grpc_health_probe-linux-amd64 && \ |
| » | chmod +x /go/bin/grpc_health_probe |
| | |
| | FROM scratch |
| | COPY --from=build /go/bin/proglog /bin/proglog |
| » | COPY --from=build /go/bin/grpc_health_probe /bin/grpc_health_probe |
| | ENTRYPOINT ["/bin/proglog"] |
The last resource we need to define in our Helm chart is the Service.
A Service in Kubernetes exposes an application as a network service. You define a Service with policies that specify what Pods the Service applies to and how to access the Pods.
Four types of services specify how the Service exposes the Pods:
ClusterIP exposes the Service on a load-balanced cluster-internal IP so the Service is reachable within the Kubernetes cluster only. This is the default Service type.
NodePort exposes the Service on each Node’s IP on a static port—even if the Node doesn’t have a Pod on it, Kubernetes sets up the routing so if you request a Node at the service’s port, it’ll direct the request to the proper place. You can request NodePort services outside the Kubernetes cluster.
LoadBalancer exposes the Service externally using a cloud provider’s load balancer. A LoadBalancer Service automatically creates ClusterIP and NodeIP services behind the scenes and manages the routes to these services.
ExternalName is a special Service that serves as a way to alias a DNS name.
I don’t recommend using NodePort services (aside from the ones LoadBalancer services create for you). You have to know your nodes’ IPs to use the services, you must secure all your Nodes, and you have to deal with port conflicts. Instead, I recommend using a LoadBalancer or a ClusterIP service if you’re able to run a Pod that can access your internal network.
Create a deploy/proglog/templates/service.yaml for your service template with the following code:
| | apiVersion: v1 |
| | kind: Service |
| | metadata: |
| | name: {{ include "proglog.fullname" . }} |
| | namespace: {{ .Release.Namespace }} |
| | labels: {{ include "proglog.labels" . | nindent 4 }} |
| | spec: |
| | clusterIP: None |
| | publishNotReadyAddresses: true |
| | ports: |
| | - name: rpc |
| | port: {{ .Values.rpcPort }} |
| | targetPort: {{ .Values.rpcPort }} |
| | - name: serf-tcp |
| | protocol: "TCP" |
| | port: {{ .Values.serfPort }} |
| | targetPort: {{ .Values.serfPort }} |
| | - name: serf-udp |
| | protocol: "UDP" |
| | port: {{ .Values.serfPort }} |
| | targetPort: {{ .Values.serfPort }} |
| | selector: {{ include "proglog.selectorLabels" . | nindent 4 }} |
This snippet defines our “headless” Service. A headless Service doesn’t load balance to a single IP. You use a headless Service when your distributed service has its own means for service discovery. By defining selectors on our Service, Kubernetes’ endpoint controller changes the DNS configuration to return records that point to the Pods backing the Service. So, each pod will get its own DNS record similar to proglog-{{id}}.proglog.{{namespace}}.svc.cluster.local, and the servers will use these records to discover each other.