Modern advertising requires a modern ad platform that enables advertisers to engage the best audiences with a consistent and relevant ad experience. Adobe Advertising Cloud a platform that manages advertising across traditional TV and digital formats. Like most Fortune 500 companies, Adobe uses Kubernetes for running networking, storage, security, and monitoring.
This postmortem report assesses the challenges, failures, and successes of an incident that affected one of the stacks team at Adobe.
- They are running ~300 worker nodes
- 6 OpenStack data centers across 4 regions
- Running on VMs
- No persistent storage
- No autoscaling; fixed footprint
- Smaller but growing
- 3 AWS clusters in us-east-1
- Running on m5d.12 x large ec2 instances
- EBS volumes for persistent storage
- Uses cluster-autoscaler
- Autoscaling events many times per hour
- Prometheus for monitoring
- Dozens of Machine Learning workloads (predictive modeling algorithms) in AWS
- Reason for frequent autoscaling events
- Cluster updates are done via a new image and a rolling update of existing nodes
- Updates are deployed every 4 weeks
One of their stacks team was having problems with the production usage of the Horizontal Pod Autoscaler. Kubernetes has this ability to adjust the number of replicas for you automatically based on the load. HPA watches a specified Kubernetes Deployment, monitoring a specific metric, and it can increase the number of Pods if there is heavy traffic, or reduce it to scale down the Deployment if there are idle Pods.
You can easily scale on CPU or Memory usage using HPA. Here's an example of an HPA based on CPU utilization:
- type: Resource
HPA also allows you to scale on custom metrics like http_requests from Ingress resources. You can even install plugins that allow you to do things like pulling metrics from Prometheus to do custom auto-scaling actions. A common use case for this is if you have a job queue, when the queue reaches a certain size, you need to launch more workers in order to handle that extra load so that you don't have to wait forever for that job queue to complete processing.
After switching from using a fixed number of replicas to using the HPA, the team ran into an issue. If there was a new release of the code that they needed to roll out to the Kubernetes cluster:
kubectl apply -f adobe-deployment.yml
After deploying, the replica count would be brought down from let's say 10 to 3. So the HPA provisions 10 replicas but after the Deployment Kubernetes overrides this behavior.
The Root Cause
Originally they were using a fixed number of replicas. They used to set the desired number of replicas inside their Deployment manifests. So, in this case, they had set a minimum of 3 replicas. The Horizontal Pod Autoscaler would say it wants 10 Pods and it would scale up to this number but the next time that deployment had been applied with 3 because perhaps there was a new version of code, it would scale down to 3 then back up to 10.
If you're using the HPA, don't set your Deployment replicas. Kubernetes will not honor HPA because HPA should not block the manual scaling of Pods. It gets triggers only from resources (CPU, Memory, etc). Instead, set it solely in the HPA. That way if there are 10 running, the Deployment will apply against those 10 Pods and not shrinking back down.
In summary, we have seen that a Kubernetes Deployment doesn't do a good job of detecting scaling behavior. We should, therefore, not set the replicas field in the Deployment.
Validation is also an important step in the lifecycle of a Kubernetes cluster. kubectl-val provides a mechanism to validate the conformance of the kubernetes clusters' configurations. It ensures that the system has a consistent and reliable behavior. This tool will catch hard-to-test bugs before users do. kubectl-val comes as a simple kubectl plugin. You can download the latest release here.