Added a playbook to restart a deployment on certain prometheus alert #321
Conversation
|
Nikhil Maheshwari seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
| Restart Based on prometheus alert | ||
| ######################## | ||
|
|
||
| These actions are useful for automatic restart of deployment based on certain prometheus alert. |
There was a problem hiding this comment.
Can you emphasize that we restart the deployment by the configured name, and not necessarily the deployment related to the alert
|
|
||
|
|
||
| @action | ||
| def deployment_restart(alert: PrometheusKubernetesAlert, params: DeploymentParams): |
There was a problem hiding this comment.
Do you mind naming it deployment_restart_by_name.
We might add another action in the future, that will do the same but using the deployment in the alert (according to labels). So we want to save a name like this for that one
| f"cannot run deployment_restart on alert with no deployment: {alert}" | ||
| ) | ||
| return | ||
| my_deployment.spec.template.metadata.annotations['kubectl.kubernetes.io/restartedAt'] = now |
There was a problem hiding this comment.
Is kubernetes.io reserved? (See
https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#:~:text=The%20kubernetes.io/%20and%20k8s.io/%20prefixes%20are%20reserved%20for%20Kubernetes%20core%20components.)
maybe 'robusta.dev/restartAt' ?
|
|
||
| if not my_deployment: | ||
| logging.error( | ||
| f"cannot run deployment_restart on alert with no deployment: {alert}" |
There was a problem hiding this comment.
This will happen if the deployment doesn't exist.
It might be useful to add the deployment name and namespace to the error log
No description provided.