Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

generating backpressure dd events #71086

Merged
merged 3 commits into from
May 21, 2024
Merged

Conversation

nirajdewani
Copy link
Contributor

@nirajdewani nirajdewani commented May 16, 2024

Description

Now that sentry supports Datadog events,
updating backpressure code to generate events as well (in addition to metrics).

Next steps

  • Send events using dogshell
  • Update event body if the need be

@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label May 16, 2024
@lynnagara
Copy link
Member

What is the motivation for this? Does it tell us anything that the metric doesn't already?

@mwarkentin
Copy link
Member

Hey @lynnagara there was some more detail in the original PR / ticket: https://getsentry.atlassian.net/browse/OPS-5392

This would enable us to do things like:

  • overlay events on top of dashboard widgets
  • add tags to events to support analysis / grouping
  • Trigger DD monitors for alerting purposes (eg. notifying slack / PagerDuty)

@mwarkentin
Copy link
Member

Here's an example time period where we experienced backpressure on May 6:

image

And split:

image

Some things that aren't clear:

  • Why is everything a multiple of 10?
  • When exactly did backpressure trigger?
  • When did it resolve?

Here is one ingest consumer lag graph of the same time:

image

With events we can have discrete events that could include information from the backpressure system for more context. These can be overlaid on top of graphs:

image

Copy link

codecov bot commented May 17, 2024

Codecov Report

Attention: Patch coverage is 75.00000% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 77.89%. Comparing base (7eb565d) to head (51ae4cd).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #71086      +/-   ##
==========================================
- Coverage   77.89%   77.89%   -0.01%     
==========================================
  Files        6525     6525              
  Lines      290597   290601       +4     
  Branches    50276    50276              
==========================================
- Hits       226375   226372       -3     
- Misses      57975    57979       +4     
- Partials     6247     6250       +3     
Files Coverage Δ
src/sentry/processing/backpressure/health.py 83.78% <75.00%> (-0.51%) ⬇️

... and 11 files with indirect coverage changes

Copy link
Member

@iambriccardo iambriccardo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nirajdewani nirajdewani merged commit 6316000 into master May 21, 2024
49 of 50 checks passed
@nirajdewani nirajdewani deleted the OPS-5392/send-backpressure-events branch May 21, 2024 20:27
@github-actions github-actions bot locked and limited conversation to collaborators Jun 6, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Scope: Backend Automatically applied to PRs that change backend components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants