Repository navigation
Investigate cluster CPU spikes #727
Description
Activity
The job is searching for orphaned documents across all time. Since we are running this on a regular basis, we can probably restrict it to only check events within the last 3 days or so (forgot how far back we allow events to be post dated).
I guess what would be better is to find recently deleted stacks, projects, orgs and see if there is any matching events. Ideally we wouldn't have to worry about this happening. We need to bullet proof the process of deleting projects, stacks and orgs and make sure it's impossible for for them to be deleted if there are any matching events. We might have a concurrency issue where a stack is deleted but a new event comes in at the same time and gets added to that stack even though it was just deleted.
- linked a pull request that will close this issueScope orphaned-event cleanup to recent ingest window to eliminate 8-hour ES CPU spikes #2279
on Jun 1, 2026 We're closing this as part of an open issue cleanup. This looks like a historical production/cluster investigation from 2020 rather than a reproducible current product bug.
If the same 8-hour CPU spike is visible in current telemetry, please open a new issue with recent metrics, traces, and the affected deployment details so we can investigate from current evidence.
Happening every 8 hours. Orphan data cleanup is the only thing that happens every 8 hours. Doesn't look like the job itself is taking a lot of CPU, but ES CPU spikes during that time. So my guess would be whatever we are doing in orphan data cleanup is very expensive in ES.