Repository navigation
Fix/hash correctness - #16392
Open
wynonaj wants to merge 33 commits into
Open
Fix/hash correctness#16392wynonaj wants to merge 33 commits into
wynonaj wants to merge 33 commits into
Conversation
…ter-into-dev-3.4.101 Release: Merge back 3.4.101 into dev from: release/merge-master-into-dev-3.4.101
…0.2 (.github/workflows/release-x-manual-docker-containers.yml) (DefectDojo#16330) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…8.0.2 (.github/workflows/rest-framework-tests.yml) (DefectDojo#16329) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…44.158.0 (.github/workflows/renovate.yaml) (DefectDojo#15938) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Bumps and [brace-expansion](https://github.com/juliangruber/brace-expansion). These dependencies needed to be updated together. Updates `brace-expansion` from 1.1.18 to 1.1.21 - [Release notes](https://github.com/juliangruber/brace-expansion/releases) - [Commits](juliangruber/brace-expansion@v1.1.18...v1.1.21) Updates `brace-expansion` from 5.0.9 to 5.0.12 - [Release notes](https://github.com/juliangruber/brace-expansion/releases) - [Commits](juliangruber/brace-expansion@v1.1.18...v1.1.21) --- updated-dependencies: - dependency-name: brace-expansion dependency-version: 1.1.21 dependency-type: indirect - dependency-name: brace-expansion dependency-version: 5.0.12 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…efectDojo#16311) Bumps [source-map-js](https://github.com/7rulnik/source-map-js) from 1.2.1 to 1.2.2. - [Release notes](https://github.com/7rulnik/source-map-js/releases) - [Changelog](https://github.com/7rulnik/source-map-js/blob/main/CHANGELOG.md) - [Commits](7rulnik/source-map-js@v1.2.1...v1.2.2) --- updated-dependencies: - dependency-name: source-map-js dependency-version: 1.2.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
….8 to v (dockerfile.integration-tests-debian) (DefectDojo#16328) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com>
…#16348) * docs(changelog): add DefectDojo Pro 3.4.101 release notes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): apply the drafting rules to the 3.4.101 entry Security fixes name feature areas only, the Orchestrator replaces its internal name, and area tags come from the fixed vocabulary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…jo#16317) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…16347) * docs: add Migrating from Open Source to DefectDojo Cloud Cloud migrations had only a one-line pointer to support on the self-hosted migration page. Add a Cloud page covering supported versions, what to export, secure delivery, encryption keys, users and SSO, cutover planning, and a checklist for support. Link it from the self-hosted page and back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: state confirmed Cloud handling of encryption keys and PostgreSQL version Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: address review on Cloud migration page Cover open source 3.x sources and Authorized Users reconciliation, warn that a large version gap lengthens the schema upgrade, scope Validate Config to OAuth/OIDC, note per-provider username matching, clarify what each key does, and state that existing API tokens carry over. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…Dojo#16357) * perf(findings): index findings by test and numerical severity Listing one test's findings ordered by numerical severity with a small limit (test=<id>&o=-numerical_severity&limit=10) has no composite index to walk. On tests with hundreds of thousands of findings Postgres sorts every finding in the test or walks the severity-only index and filters by test. Add idx_finding_test_numsev on (test, numerical_severity, id), built with AddIndexConcurrently in a non-atomic migration so it takes no exclusive lock on dojo_finding. The trailing id serves an id tiebreaker for stable pagination, and the btree can be walked in either direction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(findings): say the index serves same-direction tiebreaks only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ls (DefectDojo#16315) sla_compute_and_notify() stopped at the first finding that raised, and notifications are only created after the loop, so one bad finding meant no SLA notification went out at all. It also loaded every open finding and fetched its test, engagement and product one query at a time. - Evaluate each finding in its own try and savepoint; guard each notification, JIRA comment and combined notification separately; still send what was collected if reading further findings fails. - Only fetch findings whose SLA ends inside the notify window, stream them in chunks, and prefetch their related rows per chunk. - Send each JIRA SLA comment through that finding's own JIRA instance, and stop a product with no JIRA project from raising UnboundLocalError. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…worker (DefectDojo#16325) * docs(pro): describe rehash progress and how a rehash survives a lost worker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(pro): correct how rehash batches run, recover and report Batches run in parallel, a lost batch is reported as Failed, redelivery happens at the latest after the visibility timeout, and the identity rehash item stays Running and cannot be canceled while its rehashes run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(pro): a single-batch rehash also runs in its own process Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(pro): document how a failed rehash recovers and which batches are retried A single-batch rehash is not retried after a kill, failed rehashes stay on the card for seven days, a rehash that has not moved for a day reads Failed, Run Again does not repeat a failed identity rehash, and manage.py dedupe --hash_code_only re-hashes a tool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(pro): a rehash is lost only after a day without progress; recovery skips duplicates Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(pro): say what a lost rehash shows when a batch finishes after all Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ectDojo#16360) django-ratelimit counts in fixed windows and puts the window end in the cache key. If test_failed_attempts_are_limited_when_blocking_is_on crossed the key's window edge between its failed attempts and the final request, the count reset and the request got 200 instead of 429. Freeze the clock django_ratelimit.core reads for each test in BasicAuthAccountRulesTest. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…16261) * docs(pro): document in-app walkthroughs and the App Tour Add an In-App Walkthroughs page: the lightbulb launcher, the App Tour, Resume and Restart, the Learn more links, the first-visit offer, the per-user switches on the User Profile page, and what is stored. * docs(pro): cover the walkthrough gates, All Walkthroughs and the no-data tour The lightbulb lists only walkthroughs for features the user can use, All Walkthroughs lists every one in a table, the App Tour runs without data, a missing step offers Try Again and Skip, and the tour steps aside for a dialog opened during a step. * docs(pro): describe the walkthrough notice that points at the sandbox Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…one manager (DefectDojo#16324) * fix(notifications): keep the daily SLA run going when one finding fails sla_compute_and_notify() stopped at the first finding that raised, and notifications are only created after the loop, so one bad finding meant no SLA notification went out at all. It also loaded every open finding and fetched its test, engagement and product one query at a time. - Evaluate each finding in its own try and savepoint; guard each notification, JIRA comment and combined notification separately; still send what was collected if reading further findings fails. - Only fetch findings whose SLA ends inside the notify window, stream them in chunks, and prefetch their related rows per chunk. - Send each JIRA SLA comment through that finding's own JIRA instance, and stop a product with no JIRA project from raising UnboundLocalError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(notifications): batch the daily SLA run's notifications through one manager The SLA run sent each notification through a new NotificationManager, so every notified finding re-read the system notifications row, re-resolved its product's recipients and wrote its alerts in an INSERT of its own: about 7 queries per finding, 368k for 45k findings. notification_batch() sends every create_notification() made inside it through one manager. The SLA run opens one batch: the system row is read once, each product's recipients once, and alerts share bulk INSERTs, while each notification still renders its own messages. Alerts are written only between notifications, outside their savepoints, so one notification's rollback cannot erase another's. In a batch a failed INSERT is retried row by row, and after 5 consecutive failures the rest is reported in one fallback alert. Outside a batch nothing changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(notifications): let the SLA period method load what it reads with the run A FINDING_SLA_PERIOD_METHOD can read more of each finding than the SLA run loads. DefectDojo Pro's reads each finding's companion row for risk-based and KEV-capped configurations, which cost one query per notified finding. FINDING_SLA_NOTIFY_QUERYSET_METHOD, when set, receives the run's findings queryset and returns it with what the period method needs, so that arrives with each chunk. Unset, the run is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(notifications): keep batched SLA notifications isolated and bounded Fixes from a review of the batched SLA run: - merge_notifications_list returned its first entry itself, and callers set .user on the result. For a superuser with no settings that entry is the manager's system notifications row, so in a batch every later notification's system-level copy went to that superuser. It now merges into a copy. - A notification whose every alert row fails (5 or more recipients) tripped the outage bound and cost the rest of the batch. The bound now counts a streak only when rows of more than one notification fail. - The bound held per flush, not per run. Failure state now lives on the batch: once tripped, later flushes are not attempted, and everything not written is reported in one fallback alert when the batch ends. - notification_batch() falls back to unbatched sending when the configured NOTIFICATION_MANAGER has no batch() or cannot be built. - A Slack user id resolved during the run is kept on the cached user, so it is looked up once rather than once per notification. - A FINDING_SLA_NOTIFY_QUERYSET_METHOD that raises, is misnamed, or yields a query that cannot compile is logged and the run reads its findings without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(notifications): bound outages by connection errors and read through a broken hook Second-round review fixes for the batched SLA run: - An outage is detected after 5 consecutive connection-level failures (OperationalError, InterfaceError) whatever notification they belong to, so it no longer costs one attempt per recipient of the first failing notification. Data errors keep the per-notification rule. One traceback is logged per flush. - The end-of-batch report of unwritten alerts is guarded, so a failure to write it cannot escape the run or hide an earlier exception. - A FINDING_SLA_NOTIFY_QUERYSET_METHOD whose query cannot be read on the first chunk (a bad prefetch_related, a non-query) falls back to the plain query, instead of reading nothing. The compile-time check is gone, so a hook whose query is valid but empty is respected rather than ignored. - Only the current product's recipients are cached for the batch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Document the new StackHawk upstream connector: prerequisites (an API key from Settings > API Keys, on a plan with API access), the connector mappings, how applications, environments and alerts map to Records and findings, and that synced findings share the StackHawk HawkScan test type and dedupe key with imported webhook reports. Add StackHawk to the upstream connector lists. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…group members open their own groups (DefectDojo#16364) Changing or removing a Member who holds the Owner role now takes the same permission as granting Owner. A Member of a user group sees the groups they belong to and can open those groups' pages without the View Group configuration permission; group history keeps that permission. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 2.1.3 to 2.1.4. - [Release notes](https://github.com/sqlalchemy/sqlalchemy/releases) - [Changelog](https://github.com/sqlalchemy/sqlalchemy/blob/main/CHANGES.rst) - [Commits](https://github.com/sqlalchemy/sqlalchemy/commits) --- updated-dependencies: - dependency-name: sqlalchemy dependency-version: 2.1.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
) Bumps [@scalar/api-reference](https://github.com/scalar/scalar/tree/HEAD/packages/api-reference) from 1.72.4 to 1.73.1. - [Release notes](https://github.com/scalar/scalar/releases) - [Changelog](https://github.com/scalar/scalar/blob/main/packages/api-reference/CHANGELOG.md) - [Commits](https://github.com/scalar/scalar/commits/HEAD/packages/api-reference) --- updated-dependencies: - dependency-name: "@scalar/api-reference" dependency-version: 1.73.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…2.9.0 (.github/workflows/test-helm-chart.yml) (DefectDojo#16344) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
….github/workflows/validate_docs_build.yml) (DefectDojo#16342) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…idator action from v2.2.0 to v2.4.1 (.github/workflows/renovate.yaml) (DefectDojo#16345) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…jo#16355) Bumps [djangorestframework](https://github.com/encode/django-rest-framework) from 3.18.1 to 3.18.3. - [Release notes](https://github.com/encode/django-rest-framework/releases) - [Commits](encode/django-rest-framework@3.18.1...3.18.3) --- updated-dependencies: - dependency-name: djangorestframework dependency-version: 3.18.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ctDojo#16363) * fix(parsers): read CVE/GHSA ids from SARIF tags and OSV aliases The SARIF parser set a vulnerability id only when the ruleId contained a CVE. Tools that key a result by a GHSA or RUSTSEC id and list the CVE as a rule or result tag produced Findings with no CVE at all. Tags that are exactly a CVE or GHSA id now also become vulnerability ids, CVEs first, and stay on the Finding as tags. The OSV Scanner parser stored only the advisory id and dropped its aliases, so a GHSA-keyed advisory lost its CVE. The aliases now follow the id in the vulnerability ids. Neither field feeds the default hash code for these parsers, so deduplication is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(parsers): harden SARIF tag and OSV alias id extraction - OSV: ignore an aliases value that is null or not a list, and keep only non-empty strings, stripped. A null used to raise TypeError and a string was split into one id per character. - SARIF: match CVE/GHSA ids with ASCII-only patterns, so non-ASCII digits and look-alike letters do not make an id, and strip surrounding spaces. - SARIF: a ruleId that is a GHSA id is kept as a vulnerability id after the CVEs, matching how the OSV parser keeps its GHSA id. - Docs: describe the case-insensitive, normalized matching. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ags page (DefectDojo#16368) Add pages for the five features on the Feature Flags page that had no documentation: Celery Status, Location Map, Remediation Center, Certification Control Catalogs and Regulatory Profile. Location Map and Remediation Center were already linked from the product, so those links returned 404 until now. Add a Feature reference table to the Feature Flags page listing every flag on that page with its default and a link to its documentation. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…efectDojo#16367) * docs(sensei): Bedrock Test connection uses the API Base URL endpoint Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(sensei): a new Bedrock base URL needs the AWS secret entered again Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…efectDojo#16335) Rewrite the sidebar page's layout-switching section now that the reorganized sidebar is the only one, and add 3.4.200 upgrade notes. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…les, Card Facts, partial select matching (DefectDojo#16372) Documents the asset metadata (product.meta) and asset contact paths every rule item carries, asset custom fields in Finding and alert rules, the Card Facts setting on the Slack and Teams nodes, the webhook body additions, and contains / starts with / ends with on select custom fields. Also corrects the Asset field table, which listed fields the envelope does not carry and named the asset type key asset_type instead of kind. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…cy fallback to WARNING
hash_fields() previously called .casefold().encode('utf-8').strip(), running
strip() on bytes. bytes.strip() only removes bytes 0x00-0x20 (ASCII control
chars and space), so Unicode whitespace (U+00A0 NBSP, U+2003 EM SPACE, etc.)
was left embedded in the data before hashing. The fix applies strip() on the
string before encoding: .casefold().strip().encode('utf-8').
Compatibility: all real-world hash fields (CVEs, file paths, titles, component
names) are ASCII with no leading/trailing whitespace, so no stored hash_code
values are affected. A hash difference can only arise for values with leading
or trailing non-ASCII whitespace, which no supported parser produces.
Four silent DEBUG-level messages in compute_hash_code() are promoted to WARNING,
each now including the scan type name so operators can act on them:
- missing HASHCODE_* configuration
- scan type not in HASHCODE_FIELDS_PER_SCANNER
- field not in HASHCODE_ALLOWED_FIELDS
- cwe is 0 and HASHCODE_ALLOWS_NULL_CWE is False
One silent DEBUG fallback in do_dedupe_finding_task_internal() is also promoted
to WARNING with the scan type name and a suppression hint.
Tests: add TestHashFieldsNormalisation (7 tests) covering determinism,
case-folding, ASCII and Unicode whitespace stripping, internal space
preservation, output format, and a pinned regression value. Add
TestLegacyFallbackWarnings (2 tests) asserting WARNING-level emission and scan
type inclusion. All 115 tests in the four target suites pass (skipped=2).
|
This pull request modifies sensitive files in the
Configured Sensitive Codepath Modified by Non-Allowed Author in
|
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/finding/deduplication.py' matches configured sensitive codepath pattern 'dojo/finding/*.py' and was modified by 'wynonaj' (commit 5ed6aaf) who is not in the allowed authors list. |
Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/finding/models.py (drs_0037bac2)
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/finding/models.py' matches configured sensitive codepath pattern 'dojo/finding/*.py' and was modified by 'wynonaj' (commit 5ed6aaf) who is not in the allowed authors list. |
Comment to provide feedback on these findings.
Report false positive: @dryrunsecurity fp [FINDING ID] [FEEDBACK]
Report low-impact: @dryrunsecurity nit [FINDING ID] [FEEDBACK]
Example: @dryrunsecurity fp drs_90eda195 This code is not user-facing
All finding details can be found in the DryRun Security Dashboard.
- Move hashlib and unittest.mock imports to top-level (PLC0415) - Change .encode() call to bytes literal (UP012/RUF010) - Add blank line between class declaration and docstring (D211) - Replace EN DASH with hyphen in docstring (RUF003) - Collapse nested with-blocks into parenthesized multi-context form (SIM117) - Simplify _make_finding() to return directly (RET504)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We don't want to waste your time, so if you're unsure whether your hypothetical enhancement meets the criteria for approval, please file an issue to get pre-approval before beginning work on a PR.
Learn more here: https://github.com/DefectDojo/django-DefectDojo/blob/master/readme-docs/CONTRIBUTING.md#submission-pre-approval
Description
Describe the feature / bug fix implemented by this PR.
If this is a new parser, the parser guide may be worth (re)reading.
Test results
Ideally you extend the test suite in
tests/anddojo/unitteststo cover the changed in this PR.Alternatively, describe what you have and haven't tested.
Documentation
Please update any documentation when needed in the documentation folder)
Checklist
This checklist is for your information.
dev.devbranch.Extra information
Please clear everything below when submitting your pull request, it's here purely for your information.
Moderators: Labels currently accepted for PRs:
Contributors: Git Tips
Rebase on dev branch
If the dev branch has changed since you started working on it, please rebase your work after the current dev.
On your working branch
mybranch:In case of conflict:
When everything's fine on your local branch, force push to your
myOriginremote:To cancel everything:
Squashing commits
pickbyfixupon the commits you want squashed outpickbyrewordon the first commit if you want to change the commit messageForce push to your
myOriginremote: