Repository navigation
Release 2.2 - #23
Merged
Merged
Release 2.2#23
Conversation
The lexer created a Token object for every piece of every file, although most are never read. It now records rows in a compact TokenTable and hands out LazyTokens that build a token only when variable detection or an engine asks for it. Type chains and stream items are computed once per Pygments type, and SingleTokenImprover decides from the current token and the two before it instead of rebuilding tokens. Output is identical to 2.1.1: test_lexer_equivalence compares every fixture with a frozen copy of the 2.1.1 tokenizer.
The model expanded its bloom filter into a list of ints at import: 585 ms and 31 MB in every worker. The bits now stay packed in bytes and are tested in place, with the same answers.
Random letters-only strings (an AWS key id in capitals, a keyboard-mash password) could score as English because the dictionary's short entries and the bloom filter's false positives cover them. A string whose whole-string trigram score is below PLAUSIBLE_TRIGRAMS is now not language (KI-ENG-02). Two known_imbalance cases pass and lose their marker.
`patterns` in Rule.fill_confidence_and_file_patterns was written with a Cyrillic 'е' (KI-DM-02).
- Workers start from a forkserver where the platform has one (Linux, macOS), spawn elsewhere; the server preloads the scanner and freezes the garbage collector, so workers share it. --multiprocessing-context keeps choosing spawn or fork. The progress manager uses the pool's context. - CI mode: no live display and no manager process, a plain progress line every 30 s. On inside CI services and when output is not a terminal; --ci/--no-ci force it. - A worker killed outright (OOM killer, SIGKILL) no longer hangs the scan: its file is failed with WorkerLostError (KI-CLI-37). - A worker that cannot load the analyzer bundle stops the scan with WorkerStartupError instead of hanging it; a bundle deleted mid-scan is staged again (KI-DM-30). - Multiprocessing sockets go to a short directory when TMPDIR is too long (KI-CLI-40), and the CLI waits for the forkserver after the scan so the workers' CPU time reaches the caller (KI-CLI-42).
SEM_VAR_FILE_PATHS reads nothing but the file path, so its answer is the same for every variable of a file; the evaluator now remembers it per rule.
Files larger than --deep-max-size (default 250,000 bytes) get the regex rules and the cheap variable search without the language lexer, which cost most of the scan time on them and rarely found more. 0 analyses every file fully. The hashed engine keeps the lexer on every file. --report-diagnostics records each file's depth.
S106 reported a low-entropy value under any secret-like name. A value of 3 bits per character or less is now reported only when a rule marked allows_low_entropy fires on the name (SEM_VAR_HUMAN_CHOSEN_SECRET_NAMES: passwords and secrets people choose), not for issued keys and tokens. The diagnostics tracer predicts the rule the same way.
- choose_final_rule breaks confidence ties by the rule's position in its ruleset, then its id, instead of the set order that varied with the hash seed (KI-DM-15). - Every confidence tier carries security-severity 10.00: severity is how bad a leak is and does not depend on confidence.
A regex match is a candidate, judged in RegexEngine by its own scoring rules (rules/regex_candidate_scoring_rules.json, --regex-candidate-scoring-rules), the way a variable is judged by the variable-scoring rules. - Rules declare an evidence class (format or shape) and type their capture groups (value, name, user, host, kind); a value group can declare its encoding (base64, base32, hex, base58check). - Key rules judge the body between the markers: the payload without line breaks, quotes, concatenations, headers and decoration, then its foreign share, randomness and checksum. S5/S26 lose their negative patterns; S18 reports the service account's private key. - A rejected candidate is dropped, or kept flagged at confidence 0 when asked; a match covering a variable the semantic engine reported is corroborated and keeps its rule's confidence. - Payload patterns run in linear time; a comparison on a text field is refused when the rules load; an empty match is never reported.
all, low (default), medium, high or very-high: the lowest confidence to report. all also reports the regex candidates the evaluation rejected, at confidence 0 with an -INFO rule id.
- Regex rules S42-S56 for SonarQube, GitHub classic tokens, Algolia, Flickr, Bitly, Okta, Snyk, Telegram, Slack webhooks, Stripe, Facebook, Dropbox and Amplitude, in a keyword-first form adapted from gitleaks and Betterleaks. - The cheap variable search skips HTML tags within bounds where the lexer also reads the file (the wide skip stays above the size tier on purpose, KI-TOK-36), and reads command-line options and <add key value> settings.
Also skip minified JavaScript, source maps and bundles (*.min.js, *.map, *.bundle.js). Off by default: the size tier already makes most of them cheap, and skipping them outright loses more secrets.
LexerFinder re-executed the jsx-lexer source for every scanned file and the lexer was never chosen: Pygments' own JsxLexer answers the jsx and react aliases and file names (KI-TOK-18).
The tracer replays the size tier, the cheap search's tier-dependent detectors, the regex-candidate judging and the allows_low_entropy gate, and names the stage where a value was lost (schema version 5).
The config was a process-wide singleton that parse_arguments mutated, so a second run in the same process inherited the first one's exclusion files, rulesets and flags; after --disable-masking, a masked run still wrote plaintext secrets (KI-CLI-12). Each parse now builds its own Config and the module-level instance is gone.
iscan_mode called setup_interrupts_for_subprocess() at import, which made any non-main process of an application that merely imported the scanner ignore Ctrl+C. init_worker already sets it for every start method (KI-CLI-17).
A value that produced no finding hid every later copy of it in the file, even under a secret-like name (SMTP_USER = 'superset' then SMTP_PASSWORD = 'superset'), and in 2.2 a rejected regex candidate did the same. The cache now keys on content and name (KI-ENG-08).
- A reused CheapVarSearchTokenizer or LexerTokenizer returns only the current file's tokens. - The lexer iterates Pygments' token generator instead of building a list. - Invalid escape sequences in variable-detection patterns are doubled, so Python 3.12+ compiles the module without a SyntaxWarning.
- The worker's last progress write used the literal key 'task_id' and lost the file analyzer's counts (KI-CLI-04). Tokens now travel in the result, and File Errors are recounted from the outcomes, so CI mode and the live display print the same summary. - The live display starts after the pool has created its workers, so fork workers are not forked from a process with the display's thread. - A file that was never analysed reports no depth.
- Version 2.2.0; requires Python 3.11 (tomllib). - --outformat json is refused before the scan, as announced in 2.1.0. - start() split into plan, summary, report and release steps; the report is built before its file is opened. - The --outfile directory is checked while parsing; a bare --hashed-values disables the check; --skip-bundles works with custom --excluded-paths; exclusion files keep their order; --multiprocessing-context offers only the platform's start methods. - README: new flags, confidence tiers, Python version.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.