Skip to content

feat(detector): inventory Go modules and audit Go configuration - #230

Merged
ashishkurmi merged 5 commits into
step-security:mainfrom
raysubham:feat/go-module-inventory
Sep 30, 2026
Merged

ashishkurmi merged 5 commits into
step-security:mainfrom
raysubham:feat/go-module-inventory

Conversation

@raysubham

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds Go evidence to enterprise telemetry as two optional sections, go_inventory and go_config_audit (each schema_version: 1). Both are full bounded snapshots sent on every run, independent of the npm/Python deltas.

go_inventory (static, no Go toolchain needed)

  • Projects and workspaces: go.mod / go.work found by one bounded walk of the configured search roots. Requirements, replacements, excludes, tools and workspace membership, parsed with golang.org/x/mod/modfile.
  • Alternate manifests: an absolute -modfile from GOFLAGS, split the same way cmd/go splits it.
  • Vendored modules: vendor/modules.txt, counted only when a package directory exists on disk.
  • Module cache: extracted module@version directories and cache/download archives. ZIP metadata is checked but never decompressed. Extraction completeness follows Go's .partial / .ziphash rules.
  • Installed tools: binaries in the resolved GOBIN (or first GOPATH/bin), read via debug/buildinfo.
  • Recorded checksums from go.sum, go.work.sum, alternate .sum, .ziphash and BuildInfo Sum. These attach only to existing evidence, and are always not_verified.

go_config_audit

  • An allowlist of Go env settings from the user go/env file, the process environment (only when the agent runs as the developer), and GOROOT/go.env.
  • Findings go-001..go-006: sumdb off, GOINSECURE, private proxy falling back to a public one, credentials in URLs, GOAUTH command, malformed or duplicate keys.

Safety properties

  • No commands, shell sourcing, network calls or UserAwareExecutor. Every read goes through guarded, bounded executor file methods.
  • New opt-in Executor.ReadDirLimit (bounded ReadDir). Existing ReadDir / ReadFile are unchanged.
  • Protected directories stay excluded even with include_tcc_protected. The only exception is the exact macOS default ~/Library/Application Support/go/env file.
  • Redaction happens on the device. URL userinfo, query and fragment are stripped. GOAUTH arguments and non-module GOFLAGS are dropped. BuildInfo settings are never read.
  • Any refusal, failure or limit marks the affected source partial with a reason code, so it never reads as absence. Record and output caps (16 MiB for both sections) keep a status envelope.
  • Stable source IDs and deterministic ordering, so an unchanged machine produces identical bytes.

Wiring: a new go_scan phase after browser_extensions_scan reuses the resolved browser target user. Community scan output is unchanged. golang.org/x/mod v0.39.0 becomes a direct dependency, with no other module changes.

Contract: the golden fixture internal/model/testdata/go_inventory_v1_golden.json is strictly decoded and round-tripped. Tests also check reference integrity and coverage of the whole vocabulary. The same bytes are meant for the Agent API tests.

Type of change

  • Bug fix
  • Enhancement
  • Documentation (SCAN_COVERAGE.md, README.md)

Testing

  • Tested on macOS (version: 27.0)
  • Binary runs without errors (make smoke: 45/45)
  • JSON output is valid: not applicable, the sections are enterprise telemetry only and community JSON is unchanged
  • No secrets or credentials included
  • Lint passes: make lint (gofmt, vet, golangci-lint: 0 issues)
  • Tests pass: make test (go test ./... -race -count=1)

Also:

  • CGO_ENABLED=0 builds for linux/amd64, darwin/arm64 and windows/amd64; go mod tidy shows no drift.
  • gosec: one new G304 (file path from a variable), the raw os.Open in Real.ReadDirLimit. It mirrors the existing ReadDir / ReadFile.
  • Local end-to-end: two send-telemetry --telemetry-out runs with an isolated config and fake endpoint, reading the real home.
    • Both Go sections were present and byte-identical across the runs.
    • Fake GOPROXY / GOAUTH credentials never appeared in the JSON, stderr or execution logs.
    • go_scan took about 5 s.

Not yet done (release checks):

  • Three-VM acceptance (macOS, Linux, Windows) with Go 1.27.1 fixtures.
  • Native Windows test run; most scanner tests skip there, but they compile and vet.
  • A second scan with npm/Python deltas enabled.
  • Payload sizing on representative machines. On my Mac the Go sections were about 6.9 MB of an 11.6 MB payload.

Related Issues

Add two optional enterprise telemetry sections, go_inventory and
go_config_audit (schema_version 1), sent as full bounded snapshots on
every run, independent of the npm/Python deltas.

go_inventory statically collects go.mod/go.work declarations, alternate
-modfile manifests, vendor/modules.txt, the module cache and binaries in
the resolved GOBIN, with recorded (never verified) checksums from go.sum,
go.work.sum, .ziphash and BuildInfo. go_config_audit reports an allowlist
of Go env settings from the user go/env file, the verified process
environment and GOROOT/go.env, redacted on the device, with findings
go-001..go-006.

The collector runs no commands, sources no shell and makes no network
calls. Every read goes through guarded, bounded executor file methods
(new opt-in ReadDirLimit), protected directories stay excluded even with
include_tcc_protected, and any refusal, failure or limit marks the
affected source partial so it never reads as absence.
Comment thread internal/executor/executor.go Fixed

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Collection bounds, Windows path handling, archive validation, and nil-logger safety have unresolved correctness issues.

Review effort: Balanced
Findings: 2 High severity · 2 Medium severity

Open (4)
What changed in this PR

Adds bounded Go module inventory and configuration auditing to enterprise telemetry.

Changes:

  • Inventories projects, workspaces, vendor/cache modules, tools, and checksums.
  • Audits and redacts Go environment configuration.
  • Adds guarded directory limits, telemetry wiring, documentation, and tests.
File Description
SCAN_COVERAGE.md Documents Go scan coverage and safety.
README.md Advertises Go inventory and configuration auditing.
go.mod Adds golang.org/x/mod.
go.sum Records dependency checksums.
internal/​telemetry/​telemetry.go Runs and emits the Go scan.
internal/​telemetry/​telemetry_out_test.go Tests optional telemetry sections.
internal/​executor/​executor.go Adds bounded directory reads.
internal/​executor/​mock.go Implements bounded reads in the mock.
internal/​executor/​mock_test.go Tests bounded executor reads.
internal/​executor/​user_aware.go Delegates the new executor method.
internal/​safepath/​reader.go Adds guarded bounded directory reads.
internal/​safepath/​reader_test.go Tests limits and guard behavior.
internal/​model/​model.go Defines Go telemetry contracts.
internal/​model/​go_golden_test.go Validates schema vocabulary and references.
internal/​model/​testdata/​go_inventory_v1_golden.json Provides the schema golden fixture.
internal/​detector/​goscan.go Implements Go evidence collection.
internal/​detector/​goscan_test.go Exercises scanner behavior and limits.
internal/​detector/​gometadata.go Parses Go manifests, caches, and build metadata.
internal/​detector/​gometadata_test.go Tests metadata parsing and checksums.
internal/​detector/​configaudit/​goenv.go Audits and sanitizes Go configuration.
internal/​detector/​configaudit/​goenv_test.go Tests configuration safety and findings.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +53 to +55
func NewGoScanner(exec executor.Executor, log *progress.Logger) *GoScanner {
return &GoScanner{exec: exec, log: log, protection: configaudit.GoProtection}
}
Comment thread internal/detector/goscan.go Outdated

// Record budget, in sorted order so the same records survive every run.
// Dropped rows always mark their owning source incomplete.
budget := maxGoRecords
Comment thread internal/detector/gometadata.go Outdated
Comment on lines +272 to +273
if !strings.HasPrefix(f.Name, prefix) {
return model.GoArtifactUnreadable
Comment on lines +455 to +458
// goPathWithin reports whether path is strictly below dir, lexically.
func goPathWithin(path, dir string) bool {
return len(path) > len(dir) && strings.HasPrefix(path, dir) &&
(path[len(dir)] == filepath.Separator || strings.HasSuffix(dir, string(filepath.Separator)))
Spend the Go record budget as records and sources are collected instead
of trimming output in finish(): once it runs out nothing more is read,
and the truncated source and every enclosing root are marked partial
with record_limit.

Compare filesystem paths case-insensitively on Windows and macOS for
root dedup and overlap, workspace members, -modfile containment, the
manifest lookup and vendor dedup. Module paths, emitted paths and
source IDs keep their original case.

Validate module ZIP entry paths as x/mod/zip's checkZip does.
@ashishkurmi
ashishkurmi merged commit 7f776e3 into step-security:main Sep 30, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants