You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: onboard a new non-LF organization as one project with every repository in one call (lfx-cdp-insights-ops#817) - #5030
First half of the last #817 step (after #5015, #5026 and #5027). Today a GitHub organization that resolves to a new non-LF project goes to the CDP team as a by-hand card, because approving it would have created one project per repository. This PR adds the onboarding side: one project, every repository, one integration call. Nothing changes for users yet; the next PR replaces the by-hand card with one evaluation per organization and an Approve button that lands here.
What changes
One project for the organization.onboardOrganizationProject resolves one segment for the whole list and connects every repository through the existing connectReposToSegment path (one github-nango-connect call, existing repositories sent back on their segments). The project name is the requested one, otherwise derived from the owner (ai-dynamo → Ai Dynamo); when that name already belongs to another connected project the fallback is Name (github.com/owner), each candidate with its own slug so the fallback cannot collide on the first candidate's slug.
Retries land on the same project.resolveProjectSegment now takes the list of requested repositories: a segment left by a failed earlier attempt, connected to some of them, is reused instead of raising the "already used by a project with a GitHub integration" error. Single-repository behaviour is unchanged (list of one).
Worker.groupOnboardingUnits groups the create_segment rows of an organization like the update_integration / create_integration ones; runOrganizationProposedAction sends such a group to onboardOrganizationProject and still refuses a mixed group.
Fork lookups.fetchGithubOrgsSettings ran one GitHub request per repository at once; they now go through mapWithConcurrency (new in @crowd/common, five at a time, stops scheduling after the first failure).
Out of scope (next PR)
Remove the by-hand routing in classifyAndRoute and evaluate a new non-LF organization once, sharing the verdict with the sibling rows.
Organization review reason that shows the name the project would get.
Tests
onboardOrganizationProject: new project, owner-derived name, retry on an existing segment, fallback name, empty / non-GitHub list, missing configuration.
resolveProjectSegment with a subset of the requested repositories already connected.
runOrganizationProposedAction creation group, groupOnboardingUnits with create_segment rows.
…ository in one call
- onboardOrganizationProject creates one segment (requested name, else the
owner's; "Name (github.com/owner)" when the name belongs to another
project, each candidate with its own slug) and connects every repository
with one integration call
- resolveProjectSegment reuses a segment left by a failed attempt when its
repositories are among the requested ones
- the onboarding worker groups the create_segment rows of an organization
into one unit; mixed groups stay refused
- GitHub fork lookups run at most five at a time through mapWithConcurrency
in @crowd/common
Refs linuxfoundation/lfx-cdp-insights-ops#817
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Medium Risk
Changes core onboarding, segment resolution, and integration wiring with new locking; mistakes could create duplicate projects or corrupt repo-to-segment mappings, though behavior is heavily tested.
Overview
Adds organization-wide onboarding for new non-LF projects: repos with create_segment in the same GitHub org are grouped and handled by onboardOrganizationProject, which resolves one segment, names it from the request or owner (with owner-qualified fallback), and connects all repos in a single integration call instead of one project per repo.
resolveProjectSegment now works over multiple repo URLs (and optional org owner) so retries reuse a segment already tied to some of those repos or other repos from the same owner. The automatic onboarding worker groups create_segment org rows like join-existing flows, routes homogeneous groups through the new onboarder, and still rejects mixed proposed actions.
Concurrency and races: new mapWithConcurrency caps parallel GitHub fork lookups; lockProjectCatalogOnboardingTarget accepts arbitrary lock keys (including organization:<owner>), with claims for new-org creation taken under that lock in one transaction so parallel runs cannot split one org across multiple projects.
Reviewed by Cursor Bugbot for commit 248a712. Bugbot is set up for automated code reviews on this repo. Configure here.
Adds the onboarding foundation for creating one non-LF project per GitHub organization, extending the existing grouped integration flow. User-facing approval remains deferred.
Changes:
Adds organization-wide project creation, naming fallback, and partial-retry reuse.
Routes grouped create_segment rows through the new onboarding path.
…repo creation unit whole
- a creation group whose rows were claimed in part by another run releases
its claims instead of creating a second project for the rest
- an organization with one repository still goes through
onboardOrganizationProject, so the requested name is used
Refs linuxfoundation/lfx-cdp-insights-ops#817
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
The first repository's fork lookup runs outside the limiter through fetchGithubOrgSettings, so an organization with at least six repositories can issue six concurrent fork requests despite the limit of five. Each owner also gets a separate pool, multiplying concurrency for multi-owner inputs. Run all repositories' fork lookups through one shared mapper, then group the results and fetch each owner's logo.
…ing group
A run reads the organization without the rows another run holds at that moment, so a
creation group is re-read and claimed under an advisory lock on the organization; the
lock helper takes any target, not only a PCC project id.
Refs linuxfoundation/lfx-cdp-insights-ops#817
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Version this grouping change with a new Temporal patch. A lone organization row with create_segment previously scheduled onboardAndUpdateProject; it now schedules skipRowsOfDeletedProjects and onboardAndUpdateOrganization. Such rows can already reach onboarding because the manual-routing guard in classifyAndRoute.ts:74 excludes single-repository organizations. Replaying an existing workflow that scheduled the old activity can therefore fail with nondeterminism. The existing issue-817-organization-onboarding marker in workflows/onboardProjects.ts:71 covers both implementations; preserve the previous grouping for histories without a new marker.
Enforce one concurrency limit for all repository fork lookups
The first repository's fork lookup runs outside mapWithConcurrency, so an organization with six or more repositories can start six fork requests despite the limit of five. Each owner also gets a separate pool, so multi-owner inputs have no shared limit. Run every repository's fork lookup through one bounded map, then group the results by owner.
… that already holds it
- the claims of a creation group are taken under the organization lock in one
transaction, before any external call, so no transaction spans GitHub or
backend requests
- a candidate project already connected to repositories of the same owner is
that organization's project: a row left behind by a dead claim or a failed
attempt joins it instead of creating a second one
Refs linuxfoundation/lfx-cdp-insights-ops#817
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Replaying a history that onboarded a lone organization create_segment row now schedules skipRowsOfDeletedProjects instead of the recorded onboardAndUpdateProject. Such rows are already possible: newOrganizationProjectByHand leaves single-repository organizations unchanged. The existing issue-817-organization-onboarding patch marker predates this grouping change, so it does not protect those histories from a replay mismatch. Gate the new grouping with a fresh Temporal patch marker and preserve the previous grouping for old histories.
The first repository's fork lookup runs inside fetchGithubOrgSettings, outside the limiter. An organization with at least six repositories can therefore run six fork requests at once despite the configured limit of five. Separate pools per owner multiply the limit for multi-owner inputs. Route all fork lookups, including the first repository, through one shared pool before assembling the organization settings.
…ganization
Two requests for one organization join the same project and the backend replaces the
integration's repository list on every connect, so their resolution and connect run under
one lock keyed by the organization, like the PCC project lock for existing projects.
Refs linuxfoundation/lfx-cdp-insights-ops#817
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
This changes the activity sequence for existing Temporal histories containing organization create_segment rows. workflows/onboardProjects.ts:71–73 calls this helper under the existing issue-817-organization-onboarding patch: those rows previously scheduled onboardAndUpdateProject, but now schedule skipRowsOfDeletedProjects and onboardAndUpdateOrganization, even for a single row. Replaying such a history after deployment can fail with nondeterminism. Add a new workflow patch marker for creation-group handling and preserve the previous grouping when that marker is absent from a replayed history.
Different organizations requesting the same project name take different owner locks but can still resolve to the same segment. If one run creates Acme and has not connected its integration yet, another organization's run can reuse that segment because resolveProjectSegment accepts segments with no GitHub integration. Both groups can then connect to one project instead of the second using its owner-qualified fallback, and their integration writes are not serialized. Serialize shared candidate-name resolution through connection, or lock the resolved segment and revalidate ownership before connecting.
Include initial repository lookup in the concurrency limiter
For an organization with at least six repositories, this can start six fork lookups concurrently: five through the limiter and one inside fetchGithubOrgSettings(first, ...). The first repository therefore bypasses the configured five-lookup cap. Include all of the organization's repositories in the limiter and fetch its logo separately.
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
First half of the last #817 step (after #5015, #5026 and #5027). Today a GitHub organization that resolves to a new non-LF project goes to the CDP team as a by-hand card, because approving it would have created one project per repository. This PR adds the onboarding side: one project, every repository, one integration call. Nothing changes for users yet; the next PR replaces the by-hand card with one evaluation per organization and an Approve button that lands here.
What changes
One project for the organization.
onboardOrganizationProjectresolves one segment for the whole list and connects every repository through the existingconnectReposToSegmentpath (onegithub-nango-connectcall, existing repositories sent back on their segments). The project name is the requested one, otherwise derived from the owner (ai-dynamo→Ai Dynamo); when that name already belongs to another connected project the fallback isName (github.com/owner), each candidate with its own slug so the fallback cannot collide on the first candidate's slug.Retries land on the same project.
resolveProjectSegmentnow takes the list of requested repositories: a segment left by a failed earlier attempt, connected to some of them, is reused instead of raising the "already used by a project with a GitHub integration" error. Single-repository behaviour is unchanged (list of one).Worker.
groupOnboardingUnitsgroups thecreate_segmentrows of an organization like theupdate_integration/create_integrationones;runOrganizationProposedActionsends such a group toonboardOrganizationProjectand still refuses a mixed group.Fork lookups.
fetchGithubOrgsSettingsran one GitHub request per repository at once; they now go throughmapWithConcurrency(new in@crowd/common, five at a time, stops scheduling after the first failure).Out of scope (next PR)
classifyAndRouteand evaluate a new non-LF organization once, sharing the verdict with the sibling rows.Tests
onboardOrganizationProject: new project, owner-derived name, retry on an existing segment, fallback name, empty / non-GitHub list, missing configuration.resolveProjectSegmentwith a subset of the requested repositories already connected.runOrganizationProposedActioncreation group,groupOnboardingUnitswithcreate_segmentrows.mapWithConcurrency: order, peak concurrency, failure propagation.Refs linuxfoundation/lfx-cdp-insights-ops#817
🤖 Generated with Claude Code