Repository navigation
Speed up resolution - #61
Merged
Merged
Conversation
PackageLister looked up root dependencies with sub dependencies by scanning the list of root dependencies for every version and dependency query, most of which don't match any. Resolving hexpm's dependencies made 13,720 such lookups over 126 root dependencies. Index them by repo and package when the lister is created, keeping the first declaration like Enum.find/2 did.
PackageLister asked the registry for a package's versions every time it considered the package, which pick_package/2 does for every unsatisfied package on every decision. Resolving hexpm's dependencies through Hex made over 10,000 such calls for 155 packages, each through Hex's registry process. Keep the result in the lister for the rest of the solve. This also means the solver works with one list of versions per package even if the registry filters versions by time.
pick_package/2 filters every unsatisfied package's versions by its constraint on every decision, though most packages stay unsatisfied with the same constraint over many decisions. Resolving hexpm's dependencies evaluated 9,878 package ranges and 256,410 version checks in 156 decisions. Cache the number of allowed versions and the preferred one per repo, package and constraint for the solve. Package ranges are still evaluated in name order, so ties and errors pick the same package as before.
dependencies_as_incompatibilities/4 loaded the dependencies of every version of a package to find the range of versions around the selected one that depend on each dependency with the same constraint. Resolving hexpm's 153 dependencies through Hex made 3,689 dependency lookups. Walk out from the selected version instead, loading versions only until the constraint changes, and keep loaded dependencies for the solve so revisiting a package doesn't load them again. Versions outside these runs are no longer loaded, so a registry that warns about invalid requirements when loading dependencies, like Hex, only warns about the versions the solver looked at.
Cover the package lister returning cached candidates for the same package range, keying them on the repository and constraint, picking the first package by name among those with the fewest allowed versions, and returning the first package by name without versions after other package ranges' candidates are cached.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These make
PackageListerdo less work per decision, most of all in projects with many dependencies. The counts below come from profiling Hex's offlinemix deps.get --check-lockedon a copy of hexpm'smix.exsandmix.lock(155 locked packages) at hexpm/hex@eaa8bd6, which vendors hex_solver 0.3.0.versions/2calls for 155 packagesdependencies/3calls for 153 resolved packagesHex vendors this branch in hexpm/hex#1272. Together with the Hex changes there, resolution in offline
mix deps.update --allon the same project went from 213 ms to 22 ms (medians of 9 runs, macOS, Elixir 1.20.4 / OTP 29.1.1), and it writes the samemix.lockas before.Registries see two differences.
versions/2is called once per package per solve, so the solver works with one list of versions per package even if the registry filters versions by time.dependencies/3is only called for the versions the solver looks at and the neighbouring versions it compares constraints with, so a registry that warns about or rejects dependencies when loading them, like Hex, only does that for those versions. Package ranges are still evaluated in name order, so ties and the missing package error pick the same package as before.The new
PackageListertests cover fetching versions once, loading dependencies only as far as the constraint stays the same, candidate caching and tie-breaking. The StreamData properties also pass withMAX_RUNS=20000.Changelog entries go in with the 0.3.2 release, which hexpm/hex#1272 needs before it merges.