meta_package_manager.dispatch module

Cross-manager dispatch: scheduling many package managers at once.

Where meta_package_manager.execution runs one manager’s CLI in one subprocess, this module schedules many managers concurrently: the job-count policy that decides sequential-vs-concurrent (effective_jobs()), the up-front availability probe used during selection (warm_availability()), the two progress-wrapped fan-out primitives the CLI subcommands drive (collect_from_managers(), collect_per_package()) with their shared dispatch() engine, the backend-lock catalog that serializes conflicting managers (SHARED_LOCK_FAMILIES and merge_into_lock_lanes()), and the manager-bound ✓/✗ ledger (OperationTrail) that the concurrent and sequential paths both report through.

The generic layers live upstream in click-extra: the concurrency primitives in click_extra.execution (run_lanes driven by mpm --jobs) and the batch-reporting trail in click_extra.spinner (OperationTrail with its trail_glyph/trail_line atoms). This module keeps what is package-manager policy: which managers must never overlap, how the trail binds to the pool’s --progress state, and when a batch collapses to a sequential pass.

class meta_package_manager.dispatch.LockFamily(backend, members, contention)

Bases: object

A set of managers contending for one backend lock.

backend: str

Short name of the contended resource, labelling the family in the docs.

Rendered as the middle level of meta_package_manager._docs.lock_families_sankey() and as the first column of meta_package_manager._docs.lock_families_table(), so it reads as a thing managers queue on rather than as a tool: pacman database, not pacman.

That phrasing is also load-bearing for the diagram. Mermaid identifies a sankey node by its label alone, so a family named after a member (pkg, conda, pacman and scoop all name both) would fold the two levels into one self-linked node. test_lock_family_backends_are_distinct holds every name clear of the pool.

members: frozenset[str]

Manager ids that must never mutate at the same time.

contention: str

Why they collide, written to complete the sentence “``mpm`` never runs X at the same time as Y: …”.

Rendered on every member’s own documentation page by meta_package_manager._docs.manager_concurrency(), so it addresses a user of those managers rather than a reader of this module: keep it to the fact and its consequence, and leave the maintenance rationale to the notes below.

meta_package_manager.dispatch.SHARED_LOCK_FAMILIES: Final[tuple[LockFamily, ...]] = (LockFamily(backend='dpkg lock', members=frozenset({'pacstall', 'deb-get', 'nala', 'apt-mint', 'apt'}), contention='they all install through `dpkg` and serialize on its `/var/lib/dpkg/lock`'), LockFamily(backend='Homebrew update lock', members=frozenset({'cask', 'brew'}), contention="they are the same `brew` binary, and two concurrent `brew update` collide on Homebrew's own update lock"), LockFamily(backend='conda environment prefix', members=frozenset({'micromamba', 'mamba', 'conda'}), contention='they act on one environment prefix and one package cache, and `conda` honors none of the locks `mamba` takes on them'), LockFamily(backend='RPM database', members=frozenset({'urpmi', 'yum', 'zypper', 'dnf5', 'dnf'}), contention='they all reach the RPM database'), LockFamily(backend='pacman database', members=frozenset({'trizen', 'yay', 'pikaur', 'pacman', 'paru', 'pacaur', 'pamac'}), contention='they all reach the pacman database (`/var/lib/pacman/db.lck`), and two of them mutating at once fail to init their transaction'), LockFamily(backend='pkg install database', members=frozenset({'pkg', 'ports'}), contention='`ports` keeps no registry of its own and registers what it builds through `pkg`, whose advisory lock on that shared install database refuses a second writer'), LockFamily(backend='Scoop tree', members=frozenset({'sfsu', 'scoop'}), contention='they work on the same `~/scoop` tree, `sfsu` delegating its mutating operations to the `scoop` binary itself'))

Managers that contend for one shared backend lock, grouped by backend.

Different managers are otherwise independent processes over disjoint state, so running them in parallel is safe. The exception is a handful that drive a shared backend and serialize on its lock:

• apt, apt-mint, deb-get, nala and pacstall all reach dpkg (/var/lib/dpkg/lock). pacstall belongs here despite its pac prefix and its AUR-inspired design: it builds its pacscripts into .deb archives and installs those, so it contends with the Debian family and never touches pacman’s database.

• brew and cask are the same brew binary and serialize on Homebrew’s own update lock: two concurrent brew update (which mpm sync issues identically for both, as the formula/cask split does not apply to it) collide, one failing with “Another active Homebrew update process is already running”.

• conda, mamba and micromamba act on one environment prefix and one package cache. This is the family that does not get the guarantee below: mamba takes a real lock on the prefix and on every cache directory for the length of a transaction, and conda honors none of them, its own locking covering the repodata cache alone. Concurrent runs corrupt rather than block, which upstream closed as not planned (conda/conda#13037). Serializing them here is what keeps that out of reach.

• dnf, dnf5, yum, zypper and urpmi all reach the RPM database. urpmi fronts librpm directly, having no listing of its own, and the Mandriva lineage it serves ships dnf alongside it, so the two genuinely coexist on one host.

• pacman and the AUR helpers pacaur, pamac, paru, pikaur, trizen and yay all reach the pacman database (/var/lib/pacman/db.lck). The helpers are front-ends rather than reimplementations: each shells out to sudo pacman for the privileged steps, pamac reaching the same libalpm through Manjaro’s libpamac. Two of them mutating at once fail to init their transaction.

• pkg and ports share the install database pkg maintains, which every mutating operation but sync reaches: ports has no registry of its own, builds from /usr/ports and registers the result through pkg, whose advisory lock on that database refuses a second writer. Their sync is the one pair that would not collide, refreshing a git tree and a package catalog respectively, and is serialized along with the rest rather than splitting the family per operation.

• scoop and sfsu work on the same ~/scoop tree. sfsu reimplements Scoop’s read paths only, delegating install, remove and both upgrades to the scoop binary itself, so those are literally the same command twice; its own update and cleanup then reach the same buckets and cache Scoop’s do. Concurrent bucket refreshes are two git pull in one repository, which fails on the index lock.

dkp-pacman is deliberately not in the pacman family, and it is the one exclusion worth stating: it is Pacman by inheritance and would look like an oversight. But devkitPro ships it precisely so it can sit beside a distribution’s own pacman without colliding, pointed at its own repositories and its own database, so it contends with nothing.

pkcon is the case this model cannot express, and it is left out knowingly rather than filed under a guess. PackageKit is a client for whatever backend the host provides (apt, dnf, zypp, alpm), so the family it belongs to is a property of the machine rather than of the manager, and a frozenset here is fixed at import. It needs no protection from itself, packagekitd queuing its own transactions, but a pkcon mutation still contends with a native manager mpm drives in the same run. Expressing that would mean resolving the backend at dispatch time.

Concurrency is safe across families and unsafe within one, just as it is unsafe within a single manager (which is why a manager’s own packages stay serial). For every family above except the conda one, two members running at once block or fail rather than corrupt: each backend holds a real lock and the loser is told so. That is a property of those backends rather than of this mechanism, so a family added later earns the guarantee only by inspection, and conda is the standing proof that some do not.

Enforced for the mutating fan-outs only: merge_into_lock_lanes() collapses each family’s members into a single dispatch() lane, so they run serially while distinct families still run in parallel. The read-only queries (installed/outdated/search) take no backend lock, so they keep one lane per manager and stay fully concurrent. Members of a lane also share a command cache (see run_cache), so two that resolve to a byte-identical invocation (brew and cask for sync and cleanup) run the subprocess once.

Adding a newly-conflicting set of managers is one entry here: a LockFamily naming the backend and its members, after which the serialization, the command cache, the Concurrency section of every member’s documentation page and both renderings of docs/concurrency.md all pick it up.

meta_package_manager.dispatch.FAN_OUT_CONCURRENT: Final[str] = 'concurrent'

Every selected manager runs at once, one dispatch() lane each.

meta_package_manager.dispatch.FAN_OUT_GROUPED: Final[str] = 'grouped'

Same, but SHARED_LOCK_FAMILIES members are merged into one lane.

meta_package_manager.dispatch.FAN_OUT_SEQUENTIAL: Final[str] = 'sequential'

One manager at a time, whatever mpm --jobs says.

meta_package_manager.dispatch.FAN_OUT_NONE: Final[str] = 'none'

Runs no package operation, so there is nothing to spread.

class meta_package_manager.dispatch.FanOut(invocation, mode)

Bases: object

How one way of invoking a subcommand spreads over the selected managers.

invocation: str

The subcommand, plus whichever argument changes the answer.

install appears twice: a package tied to a manager rides the per-package fan-out, while one left untied needs the priority search that cannot be parallelized.

mode: str

One of the four FAN_OUT_* constants above.

property command: str

Bare subcommand name, as the CLI registers it.

meta_package_manager.dispatch.COMMAND_FAN_OUT: Final[tuple[FanOut, ...]] = (FanOut(invocation='cleanup', mode='grouped'), FanOut(invocation='config-template', mode='none'), FanOut(invocation='doctor', mode='grouped'), FanOut(invocation='dump', mode='concurrent'), FanOut(invocation='help', mode='none'), FanOut(invocation='install', mode='grouped'), FanOut(invocation='install <untied package>', mode='sequential'), FanOut(invocation='installed', mode='concurrent'), FanOut(invocation='managers', mode='none'), FanOut(invocation='orphans', mode='concurrent'), FanOut(invocation='outdated', mode='concurrent'), FanOut(invocation='remove', mode='grouped'), FanOut(invocation='restore', mode='grouped'), FanOut(invocation='sbom', mode='concurrent'), FanOut(invocation='search', mode='concurrent'), FanOut(invocation='sync', mode='grouped'), FanOut(invocation='upgrade', mode='grouped'), FanOut(invocation='which', mode='none'))

Fan-out mode of every mpm subcommand, rendered on docs/concurrency.md.

The three fan-out shapes above are visible from inside this module; which subcommand takes which is not, being an argument at each call site. This is where the two meet, so a reader can answer “does this command parallelize?” without following report_state=True through four CLI modules.

Kept complete rather than restricted to the commands that fan out: test_fan_out_covers_every_subcommand holds it equal to the CLI’s own command list, so a new subcommand fails the suite until someone decides its mode. The FAN_OUT_NONE entries are that decision recorded, and meta_package_manager._docs.concurrency_table() leaves them out of the rendered table.

Caution

Hand-maintained, and the one thing here that can drift from the code silently. A subcommand switching between collect_from_managers() and collect_per_package(), or gaining report_state=True, has to be reflected in the same commit: no test can read the mode back off a call site.

meta_package_manager.dispatch.effective_jobs(ctx, count)

Resolve how many worker threads to use for a batch of count items.

Thin wrapper over click_extra.execution.resolve_jobs() pinning mpm’s policy: always collapse to a single (sequential) worker at DEBUG verbosity, where coherent per-manager log narration matters more than the speed-up (interleaved threads would scramble it). The base helper also collapses to sequential with no active CLI context, for a single item, or at mpm --jobs 1; otherwise the mpm --jobs value wins, capped at count (no point spinning up more workers than there are items).

Return type:

int

meta_package_manager.dispatch.probe_signature(manager)

Static signature of the command manager’s version probe would spawn.

Built from class attributes alone: no filesystem lookup, no subprocess. Two managers sharing a signature search the same directories for the same binary names and pass it the same arguments, so their probes may resolve to a byte-identical command line. brew and cask do, and so do uv/uvx and yarn/yarn-berry.

Deliberately conservative in the safe direction. It never splits two managers that would spawn the same command, which is what merge_into_probe_lanes() needs to put them on one lane; it may however merge two that turn out to differ (the Zsh plugin managers all probe zsh, some with --version and some with version). A wrong merge costs the two a shared lane, where the second simply misses the cache and spawns as it would have anyway.

Not the resolved command line, on purpose: resolving it means walking PATH for every candidate up front, which measured slower than the redundant subprocesses it would save.

Return type:

tuple

meta_package_manager.dispatch.merge_into_probe_lanes(managers)

Group managers into warm_availability() lanes by probe_signature().

The probe counterpart of merge_into_lock_lanes(): managers whose version probe may resolve to the same command line land on one lane and run serially, while unrelated managers keep a lane each and run concurrently. Lanes come out in first-seen order, so a run is reproducible.

Return type:

list[tuple[PackageManager, ...]]

meta_package_manager.dispatch.warm_availability(managers)

Probe several managers’ available concurrently.

Reading available forces a manager’s --version detection, whose result (and the cli_path / executable / version it depends on) is cached on the instance. Warming the candidate set up front turns the sequential string of probes into a single round bounded by the slowest one, shaving startup latency off any command that touches many managers.

Managers on distinct lanes are distinct instances with their own cached attributes and subprocess, so their probes are independent and thread-safe; the GIL is released while each waits. The executor barrier publishes every cached value before the caller reads it back.

Probes run in merge_into_probe_lanes() lanes rather than one flat batch, so the managers that would spawn a byte-identical --version call take turns on one worker and share a run_cache: the first spawns, the rest replay its result. That is the same mechanism dispatch() gives a lock family, and it needs the lane for the same reason — a cache handed to managers running concurrently would just race them both into spawning.

Sized by effective_jobs() over the lane count: a no-op (leaving the probes to lazy, sequential evaluation) without an active context, at DEBUG verbosity, for a single lane, or at mpm --jobs 1.

Return type:

None

class meta_package_manager.dispatch.OperationTrail(managers, *, label='', unit='', total=0, jobs=1, coverage=False)

Bases: OperationTrail

click_extra.spinner.OperationTrail bound to the manager pool.

The upstream class owns the two renderings (sequential echoed lines, or one aggregate indicator with buffered-then-streamed lines) and the interactive gating; this subclass supplies mpm’s policy around it:

• Enablement follows ``–progress``, folded into each manager’s progress flag by the CLI (a TTY, no serialized output, not at DEBUG verbosity): any enabled manager turns the trail on, auto-gated on an interactive stderr.

• A concurrent batch mutes the managers’ own per-call spinners (which would collide on stderr) for the duration of the aggregate one.

• A concurrent batch’s aggregate indicator is a determinate progress bar, not an indeterminate spinner: every dispatch() batch counts its work up front (one task per manager, or per package-manager pair), so the bar always has a length to render against.

• ``coverage`` keeps the read-command semantics: their result table is the real output and each manager keeps its per-call spinner, so the sequential rendering stays silent (upstream’s echo_sequential=False).

The ordering-bound sequential state changers (install’s priority search) construct it bare; every dispatch() batch drives it as a context manager.

Parameters:

• managers (Iterable[PackageManager]) – the batch’s managers, read for the --progress gate and (when concurrent) to mute their per-call spinners.

• label (str) – present-tense verb for the running indicator (“Searching”).

• unit (str) – the noun counted in the indicator tally (“managers”, “packages”).

• total (int) – how many outcomes are expected, for the done/total count and the progress bar’s length.

• jobs (int) – the worker count from effective_jobs(); > 1 selects the concurrent rendering.

• coverage (bool) – when set, a sequential run stays silent (the caller has another output, its result table). Unused when concurrent.

Configure (but do not start) the trail.

Parameters:

• label (str) – present-tense verb for the running aggregate indicator ("Fetching"), composed into its {label} {done}/{total} {unit} tally.

• unit (str) – the noun counted in the tally ("files", "feeds").

• total (int) – how many outcomes are expected, for the done/total count.

• jobs (int) – the batch’s worker count; > 1 selects the concurrent rendering (one aggregate spinner), <= 1 the sequential one (plain echoed lines).

• spinner – a SpinnerPreset from the SPINNERS catalog (spinner=SPINNERS["moon"]) for the concurrent aggregate spinner. Ignored by the sequential and progress-bar renderings, and mutually exclusive with progress_bar.

• progress_bar – render the aggregate indicator as a determinate click.progressbar() instead of a spinner, for a sequential or concurrent batch alike. Requires a positive total (a bar needs a length) and is mutually exclusive with spinner.

• timer – append each operation’s and the batch’s elapsed time to the trail lines and the finisher. None (the default) follows the CLI’s --time / --no-time flag; True forces timing on with format_duration()’s compact clock, a callable (seconds: float) -> str forces it on with a custom format, and False forces it off. Per-operation times come from a seconds argument to mark(), filled in automatically by an operation() handle.

• clock – whether a running aggregate indicator shows elapsed time ("elapsed", the default: a stopwatch counting up, visible from the start) or remaining time ("eta": an estimate from the batch’s rate, appearing only once an outcome lets it be computed). Both the progress bar and the concurrent spinner honor "eta" (the spinner reuses Click’s progress-bar estimate, since the trail knows its total). Per-operation and finisher times are always elapsed.

• enabled – force the trail on or off. None (the default) auto-detects: the sequential echo renders only on an interactive stream, and the aggregate indicator applies its own TTY gate.

• echo_sequential – whether a sequential batch echoes its outcome lines and finisher at all. Turn it off when the batch has another output that is the real product (a result table) and the trail would be noise; an aggregate indicator is unaffected.

• delay – seconds before the aggregate indicator first draws: a fast batch then completes without ever flashing one.

• stream – where to render; defaults to sys.stderr so the trail never mixes into stdout data.

Raises:

ValueError – if progress_bar is set without a positive total, or together with spinner, or if clock is neither "elapsed" nor "eta".

meta_package_manager.dispatch.dispatch(label, done_label, unit, lanes, *, coverage=False, ctx=None)

Fan a set of work lanes out across managers, narrating a ✓/✗ trail.

The single scheduling primitive behind both collect_from_managers() and collect_per_package(). A lane is one or more managers paired with a list of callables; lanes run concurrently (one worker each) while a lane’s own callables run serially, because a package manager cannot safely run two of its own invocations at once, nor can two managers sharing a backend lock (see SHARED_LOCK_FAMILIES). A lane usually wraps a single manager; merge_into_lock_lanes() is what bundles a whole lock family into one, and such a lane also gets a shared command cache (see run_cache) so its members collapse identical invocations.

Each callable does its work, records its own outcome (output to INFO, failures into a caller-owned list) and returns (ok, message) for the trail. The whole batch reports through one OperationTrail: a per-outcome ✓/✗ line plus a finisher, behind a single aggregate progress bar when concurrent (a slow batch on a terminal) and silent otherwise.

Concurrency is sized by effective_jobs() (driven by mpm --jobs): it collapses to a sequential pass — preserving each manager’s own per-call spinner — for a single lane, at --jobs 1, or at DEBUG verbosity.

Parameters:

• coverage (bool) – forwarded to OperationTrail. Read commands set it (their result table is the output, so the sequential pass stays silent and the finisher reports coverage, {done_label} N {unit}, always ✓). Maintenance and state-changing commands leave it False (the trail is their output, so the finisher reports the success count, {done_label} N/M {unit}, ✗ on any failure).

• ctx (Context | None) – the active click context, read only to size concurrency (effective_jobs()). Defaults to the current context, so a command need not thread it; tests pass an explicit stand-in.

Return type:

None

meta_package_manager.dispatch.merge_into_lock_lanes(pairs)

Group (manager, task) pairs into dispatch() lanes, one per lock family.

Managers sharing a SHARED_LOCK_FAMILIES entry collapse into a single lane so their tasks run serially (the lane is dispatch()’s unit of mutual exclusion), while unrelated managers each keep their own lane and run concurrently. A manager not in any family keys on its own id, so its tasks still group together (a manager’s own invocations cannot overlap either). First-seen order is preserved, both across lanes and within a lane’s task list.

Used by the mutating fan-outs only: the state changers through collect_per_package(), and sync/cleanup/upgrade --all through collect_from_managers(). The read commands take no backend lock and skip this, keeping one lane per manager.

Return type:

list[tuple[tuple[PackageManager, ...], list[Callable[[], tuple[bool, str]]]]]

meta_package_manager.dispatch.collect_from_managers(label, done_label, managers, work, *, report_state=False, ctx=None)

Run work(manager) for every manager concurrently, results in input order.

The fan-out primitive for the read-only commands (installed/outdated/ search) and the independent maintenance commands (sync/cleanup/ upgrade --all). It adapts each manager into a dispatch() unit that runs work and stashes the (id, data) result in input position, so the returned list mirrors managers regardless of completion order. The maintenance commands (report_state) then merge lock-family members into shared serial lanes (merge_into_lock_lanes()); the read commands keep one lane per manager.

work returns this manager’s (id, data); it must handle its own meta_package_manager.execution.CLIError (each manager owns its subprocess and error list, so the call is thread-safe per manager). A truthy data["errors"] (or data["failed"]) marks that manager’s trail line ✗; an optional data["label"] overrides its text (upgrade --all uses it for cooldown skips).

Parameters:

report_state (bool) – maintenance commands set it (their only output is the trail). It flips the finisher to a success count, keeps the trail in the sequential fallback, and turns on lock-family serialization. Read commands leave it False: their table is the output, so the sequential fallback is silent and the finisher reports coverage. Passed to dispatch() as the inverse of coverage.

Return type:

list[tuple[str, dict]]

meta_package_manager.dispatch.collect_per_package(label, done_label, tasks, *, ctx=None)

Run per-package operations across managers concurrently, serial within each.

The fan-out primitive for the ordering-free state changers that act on many (package, manager) pairs: remove, upgrade <packages>, restore and the manager-tied specs of install. Takes a flat list of (manager, task) pairs and groups them into lanes by lock family (merge_into_lock_lanes()) — so a manager’s own packages, and any lock-family peers, stay serial while unrelated managers run in parallel — then drives dispatch(). Each task returns (ok, message) after doing its CLI call and recording its own outcome. The unmatched-package priority search of install is not routed here: it has genuine cross-manager ordering (stop at the first manager that has the package) and stays sequential on its own.

Return type:

None

meta_package_manager.dispatch.warn_jobs_ignored(ctx)

Note that --jobs does not parallelize this run.

Only install with at least one untied package reaches this: those packages need a priority search (install with the first manager that has the package, skip the rest), which is cross-manager-sequential, so the whole command runs serially. The other state changers (remove, upgrade <packages>, restore, and install of fully manager-tied specs) now fan out through collect_per_package(). When the user explicitly raised mpm --jobs above 1, say so once at INFO: the request simply has no effect on this run, which is narration, not a problem.

Return type:

None