feat(autoscaling): add ThroughputAutoscaledPool and the autoscaled_pool_class crawler option

[Mantisus]

Description

• This PR adds an experimental, opt-in ThroughputAutoscaledPool that looks for the concurrency at which the crawl finishes the most tasks, and an autoscaled_pool_class option on every crawler to select it. The default stays AutoscaledPool, unchanged.
• The pool helps whenever tasks slow down under load for a reason outside the machine. The target host is the usual one, but the network, proxies, and the crawler's own parts, such as storage speed, slow tasks down the same way.

How the pool decides:

• It reads throughput, finished tasks per second by Little's law, from windows of at least 10 tasks once 80% of a window has finished.
• It doubles the concurrency while each doubling delivers at least 25% more. Then it measures a level above and a level below the current concurrency and moves toward the one that delivers at least 10% more. When the two tie at the narrowest spread, it holds for six windows and then measures again.
• A window is given up early only when even its best possible reading already loses.
• All CPU, memory and event loop checks of AutoscaledPool still apply. If the machine reports an overload right after a raise, before the raised level has been measured, the raise is undone.
• It learns the peak from finished tasks, so it pays off best on long-running crawls.

Known limitations:

• Each step waits for its tasks to finish, so on slow pages it ramps up more slowly than AutoscaledPool.
• Every finished task counts as work, so a site that answers fast with errors such as HTTP 429 looks like it can take more.
• It looks for one concurrency for the whole crawl, which is unstable across many hosts with different peaks.
• With max_tasks_per_minute it settles at whatever rate the limit lets through, so a fixed rate belongs with the default pool.

Issues

• Closes: AutoscaledPool controller does not converge to optimal desired_concurency #1224

Testing

• Unit tests drive the pool through _autoscale() and _worker_task() with a fake clock and assert on desired_concurrency.
• Measured on a local simulation bench covering both the base cases and various edge cases.

✍️ Drafted by Claude Code

[janbuchar]

Hi @Mantisus, thanks for looking into this. On the JS side, we decided to extract the IConcurrencySystem from the AutoscaledPool and make it "pluggable" into your crawler. Allowing passing in autoscaled_pool_class introduces a parity gap.

Could you please look into IConcurrencySystem and see if you can bend the design of your PR to be in line with that?

[Mantisus]

Hi @janbuchar, thanks for the pointer.

Could you please look into IConcurrencySystem and see if you can bend the design of your PR to be in line with that?

I've created an issue for a small change to the IConcurrencySystem interface: apify/crawlee#4127

As for this PR, I agree that it doesn't make sense to add autoscaled_pool_class as a temporary option before refactoring AutoscaledPool. One question, though: can we do that refactoring in v1, since _autoscaling is a private module (even though AutoscaledPool shows up in the API docs), or should we postpone it, together with this PR, until v2? @vdusek, what do you think?