Julien
09/02/2025, 12:37 PMMarvin
09/02/2025, 12:37 PMMarvin
09/02/2025, 12:40 PMProcessPoolTaskRunner
is available at prefect.task_runners
, so an ImportError typically indicates the prod install predates when this class was included.
Quick checks
- Please run prefect version
in both environments and share the outputs. This will confirm the version mismatch.
- If you’re using Docker, what base image/tag or requirements pin are you using in prod?
Why this happens
- ProcessPoolTaskRunner
is present in modern Prefect (including 3.x). If prod is on an older Prefect 2.x that doesn’t include it, the import will fail there but not locally.
- Source showing it exists in current Prefect: prefect/task_runners.py
How to fix
- Preferred: align the Prefect versions between local and prod (pin the same version).
- Upgrade prod to match local (or vice versa), e.g., bump the Prefect version in your requirements/lockfile or update the Docker image tag.
- If you must stay on an older Prefect in prod, change your code to use a task runner available in that version (or avoid setting one and use the default). But the most robust approach is to pin a consistent Prefect version across environments.
Optional defensive pattern (temporary workaround)
try:
from prefect.task_runners import ProcessPoolTaskRunner
task_runner = ProcessPoolTaskRunner(max_workers=4)
except ImportError:
# Fallback for older installs; consider pinning versions instead of keeping this
from prefect.task_runners import ThreadPoolTaskRunner
task_runner = ThreadPoolTaskRunner(max_workers=4)
If it turns out the versions do match and you still see the ImportError, we should check for:
- Package shadowing (e.g., a prefect.py
file or prefect
directory in your project on prod)
- Mixed environments/paths at runtime
- How your deployment image is built (sometimes an old layer is cached)
Links
- Current task runners in source: prefect.task_runners
- Prefect 3 release notes: v3.0 release notes
- Prefect 2 task runner concepts (for reference if you’re on 2.x): 2.x task runners
Send me:
- Output of prefect version
from local and prod
- How you deploy (Docker image/requirements)
… and I’ll guide you to the exact fix.