<@ULVA73B9P> Seeing the following error on all (un...
# ask-marvin
m
@Marvin Seeing the following error on all (unchanged and previously working) modal push runs recently: “Flow run could not be submitted to infrastructure: deprecation_warning() takes 2 positional arguments but 3 were given”
m
thought for 354.9 seconds
Thanks for flagging this — that error message strongly suggests a version mismatch between Prefect and the Modal integration. What’s likely happening - The Modal worker/plugin is calling an internal Prefect deprecation utility with an old call signature. In Prefect 3.x, some deprecation utilities moved/changed signatures, so older prefect-modal builds can break with “deprecation_warning() takes 2 positional arguments but 3 were given” during submit-to-infrastructure. - I couldn’t find an existing issue that matches this exact error, which further points to a local compatibility problem rather than a known platform outage. What I recommend 1) Share versions (so we can confirm compatibility):
Copy code
prefect version
python -c "import prefect, sys; print('prefect:', prefect.__version__); \
           import importlib; m = importlib.import_module('prefect_modal'); \
           print('prefect-modal:', getattr(m, '__version__', 'unknown')); \
           import modal; print('modal:', getattr(modal, '__version__', 'unknown'))"
2) Upgrade Prefect and prefect-modal together, then rebuild/redeploy your Modal image/app used by the work pool:
Copy code
pip install -U prefect prefect-modal modal
# or with uv
uv pip install -U prefect prefect-modal modal
- If your Modal push worker builds an image, make sure that image includes the updated packages and gets rebuilt/pushed before the next run. 3) Re-run your deployment. If the error persists, please grab the full traceback from the worker logs (set
PREFECT_LOGGING_LEVEL=DEBUG
if possible) and share it here. 4) Temporary workaround if you need to unblock immediately: pin Prefect back to the last known-good version used by your previously working runs and rebuild your Modal image with that exact version. Then we can work forward from there. Useful references - Modal guide: Run on Modal (Prefect 3.x) - Modal provisioner API (3.x): Modal provisioner API - Plugin repo (to check releases/issues): Prefect Modal integration If you can share: - Prefect, prefect-modal, and modal versions - A snippet of your work pool type and any relevant
prefect.yaml
bits for the Modal build/push - The full stack trace around the error …I’ll dig in and help you get this resolved quickly.
m
@Marvin -It’s very hard to debug this because prefect-modal isn’t open source. -I have tried pinning the prefect version at both 3.1 and 3.4, with no luck. I have set the PREFECT_LOGGING_LEVEL env variable to DEBUG in the deployment configuration. Despite that, I see only the log pasted above in the Prefect run dashboard. And, in the Modal dashboard, I see 0 logs (though it does say an app with the expected name was created, no logs are present).
m
thought for 140.7 seconds
Thanks for the extra detail — that’s really helpful. Given: - It happens on all Modal push runs - You’ve pinned Prefect to 3.1 and 3.4 with no change - There are no Modal app logs (so it fails before Modal code runs) - Prefect UI shows only the single “submit to infrastructure” error This points to a server-side error in the Modal provisioner for push pools (likely a deprecation utility signature mismatch introduced recently). Pinning your local Prefect version won’t impact the submit step for push pools because that submit happens in the provisioner/worker environment managed by Prefect. To move this forward quickly: Immediate options to get unstuck - Switch the deployment temporarily to another work pool type (e.g., Process or Docker) so your schedules can continue while we investigate Modal push. - Or, if feasible, run your flows via a pull-based worker you control (local or infra you manage) until Modal push is fixed. Helpful diagnostics (please share) - Workspace name (or slug), region, and the Modal work pool name - One or more recent flow run IDs showing this error and approximate timestamps - Output of:
Copy code
prefect work-pool inspect "<your-modal-push-pool>"
If you can run this, it’ll confirm the pool type and configuration we’re troubleshooting. Optional: try to re-provision the pool (may surface more error context)
Copy code
prefect work-pool provision-infrastructure "<your-modal-push-pool>"
What I suspect - A regression in the Modal provisioner’s deprecation warning call (a signature change) on the server/plugin side. That would explain why your runs started failing “recently” and why local pinning hasn’t helped. Next steps - If you can share the details above, I’ll escalate to our engineering team right away. This looks like something we need to fix on our side. - If you prefer, you can also open a ticket with those details so we can correlate logs and timelines from the backend. Refs - Modal guide (3.x): Run on Modal (Prefect 3.x) - Modal provisioner API docs: Modal provisioner API I know this is frustrating without visibility into the plugin internals — thank you for your patience. If you share your workspace/pool/run info here (or DM), I’ll get this in front of the team and follow up with status.
m
@Bianca Hoch Could this issue be related to the earlier issue I encountered: https://prefect-community.slack.com/archives/CL09KU1K7/p1734456521326529? “Flow run could not be submitted to infrastructure: (Status.INVALID_ARGUMENT: 3, ‘Version 0.62.203 of
modal
will be deprecated soon, and cannot be used to create new apps.’, None)”