https://prefect.io logo
Title
c

Chris Gunderson

10/19/2022, 2:47 PM
Hi - When deploying a flow with a S3 block and Docker infrastructure block, will the flow be looking for the code to run in the docker container, or S3 block? The deployment is sending our code to S3, but I'm having issues running the flow. It appears to be looking for the code in the docker image. CLI Deployment script:
prefect deployment build -n fidelity-allocations-deployment -q default -sb s3/prefect-training -ib docker-container/prefect-training -a src/main/prefect/flows/allocations/prefect_fidelity_allocations.py:FidelityAllocationsFlow --cron "13 21 * * 1-5"
Error message from the flow: Flow could not be retrieved from deployment. Traceback (most recent call last): File "<frozen importlib._bootstrap_external>", line 846, in exec_module File "<frozen importlib._bootstrap_external>", line 982, in get_code File "<frozen importlib._bootstrap_external>", line 1039, in get_data FileNotFoundError: [Errno 2] No such file or directory: 'src\\main\\prefect\\flows\\allocations\\prefect_fidelity_allocations.py' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/engine.py", line 257, in retrieve_flow_then_begin_flow_run flow = await load_flow_from_flow_run(flow_run, client=client) File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/orion.py", line 82, in with_injected_client return await fn(*args, **kwargs) File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/deployments.py", line 70, in load_flow_from_flow_run flow = await run_sync_in_worker_thread(import_object, str(import_path)) File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 57, in run_sync_in_worker_thread return await anyio.to_thread.run_sync(call, cancellable=True) File "/opt/pysetup/.venv/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/opt/pysetup/.venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/opt/pysetup/.venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, *args) File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/utilities/importtools.py", line 193, in import_object module = load_script_as_module(script_path) File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/utilities/importtools.py", line 156, in load_script_as_module raise ScriptError(user_exc=exc, path=path) from exc prefect.exceptions.ScriptError: Script at 'src\\main\\prefect\\flows\\allocations\\prefect_fidelity_allocations.py' encountered an exception
1
r

Ryan Peden

10/19/2022, 2:54 PM
Hi Chris - I think this is a Windows pathname issue we are currently working to fix. If you run the deployment CLI command without the
-a
and then open the YAML file it generates, do you see backslashes instead of forward slashes in the script path? If so, try changing them to forward slashes and then run
prefect deployment apply <your_deployment_file>.yaml
🙌 1
c

Chris Gunderson

10/19/2022, 2:56 PM
Thanks Ryan. I forgot about that
@Ryan Peden Just to confirm, it is retrieving the code from S3, right?
r

Ryan Peden

10/19/2022, 3:05 PM
Right - it should only look for code in the Docker image if the deployment has no storage block.
👍 1
c

Chris Gunderson

10/19/2022, 3:05 PM
Thank you
r

Ryan Peden

10/19/2022, 3:06 PM
Always happy to help 😄
c

Chris Gunderson

10/19/2022, 7:49 PM
@Ryan Peden I tried to add parameters to the deployment with no joy... Step 1 prefect deployment build -n wedbush-allocations-deployment -q default -sb s3/prefect-training -ib docker-container/prefect-training src/main/prefect/flows/allocations/prefect_wedbush_allocations.py:WedbushAllocationsFlow --cron "13 15 * * 1-5" --param toAddresses='cgunderson@spiderrockadvisors.com' Step 2 prefect deployment apply WedbushAllocationsFlow-deployment.yaml
if __name__ == "__main__":
    emailAddress = EmailAddress(toAddresses=['<mailto:cgunderson@spiderrockadvisors.com|cgunderson@spiderrockadvisors.com>'])
    WedbushAllocationsFlow(toEmail=emailAddress)
class EmailAddress(BaseModel):
    toAddresses: list[str]
Error
I changed the yaml to get it to work
r

Ryan Peden

10/19/2022, 8:10 PM
Thanks for the update. I think we could do more to document how to map parameters from the YAML through to Python, so I'll add it to our content roadmap
👍 1
Apologies for the slow reply; my basement is flooding for the second time this week so I've been scrambling to deal with that over the past hour. If you run into any other issues, I will ask one of my colleagues to come in and assist so you aren't blocked by Prefect 🙂
c

Chris Gunderson

10/19/2022, 8:13 PM
ekk, will do. Thanks Ryan. Hope all goes well with the basement
r

Ryan Peden

10/19/2022, 8:14 PM
Thank you! And thanks for the great feedback you shared last week at PACC.