Hello I need help to configure s3 storage for flows I have a

Question

Hello I need help to configure s3 storage for flows I have an not AWS S3 compatible provider already works great with minio client and such but i failed to configure it with prefect blocks as storage provider I used the block with configured through the UI with my custom endpoint then I built with `prefect deployment build flow py check n k8sjob sb s3 bucket my s3 i kubernetes job override env EXTRA PIP PACKAGES= s3fs prefect aws a` The build ends with success and I can see the flow py being uploaded in my bucket When I launch the flow it fails to retrieve the code stack trace in thread

Emma Rizzi · Answer

Error when launching the flow ```Downloading flow code from storage at 04 47 41 PM prefect flow runs Flow could not be retrieved from deployment Traceback most recent call last File lt frozen importlib bootstrap external gt line 839 in exec module File lt frozen importlib bootstrap external gt line 975 in get code File lt frozen importlib bootstrap external gt line 1032 in get data FileNotFoundError Errno 2 No such file or directory flow py The above exception was the direct cause of the following exception Traceback most recent call last File usr local lib python3 8 site packages prefect engine py line 274 in retrieve flow then begin flow run flow = await load flow from flow run flow run client=client File usr local lib python3 8 site packages prefect client utilities py line 47 in with injected client return await fn args kwargs File usr local lib python3 8 site packages prefect deployments py line 187 in load flow from flow run flow = await run sync in worker thread load flow from entrypoint str import path File usr local lib python3 8 site packages prefect utilities asyncutils py line 91 in run sync in worker thread return await anyio to thread run sync File usr local lib python3 8 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File usr local lib python3 8 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File usr local lib python3 8 site packages anyio backends asyncio py line 867 in run result = context run func args File usr local lib python3 8 site packages prefect flows py line 809 in load flow from entrypoint flow = import object entrypoint File usr local lib python3 8 site packages prefect utilities importtools py line 197 in import object module = load script as module script path File usr local lib python3 8 site packages prefect utilities importtools py line 160 in load script as module raise ScriptError user exc=exc path=path from exc prefect exceptions ScriptError Script at flow py encountered an exception FileNotFoundError 2 No such file or directory 04 47 42 PM prefect flow runs Downloading flow code from storage at 04 48 20 PM prefect flow runs Flow could not be retrieved from deployment Traceback most recent call last File lt frozen importlib bootstrap external gt line 839 in exec module File lt frozen importlib bootstrap external gt line 975 in get code File lt frozen importlib bootstrap external gt line 1032 in get data FileNotFoundError Errno 2 No such file or directory flow py The above exception was the direct cause of the following exception Traceback most recent call last File usr local lib python3 8 site packages prefect engine py line 274 in retrieve flow then begin flow run flow = await load flow from flow run flow run client=client File usr local lib python3 8 site packages prefect client utilities py line 47 in with injected client return await fn args kwargs File usr local lib python3 8 site packages prefect deployments py line 187 in load flow from flow run flow = await run sync in worker thread load flow from entrypoint str import path File usr local lib python3 8 site packages prefect utilities asyncutils py line 91 in run sync in worker thread return await anyio to thread run sync File usr local lib python3 8 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File usr local lib python3 8 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File usr local lib python3 8 site packages anyio backends asyncio py line 867 in run result = context run func args File usr local lib python3 8 site packages prefect flows py line 809 in load flow from entrypoint flow = import object entrypoint File usr local lib python3 8 site packages prefect utilities importtools py line 197 in import object module = load script as module script path File usr local lib python3 8 site packages prefect utilities importtools py line 160 in load script as module raise ScriptError user exc=exc path=path from exc prefect exceptions ScriptError Script at flow py encountered an exception FileNotFoundError 2 No such file or directory ```

Emma Rizzi · Answer

The flow if relevant just copied from a tutorial ```from prefect import flow get run logger from platform import node platform flow def check logger = get run logger f Network node f Instance platform if name == main check ```

Emma Rizzi · Answer

and the deployment yaml just in case i missed something ``` A complete description of a Prefect Deployment for flow check name k8sjob description null version e433c7d8474dc025c33ba4ec3a690abe The work queue that will handle this deployment s runs work queue name default work pool name default agent pool tags parameters schedule null is schedule active true infra overrides env EXTRA PIP PACKAGES s3fs prefect aws infrastructure type kubernetes job env labels name null command null image prefecthq prefect 2 8 7 python3 8 namespace default service account name null image pull policy null cluster config null job apiVersion batch v1 kind Job metadata labels spec template spec parallelism 1 completions 1 restartPolicy Never containers name prefect job env customizations job watch timeout seconds null pod watch timeout seconds 60 stream output true finished job ttl null block type slug kubernetes job block type slug kubernetes job DO NOT EDIT BELOW THIS LINE flow name check manifest path null storage bucket name prefect minio credentials null aws credentials null basepath endpoint url null credentials aws access key id aws secret access key aws session token null profile name null region name eu west 1 aws client parameters api version null use ssl true verify true verify cert path null endpoint url config null block document id block document name s3 credentials is anonymous false block type slug aws credentials bucket folder prefect block document id block document name my s3 is anonymous false block type slug s3 bucket block type slug s3 bucket path entrypoint flow py check parameter openapi schema title Parameters type object properties required null definitions null timestamp 2023 03 30T14 45 30 993032+00 00 ```

Emma Rizzi · Answer

When checking the documentation it seems that S3 Bucket block is used inside tasks not as storage configuration I wonder if it s then compatible for this purpose but the S3 Filesystem doesn t provide an endpoint parameter either

redsquare · Answer

the S3 block is the one you need it has a bucket path

Christopher Boyd · Answer

add a at the end of your path

Christopher Boyd · Answer

Emma Rizzi · Answer

< Christopher Boyd> thanks that was it slightly smiling face