https://prefect.io logo
Channels
Powered by
c

Clint M

08/18/2022, 2:11 PM
I’ve got a question RE: deployments with 
null
(local) storage
The path that get’s hard coded in the deployment yaml file looks like my source code path. I just tested changing it to the local path in my docker file (oooohh I know I’m not supposed to do that… it event says it in the file) but I got it to work running in a container in eks
I really don’t want to provide you with my aws creds
so I’m willing to live with this hack for now but my question is
why in the hell do I need to give you my AWS creds when this works
why don’t you just let me run my code in the prefect agent… i.e. I don’t want it uploaded to s3
here’s the relevant bit I’m talking about
Copy code
###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: etl
manifest_path: null
storage: null
path: /Users/clint/code/github/flowcast/till/prefect-example <------------- bad
entrypoint: src/prefect_example/flow.py:etl
parameter_openapi_schema:
  title: Parameters
  type: object
  properties: {}
  required: null
  definitions: null
I tried setting it to a relative path like 
./
but the agent runs the code in some temp dir
can you let me use a variable there so I don’t have to munge the path as part of my script to build the deployment.yaml
here’s the error when I tried to use 
./
Copy code
prefect-agent-84c5c6cd64-2r8gz prefect-agent FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp1b7dnxlfprefect/src/prefect_example/flow.py'
so if I just change that to the root of my code for the container everything works fine (e.g. 
/home/app/code
)
o

Oscar Björhn

08/18/2022, 2:23 PM
I believe they're working on an official solution for this, shouldn't be far off. Not sure if that answers your question. Probably not.. 🙂
c

Clint M

08/18/2022, 2:34 PM
thanks Oscar… it’s always nice to know I’m not just shouting into the void… so thanks for the response 🙂
o

Oscar Björhn

08/18/2022, 2:39 PM
Haha, no worries. I'm looking forward to being able to pack my flows into docker images myself, although I settled on keeping them in blob storage, for now. While you wait for the official feature, did you consider creating separate aws storages (not sure of the right term, never used aws) solely for storing the prefect flows? That way you wouldn't have to worry too much about giving away your credentials.
c

Clint M

08/18/2022, 2:44 PM
I can lock down the creds so that they only have access to the one s3 bucket so I’m not overly concerned about uploading them to a 3rd party… I just don’t want to do that work for every new project/env combination that want’s to use prefect… I’m also just bothered by the fact that it seems like it’s required when it shouldn’t be… let me worry about my own code on my own infra… it feels like it breaks their hybrid model
o

Oscar Björhn

08/18/2022, 2:48 PM
That's fair. I've got the impression that deployments were launched in kind of a minimum state. It seems like they're working on a bunch of new options for how you build, package flows etc. Just gotta watch out for those release notes. 🙂
🙏 1
Powered by