Have a few questions on flow boilerplate and k8's templates
Dylan
04/26/2022, 7:22 PM
Some constraints:
1. We can’t pull code from Github during a flowRun
2. We don’t want to store flowRun results in a place like S3 since we plan on using prefect cloud
What storage type and flow declaration should we use if we assume our flow-run docker image has all the source code in it needed to run? And is that doable with a k8's spec that references that image (full image of resolved dependencies and flow-run tasks, modules etc)
Dylan
04/26/2022, 7:23 PM
After discussing with our SRE we basically can’t go the approach we were going to of pulling the task code from GH when the flow kicks off
k
Kevin Kho
04/26/2022, 7:23 PM
Local storage using
stored_as_script=True
and
path=flow_path_in_container.py
and then it will look for the Flow relative to the image
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.