Sam Dyson
07/28/2022, 10:10 PMDeployment
function to handle the deployment, as well as specifying FilePackager
and PickleSerializer
configurations so that we could create a single file from our (rather large) list of flow files and supporting library files. This file would then be pushed to an S3 bucket and pulled down during job execution on K8s using a standard docker image for the core dependencies.
For our use-case, it doesn't make sense to upload the entire directory of 200+ files for each deployment, so we really like the ability to package our flow into a single file. We also don't want to have to create a single storage block for each flow, which would multiply the number of files we're storing in s3 (and therefore have to pull down for each flow run). The documentation now states that the Deployment
function is no longer usable, but the YAML file and manifests provide no way to implement the FilePackager
and PickleSerializer
classes.
My question for the prefect team - are the FilePackager
and supporting serializers going to be available for deployments? Or is the new "upload all files" approach going to be the standard for Orion forever? If I were to hack the packager/serializer approach into our deployment methodology, will I run the risk of having the packager/serializer classes removed from the prefect package in the future? Anna has mentioned in previous comments that the prefect team is just getting started with deployments, but the lack of clarity and the fast breaking changes over the course of a week has made it quite difficult to plan our migration effectively - any clarity on this topic would be greatly appreciated!Anna Geller
Nico Neumann
07/29/2022, 12:46 AMKalise Richmond
08/02/2022, 12:38 AM