https://prefect.io logo
Title
s

Sam Dyson

07/28/2022, 10:10 PM
Good morning community, we are really excited to start the upgrade to Orion but are having some issues with deployments. Previously during our beta PoC/migration development, we were using the python
Deployment
function to handle the deployment, as well as specifying
FilePackager
and
PickleSerializer
configurations so that we could create a single file from our (rather large) list of flow files and supporting library files. This file would then be pushed to an S3 bucket and pulled down during job execution on K8s using a standard docker image for the core dependencies. For our use-case, it doesn't make sense to upload the entire directory of 200+ files for each deployment, so we really like the ability to package our flow into a single file. We also don't want to have to create a single storage block for each flow, which would multiply the number of files we're storing in s3 (and therefore have to pull down for each flow run). The documentation now states that the
Deployment
function is no longer usable, but the YAML file and manifests provide no way to implement the
FilePackager
and
PickleSerializer
classes. My question for the prefect team - are the
FilePackager
and supporting serializers going to be available for deployments? Or is the new "upload all files" approach going to be the standard for Orion forever? If I were to hack the packager/serializer approach into our deployment methodology, will I run the risk of having the packager/serializer classes removed from the prefect package in the future? Anna has mentioned in previous comments that the prefect team is just getting started with deployments, but the lack of clarity and the fast breaking changes over the course of a week has made it quite difficult to plan our migration effectively - any clarity on this topic would be greatly appreciated!
👍 1
a

Anna Geller

07/28/2022, 10:30 PM
Great question. I don't want to spoil too much of future plans but we are working on a really cool feature that will allow you to not upload everything to a file system during deployment build
but it will work the same way as it works right now, just giving you more flexibility with respect to what gets uploaded during build
n

Nico Neumann

07/29/2022, 12:46 AM
I also miss the FilePackager/PickleSerializer. I use bazel as a build system which automatically resolves (custom) dependencies when starting trough the bazel cli. When using the Manifest to specify the flow path it does not automatically find the dependencies specified through the build system. This worked fine with Prefect 2.0b15 and Prefect 1.2.4. Also parameters could be provided more dynamically through python. I’m new to Prefect 2.0, so maybe I overlooked something or there is another way. I am looking forward to the migration to 2.0, I really love the new features and the team did a great work!❤️
:thank-you: 2
k

Kalise Richmond

08/02/2022, 12:38 AM
Circling back here, I just created a enhancement request that would extend compression formats like .zip or tarballs https://github.com/PrefectHQ/prefect/issues/6167