Serge Tarkovski
03/16/2022, 8:45 AMAnna Geller
03/16/2022, 11:07 AMShellTask
or via a Docker task, and as you adopt Prefect more you can slowly migrate those pipelines to Prefect - check out Laura's tutorial on that subject
I saw many users who have their tasks specified in Python modules that get imported within the flow file, but then you need to make sure to package that properly so that at execution time Prefect can import those custom modules. Check out this post describing how you can do that https://discourse.prefect.io/t/the-simple-guide-to-productionizing-data-workflows-with-docker-by-kevin-kho/453
What agent type do you use?Serge Tarkovski
03/16/2022, 11:29 AMAnna Geller
03/16/2022, 11:54 AMso a Docker-based approach is a proven way to put the dependencies inside a containerThat's correct!
we're at a research phase (could take a month or so), would you recommend starting straight from Orion, skipping Prefect 1.0?It's totally up to you, you can certainly start directly with Orion, and in fact, this would be quite beneficial for us since you could provide us (incredibly valuable) feedback as an early adopter. But 2.0 is not yet fully production-ready and it may take a couple of months to bring 2.0 to that stage - check out our latest announcement for more details on that https://discourse.prefect.io/t/the-second-launch-week-initiative-announcing-prefect-2-0/499. I would honestly encourage you to still explore Prefect 1.x, sign up for a free tier of Prefect Cloud and put some pipelines to production already to get a feeling of how it is to work with Prefect in general. The user experience shouldn't change - you can still run your workflows locally first by adding a couple of decorators to your workflow. Then you can build/deploy your flows - in 1.0 this step is called flow registration, in 2.0 it's creating a deployment. Having used and understood 1.0 first will likely make you appreciate many Orion features more 🙂
Serge Tarkovski
03/16/2022, 1:32 PMAnna Geller
03/16/2022, 1:42 PMSerge Tarkovski
03/16/2022, 2:32 PMMatthias
03/16/2022, 2:37 PMAnna Geller
03/16/2022, 2:56 PMrelease-notes
tag on discourse to get an email update once we release that 🙂Matthias
03/16/2022, 2:59 PMAnna Geller
03/16/2022, 3:11 PMKubernetesFlowRunner
or DockerFlowRunner
Matthias
03/16/2022, 3:14 PMAnna Geller
03/16/2022, 4:24 PMDeploymentSpec
Matthias
03/16/2022, 7:30 PM