https://prefect.io logo
j

James Phoenix

10/09/2020, 2:46 PM
Hey everyone should I start with prefect cloud or prefect server?
n

nicholas

10/09/2020, 2:51 PM
Hi @James Phoenix - my recommendation would be to start with Prefect Cloud; Prefect Cloud has a free developer tier and requires you only to set up your agent to get started. As you build your workflows and you require more flexibility/scalability, you may consider migrating to a team tier on Cloud or moving to Prefect Server (a persistent version of which requires more activation energy).
👍 1
j

James Phoenix

10/09/2020, 2:52 PM
Cool thanks Nicholas.
n

nicholas

10/09/2020, 2:53 PM
Happy to help, let me know if you run into any issues as you get started 👍
j

James Phoenix

10/09/2020, 4:35 PM
@nicholas if I'm using a Docker Prefect.io agent, how do I add python packages to it etc?
Where is the execution taking place for my prefect.io tasks, if it is in the agent container, what is the best structure for adding extra python packages to run my functions/execution code etc?
n

nicholas

10/09/2020, 4:40 PM
Hi @James Phoenix - the Agent is only responsible for orchestrating flows by submitting jobs. Execution and storage are configured on a per-flow basis, for maximum flexibility (and to keep agents very light). So to make sure python packages are available at your flow's runtime, you can pass the
python_dependencies
kwarg a list to your
Docker
storage object. Take a look at https://docs.prefect.io/orchestration/tutorial/docker.html#persisting-your-flow-with-docker-storage, and https://docs.prefect.io/orchestration/execution/overview.html#execution-overview
j

James Phoenix

10/09/2020, 4:41 PM
Cool that makes sense.
So its just an agent and I'm creating individual docker containers on a per flow basis 👍
Thanks and have a good weekend!
n

nicholas

10/09/2020, 4:41 PM
exactly!
no problem, you as well @James Phoenix 😄
👍 1
j

James Phoenix

10/13/2020, 1:52 PM
@nicholas would you recommend using Prefect on a VM? Surely the agent needs to be consistently running to receive tasks from the API? For example how could I use prefect with AppEngine or CloudFunctions?
n

nicholas

10/13/2020, 1:55 PM
At the moment, no, running prefect in a VM is only useful for development purposes for the reason you state, that an agent should be polling to retrieve work. We have some stuff on the roadmap that will make using cloud functions and other serverless methods available.... more to come on that in the next few months!
🎉 1
j

James Phoenix

10/14/2020, 2:11 PM
Okay great.
What are the other recommended methods for making the agent available in a good setup/ deployment then please? 🙏
n

nicholas

10/14/2020, 4:08 PM
Hi @James Phoenix - your agent setup is pretty specific to your infrastructure but I'd recommend taking a look at the Agents documentation for some information on the process and health checks.