Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hey everyone should I start with prefect cloud or prefect server?

Hi <@U01AKLVKQMP> - my recommendation would be to start with Prefect Cloud; Prefect Cloud has a free developer tier and requires you only to set up your agent to get started. As you build your workflows and you require more flexibility/scalability, you may consider migrating to a team tier on Cloud or moving to Prefect Server (a persistent version of which requires more activation energy).

Happy to help, let me know if you run into any issues as you get started :thumbsup:

<@UN6FTLFAS> if I'm using a Docker <http://Prefect.io|Prefect.io> agent, how do I add python packages to it etc?

Where is the execution taking place for my <http://prefect.io|prefect.io> tasks, if it is in the agent container, what is the best structure for adding extra python packages to run my functions/execution code etc?

Hi <@U01AKLVKQMP> - the Agent is only responsible for orchestrating flows by submitting jobs. Execution and storage are configured on a per-flow basis, for maximum flexibility (and to keep agents very light). So to make sure python packages are available at your flow's runtime, you can pass the `python_dependencies` kwarg a list to your `Docker` storage object. Take a look at <https://docs.prefect.io/orchestration/tutorial/docker.html#persisting-your-flow-with-docker-storage>, and <https://docs.prefect.io/orchestration/execution/overview.html#execution-overview>

So its just an agent and I'm creating individual docker containers on a per flow basis :thumbsup:

no problem, you as well <@U01AKLVKQMP> :smile:

<@UN6FTLFAS> would you recommend using Prefect on a VM? Surely the agent needs to be consistently running to receive tasks from the API?

For example how could I use prefect with AppEngine or CloudFunctions?

At the moment, no, running prefect in a VM is only useful for development purposes for the reason you state, that an agent should be polling to retrieve work. We have some stuff on the roadmap that will make using cloud functions and other serverless methods available.... more to come on that in the next few months! 

What are the other recommended methods for making the agent available in a good setup/ deployment then please? :pray:

Hi <@U01AKLVKQMP> - your agent setup is pretty specific to your infrastructure but I'd recommend taking a look at the <https://docs.prefect.io/orchestration/agents/overview.html|Agents documentation> for some information on the process and health checks.