Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Basic question about Prefect cloud. Please bear with me.

So far have been using Prefect locally as follows:

1. pip install prefect
2. For the deployment, use a Python script inside which import Prefect and use the 'Deployment'
3. Then run 'prefect orion start' and it shows the scheduled runs and status at 127.0.0.1:4200
4. Then run 'prefect agent start' with queue name to serve the scheduled runs.
Now looking at Prefect cloud, and have the following question:

Should I do 'pip install prefect' anymore? If not and if prefect cloud takes care of it, how then would the Python program know about the 'Deployment', i.e. where do I import 'Deployment' from?

Or should I stop the Python part altogether and schedule the deployment from Prefect Cloud? If yes, I presume that in the Prefect Cloud, I will have to specify the script that needs to be run at the scheduled time? In this case, my script will exist in an AWS EC2 instance, how do I give access to it to the Prefect Cloud?

Thanks.

Hi <@U03V8RLCLAJ>.

&gt; Should I do ‘pip install prefect’ anymore? If not and if prefect cloud takes care of it, how then would the Python program know about the ‘Deployment’, i.e. where do I import ‘Deployment’ from?

You do need to have Prefect installed in which ever environment is running your flows. So if your agent is running in ECS or a Docker container or locally you do need Prefect installed in that environment.


&gt; Or should I stop the Python part altogether and schedule the deployment from Prefect Cloud? If yes, I presume that in the Prefect Cloud, I will have to specify the script that needs to be run at the scheduled time? In this case, my script will exist in an AWS EC2 instance, how do I give access to it to the Prefect Cloud?

You can schedule the deployment from the Prefect Cloud UI, or the Python Deployment script.

You probably want the flow code script to be run to live in an S3 bucket. You can create an S3 block from the Prefect Cloud UI under Blocks and then

just make sure you pip install s3fs when you create your deployment and follow <https://docs.prefect.io/concepts/deployments/#create-a-deployment-from-a-python-object|the example in the docs>.

Hi <@U03KWSKFN8L>, thanks! Let me grasp all this info. and see how I can implement them.

In the mean time, when I install prefect and do 'prefect orion start' in an AWS ECS instance (a t2.micro to experiment), it says prefect depends on SQLite, and the required verison of SQLite does not exist. And it does not let me upgrade the SQLite version because I am not the root user. Is there a way I can work around this issue? Should I use a different EC2 type? Thanks.

You don’t need `prefect orion start` if you’re using Prefect Cloud. Either you do `prefect orion start` to start a server in your environment OR you use Prefect Cloud.