python package. Can I use my own ec2 to run py script. and use prefect to maintain logs
✅ 1
n
Nate
06/27/2023, 2:44 PM
yes! you can run an worker on the EC2 with your env vars (like PREFECT_API_KEY and PREFECT_API_URL) and schedule work there - if you have something like free tier prefect cloud, the logs will be streamed back to your flow runs page in the cloud ui, otherwise the same would be true for a self hosted server
k
Krishna Lodha
06/27/2023, 2:46 PM
Hi @Nate thanks for your response, can you please paste any link of blog/video for me to check
n
Nate
06/27/2023, 2:50 PM
• get prefect cloud free tier
• create a simple Process work pool
• run a worker on EC2 (process type) against that work pool
• now locally, login to a workspace with
prefect cloud login
• write your flow
•
prefect deploy
and select that work pool
👀 1
✅ 1
s
Soham
09/01/2023, 8:08 AM
@Nate: I was looking for something similar and found this thread! I was wondering from where will the EC2 pull the code to execute the flows.
n
Nate
09/01/2023, 1:16 PM
hi @Soham - it depends how you define your
pull
step
for example, if you use
git_clone
- the worker will clone your code from the specified remote repo (like this) when executing the
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.