https://prefect.io
Join Slack
Hi all! I'm new to AWS, Docker, and orchestration :sweat_smile: I'm starting a data team at a new co...
# ask-community
j
Hi all! I'm new to AWS, Docker, and orchestration 😅 I'm starting a data team at a new company and I'm trying to run Prefect as a proof of concept. Right now my plan is to host the server and database on a single EC2 instance inside two containers. The server uses the 
prefecthq/prefect:2-latest
image. It's been a pain having to remote into the EC2 instance, then 
docker cp
my flow into the server container, then 
docker exec
into the container to run 
prefect deploy
then copy out that yaml file so I can modify the pull step, and then 
docker cp
it back into the server container. It doesn't have ssh or git or I'd just clone the repo that way. Is this what most people do when they self host? If I don't use the yaml file then I have to define my deployment in the code and I'm getting all mixed up. It doesn't feel right to me, so any opinions on what a good workflow for this is would be greatly appreciated!
a
Welcome @Jack Goslin!
When I used self-hosted I used fargate for this, and configured github actions to do a lot of this stuff so wanted to just post here to follow and see what others do. fwiw when I was a prefect user (before I started working here!) I ended up just using cloud since my use case fit into the free tier (like 400 requests/min at the time or something)
👀 1
k
+1 to using Cloud on the free tier if you're doing a POC that doesn't have massive, high-frequency orchestration volume.
Maybe a less dismissive version of my take on this is, if you really want to self-host that's awesome, but if operating a server is getting in the way of actually running workflows, start with Cloud and make running the server the last thing you learn. It's easier to pick up how all the pieces fit together - workers, work pools, deployments, etc. - and enjoy the feeling of stuff working without all the extra effort of hosting. Then you can start up a server wherever you want and the whole process will feel natural.
smart 1
j
Smart! I might do that and then try to run them side by side once I get comfortable. Thanks!
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by