Hi, I have a question about self hosting and whether prefect is the correct tool for the job. My team is distributed and we have high end desktops and some headless servers located at their homes and office locations. Generally speaking, the machines are not able to accept incoming connections because they sit behind firewalls or people's routers.
Is there a way that I can install some sort of daemon on these machines that fetches jobs from the que (hosted on prefect's cloud), they execute them, and then transmit the results back?
The problem we are trying to solve is that we have underutilized hardware at these locations, and I would like to try to use prefect as a way of making these resources available to whoever needs them on the team remotely in a relatively painless way.
k
Kevin Grismore
01/12/2024, 3:19 PM
yep, those would be workers. they only use outbound connections to a specific ip address to poll for work
Kevin Grismore
01/12/2024, 4:30 PM
this is actually what our hybrid model is all about (the part of the page that has the Secure Infrastructure header)
i
Isaac Tamblyn
02/15/2024, 4:56 PM
Is there a simple tutorial how to get this working for a hello world example?
k
Kevin Grismore
02/15/2024, 4:59 PM
before I make any recommendations, how do you make your workloads available? through git? docker images?
Kevin Grismore
02/15/2024, 5:03 PM
if you package your code and dependencies into docker images and push them to some remote registry, as long as that registry is accessible from your machines, a docker worker could pull and execute them
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.