hi folks! I think I'm having a mental disconnect, coming from temporal.io
I've created an ECS cluster with a long running agent that's happily registered. I'm not trying to register a task, and a flow. However, when I try register it using
it sets the flow to only run locally. More in 🧵
Lee Briggs
03/24/2022, 9:16 PM
Ideally what I want to be able to do is define some code locally, and then set it to execute remotely. I can't seem to figure that out..any hints or tips?
k
Kevin Kho
03/24/2022, 9:19 PM
I think you need a RunConfiguration set to use ECSRun to point to your Cluster. You also then need to store the Flow somewhere the ECS task can pull it.
You can see Anna’s article here on how to get a Flow running on ECS
Kevin Kho
03/24/2022, 9:21 PM
Do you run into a specific error?
Kevin Kho
03/24/2022, 9:23 PM
S3 is the most common storage used with ECS so the ECS Task can pull the Flow from S3 and then run it
l
Lee Briggs
03/24/2022, 9:25 PM
You also then need to store the Flow somewhere the ECS task can pull it
Okay, I think that's the missing piece
I've defined some run configuration like so:
but it keeps adding the local hostname label when I run the code locally
k
Kevin Kho
03/24/2022, 9:28 PM
Ah because the default Storage is local storage so you can use something like Github or S3 instead. For testing, public Github is easiest. For S3, you need to give appropriate IAM permissions to the ECS task
l
Lee Briggs
03/24/2022, 9:29 PM
okay, think I'm following now, thanks so much for the help! I'll give it a try
k
Kevin Kho
03/24/2022, 9:29 PM
Also, if you are testing, the base prefect image is
prefecthq/prefect
, which might reduce the number of possible errors
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.