I m feeling a bit lost and embarrassed to be lost but have a

Question

I m feeling a bit lost and embarrassed to be lost but have a question about what seems like a simple deployment Here is what I m trying to do

Greg Desmarais · Answer

This seems to get very far but then gets stuck All the code is running on my local and I have full access to the VPC through VPN The flow is written to S3 just fine the ECS cluster is created I can telnet to the cluster scheduler at 8786 and 8787 and the task in ECS seems correct It gets to this point in the logs ```2020 07 17T16 45 04 792 04 00 distributed scheduler INFO Scheduler at <tcp 10 72 9 106 8786> 2020 07 17T16 45 04 792 04 00 distributed scheduler INFO dashboard at 8787 2020 07 17T16 45 09 566 04 00 distributed scheduler INFO Receive client connection Client 675482da c86e 11ea b8be acde48001122 2020 07 17T16 45 09 577 04 00 distributed core INFO Starting established connection```

Greg Desmarais · Answer

I have connectivity from my local machine issuing the flow run to the prefect services I started on an EC2

Greg Desmarais · Answer

By the way I threw together the diagram quickly There is a decent chance I forgot something

Greg Desmarais · Answer

The dask scheduler task sits there forever with `Starting established connection` and the client issuing the flow run sits there with `Starting flow run`

Greg Desmarais · Answer

I have a feeling I m missing something with an agent or some other process It seems all my networking is set up correctly as I can hit everything from my client the prefect servers seem happy the ecs cluster starts up fine etc

Jeremiah · Answer

Hi Greg sorry you re feeling lost Server deploys are very tricky for us to debug due to the uniqueness of each situation but a few things to look for I don t see an agent running in your diagram Server expects an agent to be running in fargate probably to receive the `run` command and launch the flow into the Dask cluster Similarly when you say you re calling `flow run ` you ll probably actually want to call `flow register ` and then issue either a GQL command to run the flow via Agent or click the `run` command in the Prefect UI

Jeremiah · Answer

Ah never mind I just looked closely at your code and I think you re not necessarily running a server deploy You re running a script that should be connecting to a remote Dask cluster

Jeremiah · Answer

In order to turn this into a server deploy you ll want to attach an <https docs prefect io orchestration execution overview html|Environment> to your flow and then call `flow register ` against the server backend

Jeremiah · Answer

Then you won t call flow run in your script instead you ll register that flow and its environment to the server which will issue the instruction to run to an <https docs prefect io orchestration agents overview html|Agent>

Greg Desmarais · Answer

Ok what you are saying makes sense I ve never really known where to put what agent so this agent which I m not reading about if I start it programmatically which one do I start e g prefect agent docker DockerAgent or other Or is it that the different agents know which flows they should be pulling off the run queue

Greg Desmarais · Answer

And where should I be running that agent On the same box as the EC2 instance where I ran `prefect server start` That seems like a logical place to me But when I tried that I got ```prefect utilities exceptions AuthorizationError No agent API token provided ``` My guess is that I need to update my prefect config toml to point the agent at my server deployment

Greg Desmarais · Answer

But I do t know where to find docs on the configs so I m somewhat guessing maybe `prefect context config cloud api` or `prefect context config server host`

Greg Desmarais · Answer

also in the described process I create the fargate cluster then call run which I will flip to register Then as I understand it the flow is in the prefect server and the agent notices this picks up the flow and runs the executors for it am I right Is it possible to have the agent spin up the cluster on its own It would be great to do that as I would not need to give the person writing the script all sorts of permissions to create resources I could consolidate into a role

Greg Desmarais · Answer

At the least thanks < Jeremiah> you are getting me further Hopefully if I figure out a good recipie it can be of use to the community

Greg Desmarais · Answer

I think there ll be a bunch of people that will want to use it in a similar way

Greg Desmarais · Answer

ok realized the agent was looking for the api token when the config was in cloud mode switching it to server let the agent start up

Robin · Answer

Hey < Greg Desmarais> using e g ```prefect agent start kubernetes token MY TOKEN``` with a token that you can create via the prefect cloud UI >> Team >> API Tokens Hope that helps slightly smiling face

Greg Desmarais · Answer

Hi < Robin> thanks In my case I m not using cloud or k8s I don t think this token helps me regardless I appreciate the suggestion and I found the problem I didn t set the server backend to `server`

Robin · Answer

Alright happy that it works now