https://prefect.io logo
Powered by
Title
g

Greg Desmarais

07/17/2020, 9:50 PM
I'm feeling a bit lost and embarrassed to be lost, but have a question about what seems like a simple deployment. Here is what I'm trying to do:
This seems to get very far, but then gets stuck. All the code is running on my local, and I have full access to the VPC through VPN. The flow is written to S3 just fine, the ECS cluster is created, I can telnet to the cluster scheduler at :8786 and :8787, and the task in ECS seems correct. It gets to this point in the logs: 
2020-07-17T16:45:04.792-04:00
distributed.scheduler - INFO - Scheduler at: <tcp://10.72.9.106:8786>

2020-07-17T16:45:04.792-04:00
distributed.scheduler - INFO - dashboard at: :8787

2020-07-17T16:45:09.566-04:00
distributed.scheduler - INFO - Receive client connection: Client-675482da-c86e-11ea-b8be-acde48001122

2020-07-17T16:45:09.577-04:00
distributed.core - INFO - Starting established connection
I have connectivity from my local machine issuing the flow.run to the prefect services I started on an EC2.
By the way, I threw together the diagram quickly. There is a decent chance I forgot something.
The dask-scheduler task sits there forever with 
Starting established connection
and the client issuing the flow.run() sits there with 
Starting flow run
.
I have a feeling I'm missing something with an agent or some other process. It seems all my networking is set up correctly as I can hit everything from my client, the prefect servers seem happy, the ecs cluster starts up fine...etc.
j

Jeremiah

07/17/2020, 10:16 PM
Hi Greg - sorry you’re feeling lost! Server deploys are very tricky for us to debug due to the uniqueness of each situation, but a few things to look for: • I don’t see an agent running in your diagram; Server expects an agent to be running (in fargate, probably) to receive the 
run
command and launch the flow into the Dask cluster • Similarly, when you say you’re calling 
flow.run()
you’ll probably actually want to call 
flow.register()
and then issue either a GQL command to run the flow via Agent, or click the 
run
command in the Prefect UI
Ah, never mind - I just looked closely at your code and I think you’re not necessarily running a server deploy! You’re running a script that should be connecting to a remote Dask cluster.
In order to turn this into a server deploy, you’ll want to attach an Environment to your flow, and then call 
flow.register()
against the server backend.
Then you won’t call flow.run() in your script; instead, you’ll register that flow and its environment to the server, which will issue the instruction to run to an Agent.
g

Greg Desmarais

07/18/2020, 2:48 AM
Ok - what you are saying makes sense. I've never really known where to put what agent...so, this agent which I'm not reading about - if I start it programmatically, which one do I start? e.g. prefect.agent.docker.DockerAgent or other? Or is it that the different agents know which flows they should be pulling off the run queue?
And - where should I be running that agent? On the same box as the EC2 instance where I ran 
prefect server start
? That seems like a logical place to me. But...when I tried that, I got: 
prefect.utilities.exceptions.AuthorizationError: No agent API token provided.
My guess is that I need to update my .prefect/config.toml to point the agent at my server deployment.
But, I do't know where to find docs on the configs, so I'm somewhat guessing. maybe 
prefect.context.config.cloud.api
, or 
prefect.context.config.server.host
?
also, in the described process, I create the fargate cluster, then call run (which I will flip to register). Then, as I understand it, the flow is in the prefect server, and the agent notices this, picks up the flow and runs the executors for it. (am I right?) Is it possible to have the agent spin up the cluster on its own? It would be great to do that as I would not need to give the person writing the script all sorts of permissions to create resources - I could consolidate into a role.
At the least - thanks @Jeremiah - you are getting me further. Hopefully if I figure out a good recipie it can be of use to the community.
I think there'll be a bunch of people that will want to use it in a similar way.
ok, realized the agent was looking for the api token when the config was in cloud mode - switching it to server let the agent start up.
r

Robin

07/18/2020, 7:06 AM
Hey @Greg Desmarais, using e.g. 
prefect agent start kubernetes --token MY_TOKEN
with a token that you can create via the prefect cloud UI >> Team >> API Tokens Hope that helps 🙂 https://docs.prefect.io/orchestration/agents/overview.html
g

Greg Desmarais

07/18/2020, 2:54 PM
Hi @Robin - thanks. In my case, I'm not using cloud or k8s...I don't think this token helps me? regardless, I appreciate the suggestion, and I found the problem - I didn't set the server backend to 
server
.
r

Robin

07/18/2020, 3:09 PM
Alright, happy that it works now!
2 Views
#prefect-serverJoin Slack
Powered by