https://prefect.io logo
Title
n

Nimesh Kumar

01/09/2023, 5:09 PM
Hi everyone, I am just getting started with Prefect by setting up my server and writing a few workflows locally, i get stuck in a situation where i need to trigger a workflow using an API like we do in Python Requests module. I have come across a few links where they mention that it is possible using the AWS lambda function. So I wanted to know if the Python Requests module can trigger a workflow locally or not. I am using prefect 2.7.7 version
1
m

Mason Menges

01/09/2023, 5:42 PM
Hey @Nimesh Kumar you can trigger flow runs from the prefect client API, you'll need to have created a deployment that can be accessed from the API this discourse article is also a good place to start with this as well https://discourse.prefect.io/t/event-driven-data-pipelines-with-aws-lambda-prefect-and-github-actions/1597.
n

Nimesh Kumar

01/09/2023, 6:53 PM
Thank you for the response. , As per your suggestion, I have created a deployment and tried triggering using the below code
t = await client.create_flow_run_from_deployment(deployment_id="a49ac4d8-0855-4000-9cfc-c352319eba0c", name="TriggeredExternally")
In the dashboard it is reflecting that the flow is triggered but the state is not changing and shows "Late"
This is solved..... Actually there was a problem in Queue.
DOUBT Hi Let suppose i have created a workflow "X" and for that i have created a deployment "Y" with a queue "Z" I am triggering the workflow deployment using deployment externally after running
prefect agent start --work-queue "Z"
So i have doubt that when the deployment is triggered from multiple system it will still use the same queue i.e. "Z" Or we can assign the queue while triggering the deployment.? Or we can create multiple queues initially, and if multiple requests come it will automatically get assigned to a queue (but then how is the prefect deciding which queue to use) ?
1
r

redsquare

01/10/2023, 10:50 AM
The deployment is assigned to a queue and will always use that queue when triggered
n

Nimesh Kumar

01/10/2023, 10:51 AM
So a single deployment cannot have a multiple queues? or there is be multiple workers for a queue which I can configure
r

redsquare

01/10/2023, 11:01 AM
Typically the queue doesnt do much work - just create a job for the flow to run
n

Nimesh Kumar

01/10/2023, 11:03 AM
so if I trigger deployment let say 1000 time in 30 sec then queue will be a bottle neck
so if I trigger deployment let say 1000 time in 30 sec then queue will not be a bottle neck ?
r

redsquare

01/10/2023, 11:15 AM
Too hard to say, depends on whats its doing, the size of the vm, you can have multiple agent processes to be started for a single work queue
n

Nimesh Kumar

01/10/2023, 11:22 AM
Can you please provide me any links related to setting up multiple agents for a single work queue. I tried searching but couldn't found any
r

redsquare

01/10/2023, 11:23 AM
n

Nimesh Kumar

01/10/2023, 5:22 PM
Hi, I apologize if I am bothering you too much, but I did go through the link you provided, but I am still confused. you are saying that starting multiple agent for a single queue can help but I am not able to figure out how can I start multiple agent
prefect agent start -q "my-queue"
This will start one agent and create a new queue. Now, I need to create a another agent for the same queue i.e.. "my-queue" and this where i got stuck.
r

redsquare

01/10/2023, 5:25 PM
yeah you can just create another agent listening to the same queue
n

Nimesh Kumar

01/10/2023, 5:28 PM
prefect agent start -q "my-queue"
prefect agent start -q "my-queue"
This will create two agents that will point to the same queue, "my-queue," right ?
r

redsquare

01/10/2023, 5:32 PM
yeah
n

Nimesh Kumar

01/10/2023, 5:33 PM
Thanks a lot 🙌
👍 1