https://prefect.io
Join Slack
Hi everyone, I am just getting started with Prefect by setting up my server and writing a few workfl...
# prefect-getting-started
n
Hi everyone, I am just getting started with Prefect by setting up my server and writing a few workflows locally, i get stuck in a situation where i need to trigger a workflow using an API like we do in Python Requests module. I have come across a few links where they mention that it is possible using the AWS lambda function. So I wanted to know if the Python Requests module can trigger a workflow locally or not. I am using prefect 2.7.7 version
✅ 1
m
Hey @Nimesh Kumar you can trigger flow runs from the prefect client API, you'll need to have created a deployment that can be accessed from the API this discourse article is also a good place to start with this as well https://discourse.prefect.io/t/event-driven-data-pipelines-with-aws-lambda-prefect-and-github-actions/1597.
n
Thank you for the response. , As per your suggestion, I have created a deployment and tried triggering using the below code
Copy code
t = await client.create_flow_run_from_deployment(deployment_id="a49ac4d8-0855-4000-9cfc-c352319eba0c", name="TriggeredExternally")
In the dashboard it is reflecting that the flow is triggered but the state is not changing and shows "Late"
This is solved..... Actually there was a problem in Queue.
DOUBT Hi Let suppose i have created a workflow "X" and for that i have created a deployment "Y" with a queue "Z" I am triggering the workflow deployment using deployment externally after running
Copy code
prefect agent start --work-queue "Z"
So i have doubt that when the deployment is triggered from multiple system it will still use the same queue i.e. "Z" Or we can assign the queue while triggering the deployment.? Or we can create multiple queues initially, and if multiple requests come it will automatically get assigned to a queue (but then how is the prefect deciding which queue to use) ?
✅ 1
r
The deployment is assigned to a queue and will always use that queue when triggered
n
So a single deployment cannot have a multiple queues? or there is be multiple workers for a queue which I can configure
r
Typically the queue doesnt do much work - just create a job for the flow to run
n
so if I trigger deployment let say 1000 time in 30 sec then queue will be a bottle neck
so if I trigger deployment let say 1000 time in 30 sec then queue will not be a bottle neck ?
r
Too hard to say, depends on whats its doing, the size of the vm, you can have multiple agent processes to be started for a single work queue
n
Can you please provide me any links related to setting up multiple agents for a single work queue. I tried searching but couldn't found any
r
n
Hi, I apologize if I am bothering you too much, but I did go through the link you provided, but I am still confused. you are saying that starting multiple agent for a single queue can help but I am not able to figure out how can I start multiple agent
Copy code
prefect agent start -q "my-queue"
This will start one agent and create a new queue. Now, I need to create a another agent for the same queue i.e.. "my-queue" and this where i got stuck.
r
yeah you can just create another agent listening to the same queue
n
Copy code
prefect agent start -q "my-queue"
prefect agent start -q "my-queue"
This will create two agents that will point to the same queue, "my-queue," right ?
r
yeah
n
Thanks a lot 🙌
👍 1
13 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by