https://prefect.io logo
#prefect-contributors
Title
# prefect-contributors
j

jcozar

03/04/2022, 11:35 AM
Hi! Has anyone tried to implement a new type of prefect agent? I would like to read more about how it works. I don’t know if Prefect Agent is always asking Prefect Server/Cloud to run new flow runs, or if Prefect Server/Cloud sends actively the flow run to an active agent. I’m thinking of a Prefect Webhook Agent that, when there is a new flow run to be executed, invoke a webhook that executes the flow run. For example, in AWS it could be API Gateway running a AWS Lambda function or Fargate task to run the flow run. Do you think it would be possible? Thank you in advance!
a

Anna Geller

03/04/2022, 11:51 AM
I think this would be possible - if you look at the agent.py: "Agent is a standard point for executing Flows through the Prefect API. It is meant to have subclasses which inherit functionality from this class. The only piece that the subclasses should implement is the ``deploy_flows`` function, which specifies how to run a Flow on the given platform. It is built in this way to keep Prefect API logic standard but allows for platform specific customizability." But long-term it may be beneficial to instead think about contributing a custom
FlowRunner
to Orion (which will become Prefect 2.0 in near future) instead of implementing this for Prefect 1.0. To read more about flow runners check out: the FlowRunner concept docs.
j

jcozar

03/04/2022, 11:59 AM
Ok thank you @Anna Geller, I have Orion in my TODO list, I will prioritize it! 😄
🙌 1
z

Zanie

03/04/2022, 3:05 PM
This would be feasible and is a fun idea! A flow runner / agent needs to be able to do a couple things in the infrastructure it spawns • Set environment variables • Call a Python function (generally, via a CLI command)
j

jcozar

03/15/2022, 10:21 PM
Hi @Zanie, thanks for your response! Sorry but I couldn’t reply before. I read the documentation of Prefect Orion. I found information about flow runners and how to implement my own one, but I want something more related to the the agent itself, not how the agent execute flows: I think that it would be awesome to implement a webhook (I am thinking of an endpoint in AWS API Gateway) that Prefect Server/Cloud could invoke when a flow needs to be run. That endpoint should use the same flow as agents to execute the flow run and send a notification back to the Prefect Server / Cloud, as if it were a Prefect Agent. Do you think that it is feasible? Where can I find documentation to contribute to Prefect with this idea? Thank you very much!
I think that the code in the webhook should be something similar than in https://github.com/PrefectHQ/prefect/blob/orion/src/prefect/cli/agent.py (
await agent.get_and_submit_flow_runs()
). But I don’t know where Onion should invoke the endpoint.
4 Views