https://prefect.io logo
Join Slack
Powered by
Hi <@U01DYQ1EVQQ>! Take a look at examples you can...
# show-and-tell
a
Hi @tsar! Take a look at examples you can find in Prefect github repo. https://github.com/PrefectHQ/prefect/tree/master/examples/tutorial You can find some good starting points regarding your use case
t
hi, thanks for answering. I'm mostly asking about the logic here not the actual code. Not sure how these examples can help me ><
in specific, I'm not sure how to do the following logic: so as I said, there is an api that gives me data on tasks that need to be run on a specific time and I would like to schedule them and this I'm not sure how to do. I have no problem writing a python script that does it all so the programming is not a problem but I would like to do this with Perfect
the tasks are the same just different input
so lets say api gives me a json that says I have to run 2 tasks with this input on this date and 3 tasks with this input on another date.
logically with a python script I would have a script that fetches the tasks from the api and puts them in a database with the time that it needs to be run and another script that checks the database every X seconds to see if there are tasks to be run on the specific time that the script is running and if any it runs them.
a
Prefect provides out of the box scheduling and parameterization. You can get flow info by using Prefect GraphQL client. You can also use Prefect Client to trigger flow execution
By the way, I strongly suggest you take a look at the docs to get a general understanding of Prefect architecture
t
but I would like to do this whole process automatically .. . the check for new tasks from the my api should be done every X seconds and schedule tasks if any. I will look into graph client but it sounds to me like something I have to call from cli or a script.
I already have taken a look at the architecture. for sure I'm missing something since solutions are not very clear.
looking at the docs of the client looks like I have to create a flow which uses the client to create more flows. is that right ? thnx
a
Sorry @tsar but to me it's not very clear what your use case looks like 😅
t
no problem, I understand it can be hard to comprehend. Let me try again.
I have an application which has a rest api that I want to query and will provide me with input data and the time/date which this data should be processed. I want to query this api every 1minute.
the api would return for example that two jobs need to be completed in different time/date. job1 : Tuesday 1PM, input 10 job2 : Wednesday 3PM, input 20
so I would want to schedule these jobs to run on that specific time and do some tasks based on the initial input. so for example on Tuesday I need to run X number of tasks for input 10 and so forth.
I would like to automate this process with Perfect 🙂
a
You could create Prefect schedules using data you get from the rest api. I think schedules can be created programmatically using Prefect client But flows has to be already registered in Prefect Server or Cloud.
🙏 1
t
so I have a flow which runs every 1 minute and uses : *prefect.client.client.Client.create_flow_run*(flow_id=None, context=None, parameters=None, labels=None, scheduled_start_time=None, idempotency_key=None, run_name=None, version_group_id=None) to create a new flow scheduled at the time I want. You said that the flow has to be there which is fine. I assume I have to provide that flow with flow_id. How do I provide the input of this flow ? Is it with context ?
a
The easiest way to get the flow_id would be using GraphQL client I guess
t
I don't mean the input for the flow id but for the flow itself. In my case the input is "10". I assume flow id I can get and have it statically defined since it wont change.
a
If your inputs can be represented with simple values (int, string, bool, list, etc), then you can refactor your flow to accept Parameters. Then, when you are triggering flow run, you can pass param values
t
*prefect.client.client.Client.create_flow_run*(flow_id=None, context=None, parameters=None, labels=None, scheduled_start_time=None, idempotency_key=None, run_name=None, version_group_id=None) the flow run will be created by this calls. where do I pass the parameters ?
a
There’s a parameters argument in the call 🙂
t
silly me. 😛 thanks !
a
👍
3 Views
Open in Slack
PreviousNext
Powered by