Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
a
ale
10/24/2020, 11:02 AMHi @tsar!
Take a look at examples you can find in Prefect github repo.
https://github.com/PrefectHQ/prefect/tree/master/examples/tutorial
You can find some good starting points regarding your use case
t
tsar
10/24/2020, 11:05 AMhi, thanks for answering. I'm mostly asking about the logic here not the actual code. Not sure how these examples can help me ><
in specific, I'm not sure how to do the following logic:
so as I said, there is an api that gives me data on tasks that need to be run on a specific time and I would like to schedule them and this I'm not sure how to do.
I have no problem writing a python script that does it all so the programming is not a problem but I would like to do this with Perfect
the tasks are the same just different input
so lets say api gives me a json that says I have to run 2 tasks with this input on this date and 3 tasks with this input on another date.
logically with a python script I would have a script that fetches the tasks from the api and puts them in a database with the time that it needs to be run and another script that checks the database every X seconds to see if there are tasks to be run on the specific time that the script is running and if any it runs them.
a
ale
10/24/2020, 11:50 AMPrefect provides out of the box scheduling and parameterization.
You can get flow info by using Prefect GraphQL client.
You can also use Prefect Client to trigger flow execution
By the way, I strongly suggest you take a look at the docs to get a general understanding of Prefect architecture
t
tsar
10/24/2020, 11:57 AMbut I would like to do this whole process automatically .. . the check for new tasks from the my api should be done every X seconds and schedule tasks if any. I will look into graph client but it sounds to me like something I have to call from cli or a script.
I already have taken a look at the architecture. for sure I'm missing something since solutions are not very clear.
looking at the docs of the client looks like I have to create a flow which uses the client to create more flows. is that right ? thnx
a
ale
10/24/2020, 12:22 PMSorry @tsar but to me it's not very clear what your use case looks like 😅
t
tsar
10/24/2020, 12:23 PMno problem, I understand it can be hard to comprehend. Let me try again.
I have an application which has a rest api that I want to query and will provide me with input data and the time/date which this data should be processed. I want to query this api every 1minute.
the api would return for example that two jobs need to be completed in different time/date.
job1 : Tuesday 1PM, input 10
job2 : Wednesday 3PM, input 20
so I would want to schedule these jobs to run on that specific time and do some tasks based on the initial input.
so for example on Tuesday I need to run X number of tasks for input 10 and so forth.
I would like to automate this process with Perfect 🙂
a
ale
10/24/2020, 1:57 PMYou could create Prefect schedules using data you get from the rest api.
I think schedules can be created programmatically using Prefect client
But flows has to be already registered in Prefect Server or Cloud.
🙏 1
t
tsar
10/25/2020, 9:56 AMso I have a flow which runs every 1 minute and uses :
*prefect.client.client.Client.create_flow_run*(flow_id=None, context=None, parameters=None, labels=None, scheduled_start_time=None, idempotency_key=None, run_name=None, version_group_id=None)
to create a new flow scheduled at the time I want.
You said that the flow has to be there which is fine. I assume I have to provide that flow with flow_id.
How do I provide the input of this flow ? Is it with context ?
a
ale
10/26/2020, 7:57 AMThe easiest way to get the flow_id would be using GraphQL client I guess
t
tsar
10/26/2020, 7:59 AMI don't mean the input for the flow id but for the flow itself. In my case the input is "10".
I assume flow id I can get and have it statically defined since it wont change.
a
ale
10/26/2020, 8:26 AMIf your inputs can be represented with simple values (int, string, bool, list, etc), then you can refactor your flow to accept Parameters.
Then, when you are triggering flow run, you can pass param values
t
tsar
10/26/2020, 8:27 AM*prefect.client.client.Client.create_flow_run*(flow_id=None, context=None, parameters=None, labels=None, scheduled_start_time=None, idempotency_key=None, run_name=None, version_group_id=None)
the flow run will be created by this calls. where do I pass the parameters ?
a
ale
10/26/2020, 9:01 AMThere’s a parameters argument in the call 🙂
t
tsar
10/26/2020, 9:02 AMsilly me. 😛
thanks !
a
ale
10/26/2020, 9:02 AM👍