https://prefect.io
Join Slack
Hi, is there a way to create a workflow by reading all tasks and dependencies from a json/xml ?
# ask-community
a
Hi, is there a way to create a workflow by reading all tasks and dependencies from a json/xml ?
@Jeremiah Can you please help me with this. Thanks.
j
Hi @AkashB, sorry I missed this before. We don’t ship a way to do this right now, but since Prefect is pure-Python, you could write a parser yourself that loaded the JSON and programmatically generated the flow.
a
Hey @Jeremiah, Thanks for answering. I already did it using a parser and a program. I was just curious if there was already some way in prefect library to do it.
j
We’re going to lock down the Python API and then start working on bindings. We already use JSON as a serialization layer, so there’s some groundwork — but unfortunately we aren’t shipping any builtin parser yet, sorry.
a
Okay. Thanks for your help.
@Jeremiah Can we change the tasks or dependencies in a workflow when it's running ?
And how to use ifelse/switch in imperative flow ?
j
@AkashB your general workflow must be defined before it starts. However there are tools like 
ifelse
and 
map
that can help you take different paths through the workflow or generate dynamic tasks.
Great question about using 
ifelse
in an imperative flow — we should document this in a FAQ. Because functions like 
ifelse
generate intermediate tasks, I’d actually recommend switching to the functional API just for that one call. Something like:
Copy code
# ... imperative flow building
flow.add_task(condition)
# switch to functional for ifelse
with flow:
    result = ifelse(condition, <true>, <false>)
# back to imperative
flow.add_task(...)
you could introspect ifelse, rebuild it yourself, and capture all the intermediate tasks, but this will accomplish the same thing. You can switch between functional and imperative APIs at any time.
However this makes me think we could just add an optional 
flow
argument to 
ifelse
so it could be used in the imperative API without modification. (cc @Chris White for opinion?)
c
Yea I’m on board with that- I think maintaining compatibility with both APIs is important. Let’s open an issue for the ifelse task!
a
Thanks @Jeremiah and @Chris White.
Hey guys, when I try to add a new task to a running workflow, its giving me KeyError. I am able to do 
task.run()
. But when using 
add_task
, it's giving me error. I even tried 
task.set_upstream
. It's also giving the same error.
I am trying to design a dynamic workflow in which I'll be able to add any task to a running workflow at any time.
c
Hi @AkashB - Prefect does not support fully dynamic workflows like you are describing (yet)
a
Okay. Thanks. When will the support for dynamic workflows be available in prefect ?
c
Honestly can’t say - I wouldn’t expect it anytime soon for arbitrary dynamic flows like you are describing; could you explain what use case you have that requires this level of customization?
a
We thought that we had that requirement for dynamic workflow. But now we think we can manage with the existing workflow that we have. Thanks for your help.
👍 1
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by