https://prefect.io logo
Channels
Powered by
#prefect-community
Title
# prefect-community
a

AkashB

08/22/2019, 6:04 AM
Hi, is there a way to create a workflow by reading all tasks and dependencies from a json/xml ?
@Jeremiah Can you please help me with this. Thanks.
j

Jeremiah

08/23/2019, 4:20 AM
Hi @AkashB, sorry I missed this before. We don’t ship a way to do this right now, but since Prefect is pure-Python, you could write a parser yourself that loaded the JSON and programmatically generated the flow.
a

AkashB

08/23/2019, 5:02 AM
Hey @Jeremiah, Thanks for answering. I already did it using a parser and a program. I was just curious if there was already some way in prefect library to do it.
j

Jeremiah

08/23/2019, 5:04 AM
We’re going to lock down the Python API and then start working on bindings. We already use JSON as a serialization layer, so there’s some groundwork — but unfortunately we aren’t shipping any builtin parser yet, sorry.
a

AkashB

08/23/2019, 5:17 AM
Okay. Thanks for your help.
@Jeremiah Can we change the tasks or dependencies in a workflow when it's running ?
And how to use ifelse/switch in imperative flow ?
j

Jeremiah

08/24/2019, 2:26 PM
@AkashB your general workflow must be defined before it starts. However there are tools like 
ifelse
and 
map
that can help you take different paths through the workflow or generate dynamic tasks.
Great question about using 
ifelse
in an imperative flow — we should document this in a FAQ. Because functions like 
ifelse
generate intermediate tasks, I’d actually recommend switching to the functional API just for that one call. Something like:
Copy code
# ... imperative flow building
flow.add_task(condition)
# switch to functional for ifelse
with flow:
    result = ifelse(condition, <true>, <false>)
# back to imperative
flow.add_task(...)
you could introspect ifelse, rebuild it yourself, and capture all the intermediate tasks, but this will accomplish the same thing. You can switch between functional and imperative APIs at any time.
However this makes me think we could just add an optional 
flow
argument to 
ifelse
so it could be used in the imperative API without modification. (cc @Chris White for opinion?)
c

Chris White

08/24/2019, 6:29 PM
Yea I’m on board with that- I think maintaining compatibility with both APIs is important. Let’s open an issue for the ifelse task!
a

AkashB

08/26/2019, 4:09 AM
Thanks @Jeremiah and @Chris White.
Hey guys, when I try to add a new task to a running workflow, its giving me KeyError. I am able to do 
task.run()
. But when using 
add_task
, it's giving me error. I even tried 
task.set_upstream
. It's also giving the same error.
I am trying to design a dynamic workflow in which I'll be able to add any task to a running workflow at any time.
c

Chris White

08/26/2019, 2:21 PM
Hi @AkashB - Prefect does not support fully dynamic workflows like you are describing (yet)
a

AkashB

08/27/2019, 4:06 AM
Okay. Thanks. When will the support for dynamic workflows be available in prefect ?
c

Chris White

08/27/2019, 4:09 AM
Honestly can’t say - I wouldn’t expect it anytime soon for arbitrary dynamic flows like you are describing; could you explain what use case you have that requires this level of customization?
a

AkashB

08/27/2019, 11:22 AM
We thought that we had that requirement for dynamic workflow. But now we think we can manage with the existing workflow that we have. Thanks for your help.
👍 1
Powered by