a

    Andrew Nichol

    1 year ago
    Hey y'all - really struggling on how to create a parametrized dependent flow. https://docs.prefect.io/core/idioms/flow-to-flow.html#running-a-parametrized-flow the description given is exactly what i want to do:
    Let's say that you want to always run a flow with parameters that are generated by another flow. Naively, you might manually wait for one flow to finish and then manually trigger the next flow to run with the appropriate parameter values. Prefect makes this pattern easy to automate via the StartFlowRun:
    but the example code given represents a flow which is dependent upon a task that returns parameters, not a flow that is dependent upon another flow. i think the main piece im struggling with is how to return/access some values of the preceding flow from the succeeding flow. (basically getting the input to the
    parameters
    arg of the succeeding flow)
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Andrew Nichol! There are a couple of options to do this but it’s not as simple as feeding the Flow output of a
    StartFlowRun
    task in the next
    StartFlowRun
    . The docs is more about just setting the dependencies. First class support for passing data is something we are working on. The first option is persisting that output somewhere like S3 for downstream Flow runs to load. The second one is more involved in using the
    Client
    to query the previous flow and get the appropriate information (this would be more appropriate for things like state).
    a

    Andrew Nichol

    1 year ago
    okay very helpful - thanks @Kevin Kho! for the first option, are you suggesting using some of the built-in Results api stuff for that? or managing that push+pull to s3 myself?
    Kevin Kho

    Kevin Kho

    1 year ago
    The result interface might be easier but I’ve seen people make their own interface as well
    a

    Andrew Nichol

    1 year ago
    sounds good. something else i noticed is that when running flows with StartFlowRun i lose meaningful logs when tasks within those flows fail. instead i always just get somethign along the lines of
    FAIL signal raised: FAIL('6852b795-df89-4307-971c-c7573200f19c finished in state <Failed: "Some reference tasks failed.">')
    and no stacktrace from the task code - making it quite difficult to debug
    Kevin Kho

    Kevin Kho

    1 year ago
    I know what you mean. In general, the parent log does not get the logs of child Flows. You’re need to go to the child flow runs triggered and get the logs from there.
    Feel free to read and chime in to this issue by the way,
    a

    Andrew Nichol

    1 year ago
    okay that works fine i can navigate to child flows for now
    thanks will check that out!