Is there a reason that flow.result and flow.executor are not serialized for flow.register, unlike fl...

xyzy

04/07/2021, 6:08 PM

Is there a reason that flow.result and flow.executor are not serialized for flow.register, unlike flow.run_config and flow.storage?

Kevin Kho

04/07/2021, 7:03 PM

Hey @xyzy, I’ll follow up if I find a better answer but I think this is because different results need different types of serializers. Is there a use case you’re asking this for?

xyzy

04/07/2021, 8:02 PM

I'm using a script to register flows for the server and would like to keep the main flow files DRY. Currently I need to specify to Executor used by server in the top level of the flow py and the Executor used for direct local execution inside the

if __name__ == "__main__

branch that also calls flow.run

Kevin Kho

04/07/2021, 8:16 PM

I think I get what you mean, but could you give me a simple flow with the

if __name__ == "__main__"

and the Executor being defined?

Kevin Kho

04/07/2021, 8:22 PM

Also the script to register flows would help.

Kevin Kho

04/08/2021, 1:55 PM

I took a look. This all looks good. I don’t see another way other than to overwrite the executor like what you’re doing. The only thing I’ll mention is that we have a Prefect CLI for registering flows that might help. https://docs.prefect.io/api/latest/cli/register.html . Is there something in your register script that isn’t covered?

xyzz

04/09/2021, 7:49 AM

Hi Kevin, the script does a few things thant can't be done with the CLI tool... E.g. automatically selecting project names, storage, run_config and applying schedules or kubernetes spec files depending on the execution mode and what's defined in the flow file

👍 1

Kevin Kho

04/09/2021, 1:03 PM

Got it

4 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.