https://prefect.io logo
Title
b

bardovv

03/18/2020, 6:58 PM
is there any open source alternative to prefect cloud?
l

Laura Lorenz (she/her)

03/18/2020, 7:03 PM
Hi! Any specific reason you are in need of an open source alternative?
b

bardovv

03/19/2020, 6:13 AM
I want to use the dynamic visualization feature of visualization, my needs will be larger than a free tier requirement and I can't afford the pricing, is there some free open source alternative to prefect cloud where I can use the prefect core?
l

Laura Lorenz (she/her)

03/19/2020, 3:02 PM
Gotcha, for the dynamic visualization point I think someone else has mentioned dask UI (https://distributed.dask.org/en/1.12.2/web.html#example-computation) if you are using the dask executor, but dask doesn’t know how the flows are organized, so its more like visualizing overall task throughput than how the flows are going. I don’t personally know of any published open source alternatives to the dynamic visualization of Prefect flows like in Cloud/Cloud UI (maybe someone else will jump in with one or one in progress). I know at least one other person in this channel was interested in something like flow.visualize() to depict states of flows while they are running, which sounds a bit like the usage you’re interested in. Internally we are also considering a terminal-based UI like k9s (https://github.com/derailed/k9s#screenshots) in core to watch and report on flows that are happening in-process. Let me know if either of those are the kind of thing you are looking for so we can prioritize it in core better.
b

bardovv

03/20/2020, 8:00 AM
can prefect core be used with dagster together?
Yes I am looking for the things you have mentioned about.
l

Laura Lorenz (she/her)

03/20/2020, 2:02 PM
Great, I will keep you posted about progress on the core terminal UI or enriching flow.visualize() projects. Right now they exist as whiteboard ideas for us, but once there are github issues I will tag you. As for prefect + dagster, I haven’t tried, but I think dagster’s UI/scheduling features depend on your pipelines being described with their API (as in decorated as
@solid
or
@pipeline
as opposed to Prefect’s
@task
and
with Flow
, etc). Is it that you want to use Prefect’s engine API or task library (or something), but want Dagster’s UI? To be honest though I haven’t dug into dagster too much so I can give it a whirl next week and be able to comment more intelligently afterwards 🙂
b

bardovv

03/22/2020, 3:25 PM
Yes use Prefect engine API with Dagster UI, also is it possible to have a workflow withing a workflow, and have some parallel workflows, I couldn't find examples for the same.