👋 do you all have thoughts on when you'd use prefect over something like temporal/cadence? or, is the intention that you should feel comfortable using prefect for any use case that the latter aims to address?

I can't say anything about temporal, but if you tell me what problem(s) you try to solve, I can tell you whether Prefect may or may not be a good fit

fair enough about any, definitely a strong word 🙂. Our use case is basically that we generate data for people based on various inputs, but the inputs may or may not be ready at the time of derivation. so, if we were generating something like:

c = a + b

and

a

is ready and available but

b

is not, then the system is responsible for informing another system and telling

b

to go be generated, and then responsible for calculating

c

once

b

is ready. you can also imagine that

b

can change over time, so we want to not only know

c

, but we'll want to know:

c_v1 = a_v1 + b_v1
c_v2 = a_v1 + b_v2
c_v3 = a_v2 + b_v2
...

definitely doable in Prefect, as it natively supports dataflow between tasks and even between flows (subflows) if you want to store some pieces of metadata for in-between steps that you don't want to pass as data dependency, you can use sth like JSON block

that makes sense! and fwiw we've already started building this on prefect and things generally are pretty straightforward to implement (sometimes things can be a little weird when we run into weird derivation trees and representing them as flows and tasks), but i was just doing some reading and came across temporal so i didn't know if you all had any thoughts on why you'd use one or the other. a quick follow-up question: another thing that we haven't quite gotten to but are hopeful about is the scale at which we can run prefect. assuming we provide enough resources, do you all have concerns with running thousands (maybe even tens of thousands) of a given flow simultaneously? perhaps i want to

generate_data(p_1) --> generate_data(p_n)