Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

image.png

If I may point out a design inconsistency: that parameters and secrets sometimes are "values as declaration time" and sometimes "values as output by tasks" — this looks like an anti-pattern to me:

Another inconsistency: there's no default for secrets.

On the first one, yes your are right. This is a known issue and is being tackled here: <https://github.com/PrefectHQ/prefect/issues/4377>

Do you mean grabbing a default secret if one is not supplied?

I see maybe you should an issue for that one.

Remove `Secret` if it's not supposed to be used

Make everything that can be a task, possible to give as a task

I have some errands I need to run today, but I'll definitely respond to you on Monday about this :sweat_smile:

Ensure all API surface areas are available an an immutable fashion; you don't gain anything by 'currying' the state of your tasks with a two-layered 1) constructor, 2) run method; only complexity

Document the `@task` surface area; it's not always working and `kwargs` is not very discoverable when you want to override "statically" configured properties

And why reverse the source array and applicable function in `map`? Wouldn't it be easier to have an overload?

results.map(opts, task(lambda (opts, res): res + 1))

where `opts` is an object with the "unmapped" parameters?

Also, there type annotations are lacking for `map` having lots of `Any` values; I'd be happy to review fixes for that — I know my way around typed functional DSLs

I'll quickly throw in here also that secrets should probably 1) on a flow level, 2) on a group/team level, or at least be possible to set on a flow level, since they are "similar" to parameters, except that they are secret

Here are the refactored tasks <https://gist.github.com/haf/5449a94c5ab954e9f27a70d6a07afe0b>

Hey <@U01QC4HNP0D>, we have the issue open to remove the two layered constructor and run and push them to run time. One of the issues was that using parameters in construction was not possible. You can PR the refactored tasks.

Will bring up to team about 1, default secrets and 2, flow level/group level secrets.