haf

    haf

    1 year ago
    If I may point out a design inconsistency: that parameters and secrets sometimes are "values as declaration time" and sometimes "values as output by tasks" — this looks like an anti-pattern to me:
    Another inconsistency: there's no default for secrets.
    Kevin Kho

    Kevin Kho

    1 year ago
    On the first one, yes your are right. This is a known issue and is being tackled here: https://github.com/PrefectHQ/prefect/issues/4377
    Do you mean grabbing a default secret if one is not supplied?
    haf

    haf

    1 year ago
    Yes
    Useful for testing environments
    Kevin Kho

    Kevin Kho

    1 year ago
    I see maybe you should an issue for that one.
    haf

    haf

    1 year ago
    Why not unify the API?
    Remove
    Secret
    if it's not supposed to be used
    Make everything that can be a task, possible to give as a task
    Kevin Kho

    Kevin Kho

    1 year ago
    I have some errands I need to run today, but I'll definitely respond to you on Monday about this 😅
    haf

    haf

    1 year ago
    Ensure all API surface areas are available an an immutable fashion; you don't gain anything by 'currying' the state of your tasks with a two-layered 1) constructor, 2) run method; only complexity
    Document the
    @task
    surface area; it's not always working and
    kwargs
    is not very discoverable when you want to override "statically" configured properties
    And why reverse the source array and applicable function in
    map
    ? Wouldn't it be easier to have an overload? results.map(opts, task(lambda (opts, res): res + 1)) where
    opts
    is an object with the "unmapped" parameters?
    Also, there type annotations are lacking for
    map
    having lots of
    Any
    values; I'd be happy to review fixes for that — I know my way around typed functional DSLs
    I'll quickly throw in here also that secrets should probably 1) on a flow level, 2) on a group/team level, or at least be possible to set on a flow level, since they are "similar" to parameters, except that they are secret
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @haf, we have the issue open to remove the two layered constructor and run and push them to run time. One of the issues was that using parameters in construction was not possible. You can PR the refactored tasks.
    Will bring up to team about 1, default secrets and 2, flow level/group level secrets.
    You can open a issue for the map