Constantino Schillebeeckx

    Constantino Schillebeeckx

    11 months ago
    I've got a task that I'm calling from another task like:
    mount_glue_catalog.run(schema_name=schema_name, glue_db=glue_db_name, upstream_tasks=[create_catalog])
    where
    create_catalog
    is the return from another task. When I execute this I'm getting
    TypeError: mount_glue_catalog() got an unexpected keyword argument 'upstream_tasks'
    When I call this from a flow context, the invocation of
    mount_glue_catalog
    doesn't have any issues. Does the interface to running this task somehow change based on my calling it with run() vs from within a flow context?
    Kevin Kho

    Kevin Kho

    11 months ago
    So I don’t know for sure if you can use
    Task
    kwargs when calling run like this, but in general I don’t think you can because using a task like this is just the underlying Python function. With
    upstream_tasks
    specifically, there is a disconnect where
    upstream_tasks
    is a build-time concept as the DAG is created and using a task in another task is a run-time thing and you can’t create the dependency during runtime. The
    upstream_tasks
    should go in the
    Flow
    block
    Constantino Schillebeeckx

    Constantino Schillebeeckx

    11 months ago
    cool got it - as always, thanks for the speedy response