https://prefect.io logo
#prefect-community
Title
# prefect-community
a

Assaf Ben Shimon

04/21/2022, 11:01 PM
Hi all 🙂 Noob question here. I'm trying to run a very basic setup, and get the following error: AttributeError: 'Foo' object has no attribute 'set_dependencies'. (This function is called outside of the flow code) Any idea what I'm doing wrong?
k

Kevin Kho

04/21/2022, 11:02 PM
I think we only have
set_upstream
and
set_downstream
but what is Foo? Is that a task?
Can you show me some code?
I think I understand what is going on. There are a bunch of things that will break here. 1. Parameters can’t be functions. They need to be JSONSerializeable. This might work for
flow.run()
but will break for runs with Prefect Cloud because those are triggered by API requests. Instead, use a string as a default and pass it to an intermediate task to return the function/callable 2. I am unclear at the moment if
get_sequences_attr
is a task. If it is a task, you can’t use
for
loops on task outputs because
for
runs during the Flow
build
time, not during
flow.run()
. So the Flow is built, but the
for
loop doesn’t know how many items there are in the list if it’s a Task. A Task value (including Parameters) are created at runtime. Note the distinction of flow-build and flow-run time. If you need to loop over a task output, you should be looking at mapping. 3. Assuming
sequence_attr
is some task output, you can’t do stuff like
sequence_attr.get_path()
. This is because that is evaluated during build time when it is still type Task. The value doesn’t exist until runtime. Instead, you need this method class to happen inside a task so logic is deferred. This will work though if
sequence_attr
is not a task and already defined during flow-build time.
4. If
get_sequences_attr
is a function, you can’t pass a Parameter to it because Parameters don’t exist during build time. It will not behave as expected and you are passing the Parameter class as opposed to a Parameter value
It might be helpful for you to use Tasks for everything in the Flow block at first, because that is how work gets deferred to
flow.run()
I hope that helps!
a

Assaf Ben Shimon

04/21/2022, 11:24 PM
Yes, thanks a lot!!! Just to make sure I get it, to make something a task all I need is to add the
@task
annotation, right? Or is there anything else?
k

Kevin Kho

04/21/2022, 11:24 PM
Yes that’s right
a

Assaf Ben Shimon

04/21/2022, 11:28 PM
What I'm trying to do is first get a set of relevant items (sequences) to process (of unknown length), and then I want to create a separate task to perform the ETL of each of them. So I need to look into 'mapping', right?
k

Kevin Kho

04/21/2022, 11:28 PM
Yes that is the use case for mapping
a

Assaf Ben Shimon

04/21/2022, 11:29 PM
Awesome. Thanks a lot Kevin, that was super helpful!
k

Kevin Kho

04/21/2022, 11:29 PM
Of course!
6 Views