Does Prefect support multiple targets for a single task? Trying to output multiple attributes from an object as csv rather than pickling the object itself and want to require all of them to be present to skip the task
k
Kevin Kho
07/23/2021, 4:41 PM
Hey @An Hoang, sorry I’m not understanding. Do you mean multiple targets for a single task? I don’t think we have support for that. You would need to create that CSV yourself, or maybe make your own Serializer?
a
An Hoang
07/23/2021, 4:43 PM
Sorry yes I mean multiple targets for single task. So you suggest either use the
Result
object in the task then do the checking of all the files to raise skip signal, or write the Serializer and put the logic there?
k
Kevin Kho
07/23/2021, 4:51 PM
Oh I think I understand now. You want to use targets to check for the existence of multiple files and if they all exist, you want the data to be loaded and the task to be skipped right?
a
An Hoang
07/23/2021, 4:52 PM
Yes!
k
Kevin Kho
07/23/2021, 5:01 PM
Got it, unfortunately no we don’t have support for this. You probably need to load those files, check for the condition and then raise the SKIP flag yourself to skip the task like
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.