Hi! What would be “the right way” in Prefect to do what Luigi refers to as ExternalTask (e.g. a file in S3 that gets created by some external process, and the rest of the pipeline can not proceed without it) ?

Hey @Marko Mušnjak! Just a couple of questions. Are you expecting this file to start the Flow run? What do you want to happen if the file doesn’t appear? Do you think the writing of that file can be a Prefect task?

the setup with luigi is that the pipeline is started, and retried some time later if the file is not available. There could be some time between the file appearing and the pipeline continuing (it’s not time-critical)

Gotcha so the Prefect approach would just be to check for the file existence, and if not available raise the SKIP signal and this will skip all downstream flows and treat them as successes

let me check if I understood that correctly: Run is scheduled at some time we check for existence of file If it’s not there, we start another flow run at some later time, and set status as SKIP if it’s there, we continue with processing

Yes exactly

great! Thank you!