Hello. Can someone probably tell me if there is a feature to have setup task for flow to run by default before starting the flow tasks? I see that I can use add_task, but how to make it root task?

@Vadym Dytyniak can you just add it as a normal task that runs first? you could then specify that it should run first via task dependencies

Sure, but I am wondering if there is a chance to avoid adding the same task in each flow

if you need a setup and cleanup, you could have a look at resource manager: https://docs.prefect.io/core/idioms/resource-manager.html

If you don’t need observability around it, you can use a State Handler for the Flow and then create some kind of

DefaultFlow

class that contains your defaults.

@Kevin Kho Thanks. Will have a look. Are the async?

No it’s not. Async support is native in Orion though (Prefect 2.0)

So, Can I catch the moment before running flow task and do some setup?

Yeah you can. You can put whatever Python code in the state transition from Submitted to Running, which is before the tasks run. It’s just not async.

That is what I need actually

We only have the one on DockerHub. I don’t know the base OS of that.

Yes. that is what we see. We have libraries that work only in Ubuntu

What is your error? Is the module a normal Python module or custom one? Or the Prefect module?

custom one

Do you have multiple Python installations in the image?

yes, we set python3 to use what we need (3.9), but we actually not sure what prefect is using to start the flow

It uses the cli command

prefect execute flow

@Kevin Kho Have a problem to run. Downloading flow file from s3 storage and what should be the command to run it?

@Vadym Dytyniak not sure I understood the question- you don't need to set any command in your Dockerfile to run your flow, Prefect does it automatically when you schedule or start a flow run. Can you share your storage and run configuration, as well as Dockerfile?

Storage:

return S3(
    bucket=Secret('PREFECT_BUCKET').get(),
    key=f'{self.name}/{self.serialized_hash()}',
    secrets=['AWS_CREDENTIALS'],
)

Run config:

ECSRun(
    image='custom_image',
    env={
        'PIP_EXTRA_INDEX_URL': os.environ['PIP_EXTRA_INDEX_URL'],
        'EXTRA_PIP_PACKAGES': ' '.join(self.dependencies),
    },
)

Custom image: FROM base image # install tini ENV TINI_VERSION v0.19.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini RUN chmod +x /tini ENTRYPOINT ["/tini", "--"] # install prefect ENV PREFECT_VERSION 0.15.10 RUN python3.9 -m pip install prefect[aws]==${PREFECT_VERSION} # update alternatives RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.9 10 RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 10 # prepare entrypoint script COPY prefect_entrypoint.sh /usr/local/bin/entrypoint.sh RUN chmod +x /usr/local/bin/entrypoint.sh CMD ["entrypoint.sh"] entrypoint.sh copied from pure prefect image

I would strongly recommend using an official Prefect image from Dockerhub. You shouldn’t have to add this:

# prepare entrypoint script
COPY prefect_entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
CMD ["entrypoint.sh"]

Some additional notes: I usually see the ENV defined with = sign e.g.

ENV PREFECT_VERSION=0.15.10

And then I think this may need to be in quotation marks (not 100% sure):

pip install "prefect[aws]"

If you need some dependencies from this custom image, you can do a multi-stage build, and in the last part use:

FROM prefecthq/prefect:latest

@Anna Geller Main problem that we have to use Ubuntu instead of Debian

@Vadym Dytyniak I think this contains a good answer https://stackoverflow.com/a/62414670/9509388

hmm, currently don't understand how this concept works. Do you see it is possible to finally have prefect installed on ubuntu?

yes, this should be possible for sure

Do you have idea how to get output of docker run command in ECS?

yes, you could go to CloudWatch logs. You can navigate from your ECS task directly to the logs for this task

The first log I see there it is "Downloading flow from s3..."

it looks good then. Docker run is simply the output of whatever the container is doing. This should be the same what you see in Prefect Cloud UI.

But I am installing dependencies as part of docker run and I don't see these logs

not sure whether installing dependencies generates any specific logs.

We finally found the problem. We used CMD ["entrypoint.sh"] that was replaced with prefect execute flow-run during docker run

thanks for sharing @Vadym Dytyniak. Is everything resolved now?

yes, @Anna Geller and @Kevin Kho thanks for the support!