Hi all, I'm running prefect jobs on an ECS task block and I am trying to set up a datadog agent in the same task definition. However, when I run the job, the logs are not picked up by datadog. Has anyone succesfully integrated Prefect with datadog? Specifically while running on ECS?

Last time I tried it (which, granted, was a while ago), I had to add filesystem logging to Prefect's logging config and then use the Datadog host agent to pick up it up. I tried it on ECS, Azure Container Instances, as well as locally on Linux, Windows, and macOS, and it worked as expected on all of them. I wrote up the whole process here, and provided some sample logging config files: https://github.com/rpeden/prefect-datadog Although I wrote it a while ago, I just updated it to address a couple of changes in Prefect since then. If you try it and you run into any issues feel free to reply here or on GitHub!

Thanks @Ryan Peden - I guess for further clarification, we would like to set up datadog to trace our flow runs in prefect to provide insights into SQL queries that are throttling our database at times. These instructions seem to be for log ingestion, will they also provide us with trace information?

The Python plugin near the end of the write-up should give you Python traces. You'll also need to install

ddtrace

and override the run command the worker sends to the container. I think the default is

python -m prefect.engine

, so you'd need to change it to

ddtrace-run python -m prefect.engine

You can also run the Datadog agent in its own container, as long as it's running somewhere that your flow container can access it over the network. In that case you'll need to set the DD_AGENT_HOST environment variable in the flow container to tell it where to send the traces. I found the Datadog agent quick enough to start up that it was easier to just run it in the flow container, but if you have very short flows or a large volume of flows, a separate agent is probably the best choice.

I was able to get it to work by running patch_all at the beginning of the flows and adding the datadog agent to my ecs task definition. I didn't have to do the python init file or wrapping the command in ddtrace.