Hi Prefect Team I m switching our data engineering stack fro Prefect Community #ask-community

Hi Prefect Team, I’m switching our data engineeri...

Robert Banick

04/03/2024, 8:58 PM

Hi Prefect Team, I’m switching our data engineering stack from using agents to push ECS work pools and having a problems getting log settings to apply. Previously we let Prefect route logs to its default “prefect” log group and left other settings on default as well. This had the advantage of passing through a flow’s default coolname as the stream-prefix, which was advantageous for locating runs quickly (vs. randomly assigned numbers and letters). The default setting no longer appears to work and setting the log group and/or stream_prefix manually via the work pool’s

cloudwatch_logs_options

is not working, even w/

stream_output

and

configure_cloudwatch_logs

set to

True

. I am able to break flow runs (silently) by setting the

cloudwatch_log_options

incorrectly but even when ETLs go through the logs don’t appear in the specified log group. Any ideas what could be causing this? Log options settings which allow flows to pass, but don’t push logs to the specific group, are provided below

Copy code

{"awslogs-group":"prefect",  # this log group already exists
"awslogs-region":"us-east-1",
"awslogs-stream-prefix":"awslogs-example"}

Extra credit — I have to pass the above dictionary into the “default” setting under the work pool’s variables. Hard coding these into the job configuration fails silently, even though I want these settings applied any time. Any clarification you can provide on why this is / whether I can hard code would be helpful.

Robert Banick

04/03/2024, 8:59 PM

Note that if I specify

Copy code

"awslogs-create-group": "true"

in the options dictionary then the flow hangs indefinitely, even if the specified log group doesn’t yet exist.

Robert Banick

04/04/2024, 8:55 PM

@Prefect team, digging into the existing

prefect_aws

code I see that the current functionality I want to replicate happens within ecs.py in the

_prepare_task_definition

function on lines 1277-1287, like so

Copy code

if self.configure_cloudwatch_logs:
            container["logConfiguration"] = {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "prefect",
                    "awslogs-region": region,
                    "awslogs-stream-prefix": self.name or "prefect",
                    **self.cloudwatch_logs_options,
                },
            }

This is in turn called within

_create_task_and_wait_for_start

which appears to happens prior to runtime of a flow. Is there a way to recreate this functionality within the new setup of an ECS push work pool? All of these actions are attached to the to-be-deprecated

ECSTask

object. Under the new setup I’m not sure a) Where to sneak in pre-task instructions / tweaks to the log group or B) How to pass the coolname as a stream-prefix (above

self.name

) since the self in question is the defunct ECSTask object.

4 Views

Open in Slack

Previous Next