e

    Eddie Atkinson

    5 months ago
    Does anyone know if it’s possible to find out the ECS task ID of a Prefect task from Prefect or the task itself? The context is I am using the ECSAgent to run a task which dumps its logs into a cloudwatch group which is called
    prefect/task/{ECS-task-id}
    . My aim is to display those logs in our webapp to show admin users the status of the flows they’ve started Ideally I’d be able to use the ID of a flow run and use that to query the ECS task it ran on and then use that to query the logs of the flow
    Anna Geller

    Anna Geller

    5 months ago
    My aim is to display those logs in our webapp to show admin users the status of the flows they’ve started
    I'm so glad you mentioned the actual purpose of this, because this way I can show you a much better way of solving this without having to rely on CloudWatch API. Since Prefect already stores all your flow run logs in the backend, you could retrieve those easily in your web API code as follows:
    from prefect.backend.flow_run import FlowRunView
    flow_run = FlowRunView.from_flow_run_id(
        "9549c55f-1afe-45b6-afb3-1b1ed32e7c3f"
    )
    logs = flow_run.get_logs()
    This will give you logs in this format:
    FlowRunLog(timestamp=DateTime(2022, 3, 15, 10, 3, 15, 776868, tzinfo=Timezone('+00:00')), level=20, message='some log msg')
    Kevin Kho

    Kevin Kho

    5 months ago
    You can see more in this page. I agree this is a lot easier to do on the Prefect side than Cloudwatch
    e

    Eddie Atkinson

    5 months ago
    My understanding was that Prefect doesn’t retain logs all that long (7 days or something from memory)? Has that changed? Ideally I want to be able to see logs for my tasks forever
    Kevin Kho

    Kevin Kho

    5 months ago
    Yeah no we still don’t. No change. You’d need to send your logs to DataDog or a service like that
    e

    Eddie Atkinson

    5 months ago
    I’ve just checked a task from October last year and it still has its logs. Is this because I am saving my logs to cloud watch? i.e. is Prefect fetching my logs from there and deleting them internally, or am I just lucky?
    Kevin Kho

    Kevin Kho

    5 months ago
    Are you on Server or Cloud?
    Enterprise level retention is 6 months
    e

    Eddie Atkinson

    5 months ago
    Cloud
    Interesting ok