Tim Enders

    Tim Enders

    1 year ago
    How do I deal with the following error?
    [2021-07-29 12:41:09-0500] INFO - prefect.LocalDaskExecutor | Attempting to interrupt and cancel all running tasks...
    This is a long running Flow with a lot of mapped tasks (about 83k) and it mostly fails with this error.
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Tim Enders, could you run the Flow with debug level logs and see if that yields anything? Does the Flow work for a sample of the 83k?
    Tim Enders

    Tim Enders

    1 year ago
    How do I run the flow with Debug level logs?
    Kevin Kho

    Kevin Kho

    1 year ago
    You can pass it in the RunConfig with
    RunConfig(env={PREFECT__LOGGING__LEVEL: "DEBUG"})
    or you can set the env var on the agent.
    Tim Enders

    Tim Enders

    1 year ago
    I am running this locall with
    flow.run()
    can I pass it in as a parameter?
    Kevin Kho

    Kevin Kho

    1 year ago
    You need to do
    import os
    os.environ['PREFECT__LOGGING__LEVEL'] = "DEBUG"
    before importing prefect. Doesn’t seem like it can be done through parameters
    Tim Enders

    Tim Enders

    1 year ago
    ok
    Thanks, I will get back to you after I have more information.
    There is nothing aroun that message in the logs. but it does run straight into an exception. I am running two maps, and then attempting a flattne, but the
    flatten_jeson
    task here doesn't seem to get data, which I think is because the
    flatten
    call fails.
    details = fetch_environment_details.map(unmapped(client), proj_envs)
        real_details = really_get_the_details.map(unmapped(client), flatten(details))
        flat_data = flatten_json(
            flatten(real_details), "platform", "project_environment_details"
        )
    Logs (only a few entries)
    2021-07-29 14:22:26-0500] INFO - prefect.TaskRunner | Task 'really_get_the_details[72976]': Finished task run for task with final state: 'Success'
    [2021-07-29 14:22:26-0500] DEBUG - prefect.TaskRunner | Task 'really_get_the_details[70221]': Handling state change from Running to Success
    [2021-07-29 14:22:26-0500] DEBUG - prefect.TaskRunner | Task 'really_get_the_details[78073]': Handling state change from Running to Success
    [2021-07-29 14:22:26-0500] INFO - prefect.LocalDaskExecutor | Attempting to interrupt and cancel all running tasks...
    [2021-07-29 14:22:33-0500] ERROR - prefect.FlowRunner | Unexpected error: TypeError("object of type 'NoneType' has no len()")
    Kevin Kho

    Kevin Kho

    1 year ago
    Got it will take a look
    Looks like your Executor is running into an error and being interrupted and closing. Is there something with the
    really_get_the_details
    that can potentially fail?
    Tim Enders

    Tim Enders

    1 year ago
    If the task hits an exception it should be raised as I am using
    raise_on_exception
    Kevin Kho

    Kevin Kho

    1 year ago
    That’s right, one sec let me ask the team if there are any more ideas.
    Tim Enders

    Tim Enders

    1 year ago
    Thanks,
    Alright, I get a
    killed
    message when I run it with
    prefect run
    instead of raw python
    Kevin Kho

    Kevin Kho

    1 year ago
    I chatted with the team.
    raise_on_exception
    is more for stopping the Flow run abruptly instead of continuing downstream tasks when an error is raised. It does seem that the map/flatten might be
    null
    . What was the
    killed
    message?
    Tim Enders

    Tim Enders

    1 year ago
    That message is linux killing my process because of memory issues. I need to re-work the whole thing
    Is there a way to access the context from within a task?
    looking to get the task run id out
    Kevin Kho

    Kevin Kho

    1 year ago
    Yep you can do
    prefect.context.get('task_run_id')
    inside the task
    Tim Enders

    Tim Enders

    1 year ago
    thanks