Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

How do I deal with the following error? `[2021-07-29 12:41:09-0500] INFO - prefect.LocalDaskExecutor | Attempting to interrupt and cancel all running tasks...`

This is a long running Flow with a lot of mapped tasks (about 83k) and it mostly fails with this error.

Hey <@U01HV134CKG>, could you run the Flow with debug level logs and see if that yields anything? Does the Flow work for a sample of the 83k?

How do I run the flow with Debug level logs?

You can pass it in the RunConfig with `RunConfig(env={PREFECT__LOGGING__LEVEL: "DEBUG"})` or you can set the env var on the agent.

I am running this locall with `flow.run()`

You need to do
```import os
os.environ['PREFECT__LOGGING__LEVEL'] = "DEBUG"```
before importing prefect. Doesn’t seem like it can be done through parameters

Thanks, I will get back to you after I have more information.

There is nothing aroun that message in the logs. but it does run straight into an exception. I am running two maps, and then attempting a flattne, but  the `flatten_jeson` task here doesn't seem to get data, which I think is because the `flatten` call fails.
``` details = fetch_environment_details.map(unmapped(client), proj_envs)
    real_details = really_get_the_details.map(unmapped(client), flatten(details))
    flat_data = flatten_json(
        flatten(real_details), "platform", "project_environment_details"
    )```


Logs (only a few entries)
```2021-07-29 14:22:26-0500] INFO - prefect.TaskRunner | Task 'really_get_the_details[72976]': Finished task run for task with final state: 'Success'
[2021-07-29 14:22:26-0500] DEBUG - prefect.TaskRunner | Task 'really_get_the_details[70221]': Handling state change from Running to Success
[2021-07-29 14:22:26-0500] DEBUG - prefect.TaskRunner | Task 'really_get_the_details[78073]': Handling state change from Running to Success
[2021-07-29 14:22:26-0500] INFO - prefect.LocalDaskExecutor | Attempting to interrupt and cancel all running tasks...
[2021-07-29 14:22:33-0500] ERROR - prefect.FlowRunner | Unexpected error: TypeError("object of type 'NoneType' has no len()")```


Looks like your Executor is running into an error and being interrupted and closing. Is there something with the `really_get_the_details` that can potentially fail?

If the task hits an exception it should be raised as I am using `raise_on_exception`

That’s right, one sec let me ask the team if there are any more ideas.

Alright, I get a `killed` message when I run it with `prefect run` instead of raw python

I chatted with the team. `raise_on_exception` is more for stopping the Flow run abruptly instead of continuing downstream tasks when an error is raised. It does seem that the map/flatten might be `null` . What was the `killed` message?

That message is linux killing my process because of memory issues. I need to re-work the whole thing

Is there a way to access the context from within a task?

Yep you can do `prefect.context.get('task_run_id')` inside the task