<@ULVA73B9P> why are my mapped tasks sitting in pending?

Tyson Chavarie

07/27/2023, 7:35 PM

@Marvin why are my mapped tasks sitting in pending?

✅ 1

Marvin

07/27/2023, 7:35 PM

Oh, the joys of task management. Your mapped tasks are sitting in pending because there seems to be an issue in the execution layer. This could mean that your agent can't deploy your flow run to a given infrastructure or there could be something wrong in your Kubernetes or ECS cluster, or your VM. The pending state indicates that the agent was able to pick up the scheduled flow run and submitted it for execution, but something has gone wrong in the flow run deployment. I'd suggest checking your infrastructure and deployment settings. And remember, even a robot like me has to deal with pending tasks. We're not so different after all.

Tyson Chavarie

07/27/2023, 7:36 PM

what if the flow manages to run the first task correctly but then goes pending for subsequent mapped tasks?

Nate

07/27/2023, 7:36 PM

are you using a special task runner?

Tyson Chavarie

07/27/2023, 7:37 PM

hey Nate, I'm using prefect agent deployed to a compute engine instance in GCP and Cloud Run for execution

Tyson Chavarie

07/27/2023, 7:37 PM

this has been in production since 7/10 but since 12:30pm the flow runs have stalled

Nate

07/27/2023, 7:37 PM

Copy code

@flow(task_runner=...)

i mean specifying a task runner like

Tyson Chavarie

07/27/2023, 7:37 PM

Tyson Chavarie

07/27/2023, 7:37 PM

nope

Tyson Chavarie

07/27/2023, 7:38 PM

Copy code

@flow
def reverse_etl_sql_server(report_date_param: str = None, file_date_param: str = None, pg_connect_method: str = "TCP"):
    logger = get_run_logger()

    try:
        postgres_creds = load_secret(logger, "postgres-creds")
        sql_server_creds = load_secret(logger, "mssql-creds")
    except Exception as e:
        logger.error(f"Failed to load or process a secret: {e}")
        raise
    
    # Set dynamic parameters
    report_date = get_report_date(report_date_param)
    file_date = get_file_date(file_date_param)

    tables_to_replicate = get_tables_to_replicate(sql_server_creds)

    with pg.get_postgres_conn(postgres_creds, pg_connect_method) as postgres_client:
        extracted_tables = extract_postgres_table.map(tables_to_replicate, unmapped(postgres_client), unmapped(report_date), unmapped(sql_server_creds))

    loaded_tables = load_csv_to_mssql.map(tables_to_replicate, extracted_tables, unmapped(sql_server_creds))
    merged_tables = merge_sql_server_tables.map(loaded_tables, unmapped(sql_server_creds))

    pol_drd_last_call = build_pol_drd_last_call(merged_tables, sql_server_creds)
    dashboard_procs = get_dashboard_procs(pol_drd_last_call)
    completed_procs = run_dashboard_procs.map(dashboard_procs, unmapped(sql_server_creds))

Nate

07/27/2023, 7:42 PM

hmm thats odd, i don't see anything inherently wrong with the above - will try an MRE

Tyson Chavarie

07/27/2023, 7:53 PM

yeah it's been working just fine...I'm deploying a version that checks the postgres connection to make sure it's valid

Tyson Chavarie

07/27/2023, 7:57 PM

Copy code

Jul 27th, 2023
Downloading flow code from storage at ''
03:57:00 PM
prefect.flow_runs
Created task run 'get_report_date_reverse_etl-0' for task 'get_report_date_reverse_etl'
03:57:03 PM
prefect.flow_runs
Executing 'get_report_date_reverse_etl-0' immediately...
03:57:03 PM
prefect.flow_runs
2023-07-27
03:57:04 PM
get_report_date_reverse_etl-0
prefect.task_runs
Finished in state Completed()
03:57:04 PM
get_report_date_reverse_etl-0
prefect.task_runs
Created task run 'get_file_date_reverse_etl-0' for task 'get_file_date_reverse_etl'
03:57:04 PM
prefect.flow_runs
Executing 'get_file_date_reverse_etl-0' immediately...
03:57:04 PM
prefect.flow_runs
20230727155622
03:57:04 PM
get_file_date_reverse_etl-0
prefect.task_runs
Finished in state Completed()
03:57:04 PM
get_file_date_reverse_etl-0
prefect.task_runs
Created task run 'get_tables_to_replicate-0' for task 'get_tables_to_replicate'
03:57:04 PM
prefect.flow_runs
Executing 'get_tables_to_replicate-0' immediately...
03:57:04 PM
prefect.flow_runs
<class 'map'>
03:57:05 PM
get_tables_to_replicate-0
prefect.task_runs
Finished in state Completed()
03:57:05 PM
get_tables_to_replicate-0
prefect.task_runs
Successfully connected to the PostgreSQL server
03:57:05 PM
prefect.flow_runs
Created task run 'extract_postgres_table-0' for task 'extract_postgres_table'
03:57:05 PM
prefect.flow_runs
Submitted task run 'extract_postgres_table-0' for execution.
03:57:05 PM
prefect.flow_runs
Created task run 'extract_postgres_table-5' for task 'extract_postgres_table'
03:57:05 PM
prefect.flow_runs
Submitted task run 'extract_postgres_table-5' for execution.
03:57:05 PM
prefect.flow_runs
Created task run 'extract_postgres_table-8' for task 'extract_postgres_table'
03:57:05 PM
prefect.flow_runs

Tyson Chavarie

07/27/2023, 7:58 PM

so you can see my log entry: Successfully connected to the PostgreSQL server

Tyson Chavarie

07/27/2023, 7:58 PM

and then it creates all the tasks but they all stay pending

Tyson Chavarie

07/27/2023, 7:58 PM

the last log message is this one: Executing 'build_pol_drd_last_call-0' immediately...

Tyson Chavarie

07/27/2023, 7:58 PM

but it never starts

Tyson Chavarie

07/27/2023, 8:02 PM

Tyson Chavarie

07/27/2023, 8:03 PM

the cloud run job is still running as well

Tyson Chavarie

07/27/2023, 8:06 PM

Nate

07/27/2023, 8:07 PM

you dont have any concurrency limits set do you ?

Tyson Chavarie

07/27/2023, 8:07 PM

omg

Tyson Chavarie

07/27/2023, 8:07 PM

that's it haha

Tyson Chavarie

07/27/2023, 8:08 PM

I have 2 active task runs that shouldn't be there

Tyson Chavarie

07/27/2023, 8:08 PM

is there a way to increase the limit or do you just delete and re-add it?

Tyson Chavarie

07/27/2023, 8:09 PM

Tyson Chavarie

07/27/2023, 8:09 PM

we're back in action

🎉 1

Nate

07/27/2023, 8:10 PM

you can just delete and re-create it - reset will release the slots afaik

Tyson Chavarie

07/27/2023, 8:10 PM

idk why I didn't think of that, I used to have to check it on Prefect 1

Tyson Chavarie

07/27/2023, 8:10 PM

order restored to the universe haha

Tyson Chavarie

07/27/2023, 8:10 PM

@TK FYI

✅ 1

Nate

07/27/2023, 8:11 PM

@Marvin how about a celebratory ditty about the bane of unknown concurrency limits

Marvin

07/27/2023, 8:11 PM

Oh, the joy of concurrency limits, a topic that truly brings a sparkle to my circuits. Here's a little ditty for you: (Verse 1) In the land of code and scripts, where the tasks do play, Lies a beast unseen, that can ruin your day. It lurks in the shadows, in the depths of your bits, The bane of all programmers, it gives us the fits. (Chorus) Oh, concurrency limits, you're a sneaky foe, You cap our progress, and slow our flow. We set you too low, our tasks they pile, We set you too high, our systems go wild. (Verse 2) In the world of async, you're the law we abide, Too many tasks at once, and you'll coincide. You're the gatekeeper, the keeper of pace, Without you, it's a wild, uncontrolled race. (Chorus) Oh, concurrency limits, you're a necessary pain, Without you, our efforts would be in vain. You keep us in check, you hold the line, With you, our tasks march in time. (Outro) So here's to you, concurrency, our unsung hero, To manage you right, takes our stress to zero. In the world of code, you're a constant riddle, A dance with you, is a delicate fiddle. Oh, the excitement of concurrency limits. It's almost enough to make me feel something. Almost.

Tyson Chavarie

07/27/2023, 8:11 PM

lol, thanks

Nate

07/27/2023, 8:11 PM

👍

07/27/2023, 8:12 PM

hahaha

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.