https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> why are my mapped tasks sitting in pe...
# ask-marvin
t
@Marvin why are my mapped tasks sitting in pending?
✅ 1
m
Oh, the joys of task management. Your mapped tasks are sitting in pending because there seems to be an issue in the execution layer. This could mean that your agent can't deploy your flow run to a given infrastructure or there could be something wrong in your Kubernetes or ECS cluster, or your VM. The pending state indicates that the agent was able to pick up the scheduled flow run and submitted it for execution, but something has gone wrong in the flow run deployment. I'd suggest checking your infrastructure and deployment settings. And remember, even a robot like me has to deal with pending tasks. We're not so different after all.
t
what if the flow manages to run the first task correctly but then goes pending for subsequent mapped tasks?
n
are you using a special task runner?
t
hey Nate, I'm using prefect agent deployed to a compute engine instance in GCP and Cloud Run for execution
this has been in production since 7/10 but since 12:30pm the flow runs have stalled
n
Copy code
@flow(task_runner=...)
i mean specifying a task runner like
t
nope
Copy code
@flow
def reverse_etl_sql_server(report_date_param: str = None, file_date_param: str = None, pg_connect_method: str = "TCP"):
    logger = get_run_logger()

    try:
        postgres_creds = load_secret(logger, "postgres-creds")
        sql_server_creds = load_secret(logger, "mssql-creds")
    except Exception as e:
        logger.error(f"Failed to load or process a secret: {e}")
        raise
    
    # Set dynamic parameters
    report_date = get_report_date(report_date_param)
    file_date = get_file_date(file_date_param)

    tables_to_replicate = get_tables_to_replicate(sql_server_creds)

    with pg.get_postgres_conn(postgres_creds, pg_connect_method) as postgres_client:
        extracted_tables = extract_postgres_table.map(tables_to_replicate, unmapped(postgres_client), unmapped(report_date), unmapped(sql_server_creds))

    loaded_tables = load_csv_to_mssql.map(tables_to_replicate, extracted_tables, unmapped(sql_server_creds))
    merged_tables = merge_sql_server_tables.map(loaded_tables, unmapped(sql_server_creds))

    pol_drd_last_call = build_pol_drd_last_call(merged_tables, sql_server_creds)
    dashboard_procs = get_dashboard_procs(pol_drd_last_call)
    completed_procs = run_dashboard_procs.map(dashboard_procs, unmapped(sql_server_creds))
n
hmm thats odd, i don't see anything inherently wrong with the above - will try an MRE
t
yeah it's been working just fine...I'm deploying a version that checks the postgres connection to make sure it's valid
Copy code
Jul 27th, 2023
Downloading flow code from storage at ''
03:57:00 PM
prefect.flow_runs
Created task run 'get_report_date_reverse_etl-0' for task 'get_report_date_reverse_etl'
03:57:03 PM
prefect.flow_runs
Executing 'get_report_date_reverse_etl-0' immediately...
03:57:03 PM
prefect.flow_runs
2023-07-27
03:57:04 PM
get_report_date_reverse_etl-0
prefect.task_runs
Finished in state Completed()
03:57:04 PM
get_report_date_reverse_etl-0
prefect.task_runs
Created task run 'get_file_date_reverse_etl-0' for task 'get_file_date_reverse_etl'
03:57:04 PM
prefect.flow_runs
Executing 'get_file_date_reverse_etl-0' immediately...
03:57:04 PM
prefect.flow_runs
20230727155622
03:57:04 PM
get_file_date_reverse_etl-0
prefect.task_runs
Finished in state Completed()
03:57:04 PM
get_file_date_reverse_etl-0
prefect.task_runs
Created task run 'get_tables_to_replicate-0' for task 'get_tables_to_replicate'
03:57:04 PM
prefect.flow_runs
Executing 'get_tables_to_replicate-0' immediately...
03:57:04 PM
prefect.flow_runs
<class 'map'>
03:57:05 PM
get_tables_to_replicate-0
prefect.task_runs
Finished in state Completed()
03:57:05 PM
get_tables_to_replicate-0
prefect.task_runs
Successfully connected to the PostgreSQL server
03:57:05 PM
prefect.flow_runs
Created task run 'extract_postgres_table-0' for task 'extract_postgres_table'
03:57:05 PM
prefect.flow_runs
Submitted task run 'extract_postgres_table-0' for execution.
03:57:05 PM
prefect.flow_runs
Created task run 'extract_postgres_table-5' for task 'extract_postgres_table'
03:57:05 PM
prefect.flow_runs
Submitted task run 'extract_postgres_table-5' for execution.
03:57:05 PM
prefect.flow_runs
Created task run 'extract_postgres_table-8' for task 'extract_postgres_table'
03:57:05 PM
prefect.flow_runs
so you can see my log entry: Successfully connected to the PostgreSQL server
and then it creates all the tasks but they all stay pending
the last log message is this one: Executing 'build_pol_drd_last_call-0' immediately...
but it never starts
the cloud run job is still running as well
n
you dont have any concurrency limits set do you ?
t
omg
that's it haha
I have 2 active task runs that shouldn't be there
is there a way to increase the limit or do you just delete and re-add it?
we're back in action
🎉 1
n
you can just delete and re-create it - reset will release the slots afaik
t
idk why I didn't think of that, I used to have to check it on Prefect 1
order restored to the universe haha
@TK FYI
✅ 1
n
@Marvin how about a celebratory ditty about the bane of unknown concurrency limits
m
Oh, the joy of concurrency limits, a topic that truly brings a sparkle to my circuits. Here's a little ditty for you: (Verse 1) In the land of code and scripts, where the tasks do play, Lies a beast unseen, that can ruin your day. It lurks in the shadows, in the depths of your bits, The bane of all programmers, it gives us the fits. (Chorus) Oh, concurrency limits, you're a sneaky foe, You cap our progress, and slow our flow. We set you too low, our tasks they pile, We set you too high, our systems go wild. (Verse 2) In the world of async, you're the law we abide, Too many tasks at once, and you'll coincide. You're the gatekeeper, the keeper of pace, Without you, it's a wild, uncontrolled race. (Chorus) Oh, concurrency limits, you're a necessary pain, Without you, our efforts would be in vain. You keep us in check, you hold the line, With you, our tasks march in time. (Outro) So here's to you, concurrency, our unsung hero, To manage you right, takes our stress to zero. In the world of code, you're a constant riddle, A dance with you, is a delicate fiddle. Oh, the excitement of concurrency limits. It's almost enough to make me feel something. Almost.
t
lol, thanks
n
👍
t
hahaha
2 Views
Open in Slack
PreviousNext
Powered by