Channels
#ask-marvin
Title
t
Tyson Chavarie
07/27/2023, 7:35 PM@Marvin why are my mapped tasks sitting in pending?
✅ 1
m
Marvin
07/27/2023, 7:35 PMOh, the joys of task management. Your mapped tasks are sitting in pending because there seems to be an issue in the execution layer. This could mean that your agent can't deploy your flow run to a given infrastructure or there could be something wrong in your Kubernetes or ECS cluster, or your VM. The pending state indicates that the agent was able to pick up the scheduled flow run and submitted it for execution, but something has gone wrong in the flow run deployment.
I'd suggest checking your infrastructure and deployment settings. And remember, even a robot like me has to deal with pending tasks. We're not so different after all.
t
Tyson Chavarie
07/27/2023, 7:36 PMwhat if the flow manages to run the first task correctly but then goes pending for subsequent mapped tasks?
n
Nate
07/27/2023, 7:36 PMare you using a special task runner?
t
Tyson Chavarie
07/27/2023, 7:37 PMhey Nate, I'm using prefect agent deployed to a compute engine instance in GCP and Cloud Run for execution
this has been in production since 7/10 but since 12:30pm the flow runs have stalled
n
Nate
07/27/2023, 7:37 PM Copy code
@flow(task_runner=...)t
Tyson Chavarie
07/27/2023, 7:37 PMimage (1).png
nope
Copy code
@flow
def reverse_etl_sql_server(report_date_param: str = None, file_date_param: str = None, pg_connect_method: str = "TCP"):
logger = get_run_logger()
try:
postgres_creds = load_secret(logger, "postgres-creds")
sql_server_creds = load_secret(logger, "mssql-creds")
except Exception as e:
logger.error(f"Failed to load or process a secret: {e}")
raise
# Set dynamic parameters
report_date = get_report_date(report_date_param)
file_date = get_file_date(file_date_param)
tables_to_replicate = get_tables_to_replicate(sql_server_creds)
with pg.get_postgres_conn(postgres_creds, pg_connect_method) as postgres_client:
extracted_tables = extract_postgres_table.map(tables_to_replicate, unmapped(postgres_client), unmapped(report_date), unmapped(sql_server_creds))
loaded_tables = load_csv_to_mssql.map(tables_to_replicate, extracted_tables, unmapped(sql_server_creds))
merged_tables = merge_sql_server_tables.map(loaded_tables, unmapped(sql_server_creds))
pol_drd_last_call = build_pol_drd_last_call(merged_tables, sql_server_creds)
dashboard_procs = get_dashboard_procs(pol_drd_last_call)
completed_procs = run_dashboard_procs.map(dashboard_procs, unmapped(sql_server_creds))n
Nate
07/27/2023, 7:42 PMhmm thats odd, i don't see anything inherently wrong with the above - will try an MRE
t
Tyson Chavarie
07/27/2023, 7:53 PMyeah it's been working just fine...I'm deploying a version that checks the postgres connection to make sure it's valid
Copy code
Jul 27th, 2023
Downloading flow code from storage at ''
03:57:00 PM
prefect.flow_runs
Created task run 'get_report_date_reverse_etl-0' for task 'get_report_date_reverse_etl'
03:57:03 PM
prefect.flow_runs
Executing 'get_report_date_reverse_etl-0' immediately...
03:57:03 PM
prefect.flow_runs
2023-07-27
03:57:04 PM
get_report_date_reverse_etl-0
prefect.task_runs
Finished in state Completed()
03:57:04 PM
get_report_date_reverse_etl-0
prefect.task_runs
Created task run 'get_file_date_reverse_etl-0' for task 'get_file_date_reverse_etl'
03:57:04 PM
prefect.flow_runs
Executing 'get_file_date_reverse_etl-0' immediately...
03:57:04 PM
prefect.flow_runs
20230727155622
03:57:04 PM
get_file_date_reverse_etl-0
prefect.task_runs
Finished in state Completed()
03:57:04 PM
get_file_date_reverse_etl-0
prefect.task_runs
Created task run 'get_tables_to_replicate-0' for task 'get_tables_to_replicate'
03:57:04 PM
prefect.flow_runs
Executing 'get_tables_to_replicate-0' immediately...
03:57:04 PM
prefect.flow_runs
<class 'map'>
03:57:05 PM
get_tables_to_replicate-0
prefect.task_runs
Finished in state Completed()
03:57:05 PM
get_tables_to_replicate-0
prefect.task_runs
Successfully connected to the PostgreSQL server
03:57:05 PM
prefect.flow_runs
Created task run 'extract_postgres_table-0' for task 'extract_postgres_table'
03:57:05 PM
prefect.flow_runs
Submitted task run 'extract_postgres_table-0' for execution.
03:57:05 PM
prefect.flow_runs
Created task run 'extract_postgres_table-5' for task 'extract_postgres_table'
03:57:05 PM
prefect.flow_runs
Submitted task run 'extract_postgres_table-5' for execution.
03:57:05 PM
prefect.flow_runs
Created task run 'extract_postgres_table-8' for task 'extract_postgres_table'
03:57:05 PM
prefect.flow_runsso you can see my log entry: Successfully connected to the PostgreSQL server
and then it creates all the tasks but they all stay pending
the last log message is this one: Executing 'build_pol_drd_last_call-0' immediately...
but it never starts
image.png
the cloud run job is still running as well
image.png
n
Nate
07/27/2023, 8:07 PMyou dont have any concurrency limits set do you ?
t
Tyson Chavarie
07/27/2023, 8:07 PMomg
that's it haha
I have 2 active task runs that shouldn't be there
is there a way to increase the limit or do you just delete and re-add it?
image.png
we're back in action
🎉 1
n
Nate
07/27/2023, 8:10 PMyou can just delete and re-create it - reset will release the slots afaik
t
Tyson Chavarie
07/27/2023, 8:10 PMidk why I didn't think of that, I used to have to check it on Prefect 1
order restored to the universe haha
@TK FYI
✅ 1
n
Nate
07/27/2023, 8:11 PM@Marvin how about a celebratory ditty about the bane of unknown concurrency limits
m
Marvin
07/27/2023, 8:11 PMOh, the joy of concurrency limits, a topic that truly brings a sparkle to my circuits. Here's a little ditty for you:
(Verse 1)
In the land of code and scripts, where the tasks do play,
Lies a beast unseen, that can ruin your day.
It lurks in the shadows, in the depths of your bits,
The bane of all programmers, it gives us the fits.
(Chorus)
Oh, concurrency limits, you're a sneaky foe,
You cap our progress, and slow our flow.
We set you too low, our tasks they pile,
We set you too high, our systems go wild.
(Verse 2)
In the world of async, you're the law we abide,
Too many tasks at once, and you'll coincide.
You're the gatekeeper, the keeper of pace,
Without you, it's a wild, uncontrolled race.
(Chorus)
Oh, concurrency limits, you're a necessary pain,
Without you, our efforts would be in vain.
You keep us in check, you hold the line,
With you, our tasks march in time.
(Outro)
So here's to you, concurrency, our unsung hero,
To manage you right, takes our stress to zero.
In the world of code, you're a constant riddle,
A dance with you, is a delicate fiddle.
Oh, the excitement of concurrency limits. It's almost enough to make me feel something. Almost.
t
Tyson Chavarie
07/27/2023, 8:11 PMlol, thanks
n
Nate
07/27/2023, 8:11 PM👍
t
TK
07/27/2023, 8:12 PMhahaha