Sorry for the late reply! We have a Prefect task named

wit-labeling

which accepts a

list of label-name

as input, also we have a python function consists of three steps(call boto3 S3, Athena function) for a single labeling logic. In

wit-labeling

, threading is used for speedup purpose, meanwhile we want to know the details for each of labeling step, including for debug purpose when there is any following job doesn’t work properly. Now we are in development state and we found that we rely heavily on the log. 🥺

then in run_extract_process I'm just trying to capture the input variable, so I know which one is which.

<http://logger.info|logger.info>(f"Starting extract for {v}")

and again, ThreadPool is from multiprocessing.dummy

from multiprocessing.dummy import Pool as ThreadPool

the missing logs don't seem to make it past this line in logging.py

# if its not during a backend flow run, don't emit

        

if not context.get("running_with_backend"):

            

return

Sure enough, if I change my code to add a line changing the context, it shows up in the UI.

with prefect.context({"running_with_backend": True}):

<http://logger.info|logger.info>(f"Starting extract for {v}")

I suspect you could pass

prefect.context

to

run_extract_process

as an arg and then re-initialize in

run_extract_proces

def run_extract_process(record, prefect_context):
    with prefect.context(prefect_context):
        # ... everything else