@Marvin What is the best way to put something similar to tdqm in Prefect? I know we have progress artifact but I am not sure if that's the right place to use it

✅ thought for 324.0 seconds

well, similar to tdqm how? do you want a progress indicator in the console or the UI?

Honestly both works, I just want something as simple as tdqm where I can just

tqdm(Iterable)

but maintain the observability. I know tdqm and similar thing that relies on carriage return sometimes flood the logging space so I want to be safe before I throw in tqdm in my flow/tasks

Great question. There are a few good “tqdm-like” options in Prefect 3.x depending on where you want the progress to appear: - If you want a progress bar in the Prefect UI: use a Progress Artifact. This is the closest analogue to tqdm in Prefect. - If you want terminal output in worker logs: you can still use

tqdm

or

rich

, but most workers won’t render a live bar; you’ll see many log lines instead. For logs, a periodic

<http://logger.info|logger.info>("Processed X/Y")

is usually cleaner. - If you want structural progress: break the work into mapped tasks/subflows; the UI will show “X of Y completed”, but this isn’t a single bar. Using a Progress Artifact (recommended for UI) - Progress is a float from 0 to 100. - Create once, then update by ID during your loop. - Provide a

key

if you want to track the same artifact name/history across runs; otherwise it’s scoped to the run. Example (sync flow):

from prefect import flow, get_run_logger
from prefect.artifacts import create_progress_artifact, update_progress_artifact

@flow
def process_items(items: list[int]):
    logger = get_run_logger()
    total = len(items)
    artifact_id = create_progress_artifact(
        progress=0.0,
        key="process-items-progress",            # optional but helpful for UI/history
        description=f"Starting (0/{total})"
    )

    # throttle updates to avoid spamming the API/logs
    last_pct = 0
    for i, item in enumerate(items, start=1):
        # ... do work ...
        pct = round(100 * i / total, 1)
        if pct - last_pct >= 1 or i == total:    # update every 1% or on completion
            update_progress_artifact(
                artifact_id=artifact_id,
                progress=pct,
                description=f"Processed {i}/{total}"
            )
            last_pct = pct

    <http://logger.info|logger.info>("Done!")

Async flows can use

acreate_progress_artifact

/

aupdate_progress_artifact

similarly. Notes and best practices - Don’t update on every iteration for large loops; update on a cadence (e.g., every 1% or every N seconds). This keeps logs clean and avoids hitting API rate limits. See API rate limits. - Without a

key

, the artifact still shows on the flow run’s Artifacts tab. A

key

makes it easier to find/compare across runs. - Prefer artifacts for human-friendly, UI-visible progress; use logs for terminal-friendly updates. Docs - Concepts: Artifacts in Prefect 3 - How-to: Create and use artifacts Quick check: are you aiming for progress in the Prefect UI, or in the worker’s console logs? I can tailor the example accordingly.