👋 everyone! QQ. I am getting ready to build a little deployment that gsutil rsync's a bunch of data from s3 to gcs. Unfortunately the data in s3 doesn't really follow a nice partition hierarchy so i can't just

gsutil rsync <s3://some-key/dt=yyyy-mm-dd/>

<gs://some-key/dt=yyyy-mm-dd/>

😢 I am going to give

prefect-shell

a try for the first time and want to know if people have had a lot of experience with it? For my use case I have about 12K different rsync's I am going to need to run and I don't know which of these patterns is preferable:

for src in s3_sources_12k:
    dest = f'<gs://some-dest/{src}>'
    ShellOperation(
        commands=[f"gsutil rsync -r {src} {dest}"],
        env=env_var_map,
    ).run()

or

with ShellOperation(
    commands=[
        "gsutil rsync -r src1 dest1",
        "gsutil rsync -r src2 dest2",
        "gsutil rsync -r src3 dest3",
        ...
        "gsutil rsync -r src12k dest12k",
    ],
    env=env_var_map,
) as shell_operation:
    shell_process = shell_operation.trigger()
    shell_process.wait_for_completion()
    shell_output = shell_process.fetch_result()

Thanks for bringing this up! I think the first one with each op wrapped in a task would work.

@task
def sync_src(src):
    dest = f'<gs://some-dest/{src}>'
    ShellOperation(
        commands=[f"gsutil rsync -r {src} {dest}"],
        env=env_var_map,
    ).run()
    return

for src in s3_sources_12k:
    sync_src(src)

Thanks for the suggestion @Andrew Huang. In my original snippet I didn't share that I already have the entire for loop already inside of a single task call. You think creating ~12k tasks would be better than having 1 task that iterates through the list?

Depends on whether you want visibility / retries on the tasks and how long each sync takes. If it takes milliseconds, then using tasks probably will incur a large overhead

good point 🤔