🤔 EcsTaskRunner() 🤔 I just wanna share some idea that came up while talking with @Will Raphaelson Sharing to get more ideas and angles on this. So currently we have: • RayTaskRunner() • DaskTaskRunner() What about an EcsTaskRunner()? Ray and Dask TaskRunners allows me to .submit() a task to some “compute pool”, there the task will be attached to some one-off instance with dedicated memory and cpu. Isn’t this the exact same story when looking at AWS’s ECS? Ray and Dask are quite “exotic” technologies and often come up when you start talking about highly parallel machine learning/AI workloads. But what about a more common use case: A weekly job that needs to process many files in parallel where each file is significantly different in size:

@flow(task_runner=EcsTaskRunner())
def my_weekly_huge_files_job(names):
    files_on_s3 = get_list_of_huge_files()
    for path,file_size in files_on_s3:
        if file_size > xxx:
            process_file.submit(path, memory_size='4gb')
        else
            process_file.submit(path, memory_size='1gb')

The code above would be super robust since a certain huge file might crash, but it won’t affect the other instances. I believe such EcsTaskRunner() would be adopted by the community very quickly as many already use ECS for their ECS push work pools. This opens up many possibilities for distributed compute on remote, separated, machines, without intruding big guns like Ray or Dask. As a side note, maybe Dask and Ray are already much more accessible these days? I could swipe a credit card and use Coiled.io (managed Dask cluster) to get a very similar experience to what I’ve described above in the code. But still ECS would be much more common and affordable. Am I missing something here?

Hey thanks yaron - this will definitely be under consideration as we move toward 3.0 GA. I think this is probably deserving of a github issue where we can all discuss implementation. would you open one at https://github.com/PrefectHQ/prefect

@Will Raphaelson Sure, I’ll open a ticket.