ey !!! what do you think about that? it’s a crazy idea? imposible? … I am a bit lost

I think you’re a bit too fast 🙂 it would be much easier if you approach it step by step. 1. First, you can focus on building your basic functionality in a flow, 2. Then learn how to register it and use it with your chosen storage and agent. 3. Then maybe look at CI/CD with Github actions (we have just published this post showing an example CI/CD with CircleCI). Once all that is done you can look at how to streamline the process by modularizing some components. I think this would make troubleshooting and everything else much easier. Once that’s done, perhaps move on to the next step and adding Dask to parallelize some work.

Thanks so much !!! I already test usen just one file one flow with more than one task and its working well to now the idea is make a flow like a watcher of sftp server to get the files, categorized (because we have 3 different use cases) commit them to gihub to start actions … each task to process that 3 different files are really large by number of lines, so we need only exec the task that process one of them after that process will have to match a field (street address) to another task that will ask to an API the GPS coordinates to add that info to the DS finally that DS will be process with eland to store on a elasticsearch cluster I am really convinced use prefect and dask will be the perfect fit to our use case …. I need to learn more and to understand because, a part of the support I received by you and this channel … I can’t understand the documentation and I fell maybe will be better, Idk Sorry to this and thanks so much for your help I appreciated

I understand. You should definitely start by building your entire use case in Python and once that’s working you can think about how you can break it into Prefect tasks and organize into a flow

hummm I have already the code in python

if you already have code for that, then all you need to do is to package it into e.g. functions and call those functions within your tasks and build a DAG. You can import a function in a Prefect flow as with any other Python script, e.g.:

from prefect import task, Flow
from yourcustommodule import extract_logic, transform_logic, load_logic


@task
def get_list_of_files_from_sftp():
    extract_logic()


@task
def transform_and_load(file):
    transform_logic(file)
    load_logic(file)


with Flow("etl") as flow:
    list_of_files = get_list_of_files_from_sftp()
    transform_and_load.map(list_of_files)

Does it help? LMK if you have any specific question I can help with

Yes !!! of course !!! you are helping me a lot and I appreciated sorry if I am so clumsy

and once you want to run it in parallel, you can attach a local dask executor:

from prefect.executors import LocalDaskExecutor

with Flow("etl", executor=LocalDaskExecutor()) as flow:
    list_of_files = get_list_of_files_from_sftp()
    transform_and_load.map(list_of_files)

With LocalDaskExecutor the agent will get all python files ?

yes

wow !!! I dont know that before 🙂

I was just going through earlier messages. I think I answered this in the thread below. But yes you can use the LocalDaskExecutor with local agents. I assume they are different machines. I don’t think there is a use case where it makes sense to run multiple LocalAgents on the same machine