Samuel Hinton
02/16/2021, 6:48 PMPeyton Runyan
02/16/2021, 8:21 PMsql_task = SqlServerFetch(
db_name=prefect.config.sql_server.database,
user=prefect.config.sql_server.user,
host=prefect.config.sql_server.server,
query=get_manual_override_rows,
fetch='many',
fetch_count=3,
result=result_formatter,
name="Database Query"
# commit: bool = False,
)
with Flow("config_flow") as f:
mode = Parameter("Mode", default="dev")
password_var = set_creds(mode)
password_task = EnvVarSecret()
password = password_task(password_var)
s = sql_task(password=password)
v = view_sql(sql_task)
Suchindra
02/16/2021, 9:16 PMLocalDaskExecutor
My flow looks something like this:
list_of_files = Parameter()
list_of_frames = extract_fn.map(list_of_files) # This is indeed getting executed in parallel
How can I pass both the result of the previous mapped fn and also the input parameters to my transform fn? I understand that I could probably refactor the transform_fn to avoid using the list_of_files
but wanted to know if there is any other way of handling this.
transform_fn.map(list_of_frames, list_of_files)
As you can tell I am very new to prefect. Thanks a lot!Peyton Runyan
02/16/2021, 10:58 PMSuchindra
02/16/2021, 11:24 PMlogbook
instead of logging
from stdlib. How can I configure it to capture logs from within prefect tasks?itay livni
02/16/2021, 11:30 PMStartFlowRun
in a task with a loop signal? My concern is that with StartFlowRun
the enviroment (including modules) will be "re-initialized" on every loop. But maybe that will happen with recursion also? -- In short is the effort of trying to build a recursive Flow worth the effort from a speed perspective? (Currently running into different issues). Thanksmatta
02/17/2021, 4:11 AMSven Teresniak
02/17/2021, 9:40 AM--skip-if-exists
because this is the default behaviour. but when you're creating a tenant a 2nd time (same name and slug) you will get an exception and return value 1 from the prefect cli. it seems create-tenant
needs a --skip-if-exists
and create project
notHarshal Rane
02/17/2021, 9:51 AMHarshal Rane
02/17/2021, 9:51 AMFailed to load and execute Flow's environment: ModuleNotFoundError("No module named '/root/'")
Attached are logs from Docker agent
Has anyone faced this issue before?Michael Hadorn
02/17/2021, 10:04 AMJonathan Wright
02/17/2021, 10:51 AMMilly gupta
02/17/2021, 1:34 PMJosselin Girault
02/17/2021, 2:37 PMMatthew Blau
02/17/2021, 2:40 PMPuja Thacker
02/17/2021, 3:29 PMNikul
02/17/2021, 4:35 PMAjith Kumara Beragala Acharige Lal
02/17/2021, 5:57 PMHelm
, the Prefect server also deployed to the kubernetes via Helm, now need to try-out a flow on that Prefect-server, does anyone has example/page?Tobias Heintz
02/17/2021, 8:26 PMFina Silva-Santisteban
02/18/2021, 1:27 AMprefect ecs agent
in a terminal, and then use the prefect dashboard to trigger that run. I’d like to build a standalone app, so have a custom GUI that lets a user input flow parameters and trigger a flow run. I’m imagining that I’ll have to create a prefect ecs agent using the prefect api instead of running it in a separate terminal but I’m not sure how to go about that. Pls let me know of any deployment recipes that fit my use case! 🙏Puja Thacker
02/18/2021, 7:45 AMPuja Thacker
02/18/2021, 7:46 AMJulian
02/18/2021, 10:31 AMcrazy_monkey
, but they don't provide any actual value. Since it is quite nice (from ui perspective) to have meaningful names, we rename these flowruns during runtime like {flow_name}_{system_time}
or {flow_name}_{parameters}
. However, if a flowrun fails before executing the rename task, it won't have this "nice" name but the orignal (meaning-less) name. Also, renaming inside the flow will also overwrite flowrun names which were provided when manually starting a flowrun with name, e.g. from ui.Milly gupta
02/18/2021, 11:47 AMSiva Kumar
02/18/2021, 2:57 PMJoël Luijmes
02/18/2021, 2:59 PMtasks = dynamic_list_of_tasks()
windowed_tasks = fixed_window(tasks, window_size=5)
def process_item(item):
x = task_1()
task_2(x)
def process_window(tasks):
apply_map(process_item, tasks)
apply_map(process_window, windowed_tasks)
Jean-Marc LE FEVRE
02/18/2021, 3:04 PMJean-Marc LE FEVRE
02/18/2021, 3:05 PMKarolína Bzdušek
02/18/2021, 5:43 PMflow.storage = GitHub(repo="cividi/gemeinde_workflows",path="/workflows/residential_density.py",access_token_secret="GITHUB_ACCESS_TOKEN")
"GITHUB_ACCESS_TOKEN"
is stored in secret in Cloud.
However, when I run flow through UI, I am still getting this and not sure why.
Note: repository is private and own by our team, I am using token generated under my account.
What I am missing? Thanks for your help in advance.Craig Wright
02/18/2021, 8:08 PMFivetranSyncTask
. If anyone is even vaguely interested in more deterministic Fivetran scheduling, please reach out! https://prefect-community.slack.com/archives/CL09KTZPX/p1613678832100000Craig Wright
02/18/2021, 8:08 PMFivetranSyncTask
. If anyone is even vaguely interested in more deterministic Fivetran scheduling, please reach out! https://prefect-community.slack.com/archives/CL09KTZPX/p1613678832100000