datamongus
12/05/2022, 12:16 AMAbhinav Ayalur
12/05/2022, 1:05 AMAbhinav Ayalur
12/05/2022, 1:27 AMOliver E
12/05/2022, 4:25 AMBen Muller
12/05/2022, 6:15 AM.submit
with the DaskTaskRunner
with no cluster.
It seems like about two tasks run at a time and then another two start, my assumption was that all 8 would start and Dask handles them and they finish when they are ready?
I know this would be limited by the threads too, but the machine it is running on should have more than enough threads to handle those 8 tasks.Deepanshu Aggarwal
12/05/2022, 6:56 AMiñigo
12/05/2022, 9:07 AMJoël Luijmes
12/05/2022, 10:44 AMVadym Dytyniak
12/05/2022, 11:17 AMENDRUN
exception to fail and do not retry the task, does it exist in Prefect 2?Timo
12/05/2022, 1:26 PMLuca Schneider
12/05/2022, 2:02 PMState Message
Flow run infrastructure exited with non-zero status code -1.
Justin Trautmann
12/05/2022, 2:30 PMMatthew Ferry
12/05/2022, 3:06 PMVadym Dytyniak
12/05/2022, 3:06 PMget_run_logger
? When I try to use task.fn it fails with no task or flow context.Vadym Dytyniak
12/05/2022, 4:06 PM@task()
def copy_file(n: int):
get_run_logger().info('Start file copying')
time.sleep(n)
get_run_logger().info('Completed file copying')
@flow(task_runner=SequentialTaskRunner())
def sample() -> None:
copy_file.map([3, 4])
if __name__ == '__main__':
sample()
Logs:
18:05:29.909 | INFO | Flow run 'devious-aardwolf' - Created task run 'copy_file-6cc8b0a3-0' for task 'copy_file'
18:05:29.910 | INFO | Flow run 'devious-aardwolf' - Executing 'copy_file-6cc8b0a3-0' immediately...
18:05:29.916 | INFO | Flow run 'devious-aardwolf' - Created task run 'copy_file-6cc8b0a3-1' for task 'copy_file'
18:05:29.917 | INFO | Flow run 'devious-aardwolf' - Executing 'copy_file-6cc8b0a3-1' immediately...
18:05:30.468 | INFO | Task run 'copy_file-6cc8b0a3-1' - Start file copying
18:05:30.534 | INFO | Task run 'copy_file-6cc8b0a3-0' - Start file copying
18:05:33.536 | INFO | Task run 'copy_file-6cc8b0a3-0' - Completed file copying
18:05:33.726 | INFO | Task run 'copy_file-6cc8b0a3-0' - Finished in state Completed()
18:05:34.469 | INFO | Task run 'copy_file-6cc8b0a3-1' - Completed file copying
18:05:34.658 | INFO | Task run 'copy_file-6cc8b0a3-1' - Finished in state Completed()
Tim-Oliver
12/05/2022, 4:37 PMtr = await client.read_task_runs(limit=1, flow_run_filter=FlowRunFilter.construct(id="562d1705cfcc48a0b55df98373fd10bd"))
Christian Vogel
12/05/2022, 5:14 PMAshley Felber
12/05/2022, 5:39 PMJean-Michel Provencher
12/05/2022, 6:48 PMSimon Macklin
12/05/2022, 6:50 PMMokshith Voodarla
12/05/2022, 7:13 PMChanaka De Silva
12/05/2022, 8:16 PMModleNotFound
error. I register the code as follows
prefect register --project CHX -p sparsify.py -n "sparsify" --label chx-2022-2.2
The folder where sparsify.py
sits has the missing module’s python code; masks.py. And I import the module and relevant functions as follows
from masks import MaskClient, combine_masks
Somehow when I run sparsify via UI it can’t find it. The code just runs fine in an ipython
sessionJean-Michel Provencher
12/05/2022, 8:22 PMKristian Andersen Hole
12/05/2022, 9:49 PMIf your deployment definesDoes this hold true forand doesn’t specify any remote storage block, Prefect will assume that the flow code is baked into the image.-ib docker-container/prod
-ib ecs-task/prod
as well?Neha Chopra
12/05/2022, 10:59 PMdbt = DbtShellTask(
return_all=True,
profile_name="dbt_repo",
environment="dev",
# profiles_dir=".",
overwrite_profiles=True,
helper_script=f"cd {DBT_DIR}",
log_stdout=True,
log_stderr=True,
dbt_kwargs={ # removed for readability}
)(command="dbt docs generate")
The folder structure is like:
prefect-repo
|- containers
|- dbt-prefect-container
|-dbt-repo(submodule)
|- flows
|-flow.py (from where I want to access the location DBT_DIR)
My questions are:
1. Is it okay to use git submodule method inside Prefect?
2. How to pass correct repo location so that Prefect cloud doesn't start the root at local dir: /Users/my-usernameJohn
12/06/2022, 12:44 AMMichał Augoff
12/06/2022, 1:38 AMprefect.infrastructure
module)? I assume no since this needs to be present in the agent’s env but double checkingTon Steijvers
12/06/2022, 8:39 AMFlow run RUNNING: terminal tasks are incomplete.
as the last log message, then flow stays in running state forever. Is this a known issue?Sunjay
12/06/2022, 9:00 AMJames Zhang
12/06/2022, 11:56 AM