Sander
12/02/2022, 7:03 PMMalek Naski
12/02/2022, 7:38 PMdeployments.run_deployment()
. However, this makes it not possible to manually retry a failed subflow, is there a better way to do this? My use case is mostly the parent flow fetching inputs and then each subflow run would handle one of the inputs (I'm looping over the task that triggers the subflow deployment runs)
Thanks!Chris McClellan
12/03/2022, 2:17 AMmerlin
12/03/2022, 6:51 AMTim Galvin
12/03/2022, 8:18 AMPREFECT_LOGGING_EXTRA_LOGGERS
changed from 2.6.7
to 2.6.9
at all? I was previously using it successfully in combination with DaskTaskExecutor
and dask_jobqueue.SLURMCluster
to capture logs from another package. It seems though with an updated to 2.6.8
or 2.6.9
I have lost this ability.
I do see the logs being printed to my slurm output files, and I do see it is formatted in the prefect
style, but I am not seeing these logs being submitted through to my self-hosted Orion server (as presented by the web UI).
Any ideas?Andreas Nigg
12/03/2022, 1:33 PMTim Galvin
12/04/2022, 8:46 AMPREFECT_LOGGING_EXTRA_LOGGERS
no longer reporting logs through to orion.
I went reading through prefect/logging/configuration.py
and see in #7569 that there was some change to the logic. Now there is a test made against config['incremental']
before the orion log handler is attached to the logger of the extra module.
I am not sure exactly what incrmental
is in this sense, but when I disable this check to force the orion handler to attach itself to my extra module, things work as expected and logs are streamed to orion.
It seems that by default that this config['incremental']
is set to True
(at least I have not knowingly set it), and the test made against its value is negative when evaluating whether to attach the orion logger.
So my question is what is this incremental
configurable, and how does one set it? Is the check made against it actually intended to be negated? Once I remove the check my logs for extra modules outlined in PREFECT_LOGGING_EXTRA_LOGGERS
behave the same as pre v2.6.9 / #7569
@Anna Geller @Michael Adkins - I see both your names on the change -- please don't hate me for tagging you directly 🙂Marwan Sarieddine
12/04/2022, 2:26 PMHAITAM BORQANE
12/04/2022, 4:09 PMYaron Levi
12/04/2022, 6:57 PMprefect deployment apply jobs/selfServiceDaily.yaml
Are there any shortcuts to apply many yaml files at once?João Coelho
12/04/2022, 8:45 PMdatamongus
12/05/2022, 12:16 AMAbhinav Ayalur
12/05/2022, 1:05 AMAbhinav Ayalur
12/05/2022, 1:27 AMoliverek12
12/05/2022, 4:25 AMBen Muller
12/05/2022, 6:15 AM.submit
with the DaskTaskRunner
with no cluster.
It seems like about two tasks run at a time and then another two start, my assumption was that all 8 would start and Dask handles them and they finish when they are ready?
I know this would be limited by the threads too, but the machine it is running on should have more than enough threads to handle those 8 tasks.Deepanshu Aggarwal
12/05/2022, 6:56 AMiñigo
12/05/2022, 9:07 AMJoël Luijmes
12/05/2022, 10:44 AMVadym Dytyniak
12/05/2022, 11:17 AMENDRUN
exception to fail and do not retry the task, does it exist in Prefect 2?Timo
12/05/2022, 1:26 PMLuca Schneider
12/05/2022, 2:02 PMState Message
Flow run infrastructure exited with non-zero status code -1.
Justin Trautmann
12/05/2022, 2:30 PMMatthew Ferry
12/05/2022, 3:06 PMVadym Dytyniak
12/05/2022, 3:06 PMget_run_logger
? When I try to use task.fn it fails with no task or flow context.Vadym Dytyniak
12/05/2022, 4:06 PM@task()
def copy_file(n: int):
get_run_logger().info('Start file copying')
time.sleep(n)
get_run_logger().info('Completed file copying')
@flow(task_runner=SequentialTaskRunner())
def sample() -> None:
copy_file.map([3, 4])
if __name__ == '__main__':
sample()
Logs:
18:05:29.909 | INFO | Flow run 'devious-aardwolf' - Created task run 'copy_file-6cc8b0a3-0' for task 'copy_file'
18:05:29.910 | INFO | Flow run 'devious-aardwolf' - Executing 'copy_file-6cc8b0a3-0' immediately...
18:05:29.916 | INFO | Flow run 'devious-aardwolf' - Created task run 'copy_file-6cc8b0a3-1' for task 'copy_file'
18:05:29.917 | INFO | Flow run 'devious-aardwolf' - Executing 'copy_file-6cc8b0a3-1' immediately...
18:05:30.468 | INFO | Task run 'copy_file-6cc8b0a3-1' - Start file copying
18:05:30.534 | INFO | Task run 'copy_file-6cc8b0a3-0' - Start file copying
18:05:33.536 | INFO | Task run 'copy_file-6cc8b0a3-0' - Completed file copying
18:05:33.726 | INFO | Task run 'copy_file-6cc8b0a3-0' - Finished in state Completed()
18:05:34.469 | INFO | Task run 'copy_file-6cc8b0a3-1' - Completed file copying
18:05:34.658 | INFO | Task run 'copy_file-6cc8b0a3-1' - Finished in state Completed()
Tim-Oliver
12/05/2022, 4:37 PMtr = await client.read_task_runs(limit=1, flow_run_filter=FlowRunFilter.construct(id="562d1705cfcc48a0b55df98373fd10bd"))
Christian Vogel
12/05/2022, 5:14 PMAshley Felber
12/05/2022, 5:39 PMJean-Michel Provencher
12/05/2022, 6:48 PMJean-Michel Provencher
12/05/2022, 6:48 PMRob Freedy
12/06/2022, 4:18 PM--skip-upload
flag which will skip uploading all of the files to remote storage. You can submit a feature request for specifying files here: https://github.com/PrefectHQ/prefect/issues/new/choose
The prefect ignore file has the same syntax as a gitignore file, which should support wildcards: https://git-scm.com/docs/gitignore