Ben Muller
11/17/2022, 12:57 AM@flow
def my_flow(
start: str = (datetime.utcnow() - timedelta(days=7)).strftime("%Y-%m-%d"),
end: str = datetime.utcnow().strftime("%Y-%m-%d"),
):
Will these default parameters be set at deploy time or will they update at run time ?Nick Batchelder
11/17/2022, 1:03 AMusername=nick
at runtime.Deepanshu Aggarwal
11/17/2022, 8:15 AMhttpx.RemoteProtocolError: <ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:19999, additional_data:None>
anyone who faced similar issue ?
adding the trace stack in commentsBen Muller
11/17/2022, 8:31 AMlock document has schema checksum sha256:96937dc8f6bd314a1cbeae7623159848872249e742b3bac8d7af39da4e2dfe79 which does not match the schema checksum for class 'ECSTask'. This indicates the schema has changed and this block may not load.
How do i debug and or make sure everything is all good?Mohit Singhal
11/17/2022, 8:41 AMManuel Garrido Peña
11/17/2022, 9:32 AMparams= X=y
, why did we remove such awesome feature?Zinovev Daniil
11/17/2022, 11:26 AMHi all. I had a problem right after installing prefect orion. UI is not loading. I am getting a blank page with a title. JS is not fully loaded. It breaks at 1.1 mb. Perhaps someone solved such a problem?
Rajeshwar Agrawal
11/17/2022, 1:27 PMJared Robbins
11/17/2022, 2:06 PMJon Young
11/17/2022, 2:19 PMcreate_flow_run.map
. How to get the results of the mapped flow?
I'm getting this error:
└── 09:11:14 | ERROR | Task 'get_task_run_result': Exception encountered during task execution!
Traceback (most recent call last):
File "/Users/jonyoung/Dev/workflows/.venv/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
value = prefect.utilities.executors.run_task_with_timeout(
File "/Users/jonyoung/Dev/workflows/.venv/lib/python3.9/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/Users/jonyoung/Dev/workflows/.venv/lib/python3.9/site-packages/prefect/tasks/prefect/flow_run.py", line 220, in get_task_run_result
flow_run = FlowRunView.from_flow_run_id(flow_run_id)
File "/Users/jonyoung/Dev/workflows/.venv/lib/python3.9/site-packages/prefect/backend/flow_run.py", line 585, in from_flow_run_id
flow_run_data = cls._query_for_flow_run(where={"id": {"_eq": flow_run_id}})
File "/Users/jonyoung/Dev/workflows/.venv/lib/python3.9/site-packages/prefect/backend/flow_run.py", line 627, in _query_for_flow_run
result = client.graphql(flow_run_query)
File "/Users/jonyoung/Dev/workflows/.venv/lib/python3.9/site-packages/prefect/client/client.py", line 464, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'message': 'parsing UUID failed, expected String, but encountered Array', 'locations': [{'line': 2, 'column': 5}], 'path': ['flow_run'], 'extensions': {'path': '$.selectionSet.flow_run.args.where.id._eq', 'code': 'parse-failed', 'exception': {'message': 'parsing UUID failed, expected String, but encountered Array'}}}]
Taylor Curran
11/17/2022, 2:47 PMKirill Popov
11/17/2022, 3:21 PMJai P
11/17/2022, 4:18 PMretry_on
functionality is going to be added to Prefect 2 tasks as well? I see it was added in prefect 1.2jack
11/17/2022, 5:58 PMFernando Silveira
11/17/2022, 6:21 PMingestion_flow
). A couple of notes about this:
• ingestion_flow
runs every day and to ingests a data partition for that date. The flow is parameterized with the date
to ingest.
• the flow is organized into a few tasks to ingest data from different clients. So the flow just runs a for loop running client_ingestion_task(client, date)
for each client
Once ingested this data is useful for a bunch of downstream pipelines. So I'd like to re-use ingestion_flow
as a starting point for two other flows, say, training_flow_1
and training_flow_2
.
The way I thought this could be accomplished was by simply running ingestion_flow
as a subflow at the beginning of both training_flow_{1,2}
. Obviously I don't want to ingest the same data twice. I created a caching key for client_ingestion_task
using the client
and date
parameters. However I quickly stumbled upon the fact that task caches only rely on local storage. I'm running these flows in kubernetes and I was hoping for flows to run concurrently ensuring that ingestion only happens once and the cached status gets re-used in the second flow to run.
The fact that task caches are restricted to local storage tells me this is not the way to implement what I wanted. Can someone point me in the right direction here? Happy to discuss more details if that's needed.Heather DeHaven
11/17/2022, 6:41 PMKevin Wang
11/17/2022, 7:14 PMVishy ganesh
11/17/2022, 8:53 PMbasepath
option but, s3_upload has no basepath..
How do we associate a basepath to the s3_upload option
Essentially I don't want the files to land in the bucket rootJean-Michel Provencher
11/17/2022, 9:24 PMgit clone <https://github.com/jmprovencher/prefect-tutorial>
and follow the steps in README.md
and TUTORIAL.md
Erik Mathis
11/17/2022, 9:37 PMKalise Richmond
11/17/2022, 9:59 PMMadison Schott
11/17/2022, 10:55 PMawait
from the function, any ideas?
A 'sync_compatible' method was called from a context that was previously async but is now sync. The sync call must be changed to run in a worker thread to support sending the coroutine for 'load' to the main thread.
Ben Muller
11/18/2022, 12:51 AMrun_deployment
to essentially launch 30 concurrent ecs tasks. What I notice is that 5 of the deployment runs start and the other 25 are pending.
How do I make this behave similar to prefect 1.0 where I was able to have all of these run at the same time ? Is this a limitation of my agent ? I was under the impression all the agent does is orchestrate the ecs tasks in my aws environment?Howard
11/18/2022, 6:05 AMAndreas Nigg
11/18/2022, 7:27 AMTim-Oliver
11/18/2022, 11:03 AMDaskTaskRunner
workers with the current main-branch version of Prefect, which works now locally thanks to some very recent changes (thanks a lot 💐). However, I am having troubles getting the logs from tasks which are running on a DaskTaskRunner
which uses dask_jobqueue.SLURMCluster
. The logs from tasks are written into the slurm-output file, but not propagated back to the flow-log or the cloud UI. Happy, to test some things if it would be helpful.Joshua Greenhalgh
11/18/2022, 12:02 PMWARNING:urllib3.connectionpool:Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)")': /
Tushar Gupta
11/18/2022, 12:50 PMVadym Dytyniak
11/18/2022, 1:53 PMprefetch-seconds
. It doesn't work like we expected. Details in the thread.Dave
11/18/2022, 3:19 PMget_run_logger().setLevel(level)
within each task and flow. Would this require overriding flow and task initiation, or similar? Thank you in advance for any help!Dave
11/18/2022, 3:19 PMget_run_logger().setLevel(level)
within each task and flow. Would this require overriding flow and task initiation, or similar? Thank you in advance for any help!redsquare
11/18/2022, 3:21 PMDave
11/18/2022, 3:24 PMRyan Peden
11/18/2022, 3:26 PMPREFECT_LOGGING_LEVEL
. Some loggers use that, and some use PREFECT_LOGGING_SERVER_LEVEL
.Dave
11/18/2022, 3:38 PMPREFECT_LOGGING_SERVER_LEVEL
, which might explain the inconsistent behavior, unsure. I'll give that a try. Open to other suggestions as well.
import os
os.environ["PREFECT_LOGGING_LEVEL"] = log_level
os.environ["PREFECT_LOGGING_ROOT_LEVEL"] = log_level
os.environ["PREFECT_LOGGING_HANDLERS_CONSOLE_LEVEL"] = log_level
os.environ["PREFECT_LOGGING_HANDLERS_CONSOLE_FLOW_RUNS_LEVEL"] = log_level
os.environ["PREFECT_LOGGING_HANDLERS_CONSOLE_TASK_RUNS_LEVEL"] = log_level
Ryan Peden
11/18/2022, 3:53 PMos.environ
may or may not work depending on when you run those commands. I think it will work if you run them before you import Prefect. Otherwise, I believe Prefect reads the settings when its module initializes - but I'm going to go double check to code to verify that.Dave
11/18/2022, 4:10 PMPREFECT_LOGGING_SERVER_LEVEL
environment variable forced within Python things seemed to work as expected. Appreciate the help!Ryan Peden
11/18/2022, 4:10 PM