Wei Mei
03/18/2022, 6:05 PMUnexpected error while running flow: KeyError('Task slug connect_source-1 is not found in the current Flow. This is usually caused by a mismatch between the flow version stored in the Prefect backend and the flow that was loaded from storage.\n- Did you change the flow without re-registering it?\n- Did you register the flow without updating it in your storage location (if applicable)?')
Jean-Michel Provencher
03/18/2022, 7:05 PMupstream_tasks
with methods that actually requires parameter?
The documentation is not really clear regarding how to chain multiple upstream_tasks to create dependencies between them and I was wondering if some of you had some more complex examples.
For example, I don’t think I can do this
with Flow(f"{environment_prefix}-test", storage=S3(bucket=storage_location_bucket_name)) as flow:
dbt_run(organization_id_param, data_processing_start_date_param, data_processing_end_date_param, should_process_last_period, period, period_value,
upstream_tasks=[pull_snowflake_secret(a,b), pull_repo(b,c)])
Tony Liberato
03/18/2022, 7:44 PMMichał Augoff
03/18/2022, 8:12 PMIMAGE_PULL_SECRETS
for the Orion k8s agent/flow runner? I couldn’t find anything in the docs and the codeDarshan
03/18/2022, 9:06 PMTabari Brannon
03/18/2022, 9:35 PMMilton
03/18/2022, 10:01 PMMichał Augoff
03/18/2022, 10:28 PMKubernetesRun
restrict the flow only to kubernetes agents or can it still be picked up by any agent as long as the labels match? I was under this impression after reading the docs but when I created 1 docker agent and 1 k8s agent with the same set of labels, my kubernetes flow got picked up by the docker agentRoyzac
03/18/2022, 11:40 PMSerge Tarkovski
03/19/2022, 12:34 PMprefect==2.0b2
, Python 3.9, Linux):
• created an S3 bucket and a storage (isn't it too much for a local run?)
• started a local API server and configured PREFECT_API_URL
• created a queue
• started an agent for that queue
• created a deployment with DockerFlowRunner and no parameters
Then I ran the deployment and the agent output wasn't very informative, see below. In the UI the flow run "dag-sample/caped-antelope" is shown as failed without any logs available. Anyone can explain?
$ prefect agent start 6aa0c2b2-e895-4b8c-aed3-8bef5b2c88ab
Starting agent connected to <http://127.0.0.1:4200/api>...
___ ___ ___ ___ ___ ___ _____ _ ___ ___ _ _ _____
| _ \ _ \ __| __| __/ __|_ _| /_\ / __| __| \| |_ _|
| _/ / _|| _|| _| (__ | | / _ \ (_ | _|| .` | | |
|_| |_|_\___|_| |___\___| |_| /_/ \_\___|___|_|\_| |_|
Agent started! Looking for work from queue '6aa0c2b2-e895-4b8c-aed3-8bef5b2c88ab'...
13:29:31.446 | INFO | prefect.agent - Submitting flow run '8eb8aa0b-82da-4081-9a42-aae1f22b0525'
/home/tarkovskyi/miniconda3/envs/prefect_exp39/lib/python3.9/site-packages/prefect/flow_runners.py:697: UserWarning: `host.docker.internal` could not be automatically resolved to your local ip address. This feature is not supported on Docker Engine v19.3.15, upgrade to v20.10.0+ if you encounter issues.
warnings.warn(
13:29:32.215 | INFO | prefect.flow_runner.docker - Flow run 'caped-antelope' has container settings = {'image': 'prefecthq/prefect:2.0b2-python3.9', 'network': None, 'network_mode': 'host', 'command': ['python', '-m', 'prefect.engine', '8eb8aa0b-82da-4081-9a42-aae1f22b0525'], 'environment': {'PREFECT_API_URL': '<http://127.0.0.1:4200/api>'}, 'auto_remove': False, 'labels': {'io.prefect.flow-run-id': '8eb8aa0b-82da-4081-9a42-aae1f22b0525'}, 'extra_hosts': {}, 'name': 'caped-antelope', 'volumes': []}
13:29:33.547 | INFO | prefect.agent - Completed submission of flow run '8eb8aa0b-82da-4081-9a42-aae1f22b0525'
13:29:33.584 | INFO | prefect.flow_runner.docker - Flow run container 'caped-antelope' has status 'running'
13:29:44.923 | INFO | prefect.flow_runner.docker - Flow run container 'caped-antelope' has status 'exited'
Taylor Harless
03/19/2022, 5:29 PMfrom prefect import Flow
from prefect.tasks.gcp.storage import GCSUpload
with Flow("google-cloud-test") as flow:
GCSUpload(bucket="test-upload", create_bucket=True)(
data="test-file.csv", credentials="GCP_CREDENTIALS"
)
flow.run()
and received an error: AttributeError: 'str' object has no attribute 'keys'
. I've incorporated the feedback from a similar error discussion recently, but can't figure out what the issue is. Any help is much appreciated.Omar Sultan
03/20/2022, 9:27 AMFile "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 341, in _raise_timeout
self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='prefect-apollo.prefect', port=4200): Read timed out. (read timeout=15)
This happens especially when we use the Task StartFlowRun it does not happen very often , but I was wondering if there was a way to force retry or if anyone knows why this would be happening? ThanksTomer Cagan
03/20/2022, 12:48 PMworker_client
and then submit new tasks to the same cluster (code inside), I see that printout from the first call of the function is logged in prefect and shipped to server, but consecutive calls are not (see log inside).
Is there some trick I can use?Trip Humphrey
03/21/2022, 3:15 AMMadhup Sukoon
03/21/2022, 4:28 AMprefect get flows
), and
2. De-register some of these flows through python.Ryan Sattler
03/21/2022, 5:51 AMmoti shakuri
03/21/2022, 11:04 AMAndreas Nord
03/21/2022, 1:32 PMAzer Rustamov
03/21/2022, 2:05 PMKubernetesFlowRunner
?Jared Robbins
03/21/2022, 2:33 PMJared Robbins
03/21/2022, 2:44 PMJared Robbins
03/21/2022, 3:10 PMPaul Gierz
03/21/2022, 3:21 PMEmma Rizzi
03/21/2022, 3:52 PMSamay Kapadia
03/21/2022, 4:07 PM@prefect.task()
def get_backfill_or_scheduled_date():
time = prefect.context.get("backfill_time") or prefect.context.get("scheduled_start_time")
print(time)
print(type(time))
However I see that this task creates a pod in kubernetes with the image prefecthq/prefect:0.15.12
, is there a way that I can specify the python version for this image? The mismatch between the build and run environment is causing an errorFlorian Guily
03/21/2022, 4:36 PMMichael Moscater
03/21/2022, 6:15 PMFuETL
03/21/2022, 6:19 PMHedgar
03/21/2022, 7:24 PM<tel:18-03-2022|18-03-2022>.csv
instead of <tel:21-03-2022|21-03-2022>.csv
like I have said before my code is on ec2 instance that through a lambda start and stops at certain time of each day. What could be amiss?