iKeepo w
10/24/2022, 3:49 AMprefect deployment build
?Stephen Herron
10/24/2022, 3:56 AMdbt_projects/*
!dbt_projects/.gitkeep
Tim Enders
10/24/2022, 2:08 PMisinstance(blah, Failed)
but Failed isn't a type. Here is a code snippet:
if isinstance(bq_result, LoadJob) and bq_result.state == "DONE":
return Completed(message="Load Finished!")
elif isinstance(bq_result, Failed):
slack_webhook_block = SlackWebhook.load("data-pipeline-notifications")
slack_webhook_block.notify("Hello from Prefect 2.0!")
return bq_result
else:
iñigo
10/24/2022, 4:03 PMDavid Elliott
10/24/2022, 4:08 PMAlejandro
10/24/2022, 4:49 PMBlock
inside the class in which another Block
is defined using the Python API?
class Foo(Block):
target_name: str
_block_type_slug = "foo-block"
def block_initialization(self) -> None:
bar_block = JSON.load("bar-block").value
another_block = JSON.load(target_name).value
I have tried with the above code but I get the following warning, which results in a coroutine being returned instead of a block:
RuntimeWarning: coroutine 'Block.load' was never awaited
redsquare
10/24/2022, 5:22 PMAgent started! Looking for work from queue(s): default-work-queue...Mon, Oct 24 2022 6:17:45 pm17:17:45.536 | ERROR | prefect.agent - Failed to create work queue 'default-work-queue'.Mon, Oct 24 2022 6:17:45 pm
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<https://api.prefect.cloud/api/accounts/9396b5ff-23c0-42b4-91d9-5892450xxxx/workspaces/de803906-c6f7-45%0A77-a3aa-d300b56fxxx/work_queues/>'
Tim Enders
10/24/2022, 6:52 PMif __name__ == "__main__":
try:
flow_result = main()
except Exception as e:
slack_webhook_block = SlackWebhook.load("data-pipeline-notifications")
slack_webhook_block.notify(f"{e.__class__}: {str(e)}")
Tim Enders
10/24/2022, 9:13 PMlink89
10/25/2022, 1:19 AMrm -rf ~/.prefect/storage/*
to clear cache so that the task would run again next time.
But I find it won't work this time and I have to clean the whole ~/.prefect
directory instead of just storage to make it work.
Is this a bug or I just miss something? Does prefect provide some command line interface to clear cache?Saurabh Indoria
10/25/2022, 3:04 AM[20 Oct 2022 12:22pm]: Error during execution of task: SSLError(MaxRetryError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)')))"))
Kishan
10/25/2022, 3:33 AMpip install prefect[gcp]
as per the documentation but I'm getting WARNING: prefect 2.6.4 does not provide the extra 'gcp'
. Maybe the docs are not up to date? Anyone know how to install the vertex agent now?Jake Loo
10/25/2022, 3:54 AMJon Ruhnke
10/25/2022, 4:11 AMKhyaati Jindal
10/25/2022, 5:23 AMraise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/xxxxxx>
return asynclib.run(func, *args, **backend_options)
Usually re - running the agent works fine but given the sensitive nature of my prjects, I would like to avoid this in future, hence i am trying to understand why is this happeningvholmer
10/25/2022, 8:32 AMPrefectHTTPStatusError: Client error '403 Forbidden' for url
'<https://api.prefect.cloud/api/accounts/><ACCOUNTGUID>/wor
kspaces/<WORKSPACEGUID>/work_queues/<WORKQUEUEGUID>/get_runs'
At /usr/local/lib/python3.8/dist-packages/prefect/client/base.py:125. Submitting any flows after this exception leads to them being visible in the Prefect cloud web interface as "Scheduled" or "Late", basically the agent completely dies after this. Is this a known issue or could this be due to something else I'm doing incorrectly?Lukasz Pakula
10/25/2022, 9:32 AMINFO - Retiring workers [154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185]
INFO - Adaptive stop
INFO - Adaptive stop
ERROR - prefect.CloudFlowRunner | Unexpected error: KilledWorker('<name>', <WorkerState 'tcp://<ip>', name: 47, status: closed, memory: 0, processing: <number>, 3)
Restarting the flow is resolving the issue.
Is there any sensible explanation of why upgrading kubernetes cluster could cause it? Or i must be missing something elsewhere ?Stanislav Kotsiievskyi
10/25/2022, 12:45 PMFlow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 247, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 159, in load_flow_from_flow_run
await storage_block.get_directory(from_path=deployment.path, local_path=".")
File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 146, in get_directory
shutil.copytree(from_path, local_path, dirs_exist_ok=True)
File "/usr/local/lib/python3.10/shutil.py", line 556, in copytree
with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp2ywchethprefect/flows/hello_world'
Have you got experience with such issues?Jessica Smith
10/25/2022, 1:41 PMNic
10/25/2022, 1:52 PMNic
10/25/2022, 2:23 PMA list of volume mount strings in the format of "local_path:container_path".
Would following bind my Local G:/ Drive to a G:/ in the docker, or should I write it differently?Tim Enders
10/25/2022, 2:53 PMTraceback (most recent call last):
File "/home/tenders/Documents/code/prefect-orion/platform_prefect/accounts/accounts_subscriptions.py", line 359, in <module>
flow_result = main()
File "/home/tenders/.cache/pypoetry/virtualenvs/prefect-orion-HonJDUqB-py3.10/lib/python3.10/site-packages/prefect/flows.py", line 439, in __call__
return enter_flow_run_engine_from_flow_call(
File "/home/tenders/.cache/pypoetry/virtualenvs/prefect-orion-HonJDUqB-py3.10/lib/python3.10/site-packages/prefect/engine.py", line 150, in enter_flow_run_engine_from_flow_call
return anyio.run(begin_run)
File "/home/tenders/.cache/pypoetry/virtualenvs/prefect-orion-HonJDUqB-py3.10/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/home/tenders/.cache/pypoetry/virtualenvs/prefect-orion-HonJDUqB-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/home/tenders/.cache/pypoetry/virtualenvs/prefect-orion-HonJDUqB-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/home/tenders/.cache/pypoetry/virtualenvs/prefect-orion-HonJDUqB-py3.10/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/home/tenders/.cache/pypoetry/virtualenvs/prefect-orion-HonJDUqB-py3.10/lib/python3.10/site-packages/prefect/engine.py", line 229, in create_then_begin_flow_run
return await state.result(fetch=True)
File "/home/tenders/.cache/pypoetry/virtualenvs/prefect-orion-HonJDUqB-py3.10/lib/python3.10/site-packages/prefect/states.py", line 86, in _get_state_result
raise MissingResult(
prefect.exceptions.MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.
It seems like Prefect is still expecting the State data to exist, even though it doesn't by default? (I may be misunderstanding things too)Mary Clair Thompson
10/25/2022, 3:41 PM@dataclass
class Config(YamlDataClassConfig):
EMAIL_FROM: str = None
EMAIL_TO: List[str] = None
as well as a task intended to instantiate the instance of the class and load data to it:
@task(name='load-configs')
def load_configs(config_path: str) -> Config:
configs = Config()
configs.load(config_path)
return configs
The task seems to instantiate the config object, but breaks when we try to actually load configs from the provided path. Again, this works perfectly fine outside of the context of a task, so presumably there is some internal prefect logic I’m futzing with here. I’d appreciate any thoughts!David Elliott
10/25/2022, 3:47 PMCarlo
10/25/2022, 4:55 PMClaire Herdeman
10/25/2022, 8:27 PMDavid Stern
10/26/2022, 4:40 AMDavid Huang
10/26/2022, 7:09 AMlink89
10/26/2022, 7:27 AMsignal
by themselves, and due to this limitation
When threads are enabled, this function can only be called from the main thread of the main interpreter; attempting to call it from other threads will cause ahttps://docs.python.org/3/library/signal.html#signal.signal It will fail when execute as task of prefect.exception to be raised.ValueError
Khyaati Jindal
10/26/2022, 8:11 AMKhyaati Jindal
10/26/2022, 8:11 AMRob Freedy
10/26/2022, 12:54 PMKhyaati Jindal
10/31/2022, 6:58 AM