James Phoenix
08/14/2022, 9:03 PMJames Phoenix
08/14/2022, 9:03 PMprefect deployment build base_flow.py:basic_flow --name dev --tag dev --storage-block gcs/dev
James Phoenix
08/14/2022, 9:04 PMJames Phoenix
08/14/2022, 9:04 PMgs://
instead of gcs://
?James Phoenix
08/14/2022, 9:21 PM@property
def basepath(self) -> str:
return f"gcs://{self.bucket_path}"
@property
def filesystem(self) -> RemoteFileSystem:
settings = {}
if self.service_account_info:
try:
settings["token"] = json.loads(
self.service_account_info.get_secret_value()
)
except json.JSONDecodeError:
raise ValueError(
"Unable to load provided service_account_info. Please make sure that the provided value is a valid JSON string."
)
remote_file_system = RemoteFileSystem(
basepath=f"gcs://{self.bucket_path}", settings=settings
)
return remote_file_system
James Phoenix
08/14/2022, 9:21 PMLuke Segars
08/15/2022, 2:45 AM--env
at agent startup is a great solution, but the env variable is null for all three of these approaches.
1. passing as arg in prefect agent kubernetes start --env SERVICES_ENDPOINT=XYZ
2. passing directly as an env var to the agent (dont think this is expected to work)
3. passing as PREFECT___CLOUD____AGENT__ENV_VARS_
any thoughts on what i could be doing wrong? i tried each of these individuallyIuliia Volkova
08/15/2022, 3:07 AMBeizhen
08/15/2022, 7:48 AMClientError
message? All upstream tasks succeeded.
Setup: prefect 1.2.2
, prefect cloud.
Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID f0614b56-3c3b-41f6-9481-878c9f4bee78: provided a running state but associated flow run 8647f325-a45d-4124-9995-b68d43faaf15 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 1604, in set_task_run_state
result = self.graphql(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 464, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID f0614b56-3c3b-41f6-9481-878c9f4bee78: provided a running state but associated flow run 8647f325-a45d-4124-9995-b68d43faaf15 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
Sven Aoki
08/15/2022, 8:22 AMprefect deployment build fx_run.py:fx_rates_flow -n fx_deploy --infra docker-container -sb remote-file-system/minio
and then I also created a custom docker image with s3fs installed which I pull during deployment. However I keep getting the error: No such file or directory: 'docker': 'docker'
. Can anyone advise?Chris L.
08/15/2022, 8:32 AMworker_client
context manager works as expected in Prefect 1.0 (see https://discourse.prefect.io/t/how-to-use-dask-without-mapping-in-prefect-1-0-using-das[…]-client-to-call-client-submit-inside-a-prefect-task/470/13) but this behavior doesn't seem to migrate over to 2.0Hawkar Mahmod
08/15/2022, 8:53 AMDekel R
08/15/2022, 1:51 PMTask 'upload_data_to_bq_task': Exception encountered during task execution!
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
value = prefect.utilities.executors.run_task_with_timeout(
File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/Users/dekelr/PycharmProjects/similarity-filter-layer/prefect_tasks/upload_data_to_bq.py", line 24, in upload_data_to_bq_task
SystemError: unknown opcode
The code fails when running this specific row - (batch size is either an int or None)
if batch_size is None:
Everything works fine when I change this row to “if not batch_size:”
After some troubleshooting - I found this thread - https://github.com/PrefectHQ/prefect/issues/3635
Running the same flow with the original row of -
if batch_size is None:
Still doesn’t work when registering with python 3.8 (same python version as in the container)
Can you please explain this really odd behavior?
Thanksdammy arinde
08/15/2022, 2:05 PMOscar Björhn
08/15/2022, 2:22 PMSam Garvis
08/15/2022, 3:15 PMHamza Naanani
08/15/2022, 3:24 PMJames Brady
08/15/2022, 4:58 PMLana Dann
08/15/2022, 5:57 PMPedro Machado
08/15/2022, 8:04 PMJai P
08/15/2022, 8:55 PMdatamongus
08/15/2022, 11:25 PMprefect-gcp
seems to only refer to Prefect 2.0James Brady
08/16/2022, 5:24 AMemo loic
08/16/2022, 9:20 AMAndreas Nigg
08/16/2022, 10:18 AMOscar Björhn
08/16/2022, 10:38 AMMarcin Grzybowski
08/16/2022, 11:26 AMParwez Noori
08/16/2022, 1:45 PMhttps://www.youtube.com/watch?v=m9FYoOIfWMc&ab_channel=Prefect▾
Andreas Nigg
08/16/2022, 1:50 PMTim Enders
08/16/2022, 1:59 PMsqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
Tim Enders
08/16/2022, 1:59 PMsqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
Marcin Grzybowski
08/16/2022, 2:19 PMAnna Geller
08/16/2022, 2:28 PMIs it recommended to use a Postgres backend when using DaskIMO yes since you will most likely come across issues with concurrent writes which won't work with SQLite Postgres or just using Cloud - both works: https://app.prefect.cloud/
Tim Enders
08/16/2022, 2:37 PM