Matt Delacour
08/12/2022, 6:43 PMAlex Shea
08/12/2022, 7:10 PMPREFECT__CLOUD__AGENT__ENV_VARS
agent variable?Chris Marchetti [Datateer]
08/12/2022, 7:46 PM[2022-08-12 19:42:34+0000] INFO - prefect.Docker | Pushing image to the registry...
The error is:
InterruptedError: EOF
Complete traceback in thread. Anyone seen this before?Tim Helfensdörfer
08/12/2022, 8:30 PMEmil Ordoñez
08/12/2022, 9:52 PMEdmondo Porcu
08/12/2022, 9:54 PMBlake Hamm
08/12/2022, 10:24 PMdict
, it still reads it in as a string in the UI and can't seem to parse it out. I'm pretty much stuck with the default dictionary parameter or I can trigger it from the CLI. Do I essentially have to use pydantic
and explicitly specify the model when working with dictionary parameters?
Some other funny parameter behavior, the UI displays the default parameter as [object Object]
when I'm creating a custom flow run. But on the flow run (with defaults), it looks like more like a dictionary.Brad
08/12/2022, 10:59 PMRohan Chutke
08/13/2022, 12:25 AMMasatoShima
08/13/2022, 1:38 PMArtur Spatari
08/13/2022, 2:44 PMMoises Vera
08/13/2022, 6:10 PMMarwan Sarieddine
08/13/2022, 6:40 PMRio McMahon
08/13/2022, 7:11 PMhttps://
prefix but s3://
is hardcoded in the S3 filesystem object). I did this by copying the S3 storage object code and removing the hardcoded prefix but leaving everything else the same (except name and credential arguments). The block successfully registers to prefect cloud. However when I try to attach it as the storage block I get the error in the comments.
From what I can tell this is happening during the validation step. At some point prefect is looking at the registry for the Block
type via:
from prefect.utilities.dispatch import lookup_type, get_registry_for_type
from prefect.blocks.core import Block
registry = get_registry_for_type(Block)
print(registry)
which returns all the prefect defined blocks. If I try to build a deployment it will fail because the custom filesystem class I created does not show up in that registry. Currently the remote file system object stores all ‘settings’ as plain text which isn’t ideal because I’m storing access credentials. Thoughts on the best way to proceed?Dev Dabke
08/13/2022, 7:43 PMMarwan Sarieddine
08/13/2022, 8:32 PMOscar Björhn
08/14/2022, 11:22 AMJames Brady
08/14/2022, 1:40 PMdask_kubernetes.KubeCluster
class – trying to get a flow to run on a GPU-enabled node. What's happening:
• The pod_template I'm using for the flow specifies <http://nvidia.com/gpu|nvidia.com/gpu>: 1
, per the docs
• karpenter starts a new node, which has a GPU (🙌)
• However, the new node can't accommodate the dask client because it doesn't have the right resource annotation ("0/3 nodes are available: … , 3 Insufficient nvidia.com/gpu")
I realise this might be a question better-suited to the dask community, but would appreciate any stories of people successfully running Prefect 2 workloads on GPU-accelerated nodes and/or help figuring out the above specific issue.James Phoenix
08/14/2022, 9:02 PMJames Phoenix
08/14/2022, 9:02 PMFile system created with scheme 'gcs' from base path '<gcs://prefect_source_data>' could not be created. You are likely missing a Python module required to use the given storage protocol.
James Phoenix
08/14/2022, 9:03 PMJames Phoenix
08/14/2022, 9:03 PMprefect deployment build base_flow.py:basic_flow --name dev --tag dev --storage-block gcs/dev
James Phoenix
08/14/2022, 9:04 PMJames Phoenix
08/14/2022, 9:04 PMgs://
instead of gcs://
?James Phoenix
08/14/2022, 9:21 PM@property
def basepath(self) -> str:
return f"gcs://{self.bucket_path}"
@property
def filesystem(self) -> RemoteFileSystem:
settings = {}
if self.service_account_info:
try:
settings["token"] = json.loads(
self.service_account_info.get_secret_value()
)
except json.JSONDecodeError:
raise ValueError(
"Unable to load provided service_account_info. Please make sure that the provided value is a valid JSON string."
)
remote_file_system = RemoteFileSystem(
basepath=f"gcs://{self.bucket_path}", settings=settings
)
return remote_file_system
James Phoenix
08/14/2022, 9:21 PMLuke Segars
08/15/2022, 2:45 AM--env
at agent startup is a great solution, but the env variable is null for all three of these approaches.
1. passing as arg in prefect agent kubernetes start --env SERVICES_ENDPOINT=XYZ
2. passing directly as an env var to the agent (dont think this is expected to work)
3. passing as PREFECT___CLOUD____AGENT__ENV_VARS_
any thoughts on what i could be doing wrong? i tried each of these individuallyIuliia Volkova
08/15/2022, 3:07 AMBeizhen
08/15/2022, 7:48 AMClientError
message? All upstream tasks succeeded.
Setup: prefect 1.2.2
, prefect cloud.
Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID f0614b56-3c3b-41f6-9481-878c9f4bee78: provided a running state but associated flow run 8647f325-a45d-4124-9995-b68d43faaf15 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 1604, in set_task_run_state
result = self.graphql(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 464, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID f0614b56-3c3b-41f6-9481-878c9f4bee78: provided a running state but associated flow run 8647f325-a45d-4124-9995-b68d43faaf15 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
Sven Aoki
08/15/2022, 8:22 AMprefect deployment build fx_run.py:fx_rates_flow -n fx_deploy --infra docker-container -sb remote-file-system/minio
and then I also created a custom docker image with s3fs installed which I pull during deployment. However I keep getting the error: No such file or directory: 'docker': 'docker'
. Can anyone advise?