Blake Stefansen
09/21/2022, 3:54 PMNathaniel Russell
09/21/2022, 4:08 PMFile "C:\work\testing_service\sand02-cumulus-pipeline\service.py", line 7, in <module>
from prefect.client import get_client
File "C:\work\testing_service\sand02-cumulus-pipeline\prefect\__init__.py", line 24, in <module>
from prefect.orion.schemas.states import State
File "C:\work\testing_service\sand02-cumulus-pipeline\prefect\orion\schemas\__init__.py", line 1, in <module>
from . import states, schedules, data, core, sorting, filters, responses, actions
File "C:\work\testing_service\sand02-cumulus-pipeline\prefect\orion\schemas\states.py", line 14, in <module>
from prefect.orion.schemas.data import DataDocument
File "C:\work\testing_service\sand02-cumulus-pipeline\prefect\orion\schemas\data.py", line 8, in <module>
from prefect.orion.utilities.schemas import PrefectBaseModel
File "C:\work\testing_service\sand02-cumulus-pipeline\prefect\orion\utilities\schemas.py", line 12, in <module>
import orjson
File "C:\work\testing_service\sand02-cumulus-pipeline\orjson\__init__.py", line 1, in <module>
from .orjson import *
ModuleNotFoundError: No module named 'orjson.orjson'
The extra weird thing is if I start python in anaconda prompt and simply try to import get_client or orjson it doesn't error, but running a python file it does.Mohamed Alaa
09/21/2022, 4:20 PMfrom prefect.blocks.core import Block
cube = Block.load('Cube/cube-test')
where Cube is the name of the registered block and cube-test is an instance of it.
i would always get
ValueError: Unable to find block document named cube-test for block type Cube
I ran this code to find all the registered blocks
from prefect.utilities.dispatch import lookup_type, get_registry_for_type
from prefect.blocks.core import Block
registry = get_registry_for_type(Block)
print(registry)
And it results in all the default prefect block without my custom blocks
Am I missing something?
Thanks in advanceMax
09/21/2022, 6:00 PMSam Garvis
09/21/2022, 6:56 PMAdam Green
09/22/2022, 1:40 AMSeth Coussens
09/22/2022, 2:05 AMDeepanshu Aggarwal
09/22/2022, 5:36 AMale
09/22/2022, 7:10 AMErik
09/22/2022, 7:48 AMEmail/test
I have tried two different methods to access the block in my code.
Method 1:
from prefect.blocks.notifications import NotificationBlock
email = NotificationBlock().load('Email/test')
and I get the following error
File PYTHON/site-packages/prefect/blocks/core.py:836, in Block.__new__(cls, **kwargs)
834 return m
835 else:
--> 836 m = super().__new__(cls)
837 object.__setattr__(m, "__dict__", kwargs)
838 object.__setattr__(m, "__fields_set__", set(kwargs.keys()))
TypeError: Can't instantiate abstract class NotificationBlock with abstract method notify
It turns out this error comes from any attempt to instantiate NotificationBlock()
. Can anyone else confirm this error when instantiating?
Method 2:
from prefect.blocks.core import Block
email = Block().load('Email/test')
and I get the following error
File PYTHON/site-packages/prefect/blocks/core.py:646, in Block.load(cls, name)
642 block_document = await client.read_block_document_by_name(
643 name=block_document_name, block_type_slug=block_type_slug
644 )
645 except prefect.exceptions.ObjectNotFound as e:
--> 646 raise ValueError(
647 f"Unable to find block document named {block_document_name} for block type {block_type_slug}"
648 ) from e
649 return cls._from_block_document(block_document)
ValueError: Unable to find block document named test for block type Email
even though the block has def been created in the UI.
Have I tried the right things? Is there some other way to access my Email block? Or is this a real bug?
PS (I see Mohamed Alaa write about a similar bug in this thread a few hours ago)Steve Smith
09/22/2022, 1:34 PMentrypoint="app\\flows\\my_flow.py:my_flow"
), so when I try to run it the job cannot find the flow source code. My fix is to manually replace "\\" with "/" in the entrypoint to make it a Linux-style path before applying the deployment. Is this a known issue?Jan Domanski
09/22/2022, 2:18 PMprefect cloud login --workspace my-account/my-workspace --key ....
Creating a profile for this Prefect Cloud login. Please specify a profile name: default
Profile 'default' already exists.
I haven’t expected to be prompted about anything, it’s impossible for me to respond to the prompt. Is there another way to provide the key? An ENV var?
Is there a way to “purge” the prefect config to be able to login without the prompt complainingChing
09/22/2022, 3:44 PMDavid Cupp
09/22/2022, 3:45 PM2.4.0
) output text without the fancy borders and column headers?
I would like to pass the output through awk, e.g. prefect deployment ls | awk '(extract deployment id)' | xargs -n 1 (some command)
Nathaniel Russell
09/22/2022, 4:33 PMroot
├── orjson
| ├── __pycache__
| ├── __init__.py
| ├── __init__.pyi
| ├── orjson.cp39-win_amd64.pyd
| └── py.typed
└── service.py
In service.py I am trying to import import prefect.client that then imports orjson
import orjson
This is what init.py looks like
from .orjson import *
__doc__ = orjson.__doc__
if hasattr(orjson, "__all__"):
__all__ = orjson.__all__
however importing orjson throws this error:
File "C:\work\testing_service\sand02-cumulus-pipeline\service.py", line 7, in <module>
from prefect.client import get_client
File "C:\work\testing_service\sand02-cumulus-pipeline\prefect\__init__.py", line 24, in <module>
from prefect.orion.schemas.states import State
File "C:\work\testing_service\sand02-cumulus-pipeline\prefect\orion\schemas\__init__.py", line 1, in <module>
from . import states, schedules, data, core, sorting, filters, responses, actions
File "C:\work\testing_service\sand02-cumulus-pipeline\prefect\orion\schemas\states.py", line 14, in <module>
from prefect.orion.schemas.data import DataDocument
File "C:\work\testing_service\sand02-cumulus-pipeline\prefect\orion\schemas\data.py", line 8, in <module>
from prefect.orion.utilities.schemas import PrefectBaseModel
File "C:\work\testing_service\sand02-cumulus-pipeline\prefect\orion\utilities\schemas.py", line 12, in <module>
import orjson
File "C:\work\testing_service\sand02-cumulus-pipeline\orjson\__init__.py", line 1, in <module>
from .orjson import *
ModuleNotFoundError: No module named 'orjson.orjson'
I have no clue why this import doesn't work because if I simply have orjson in an anaconda environment and import it it works fine.
Here is init.pyi if that is of any use:
import json
from typing import Any, Callable, Optional, Union
__version__: str
def dumps(
__obj: Any,
default: Optional[Callable[[Any], Any]] = ...,
option: Optional[int] = ...,
) -> bytes: ...
def loads(__obj: Union[bytes, bytearray, memoryview, str]) -> Any: ...
class JSONDecodeError(json.JSONDecodeError): ...
class JSONEncodeError(TypeError): ...
OPT_APPEND_NEWLINE: int
OPT_INDENT_2: int
OPT_NAIVE_UTC: int
OPT_NON_STR_KEYS: int
OPT_OMIT_MICROSECONDS: int
OPT_PASSTHROUGH_DATACLASS: int
OPT_PASSTHROUGH_DATETIME: int
OPT_PASSTHROUGH_SUBCLASS: int
OPT_SERIALIZE_DATACLASS: int
OPT_SERIALIZE_NUMPY: int
OPT_SERIALIZE_UUID: int
OPT_SORT_KEYS: int
OPT_STRICT_INTEGER: int
OPT_UTC_Z: int
Clovis
09/22/2022, 4:33 PMFlow Runs
dashboards deployment filter for instance), I see that deployments are not grouped by similar name but by associated flow apparently.
So I think I may misunderstood the prefect deployment philosophy and wanted to ask you all if there is any literature or documentation about the naming convention in prefect P ?Xavier Babu
09/22/2022, 5:04 PMBen Ayers-Glassey
09/22/2022, 5:17 PMAaron Goebel
09/22/2022, 5:19 PMBaseDockerLogin
and defines login as grabbing a token and calling login with the parsed out username, password etc.
I pass the registry to the DockerContainer
infrastructure and save the blocks.
When I create the blocks (registry and container), my prefect agents are initially able to execute deployments and pull the latest containers from ECR. But after 12 hours or so (I assume token expiration), all deployments to the agent :
docker.errors.APIError: 500 Server Error for <http+docker://localhost/v1.41/auth>: Internal Server Error ("login attempt to https://{My_ECR_Registry}/v2/ failed with status: 403 Forbidden
I'm kind of confused why it works right after creating a block and deployment, but seems like reauthentication does notQ
09/22/2022, 5:32 PMRemoteFileSystem(basepath="<webhdfs://basepath>", settings=params).write_path(path="subpath/filename", content=content)
I get an exception like this:
null for uri: <https://hdfs.example.com>:port/webhdfs/v1webhdfs%3A//basepath/subpath?op=MKDIRS
Which is unsurpring, since URL above is clearly not valid.
The problem here is that `fsspec.WebHDFS`'s methods makedirs/mkdir expect a slash-prefixed path w/o scheme (in this case /basepath/subpath
), while prefect.RemoteFileSystem
just passes <scheme://basepath/subpathdir>
and expects fsspec to do the rest.
2. A lot of fsspec.AbstractFileSystem methods rely on fsspec.infer_storage_options, most importantly open
, which is used by prefect.RemoteFileSystem.write_path
and prefect.RemoteFileSystem.read_path
. infer_storage_options
calls urlsplit and returns urlsplit(path).path
, i.e. strips away scheme+netloc.
Here the problem is that netloc is whatever comes after the scheme and before the first slash, e.g. for <webhdfs://home/user/project>
home
would get stripped away erroneosly.
This means one would have to prepend an extra segment to basepath to be sacrificed to fsspec's implementation (e.g. <webdhfs://thisgetslost/home/user/project>
).
I can subclass RemoteFileSystem
and override write_path
to solve both problems.
But then I would, it seems, need to make this class definition available and import it before calling Block.load
.
What I don't understand is how I can make an agent use this to fetch code. Right now it just fails with KeyError: "No class found for dispatch key 'subclassname' in registry for type 'Block'."
. Any ideas?
Maybe there is a better way of doing this that doesn't involve subclassing?Vaikath Job
09/22/2022, 6:46 PMIlya Galperin
09/22/2022, 7:02 PMSam Garvis
09/22/2022, 7:15 PMAmol Shirke
09/22/2022, 7:23 PMLuis Henrique
09/22/2022, 7:33 PMkiran
09/22/2022, 8:29 PMMichał Augoff
09/22/2022, 9:01 PMCan't connect to Orion API at https://<internal-url>/api. Check that it's accessible from your machine.
even though the connection between the API and UI works fine and we can see the data in the UI. Could that just be a false warning? Any way to get rid of it? We set the url via an env var as part of the k8s deploymentTony Popov
09/22/2022, 9:26 PMKubernetesJob
block work? There’s almost no info in the docs and github examples are quite confusing.
Context is setting up the CI deployment of Prefect flowsTony Popov
09/22/2022, 9:49 PMprefect-cli
talks to orion
via kubectl port-forward
2. prefect-agent
knows about orion
via the k8s service (setting api url env var)
Hence the question: how does prefect deploy
command interact with an prefect-agent
if orion
does not know about agent as well?Tony Popov
09/22/2022, 11:47 PMprefect deployment build ./path/to/flow.py …
cli uploads all files in ./
to Storage