Vincent
10/21/2020, 7:54 PMBilly McMonagle
10/21/2020, 9:37 PMMike Marinaccio
10/21/2020, 9:42 PMprefect.context.parameters
. Am I correct to say that I can always rely on the context params being set before any other task is run in an async / Dask environment?Jimmy Le
10/21/2020, 10:20 PMTypeError: cannot pickle '_thread.lock' object
.
Has anyone run into a similar problem? Any suggestions would be appreciated!Zach
10/21/2020, 11:35 PMZach
10/21/2020, 11:36 PMPrefectResult()
result handler doesn't work since a class instance is not JSON serializableZach
10/21/2020, 11:37 PMbral
10/22/2020, 3:29 AMAlberto de Santos
10/22/2020, 2:14 PMnohup
, however, in order to do that, I have to execute everything with the flow.run_agent()
option and then comment that line of code to execute the flows.
The alternative is through CLI, however, I couldn’t make it work (it doesn’t file the libraries I need to execute, despite of using the -p
param)
Which is your experience/view here?Dennis Schneidermann
10/22/2020, 2:31 PMJames Phoenix
10/22/2020, 3:55 PMJames Phoenix
10/22/2020, 3:55 PMJames Phoenix
10/22/2020, 3:56 PMJames Phoenix
10/22/2020, 3:56 PMJames Phoenix
10/22/2020, 3:56 PMNewskooler
10/22/2020, 5:29 PMdef test_smth():
with Flow("test") as flow:
outcome = my_funct()
state = flow.run()
assert state.result[outcome].is_failed()
ale
10/22/2020, 5:59 PMBilly McMonagle
10/22/2020, 6:02 PMFargateAgent
. I am running this agent locally, and it seems to work intermittently. However, I am repeatedly getting this error:
[2020-10-22 17:59:48,212] INFO - agent | Starting FargateAgent with labels ['XXX']
[2020-10-22 17:59:48,212] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-10-22 17:59:48,212] INFO - agent | Agent connecting to the Prefect API at <https://api.prefect.io>
[2020-10-22 17:59:48,269] INFO - agent | Waiting for flow runs...
[2020-10-22 17:59:48,402] ERROR - agent | [{'path': ['get_runs_in_queue'], 'message': "'NoneType' object has no attribute 'flow_group_id'", 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
Mitchell Bregman
10/22/2020, 7:54 PM# $ROOT/src/flow.py
from prefect import Flow
flow = Flow(name="test flow")
Now from the command line in dir $ROOT
, I run:
prefect register flow --file src/flow.py --name "test flow"
I get the error
KeyError: "'__name__' not in globals"
What am I doing wrong? I’d like to use the CLI as it will be more simplistic from a CI/CD standpointfabian wolfmann
10/22/2020, 9:40 PMAlberto de Santos
10/22/2020, 10:06 PMJesper van Dijke
10/22/2020, 11:50 PMauth0-spa-js must run on a secure origin
Which of course is completely ok because not running on localhost but rather on an ubuntu image and connect to <http://192.168.88.133:8080>
in chrome address bar <chrome://flags/#unsafely-treat-insecure-origin-as-secure>
added this domain.
No luck, next error : Refused to frame '<https://login.prefect.io/>' because an ancestor violates the following Content Security Policy directive: "frame-ancestors 'none'".
Anyone solved or run it on a different machine?
Alternatively I end up running SSH and tunnel to it...
Putty, ssh tunnel forward 8080 and 4200, and all is good.
Maybe a nice addendum to the documentation.Scott Asher
10/23/2020, 2:54 AMprefect.utilities.exceptions.ClientError: Malformed response received from Cloud - please ensure that you have an API token properly configured.
Scott Asher
10/23/2020, 2:54 AMScott Asher
10/23/2020, 3:13 AMflow.register()
Alberto de Santos
10/23/2020, 9:29 AMAlberto de Santos
10/23/2020, 9:30 AMmap
over a DataFrame
, do you have any idea? I see very clear how to make it over a list
. To my mind, it seems straightforward to convert the DataFrame
into a list
. So, I would like to know your opinion.Zach
10/23/2020, 2:55 PMFailed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': "can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE", 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 111, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1177, in set_task_run_state
result = self.graphql(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 226, in graphql
raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': "can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE", 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
I ran this flow 89 times with different input, and one time it failed with this message.Zach
10/23/2020, 3:46 PMMitchell Bregman
10/23/2020, 6:09 PMModuleNotFoundError: No module named src
during the flow healthcheck, traceback here. Am I doing something wrong in terms of __init__
packaging? This is a followup to thread yesterday.Mitchell Bregman
10/23/2020, 6:09 PMModuleNotFoundError: No module named src
during the flow healthcheck, traceback here. Am I doing something wrong in terms of __init__
packaging? This is a followup to thread yesterday.nicholas
10/23/2020, 6:13 PMsrc
is referenced, it's from within the src
directory, shouldn't the __init__
module reference it with from flow import Flow
?Mitchell Bregman
10/23/2020, 6:14 PMMichael Adkins
10/23/2020, 6:15 PMMitchell Bregman
10/23/2020, 6:15 PMMichael Adkins
10/23/2020, 6:16 PMsrc
init file you should still reference the full path to the moduleMitchell Bregman
10/23/2020, 6:16 PM"""Top-level module."""
from src.flow import flow
__all__ = ["flow"]
flow.py
to build.py
Michael Adkins
10/23/2020, 6:17 PM/opt/prefect/healthcheck.py:147: UserWarning: Flow uses module which is not importable. Refer to documentation on how to import custom modules <https://docs.prefect.io/api/latest/environments/storage.html#docker>
flows = cloudpickle_deserialization_check(flow_file_paths)
src
which is not installed within the docker containerMitchell Bregman
10/23/2020, 6:18 PMimport src
all works just fineMichael Adkins
10/23/2020, 6:18 PMpip install -e
is not run within the docker container thoughMitchell Bregman
10/23/2020, 6:19 PMnicholas
10/23/2020, 6:20 PMsrc
folder to the docker container?Michael Adkins
10/23/2020, 6:20 PMextra_dockerfile_commands
kwarg or include your module like so
Docker(
files={
# absolute path source -> destination in image
"/Users/me/code/mod1.py": "/modules/mod1.py",
"/Users/me/code/mod2.py": "/modules/mod2.py",
},
env_vars={
# append modules directory to PYTHONPATH
"PYTHONPATH": "$PYTHONPATH:modules/"
},
)
Mitchell Bregman
10/23/2020, 6:22 PMMichael Adkins
10/23/2020, 6:23 PM"/path/in/ci/to/module": "/modules"
Mitchell Bregman
10/23/2020, 6:24 PMMichael Adkins
10/23/2020, 6:24 PMpip install -e /modules/yourmodule
(via the extra cmds) or add it to the PYTHONPATH using env_vars
Mitchell Bregman
10/23/2020, 6:38 PMextra_dockerfile_commands="pip install -e /modules",
files={f"{os.path.join(os.path.expanduser('~'), 'project')}": "/modules"},
flow.storage = Docker(
env_vars=config.ENVIRONMENT_VARIABLES,
extra_dockerfile_commands="pip install -e /modules",
files={f"{os.path.join(os.path.expanduser('~'), 'project')}": "/modules"},
image_name=config.DOCKER_IMAGE_NAME,
image_tag=config.DOCKER_IMAGE_TAG,
python_dependencies=config.PYTHON_DEPENDENCIES,
registry_url="<http://parkmobile-docker.jfrog.io|parkmobile-docker.jfrog.io>",
tls_config=tls_config,
)
os.path.join(os.path.expanduser('~'), 'project'
resolves to home/circleci/project
(all the code if you were to clone it lives here)/modules
Michael Adkins
10/23/2020, 6:40 PMMitchell Bregman
10/23/2020, 6:40 PMdocker.errors.APIError: 400 Client Error: Bad Request ("Dockerfile parse error line 16: unknown instruction: P")
Michael Adkins
10/23/2020, 6:41 PMMitchell Bregman
10/23/2020, 6:41 PMMichael Adkins
10/23/2020, 6:41 PM["RUN …"]
Mitchell Bregman
10/23/2020, 6:41 PMMichael Adkins
10/23/2020, 6:53 PMMiha Sajko
12/19/2020, 4:56 PMfiles
argument as discussed in this thread or is there a better way?Michael Adkins
12/19/2020, 6:14 PMfrom my_project import PROJECT_PATH, PROJECT_NAME
from prefect.storage.docker import Docker
def ProjectDockerStorage(
project_path: str = PROJECT_PATH, project_name: str = PROJECT_NAME, **kwargs
) -> Docker:
"""
A thin wrapper around `prefect.storage.Docker` with installation of a local project,
defaulting to installing this project
Cannot be a class because then it is not a known serializable storage type so this
is just an instance factory for Docker storage
"""
# Copy this namespace into the docker image
kwargs.setdefault("files", {})
kwargs["files"][str(project_path)] = project_name
# Install the namespace so it's on the Python path
kwargs.setdefault("extra_dockerfile_commands", [])
kwargs["extra_dockerfile_commands"].append(f"RUN pip install -e {project_name}")
return Docker(**kwargs)
then
flow.storage = ProjectDockerStorage()
Sagun Garg
12/28/2020, 8:21 AM