Joseph Haaga
11/12/2020, 4:02 PMDockerStorage.create_dockerfile_object(directory)
contains a COPY statement to add healthcheck.py
to the image, but uses an absolute path (which docker build
treats as a relative path from the build context)
COPY /absolute/path/on/my/machine/healthcheck.py /opt/prefect/healthcheck.py
This causes the following error:
Step 6/16 : COPY /absolute/path/on/my/machine/healthcheck.py /opt/prefect/healthcheck.py
COPY failed: stat /var/lib/docker/tmp/docker-builder852791584/absolute/path/on/my/machine/healthcheck.py: no such file or directory
However, this works fine when I manually update the Dockerfile to use a relative path to healthcheck.py
Would this constitute an issue? Or am I circumventing an intentional design decision?bral
11/12/2020, 4:03 PMZach
11/12/2020, 4:33 PM"RANDOM_TOOL_VERSION": ": Guppy Basecalling Software, (C) Oxford Nanopore Technologies, Limited. Version 4.2.2+effbaf84"
Here is a sample flow registration that is similar to what I am doing:
(The error I was getting is in a comment at the bottom of the file)ale
11/12/2020, 4:35 PMNicolas Bigaouette
11/12/2020, 6:48 PMreplica
setting. In addition, the app is behind guvicorn. All this means that we have multiple of our backends running concurrently. As such, if we register a flow in each backends prefect will receive multiple register request for the same flow, which is obviously wrong...
How should I handle this use case? How can I have multiple instances of my application that use the same flow? We though of performing a search for the flow and creating it if not present. But then the flow's name (or anything that is used to perform the search) would become the unique key to identify a flow. From what I'm reading about prefect a flow name is not the unique key to identify a flow.
Any idea?
Thanks!!Pedro Machado
11/12/2020, 9:14 PMLuke Orland
11/12/2020, 9:30 PMSerializer
that serializes str
?Riley Hun
11/13/2020, 12:30 AMstorage.build()
. I get the following error:
InterruptedError: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: <https://cloud.google.com/container-registry/docs/advanced-authentication>
I don't think this is a permissions issue to the GCR images because I tried building the docker file and pushing it using gcloud commands, and it worked successfully.
The flow also failed on my prefect core server deployed on GCP with the same kind of authentication error:Pedro Machado
11/13/2020, 2:49 AMM Taufik
11/13/2020, 4:03 AMPedro Machado
11/13/2020, 6:21 AMidempotency_key=flow.serialized_hash()
when registering a flow. The hash is changing even when nothing has changed in the flow. Any ideas?Joël Luijmes
11/13/2020, 8:25 AMSteven Hamblin
11/13/2020, 9:20 AMsimone
11/13/2020, 2:37 PMFailed to load and execute Flow's environment: ModuleNotFoundError(
I followed the instruction seen in other slacks discussion on the error and restarted the Agent pointing to the path to my package p`refect agent local start --api http://172.19.0.6:4200 -p /home/simone/tmp_code/pysmFISH_auto -p /home/simone/tmp_code/pysmFISH_auto/pysmFISH --show-flow-logs` This seems to solve the module error but the flow doesn’t run. Looking at the logs from the Agent is because ModuleNotFoundError: No module named 'numpy'
, numpy is install in the env. If I restart the agent adding the path to the conda env I am using prefect agent local start --api <http://172.19.0.6:4200> -p /home/simone/tmp_code/pysmFISH_auto -p /home/simone/tmp_code/pysmFISH_auto/pysmFISH -p /home/simone/miniconda3/envs/prefect_dev --show-flow-logs
I get again the same initial error of missing module.
I really cannot understand what is going on. Any help is really appreciated! thanks!
Here is the flow I have been running Flow GIST
------UPDATE---------
It looks like that the agent cannot load the flow from the local directory. The local directory is there and contains the flow. The error persist even if I change the directory where to save the flowsBen Fogelson
11/13/2020, 7:55 PMParameters
for some of its call method arguments. This can lead to a lot of boilerplate:
from prefect import Flow, Parameter, task
@task
def task_with_many_args(a, b, c, d, e=None, f=5, g='foo'):
pass
with Flow('flow') as flow:
a = some_other_task()
task_with_many_args(
a=a,
b=Parameter('task_with_many_args.b'),
c=Parameter('task_with_many_args.c'),
d=Parameter('task_with_many_args.d'),
e=Parameter('task_with_many_args.e', default=None),
f=Parameter('task_with_many_args.f', default=5),
g=Parameter('task_with_many_args.g', default='foo'),
)
This comes up often enough that we’ve written a helper function to populate these parameters by inspecting the task’s call signature.
with Flow('flow') as flow:
a = some_other_task()
call_and_populate_parameters(task_with_many_args, a=a)
These two examples produce identical flows.
What I’m wondering from the Prefect team is whether there’d be interest in incorporating some sort of populate_paramaters
kwarg directly into the Task.__call__
method. Happy to take a stab at implementing it myself, but wanted to check before I put in the effort.Newskooler
11/13/2020, 8:21 PMValueError: Could not infer an active Flow context.
I find it quite cryptic. Can anyone please shed some light on what my issue may be? I have no clue right now : /Joseph Solomon
11/13/2020, 9:06 PMDJ Erraballi
11/14/2020, 9:57 AMDJ Erraballi
11/14/2020, 9:57 AMDJ Erraballi
11/14/2020, 9:58 AMMarwan Sarieddine
11/14/2020, 4:18 PMclient.create_flow_run
which is how we programmatically run flows ...bral
11/14/2020, 6:51 PMHello! Can CASE be used with Parameter? I need, if the parameter was passed, then perform some tasks, otherwise others
Felix Vemmer
11/14/2020, 10:56 PMpython3 get_medium_stats.py
in the terminal which executes these parts in the script…
flow.storage = Local(
path=os.path.abspath(python_file),
stored_as_script=True
)
flow.register(project_name="Social Media Automation")
flow.run_agent()
As expected I can see that the agent is running in the terminal.
Checkin the cloud I can also confirm that the local agent is running:
AGENT ID: 1132a46f-8016-4880-b028-6616487a1785
Finally, also running prefect get flows
shows the flow that I want to run: fa7eb2ff-1220-44bd-b302-14fe7f8e3a22 Social Media Automation
However, the flow is stuck in submission and doesn’t execute. However I noticed when I stop the script/local agent in the terminal with ctrl + c
at timestamp [2020-11-14 22:49:50,984] INFO - agent | Keyboard Interrupt received: Agent is shutting down.
I can see in the logs in the cloud that the process now kicks off 23:49:52 Beginning Flow run for 'Medium Stats Scraper'
.
So it seems that something is blocking the execution 🧐
I would very much appreciate any help, thanks!Michael Hadorn
11/16/2020, 8:31 AM[2020-11-16 09:26:55+0100] ERROR - prefect.Load | Unexpected error occured in FlowRunner: ModuleNotFoundError("No module named 'xxx'")
I think I have to upload all my necessary files via: client.upload_file('xxx.py')
(see https://distributed.dask.org/en/latest/api.html#distributed.Client.upload_file)
But how can i access the dask client?
Here the client would be available while the run: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/engine/executors/dask.py#L257
But directly after, it's cleared again.
Does anybody have a hint for me?
Regards MichaelRiley Hun
11/17/2020, 12:56 AMPedro Machado
11/17/2020, 1:55 AMMichelle Wu
11/17/2020, 7:59 AMexecutor = DaskExecutor(address="<tcp://xxx>")
flow.run(executor = executor)
But when I registered the flows to run it in Prefect UI, they stopped running on my local Dask cluster. Neither scheduler or workers showed any reactions in terminal or in Bokeh. I wonder why this is happening?Alexander
11/17/2020, 3:29 PMFlow run RUNNING: terminal tasks are incomplete.
But all tasks within flow are success
. How can i debug this?Zach
11/17/2020, 7:47 PMgcs_creds_secret = PrefectSecret("GCS_SP_CREDS")
Zach
11/17/2020, 7:48 PM