Joseph Haaga11/12/2020, 4:02 PM
contains a COPY statement to add
to the image, but uses an absolute path (which
treats as a relative path from the build context)
This causes the following error:
COPY /absolute/path/on/my/machine/healthcheck.py /opt/prefect/healthcheck.py
However, this works fine when I manually update the Dockerfile to use a relative path to
Step 6/16 : COPY /absolute/path/on/my/machine/healthcheck.py /opt/prefect/healthcheck.py COPY failed: stat /var/lib/docker/tmp/docker-builder852791584/absolute/path/on/my/machine/healthcheck.py: no such file or directory
Would this constitute an issue? Or am I circumventing an intentional design decision?
bral11/12/2020, 4:03 PM
Zach11/12/2020, 4:33 PM
Here is a sample flow registration that is similar to what I am doing: (The error I was getting is in a comment at the bottom of the file)
"RANDOM_TOOL_VERSION": ": Guppy Basecalling Software, (C) Oxford Nanopore Technologies, Limited. Version 4.2.2+effbaf84"
ale11/12/2020, 4:35 PM
Nicolas Bigaouette11/12/2020, 6:48 PM
setting. In addition, the app is behind guvicorn. All this means that we have multiple of our backends running concurrently. As such, if we register a flow in each backends prefect will receive multiple register request for the same flow, which is obviously wrong... How should I handle this use case? How can I have multiple instances of my application that use the same flow? We though of performing a search for the flow and creating it if not present. But then the flow's name (or anything that is used to perform the search) would become the unique key to identify a flow. From what I'm reading about prefect a flow name is not the unique key to identify a flow. Any idea? Thanks!!
Pedro Machado11/12/2020, 9:14 PM
Luke Orland11/12/2020, 9:30 PM
Riley Hun11/13/2020, 12:30 AM
. I get the following error:
I don't think this is a permissions issue to the GCR images because I tried building the docker file and pushing it using gcloud commands, and it worked successfully. The flow also failed on my prefect core server deployed on GCP with the same kind of authentication error:
InterruptedError: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: <https://cloud.google.com/container-registry/docs/advanced-authentication>
Pedro Machado11/13/2020, 2:49 AM
M Taufik11/13/2020, 4:03 AM
Pedro Machado11/13/2020, 6:21 AM
when registering a flow. The hash is changing even when nothing has changed in the flow. Any ideas?
Joël Luijmes11/13/2020, 8:25 AM
Steven Hamblin11/13/2020, 9:20 AM
simone11/13/2020, 2:37 PM
I followed the instruction seen in other slacks discussion on the error and restarted the Agent pointing to the path to my package p`refect agent local start --api http://172.19.0.6:4200 -p /home/simone/tmp_code/pysmFISH_auto -p /home/simone/tmp_code/pysmFISH_auto/pysmFISH --show-flow-logs` This seems to solve the module error but the flow doesn’t run. Looking at the logs from the Agent is because
Failed to load and execute Flow's environment: ModuleNotFoundError(
, numpy is install in the env. If I restart the agent adding the path to the conda env I am using
ModuleNotFoundError: No module named 'numpy'
I get again the same initial error of missing module. I really cannot understand what is going on. Any help is really appreciated! thanks! Here is the flow I have been running Flow GIST ------UPDATE--------- It looks like that the agent cannot load the flow from the local directory. The local directory is there and contains the flow. The error persist even if I change the directory where to save the flows
prefect agent local start --api <http://172.19.0.6:4200> -p /home/simone/tmp_code/pysmFISH_auto -p /home/simone/tmp_code/pysmFISH_auto/pysmFISH -p /home/simone/miniconda3/envs/prefect_dev --show-flow-logs
Ben Fogelson11/13/2020, 7:55 PM
for some of its call method arguments. This can lead to a lot of boilerplate:
This comes up often enough that we’ve written a helper function to populate these parameters by inspecting the task’s call signature.
from prefect import Flow, Parameter, task @task def task_with_many_args(a, b, c, d, e=None, f=5, g='foo'): pass with Flow('flow') as flow: a = some_other_task() task_with_many_args( a=a, b=Parameter('task_with_many_args.b'), c=Parameter('task_with_many_args.c'), d=Parameter('task_with_many_args.d'), e=Parameter('task_with_many_args.e', default=None), f=Parameter('task_with_many_args.f', default=5), g=Parameter('task_with_many_args.g', default='foo'), )
These two examples produce identical flows. What I’m wondering from the Prefect team is whether there’d be interest in incorporating some sort of
with Flow('flow') as flow: a = some_other_task() call_and_populate_parameters(task_with_many_args, a=a)
kwarg directly into the
method. Happy to take a stab at implementing it myself, but wanted to check before I put in the effort.
Newskooler11/13/2020, 8:21 PM
I find it quite cryptic. Can anyone please shed some light on what my issue may be? I have no clue right now : /
ValueError: Could not infer an active Flow context.
Joseph Solomon11/13/2020, 9:06 PM
DJ Erraballi11/14/2020, 9:57 AM
DJ Erraballi11/14/2020, 9:57 AM
DJ Erraballi11/14/2020, 9:58 AM
Marwan Sarieddine11/14/2020, 4:18 PM
which is how we programmatically run flows ...
bral11/14/2020, 6:51 PM
Hello! Can CASE be used with Parameter? I need, if the parameter was passed, then perform some tasks, otherwise others
Felix Vemmer11/14/2020, 10:56 PM
in the terminal which executes these parts in the script…
flow.storage = Local(
flow.register(project_name="Social Media Automation")
As expected I can see that the agent is running in the terminal. Checkin the cloud I can also confirm that the local agent is running:
Finally, also running
AGENT ID: 1132a46f-8016-4880-b028-6616487a1785
shows the flow that I want to run:
prefect get flows
However, the flow is stuck in submission and doesn’t execute. However I noticed when I stop the script/local agent in the terminal with
fa7eb2ff-1220-44bd-b302-14fe7f8e3a22 Social Media Automation
ctrl + c
I can see in the logs in the cloud that the process now kicks off
[2020-11-14 22:49:50,984] INFO - agent | Keyboard Interrupt received: Agent is shutting down.
. So it seems that something is blocking the execution 🧐 I would very much appreciate any help, thanks!
23:49:52 Beginning Flow run for 'Medium Stats Scraper'
Michael Hadorn11/16/2020, 8:31 AM
I think I have to upload all my necessary files via:
[2020-11-16 09:26:55+0100] ERROR - prefect.Load | Unexpected error occured in FlowRunner: ModuleNotFoundError("No module named 'xxx'")
(see https://distributed.dask.org/en/latest/api.html#distributed.Client.upload_file) But how can i access the dask client? Here the client would be available while the run: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/engine/executors/dask.py#L257 But directly after, it's cleared again. Does anybody have a hint for me? Regards Michael
Riley Hun11/17/2020, 12:56 AM
Pedro Machado11/17/2020, 1:55 AM
Michelle Wu11/17/2020, 7:59 AM
But when I registered the flows to run it in Prefect UI, they stopped running on my local Dask cluster. Neither scheduler or workers showed any reactions in terminal or in Bokeh. I wonder why this is happening?
executor = DaskExecutor(address="<tcp://xxx>") flow.run(executor = executor)
Alexander11/17/2020, 3:29 PM
But all tasks within flow are
Flow run RUNNING: terminal tasks are incomplete.
. How can i debug this?
Zach11/17/2020, 7:47 PM
gcs_creds_secret = PrefectSecret("GCS_SP_CREDS")