Apoorva Desai
07/27/2022, 5:55 PMMansour Zayer
07/27/2022, 8:02 PMMars
07/27/2022, 9:08 PMAnna Geller
07/27/2022, 9:27 PMAndreas Nigg
07/27/2022, 11:16 PMprefect deployment build ../flows/r_script_automation.py:r_script_automation --name retention_cohort_analysis_deployment -t k8s -sb gcs/gcs-prefect-stprage -i kubernetes-job
If I then run "prefect apply..." and "prefect run..:", the job gets created, however it fails with:
Flow could not be retrieved from deployment.
FileNotFoundError: [Errno 2] No such file or directory: '/home/andreas/github/r_automation/flows/r_script_automation.py'
Note, that the directory /home/andreas.... is my local flow location, where I created the deployment from. It seems that somehow the job is not downloading the flow from GCS?
My GCS storage block configuration is correct (I guess), because all my subprocess-flows work - only the KubernetesJobs have this problem. Any tips for me? 🙂Seth Coussens
07/27/2022, 11:38 PMCharles Phares
07/27/2022, 11:40 PMSlackbot
07/28/2022, 12:49 AMKelvin
07/28/2022, 2:09 AMDenys Volokh
07/28/2022, 3:38 AMKV Store operations are only supported while using Prefect Cloud as a backend
and here is my setup:
1. My flow and run config using KubernetesRun:
with Flow(
name="Import Benchmark Index Data",
storage=Local(
path="/app/workflows/benchmarks/flow_import_index_data.py",
stored_as_script=True,
),
run_config=KubernetesRun(
image="prefect-workflows:latest",
image_pull_policy="Always",
),
) as flow:
benchmarks_states = task_fetch_benchmarks_state()
2. Docker file for prefect-workflows image
FROM prefecthq/prefect:1.2.4-python3.9
WORKDIR /app
ADD . .
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
RUN pip install -e .
RUN prefect backend cloud
3. Agent configuration:Denys Volokh
07/28/2022, 3:50 AMJeffrey Lai
07/28/2022, 5:33 AMViet Nguyen
07/28/2022, 6:43 AM_from_ prefect.utilities.asyncio _import_ Sync
hi everyone, I wonder if what is the equivalent thing for this in prefect 2.0 ? I'm trying to implement something similar to as_completed()
from Dask, thank you.Michiel Verburg
07/28/2022, 7:17 AMRajvir Jhawar
07/28/2022, 9:22 AMAndreas Nigg
07/28/2022, 9:26 AMtest
|___ deployment.ymal
|___ kayak_database_flows.py
My build command is (run from inside the folder "test")
prefect deployment build kayak_database_flows.py:integrate_kayak_details --name integrate_kayak_db_details -t rmesbtest1 -sb gcs/gcs-prefect-stprage
Then i apply the deployment and run the deployment. Everything fine so far, the flow get's picked up by the corresponding agent
However, the agent downloads ALL of the content of the GCS bucket. So really all of the content in this bucket, not only the stuff related to this deployment. Is this expected?
If so, do I need a separate GCS storage block for any of the deployments?David
07/28/2022, 10:03 AMPaul Lucas
07/28/2022, 10:14 AMAndreas
07/28/2022, 10:45 AMprefect deployment create deployments.yaml
we were able to deploy them all at once in Prefect Server. I have seen that there are a lot of changes in GA compared to latest beta and I took a look at the updated docs about it. However I think that some (important to me) functionality is missing compared to what we had in beta. Is there an easy way in the new GA deployments procedure to have multiple flows in a yaml file or an alternative way to deploy multiple flows at once without having to call prefect deployment build ......
and prefect deployment apply deployment.yaml
manually for each single flow that we have?Tarek
07/28/2022, 11:21 AMlabels='prod'
however the registered flow also uses the id of the docker instance it is built into (See screenshot): Is there anyway to delete that, so that only the labels I give are used?haris khan
07/28/2022, 11:31 AMChu
07/28/2022, 11:42 AMash
07/28/2022, 12:06 PMexit error 2, Instance terminated
on dashboard and on pod logs we are getting No such command "execute"
Riccardo Tesselli
07/28/2022, 12:09 PMJehan Abduljabbar
07/28/2022, 12:26 PMNelson Griffiths
07/28/2022, 12:59 PMLucien Fregosi
07/28/2022, 1:00 PMAlix Cook
07/28/2022, 1:24 PMT B
07/28/2022, 1:32 PMlocal file system
storage/block: my-local-storage.
However when I try to build with the parameter "prefect deployment build --storage-block local-file-system/my-local-storage" I get an exception :
File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 114, in put_directory
shutil.copytree(from_path, local_path, dirs_exist_ok=True)
NameError: name 'from_path' is not defined
Looking at the code : https://github.com/PrefectHQ/prefect/blob/2.0.0/src/prefect/filesystems.py#L114
it seems the variable "from_path" is indeed not defined.Troels Bjørnskov
07/28/2022, 1:47 PMprefect_test_harness
located? It tries to connect to a server, I want to skip that stepTroels Bjørnskov
07/28/2022, 1:47 PMprefect_test_harness
located? It tries to connect to a server, I want to skip that stepBianca Hoch
07/28/2022, 2:28 PMprefect_test_harness
here. I believe there is an open issue pertaining to prefect_test_harness attempting to connect to the server as well. I mentioned this to the team, so expect a resolution soon.Troels Bjørnskov
07/28/2022, 2:47 PMPREFECT_API_URL
is set in the environment, it will try to connect to the api regardless of prefect_test_harness
being used.Anna Geller
07/28/2022, 2:48 PMTroels Bjørnskov
07/28/2022, 2:55 PMAnna Geller
07/29/2022, 12:33 AM