matta
06/15/2020, 11:08 PMpsimakis
06/16/2020, 9:45 AMDarragh
06/16/2020, 1:20 PMAn error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.
Appears to be amazon complaining about the job family name, but I don’t know if there’s a way around it in the agent config.Thomas Hoeck
06/16/2020, 2:43 PMMatthias
06/16/2020, 3:07 PMcheckpoint=False
but still data is stored in the results folder. Do you know of any other place I need to check/set something?Sandeep Aggarwal
06/16/2020, 3:59 PMRafal
06/16/2020, 5:32 PMDylan
Dylan
jars
06/17/2020, 3:23 AMfrom lib.package1 import fn1
from lib.package2 import fn2
These correspond to files alongside my flow like: lib/package1.py
and lib/package2.py
.
When calling flow.register()
it looks like my Docker Storage healthcheck fails because it cannot find the lib module...
Traceback (most recent call last):
File "/opt/prefect/healthcheck.py", line 135, in <module>
flows = cloudpickle_deserialization_check(flow_file_path)
File "/opt/prefect/healthcheck.py", line 40, in cloudpickle_deserialization_check
flows.append(cloudpickle.load(f))
ModuleNotFoundError: No module named 'lib'
Any tips on how to make that work?Eamon Keane
06/17/2020, 10:10 AMenv_var
with 10 parameters, fan out to 10 pods when an initial task has finished. Basically I mainly want to use prefect as a plain orchestrator of pods like airflow pod operator.
I see in the docs it documents a similar use case with Dask, but couldn't see an example which did similar with kubernetes pods.Darragh
06/17/2020, 11:46 AMprefecthq/prefect:master
in flows - it has a feature I need [DFE execution on mapped tasks], but I can’t seem to figure out how to actually use master. I’ve updated both my Prefect Server and the image I base all our flows off [FROM prefecthq/prefect:master
] but even with all that I still see the following when I build the flow storage:
Step 3/9 : RUN pip show prefect || pip install git+<https://github.com/PrefectHQ/prefect.git@0.11.5#egg=prefect[kubernetes]>
And in the UI it still reports 0.11.5 as Prefect Core Version, and the expected DFE mapping behaviour is not there, i.e. mapped tasks are executed one by one.
.Any ideas on how to get around this?Sandeep Aggarwal
06/17/2020, 12:28 PMStanley Ndagi
06/17/2020, 12:49 PMDylan
josh
06/17/2020, 5:54 PM0.12.0
has been released! Here are a few notable highlights since `0.11.0`:
🗺️ Depth-first execution for mapping
🎈 Lightweight Dask serialization for lower memory footprint
🌡️ API health check tile on server UI
💎 Configure volumes in server for persistent orchestration state
📑 Task signature tab completion
🔎 New Great Expectations task library task
🛠️ Lots of enhancements and fixes
Read the full changelog here: https://github.com/PrefectHQ/prefect/releases/tag/0.12.0
Special thanks to all of the contributors who have assisted in discussions, issues, and PRs that have contributed to this release!jars
06/18/2020, 3:02 AMTraceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 986, in get_task_run_state
result = self.result.write(value, filename="output", **prefect.context)
File "/usr/local/lib/python3.7/site-packages/prefect/engine/results/gcs_result.py", line 73, in write
self.gcs_bucket.blob(new.location).upload_from_string(binary_data)
File "/usr/local/lib/python3.7/site-packages/prefect/engine/results/gcs_result.py", line 35, in gcs_bucket
from prefect.utilities.gcp import get_storage_client
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/gcp.py", line 6, in <module>
from google.cloud import bigquery, storage
ImportError: cannot import name 'bigquery' from 'google.cloud' (unknown location)
It seems the process cannot find a prefect core bigquery module inside of google.cloud.
I thought perhaps it had something to do with setting PYTHONPATH
to my own application & lib directories in my Flow's Dockerfile:
ENV PYTHONPATH="/app:/app/lib"
And now Prefect core cannot find it's own packages?
But experimenting with the alternative "extension" of PYTHONPATH
instead of overwrite:
ENV PYTHONPATH="/app:/app/lib:${PYTHONPATH}"
simply yields a PYTHONPATH
with a colon on the end (/app:/app/lib:
), so I opted not to extend, since there is no default.
My suspicions about PYTHONPATH
could very well be a red herring... Any ideas?Emmanuel Klinger
06/18/2020, 8:05 AMAndrey Alekseev
06/18/2020, 8:40 AMRafal
06/18/2020, 9:15 AMHoward Cornwell
06/18/2020, 2:06 PMHoward Cornwell
06/18/2020, 2:06 PMwilsojb
06/18/2020, 2:09 PMprefect server start
where its not running in the foreground? I was expecting to find a -d
option (similar to docker-compose). I'm sure this has been asked before - just looking to be pointed in the right direction! Thanks! 🙇Jackson Maxfield Brown
06/18/2020, 4:13 PM0.12.0
, all tasks on dask
/ bokeh
are labeled as "run_task" instead of the function name. Did the 0.12.0
update make it so I need to explicitly attach a name to each task?
bokeh==2.0.2
dask==2.18.1
distributed==2.18.0
Lewis Bails
06/18/2020, 5:11 PMKevin Weiler
06/18/2020, 5:58 PMChris Vrooman
06/18/2020, 6:39 PMChris Vrooman
06/18/2020, 6:39 PMjames.lamb
06/18/2020, 7:56 PMKubernetesAgent
(https://github.com/PrefectHQ/prefect/pull/2796). I'm struggling with something and hoping someone can help.
I have the following setup:
1. Flow code uses KubernetesJobEnvironment
+ S3
storage, using flow.register()
to register flows with Prefect Cloud
2. Running a KubernetesAgent
I can see that the agent is successfully communicating with Prefect Cloud. When I run a flow from the Prefect Cloud UI, I can see it in the agent's logs and I see a Kubernetes job created. That job is now failing with this error
[2020-06-18 194454] INFO - prefect.S3 | Downloading hello-flow-w-s3-k8s-env/2020-06-18t19-18-00-707752-00-00 from prefect-d94f436a-25b1-1699546c3
... big stacktrace ...
botocore.exceptions.NoCredentialsError: Unable to locate credentialsMy flow code is just the hello world example that prints to the
prefect
logger, so that error must come from the prefect
code that is trying to pull the flow from storage in S3. In the manifest for the KubernetesAgent
, I've set up the environment based on the directions in https://docs.prefect.io/core/concepts/secrets.html#default-secrets
env:
- name: PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS
value: '{"ACCESS_KEY": "REDACTED", "SECRET_ACCESS_KEY": "REDACTED"}'
I expected that setting this on the agent would mean that every job it creates has access to those credentials to download the flow. What am I doing wrong? Happy to provide more context in thread.Will Milner
06/18/2020, 8:49 PM