Hi! My flow runs all tasks but does not send the Flow Succesful status before going to 'exited', the agent log in the environment looks like this:

08:57:40.650 | INFO    | Task run 'task_name-6b80d8b6-24' - Finished in state Completed()
08:57:59.061 | INFO    | prefect.infrastructure.docker-container - Docker container 'adaptable-quoll' has status 'exited'

And the Prefect Cloud log like this:

Created task run 'task_name-6b80d8b6-24' for task 'task_name'
10:57:39 AM
Executing 'task_name-6b80d8b6-24' immediately..

The flow run is stuck at 'running' in the Prefect Cloud UI. This is only the case for one of my flows, but the final tasks of the flow are the same tasks as for my other flows.

Hey all, am currently running Prefect 2.3.2 on a minikube cluster and while the agent connects to Orion just fine, it does not seem to be reading jobs from the queue. I have a pending job and the agent just doesn’t pick it up. Any ideas?

Hello, Is it possible to implement something like DateClock in Prefect 2? I was trying to implement custom Schedule, but getting validation error:

pydantic.error_wrappers.ValidationError: 9 validation errors for Deployment
schedule -> interval
  field required (type=value_error.missing)
schedule -> calendar_id
  extra fields not permitted (type=value_error.extra)
schedule -> run_times
  extra fields not permitted (type=value_error.extra)
schedule -> cron
  field required (type=value_error.missing)
schedule -> calendar_id
  extra fields not permitted (type=value_error.extra)
schedule -> run_times
  extra fields not permitted (type=value_error.extra)
schedule -> rrule
  field required (type=value_error.missing)
schedule -> calendar_id
  extra fields not permitted (type=value_error.extra)
schedule -> run_times
  extra fields not permitted (type=value_error.extra)

Heya! Probably a really stupid question, but how do I pass arguments to deployment runs via the CLI? I couldn’t figure out a way to do it yet. What I am looking for is something like this:

prefect deployment run flow/deployment paramA=1 paramB=2

Am I missing something super obvious here? 😅 Thanks!

Hi prefect friends, I haven’t checked in in a bit. Do we have

Result

equivalent in prefect 2.0? 🙂

Hi. Does RRuleSchedule support something like? I was able to deploy the flow, but don't see any scheduled flow runs.

RDATE:20220920T200000Z,20220921T133000Z

UI says: 'Every year' schedule:

Is the helm prefect-agent version 1.0.0 released?

Hi everyone, I am trying to run a simple task against a spark cluster using Prefect 1.2 and i get this error:

TypeError: cannot pickle '_thread.RLock' object

I can see the application being registered inside the spark cluster. Thanks!

Greetings friends; great product! I’m having a lot of fun diving in. I was looking into using an ECSTask infra block, and I observed that there’s no

conda

package for

prefect-aws

. I can just use

pip

to keep moving forward, but I was curious about the inconsistency. Other prefect packages are available.

Hi Team - In Blocks - Email Server Credentials. What needs to be put into the Smtp Type and smtp server for office 365?

Hi team! How come Docker is no longer a storage option? Is the local file a replacement for this? I remember reading in the docs that Docker was no longer required but it looks like a replacement is still needed for storage https://docs.prefect.io/api-ref/prefect/filesystems/#prefect.filesystems.GitHub

I'm currently looking at this example https://discourse.prefect.io/t/how-can-i-run-my-flow-in-a-docker-container/64 which looks pretty similar to what I have in 1.0, except I use Docker for storage and ECSRun as my run config. What would this look like in 2.0?

with Flow("data_pipeline_test_new_account", storage=STORAGE, run_config=RUN_CONFIG) as data_pipeline_test_new_account:

Hi Team - I'm currently attempting to login to prefect via the docker image (DockerFile). I'm having an issue where it is asking for the profile name. Initially, there was an issue with the workspace, but I added that in the same line as the login. I'm not sure how to resolve this, because there is not a parameter for profile name like there is for workspace. Is there another way to do this?

RUN /bin/sh -c "prefect cloud login -k $(cat ./api_key.txt) --workspace $(cat ./workspace.txt)"

Hi All, With prefect 2.4.0, prefect-dask 0.2.0 I am receiving Internal server error and these errors are random, sometime the flow works fine next time flow crashes with

internal server error

. Other errors that appears from agent logs are,

RunTime error: The connection pool was closed while 9 HTTP requests/responses were still in flight.

Hi - we have an API that’s trying to register and run flows programatically via Prefect 1 Cloud, using the Prefect Python lib. We need to do this against two different Tenants in Prefect Cloud. This leads to an issue with credentials - since Prefect’s credentials seemingly need to be stored globally either in an env var or in a file, trying to switch between tenant auths while concurrently publishing multiple flows (eg one to tenant 1, one to tenant 2) is leading to a race condition. We might be able to work around this via multiprocessing but is there a cleaner way to do it, eg some way to set auth per call rather than globally?

Hi all, while evaluating orchestration engines, I discovered prefect today. So I decided to give it a whirl, and the pip install failed on both Windows 10 and Mac OS 12.4. I'm running python 10.3.6 on both This is the Mac output after running "pip install --pre prefect" for Mac: × Running setup.py install for aiohttp did not run successfully. │ exit code: 1 ╰─> [511 lines of output] ********************** * Accellerated build * ********************** /Users/danhorne/Env/prefect/lib/python3.10/site-packages/setuptools/config/setupcfg.py508 SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead. warnings.warn(msg, warning_class) running install /Users/danhorne/Env/prefect/lib/python3.10/site-packages/setuptools/command/install.py34 SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( running build running build_py creating build creating build/lib.macosx-12-arm64-cpython-310 creating build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_ws.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/worker.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/multipart.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_response.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/signals.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/client_ws.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/test_utils.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/tracing.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_exceptions.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_middlewares.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/http_exceptions.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_app.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/streams.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_protocol.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/log.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/client.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_urldispatcher.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_request.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/http_websocket.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/client_proto.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/locks.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/__init__.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_runner.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_server.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/base_protocol.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/payload.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/client_reqrep.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/http.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_log.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/resolver.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/formdata.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_routedef.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/connector.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/client_exceptions.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/typedefs.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/hdrs.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/web_fileresponse.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/http_writer.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/tcp_helpers.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/helpers.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/frozenlist.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/http_parser.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/cookiejar.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/pytest_plugin.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/abc.py -> build/lib.macosx-12-arm64-cpython-310/aiohttp running egg_info writing aiohttp.egg-info/PKG-INFO writing dependency_links to aiohttp.egg-info/dependency_links.txt writing requirements to aiohttp.egg-info/requires.txt writing top-level names to aiohttp.egg-info/top_level.txt reading manifest file 'aiohttp.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching 'aiohttp' anywhere in distribution warning: no previously-included files matching '*.pyc' found anywhere in distribution warning: no previously-included files matching '*.pyd' found anywhere in distribution warning: no previously-included files matching '*.so' found anywhere in distribution warning: no previously-included files matching '*.lib' found anywhere in distribution warning: no previously-included files matching '*.dll' found anywhere in distribution warning: no previously-included files matching '*.a' found anywhere in distribution warning: no previously-included files matching '*.obj' found anywhere in distribution warning: no previously-included files found matching 'aiohttp/*.html' no previously-included directories found matching 'docs/_build' adding license file 'LICENSE.txt' writing manifest file 'aiohttp.egg-info/SOURCES.txt' copying aiohttp/_cparser.pxd -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_find_header.c -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_find_header.h -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_find_header.pxd -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_frozenlist.c -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_frozenlist.pyx -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_headers.pxi -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_helpers.c -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_helpers.pyi -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_helpers.pyx -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_http_parser.c -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_http_parser.pyx -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_http_writer.c -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_http_writer.pyx -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_websocket.c -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/_websocket.pyx -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/frozenlist.pyi -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/py.typed -> build/lib.macosx-12-arm64-cpython-310/aiohttp copying aiohttp/signals.pyi -> build/lib.macosx-12-arm64-cpython-310/aiohttp running build_ext building 'aiohttp._websocket' extension creating build/temp.macosx-12-arm64-cpython-310 creating build/temp.macosx-12-arm64-cpython-310/aiohttp clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/Users/danhorne/Env/prefect/include -I/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c aiohttp/_websocket.c -o build/temp.macosx-12-arm64-cpython-310/aiohttp/_websocket.o aiohttp/_websocket.c216622: warning: '_PyUnicode_get_wstr_length' is deprecated [-Wdeprecated-declarations] (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : ^ /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10/cpython/unicodeobject.h2617: note: expanded from macro 'PyUnicode_GET_SIZE' PyUnicode_WSTR_LENGTH(op) : \ ^ /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10/cpython/unicodeobject.h45135: note: expanded from macro 'PyUnicode_WSTR_LENGTH' #define PyUnicode_WSTR_LENGTH(op) _PyUnicode_get_wstr_length((PyObject*)op) ^ /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10/cpython/unicodeobject.h4451: note: '_PyUnicode_get_wstr_length' has been explicitly marked deprecated here Py_DEPRECATED(3.3) ^ /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10/pyport.h51354: note: expanded from macro 'Py_DEPRECATED' #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((deprecated)) ^ aiohttp/_websocket.c216622: warning: 'PyUnicode_AsUnicode' is deprecated [-Wdeprecated-declarations] (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : ... lots more messages ... aiohttp/_http_parser.c2327619: error: implicit declaration of function '_PyGen_Send' is invalid in C99 [-Werror,-Wimplicit-function-declaration] ret = _PyGen_Send((PyGenObject*)yf, value == Py_None ? NULL : value); ^ aiohttp/_http_parser.c2327617: warning: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion] ret = _PyGen_Send((PyGenObject*)yf, value == Py_None ? NULL : value); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ aiohttp/_http_parser.c2336019: error: implicit declaration of function '_PyGen_Send' is invalid in C99 [-Werror,-Wimplicit-function-declaration] ret = _PyGen_Send((PyGenObject*)yf, NULL); ^ aiohttp/_http_parser.c2336017: warning: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion] ret = _PyGen_Send((PyGenObject*)yf, NULL); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 21 warnings and 5 errors generated. error: command '/usr/bin/clang' failed with exit code 1 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure × Encountered error while trying to install package. ╰─> aiohttp note: This is an issue with the package mentioned above, not pip. Has anyone else received these messages and resolved the problem? Thanks

Hi 👋 We have a GitHub block defined in the Prefect Cloud UI and it’s working. But the password is embedded inside the link repository like so: https://yuvital-render:xxxxx@github.com/yuvital/prefect.git Is there a way to make it work without embedding the password like this, or at least hide it in some way (for example like it’s done in the S3 Block)?

Suggestion to improve the tutorial at this page. For someone who is tinkering with Prefect and Docker containers for the first time it might not be obvious that the agent cannot be running with the ephemeral API. Specifically, here, the documentation only says to start an agent polling from the “test” queue. This works if the PREFECT_API_URL has been set before hand. It would be nice for the tutorial to include a note about setting this environment variable using the CLI or starting the agent with the --api flag set to the API endpoint.

Cached version of virtual env documentation states to use subflowrunnes to pass a env argument to the deploy, if you wish the flow to run in a different environment than your agent. Currently there's no documentation on the page and it returns 404 - Looking into subflowrunners, it's deprecated and states that you should use the process block. So far so good, but how would I declare my virtual env in the process block?

Hey! I want to schedule my flows with IntervalSchedule. How can I make first run right after the flow is deployed?

Hello everyone I have started using Prefect for the first time, and now I am in a dilemma whether to continue using it or not. I am developing a high performance REST API, and it has ETL functionality, so I thought Prefect would be best to orchestrate the whole workflow. Anyway, the problem is that with the same code, the response time of the API varies drastically based on I am using prefect or not. (I understand that the first run of the workflow will be very slow, but I have a problem with the subsequent invocations being very slow). I am attaching images of both scenarios here. My question is whether I am doing something wrong, or is it something I will have to live with if I decide to use Prefect?

Is there a way to use a specific profile when you use cloud login? I've created the profile locally, but when using git action to login on the same key workspace, i get errors because it's waiting for input to create a profile

I’m experimenting with the new ECSTask in 2.4.0, as we run prefect 1. One option in prefect 1 was the ability to pass in

networkConfiguration

and thus

awsvpcConfiguration

, to allow the flow to run under specific subnets/security groups but this doesn’t seem possible in prefect2 using this task at the moment?

Hello, I have ran into an issue here. When i use LocalDaskExecutor as an executor. One of my task is already using concurrent.futures (multiprocessing) module to run multiple processes. After using LocalDaskEcecutor in prefect flow i am getting AssertionError : daemonic processes are not allowed to have children. Anyone have faced this issue and how to resolve it?

Hello, have a quick question. What would be the best way to set an environment variable within a flow in prefect 2.0?

Hi! I'm using Prefect on K8s clusters and I would need to track costs per projects and namespace. To achieve this, my plan was to add labels to the jobs. Before initializing

KubernetesRun

objects, I tried to modify the `metadata.labels`field, but apparently it gets overwritten by Prefect Cloud. I've also tried to modify

annotations

but that also didn't appear when checking

kubectl describe pod

. Is there a workaround or a different approach that comes to your mind?

Hi, I am trying to solve a problem with different repos using prefect 2. I have a repo for prefect itself and then repos for the DS projects. Each have virtual environments setup. I have simple flows setup that: 1. Creates a yaml file for each deployment and applying a schedule for each deployment visible in prefect cloud. 2. Run the scheduled tasks as k8s jobs. However when I'm pushing deployments from the DS repo it is registered but never actually run/triggered. I have also created another storage block for DS project, but this has not solved the problem.

Hi! New to prefect, trying to create a deployment locally with Orion UI and flow source code from my pycharm project with conda environment. But every time face the same exception ‘flow could not be retrieved from deployment’. Tried all tips from the chat, e.g. absolute path, but every time get this error + ‘script encountered an exception’. When i run the flow as a python file, everything works fine. I think there is a problem somewhere with deployment

Reposting again : - Hello, I have ran into an issue here. When i use LocalDaskExecutor as an executor. One of my task is already using concurrent.futures (multiprocessing) module to run multiple processes. After using LocalDaskEcecutor in prefect flow i am getting AssertionError : daemonic processes are not allowed to have children. Anyone have faced this issue and how to resolve it?