Hi I just posted a new bug. Issue # 9933. https://github.com/PrefectHQ/prefect/issues/9933

Issue added re long waits between tasks -> https://github.com/PrefectHQ/prefect/issues/9934 - Help would be much appreciated - my only real solution at the moment is to not use tasks at all...

Hello! Can anyone tell me how to get 'task_run_id' in Prefect 2.0? In version 1.0, we got it like this:

prefect.context.get('task_run_id')

I wonder what could be causing this error? The relevant poetry dependency specification is a git dependency with a revision specified:

altair-saver = {git = "<https://github.com/PaleNeutron/altair_saver>", rev = "3a8cf9c3a7a3c8c8b642307cd1aec33ae85cc1e3"}

Hi! How to get logging messages from imported modules? In version 1.0, we added the line logging = prefect.context.get("logger") to each module and then used the logging.debug lines for logging.

Hi, when I and my colleague @Cristian Federiconi debug code with Pycharm the prefect flows (@flow) return the state (State object) instead of actual data, as, on the other hand, normally happens when I run the code with "Run" or in the cloud. This causes the code we wrote to stop working when debugging because the State object obviously does not behave as the data type we expect. Is there a particular reason why this happens? Is it possible to set a parameter on the flow decorator to stop this behaviour? Thank you in advance for your response.

What is the difference between a prefect worker and prefect agent

Hello! I'm applying a Deployment through the

Deployment.build_from_flow

method, and I was wondering if it's possible to reference the flow through the package name instead of the file. Like saying "you need to call the flow from `my.python.package:my_flow`" instead of the full file path.

I am getting the following error when I try to either edit or create a work pool in my hosted opensource environment. The ui call is getting a 500 response http://prefect.local/api/collections/views/aggregate-worker-metadata

16:54:29.583 | ERROR   | prefect.server - Encountered exception in request:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpcore/backends/asyncio.py", line 114, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 216, in connect_tcp
    async with create_task_group() as tg:
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 593, in __aexit__
    raise CancelledError
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/usr/local/lib/python3.11/site-packages/httpcore/backends/asyncio.py", line 113, in connect_tcp
    with anyio.fail_after(timeout):
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_tasks.py", line 119, in __exit__
    raise TimeoutError
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 261, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 92, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 69, in handle_async_request
    stream = await self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 117, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/backends/auto.py", line 31, in connect_tcp
    return await self._backend.connect_tcp(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/backends/asyncio.py", line 112, in connect_tcp
    with map_exceptions(exc_map):
  File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await <http://self.app|self.app>(scope, receive, _send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 24, in __call__
    await responder(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 44, in __call__
    await <http://self.app|self.app>(scope, receive, self.send_with_gzip)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await <http://self.app|self.app>(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await <http://self.app|self.app>(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await <http://self.app|self.app>(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/server/utilities/server.py", line 103, in handle_response_scoped_depends
    response = await default_handler(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/server/api/collections.py", line 18, in read_view_content
    return await get_collection_view(view)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/server/api/collections.py", line 43, in get_collection_view
    resp = await client.get(repo_url + f"/views/{view_filename}")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1757, in get
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout
Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 428, in run_asgi

Hey Prefect team. Having some interesting issues with a custom docker image we've made for k8 pods based off of your

prefecthq/prefect:2-python3.8-kubernetes

image pretty simple dockerfile, we just pull down your image, remove existing pip packages, and install our own based on the

requirements.txt

file we maintain.

# syntax=docker/dockerfile:1
FROM prefecthq/prefect:2-python3.8-kubernetes

ADD ./Prefect /home/prefect/storage/repo-files

#RUN chmod 777 -R /home/prefect/storage/repo-files/

RUN pip install --upgrade pip \
&& pip freeze > /home/prefect/storage/repo-files/current_packages.txt \
&& pip uninstall -r /home/prefect/storage/repo-files/current_packages.txt -y \
&& rm /home/prefect/storage/repo-files/current_packages.txt \
&& pip install -e /home/prefect/storage/repo-files

One thing to note is that since we use openshift, we had some permissions issues with cloning our git repo to the image, and found a work around by overriding the PREFECT_HOME environment variable to generic

/home/prefect

directory after that fix, the Image is working fine when we are deploying images via git cloning, but another team member starting to look into deploying with the API using the local paths of python files we're already storing on the image in the ADD layer. when I build the image locally, I see the directory structure created as specified as shown in the first attached screenshot. When we check the directory structure of pods spun up by our worker however, the files do no exist there. Actually nothing exists under

/home/prefect/

. I also notice from these docs that there is a PREFECT_LOCAL_STORAGE_PATH env var that should be set as

PREFECT_LOCAL_STORAGE_PATH='${PREFECT_HOME}/storage'

(hence why I added our files to a directory underneath that location in the dockerfile). Trying to confirm what that value got set to on the pod returns an empty string as shown in the other attached screenshot. questions here are • why do we not see our files on the pod when we know the local image has them? Is it possible that since we write these files there before the pod spins up and PREFECT_HOME is overwritten, the dir gets purged during spin up? • why do we not see a value for the

PREFECT_LOCAL_STORAGE_PATH

on the pod? is that related to not being able to see the files? • is there a better or recommended way to achieve running flow deployments with local files on an image?

Is prefect having trouble scheduling flows right now??

What would cause a worker to not pull flow_runs from a queue in a work_pool. I can see the logs from the worker querying the server, but the queue is not turning healthy

Hello, I would like to distribute tasks on different servers/workers depending on several criteria (for example: GPU availability, available memory, location, ...) My original plan was to use a Dask cluster and the dask annotation system. However, it seems that it is not currently possible to use Dask annotations with an existing cluster. Would there be an alternative way of achieving the same result? If I understand correctly, I could set up different prefect work pools for different types of servers. But it will assign the whole flow to the same server (not per task) so I would need to replace each task with a subflow. Is that the proper way achieving this? I am thinking to schedule 100s of such tasks at once and I am unsure subflows are suited for this. Thank you!

Hi, Is there a way to re-run a Prefect flow, but only from some point on? E.g. I have 10 tasks in my flow and I'd like to use the cache for the first 4 tasks, but I want everything else to be computed again. In Prefect 1, I'd normally delete the cache files and run the flow again - when cache file was present, cache would be used. And when cache file would not exist, Prefect would run these tasks.

Hi All...........Can someone please help me how to get all the logs to a file using Prefect 2 I referred this https://discourse.prefect.io/t/how-to-stream-prefect-logs-to-a-file/1963 But it's not working........Can someone please suggest

Hello, is

refresh_cache

supposed to reset the value of

cache_expiration

as well? From my tests it does not, is this intended or an issue?

Hey, I have a worker in ECS and a few Prefect deployments. I want to enable cache for Prefect Tasks. How does that work in ECS Tasks spawned by the worker?

What would cause my queues to not be healthy when I can see the worker are actively polling?

Hey team - is there a way to have local flow runs NOT show up in the UI on prefect cloud? We are doing a lot of testing and there are a crazy amount of flows which are showing up in our console. Also, at times when we close the debugger (spins up docker container with prefect in it) or even running with virtual env, the flow run on the UI shows running still and does not exit. Any help would be greatly appreciated 🙂

Hello I have a strange error when I run a deployment. To add context, I deploy a worker on Kubernetes with Helm (with this: https://docs.prefect.io/2.10.13/guides/deployment/helm-worker/ ). Then I deploy a flow with build_docker_run (with

namespace: prefect

specified in my deployment.yaml (I use projects concept). This is my error at launch :

kubernetes.client.exceptions.ApiException: (422)
Reason: Unprocessable Entity
HTTP response headers: HTTPHeaderDict({'Audit-Id': '90369801-cb01-411b-b3a8-43018a11c756', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Warning': '299 - "unknown field \\"spec.template.spec.completions\\"", 299 - "unknown field \\"spec.template.spec.parallelism\\""', 'X-Kubernetes-Pf-Flowschema-Uid': 'ff4e4556-b309-4791-8d69-60463f550a9d', 'X-Kubernetes-Pf-Prioritylevel-Uid': '972bc841-dafb-496e-89cd-a53f07f46a9f', 'Date': 'Thu, 15 Jun 2023 15:22:37 GMT', 'Transfer-Encoding': 'chunked'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Job.batch \"-fqr7x\" is invalid: [metadata.generateName: Invalid value: \"-\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. '<http://example.com|example.com>', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), metadata.name: Invalid value: \"-fqr7x\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. '<http://example.com|example.com>', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.template.labels: Invalid value: \"-fqr7x\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')]","reason":"Invalid","details":{"name":"-fqr7x","group":"batch","kind":"Job","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \"-\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. '<http://example.com|example.com>', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","field":"metadata.generateName"},{"reason":"FieldValueInvalid","message":"Invalid value: \"-fqr7x\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. '<http://example.com|example.com>', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","field":"metadata.name"},{"reason":"FieldValueInvalid","message":"Invalid value: \"-fqr7x\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')","field":"spec.template.labels"}]},"code":422}

I try to deploy worker and flow on default namespace but with same results. I try to upgrade my GCP clusters to be with the last stable k8s version but nothing change. Do you have any idea?

Does anyone have a good way to keep track resource usage for all Prefect flows running in kubernetes? Like maybe with Prometheus?

How to monitor ECS + Fargate runs? I am using Prefect cloud, an AWS hosted agent, I created a deploy with an ECS task from

prefect-aws

The agent logs are only showing resource allocation + cleanup, the prefect cloud web UI is showing the flow as crashed. I have some guesses about what's wrong (a pip package hosted in a private repository) but I would like to know where I could look.

I'm trying to dynamically add tags to a flow by iterating through a list. Tried a few different things, but keep getting

TypeError: 'list' object does not support the context manager protocol

or

TypeError: 'generator' object does not support the context manager protocol

Here's one of my attempts. Any suggestions on how to do this?

from prefect import flow, task, tags
from prefect.context import TagsContext

def add_tags(tag):
    with tags(tag):
        yield

@flow()
def my_flow():
    # print(TagsContext.get().current_tags)
    print('inside flow')


tag_list = ['mytag1', 'mytag2']

with [add_tags(x) for x in tag_list]:
    my_flow()

Prefect Team have you seen an instance where you have worker polling a workpool, but the pool stays unhealthy and the workers don't discover the queued flows?

Does anyone have experience passing in a task as a parameter to a deployment? I have a deployment being created using a flow. I'm passing in a Task object within a list as a parameter during the Deployment build process. But I'm running into this error:

deployment = Deployment.build_from_flow(
...
parameters={"foo_tasks": [FooTask]}

Object of type 'Task' is not JSON serializable

https://prefect.status.io/ I think the side effects of this incident are more than just "Increased latency loading flow runs"

Hey team - since the outage, my flows are all of a sudden unable to find the .py files in S3 which they are getting the script from. I will put the error in the thread.

Hi - has anyone have an issue with flows "exiting silently" without it being reflected on the UI? In the image below, it looks like the flow just exited quietly halfway through, and the UI is still says its running?

Hi everyone, i am trying to add a kubernetes block to run each flow-runs in a pod, i have crearted both the kubernetes block but the pods are not getting created, can anyone please help me on this

If/when this goes into effect, do we lose the ability to deploy from a script? https://github.com/PrefectHQ/prefect/pull/9881