Is there any reason to use the

...-kubernetes

Docker images (eg

prefect/2.10.11-python3.11-kubernetes

) if you're running on kubernetes but don't plan to use any functionality from the

prefect-kuberentes

Python package?

@Chris Arderne no specific requirement yet, it just came from helm package as we are just starting with prefect so we started to deploy prefect on Kubernetes to start with perfect. we just deployed the helm actually without changes related to images.

hi folks, I'm a bit confused with the open source status of Prefect 2.0. this page suggests that it's possible to host Prefect 2.0 locally https://docs.prefect.io/2.10.11/host/ however, in the

PrefectHQ/server

install metadata, I see

install_requires = [
    "prefect >= 1.0.0, < 2.0.0",

https://github.com/PrefectHQ/server/blob/c6e4627bb6d280fd76924ef245ed870089164bdb/setup.py#L7-L8 is it possible to fully self-host Prefect 2.0 yes or no?

Hey all! Is there a way to use docker image from a private image registry? I defined a K8S block as my infrastructure to run my flows and I couldn’t figure out how to add the credentials for my docker image. Thanks.

Hi We are trying to integrate our prefect flows with github. But I am getting segmentation fault while trying to deploy. And on running the flow it give below E*rror,* Could you please help me idenitify the issue here?

State Message
Flow could not be retrieved from deployment. Traceback (most recent call last): File "<frozen importlib._bootstrap_external>", line 846, in exec_module File "<frozen importlib._bootstrap_external>", line 982, in get_code File "<frozen importlib._bootstrap_external>", line 1039, in get_data FileNotFoundError: [Errno 2] No such file or directory: 'flow_name.py' The above exception was the direct cause of the following exception: ScriptError: Script at 'flow_name.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')

Below is our dir structure in github

└── src
    ├── dev
    │   └── flows
    │       ├── flow
    │       │   ├── README.md
    │       │   ├── flow_name.py
    │       │   ├── flow_deployment.py

And the deployment file is like this

from flow import flow
from prefect.blocks.system import JSON
from prefect.deployments import Deployment
from prefect.filesystems import GitHub

storage = GitHub.load("gh-poc")

FLOW_NAME = "flow_name"


deployment = Deployment.build_from_flow(
    flow=flow_name,
    name=FLOW_NAME,
    parameters={},
    storage=storage,
    path="src/dev/flows/flow_name",
    skip_upload=True,
)

if __name__ == "__main__":
    deployment.apply()
    print("Deployment applied")

Hi everyone. I use

ShellOperation

in some of my Prefect task for long running processes. Is there a way to see the output of such process while it’s still running? Currently the output only shows in Prefect logs after the process has finished.

Hi I think Ive found a major bug related to subflows. I have no idea on how to replicate it. WIll continue as thread.

hi , is there a way to use the prefect flow context as a cache layer ? i want to pass variable’s values between tasks , but the number and kind of the variables is different between tasks. can i set the variable value in some shared context and use it when i need inside some task , or i need to handle it outside of prefect ?

Hey all! New to the slack, glad to be here! I am a project manager and am trying to get a sense of how difficult (time and resource intensive) it is to migrate from Prefect v1 to v2? TIA

Hey team - I have a prefect 1 flow in which one of the tasks is being called twice even though I only have one invocation of it. I will post a mock snippet in the thread.

Hi All, are there any changes to how to set up a reverse proxy between version 2.7.12 and 2.10.10? The reverse proxy is having issues after the update.

Further information on the reverse proxy issue: MS IIS 10, seeing a 500.52 URL Rewrite error with the info 'Outbound rewrite rules cannot be applied when the content of the HTTP response is encoded ("gzip")'

I've been asking Marvin to help me get the flow logs copied to a cloudwatch log stream and I think I'm getting somewhere (https://prefect-community.slack.com/archives/C04DZJC94DC/p1685573638461379) However - I'm stuck on how you deploy a flow to use a custom logging.yml file? I have it part of my deployment and I can see it there - I also have this line in my deployment.py to try and use it

infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "INFO", "PREFECT_LOGGING_CONFIG": "logging.yml"}},

But I can't see any changes being reflected in the logging output. Any ideas? Thanks

Hi there! I'm running into an issue where tasks in v2 don't seem to clean up pycuda context correctly. This issue doesn't appear in prefect v1. Here's a minimal example:

from prefect import task, flow

@task
def initialize_pycuda():
    import pycuda.autoinit

    print("Initialized...")

@flow(log_prints=True)
def my_flow():
    f = initialize_pycuda()

if __name__ == "__main__":
    my_flow()

When I run, I get:

00:38:51.036 | INFO    | prefect.engine - Created flow run 'smiling-hog' for flow 'my-flow'
00:38:51.039 | INFO    | Flow run 'smiling-hog' - View at <https://app.prefect.cloud/account/>...
00:38:51.600 | INFO    | Flow run 'smiling-hog' - Created task run 'initialize_pycuda-0' for task 'initialize_pycuda'
00:38:51.601 | INFO    | Flow run 'smiling-hog' - Executing 'initialize_pycuda-0' immediately...
00:38:52.864 | INFO    | Task run 'initialize_pycuda-0' - Initialized...
-------------------------------------------------------------------
PyCUDA ERROR: The context stack was not empty upon module cleanup.
-------------------------------------------------------------------
A context was still active when the context stack was being
cleaned up. At this point in our execution, CUDA may already
have been deinitialized, so there is no way we can finish
cleanly. The program will be aborted now.
Use Context.pop() to avoid this problem.
-------------------------------------------------------------------
Aborted (core dumped)

When I remove the @task decorator, this flow works perfectly.

Hello, (how) does prefect work with asyncio? E.g. with asyncpg?

why not show a list of types? and why every endpoint returning 405. its helm deployment. help needed.

why not show a list of types? and why every endpoint returning 405. its helm deployment. help needed

Good morning everyone. Has anyone else experienced pending and completed tasks blocking the task run concurrency limits? I’ve found I can clear the completed tasks by changing their state to completed. I’m currently using prefect 2.8.6

Hi, we recently encountered an issue again that was supposed to be fixed (see this thread). The issue causes the same flow run to be started twice (run count is 2 and Results tab shows two runs started at the same time). This can be very problematic since we're doing database updates in our flow runs. Is this a known issue again / what can we do to bypass the issue?

Hi there, I'm currently running my flows in cloud run infrastructure. However, cloud run jobs have a max time limit of 1h, so the flow needs to finish before that. Is there an elegant way to early stop the running flow (by measuring either the time passed or the "work done") and trigger another one with the same deployment so that it continues the work that is left in a new flow run / cloud run job? Is there an elegant way of passing job metadata (for example, records processed / failed ) from one flow run to another? Thanks in advance!

Does prefect still use

pg_trgm

and how is it used? All the starting guides just have you spin up a postgres container in docker which does not have this extension turned on?

I want to configure my existing work pool from ci/cd, especially the

Base Job Template

section. So far, I found the only way is to use the

PATCH

REST API, or is there a way to change settings via cli?

Hi all, Using prefect cloud here; is there any way of automatically cleaning up old flow-runs, say I don't need anything > 24 hours old? Its starting to clog up a bit and running a script manually feels a bit strange since I don't like running manual stuff -- sort of what I got prefect for in the first place 🙂 Thanks!

Does Prefect 2 have an equivalent of callable task name generators from Prefect 1? For flows that map inputs over tasks. Prefect 1 example (this names the mapped children tasks in cloud UI):

def generate_task_run_name_foo(dataset: dict, **kwargs):
    return 'Foo:  ' + dataset['name']


@task(max_retries=RETRY_COUNT, retry_delay=RETRY_DELAY, log_stdout=True, task_run_name=generate_task_run_name_foo)
def foo(dataset: dict):
	...

How do you handle model registry / deployment for Prefect deployments? TL;DR we want revisions and be able to rollout / rollback a ML model for a daily running deployment. We have a Kedro data science inference pipeline that runs for 3 weeks (RRule). During the second week we would like to update the reference to the training model (the path to the pickle / mlflow run id) that is associated with the Prefect deployment. Is there best practice to do that? Kedro is using MLFlow for experiment manager, but I’m not sure how to propagate the pickle / mlflow run id to the deployed Prefect deployment. I could reinvent the wheel and add an S3 pointer to the “latest” pickle, but I want Amazon SageMaker / MLFlow like capability to review and deploy models.

i.e. what’s the Prefect way to perform

mlflow.update_registered_model()

?

Hi guys, I am following Prefect's Anna Geller's Medium blog post on how to use Prefect in AWS Lambdas using Docker images. Following the blog, I did the following: • Wrote a simple Python script with a simple flow • Added a Dockerfile that uses the AWS Lambda Docker base image, installs dependencies and adds the script as a handler • Build the image • Ran the image • Tested the image locally (following AWS's docs) I kept getting weird errors that 'State' object has no attribute 'services', and I couldn't figure out why. After some googling, turns out that the sqlite3 version of AWS Lambdas Docker image runtime is 3.7, but Prefect needs >3.24. So I got curious, and cloned the blog's repo and tried following the exact same steps with her code, because how come her code would work with exactly the same steps but mine wouldn't? Conclusion is that her code also doesn't work, which is weird, because it must have worked when she wrote the blog post, so what is different now? Using Docker + the same AWS Lambda base image should prevent exactly such issues, or in other words, ensure it runs anywhere, but in this case it doesn't. Am I forgetting something important here? Am I overlooking something? Do you guys have any ideas what could be wrong, or how I could get this to work? See the blog post here: https://medium.com/the-prefect-blog/event-driven-data-pipelines-with-aws-lambda-prefect-and-github-actions-b3d9f84b1309 See the repo here: https://github.com/anna-geller/prefect-aws-lambda/tree/master

Hi everyone. I have a flow with some conditional logic. I want to exit cleanly if a particular condition is met. I am using

exit(0)

but Prefect sees that as a crash. What is the recommended approach? I am doing something like this:

@flow
def my_flow():
   if not condition:
       run_task()
       exit(0)
    do_other_stuff()

Prefect cloud scheduler bugged out and has scheduled dozens or hundreds of simultaneous scheduled runs for one of our deployments today

Hi everyone - I have configured a block to connect to SQL Server using sqlalchemy connnector. And I am specifying { "driver": "ODBC Driver 17 for SQL Server"} as additional connection arguments. When I try to get the engine, I get a warning - SAWarning: No driver name specified; this is expected by PyODBC when using DSN-less connections engine = create_engine(**engine_kwargs). I am lost what I am doing wrong. Also I did a print of engine_kwargs and I got this as part of my print statement - {'url': mssql+pyodbc://testuser:***@test/reportserver, 'connect_args': {'driver': 'ODBC+Driver+17+for+SQL+Server'}} so it looks like the additional configuration is getting passed. This just seems to be a warning and the code still works fine that references that engine object.