Hi all,

prefect deployment build flows/hello.py:hello --name docker-custom --tag dev -sb s3/dev -ib docker-container/docker-custom-image

how do I expose the Port`8787` on docker-custom-image. I have configured volumes, networks on Docker Container block but didn't find any option about ports. I want to access Dask dashboard when I run flows from UI which spin up a separate container of my docker-custom-image.

where did flow packaging go? I'm unable to find any references to if after the public release

I have a question: I have been using Prefect to run pipelines with no problems until recently. However, a colleque of mine wanted to try the email notification and had installed prefect-email. Now I keep getting import error while running the existing code to run the pipeline that was working just fine before installing prefect-email. from prefect import task, Flow, Parameter Exception has occurred: ImportError cannot import name 'Parameter' from 'prefect' (/home/ubuntu/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/__init__.py) We are using prefect 2.0.1. Tried downgrading to 2.0.0 as well as tried clean installations many times with no luck. Could someone help diagnose the issue and help me resolve it?

Hey Guys, we're currently using Prefect for handling all of our data pipelines, and we have another tool in place (Temporal) for handling non-data related pipelines or workflows. These are mainly workflows that handle 1 specific action (like activating a customer record, creating a new entity, etc.) My question is have anyone tried to use Prefect in order to implement use cases like those i have mentioned? The reason we're asking is because we are thinking about unifying the technology stack but we're not sure if Prefect is well suited for handling such use cases

Hey, I haven't managed to find anything similar to how you have separate flow domains with

projects

in prefect 1.0, in prefect 2.0. Is that something that exists and I'm just missing it? I see workspaces but that's a different paradigm..

while using prefect 2.0 beta version i was creating my multiple deployemnets using the following code and then running prefect deployemnt create {name of the file } and my multiple deployments were getting created but now there is not any option to create muliple deployments and neither ist specified in the docs and my previous deployments are shown deprecated . please help

UPDATE -- see edit 3 below. This works fine on a dedicated linux computer. This problem seems to be specific to Windows (maybe just Windows 11), regardless of the combination of agent running in WSL2/Powershell, docker WSL2 backend/Hyper-V. Prefect 2.0 is quite fun to use so far. One issue that I'm running into is with "auto remove" in the Docker Container block. Since Prefect 1.0 behavior is that the container is removed after the flow run, my assumption is that it made sense to set auto remove to True in Prefect 2.0 as well (otherwise containers start to pile up). When I run a flow using the default "auto remove"=False, the flow completes successfully in Cloud 2.0, the agent resumes waiting for the future runs, and the stopped container persists for the Docker Engine instance. But if I change "auto remove"=True: • the flow still completes successfully in Cloud 2.0 • but the agent completely crashes, stating it can't find the container marked for removal -- and stops listening for future runs • at the same time, the container is indeed successfully removed So, it's completing 2/3 steps successfully. It almost seems like the container is marked for removal, gets removed, and the agent tries to remove it again, or tries to verify that it was successfully removed but runs into an error when it doesn't find it rather than seeing that as successful. If this is a bug it wouldn't be quite as much of an issue if the agent continued running after the error message. Attached is the logout for the error message that appears after it logs all tasks completed. EDIT: I should add: This is using WSL2 Ubuntu in Windows 11. Docker Engine is running WSL2 backend. I get the same error if I run the agent from Powershell in Windows 11 and if I run the agent in a separate WSL2 Ubuntu instance. Both see the Docker Engine just fine, and run the flow in a container just fine, but also both crash with the output in the attached log above. EDIT 2: I tried changing Docker Desktop on windows to use Hyper-V rather than WSL2, and ran the agent in Powershell so it's all in the same environment. Got pretty much the exact same error message. Later I will try the agent and docker with "auto remove" on a dedicated linux machine. EDIT 3: This works fine on a dedicated linux machine (Ubuntu 22.04, docker pulling from ghcr.io, flow pulled from S3, successfully completed, successfully removed container, agent continued running fine afterwards).

Hi, is it possible to start two Prefect Orion servers mapped to the same database? More specifically, I want to run one server for 1. outbound access. It should be used for users to view the UI 2. internal virtual network access. It should be used by agents to talk with the Prefect API Can I specify the same Postgres connection string for both servers? Or do you have any other solution ideas for my setup?

Through the following code, I am trying to run a flow on prefect cloud. But it was getting in scheduled and pending state then I added .start() line of code with agent, the flow was no more scheduled but when I am trying to do the same with a new flow it is just running an agent. Is this a right way bcz its just running agent but not registering the new version until I stop the agent*?*

@task(log_stdout=True, result=LocalResult())
def hello_world():
    text = f"hello to {FLOW_NAME}"
    print(text)
    return text

from prefect.agent.local import LocalAgent
from prefect.run_configs import LocalRun

with Flow(
    FLOW_NAME, storage=STORAGE, run_config=LocalRun(labels=["xyz"])
) as flow:
    data = hello_world()

LocalAgent(labels=["xyz"]).start()

Hey Everyone, when using Prefect 1.0 we had a helm chart and our entire deployment was running on a local kubernetes cluster. other than running the orion server on K8 and giving it access to a database, I can't find any information in the docs about setting up a cluster on k8

I'm wondering if anyone has thoughts on writing a flow with a combination of sync and async tasks. or perhaps a flow of flows, with some sync and some async? I currently have a flow where the first task needs to be async because it's calling coroutines in another async framework, but I'd like the following tasks to not be async - especially because if I define all the tasks as async, even though I can ask them to

wait_for

the other tasks, they are still visualized as happening concurrently, which makes the visual less fun.

I am trying to run the orion server inside a docker container, but I can’t get it to work. This is my command:

docker run --platform linux/amd64 -ti -p 4200:4200 --expose=4200 prefecthq/prefect:2.0.1-python3.9

Once my container is ready, I execute

prefect orion start

inside of it I see the in the logs that the server successfully started, however I don’t get a connection when I open http://localhost:4200 in my browser.

This message was deleted.

👋 do you all have thoughts on when you'd use prefect over something like temporal/cadence? or, is the intention that you should feel comfortable using prefect for any use case that the latter aims to address?

Hey team! My workflow is paused after submission, prefect agent is also up. What could be the reason? This is the already working solution, now I am trying it with different GCP environment and it is stucked.

Hey community! Any guide on this error, To which connection does this error refer to??

ERROR
agent
HTTPSConnectionPool(host='34.66.191.39', port=443): Max retries exceeded with url: /apis/batch/v1/namespaces/default/jobs (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f70cc17afd0>: Failed to establish a new connection: [Errno 111] Connection refused'))

Hi Prefect family. I'm running the example below in Visual Studio Code in my local machine Windows 10 and getting the error displayed. Any guidance?

Hi everyone, I realize it's Sunday night after a long week, so no worries if you wait until tomorrow to respond. I was using orion for a good portion of the beta, and some of the last-minute changes to Deployments are confusing me. For starters, I would dynamically create deployments like this:

deployment = Deployment(
    name="example-deploy",
    flow=my_flow_object,  # defined elsewhere
    packager=OrionPackager(serializer=PickleSerializer()),
    tags=[
        "some-tag1",
        "some-tag2",
    ],
)

deployment_id = await deployment.create()

But this code is now broken and the docs recommend sticking to yaml files. How would I go about updating this? Can we still specify a

packager

and is it possible to create a deployment in python without creating a yaml file?

How do I programmatically assign

Settings

values? E.g. I want to set

PREFECT_API_URL

from inside the python script?

Hey community ! That might be a noob question, but I have some dififculties finding reliable materials on Prefect 2 👀. I already have a working job example.py file and want to start scheduling it Struggles are on the deployment part mentionned on the Documentation + The command does not seem to exist. ~/workspace/prefect ❱ prefect --version

2.0.1

~/workspace/prefect ❱ prefect deployment create example.py

Usage: prefect deployment [OPTIONS] COMMAND [ARGS]...

Try 'prefect deployment --help' for help.

╭─ Error ──────────────────────────────────────────

│ No such command 'create'.

╰──────────────────────────────────────────────────

I really enjoyed the file-less workflow of uploading flows to the Prefect 1.0 server and having them execute in the k8s infrastructure (via the agent, as k8s jobs). It seems it’s not possible anymore with Orion, is it? I don’t fully understand the benefits of the json/yaml files as opposed to the former API. Should I really embrace calling CLI and dealing with files as it is a superior approach? One obstacle in my workflow is I’m doing interactive work in a Jupyter kernel, running in the cluster, so getting a shell at the same host is not straightforward

Is there a public roadmap for future Prefect development and features?

Hi guys, I'm just getting started with prefect 2.0 in docker. I mounted file into /flows folder - but unfortunately I can't see it in the flows section in the UI... what am I missing?

root@24dc674d1f7d:/flows# cat flow1.py

from prefect import flow

@flow
def my_favorite_function():
    print("What is your favorite number?")
    return 42

print(my_favorite_function())

Hi community, Lately we have noticed the following error that occasionally caused failure for multiple unrelated flows (here is log example for one):

14:25:23 INFO Submitted for execution: Job prefect-job-2444a873
14:25:49 ERROR Failed to load and execute flow run: ConnectionError(MaxRetryError("HTTPSConnectionPool(host='<http://www.scorpiotankers.com|www.scorpiotankers.com>', port=443): Max retries exceeded with url: /our-fleet/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efdbc010d00>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"))
14:26:00 INFO Beginning Flow run for 'Destination Predict'
14:26:00 DEBUG Using executor type LocalDaskExecutor
14:26:00 INFO Flow run has already finished.
14:26:32 ERROR k8s-infra Pod prefect-job-2444a873-8887r failed.
	Container 'flow' state: terminated
		Exit Code:: 1
		Reason: Error

The specific error seems to reflect code from another unrelated flow. Any idea how one flow can throw an error in several other completely unrelated flows? before even the flow starts?

Hi, happy 2.0 GA 🎉 What is your opinion on how to structure Prefect 2.0 code within an organization? One big mono-repo? One repo per Flow? One repo per business value stream? Would love to hear your experience and perspectives! prefect duck

I've spent a few days trying to get accustomed to 2.0 after running 1.0 for over a year and a half. It looks amazing so far! However, there are two things I'm still trying to wrap my head around: Docker containers and storage. I've been trying to learn as much as I can from the documentation and this slack workspace but it's still a bit unclear. Previously, we have been building each flow as its own docker container and we have never specified any sort of local/remote storage beyond setting flow.storage to Docker. I suppose this is no longer supported in 2.0? The impression I'm getting from reading the documentation and slack is that people generally build a few base images, upload it to their container registry (Azure, in our case) and then store the flow .py files themselves in a remote storage location. Is this correct? If so, since I'm using Azure, can I even move forward with the migration right now or do I have to wait for blob storage support? In case it makes a difference, we run our agents in VMs and we're primarily using the cloud versions of the web service, other than for local development.

What I really liked about Prefect 1.0 was the

log_to_stdout

Parameter which automatically logs all print statements. Is there something similar for Prefect 2?

Is the Prefect 2.0 sign up page broken?

Hello guys! I've just found out about prefect not long ago and currently it a top candidate for our data workflow and pipeline. While i was going through the documentation, it stated that prefect 2.0 is under active development and prefect 1.0 is recommended for production use. Since prefect was officailly released a couple of days ago and is now the default installing package for pip, is it still true that prefect 2.0 is not recommended for production use? Is there also a way to visualize my workflow like in prefect 1.0's

flow.visuallize()

And also im really loving prefect and cannot wait to start working with it and exploring Thanks alot

Is it possible that some errors are not caught by Prefect when running multi-threaded code? PS: wasn’t sure if this is the right place to post since #prefect-server was archived.