Is there any way to get a link to the flow run log in the prefect cloud from within a flow run? I dont see anything like that in the FlowRunContext.

Hi! I’ve setup a local prefect server to test some workflows out on. The server and agent are run in docker containers and the code is executed in a DockerContainer infrastructure block. I’m having a little trouble with re-running a flow when 1 of my tasks fails. I have 1 task that always raises an exception (on purpose). In the UI I manually set the state to Complete, then re-run the flow. I’m expecting the task that failed (but is marked as complete) to be skipped on the next run; however I get an exception

prefect.exceptions.MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.

I’ve set

PREFECT_RESULTS_PERSIST_BY_DEFAULT=true

in the environment but this doesn’t change the behavior. Any advice would be appreciated. I’m running prefect 2.10.2.

Hello! I’m trying to deploy prefect via helm and am running into this error when validating my overrides:

> helm template . --values deployments/values-prod.yaml --debug
install.go:194: [debug] Original chart version: ""
install.go:211: [debug] CHART PATH: /tools/prefect


Error: template: prefect/charts/prefect-server/templates/hpa.yaml:2:15: executing "prefect/charts/prefect-server/templates/hpa.yaml" at <include "common.capabilities.hpa.apiVersion" .>: error calling include: template: prefect/charts/prefect-server/charts/common/templates/_capabilities.tpl:133:7: executing "common.capabilities.hpa.apiVersion" at <semverCompare "<1.23-0" (include "common.capabilities.kubeVersion" .context)>: error calling semverCompare: Invalid Semantic Version
helm.go:84: [debug] template: prefect/charts/prefect-server/templates/hpa.yaml:2:15: executing "prefect/charts/prefect-server/templates/hpa.yaml" at <include "common.capabilities.hpa.apiVersion" .>: error calling include: template: prefect/charts/prefect-server/charts/common/templates/_capabilities.tpl:133:7: executing "common.capabilities.hpa.apiVersion" at <semverCompare "<1.23-0" (include "common.capabilities.kubeVersion" .context)>: error calling semverCompare: Invalid Semantic Version

I can’t figure out where this value is coming from

<1.23-0

Looks like a bug in the autoscaling setup: https://github.com/PrefectHQ/prefect-helm/blame/abe3dd1ce7979102b0f7209d77d04ad57f5c961a/charts/prefect-server/templates/hpa.yaml#L27

Hello, I'm upgrading from Prefect Cloud 1 to Prefect Cloud 2. We are on the standard plan in Prefect Cloud 1. Is there a way to have that plan active in both Prefect Cloud 1 and 2? I want to move to Prefect Cloud 2 while still having access to everything in my existing Prefect Cloud 1.

When I click on Organizations in Prefect Cloud 2 I get a billing page, even though we have that set up in Prefect Cloud 1, so I'm not sure if there is a way to bring things over.

Deployment schedule anchor dates aren't working as I expect. I suspect a timezone issue but not sure how to resolve. I set:

Deployment.schedule = IntervalSchedule(
    interval=datetime.timedelta(days=1),
    anchor_date=datetime.datetime.today().replace(
       hour=9, minute=0, second=0
    ),
    timezone="America/New_York"
)

I would expect the Deployment to be scheduled to run at 9am EST every day. Instead, it is scheduled 4 hours later than that. If it is relevant, the computer I'm executing this from is in US/Pacific time. Here's my prefect version info:

Version:             2.8.2
API version:         0.8.4
Python version:      3.10.9
Git commit:          afbed19d
Built:               Fri, Feb 17, 2023 10:02 AM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         cloud

Hi Team, i have a query regarding pause and resume prefect flows. Use case :

Let supose i have sent 3 zip file for inferencing, and number of images in 1,2,3 zip files are 99,147,810. And when i trigger the flow let suppose zip file with 810 images started first and 99 is next in line. So what i want is the moment the zip file with the lesser image came for inferencing compare to current execution in flow run with 810 images get pause and zip file with 99 images (which currently is in late state because concurrency is 1 and i cannot increase it) starts running.

Can anyone please help in this.

Hello, not sure if this is the place to ask. If I should put questions in another place please let me know. I have been following the deployments tutorial at https://docs.prefect.io/latest/tutorials/deployments/#run-a-prefect-server When I try to run the

prefect profile use 'local'

command I get the error below, but the profile is still changed. I'm not sure if the error is going to cause issues down the road.

## Running on powershell in Windows 11

##
## View Current profile
##
❯❯❯  prefect profile ls
┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Available Profiles: ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│             default │
│             * local │
└─────────────────────┘
   * active profile
##
## Change to 'default'. Error gets thrown
##

❯❯❯  prefect profile use 'default'
⠸ Checking API connectivity...
Connected to Prefect Cloud using profile 'default'
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000002A9D9185310>
Traceback (most recent call last):
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\asyncio\base_events.py", line 719, in call_soon
    self._check_closed()
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\asyncio\base_events.py", line 508, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

##
## See that profile has changed
##

❯❯❯  prefect profile ls
┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Available Profiles: ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│           * default │
│               local │
└─────────────────────┘
   * active profile

##
## Change back to 'local'. Error is thrown again.
##

❯❯❯  prefect profile use 'local'
⠋ Checking API connectivity...
Connected to Prefect server using profile 'local'
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000001F385726310>
Traceback (most recent call last):
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\asyncio\base_events.py", line 719, in call_soon
    self._check_closed()
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\asyncio\base_events.py", line 508, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000001F385726310>
Traceback (most recent call last):
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\asyncio\base_events.py", line 719, in call_soon
    self._check_closed()
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\asyncio\base_events.py", line 508, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

##
## Show that profile was switched
##

❯❯❯  prefect profile ls
┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Available Profiles: ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│             default │
│             * local │
└─────────────────────┘
   * active profile

I've been seeing RecursionError when a flow finishes executing, literally after all the flow code is totally done, it's a crash on exit.

RecursionError: maximum recursion depth exceeded in comparison

Here's my prefect version info:

Version:             2.8.2
API version:         0.8.4
Python version:      3.10.9
Git commit:          afbed19d
Built:               Fri, Feb 17, 2023 10:02 AM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         cloud

Hello, I’m having issue submitting a flow onto our k8s cluster. I’m getting a 403 on the submission. More details in 🧵

Hello, how do I create a single project which runs on local server as well as cloud (when we have internet connection). One way I could think was checking the internet connection in flow #0 and then loading profile based on that. But the problem is I end up having 2 dashboards (local and cloud). Any workaround for this?

Hi! I'm new to prefect and trying to understand how to correctly manage memory. I posted details on the discourse a few days ago but in short, is there a way to not keep the results of a task in memory once it's completed? Thanks for your help 🙏

What is the recommended way of building custom images for flows? More details in thread 🧵

how might one deploy from a relative directory? I have my core logic in some

src/myproj/file.py

and the boilerplate for flows and configs in another

src/myproj

, and I was doing

prefect deploy ../../src/myproj/file.py:foo

. unsurprisingly this didn't work. what's the recommended pattern here? is it possible to include

src/myproj

in the

push

section of my prefect.yaml?

Hey, currently scoping out prefect... looking for sample deployments using docker image from a local registry

Can anyone help me understand how to add parameters to a deployment I have? File is main.py -- have two parameters that need to be modifiable, startdate / enddate. Not exactly sure how to make this possible

Hi everyone, I was trying to figure out how to get my blocks that I set up in Prefect cloud when running through a docker container. It seems that I am not able to do this.. I was thinking I needed to be able to do this to get "Secrets" at run time (for passwords, etc). I am able to grab these when running on my Windows environment, but when trying to run in the environment in my docker container, it is giving this error for example: "ValueError: Unable to find block document named imdbpassword for block type secret". This goes for any block that I try to access when running the container, so I thought that it might be something with my Dockerfile. But I am able to print out my PREFECT_API_URL.. I have these commands as well in my Dockerfile, which I am pretty sure signs me in:

RUN az login --service-principal -u ${SP_ACCESS_KEY_ID} -p ${SP_SECRET_ACCESS_KEY} --tenant ${SP_TENANT_ID} \
    && /bin/sh -c "prefect config set PREFECT_API_URL=$(az keyvault secret show --name PREFECT-API-URL --vault-name secretsmanager --query value)" \
    && /bin/sh -c "prefect config set PREFECT_API_KEY=$(az keyvault secret show --name PREFECT-API-KEY --vault-name secretsmanager --query value)" \
    && /bin/sh -c "prefect block register -m prefect_azure" \
    && /bin/sh -c "prefect block register -m prefect_azure.container_instance"

Where does prefect store the logs that it captures? I’m assuming it stores them in postgres since I don’t see where else it would be stored.

Hello everyone! I am just starting to evaluate Prefect with DaskTaskRunners, migrating from Luigi. The workflow is highly parametrized - for now users would call a prefect script from the command line and input a setting file, but I would like it to be compatible to the "deployment parameters" in the future It is unclear to me where I find more information about flow parametrization? I have seen this article which seems to use Prefect 1.x (https://www.prefect.io/guide/blog/how-to-make-your-data-pipelines-more-dynamic-using-parameters-in-prefect/ ) and then the tutorial using Pydantic: https://docs.prefect.io/latest/concepts/flows/#parameters

What’s the recommended approach for a prefect waiter? Right now we have a shell script that checks the flow run state periodically. We could make this into an exponential back off but is that the best? Is there something similar to an AWS waiter for prefect?

Where is the best place to ask questions in general? This slack community or discourse? https://discourse.prefect.io/

Hi, what’s the way to get run history? I am unable to figure out what credentials to pass to the REST API and also can’t access it with the SDK using task_run_history.

Hi, thanks for the great tool! I’ve been using for around a month with great results. Recently, however, I’ve been getting this type of error: `A flow named ... and defined at ... conflicts with another flow. Consider specifying a unique

name

parameter in the flow definition`. I’ve checked everywhere and can’t find any sort of double definition. Looked around online, and couldn’t find almost any help. Any ideas? (using the most recent release

2.10.5

)

Hello everyone, hopefully not too silly of a question -- trying deployment with a custom docker image that has flow code baked into it & I am running into this error. I'm sure it's something relatively silly... CMD:

prefect deployment build acc_speed_loader.py:run_all --name acc_test --infra-block docker-container/prefect

ERROR:

ValueError: Unable to find block document named prefect for block type docker-container

An exception occurred

[more info in thread]

How can I change the flow run name, to be something more useful than

certain-stingray

discreet-mole

etc 🙂 Ideally in runtime, since I want it to reflect the parameters that was sent in, so lets say that I have

@flow
def my_flow(
  name: str,
  some_number: int,
):
  somehow_set_flow_run_name_to(f"{name}: {some_number}")

Something like that 🙂

Hi, I'm currently trying out the dask task runner. How is the number of tasks running in parallel determined? This is the decorator I used.

@flow(task_runner=DaskTaskRunner(cluster_kwargs={"processes": False}))

Hi, Ive been trying to set up a local prefect server on EC2 instance to demo Prefect. After some initial success with making flows and deployments and running them, I wanted to clean up the repo and restart everything. After creating a new python venv to run prefect from, I am now no longer seeing anything on my Prefect UI. Flow run history is gone, I cant create Work Pools (hangs after pressing create). Ive tried deleting the database in .prefect/ and followed instructions on how to restart it with

prefect server database restart -y

but the problem is persisting. I'm not sure what is going on here.

Hi, I’ve been trying to set up Prefect with docker-compose but I’m getting

This page isn't working

. I have 2 services in my yaml file prefect-server & prefect-agent. prefect-server’s log is attached, prefect-agent is running fine. will appreciate any help to set this up. TIA!