Hello - can I get some help with debugging a failing workflow?

I feel like I'm going to gain a reputation as the guy who only ever asks about type safety but... I've got pyright integrated into my environment now, which is working far better with Prefect than Mypy did. It just requires one small hack, and it makes me wonder why this isn't part of the library: I had to add

py.typed

on my own. I see that there is a

py.typed

on the master branch; was there a conscious decision made not to include this marker file in 2.0? If so, why? Without the marker file, I get:

from prefect.flows import flow

reveal_type(flow)  # Unknown

With the marker file, I get:

from prefect.flows import flow

reveal_type(flow)  # Overload[(__fn: (**P@flow) -> R@flow, /) -> Flow[P@flow, R@flow], (*, name: str = None, version: str = None, task_runner: BaseTaskRunner = ConcurrentTaskRunner, description: str = None, timeout_seconds: int | float = None, validate_parameters: bool = True) -> (((**P@flow) -> R@flow) -> Flow[P@flow, R@flow]), (__fn: Unknown | None = None, *, name: str = None, version: str = None, task_runner: BaseTaskRunner = ConcurrentTaskRunner, description: str = None, timeout_seconds: int | float = None, validate_parameters: bool = True) -> (Flow[P, R] | (((**P) -> R) -> Flow[P, R]))]

which is much better. For example, now when I decorate a function with

@task

and use it in an

@flow

, the return type of the task is known to have a

.result()

method. When it was just an Unknown, pyright would complain that I'm accessing a method that doesn't exist. For someone who enforces fully passing mypy (now pyright) checks on every pull request, this is kind of a necessity.

Hello, when I run prefect 2.0b5 on windows machine, there is an error message "ValidationError: 1 validation error for Settings PREFECT_ORION_API_PORT value is not a valid integer (type=type_error.integer)". I try to reset it with "prefect config set PREFECT_ORION_API_PORT=4200". It showed the same error message. Any idea to fix it? Thanks.

Hi. I continue to ask my questions here - super grateful for the help i got so far. I have now setup a flow that launches a number of subflows that fetches data from Google Cloud Storage and writes the data to Big Query. I’m running the flow locally (prefect 2.0) and it’s connected to prefect cloud. However, i sometimes run into an error i cannot really understand and i don’t know how to troubleshoot it. It only happens occasionally and i cannot really reproduce it. I moved the trace to the thread. The logging message @

16:34:10.826

is the last thing that happens in the task read_blob in which the error occurs in. So it looks to me that something goes wrong when reporting the task. The error message doens’t tell me that much - so if you have any advices on how i should troubleshoot this i would be really helpful (or if you can guess on what might be wrong?)

is something funky going on with logging in the prefect UI? for flows that failed, i'm not getting the full logs (in the UI) - I send those logs to AWS cloudwatch and can confirm that full expected logs are there 😞

Hi everyone 👋 I have some troubles with the running of my flow on kubernetes. My flow includes docker command such as

docker volume create

/

docker-compose up

so i need docker daemon but it cannot be outside my job. Here is my job template

apiVersion: batch/v1
kind: Job
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: flow-container
        - name: dind-daemon
          image: docker:stable-dind
          env:
            - name: DOCKER_TLS_CERTDIR
              value: ""
          securityContext:
            privileged: true
      imagePullSecrets:
        - name: regcred

It works but my docker daemon container never stop...... any ideas ?

any suggestions / best practices for triggering Prefect jobs within a CI/CD pipeline? suppose we're building a Docker image within the pipeline and pushing it to a container registry, and want to execute a flow using that container image, guaranteeing that we're always using the latest (can we do this with a Docker agent?)

Hi wonder if anyone could help me with a problem I have working with the Dask KubeCluster? So the issue I am having is that various secrets that I have mounted to the usual flow jobs don't get carried over to the pods that are started by dask - there is an added complexity that I am using two images a dev one and a non dev one tied to two different prefect projects - I am able to do something like this to switch the image;

DEV_TAG = os.environ.get("DEV", "") != ""

JOB_IMAGE_NAME = f"blah/flows{':dev' if DEV_TAG else ''}"

And then in each flow I ref the

JOB_IMAGE_NAME

- this just changes the image but otherwise uses the job template I have defined on the agent;

apiVersion: batch/v1
kind: Job
spec:
  template:
    spec:
      containers:
        - name: flow
          imagePullPolicy: Always
          env:
            - name: SOME_ENV
              valueFrom:
                secretKeyRef:
                  name: secret-env-vars
                  key: some_env
                  optional: false

Now when I specify the dask setup I do the following;

executor=DaskExecutor(
        cluster_class=lambda: KubeCluster(make_pod_spec(image=JOB_IMAGE_NAME)),
        adapt_kwargs={"minimum": 2, "maximum": 3},
    )

But this is obviously missing the env part of my default template - I would like to not have to respecify it (its much bigger then the above snippet) - is it possible to grab a handle on the default template and just override the image name?

I am trying to write a DeploymentSpec YAML config file referring to this example:

$ prefect deployment inspect 'hello-world/hello-world-daily'
{
    'id': '710145d4-a5cb-4e58-a887-568e4df9da88',
    'created': '2022-04-25T20:23:42.311269+00:00',
    'updated': '2022-04-25T20:23:42.309339+00:00',
    'name': 'hello-world-daily',
    'flow_id': '80768746-cc02-4d25-a01c-4e4a92797142',
    'flow_data': {
        'encoding': 'blockstorage',
        'blob': '{"data": "\\"f8e7f81f24512625235fe5814f1281ae\\"", "block_id":
"c204821d-a44f-4b9e-aec3-fcf24619d22f"}'
    },
    'schedule': {
        'interval': 86400.0,
        'timezone': None,
        'anchor_date': '2020-01-01T00:00:00+00:00'
    },
    'is_schedule_active': True,
    'parameters': {},
    'tags': ['earth'],
    'flow_runner': {'type': 'universal', 'config': {'env': {}}}
}

Is there any extensive example available to write the complete config for a flow??

Can't get the reason why there are so many late runs piling up Have scheduled about 6-7 flows on single node cluster via minikube

Hi there what’s the best practice for passing database parameters into the prefect flows? I have a flow that I want to connect to different DBs (alpha/beta/prod), should I just use the Parameter mechanism for this?

Not sure if others have faced this as well, but we experienced an issue at 4:00pm EST where all flows that were scheduled to run on prefect cloud at 4:00 pm EST were only picked up by our agents at 4:25 pm EST. It seems that things are back to normal now, given runs are being picked up by our agents promptly

FYI, we experience flow run issues from 1PM - 1:30 PDT time. The scheduled flow runs were not triggered or picked up to run.

@Marwan Sarieddine and @Hui Zheng We've seen some spike in traffic during that time, but the issue has been resolved now. We apologize, and we'll keep you up-to-date as we monitor the services.

Hi. I'm using client 2.0b4 and orion cloud. Got problem recently with the same setup, my agent fails to pick up runs from queue. All IDs are correct (was working fine for weeks), yet it keeps on failing. What's wrong?

httpx.HTTPStatusError: Server error '500 Internal Server Error' for url '<https://api-beta.prefect.io/api/accounts/ff4de07a-3c0a-4831-a96f-236ce6513b52/workspaces/c36fd28e-d969-4a32-bb11-0f7712f96635/work_queues/cfbcef84-6668-41ff-83dd-0053055deffb/get_runs>'

Hey @Prefect Cloud - our production alert system just alerted us. It looks like Flows are not kicking off in Prefect Cloud. They are getting stuck in the Scheduled and Submitted States. Potential repeat of this afternoon?

Hi, everyone. We see some performance degradation in Prefect Cloud. We are aware of the issue and working on a resolution. We apologize, and we'll keep you up-to-date. cc @jars @Quan Cao

Thanks @Anna Geller

I observed that the pods start and keep failing and restarting While no resources show up when we get pods for default namespace

Hi 🙂 I'm getting error 500 when running my flow using Orion. Any idea what's wrong?

httpx.HTTPStatusError: Server error '500 Internal Server Error' for url '<https://api-beta.prefect.io/api/accounts/927cf1b8-198e-4236-841e-36098d433977/workspaces/be783b36-6bf5-4c3e-b919-8da5b8ea57cc/task_runs/277f8cff-c1aa-47b4-a5d7-126ce2957ad3/set_state>'

Starting to see Flow Runs kick off again now...

A short update on the above: all systems are back to normal since 11:31 AM UTC - the status page is updated and we keep monitoring the services. Thanks to everyone reporting the issue!

Hi all, I’m looking for good options of where to setup up the prefect agent and have it running. Currently I have a virtual machine with a terminal open listening. Does anyone have resources I can look at to compare? I’m on Azure for if that matters.

Something strange happened with one of my flow runs today and I wonder if anyone could help me understand? A flow was supposed to start on the hour but didn't actually start until 40 mins later - I am running on k8s - this is the logs for the agent;

With the recent release of the Cloud Run Jobs feature on Google Cloud, are there any plans to make an Agent out of it? Am I correct in thinking that would be a great fit for prefect flows?

Is there an architectural diagram anywhere to explain how all of the different Prefect components fit together (e.g. agents, executors etc). Trying to wrap my head around some of the terminology and concepts after coming from a heavy Airflow background.

I had a thought this morning for a personal project that I could use Prefect for. As always I'd like to start with 2.0, but it leads me to a pretty important question about security. The idea I had would center around using my banking data via a python "faux API" (one of those screen scraper packages that tries to stand in place of an API). Obviously to use this, I would require two extremely sensitive secrets (card and password). Given that 2.0 is in a beta, I guess my question is, are Secrets currently in a place where I could insert these two pieces of data and use them in a Flow? Or am I better off for now using a different secret manager and accessing it from within a task?

question about

shell_run_command

in orion. I’m not able to pass a

retries

parameter to the task.

TypeError: got an unexpected keyword argument 'retries'

any ideas?

Hey Team! What's the prefecthq image latest release?

I definitely sound like a broken record at this point, but I'm still stuck on trying to understand the reasoning behind [these overloads](https://github.com/PrefectHQ/prefect/blob/orion/src/prefect/tasks.py#L231-L255) on the base Task class. Namely, the use of NoReturn in the first overload. Here's a minimal example where I use NoReturn in a Task skeleton, and the resulting

reveal_type

of a task run according to mypy: (threaded to avoid an obnoxiously long message)