prefect-community #prefect-cloud

Is there a way to get timestamps on the prefect agent logs? We are running the agent on a windows server using nssm, and the STDERR appears to have no timestamps.

Darren Liu

05/24/2023, 8:20 PM

Hi, I am using the new variables feature. It worked early on but now just freezes up indefinitely

Copy code

@flow(name="test", flow_run_name="dev > test")
def test():
    return variables.get("test")

Brayyan Yanes

05/24/2023, 8:24 PM

👋 Hello, team!

👋 3

Nicholas Torba

05/25/2023, 1:16 PM

Hello, Can someone point me to docs that can help me understand how cancellation of subflows works? The behavior I'm seeing is that when I cancel a flow with sublfows, the subflows don't get cancelled as well, so I end up with a lot of ghost flows continuously running, because I thought they would have been cancelled. Also, most of my sublows don't have an option to cancel. They only have the option to copy or delete (from the UI). What causes that behavior to a flow?

Nicholas Torba

05/26/2023, 3:14 PM

hey @Jenny has your team ever seen prefect agents running locally on a windows laptop stalling at the download code from storage phase? It is a local process, so trying to copy code from a local directory, but it never completes. The same code works on my ubuntu laptop. We tried running the agent in a docker container as well, but still stalls at the same step

Jane Jeong

05/26/2023, 8:50 PM

Reporting a bug - I have a cloud run job block that I used to have a command for. I've erased everything in the "command" section, but now runs of

prefect deployment

are failing with:

Copy code

pydantic.error_wrappers.ValidationError: 1 validation error for CloudRunJob
command
  value is not a valid list (type=type_error.list)

The value is now null - I don't think "command" should still be checked at this point

Jezreel Zamora

05/29/2023, 1:17 AM

Hello, Can someone explain to me how the

Zombie Killer

process works in V1? From what i understand, it should mark the flows as failed if the flows failed to send heartbeats on a X number of times. I’m testing this behavior with a flow that is running in ECS by simulating Out of Memory. I can see that the ECS Task failed and stopped running in AWS Console but the flow run in Prefect Cloud remains in Running state. This cause our schedule flows to not run until we manually set the status to Failed for the stuck flow. Is there anything I’m missing or something we need to setup to make this work properly?

Wei Hong Ho

05/29/2023, 10:47 AM

Hi, i am running a script every 2 minutes, trying to get market data from an API on an ec2. However, my EC2 suddenly crashed and I am unable to connect. Looking a htop, my computer ran out of CPU. looking at

ps ax

i realized, there's a lot of

prefect.engine

not sure whether they are zombie processes or not. any tips on how to prevent this in the future?

Zhang David

05/29/2023, 8:56 PM

Hi guys - I'm running into this error while using prefect cloud automation to send slack msgs on different states of flow run:

Copy code

"The notification block was invalid: KeyError(\"No class found for dispatch key 'slack-incoming-webhook' in registry for type 'Block'.\")"

Has anyone run into this as well ? Below is the full log:

Copy code

{
  "id": "77b5b58d-e9fa-4fff-bb5b-5c2157314dba",
  "account": "94a1563f-c82c-4049-854c-df6f4547b850",
  "event": "prefect-cloud.automation.action.failed",
  "occurred": "2023-05-29T21:52:20.401Z",
  "payload": {
    "reason": "The notification block was invalid: KeyError(\"No class found for dispatch key 'slack-incoming-webhook' in registry for type 'Block'.\")",
    "invocation": "8ff1536d-55b2-4170-9af4-10d6dd13243f",
    "action_type": "send-notification",
    "action_index": 0
  },
  "received": "2023-05-29T21:52:20.401Z",
  "related": [],
  "resource": {
    "prefect.resource.id": "prefect-cloud.automation.fcac6a63-01bb-4ade-94f0-e27664583542",
    "prefect-cloud.posture": "Reactive",
    "prefect.resource.name": "prod-wf-state"
  },
  "workspace": "1e3683bb-8412-4b7f-a2a8-c858d613af44"
}

✅ 1

Matthew Scanlon

05/30/2023, 1:18 PM

Can you pass pedantic models between flows? Furthermore, if you can, can you pass models that allow abstract types

✅ 1

Nicholas Torba

05/30/2023, 3:58 PM

Hey @Jenny (sorry for tagging you again, this seems to be the best way to get messages seen in this channel, lmk if there is a better option) I'm stuck on trying to figure out a reliable way to cancel a flow with subflows with python. I can cancel a parent flow via the api, via the cli, and via the UI, and in all three cases, my subflows are left in the running state. I can cancel each of them via the CLI, when i copy and paste their id's. I haven't been able to figure out how to get the subflow id's from the api response of the parent flow. Do you think you could help me figure out a solution for this? flow and subflow cancellation is a must for our app

Ying Sheng Lam

06/01/2023, 2:41 AM

Hi @Jenny wanted to ask if it's possible to exclude specific deployments within a flow here?

Matt Alhonte

06/05/2023, 11:17 PM

What would be the best way to access variables from within a Flow in a State Handler? I'm making a Flow that runs Jupyter notebooks, and on failure I want to post a link to the failed notebook file to slack. There's a task earlier in the flow that generates the filepath.

john

06/06/2023, 12:52 PM

Hi, I'm John. I'm planning on using Prefect Cloud free plan and was wondering what would happen to flow runs that take more than 7 days to run, given the 7 days history limit for the free plan? Are they automatically terminated and removed?

✅ 1

Nicholas Torba

06/07/2023, 4:54 PM

Hey Prefect Team, I am trying to trigger an email when my work queues go into an unhealthy state. I setup an email block and the automation, but don't see it triggered when I stop my agent from running and see the work queues go unhealthy. has anyone else had this problem?

Joseph Thickpenny Ryan

06/08/2023, 10:43 AM

Given the API rate limits that are in place, is there any way to see our actual API usage?

👀 2

Ahmed Maged

06/11/2023, 4:11 PM

Hi, i am having a problem, although I deployed my scripts to prefect cloud when they run while closing my vs code they show a status of late and do not run

Brett

06/12/2023, 9:36 PM

response from kubernetes when attempting to start a job

Brett

06/12/2023, 9:36 PM

I'm unable to start flows from the UI for my cluster - which is running

prefect 2.10.13

it appears that there is some error with using

metadata.generateName

where it is only generating the "-value". Example response in the snippet above. Forcing it back to

prefect 2.10.9

allows it to work properly. Is this a known bug?

Syed Areeb Amjed

06/14/2023, 8:01 AM

Hey, Wanted to ask that I am running a prefect flow that what does it do is reads the folder and in that the .txt files are converted to csv but then when I push to postgres it throws error

Gosia

06/14/2023, 8:11 AM

Hi, as of today I cannot log into my prefect cloud account. The magic sign-in link doesn’t arrive, of if it arrives, I get the ‘Code has expired or is invalid.’ error. Anybody else with this issue? Where can I get help with this?

alex

06/14/2023, 6:02 PM

Hello, I am running into an issue where my flows are not being scheduled as often as they should, leaving my work queue unhealthy. I'm using Kubernetes as my infrastructure, my agents are up and they are healthy. I can also confirm the queue is being polled actively. I have a concurrency limit of 9, but I am only seeing one flow at a time being scheduled and having a "Running" state. No task limits are being used. I'm on v2.10.9. How can I identify the cause of this bottleneck?

Robert Esteves

06/15/2023, 6:43 PM

Hi Everyone, I just wanted to check to see if anybody else is getting a 503 Error from Prefect Cloud. prefect.exceptions.PrefectHTTPStatusError: Server error '503 Service Unavailable' for url

hard same 1

✅ 1

Mitch

06/15/2023, 9:13 PM

Hey team - starting at 1250pm CT my flows in prefect cloud cannot find the files which are their entrypoints. Nothing configuration wise changed from what I am aware of.

Copy code

Importing flow code from 'XXXX.py:yy'
Flow could not be retrieved from deployment.

Sergey Zmeyev

06/15/2023, 9:44 PM

Hi I need technical support to regain admin access to our account.

✅ 1

Claire Scott

06/16/2023, 6:33 AM

Hrmm, I seem to be having the trouble I have a couple of months back, where (In this case, only one of) my deployments will generate the initial batch of future task runs if it's switched off and on again, but it will run through those without creating any new task runs, after the initial batch... the deployment in question is https://app.prefect.cloud/account/4a2c5dd6-f543-4178-b778-f81c8a3d23e7/workspace/0cf4[…]/deployments/deployment/0559c1a4-20d7-4206-871e-95bae464f97c. and the previous thread is at https://prefect-community.slack.com/archives/CL09KU1K7/p1682558283761859 is there anything going on with the scheduler processery thing atm?

Kees

06/19/2023, 11:59 AM

Hi guys, with the transition to Prefect2 we have lately running into some problems with prefect-gcp. While trying to query from BigQuery and store in a dataframe, BigQuery states "success" long before this is handled in Prefect. As a result, it can take up to 2 hours before the next task is started. Has anyone else ran into these problems? And is there a fix to this?

Ben Muller

06/20/2023, 1:02 AM

Hey team, I was looking to migrate my team over to the new

deployments ( old projects )

thingy, we store our docker images in ecr to run on ecs (AWS) and use git storage in github. I noticed on prefect-aws that the steps required for the build of the

prefect.yaml

doesnt have anything yet for a

build_docker_image

from ecs. Am I missing something or can I use a vanilla - prefect_docker.deployments.steps.build_docker_image:
and reference a ecr image and it will handle it?

Jey Farmer

06/20/2023, 3:06 PM

Trying to get k8s worker in AWS EKS Cluster connected to Prefect Cloud Account. • I’ve followed the docs here for using helm chart for prefect workers • Only difference is we’re using weave-gitops instead of installing the helm chart manually • I’ve added my API key (no expiration) and base64 encoded it, added as secret and ref’d the secret in helm chart • Ensured that the cloudApiConfig is filled out (i.e accountId , workspaceId) • Ensured that the workPool is filled out with workpool name ◦ I tried leaving this empty and then also specifying a k8s work pool I created in the ui • Wondering if my cloud url is proper ? I’ve tried 127.0.0.1 / 0.0.0.0 / and https://api.prefect.cloud/api ◦

apiUrl: <http://127.0.0.1:4200/api>

- this is default in the helm chart and docs don’t mention changing this Error I receive when looking at the logs of the worker

Copy code

Worker 'KubernetesWorker 3f629027-a4e8-4b08-b856-2e5c62e38933' started!
15:01:39.075 | DEBUG   | prefect.worker.kubernetes.kubernetesworker 3f629027-a4e8-4b08-b856-2e5c62e38933 - Exiting worker context...
15:01:39.075 | DEBUG   | prefect.worker.kubernetes.kubernetesworker 3f629027-a4e8-4b08-b856-2e5c62e38933 - Tearing down worker...
15:01:39.076 | DEBUG   | MainThread   | prefect._internal.concurrency - <AsyncCancelScope, name='start' COMPLETED, runtime=2.02> exited
15:01:39.076 | DEBUG   | MainThread   | prefect._internal.concurrency - Encountered exception in async call start(worker_name=None, work_pool_name='jey-test-custom-k8s-work-pool', work_queues=[], worker_type='kuber...)
Traceback (most recent call last):

..... shortened logs.....

prefect.exceptions.PrefectHTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/><REMOVED_ACCOUNT_ID>/workspaces/<REMOVED_WORKSPACE_ID>/work_pools/jey-test-custom-k8s-work-pool'
Response: {'detail': 'Unauthorized'}
For more information check: <https://httpstatuses.com/403>

Mr Eli W Jones

06/20/2023, 11:36 PM

Hmm, maybe someone in here can help me since we are using Prefect Cloud currently. Say I have 1000 flows that all share the same lib module that gets uploaded to our S3 storage block during the normal prefect deployment build/apply step. Do I need to re-deploy every flow whenever the shared module changes (so we can track the versions)? Is there a cleaner, prefect approved way to do this?: https://prefect-community.slack.com/archives/C048ZHT5U3U/p1687204304186759