Hey guys! Since yesterday we've been having issues with new runs being scheduled. We've noticed this issue in the past once or twice but not this severe. Any idea what might be happening? We are on prefect 1.0 still.

Is there anyway to restrict Prefect cloud account by API keys. Basically I am trying to find the easy way to deploy code to prod and non-prod env by API keys such that one env API key cannot access another.

👋 everyone! QQ. I am getting ready to build a little deployment that gsutil rsync's a bunch of data from s3 to gcs. Unfortunately the data in s3 doesn't really follow a nice partition hierarchy so i can't just

gsutil rsync <s3://some-key/dt=yyyy-mm-dd/>

<gs://some-key/dt=yyyy-mm-dd/>

😢 I am going to give

prefect-shell

a try for the first time and want to know if people have had a lot of experience with it? For my use case I have about 12K different rsync's I am going to need to run and I don't know which of these patterns is preferable:

for src in s3_sources_12k:
    dest = f'<gs://some-dest/{src}>'
    ShellOperation(
        commands=[f"gsutil rsync -r {src} {dest}"],
        env=env_var_map,
    ).run()

or

with ShellOperation(
    commands=[
        "gsutil rsync -r src1 dest1",
        "gsutil rsync -r src2 dest2",
        "gsutil rsync -r src3 dest3",
        ...
        "gsutil rsync -r src12k dest12k",
    ],
    env=env_var_map,
) as shell_operation:
    shell_process = shell_operation.trigger()
    shell_process.wait_for_completion()
    shell_output = shell_process.fetch_result()

Hi all! We suddenly started to have issues with flow runs on Prefect Cloud not picking up on their input parameters. Our flow runs are getting

null

for all parameters, even when there are defaults set and when we choose

Customize Run

to input parameters. Is this a known issue? This seems to have begun when the latest release of Prefect was pushed, about 3 hours ago.

I just went in to cloud for the first time in a day or two and it looks like my notification rules were migrated. They didn’t pick up a failed run from this AM. Additionally, the notification method (slack) is now sealed up in an “anonymous block” that I can’t see or edit. Do I need to recreate the automations?

Hey folks 👋 I've been playing around with custom Prefect logging. It works great when I am running flows locally, but I cannot seem to get my deployments to pick up a custom

logging.yml

set with

PREFECT_LOGGING_SETTINGS_PATH

. Does anyone have examples of how this is done for cloud deployments?

Hello everyone! Prefect looks awesome — but I’ve been having some issues with flow runs. I am running the agent as a kubernetes pod with

prefect agent start --pool "$PREFECT_ENV" --work-queue default

. 1. It takes really long for the flow to be submitted. Notice the time here is more than 30 seconds for the task to be created, before even running:

15:25:43.676 | INFO    | prefect.agent - Submitting flow run '0dda37d3-87e4-46e2-9266-920e7dae9113'
15:25:44.499 | INFO    | prefect.infrastructure.process - Opening process 'congenial-falcon'...
15:25:44.998 | INFO    | prefect.agent - Completed submission of flow run '0dda37d3-87e4-46e2-9266-920e7dae9113'
<frozen runpy>:128: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
15:26:01.483 | INFO    | Flow run 'congenial-falcon' - Downloading flow code from storage at '/app'
15:26:16.402 | INFO    | Flow run 'congenial-falcon' - Created task run 'return_value-0' for task 'return_value'

2. It downloads the code for every flow, making the pod go out of memory very quickly:

Pod ephemeral local storage usage exceeds the total limit

I am probably missing something here, but would love some help 🙏

QQ for those with experience using GCP's Cloud Run parallelism abilities. I am going to try to run a bunch of commands in parallel, but I am not sure if I should first try to keep everything in Prefect-land and try to use the ``ConcurrentTaskRunner`` and maybe chunk my commands into groups of like 10 (10 would be more than enough).........or if I should try to use the native parallel options available to Cloud Run.

You can structure a job as a single task or as multiple, independent tasks (up to 10,000 tasks) that can be executed in parallel. Each task runs one container instance and can be configured to retry in case of failure. Each task is aware of its index, which is stored in the

CLOUD_RUN_TASK_INDEX

environment variable. The overall count of tasks is stored in the

CLOUD_RUN_TASK_COUNT

environment variable. If you are processing data in parallel, your code is responsible for determining which task handles which subset of the data.

Hi everyone! I’ve started using automations in prefect cloud, and set up an automation that should notify me (pr. email) when a stuck in state “Running”, “Paused”, “Late”, “Resuming” or “Pending” for more than 10 minutes. However, for some reason I occasionally get notifications from this automation when some flow runs completes. Anyone experiencing the same behavior? Any suggestion on how to stop this? Notification example: Flow run get_data entered state

Completed

at 2023-02-22T110744.757581+00:00. Flow ID: x Flow run ID: y Flow run URL: z State message: All states completed. Using prefect 2.8.1

Is there a way to see the flow runs for a flow that occurred more than 7 days ago? We have some flows that only run once a month and can't tell whether they successfully ran last time if it was more than 7 days ago.

Hello everyone Quick question, I want to ask please help me out, I am unable to find out. I have a project named ABC in which flow name is let say XYZ now whenever I run my XYZ flow as everyone knows prefect give random names to the runs. Now I want to get the time at which the latest success run occurred using python. Please help me out. Thanks in advance!

We have a rather urgent problem with Prefect Cloud v1.0. We have zombie tasks that cannot be viewed in the UI, but are queryable via the API and are actively blocking resources. Though we can SEE these zombie tasks we cannot kill them; when we try, we see:

[{'path': ['set_task_run_states'], 'message': "'NoneType' object has no attribute 'flow_id'", 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

Please help.

Take note of the upcoming Prefect Cloud 2 maintenance window:

My agents are not able to push logs to the Prefect API. I’m on 2.7.9. Am I hitting a rate limit for the API?

--- Orion logging error ---
Traceback (most recent call last):
  File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/prefect/logging/handlers.py", line 151, in send_logs
    await client.create_logs(self._pending_logs)
  File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/prefect/client/orion.py", line 1843, in create_logs
    await <http://self._client.post|self._client.post>(f"/logs/", json=serialized_logs)
  File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/httpx/_client.py", line 1855, in post
    extensions=extensions,
  File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/prefect/client/base.py", line 253, in send
    response.raise_for_status()
  File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/httpx/_models.py", line 736, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '429 Too Many Requests' for url '<https://api.prefect.cloud/api/accounts/><redacted>/workspaces/<redacted>/logs/'

Hi everyone. Is there a problem with site cloud.prefect.io ? Our screen is like that:

image.png

👋 Hi everyone, Heads up - for that past week, we've been chasing down the root cause of an issue causing Cloud 2 automations with a “Stays in” trigger to intermittently misfire. This issue only causes false positives - unwanted triggers - not false negatives - omitted triggers. So while this issue is creating unnecessary noise, the triggers do not miss any cases in which their condition is satisfied. We have made some progress towards a resolution, but we don't have a clear path to a complete fix. We'll provide an update as soon as we're confident that we've resolved the issue. In the meantime, we appreciate your understanding and patience.

Something I just noticed was that some deployments I just pushed to Prefect Cloud have parameters that don't seem to be showing up correctly in the UI. In my

deployments.py

I wrote:

def deploy_factory(name:str, flow, param: dict = dict(), cron: str = '') -> Deployment:

    kwargs = {
        'flow': flow,
        'name': name,
        'infrastructure': CloudRunJob.load(f"aether-flows-cloud-run-job-{ENV}"),
        'work_queue_name': f"{ENV}_aether",
        'version': VERSION,
        'output': f'{name}.yaml',
        'skip_upload': True,
    }

    if param:
        kwargs['parameters'] = param,
    if cron:
        kwargs['schedule'] = CronSchedule(cron=cron)

    return Deployment.build_from_flow(**kwargs)


aether_space_metrics_1h_parquet_rsync = deploy_factory(
    name=f'aether_space_metrics_1h_parquet_rsync_{ENV}',
    flow=space_metrics_parquet_rsync,
    param={'date': 'current', 'agg_window': '1h'},
    cron=sched('0 23 * * *')
)
aether_space_metrics_1h_parquet_rsync.apply(work_queue_concurrency=20)

i'm having a bit of issues with the docs, i'm on the latest version of prefect and i don't understand why not all of the methods that are documented in docs are actually present when trying to use the prefect cli, here for example trying to use prefect cloud cli.. ex: where is the

check_key_is_valid_for_login

call or all the other ones that are in the docs but not actually available in the api

Hi Vera, those API docs are autogenerated and represent the Python code that's in the CLI modules, but not all of the methods are runnable via the CLI. That's a bit confusing, so I'll see if we can make some changes to ensure only commands you can run via the CLI show up in the CLI section of the docs. For now, I think the best way to see all available commands is to use

--help

on the CLI itself; so, for example,

prefect cloud --help

will show you all the cloud-related CLI commands.

HELP~ I'm having an issue where prefect cloud doesn't log anything. whereas Orion does with the same script. (Can see it in the attached screen shots) The Script:

from prefect import flow

@flow(log_prints=True)
def hello():
    print("whats up prefectttt!!")


if __name__ == "__main__":
    hello()

What I've already done: • Create and work in a virtual env where prefect is up to date. • I've set the PREFECT_API_URL so it is connected to my prefect-cloud workspace (Link) • Connected to prefect-cloud via CLI with

prefect cloud login

I've looked around and have yet to find anyone with this specific situation....

I have 3 flows for a data workflow (flow1=extracts the data, flow2= makes data ready, flow 3= creates training set) For the next step of work, we want to run this workflow (3 flows) in parallel for different parameters +1000 times . Our concerns are: 1- memory 2- how to set up the pipeline to run +1000 flows in parallel + run the subflows sequentially Any thoughts?

I read somewhere that Prefect Cloud task runs are limited to 10,000/month on the free tier. Is this true? I don't see that limit listed on the pricing page currently.

Hello! Yesterday at 13:00 PM we got a flow that had a Bad Gateway (502) crash towards the Cloud API. Either way, what would be the best practice to catch these sort of errors and retry the flow automatically? This seems to be low level prefect lib crash, hence not sure where to even start..

Feb 27 13:00:05 dataplatform-jp-sandbox python[105783]: httpx.HTTPStatusError: Server error '502 Bad Gateway' for url '<https://api.prefect.cloud/api/accounts/c1397d5f-b9f3-49e8-abb6-bce7d7b1412e/workspaces/32dfe242-315b-4405-b06d-8b6308d6b631/flow_runs/d3f64d7f-acef-46f4-8114-8b1b1824e2c5>'

Full trace in attachment

Heya! I'm trying to use Prefect 2 REST API to create a flow run. With the help of API docs api/accounts/{account_id}/workspaces/{workspace_id}/flow_runs/ I am able to get the flow run registered on Prefect Cloud, but I can not get it to be put into a work queue for my agent to pick it up 😕

curl --request POST \
  --url "<https://api.prefect.cloud/api/accounts/$PREFECT_ACCOUNT_ID/workspaces/$PREFECT_WORKSPACE_ID/flow_runs>" \
  --header "Authorization: Bearer $PREFECT_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "flow_id": "0c6a7af0-e534-44dd-a88a-151db5c289fc",
	"deployment_id": "03e816d7-38f9-4c1f-b036-efc37a1084c1"
}'

The flow just sits there, queue-less in PENDING state. How can I tell it to place the flow run in a queue for one of my agent's to pick it up and run it? Alternatively, since the deployment is associated with a queue, why is Prefect not putting this flow there implicitly? However, the method described by Anna G. in a blog post "How to trigger a flow run from a deployment via API call using Python requests library or from a terminal using curl?" works!

❯ curl --request POST \
  --url "<https://api.prefect.cloud/api/accounts/$PREFECT_ACCOUNT_ID/workspaces/$PREFECT_WORKSPACE_ID/deployments/$PREFECT_DEPLOYMENT_ID/create_flow_run>" \
  --header "Authorization: Bearer $PREFECT_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{"name": "your_flow_run", "state": {"type": "SCHEDULED"}}'

Is the API docs wrong, and to create a flow, one must POST to

/create_flow_run

and not

/flow_runs

? 😕 P.S. Originally posted in #prefect-community, but now moved to #prefect-cloud

Trying to transfer a workspace in Prefect Cloud (app.prefect.cloud) but the UI just hangs without populating the list of potentiall transferees. Is this a known issue?

Hi all, anybody else having problems using prefect 2 in Safari ? I am now just getting an empty window when I click on flows in Safari. In Chrome everything seems to work fine.

hey all a question. We're currently moving our Prefect setup from GCP to AWS. When pushing up the flows in DockerStorage, it takes in a

registry_url

. GCR supports virtual folders, so i can have

<http://gcr.io/{project}/flows|gcr.io/{project}/flows>

be the

registry_url

and each flow will be appended after that to have

<http://gcr.io/{project}/flows/{flow1_name}/|gcr.io/{project}/flows/{flow1_name}/>

and

<http://gcr.io/{project}/flows/{flow2_name}/|gcr.io/{project}/flows/{flow2_name}/>

etc it doesn't seem like ECR supports this suffix style naming, how do people resolve this? do you create a separate repository per flow? or is there a way to get around that

Hi all, While i execute flow manually ( from my IDE ) it works but the same if i run from the cloud UI it gets scheduled but moved to "Late" state or not running. Help Appreciated

image.png