https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-cloud
  • c

    Carlos Cueto

    02/10/2023, 1:54 PM
    Woke up this morning to the Cloud's Prefect 2.8.0 update with work pools. I know that all existing work-queues went into the default work pool, however, flow runs are being tagged with either
    work queue
    or
    work pool
    in what seems to be random. I liked the old way of seeing the work queue because it gave me an idea of the execution environment of said flow. Is this intended?
    :upvote: 1
    👍 1
    m
    • 2
    • 3
  • t

    Tanay Kothari

    02/12/2023, 3:18 AM
    Hey Prefect team,
    Task run '<id>' received abort during orchestration: This run cannot transition to the RUNNING state from the RUNNING state. Task run is in RUNNING state.
    
    prefect.exceptions.MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.
    I’m getting this error. I have a Dask cluster on EKS. When I run the prefect flow locally, it connects to the dask cluster and runs it just fine. When I deploy it to the cloud, my prefect agent on Fargate runs the first task in my DAG and then crashes with this error a few seconds later. I’m using Prefect 2.8.0.
    • 1
    • 1
  • a

    Alex Turek

    02/13/2023, 6:29 PM
    where is Prefect Cloud’s maintenance schedule documented?
    c
    j
    • 3
    • 12
  • s

    Stéphan Taljaard

    02/13/2023, 8:24 PM
    set the channel description: Hi 👋 - post comments & code in 🧵 - ☁️ docs: https://docs.prefect.io/ui/cloud/ - status updates: https://prefect.status.io/
    🙌 1
    :thank-you: 2
    👍 1
  • d

    Data Ops

    02/14/2023, 2:13 PM
    Hi everyone! I need speak with someone from financial department about our plan (Logcomex). Our Contact is Richard Christie
    ✅ 1
    r
    • 2
    • 1
  • j

    Joseph Thickpenny Ryan

    02/14/2023, 4:22 PM
    Is there somewhere I can subscribe to upcoming change notifications for Prefect Cloud? I'm working on a v1 to v2 migration at the moment and was confused when work queues went missing (I've found them now) from the UI since the last time I looked
    ✅ 1
    r
    • 2
    • 1
  • s

    Sidharth Ramalingam

    02/15/2023, 3:36 PM
    Hi community is there a way where I could run a deployment on prefect cloud without keeping my local PC switched on. My question is after creating a deployment and scheduling will the flow run even when my PC is disconnected.
    ✅ 1
    s
    • 2
    • 5
  • j

    Jai P

    02/15/2023, 4:01 PM
    hi there! through some wackiness, i've ended up with a couple of prefect cloud accounts based on the aliases of my e-mail address. Is there a way in prefect cloud that i can request for one of those accounts to be deleted?
  • j

    Joseph Thickpenny Ryan

    02/15/2023, 4:08 PM
    Is the automations limit of 3 per workspace a free tier limit or a forever limit?
    ✅ 1
    w
    • 2
    • 2
  • n

    Nikhil Jain

    02/15/2023, 8:31 PM
    Hi, we are getting some spurious alerts from prefect-cloud: This alert is configured to trigger if a flow-run remains in
    Late
    or
    Pending
    state for over 10 minutes, but we are seeing this even if the flow completed successfully and in less than 10 minutes.
    Prefect flow Late or Pending for over 10 minutes
    Flow run build-meta/spectral-anteater entered state
    Completed
    at 2023-02-15T19:39:53.508606+00:00.
    Flow run URL: https://app.prefect.cloud/account/adee12e2-25d9-428a-9a77-32d8d7ca2ac7/workspace/28562586-0812-4e92-803f-99ce64384cbd/flow-runs/flow-run/81433951-84a0-4bb1-95e9-22a942912d3a
    State message: All states completed.
    w
    c
    i
    • 4
    • 7
  • t

    Thet Naing

    02/16/2023, 1:43 AM
    Since the new waterfall visualization was introduced to Prefect Cloud, we've been experiencing frequent freezing and crashing on the UI when using
    map
    for tasks. Is there a way to simplify the visualization so that it doesn't show every single task mapping on the UI? It also makes it rather difficult to see and understand what's happening in the flow
    s
    • 2
    • 6
  • j

    JP

    02/16/2023, 9:06 AM
    Heya! Last night we received quite many slack notifications of flow being healthy from single Automation over span of roughly 2 hours. All but 2 were with valid payload 😕 The amount of notifications are to be attributed to following config • Trigger Type: Work queue health • _*Work Queues*_: <single production work queue> • _*Work Queue*_: Stays in > Healthy • _*For*_: 10 Seconds Our Slack notification template looks as follows
    Name: {{ work_queue.name }}
    Last polled: {{ work_queue.last_polled }}
    Late run count: {{ work_queue.late_runs_count }}
    URL: {{ work_queue|ui_url }}
    But.. why are all but 2 notifications with no values? (see screenshot below). The two valid notifications are 1. The first one at 21:53 CEST 2. At 22:54 CEST (visible in screenshot) edit: For the record.. We do not intend to run such trigger condition (it was a mis-configuration). We do not really care being notified every 10 seconds about a queue being healthy 😅 Hence the subject of the notifications in screenshot is mis-leading
    ✅ 1
    w
    s
    • 3
    • 10
  • k

    karteek

    02/16/2023, 4:05 PM
    Hey guys! Since yesterday we've been having issues with new runs being scheduled. We've noticed this issue in the past once or twice but not this severe. Any idea what might be happening? We are on prefect 1.0 still.
  • n

    nicholasnet

    02/16/2023, 5:54 PM
    Is there anyway to restrict Prefect cloud account by API keys. Basically I am trying to find the easy way to deploy code to prod and non-prod env by API keys such that one env API key cannot access another.
    s
    • 2
    • 2
  • a

    Aaron Gonzalez

    02/16/2023, 10:53 PM
    👋 everyone! QQ. I am getting ready to build a little deployment that gsutil rsync's a bunch of data from s3 to gcs. Unfortunately the data in s3 doesn't really follow a nice partition hierarchy so i can't just
    gsutil rsync <s3://some-key/dt=yyyy-mm-dd/>
    <gs://some-key/dt=yyyy-mm-dd/>
    😢 I am going to give
    prefect-shell
    a try for the first time and want to know if people have had a lot of experience with it? For my use case I have about 12K different rsync's I am going to need to run and I don't know which of these patterns is preferable:
    for src in s3_sources_12k:
        dest = f'<gs://some-dest/{src}>'
        ShellOperation(
            commands=[f"gsutil rsync -r {src} {dest}"],
            env=env_var_map,
        ).run()
    or
    with ShellOperation(
        commands=[
            "gsutil rsync -r src1 dest1",
            "gsutil rsync -r src2 dest2",
            "gsutil rsync -r src3 dest3",
            ...
            "gsutil rsync -r src12k dest12k",
        ],
        env=env_var_map,
    ) as shell_operation:
        shell_process = shell_operation.trigger()
        shell_process.wait_for_completion()
        shell_output = shell_process.fetch_result()
    a
    • 2
    • 6
  • t

    Thet Naing

    02/17/2023, 7:45 PM
    Hi all! We suddenly started to have issues with flow runs on Prefect Cloud not picking up on their input parameters. Our flow runs are getting
    null
    for all parameters, even when there are defaults set and when we choose
    Customize Run
    to input parameters. Is this a known issue? This seems to have begun when the latest release of Prefect was pushed, about 3 hours ago.
    j
    • 2
    • 2
  • c

    Chris Whatley

    02/17/2023, 9:27 PM
    I just went in to cloud for the first time in a day or two and it looks like my notification rules were migrated. They didn’t pick up a failed run from this AM. Additionally, the notification method (slack) is now sealed up in an “anonymous block” that I can’t see or edit. Do I need to recreate the automations?
    ✅ 1
    w
    c
    • 3
    • 7
  • t

    Thet Naing

    02/20/2023, 3:38 PM
    Hey folks 👋 I've been playing around with custom Prefect logging. It works great when I am running flows locally, but I cannot seem to get my deployments to pick up a custom
    logging.yml
    set with
    PREFECT_LOGGING_SETTINGS_PATH
    . Does anyone have examples of how this is done for cloud deployments?
  • a

    Adam Gold

    02/20/2023, 3:58 PM
    Hello everyone! Prefect looks awesome — but I’ve been having some issues with flow runs. I am running the agent as a kubernetes pod with
    prefect agent start --pool "$PREFECT_ENV" --work-queue default
    . 1. It takes really long for the flow to be submitted. Notice the time here is more than 30 seconds for the task to be created, before even running:
    15:25:43.676 | INFO    | prefect.agent - Submitting flow run '0dda37d3-87e4-46e2-9266-920e7dae9113'
    15:25:44.499 | INFO    | prefect.infrastructure.process - Opening process 'congenial-falcon'...
    15:25:44.998 | INFO    | prefect.agent - Completed submission of flow run '0dda37d3-87e4-46e2-9266-920e7dae9113'
    <frozen runpy>:128: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
    15:26:01.483 | INFO    | Flow run 'congenial-falcon' - Downloading flow code from storage at '/app'
    15:26:16.402 | INFO    | Flow run 'congenial-falcon' - Created task run 'return_value-0' for task 'return_value'
    2. It downloads the code for every flow, making the pod go out of memory very quickly:
    Pod ephemeral local storage usage exceeds the total limit
    I am probably missing something here, but would love some help 🙏
    t
    • 2
    • 4
  • a

    Aaron Gonzalez

    02/20/2023, 4:07 PM
    QQ for those with experience using GCP's Cloud Run parallelism abilities. I am going to try to run a bunch of commands in parallel, but I am not sure if I should first try to keep everything in Prefect-land and try to use the ``ConcurrentTaskRunner`` and maybe chunk my commands into groups of like 10 (10 would be more than enough).........or if I should try to use the native parallel options available to Cloud Run.
    You can structure a job as a single task or as multiple, independent tasks (up to 10,000 tasks) that can be executed in parallel. Each task runs one container instance and can be configured to retry in case of failure. Each task is aware of its index, which is stored in the
    CLOUD_RUN_TASK_INDEX
    environment variable. The overall count of tasks is stored in the
    CLOUD_RUN_TASK_COUNT
    environment variable. If you are processing data in parallel, your code is responsible for determining which task handles which subset of the data.
    t
    • 2
    • 2
  • e

    Emil Ostergaard

    02/22/2023, 1:13 PM
    Hi everyone! I’ve started using automations in prefect cloud, and set up an automation that should notify me (pr. email) when a stuck in state “Running”, “Paused”, “Late”, “Resuming” or “Pending” for more than 10 minutes. However, for some reason I occasionally get notifications from this automation when some flow runs completes. Anyone experiencing the same behavior? Any suggestion on how to stop this? Notification example: Flow run get_data entered state
    Completed
    at 2023-02-22T11:07:44.757581+00:00. Flow ID: x Flow run ID: y Flow run URL: z State message: All states completed. Using prefect 2.8.1
    i
    • 2
    • 2
  • c

    Carlos Cueto

    02/22/2023, 2:33 PM
    Is there a way to see the flow runs for a flow that occurred more than 7 days ago? We have some flows that only run once a month and can't tell whether they successfully ran last time if it was more than 7 days ago.
    ✅ 1
    c
    s
    • 3
    • 6
  • t

    Tushar Gupta

    02/22/2023, 4:05 PM
    Hello everyone Quick question, I want to ask please help me out, I am unable to find out. I have a project named ABC in which flow name is let say XYZ now whenever I run my XYZ flow as everyone knows prefect give random names to the runs. Now I want to get the time at which the latest success run occurred using python. Please help me out. Thanks in advance!
  • j

    Jason Vertrees

    02/22/2023, 4:34 PM
    We have a rather urgent problem with Prefect Cloud v1.0. We have zombie tasks that cannot be viewed in the UI, but are queryable via the API and are actively blocking resources. Though we can SEE these zombie tasks we cannot kill them; when we try, we see:
    [{'path': ['set_task_run_states'], 'message': "'NoneType' object has no attribute 'flow_id'", 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
    Please help.
  • s

    Stéphan Taljaard

    02/22/2023, 6:08 PM
    Take note of the upcoming Prefect Cloud 2 maintenance window:
    :thank-you: 4
    👀 1
    :upvote: 1
    a
    • 2
    • 1
  • a

    Andrew Richards

    02/23/2023, 4:28 PM
    My agents are not able to push logs to the Prefect API. I’m on 2.7.9. Am I hitting a rate limit for the API?
    --- Orion logging error ---
    Traceback (most recent call last):
      File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/prefect/logging/handlers.py", line 151, in send_logs
        await client.create_logs(self._pending_logs)
      File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/prefect/client/orion.py", line 1843, in create_logs
        await <http://self._client.post|self._client.post>(f"/logs/", json=serialized_logs)
      File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/httpx/_client.py", line 1855, in post
        extensions=extensions,
      File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/httpx/_client.py", line 1527, in request
        return await self.send(request, auth=auth, follow_redirects=follow_redirects)
      File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/prefect/client/base.py", line 253, in send
        response.raise_for_status()
      File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/httpx/_models.py", line 736, in raise_for_status
        raise HTTPStatusError(message, request=request, response=self)
    httpx.HTTPStatusError: Client error '429 Too Many Requests' for url '<https://api.prefect.cloud/api/accounts/><redacted>/workspaces/<redacted>/logs/'
    j
    • 2
    • 1
  • d

    Data Ops

    02/23/2023, 9:06 PM
    Hi everyone. Is there a problem with site cloud.prefect.io ? Our screen is like that:
    ✅ 1
    j
    • 2
    • 2
  • d

    Data Ops

    02/23/2023, 9:07 PM
    image.png
  • j

    justabill

    02/24/2023, 2:24 PM
    👋 Hi everyone, Heads up - for that past week, we've been chasing down the root cause of an issue causing Cloud 2 automations with a “Stays in” trigger to intermittently misfire. This issue only causes false positives - unwanted triggers - not false negatives - omitted triggers. So while this issue is creating unnecessary noise, the triggers do not miss any cases in which their condition is satisfied. We have made some progress towards a resolution, but we don't have a clear path to a complete fix. We'll provide an update as soon as we're confident that we've resolved the issue. In the meantime, we appreciate your understanding and patience.
  • a

    Aaron Gonzalez

    02/24/2023, 4:57 PM
    Something I just noticed was that some deployments I just pushed to Prefect Cloud have parameters that don't seem to be showing up correctly in the UI. In my
    deployments.py
    I wrote:
    def deploy_factory(name:str, flow, param: dict = dict(), cron: str = '') -> Deployment:
    
        kwargs = {
            'flow': flow,
            'name': name,
            'infrastructure': CloudRunJob.load(f"aether-flows-cloud-run-job-{ENV}"),
            'work_queue_name': f"{ENV}_aether",
            'version': VERSION,
            'output': f'{name}.yaml',
            'skip_upload': True,
        }
    
        if param:
            kwargs['parameters'] = param,
        if cron:
            kwargs['schedule'] = CronSchedule(cron=cron)
    
        return Deployment.build_from_flow(**kwargs)
    
    
    aether_space_metrics_1h_parquet_rsync = deploy_factory(
        name=f'aether_space_metrics_1h_parquet_rsync_{ENV}',
        flow=space_metrics_parquet_rsync,
        param={'date': 'current', 'agg_window': '1h'},
        cron=sched('0 23 * * *')
    )
    aether_space_metrics_1h_parquet_rsync.apply(work_queue_concurrency=20)
    • 1
    • 1
Powered by Linen
Title
a

Aaron Gonzalez

02/24/2023, 4:57 PM
Something I just noticed was that some deployments I just pushed to Prefect Cloud have parameters that don't seem to be showing up correctly in the UI. In my
deployments.py
I wrote:
def deploy_factory(name:str, flow, param: dict = dict(), cron: str = '') -> Deployment:

    kwargs = {
        'flow': flow,
        'name': name,
        'infrastructure': CloudRunJob.load(f"aether-flows-cloud-run-job-{ENV}"),
        'work_queue_name': f"{ENV}_aether",
        'version': VERSION,
        'output': f'{name}.yaml',
        'skip_upload': True,
    }

    if param:
        kwargs['parameters'] = param,
    if cron:
        kwargs['schedule'] = CronSchedule(cron=cron)

    return Deployment.build_from_flow(**kwargs)


aether_space_metrics_1h_parquet_rsync = deploy_factory(
    name=f'aether_space_metrics_1h_parquet_rsync_{ENV}',
    flow=space_metrics_parquet_rsync,
    param={'date': 'current', 'agg_window': '1h'},
    cron=sched('0 23 * * *')
)
aether_space_metrics_1h_parquet_rsync.apply(work_queue_concurrency=20)
I'm currently still testing by launching some custom runs where I am manually entering a date (e.g.
2023-02-01
) and agg_window (
1h
), but I'm curious if this is just a UI bug or an actual prod run with default params is going to use what I defined in code 🤔
View count: 2