• s

    Shawn Marhanka

    1 year ago
    Hi Prefect! On Prefect Cloud, is there a limit to how many runs you can have in scheduled state or queued up to be ran by the next available agent (we have 2 slots). My co-worker tried to queue 80 runs last night and said not all of them made it. Thank you!
    s
    Jim Crist-Harif
    5 replies
    Copy to Clipboard
  • Raghav Thirumulu

    Raghav Thirumulu

    1 year ago
    hi everyone! i'm attempting to get DBT shell tasks running via prefect. attaching my code below. anyone have any ideas on what i'm getting wrong? i can confirm that the
    profiles.yml
    was created correctly. im able to run dbt commands with the newly created
    profiles.yml
    outside of the prefect package
    Raghav Thirumulu
    h
    +1
    9 replies
    Copy to Clipboard
  • h

    Hui Zheng

    1 year ago
    Hello, Is there a way to set the MaximumDeadline for a flow, such that regardless retries or long-running tasks. The total duration of the flow shall not exceed certain amount of time. Once exceeded, the flow run will be marked as
    FAIL
    ?
    h
    1 replies
    Copy to Clipboard
  • Matt Drago

    Matt Drago

    1 year ago
    Hey folks, I'm looking for some guidance on using the
    GCRResult
    . I have Prefect running in a kubernetes cluster, using the
    KubernetesRun
    with
    Docker
    storage. I've set the created the
    GCRResult
    using:
    result = GCSResult(
        bucket='REDACTED',
        location='flow-results/{flow_name}/{date:%Y/%m/%d}/{task_name}_{task_run_id}.prefect'
    )
    And assigned the result to the flow with:
    with Flow("Redacted", schedule=schedule, result=result) as flow:
    Weird thing is that the bucket name is being used to store the Result files, but not using the location template that I have provided. One thing that I noticed was that for one task (a function with the
    @task
    decorator), I passed in an argument with the name
    date
    and that Task did have it's Results stored in the location according to the template.
    Matt Drago
    Chris White
    9 replies
    Copy to Clipboard
  • Marc Lipoff

    Marc Lipoff

    1 year ago
    what iam permissions are needed for an ecs agent? A complete list would be great, rather than trying to go one by one. ATM, I'm getting this error
    An error occurred (UnauthorizedOperation) when calling the DescribeVpcs operation: You are not authorized to perform this operation.
    Marc Lipoff
    Kyle Moon-Wright
    +1
    6 replies
    Copy to Clipboard
  • Amanda Wee

    Amanda Wee

    1 year ago
    Hi friends, a question on memory usage. Currently, my team is only using Prefect Core, which we setup as a bunch of ECS tasks (each corresponding to what Prefect Cloud/Server calls a project) that runs the flows using flow.run() with a schedule. It works, but we found that the memory consumed by the flows was not released after the flow ended. As a workaround, we kill the ECS task after detecting that all the flows for that task have been run for the day, and then ECS kindly restarts the task for us with the memory usage reset. Not ideal, but the hack works so that subsequent flows for the day can reuse the memory released to the EC2 instance. In the coming few days, we're transitioning to Prefect Server using a single local agent with multiple projects, with both agent and flows (since the agent is local) in one ECS task. At startup, the ECS task will create the projects and register the flows, then start the local agent. Eventually we aim to move to use an ECS agent with Fargate, but for now: will we experience memory hogging with this setup?
    Amanda Wee
    Chris White
    3 replies
    Copy to Clipboard
  • m

    mithalee mohapatra

    1 year ago
    if name == 'main':     schedule = Schedule(clocks=[IntervalClock(start_date=pendulum.datetime(2020, 12, 16,hour=2,minute=14,second=0),interval=timedelta(minutes=2))]) #schedule.next(5) with Flow("Hello", schedule) as flow:     say_hello() executor=LocalDaskExecutor(scheduler="processes", num_workers=6) flow.run(executor=executor) I am trying to run LocalDaskExecutor with prefect schedule but getting the below error. Just Flow.run() works fine with schedule though. with Flow("Hello", schedule) as flow: NameError: name 'schedule' is not defined
    m
    Chris White
    6 replies
    Copy to Clipboard
  • m

    Mitchell Bregman

    1 year ago
    Has anyone gotten this error from Docker API when registering flow to cloud?
    docker.errors.APIError: 500 Server Error for <https://35.229.119.149:2376/v1.32/build?t=xxx-xxx.jfrog.io%2Fprefect%2Fprefect_qisvgvrv-compare_sqlserver_and_snowflake%3A0.0.0&q=False&nocache=False&rm=False&forcerm=True&pull=False&dockerfile=Dockerfile>: Internal Server Error ("Syntax error - can't find = in "Driver". Must be of the form: name=value")
    m
    1 replies
    Copy to Clipboard
  • Алексей Филимонов

    Алексей Филимонов

    1 year ago
    Hi folks! Sometimes we face a problem: flow starts few times (according to the logs) even though in the timeline we can see only one run. The same about tasks inside flow - sometimes they start few times.Expected behavior: flow (and tasks) starts one time using one pid. Additional info: prefect server
    v 0.13.12
    , flows doesn't contain map and task runs without restart option.
    Алексей Филимонов
    Allyson Lubimir
    2 replies
    Copy to Clipboard