• Michael Welsh

    Michael Welsh

    1 year ago
    Hi general question here When running flows lately I have been noticing lately long waits for workers, hour+, and when I do happen to get a worker upon creation of a cluster its only ever one worker. Is this common? I tried just creating coiled clusters and they have no problem getting workers. Any help is appreciated thanks.
    Michael Welsh
    Kevin Kho
    14 replies
    Copy to Clipboard
  • Aric Huang

    Aric Huang

    1 year ago
    Hi, it seems that when running flows locally with the Prefect CLI it will abort if passed parameters that don't exist in the flow, but when running registered flows it will pass them through and run with no errors. Is there a way for the CLI to validate parameters and show an error if a param doesn't exist? We'd like to catch this in case someone passes a misspelled param name, for example.
    # Running the flow locally
    aric@aric-dev:~/src/conductor$ prefect run -p conductor/samples/hello_world_flow.py --param foo=bar
    Retrieving local flow... Done
    Configured local flow run
    └── Parameters: {'foo': 'bar'}
    Running flow locally...
    Flow runner encountered an exception!
      Traceback (most recent call last):
        File "/home/aric/.local/lib/python3.8/site-packages/prefect/cli/run.py", line 590, in run
        result_state = flow.run(parameters=params_dict)
        File "/home/aric/.local/lib/python3.8/site-packages/prefect/core/flow.py", line 1250, in run
        raise ValueError(
      ValueError: Flow.run received the following unexpected parameters: foo
    
    Flow run failed!
    
    # Running the same flow registered on Prefect Cloud
    aric@aric-dev:~/src/conductor$ prefect run -n "Hello World" --param foo=bar
    Looking up flow metadata... Done
    Creating run for flow 'Hello World'... Done
    └── Name: monumental-bat
    └── UUID: 39c7896b-0bee-4015-be99-648d90da201b
    └── Labels: []
    └── Parameters: {'foo': 'bar'}
    └── Context: {}
    └── URL: <https://cloud.prefect.io/><URL>
    Aric Huang
    Michael Adkins
    4 replies
    Copy to Clipboard
  • Sepehr Sadighpour

    Sepehr Sadighpour

    1 year ago
    Hi all, has anyone toyed with deployment of flows from S3? I’d like to speed up our CI/CD process, and wondering if it’d make sense to have a “meta” Flow fetch a python package of other Flows from S3, download and register them, etc. Airflow has had this on their to-do list for years (https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher), and I’d love for Prefect to beat them to the punch. Reducing deployment from 20-30 minutes to seconds would be a game-changing differentiator
    Sepehr Sadighpour
    Kevin Kho
    3 replies
    Copy to Clipboard
  • Aiden Price

    Aiden Price

    1 year ago
    What is the best approach to calling an async function within a Prefect task? I have to call the Azure Eventhub client which is async all the way down, from my straight synchronous Prefect task. I've tried calling the async functions like so;
    asyncio.run(client.send_batch(event_data_batch))
    but I get errors like
    RuntimeError: Event loop is closed
    Any advice?
    Aiden Price
    Noah Holm
    +1
    5 replies
    Copy to Clipboard
  • m

    Maria

    1 year ago
    Hi all, I'm triggering my flow with AWS lambda and this great example answered most of my questions: https://github.com/cicdw/prefect-cloud-lambda . But I'm just wondering, is there an easy way to get latest flow_id for my flow_group, rather than hardcoding flow_id in there (and changing every time I deploy a new version)? Is there another env var I can specify maybe?
    m
    Bouke Krom
    +1
    3 replies
    Copy to Clipboard
  • Pedro Machado

    Pedro Machado

    1 year ago
    Hi everyone. I have a flow that triggers another flow using
    create_flow_run
    and
    get_task_run_result
    . I am using these two tasks inside of another task so this can be retried as a unit (see code in thread). Last night, the child flow was triggered three times (triggered by the retries in the parent flow) and all three times the flow succeeded so it looks like there was an issue where the parent flow misread the child flow's status. I am attaching the logs in the thread. Am I misusing these tasks? Any ideas as to why this could be happening? This has been running every night for a couple of weeks and it's the first time this problem happens. Thanks!
    Pedro Machado
    s
    +2
    15 replies
    Copy to Clipboard
  • b

    Bastian Röhrig

    1 year ago
    Hey everyone, do you know any resources for best practises / guides regarding production deployment of agents? I'm looking at https://docs.prefect.io/orchestration/agents/overview.html#agent-types and am wondering whether a single vm using a docker agent would be fine. I am looking for something with very little overhead as our team is very small (3 people, so I feel kubernetes agent is not really an option). Thanks!
    b
    Kyle McChesney
    +1
    6 replies
    Copy to Clipboard
  • Dmitry Kuleshov

    Dmitry Kuleshov

    1 year ago
    Hi all! Can someone rub me into the documentation where I can find information on how prefect works with long-run jobs. We are interested in what will happen if there will be temporary network issue. What will happen with the job and do we have any choices to configure behaviour for that.
    Dmitry Kuleshov
    Kevin Kho
    3 replies
    Copy to Clipboard
  • David McGuire

    David McGuire

    1 year ago
    Has anyone had trouble with a localized
    CronClock
    schedule skipping
    Flow
    runs, entirely?
    David McGuire
    Kevin Kho
    23 replies
    Copy to Clipboard
  • Constantino Schillebeeckx

    Constantino Schillebeeckx

    1 year ago
    Does prefect support intersphinx for documentation purposes?
    Constantino Schillebeeckx
    Kevin Kho
    8 replies
    Copy to Clipboard