• k

    Kurt Rhee

    1 year ago
    Realistically speaking, how many people does it take to run and maintain prefect server? Is it like 0.5 people, 2 people?
    k
    Kevin Kho
    2 replies
    Copy to Clipboard
  • Lawrence Finn

    Lawrence Finn

    1 year ago
    How does the scheduling component of Prefect actually work? I get that the agent asks for work, but how does the API know which flows should be ran? I am guessing it has the next N scheduled times computed, but is that pre computed on some schedule? Or calculated on API request?
    Lawrence Finn
    Kevin Kho
    7 replies
    Copy to Clipboard
  • Eric Mauser

    Eric Mauser

    1 year ago
    Hey everyone, I'm using the ECS agent and want to pass a secret from AWS Secrets Manager into each task. I saw the documentation about providing a task template for each task to build upon. do I have to create a fully defined template for each task? or can I just create the secrets{} part and let prefect auto generate the rest of it with defaults?
    Eric Mauser
    1 replies
    Copy to Clipboard
  • TOMAS IGNACIO ACUÑA RUZ

    TOMAS IGNACIO ACUÑA RUZ

    1 year ago
    Hello there, I would like to ask how to shut down all process of my 'prefect server start'. I have found1. https://github.com/PrefectHQ/prefect/issues/2348 2. https://github.com/PrefectHQ/prefect/issues/3660 But I can't find any documentation associated with prefect server stop Could you show me how to perform a docker-compose down similar behavior. Best reagards
    TOMAS IGNACIO ACUÑA RUZ
    Kevin Kho
    +1
    8 replies
    Copy to Clipboard
  • s

    Sam Cook

    1 year ago
    Is there a utility, graphql mutation, or canned SQL query available for cleaning up the postgres database? I have a kubernetes deployment and I want to regularly clean up some of the larger tables (logs, task runs) while leaving recent data from the past 30 days intact so users don't suddenly lose all of their recent logs, tasks, etc.
    s
    Kevin Kho
    3 replies
    Copy to Clipboard
  • Lawrence Finn

    Lawrence Finn

    1 year ago
    Is there a way to set the api_token in code rather than relying on env or config file?
    Lawrence Finn
    Kevin Kho
    11 replies
    Copy to Clipboard
  • m

    Matt Klein

    1 year ago
    We have some Prefect flows that sometimes need to run for many hours. For particularly long-running flows, once the flow has been running for more than 12 hours, we’re often seeing that flow fail before it completes -- the Prefect UI shows its last state message as “Unexpected error: CancelledError()“. This doesn’t happen as a result of the code we’ve written to launch or monitor flows. It appears to be a result of an action that Prefect (or Dask?) is taking to automatically cancel long-running flows. However, I don’t see anything in the Prefect or Dask docs indicating that this is expected behavior, or how it could be controlled (e.g., disabled, or increased the allowable duration, etc.). Can anybody provide any guidance on how to deal with flows failing with this
    CancelledError
    ? Any clues on how we can configure Prefect or Dask to allow flows to run past the 12 hour mark?
    m
    Kevin Kho
    4 replies
    Copy to Clipboard
  • Lawrence Finn

    Lawrence Finn

    1 year ago
    If I have a flow run in scheduled state with one parameter, then i try to run that flow again with a different param, why does it cancel the first flow run?
    Lawrence Finn
    Kevin Kho
    14 replies
    Copy to Clipboard
  • Alfie

    Alfie

    1 year ago
    Hi Team, about the the storage options, can I use any DB for it, such as Postgres? If it’s not supported so far, is there any guide I can reference to and implement it by myself? Thanks
    Alfie
    Kevin Kho
    6 replies
    Copy to Clipboard
  • Ryan Sattler

    Ryan Sattler

    1 year ago
    Hi - I’ve been going through the Prefect tutorials using a free account. I’ve gotten the basic Hello World flow to run using the local agent, but now I’m trying to get it to run via the kubernetes agent (using MacOS desktop docker). I’ve run the command-line k8s agent as follows:
    prefect agent kubernetes start
    . When I kick off the hello world flow, this indeed starts a job container on my desktop k8s which appears to start up fine and have the values that I configured (eg correct image). However it stays in a “running” state for several minutes until it errors out as follows:
    urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5455f07710>: Failed to establish a new connection: [Errno -3] Tempor
    Here’s the code for my flow:
    import prefect
    from prefect import task, Flow
    from prefect.run_configs import KubernetesRun
    
    @task
    def hello_task():
        logger = prefect.context.get("logger")
        <http://logger.info|logger.info>("Hello world!")
    
    flow = Flow("hello-flow2", tasks=[hello_task])
    
    # flow.run() We could run our flow locally using the flow's run method but we'll be running this from Cloud!
    
    flow.run_config = KubernetesRun(
        env={"PREFECT__LOGGING__LEVEL": "DEBUG"},
        image="prefecthq/prefect:latest",
        labels=[]
    )
    flow.register(project_name="tester")
    Ryan Sattler
    Kevin Kho
    9 replies
    Copy to Clipboard