• Ame

    Ame

    2 years ago
    💃 Just arrived!
    Ame
    1 replies
    Copy to Clipboard
  • j

    John Ramirez

    2 years ago
    Are you guys doing a cloud release cause I am unable to view my flow in Cloud
    j
    Chris White
    3 replies
    Copy to Clipboard
  • Ame

    Ame

    2 years ago
    Hello everyone, I got some questions regarding the Prefect Agent. Firstly, how should I choose the number of agents, is there a way to calculate or common way to decide this number? Secondly, is a single agent enough to handle (maximum) 1000 querys per hour, the server is running with a local environment and using a DaskExecutor(a local Dask cluster)
    Ame
    Chris White
    +1
    6 replies
    Copy to Clipboard
  • Alfie

    Alfie

    2 years ago
    Hello guys, I’d like to log some info in state handlers, such as flow version, flow run id, is that possible? How can I get them? thanks
    Alfie
    Chris White
    2 replies
    Copy to Clipboard
  • l

    Lewis Bails

    2 years ago
    Hi all, Is it possible to control how many mapped tasks are running at any one time? I have a memory issue, as my mapped task requires loading in some data, and no matter how many workers or threads I specify in the DaskExecutor, the tasks just keep spawning.
    l
    Jeremiah
    +1
    7 replies
    Copy to Clipboard
  • Simone Cittadini

    Simone Cittadini

    2 years ago
    Hi all, question about latest server/ui ; I have prefect installed as a docker swarm, if pull the latest images ( tried "alpha" and "latest") the UI asks to create a tenant with command
    prefect server create-default-tenant --name default
    but from the shell ( inside the graphql container ) I got
    Error: No such command 'create-default-tenant'.
    prefect version
    0.13.1+14.g303356902
    EDIT: worked with master ( prefect v 0.13.2 ) great !
  • Adam

    Adam

    2 years ago
    Happy Monday everyone! Are there any best practices for sharing “clients” between tasks? By clients I mean database clients/connections, API clients etc. I would assume there’s some care to be taken here due to the distributed nature of the tasks.
    Adam
    Jeremiah
    2 replies
    Copy to Clipboard
  • a

    Alex Papanicolaou

    2 years ago
    Hi, we made a small adaptation to the task runner to make it run a little faster locally so we could use prefect as a slick local DAG executor. When running one of these flows within a regular flow, we get all the logs for both flows when we really don’t want the inner flow logs. How would we shut off logs for the inner flow? Was thinking something like wrapping the inner flow like this:
    with prefect.context(loglevel="ERROR"):
        inner_flow.run()
    a
    Kyle Moon-Wright
    8 replies
    Copy to Clipboard
  • m

    Mitchell Bregman

    2 years ago
    Hi all, I am looking to create a dynamic task DAG, which essentially calls a database for all task definitions and dynamically generates tasks. Here is a code snippet:
    class DS242(ParkMobileFlow):
    
        @property
        def tasks(self):
            """Loads all tasks for data quality tests."""
    
            # Connect to ParkMobile client
            pm = ParkMobile()
    
            # Get Snowflake connection
            sf = pm.get("snowflake")
            sf.connect("sqlalchemy")
    
            # Call all tasks and task definitions
            records = sf.execute("""
                SELECT *
                FROM DEV_DB.DATA_QUALITY_TESTS.QUALITY_TASKS
                WHERE is_active = true;
            """)
    
            return records
    
        @property
        def flow(self):
            """Returns the Prefect Flow object."""
            return self.build()
    
        def build(self):
            """Builds the workflow."""
            tasks = []
            for task in self.tasks:
                tasks.append(
                    DataQualityTask(
                        uuid=task.task_uuid,
                        name=task.task_name,
                        description=task.task_description,
                        source_query=task.source_query,
                        target_query=task.target_query,
                    )
                )
    
            with Flow("DS242__data-quality-tests") as flow:
                for task in tasks:
                    task()
    
            return flow
    Now, I am trying to
    register
    this flow to the backend server, and am getting a serialization error, which is understandable. My question is, can I create a first
    Task
    , that queries the DB for all task definitions, and then dynamically creates new tasks to the
    Flow
    which performs all of the tasks? I can certainly build a custom Dockerfile that handles deployment of this, but would be awesome to connect to
    prefect server
    m
    Kyle Moon-Wright
    3 replies
    Copy to Clipboard
  • j

    Julien Allard

    2 years ago
    Hello all! I have a question about scaling to Kubernetes+Dask in GCP. I am trying to run a flow that uses pandas for data transformation that requires a large amount of memory, more than what can be allocated for a single node of our current cluster configuration. Here are some of the ideas, I had to fix the problem1. I could parallelize my flow using Dask Dataframe but that seems to be a lot of effort. 2. We could vertically scale the nodes, but then we would pay for more compute that we need most of the time 3. Create a preemtible node pool where the flow can then be executed I feel like the 3rd point could be a good solution, but I am not sure how it could work to allocate a preemtible node pool at the start of a flow and then execute the flow on the new node pool since we have already deployed the job or dask cluster on the other nodes. So I guess my question is what is the best way to achieve what i'm trying to do without having to vertically scale the cluster? Thank you!
    j
    Kyle Moon-Wright
    +1
    6 replies
    Copy to Clipboard