• Hemabh Kamboj

    Hemabh Kamboj

    3 months ago
    Hi folks, Is there a way to get meta of flow run ( like, flow run name, started time etc.) inside the flow run. For example, few of my task inside the flow run inserts data in a database, so for monitoring and logging purpose I want to map flow run to those inserts, so think of like adding few columns to the table in db, named
    prefect_flow_run_id
    and
    prefect_flow_version
    and inserting those params in the db
    Hemabh Kamboj
    Tyler Rouze
    2 replies
    Copy to Clipboard
  • m

    Matt Alhonte

    3 months ago
    Is there an easy way to set/change the Name/Slug of a Task based on the arguments? (ie, a task for running Jupyter notebooks that names the task after the filename of the notebook)
    m
    Kevin Kho
    +1
    9 replies
    Copy to Clipboard
  • William Jamir

    William Jamir

    3 months ago
    Hi, I’m receiving this error when trying to use
    upstream_tasks
    Does someone know what I’m doing wrong? My intention is to execute
    task_3
    only after
    task_1
    and task_2is finished (since it depends on the output of them) and start
    task_4
    only after the
    task_3
    is finished How can I accomplish that? (Code and output error in thread)
    William Jamir
    Anna Geller
    +1
    5 replies
    Copy to Clipboard
  • d

    Daniel

    3 months ago
    Dear Prefect Community, I have written some custom python to extract an load data into a snowflake database. It works well from my local machine but attempts to orchestrate these pipelines with Prefect cloud fail. I have followed @Anna Geller 's brilliant article on using prefect and AWS ACS Fargate as a serverless pipeline solution using the
    prefecthq/prefect:latest-python3.10
    docker image with some additional packages including the snowflake python connector and it's dependencies. Flows without snowflake interactions work perfectly when run from Prefect cloud however my EL flows which insert data into snowflake tables fail to load and execute returning >ModuleNotFoundError("No module named 'snowflake'"). Haven't been able to fine any similar reports among the community so wondering if someone could suggest what I may have done wrong. With thanks, Daniel.
    d
    Kevin Kho
    +1
    5 replies
    Copy to Clipboard
  • x

    Xavier Witdouck

    3 months ago
    Hi all, is it possible to override a default parameter value when register a flow via the CLI? For example, our flows have an environment parameter (eg dev, qa, prod) and we don’t want to hard code that in the flow code. Ideally when we register the flow, we want to set the default value for that parameter. Thanks in advance.
    x
    Anna Geller
    +1
    7 replies
    Copy to Clipboard
  • marque

    marque

    3 months ago
    Hi, I am getting a
    prefect.exceptions.ClientError: [{'path': ['create_project'], 'message': 'Uniqueness violation.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
    when I ran
    prefect delete project
    -->
    prefect create project
    (using Prefect v1.0, with Prefect Cloud as backend). Can't seem to recreate a project with the same name
    marque
    Kevin Kho
    +2
    15 replies
    Copy to Clipboard
  • Bharadwaj Yadati

    Bharadwaj Yadati

    3 months ago
    Hi , I’m planning to use prefect as a replacement of our internal framework, its worked like charm while using local agent but when i’m trying ecs agent , im facing issues , i want to run the same etl job on ec2 instances . The guide provided on prefect website (https://docs.prefect.io/orchestration/agents/ecs.html#running-ecs-agent-in-production) is not working with issue ( i even provided aws keys as env variables for task ), if i dont provide args it throws any error asking for it. I’m kind of stuck on this , kindly help me out. Thanks in advance
    Bharadwaj Yadati
    Anna Geller
    +1
    23 replies
    Copy to Clipboard
  • Tarek

    Tarek

    3 months ago
    hi, i am getting sometimes "error occured while creating new work queue" when creating new work-queue from the ui [prefect 2.0]
    Tarek
    Anna Geller
    +3
    18 replies
    Copy to Clipboard
  • Michal Zawadzki

    Michal Zawadzki

    3 months ago
    Prefect 2.0: Hi, I can't seem to be able to run a deployed flow with tags -- it's not being picked up by the work queue/agent that's supposed to be handling that tag. 1. I created a work queue named
    dev_queue
    with the label
    dev
    and supporting all flow runners. It's the only work queue I have set up. 2. I ran an agent with
    prefect agent start dev_queue
    3. I created a deployment with the tag `dev`:
    name: test_platform_flow_first_deployment
    flow_name: Data Platform Demo
    flow_location: ./test_platform_flow.py
    parameters:
      to_print: "Hello from first deployment!"
    tags:
      - dev
    I verified in the UI that it has the right tag and flow runner. However, when I run the deployment, the flow run is never picked up. One suspicious thing I noticed is that the flow run doesn't inherit the
    dev
    tag from the deployment (although I don't know if it's a bug or a feature). When I remove all labels from the work queue, the flow runs are picked up correctly.
    Michal Zawadzki
    Anna Geller
    23 replies
    Copy to Clipboard
  • Michal Zawadzki

    Michal Zawadzki

    3 months ago
    Is there a way to specify
    flow_runner
    in the deployment? I get ValueError: Unregistered flow runner 'DockerFlowRunner'when running
    prefect deployment create my_deployment.yaml
    . My deployment looks like this:
    name: test_platform_flow_first_deployment
    flow_name: Data Platform Demo
    flow_location: ./test_platform_flow.py
    parameters:
      to_print: "Hello from first deployment!"
    tags:
      - dev
    flow_runner: 
      type: DockerFlowRunner
      config:
        image: viadot:orion
    Unfortunately the
    flow_runner
    config is not documented anywhere so it's hard for me to say if I'm specifying it incorrectly or it's not supported at all.
    Michal Zawadzki
    Anna Geller
    +1
    12 replies
    Copy to Clipboard