• Todd de Quincey

    Todd de Quincey

    4 months ago
    Prefect 2.0 - Snowflake PUT command from local file error
    Todd de Quincey
    Anna Geller
    19 replies
    Copy to Clipboard
  • Ed Burroughes

    Ed Burroughes

    4 months ago
    Hey All, Apologies if this is the wrong channel for this. We've come into a bit of an issue with creating a dynamic flow in Prefect 1.0, see below:
    with Flow("tax_assessor_inc") as flow:
        
        raw_s3_keys = Parameter("raw_s3_keys")
        
        flow_tasks = []
        for s3_key in raw_s3_keys:
            _clean = clean([s3_key])
            _load_data = load_data(_clean)
            _test_raw_data = test_raw_data()
            _transform_task = transform_task()
            flow_tasks.append([_clean, _load_data, _test_raw_data, _transform_task])
    
        reduced_flow_tasks = sum(flow_tasks, [])
        for i in range(len(reduced_flow_tasks) - 1):
            reduced_flow_tasks[i].set_downstream(reduced_flow_tasks[i + 1])
    The issue arises when trying to iterate over the
    raw_s3_keys
    parameter as it's not iterable. It works when hardcoding the s3 paths but obviously that's not what we want, as it would great to dynamically define that list. Also annoyingly each task has to occur synchronously otherwise we'd use something like prefect map. Firstly is it possible to iterate the output of a parameter object. If not is there a better way of solving this problem. Sorry for the long winded message, Thank you 🙂
    Ed Burroughes
    Anna Geller
    9 replies
    Copy to Clipboard
  • Frank Embleton

    Frank Embleton

    4 months ago
    What are my options for alerting with Prefect 2.0? I guess I can create alert tasks and then call them as part of a flow on failure. Is there any plan or ability to handle that on a higher level? Such as on Cloud?
    Frank Embleton
    Anna Geller
    2 replies
    Copy to Clipboard
  • Rainer Schülke

    Rainer Schülke

    4 months ago
    Good afternoon 🙂 I am on Prefect 1.0 and was asking myself - mainly out of curiosity - if it is possible to exclude a specific element while calling .map(), store that element and process it at a later point within the flow? (Not talking about unmapped)
    Rainer Schülke
    Anna Geller
    +1
    7 replies
    Copy to Clipboard
  • Vamsi Reddy

    Vamsi Reddy

    4 months ago
    Hi there, we are using an ECS agent to run our flows. is it possible to set the cpu and memory per flow. I tried setting the cpu and memory for the flow in ECSRun config but the task that gets spun up still uses the default that was specified in the yaml file i.e cpu 1024 and memory 2048. i want to change the cpu and memory per flow. my current ecs run config is as follows:
    RUN_CONFIG = ECSRun(
        labels=["dev"],
        task_role_arn="arn:aws:iam::xxxxx:role/prefectTaskRole",
        execution_role_arn="arn:aws:iam::xxxxx:role/prefectECSAgentTaskExecutionRole",
        image="<http://xxxxxx.dkr.ecr.us-east-1.amazonaws.com/prefect-orchestration:latest|xxxxxx.dkr.ecr.us-east-1.amazonaws.com/prefect-orchestration:latest>",      # this is our custom image that also has our flows
        run_task_kwargs=dict(cluster="prefectEcsClusterDev", launchType="FARGATE", overrides=dict(
            containerOverrides=[
                dict(
                    name="flow",
                    cpu=4096,
                    memory=8192,
                )])),
        cpu=4096,                    #tried specifying here but still no luck
        memory=8192
    )
    Vamsi Reddy
    Kevin Kho
    20 replies
    Copy to Clipboard
  • Todd de Quincey

    Todd de Quincey

    4 months ago
    Prefect 2.0 Add users & edit permissions
    Todd de Quincey
    Kevin Kho
    +1
    9 replies
    Copy to Clipboard
  • JK

    JK

    4 months ago
    Hello, I’m not seeing logs on our flow runs, have a feeling that it may be something on our account — is there a way to double check?
    JK
    1 replies
    Copy to Clipboard
  • Madison Schott

    Madison Schott

    4 months ago
    Hi 🙂 wondering if anyone has resources for creating a (near) real-time data pipeline using Fivetran and Prefect?
    Madison Schott
    Kevin Kho
    4 replies
    Copy to Clipboard
  • Constantino Schillebeeckx

    Constantino Schillebeeckx

    4 months ago
    Question about environment variables, if I set something like
    "FOO": "BAR"
    in the run config, and then do something like
    os.environ["FOO"]: "BAZ"
    in the flow.py - what will
    FOO
    bet set to when the flow runs?
    Constantino Schillebeeckx
    2 replies
    Copy to Clipboard
  • Patrick Tan

    Patrick Tan

    4 months ago
    Hi, today I setup a personal account for development using personal email address. Projects and flows are registered under my account. For production deployment, we want multiple people to be able to monitor flow run and access Prefect Cloud UI. What is best practice on setting this up? Should we setup a service account (share email address) and everyone login to same account?
    Patrick Tan
    Kevin Kho
    3 replies
    Copy to Clipboard