• Sandip Viradiya

    Sandip Viradiya

    1 year ago
  • Sandip Viradiya

    Sandip Viradiya

    1 year ago
    Thanks in advance everyone
  • m

    Milly gupta

    1 year ago
    Hi All, what is the best way to send email notification when flow is completed. Is it state handler?
    m
    Abhishek
    +1
    19 replies
    Copy to Clipboard
  • John Lee

    John Lee

    1 year ago
    Hi all, I wonder can someone help me make sense of some errors I am encountering with gcp auth. I am trying to store gcp credentials on the agent as described here so that my tasks are able to use google storage/big query. I am setting PREFECT__CONTEXT__SECRETS__GCP_CREDENTIALS on the agent via a helm chart to a string containing the json credentials. This seems to propagate GOOGLE_APPLICATION_CREDENTIALS to each prefect job, and the creds are different to what I set on the agent, but this var is set to the json contents rather than a path in the container containing the credentials. This causes errors for the prefect google utilities and the google api in python. I can hack a fix for this by running something like the following but I am wondering if this is expected behaviour or I am setting up the agent incorrectly?
    from pathlib import Path
    import tempfile
    import os
    import google.auth
    
    creds = Path(tempfile.NamedTemporaryFile().name)
    creds.write_text(os.environ["GOOGLE_APPLICATION_CREDENTIALS"])
    os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = str(creds)
    google.auth.default()
    John Lee
    Kevin Kho
    +1
    36 replies
    Copy to Clipboard
  • Ruslan Aliev

    Ruslan Aliev

    1 year ago
    Hello Everyone! I use
    local agent
    and try to bind local folder with folder inside docker container using
    volumes
    , but looks like it doesn’t work or I do it wrong 👀:
    container_name = StringFormatter(name="Container name", template="scenario_{scenario_id}_task_{task_id}")
    container_id = create_container(image_name=image_name,
                                    container_name=container_name(scenario_id=scenario_id, task_id=task_id),
                                    # command=command,
                                    command=['ls'],
                                    volumes=['data:/home/data',
                                             'temp:/home/temp'],
                                    ).set_upstream(input_file_path)
    started = start_container(container_id=container_id)
    status_code = wait_on_container(container_id=container_id, upstream_tasks=[started])
    logs = get_container_logs(container_id=container_id, upstream_tasks=[status_code])
    log([logs])
    The
    log([logs])
    shows, that the
    data
    and
    temp
    folders are missed. What am I missing here?
    Ruslan Aliev
    Kevin Kho
    9 replies
    Copy to Clipboard
  • Barbara Abi Khoriati

    Barbara Abi Khoriati

    1 year ago
    Hey everyone, I'm new to Prefect (and AWS) and have a quick question. When deploying your ECS Agent inside a EC2 in same AWS environment, do I still need to set authentication for the container deployments, and the S3 bucket with my scripts? Thanks!
    Barbara Abi Khoriati
    Kevin Kho
    3 replies
    Copy to Clipboard
  • Issam Assafi

    Issam Assafi

    1 year ago
    Hey, how to do nested mapping?
    Issam Assafi
    Ruslan Aliev
    +3
    15 replies
    Copy to Clipboard
  • Steve s

    Steve s

    1 year ago
    hey everyone, I have a question about passing data between flows. My scenario is that I need to pass a pandas DataFrame from one flow to another. Right now I'm converting the DataFrame to JSON and passing it in as a Parameter. It works well, but I've only tried for relatively small datasets. I'm wondering: what are the limits of scale for this approach, and should I be handling this differently?
    Steve s
    emre
    +1
    4 replies
    Copy to Clipboard
  • Dmitry Kuleshov

    Dmitry Kuleshov

    1 year ago
    Hi all! Didn't find information if I can limit number of parallel jobs created by prefect agent (kubernetes agent). Can someone either throw me a link or answer. Thanks
    Dmitry Kuleshov
    Kevin Kho
    2 replies
    Copy to Clipboard
  • Anze Kravanja

    Anze Kravanja

    1 year ago
    Hello! I’m curious if in Prefect Server UI you can unarchive a flow? This would be the use case, I have multiple versions of the flow, the latest one is running, but I find there is a bug and I want to roll-back to the previous version. I know I can delete the latest version, but that doesn’t make the now latest version unarchived. Thank you!
    Anze Kravanja
    Kevin Kho
    2 replies
    Copy to Clipboard