• Dor Menachem

    Dor Menachem

    1 year ago
    is it possible to run an agent in the cloud and not in my local machine?
    Dor Menachem
    Raphaël Riel
    +1
    5 replies
    Copy to Clipboard
  • Felipe Netto

    Felipe Netto

    1 year ago
    Hi guys! I'm new to Prefect and I'm wondering if it has some built-in task to interact with ASW EMR...
    Felipe Netto
    Dylan
    2 replies
    Copy to Clipboard
  • Walt Wells

    Walt Wells

    1 year ago
    Long time listener, first time caller. Wondering if there are any best practices for teams using Prefect Cloud to ensure that their execution environment is up to date? Are there any known issues for when Prefect Cloud moves to a new release and the execution environment a flow is running in doesn't move along side? What is Prefect's philosophy on versioning/breaking changes?
    Walt Wells
    Dylan
    5 replies
    Copy to Clipboard
  • Joël Luijmes

    Joël Luijmes

    1 year ago
    With the schedule concept, can I also set a maximum deadline for which a missed flow will ran? I just tested something while my agent wasn’t running, and now I got 40ish jobs it still will run.
    Joël Luijmes
    Dylan
    2 replies
    Copy to Clipboard
  • r

    Rodrigo Neves

    1 year ago
    There are any resources/examples about a ETL pipeline for live data ingestion reading from a Queue and for each new event it creates a new flow that is executed by a central dask cluster with auto scaling? (actually this is a good use case for prefect? or should we stick with batch jobs?)
    r
    Dylan
    3 replies
    Copy to Clipboard
  • m

    Mac Gréco Péralte Chéry

    1 year ago
    Hi Everyone, I have a flow that have mapped tasks that use DaskCloudProvider Environnement and i have set the number of workers to 4. For some reasons only one worker is created. Here is the code:
    flow.environment = DaskCloudProviderEnvironment(
        provider_class=FargateCluster,
        # task_role_arn="arn:aws:iam::497427061914:role/ecsTaskRole",
        execution_role_arn="arn:aws:iam::497427061914:role/ecsTaskRole",
        n_workers=4,
        scheduler_cpu=256,
        scheduler_mem=512,
        worker_cpu=512,
        worker_mem=1024,
        labels=["fargate"]
    )
    m
    Dylan
    3 replies
    Copy to Clipboard
  • Joseph Haaga

    Joseph Haaga

    1 year ago
    I’m working on a Python utility to automate the registration/deployment of Flows for our Prefect users; I noticed that the Dockerfile generated by
    DockerStorage.create_dockerfile_object(directory)
    contains a COPY statement to add
    healthcheck.py
    to the image, but uses an absolute path (which
    docker build
    treats as a relative path from the build context)
    COPY /absolute/path/on/my/machine/healthcheck.py /opt/prefect/healthcheck.py
    This causes the following error:
    Step 6/16 : COPY /absolute/path/on/my/machine/healthcheck.py /opt/prefect/healthcheck.py
    COPY failed: stat /var/lib/docker/tmp/docker-builder852791584/absolute/path/on/my/machine/healthcheck.py: no such file or directory
    However, this works fine when I manually update the Dockerfile to use a relative path to
    healthcheck.py
    Would this constitute an issue? Or am I circumventing an intentional design decision?
    Joseph Haaga
    1 replies
    Copy to Clipboard
  • bral

    bral

    1 year ago
    Hi all! Is there possibility looping of flow ? We have long running task
    bral
    z
    +1
    9 replies
    Copy to Clipboard
  • z

    Zach

    1 year ago
    Are there restrictions on what I can store in an environment variable? I am having a flow registration fail because of an environment variable. The environment variable looks like this:
    "RANDOM_TOOL_VERSION": ": Guppy Basecalling Software, (C) Oxford Nanopore Technologies, Limited. Version 4.2.2+effbaf84"
    Here is a sample flow registration that is similar to what I am doing: (The error I was getting is in a comment at the bottom of the file)
    z
    j
    +1
    4 replies
    Copy to Clipboard
  • ale

    ale

    1 year ago
    Hey folks, we’re interested in using the brand new ECS Agent. However, it is not completely clear how we should setup the RunConfig. Is this something we have to setup at flow or flow environment level?
    ale
    j
    7 replies
    Copy to Clipboard