I have an interesting issue that I am running into. I have a flow that looks roughly like:

from settings import JOB_QUEUE, JOB_DEF

@task
def trigger_batch_job(input):
    ...
    return BatchSubmit(
        job_name=job_name,
        job_definition=JOB_DEF,
        job_queue=JOB_QUEUE,
        batch_kwargs=...,
    ).run()


with Flow(...):
    inputs = some_task()
    wait_res = AWSClientWait(
        client='batch',
        waiter_name='JobComplete',
    )(
        waiter_kwargs={
            'jobs': [
                trigger_batch_job(inputs),
            ],
            'WaiterConfig': {
                'Delay': 10,
                'MaxAttempts': 10000,
            },
        },
    )

And a settings.py file that looks like:

# settings.py
import os

JOB_DEF = os.environ.get(
    'PREFECT_FLOWS_JOB_DEF',
    'job-latest',
)

JOB_QUEUE = os.environ.get(
    'PREFECT_FLOWS_JOB_QUEUE',
    'job-gpu-dev',
)

The idea here is to have dynamic values for these settings based on what environment this is running in (we have dev/test/prod AWS accounts)

I ran a job in our test environment, and there was an error trying to submit the job to the dev queue (i.e. the default value here) I 2x checked and the environment variable is set on the docker image used to run the task

is it possible that this value is being predefined when the flow is built/registered and thus not using the most up to date value (the one at flow execution time)?

Is there a SLURM executor?

I've been trying to figure out how to properly expose the Dask Dashboard so it can be accessed externally outside of a flow's Docker Container on a flow run w/ DockerRun and DaskCluster Executor (ephemeral Local Dask Cluster), and the initial path I went down wasn't successful, so looking for some advice. Right now I have the Flow Runs running with containers on a user-defined Docker bridge network. Currently, can't reach the Dask Dashboard, which I assume is because its port (8787) isn't accessible outside the container. So, I figured the most straightforward way would be to enable port forwarding for the 8787 port (which Dask Dashboard runs on by default) for the containers. I was planning on adding in that extra config for port-forwarding with the

host_config

arg of

DockerRun

, however looking at the Docker Python SDK docs, it appears the ports for forwarding have to additionally be passed into the

create_container()

function (https://docker-py.readthedocs.io/en/stable/api.html#docker.api.container.ContainerApiMixin.create_container), which looking at the prefect source code, is not an additional arg I can pass into the run configs of

DockerRun

to thus be sent into

create_container()

. So, doesn't seem possible. This feels like something pretty common and doable, and I'm guessing I'm just looking at it the wrong direction. Has anyone accomplish this before and have any suggestions? Thanks!

I am trying to understand the correct way to separate my registration process from my flow code. I'm using S3 storage and KubernetesRun, with a custom docker image. Because the flow dependencies are not available in my build environment, I feel like I'm in a catch22 situation... my register script is failing on import errors for modules that are not required during the build/register process at all. Could you please offer some conceptual advice here? Happy to share code although I'm not sure which parts are pertinent.

Hi folks I am having issues using the prefect backend key-value store. (Please see the exception in the thread)

Hi, I'm wondering if it is possible to run the executor on the same EC2 instance that is running the agent?

Hi everyone, Wondered if exponential backoff on retries was some that Prefect might have on a roadmap, or people have interest in? Airflow has this feature (retry_exponential_backoff). My use cases are APIs with rate limits, and backing off automatically when we hit them

Hi folks, Is there a way to ignore the schedule of a flow when running using the CLI

prefect run

for a local flow?

Hi all, has anyone deployed an ECSAgent as a Fargate Service with an application load balancer? Trying to figure out: 1. When registering an

ECSAgent

we can use

agent_address=<address:port>

. Should that be the load balancer's DNS ? 2. What is the health-check command to set in the application balancer?

Hello everyone ! I am a data scientist who is currently testing the Prefect library. I am a little bit stuck on using prefect flows with Docker. Do you have any example on flows configured with DockerRun and ran with a Local DockerAgent ? My flow is registered but is stuck in 'Submitted for execution'. Thanks for your help ! 🙂

Hi all! I have a very specific technical question about prefect related to task configuration.

Hi all, I have a quick question about how registering flows with s3 storage works.

Hi all! I wanted to have a try to Orion, but I encountered strange stuff while installing it. I am running Ubuntu on a GCP VM. I did a virutalenv, activated it, pip installed prefect and run

prefect version

.

virutalenv env
source env/bin/activate
pip install prefect==2.0a2
prefect version

This outputs:

0+unknown

Hi all! Can I ask if there's a way to setup custom log handlers on the prefect logger? I haven't tested it yet, but I'm pretty sure I can attach a handler, or check if a handler is already attached, when I call

prefect.context.get("logger")

, but is there a better place where I can hook up my own log handlers? For example, we are going to write a log handler that can send agent/flow logs to a central location like kafka or s3 or datadog or whatever. We haven't exactly decided the target location yet, but we do know we want a central space, and I'm just trying to find the best place to configure and hook up our custom log handler.

hi, is this a bug? the same flow showing many times:

btw, I’ve seen this a while that I have to restart the flow twice to make it really restarted. The first time click just logged a line but nothing in effect until the second time click. Is this a known issue?

Thank you so much to those who have completed our first ever user survey. We’ll close the survey after Friday, so please help us make Prefect better — and have a chance to get cool swag! https://prefect.qualtrics.com/jfe/form/SV_41MC3mkByz18Ok6

I'm currently working on a flow that triggers several flows via StartFlowRun, and I'm seeing that the RunConfig that I pass is being ignored s.t. the run config for flow on Prefect UI is just an empty dict (IE the default). Is this a known issue? I'm on a slightly older version,

0.14.22

, but a quick dig through github issues didn't reveal any known problem with this (or any changes to that task definition for quite a while), so I may just being doing something wrong.

Is there any way to wait for flows started with

create_flow_run

or do we need to use

StartFlowRun

in that scenario?

this is a simple question, but is there a way on the prefect cloud ui to see the flow run id on the flow run page?

Quick question: Is there way to embed flow readme in flow source code? https://docs.prefect.io/orchestration/ui/flow.html#read-me

Wondering if anyone has tried to set

secret

using GraphQL mutation https://docs.prefect.io/orchestration/concepts/secrets.html#graphql It’s erroring out for me -

"message": "Expected type JSON!, found \"MY SECRET VALUE\"; Expecting value: line 1 column 1 (char 0)",

I’m trying out Orion. When I register a deployment, it doesn’t show up in the UI, however, I can list and run it from the CLI. Am I doing something wrong?

Hi - Also trying out Orion today!

2.0.a3

. Is there a way to import prefect secrets? Previously I used

from prefect.tasks.secrets import PrefectSecret

. Thanks in advance...

Hi I've just recently started receiving

'connection reset by peer'

errors from tasks in various flow. Do you have any pointers to troubleshoot this?

Hi, just wondering here about the possibility of prefect to develop a visual pipeline builder like aws step functions or orchest? Are you planning on this or let the community do it?

I'm having issues getting a flow to run locally using Prefect Server, Docker Agent, DockerRun, and Script Storage (via gitlab) (I think that is an ok combo...). When the flow is triggered, the container is built, but the agent is unable to pull the flow from gitlab

gitlab.exceptions.GitlabGetError: 404: 404 Not Found.

This is a self-hosted gitlab, but a public repo. Does this error suggest authentication issues or am I missing something else? Thanks! agent is launched with:

prefect agent docker start --api <http://localhost:4200> --show-flow-logs --label docker_local

storage configs:

GitLab(host = selfhost_gitlab_url, repo='rgaffney/simple_orchestration', path='HelloWorld_flow.py',ref='prefect_v1')

run_configs:

DockerRun(image="prefecthq/prefect:core-0.15.5", env={"EXTRA_PIP_PACKAGES": 'asyncio aiohttp aiofiles prefect[gitlab]'}, labels=['docker_local'])