Is there a way to use

nvidia-docker

with the prefect docker agent?

Is there a more efficient way to define Paremeters/Values than a single assignment per param? They each turn into a Task and I am looking at lowering my Task count and this seems like a prime candidate.

Hello - When using ECS/Fargate is there a way to set the dnsServers on the task definition without providing a custom task definition?

Hey folks 😒imple_smile: We want to trigger a Prefect flow from another Python app….which is written in Python 2 😅 , hence we cannot use Prefect Client. What approach would you suggest?

Hey guys, I am new to prefect. It is nice to join this community and work on such software. I am actually curious is the

Prefect Team

option disabled from Prefect Core's server ?

Hi All, I only have one flow enabled at a single time using a local agent.

[2021-05-12 12:26:05-0700] INFO - prefect.task_name | Waiting for next scheduled run at 2021-05-13T10:00:00+00:00
2021-05-12 12:26:05,221 - prefect.task_name - INFO - Waiting for next scheduled run at 2021-05-13T10:00:00+00:00

But the local agent somehow runs two of the same flow twice. How should I debug this issue?

Hey y'all - really struggling on how to create a parametrized dependent flow. https://docs.prefect.io/core/idioms/flow-to-flow.html#running-a-parametrized-flow the description given is exactly what i want to do:

Let's say that you want to always run a flow with parameters that are generated by another flow. Naively, you might manually wait for one flow to finish and then manually trigger the next flow to run with the appropriate parameter values. Prefect makes this pattern easy to automate via the StartFlowRun:

but the example code given represents a flow which is dependent upon a task that returns parameters, not a flow that is dependent upon another flow. i think the main piece im struggling with is how to return/access some values of the preceding flow from the succeeding flow. (basically getting the input to the

parameters

arg of the succeeding flow)

What if I don't want to store a result? Is it possible to store in memory just in that one run? My concern is that running this data loader every day, I was seeing 1GB result files each day

Is there a way to get help on how to run an Agent on AWS EC2 ? I think I've done everything properly but it still doesn't work like I expect/want. Am I expecting the right thing ? I have an existing EC2 (t2.micro) that I want to run an agent on and the python scripts are stored in a folder on the EC2. Is there a more recommended/cheaper architecture than this ?

Hi guys, I was working on a Dask deploy on EKS when the

DaskKubernetesEnvironment

was supported and that worked great, now Im using the

run_configs

and

executors

and Im having the botocore region issue displayed, not sure Im missing any argument for the

DaskExecutor

please check the code bellow:

flow.storage = Docker(registry_url=ecr,
                      python_dependencies=python_dependencies,
                      files={f'{getcwd()}/src/dask_flow': '/modules/dask_flow'},
                              env_vars={"PYTHONPATH": "$PYTHONPATH:modules/"},
                              image_tag='latest')

flow.run_config=KubernetesRun()
flow.executor=DaskExecutor(
   cluster_class="dask_cloudprovider.aws.FargateCluster",
   cluster_kwargs={'n_workers': 5, 'region_name':aws_region})

Hi, is there a way to access the dask's

client

object from Prefect's DaskExecutor? I need to be able to upload a python egg to the Dask cluster before starting executing Tasks (which I would achieve usually with

client.upload_file)

Thank you in advance!

Hi, how do I make prefect to send all the required side libraries to the underlying Dask scheduler? I.e. running the 02_etl_flow.py example from the tutorial when I send the flow to a remote dask scheduler I get the following error:

ModuleNotFoundError: No module named 'aircraftlib'
[2021-05-13 09:27:29+0200] ERROR - prefect.etl | Unexpected error occured in FlowRunner: ModuleNotFoundError("No module named 'aircraftlib'")

aircraftlib is a module from the tutorial imported by the flow. it works when I run it all locally, it does not work if i send it to a remote dask scheduler. I could place it all in a single file "flow.py" but is there an elegant way to do this? thanks.

Hi all again, I am looking into PrefectHQ/server kubernetes installation through Helm https://github.com/PrefectHQ/server/blob/master/helm/prefect-server when I try to modify the securityContext for e.g. hasura service the helm install fails:

helm install prefect prefecthq/prefect-server -f values.yaml

with the following error:

Error: YAML parse error on prefect-server/templates/hasura/deployment.yaml: error converting YAML to JSON: yaml: line 31: mapping values are not allowed in this context

Attached you can find the values.yaml file that I try to provide for the helm install command. The only modification is the securityContext:

securityContext:
  runAsUser: 1000
  runAsGroup: 1000

instead of

securityContext: {}

for hasura service The errored line 31 of the hasura deployment.yaml looks as follows:

securityContext:
    {{- toYaml . | nindent 8 }}
  {{- end }}

Thanks for help.

👋 I’m here! What’d I miss?

Hello community - I'm getting started with my first flow. I've set up an EC2 instance to house my code and the agent, registered a flow, fired up the agent (which is seen by the cloud server). When I run my flow . . . nothing happens. The agent just says "Waiting for flow runs . . ." What am I missing?

👋 Hey folks. Just bumped from

0.14.17

->

0.14.19

and I'm getting an error when my Dask Scheduler pod starts up:

Traceback (most recent call last):	
File "/usr/local/bin/dask-scheduler", line 5, in <module>	
from distributed.cli.dask_scheduler import go	
File "/usr/local/lib/python3.8/site-packages/distributed/cli/dask_scheduler.py", line 120, in <module>	
def main(	
File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 247, in decorator	
_param_memo(f, OptionClass(param_decls, **option_attrs))	
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 2467, in __init__	
super().__init__(param_decls, type=type, multiple=multiple, **attrs)	
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 2106, in __init__	
raise ValueError(	
ValueError: 'default' must be a list when 'multiple' is true.

I have a question regarding Prefect versioning compatibility. Our flows will always be registered with a specific version of Prefect (let’s say 14.7 for example). The image used by our Dask schedulers and workers will always use the same Prefect version (14.7 in this case). However, our agent will most likely be running an older version of Prefect which is not updated as frequently (14.3 in this case). Is there the potential for Flow deserialization issues or failures due to this mismatch?

Is there a way we can run prefect map function outside the flow?

Hi 👋 I have a very simple ETL where I extract data from a Rest API and save it to a DB. I notice that my RAM keep increasing until the whole Flow has finished. Why is that? This happens despite me using a depth first search in the flow, so I don’t really keep or need the data, but the Flow keeps it in memory it seems. Can anyone please advise?: )

Hey Team, 🙂 We've recently signed on board w/ prefect. After using the new Automations tab for a bit, we have some feedback: 1. UI: After selecting create PagerDuty Action and filling out the config information, the next button is still grayed. I found a work around was to click back to altering the PagerDuty message and click Next/Save. 2. API Key: With regards to using an API Key, it’s not generally required to send alerts to PagerDuty. By creating a general service pagerduty API Key, we don’t have a way to scope it and unfortunately that makes a read/write key that can do anything in pagerduty. 3. Payload: While we’re able to edit the message in the Automations tab, It doesn’t seem there’s a way to edit the Payload. Can this function be available? 😄 appreciate all the efforts. Thanks!

So I have a situation where on my computer I would like to be able to easily switch between my personal "team" and my organizations team so that I can run personal projects on Prefect also. It seems like this is a feature through tenants, but I can't seem to figure out how I would add a second tenant to my account so I can easily switch between the two. I can't find any documentation on it either. Is what I am trying to do possible and if so could someone point me to some docs?

running into this error using a docker agent, the task sometimes runs but other times I get this:

Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/root/'")

Are there any details on what is expected here? I added this to see what would happen and now I cannot delete it in cloud.

Hey all - is there a way to have multiple

IntervalSchedule

similar to having multiple clocks in the docs (https://docs.prefect.io/core/concepts/schedules.html#clocks)? What would be the recommended way to have a single flow that can run on 2 schedules (say every 5 minutes and every 30 minutes)? I guess we could probably achieve this with a 1 minute schedule and check within the flow run if

now().minute == 30

or

now().minute % 5 == 0

or some similar method but just wondering what would "prefecters" do...

Hello everyone, I organize the Orlando Data Science Meetup and I am planning an event around Prefect. I was wondering if there’s anyone here interested speaking at the meetup with me about their Prefect use case? You can also speak alone if you want the full stage. 😆

Hi all. I am trying to setup ECS agent to run flows in Fargate. Assuming that I have an ECS Fargate cluster ready, here is how I start my agent

from prefect.agent.ecs import ECSAgent

agent = ECSAgent(
    name='Test Agent',
    region_name='eu-west-1',
    cluster='my-test-cluster',
    launch_type='FARGATE',
    run_task_kwargs_path='./config.yaml'
)
agent.start()

This then failed with the following message

[2021-05-14 12:46:54,069] ERROR - Test Agent | Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`
Traceback (most recent call last):
  File "/Users/kha/Desktop/foo/agent.py", line 3, in <module>
    agent = ECSAgent(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/zoined/lib/python3.9/site-packages/prefect/agent/ecs/agent.py", line 234, in __init__
    ] = self.infer_network_configuration()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/zoined/lib/python3.9/site-packages/prefect/agent/ecs/agent.py", line 274, in infer_network_configuration
    raise ValueError(msg)
ValueError: Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`

I cannot run a simple flow on ECS. I have setup a simple flow like this

import prefect
from prefect import task, Flow
from prefect.run_configs import ECSRun


@task
def hello_task():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello World!")
    return 0


task_definition = {
    'networkMode': 'awsvpc',
    'cpu': 1024,
    'memory': 2048,
    'containerDefinitions': [
        {'name': 'flow'}
    ]
}


run_config = ECSRun(
    image='prefecthq/prefect',
    task_definition=task_definition
)

flow = Flow("hello-flow",
            tasks=[hello_task],
            run_config=run_config)

flow.register(project_name='Prefect_Tutorials', labels=['aws', 'ecs', 'fargate'])

The ECS agent can successfully trigger an ECS task. However, the flow run failed with error

Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/Users/kha/'")

. Why does it include something specific to my name in the flow?

Hi all, does anyone have any tricks to only respect a schedule based on a project name? Our use case is that we use prefect projects for our environments/branches (e.g. we have a master project, a staging project a branch_1 project etc etc), but we want to turn off all scheduling for everything apart from the master branch. We currently do some monkey patching at register time to set the schedule=None, but we'd rather have something in the prefect CLI. For example

prefect register flow_name --label=staging --no-schedule

We've also messed about with setting a bad label on the schedule itself, which "kind of" works, as in the flow won't run because it can't find an agent, but, the UI still get overwhelmed with flows being scheduled, but not being able to be run.

is there a way to use secrets when registering a flow? E.g, I want to register flows via a CI pipeline, however I keep getting “local secret not found”. I would have thought that logging in w/ the cli and setting my backend to cloud wouldve forced prefect to look at cloud secrets, but maybe I am missing something? Setting environment variables does work, but I’d like to try to avoid specifying secrets via the UI and in circleci if possible.