https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • v

    Vincent

    05/12/2021, 1:10 PM
    Is there a way to use
    nvidia-docker
    with the prefect docker agent?
    k
    10 replies · 2 participants
  • t

    Tim Enders

    05/12/2021, 1:20 PM
    Is there a more efficient way to define Paremeters/Values than a single assignment per param? They each turn into a Task and I am looking at lowering my Task count and this seems like a prime candidate.
    m
    k
    3 replies · 3 participants
  • r

    Robert Bastian

    05/12/2021, 1:40 PM
    Hello - When using ECS/Fargate is there a way to set the dnsServers on the task definition without providing a custom task definition?
    k
    4 replies · 2 participants
  • a

    ale

    05/12/2021, 3:39 PM
    Hey folks 😒imple_smile: We want to trigger a Prefect flow from another Python app….which is written in Python 2 😅 , hence we cannot use Prefect Client. What approach would you suggest?
    t
    c
    4 replies · 3 participants
  • s

    Sami Serbey

    05/12/2021, 5:55 PM
    Hey guys, I am new to prefect. It is nice to join this community and work on such software. I am actually curious is the
    Prefect Team
    option disabled from Prefect Core's server ?
    k
    2 replies · 2 participants
  • m

    Max Lei

    05/12/2021, 7:46 PM
    Hi All, I only have one flow enabled at a single time using a local agent.
    [2021-05-12 12:26:05-0700] INFO - prefect.task_name | Waiting for next scheduled run at 2021-05-13T10:00:00+00:00
    2021-05-12 12:26:05,221 - prefect.task_name - INFO - Waiting for next scheduled run at 2021-05-13T10:00:00+00:00
    But the local agent somehow runs two of the same flow twice. How should I debug this issue?
    k
    m
    32 replies · 3 participants
  • a

    Andrew Nichol

    05/12/2021, 10:02 PM
    Hey y'all - really struggling on how to create a parametrized dependent flow. https://docs.prefect.io/core/idioms/flow-to-flow.html#running-a-parametrized-flow the description given is exactly what i want to do:
    Let's say that you want to always run a flow with parameters that are generated by another flow. Naively, you might manually wait for one flow to finish and then manually trigger the next flow to run with the appropriate parameter values. Prefect makes this pattern easy to automate via the StartFlowRun:
    but the example code given represents a flow which is dependent upon a task that returns parameters, not a flow that is dependent upon another flow. i think the main piece im struggling with is how to return/access some values of the preceding flow from the succeeding flow. (basically getting the input to the
    parameters
    arg of the succeeding flow)
    k
    8 replies · 2 participants
  • j

    Joseph Loss

    05/12/2021, 10:09 PM
    What if I don't want to store a result? Is it possible to store in memory just in that one run? My concern is that running this data loader every day, I was seeing 1GB result files each day
    a
    2 replies · 2 participants
  • c

    Chris McClellan

    05/12/2021, 11:08 PM
    Is there a way to get help on how to run an Agent on AWS EC2 ? I think I've done everything properly but it still doesn't work like I expect/want. Am I expecting the right thing ? I have an existing EC2 (t2.micro) that I want to run an agent on and the python scripts are stored in a folder on the EC2. Is there a more recommended/cheaper architecture than this ?
    k
    j
    7 replies · 3 participants
  • l

    Leandro Mana

    05/12/2021, 11:33 PM
    Hi guys, I was working on a Dask deploy on EKS when the
    DaskKubernetesEnvironment
    was supported and that worked great, now Im using the
    run_configs
    and
    executors
    and Im having the botocore region issue displayed, not sure Im missing any argument for the
    DaskExecutor
    please check the code bellow:
    flow.storage = Docker(registry_url=ecr,
                          python_dependencies=python_dependencies,
                          files={f'{getcwd()}/src/dask_flow': '/modules/dask_flow'},
                                  env_vars={"PYTHONPATH": "$PYTHONPATH:modules/"},
                                  image_tag='latest')
    
    flow.run_config=KubernetesRun()
    flow.executor=DaskExecutor(
       cluster_class="dask_cloudprovider.aws.FargateCluster",
       cluster_kwargs={'n_workers': 5, 'region_name':aws_region})
    k
    29 replies · 2 participants
  • s

    Sébastien Arnaud

    05/13/2021, 3:40 AM
    Hi, is there a way to access the dask's
    client
    object from Prefect's DaskExecutor? I need to be able to upload a python egg to the Dask cluster before starting executing Tasks (which I would achieve usually with
    client.upload_file)
    Thank you in advance!
    k
    14 replies · 2 participants
  • m

    Matej

    05/13/2021, 7:31 AM
    Hi, how do I make prefect to send all the required side libraries to the underlying Dask scheduler? I.e. running the 02_etl_flow.py example from the tutorial when I send the flow to a remote dask scheduler I get the following error:
    ModuleNotFoundError: No module named 'aircraftlib'
    [2021-05-13 09:27:29+0200] ERROR - prefect.etl | Unexpected error occured in FlowRunner: ModuleNotFoundError("No module named 'aircraftlib'")
    aircraftlib is a module from the tutorial imported by the flow. it works when I run it all locally, it does not work if i send it to a remote dask scheduler. I could place it all in a single file "flow.py" but is there an elegant way to do this? thanks.
    k
    19 replies · 2 participants
  • m

    Matej

    05/13/2021, 9:23 AM
    Hi all again, I am looking into PrefectHQ/server kubernetes installation through Helm https://github.com/PrefectHQ/server/blob/master/helm/prefect-server when I try to modify the securityContext for e.g. hasura service the helm install fails:
    helm install prefect prefecthq/prefect-server -f values.yaml
    with the following error:
    Error: YAML parse error on prefect-server/templates/hasura/deployment.yaml: error converting YAML to JSON: yaml: line 31: mapping values are not allowed in this context
    Attached you can find the values.yaml file that I try to provide for the helm install command. The only modification is the securityContext:
    securityContext:
      runAsUser: 1000
      runAsGroup: 1000
    instead of
    securityContext: {}
    for hasura service The errored line 31 of the hasura deployment.yaml looks as follows:
    securityContext:
        {{- toYaml . | nindent 8 }}
      {{- end }}
    Thanks for help.
    values.yaml
    k
    6 replies · 2 participants
  • a

    Alessio Civitillo

    05/13/2021, 12:22 PM
    👋 I’m here! What’d I miss?
    k
    1 reply · 2 participants
  • l

    Lone Pine Account

    05/13/2021, 1:37 PM
    Hello community - I'm getting started with my first flow. I've set up an EC2 instance to house my code and the agent, registered a flow, fired up the agent (which is seen by the cloud server). When I run my flow . . . nothing happens. The agent just says "Waiting for flow runs . . ." What am I missing?
    k
    42 replies · 2 participants
  • c

    ciaran

    05/13/2021, 1:52 PM
    👋 Hey folks. Just bumped from
    0.14.17
    ->
    0.14.19
    and I'm getting an error when my Dask Scheduler pod starts up:
    Traceback (most recent call last):	
    File "/usr/local/bin/dask-scheduler", line 5, in <module>	
    from distributed.cli.dask_scheduler import go	
    File "/usr/local/lib/python3.8/site-packages/distributed/cli/dask_scheduler.py", line 120, in <module>	
    def main(	
    File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 247, in decorator	
    _param_memo(f, OptionClass(param_decls, **option_attrs))	
    File "/usr/local/lib/python3.8/site-packages/click/core.py", line 2467, in __init__	
    super().__init__(param_decls, type=type, multiple=multiple, **attrs)	
    File "/usr/local/lib/python3.8/site-packages/click/core.py", line 2106, in __init__	
    raise ValueError(	
    ValueError: 'default' must be a list when 'multiple' is true.
    k
    3 replies · 2 participants
  • s

    Sean Harkins

    05/13/2021, 3:56 PM
    I have a question regarding Prefect versioning compatibility. Our flows will always be registered with a specific version of Prefect (let’s say 14.7 for example). The image used by our Dask schedulers and workers will always use the same Prefect version (14.7 in this case). However, our agent will most likely be running an older version of Prefect which is not updated as frequently (14.3 in this case). Is there the potential for Flow deserialization issues or failures due to this mismatch?
    k
    2 replies · 2 participants
  • r

    Ravikiran Sharvirala

    05/13/2021, 4:13 PM
    Is there a way we can run prefect map function outside the flow?
    k
    4 replies · 2 participants
  • n

    Newskooler

    05/13/2021, 4:24 PM
    Hi 👋 I have a very simple ETL where I extract data from a Rest API and save it to a DB. I notice that my RAM keep increasing until the whole Flow has finished. Why is that? This happens despite me using a depth first search in the flow, so I don’t really keep or need the data, but the Flow keeps it in memory it seems. Can anyone please advise?: )
    k
    17 replies · 2 participants
  • c

    Charles Leung

    05/13/2021, 7:24 PM
    Hey Team, 🙂 We've recently signed on board w/ prefect. After using the new Automations tab for a bit, we have some feedback: 1. UI: After selecting create PagerDuty Action and filling out the config information, the next button is still grayed. I found a work around was to click back to altering the PagerDuty message and click Next/Save. 2. API Key: With regards to using an API Key, it’s not generally required to send alerts to PagerDuty. By creating a general service pagerduty API Key, we don’t have a way to scope it and unfortunately that makes a read/write key that can do anything in pagerduty. 3. Payload: While we’re able to edit the message in the Automations tab, It doesn’t seem there’s a way to edit the Payload. Can this function be available? 😄 appreciate all the efforts. Thanks!
    👍 1
    n
    m
    +1
    6 replies · 4 participants
  • n

    Nelson Griffiths

    05/13/2021, 7:56 PM
    So I have a situation where on my computer I would like to be able to easily switch between my personal "team" and my organizations team so that I can run personal projects on Prefect also. It seems like this is a feature through tenants, but I can't seem to figure out how I would add a second tenant to my account so I can easily switch between the two. I can't find any documentation on it either. Is what I am trying to do possible and if so could someone point me to some docs?
    k
    1 reply · 2 participants
  • j

    Joseph Loss

    05/13/2021, 8:11 PM
    running into this error using a docker agent, the task sometimes runs but other times I get this:
    Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/root/'")
    k
    9 replies · 2 participants
  • b

    Braun Reyes

    05/13/2021, 9:40 PM
    Are there any details on what is expected here? I added this to see what would happen and now I cannot delete it in cloud.
    k
    d
    4 replies · 3 participants
  • j

    Jeremy Yeo

    05/14/2021, 12:33 AM
    Hey all - is there a way to have multiple
    IntervalSchedule
    similar to having multiple clocks in the docs (https://docs.prefect.io/core/concepts/schedules.html#clocks)? What would be the recommended way to have a single flow that can run on 2 schedules (say every 5 minutes and every 30 minutes)? I guess we could probably achieve this with a 1 minute schedule and check within the flow run if
    now().minute == 30
    or
    now().minute % 5 == 0
    or some similar method but just wondering what would "prefecters" do...
    ✅ 1
    n
    4 replies · 2 participants
  • k

    Kevin Kho

    05/14/2021, 12:48 AM
    Hello everyone, I organize the Orlando Data Science Meetup and I am planning an event around Prefect. I was wondering if there’s anyone here interested speaking at the meetup with me about their Prefect use case? You can also speak alone if you want the full stage. 😆
    :upvote: 3
    🚀 2
  • k

    Kha Nguyen

    05/14/2021, 12:51 PM
    Hi all. I am trying to setup ECS agent to run flows in Fargate. Assuming that I have an ECS Fargate cluster ready, here is how I start my agent
    from prefect.agent.ecs import ECSAgent
    
    agent = ECSAgent(
        name='Test Agent',
        region_name='eu-west-1',
        cluster='my-test-cluster',
        launch_type='FARGATE',
        run_task_kwargs_path='./config.yaml'
    )
    agent.start()
    This then failed with the following message
    [2021-05-14 12:46:54,069] ERROR - Test Agent | Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`
    Traceback (most recent call last):
      File "/Users/kha/Desktop/foo/agent.py", line 3, in <module>
        agent = ECSAgent(
      File "/opt/homebrew/Caskroom/miniforge/base/envs/zoined/lib/python3.9/site-packages/prefect/agent/ecs/agent.py", line 234, in __init__
        ] = self.infer_network_configuration()
      File "/opt/homebrew/Caskroom/miniforge/base/envs/zoined/lib/python3.9/site-packages/prefect/agent/ecs/agent.py", line 274, in infer_network_configuration
        raise ValueError(msg)
    ValueError: Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`
    c
    7 replies · 2 participants
  • k

    Kha Nguyen

    05/14/2021, 2:30 PM
    I cannot run a simple flow on ECS. I have setup a simple flow like this
    import prefect
    from prefect import task, Flow
    from prefect.run_configs import ECSRun
    
    
    @task
    def hello_task():
        logger = prefect.context.get("logger")
        <http://logger.info|logger.info>("Hello World!")
        return 0
    
    
    task_definition = {
        'networkMode': 'awsvpc',
        'cpu': 1024,
        'memory': 2048,
        'containerDefinitions': [
            {'name': 'flow'}
        ]
    }
    
    
    run_config = ECSRun(
        image='prefecthq/prefect',
        task_definition=task_definition
    )
    
    flow = Flow("hello-flow",
                tasks=[hello_task],
                run_config=run_config)
    
    flow.register(project_name='Prefect_Tutorials', labels=['aws', 'ecs', 'fargate'])
    The ECS agent can successfully trigger an ECS task. However, the flow run failed with error
    Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/Users/kha/'")
    . Why does it include something specific to my name in the flow?
    k
    30 replies · 2 participants
  • b

    Ben Davison

    05/14/2021, 4:13 PM
    Hi all, does anyone have any tricks to only respect a schedule based on a project name? Our use case is that we use prefect projects for our environments/branches (e.g. we have a master project, a staging project a branch_1 project etc etc), but we want to turn off all scheduling for everything apart from the master branch. We currently do some monkey patching at register time to set the schedule=None, but we'd rather have something in the prefect CLI. For example
    prefect register flow_name --label=staging --no-schedule
    We've also messed about with setting a bad label on the schedule itself, which "kind of" works, as in the flow won't run because it can't find an agent, but, the UI still get overwhelmed with flows being scheduled, but not being able to be run.
    k
    m
    +1
    8 replies · 4 participants
  • z

    Zach Schumacher

    05/14/2021, 4:54 PM
    is there a way to use secrets when registering a flow? E.g, I want to register flows via a CI pipeline, however I keep getting “local secret not found”. I would have thought that logging in w/ the cli and setting my backend to cloud wouldve forced prefect to look at cloud secrets, but maybe I am missing something? Setting environment variables does work, but I’d like to try to avoid specifying secrets via the UI and in circleci if possible.
    k
    8 replies · 2 participants
  • l

    Lone Pine Account

    05/14/2021, 5:14 PM
    There's a DatabricksRunNow task that I'm using for flows - but that task doesn't return any useful information when the databricks notebook runs. What I
Powered by Linen
Title
l

Lone Pine Account

05/14/2021, 5:14 PM
There's a DatabricksRunNow task that I'm using for flows - but that task doesn't return any useful information when the databricks notebook runs. What I
View count: 1