https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • c

    Carlos Cueto

    06/08/2022, 4:15 PM
    Hi everyone, having a bit of a configuration problem. I have a script that runs fine when I run it locally but when I register it with Cloud, it fails to execute on there with the following error:
    Failed to load and execute Flow's environment: ValueError('No flows found in file.')
    This is the flow definition:
    if __name__ == '__main__':
    with Flow('Scouter-Solr-Script') as flow:
    snowflake_user = PrefectSecret('snowflake_usr')
    snowflake_pwd = PrefectSecret('snowflake_pwd')
    snowflake_to_solr(snowflake_user, snowflake_pwd)
    flow.run_config = LocalRun(labels=['SVRNAME1'])
    flow.storage = Git(repo="Prefect-Flows", flow_path="Python/Scouter/snowflake_to_solr.py", git_clone_url_secret_name="azure_devops_clone_url")
    flow.register(project_name='Scouter')
    I'm assuming it has to do with the
    if __name__ == '__main__'
    part on top of the Flow class definition, but I don't know how to go about fixing this. I need that for multiprocessing that happens within the main task of the flow.
    k
    f
    11 replies · 3 participants
  • m

    Michał Augoff

    06/08/2022, 4:54 PM
    hi everyone, is it possible to add annotations to k8s jobs created by 2.0 agents? currently we inject Vault secrets via annotations and it works fine in 1.0 (we have a custom job template deployed with the agent) but I’m trying to scope the migration process and I haven’t seen any equivalent in 2.0 (I also looked at the agent code for 2.0b6)
    m
    2 replies · 2 participants
  • r

    Robin Doornekamp

    06/08/2022, 4:56 PM
    Hi all, relatively new to prefect 2.0 and using the cloud to get going. Had two questions that somewhat baffle me so far and I couldn't find in the docs: 1. regarding the storage. Configured remote S3 storage... what I'm not sure about is if this storage is used somehow in prefect? I had to give aws keys and I wasn't sure if this is somehow kept on prefect side (cloud). 2. i made a scheduled deployment, every 300 secs. when i look in the UI it seems to have put there about 100 flow runs into the future, but I can't figure out why approx 100... why not 20.. or 50.... I don't remember seeing any configuration for this.
    ✅ 1
    k
    8 replies · 2 participants
  • p

    Paco Ibañez

    06/08/2022, 5:37 PM
    Hello! where can I find out more info about what changed in the block api for beta6? I have a script that configures minio as default storage and I cant get it to work with the new apis. Before beta6 there were 2 abstractions bloc_spec and block and now there are 3? block_type, block_document and block_schema? thanks! https://github.com/fraibacas/prefect-orion/blob/main/flows/init_orion.py#L15
    m
    a
    3 replies · 3 participants
  • y

    YD

    06/08/2022, 5:42 PM
    Custom Roles Is it possible to limit access of users to specific projects in their "workspace" using "Custom Roles" in the enterprise edition ? If yes, is there an example? I do not see it in https://docs.prefect.io/orchestration/rbac/overview.html
    k
    1 reply · 2 participants
  • a

    aaron

    06/08/2022, 6:55 PM
    hey all — is there a straightforward way to run a flow continuously, sequentially and in perpetuity? like an infinite loop.
    ✅ 1
    a
    k
    4 replies · 3 participants
  • b

    Ben Ayers-Glassey

    06/08/2022, 7:14 PM
    Is it just me, or is the UI broken for flow runs at the moment? https://cloud.prefect.io/zesty-ai/flow-run/fc3a37ca-bcce-44b9-ae05-572ecf91328d
    m
    6 replies · 2 participants
  • t

    Tim Enders

    06/08/2022, 7:41 PM
    Where can I find the signals FAIL, SKIP, etc. in Prefect 2.0? I am getting an error when importing them from the old place.
    ✅ 1
    a
    k
    6 replies · 3 participants
  • d

    Derek Heyman

    06/08/2022, 8:10 PM
    I'm attempting to use map to reuse tasks and am running into the following error. Any insight into what I might be doing wrong?
    Unexpected error: KeyError(0)
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
        new_state = method(self, state, *args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/prefect/engine/flow_runner.py", line 569, in get_flow_run_state
        executors.prepare_upstream_states_for_mapping(
      File "/usr/local/lib/python3.8/site-packages/prefect/utilities/executors.py", line 681, in prepare_upstream_states_for_mapping
        value = upstream_state.result[i]
    KeyError: 0
    k
    12 replies · 2 participants
  • f

    FuETL

    06/08/2022, 8:52 PM
    Hey guys, i started getting this error when running my flows on ECS Agent:
    An error occurred (ThrottlingException) when calling the RegisterTaskDefinition operation (reached max retries: 2): Rate exceeded
    This means that i should increase my Rate limit on my AWS account?
    m
    6 replies · 2 participants
  • v

    Volker L

    06/08/2022, 9:03 PM
    Hi. is it possible in prefect 2.0 to delete all deployments at once? Thanks
    ✅ 1
    a
    f
    11 replies · 3 participants
  • r

    Robin

    06/08/2022, 9:04 PM
    Is there an ECS flow runner planned for 2.0? Couldnt find anything in the v2 tagged github issues... Thanks!
    ✅ 1
    m
    a
    +1
    6 replies · 4 participants
  • t

    Tim Enders

    06/08/2022, 9:06 PM
    Getting this message.
    TypeError: cannot unpack non-iterable PrefectFuture object
    Do I just need to do
    @task(nout=2)
    in Prefect 2.0?
    m
    2 replies · 2 participants
  • t

    Tim Enders

    06/08/2022, 9:17 PM
    One last question for today... Mapping Tasks... is this still TODO on Prefect 2.0?
    m
    11 replies · 2 participants
  • j

    Jenia Varavva

    06/08/2022, 11:08 PM
    Is the Cloud RBAC granularity at the tenant level? I.e. I’d like to have everyone be a reader, but some users to only be able to “write” some flows but not others. Separately, I’d also like to have “per-team” agents and some way to limit which agents can a user submit flows to. Am I better sticking to just separate tenants/separate servers for on-prem deployment, than considering the RBAC for this? Is
    workspace
    the 2.0 cloud analogue to the 1.0 tenant?
    ✅ 1
    a
    k
    2 replies · 3 participants
  • j

    Jose Daniel Posada Montoya

    06/08/2022, 11:08 PM
    Hi, someone know if there is a way of installing just the core of Prefect Orion, without the REST API, UI, CLI, just the python API? Thanks in advance.
    ✅ 1
    a
    k
    7 replies · 3 participants
  • d

    Daniel Lomartra

    06/08/2022, 11:31 PM
    Is there a way to force a specific mapped task to run in sequence even though the flow is using a local dask executor? I have a flow with a bunch of mapped tasks that I want to run in parallel but one mapped task that i'd like run in sequence.
    ✅ 1
    a
    k
    3 replies · 3 participants
  • k

    Khang Lam

    06/08/2022, 11:35 PM
    Hi folks, I'm trying to schedule a flow but keep getting
    Something went wrong. Please wait a few moments and try again
    . Flow runs fine locally. Any idea how to troubleshoot this?
    ✅ 1
    a
    5 replies · 2 participants
  • c

    Carlos Cueto

    06/09/2022, 3:56 AM
    Hi everyone. Has somebody successful ran flows that use Dask Dataframes or multiprocessing library on a Windows Local Agent run deployed on Cloud? I am finding it impossible to define a flow and wrap it inside the necessary
    if __name__ == ‘__main__’
    . It all works fine when running locally with
    flow.run()
    but it tells me it can't find a Flow when I deploy it to Cloud and run it from there.
    k
    2 replies · 2 participants
  • t

    Thomas Fredriksen

    06/09/2022, 6:30 AM
    Hi there, is it possible to obtain the dask client when running with the `DaskExecutor`/`LocalDaskExecutor` or the
    DaskTaskRunner
    in Orion?
    ✅ 1
    a
    k
    3 replies · 3 participants
  • j

    Joshua Greenhalgh

    06/09/2022, 9:10 AM
    Morning - where there any issues with cloud this morning? I have a flow that failed (sent no slack message) - error was;
    raise ClientError(result["errors"])
    prefect.exceptions.ClientError: [{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
    👍 1
    ✅ 1
    a
    k
    12 replies · 3 participants
  • s

    Samuel Hinton

    06/09/2022, 10:05 AM
    Hey team! With prefect 2 coming soon, would someone be able to point me in the direction of the benefits prefect cloud will have over hosting our own server? I notice the pricing page highlights the support and escalation, I was wondering if there was a similar breakdown to the technical benefits (for example, right now I know prefect cloud supports flow concurrency limits, while prefect server 1.x does not).
    ✅ 1
    a
    2 replies · 2 participants
  • m

    Marcin Grzybowski

    06/09/2022, 11:32 AM
    Hi all, how do we build docker image from orion branch ?
    Executing task: docker build --pull --rm -f "Dockerfile" -t myprefect:latest "." <
    
    Sending build context to Docker daemon  101.5MB
    Step 1/38 : ARG PYTHON_VERSION=3.8
    Step 2/38 : ARG BUILD_PYTHON_VERSION=3.8
    Step 3/38 : ARG NODE_VERSION=14
    Step 4/38 : FROM node:${NODE_VERSION}-bullseye-slim as ui-builder
    14-bullseye-slim: Pulling from library/node
    Digest: sha256:f7137af1e34927cb3251a6e091edff4592c49a89422196680a2f087edcbc6e4d
    Status: Image is up to date for node:14-bullseye-slim
     ---> e2eb23c871f2
    Step 5/38 : WORKDIR /opt/orion-ui
     ---> Using cache
     ---> 74138fe30a86
    Step 6/38 : RUN apt-get update &&     apt-get install --no-install-recommends -y         chromium     && apt-get clean && rm -rf /var/lib/apt/lists/*
     ---> Using cache
     ---> 77731060b4e3
    Step 7/38 : RUN npm install -g npm@8
     ---> Using cache
     ---> ac23c1b14bea
    Step 8/38 : COPY ./orion-ui/package*.json .
    When using COPY with more than one source file, the destination must be a directory and end with a /
    The terminal process "/usr/bin/bash '-c', 'docker build --pull --rm -f "Dockerfile" -t myprefect:latest "."'" terminated with exit code: 1.
    ✅ 1
    a
    8 replies · 2 participants
  • a

    Apostolos Papafragkakis

    06/09/2022, 12:58 PM
    Hello all, I recently started experimenting with Prefect for one of my projects. I would like to ask whether it is possible to use the asynchronous functionality provided by dask; there are some tasks that involve IO output and as such they could be executed asynchronously (I am using the latest release of Prefect 2.0).
    ✅ 1
    a
    m
    26 replies · 3 participants
  • f

    Florian Guily

    06/09/2022, 1:09 PM
    hey everyone, is there a way to use variables that are global to every task like an increment counter ?
    ✅ 1
    a
    4 replies · 2 participants
  • a

    Anna Geller

    06/09/2022, 1:10 PM
    Cross-posting here for visibility - this is a fantastic piece of content by Peyton that will: • walk you through getting started writing data pipelines in Python • showing challenges of operationalizing such Python script - the negative engineering part • walk you through how to turn this script onto a Prefect 2.0 flow • show you around the new Prefect 2.0 UI
    👍 1
    :wizard2: 1
  • f

    Florian Guily

    06/09/2022, 1:49 PM
    Hey, was wondering if mapped tasks are able to output multiple values with nout ? i have an error when trying with dummy code:
    from asyncio.log import logger
    import prefect, pymongo, datetime
    from prefect import task, Flow, flatten
    from prefect.tasks.secrets import PrefectSecret
    from prefect.tasks.mysql.mysql import MySQLFetch
    from prefect.storage import GitHub
    
    @task(nout= 3)
    def produce_output(input):
        return input*2-1, input*2, ["val"+str(input*2-1), "val"+str(input*2)]
    
    @task
    def reduce(a):
        return sum(a)
    
    @task
    def retry_post(data):
        logger = prefect.context.get("logger")
        <http://logger.info|logger.info>(data)
    
    with Flow("abcdef") as flow:
        success, fail, to_retry = produce_output.map([1, 2, 3, 4])
        total_success = reduce(success)
        total_fail = reduce(fail)
        retry_post.map(flatten(to_retry))
    
    flow.run()
    ✅ 1
    a
    2 replies · 2 participants
  • s

    Samarth

    06/09/2022, 2:35 PM
    Hey, have a question: • Using Prefect 1.0 • Using
    Local
    storage and
    Prefect Cloud
    I want to use
    stored_as_script=True
    for my flow (pickle storage is not working with an SQLAlchemy object for some reason), and so I need to specify
    path
    — but that is giving me an error. Sharing the relevant code snippet and error.
    with Flow(
            "eco_flow", 
            schedule=schedule, 
            storage=Local(stored_as_script=True, path='~/.prefect/flows/test-flow.py')
        ) as flow:
    
        cities = ['mumbai', 'new-delhi']
        
        for city in cities:
            data = get_traffic_data(city)  
            clean_data = clean_traffic_data(data)
            insert_to_db(clean_data)
    Error after registering the flow:
    Failed to load and execute flow run: ModuleNotFoundError("No module named '~/'")
    I guess I am defining the
    path
    incorrectly?
    k
    2 replies · 2 participants
  • t

    Tim Enders

    06/09/2022, 2:39 PM
    Trying to use the DaskTaskRunner in Prefect 2.0 and getting this error. I am following the published documentation.
    from prefect.task_runner import DaskTaskRunner
    ModuleNotFoundError: No module named 'prefect.task_runner'
    Is there an extra that needs installed?
    ✅ 1
    a
    k
    12 replies · 3 participants
  • t

    Tim Enders

    06/09/2022, 2:51 PM
    OK, So I am trying to use the DaskRunner with a for loop to concurrently run a task over a list.
    @flow(name="Subscriptions Flow",
          task_runner=DaskTaskRunner())
    def main():
       *snip*
    
        pages_list = get_pages_list(client, "subscriptions", params)
        for page in pages_list:
            item_list = get_items_list(client, "subscriptions", page)
     
    
    if __name__ == "__main__":
        flow_result = main()
        print(flow_result)
    But I get a RuntimeError from the multiprocessing. Can I get some help on what I am doing wrong with the DaskRunner. I want to parallel run over what is in
    pages_list
    k
    m
    6 replies · 3 participants
Powered by Linen
Title
t

Tim Enders

06/09/2022, 2:51 PM
OK, So I am trying to use the DaskRunner with a for loop to concurrently run a task over a list.
@flow(name="Subscriptions Flow",
      task_runner=DaskTaskRunner())
def main():
   *snip*

    pages_list = get_pages_list(client, "subscriptions", params)
    for page in pages_list:
        item_list = get_items_list(client, "subscriptions", page)
 

if __name__ == "__main__":
    flow_result = main()
    print(flow_result)
But I get a RuntimeError from the multiprocessing. Can I get some help on what I am doing wrong with the DaskRunner. I want to parallel run over what is in
pages_list
k

Kevin Kho

06/09/2022, 2:53 PM
Can you share the error here?
t

Tim Enders

06/09/2022, 2:54 PM
I hadn't saved a typo fix 🤦
Now I get maximum recursion depth exceeded... which is a whole other can of worms
🤦‍♂️ 1
m

Michael Adkins

06/09/2022, 2:56 PM
You’ll need to iterate over
pages_list.result()
Not sure why you’re getting a recursion error though
t

Tim Enders

06/09/2022, 2:57 PM
Oh that is my API client failing to get a token... which is not a Prefect problem
View count: 3