• Samuel Hinton

    Samuel Hinton

    3 months ago
    Hey team! With prefect 2 coming soon, would someone be able to point me in the direction of the benefits prefect cloud will have over hosting our own server? I notice the pricing page highlights the support and escalation, I was wondering if there was a similar breakdown to the technical benefits (for example, right now I know prefect cloud supports flow concurrency limits, while prefect server 1.x does not).
    Samuel Hinton
    Anna Geller
    2 replies
    Copy to Clipboard
  • m

    Marcin Grzybowski

    3 months ago
    Hi all, how do we build docker image from orion branch ?
    Executing task: docker build --pull --rm -f "Dockerfile" -t myprefect:latest "." <
    
    Sending build context to Docker daemon  101.5MB
    Step 1/38 : ARG PYTHON_VERSION=3.8
    Step 2/38 : ARG BUILD_PYTHON_VERSION=3.8
    Step 3/38 : ARG NODE_VERSION=14
    Step 4/38 : FROM node:${NODE_VERSION}-bullseye-slim as ui-builder
    14-bullseye-slim: Pulling from library/node
    Digest: sha256:f7137af1e34927cb3251a6e091edff4592c49a89422196680a2f087edcbc6e4d
    Status: Image is up to date for node:14-bullseye-slim
     ---> e2eb23c871f2
    Step 5/38 : WORKDIR /opt/orion-ui
     ---> Using cache
     ---> 74138fe30a86
    Step 6/38 : RUN apt-get update &&     apt-get install --no-install-recommends -y         chromium     && apt-get clean && rm -rf /var/lib/apt/lists/*
     ---> Using cache
     ---> 77731060b4e3
    Step 7/38 : RUN npm install -g npm@8
     ---> Using cache
     ---> ac23c1b14bea
    Step 8/38 : COPY ./orion-ui/package*.json .
    When using COPY with more than one source file, the destination must be a directory and end with a /
    The terminal process "/usr/bin/bash '-c', 'docker build --pull --rm -f "Dockerfile" -t myprefect:latest "."'" terminated with exit code: 1.
    m
    Anna Geller
    8 replies
    Copy to Clipboard
  • Apostolos Papafragkakis

    Apostolos Papafragkakis

    3 months ago
    Hello all, I recently started experimenting with Prefect for one of my projects. I would like to ask whether it is possible to use the asynchronous functionality provided by dask; there are some tasks that involve IO output and as such they could be executed asynchronously (I am using the latest release of Prefect 2.0).
    Apostolos Papafragkakis
    Anna Geller
    +1
    26 replies
    Copy to Clipboard
  • Florian Guily

    Florian Guily

    3 months ago
    hey everyone, is there a way to use variables that are global to every task like an increment counter ?
    Florian Guily
    Anna Geller
    4 replies
    Copy to Clipboard
  • Anna Geller

    Anna Geller

    3 months ago
    Cross-posting here for visibility - this is a fantastic piece of content by Peyton that will: • walk you through getting started writing data pipelines in Python • showing challenges of operationalizing such Python script - the negative engineering part • walk you through how to turn this script onto a Prefect 2.0 flow • show you around the new Prefect 2.0 UI
  • Florian Guily

    Florian Guily

    3 months ago
    Hey, was wondering if mapped tasks are able to output multiple values with nout ? i have an error when trying with dummy code:
    from asyncio.log import logger
    import prefect, pymongo, datetime
    from prefect import task, Flow, flatten
    from prefect.tasks.secrets import PrefectSecret
    from prefect.tasks.mysql.mysql import MySQLFetch
    from prefect.storage import GitHub
    
    @task(nout= 3)
    def produce_output(input):
        return input*2-1, input*2, ["val"+str(input*2-1), "val"+str(input*2)]
    
    @task
    def reduce(a):
        return sum(a)
    
    @task
    def retry_post(data):
        logger = prefect.context.get("logger")
        <http://logger.info|logger.info>(data)
    
    with Flow("abcdef") as flow:
        success, fail, to_retry = produce_output.map([1, 2, 3, 4])
        total_success = reduce(success)
        total_fail = reduce(fail)
        retry_post.map(flatten(to_retry))
    
    flow.run()
    Florian Guily
    Anna Geller
    2 replies
    Copy to Clipboard
  • Samarth

    Samarth

    3 months ago
    Hey, have a question: • Using Prefect 1.0 • Using
    Local
    storage and
    Prefect Cloud
    I want to use
    stored_as_script=True
    for my flow (pickle storage is not working with an SQLAlchemy object for some reason), and so I need to specify
    path
    — but that is giving me an error. Sharing the relevant code snippet and error.
    with Flow(
            "eco_flow", 
            schedule=schedule, 
            storage=Local(stored_as_script=True, path='~/.prefect/flows/test-flow.py')
        ) as flow:
    
        cities = ['mumbai', 'new-delhi']
        
        for city in cities:
            data = get_traffic_data(city)  
            clean_data = clean_traffic_data(data)
            insert_to_db(clean_data)
    Error after registering the flow:
    Failed to load and execute flow run: ModuleNotFoundError("No module named '~/'")
    I guess I am defining the
    path
    incorrectly?
    Samarth
    Kevin Kho
    2 replies
    Copy to Clipboard
  • Tim Enders

    Tim Enders

    3 months ago
    Trying to use the DaskTaskRunner in Prefect 2.0 and getting this error. I am following the published documentation.
    from prefect.task_runner import DaskTaskRunner
    ModuleNotFoundError: No module named 'prefect.task_runner'
    Is there an extra that needs installed?
    Tim Enders
    Anna Geller
    +1
    12 replies
    Copy to Clipboard
  • Tim Enders

    Tim Enders

    3 months ago
    OK, So I am trying to use the DaskRunner with a for loop to concurrently run a task over a list.
    @flow(name="Subscriptions Flow",
          task_runner=DaskTaskRunner())
    def main():
       *snip*
    
        pages_list = get_pages_list(client, "subscriptions", params)
        for page in pages_list:
            item_list = get_items_list(client, "subscriptions", page)
     
    
    if __name__ == "__main__":
        flow_result = main()
        print(flow_result)
    But I get a RuntimeError from the multiprocessing. Can I get some help on what I am doing wrong with the DaskRunner. I want to parallel run over what is in
    pages_list
    Tim Enders
    Kevin Kho
    +1
    6 replies
    Copy to Clipboard
  • Tim Enders

    Tim Enders

    3 months ago
    What is the equivalent to this in 2.0?
    @task(checkpoint=False)
    If there is one currently
    Tim Enders
    Kevin Kho
    +2
    5 replies
    Copy to Clipboard