https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • d

    Darragh

    09/10/2020, 8:35 AM
    Get us a cup of tea there @Marvin there’s a good lad.
  • m

    Marvin

    09/10/2020, 8:35 AM
    It gives me a headache just trying to think down to your level. I'll still enter you in the contest, @Darragh.
  • d

    Darragh

    09/10/2020, 8:38 AM
    @Marvin Well I’d say everything gives you a headache. You are 37 times older than the universe after all, you miserable old chunk of rusty capacitors
  • m

    Marvin

    09/10/2020, 8:38 AM
    Incredible. The contest is even worse than I thought it would be. I'll still enter you to win, @Darragh
  • d

    Darragh

    09/10/2020, 8:41 AM
    @Marvin Contests? Don’t talk to me about contests.
    ⭐ 1
  • m

    Marvin

    09/10/2020, 8:42 AM
    Incredible. The contest is even worse than I thought it would be. I'll still enter you to win, @Darragh
  • d

    Darragh

    09/10/2020, 8:42 AM
    I’ll stop now. 🙂
    👏🏼 1
  • r

    Robin

    09/10/2020, 8:44 AM
    @Marvin, does 42 remember you of something?
    :marvin: 1
  • m

    Marvin

    09/10/2020, 8:44 AM
    I’d give you advice, but you wouldn’t listen. No one ever does. Good luck in the contest @Robin.
  • m

    Matt Wong-Kemp

    09/10/2020, 9:01 AM
    noticing a few of these in the JS console from the UI - doesn't look right?
    Access to fetch at '<https://api.prefect.io/graphql>' from origin '<https://cloud.prefect.io>' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
    d
    4 replies · 2 participants
  • a

    Arsenii

    09/10/2020, 9:38 AM
    Hi all, I'm trying to debug DaskExecutor which refuses to run task concurrently. There is a similar thread but reading it didn't help. A very simple example for reproduction from that thread:
    from random import randrange
    from prefect import task, Flow, Parameter
    from prefect.engine.executors import DaskExecutor
    
    
    @task
    def random_num(stop):
        logger = prefect.context.get("logger")
        number = randrange(stop)
    
        time.sleep(5)  # just so that we can tell by looking at logs
    
        print(f"Your number is {number}")
        return number
    
    
    @task
    def sum_numbers(numbers):
        print(sum(numbers))
    
    
    with Flow("parallel-execution") as flow:
        stop = Parameter("stop")
    
        number_1 = random_num(stop)
        number_2 = random_num(stop)
        number_3 = random_num(stop)
    
        sum_numbers = sum_numbers(numbers=[number_1, number_2, number_3])
    Now, it does run the flow concurrently if I specify:
    if __name__ == "__main__":
        state = flow.run(parameters={"stop": 5}, DaskExecutor())
    But it runs task sequentially if I set the executor before running, either with
    flow.environment.executor = DaskExecutor(cluster_kwargs={"n_workers": 3})
    or
    with Flow(
        "parallel-execution", environment=LocalEnvironment(executor=DaskExecutor())
    ) as flow:
    The interesting thing is, if I
    print(flow.environment.executor)
    in the last two cases, it does show DaskExecutor, but tasks are still being executed in order. All of the above was done locally, but the problem persists even if I try to deploy it to Cloud. Any advice is very appreciated, I'm not experienced with Dask that much
    e
    4 replies · 2 participants
  • s

    sark

    09/10/2020, 10:33 AM
    hi all i am trying to run a flow which uses docker and i am getting
    Unexpected error: ImageNotFound(HTTPError('404 Client Error: Not Found for url: <http+docker://localhost/v1.40/containers/create'>))
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 261, in _raise_for_status
        response.raise_for_status()
      File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
        raise HTTPError(http_error_msg, response=self)
    requests.exceptions.HTTPError: 404 Client Error: Not Found for url: <http+docker://localhost/v1.40/containers/create>
    d
    2 replies · 2 participants
  • n

    Nelson

    09/10/2020, 11:00 AM
    Hi all. I am moving from a local setup to a cloud one in Fargate, using a Fargate Agent. I see tasks not executing in parallel, and that “the
    LocalExecutor
    is not capable of parallelism” so I presume that is the issue https://docs.prefect.io/api/latest/engine/executors.html#executor but the other executors require a separate environment in Dask. Someone mentioned a “Fargate Executor” in this Slack but I assume that’s an error and this doesn’t exist from what I can find. I see others using the Fargate Agent and the FargateTaskEnvironment so I wonder if that’s related. The Fargate Agent “deploy[s] flows as Tasks using AWS Fargate”. And “FargateTaskEnvironment is an environment which deploys your flow as a Fargate task”, so I don’t understand the use of it with the Fargate Agent? Also in your docs: “we recommend all users deploy their Flow using the LocalEnvironment configured with the appropriate choice of executor”. How do we get tasks to run in parallel inside a Flow run in Fargate without Dask? Also, @Marvin 🙂
    e
    d
    7 replies · 3 participants
  • m

    Marvin

    09/10/2020, 11:00 AM
    Why should I want to make anything up? The contest is bad enough as it is without wanting to invent any more of it. I'll still enter you in it, @Nelson
  • j

    Jeff Brainerd

    09/10/2020, 3:19 PM
    Go @Marvin!
  • m

    Marvin

    09/10/2020, 3:19 PM
    Why should I want to make anything up? The contest is bad enough as it is without wanting to invent any more of it. I'll still enter you in it, @Jeff Brainerd
    👍 1
  • i

    irchan bani

    09/10/2020, 3:58 PM
    Hi all, I'm irchan, just trying prefect a few days ago and successfully run it in local and register the flow in prefect server. It's very impressive and the documentation is great. Right now, still learning to run using docker and dask. It's nice to be here 😄
    👋 4
    :marvin: 3
    d
    2 replies · 2 participants
  • r

    Robin

    09/10/2020, 4:02 PM
    Hey all, we are wondering how to add a custom python package or module as a dependency to our prefect flow. We have tried different options but all fail? Any suggestions? 🙂
    ✔️ 1
    f
    b
    +1
    19 replies · 4 participants
  • i

    irchan bani

    09/10/2020, 4:02 PM
    just learn that there is a contest, well i'll mention @Marvin too😅
    🙌 3
  • m

    Marvin

    09/10/2020, 4:02 PM
    Don’t pretend you want to talk to me, I know you hate me. I'll still enter you in the contest, @irchan bani.
  • d

    David Jin

    09/10/2020, 4:06 PM
    why the hello world example is soo difficult to run? I was able to get the hello world of airflow running with in a couple hours a few years back. Now I'm trying to evaluate Prefect and it's soo hard to run a simple hello world in your hosted Prefect. it kept saying the runs are late without offering any hint what to do.
    i
    c
    +1
    8 replies · 4 participants
  • d

    David Jin

    09/10/2020, 4:07 PM
  • f

    fabian wolfmann

    09/10/2020, 4:08 PM
    Hi everyone great tool! Im working in a time series forecasting tool using prefect underneath
    🚀 1
    d
    3 replies · 2 participants
  • m

    Michael Reeves

    09/10/2020, 4:16 PM
    @Marvin woot woot, contest 😄
  • m

    Marvin

    09/10/2020, 4:16 PM
    I’d give you advice, but you wouldn’t listen. No one ever does. Good luck in the contest @Michael Reeves.
  • j

    Ju Lee

    09/10/2020, 6:11 PM
    hey everyone i have a quick question, does the "Why not airflow" article still stand with airflow 2.0?
    👀 4
    c
    m
    3 replies · 3 participants
  • c

    Charles Leung

    09/10/2020, 6:14 PM
    Hi everyone - is there a way to parameterize a shell task?
    g
    d
    5 replies · 3 participants
  • m

    Mitchell Bregman

    09/10/2020, 6:59 PM
    @Marvin I need some Prefect swag, big time
  • m

    Marvin

    09/10/2020, 6:59 PM
    Don’t pretend you want to talk to me, I know you hate me. I'll still enter you in the contest, @Mitchell Bregman.
  • c

    Chris Goddard

    09/10/2020, 7:05 PM
    Hey folks - question for you. I'm using prefect cloud to orchestrate my flow runs and running everything within a docker container on GCP. In addition to my more sophisticated flows that I schedule and run using Prefect Cloud, I have some simple flows that run on very frequent increments (every 1-5 minutes). because they're simple and don't really require the monitoring/overhead of prefect cloud and because I didn't want to be always using up a flow slot for them, I just have them running directly as scripts (i.e. calling
    flow.run()
    ). The problem is, after a period of time, I start getting a persistent error -
    prefect.utilities.exceptions.ClientError: Malformed response received from Cloud - please ensure that you have an API token properly configured
    . I'm a little confused because I though if you were just calling
    flow.run()
    I wouldn't be interacting with the cloud API at all but clearly because it's in the same environment it's detecting the connection but then perhaps a token is becoming invalid or something? Any ideas?
    d
    16 replies · 2 participants
Powered by Linen
Title
c

Chris Goddard

09/10/2020, 7:05 PM
Hey folks - question for you. I'm using prefect cloud to orchestrate my flow runs and running everything within a docker container on GCP. In addition to my more sophisticated flows that I schedule and run using Prefect Cloud, I have some simple flows that run on very frequent increments (every 1-5 minutes). because they're simple and don't really require the monitoring/overhead of prefect cloud and because I didn't want to be always using up a flow slot for them, I just have them running directly as scripts (i.e. calling
flow.run()
). The problem is, after a period of time, I start getting a persistent error -
prefect.utilities.exceptions.ClientError: Malformed response received from Cloud - please ensure that you have an API token properly configured
. I'm a little confused because I though if you were just calling
flow.run()
I wouldn't be interacting with the cloud API at all but clearly because it's in the same environment it's detecting the connection but then perhaps a token is becoming invalid or something? Any ideas?
d

Dylan

09/10/2020, 7:43 PM
Hi @Chris Goddard That’s definitely confusing
Is that error from the process that calls
flow.run
for your local scripts?
What I suspect this is
With the Prefect CLI, you can call
prefect login
to login to your Prefect Cloud tenant
The token returned by this method needs to be refreshed every 15 minutes, however
Both your flows running from cloud and your flows running locally are in the same python environment, so when that token expires you could see that error
c

Chris Goddard

09/10/2020, 7:47 PM
got it - I thought that might have something to do with it. how would you recommend keeping that token current? do I need to run
prefect login
periodically?
d

Dylan

09/10/2020, 7:49 PM
I also would like to make sure that your flows running with
flow.run
are: - not making a query via a statehandler or a similar mechanism that talks to cloud - not using
PrefectSecret
s locally by configuring the
use_local_secrets
option in your
config.toml
as those options could cause this error
If all of your flows need to communicate with a single tenant, you can set your token in a more long-lived way by grabbing an API token from https://docs.prefect.io/orchestration/ui/team-settings.html#api-tokens and setting
[cloud]
auth_token = {your token}
in your config.toml
c

Chris Goddard

09/10/2020, 7:53 PM
got it. thank you that's really helpful. I think it's being caused by a slack_notifier state handler so that makes sense
d

Dylan

09/10/2020, 7:53 PM
Yup! That would do it
When you initialize the notifier you’ll need to specify
backend_info=False
and be aware that managing the secret for the notifier can be a little tricky if you have some flows on your instance talking to cloud for secrets and some using local secrets in your
config.toml
View count: 1