https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • l

    Luke Orland

    11/24/2020, 6:02 PM
    is there a way to test-run a flow locally in Docker, something like
    flow.run(run_on_schedule=False)
    or using FlowRunner, without using an agent?
    s
    7 replies · 2 participants
  • j

    josh

    11/24/2020, 8:02 PM
    Hey team, Prefect version 
    0.13.17
     has been released and here are a few notable changes:   😛refect-duck:😛refect-duck: Nicer support for tasks with multiple results   ⚠️   Improved logs and warnings   ♻️   Cleaner flow-run restarts when using resource managers #️⃣ Squashed bug in flow-hashing for idempotency A big thank you to our contributors who helped out with this release! Full changelog: https://github.com/PrefectHQ/prefect/releases/tag/0.13.17
    Untitled
    🚀 9
    :prefect-duck: 3
    :marvin: 6
    :prefect: 5
    j
    1 reply · 2 participants
  • m

    Mac Gréco Péralte Chéry

    11/24/2020, 8:37 PM
    Hello Eveveryone! I am using a CI/CD process with AWS Codepipeline (source: Github, build: Codebuild) the output is a docker image i push to an AWS ECR repository for each flow. I my github project i have a buildspec file where i put this code:
    build:
        commands:
          - python --version
          - pyenv global 3.7.7
          - python --version
          - pip install $(grep -ivE "pywin32" requirements.txt)
          - prefect backend cloud
          - prefect auth login -t $PREFECT_ACCESS_TOKEN
          - aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin <http://xxxxxxxx.dkr.ecr.us-east-2.amazonaws.com|xxxxxxxx.dkr.ecr.us-east-2.amazonaws.com>
          - python3.7 flows/my_flow.py
    From November 20 docker introduced pull limits for images what cause me to have this error in Code build
    File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 619, in pull_image
        output = client.pull(self.base_image, stream=True, decode=True)
      File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/docker/api/image.py", line 415, in pull
        self._raise_for_status(response)
      File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/docker/api/client.py", line 261, in _raise_for_status
        raise create_api_error_from_http_exception(e)
      File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
        raise cls(e, response=response, explanation=explanation)
    docker.errors.APIError: 500 Server Error: Internal Server Error ("toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: <https://www.docker.com/increase-rate-limit>")
    When i go to the site i can read: "Anonymous free users will be limited to 100 pulls per six hours, and authenticated free users will be limited to 200 pulls per six hours." So i created a account on docker to raise my limits to 200 pull per 6 hours. So i understand the problem occurs when prefect is pulling the base prefect docker image to build my flow image because i have reached my 100 pulls / 6 hours anonymous limit My question is where should i put my docker account token to pull the prefect base image via my newly created docker account?
    m
    j
    4 replies · 3 participants
  • k

    Krzysztof Nawara

    11/24/2020, 11:22 PM
    message has been deleted
  • h

    Hui Zheng

    11/24/2020, 11:29 PM
    hello Prefect Support, I have a flow run that, right after it started the first task
    init_run
    ,
    Task 'init_run': Starting task run...
    it got this error message ( see thread) and, the log message said
    Task 'init_run': Finished task run for task with final state: 'ClientFailed'
    After that, All remaining tasks stuck in
    Pending
    state for hours. Any ideas what happened? Maybe my flow need to know how to handle this
    ClientFailed
    state ?
    j
    3 replies · 2 participants
  • k

    Kilian

    11/24/2020, 11:50 PM
    Hi all, been using prefect for a while now, and big fan of it. I have the following use case and am not sure if it's supported by prefect. I have a flow that first downloads quite a large file and then processes it using a GPU. I would like to set this up in a parallel cluster of some sort. However, I wonder how can I ensure that the file is downloaded and processed on the same node. To my understanding, a task can be executed on any machine if it is ready to be processed and only waiting for the upstream tasks to finish. Meaning, it might happen that the file downloaded on one machine, the gpu processing is waiting but might start on another machine and then can obviously not find the file. One workaround would be to make this one big task and request a gpu for it, but then if the download takes quite long the GPU could have been utilised elsewhere. Are there any other options. Basically enforcing that two tasks are executed on any yet the same machine? Thank you
  • l

    Lior

    11/25/2020, 11:26 AM
    Hey, I am looking to persist to local storage tasks that output pandas dataframes, do you have a relevant example?
    j
    c
    8 replies · 3 participants
  • a

    Adam

    11/25/2020, 11:57 AM
    Any uptime issues at the moment? The Prefect Cloud UI is not loading anything and I see errors in the console like
    Access to fetch at '<https://api.prefect.io/graphql>' from origin '<https://cloud.prefect.io>' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled
    j
    3 replies · 2 participants
  • e

    Enrique

    11/25/2020, 1:05 PM
    Hi prefect team! I have an issue when registering a flow using docker storage. I have a Github Action that downloads the latest version of prefect and register the new flow when the branch changes. I haven't introduced any change since October 30th using Prefect Version 0.13.13. Today when I tried to update my flow this error has appeared. I've tried to do the same from my computer (using version 0.13.13) and it works fine. In our team we follow the update announcements of new versions of Prefect but we haven't seen anything that could affect to our flow. Do you have any idea of what the problem could be? We are using the following command to register if it can help to find the problem:
    flow.register(project_name=env_label, add_default_labels=False)
    We use also the
    flow.storage = Docker(...)
    command. Thanks for your time, and sorry if it is an obvious question 😅
    j
    m
    4 replies · 3 participants
  • s

    Saulius Beinorius

    11/25/2020, 1:45 PM
    Hi, it seems that when cancelling a flow run when there are tasks within a resource manager segment, the resource manager cleanup is not called, is that intentional (or can be enabled somewhere)?
    j
    j
    5 replies · 3 participants
  • p

    Peter Peter

    11/25/2020, 3:55 PM
    Hello, We are currently looking at shelltask and dbtshelltask and we are wondering why helper_script is only in the constructor and why it cannot set when we call run. There must be a reason for this. If someone could shed some light on this. Thanks Peter
    j
    m
    4 replies · 3 participants
  • p

    Peter Peter

    11/25/2020, 4:10 PM
    Hello, In the documentation it show that when a flow is in pause state that it will show up in the dashboards notifications. We are not seeing this, Is there some setting we need to change or enable notification in our flow? here is where we see it. https://docs.prefect.io/core/idioms/pause-for-approval.html#pause-for-approval Thanks
    j
    3 replies · 2 participants
  • r

    Riley Hun

    11/25/2020, 6:10 PM
    Hi everyone - This question pertains to the helm chart for deploying Prefect Server on k8s. I'm trying to use an external database as was recommended in the docs. However, I am dealing with an issue when trying to connect to a postgres cloudsql instance on GCP.
    m
    50 replies · 2 participants
  • v

    Vinod Sugur

    11/25/2020, 7:26 PM
    Hi everyone - How to convert prefect python dict parameter to JSON string? I am looking at JSONCompatible. Please point me to some examples.
    j
    c
    5 replies · 3 participants
  • j

    jack

    11/26/2020, 5:09 AM
    Heya - can anyone point me in the right direction/documentation/example if I wanted to register a flow on Prefect Cloud by pushing changes to a repo in Azure DevOps?
  • h

    Hamed Sheykhlou

    11/26/2020, 7:18 AM
    Hi guys. is there any way to doing the integration testing with prefect flow? I mean how can I doing integration testing of my docker based flow?
  • l

    Lior

    11/26/2020, 9:54 AM
    Is there a way to stop a mapping task if one of the tasks in the map failed? (stop all the other tasks). It's possible with dask Variable
  • j

    Jonas Hanfland

    11/26/2020, 1:13 PM
    Hi guys, I am having an issue with one of my flows not terminating due to a single mapped task ("mapped child 1") being stuck in pending. When checking the logs for that mapped task, I found what seems to be an internal server error (see thread). I think there have been one or two flows in the past, with the same kind of issue (one mapped task stuck in pending), but I am not able to retrieve their logs so I can't be sure. For the old runs, subsquent re-runs finished successfully, but this time the re-run seems to run into the same issue (+ same exception in logs). Is that exception the cause for the problem? Is the issue on prefect's side or on mine?
    c
    3 replies · 2 participants
  • i

    Iain Dillingham

    11/26/2020, 1:22 PM
    Hi everyone. I recall that it was once possible to store parameter values to the Prefect DB. I can't find a reference in the docs, though. Did I recall wrong? Should I look harder? 🙂 Thanks for your help!
    1 reply · 1 participant
  • a

    Analytics BSTD

    11/26/2020, 3:38 PM
    Hi everyone, I've just started to use prefect and have created a simple test flow. It's working good localy, but when I added agent, I have an error: "ERROR - agent | Error while deploying flow: FileNotFoundError(2, ..." The code is here https://codeshare.io/5MO9xy How to fix this error?
    c
    1 reply · 2 participants
  • s

    Sque

    11/26/2020, 3:55 PM
    Hi! I am a bit lost in the documentation! Can you point me in a reference on how to update programmatically all pipelines of a project? I find that you can do
    flow.register()
    but what about flows that you want to be removed? How do you remove deprecated flows?
    c
    1 reply · 2 participants
  • r

    Robin

    11/26/2020, 5:06 PM
    Dear prefect community, Is there an (elegant) solution to run only those tasks provided as a list in a (default) parameter? I am having problems applying basic list operations on a parameter in the flow context:
    tasks_to_be_executed = Parameter("tasks_to_be_executed", default=["task_a", "task_c"])
    The following error is returned when checking `if "get_number_of_all_systems" in tasks_to_be_executed`:
    Traceback (most recent call last):
      File ".\awesome_flow.py", line 112, in <module>
        if "get_number_of_all_systems" in tasks_to_be_executed:
    TypeError: argument of type 'Parameter' is not iterable
    c
    s
    12 replies · 3 participants
  • j

    Joseph Finlayson

    11/27/2020, 1:08 AM
    Hello. I hope this is the right place. I'm really having a nightmare with the agent creating jobs on my prefect deployment. the job itself has "ErrImgPull" as it somehow cannot get access to the image.
    │   ----     ------          ----              ----                  -------                                                                 │
    │   Normal   Scheduled       <unknown>         default-scheduler     Successfully assigned prefect/prefect-job-3afb5e32-qx8wd to first-3kwtz │
    │   Normal   Pulling         13s               kubelet, first-3kwtz  Pulling image "<http://registry.digitalocean.com/unmanaged-registry/prefect-job|registry.digitalocean.com/unmanaged-registry/prefect-job> │
    │ s:ac673b8facdc4d18a75da5e8fb98b63b"                                                                                                        │
    │   Warning  Failed          13s               kubelet, first-3kwtz  Failed to pull image "<http://registry.digitalocean.com/unmanaged-registry/pref|registry.digitalocean.com/unmanaged-registry/pref> │
    │ ect-jobs:ac673b8facdc4d18a75da5e8fb98b63b": rpc error: code = Unknown desc = Error response from daemon: Get <https://registry.digitalocean> │
    │ .com/v2/unmanaged-registry/prefect-jobs/manifests/ac673b8facdc4d18a75da5e8fb98b63b: unauthorized: authentication required                  │
    │   Warning  Failed          13s               kubelet, first-3kwtz  Error: ErrImagePull
    Agent YAML looks like this
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: prefect
      name: prefect-agent
      namespace: prefect
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: prefect
      template:
        metadata:
          labels:
            app: prefect
        spec:
          containers:
          - args:
            - prefect agent start kubernetes         command:
            - /bin/bash
            - -c
            env:
            - name: NAMESPACE
              value: prefect
            - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
              value: XXXXX
            - name: PREFECT__CLOUD__API
              value: <https://api.prefect.io>
            - name: IMAGE_PULL_SECRETS
              value: unmanaged-registry
    The generated job itself doesn't seem to have anything set regarding an imagepull secret
    m
    5 replies · 2 participants
  • j

    Jessica Stewart

    11/27/2020, 4:37 AM
    Hi there I have been considering Prefect as it seems like a robust and user friendly tool :) I am curious regarding scalability testing 1. Have you done scalability testing for concurrently running tasks? What procedure did you use? 2. How many tasks were you able to scale up to? What bottlenecks were hit? Thanks so much in advance 🙇
  • j

    Joseph Finlayson

    11/27/2020, 12:51 PM
    When calling
    flow.register
    I'm seeing the following strange stack:
    flow.register(project_name="default")
      File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/core/flow.py", line 1655, in register
        registered_flow = client.register(
      File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/client/client.py", line 771, in register
        serialized_flow = flow.serialize(build=build)  # type: Any
      File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/core/flow.py", line 1462, in serialize
        storage = self.storage.build()  # type: Optional[Storage]
      File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 359, in build
        self._build_image(push=push)
      File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 423, in _build_image
        self._parse_generator_output(output)
      File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 658, in _parse_generator_output
        output = (
    AttributeError: 'NoneType' object has no attribute 'strip'
      `
    👍 1
    🤷‍♂️ 1
    f
    s
    +3
    11 replies · 6 participants
  • s

    Scott Moreland

    11/27/2020, 1:13 PM
    What's the best way to ignore an existing checkpoint and overwrite it?
    c
    m
    +1
    13 replies · 4 participants
  • j

    Joseph Finlayson

    11/27/2020, 1:52 PM
    What's the recommended, simplest way of deploying prefect onto a kubernetes cluster, the docs are in a transitional state between environments and runconfigs.
  • g

    Gabriel Montañola

    11/27/2020, 4:57 PM
    StartFlowRun
    tasks are resulting in failed tasks for me since
    0.13.17
    in Prefect Cloud 😛refect-cloud:
    Unexpected error: ClientError([{'path': ['user'], 'message': 'field "user" not found in type: \'query_root\'', 'extensions': {'path': '$.selectionSet.user', 'code': 'validation-failed', 'exception': {'message': 'field "user" not found in type: \'query_root\''}}}])
    Is this related somehow with https://github.com/PrefectHQ/prefect/pull/3692?
    m
    j
    5 replies · 3 participants
  • a

    Andrey Tatarinov

    11/28/2020, 6:40 PM
    Hi! What would be practical number of tasks to
    map
    ? I have a collection that might vary in size from 500 to 50K, is it reasonable to process it with map?
  • l

    Lior

    11/29/2020, 6:49 AM
    How can I create flows inside of other flows? (a task that runs multiple tasks inside of it?)
    m
    1 reply · 2 participants
Powered by Linen
Title
l

Lior

11/29/2020, 6:49 AM
How can I create flows inside of other flows? (a task that runs multiple tasks inside of it?)
m

Marwan Sarieddine

11/29/2020, 7:07 PM
You’ll probably want to use a flow of flows https://docs.prefect.io/core/idioms/flow-to-flow.html#scheduling-a-flow-of-flows
:upvote: 1
View count: 1