is there a way to test-run a flow locally in Docker, something like

flow.run(run_on_schedule=False)

or using FlowRunner, without using an agent?

Hey team, Prefect version 

0.13.17

 has been released and here are a few notable changes:   😛refect-duck:😛refect-duck: Nicer support for tasks with multiple results   ⚠️   Improved logs and warnings   ♻️   Cleaner flow-run restarts when using resource managers #️⃣ Squashed bug in flow-hashing for idempotency A big thank you to our contributors who helped out with this release! Full changelog: https://github.com/PrefectHQ/prefect/releases/tag/0.13.17

Hello Eveveryone! I am using a CI/CD process with AWS Codepipeline (source: Github, build: Codebuild) the output is a docker image i push to an AWS ECR repository for each flow. I my github project i have a buildspec file where i put this code:

build:
    commands:
      - python --version
      - pyenv global 3.7.7
      - python --version
      - pip install $(grep -ivE "pywin32" requirements.txt)
      - prefect backend cloud
      - prefect auth login -t $PREFECT_ACCESS_TOKEN
      - aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin <http://xxxxxxxx.dkr.ecr.us-east-2.amazonaws.com|xxxxxxxx.dkr.ecr.us-east-2.amazonaws.com>
      - python3.7 flows/my_flow.py

From November 20 docker introduced pull limits for images what cause me to have this error in Code build

File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 619, in pull_image
    output = client.pull(self.base_image, stream=True, decode=True)
  File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/docker/api/image.py", line 415, in pull
    self._raise_for_status(response)
  File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/docker/api/client.py", line 261, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: <https://www.docker.com/increase-rate-limit>")

When i go to the site i can read: "Anonymous free users will be limited to 100 pulls per six hours, and authenticated free users will be limited to 200 pulls per six hours." So i created a account on docker to raise my limits to 200 pull per 6 hours. So i understand the problem occurs when prefect is pulling the base prefect docker image to build my flow image because i have reached my 100 pulls / 6 hours anonymous limit My question is where should i put my docker account token to pull the prefect base image via my newly created docker account?

message has been deleted

hello Prefect Support, I have a flow run that, right after it started the first task

init_run

,

Task 'init_run': Starting task run...

it got this error message ( see thread) and, the log message said

Task 'init_run': Finished task run for task with final state: 'ClientFailed'

After that, All remaining tasks stuck in

Pending

state for hours. Any ideas what happened? Maybe my flow need to know how to handle this

ClientFailed

state ?

Hi all, been using prefect for a while now, and big fan of it. I have the following use case and am not sure if it's supported by prefect. I have a flow that first downloads quite a large file and then processes it using a GPU. I would like to set this up in a parallel cluster of some sort. However, I wonder how can I ensure that the file is downloaded and processed on the same node. To my understanding, a task can be executed on any machine if it is ready to be processed and only waiting for the upstream tasks to finish. Meaning, it might happen that the file downloaded on one machine, the gpu processing is waiting but might start on another machine and then can obviously not find the file. One workaround would be to make this one big task and request a gpu for it, but then if the download takes quite long the GPU could have been utilised elsewhere. Are there any other options. Basically enforcing that two tasks are executed on any yet the same machine? Thank you

Hey, I am looking to persist to local storage tasks that output pandas dataframes, do you have a relevant example?

Any uptime issues at the moment? The Prefect Cloud UI is not loading anything and I see errors in the console like

Access to fetch at '<https://api.prefect.io/graphql>' from origin '<https://cloud.prefect.io>' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled

Hi prefect team! I have an issue when registering a flow using docker storage. I have a Github Action that downloads the latest version of prefect and register the new flow when the branch changes. I haven't introduced any change since October 30th using Prefect Version 0.13.13. Today when I tried to update my flow this error has appeared. I've tried to do the same from my computer (using version 0.13.13) and it works fine. In our team we follow the update announcements of new versions of Prefect but we haven't seen anything that could affect to our flow. Do you have any idea of what the problem could be? We are using the following command to register if it can help to find the problem:

flow.register(project_name=env_label, add_default_labels=False)

We use also the

flow.storage = Docker(...)

command. Thanks for your time, and sorry if it is an obvious question 😅

Hi, it seems that when cancelling a flow run when there are tasks within a resource manager segment, the resource manager cleanup is not called, is that intentional (or can be enabled somewhere)?

Hello, We are currently looking at shelltask and dbtshelltask and we are wondering why helper_script is only in the constructor and why it cannot set when we call run. There must be a reason for this. If someone could shed some light on this. Thanks Peter

Hello, In the documentation it show that when a flow is in pause state that it will show up in the dashboards notifications. We are not seeing this, Is there some setting we need to change or enable notification in our flow? here is where we see it. https://docs.prefect.io/core/idioms/pause-for-approval.html#pause-for-approval Thanks

Hi everyone - This question pertains to the helm chart for deploying Prefect Server on k8s. I'm trying to use an external database as was recommended in the docs. However, I am dealing with an issue when trying to connect to a postgres cloudsql instance on GCP.

Hi everyone - How to convert prefect python dict parameter to JSON string? I am looking at JSONCompatible. Please point me to some examples.

Heya - can anyone point me in the right direction/documentation/example if I wanted to register a flow on Prefect Cloud by pushing changes to a repo in Azure DevOps?

Hi guys. is there any way to doing the integration testing with prefect flow? I mean how can I doing integration testing of my docker based flow?

Is there a way to stop a mapping task if one of the tasks in the map failed? (stop all the other tasks). It's possible with dask Variable

Hi guys, I am having an issue with one of my flows not terminating due to a single mapped task ("mapped child 1") being stuck in pending. When checking the logs for that mapped task, I found what seems to be an internal server error (see thread). I think there have been one or two flows in the past, with the same kind of issue (one mapped task stuck in pending), but I am not able to retrieve their logs so I can't be sure. For the old runs, subsquent re-runs finished successfully, but this time the re-run seems to run into the same issue (+ same exception in logs). Is that exception the cause for the problem? Is the issue on prefect's side or on mine?

Hi everyone. I recall that it was once possible to store parameter values to the Prefect DB. I can't find a reference in the docs, though. Did I recall wrong? Should I look harder? 🙂 Thanks for your help!

Hi everyone, I've just started to use prefect and have created a simple test flow. It's working good localy, but when I added agent, I have an error: "ERROR - agent | Error while deploying flow: FileNotFoundError(2, ..." The code is here https://codeshare.io/5MO9xy How to fix this error?

Hi! I am a bit lost in the documentation! Can you point me in a reference on how to update programmatically all pipelines of a project? I find that you can do

flow.register()

but what about flows that you want to be removed? How do you remove deprecated flows?

Dear prefect community, Is there an (elegant) solution to run only those tasks provided as a list in a (default) parameter? I am having problems applying basic list operations on a parameter in the flow context:

tasks_to_be_executed = Parameter("tasks_to_be_executed", default=["task_a", "task_c"])

The following error is returned when checking `if "get_number_of_all_systems" in tasks_to_be_executed`:

Traceback (most recent call last):
  File ".\awesome_flow.py", line 112, in <module>
    if "get_number_of_all_systems" in tasks_to_be_executed:
TypeError: argument of type 'Parameter' is not iterable

Hello. I hope this is the right place. I'm really having a nightmare with the agent creating jobs on my prefect deployment. the job itself has "ErrImgPull" as it somehow cannot get access to the image.

│   ----     ------          ----              ----                  -------                                                                 │
│   Normal   Scheduled       <unknown>         default-scheduler     Successfully assigned prefect/prefect-job-3afb5e32-qx8wd to first-3kwtz │
│   Normal   Pulling         13s               kubelet, first-3kwtz  Pulling image "<http://registry.digitalocean.com/unmanaged-registry/prefect-job|registry.digitalocean.com/unmanaged-registry/prefect-job> │
│ s:ac673b8facdc4d18a75da5e8fb98b63b"                                                                                                        │
│   Warning  Failed          13s               kubelet, first-3kwtz  Failed to pull image "<http://registry.digitalocean.com/unmanaged-registry/pref|registry.digitalocean.com/unmanaged-registry/pref> │
│ ect-jobs:ac673b8facdc4d18a75da5e8fb98b63b": rpc error: code = Unknown desc = Error response from daemon: Get <https://registry.digitalocean> │
│ .com/v2/unmanaged-registry/prefect-jobs/manifests/ac673b8facdc4d18a75da5e8fb98b63b: unauthorized: authentication required                  │
│   Warning  Failed          13s               kubelet, first-3kwtz  Error: ErrImagePull

Agent YAML looks like this

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prefect
  name: prefect-agent
  namespace: prefect
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect
  template:
    metadata:
      labels:
        app: prefect
    spec:
      containers:
      - args:
        - prefect agent start kubernetes         command:
        - /bin/bash
        - -c
        env:
        - name: NAMESPACE
          value: prefect
        - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
          value: XXXXX
        - name: PREFECT__CLOUD__API
          value: <https://api.prefect.io>
        - name: IMAGE_PULL_SECRETS
          value: unmanaged-registry

The generated job itself doesn't seem to have anything set regarding an imagepull secret

Hi there I have been considering Prefect as it seems like a robust and user friendly tool :) I am curious regarding scalability testing 1. Have you done scalability testing for concurrently running tasks? What procedure did you use? 2. How many tasks were you able to scale up to? What bottlenecks were hit? Thanks so much in advance 🙇

When calling

flow.register

I'm seeing the following strange stack:

flow.register(project_name="default")
  File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/core/flow.py", line 1655, in register
    registered_flow = client.register(
  File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/client/client.py", line 771, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/core/flow.py", line 1462, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 359, in build
    self._build_image(push=push)
  File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 423, in _build_image
    self._parse_generator_output(output)
  File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 658, in _parse_generator_output
    output = (
AttributeError: 'NoneType' object has no attribute 'strip'
  `

What's the best way to ignore an existing checkpoint and overwrite it?

What's the recommended, simplest way of deploying prefect onto a kubernetes cluster, the docs are in a transitional state between environments and runconfigs.

StartFlowRun

tasks are resulting in failed tasks for me since

0.13.17

in Prefect Cloud 😛refect-cloud:

Unexpected error: ClientError([{'path': ['user'], 'message': 'field "user" not found in type: \'query_root\'', 'extensions': {'path': '$.selectionSet.user', 'code': 'validation-failed', 'exception': {'message': 'field "user" not found in type: \'query_root\''}}}])

Is this related somehow with https://github.com/PrefectHQ/prefect/pull/3692?

Hi! What would be practical number of tasks to

map

? I have a collection that might vary in size from 500 to 50K, is it reasonable to process it with map?