Hello Eveveryone! I am using a CI/CD process with AWS Codepipeline (source: Github, build: Codebuild) the output is a docker image i push to an AWS ECR repository for each flow. I my github project i have a buildspec file where i put this code:

build:
    commands:
      - python --version
      - pyenv global 3.7.7
      - python --version
      - pip install $(grep -ivE "pywin32" requirements.txt)
      - prefect backend cloud
      - prefect auth login -t $PREFECT_ACCESS_TOKEN
      - aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin <http://xxxxxxxx.dkr.ecr.us-east-2.amazonaws.com|xxxxxxxx.dkr.ecr.us-east-2.amazonaws.com>
      - python3.7 flows/my_flow.py

From November 20 docker introduced pull limits for images what cause me to have this error in Code build

File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 619, in pull_image
    output = client.pull(self.base_image, stream=True, decode=True)
  File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/docker/api/image.py", line 415, in pull
    self._raise_for_status(response)
  File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/docker/api/client.py", line 261, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: <https://www.docker.com/increase-rate-limit>")

When i go to the site i can read: "Anonymous free users will be limited to 100 pulls per six hours, and authenticated free users will be limited to 200 pulls per six hours." So i created a account on docker to raise my limits to 200 pull per 6 hours. So i understand the problem occurs when prefect is pulling the base prefect docker image to build my flow image because i have reached my 100 pulls / 6 hours anonymous limit My question is where should i put my docker account token to pull the prefect base image via my newly created docker account?

hello Prefect Support, I have a flow run that, right after it started the first task

init_run

,

Task 'init_run': Starting task run...

it got this error message ( see thread) and, the log message said

Task 'init_run': Finished task run for task with final state: 'ClientFailed'

After that, All remaining tasks stuck in

Pending

state for hours. Any ideas what happened? Maybe my flow need to know how to handle this

ClientFailed

state ?

Hi all, been using prefect for a while now, and big fan of it. I have the following use case and am not sure if it's supported by prefect. I have a flow that first downloads quite a large file and then processes it using a GPU. I would like to set this up in a parallel cluster of some sort. However, I wonder how can I ensure that the file is downloaded and processed on the same node. To my understanding, a task can be executed on any machine if it is ready to be processed and only waiting for the upstream tasks to finish. Meaning, it might happen that the file downloaded on one machine, the gpu processing is waiting but might start on another machine and then can obviously not find the file. One workaround would be to make this one big task and request a gpu for it, but then if the download takes quite long the GPU could have been utilised elsewhere. Are there any other options. Basically enforcing that two tasks are executed on any yet the same machine? Thank you

Hey, I am looking to persist to local storage tasks that output pandas dataframes, do you have a relevant example?

Any uptime issues at the moment? The Prefect Cloud UI is not loading anything and I see errors in the console like

Access to fetch at '<https://api.prefect.io/graphql>' from origin '<https://cloud.prefect.io>' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled

Hi prefect team! I have an issue when registering a flow using docker storage. I have a Github Action that downloads the latest version of prefect and register the new flow when the branch changes. I haven't introduced any change since October 30th using Prefect Version 0.13.13. Today when I tried to update my flow this error has appeared. I've tried to do the same from my computer (using version 0.13.13) and it works fine. In our team we follow the update announcements of new versions of Prefect but we haven't seen anything that could affect to our flow. Do you have any idea of what the problem could be? We are using the following command to register if it can help to find the problem:

flow.register(project_name=env_label, add_default_labels=False)

We use also the

flow.storage = Docker(...)

command. Thanks for your time, and sorry if it is an obvious question 😅

Hi, it seems that when cancelling a flow run when there are tasks within a resource manager segment, the resource manager cleanup is not called, is that intentional (or can be enabled somewhere)?

Hello, We are currently looking at shelltask and dbtshelltask and we are wondering why helper_script is only in the constructor and why it cannot set when we call run. There must be a reason for this. If someone could shed some light on this. Thanks Peter

Hello, In the documentation it show that when a flow is in pause state that it will show up in the dashboards notifications. We are not seeing this, Is there some setting we need to change or enable notification in our flow? here is where we see it. https://docs.prefect.io/core/idioms/pause-for-approval.html#pause-for-approval Thanks

Hi everyone - This question pertains to the helm chart for deploying Prefect Server on k8s. I'm trying to use an external database as was recommended in the docs. However, I am dealing with an issue when trying to connect to a postgres cloudsql instance on GCP.

Hi everyone - How to convert prefect python dict parameter to JSON string? I am looking at JSONCompatible. Please point me to some examples.

Heya - can anyone point me in the right direction/documentation/example if I wanted to register a flow on Prefect Cloud by pushing changes to a repo in Azure DevOps?

Hi guys. is there any way to doing the integration testing with prefect flow? I mean how can I doing integration testing of my docker based flow?

Is there a way to stop a mapping task if one of the tasks in the map failed? (stop all the other tasks). It's possible with dask Variable

Hi guys, I am having an issue with one of my flows not terminating due to a single mapped task ("mapped child 1") being stuck in pending. When checking the logs for that mapped task, I found what seems to be an internal server error (see thread). I think there have been one or two flows in the past, with the same kind of issue (one mapped task stuck in pending), but I am not able to retrieve their logs so I can't be sure. For the old runs, subsquent re-runs finished successfully, but this time the re-run seems to run into the same issue (+ same exception in logs). Is that exception the cause for the problem? Is the issue on prefect's side or on mine?

Hi everyone. I recall that it was once possible to store parameter values to the Prefect DB. I can't find a reference in the docs, though. Did I recall wrong? Should I look harder? 🙂 Thanks for your help!

Hi everyone, I've just started to use prefect and have created a simple test flow. It's working good localy, but when I added agent, I have an error: "ERROR - agent | Error while deploying flow: FileNotFoundError(2, ..." The code is here https://codeshare.io/5MO9xy How to fix this error?

Hi! I am a bit lost in the documentation! Can you point me in a reference on how to update programmatically all pipelines of a project? I find that you can do

flow.register()

but what about flows that you want to be removed? How do you remove deprecated flows?

Dear prefect community, Is there an (elegant) solution to run only those tasks provided as a list in a (default) parameter? I am having problems applying basic list operations on a parameter in the flow context:

tasks_to_be_executed = Parameter("tasks_to_be_executed", default=["task_a", "task_c"])

The following error is returned when checking `if "get_number_of_all_systems" in tasks_to_be_executed`:

Traceback (most recent call last):
  File ".\awesome_flow.py", line 112, in <module>
    if "get_number_of_all_systems" in tasks_to_be_executed:
TypeError: argument of type 'Parameter' is not iterable

Hello. I hope this is the right place. I'm really having a nightmare with the agent creating jobs on my prefect deployment. the job itself has "ErrImgPull" as it somehow cannot get access to the image.

│   ----     ------          ----              ----                  -------                                                                 │
│   Normal   Scheduled       <unknown>         default-scheduler     Successfully assigned prefect/prefect-job-3afb5e32-qx8wd to first-3kwtz │
│   Normal   Pulling         13s               kubelet, first-3kwtz  Pulling image "<http://registry.digitalocean.com/unmanaged-registry/prefect-job|registry.digitalocean.com/unmanaged-registry/prefect-job> │
│ s:ac673b8facdc4d18a75da5e8fb98b63b"                                                                                                        │
│   Warning  Failed          13s               kubelet, first-3kwtz  Failed to pull image "<http://registry.digitalocean.com/unmanaged-registry/pref|registry.digitalocean.com/unmanaged-registry/pref> │
│ ect-jobs:ac673b8facdc4d18a75da5e8fb98b63b": rpc error: code = Unknown desc = Error response from daemon: Get <https://registry.digitalocean> │
│ .com/v2/unmanaged-registry/prefect-jobs/manifests/ac673b8facdc4d18a75da5e8fb98b63b: unauthorized: authentication required                  │
│   Warning  Failed          13s               kubelet, first-3kwtz  Error: ErrImagePull

Agent YAML looks like this

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prefect
  name: prefect-agent
  namespace: prefect
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect
  template:
    metadata:
      labels:
        app: prefect
    spec:
      containers:
      - args:
        - prefect agent start kubernetes         command:
        - /bin/bash
        - -c
        env:
        - name: NAMESPACE
          value: prefect
        - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
          value: XXXXX
        - name: PREFECT__CLOUD__API
          value: <https://api.prefect.io>
        - name: IMAGE_PULL_SECRETS
          value: unmanaged-registry

The generated job itself doesn't seem to have anything set regarding an imagepull secret

Hi there I have been considering Prefect as it seems like a robust and user friendly tool :) I am curious regarding scalability testing 1. Have you done scalability testing for concurrently running tasks? What procedure did you use? 2. How many tasks were you able to scale up to? What bottlenecks were hit? Thanks so much in advance 🙇

When calling

flow.register

I'm seeing the following strange stack:

flow.register(project_name="default")
  File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/core/flow.py", line 1655, in register
    registered_flow = client.register(
  File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/client/client.py", line 771, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/core/flow.py", line 1462, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 359, in build
    self._build_image(push=push)
  File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 423, in _build_image
    self._parse_generator_output(output)
  File "/home/joe/.local/share/virtualenvs/flows/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 658, in _parse_generator_output
    output = (
AttributeError: 'NoneType' object has no attribute 'strip'
  `

What's the best way to ignore an existing checkpoint and overwrite it?

What's the recommended, simplest way of deploying prefect onto a kubernetes cluster, the docs are in a transitional state between environments and runconfigs.

StartFlowRun

tasks are resulting in failed tasks for me since

0.13.17

in Prefect Cloud prefect cloud

Unexpected error: ClientError([{'path': ['user'], 'message': 'field "user" not found in type: \'query_root\'', 'extensions': {'path': '$.selectionSet.user', 'code': 'validation-failed', 'exception': {'message': 'field "user" not found in type: \'query_root\''}}}])

Is this related somehow with https://github.com/PrefectHQ/prefect/pull/3692?

Hi! What would be practical number of tasks to

map

? I have a collection that might vary in size from 500 to 50K, is it reasonable to process it with map?

How can I create flows inside of other flows? (a task that runs multiple tasks inside of it?)

Hi, I want to run a flow on a remote dask-worker using the

dask-resource

in

tags

as argument for

@task

:

running_server = "name_of_remote_server"
resource_tag = "dask-resource:{}=1".format(running_server)
@task(log_stdout=True, state_handlers=[email_on_failure], tags=[resource_tag])
def test_task():
    print(1/0)

When I started the remote dask-worker, I used command like this:

dask-worker tcp://<address of local dask-scheduler>:8786 --nprocs 4 --nthreads 1 --worker-port xxx --resources "name_of_remote_server=1"

This connected local scheduler with remote worker perfectly. However, when I actually started the flow on local machine, it failed running on the remote worker first because of

ModuleNotFoundError: No module named 'prefect'

. After I installed prefect for the remote worker, another error occurred on it:

[2020-11-24 17:02:05+0800] ERROR - prefect.CloudTaskRunner | Failed to set task state with error: ConnectionError(MaxRetryError("HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff6099fcf10>: Failed to establish a new connection: [Errno 111] Connection refused'))"))
Traceback (most recent call last):
  File "/anaconda3_a/envs/py37/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/anaconda3_a/envs/py37/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/anaconda3_a/envs/py37/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
...
2020-11-24 17:02:05+0800] INFO - prefect.CloudTaskRunner | Task 'test_task': Finished task run for task with final state: 'ClientFailed'
[2020-11-24 09:02:10+0000] CRITICAL - CloudHandler | Failed to write log with error: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff6097590d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2020-11-24 09:02:10+0000] CRITICAL - CloudHandler | Failed to write log with error: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff6097590d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2020-11-24 09:02:10+0000] CRITICAL - CloudHandler | Failed to write log with error: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff6097590d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2020-11-24 09:02:10+0000] CRITICAL - CloudHandler | Unable to write logs to Prefect Cloud
[2020-11-24 09:02:10+0000] CRITICAL - CloudHandler | Unable to write logs to Prefect Cloud
[2020-11-24 09:02:10+0000] CRITICAL - CloudHandler | Unable to write logs to Prefect Cloud

I tried opening port 4200 of local scheduler for the remote worker, but the same error occurred anyway. Wondering what it is that I’ve been doing wrong? 😶

Hey all - Not sure if this is the right channel, but what the heck 🙂 I would love to contribute to Prefect Task library. Specifically, I want to add a task that fetches data from Dremio. Now I haven’t noticed anything related to Dremio on GitHub and here on Slack, so I went and implemented a first draft of a solution on my fork of prefect. This is my very first try in contributing to open source and as you will be able to notice from the code, my programming experiences are limited. Nevertheless, I am enthusiastic to contribute 🤓 Of course I did my best to follow the docs (Contributing). And here is my question: what should be my next step to move this forward? Should I open a PR and get feedback there or are there other steps to be taken beforehand?

Hi folks, I have a question around the hybrid model. As mentioned in the hybrid model - medium blog, once I registered my flow on the Prefect Cloud (step 2), I could run an agent on my infra (step 3). That is, the Prefect Cloud and the agent (using docker agent) are two different machines. And I set the same env as mentioned above. The question is, in the agent machine, I use the command “`prefect agent docker start --api http://<Prefect Cloud IP>:4200` (backend: server) and the agent will query the prefect core in my cloud server. But I change backend to cloud, using “`prefect agent docker start -t <MY_TOKEN> --api http://<Prefect Cloud IP>:4200`” will show the following message:

prefect.utilities.exceptions.ClientError: 400 Client Error: Bad Request for url: http://<Prefect Cloud IP>:4200

This is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:

query {
  query { auth_info { api_token_scope } }
}
variables {
  null
}

Is there any missing step when I set Prefect agent? Thank you very much!