is there some scope problem here?

Hello! I’m seeing an issue where mapped tasks are limited to 2 in Prefect Cloud but are unlimited locally. Is this a standard plan limitation?

I'm trying to find the

Secrets page

in the cloud UI: I know I've been there, but I can't for the life of me find my way back there... Where is it? Could someone provide a link? Google says nothing and the docs link back to itself.

If I may point out a design inconsistency: that parameters and secrets sometimes are "values as declaration time" and sometimes "values as output by tasks" — this looks like an anti-pattern to me:

When trying to set parameters in the Cloud UI, the parameters input textarea alternates between the existing value, an empty object after I start to change it (back and forth making it impossible to insert anything there)

The docs aren't reflecting the latest release, I just noticed. https://github.com/PrefectHQ/prefect/releases

I have been looking at the Executor options, and there is a DaskExecutor

executor = DaskExecutor(
    cluster_class="dask_cloudprovider.FargateCluster",
    cluster_kwargs={
        "image": "prefecthq/prefect:latest",
        "n_workers": 5,
        ...
    },
)

Does it mean that Prefect can also create a Dask cluster when there is some flows to run, and tear down the Dask cluster on Fargate once all flows have been run? I already have a setup using ECSAgent. Does this executor mean that I don’t need that ECS agent any longer?

I mainly use Prefect for forecasting. An important feature that I am looking for is to pass data (large pandas dataframe of about 1.5GB in csv, or numpy array) between tasks. Does Prefect already support serialising significant data between tasks? I saw something called PandasSerializer, but not sure what it is for.

Hi i am going through the tutorials and https://docs.prefect.io/core/examples/mapping.html

from prefect import task

@task
def add(x, y):
    return x + y

add(1, 2)  # 3

this example throws me:

ValueError: Could not infer an active Flow context while creating edge

and suggests using add.run() instead. In the tutorial however, they use add(1,2) regularly. Are the tutorials outdated for the newest version of prefect? edit: I am also unable to use .map as it is used in the tutorial:

add.map([1, 10], [2, 3])

throws the same error? How am I supposed to experiment with tasks without flow context?

One more question, does prefect support looping of a subflow? that a part of the flow is repeated? It seems loop only works on a single task, what I have in mind is that I could loop through a chain of tasks.

We're seeing duplicate flow runs scheduled in Prefect Cloud. This flow id, for example, is currently duplicated: 5842f8d7-2f86-4c92-a291-3fcc87e6fbf7

Hi there. I am trying to use docker storage with a custom Dockerfile that is based on an image that contains a conda environment. I am having an issue with the health checks failing, but the flow itself runs. Can you tell if I am doing something wrong? More details in thread.

Hi. Any issues with using poetry for dependency management (for Server + Agent on a single node)? Here and there there seems to have been issues, but besides potentially this, I can't see any reason to not use poetry?

Hey folks is anyone else getting a deprecation warning on marshmallow? Something like:

...
/home/rof/.pyenv/versions/3.9.1/lib/python3.9/site-packages/marshmallow/fields.py:198
/home/rof/.pyenv/versions/3.9.1/lib/python3.9/site-packages/marshmallow/fields.py:198: RemovedInMarshmallow4Warning: Passing field metadata as a keyword arg is deprecated. Use the explicit `metadata=...` argument instead.
...

Tried to update to the latest version of prefect and still no joy!

Hello prefect, I'm trying to use secrets which are saved in the kubernetes. To access them, I'm using KubernetesRun agent parameter:

job_template_path

where there is a path to yaml file which is supposed to create environment variables and used them in the prefect code:

apiVersion: v1
kind: Pod
metadata:
  name: secret-test-pod
spec:
  containers:
    - name: test-container
      image: <http://k8s.gcr.io/busybox|k8s.gcr.io/busybox>
      command: [ "/bin/sh", "-c", "env" ]
      envFrom:
      - secretRef:
          name: mysecret
  restartPolicy: Never

Yaml file template is took from kubernetes documentation(https://kubernetes.io/docs/concepts/configuration/secret/#use-cases) However, when I try to run it on prefect server, I got this error:

"'spec'"

(will attach an error log screenshot) Is there are anything wrong with the template? Is this approach is a good way to define and use secrets in the kubernetes agent? P.S I tried to define

env

parameter with secrets by using KubernetesRun agent but it is not a good option for this case since all secrets could be visible as a plain text in the prefect server UI.

Hi everyone. I am reposting this message from Sunday which may be overlooked. Thanks for your help!

When i specify a

job_template

in my

KubernetesRun

config, I’m stilling seeing default set in the UI. I believe previously I would see the template there. Anyone know what is going on? We are running prefect cloud and my core version is

0.14.19

I have a mapped task that needs to wait until a previous mapped task is 100% complete before starting. If I add upstream_tasks=[fp_validation_predicted_locations] it kicks off task 1 when task 1 of fp_validation_predicted_locations finishes. How can I have it wait until they are all finished? Can I do upstream_tasks=[unmapped(fp_validation_predicted_locations)]?

does anyone have an example of using "on_failure" in a flow argument to re-run the whole flow after a certain period? I see the examples for specific tasks, would it work the same way for a flow I'm guessing?

Editing schedules isn't loading the schedule information in Cloud.

I am trying out Prefect cloud Enterprise and the account page shows I can add 100 users but when I try to add more members I get

Your team has no users available; upgrade your plan to get more users and access to more features!

Any ideas?

Hi Everyone, I'm looking at switching to Prefect, but I have a question: I currently have a scheduling system that is organized by different "jobs". These jobs are essentially python classes that can be instantiated with varying parameters. For example, I have a job class that moves files from an SFTP to a local directory and I create multiple instances of it through the UI and specify the parameters through the UI as well without creating additional code. Is it possible to do something similar with Prefect? Part of the advantage of the above system is that I can have non-technical people add jobs without adding any code or doing a release. Perhaps I'm not thinking about Prefect's tasks and flows correctly, but any hep or clarity would be appreciated.

Is there a way to remove the local label added by

flow.register()

function? I notice that if I use

prefect register

command line tool, the label is exactly what I provided, but if I use python code, it will be added. I read the doc and it says this is the desired behavior, wonder if there is a way to get it removed?

One more question, in the docs

A result subclass that matches the storage backend of your 

prefect.config.flows.storage

 setting will automatically be applied to all tasks, if available; notably this is not yet supported for Docker Storage

Does this mean we can not use checkpointing if using Docker Storage?

Hey folks, we are trying to move our task execution from ECS to EKS but running into this error as the agent tries to roll out pods for the job:

TypeError: '<' not supported between instances of 'NoneType' and 'datetime.datetime'
[2021-05-17 23:14:44,507] ERROR - agent | Error while managing existing k8s jobs
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 383, in heartbeat
    self.manage_jobs()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 236, in manage_jobs
    pod_name
TypeError: '<' not supported between instances of 'NoneType' and 'datetime.datetime'

we’re using the config generated from the prefect cli for kubernetes agent. Any idea on how to work resolve it?

sorry I know that's a lot, maybe add it to the task example library?

Hi some of our scheduled tasks have started failing with

[Errno 11] Resource temporarily unavailable

any hints on what could be the underlying issue?

Hey folks, we use flows in conjunction with parameterized schedules A LOT and I was wondering if it’s possible to alter the name of a Flow Run to contain some identifier based on Flow parameters? So for example if one of the flow parameters is a table name, is it possible to have the flow name be something like

"my_table-intrepid-iguana"

or something like that. This would make it easier to navigate the flow runs in the Cloud UI.

I'm using Prefect Cloud and have a problem with a flow that takes a long time to run (a lot of heavy processing). Is there a way to send a heartbeat (in my code) to ensure that the flow is not marked as Failed ?

Hello guys, I would like to ask if anyone have an example how to mount CephFS with prefect flow? For now I'm trying to use

job_template_path

parameter in

KubernetesRun

by passing yaml file which looks like this:

apiVersion: batch/v1
kind: Job
metadata:
  name: prefect-job
  labels: {}
spec:
  template:
    metadata:
      labels: {}
    spec:
      containers:
        - name: flow
          image: test-image:1.2
          imagePullPolicy: IfNotPresent
          command: ["/bin/sh", "-c"]
          args: ["prefect execute flow-run"]
          env:
            - name: PREFECT__CONTEXT__SECRETS__database_psw
              value: veryStrongPasswrod
          volumeMounts:
            - name: storage
              mountPath: /tnt/test/domantas_test
      restartPolicy: Never
      imagePullSecrets:
      - name: registry-secret
      volumes:
        - name: storage
          cephfs:
            monitors:
                - 59.11.129.131
                - 59.11.129.132
                - 59.11.129.133
                - 59.11.129.134
                - 59.11.129.135
            path: /test
            user: storage
            secretRef:
                name: ceph-secret

However, when I try to execute in prefect server UI, it seems the flow get stuck with "Submitted for execution" status with no useful logs. I took job_template example from here: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/kubernetes/job_spec.yaml and additionally added cephFS necessary parameters.