How do you change your account username and email address in Prefect Cloud?

I'm experiencing a strange issue with version 2.1.1 where a scheduled flow run will try to run a task on all three of my Kubernetes pods simultaneously. This results in one of the tasks succeeding and two of them failing. Each pod is running on its own node. I am using SequentialTaskRunner with Prefect Shell Task and Process Infrastructure. I do not want to use Kubernetes Infrastructure because I want control over using my own kubectl commands. I'd prefer to spin up Prefect Agent pods like I would with any other process I run in Kubernetes. When I manually run the flow through the UI, it works perfectly and only schedules the task to one pod in the cluster. When I auto-schedule the flow run, it attempts to run the task on all pods in the cluster simultaneously despite them running on their own nodes. The UI thinks it is only running one task which can be seen in the screenshots. Anyone know why this might be?

Is there a way to declare docker image in the prefect deployment build command?

Hi! I was curious if there was any good docs or known methods for creating rotating secrets, where behind the scenes we are updating the value of a secrets

Hi all, I am trying to run a deployment locally on my M1 Mac. However, I am continuously getting FileNotFoundError: [Errno 2] No such file or directory error. I tried to look around for a solution in the channels and discourses but I haven’t found the same problem just some similar ones. I am trying to run with the prefect cli and I am in my flows folder.

prefect deployment build mssql_dbx_flow.py:dbx_trigger_sync_flow -n airbyte-prefect-dbx-elt -t testing

I have created an agent already and I’m using prefect 2.1.1. I’ve attached my orion settings, as well. Could you advise me on what am I missing?

This message was deleted.

Quick question. I'm going to be building a complex data processing workflow in AWS and we plan to use our own Prefect installation for this. I'm trying to determine if it should use AWS EKS (kubernetes) or AWS ECS (elastic container service). We plan to dockerize most of our worker nodes and call those from Prefect (they'll be running on windows and need custom installed software often, not just raw python, so we need to use EC2 instances not Fargate). Would anyone have a recommendation for going with EKS or ECS for a use case like this? My assumption is that there will be more community support when using Prefect with EKS as that's a more commonly used open source solution? Just want to go with whatever makes our lives the easiest.

Am I putting the default label flag in the wrong place? I am getting this error:

TypeError: __init__() got an unexpected keyword argument 'add_default_labels'

My configuration looks like this:

# Configure a custom image
f.run_config = DockerRun(
    labels=["grls"],
    add_default_labels=False,
)

Q from @Chris Gunderson - Hi Prefect community - I'm having an issue with the parameters used in a task 🧵

is it possible to pass result data between prefect tasks that are run on different agents? Or does this require stashing the intermediate data in something like a file on S3

Hello, quick question, I am using

trigger=any_failed

in my task. But I really only want to do something if the previous task has failed, and not consider all upstream tasks. Can someone point me in the right direction?

Hello. I tried to deploy this locally but get an exception.

- name_input: field required

I see that in the do not edit below line part of the deployment.yaml it has name_input set to required.

from prefect import flow, task
from prefect.tasks import task_input_hash
from datetime import timedelta

@task(cache_key_fn=task_input_hash, cache_expiration=timedelta(minutes=1))
def hello_task(name_input):
    # Doing some work
    print(f"Saying hello {name_input}")
    return "hello " + name_input

@flow
def hello_flow(name_input):
    hello_task(name_input)

if __name__ == "__main__":
    hello_flow("Marvin")

Trying to get the hang of prefect 2 for future transition, however I have the following issues After running the file a had a runtime error:”” “Orion requires sqlite>=3.24.0 but we found version 3.22.0” Secondly how can I effect schedule from local file/environment

Hello. Working on the migration to 2.0, I can’t seem to find any documentation for how to migrate Results storage. In 1.0 I had an S3 bucket with a short lifecycle policy to store results/cache data between runs, but I’m not seeing where I would set that up for 2.0.

from prefect import task, Parameter, Flow
from prefect.storage import S3
import prefect


@task
def main_task():
    print("foobar")
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("foobar")


storage = S3(bucket="prefect-task-behaviour", stored_as_script=True, local_script_path="main.py")

def build_flow():
    with Flow("s3-to-prefect-flow", storage=storage) as building_flow:
        main_task()
    return building_flow


print(build_flow())

I have an S3 bucket with a copy of this code (named main.py) in it, yet I am not seeing it print out foobar, why is that?

In prefect 2.0 what is the migration path for running workflows on a simple interval schedule? I tried creating a deployment yaml and including a schedule like this

schedule:
  interval: 75
  timezone:

However the flow doesn't actually run and I see a bunch of

Scheduled

flow runs that eventually become

Late

Hello! I am trying to run a simple flow using a DockerContainer infrastructue and minio storage block. When the deployment is created, the flow is uploaded to minio but when I run the flow I get an error. Any ideas? Flow and error in thread

Hey all, I’m kind of new of Prefect, and I have the Kubernetes agent installed and wired up to my Prefect cloud 2.0 tenant on a local cluster as a POC. Can anyone point me in the direction of a docs page that can get me started with triggering a flow on my k8s cluster using the Agent? I may be missing it but I haven’t been able to find anything in the docs that gives a succinct end to end example and this page seems to just give me back a 404. Do I just need to define a Deployment and specify that it be run in Kubernetes somehow?

hullo! i'm attempting to build a flow that kicks off subflows asynchronously in Prefect 2.0, and seeing an error that i'm a bit stumped on:

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked

details in the thread!

I am having an issue with the prefect client/accessing flow runs. When I run

import prefect
pc = prefect.get_client()
await pc.read_flow_runs()

it returns a cloudpickle error

TypeError: __init__() missing 1 required positional argument: 'path'

(see comments for full stack trace). Previously I was using python 3.7 and was getting another cloudpickle error about version 5 not working. Other prefect client functions work (including

read_flow_run

and

read_flow_run_states

) version stuff:

$ prefect version
Version:             2.1.1
API version:         0.8.0
Python version:      3.9.12
Git commit:          dc2ba222
Built:               Thu, Aug 18, 2022 10:18 AM
OS/Arch:             darwin/x86_64
Profile:             goodkiwi
Server type:         hosted

$ python --version
Python 3.9.12

Any thoughts on how to fix?

Hi! After playing around a bit with the prefect 2-beta i am now trying to get the public version up and running - but running into issues with the new deployment. My scenario is that i would like to build the deployments locally and have a VM (GCE) running as my agent. The problem is that when i am building the deployment file. my local path is hard-coded as a path - which results in an error when the agent tries to fetch the flow. Is there a way to solve this? Or is this a complete abuse of how it’s supposed to work?

Hi, I am just wondering what are all the possible options prefect provides us for executing spark/pyspark jobs. Like, in kubeflow pipelines we can use spark-operator to do this. In our use case we want to do everything on K8 clusters so using prefect how can we do that. Using this example, we can only run spark in local mode. We want to run spark/pyspark jobs on our K8 cluster with proper scaling and all. cc @Kevin Kho @Anna Geller

Hi, I was just wondering whether anyone knows if you can pause a deployment with python/CLI, as there is the option to pause a deployment in the UI. Cheers :)

I’m trying to build an AzDo pipeline for our Prefect code. To login to the Prefect cloud, the command

prefect cloud login -k *** -w myaccount/test

will ask for a profile name. Is there a way to pass the profile name in the parameters?

Do I need to download prefect.schedules module separately? If yes, I'm unable to install it. I'm getting below error. ERROR : Could not find a version that satisfies the recurretient prefect schedules ( VE DE ERROR: No matching distribution Fourd for prefect, scandules

Hi, i tried to install orion server on eks using helm following steps here. For some reason it only creates prefect orion api and postgresql pods and the api pod keeps crashing and never manages to get up. For prefect server 1.0, if i remember correctly, 5-6 pods were created. Did i miss something here?

Question about Prefect 1.0 memory usage: there seems to be some kind of memory leak even though each task is run as an isolated Shell task that completely exits once it’s done ----- what could be the cause of this?

Does Prefect 2.X version supports prefect agent docker start?

Do we need to authenticate with prefect cloud via

prefect cloud login -k $PREFECT_API_KEY -w $PREFECT_WORKSPACE

in order to run a

block.save

Python method? If we try authenticating through simply setting our default profile and running

prefect config set PREFECT_API_KEY/PREFECT_API_URL

but not explicitly calling prefect cloud, we get an error when trying to save a storage block (noted in replies). Is there a recommended pattern to use for CI/CD to a save a block otherwise?

Prefect 1.0: que about CronScheduler: is it possible to modify Parameter defaults for runs that are started via CronScheduler? the use case is that manual runs are assumed to be for testing and we want to skip a certain task by default. But tasks run via scheduler should not skip that task.