Hey guys, so I managed to do my first deployment (🦜 ) but the scheduled tasks are failing with this weird error 😞 Any help appreciated

also pls mind I'm more oriented towards devops side of things, so I'm not 100% sure how all python stuff works yet. The prefect install has it's own virtualenv, the script should run in the same venv (for now)

This is probably a dumb question, but is it possible to change or unregister a task when I'm developing a script in interactive mode? When I define a task it works fine, but if I edit the function for the task slightly and try to run the code to define it again I get the error

OSError: could not get source code

.

del the_task

does remove it from my environment, but then if I try to run the task definition again I still get the error. I'm not totally sure if this is a bug or expected behavior. Is my workflow here just wrong for prefect?

When I run this

prefect deployment build name.name -n name_dev -t dev_wq_k8s -sb gcs/dev --work-queue=dev_wq_k8s -ib kubernetes-job/dev-k8s-job

I get this error

Job is missing required attributes at the following paths: /apiVersion, /kind, /metadata, /spec (type=value_error)

Even though in the Prefect 2.0 UI for creating a k8s job block, it says

Job (Optional)

Hello everyone, I am using prefect server deployed on K8 and using git as storage. I am using a flow with date string as parameter shown in mage 1 and getting error shown in image 2. Can anyone guide or help me on how to resolve this error.

does someone have experience with Prefect Disaster Recovery? What are the best practices to restore the workflow and also the history of the runs?

Hi all! I'm just getting into Prefect and trying to understand how everything works. I'm not starting a prefect project from scratch, but trying to build on top of the existing one (mainly working on fixes now). The issue I'm running into right now that I can't find the answer to online is that

ShellTask

is not logging all the outputs from the script that it's running. I am looking at https://docs-v1.prefect.io/api/latest/tasks/shell.html#shelltask and seems like just setting

stream_output=True

should do the job, but I'm still seeing just

Command failed with exit code 1

. I also tried setting

return_all=True

and

log_stderr=True

, but still the same behavior. Has anyone ran into this issue in the past?

I am coming across this error. We are using saturn cloud to manage kubernetes. I have reached out to them, but hoping someone here can point me in a direction I can start debugging. This is prefect 1.0 still (we have put time aside to migrate before the end of the year). The container seems to terminate mid-task

Hi all! Does anyone know why

deployment.apply()

doesn't work from a jupyter notebook cell, but does work from

.py

script file. Also, is there anyway to deploy a flow to prefect cloud using jupyter notebook ?

Hey guys, quick Q. Just wondering if anybody knew how stable the deployment YAML for agents would be going forward in Prefect 2.0. I know there were some changes in the past few weeks, but was wondering how often, if ever, the deployment YAML for agents will change in the future as we are planning to put our deployment YAMLs for agents in source control yet we would like to avoid having to manually reconfigure things if they change between updates. Any assistance in the matter would be greatly appreciated

Hi, I'm running a flow run that downloads a file to local file system within docker container and getting [Errno 13] Permission denied. is there best practice to bypass this?

Prefect 2.0.4, dask, prefect UI displays logs from flows and subflows only, I have set loggers in task but those logs are not showing up in UI, I am using info logs only

Hi guys, in order to transfer big files between 2 saas systems I have to download locally first? and if so within Docker container is there a best practice?

Hello! I have two small questions related to blocks: • Even if the

prefect_dbt

and

prefect_airbyte

collections exists, I was wondering if it makes sense to have something like a dedicated block for Airbyte (which allow us to store the

server_url

but also invoke a

trigger_sync

function) and the same question with a DBT block (and we store all the information that we can find in the

profiles.yml

, for exemple)? • What is the best way to log from a function of a

Block

subclass? Can we use the

get_run_logger()

or is preferable to use some sort of callbacks that are going to be invoked inside a

@task

or something? Thanks!

Hi #prefect-community, getting my hands dirty on installing prefect for certain flows... As per the requirement we are supposed to use GraphQL agent. which i have no idea how to integrate into the local prefect system!! has anybody Worked on this? or any idea how to move forward?

Hi, has anyone experienced the following error that the Kubernetes Jobs cannot pull an public image. It gives the following error: Inserted in the thread.

Hi all, am using v2.1.1 and deployed prefect UI to kubernetes via helmchart. for some reason all the blocks and deplyoments dissappeared from the UI after a while, not immediately , could be this a prefect issue, or could be some configuration in the helm chart..thanks..

Is there any way to attach some kind of listener to a flow as it changes states? In Prefect 1 I would have done this with state handlers, but those don’t seem to be an option if I’m looking to monitor the flow status instead of task status. My use case is to sync the Flow status with our legacy (homegrown) orchestrator to help perform the migration, so it can almost be thought of as custom notifications.

Hi again, I'm trying to start flow run from REST API with Postman - but it's stuck in status PENDING. The agent started but it's not picking it up.... any ideas ??? http://localhost:4200/api/deployments/aa64fe41-a2f2-43c3-80f8-38e9d5c1ad26/create_flow_run

Well, shoot. How do I solve the lovely

_pickle.PicklingError: Pickling client objects is explicitly not supported.
Clients have non-trivial state that is local and unpickleable.

Pickle error on clients in Prefect 2.0?

Hi there! We are seeing that when registering flows on Prefect1, the code inside the task gets executed. This makes some trouble for example when doing queries taking long time, as the task registering sort of times out. Is it thought to be this way?

Not sure if this is the right channel to post this but prefect cloud seems to be very unresponsive/slow in the last few hours. We aren't able to deploy any new flows and having problems scheduling existing ones. Wanted to know if anyone at @Prefect has any insights regarding the problem. We have some critical jobs production jobs scheduled on prefect.

Hey all, has anyone seen this error when attempting to use a GCS storage backend?

RuntimeError: File system created with scheme 'gcs' from base path '<gcs://mattmelgard-test-bucket-01>' could not be created. You are likely missing a Python module required to use the given storage protocol.

This is the CLI command I’m using to build the deployment:

prefect deployment build tutorial-flow/flow/tutorial-flow.py:my_flow -n tutorial-flow-k8s -t test -i kubernetes-job -sb gcs/tutorial-bucket

and I have a block defined for the bucket with the name

tutorial-bucket

but I’m not sure what could be missing here, I have pretty much all the python packages installed via

pip install 'prefect[all_extras]'

@Tim Enders I hope you don't mind that I move your message here - the other channel is for sharing tutorials I have this snippet from Prefect 1.0. What are the analogs to

results

and

state_handler

in 2.0?

with Flow(
    "Subscriptions API Import",
    result=GCSResult(bucket="platformsh-vendor-prefect"),
    # executor=LocalExecutor(),
    executor=LocalDaskExecutor(scheduler="threads", num_workers=5),
    state_handlers=[flow_failure],

Hi there I'm working on migration from Prefect 0.x to 2.x. To deploy, I'm using

prefect.environments.storage.Docker

to create an image with my flow, context, scheduler and all the required virtual environnement. Then I register flow to Prefect with

prefect.environments.DaskKubernetesEnvironment

as environment. I'm using

prefect.executors.DaskExecutor

to run it. To deploy on 2.x, I simply try to use

DockerContainer

but it doesn't work without storage. So I try to install main branch of Prefect because of a merged PR ( https://discourse.prefect.io/t/how-to-build-deployments-with-flow-code-and-dependencies-being-baked-into-a-docker-image/1341 ) to use

DockerPackager

but it's not compatible with

Deployment.build_from_flow

. I also try to push directly my Docker image with

push_image

but push doesn't already work. So, I'm a bit confused because I can't find explicit documentation on Discourse and I read different responses because of the different releases on 2.x. Is

DockerPackager

the best practice to deploy a completely self contained flow? In this case, do you have any idea of the release roadmap to fix this (and to update documentation) ? If not, what can I should use? Thank you !

Hi. I using prefect 2.2. I have a use case that looks something like this: • A data file has been created, I create a flow_run using the client API with the file path as a parameter • At the end of the flow, if it has not errored, then I want more tasks to run, but I need it to wait for a particular interval (day, week) I could do this with my own external scheduling system (that I have not written), but I would love to do that within prefect if possible. So my question…within a

flow

, is there a way to schedule a subflow to start at a particular time in the future? Or some other way to accomplish a long-delayed subflow/task?

Prefect 1.0 - Is there a recommended way to reroute/duplicate the pip logs for

EXTRA_PIP_PACKAGES

to Cloud UI? When using Kubernetes agent, those logs only show up in job pods logs and are not available in Cloud UI.

[RESOLVED] Hi Guys, sorry if my question is very basic, but I'm starting in prefect 😅. I'm trying to find out how I can send an HTTP request to the prefect API to create a flow that runs from a deployment. With Orion running locally, I was able to perform this request through the url

<http://127.0.0.1:4200/api/deployments/{id}/create_flow_run>

and it worked normally (I used Insomnia). Now, I need to make this same request through Prefect Cloud. I was already able to create the deployment and I was able to run the flow normally through the Prefect UI with the agent active. Can you help me?

Hello, i’m just setting up Prefect 2.0 and i’m trying to understand options for setting up agent/work queues in a production environment. I haven’t been able to find resources on pros/cons of different options. Any suggestions on where to look?

Hi! I’ve searched in the docs but couldn’t find an answer so I’m asking here, is there a way to run a task even if a previous task fails? Here’s an example:

@flow
async def test():
    connected = await connect_db.submit()

    task1 = await db_stuff.submit(wait_for=[connected])
    await disconnect_db.submit(wait_for=[task1])

If

db_stuff

fails,

disconnect_db

never is executed but I’d like it to do it