Hey folks! I'm new to Prefect and I'm sure I'm doing something dumb. I have a very simple task (like a few simple string manipulations) that I'm mapping over about 4000 list items, and I'm noticing that Prefect takes around 5 minutes to complete the tasks, is causing about 500MB of network traffic, and postgres is writing 600MB to disk. I'll provide more info in the thread below. Any advice would be much appreciated.

Hello friends! I'm trying to implement the scenario of the prefect's work with a dask cluster of 20 ~ 30 workers and it won't work. I'm asking for the best practice. I have the following tools: Debian server with a running prefect version 15.5 and an installed dask-2021.9.0 distributed-2021.9.0.  I run dask-scheduler on the Debian server and one of the dask-workers on another computer. Details are in my messages

I'm trying to run prefect in the docker container, using a local agent for it but the flow run does not complete the execution. In terminal get these logs, it goes as

deploying-->completed deployment-->returned with exit code -11!

, Attaching the screenshot. What is the

exit code -11!

here and how can I solve this issue.

Hello, guys. I have a KubernetesRun with image

image=f"project_name/prefect:etl_tools-{sha}"

where sha is a git commit SHA. And I have a registration bash file that call

prefect register --project etl -p "/flows/etl/**/**.py"

. Obvious that every commit to repo change SHA, change the image, and registration process bump version for all flows. There is a way to NOT bump all version and exclude image name from the idempotency key?

Hi guys (and girls), Currently we are using Prefect server with 1 agent and a default Dask executor with Dask scheduler and workers. We are running into problems with using local imports and then registering these imports in our flow. We get a "ModuleNotFoundError" for the given module, for example:

from test.package1 import test1

Would result into "ModuleNotFoundError: No module named 'test'". We are not using Docker images thus on registering we want this to be resolved instead of installing our package somewhere, which we constantly need to update. The same issue happens here https://github.com/PrefectHQ/prefect/issues/2797 except here they are using Docker images, while we are not.

Hi Everyone, we seem to be experiencing an unexpected issue with one of our ECS agents. When kicking off a flow, we get an error message:

Unable to locate credentials

. This happens right after the flow is scheduled and submitted (before any of the tasks can even begin to kick off). We don't think it is related to boto because our EC2 instance is able to read from aws buckets (confirmed by ssh'ing on the instance) via an iam policy attached to the EC2 instance. Unfortunately, running with debug mode didn't produce any more verbose results or a stack trace. We did have to reboot this instance recently after it became unresponsive due to an errant task we had running that consumed too much memory and stalled out. Has anyone experienced this before?

Hi Folks, Is there an example of sending emails using Sendgrid using prefect? I can send emails via sendgrid API without issues. However, while using it with prefect I am getting a

ForbiddenError: HTTP Error 403: Forbidden

Hi Everyone, Quick question on alerting, is there a way to set up alerts to receive emails or slack messages when agents are unhealthy? or if no run has completed successfully in the last say 24hours? Thank you!

I noticed after upgrading to prefect 0.15.5 that the argument

set_schedule_active

is now defaulted to

True

(use to be

False

i believe) which causes a warning if you register your flow without a schedule in the prefect UI. I was wondering if this was intended as I didn't see anything about this change in the release notes.

We are trying to figure out how a flow running in Prefect Cloud might fail with

Failed to find current tenant None in result  {'data': {'tenant': [{'id': 'foo-400d-4288-8020-...', 'slug': '<redacted>'}]}}

when it attempts to trigger another flow run via

StartFlowRun

.

Was there ever any consideration for including a

--tag

or

--image-tag

override in the

build

CLI command?

hi there! I'm trying to add the

AWS_CREDENTIALS

secret to the Prefect Cloud UI but I"m unclear how exactly to format it, which one of these is it?

{"ACCESS_KEY": "abcdef", "SECRET_ACCESS_KEY": "ghijklmn"}
or
'{"ACCESS_KEY": "abcdef", "SECRET_ACCESS_KEY": "ghijklmn"}'

Also, which "type" should I choose when entering the secret?

Hi Humans, I have a hopefully quick question that I need help with. I am supporting a Prefect installation which I would like to clean up the code base since there is a lot of boilerplate, I am considering using a single python source file and add the necessary flow name, parameters and whatever else from a database query . But I realized that each flow must be registered so using a single python file this way may not work... So my question is will it work, and how would I register the individual flows? Thanks!!

Hi just started installing the prefect and I am facing an issue while creating a project from cli and getting ValueError: invalid literal for int() with base 0

seems the auth_token is getting loaded into int type variable

Traceback (most recent call last):  File "/usr/local/lib/python3.9/site-packages/toml/decoder.py", line 511, in loads   ret = decoder.load_line(line, currentlevel, multikey,  File "/usr/local/lib/python3.9/site-packages/toml/decoder.py", line 778, in load_line   value, vtype = self.load_value(pair[1], strictly_valid)  File "/usr/local/lib/python3.9/site-packages/toml/decoder.py", line 918, in load_value   v = int(v, 0) ValueError: invalid literal for int() with base 0

Is it possible to tell a flow to not run if it is already running?

I recently had to work with kedro, an open-source Python framework for creating modular workflows. Even though I am not a 100% fan of the framework, there are several features I really like. One of which is how they handle the concept of a flow of flows (which they call modular pipelines). Hence the reason I am posting it here, mainly because I thought it could serve as inspiration for additional features in Prefect...

Hey there! Is there anything equivalent to

prefect get

(api here) in the Python client? In particular, is there any way to get a Flow ID by it’s name, project, & version as there is with the CLI?

Hey everyone, I am using the

create_flow_run

and

get_task_run_result

to interact with child flows. Now I would love to set some general task properties like

state_handlers

and

task_run_name

on these. Maybe I am missing something, but I can't figure out how to do this without writing my own wrapper task. Is there a way to do this?

Hey folks! I'm struggling with implementing conditional logic inside my Prefect flow. This script works but this one with the conditional logic fails with a rather obscure repetitive error message. AFAICT I'm following the guidelines outline in the Prefect docs here. Would really appreciate any pointers on what might be going wrong here!

File "/Users/rpelgrim/mambaforge/envs/prefect/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/Users/rpelgrim/mambaforge/envs/prefect/lib/python3.9/multiprocessing/spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "/Users/rpelgrim/mambaforge/envs/prefect/lib/python3.9/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

I'm having a strange error that just popped up one day after a completely unrelated code change:

Unexpected error: ValueError('Cell is empty')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 49, in dumps
    result = pickle.dumps(x, **dump_kwargs)
_pickle.PicklingError: Can't pickle <function create_job at 0x7fabe956fc20>: it's not the same object as tasks.common.create_job

None of my Prefect tasks have been working since it first appeared. I tried this Stack Overflow answer but it didn't help: https://stackoverflow.com/a/1413299 What causes this error? The

create_job

function is always referred to by its full Python import path (

tasks.common.create_job

) due to the Stack Overflow answer.

Hello everybody! When i use prefect to ETL data. I cant track mapping from fields data source to fields destination

hey all, I'm trying to setup prefect on an ubuntu server instance. I've got it installed with sudo, python3.8, but I'm having trouble finding the config.toml file

Hi - I am seeing some an issue on my flow. I was hoping someone could shed some light on how to resolve this one. Here is what I am seeing in the log:

06:42:18
ERROR
CloudTaskRunner
Failed to set task state with error: HTTPError('400 Client Error: Bad Request for url: <https://api.prefect.io/graphql>')
Traceback (most recent call last):
  File "c:\program files\python37\lib\site-packages\prefect\engine\cloud\task_runner.py", line 128, in call_runner_target_handlers
    cache_for=self.task.cache_for,
  File "c:\program files\python37\lib\site-packages\prefect\client\client.py", line 1407, in set_task_run_state
    version=version,
  File "c:\program files\python37\lib\site-packages\prefect\client\client.py", line 281, in graphql
    retry_on_api_error=retry_on_api_error,
  File "c:\program files\python37\lib\site-packages\prefect\client\client.py", line 237, in post
    retry_on_api_error=retry_on_api_error,
  File "c:\program files\python37\lib\site-packages\prefect\client\client.py", line 401, in _request
    session=session, method=method, url=url, params=params, headers=headers
  File "c:\program files\python37\lib\site-packages\prefect\client\client.py", line 333, in _send_request
    response.raise_for_status()
  File "c:\program files\python37\lib\site-packages\requests\models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <https://api.prefect.io/graphql>

has anybody setup slack messages for prefect from within a task, basically I am looping over some SQL table builds, and I wrap them in a try/except for each table. the try/except allows the task to continue running if any one table fails, but besides logging the warning, I don't get a slack notif for any of the individual tables if they fail... i have the slack notifs setup for the flow as a whole, and could always set them up with the native python slack integration, but was curious if anyone had set that up within prefect natively, thanks!

hey, I tried to register a new version of a flow while the previous was running and it failed the run with an error

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 861, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/executors.py", line 327, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.8/site-packages/prefect/tasks/prefect/flow_run.py", line 235, in wait_for_flow_run
    for log in watch_flow_run(
  File "/usr/local/lib/python3.8/site-packages/prefect/backend/flow_run.py", line 94, in watch_flow_run
    flow_run = flow_run.get_latest()
  File "/usr/local/lib/python3.8/site-packages/prefect/backend/flow_run.py", line 422, in get_latest
    return self.from_flow_run_id(
  File "/usr/local/lib/python3.8/site-packages/prefect/backend/flow_run.py", line 568, in from_flow_run_id
    flow_run_data = cls._query_for_flow_run(where={"id": {"_eq": flow_run_id}})
  File "/usr/local/lib/python3.8/site-packages/prefect/backend/flow_run.py", line 620, in _query_for_flow_run
    raise ValueError(
ValueError: No flow runs found while querying for flow runs where {'id': {'_eq': 'b1533b47-859e-46c7-acdc-dbe4976ac3e7'}}

How do I get access to the labels on a Flow? Are they in the Context object?

Hello! The new config variable

max_scheduled_runs_per_flow

. Is there a way to configure it via ENV variable? (We use Prefect in docker-compose)