This message was deleted.

Hi all, are there any plans relating to a prometheus exporter for Kubernetes deployed agents - thought I would check before I go about building some form of a custom exporter

how can i change the create_flow_run task name?? see this schematic has three create_flow_run, if we can’t change the displayed name, it’s hard to debug if one fails

I’d like to be able to use the

run_task_kwargs_path

for

prefect agent ecs start

as it’s not picking up my VPC (not the default one) but the docs don’t say what format

options.yml

should be in. Saw this issue here, but hasn’t been an update recently https://prefect-community.slack.com/archives/CL09KU1K7/p1653756981241529

Hi! I'm running into an error when trying to run a Prefect 2.0 flow using Comet.ml and a Ray task runner. The flow seems to fail at the comet import statement:

File "C:\Users\...\lib\site-packages\comet_ml\connection.py", line 35, in <module>
    from comet_ml import config
  File "C:\Users\...\lib\site-packages\comet_ml\config.py", line 59, in <module>
    from everett.manager import ConfigIniEnv as UpstreamConfigIniEnv
ImportError: cannot import name 'ConfigIniEnv' from 'everett.manager'

Any thoughts?

I'm trying to set up S3 storage for flow runs (using prefect 1)

with Flow(
    "test-flow",
    storage=S3(bucket="my-bucket"),
    run_config=ECSRun(...)

but getting this error:

Failed to load and execute flow run: NoCredentialsError('Unable to locate credentials')

I added a Secret named

AWS_CREDENTIALS

through the prefect cloud UI (https://cloud.prefect.io/team/secrets) with a json of the form

{
  "ACCESS_KEY": "...",
  "SECRET_ACCESS_KEY": "..."
}

but still getting this error. Curious if anyone one has tips to troubleshoot.

Has anyone seen the error:

'prefect.engine' is a package and cannot be directly executed

when trying to run a DockerContainer flow in Prefect 2.0?

For Prefect 2.0, how do I correctly pass the env variable EXTRA_PIP_PACKAGES to the runtime infrastructure? I tried below, but am still running into an error that s3fs is not installed - which is required to pull storage from S3

In prefect 1.0, is there any kind of support for dynamically adding parameters to a current flow run or updating the values of parameters

Hi Community! I am having trouble increasing the memory of my tasks when using the ECSRun, I followed the documentation from Prefect 1.0 but for some reason the value remains in 1024 cpu units and 2048 mb ram I need 4096 mb of ram, anyone has done this that can help me ?

any guide on this error message for prefect 1.0 cloud

HTTPSConnectionPool(host='34.66.191.39', port=443): Max retries exceeded with url: /apis/batch/v1/namespaces/default/jobs (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f3196f62eb0>: Failed to establish a new connection: [Errno 110] Connection timed out'))

Hi all, in the realtime examples I’m coming across, all of them use polling in the form of

if __name__ == "__main__": while True:

- are there any examples where flows can be invoked via realtime events, e.g. S3

ObjectCreated

Furthermore, are there any Prefect components in place so that if you change the flow code, it can replay all of the prior events?

Hi there - does this look right?

import asyncio
from prefect_snowflake import SnowflakeCredentials

from prefect import flow
creds = SnowflakeCredentials.load("credentials")

from prefect_snowflake.database import snowflake_query


@flow(name="Snowflake Query")
async def select_something():

    result = await snowflake_query(
        "select 1 as test",
        creds,
    )

if __name__ == "__main__":
    asyncio.run(select_something())

Seems to work locally but when I try to run ad-hoc via a deployment I get the error:

...
prefect_snowflake/database.py", line 69, in snowflake_query
    with snowflake_credentials.get_connection(**connect_params) as connection:
AttributeError: 'coroutine' object has no attribute 'get_connection'

I'm wondering can Parameter () take a list as default values? Since I need to map it to do some parallel running

does anyone have an example of what CI/CD should look like with prefect 2.0’s new deployments? and should i be committing deployment yamls and manifests to source control?

Hi all, is there any existing documentation / discussion about the difference between prefect 2.0 self-host vs cloud version?

Hi all, I have some questions/thoughts about the deployment process with Prefect 2.0. (I am using Kubernetes jobs) After every change the deployment build has to be run to upload the files. But every time the yaml has to be edited again, e.g. changing the image. Is there a better way to do this and not manually? Maybe specifying a template for the build would solve this manual work. Or a CLI option to keep the content. Also all files are uploaded, like pycaches. There should be a ignore file to handle this, like gitignore. When files are moved inside the folder stucture underneath the manifest or renamed and the deployment is rebuilded, the old files are still in the storage. This can end up with duplicated files. Since just clearing the storage before rebuilding/uploading could crash current pulling deployments, I would think about a version folder between the path given in the storage block and the manifest-root. A 'build' would upload the files to the new version-folder and an 'apply' would switch to the new folder. Our Flows are all event triggered with parameters. How can I trigger a flow run with parameters? I would expect an option --parameters in 'deployment run'. In our current Prefect 1 setup I am using a state_handler to do some post processing after a failed flow (state_handlers=...). How would this look like in Orion? Should I implement a very last task in a flow to do this?

Hi there! I’m trying to forward logs form an external library, more specifically

scrapy

. I have configured scrapy to log to stdout, and configured my Prefect task to

log_stdout=True

. Should not Prefect then pick up the logs from scrapy or am I missing something?

Hi all, great to hear that prefect 2 is now out of beta - the announcement doesn't give much clarity about how long prefect 1 cloud is going to be around - are there more details around this?

Is it easy to upgrade prefect 1 flows to prefect 2? And will the vertex-ai runner be available in prefect 2?

In Prefect 2.0, I have a task

@task
def run(x: int, y: int) -> int:
    return x + y

@flow
def my_flow():
    results = run.map([1, 2, 3, 4], y=1)

How can I map only on

x

, and use the same

y

argument?

When I use map to submit about 10000 tasks, the run failed. Is there a limit to how many tasks I can submit?

I'm back with another question, having a hard time finding anything in the docs or slack regarding this: Is there an equivalent to the old StartFlowRun or the newer create_flow_run in Prefect 2.0? Or some other way of having a flow run a subflow without knowing about it beforehand (since it's deployed in a separate docker container)?

I cannot log in a task. Logging in a flow is okay. A task is called using submit.

Is Prefect specialized for streaming pipeline schedule?

what does this error implies:

TypeError: `fn` must be callable

I’m calling a task inside a With Flow statement, and the Task should return me a list of strings

Hi there! I am having an issue with Prefect 1.2.4 running a flow of flows. I followed the tutorial https://www.prefect.io/guide/blog/flow-of-flows-orchestrating-elt-with-prefect-and-dbt/ that is great and helped me a lot to connect DBT with other flows. This was working fine but suddenly start to fails with this error: Error during execution of task: ValueError("Flow 'ETL' not found.")

Hey there, for the run history, how long it keeps the logs? If a flow runs extensively long time, will it continue to finish even though the run history is wiped out? How can i track the logs for those long running flows?

hey y'all! the Prefect Helm Chart doesn't have any resources defined (cpu/mem limits/requests)... are there any good recommendations for a starting point?