any guide on this error message for prefect 1.0 cloud

HTTPSConnectionPool(host='34.66.191.39', port=443): Max retries exceeded with url: /apis/batch/v1/namespaces/default/jobs (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f3196f62eb0>: Failed to establish a new connection: [Errno 110] Connection timed out'))

Hi all, in the realtime examples I’m coming across, all of them use polling in the form of

if __name__ == "__main__": while True:

- are there any examples where flows can be invoked via realtime events, e.g. S3

ObjectCreated

Furthermore, are there any Prefect components in place so that if you change the flow code, it can replay all of the prior events?

Hi there - does this look right?

import asyncio
from prefect_snowflake import SnowflakeCredentials

from prefect import flow
creds = SnowflakeCredentials.load("credentials")

from prefect_snowflake.database import snowflake_query


@flow(name="Snowflake Query")
async def select_something():

    result = await snowflake_query(
        "select 1 as test",
        creds,
    )

if __name__ == "__main__":
    asyncio.run(select_something())

Seems to work locally but when I try to run ad-hoc via a deployment I get the error:

...
prefect_snowflake/database.py", line 69, in snowflake_query
    with snowflake_credentials.get_connection(**connect_params) as connection:
AttributeError: 'coroutine' object has no attribute 'get_connection'

I'm wondering can Parameter () take a list as default values? Since I need to map it to do some parallel running

does anyone have an example of what CI/CD should look like with prefect 2.0’s new deployments? and should i be committing deployment yamls and manifests to source control?

Hi all, is there any existing documentation / discussion about the difference between prefect 2.0 self-host vs cloud version?

Hi all, I have some questions/thoughts about the deployment process with Prefect 2.0. (I am using Kubernetes jobs) After every change the deployment build has to be run to upload the files. But every time the yaml has to be edited again, e.g. changing the image. Is there a better way to do this and not manually? Maybe specifying a template for the build would solve this manual work. Or a CLI option to keep the content. Also all files are uploaded, like pycaches. There should be a ignore file to handle this, like gitignore. When files are moved inside the folder stucture underneath the manifest or renamed and the deployment is rebuilded, the old files are still in the storage. This can end up with duplicated files. Since just clearing the storage before rebuilding/uploading could crash current pulling deployments, I would think about a version folder between the path given in the storage block and the manifest-root. A 'build' would upload the files to the new version-folder and an 'apply' would switch to the new folder. Our Flows are all event triggered with parameters. How can I trigger a flow run with parameters? I would expect an option --parameters in 'deployment run'. In our current Prefect 1 setup I am using a state_handler to do some post processing after a failed flow (state_handlers=...). How would this look like in Orion? Should I implement a very last task in a flow to do this?

Hi there! I’m trying to forward logs form an external library, more specifically

scrapy

. I have configured scrapy to log to stdout, and configured my Prefect task to

log_stdout=True

. Should not Prefect then pick up the logs from scrapy or am I missing something?

Hi all, great to hear that prefect 2 is now out of beta - the announcement doesn't give much clarity about how long prefect 1 cloud is going to be around - are there more details around this?

Is it easy to upgrade prefect 1 flows to prefect 2? And will the vertex-ai runner be available in prefect 2?

In Prefect 2.0, I have a task

@task
def run(x: int, y: int) -> int:
    return x + y

@flow
def my_flow():
    results = run.map([1, 2, 3, 4], y=1)

How can I map only on

x

, and use the same

y

argument?

When I use map to submit about 10000 tasks, the run failed. Is there a limit to how many tasks I can submit?

I'm back with another question, having a hard time finding anything in the docs or slack regarding this: Is there an equivalent to the old StartFlowRun or the newer create_flow_run in Prefect 2.0? Or some other way of having a flow run a subflow without knowing about it beforehand (since it's deployed in a separate docker container)?

I cannot log in a task. Logging in a flow is okay. A task is called using submit.

Is Prefect specialized for streaming pipeline schedule?

what does this error implies:

TypeError: `fn` must be callable

I’m calling a task inside a With Flow statement, and the Task should return me a list of strings

Hi there! I am having an issue with Prefect 1.2.4 running a flow of flows. I followed the tutorial https://www.prefect.io/guide/blog/flow-of-flows-orchestrating-elt-with-prefect-and-dbt/ that is great and helped me a lot to connect DBT with other flows. This was working fine but suddenly start to fails with this error: Error during execution of task: ValueError("Flow 'ETL' not found.")

Hey there, for the run history, how long it keeps the logs? If a flow runs extensively long time, will it continue to finish even though the run history is wiped out? How can i track the logs for those long running flows?

hey y'all! the Prefect Helm Chart doesn't have any resources defined (cpu/mem limits/requests)... are there any good recommendations for a starting point?

Hi there! Does anyone know how to run a flow from a specific docker image (not the prefect image) in Prefect 2.0? I’ve been trying with this, passing the

image=<my_images_name>

argument to the DockerContainer, but doesn’t work, or at least, it doesn’t run the

CMD

specified in the Dockerfile of that image

Hi everyone. I started recieving an error message for all flows:

404 Client Error for <http+docker://localhost/v1.41/containers/create?name=elastic-salamander>: Not Found ("No such image: <http://26123096XXXX.dkr.ecr.us-east-1.amazonaws.com/cal_val_etl_flows:latest-165946XXXX%22|26123096XXXX.dkr.ecr.us-east-1.amazonaws.com/cal_val_etl_flows:latest-165946XXXX">)

I can go on AWS ECR, see the container, and see that the URI matches with the one from the error so it does exist…I have changed some settings recently regarding the python version and docker build, but cannot understand why i would get a container not found error edit: formatting

while running the deployemnt from ui in prefect 2.0.1 . athough the directory exists

Which syntax I can use to get current flow run's parameters value

Hello. In Prefect 1: Context: we have a flow of flows, and to manage dependencies, we have 2 solution. Either make sure the infra registers the flows in order, or to import and register the underlying flows in the flow-of-flows. What could go wrong if you do register a flow in the flow-of-flows like

from underlying_flows.flow_a import flow_a

flow_a.register(project...)

With Flow(...):
    ...

Because when we do that, prefect throws a warning

UserWarning: Attempting to call "flow.register" during execution of flow file will lead to unexpected results.

For launching an ECS instance via the

--cli-input-json

tag for cli, should there be an extra argument in the

prefect-agent-td.json

file called

agentRoleArn

? There are two for

executionRoleArn

and

taskRoleArn

but if the agent itself needs a Role in AWS should that be specified or can it be included in the other variables?

Hello! I have a 2.0 question about registering and using blocks, specifically infrastructure blocks.

Following these links for running prefect sample flow on cloud 2.0 : https://orion-docs.prefect.io/ui/cloud-getting-started/ + https://orion-docs.prefect.io/concepts/storage/ Executed this command:

prefect deployment build ./sample.py:samplefunc --name "ALS Deployment" --storage-block recommenderprefectserver

Error: Deployment.yaml is not created with following exception

Hi all! I could use some help here. I’m trying to query a DB to get a collection of inputs to process, and to dispatch each input to a sub-flow:

@task()
def get_inputs() -> List:
    # This task retrieves a list of inputs to process from a DB and returns it
    pass


with Flow(name='dispatcher') as dispatcher:

    inputs = get_inputs()

    for i in inputs:
        create_flow_run('worker', parameters={'input': i})

It doesn’t work as I need to provide either nouts or return a Tuple in order to iterate over my inputs, however I don’t know the size of it. What would be a correct approach here? Thanks!

Hi! I'm trying to specify result location per task in Orion but I can't find any documentation on the issue. Was the task result functionality removed in Orion?

Prefect Agent and and Orion both are up, yet the flow on cloud: 2.0 is scheduled only and paused in the late state