HI, i have set the prefect server using helm chart on gke following this video on prefect channel. I can see on my cluster prefect server and all other dependencies like apollo etc are deployed including prefect agent that i deployed using --set flag with helm install. For the first time this ui comes on prefect server

Hi all, I'm new to Prefect and trying to set up a local server on an Azure (linux) VM inside a VNET. I'm having issues getting the up and running in the first place. I'm in a corporate environment; unfortunately this means MANY blocked ports. If I do an NMAP scan on the VM I get the first picture below. To even see the server I have to use the expose flag, and set the ui port to 80. With some testing it seems if I set the server port to 1720 it all starts okay (

prefect server start --expose --ui-port=80 --server-port=1720

), I can at least get to the UI then, but the UI gives me a failure message about not connecting to graphql and a link to this slack (second picture below). I've tried setting the graphql port flag 5060 or 443 or even also to 1720 or using the no port mapping flag (https://docs.prefect.io/api/latest/cli/server.html), all to no avail. I'm new to this so I'm probably doing something stupid but I can't figure out what; any thoughts? In case it's related, I'm also unable to start a user agent. When I try

prefect agent local start

I get

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=1720): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1d60a34e20>: Failed to establish a new connection: [Errno 111] Connection refused'))

. I'm unclear on whether the server is automatically starting agents of its own or if I'd have to do this myself in addition to the server. I could potentially have the option to open some additional ports, but it takes several days, a long approval process that might get rejected, and I'm not even clear on which I'd need to open or why. After all, these are mostly internal ports to the same machine, right? I know that's a lot but I'm a bit at a loss here; any help?

i have set the prefect server and agent using helm chart on gke. I have set the value in .prefect/config.toml

[server]

host = "http://<EXTERNAL-IP>"

as mentioned in docs to point the cli to server. i have created the tenat using command

prefect backend server && prefect server create-tenant --name default --slug default

now when i open the server ui i shows the getting-started page and if i click on dashboard

Don't we have secret option available for server Ui? as my flow is stored on github repo which is private so i need secretes there

Hello, i'm trying to test out prefect locally on a linux machine and running into something (that it seems a bunch of others have run into) but not sure how to solve it

and, on a separate issue, when i try to test out prefect cloud and lift up a local agent (according to instruction given) - i'm seeing the following:

I have a basic question about the use of

Module

storage in a Docker container. I've been trying to get it working for half a day now and I'm stumped. I'm sure it's a very simple problem but I'm unable to find any working examples as a basis. So I have a flow that is installed (via

pip install

) in a Docker image. The flow is called

flows.docker_2

. It looks like this:

import prefect
from prefect import task, Flow
from prefect.run_configs import DockerRun
from prefect.storage import Module

@task
def say_hello():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello, docker 2!")

with Flow("docker_2_flow") as flow:
    flow.storage = Module("flows.docker_2")
    flow.run_config = DockerRun(
        image="prefect_docker_2_image",
        host_config={
            "auto_remove": False
        }
    )
    say_hello()

if __name__ == "__main__":
    flow.register(project_name="docker_2_project", labels=["docker_flows"])

I am able to register the flow by executing it inside the container:

docker run --network prefect-server --rm --name prefect_docker_2 -it prefect_docker_2:latest python -m flows.docker_2

So I know that the module is installed and can be run inside the container. However, when I run the flow, the module is not found. The docker agent reports:

File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'flows'

Is there something else I have to do (other than `pip install`ing the module) to make the flow code visible?

Hi everyone, This might sound pretty straight forward but I have spent a few days on it but I am not sure what's possible. Is it possible to run prefect server in a container (I have read that running docker compose in a container doesn't work)? Q2: Has anyone been able to run prefect server in an Azure Web App? Thanks

Hi, I’m trying to get some basic (short-term) persistence going with the included Postgresql in the Prefect Server Helm Chart. When I change

postgresql: persistence: enabled:

from

false

to

true

in the chart and

helm install

, the chart appears to install successfully as normal and the various server pods are created, but the Postgres pod never appears and there are no error messages, nothing in

kubcetl get deployments

or

kubectl describe replicasets

etc. With the value

false

it works fine (except for the part where it loses all data when the pod dies). Does anyone know how to debug this?

Hello ! Is it possible to use the

create_flow_run

paradigm to create multiple flow runs with differing parameters using

map

? Been trying it out and can't seem to make it work. Code below.

Hey again, Got another question, sorry for bothering. How can I get the result of a subflow to pass as input to another task ? I'd like to do that because I have a specific task in my flow that is quite resource hungry and I would like to run it as a separate flow in order to optimize costs. The issue is that the output of this task is currently used downstream and I'd like to do the same with it running as its own flow. I may have missed it in the documentation, but I don't think I saw this use case described. Basically, I'd like to do something like this :

with Flow(name="Parent Flow") as flow:
    docs = Parameters("docs", default=[1, 2, 3])
    small_task = SmallTask().map(docs)
    big_task = create_flow_run.map(flow_name=unmapped("Child flow"), project_name="whatever", parameters=small_task)
    wait_big_task = wait_for_flow_run.map(big_task, raise_final_state=unmapped(True))
    another_small_task = AnotherSmallTask().map(wait_big_task)

but I'm not sure how or where to define the result of a flow in order to use it downstream.

shown left: I run a flow, flow enters subflow, completes successfully, and enters the "clean" task successfully when there is a constantly running "keep-alive" task running in parallel. The keep-alive task just logs "keeping alive" every 10 seconds for an indefinite period of time. shown right: I run a flow, flow enters subflow, subflow completes successfully but the main flow does not understand that (hence the grey status for the subflow), main flow restarts instead of entering the clean task (hence the greyed out box in one of the prior tasks --> where I cancelled this flow). Anybody understand why this happens, and a non-hackish fix for this?

Hi I am Volker from Inqbus Scientific Computing. We are currently evaluating Prefect/Orion as the future workflow engine for a not so small customer. Prefect is out of question but we like to give Orion a try. We have some deeply reaching questions about Orion: Class decorators, optimization without knowing the DAG beforehand, future development of the UI etc. Where do we place these questions best?

@Kevin Kho Thanks for the fast answer!

OK here we go. I noticed that the UI of Orion is way back behind the funtionality of Prefect. When can one expect a catchup?

Has feature development fully moved to Orion? When can we expect a stable version for Orion?

Hi. This sounds like a noob question. How do I change the default folder for results? (I don't mind, or actually want to, move the whole config folder) I've tried adding

home_dir = "/mnt/disks/prefect_data/.prefect"

to

.prefect/config.toml

, but results are still saved on my main disk

Hi, honestly I’m not sure if it’s prefect related, but, when I try to run the graphql query to turn off a specific flow:

mutation {
  set_schedule_inactive(input: {flow_id: "67d364da-375d-4d0f-b151-d46435351c50"}) {
    success
  }
}

I get:

"message": "field \"set_schedule_active\" not found in type: 'mutation_root'"

I’m basically trying to use the query from the documentation here

Hey ! Are tasks results available in flow-level state handlers ? I'd like to use those to generate some kind of report and send it to Slack.

Hello all. I have been trying to configure Prefect Server on a Debian/GNU Linux 10 VM and I encounter a problem. I have a conda env with prefect installed, docker and docker-compose installed as well. I can launch the server with prefect server start --expose (cannot seem to access the UI from my computer without the "--expose" part), but I got a "Couldn't connect to Prefect Server at http://localhost:4200/graphql" issue. It seems to be the same problem that the one Krishna had here: https://prefect-community.slack.com/archives/CL09KU1K7/p1636636632389000. I have attached the return of a prefect diagnostics command, and it seems that my docker containers have enough memory (16GB per container). My company is against the fact that I use Prefect Cloud. Finally, I can run Flows manually with 'prefect run -n "flow_to_test" --execute'. Anyone has any idea about my issue? Thank you very much!

Hey, so I’m using prefect server on AKS. For the job run, I want to pass a few environmental variables to the image by default. What is the best way to add this? On my local machine, I start the kubernetes agent with —env, can I do similar thing for the Kubernetes agent deployed on AKS?

Hi there! I’m attempting to set up an ECS Agent as an ECS Fargate service but I need to scope the IAM permissions to the minimum requirements. I was wondering if someone could explain why we need these permissions so that I can figure out how to scope them out:

"ec2:AuthorizeSecurityGroupIngress",
"ec2:CreateSecurityGroup",
"ec2:CreateTags",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVpcs",
"ec2:DeleteSecurityGroup"

Hey there! I’m trying out the artifact task, and I think it works well. However, I’m trying to get it work with larger artifacts. I submitted earlier a PR to make Apollo’s body size configurable. But I’m now running into another limitation. In the code there is a hard coded limit of 1Mb. I’m willing to submit a PR, but am unfamiliar with this codebase (with graphql in general actually). Can someone point me how to make this limit check configurable?

if data_size > 1000000:  # 1 mb max
    raise ValueError("Artifact data payload exceedes 1Mb limit.")

Hi all. Another question. Sorry for bothering. So I am trying to use a custom PostgreSQL database. I have set the a prefectdb postgre db with a username prefect and a given password. I have add the following lines of code into my config.toml file: [server.database] connection_url = "postgres://prefect:_mypassword_@0.0.0.0:5432/prefectdb" I launch my server with the command : prefect server start --external-postgres --expose However, I have the following error log coming up: hasura_1 | {"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"0.0.0.0\" I believe there is a mistake in my host definition, or a docker container link issue. I have tried to replace 0.0.0.0 with localhost and the hostname of my virtual machine, but I still find the same problem. Is there any more changes to add to the prefect server config file? Moreover, I have another error with the graphql container: Error: Can't load plugin: sqlalchemy.dialects:postgres And I am wondering whether I should set the connection_url element in config.toml as connection_url = "postgresql://...", as stated here: https://stackoverflow.com/questions/62688256/sqlalchemy-exc-nosuchmoduleerror-cant-load-plugin-sqlalchemy-dialectspostgre Could you please help me solve that issue? Thanks in advance, again!

Hi, I can not find anywhere info about how to upgrade prefect server and UI to the latest version. Can someone paste here the snippet?

Hello there ! I'm having trouble using

get_task_run_result

to access data from a subflow. This example should reproduce the issue :

from prefect import Flow, task, Parameter, unmapped
from prefect.tasks.prefect import create_flow_run, get_task_run_result

@task(slug="plus-one")
def plus_one(x: int):
    return x+1

with Flow("Child") as child_flow:
    x = Parameter("x", default=10)
    po = plus_one(x)

@task
def get_params():
    return [{"x": i} for i in range(5)]

with Flow("Parent") as parent_flow:
    params = get_params()
    flow_run_ids = create_flow_run.map(flow_name=unmapped("Child"), parameters=params)
    results = get_task_run_result.map(flow_run_ids, unmapped("plus-one"))

When running the parent flow, I get issues that look like :

Error during execution of task: ValueError("No task runs found while querying for task runs where {'task': {'slug': {'_eq': 'plus-one'}}, 'flow_run_id': {'_eq': '3316e187-adda-461d-814f-29a6ffc059a4'}, 'map_index': {'_eq': -1}}")

Is there something I'm missing with how slugs work ?

Hi all! Could anyone explain please, if I register flow with some parameters, for example: a=10. And in the flow i have:

with Flow('flow_name') as flow:
  a = prefect.Parameter('a', 'required'=True)
  <http://logger.info|logger.info>(a)

how can i get the parameter value to be used? So i want

a

to be 1, but I get:

a = <Parameter: a>

Im running the flow in the following way:

prefect_client.graphql(query= """
                                mutation{
                                    create_flow_run(input: { flow_id: %s , parameters: %s}) {
                                    id
                                    }
                                }
                                """ % (train_flow_id, json.dumps(flow_params).replace('"', '\"'))
                                )

where flow_params is a dictionary

Hi All, We are using Prefect Cloud along with Kubernetes agent for our deployment. Right now, we just have one prefect-agent pod running in Kubernetes, which in turn creates Kubernetes Jobs for all flow runs. I was wondering if this is the best approach? Should we run multiple prefect-agent pods instead? We are very close to our production release, and would really appreciate your inputs!

Hi, I’m trying to connect the Prefect Server Helm Chart to an existing postgres database. I’ve configured the username and password fields inside the values file. However my pods fail to start with this error:

Error: secret "prefect-server-postgresql" not found

. I thought this secret would be created automatically? Am I supposed to create it manually and if so how?