I find an interesting behavior of the local server. I used to launch server on my host A and created some data. Today I try to start server on host B and access its UI with browser on host A, I find that all my flows/projects etc are visible to host B UI because the resources are loaded as

<localhost://xxxx>

, others who access server B UI can’t see it

Dear Prefect Community. I am new to prefect and I'm evaluating it vs Airflow and the why-not-airflow paper on the website was very convincing but my first experience was not positive - thus reaching out here because I dont want to give up on a technology that might just make my life easier later by giving up too early. I will paste the startup error in this thread - but the problem is on prefect server start - I am getting connection errors, and then later when I pull up the UI :8080 I get to the getting started screen but the graphql connect fails with more docker error logs scrolling. I can move the UI question to the other room if this is the wrong place.

I get so confused by how the

graphql

works. Here is my operations • First I try

prefect server config > origin.yml

then launch it with docker-compose. App works on

localhost:8080

• Under

getting started

tab, I put

<http://localhost:4200/graphql>

as graphql end point and it works. So far so good • Then I try

prefect server config --graphql-port 40002 > my-prefect.yml

, launch with docker composer. App still works on the same address • However, I tried to give ui the graphql endpoint with values

<http://localhost:4200/graphql>, <http://localhost:40002/graphql>, <http://graphql:40002/>, <http://localhost:40002/>

with/without last

/

. None of them works. I try in browser

<http://localhost:40002/graphql/>

, it gives me a console to play. As long as I touched graphql port, I will not be able to set up a workable local sever. Really appreciate if someone can help me take a look

Re my message above. Eventually my workaround is: • No touch on

graphql

. Only move ui and server port

prefect server start --ui-port 30001 --server-port 30002 --no-postgres-port --no-hasura-port --no-graphql-port

• On

prefect server

tab on the web application, type in

<http://localhost:30002/graphql>

to reflect the change • You will be blocked on next tap when creating tenant, add this to your ~/.prefect/config.toml. Otherwise command

prefect agent start --name "Default Agent"

will by default access port

4200

instead

server]
host = "<http://localhost>"
port = "4200"
host_port = "30002"
endpoint = "${server.host}:${server.host_port}"

Then everything all set

Hey! Can anyone confirm for me that if a schedule contains more than 10 (concurrent) runs, this is limited to 10 runs only. My schedule contains 20 runs (with different parameters for a flow) where many start at the same time. However, when I deploy this schedule to Prefect only 10 runs are scheduled instead of the 20. This is on Prefect's website: There is no limit to the number of flow runs you can create, even if they start at the exact same time with the exact same parameters: Prefect supports unlimited concurrent execution. This does not seem the case for me.

what is the max size you can give as parameters?

Hello all, We will have 3 flows for 3 environments (DEV, QA, PROD) ... Those flows are all the same except the AWS resources. Instead of creating 3 different python files, is there a way to pass resource names as a parameter with this command to the python file? prefect register flow --file prefectDemo.py --project \"Data Loading\"

Hi, Prefect community! I successfully created my first flow and was able to see it in the UI. But I have some questions. 1). At the moment I am connecting to the remote machine via ssh. When Prefect is running, I can access the UI through both localhost and the ip of my remote machine, for example xxx.xx.xxx.xxx:8080 or localhost:8080. But I cannot view my flows from my phone, other computer, etc. What confuses me is that the UI is visible on all devices, but flows are displayed only on my laptop at the specified addresses. What could be the problem? 2). How to start agent in detached mode? Now I am running

prefect agent local start

every time.

Hi community, I am planning to deploy to Docker Swarm, is there a more official version than https://github.com/flavienbwk/prefect-docker-compose around?

Hi all! We’re deploying the prefect ecs agent to Fargate. Works fine locally, but on Fargate we’re getting neither a successful health check nor any logs appearing in Cloudwatch. Bear in mind that we can deliberately bork the build and see logs. So I’m wondering if anyone else has had a similar experience. Would it be better to start with the docker version of the agent and try to get that working first?

hello, I'm new to Prefect, just got it running yesterday. I am coming from a history of using Airflow and was curious how Prefect users manage variable and connetions? I saw on the main Prefect docs that secrets aren't supported by the

prefect-server

and that you need to use the

config.yml

. This seems like an all around disaster waiting to happen. I am not able to use the Prefect Cloud due to where I am setting things up, so are there alternative methods people use? TIA

how, how to setup flow for not start if last execution not end yet?

I have this weird problem now with one of my lab prefect servers. I can't run jobs, they are only scheduled for 24 hours later.

Last noob question before I actually run a POC I promise "Where is an example of prefect task that just runs a pre-built Docker container." I've read the docs extensively, and I think this is covered with LocalExecutor + DockerRun, but the examples are confusing me -- they seem to all be set up for executing some prefect flow python code from the flow run on various environments. Here is my use case: • I have data transform code written in Julia, in a docker container already built with an environment to execute that Julia code, along with permissions to access s3 for read/write on either end of the execution. • I just want to schedule the running of this container (Fargate, EKS?) • Does Prefect need to be installed on the container? • Does the flow code need to execute some python commands in the container to start it off? This probably seems obvious, but its not to me. I want to use Prefect as an orchestrator for many kinds of jobs (Spark, julialang, shell calls to internal team services), and the API is full of environment-specific tasks like "AWS Tasks", "DBT Tasks", "Shell Tasks", etc. I am confused if generic docker containers can be run or if only the kind of tasks from the API documentation can be run.

Hey folks, We're currently using Server until we can get Cloud approved at a client. There's some GraphQL queries/mutations we're looking at adding to Server (more metadata aggregation, schedule updates, etc. - probably no new resolvers) and we're wondering if adding to the GraphQL Schema will be an option in Cloud as well.

Hello, we are intermittently getting “ No heartbeat detected from the remote task, marking the run as failed.” When we look at the slack channel, we detect two solutions. 1. Adding the "cluster-autoscaler.kubernetes.io/safe-to-evict": "false." 2. Another solution might be disabling the Lazarus toggle in the Prefect UI for our Plan Data Loading Flow -> Done We applied them but still getting the same error. What is your recommendation?

Hey everyone, Can someone explain to me why I keep getting this error in my flow runs 

requests.exceptions.ReadTimeout: HTTPConnectionPool(host='prefect-apollo.prefect', port=4200): Read timed out. (read timeout=15)

? It does not happen all the time but it happens so often that a lot of tasks fail because of it. Our Prefect server is deployed using Kubernetes on GCP and the pod seems to be getting enough resources.

Hey everyone, Does anyone know why I get this errors (sometimes) with scheduled tasks?

Failed to retrieve task state with error: ClientError([{'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 5}], 'path': ['get_or_create_task_run_info'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: '}}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
    task_run_info = self.client.get_task_run_info(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1399, in get_task_run_info
    result = self.graphql(mutation)  # type: Any
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 5}], 'path': ['get_or_create_task_run_info'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: '}}}]

Our Prefect server is deployed using Kubernetes on GCP with autoscaling enabled, maybe it has something to do with that?

Hi everyone I've got a prefect agent running on a Kubernetes cluster for a year or so and everything was fine, then it started to give me this error few days ago, without any change in anything:

[2021-05-07 08:16:03,101] ERROR - agent | Error while deploying flow: ValidationError({'_schema': 'Invalid data type: None'},)

I've tried to search, but couldn't find any hint on what to look, anyone could help?

Does anyone have any tips on managing environments with Prefect Cloud? Did you set up different projects for dev/qa/prod? Different flows within a single project? I'd love to hear about some examples of successful architectures if anyone has any.

Hi all, Does anyone have a good recipe that is pretty plug&play CI for deploying flows either with dind or mounting the docker sock. (preferably using gitlab.com runners)

Does anyone know if the local Prefect server is supposed to work with

DockerAgent

? I'm trying the example flow from the tutorial:

from prefect import Flow
from prefect import task
from prefect.run_configs import DockerRun


@task
def add(x, y=1):
    return x + y


with Flow("My first flow!", run_config=DockerRun()) as flow:
    first_result = add(1, y=2)
    second_result = add(x=first_result, y=100)

flow.register(project_name="test-project")

I submit the flow using the "Quick Run" button in the UI and start a Docker agent:

prefect agent docker start --show-flow-logs --log-level DEBUG

But the run forever stays in the pending state and I only ever see

Querying for flow runs / No flow runs found

in the agent logs. I've tried

DockerRun()

as above and also using a custom image containing only this:

FROM python:3.8-buster
RUN pip install prefect

The UI shows it can reach my agent, though. Any ideas?

Hi folks, we experienced a flow run failure that seems to stem from prefect cloud given the failure messages we see. Re-running the flow resolves the issue but I thought I would flag it in case it helps.

Help needed / Question -- I have a local agent running in a docker container that is properly picking up a flow i publish from another container but failing with the following error message. "Failed to load and execute Flow's environment: ValueError('Flow is not contained in this Storage')" The flow is assigned a UniversalRun (i tried a LocalRun with the same effect), a LocalExecutor, LocalStorage, and labels ABC and DEF. The Flow has labels ABC, DEF and is properly picked up by the agent.

Hi I am trying to connect to the prefect server on localhost:8080. But i am working in a firm where we use HTTP and HTTPS proxies. The prefect server doesnot connect if i don't bipass the proxy. And when I bypass the proxy the flow does not run, as one of the task is api data fetching which needs Proxies. What is the way around it? You can see in picture I am trying to set the Http and https env in the run config as well. If i don't put the os.environ to empty. Then I get the following error: "error 503: cannot connect to localhost:4200" which is my apollo docker port.

Hi Prefect family! i want see DEBUG logs in UI I tried

prefect.config.logging.level='DEBUG'

prefect.config.debug=True

and

os.environ["PREFECT__LOGGING__LEVEL"]='DEBUG'

However, it doesn't work How can i set DEBUG log level?

Hi, I am new to Prefect for task orchestration. And have couple questions here: 1. Understand that we have Prefect Cloud which is a managed service. But could we deploy prefect server as cluster? 2. Is it possible to setup a dependency between "flows" instead of "tasks"?

Is there a way to send a SUCCESS signal but continue the process running? For context i have a setup heavy process that ideally has a cleanup but as soon as the data is done, I want to continue processing. However, I don’t want to finish the task and move to another process for cleanup because that’s fairly expensive

My team is running Prefect Server on ECS (apollo, hasura, graphql, and towel as separate containers in a single ECS task; postgres as an aurora cluster; Prefect UI as another ECS task), with a single local Prefect agent also on ECS. From time to time we encounter HTTP 502 errors and/or read timeout errors, and also a flow where we had a large task for which we would get heartbeat timeouts, but after breaking it up, Prefect UI would then report that at one of the broken up tasks, the flow runs "forever". The errors seem pretty inconsistent though, but started to occur a bit more frequency recently. Prefect Server itself has remained up, so I originally thought that there was something finicky about the internal load balancer, yet it does work. Would anyone have any ideas what might be going wrong?