Hi Team, I am new to prefect & currently got assigned to perform the necessary setup. I am using helm prefect server repo : https://github.com/PrefectHQ/prefect Somehow the agent ( prefect:0.14.5 tag) is not coming up & I am getting "No Tenant found" in pod logs. Can someone please suggest what I am missing here?

Hi all, I'm having a bit of trouble getting prefect server to work with github. The environment variable where the agent is running is set

PREFECT__CONTEXT__SECRETS__GITHUB_ACCESS_TOKEN= <my_token>

The flow is

storage = GitHub(
    repo="my_repo",
    path="simple_flow.py",
    secrets=["GITHUB_ACCESS_TOKEN"]
)

with Flow("github_flow", storage=storage) as f:

If I remove storage and run it, it runs fine. With storage, it hangs indefinitely on

scheduled

. Any ideas?

Hi Prefect community, I'm having trouble getting the Docker Agent working when deployed in a Docker container. I have registered a flow with Docker storage in the cloud. I'm building the Docker image in a slightly non-standard way, but the flow works fine as long as I execute it though a Docker agent running on my laptop (i.e. not in a docker container). Next, I try to deploy an agent using docker stack and a simple docker-compose.yml, but this seems unable to execute the same flow correctly. Here is what I see: - There is connection to cloud, agent says: "Found 1 flow run(s) to submit for execution." - There is connection to the private registry where flow is stored, agent says: "Successfully pulled image *****/template-flow:dbea2161" - I think flow container works, agent says: " Docker container ***** started" - Strangely: - agent reports job complete (?): "Completed flow run submission (id: a507921f-b9c6-400a-943f-a984b99eadbc)" - cloud UI shows both tasks for that flow run as pending. If I enable " --show-flow-logs" for the Docker-agent-in-Docker, the agent prints below traceback (after above list of output). So my guess is that something inside my flow container is trying to talk to the cloud API, but somehow failing due to missing credentials. I don't understand why it's failing in the Docker-agent-in-Docker case and not in the Docker-agent-from-terminal case. Any hints appreciated.

Traceback (most recent call last):
   File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 451, in _request
     json_resp = response.json()
   File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 900, in json
     return complexjson.loads(self.text, **kwargs)
   File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads
     return _default_decoder.decode(s)
   File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
   File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
     raise JSONDecodeError("Expecting value", s, err.value) from None
 json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
 
 The above exception was the direct cause of the following exception:
 
 Traceback (most recent call last):
   File "/usr/local/bin/prefect", line 8, in <module>
     sys.exit(cli())
   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
     return self.main(*args, **kwargs)
   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
     rv = self.invoke(ctx)
   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
     return _process_result(sub_ctx.command.invoke(sub_ctx))
   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
     return _process_result(sub_ctx.command.invoke(sub_ctx))
   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
     return ctx.invoke(self.callback, **ctx.params)
   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
     return callback(*args, **kwargs)
   File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 49, in flow_run
     result = client.graphql(query)
   File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 298, in graphql
     result = <http://self.post|self.post>(
   File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 213, in post
     response = self._request(
   File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 454, in _request
     raise ClientError(
 prefect.utilities.exceptions.ClientError: Malformed response received from Cloud - please ensure that you have an API token properly configured.

How can I set the logging level of the prefect graphql service? I tried adding both

PREFECT_SERVER__LOGGING__LEVEL: WARN

and

PREFECT__LOGGING__LEVEL: WARN

environment variables to the docker-compose file, however I still see

INFO

log messages.

Is it possible to create your own executor, esp., one in a different language? I am concerned about how Python centric everything looks - I’m looking for the ability to run both Python and Julia tasks (the Julia tasks very well might be calling into Python via PyCall, so I’d need to make sure that that could be handled also)

Hi. I'm new to prefect (and docker by the way). I'm able to start prefect server via terminal inside Pycharm. I'm starting a local agent from a terminal in Pycharm also. Then I'm registering a flow. I open the UI and the flow is correctly registered and scheduled. I'm letting the flow run for a few times and it executes successfully. Then I close Pycharm and that terminates the terminals where I have started prefect server and the agent. If I'm understanding it well, prefect server is still running on a docker container (I can see that via Docker Desktop) but I need to start a new agent. So I run a python script to start the agent and it is successful:

from prefect.agent.local import LocalAgent

LocalAgent().start()

After that, all scheduled executions of the flow are failing with:

Failed to load and execute Flow's environment: StorageError('An error occurred while unpickling the flow:\n ModuleNotFoundError("No module named \'src\'")\nThis may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')

What am I doing wrong? I am using Windows 10, python 3.9 and a virtual environment (venv).

Q: Is the Prefect server multitenant i.e. multi users with different logins can submit workflow thru a single server instance??

Hello guys, I just did something really silly, I ran into an infinite loop which triggered all flows to run. Luckily I could stop that. However, my run history is somewhat spoiled now (see picture). Any way I can reset the run history?

has anyone ever run into this issue before? i was running

prefect server start

just fine the other day, and in the last 10-15 minutes when I've tried running it, i'm getting sporadic errors for the various docker compose services that need to be pulled. one time i might get

ERROR: for apollo unauthorized: authentication required

, another time i get

ERROR: for postgres unauthorized: authentication required

, a 3rd time i get the same for

graphql

, or some combination of the above

If anyone is currently experiencing authentication/authorization issues with pulling the Prefect server images it is likely due to the current DockerHub service interruptions where the images are being hosted https://status.docker.com/pages/533c6539221ae15e3f000031

Hi all, I'm trying to use a remote Kubernetes cluster to do my computations, but submit them directly from a local machine without having to first register a flow, and after the flow has run I want to avoid having to collect the results, eg from GCS (so I would like the computations to be interactive). Is this possible with Prefect? Thank you (The KubernetesAgent and Prefect Server are both running in-cluster).

Is there anyway to batch mapped tasks? Specifically I have a flow that will kick of many runs of another parametrised run but I need to minimise the number of concurrent runs

does a prefect task have any knowledge of what project context it's being executed in?

Hi there, I've got a question about

flow.register( idempotency_key=flow.serialized_hash(),...)

. If I register my flows with this method, then, if I change the code of a single task, it creates no new version. Only if I make a change at the flow itself. Is it possible to make a new version if the code of the task changes (but not every time, like without idempotency_key)? Thanks

Hello fellow prefect user, have you guys ever experienced this type of error?

Unexpected error: AttributeError("'StartFlowRun' object has no attribute 'run_config'")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 867, in get_task_run_state
    logger=self.logger,
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 298, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 453, in method
    kwargs.setdefault(attr, getattr(self, attr))
AttributeError: 'StartFlowRun' object has no attribute 'run_config'

Hi there :) My Problem is: the prefect agent on the server doesn't deploy my flow run. (he's only waiting for flow runs) In the UI the flow is registered and scheduled, but the agent does not execute it. I see the flow if I do: prefect get flow-runs

> prefect get flow-runs

NAME FLOW NAME STATE AGE START TIME ID

spiffy-scorpion Load Scheduled 13 minutes ago 17a13e07-d217-4a10-a821-5947657a1a48

I use the local agent. In the UI can see the agent and also if I stop the UI, the agent complains. Also the prefect server start env is used for the whole setup. Would be very amazing to get some help. Thanks

Hi. I have a flow that passes a results object (the results of session.query - sqlalchemy) between tasks. If I run the flow with flow.run() it executes ok. If I run it from the UI it fails with this error:

Unexpected error: TypeError("cannot pickle 'weakref' object")
Traceback (most recent call last):
  File "d:\302_investing\venv\lib\site-packages\prefect\engine\runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "d:\302_investing\venv\lib\site-packages\prefect\engine\task_runner.py", line 891, in get_task_run_state
    result = self.result.write(value, **formatting_kwargs)
  File "d:\302_investing\venv\lib\site-packages\prefect\engine\results\local_result.py", line 116, in write
    value = self.serializer.serialize(new.value)
  File "d:\302_investing\venv\lib\site-packages\prefect\engine\serializers.py", line 70, in serialize
    return cloudpickle.dumps(value)
  File "d:\302_investing\venv\lib\site-packages\cloudpickle\cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "d:\302_investing\venv\lib\site-packages\cloudpickle\cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
TypeError: cannot pickle 'weakref' object

Why the different behavior between Core and Server?

Has anyone successfully launched an agent using docker-compose that talks to the apollo server? Screenshotting my current agent container in case theres an obvious error - currently it runs and connects but maybe the config is messed up because it never finds any flows to run. If I run

prefect local agent start --label any

in my terminal, it works beautifully. (Edit: Actually now when I run it its stuck on “Waiting for flow runs”). Inside a docker container (using local or docker agents) - connects fine by then always says “No flow runs found”. This is the same whether I run a docker or local agent in the docker-compose, with and without attaching labels to anything, with both local storage and S3, and with local, universal and docker run configs explicitly set.

I am running a very simple flow with a custom executor (I have an executor that is similar to Dask for a very large private compute cluster). When I run the flow locally (

flow.run()

) it successfully completes and using diagnostic tools for my cluster I can see the jobs complete. I can also register the flow with the server (

flow.register

). But when I quick run the flow, from the UI’s perspective it looks as if it never completes (it is stuck in the “running” state). Using my cluster diagnostics I can see the actual jobs get scheduled and complete. So I’d guess whatever sort of events are meant to be sent to the server in order to indicate state changes in the jobs are not being sent? How can I diagnose what’s happening?

Hi everyone, I am new to prefect, when using kubernetes agent, tasks got stuck at scheduled state. Same works fine with local agent.

Hi there, I'm new in Prefect and I'm facing some issues in an Flow I need to implement. I need a Flow with few tasks in sequence (Task1 -> Task2 -> Task3). But depending on diferent conditions each Task can be SKIPPED according with its respective condition. I need each task (that are not skipped) process a dataset and pass this data precessed to the next NOT SKIPPED task. There a two main issues: 1. If Task1 and Task3 a forced to run, and Task2, in the middle, forced to be SKIPPED, the downstream Task3 is SKIPPED too (if skip_on_upstream_skip=True). Although, if I set skip_on_upstream_skip to False (in all 3 tasks), Task2 is NOT skipped as should be [image].  2. How can I keep the data flow through a sequence of tasks, if one of then is skipped? When a task is skipped, I lost the data flow through it. Woudl be the best practice use a shared path for all tasks (a hive table that is read and overwrited for each task, for example)  is OR there is a more Prefectonic way to do that? May someone help me, please?

with Flow("conditional-branches") as flow:

    cond1 = check_condition1()
    with case(cond1, True):
        val1 = task1()

    cond2 = check_condition2()
    with case(cond2, True):
        val2 = task2()

    cond3 = check_condition3()
    with case(cond3, True):
        val3 = task3()



    flow.set_dependencies(
    task=val1,
    downstream_tasks=[val2])

    flow.set_dependencies(
    task=val2,
    downstream_tasks=[val3])

Hi Everyone, I am sure I am making a small mistake here somewhere. After I run creating the tenant command

$ prefect backend server && prefect server create-tenant --name default --slug default

This is the error I get on my CLI

raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'message': '[Errno -2] Name or service not known', 'locations': [{'line': 2, 'column': 5}], 'path': ['create_tenant'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': '[Errno -2] Name or service not known'}}}]

I am getting this error in my prefect-graphql container

2021-02-07 01:57:13[Errno -2] Name or service not known7e86527e64b44aa992ab290af4938b13
2021-02-07 01:57:13GraphQL request:2:37e86527e64b44aa992ab290af4938b13
2021-02-07 01:57:131 | mutation ($input: create_tenant_input!) {7e86527e64b44aa992ab290af4938b13
2021-02-07 01:57:132 | create_tenant(input: $input) {7e86527e64b44aa992ab290af4938b13
2021-02-07 01:57:13| ^7e86527e64b44aa992ab290af4938b13
2021-02-07 01:57:133 | id7e86527e64b44aa992ab290af4938b13
2021-02-07 01:57:13Traceback (most recent call last):7e86527e64b44aa992ab290af4938b13
2021-02-07 01:57:13File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 628, in await_result7e86527e64b44aa992ab290af4938b13
2021-02-07 01:57:13return await result7e86527e64b44aa992ab290af4938b13
2021-02-07 01:57:13File "/prefect-server/src/prefect_server/graphql/extensions.py", line 52, in resolve7e86527e64b44aa992ab290af4938b13
2021-02-07 01:57:13result = await result7e86527e64b44aa992ab290af4938b13

I have setup prefect-server on AWS Fargate ECS using the docker-compose.yml as a guidleine to create my own task definition where I am doing a multi-container setup with ENV defined in Key, Values of each container. I think there is some issue in ENV Hasura Admin Secret in prefect-graphql, prefect-apollo as I see some errors in headers while the tenant is being created. So I have removed the Hasura Admin password from my prefect-hasura container. I am still getting the above errors. ANY HELP IN THIS REGARD WILL BE APPRECIATED...I THINK I AM 1 STEP AWAY FROM SEEING PREFECT-SERVER RUN AGAIN 🙂 I HAD IT RUNNING BEFORE BUT HIT THIS ISSUE.

Did anyone else noticed the issue that a KubernetesAgent is labeled as unhealthy in the UI but really isn't? It happens when the agent loses it's connection with the server, reconnects after a while but in the UI still shows as an unhealthy agent. It is picking up flows so the agent does work as expected. The UI just doesn't reflect the state correct. If I restart the pod where the agent runs on and it connects after this restart the agents is showing as healthy again. I'm running Prefect server version 0.14.3. Or might this be something in my configuration? We have only 1 agent running.

I'm having a pretty buggy experience on Prefect Cloud lately - the page will hang/freeze and now often redirect to cloud.prefect.io/404

I'm having trouble connecting to graphql via the Prefect Server webUI. I can reach it on localhost:4200, <ip>:4200, and <fully-qualified-name>:4200, but I always get

Oops! It looks like something went wrong when trying to connect; make sure Prefect Server is running at the URL above and try again.

Hi all, I’m having trouble running the hello-world flow based on the tutorial (First Flow). It’s been modified and runs on all my coworkers computers but for some reason, when I run it, I get the

Name or service not known

error message. I confirmed that GraphQL is not running based on the error message on the top right of the prefect UI. Stack trace:

apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...
apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...
graphql_1   | 
graphql_1   | Running Alembic migrations...
graphql_1   | INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
graphql_1   | INFO  [alembic.runtime.migration] Will assume transactional DDL.
graphql_1   | 
graphql_1   | Could not upgrade the database!
graphql_1   | Error: HTTPConnectionPool(host='hasura', port=3000): Max retries exceeded with url: /v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x40133ab7d0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...

Based on the stack trace, it confirms that there is an issue with graphql, I’m just not sure how to dive deeper. I have tried Dagster and initially came across some issues with compatibility with the M1 chip so I created an EC2 instance and tried the prefect first flow tutorial in that one as well. I hit an issue with docker compose not being found so rather than splitting into two separate paths, I’m going to try to focus on getting the first flow tutorial working on my local instance (macos)

What happens if I update Prefect (or restart it) while running a flow (specifically Kubernetes Agent)? Will the new agent correctly monitor the flow or could it introduce undefined behhavior?

I’m having trouble finding this in the docs - is there a way to specify package dependencies for a run? I see you can use a kubernetesExecutor like airflow and specify a docker image, but that would involve rebuilding your image every time dependencies change. Is there a way to include a

requirements.txt

or

conda.yaml

with a job? Thanks!

Hi there 🙂 One question about dynamic flows (build at runtime) and the prefect server/cloud. Our ETL process contains entities to load (stored in the database). The list of these entities can change over time (because of development). How can I achieve a dynamic load, which will load all entities existing at runtime and be able to manage this via the Server/Cloud? I can not use the task-mapping function, because the entities have own dependencies (also stored in the database) and these will be dynamically resolved at runtime. Is there a way to solve this kind of problem with prefect/UI? I developed this with the localDaskRun and the explicit run-call. Then its working, because the flow is built everytime. Now via Server, I got errors when the task is changing:

KeyError: 'Task slug XX not found in the current Flow; this is usually caused by changing the Flow without reregistering it with the Prefect API.'

Best Michael

Hi all, I am looking to use the helm chart to deploy on a AWS EKS cluster. Having little knowledge and spending some time reading through the docs, I was wondering if anyone else has done this and have any notes they can send me so I can attempt to set this up today? Thanks in advance, any advice would be appreciated!