What are the capabilities of perfect?

Is it possible to extend the prefect Flow with my own flow class that implements some common patterns? If I do that, and I use my own flow class in the context manager during a flow creation on the client, which downstream services need that flow class code? I am seeing this in my prefect server ui:

Last State Message
[9:35am]: Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'rightsize'")

I'm working on getting Results to be available for registered flows. I have a server deployed that is executing my flows nicely with a Dask cluster, and can run those flows when they are not registered. When I register a flow and create a flow run, I don't seem to be able to get a hold of any Results.

using the Client class, how do i use the flow name rather that flow uuid? e.g.

client.create_flow_run("a14f9774-e94d-40cc-b019-cb9044e120d4", run_name=f"manually_run_flow")

I understand that there can be multiple versions of the same flow. However, is there a way to query the latest version of UUID for a given flow from the client class itself and not using the graphql api? Thanks

Hi! I have defined a flow and can run it with

flow.run()

. However, these runs do not show up in the Prefect Dashboard, even when I

.register()

them (the flow shows up in the web interface and I can run it from there, but starting it with

.run()

does not show them as a Run). Is there a way to make them show up without running an agent?

If I have the server running locally (with

prefect backend server

), is there a way to run a flow every 10 seconds. Right now, I take an error saying that the cloud runner's minimum time delta is 1 minute? Thanks

Hey there, I’ve just started exploring Prefect locally with the Prefect OSS server and I came across a couple of basic issues: • I can’t create any project from the local UI: I get a failure message (with no explanation and nothing specific in the logs the docker containers). I could create the project tho by direct insertion in the PG database. • When I run a local agent locally, it registers properly with the graphql server but it doesn’t show up in the local ui agent dashboard (or I didn’t find that part of the dashboard) • After registering a flow, I can’t get it to start from the UI, it complains about

tenant

being

None

🤷‍♂️

^^ ok found the issue, since I could not create a project from the UI when creating it directly in PG I had to attach it to a valid tenant (default one actually)… too bad the project creation didn’t work from the UI

Hi everyone, I would like to use specific version of Prefect. When testing things locally

prefect server start --version 0.13.2

gives me an error. It's looking for the manifest for

prefecthq/apollo:0.13.2

which doesn't exist. Using the

latest

works, but I would rather not use the latest tag in a production system as it may change when you don't want to. Looking at tags of prefecthq/server, prefecthq/apollo and prefecthq/ui the latest common version was

0.12.6

. Where can I find which versions of those Prefect components are compatible?

set the channel topic: https://github.com/PrefectHQ/server

Hi. We just tried setting up a clean install with 0.13.2 and the ui is not finding a tenant, and the create-default-tenant command noted in the web ui doesn't seem to be working.

Just a comment we have a running list of fixes to apply when spinning up prefect server. 1. docker-compose in the conda environment (as its not a dependency, perhaps not something that should be fixed) 2. sed on the docker-compose.yml to deal with the fact that we use a proxy for the docker repos 3. sed in the running server container to adjust the graphql url. (not sure if handling of an environment variable fixes this, but haven't seen it working so far) That last point also fixed the missing tenant error.

Hi friends, is there any helm chart for Prefect Server or a guide to installing on Kubernetes?

Hi all, Now that I have my flow running on a local server working beautifully, I would like to transition to the Cloud. I already have a remote server up and running on compute engine on GCP. So what are the next steps for registering a flow to a remote Prefect Server hosted on GCP? I should also add that I have my flows stored as docker images on GCS. Thanks in advance.

Hi Everyone I've just started trying out the Prefect Cloud and unfortunately I can't seem to get any tasks to run I have a simple task that looks like this:

from datetime import timedelta

import prefect
from prefect import Flow, task
from prefect.schedules import IntervalSchedule

prefect.config.debug = True
prefect.config.logging.level = "DEBUG"

@task(name="orchestrate", log_stdout=True)
def orchestrate():
    logger = prefect.context["logger"]
    logger.info("Orchestrating")
    for i in range(1, 60):
        logger.error(f"Counting {i}")
    return True


flow = Flow("Cloud Flow", schedule=IntervalSchedule(
    interval=timedelta(minutes=1),
))
with flow:
    orchestrate()

flow.register(project_name="dev")
flow.run_agent(token="XXXX")

And I start the agent with:

python example.py

Agent seems to come up fine but just hangs:

(venv) ibraflow (master|✚5…) python example.py
Result check: OK
Flow: <https://cloud.prefect.io/ysft/flow/b7f56496-54df-4eeb-a0b6-2e274c39ed80>

 ____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                           |___/

[2020-08-16 10:34:38,874] INFO - agent | Starting LocalAgent with labels ['MacBook-Pro-2.local', 'azure-flow-storage', 'gcs-flow-storage', 's3-flow-storage', 'github-flow-storage', 'webhook-flow-storage']
[2020-08-16 10:34:38,874] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-08-16 10:34:38,875] INFO - agent | Agent connecting to the Prefect API at <https://api.prefect.io>
[2020-08-16 10:34:39,009] INFO - agent | Waiting for flow runs...

On the cloud's UI I see the correct version and task, so that rules out authentication and network issues? But the tasks never run and are always marked "late" - attachment:

Hello! Hope everyone had a good weekend. Are there any plans to release a tutorial for adding security to the Prefect Core Server that's deployed to GCP? I'm looking into adding an nginx reverse proxy on my GCP Compute Engine VM where the Prefect Server is hosted but no success. Not sure if anyone has done something like that before and can point me to some available online resources? Would be much appreciated.

Hello prefect team !! With the new prefect UI (that was released maybe 3 weeks ago), I am getting this error: "Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:4200/graphql. (Reason: CORS request did not succeed)." Any idea what is going on? Thanks

Hi, I have been asked to deploy Prefect by our Data team. Since we operate in AWS I would like to use Aurora Postgres for the database and Fargate for the containers. Is there some documentation on this deployment?

Hi Folks, is there a link to monitor the performance of prefect cloud i.e. to track any service issues if/when they arise ?

Hello folks, I am new to prefect and trying to build a process to load millions of files from a FTP location to one of the local database. While doing this we have the typical ETL transformation which happens at each file. My question here is on how can i design this using prefect? Right now we have a in house process which leverages Celery to queue and process the files but the way it runs currently, it downloads all the files and then perform the ETL process on each one of it in parallel. Shall we maintain it the same way or download each file and then perform ETL operation? Maintaining error files state even if retry logic fails. Also the current time which this process takes is about 20-24 hours as it processes Billions of records. Any insight on how to design this would be helpful.

Hey Prefect team, I"m having issues using docker storage when the base_image is being pulled from my team's on-prem repository; it seems docker-py doesn't support that at the moment?

I'm getting an error when the graphql service tries to run it's Alembic migrations against an RDS postgres instance. Has anyone seen anything like this before?

Error: (psycopg2.errors.SyntaxError) syntax error at or near "REFERENCING"
LINE 61: ...rting_state AFTER INSERT ON public.flow_run_state REFERENCIN...

Hey team! I'm having an issue with running prefect server start, but against my private docker registry. how do i configure this?

Hey Team, i'm having an issue when running "prefect server create-tenant". I am getting the error [requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=4200): Read timed out. (read timeout=30)], however i have set the environment variable GRAPHQL_URL to my hostname where it's hosted; it seems not to be registering the env variable. Any advice on how to properly change the Graphql url?

Question team, if i have multiple agents with the same label, how does the cloud-server / core-server choose which agent to execute on? is it arbitrary?

i have trouble running dask on prefect, https://github.com/PrefectHQ/prefect/issues/3226 can anyone helps? i did try similar example using older version of prefect and it was working

Unexpected error: AttributeError("'NoneType' object has no attribute 'is_finished'")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 359, in check_upstream_finished
    if not all(s.is_finished() for s in all_states):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 359, in <genexpr>
    if not all(s.is_finished() for s in all_states):
AttributeError: 'NoneType' object has no attribute 'is_finished'

any comment?

Hey Team, I have a query, I am running flow.register() from within the Flask app and I could see as many time Flask app is run, I could see new version from UI. How do I avoid re-registering of the flow? Thanks

Hey folks - I'm trying to get prefect server set up on google cloud compute. i'm looking through the tutorials on youtube and medium - I'm confused why the graphql url has to be the external IP. if you're just running on one node couldn't it just be on localhost? also, any instructions/tips on how to secure an instance? I could connect to the ui through an ssh tunnel but I'd rather not have anything publicly accessible if I can help it

Hey everyone, I'm trying to run the cloud tutorial "Running a Flow", I've authenticated using my personal token and then run the agent with its token and this works fine, but "Run Demo Flow" simply results in an error message

"Failed to start flow run. Please wait a few moments and try again"

😅 Any ideas with what could be wrong? Searching google and this channel didn't turn anything up :/

Are logs from flow runs stored in the postgres instance for the prefect server? If so, is there any kind of retention policy or cleanup process in place to delete them if they pass a certain age? I'm wondering about running out of space as runs accumulate in the history