With the latest release, are the deployments from the examples here portable to the new version? In particular the second one where the flow file is baked into the image?

And as a follow up question, it was mentioned in several locations that blocks allow you to securely store info. Is the data somehow encrypted first then?

Trying to deploy my flow and following 2.0 tutorial. Prefect deployment build won't run because there is no build command. Looking at prefect deployment - - help there is no such command. Using 2.0b6. What did I miss?

Quick question regarding *Prefect 2.0 subflow + DaskTaskRunner*… I currently have a simple ETL flow that I define and can run sequentially no problem. I now want to use this flow in inside another flow that basically wraps and executes it for many files in parallel on a DaskTaskRunner… I know that I have to use my_task.submit() on individual tasks if I want to run them concurrently on a dask cluster. But is it possible to tell a subflow that all it’s tasks should be scheduled with a submit() after the fact? Hope this was not to confusing… Thanks y’all

Question here about the new deployment mechanism: Can I load a storage block from file instead of having to query prefect? For instance: prefect deployment build -sb <point_to_file> I want to remove any failure condition related to networking when using a CI/CD pipeline.

Hey everyone, is there any documentation on how to setup a docker-compose configuration for Prefect 2.0? I've search this slack channel, the closest I found was this repo but looks a bit outdated and non-official. https://github.com/flavienbwk/prefect-docker-compose

I'm on 0.14.22, which i know is quite behind the times at this point (definitely want to migrate to v2, but not had a chance yet).... i'm trying to use Github for storage, and need the complete repo, not just one file. According to the docs the

Git

storage clones the full repo, so I'm trying to use that rather than the

GitHub

storage as that looks to just clone the one flow file (having looked at the code). However, when using

Git

storage I get the following error as the only log in my flow:

{'type': ['Unsupported value: Git']}

In the attached link, you explain that the new storage blocks make it easier to include not just flow code in the storage but also other python dependencies and code. But - there is no explanation there (or in the link inside that leads to the documentation about deployments) for how exactly this can be done. The examples for deployments only explain how to include flow code in the storage. Are there such examples somewhere, or can you elaborate more? https://discourse.prefect.io/t/deployments-are-now-simpler-and-declarative/1255

Hey anybody got any way to wait on multiple flows at the same time that are created by

create_flow_run

? Want to do something like;

Will prefect 2.0 be supported via kubernetes GKE auto-pilot?

I’m getting some weird behaviour with mapped tasks after failure. It seems like rather than being automatically failed by the runner when an upstream task failed, they’re getting run but with a signal object

TRIGGERFAIL

passed to them as their first argument. If this sounds completely bizarre I can see if I can narrow down a minimal example

Hello! I was wondering if there’s any way to externally feed Prefect 2.0 with logs instead of directly connecting with Prefect cloud/UI

Prefect 2.0.0 Deployment - flow lives in a dedicated Python library/module. I have an internal Python library called

orchester

in which I organize all my orchestration code. In Prefect 1.x the deployment pattern was akin to this:

from orchester.flows import hello_flow
hello_flow.storage = Module(__name__ + ".flows")
hello_flow.register()

In Prefect 2.x-beta I was able to achieve something similar like this:

from orchester.flows import hello_flow
Deployment(
    name="entw",
    flow=hello_flow,
    packager=OrionPackager(serializer=ImportSerializer()),
)
# After that I created the deployment on the local Orion server with the CLI.

Now with Prefect 2.0.0 I am unsure what the deployment pattern would look like. My guess would be to run

prefect deployment build .\orchester\flows\hello_flow.py:hello_flow --name entw

from the cli and to then manually edit the

hello_flow-manifest.json

. I am not sure whether this is the the best approach or whether it needs to be done differently.

{
    "flow_name": "hello_flow",
    "import_path": ".\\orchester\\flows\\hello_flow.py:hello_flow",  # Do I just need to change this line to a module path? 
    "parameter_openapi_schema": {
        "title": "Parameters",
        "type": "object",
        "properties": {
            "name": {
                "title": "name",
                "type": "string"
            }
        },
        "required": [
            "name"
        ],
        "definitions": null
    }
}

TLDR; Does anyone have a good suggestion how to deploy flows that live in their own Python module with Prefect 2.0.0?

when will 2.0 be available for prefect cloud customers?

Hey everyone, Is there any version of Prefect 2 that is compatible with sqlalchemy version < 1.4 and >= 1.3.6? Want to try it out for my project but I’m getting compatibility issues ...

Hey, can we use local and Kubernetes both prefect agents simultaneously through different pipeline in prefect cloud

Hi Community, do we have to create deployments.yaml for each flow we submit to Prefect Cloud?

Hi, I'm running into a few issues while trying to use Prefect 2.0 block storage. I'm receiving a CORS error in Chrome when I try to delete a block in the UI at app.prefect.cloud. I have no issue creating block storage through the UI or through Python.

Access to XMLHttpRequest at '<https://api.prefect.cloud/api/accounts/><MY ACCOUNT ID>/workspaces/<MY WORKSPACE ID>/block_documents/<MY BLOCK ID>' from origin '<https://app.prefect.cloud>' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

It seems like CORS may be unhappy with the UI trying to do a cross-domain request from app.prefect.cloud to api.prefect.cloud?

Hi everyone, evaluating Prefect and hoping to get your advice on getting to a working cloud POC as fast as possible. Our situation: • Mostly Windows/Azure shop. Preference to run on Windows, all else equal, but can work with Linux VMs if necessary. • We have nothing in our ecosystem yet running on VMs (serverless only) and no Docker/Kubernetes setup. How would you suggest we set up a POC that deploys and runs flows on Azure and connects to Prefect Cloud: • POC should include demo for publishing and running flows to Azure from local dev machines • Primary goal is to set up as fast as possible: • Set up on a bare VM? Install and run under Docker? Try Azure AKS? (no experience at present with Kubernetes/AKS, but can be good case to acquire such experience in-house) Thanks much in advance!

Has anyone ever run into this issue when trying to apply a kubernetes agent?

prefect.exceptions.ClientError: Malformed response received from API.

Here is the full stack trace:

return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 305, in start
    start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 128, in start_agent
    agent.start()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 186, in start
    self._setup_api_connection()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 870, in _setup_api_connection
    self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 831, in _register_agent
    agent_config_id=self.agent_config_id,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1788, in register_agent
    tenant_id=self.tenant_id,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 264, in tenant_id
    self._tenant_id = self._get_default_server_tenant()
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 176, in _get_default_server_tenant
    response = self.graphql({"query": {"tenant": {"id"}}})
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 449, in graphql
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 405, in post
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 646, in _request
    raise ClientError("Malformed response received from API.") from exc
prefect.exceptions.ClientError: Malformed response received from API.

Here is the .yml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prefect-agent
  namespace: prefect
  name: prefect-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect-agent
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      containers:
      - args:
        - prefect agent kubernetes start --job-template "<s3://xxx>"
        command:
        - /bin/bash
        - -c
        env:
        - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
          value: ''
        - name: PREFECT__CLOUD__API
          value: <https://api.prefect.io>
        - name: NAMESPACE
          value: prefect
        - name: IMAGE_PULL_SECRETS
        - name: PREFECT__CLOUD__AGENT__LABELS
          value: '[''xxx'', ''xxxxx'']'
        - name: JOB_MEM_REQUEST
          value: ''
        - name: JOB_MEM_LIMIT
          value: ''
        - name: JOB_CPU_REQUEST
          value: ''
        - name: JOB_CPU_LIMIT
          value: ''
        - name: IMAGE_PULL_POLICY
          value: Always
        - name: SERVICE_ACCOUNT_NAME
          value: ''
        - name: PREFECT__BACKEND
          value: cloud
        - name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
          value: http://:8080
        - name: PREFECT__CLOUD__API_KEY
          value: xxx
        - name: PREFECT__CLOUD__TENANT_ID
          value: xxx
        - name: PREFECT__LOGGING__EXTRA_LOGGERS
          value: "['xxxxx', 'xxxxxxxx']"
        image: prefecthq/prefect:1.2.2-python3.7
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 2
          httpGet:
            path: /api/health
            port: 8080
          initialDelaySeconds: 40
          periodSeconds: 40
        name: agent

Hi everybody! Just started recently using Prefect and it looks great! I have a simple question which I am finding hard to find the solution. If I have a flow with a schedule, say every minute, what is the proper way to stop that flow to run and all the following runs? Maybe using states?

Hey in the prefect 2.0 docs it says

All tasks must be called from within a flow. Tasks may not call other tasks directly.

Does this mean we can't call task.run() like in 1.0? That was quite useful...

Hey all, I'm hoping I can get a little guidance on Prefect and whether it would be the right tool for what my team needs to do. We are designing a data processing workflow for the cloud. It currently exists on-prem. It works with a lot of python libraries and tools on Windows. I was planning on building each step of the process into a docker container, and then using AWS ECS (Elastic Container Service, orchestration service like Kubernetes) to manage those containers and the dataflow, and to use a custom built web app and message queue etc. to manage the workflow. Someone on my team is recommending we used Prefect instead tho. And I'm trying to figure out if Prefect is a more appropriate solution for our problem. I think what I'm most stuck on is is Prefect made to manage multiple separate tools that are built as Docker containers? I want to sandbox these data processing steps from each other in terms of their dependencies, library versions etc. But it feels like Prefect is more intended to be a monolithic app? How would you normally manage running tools that need to use python libraries with different versions in prefect, etc? Sorry if this is a bit newbish in terms of a question. My background is as a fullstack webapp developer, and not with Python. So figuring out how to properly architect a complex data processing flow with Python is a stretch for me!

I think I'm trying to wrap my head around whether Prefect is a tool that is good for our very specific problem. When I read about the case studies, etc. it sounds like it is solving different problems. An example of our problem and what I'm trying to solve: We have a multi-terabyte drive of data we need to upload to the cloud. We upload it to the cloud. We run multiple proprietary tools on servers to extract the data. Transform and process it multiple times. Say 20 different steps in one workflow. The process will break multiple times during this process and we need to inspect and potentially even manually fix problems inside the files themselves and debug / rerun individual steps. So none of this is ever customer data, we're never trying to gain insight about big data generated from users or anything. This is purely a technical GIS data processing project.

I'm trying to set up CI to build and apply deployments via GitHub Actions, but the process fails after the login command:

EOFError EOF when reading a line
Creating a profile for this Prefect Cloud login. Please specify a profile name: An exception occurred.
Error: Process completed with exit code 1.

I don't think it's possible for me to respond to a prompt like that via Actions. Is there a way around this?

Hi Community, Does anyone know how to configure dependencies at the flow level in Prefect Cloud 2.0?

Is it possible to prevent or re-schedule a flow run if the flow is still running

Each time I run

prefect deployment build

, the

deployment.yaml

file is overwritten ?

Hi. Question about the S3 storage. I got this error in order to run the command to build the deployment. [Thread]

This message was deleted.