prefect-community #prefect-community

prefect-community

John Mizerany

07/28/2022, 9:01 PM

When using the GraphQL API is there a certain way we should be passing our flow ID? I believe to be passing the correct flow ID but I get an API response of

Flow __ not found

so was not sure if passing the type wrong?

Chu

07/28/2022, 9:13 PM

return create_flow_run(…) under a task, why it gives me this :

aise TypeError("`fn` must be callable")
TypeError: `fn` must be callable

Sam Dyson

07/28/2022, 10:10 PM

Good morning community, we are really excited to start the upgrade to Orion but are having some issues with deployments. Previously during our beta PoC/migration development, we were using the python

Deployment

function to handle the deployment, as well as specifying

FilePackager

and

PickleSerializer

configurations so that we could create a single file from our (rather large) list of flow files and supporting library files. This file would then be pushed to an S3 bucket and pulled down during job execution on K8s using a standard docker image for the core dependencies. For our use-case, it doesn't make sense to upload the entire directory of 200+ files for each deployment, so we really like the ability to package our flow into a single file. We also don't want to have to create a single storage block for each flow, which would multiply the number of files we're storing in s3 (and therefore have to pull down for each flow run). The documentation now states that the

Deployment

function is no longer usable, but the YAML file and manifests provide no way to implement the

FilePackager

and

PickleSerializer

classes. My question for the prefect team - are the

FilePackager

and supporting serializers going to be available for deployments? Or is the new "upload all files" approach going to be the standard for Orion forever? If I were to hack the packager/serializer approach into our deployment methodology, will I run the risk of having the packager/serializer classes removed from the prefect package in the future? Anna has mentioned in previous comments that the prefect team is just getting started with deployments, but the lack of clarity and the fast breaking changes over the course of a week has made it quite difficult to plan our migration effectively - any clarity on this topic would be greatly appreciated!

👍 1

Austin Anderson

07/28/2022, 10:13 PM

Congrats on the 2.0 release! Just FYI, the "DockerFlowRunner" and "KubernetesFlowRunner" hyperlinks return "404" errors in the docs: https://docs.prefect.io/tutorials/docker-flow-runner/ https://docs.prefect.io/tutorials/kubernetes-flow-runner/

🙏 1

✅ 1

:gratitude-thank-you: 1

Chris Hansen

07/28/2022, 10:52 PM

Are there instructions on how to set up Kubernetes flows in Prefect Cloud 2.0?

✅ 1

Geoffrey Greenleaf

07/29/2022, 1:36 AM

Has anyone done anything with clickhouse and prefect that minds sharing some of their tasks how they are inserting data? Current process is we have a spark job that does our ETL and inserts into our sql server database. but I'm currently exploring clickhouse as our new data store and looking at prefect to kinda orchestrate and monitor our ETLs

Faheem Khan

07/29/2022, 2:30 AM

Hi all, prefect 2.0. I am getting the following error while building a deployment.

ValueError: Unable to find block document named mini1 for block type RemoteFileSystem

Pedro Machado

07/29/2022, 2:49 AM

Hi. We have a number of flows that are running on Prefect 0.14.22. I am wondering if I can progressively upgrade them to 1.2. If I run a 1.2 agent, will it work with flows written using an older version of Prefect? I know it's a weird question to ask when 2.0 just went GA 🙂

Slackbot

07/29/2022, 4:42 AM

This message was deleted.

Viet Nguyen

07/29/2022, 6:25 AM

hi everyone, I'm using Prefect 2.0 to orchestrate

xarray

dataset conversion, using

DaskTaskRunner

, the created dask client (

task_runner

) doesn't automatically pickup the

xarray

functions such as

xarray.dataset.concat()

<http://xarray.dataset.to|xarray.dataset.to>_zarr

to fill up the works to all CPU threads like without Prefect. To bring the tasks to dask client with Prefect, I have to explicitly submit the task with

@task

decorator and calling

.submit()

This way, however, disallow

xarray

functions to be executed in all CPU threads, only the wrapped task function to be executed in individual thread, which is not ideal. If anyone has come across this issue before and have a solution, much appreciated . Thanks a lot.

✅ 1

Matthias

07/29/2022, 8:03 AM

With the latest release, are the deployments from the examples here portable to the new version? In particular the second one where the flow file is baked into the image?

✅ 1

👀 1

Matthias

07/29/2022, 8:05 AM

And as a follow up question, it was mentioned in several locations that blocks allow you to securely store info. Is the data somehow encrypted first then?

flurven

07/29/2022, 9:20 AM

Trying to deploy my flow and following 2.0 tutorial. Prefect deployment build won't run because there is no build command. Looking at prefect deployment - - help there is no such command. Using 2.0b6. What did I miss?

✅ 1

Christian

07/29/2022, 9:29 AM

Quick question regarding *Prefect 2.0 subflow + DaskTaskRunner*… I currently have a simple ETL flow that I define and can run sequentially no problem. I now want to use this flow in inside another flow that basically wraps and executes it for many files in parallel on a DaskTaskRunner… I know that I have to use my_task.submit() on individual tasks if I want to run them concurrently on a dask cluster. But is it possible to tell a subflow that all it’s tasks should be scheduled with a submit() after the fact? Hope this was not to confusing… Thanks y’all

✅ 1

Rajvir Jhawar

07/29/2022, 10:14 AM

Question here about the new deployment mechanism: Can I load a storage block from file instead of having to query prefect? For instance: prefect deployment build -sb <point_to_file> I want to remove any failure condition related to networking when using a CI/CD pipeline.

✅ 1

Leonardo Melo

07/29/2022, 11:57 AM

Hey everyone, is there any documentation on how to setup a docker-compose configuration for Prefect 2.0? I've search this slack channel, the closest I found was this repo but looks a bit outdated and non-official. https://github.com/flavienbwk/prefect-docker-compose

✅ 1

Daniel Manson

07/29/2022, 12:29 PM

I'm on 0.14.22, which i know is quite behind the times at this point (definitely want to migrate to v2, but not had a chance yet).... i'm trying to use Github for storage, and need the complete repo, not just one file. According to the docs the

Git

storage clones the full repo, so I'm trying to use that rather than the

GitHub

storage as that looks to just clone the one flow file (having looked at the code). However, when using

Git

storage I get the following error as the only log in my flow:

{'type': ['Unsupported value: Git']}

Tom Klein

07/29/2022, 12:55 PM

In the attached link, you explain that the new storage blocks make it easier to include not just flow code in the storage but also other python dependencies and code. But - there is no explanation there (or in the link inside that leads to the documentation about deployments) for how exactly this can be done. The examples for deployments only explain how to include flow code in the storage. Are there such examples somewhere, or can you elaborate more? https://discourse.prefect.io/t/deployments-are-now-simpler-and-declarative/1255

✅ 1

Joshua Greenhalgh

07/29/2022, 1:04 PM

Hey anybody got any way to wait on multiple flows at the same time that are created by

create_flow_run

? Want to do something like;

James Phoenix

07/29/2022, 1:24 PM

Will prefect 2.0 be supported via kubernetes GKE auto-pilot?

Eli Treuherz

07/29/2022, 2:00 PM

I’m getting some weird behaviour with mapped tasks after failure. It seems like rather than being automatically failed by the runner when an upstream task failed, they’re getting run but with a signal object

TRIGGERFAIL

passed to them as their first argument. If this sounds completely bizarre I can see if I can narrow down a minimal example

07/29/2022, 2:13 PM

Hello! I was wondering if there’s any way to externally feed Prefect 2.0 with logs instead of directly connecting with Prefect cloud/UI

👍 1

✅ 1

Sebastian

07/29/2022, 2:27 PM

Prefect 2.0.0 Deployment - flow lives in a dedicated Python library/module. I have an internal Python library called

orchester

in which I organize all my orchestration code. In Prefect 1.x the deployment pattern was akin to this:

from orchester.flows import hello_flow
hello_flow.storage = Module(__name__ + ".flows")
hello_flow.register()

In Prefect 2.x-beta I was able to achieve something similar like this:

from orchester.flows import hello_flow
Deployment(
    name="entw",
    flow=hello_flow,
    packager=OrionPackager(serializer=ImportSerializer()),
)
# After that I created the deployment on the local Orion server with the CLI.

Now with Prefect 2.0.0 I am unsure what the deployment pattern would look like. My guess would be to run

prefect deployment build .\orchester\flows\hello_flow.py:hello_flow --name entw

from the cli and to then manually edit the

hello_flow-manifest.json

. I am not sure whether this is the the best approach or whether it needs to be done differently.

{
    "flow_name": "hello_flow",
    "import_path": ".\\orchester\\flows\\hello_flow.py:hello_flow",  # Do I just need to change this line to a module path? 
    "parameter_openapi_schema": {
        "title": "Parameters",
        "type": "object",
        "properties": {
            "name": {
                "title": "name",
                "type": "string"
            }
        },
        "required": [
            "name"
        ],
        "definitions": null
    }
}

TLDR; Does anyone have a good suggestion how to deploy flows that live in their own Python module with Prefect 2.0.0?

✅ 1

👀 4

Zach Schumacher

07/29/2022, 2:54 PM

when will 2.0 be available for prefect cloud customers?

✅ 1

Muhammad Mikaal Saleem Anwar

07/29/2022, 3:27 PM

Hey everyone, Is there any version of Prefect 2 that is compatible with sqlalchemy version < 1.4 and >= 1.3.6? Want to try it out for my project but I’m getting compatibility issues ...

Hafsa Junaid

07/29/2022, 3:43 PM

Hey, can we use local and Kubernetes both prefect agents simultaneously through different pipeline in prefect cloud

✅ 1

Jeffrey Lai

07/29/2022, 4:27 PM

Hi Community, do we have to create deployments.yaml for each flow we submit to Prefect Cloud?

Andrew Richards

07/29/2022, 5:28 PM

Hi, I'm running into a few issues while trying to use Prefect 2.0 block storage. I'm receiving a CORS error in Chrome when I try to delete a block in the UI at app.prefect.cloud. I have no issue creating block storage through the UI or through Python.

Access to XMLHttpRequest at '<https://api.prefect.cloud/api/accounts/><MY ACCOUNT ID>/workspaces/<MY WORKSPACE ID>/block_documents/<MY BLOCK ID>' from origin '<https://app.prefect.cloud>' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

It seems like CORS may be unhappy with the UI trying to do a cross-domain request from app.prefect.cloud to api.prefect.cloud?

✅ 1

Oleg Sheyner

07/29/2022, 5:29 PM

Hi everyone, evaluating Prefect and hoping to get your advice on getting to a working cloud POC as fast as possible. Our situation: • Mostly Windows/Azure shop. Preference to run on Windows, all else equal, but can work with Linux VMs if necessary. • We have nothing in our ecosystem yet running on VMs (serverless only) and no Docker/Kubernetes setup. How would you suggest we set up a POC that deploys and runs flows on Azure and connects to Prefect Cloud: • POC should include demo for publishing and running flows to Azure from local dev machines • Primary goal is to set up as fast as possible: • Set up on a bare VM? Install and run under Docker? Try Azure AKS? (no experience at present with Kubernetes/AKS, but can be good case to acquire such experience in-house) Thanks much in advance!

👀 4

Jake

07/29/2022, 6:46 PM

Has anyone ever run into this issue when trying to apply a kubernetes agent?

prefect.exceptions.ClientError: Malformed response received from API.

Here is the full stack trace:

return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 305, in start
    start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 128, in start_agent
    agent.start()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 186, in start
    self._setup_api_connection()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 870, in _setup_api_connection
    self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 831, in _register_agent
    agent_config_id=self.agent_config_id,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1788, in register_agent
    tenant_id=self.tenant_id,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 264, in tenant_id
    self._tenant_id = self._get_default_server_tenant()
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 176, in _get_default_server_tenant
    response = self.graphql({"query": {"tenant": {"id"}}})
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 449, in graphql
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 405, in post
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 646, in _request
    raise ClientError("Malformed response received from API.") from exc
prefect.exceptions.ClientError: Malformed response received from API.

Here is the .yml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prefect-agent
  namespace: prefect
  name: prefect-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect-agent
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      containers:
      - args:
        - prefect agent kubernetes start --job-template "<s3://xxx>"
        command:
        - /bin/bash
        - -c
        env:
        - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
          value: ''
        - name: PREFECT__CLOUD__API
          value: <https://api.prefect.io>
        - name: NAMESPACE
          value: prefect
        - name: IMAGE_PULL_SECRETS
        - name: PREFECT__CLOUD__AGENT__LABELS
          value: '[''xxx'', ''xxxxx'']'
        - name: JOB_MEM_REQUEST
          value: ''
        - name: JOB_MEM_LIMIT
          value: ''
        - name: JOB_CPU_REQUEST
          value: ''
        - name: JOB_CPU_LIMIT
          value: ''
        - name: IMAGE_PULL_POLICY
          value: Always
        - name: SERVICE_ACCOUNT_NAME
          value: ''
        - name: PREFECT__BACKEND
          value: cloud
        - name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
          value: http://:8080
        - name: PREFECT__CLOUD__API_KEY
          value: xxx
        - name: PREFECT__CLOUD__TENANT_ID
          value: xxx
        - name: PREFECT__LOGGING__EXTRA_LOGGERS
          value: "['xxxxx', 'xxxxxxxx']"
        image: prefecthq/prefect:1.2.2-python3.7
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 2
          httpGet:
            path: /api/health
            port: 8080
          initialDelaySeconds: 40
          periodSeconds: 40
        name: agent

✅ 1

Title

Jake

07/29/2022, 6:46 PM

Has anyone ever run into this issue when trying to apply a kubernetes agent?

prefect.exceptions.ClientError: Malformed response received from API.

Here is the full stack trace:

return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 305, in start
    start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 128, in start_agent
    agent.start()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 186, in start
    self._setup_api_connection()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 870, in _setup_api_connection
    self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 831, in _register_agent
    agent_config_id=self.agent_config_id,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1788, in register_agent
    tenant_id=self.tenant_id,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 264, in tenant_id
    self._tenant_id = self._get_default_server_tenant()
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 176, in _get_default_server_tenant
    response = self.graphql({"query": {"tenant": {"id"}}})
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 449, in graphql
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 405, in post
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 646, in _request
    raise ClientError("Malformed response received from API.") from exc
prefect.exceptions.ClientError: Malformed response received from API.

Here is the .yml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prefect-agent
  namespace: prefect
  name: prefect-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect-agent
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      containers:
      - args:
        - prefect agent kubernetes start --job-template "<s3://xxx>"
        command:
        - /bin/bash
        - -c
        env:
        - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
          value: ''
        - name: PREFECT__CLOUD__API
          value: <https://api.prefect.io>
        - name: NAMESPACE
          value: prefect
        - name: IMAGE_PULL_SECRETS
        - name: PREFECT__CLOUD__AGENT__LABELS
          value: '[''xxx'', ''xxxxx'']'
        - name: JOB_MEM_REQUEST
          value: ''
        - name: JOB_MEM_LIMIT
          value: ''
        - name: JOB_CPU_REQUEST
          value: ''
        - name: JOB_CPU_LIMIT
          value: ''
        - name: IMAGE_PULL_POLICY
          value: Always
        - name: SERVICE_ACCOUNT_NAME
          value: ''
        - name: PREFECT__BACKEND
          value: cloud
        - name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
          value: http://:8080
        - name: PREFECT__CLOUD__API_KEY
          value: xxx
        - name: PREFECT__CLOUD__TENANT_ID
          value: xxx
        - name: PREFECT__LOGGING__EXTRA_LOGGERS
          value: "['xxxxx', 'xxxxxxxx']"
        image: prefecthq/prefect:1.2.2-python3.7
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 2
          httpGet:
            path: /api/health
            port: 8080
          initialDelaySeconds: 40
          periodSeconds: 40
        name: agent

✅ 1

Nevermind, I found the issue! was using the wrong tenant value

Mason Menges

07/29/2022, 7:01 PM

Fantastic 😄

View count: 3