https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • j

    John Mizerany

    07/28/2022, 9:01 PM
    When using the GraphQL API is there a certain way we should be passing our flow ID? I believe to be passing the correct flow ID but I get an API response of
    Flow __ not found
    so was not sure if passing the type wrong?
    j
    • 2
    • 3
  • c

    Chu

    07/28/2022, 9:13 PM
    return create_flow_run(…) under a task, why it gives me this :
    aise TypeError("`fn` must be callable")
    TypeError: `fn` must be callable
    j
    • 2
    • 6
  • s

    Sam Dyson

    07/28/2022, 10:10 PM
    Good morning community, we are really excited to start the upgrade to Orion but are having some issues with deployments. Previously during our beta PoC/migration development, we were using the python
    Deployment
    function to handle the deployment, as well as specifying
    FilePackager
    and
    PickleSerializer
    configurations so that we could create a single file from our (rather large) list of flow files and supporting library files. This file would then be pushed to an S3 bucket and pulled down during job execution on K8s using a standard docker image for the core dependencies. For our use-case, it doesn't make sense to upload the entire directory of 200+ files for each deployment, so we really like the ability to package our flow into a single file. We also don't want to have to create a single storage block for each flow, which would multiply the number of files we're storing in s3 (and therefore have to pull down for each flow run). The documentation now states that the
    Deployment
    function is no longer usable, but the YAML file and manifests provide no way to implement the
    FilePackager
    and
    PickleSerializer
    classes. My question for the prefect team - are the
    FilePackager
    and supporting serializers going to be available for deployments? Or is the new "upload all files" approach going to be the standard for Orion forever? If I were to hack the packager/serializer approach into our deployment methodology, will I run the risk of having the packager/serializer classes removed from the prefect package in the future? Anna has mentioned in previous comments that the prefect team is just getting started with deployments, but the lack of clarity and the fast breaking changes over the course of a week has made it quite difficult to plan our migration effectively - any clarity on this topic would be greatly appreciated!
    👍 1
    a
    n
    k
    • 4
    • 4
  • a

    Austin Anderson

    07/28/2022, 10:13 PM
    Congrats on the 2.0 release! Just FYI, the "DockerFlowRunner" and "KubernetesFlowRunner" hyperlinks return "404" errors in the docs: https://docs.prefect.io/tutorials/docker-flow-runner/ https://docs.prefect.io/tutorials/kubernetes-flow-runner/
    🙏 1
    ✅ 1
    :gratitude-thank-you: 1
    a
    • 2
    • 2
  • c

    Chris Hansen

    07/28/2022, 10:52 PM
    Are there instructions on how to set up Kubernetes flows in Prefect Cloud 2.0?
    ✅ 1
    t
    t
    +3
    • 6
    • 21
  • g

    Geoffrey Greenleaf

    07/29/2022, 1:36 AM
    Has anyone done anything with clickhouse and prefect that minds sharing some of their tasks how they are inserting data? Current process is we have a spark job that does our ETL and inserts into our sql server database. but I'm currently exploring clickhouse as our new data store and looking at prefect to kinda orchestrate and monitor our ETLs
    k
    d
    • 3
    • 3
  • f

    Faheem Khan

    07/29/2022, 2:30 AM
    Hi all, prefect 2.0. I am getting the following error while building a deployment.
    ValueError: Unable to find block document named mini1 for block type RemoteFileSystem
    m
    a
    • 3
    • 4
  • p

    Pedro Machado

    07/29/2022, 2:49 AM
    Hi. We have a number of flows that are running on Prefect 0.14.22. I am wondering if I can progressively upgrade them to 1.2. If I run a 1.2 agent, will it work with flows written using an older version of Prefect? I know it's a weird question to ask when 2.0 just went GA 🙂
    a
    • 2
    • 5
  • s

    Slackbot

    07/29/2022, 4:42 AM
    This message was deleted.
  • v

    Viet Nguyen

    07/29/2022, 6:25 AM
    hi everyone, I'm using Prefect 2.0 to orchestrate
    xarray
    dataset conversion, using
    DaskTaskRunner
    , the created dask client (
    task_runner
    ) doesn't automatically pickup the
    xarray
    functions such as
    xarray.dataset.concat()
    or
    <http://xarray.dataset.to|xarray.dataset.to>_zarr
    to fill up the works to all CPU threads like without Prefect. To bring the tasks to dask client with Prefect, I have to explicitly submit the task with
    @task
    decorator and calling
    .submit()
    This way, however, disallow
    xarray
    functions to be executed in all CPU threads, only the wrapped task function to be executed in individual thread, which is not ideal. If anyone has come across this issue before and have a solution, much appreciated . Thanks a lot.
    ✅ 1
    a
    • 2
    • 10
  • m

    Matthias

    07/29/2022, 8:03 AM
    With the latest release, are the deployments from the examples here portable to the new version? In particular the second one where the flow file is baked into the image?
    ✅ 1
    👀 1
    m
    a
    j
    • 4
    • 14
  • m

    Matthias

    07/29/2022, 8:05 AM
    And as a follow up question, it was mentioned in several locations that blocks allow you to securely store info. Is the data somehow encrypted first then?
    a
    • 2
    • 1
  • f

    flurven

    07/29/2022, 9:20 AM
    Trying to deploy my flow and following 2.0 tutorial. Prefect deployment build won't run because there is no build command. Looking at prefect deployment - - help there is no such command. Using 2.0b6. What did I miss?
    ✅ 1
    m
    • 2
    • 7
  • c

    Christian

    07/29/2022, 9:29 AM
    Quick question regarding *Prefect 2.0 subflow + DaskTaskRunner*… I currently have a simple ETL flow that I define and can run sequentially no problem. I now want to use this flow in inside another flow that basically wraps and executes it for many files in parallel on a DaskTaskRunner… I know that I have to use my_task.submit() on individual tasks if I want to run them concurrently on a dask cluster. But is it possible to tell a subflow that all it’s tasks should be scheduled with a submit() after the fact? Hope this was not to confusing… Thanks y’all
    ✅ 1
    a
    • 2
    • 10
  • r

    Rajvir Jhawar

    07/29/2022, 10:14 AM
    Question here about the new deployment mechanism: Can I load a storage block from file instead of having to query prefect? For instance: prefect deployment build -sb <point_to_file> I want to remove any failure condition related to networking when using a CI/CD pipeline.
    ✅ 1
    d
    a
    • 3
    • 3
  • l

    Leonardo Melo

    07/29/2022, 11:57 AM
    Hey everyone, is there any documentation on how to setup a docker-compose configuration for Prefect 2.0? I've search this slack channel, the closest I found was this repo but looks a bit outdated and non-official. https://github.com/flavienbwk/prefect-docker-compose
    ✅ 1
    a
    • 2
    • 2
  • d

    Daniel Manson

    07/29/2022, 12:29 PM
    I'm on 0.14.22, which i know is quite behind the times at this point (definitely want to migrate to v2, but not had a chance yet).... i'm trying to use Github for storage, and need the complete repo, not just one file. According to the docs the
    Git
    storage clones the full repo, so I'm trying to use that rather than the
    GitHub
    storage as that looks to just clone the one flow file (having looked at the code). However, when using
    Git
    storage I get the following error as the only log in my flow:
    {'type': ['Unsupported value: Git']}
    a
    • 2
    • 6
  • t

    Tom Klein

    07/29/2022, 12:55 PM
    In the attached link, you explain that the new storage blocks make it easier to include not just flow code in the storage but also other python dependencies and code. But - there is no explanation there (or in the link inside that leads to the documentation about deployments) for how exactly this can be done. The examples for deployments only explain how to include flow code in the storage. Are there such examples somewhere, or can you elaborate more? https://discourse.prefect.io/t/deployments-are-now-simpler-and-declarative/1255
    ✅ 1
    a
    j
    a
    • 4
    • 9
  • j

    Joshua Greenhalgh

    07/29/2022, 1:04 PM
    Hey anybody got any way to wait on multiple flows at the same time that are created by
    create_flow_run
    ? Want to do something like;
    j
    • 2
    • 8
  • j

    James Phoenix

    07/29/2022, 1:24 PM
    Will prefect 2.0 be supported via kubernetes GKE auto-pilot?
    j
    r
    • 3
    • 14
  • e

    Eli Treuherz

    07/29/2022, 2:00 PM
    I’m getting some weird behaviour with mapped tasks after failure. It seems like rather than being automatically failed by the runner when an upstream task failed, they’re getting run but with a signal object
    TRIGGERFAIL
    passed to them as their first argument. If this sounds completely bizarre I can see if I can narrow down a minimal example
    a
    • 2
    • 1
  • r

    R

    07/29/2022, 2:13 PM
    Hello! I was wondering if there’s any way to externally feed Prefect 2.0 with logs instead of directly connecting with Prefect cloud/UI
    👍 1
    ✅ 1
    a
    • 2
    • 2
  • s

    Sebastian

    07/29/2022, 2:27 PM
    Prefect 2.0.0 Deployment - flow lives in a dedicated Python library/module. I have an internal Python library called
    orchester
    in which I organize all my orchestration code. In Prefect 1.x the deployment pattern was akin to this:
    from orchester.flows import hello_flow
    hello_flow.storage = Module(__name__ + ".flows")
    hello_flow.register()
    In Prefect 2.x-beta I was able to achieve something similar like this:
    from orchester.flows import hello_flow
    Deployment(
        name="entw",
        flow=hello_flow,
        packager=OrionPackager(serializer=ImportSerializer()),
    )
    # After that I created the deployment on the local Orion server with the CLI.
    Now with Prefect 2.0.0 I am unsure what the deployment pattern would look like. My guess would be to run
    prefect deployment build .\orchester\flows\hello_flow.py:hello_flow --name entw
    from the cli and to then manually edit the
    hello_flow-manifest.json
    . I am not sure whether this is the the best approach or whether it needs to be done differently.
    {
        "flow_name": "hello_flow",
        "import_path": ".\\orchester\\flows\\hello_flow.py:hello_flow",  # Do I just need to change this line to a module path? 
        "parameter_openapi_schema": {
            "title": "Parameters",
            "type": "object",
            "properties": {
                "name": {
                    "title": "name",
                    "type": "string"
                }
            },
            "required": [
                "name"
            ],
            "definitions": null
        }
    }
    TLDR; Does anyone have a good suggestion how to deploy flows that live in their own Python module with Prefect 2.0.0?
    ✅ 1
    👀 4
    a
    j
    • 3
    • 11
  • z

    Zach Schumacher

    07/29/2022, 2:54 PM
    when will 2.0 be available for prefect cloud customers?
    ✅ 1
    b
    • 2
    • 3
  • m

    Muhammad Mikaal Saleem Anwar

    07/29/2022, 3:27 PM
    Hey everyone, Is there any version of Prefect 2 that is compatible with sqlalchemy version < 1.4 and >= 1.3.6? Want to try it out for my project but I’m getting compatibility issues ...
    m
    m
    • 3
    • 4
  • h

    Hafsa Junaid

    07/29/2022, 3:43 PM
    Hey, can we use local and Kubernetes both prefect agents simultaneously through different pipeline in prefect cloud
    ✅ 1
    n
    • 2
    • 1
  • j

    Jeffrey Lai

    07/29/2022, 4:27 PM
    Hi Community, do we have to create deployments.yaml for each flow we submit to Prefect Cloud?
    m
    • 2
    • 1
  • a

    Andrew Richards

    07/29/2022, 5:28 PM
    Hi, I'm running into a few issues while trying to use Prefect 2.0 block storage. I'm receiving a CORS error in Chrome when I try to delete a block in the UI at app.prefect.cloud. I have no issue creating block storage through the UI or through Python.
    Access to XMLHttpRequest at '<https://api.prefect.cloud/api/accounts/><MY ACCOUNT ID>/workspaces/<MY WORKSPACE ID>/block_documents/<MY BLOCK ID>' from origin '<https://app.prefect.cloud>' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
    It seems like CORS may be unhappy with the UI trying to do a cross-domain request from app.prefect.cloud to api.prefect.cloud?
    ✅ 1
    a
    a
    • 3
    • 7
  • o

    Oleg Sheyner

    07/29/2022, 5:29 PM
    Hi everyone, evaluating Prefect and hoping to get your advice on getting to a working cloud POC as fast as possible. Our situation: • Mostly Windows/Azure shop. Preference to run on Windows, all else equal, but can work with Linux VMs if necessary. • We have nothing in our ecosystem yet running on VMs (serverless only) and no Docker/Kubernetes setup. How would you suggest we set up a POC that deploys and runs flows on Azure and connects to Prefect Cloud: • POC should include demo for publishing and running flows to Azure from local dev machines • Primary goal is to set up as fast as possible: • Set up on a bare VM? Install and run under Docker? Try Azure AKS? (no experience at present with Kubernetes/AKS, but can be good case to acquire such experience in-house) Thanks much in advance!
    👀 4
    m
    c
    c
    • 4
    • 6
  • j

    Jake

    07/29/2022, 6:46 PM
    Has anyone ever run into this issue when trying to apply a kubernetes agent?
    prefect.exceptions.ClientError: Malformed response received from API.
    Here is the full stack trace:
    return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 305, in start
        start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 128, in start_agent
        agent.start()
      File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 186, in start
        self._setup_api_connection()
      File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 870, in _setup_api_connection
        self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
      File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 831, in _register_agent
        agent_config_id=self.agent_config_id,
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1788, in register_agent
        tenant_id=self.tenant_id,
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 264, in tenant_id
        self._tenant_id = self._get_default_server_tenant()
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 176, in _get_default_server_tenant
        response = self.graphql({"query": {"tenant": {"id"}}})
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 449, in graphql
        retry_on_api_error=retry_on_api_error,
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 405, in post
        retry_on_api_error=retry_on_api_error,
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 646, in _request
        raise ClientError("Malformed response received from API.") from exc
    prefect.exceptions.ClientError: Malformed response received from API.
    Here is the .yml:
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: prefect-agent
      namespace: prefect
      name: prefect-agent
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: prefect-agent
      template:
        metadata:
          labels:
            app: prefect-agent
        spec:
          containers:
          - args:
            - prefect agent kubernetes start --job-template "<s3://xxx>"
            command:
            - /bin/bash
            - -c
            env:
            - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
              value: ''
            - name: PREFECT__CLOUD__API
              value: <https://api.prefect.io>
            - name: NAMESPACE
              value: prefect
            - name: IMAGE_PULL_SECRETS
            - name: PREFECT__CLOUD__AGENT__LABELS
              value: '[''xxx'', ''xxxxx'']'
            - name: JOB_MEM_REQUEST
              value: ''
            - name: JOB_MEM_LIMIT
              value: ''
            - name: JOB_CPU_REQUEST
              value: ''
            - name: JOB_CPU_LIMIT
              value: ''
            - name: IMAGE_PULL_POLICY
              value: Always
            - name: SERVICE_ACCOUNT_NAME
              value: ''
            - name: PREFECT__BACKEND
              value: cloud
            - name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
              value: http://:8080
            - name: PREFECT__CLOUD__API_KEY
              value: xxx
            - name: PREFECT__CLOUD__TENANT_ID
              value: xxx
            - name: PREFECT__LOGGING__EXTRA_LOGGERS
              value: "['xxxxx', 'xxxxxxxx']"
            image: prefecthq/prefect:1.2.2-python3.7
            imagePullPolicy: Always
            livenessProbe:
              failureThreshold: 2
              httpGet:
                path: /api/health
                port: 8080
              initialDelaySeconds: 40
              periodSeconds: 40
            name: agent
    ✅ 1
    m
    • 2
    • 2
Powered by Linen
Title
j

Jake

07/29/2022, 6:46 PM
Has anyone ever run into this issue when trying to apply a kubernetes agent?
prefect.exceptions.ClientError: Malformed response received from API.
Here is the full stack trace:
return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 305, in start
    start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 128, in start_agent
    agent.start()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 186, in start
    self._setup_api_connection()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 870, in _setup_api_connection
    self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 831, in _register_agent
    agent_config_id=self.agent_config_id,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1788, in register_agent
    tenant_id=self.tenant_id,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 264, in tenant_id
    self._tenant_id = self._get_default_server_tenant()
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 176, in _get_default_server_tenant
    response = self.graphql({"query": {"tenant": {"id"}}})
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 449, in graphql
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 405, in post
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 646, in _request
    raise ClientError("Malformed response received from API.") from exc
prefect.exceptions.ClientError: Malformed response received from API.
Here is the .yml:
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prefect-agent
  namespace: prefect
  name: prefect-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect-agent
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      containers:
      - args:
        - prefect agent kubernetes start --job-template "<s3://xxx>"
        command:
        - /bin/bash
        - -c
        env:
        - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
          value: ''
        - name: PREFECT__CLOUD__API
          value: <https://api.prefect.io>
        - name: NAMESPACE
          value: prefect
        - name: IMAGE_PULL_SECRETS
        - name: PREFECT__CLOUD__AGENT__LABELS
          value: '[''xxx'', ''xxxxx'']'
        - name: JOB_MEM_REQUEST
          value: ''
        - name: JOB_MEM_LIMIT
          value: ''
        - name: JOB_CPU_REQUEST
          value: ''
        - name: JOB_CPU_LIMIT
          value: ''
        - name: IMAGE_PULL_POLICY
          value: Always
        - name: SERVICE_ACCOUNT_NAME
          value: ''
        - name: PREFECT__BACKEND
          value: cloud
        - name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
          value: http://:8080
        - name: PREFECT__CLOUD__API_KEY
          value: xxx
        - name: PREFECT__CLOUD__TENANT_ID
          value: xxx
        - name: PREFECT__LOGGING__EXTRA_LOGGERS
          value: "['xxxxx', 'xxxxxxxx']"
        image: prefecthq/prefect:1.2.2-python3.7
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 2
          httpGet:
            path: /api/health
            port: 8080
          initialDelaySeconds: 40
          periodSeconds: 40
        name: agent
✅ 1
Nevermind, I found the issue! was using the wrong tenant value
m

Mason Menges

07/29/2022, 7:01 PM
Fantastic 😄
View count: 3