https://prefect.io logo
Join Slack
Channels
Powered by Linen
prefect-community
  • m

    Mohamed Alaa

    08/11/2022, 12:59 PM
    Hello Guys, Is there any documentation or examples for how to deploy Prefect 2.0 outside of Prefect Cloud? The requirements such as the database and so on. Thanks in advance
    :plus-one: 1
    t
    • 2
    • 2
  • o

    Oscar Björhn

    08/11/2022, 1:50 PM
    Is there currently a way of overriding a task name when calling it in Prefect 2? The following (which is what the documentation recommends, unless I'm reading it wrong) appears to have no effect: my_task_function(name="Custom Task Name") The task run name for the above task ends up being "my_task_function-6971d82e-1"
    ✅ 1
    j
    k
    +2
    • 5
    • 13
  • a

    Alex Shea

    08/11/2022, 2:36 PM
    I have a question regarding version one of the prefect agent in kubernetes. I am experiencing a similar issue to that of this question before. My error is 
    User \"system:serviceaccount:prefect:prefect-agent\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"prefect\"
    I as you might be able to tell, I have the prefect agent running in the prefect namespace. I have a service account for it called prefect agent. I have used the role and rolebinding from the prefect documentation, but updated the namespace on both and the service account name from default to prefect -agent.
    ✅ 1
    k
    c
    • 3
    • 11
  • n

    Nelson Griffiths

    08/11/2022, 2:56 PM
    Is there a way to get the logger provided by prefect to log to stdout using the RichHandler? Or do I just need to create my own logger to do that?
    a
    j
    • 3
    • 10
  • j

    James Brady

    08/11/2022, 3:31 PM
    Hello all! Is there a recommended way to install and run the Prefect agent in non-local environments? I see references to e.g. 
    prefect agent kubernetes install
    but that seems to be v1.x only…
    k
    j
    • 3
    • 7
  • b

    Bigya Man Pradhan

    08/11/2022, 3:36 PM
    Hi all, I am using 
    prefect deployment apply ...
    and 
    prefect deployment build ...
    CLI commands to deploy and build flows. Is there a python equivalent to these commands?
    k
    n
    • 3
    • 5
  • s

    Sam Garvis

    08/11/2022, 4:05 PM
    For 2.0, has anyone successfully used the k8s helm deployment.yaml I'm having trouble getting it to actually apply and create pods.
    ✅ 1
    n
    • 2
    • 11
  • a

    Alexander Belikov

    08/11/2022, 4:28 PM
    Hi all, I'm migrating from 1.0 to 2.0. I successfully went through deployment steps a) 
    prefect deployment build ...
    and b) 
    prefect deployment apply ...
    prefect deployment run ...
    runs successfully. I've created the work queue and the agent is running, but scheduled flows are not being executed, instead at time X they change their state from 
    Scheduled
    to 
    Pending
    . The work queue is empty (!). What could be the problem?
    n
    • 2
    • 10
  • c

    Chandrashekar Althati

    08/11/2022, 4:46 PM
    Hi all, I am new to Prefect. We are planning to use prefect cloud for orchestrating our workloads. For execution environment, we are planning to go on AKS in long term. But for short term, we would like to run the flows as containerized applications on Azure. But my question is, which of the following Azure container options best suites for running the Prefect flows as containers. 1. Azure Container Apps , 2. Azure Container Instances, 3. Azure Functions. Find the attached image. We would like to replace the AKS in the diagram with one of the above three Azure container options.

    https://infinitelambda.com/wp-content/uploads/2021/02/prefect-steps-2048x643.png▾

    c
    c
    • 3
    • 10
  • m

    Mars

    08/11/2022, 4:58 PM
    Hi, I’m trying to deploy a trivial flow in Prefect 2.0 using S3 and the RemoteFileSystem storage block. The block is properly registered with Orion. For some reason 
    prefect deployment build
    gives me a deployment YAML, but is no longer producing the manifest JSON file. It’s only producing the deployment YAML. How can I debug this? Here is the build command used: 
    prefect deployment build myflow.py:myflow -n myflow-deployment -t dev -ib kubernetes-job/k8dev -sb remote-file-system/dev
    j
    • 2
    • 6
  • j

    John Kang

    08/11/2022, 6:12 PM
    Does anyone else not have the ability to run custom flow runs from the UI? Yesterday I was able to, but that option seems to have disappeared today on the UI for me. Anyone else experiencing this problem?
    j
    k
    • 3
    • 5
  • c

    Chandrashekar Althati

    08/11/2022, 6:36 PM
    Hi, is Prefect Cloud 2.0 HIPAA complaint ?
    ✅ 1
  • j

    Jon Ruhnke

    08/11/2022, 7:11 PM
    In Prefect 1.0... How do I run multiple instances of the same flow, based on a list of data? An example would be I've created a generic flow to go grab 1 table from a source system. I want to be able to neatly specify dozens of tables in a convenient location (config file?) and have an instance of the flow run for each of the tables. Do I need to create a 'master' flow that loops through and runs a bunch of child flows? I've already created a working flow with the .map() function to iterate over the list of tables, but that doesn't seem like a good approach since the Prefect UI doesn't seem to neatly separate things out for debugging and retrying.
    n
    a
    • 3
    • 5
  • m

    Mars

    08/11/2022, 7:19 PM
    Hi, the KubernetesJob docs say that you need to have a remote file store configured to use it. And the Storage docs say that none of the remote file storage libraries are installed by default. Now my flow stored in S3 is failing to deploy via the k8s agent because of the missing file store libraries. How do you recommend including something like s3fs in the KubernetesJob environment, since it isn’t included in the default package set? The Storage docs say that you can use 
    EXTRA_PIP_PACKAGES
    for the Docker executor. Is there something similar for the k8s job executor?
    i
    a
    • 3
    • 7
  • y

    Yardena Meymann

    08/11/2022, 7:53 PM
    Hi, in Prefect 1.0 (Cloud) I want to register a new version of flow A from another flow B. If I am authenticated to Prefect inside flow B I should be able to do this, right?
    a
    • 2
    • 2
  • s

    Slackbot

    08/11/2022, 8:19 PM
    This message was deleted.
  • s

    Slackbot

    08/11/2022, 8:28 PM
    This message was deleted.
  • j

    Jeff LaPorte

    08/11/2022, 8:30 PM
    I always find it interesting how data / data science products don’t consider prod/stage separation an “ante up” aspect of a SaaS service.
    k
    a
    • 3
    • 4
  • j

    Joe Goldbeck

    08/11/2022, 8:48 PM
    Is there any way to achieve something like 
    prefect register --dry-run
    ? I would like to add a CI step that ensures that we won’t fail after a merge when we try to register flows in CD
    👀 2
    a
    • 2
    • 5
  • n

    Neil Natarajan

    08/11/2022, 8:59 PM
    in prefect 2.0 what is the best way to chain results between map tasks? Specifically, how should errors on specific iterations be handled if the remaining results are still desired? This seemed seemless in prefect 1.x but I am running into issues while migrating to 2.0
    k
    j
    +2
    • 5
    • 16
  • e

    Edmondo Porcu

    08/11/2022, 11:58 PM
    Hi, I am using Prefect 1.0 and I tried to create two tasks that share a lot of common code, and at a certain moment they invoke a function which uses the task library to create an instance on of a task and runs it. However, this function is not annotated via @task, while the two invoking functions they are. I observed a weird behavior where only one of the two tasks was run. Should mytask.run() only be invoked within fuction annotated with @task ?
    ✅ 1
    m
    a
    • 3
    • 47
  • i

    Ilya Galperin

    08/12/2022, 12:37 AM
    What strategies are folks using for flow storage blocks in 2.0? It seems like there are at least two distinct possibilities: 1. Save all flows using one storage block and running the 
    deployment
    build
    and 
    apply
    commands from a parent directory containing multiple flows. This way, we use only one storage block but if multiple developers are working in the same shared bucket like in a staging environment, the blast radius is bigger. For example, if a developer is working on only a single flow, there is a risk of overwriting someone else’s in-development code if you commit code from another flow/folder accidentally. 
    deployment build flow_a/flow.py:entry_point -n flow_a_deployment --storage-block s3/universal-storage-block

flows/ <working directory>
   flow_a/
   flow_b/
    2. Every flow gets its own storage block and running 
    deployment
    build
    and 
    apply
    commands from that flow’s root directory. This will obviously require us to use more storage blocks, but seems to decrease blast radius. 
    deployment build ./flow.py:entry_point -n flow_a_deployment --storage-block s3/individual-storage-block

flow_/a <working directory>
    It seems to me like option 2 is more optimal, but are there any disadvantages or limitations we should be aware of in using multiple storage blocks?
    ✅ 1
    a
    j
    • 3
    • 6
  • d

    Dean Magee

    08/12/2022, 12:59 AM
    Hi there... I need to push a geojson file to Postgres (PostGIS). What is the most efficient way to do this? One option is to write a shell command (using ogr2ogr) within a task. Is there a recommended way to do this?
    ✅ 1
    a
    • 2
    • 2
  • l

    Lennert Van de Velde

    08/12/2022, 8:00 AM
    Hi, is there any possibility of setting the write path and read path of your deployment storage to a different location, e.g. the read path to a Github Directory and the write path to a Local File System?
    ✅ 1
    a
    • 2
    • 2
  • m

    Marcin Grzybowski

    08/12/2022, 9:07 AM
    HI all, Is it safe to add a table into orion.db? or a column into flow_run ? We would need to "log" additional info about flow results, but using "normal" logging may be not optimal, as log table may be big and we would need to do searching in text column...
    ✅ 1
    a
    • 2
    • 6
  • j

    Jonathan Mathews

    08/12/2022, 12:14 PM
    Hello! In Prefect 1.0, I have my flow run_config set to call a function set_run_config. From that function, is it possible to access the labels that have been added to the flowrun?
    ✅ 1
    a
    • 2
    • 22
  • j

    James Brady

    08/12/2022, 12:50 PM
    What's the recommended / expected way to update a deployment in Prefect 2? I have an existing deployment, and used 
    prefect deployment apply …
    after updating Python files, but I can see in the S3 bucket that the Python files aren't being updated to match what's on my local computer.
    ✅ 1
    b
    o
    • 3
    • 7
  • t

    Tomas Knoetze

    08/12/2022, 1:00 PM
    Hello! Quick question: is there an equivalent for task_run_name in Prefect 2.0?
    ✅ 1
    a
    p
    • 3
    • 3
  • s

    Sean Malone

    08/12/2022, 1:30 PM
    Is there a way to “un-register” a custom block type? (2.0)
    ✅ 1
    a
    • 2
    • 1
  • a

    Andrei Tulbure

    08/12/2022, 1:50 PM
    Hi! Posted it also on the data tricks and tips channel but I see here there is more activity. ---- I am pretty new to Prefect, Metaflow and other MLOPS platforms (have a strong theoretical and MLE background though). I am now in part of a team of MLEs to perform hyper-param optimization experiments and general ML experiments. As you all know, these experiments are really hard to quantify and I have been looking the past 2 weeks at MLOPS frameworks to help us out with quantifying everything. Hence, I landed with the semi-decision to use Prefect on a AWS backend. How hard is the configuration part ? Can I expect it to be set up by myself over the weekend ? What are the typical costs incurred by my AWS Backend (VMs/ Buckets etc). And is it useful for my use case : I want to perform hyper-parameter optimization experiments and new model experiments - be able to see which configuration of parameters has the best results? And quickly deploy it ?
    ✅ 1
    a
    • 2
    • 7
Title
a

Andrei Tulbure

08/12/2022, 1:50 PM
Hi! Posted it also on the data tricks and tips channel but I see here there is more activity. ---- I am pretty new to Prefect, Metaflow and other MLOPS platforms (have a strong theoretical and MLE background though). I am now in part of a team of MLEs to perform hyper-param optimization experiments and general ML experiments. As you all know, these experiments are really hard to quantify and I have been looking the past 2 weeks at MLOPS frameworks to help us out with quantifying everything. Hence, I landed with the semi-decision to use Prefect on a AWS backend. How hard is the configuration part ? Can I expect it to be set up by myself over the weekend ? What are the typical costs incurred by my AWS Backend (VMs/ Buckets etc). And is it useful for my use case : I want to perform hyper-parameter optimization experiments and new model experiments - be able to see which configuration of parameters has the best results? And quickly deploy it ?
✅ 1
a

Anna Geller

08/12/2022, 1:54 PM
What do you mean by a quick setup on AWS? Setting up agent to be used with Cloud?
If so check out https://discourse.prefect.io/t/how-to-build-a-prefect-2-0-poc-on-aws/1066
And in the future please decide about the channel you want to post and create only one thread We'll respond if we can, no need for multiple requests
👍 1
and for AWS cost calculation, check out https://calculator.aws/#/
a

Andrei Tulbure

08/12/2022, 3:07 PM
Yes, I mean setting up an agent on AWS. Well, yes, I was just curious to know if somebody has any hands on experience regarding these costs. Like if they ran experiments or projects with prefect and maybe they could share their opinions
I am just concerned that after setting up prefect I won`t be able to run experiments for really large grid searches with Prefect. Has anybody any experience regarding that ? Because usually we would need large EC2 instances
a

Anna Geller

08/12/2022, 4:38 PM
you would need to estimate costs based on your needs and workload you estimate if you need more guidance with any pricing estimates, you can reach out to sales@prefect.io
View count: 3