https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • s

    Slackbot

    08/01/2022, 6:37 PM
    This message was deleted.
  • l

    Leon Kozlowski

    08/01/2022, 6:37 PM
    Hi all, are there any plans relating to a prometheus exporter for Kubernetes deployed agents - thought I would check before I go about building some form of a custom exporter
    n
    l
    • 3
    • 10
  • c

    Chu

    08/01/2022, 6:42 PM
    how can i change the create_flow_run task name?? see this schematic has three create_flow_run, if we can’t change the displayed name, it’s hard to debug if one fails
    a
    • 2
    • 1
  • j

    Jonathan Mathews

    08/01/2022, 7:38 PM
    I’d like to be able to use the
    run_task_kwargs_path
    for
    prefect agent ecs start
    as it’s not picking up my VPC (not the default one) but the docs don’t say what format
    options.yml
    should be in. Saw this issue here, but hasn’t been an update recently https://prefect-community.slack.com/archives/CL09KU1K7/p1653756981241529
    ✅ 1
    r
    • 2
    • 6
  • n

    Nathan Ballou

    08/01/2022, 7:52 PM
    Hi! I'm running into an error when trying to run a Prefect 2.0 flow using Comet.ml and a Ray task runner. The flow seems to fail at the comet import statement:
    File "C:\Users\...\lib\site-packages\comet_ml\connection.py", line 35, in <module>
        from comet_ml import config
      File "C:\Users\...\lib\site-packages\comet_ml\config.py", line 59, in <module>
        from everett.manager import ConfigIniEnv as UpstreamConfigIniEnv
    ImportError: cannot import name 'ConfigIniEnv' from 'everett.manager'
    Any thoughts?
    n
    • 2
    • 22
  • l

    Leo Marvin

    08/01/2022, 8:04 PM
    I'm trying to set up S3 storage for flow runs (using prefect 1)
    with Flow(
        "test-flow",
        storage=S3(bucket="my-bucket"),
        run_config=ECSRun(...)
    but getting this error:
    Failed to load and execute flow run: NoCredentialsError('Unable to locate credentials')
    I added a Secret named
    AWS_CREDENTIALS
    through the prefect cloud UI (https://cloud.prefect.io/team/secrets) with a json of the form
    {
      "ACCESS_KEY": "...",
      "SECRET_ACCESS_KEY": "..."
    }
    but still getting this error. Curious if anyone one has tips to troubleshoot.
    ✅ 1
    n
    p
    • 3
    • 9
  • s

    Seth Coussens

    08/01/2022, 8:48 PM
    Has anyone seen the error:
    'prefect.engine' is a package and cannot be directly executed
    when trying to run a DockerContainer flow in Prefect 2.0?
    ✅ 1
    a
    • 2
    • 20
  • y

    Yupei Chen

    08/01/2022, 9:02 PM
    For Prefect 2.0, how do I correctly pass the env variable EXTRA_PIP_PACKAGES to the runtime infrastructure? I tried below, but am still running into an error that s3fs is not installed - which is required to pull storage from S3
    a
    k
    r
    • 4
    • 9
  • k

    Kyle McChesney

    08/01/2022, 9:29 PM
    In prefect 1.0, is there any kind of support for dynamically adding parameters to a current flow run or updating the values of parameters
    ✅ 1
    b
    • 2
    • 3
  • s

    Sebastián Montoya Tapia

    08/01/2022, 9:33 PM
    Hi Community! I am having trouble increasing the memory of my tasks when using the ECSRun, I followed the documentation from Prefect 1.0 but for some reason the value remains in 1024 cpu units and 2048 mb ram I need 4096 mb of ram, anyone has done this that can help me ?
    r
    • 2
    • 1
  • h

    Hafsa Junaid

    08/01/2022, 9:44 PM
    any guide on this error message for prefect 1.0 cloud
    HTTPSConnectionPool(host='34.66.191.39', port=443): Max retries exceeded with url: /apis/batch/v1/namespaces/default/jobs (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f3196f62eb0>: Failed to establish a new connection: [Errno 110] Connection timed out'))
    m
    c
    • 3
    • 4
  • m

    Matt Fysh

    08/01/2022, 10:31 PM
    Hi all, in the realtime examples I’m coming across, all of them use polling in the form of
    if __name__ == "__main__": while True:
    - are there any examples where flows can be invoked via realtime events, e.g. S3
    ObjectCreated
    Furthermore, are there any Prefect components in place so that if you change the flow code, it can replay all of the prior events?
    t
    f
    • 3
    • 4
  • s

    Stephen Herron

    08/02/2022, 1:28 AM
    Hi there - does this look right?
    import asyncio
    from prefect_snowflake import SnowflakeCredentials
    
    from prefect import flow
    creds = SnowflakeCredentials.load("credentials")
    
    from prefect_snowflake.database import snowflake_query
    
    
    @flow(name="Snowflake Query")
    async def select_something():
    
        result = await snowflake_query(
            "select 1 as test",
            creds,
        )
    
    if __name__ == "__main__":
        asyncio.run(select_something())
    Seems to work locally but when I try to run ad-hoc via a deployment I get the error:
    ...
    prefect_snowflake/database.py", line 69, in snowflake_query
        with snowflake_credentials.get_connection(**connect_params) as connection:
    AttributeError: 'coroutine' object has no attribute 'get_connection'
    a
    • 2
    • 2
  • c

    Chu

    08/02/2022, 2:05 AM
    I'm wondering can Parameter () take a list as default values? Since I need to map it to do some parallel running
    ✅ 1
    r
    • 2
    • 1
  • c

    Chris Hansen

    08/02/2022, 3:15 AM
    does anyone have an example of what CI/CD should look like with prefect 2.0’s new deployments? and should i be committing deployment yamls and manifests to source control?
    👀 3
    r
    • 2
    • 3
  • h

    Ha Pham

    08/02/2022, 5:07 AM
    Hi all, is there any existing documentation / discussion about the difference between prefect 2.0 self-host vs cloud version?
    ✅ 1
    r
    • 2
    • 2
  • f

    Florian Sell

    08/02/2022, 6:48 AM
    Hi all, I have some questions/thoughts about the deployment process with Prefect 2.0. (I am using Kubernetes jobs) After every change the deployment build has to be run to upload the files. But every time the yaml has to be edited again, e.g. changing the image. Is there a better way to do this and not manually? Maybe specifying a template for the build would solve this manual work. Or a CLI option to keep the content. Also all files are uploaded, like pycaches. There should be a ignore file to handle this, like gitignore. When files are moved inside the folder stucture underneath the manifest or renamed and the deployment is rebuilded, the old files are still in the storage. This can end up with duplicated files. Since just clearing the storage before rebuilding/uploading could crash current pulling deployments, I would think about a version folder between the path given in the storage block and the manifest-root. A 'build' would upload the files to the new version-folder and an 'apply' would switch to the new folder. Our Flows are all event triggered with parameters. How can I trigger a flow run with parameters? I would expect an option --parameters in 'deployment run'. In our current Prefect 1 setup I am using a state_handler to do some post processing after a failed flow (state_handlers=...). How would this look like in Orion? Should I implement a very last task in a flow to do this?
    ➕ 3
    👀 3
    a
    • 2
    • 1
  • r

    Rasmus Lindqvist

    08/02/2022, 7:43 AM
    Hi there! I’m trying to forward logs form an external library, more specifically
    scrapy
    . I have configured scrapy to log to stdout, and configured my Prefect task to
    log_stdout=True
    . Should not Prefect then pick up the logs from scrapy or am I missing something?
    ✅ 1
    r
    k
    d
    • 4
    • 11
  • j

    Joshua Greenhalgh

    08/02/2022, 8:30 AM
    Hi all, great to hear that prefect 2 is now out of beta - the announcement doesn't give much clarity about how long prefect 1 cloud is going to be around - are there more details around this?
    👀 2
    m
    • 2
    • 3
  • d

    David

    08/02/2022, 8:49 AM
    Is it easy to upgrade prefect 1 flows to prefect 2? And will the vertex-ai runner be available in prefect 2?
    t
    • 2
    • 6
  • k

    Kha Nguyen

    08/02/2022, 10:12 AM
    In Prefect 2.0, I have a task
    @task
    def run(x: int, y: int) -> int:
        return x + y
    
    @flow
    def my_flow():
        results = run.map([1, 2, 3, 4], y=1)
    How can I map only on
    x
    , and use the same
    y
    argument?
    ✅ 1
    j
    • 2
    • 3
  • k

    Kha Nguyen

    08/02/2022, 11:21 AM
    When I use map to submit about 10000 tasks, the run failed. Is there a limit to how many tasks I can submit?
    ✅ 1
    b
    • 2
    • 16
  • o

    Oscar Björhn

    08/02/2022, 11:23 AM
    I'm back with another question, having a hard time finding anything in the docs or slack regarding this: Is there an equivalent to the old StartFlowRun or the newer create_flow_run in Prefect 2.0? Or some other way of having a flow run a subflow without knowing about it beforehand (since it's deployed in a separate docker container)?
    ✅ 1
    m
    • 2
    • 3
  • k

    Kha Nguyen

    08/02/2022, 11:43 AM
    I cannot log in a task. Logging in a flow is okay. A task is called using submit.
    👀 1
    b
    • 2
    • 6
  • c

    Chu

    08/02/2022, 12:13 PM
    Is Prefect specialized for streaming pipeline schedule?
    ✅ 1
    a
    • 2
    • 5
  • c

    Chu

    08/02/2022, 2:30 PM
    what does this error implies:
    TypeError: `fn` must be callable
    I’m calling a task inside a With Flow statement, and the Task should return me a list of strings
    ✅ 1
    n
    • 2
    • 13
  • m

    Mateo Merlo

    08/02/2022, 2:46 PM
    Hi there! I am having an issue with Prefect 1.2.4 running a flow of flows. I followed the tutorial https://www.prefect.io/guide/blog/flow-of-flows-orchestrating-elt-with-prefect-and-dbt/ that is great and helped me a lot to connect DBT with other flows. This was working fine but suddenly start to fails with this error: Error during execution of task: ValueError("Flow 'ETL' not found.")
    n
    • 2
    • 9
  • r

    Ray Shi

    08/02/2022, 3:25 PM
    Hey there, for the run history, how long it keeps the logs? If a flow runs extensively long time, will it continue to finish even though the run history is wiped out? How can i track the logs for those long running flows?
    ✅ 1
    k
    b
    • 3
    • 19
  • c

    Christian Nuss

    08/02/2022, 5:09 PM
    hey y'all! the Prefect Helm Chart doesn't have any resources defined (cpu/mem limits/requests)... are there any good recommendations for a starting point?
    a
    • 2
    • 1
  • r

    Ronald Tichauer

    08/02/2022, 5:26 PM
    Hi there! Does anyone know how to run a flow from a specific docker image (not the prefect image) in Prefect 2.0? I’ve been trying with this, passing the
    image=<my_images_name>
    argument to the DockerContainer, but doesn’t work, or at least, it doesn’t run the
    CMD
    specified in the Dockerfile of that image
    j
    k
    • 3
    • 4
Powered by Linen
Title
r

Ronald Tichauer

08/02/2022, 5:26 PM
Hi there! Does anyone know how to run a flow from a specific docker image (not the prefect image) in Prefect 2.0? I’ve been trying with this, passing the
image=<my_images_name>
argument to the DockerContainer, but doesn’t work, or at least, it doesn’t run the
CMD
specified in the Dockerfile of that image
j

Jean Luciano

08/02/2022, 5:28 PM
Are you specifying this in a yaml deployment?
k

Khuyen Tran

08/02/2022, 5:36 PM
What version of Prefect are you in? Using a Docker container infrastructure will help
r

Ronald Tichauer

08/03/2022, 11:53 PM
Thanks Jean and Khuyen! I was using a yaml deployment run by a
build deployment
but now I saw that even specifying the infrastructure within the deployment, it was not getting it when I built the yaml. It still wasn’t what I needed to do and couldn’t make it work, that was running a specific docker from a flow. Sadly I think I’ll have to go back to Airflow 😭 I was using prefect 2.0 btw
k

Khuyen Tran

08/04/2022, 2:59 PM
Could you be more specific about which infrastructure you tried to use and the command that you used?
View count: 5