https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • j

    Jacques

    04/17/2020, 2:34 PM
    I was wondering if there was a good pattern to trigger a second flow (using the next_page token as a parameter) from a task?
    j
    k
    • 3
    • 28
  • a

    An Hoang

    04/17/2020, 2:50 PM
    Hello I’m trying a simple flow of doing linear regression in batches. The flow works when doing it sequentially but when I try with Dask backend it causes memory problems. What is confusing is that there is ample memory per worker. Can someone help me identify the problem?
    simple_linreg_flow_dask.html
    • 1
    • 2
  • a

    An Hoang

    04/17/2020, 2:51 PM
    Here’s the notebook version
    simple_linreg_flow_dask.ipynb
  • r

    rmax

    04/17/2020, 4:02 PM
    I have a question about prefect cloud. If I get it correctly, they provide the scheduler and flows/jobs UI with metadata storage. On our side we have to provide the agents (nodes) where the actual computation is going to be performed. Is that correct?
    j
    • 2
    • 1
  • j

    John Ramirez

    04/17/2020, 4:08 PM
    Hey everyone - I use a parameter to pass a date to tasks downstream
    batch_run_date = Parameter(
            name='batch_run_date',
            default=(datetime.today() - timedelta(days=1)).strftime('%Y-%m-%d')
        )
    in perfect
    0.9.*
    this would alter the default value based on the current date. But in prefect
    0.10.1
    the default date is fixed. Is there a a different way I can reproduce the previous functionality
    c
    • 2
    • 2
  • a

    An Hoang

    04/17/2020, 4:27 PM
    I registered my flow (tested successfully locally) to the server following the instructions here: https://docs.prefect.io/orchestration/tutorial/first.html and got this error when triggering the flow from both the CLI and UI. April 17th 2020 at 11:22:39am | agent ERROR lens [Errno 2] No such file or directory: ‘prefect’: ‘prefect’
    c
    • 2
    • 2
  • t

    tiz.io

    04/17/2020, 5:33 PM
    Is it true that the GUI is not available in the community server?
    z
    • 2
    • 3
  • t

    tiz.io

    04/17/2020, 5:51 PM
    I've been having a vexing time getting this launched. I kick off the docker container using the master tag and it exits immediately with no logs or errors. When I shell into the container and try to launch the server, it errors out with "FileNotFoundError: [Errno 2] No such file or directory: 'docker-compose': 'docker-compose'"
    c
    • 2
    • 28
  • q

    qizzle

    04/17/2020, 6:27 PM
    is it possible to run only parts of a flow instead of the whole thing from the orchestrator?
  • q

    qizzle

    04/17/2020, 6:27 PM
    like if there were tasks a, b, and c to only run b and c
  • q

    qizzle

    04/17/2020, 6:46 PM
    and is it possible to query against the orchestrators database from a separate app?
    j
    • 2
    • 4
  • a

    asm

    04/17/2020, 6:53 PM
    Howdy! I'm currently working on adding a way to roll back a potentially bad deploy and was wondering if there is a way to have prefect basically "stop the world" on a particular flow
  • a

    asm

    04/17/2020, 6:54 PM
    I have code currently that will set all flow runs + their child task runs to cancelled but it looks like that won't stop tasks from being submitted to Dask
    c
    j
    • 3
    • 4
  • a

    asm

    04/17/2020, 6:54 PM
    I'm using the
    DaskExecutor
    if that helps
  • m

    Matthew Maldonado

    04/17/2020, 8:31 PM
    I'm using jaydebeapi on windows and prefect keeps throwing the error java.sql.SQLException: No suitable driver found for. This flow works when not using prefect server. However, it appears it can't find the jar file possibly. Where is the best place to put the file? This is also using dask distributed
    c
    • 2
    • 11
  • a

    Andrew Vaccaro

    04/17/2020, 10:06 PM
    Question about setting
    upstream_tasks
    to the same task being called twice with different arguments. I have something like
    A(abcd)
    B.set_dependencies(upstream_tasks=[A], keyword_tasks=...)
    A(defg)
    B.set_dependencies(upstream_tasks=[A], keyword_tasks=...)
    where B must run after A, and each gets called with two separate sets of Parameters. If I call
    B.set_dependencies(upstream_tasks=[A])
    twice, will the second B wait on the second A? And is this the idiomatic way to do this?
    n
    • 2
    • 10
  • m

    Matthew Perry

    04/17/2020, 11:09 PM
    Hi everyone 👋 Data engineer with a remote sensing/ IOT sensor background here. I'm looking at Prefect to orchestrate processing-intensive scientific computing workflows on AWS. Currently, we're using AWS Batch + ECS to run docker containers but it's a really limited map-reduce thing. Ideally we'd like a system that would allow us to leverage our existing docker containers and build them into more complex, parameterized DAG topologies. We've tried Airflow... and hit almost all of the limitations mentioned in prefect's docs 🙄 We're just getting started with Prefect but it looks very promising so far! Are there any other folks using prefect to tie together already-dockerized processes?
    👀 1
    j
    j
    • 3
    • 4
  • b

    Brad

    04/18/2020, 1:48 AM
    Is it possible to set the flow run name when using a Clock? I’m using one flow, multiple clocks with different parameters and it would be useful to be able to name the flow-run
    c
    • 2
    • 1
  • t

    tiz.io

    04/18/2020, 2:14 AM
    We're running into configuration issues with the graphql url. We've set server.ui.graphql_url to the proper host and path, but the UI doesn't use the config value. Is there a param for this not listed in the example config.toml?
    c
    • 2
    • 7
  • t

    tiz.io

    04/18/2020, 3:01 AM
    Lessons from the front line: Don't fire up multiple agents on one machine and background all the processes. You'll definitely regret it. 😂
    😂 1
    :marvin: 1
  • a

    Alexey Volochenko

    04/18/2020, 8:31 PM
    👋 new prefect user, currently in evaluation phase as a replacement to airflow. for reference i’m running airflow in a container, with scheduler and webserver as separate containers and multiple worker containers, all of this is on a 3 node cluster. is there a doc on how to run prefect containers similarly? i see install doc uses prefect server command which launches containers, but not a whole lot of info on how to setup this w/o that command
    s
    j
    b
    • 4
    • 10
  • s

    Scott Zelenka

    04/18/2020, 9:43 PM
    Has anyone experimented with running a K8 agent on GKE with Anthos for serverless execution of perfect jobs?
  • h

    Hui Zheng

    04/19/2020, 8:50 PM
    Hi, We had a task failure (it seems a task run time-out) last night. it seems prefect didn’t retry this tasks when it failed. We have to resolve to a manual recover this morning. Could you recommend a solution to allow prefect to handle this failure itself? (see error details in thread)
    j
    • 2
    • 8
  • m

    Mikael

    04/20/2020, 7:00 AM
    Hi! I am experimenting with the local prefect server. I know how to start it but how do I stop it?
    z
    a
    • 3
    • 5
  • a

    An Hoang

    04/20/2020, 11:53 AM
    Has anyone used prefect as a wrapper to interact with a HPC cluster? Like wrapping the
    bsub
    or
    lsub
    command into a prefect task. Basically how do you use prefect to submit jobs to a compute cluster that has no automatic callback API for job completion and you have to ping it continuously to check the status? How do you check if the HPC job is complete? Do you parse the output of the HPC’s status command, check existence of an output file or something else?
    k
    • 2
    • 1
  • r

    Romain

    04/20/2020, 1:39 PM
    Hi there! I'm giving Prefect Core a try, but I could not find answers to a few questions in the docs. Is it possible to configure multi-user access (with passwords etc.) to Prefect Server? Or is it a Cloud feature?
    j
    • 2
    • 1
  • m

    Martin Guindon

    04/20/2020, 2:24 PM
    Hello everyone 👋 I'm Martin Guindon, consultant for Montreal Analytics, based in Montreal, Canada.
    👋 8
    j
    • 2
    • 1
  • m

    Matias Godoy

    04/20/2020, 3:27 PM
    Hi guys! I have a question about task options; can I combine the
    retries
    option with the
    timeout
    ? Let me explain better: Inside my task I am polling an API waiting for something to finish. If one of those HTTP calls fail, I'd like to retry, but if it takes too long and that something hasn't finished, I want the entire task to timeout. Is that how combining those two options work? I'm afraid that a timeout would cause a retry and start all over again.
    k
    c
    m
    • 4
    • 4
  • c

    Christian

    04/20/2020, 5:00 PM
    Hi all. I currently try to test
    prefect core server
    . I have an instance of prefect server () and an agent running on a remote machine (via a VPN connection). Then I set
    export PREFECT__CLOUD__API=<http://myserver.edu:4200>
    Finally, I run my dummy flow which contains a flow.register() at the end... The output on the command is:
    Result Handler check: OK                                                                       │
    Flow: <http://localhost:8080/flow/17b029f0-fb9d-4ac5-bc54-fc4102a44631>
    Is this OK? The task does not show up on the UI. Anything I'm missing? Do I have to configure something else for this? My local laptops' diagnostic:
    {
      "config_overrides": {},
      "env_vars": [
        "PREFECT__CLOUD__API"
      ],
      "system_information": {
        "platform": "macOS-10.14.6-x86_64-i386-64bit",
        "prefect_version": "0.10.2",
        "python_version": "3.8.2"
      }
    }
    And the servers' diagnostic:
    {
      "config_overrides": {},
      "env_vars": [],
      "system_information": {
        "platform": "Linux-5.3.0-40-generic-x86_64-with-glibc2.29",
        "prefect_version": "0.10.2",
        "python_version": "3.8.2"
      }
    }
    Cheers, C
    k
    • 2
    • 4
  • c

    Christian

    04/20/2020, 6:18 PM
    Hi all (again). A quick general question if I may? 🤔 Is there some HowTo/ Tutorial about how to setup a prefect server + agent on a beefy workstation and send/ register tasks from a laptop over the same network? Sorry if this is a noob question... Surely I simply overlook something simple... I am trying to figure out how to best scale my local processing pipelines and thought that maybe I just need to read up a bit first before trying to debug with the community help... Cheers, Christian
    k
    l
    • 3
    • 3
Powered by Linen
Title
c

Christian

04/20/2020, 6:18 PM
Hi all (again). A quick general question if I may? 🤔 Is there some HowTo/ Tutorial about how to setup a prefect server + agent on a beefy workstation and send/ register tasks from a laptop over the same network? Sorry if this is a noob question... Surely I simply overlook something simple... I am trying to figure out how to best scale my local processing pipelines and thought that maybe I just need to read up a bit first before trying to debug with the community help... Cheers, Christian
k

Kyle Moon-Wright

04/20/2020, 6:29 PM
Hey Christian, I think the Orchestration Documentation will be informative in addition to the documentation on remote environments, check it out here: https://docs.prefect.io/orchestration/#orchestration https://docs.prefect.io/orchestration/execution/remote_environment.html There’s also a more step-by-step tutorial here: https://docs.prefect.io/orchestration/tutorial/configure.html
l

Laura Lorenz (she/her)

04/20/2020, 7:05 PM
^ Kyle’s suggestion is good, you definitely might consider Prefect Cloud so you don’t have to host the orchestration layer yourself but if you want to submit flow’s to your version of prefect server and you are already on the same network, you should be able to as long as the api server config matches however you need to be networked when the client gets instantiated (ref: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/client/client.py#L87)
c

Christian

04/20/2020, 9:19 PM
Thanks to both of you. Will do some lockdown reading... 👍
🔏 1
View count: 1