Hello I’m trying a simple flow of doing linear regression in batches. The flow works when doing it sequentially but when I try with Dask backend it causes memory problems. What is confusing is that there is ample memory per worker. Can someone help me identify the problem?

Here’s the notebook version

I have a question about prefect cloud. If I get it correctly, they provide the scheduler and flows/jobs UI with metadata storage. On our side we have to provide the agents (nodes) where the actual computation is going to be performed. Is that correct?

Hey everyone - I use a parameter to pass a date to tasks downstream

batch_run_date = Parameter(
        name='batch_run_date',
        default=(datetime.today() - timedelta(days=1)).strftime('%Y-%m-%d')
    )

in perfect

0.9.*

this would alter the default value based on the current date. But in prefect

0.10.1

the default date is fixed. Is there a a different way I can reproduce the previous functionality

I registered my flow (tested successfully locally) to the server following the instructions here: https://docs.prefect.io/orchestration/tutorial/first.html and got this error when triggering the flow from both the CLI and UI. April 17th 2020 at 112239am | agent ERROR lens [Errno 2] No such file or directory: ‘prefect’: ‘prefect’

Is it true that the GUI is not available in the community server?

I've been having a vexing time getting this launched. I kick off the docker container using the master tag and it exits immediately with no logs or errors. When I shell into the container and try to launch the server, it errors out with "FileNotFoundError: [Errno 2] No such file or directory: 'docker-compose': 'docker-compose'"

is it possible to run only parts of a flow instead of the whole thing from the orchestrator?

like if there were tasks a, b, and c to only run b and c

and is it possible to query against the orchestrators database from a separate app?

Howdy! I'm currently working on adding a way to roll back a potentially bad deploy and was wondering if there is a way to have prefect basically "stop the world" on a particular flow

I have code currently that will set all flow runs + their child task runs to cancelled but it looks like that won't stop tasks from being submitted to Dask

I'm using the

DaskExecutor

if that helps

I'm using jaydebeapi on windows and prefect keeps throwing the error java.sql.SQLException: No suitable driver found for. This flow works when not using prefect server. However, it appears it can't find the jar file possibly. Where is the best place to put the file? This is also using dask distributed

Question about setting

upstream_tasks

to the same task being called twice with different arguments. I have something like

A(abcd)
B.set_dependencies(upstream_tasks=[A], keyword_tasks=...)
A(defg)
B.set_dependencies(upstream_tasks=[A], keyword_tasks=...)

where B must run after A, and each gets called with two separate sets of Parameters. If I call

B.set_dependencies(upstream_tasks=[A])

twice, will the second B wait on the second A? And is this the idiomatic way to do this?

Hi everyone 👋 Data engineer with a remote sensing/ IOT sensor background here. I'm looking at Prefect to orchestrate processing-intensive scientific computing workflows on AWS. Currently, we're using AWS Batch + ECS to run docker containers but it's a really limited map-reduce thing. Ideally we'd like a system that would allow us to leverage our existing docker containers and build them into more complex, parameterized DAG topologies. We've tried Airflow... and hit almost all of the limitations mentioned in prefect's docs 🙄 We're just getting started with Prefect but it looks very promising so far! Are there any other folks using prefect to tie together already-dockerized processes?

Is it possible to set the flow run name when using a Clock? I’m using one flow, multiple clocks with different parameters and it would be useful to be able to name the flow-run

We're running into configuration issues with the graphql url. We've set server.ui.graphql_url to the proper host and path, but the UI doesn't use the config value. Is there a param for this not listed in the example config.toml?

Lessons from the front line: Don't fire up multiple agents on one machine and background all the processes. You'll definitely regret it. 😂

👋 new prefect user, currently in evaluation phase as a replacement to airflow. for reference i’m running airflow in a container, with scheduler and webserver as separate containers and multiple worker containers, all of this is on a 3 node cluster. is there a doc on how to run prefect containers similarly? i see install doc uses prefect server command which launches containers, but not a whole lot of info on how to setup this w/o that command

Has anyone experimented with running a K8 agent on GKE with Anthos for serverless execution of perfect jobs?

Hi, We had a task failure (it seems a task run time-out) last night. it seems prefect didn’t retry this tasks when it failed. We have to resolve to a manual recover this morning. Could you recommend a solution to allow prefect to handle this failure itself? (see error details in thread)

Hi! I am experimenting with the local prefect server. I know how to start it but how do I stop it?

Has anyone used prefect as a wrapper to interact with a HPC cluster? Like wrapping the

bsub

or

lsub

command into a prefect task. Basically how do you use prefect to submit jobs to a compute cluster that has no automatic callback API for job completion and you have to ping it continuously to check the status? How do you check if the HPC job is complete? Do you parse the output of the HPC’s status command, check existence of an output file or something else?

Hi there! I'm giving Prefect Core a try, but I could not find answers to a few questions in the docs. Is it possible to configure multi-user access (with passwords etc.) to Prefect Server? Or is it a Cloud feature?

Hello everyone 👋 I'm Martin Guindon, consultant for Montreal Analytics, based in Montreal, Canada.

Hi guys! I have a question about task options; can I combine the

retries

option with the

timeout

? Let me explain better: Inside my task I am polling an API waiting for something to finish. If one of those HTTP calls fail, I'd like to retry, but if it takes too long and that something hasn't finished, I want the entire task to timeout. Is that how combining those two options work? I'm afraid that a timeout would cause a retry and start all over again.

Hi all. I currently try to test

prefect core server

. I have an instance of prefect server () and an agent running on a remote machine (via a VPN connection). Then I set

export PREFECT__CLOUD__API=<http://myserver.edu:4200>

Finally, I run my dummy flow which contains a flow.register() at the end... The output on the command is:

Result Handler check: OK                                                                       │
Flow: <http://localhost:8080/flow/17b029f0-fb9d-4ac5-bc54-fc4102a44631>

Is this OK? The task does not show up on the UI. Anything I'm missing? Do I have to configure something else for this? My local laptops' diagnostic:

{
  "config_overrides": {},
  "env_vars": [
    "PREFECT__CLOUD__API"
  ],
  "system_information": {
    "platform": "macOS-10.14.6-x86_64-i386-64bit",
    "prefect_version": "0.10.2",
    "python_version": "3.8.2"
  }
}

And the servers' diagnostic:

{
  "config_overrides": {},
  "env_vars": [],
  "system_information": {
    "platform": "Linux-5.3.0-40-generic-x86_64-with-glibc2.29",
    "prefect_version": "0.10.2",
    "python_version": "3.8.2"
  }
}

Cheers, C

Hi all (again). A quick general question if I may? 🤔 Is there some HowTo/ Tutorial about how to setup a prefect server + agent on a beefy workstation and send/ register tasks from a laptop over the same network? Sorry if this is a noob question... Surely I simply overlook something simple... I am trying to figure out how to best scale my local processing pipelines and thought that maybe I just need to read up a bit first before trying to debug with the community help... Cheers, Christian

@josh - Not sure if you were the josh in during the PyCon Zoom meeting. If so, would it be possible to get the link to the docker examples again?