I'm trying to figure out how to run some Tasks in parallel, like something like this in the second stage. There is no data passed between the stages, the tasks in the previous stage just have to complete with Success I know how to run them sequentailly like task_2(task_1()) but unclear if I should be using .map?

Hi everyone 👋, Is it possible to create a task which is triggered based on some external event (e.g. email received, or website updated specific data point, etc)? I wonder how would one go about creating such a flow?:) 🤔

Hello team, previously I was asking to clarify why it is impossible to register new flow from parent flow, here https://prefect-community.slack.com/archives/CL09KU1K7/p1600240447350500?thread_ts=1600209791.334200&cid=CL09KU1K7 Could you bring some light on that case? Like cold system start, where the one parent flow creates many new flows, initially stored in git or database. What will be the way to accomplish it with current prefect funcionality?

Hello! Thanks again for your product! It's amazing! We did installation of Prect Core in AWS and we want to do : 1. SSL certificated added on prefect 2. Have a login page to the prefect UI Do you know where I can documentation about theses 2 points ? Thanks!

Hi everyone. I have a question about

LocalResult

and exceptions. We use

LocalResult

for checkpointing, so only failed tasks get executed on a workflow re-run. So far we execute the workflow without a server, using just the basic CLI. If

LocalResult

exists, the task is skipped and just tasks with missing results get executed. We would also like to store error information on the disk. But if write exception's content as a

LocalResult

through task's

target

parameter, it gets treated as successful, because something exists at the result location, whatever that may be. Is there some established way of dumping errors to disk, so they don't just remain in the console's

stdout

, but that at the same time they don't obstruct the logic of what is successful and what not? Any ideas or comments appreciated. Jovan

Hi team, thanks a lot for Prefect, having a lot of fun working with it so far! I have two questions 1. When mapping over results of a previous task, is it possible to only continue with the ones that were successful? In my use case, I first fetch feeds for a list of urls with

fetch_feed_for_url

and then save them with

save_feed_entries

(with trigger

any_successful

). However, some of the urls might not work leading to all subsequent tasks mapping over the result (and with that the flow) also failing. What I'm looking for is to map only over "successful

feeds_list

entries"

feeds_list = fetch_feed_for_url.map(url=feed_urls)
    save_feed_entries = save_feed_entries_to_db.map(feed=feeds_list)

2. Is it possible to initialize a task from a Task class using

EnvVarSecret

? For example, creating a Task using the Task Library often requires the credentials upfront. Are all the secrets required to be available at "flow building time" to work with Task classes?

fetch_mysql = MySQLFetch(
  user=EnvVarSecret("USER)
  password=EnvVarSecret("PASSWORD)
)

Thanks!

I'm trying to use the files{} parameter with the Docker storage class, it says it can take files or directories, however it only deploys if i have a file specified, not a directory. When pointed at the directory I get a "permission denied" error, https://stackoverflow.com/questions/11278066/using-shutil-copyfile-i-get-a-python-ioerror-errno-13-permission-denied/11278116 • `files (dict, optional)`: a dictionary of files or directories to copy into the image when building. Takes the format of 

{'src': 'dest'}

Hi Everyone, Has anyone experienced their flow running an hour later after DST? Just wanted to know if there's a way to disable it or would I need to update our flows manually? Thanks!

Hi, I have prefect on a RHEL box with a windows drive mounted via smb. I would like airflow to trigger some python code using the WINDOWS process, not the linux process. Is this possible?

can you mark a task as always_run in a flow? For example I get some results but sometimes I want the later tasks to run, sometimes not, so I don't want to decorate the actual task as allways_run

Hi, I have prefect 0.13.8 installed and running on Win 10, Python 3.7.9. At the very end of trying to registering the flow (

INFO - prefect.Docker | Pushing image to the registry...

) I get a

shutil PermissionError: [WinError 5] Access is denied

. Has anybody had the same issues? We are using Python 3.7.9 due to some issues with Python 3.8 of another third party package ...

Hi, We've setup a workflow with the goal of parallelizing some behaviour. I've attached the DAG from the

visualize

command. The issue is that individual

individual_preprocessing

task should be done in parallel as well, however its not. What have we have done wrong here in our task setup?

Attached is the code for the above

Still stuck on this. I have a task that is the source of some data but it may fail to gather some of it but I want the rest of the flow to run.

another way too look at it is I have a reference task that I want to feed a pipeline, even it is has some failures

worked out a way around it, I made my initial discovery task catch all errors and simply made a return item that indicated the exception, then, in the following filter I removed those items.

Hi, quick question: I deployed prefect server in a VM in azure and everything works fine except I cannot change the default

apollo_url

like described here: https://docs.prefect.io/orchestration/server/deploy-local.html#ui-configuration I do this in `~/.prefect/config.toml`:

[server]
  [server.ui]
  apollo_url="http://<server_url>:4200/graphql"

then:

prefect server start

and it still uses the default

localhost

hence I have to manually change it in the server UI. Any idea why?

Just encountered a late run for the first time. I did everything like yesterday, but somehow my agent isn't picking up the task... Any ideas?

Any news about how to run prefect with a predefined postgres database? I have a client running on windows and it works pretty well except that all flow and logs disappear on restart due to the mount volume parameter doesn’t work on windows. This is a general docker on windows bug but if you can specify your own postgres db prefect would run fine.

Hello! Having an issue with Kubernetes cluster autoscaler for long running (> 21 min) flows similar to issue 3058. I noticed the issue has been marked "closed". What was the solution?

hey all. I am trying to register a flow but its having difficulty pickling a dependency of

paramiko

which I am using for sftp. See below stack trace. it makes sense and other people are having difficulties pickling the same object in different applications. Is the best bet to use ShellTasks to sftp?

<https://github.com/paramiko/paramiko/issues/753>

<https://stackoverflow.com/questions/39321606/cant-pickle-an-rsa-key-to-send-over-a-socket>

stack trace : https://pastebin.com/8hyAdpx1

Hi - I am running into an error where a merge statement gets triggered in a case branch unnecessarily causing the flow to fail. Is this expected behavior? https://github.com/PrefectHQ/prefect/issues/3399

Hi! I'm new to Prefect, working on my first flow, and struggling a bit with config and context in Prefect Core. I'm calling this from the command line and have the parameters going in fine and am grabbing secrets from env settings. I also understand that Prefect returns these as Tasks and not strings. Are both of these (Parameters and EnvVarSecrets) only available to the Flow? I don't seem to be able to access them from the Tasks themselves and it doesn't seem like they get automatically added to prefect.context or prefect.config. Is the only way to do this to pass them to the Task (socrata.download_dataset in this case) as parameters? I've tried adding the Secrets to both context and config programmatically but they don't seem to appear when I do this although the docs suggest this should work. My first attempt at a flow had parameters getting passed every which way and I'm trying to clean this up. Would love some guidance from the community. Thanks!

Hey guys, got a noob question here. When running:

prefect create project "Test Project X"

I assume that under the hood prefect is hitting localhost:4200, is there any way to change that url?

A dumb question, how can I redirect or config prefect logging to a local file? I read to doc but didn't find any example to do it

Greetings ! Prefect Community Im using the Prefect for the first time and trying to figure out how to get the flow to execute on the Prefect Cloud . Here is what I have done. 1. Create RUNNER Token as an Admin user 2. Build a docker container to install the prefect pip pkg and run prefect agent start at the ENTRYPOINT

ENTRYPOINT prefect agent start --token  $PREFECT_RUNNER_TOKEN -l fargate-dev

3. When I run the docker container locally - The Precct agent starts inside the container and can see the output

[2020-09-30 02:53:34,447] INFO - agent | Waiting for flow runs...

4. But the flows registered in the project never start to execute I have RTFM but still missing a link here 😕 Here is the code of my registered flow

import prefect
from prefect import task, Flow
@task
def hello_task():
    print("hello")
flow = Flow("hello-flow", tasks=[hello_task])
flow.register(project_name="Demo",labels = "fargate-dev")

Hi all, I am new to Prefect. I have a simple Hello World python script that I'm orchestrating via Prefect. In my local Prefect Core server it takes 1 sec to run from submission to execution. But from Prefect Cloud, it takes about 6-9 minutes to complete. Any help/pointers will be helpful, thanks in advance.

Hey all, is there a way to set the

min_workers

and

max_workers

as flow parameters, such that they can be changed from the cloud ui?

flow.environment = DaskKubernetesEnvironment(
        min_workers=20, max_workers=30, labels=["k8s"]
    )