Hi! Quick question: is there currently any way to set flow-level concurrency limits? (rather than just task level concurrency with tags)

Is it possible to use the CLI client to search for the Flows in a given Tenant/Project by the Flow Name? I'm trying to schedule a flow, where the user only has the display name of the Flow. I see

client.create_flow_run

but that want's the UUID. I'm looking for a display name to UUID map, similar to how the UI does it.

for those hitting issue #2237 (

localhost

hardcoded in the UI) when deploying Server to an external server, this stopgap might help you get going:

socat TCP-LISTEN:4200,fork,reuseaddr TCP:your.prefect.api.host:4200

(TCP proxy from

localhost:4200

to the external Prefect API server)

Hi guys! I'm new to Prefect. I'm thrilled with the UI going Open Source, so we're planning to start using Prefect for our pipelines 🙂

First of all I wanted to thank you for your hard work and the awesome project. Also for releasing it to the community!

My fist question: I'm creating a Flow for one of the pipelines, and my Python file is getting really big already. Is it possible to split the file and import, for example, tasks from another .py file?

GM, I started working with caching this morning and got some unexpected behavior. When I run the flow without the LocalResultHandler I get the correct final log message with the transformed data.

INFO - prefect.Task: write_data | {'col_1': [6, 4, 2, 0], 'col_2': ['a', 'b', 'c', 'd']}

When I run it with the LocalResultHandler the xform_data stage is more or less skipped and I get an incorrect result.

INFO - prefect.Task: write_data | {'col_1': [3, 2, 1, 0], 'col_2': ['a', 'b', 'c', 'd']}

If I look in the log I see that the xform_data stage isn't run. Not sure why that happens, but also not sure how the data passed to write_data is the data that was passed to xform_data. It seems like the two sets of cached data are the same and xform_data sees the cached data from load_data and gets skipped.

April 15th 2020 at 8:45:25am | prefect.CloudTaskRunner
DEBUG 
Task 'xform_data': 1 candidate cached states were found
April 15th 2020 at 8:45:25am | prefect.LocalResultHandler
DEBUG 
Starting to read result from /tmp/inter/prefect-result-2020-04-15t14-45-25-356078-00-00...
April 15th 2020 at 8:45:25am | prefect.LocalResultHandler
DEBUG 
Finished reading result from /tmp/inter/prefect-result-2020-04-15t14-45-25-356078-00-00...
April 15th 2020 at 8:45:25am | prefect.CloudTaskRunner
DEBUG 
Task 'xform_data': Handling state change from Pending to Cached
April 15th 2020 at 8:45:25am | prefect.CloudTaskRunner
DEBUG 
Task 'xform_data': can't set state to Running because it isn't Pending; ending run.
April 15th 2020 at 8:45:25am | prefect.CloudTaskRunner
INFO 
Task 'xform_data': finished task run for task with final state: 'Cached'
April 15th 2020 at 8:45:25am | prefect.CloudTaskRunner
INFO 
Task 'write_data': Starting task run...

This seems to be the line that I don't expect.

Task 'xform_data': can't set state to Running because it isn't Pending; ending run.

Anyone using Prefect instead of something like Kubeflow?

👋 I’m here! My Name's Edidiong I saw Prefect has been Open Sourced. I wanna try and contribute to it's growth.

Anyone else running into the problem caused by loading yaml file when trying to do

prefect server start

? This is a fresh environment. I thought it might be incompatibility with python 3.8 but I saw the 3.8 compatibility was merged.

👋 For anyone curious, @Laura Lorenz (she/her) and @Andrew Vaccaro are hosting a virtual PyData Denver tutorial in about 15 minutes - you can join here!

Hey everyone! In our setup our stdout logs are getting shipped to DataDog and parsed to determine if they are errors or warnings, so I need to change the format of the log messages to match the parser in DataDog. I know how to do this with the logging library but I'm not sure what the smart way of doing this in a decorator task in production (no Cloud for now just cron for testing): 1. Is it possible to change the timestamp format? 2. Is it possible to change the level within a single flow?

Hi there 👋 - enjoying the new Prefect releases. Thanks to the

--postgres-port

feature I was able to deploy on my NAS at home. I’m not treating the NAS like a development environment, so I’m testing out flows on my laptop before scheduling on the NAS. My question is - how do I deploy a flow I’ve written locally to another server? Should I run something like

prefect backend server

with an option for the server address

192.168.103.210:8080

?

Hi all! I've been trying to write a ""dynamic"" flow, and wondered if anyone has any comments. The flow downloads a list of objects

A

, and then maps 5 tasks over that list. Pretty straightforward, until the part where objects

A

include information about what other objects

A'

(A dash) they depend on... And those

A'

objects have to be processed and mapped over the same tasks as

A

, before

A

. This can go several layers deep, with

A->A'->A''

dependencies that need to be taken care of dynamically. The most naive solution is just to insert the dependencies at the beginning of the original list, hence making some kind of priority queue, and mapping over the tasks. However, this would not work with a DaskExecutor -- since everything is in parallel. What I guess I need here is "sub-flows" that can be mapped over a list of lists. It seems there's some discussion on it going on https://github.com/PrefectHQ/prefect/issues/1745 , but since it's far from release yet, do y'all think a similar thing can be hacked together, now? Thanks!!

I’m trying to follow the getting started tutorial, but I can’t manage to start the prefect server in docker. I’m new to docker and trying to get it to work on my macbook. I followed https://docs.prefect.io/core/getting_started/installation.html#docker and then tried https://docs.prefect.io/core/getting_started/installation.html#docker This results in this error: FileNotFoundError: [Errno 2] No such file or directory: ‘ip’: ‘ip’ Any ideas what I might be doing wrong?

When a Flow is running, is it aware of it's own flow_run_id? I have a use case where it'd be helpful to have a Flow execute a workflow, then have a final Task where it uploads the results of that Flow to an Object Store with the flow_run_id as the filename.

Hello all, I am trying to create a project in Prefect's local backend from version 10.0, but I am getting the following error:

$ prefect create project "Test-Project"


....
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "python3.6/http/client.py", line 1354, in getresponse
    response.begin()
  File "python3.6/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "python3.6/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

As anyone experienced this before?

Hi team - I have some tasks that use the

slack_notifier

state handler (via the task kwarg), but I’d like to do some local testing without trigger the handler. Is there a

context

or

set_temp_config

I could use to mute it ?

I’m trying to make a docker image that’ll allow me to run my flows within it. I attempted to use

prefecthq/prefect:latest

as the base image, but encountered pyscopg2 installation issues, so I decided to just pip install prefect within the image. The example below is super basic (just want to be able to run a

prefect

command). However, when I run the image, I get the following error:

/bin/sh: 1: prefect: not found

I’ve also attempted the following variations in place of the

CMD prefect version

command: •

RUN ["/bin/bash", "-c", "prefect version"]

•

CMD ["prefect", "version"]

Any ideas? 😄

Hi guys im trying to install prefect, but all times this error appears. Whats the most easy way to install prefect ? Thanks

/home/luan/anaconda3/lib/python3.7/site-packages/prefect/cli/server.py:236: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read <https://msg.pyyaml.org/load> for full details.

y = yaml.load(file)

`ERROR: Couldn't connect to Docker daemon - you might need to run

docker-machine start default

.`

Exception caught; killing services (press ctrl-C to force)

`ERROR: Couldn't connect to Docker daemon - you might need to run

docker-machine start default

.`

Traceback (most recent call last):

File "/home/luan/anaconda3/lib/python3.7/site-packages/prefect/cli/server.py", line 291, in start

["docker-compose", "pull"], cwd=compose_dir_path, env=env

File "/home/luan/anaconda3/lib/python3.7/subprocess.py", line 347, in check_call

raise CalledProcessError(retcode, cmd)

subprocess.CalledProcessError: Command '['docker-compose', 'pull']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/home/luan/anaconda3/bin/prefect", line 10, in <module>

sys.exit(cli())

File "/home/luan/anaconda3/lib/python3.7/site-packages/click/core.py", line 764, in __call__

return self.main(*args, **kwargs)

File "/home/luan/anaconda3/lib/python3.7/site-packages/click/core.py", line 717, in main

rv = self.invoke(ctx)

File "/home/luan/anaconda3/lib/python3.7/site-packages/click/core.py", line 1137, in invoke

return _process_result(sub_ctx.command.invoke(sub_ctx))

File "/home/luan/anaconda3/lib/python3.7/site-packages/click/core.py", line 1137, in invoke

return _process_result(sub_ctx.command.invoke(sub_ctx))

File "/home/luan/anaconda3/lib/python3.7/site-packages/click/core.py", line 956, in invoke

return ctx.invoke(self.callback, **ctx.params)

File "/home/luan/anaconda3/lib/python3.7/site-packages/click/core.py", line 555, in invoke

return callback(*args, **kwargs)

File "/home/luan/anaconda3/lib/python3.7/site-packages/prefect/cli/server.py", line 307, in start

["docker-compose", "down"], cwd=compose_dir_path, env=env

File "/home/luan/anaconda3/lib/python3.7/subprocess.py", line 395, in check_output

**kwargs).stdout

File "/home/luan/anaconda3/lib/python3.7/subprocess.py", line 487, in run

output=stdout, stderr=stderr)

subprocess.CalledProcessError: Command '['docker-compose', 'down']' returned non-zero exit status 1.

I get the following error when I try to register a flow. Any ideas? TypeError: can't pickle JavaField objects

Hi, I'm very new to Prefect and trying to get some examples up and running. i've managed to successfully run my workflow locally by just executing the python scripts containing my flow using flow.run(). I've also managed to set up server running on my machine and have registered my flow using flow.register(). When I try to execute in my local UI it gives me a 'module not found error' understandably as my flow is dependent on other classes and I have created. I can successfully get a simple example that doesn't reference other modules running so have ruled that out. I've found other threads here advising that my

Flow

needs a

storage

keyword and that storage needs to create a

Docker

instance with a

dockerfile

keyword. I have tried this on a very basic example passing a dockerfile and I keep getting the same error

500 Server Error: Internal Server Error ("Cannot locate specified Dockerfile: .\tmp64wpj0sv\Dockerfile")

after I execute flow.register() Simple example was taken from tutorial and my dockerfile is located in the same location (note i'm pretty certain it can find the file as I get a different error message if the file can't actually be located)

import prefect

from prefect import task, Flow

from prefect.environments.storage import Docker

@task

def hello_task():

    

logger = prefect.context.get("logger")

<http://logger.info|logger.info>("Hello, Cloud!")

flow = Flow("hello-flow", tasks=[hello_task])

flow.storage = Docker(dockerfile = 'Dockerfile')

flow.register(project_name="hello-flow")

Dockerfile content:

FROM ubuntu:18.04

Any assistance on how I should actually be specifying my dockerfile to add the required modules to my actual workflow is appreciated Thanks

Hi all, is there a way to use different executors for different task ? Like if some of my tasks can't be pickled, I want it on local executor, with the remaining on daskexecutor..

If I register a flow and set the proper environment variables for dask. It should use the dask to run it correct?

Hi All, really interested in Prefect and so far enjoyed playing with it. My questions is around tasks calling other tasks. Can you pass the flow context into other tasks, so that you can set dependencies or perform map functions? Or is it by design that all task orchestration must happen at the top level within the flow context? TaskA does 100 computations, where each computation needs info from TaskB which itself does 100 computations. I'm wanting to take advantage of Prefect distributed nature of tasks out of the box. So i can spin a cluster up and get at least 100 computations all happening on their own nodes.

How do restart a prefect server and agent?

Hi all, I've been playing with prefect for a while now to see if we can move our ETL engine. It's been crazy to see how much it reduces our code size when converting our current stuff!

I have a question that is sort of related to @Ralph Willgoss’s above I think. So we have a common pattern where as part of a complex etl flow, the extract is from a paged API, with the transform and load taking the bulk of the time of the flow.

I was wondering if there was a good pattern to trigger a second flow (using the next_page token as a parameter) from a task?