Hi Prefect team, I’ve been reading a bit more on agents and I think that having diagrams in the documentation for the interactions between different agent types, the Prefect API (is this GraphQL??), and the storage and executor chosen for a Flow, would really help making it crystal clear what happens where. For example, the docs on a local agent make it very clear that it is local because the Flows that the local agent polls the Prefect API for, will be run in the same machine as where Prefect has been installed. So the logal agent is both local in the sense that it is 1) a process local to the machine where Prefect is installed, and 2) runs Flows in the same machine as where Prefect is installed. But when I read the Kubernetes agent docs, it is not clear to me whether the Kubernetes agent process runs in the same environment/machine as where Prefect is installed and pushes Flows to a Kubernetes cluster, or whether we install a Kubernetes agent in our Kubernetes cluster and it pulls Flows to the cluster. Or are both valid options?

prefect agent kubernetes start

would start a Kubernetes agent in the environment where Prefect is installed but

prefect agent install kubernetes

installs the agent in the Kubernetes cluster right? Having diagrams for these interactions would make it more intuitive to read the docs.

Hi everyone. Does any one have experience/examples of getting a large response from a http get call, and writing to s3? Prefect has the map functionality, which seems like a good way to do this. If I were not using prefect, I'd do something like this:

s3_client = boto3.client('s3')

with requests.request("GET", url, stream=True) as r:
     r.raise_for_status()

     for chunk in r:
        s3_client.put_object(Body=chunk,
                             Bucket='xxx',
                             Key='xxx')

Hi Prefect team, i’m new to prefect[the project looks amazing! and exactly what I was looking for] I have one basic question - is that possible to run locally my flow [much like flow.run()] without any agent including local agent, BUT - still send the results and logs to the server? I know I can use the client object to submit a flow run and catch it with a local agent but i want to skip this step

Hi Prefect Team, We have a prefect-server running on a Kubernetes cluster and just updated to

0.13.19.

. As environments are being deprecated I am trying to move the flows over to run_config setup. I am using the Docker storage and KubernetesRun for run_config. I noticed that the container that is built with

KubernetesRun()

is configured with:

command: /bin/sh -c
command arguments: prefect execute flow-run

but when I inject a job spec file with`KubernetesRun(job_template_path="job_specs.yaml")` (as I need to to access the Kubernetes secrets that I need for the flow to run) only the

command arguments

are written by the prefect build and the

command

is left blank. That leads to the container failing with an

"executable not found"

A workaround is to specifically include the

command

section in the job specs file

command: ["/bin/sh", "-c"]

. That doesn’t seem to be expected behaviour, or am I missing something?

Hey community, We have a mapped task in our flow, spawning several thousand mapped tasks, whose output it then forwarded to one task (unmapped). We noticed that these tasks are being called twice. Is this expected in some way? This happened to me before, but I thought this was already solved. prefect 13.1 running on dask-kubernetes

Hey all, New to the community and Prefect in general, but a bit in love with it already 🙂 I have a basic architectural question, and I hope this is a good place to ask it - otherwise, please let me know! In a nutshell, I am looking to

map

case

. More in detail, I am converting into Prefect an existing process currently run locally through a simple CLI, to fetch and handle datasets from a remote server. One of the first things in this process is checking the dataset version, using 

check_version(id)

 which returns a 

bool

. In the CLI this is then used in a simple logical block to determine how to continue. In the CLI a list of ids is handed and looped over, making that check for each element in the list (sequentially). I want to convert this into Prefect and take advantage of the parallelisation opportunities. Therefore, I thought that 

case

 and 

map

 would work great here, something like this (which is not possible):

ids = [id1, id2, id3]
with Flow('Check version') as flow:
    with case (check_version.map(id=ids), True):
        val1 = action_if_true()
    with case (check_version.map(id=ids), False):
        val2 = action_if_false()

The issue here being that the output from 

check_version(id)

 is mapped, and is therefore a list of bools - 

[True, True, False]

, which cannot be compared against a 

bool

. I found no option to map `case`. I did however see the option to map `switch`, and I could imagine using that if I have to, but since it must take strings as input (it uses a

dict

to get the different scenarios) it would require a conversion or changing the original functions in the CLI, which I would rather not do. This is a pretty simple thing, and I imagine someone else has solved this before, but could not locate anything in the docs or other places. Or perhaps I am thinking about this in the wrong way. Any input is welcome, thanks!

Hello Prefect team. I have a question regarding the prefect context. Is it possible to set prefect context in one task (so on runtime) and use those values in other tasks? It works very well when running it locally but I have no success so far when deploying it to prefect cloud. As things late dates are set dynamically, there should be some way to do that.

hey there — I am designing a flow that requires

pyodbc

to be installed in order to connect to a SQLServer instance… i have pulled Prefects

master

docker image and ran interactively to test out `pip install pyodbc`… running into gcc issues, as expected, but i am now trying to

apt-get

or

apt install

an odbc driver

unixodbc-dev

and getting weird “not found” type results… i have run into similar issues, but was relying on a conda environment which allowed me to

conda install pyodbc

but how can i replicate this here with

pip

?

Greetings! Could you please help me with a simple question? I created a duplicated prefect project, using the command:

prefect create project "my-project"

Can we also delete projects, using a similar command ?

New server, this error isn't telling much:

#> prefect auth login -t "<my token>"
Error attempting to communicate with Prefect Cloud

I can confirm that

~/.prefect/config.toml

isn't being created. Trying to use

python3 start.py

to register any flow gives:

prefect.utilities.exceptions.ClientError: Malformed response received from Cloud - please ensure that you have an API token properly configured.

which unfortunately doesn't add any info. Pinging

<http://api.prefect.io|api.prefect.io>

works:

PING <http://api.prefect.io|api.prefect.io> (34.96.72.220) 56(84) bytes of data.
64 bytes from <http://220.72.96.34.bc.googleusercontent.com|220.72.96.34.bc.googleusercontent.com> (34.96.72.220): icmp_seq=1 ttl=116 time=3.71 ms

Any ideas?

Dask not running mapped tasks in parallel - any ideas? (details: https://prefect-community.slack.com/archives/CL09KU1K7/p1607561409431800?thread_ts=1607549801.425600&cid=CL09KU1K7)

Hello all. I'm part of a team evaluating Prefect for my employer. Is there any data available on the performance of various components with very large graphs? I have scenarios that could extend up to 10M nodes and would regularly run around 1M nodes.

Hey, is it possible to use context value as a default for Parameter?

Hello, I just read somewhere on the prefect website that prefect no-longer needs docker but I can't find a blogpost of webpage detailing that. Could you please help me with a link / confirmation? Have the tutorials been updated?

Does anybody know if there's a way to make prefect agents / prefect server log tracebacks for exceptions? Some of the errors get a little hard for me to debug.

Hello hello dear folks, A very basic question - just started looking at the tool. If I understand correctly the cloud version deals only with UI, scheduling, etc. and we still need to provide our own infrastructure to host code (storage) and run the work (agents). There is no fully managed service. Am I following along correctly?

Hi everyone. is it possible to set a task to run at the end of flow in any case. I want to run somekind of ‘cleanup’ task and i want it to run even if the flow is failing

Has anyone run into this issue when trying to run

flow.visualize()

? I installed the viz package with

pip install prefect[viz]

but am still getting this:

Traceback (most recent call last):
  File "insurance_export.py", line 184, in <module>
    flow.visualize()
  File "/home/ec2-user/anaconda3/envs/python38_prefect/lib/python3.8/site-packages/prefect/core/flow.py", line 1406, in visualize
    graph.render(tmp.name, view=True)
  File "/home/ec2-user/anaconda3/envs/python38_prefect/lib/python3.8/site-packages/graphviz/files.py", line 216, in render
    self._view(rendered, self._format, quiet_view)
  File "/home/ec2-user/anaconda3/envs/python38_prefect/lib/python3.8/site-packages/graphviz/files.py", line 262, in _view
    view_method(filepath, quiet)
  File "/home/ec2-user/anaconda3/envs/python38_prefect/lib/python3.8/site-packages/graphviz/backend.py", line 323, in view_unixoid
    popen_func(cmd)
  File "/home/ec2-user/anaconda3/envs/python38_prefect/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/ec2-user/anaconda3/envs/python38_prefect/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'xdg-open'

Hi guys. is there any way to securely pass the result of a task to the next task?

If I set the

run_config

for my flow like this:

with Flow("dask_flow_test", run_config=KubernetesRun()) as flow:

Then if I run it locally, will it create a kubernetes job? If I don't want it to create a kuberentes job when I run it locally, should I instead set the

run_config

to

LocalRun

there, and then before I register the flow I can set it to

KubernetesRun

instead?

Hi Community, I am trying to save the execution log to local file system in a hybrid prefect cloud with local runner mode with the following code

import prefect
import prefect.utilities.context
from prefect.environments.storage import Local
from prefect.engine.results import LocalResult
from prefect import task, Flow

with prefect.context(a=1, b=2):
    print(prefect.context.a) # 1

@task(name='task_today')
def report_start_day():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>(prefect.context.today)
 
    <http://logger.info|logger.info>(prefect.context.flow_run_id)

with Flow('My flow', storage=Local(), result=LocalResult()) as flow:
	report_start_day()

flow.register(project_name="prefect_first_project")

I can seem to find where the log was saved to

I am running the prefect agent on a virtual python env

I was hoping the following logs in the cloud can be save in my local execution environment.

Hi all. I'm trying to implement a docker storage flow on a prefect server instance on a workstation in our network. I build a custom docker base image and push it to docker.io. However, I get an error registering my flow... Not sure where this might originate?

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: <http+docker://localhost/v1.40/images/create?tag=0.13.19-python3.9&fromImage=prefecthq%2Fprefect>

more trace ...

File "/Users/werner-ch/.pyenv/versions/3.9.0/envs/dataflow/lib/python3.9/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error for <http+docker://localhost/v1.40/images/create?tag=0.13.19-python3.9&fromImage=prefecthq%2Fprefect>: Not Found ("manifest for prefecthq/prefect:0.13.19-python3.9 not found: manifest unknown: manifest unknown")

Would anyone know where I should start digging? My base image is: https://hub.docker.com/r/cwerner/dataflow

Hi all, is anyone randomly getting errors from StartFlowRun tasks.

Unexpected error: ClientError([{'path': ['create_flow_run'], 'message': '[{\'extensions\': {\'path\': \'$.variableValues.insert_objects[0].idempotency_key\', \'code\': \'validation-failed\'}, \'message\': \'field "idempotency_key" not found in type: \\\'flow_run_insert_input\\\'\'}]', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])

We got the error on 13.16 then reverted back to 13.13 and are still getting the error.

has anyone ever encountered this issue before? I'm orchestrating a flow of flows, where the parent flow is running in a docker container and is launching its child flows in sibling docker containers, and am getting this error when my parent flow tries to launch my child flow:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 856, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/executors.py", line 298, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/tasks.py", line 449, in method
    return run_method(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/tasks/prefect/flow_run.py", line 160, in run
    flow_run_id = client.create_flow_run(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1041, in create_flow_run
    res = self.graphql(create_mutation, variables=dict(input=inputs))
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 318, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['create_flow_run'], 'message': '[{\'extensions\': {\'path\': \'$.variableValues.insert_objects[0].idempotency_key\', \'code\': \'validation-failed\'}, \'message\': \'field "idempotency_key" not found in type: \\\'flow_run_insert_input\\\'\'}]', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

all my containers are running prefect 0.13.18 on python 3.8

wow, that is crazy timing

Hi All! I’m trying to get stdout from my tasks to go to the UI (with the docker agent) and not having any luck. I’m setting the

PREFECT__LOGGING__EXTRA_LOGGERS

environment variable both when launching my agent and injecting that variable into my docker container at runtime (with the

-env

arg). Any advice on how to proceed?