What's the best way to report a bug? On this line, the key for s3 is wrong. The leading

/

needs to be removed.

Hi there. I was under the impression that I the

prefect server stop

command had been added. I lost the connection to the server. Is there a graceful way to stop it once it's running? If I wanted to use the docker compose file directly to run server, what is the easiest way to set all the variables that are needed in this file?

AFAIK the logs are stored in the database right? If I want to move that to a different backend (google bucket or stackdriver), are the logs then still retrievable from the UI? Or are there building blocks to implement that?

Hi! I'm new to Prefect and wish to get some help from experts. We have a POC system using prefect. We just do "prefect server start" to start a local server, and "prefect agent start" on the same server to run the code. Everything works fine for weeks. After some heavy load testing, we started to get error saying "Unexpected error: OSError(28, 'No space left on device') " If we run our code directly using flow.run(), it works fine. But if the same code for flow.register(), it failed with "Unexpected error: OSError(28, 'No space left on device') " We have tried to remove all images and clear docker volumes, but still getting this error. The disk space usage looks OK on that server. Can someone help to shed some light on this issue? Thanks in advance

Hi, a question about using

with case

and

upstream_tasks

together we have a set of 3 tasks that need to run one after the other (no values shared between them)

A -> B -> C

we also would like to have the ability to run only:

A -> C

that means that

B

should go inside

with case

, but then what happens is that

C

is skipped, because it has

upstream_tasks

related to

B

any solutions available?

Hello there. Seems like there is no way to "detach" local 'prefect server start' from terminal out of the box, and making own docker-compose with all env's seems overwhelming since it will have to be checked and updated manually every time prefect updates. Am i missing something, or currently there is no way to actually deploy 'official' deployment of on-premise Prefect Core in any kind of infrastructure (service in VM, deployment in k8s, etc) and only way to use it is through Prefect Cloud?

Hi, a question about agent. I created 1k flow runs. LocalAgent started to deploy all these flows using popen(prefect execute flow-run). And it created 1k processes. All this processes used all available memory. Is it possible to limit number of deployed flows? I use OSS version. Thanks in advance

Hi, has anyone had any issues with 0.14.0? I see a lot of things breaking... For example, GitLab storage does not add its default label any more. Also the agent starts with no default label other than the IP address And a bunch of name spaces not found anymore:

No module named 'prefect.engine.result_handlers'

on the agent

Hey, when pinning my helm deployed prefect to 0.14.0, I noticed the tags 0.14.0 and core-0.14.0. As I’m running it myself I probably need the core- version. However, when I used that version for the Kubernetes Agent, it fails to start (see thread for logs).

How to use any global object/variable/parameter in task definitions. For example, my flow fails when I declare the client in the global scope in the below code

#!/usr/bin/python3

from prefect import task, Flow import boto3 import json client = boto3.client('sagemaker', region_name='us-east-1'                       ) @task def list_training_jobs():     response = client.list_training_jobs()     print(response) with Flow("SageMakerFlow") as flow:     training_jobs = list_training_jobs()     print(training_jobs)      flow.register(project_name="tutorial") flow.run()

Hi- I'm trying to understand why my runs are not showing up in the run. I have the local agent running and when I run the following code, I don't see the run in the UI

import prefect

from prefect import task, Flow

@task

def say_hello():

    

logger = prefect.context.get("logger")

<http://logger.info|logger.info>("Hello, Cloud!")

with Flow("hello-flow") as flow:

    

say_hello()

# Register the flow under the "tutorial" project

flow.register(project_name="tutorial")

flow.run()

Hello all, I am trialing out Prefect in my organization and am currently trying to use prefect to run existing docker containers. However, I am not finding documentation/tutorials that show how I can run existing built docker images with a prefect flow. I have prefect locally installed and am using the OSS version of it. I am basically wanting to just run the existing container, as is, with prefect. Any suggestions or documentation that I can look at to assist in this goal?

Hi. We are trying to deploy Prefect Server at a client and their security team noticed that the server loads https://cdn.lr-ingest.io/logger-1.min.js and a js from js.stripe.com. Could you comment on the purpose of these scrips how they can be disabled? We are starting the server with this env var

PREFECT_SERVER__TELEMETRY__ENABLED=false

Thanks!

Hi! Whenever I rerun a flow, it reuses values from the previous run for the parameters declared globally. How should I handle this?

#!/usr/bin/python3

from datetime import datetime

from prefect import task, Flow, Parameter, context, case

from prefect.utilities.debug import is_serializable

logger = context.get("logger")

timestamp = datetime.now().strftime("%Y-%m-%dT%H-%M-%S")

@task

def print_timestamp():

<http://logger.info|logger.info>(timestamp)

with Flow("Test") as flow:

    

print_timestamp()

serializable = is_serializable(flow)

if serializable:

    

flow.register(project_name="Default",

                  

idempotency_key=flow.serialized_hash())

flow.run()

Attached screenshot from two runs. As you could see, the value of the global parameter

timestamp

hasn't changed

Are there plans to add prefect-server helm chart to a helm repository?

If I want to run a flow every Saturday, would I use an

IntervalClock

or a

CronClock

? Which is more in line with what Prefect expects?

Using a self-hosted server, how do I create an access token? Using the cloud option, I can click User -> User Settings -> Personal Access Tokens. But with the self-hosted server it doesn't even show a User icon in the top right.

Hey, I'm encountering an issue trying to run the server on a remote machine. The apollo server does not seem to start up properly. After running

prefect backend server

, I start the server with

prefect server start --postgres-port 4301 --hasura-port 4302 --graphql-port 4303 --ui-port 4304 --use-volume --volume-path /home/arnaud/mco/prefect/pgdata

(Mapping ports as some ports are already in use). In the config.toml, I changed the following:

[server]
	[server.ui]
	apollo_url = "<http://192.168.176.193>:${server.host_port}/graphql"

After starting the server, all services seem to start properly and

docker ps

indicates all services are up (and all ports mapped correctly). However, as you can see below, the are no logs for the apollo server. Running

prefect server create-tenant --name default --slug default

results in

requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.176.193', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4b14a1dc70>: Failed to establish a new connection: [Errno 111] Connection refused'))

and I cannot connect to http://192.168.176.193:4200/graphql (nor can the UI). Can anyone help me with this ?

Hi I'm new to prefect. I'm trying the docker image and ran into this issue

$docker run -it prefecthq/prefect:latest bash

... within the container
root@8afe017ae8ef:/# prefect backend server
Backend switched to server
root@8afe017ae8ef:/# prefect server start
Exception caught; killing services (press ctrl-C to force)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/server.py", line 347, in start
    ["docker-compose", "pull"], cwd=compose_dir_path, env=env
  File "/usr/local/lib/python3.7/subprocess.py", line 358, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/local/lib/python3.7/subprocess.py", line 339, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'docker-compose': 'docker-compose'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/server.py", line 385, in start
    ["docker-compose", "down"], cwd=compose_dir_path, env=env
  File "/usr/local/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/usr/local/lib/python3.7/subprocess.py", line 488, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'docker-compose': 'docker-compose'

It's likely I'm missing something basic. Any tips appreciated. Thanks.

Hi folks, quick question about data persistence... we are running Prefect Server on an EC2 instance, and we just upgraded from

0.13.x

to

0.14.2

by running these commands:

pip install --upgrade prefect
prefect server stop
prefect server start --detach

Unfortunately, after the upgrade had finished all of the projects, registered flows, and flow run history have been lost. Is there any way to prevent this from happening after future upgrades?

Would it be possible to extend

PostgresFetch

task to accept an optional boolean arg that indicates whether you want the column names returned with the query or not?

Hi Experts, any idea why this code cannot trigger "Quick run" via Prefect UI? this job stays

scheduled

forever without getting triggered, , any idea?

import prefect
from prefect.tasks.shell import ShellTask
from prefect import task, Flow

task = ShellTask(helper_script="cd /prefect")

with Flow("ShellScriptTest") as f:
    mv_file = task(command='cp -rf /prefect/test.py /prefect/kubernetes/')

out = f.register(project_name="ShellScriptTest")

Hi, I'm having trouble with Prefect Server. It came up, I was able to register a project in it, and I was also able to register a flow. But when I try to run the flow, I get an error:

agent@prefect-agent:~/.prefect/results$ prefect run flow --name 'My first Flow' --project test
    Traceback (most recent call last):
      File "/home/agent/.pyenv/versions/3.7.3/bin/prefect", line 10, in <module>
        sys.exit(cli())
      File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/cli/run.py", line 186, in flow
        run_name=run_name,
      File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/client/client.py", line 1094, in create_flow_run
        res = self.graphql(create_mutation, variables=dict(input=inputs))
      File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/client/client.py", line 319, in graphql
        raise ClientError(result["errors"])
    prefect.utilities.exceptions.ClientError: [{'message': "Required parameters were not supplied: {'numbers'}", 'locations': [{'line': 2, 'column': 5}], 'path': ['create_flow_run'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': "Required parameters were not supplied: {'numbers'}"}}}]

I checked the Docker logs and found this GraphQL error:

GraphQL request:2:3
    1 | mutation ($input: create_flow_run_input!) {
    2 |   create_flow_run(input: $input) {
      |   ^
    3 |     id
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 628, in await_result
        return await result
      File "/prefect-server/src/prefect_server/graphql/extensions.py", line 52, in resolve
        result = await result
      File "/prefect-server/src/prefect_server/graphql/runs.py", line 119, in resolve_create_flow_run
        run_config=input.get("run_config"),
      File "/prefect-server/src/prefect_server/api/runs.py", line 74, in create_flow_run
        run_config=run_config,
      File "/prefect-server/src/prefect_server/api/runs.py", line 215, in _create_flow_run
        raise ValueError(f"Required parameters were not supplied: {missing}")
    ValueError: Required parameters were not supplied: {'numbers'}

Can user tokens be restricted by project? Can runner tokens be restricted by project?

hey guys, question about using a dockerfile with prefect

FROM python:3
ADD hello_world/hello_world.py /~
RUN pip3 install prefect
RUN pip3 install numpy
RUN pip3 install pandas
CMD python3 /hello_world/hello_world.py

this is an example, but do the dockerfile's need a CMD line since Prefect Agent is what handles running the code? it seems like we would just need ADD and RUN statements, and then Prefect agent would handle execution

I have timeouts on all my tasks now but if a command in a mapped flow gets locked the process is never stopped and locks up my flow when it tries to finish waiting on all the processes

Hey guys! Trying to set up a schedule for one of my flows:

now = datetime.datetime.utcnow()

dev_clock = clocks.IntervalClock(start_date=now, 
                                interval=datetime.timedelta(minutes=10), 
                                parameter_defaults={"some_prop": "someval"})
schedule = Schedule(clocks=[dev_clock])

flow.schedule = schedule

and it works! the problem is that it only schedules 10 runs, and no more. Any pointers?

Hey everyone, it looks like you've done a lot with prefect server! Thanks for the awesome tool. I have a few quick questions about the updates to server/cloud: 1. While the free (dev) version of cloud has a limit of 1 concurrent flow run, is there any limit to concurrency in server? I realize the custom concurrency feature isn't available, but I'm shooting for an unlimited concurrency here. 2. Originally, agents for prefect server had to be on the same device. Is that still the case? Or can I now have any number of agents distributed accross devices & clusters -- just like with cloud? Also shoutout to making Agent requests/signaling one directional (from Agent to Prefect API). This helps soooo much with not needing to deal with firewalls and opening ports on HPC clusters. Thank you! 3. Because I'm still early in my codes development, I need to reset my Postgres database often and then start from scratch for testing. This is pretty easy for server, but what is the easiest way for cloud? The graphql interface ("interactive API")? There used to be an option to reset your data in the user settings that I liked to use.