After reading the “Why not Airflow” post for the 50th time, I get more and more convinced of Prefect - especially on dataflow. But I’m looking at the docs and cannot seem to find limitations on dataflow. What are the limits of how much data can be passed between tasks in a flow? And are there docs/further reading on where/how is that data being persisted?

If I use the prefect server start --detach then the service hangs forever. Any reason ?? Not sure what is blocking in here.

has anyone had any experience with running a local server and registering a flow to it from within a docker container?

Hi! How to restrict some flows to run in parallel? I’m looking at concurrency running labels, but they work on task, not on flows

Does the user part of the local docker-compose run server work for anyone else? I managed to get the graphql endpoint working (seems the ~/.prefect/config.toml change does not do anything, but the webform does). Now I want to create a token but, starting with:

prefect server start --use-volume  --volume-path ~/.prefect/pg_data

If I select team|Account, it goes to stripe and fails as I am http only.

IntegrationError: Live Stripe.js integrations must use HTTPS. For more information: <https://stripe.com/docs/security/guide#tls>
    at new o (<https://js.stripe.com/v3/:1:2860>)
    at fc.value (<https://js.stripe.com/v3/:1:203919>)
    at new fc (<https://js.stripe.com/v3/:1:205084>)
    at mc (<https://js.stripe.com/v3/:1:205615>)
    at Module.4120 (<http://old7.mianos.com:8080/js/team-settings--account~21833f8f.f65a711f.js:1:42224>)
    at s (<http://old7.mianos.com:8080/js/app~d0ae3f07.b8b5e05d.js:1:595>)

When creating a project:

vue-router.esm.js:2008 Uncaught (in promise) Error: Navigation cancelled from "/default?agents" to "/default/project/dd392518-7e7b-41b0-8967-cdd5d2dda5a3" with a new navigation.
    at $t (vue-router.esm.js:2008)
    at wt (vue-router.esm.js:1990)
    at v (vue-router.esm.js:2297)
    at r (vue-router.esm.js:1947)
    at vue-router.esm.js:1948
    at vue-router.esm.js:2322
    at Dashboard.vue:69
    at u (runtime.js:63)
    at Generator._invoke (runtime.js:293)
    at Generator.next (runtime.js:118)

But it does work. The tutorial does not mention creating a project during registration. It seems to be needed. Anyway, my flow seems to register, if I do name it, It just seems this is not really the same state as prefect itself.

hi there 🙂 it seems, that in the latest prefect release the backend is not working: ubuntu 18, Docker version 20.10.2, build 2291f61 pip install prefect==0.14.8

> prefect server start
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.ui.depends_on contains an invalid type, it should be an array
services.apollo.depends_on contains an invalid type, it should be an array
services.towel.depends_on contains an invalid type, it should be an array
services.graphql.depends_on contains an invalid type, it should be an array
services.hasura.depends_on contains an invalid type, it should be an array
Exception caught; killing services (press ctrl-C to force)
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.ui.depends_on contains an invalid type, it should be an array
services.apollo.depends_on contains an invalid type, it should be an array
services.towel.depends_on contains an invalid type, it should be an array
services.graphql.depends_on contains an invalid type, it should be an array
services.hasura.depends_on contains an invalid type, it should be an array
Traceback (most recent call last):
  File "/home/poc/miniconda3/envs/etlp/lib/python3.8/site-packages/prefect/cli/server.py", line 346, in start
    subprocess.check_call(
  File "/home/poc/miniconda3/envs/etlp/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker-compose', 'pull']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/poc/miniconda3/envs/etlp/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/home/poc/miniconda3/envs/etlp/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/poc/miniconda3/envs/etlp/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/poc/miniconda3/envs/etlp/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/poc/miniconda3/envs/etlp/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/poc/miniconda3/envs/etlp/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/poc/miniconda3/envs/etlp/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/poc/miniconda3/envs/etlp/lib/python3.8/site-packages/prefect/cli/server.py", line 384, in start
    subprocess.check_output(
  File "/home/poc/miniconda3/envs/etlp/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/home/poc/miniconda3/envs/etlp/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['docker-compose', 'down']' returned non-zero exit status 1.

when trying to use prefect locally via the docker image

docker run -it prefecthq/prefect:latest

, when i then run

prefect server start

I get a FileNotFoundError looking for a docker-compose file. I am running the command in the docker container

prefect server start --use-volume --volume-path ~/.prefect/pg_data .. ps -afe | grep curl | wc 23205 208846 1601151 .. root 4065141 2729330 0 14:52 ? 000000 [curl] <defunct>

LOL:

ps -afe | grep curl | wc -l
63158

Does anyone know if / how we can change the config directory from the default

~/.prefect

to something else? Tried

PREFECT__USER_CONFIG_PATH

but that didnt work

Hey All, I am a graphql noob and was wondering if anyone had an example of a mutation to run a flow with given parameters?

Hey everyone, I’m very new to Prefect and am struggling with something simple.. I’m unable to get Prefect Server to execute multiple jobs in parallel. My guess is I have something basic misconfigured, but I’m not sure what exactly. I’ve set the executor for my flow to be

LocalDaskExecutor(scheduler="threads", num_workers=5)

but it seems like I still can only serially execute my flows. Does anyone here have experience with this? Thank you so much for the help!

Hi everyone, I have installed prefect in ubuntu 18.04 server, i am trying to setup agents to run flows in prefect.io cloud. But not able to login using this command in prefect, i have already generated PAT token from my prefect account

prefect auth login -t token

Getting below error

Usage: prefect auth [OPTIONS] COMMAND [ARGS]...
Error: Auth commands with server are not currently supported.

I am new to prefect, any suggestions from community will be really helpful

Hi, While playing around with the server UI, I clicked on the cloud option in the top right corner. Now if I try to navigate to http://localhost:8080/default it automatically redirects me to https://universal.prefect.io/. I tried restarting chrome and get the same behavior. Any idea how to stop this? Thanks!

I updated to the newest prefect version backend (0.14.9), python 3.8.6. my linux crashing because a lot of new zombie processes: nothing else is running on the server

Hello all, I am new to Prefect and having a hard time getting my github access token picked up when running a flow on the kubernetes agent. I have set the env variable

PREFECT__CONTEXT__SECRETS__GITHUB_ACCESS_TOKEN

on the agent pod but I keep getting this error

ValueError('Local Secret "GITHUB_ACCESS_TOKEN" was not found.')

. Does the default secret not get created when using the kubernetes agent?

Hello Community, We have deployed Prefect in October 2020 to evaluate it (we love it!) (and we are going to use it soon on bigger projects :)). Today we encountered an unexpected thing: Docker used 836 GB of hard drive space. 717 GB due to a (rather odd?) log file (see picture) in the directory

./var/lib/docker/containers/87ee8de38f69b14c1c4b27b07c3dd591efd8ce60370a8f9a089a4306c66f5a46

. We have no idea where this comes from, I hope that it can be traced back to something I did wrong in the setup. Any experience or ideas? With regards from Berlin to you all

Hi, just wrote the first flow, the script runs the flow on its own with 0 agent and does not show up in the local UI dashboard. How to make it run on the local agent and display the runs in the dashboard? The UI only shows the flow name with 0 run.

...
with Flow('flow') as flow:
    ..

flow.register(project_name='project1')
flow.run(parameters=dict(, ..))
flow.run(parameters=dict(, ..))

[2021-02-19 ...] INFO - prefect.TaskRunner | Task '..': Starting task run..
[2021-02-19 ...] INFO - prefect.TaskRunner | Task '..': Finished task run for task with final state: 'Success'
...

Clikcing Flow URL shows 404.

Hello, I am seeing flows fail if a task with an

any_failed

trigger is not fired because no task failed. Is there any way to fix this? docs

Hi does anyone know how I can set the node selector for the postgres that comes with the helm chart? Doesn't seem to be an option in the overrides...

Is there a limit to the size of the flows when pushing to the prefect server? I have a ~1500 task flow (usually 1-4 dependencies per task) that crashes graphql when I send it, but whenever I send any subparts of the graph it seems to work fine.

Hi there If I skipped some schedules in the backend, can I replan them? Even if I recreate the schedule with an other time (cron, daily) it will no recreate the skipped schedules...

Hello, I'm trying the tutorials and have the cloud based server instance running. The issue I'm running into is the start a local agent on my machine. I followed the directions here: https://docs.prefect.io/orchestration/tutorial/overview.html#create-a-runner-token and added the returned runner token to my ~/.prefect/config.toml. When I try to launch the local agent with: prefect agent local start I get the following stack trace:

Traceback (most recent call last):

File "C:\ProgramData\Miniconda3\envs\prefect\lib\site-packages\toml\decoder.py", line 511, in loads ret = decoder.load_line(line, currentlevel, multikey, File "C:\ProgramData\Miniconda3\envs\prefect\lib\site-packages\toml\decoder.py", line 778, in load_line value, vtype = self.load_value(pair[1], strictly_valid) File "C:\ProgramData\Miniconda3\envs\prefect\lib\site-packages\toml\decoder.py", line 910, in load_value raise ValueError("This float doesn't have a leading " ValueError: This float doesn't have a leading digit During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\ProgramData\Miniconda3\envs\prefect\Scripts\prefect-script.py", line 5, in <module> from prefect.cli import cli File "C:\ProgramData\Miniconda3\envs\prefect\lib\site-packages\prefect\__init__.py", line 1, in <module> import prefect.utilities File "C:\ProgramData\Miniconda3\envs\prefect\lib\site-packages\prefect\utilities\__init__.py", line 1, in <module> import prefect.utilities.logging File "C:\ProgramData\Miniconda3\envs\prefect\lib\site-packages\prefect\utilities\logging.py", line 23, in <module> from prefect.utilities.context import context File "C:\ProgramData\Miniconda3\envs\prefect\lib\site-packages\prefect\utilities\context.py", line 69, in <module> from prefect.configuration import config File "C:\ProgramData\Miniconda3\envs\prefect\lib\site-packages\prefect\configuration.py", line 324, in <module> config = load_configuration( File "C:\ProgramData\Miniconda3\envs\prefect\lib\site-packages\prefect\configuration.py", line 300, in load_configuration user_config = load_toml(user_config_path) File "C:\ProgramData\Miniconda3\envs\prefect\lib\site-packages\prefect\configuration.py", line 177, in load_toml for key, value in toml.load(cast(str, interpolate_env_vars(path))).items() File "C:\ProgramData\Miniconda3\envs\prefect\lib\site-packages\toml\decoder.py", line 134, in load return loads(ffile.read(), _dict, decoder) File "C:\ProgramData\Miniconda3\envs\prefect\lib\site-packages\toml\decoder.py", line 514, in loads raise TomlDecodeError(str(err), original, pos) toml.decoder.TomlDecodeError: This float doesn't have a leading digit (line 3 column 1 char 39) This is on a Windows 10 machine with a conda install of prefect. Looks like it expects the token to be numeric or at least lead with a numeric value. Many thanks.

Quick question. Can multiple agents share the same auth token or does each have to have their own? For example, if I generate the auth token and use that for a K8S agent deployment with a replicaset of 5, can I create that deployment using that one token? Doing that only shows one agent in the Prefect Cloud server even though 5 pods exist. Thanks.

has anyone ever had an issue with using a logger from a custom module, whereby the custom module's logs are being appropriately by agent's flow logs, but those logs seemingly do not get forwarded to your server? I do have in my config.toml:

[logging]
# Extra loggers for Prefect log configuration
extra_loggers = "['custom_module']"
# Send logs to Prefect Cloud
log_to_cloud = true

Hey there! 👋 I’ve looked in the documentation and did some google-searching but to no avail... so I’m trying my luck here 🙂 How would you go if you wanted to check a condition based on the date of the last successful run of a flow? For instance: if the data I work with wasn’t updated since the last successful run, I can skip it. My ugly simple solution at the moment is to dump a timestamp in a file somewhere. It seems that I should be able to build that using either Result or Cache, but I can’t really figure out a way to do it. I was hoping to find this information (date of last successful run of the flow) populated in

prefect.context

. Pointers and suggestions welcome! 🙏

Hello, I was wondering if I might ask for a little assistance. I decided to try prefect.io out, so I went to the documentation site and chose a docker employment. I deployed a container via official dockerhub image and then executed "prefect backend server" and "prefect server start", which failed because there is no docker inside the prefect image nor docker-compose.

I'd like to ask - how is it intended to be deployed? There is no mention about docker-compose.yml in the docs.

Prefect cloud is giving a 403 Security violation response to a POST using httpx (an async http module) to post a mutation (file attached). The call is being made from a FastAPI app, and works fine if I use the requests module. I know Prefect expects synchronous tasks, does this extend to using http on the API calls as well?

Hi all, I’m trying to deploy Prefect in a Kubernetes cluster, but would like to use an ingress or NodePort configuration rather than two load balancers, which are somewhat expensive in AWS. I tried to set things up to use an Nginx ingress controller with /prefect serving the UI server and /graphql serving the Apollo server, but Prefect’s JavaScript setup does not seem to deal well with being run this way, and even adding rules for a bunch of the absolute paths (/css, /js, etc.) I was not able to get things running. Questions: 1. Is this a known limitation of Prefect’s server setup as presently constituted? 2. Has anyone had success deploying Prefect using an ingress controller? 3. If the answer to #2 is no, how about using NodePort for the UI and Apollo servers?