Hi, I have a flow that uses several

FlowRunTask

and run them concurrently using local dask executor. (The flow code is in the first comment). My question is, is there another way to make the

FlowRunTask

’s run concurrently without using dask? Thanks!

Hi all upgrading from 0.10.6 to 0.13.10 and looking for some advice/clarification. We used create the flow with a context manager to setup task orders and register e.g.

with Flow("Flow name") as flow:
   flow.add_task(abc)
   flow.register(project_name="Test")

In the latest version I find that this isn't possible anymore as I get a

TypeError: can't pickle generator objects

My solution has been to replace this by defining the the flow object directly and ditching the context manager e.g.

flow = Flow("Flow name")
flow.add_task(abc)
flow.register(project_name="Test")

This works absolutely fine and I'm happy to continue with it. I just wanted to check if: 1. This is the expected / best way to resolve this problem 2. [Some of the documentation](https://docs.prefect.io/core/concepts/tasks.html#overview) still refers to the use of the context manager style. Is this okay to do simply because tese example do not register a flow? Any advice is welcome 🙂 Thanks!

Hello. Is there a way to reduce the level of logging from 'DEBUG' to just periodically sending something out? I am getting complaints from my system admin that it is impacting other system operations on the server.

Hey all, I have a question regarding S3 storage. I would like to store my flow (as python script) as S3 object, such that it could be easily retrieved and run from another agent (e.g. colleagues computer, cloud etc.). I noticed, that the flow script is not stored in s3, when I set

stored_as_script = True

which I tracked down to the implemenation of the S3 build() function that returns the storage object before uploading it to s3, when this variable is set to True.

def build(self) -> "Storage":
        """
        Build the S3 storage object by uploading Flows to an S3 bucket. This will upload
        all of the flows found in `storage.flows`. If there is an issue uploading to the
        S3 bucket an error will be logged.

        Returns:
            - Storage: an S3 object that contains information about how and where
                each flow is stored

        Raises:
            - botocore.ClientError: if there is an issue uploading a Flow to S3
        """
        self.run_basic_healthchecks()

        if self.stored_as_script:
            if not self.key:
                raise ValueError(
                    "A `key` must be provided to show where flow `.py` file is stored in S3."
                )
            return self
        ..

Also, I can register the flow with s3 storage and

stored_as_script = False

, but even though it appears in UI and has a corresponding s3 object, flow_runs are not executed.

Hi, I just installed prefect on windows 10 and as I was trying

prefect server start

the first time, received this error. Any idea how to troubleshoot this?

PS C:\Users\puruz\Documents\prefect> prefect server start
Exception caught; killing services (press ctrl-C to force)
Traceback (most recent call last):
  File "C:\Users\puruz\AppData\Roaming\Python\Python39\site-packages\prefect\cli\server.py", line 331, in start
    subprocess.check_call(
  File "c:\program files\python39\lib\subprocess.py", line 368, in check_call
    retcode = call(*popenargs, **kwargs)
  File "c:\program files\python39\lib\subprocess.py", line 349, in call
    with Popen(*popenargs, **kwargs) as p:
  File "c:\program files\python39\lib\subprocess.py", line 947, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "c:\program files\python39\lib\subprocess.py", line 1416, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

Hi, I have prefect server installed and an agent running locally on the same tin. Can I have an agent running on a different server configured to communicate with the perfect server? If so, what is the required configuration? Also how to I see which agents are connecting into the server?

hi there!

First of all thanks for all the efforts on opening everything, I am in the need to deploy a custom prefect server on a Kubernetes cluster, there is some advise or guide on how to best do it? i didn't found anything

Hi all, So I started server on an ec2. With config.toml

graphql_url

pointing to

backend = "server"
graphql_url = "<http://localhost:4200/graphql>"
[server]
  [server.ui]
  graphql_url = "<http://localhost:4200/graphql>"

When I SSH-ed into the ec2 I have two portforward, 4200 and 8080. And my local

config.toml

has the same config as the server

config.toml

A local agent is running on the ec2. The issue I am facing is: When running the flow after registering from both local or in the ec2 instance, the shell tasks run into

requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=4200): Read timed out. (read timeout=30)

I reckon is my config.toml but not sure where to start, or is there something that I am missing ?

Hey there - not sure if this belongs here or in community but I'm using Prefect Cloud and I often find that the dashboard becomes unresponsive when the page is left idle for a while. Refreshing doesn't work (I think maybe the refresh is overridden via history api? not sure) - I often have to close and open the page again. Anyway, it's not a big deal but as I often have the dashboard page open in the background so I can periodically check on it - which I feel like is probably fairly common. This is on Windows/Chrome/86.0.4240.75.

Hey Everyone, I wanted to know if there are some other configurations invovled to ensure that

.map

flows work parallel-ly. We're using prefect local to test this before we decide on buying cloud. But in local I see my functions are evaluated serially.

# Generate Dates
    date_generator = generate_keys_for_nppes_urls()

    # Download ZIP Files, Instead of downloading files parallel-ly this function is executing first October, then September and then August...
    zip_files = download_nppes_data_for_time.map(date_generator)

    # Launch NPPES Spark Job
    spark_res = launch_nppes_spark_job.map(zip_files)

    # Launch NPPES Configuration Job
    config_job = nppes_configuration_job()
    config_job.set_upstream(spark_res)

HI folks, we've successfully deployed Farget Agent, but fargate tasks are quite slow due to the provisioning phase which takes ~ 1-2 minutes. We are thinking of switching from Fargate tasks to EC2 tasks. Is this option supported by Fargate Agent?

Hi folks, Has anyone any idea why, given a standard local agent, after some scheduled tasks (let’s say every 15 minutes), all the processes start to be queued and never processed? Thank you!

hi, is it possible to install prefect on windows server 2016?

i had this kind of errors

Hi Prefect Team looks like I found a bug but need your view: I had an error while running flow as script with ModuleNotFoundError and I realise that I need to start the agent with "-p" option so that Agent can find the new path though that did not help. On looking further I found the code inside the LocalAgent

if self.import_paths:
    python_path += self.import_paths

current_env["PYTHONPATH"] = ":".join(python_path)

As I am working on Windows I realise that the colon to separate the Python path might not work and it should have been Semicolon for Windows

Dear Prefect Team, first of all, congratulations and many thanks for your hard work you've put into Prefect. I have managed to setup some flows and are able to run them when running single flows with single agents. However, I made the following experience, when I want to run three different flows in three different directories using supervisord: If I use one agent as program like:

[program:fhn]
command=sudo prefect agent start local -p /usr/dcc/fhn/data_import -p /usr/dcc/fhn/calc_td -p /usr/dcc/fhn/vol_back_all -f

I can run only two flows, the third one fails with the message "Failed to load and execute Flow's environment: ModuleNotFoundError". It doesn't matter which flow is third, it is always the last one, which fails. So I have created two separate programs:

[program:fhn1]
command=sudo prefect agent start local -p /usr/dcc/fhn/data_import -p /usr/dcc/fhn/calc_td -f -l Import
[program:fhn_2]
command=prefect agent start local -p /usr/dcc/fhn/vol_back_all  -f -l Vol

to handle all three flows. But now, the success of the flows is unstable. Sometimes they fail with the "Failed to load.." message, sometimes they succeed. Even if I trigger them manually, they sometimes fail and sometimes succeed. More than 50% of flow runs fail. Here is a log from supervisord:

2020-10-18 08:55:05,888 DEBG 'fhn1' stdout output:
[2020-10-18 08:55:05] INFO - prefect.CloudFlowRunner | Beginning Flow run for 'Calculate Equations FHN'

2020-10-18 08:55:05,903 DEBG 'fhn1' stdout output:
[2020-10-18 08:55:05] ERROR - prefect.Local | Failed to load Flow from /usr/dcc/fhn/flows/back-allocation-fhn.prefect
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/prefect/environments/storage/local.py", line 103, in get_flow
    return prefect.core.flow.Flow.load(flow_location)
  File "/usr/local/lib/python3.7/dist-packages/prefect/core/flow.py", line 1495, in load
    return cloudpickle.load(f)
  File "/usr/local/lib/python3.7/dist-packages/cloudpickle/cloudpickle.py", line 562, in subimport
    __import__(name)
ModuleNotFoundError: No module named 'read_wt'

2020-10-18 08:55:05,954 DEBG 'fhn1' stdout output:
No module named 'read_wt'

The two agents are running fine. All code and agents are local, everything runs on one machine. I am running looped tasks in each of the flow. Here is the code for one of the flows:

flow = Flow("Data Import FHN")
flow.set_dependencies(loop_conns, keyword_tasks={"iloop": looplist}, mapped=True)
with Flow("Data Import FHN") as flow:
    connect = db_conn()
    mapped_result =loop_conns.map(iloop=looplist)

flow.storage = Local(directory="/usr/dcc/fhn/flows")
flow.storage.build()
flow.register(project_name="FHN")

Any help would be appreciate to understand the unstable performance better.

Any idea what could be causing a flow to fail with the error "Parameter object has no attribute 'task_run_name'" when using Docker storage? The docker image build was fine and passed its health checks, and the flow runs fine when using local storage (and local agent) instead. Note this happens only if I use a custom Dockerfile.

Hi, Is there a way to provide a default "run-name" for a flow? I know using the CLI, it can be specified manually, but I'm hoping to do it programmatically, so that it appears in the UI and can be manually edited if needed.

--run-name

For instance, I'd like Prefect Server to suggest and use the default run name including a timestamp like "run 20201018_231500"

Hey, we're seeing some duplicate flow runs, got any ideas? (more info in thread)

Hi guys, I am having some trouble figuring out the UI configuration logic, where I simply want to use another url than the 

<http://localhost:4200/graphql>

This specific setup were I'm currently testing this is running the following:

"platform": "Windows-10-10.0.19041-SP0",
"prefect_backend": "server",
"prefect_version": "0.13.11",
"python_version": "3.8.3"

I followed the guide for UI configuration and I added the following to my config (/home./prefect/config.toml):

backend = "server"
[server]
    [server.ui]
    apollo_url = "http://<url>:4200/graphql"

It works just fine, if you check `prefect diagnostics`:

{
  "config_overrides": {
    "backend": true,
    "server": {
      "ui": {
        "apollo_url": true
      }
    }
  },
  "env_vars": [],
  "system_information": {
    "platform": "Windows-10-10.0.19041-SP0",
    "prefect_backend": "server",
    "prefect_version": "0.13.11",
    "python_version": "3.8.3"
  }
}

I can also see that the UI indeed understand that the configuration have been added to the config.toml where it picks up the specific URL at: 

http://<url>:8080/settings.json

 :

{
  "server_url": "http://<url>:4200/graphql"
}

When I then dig a littler deeper into the network requests happening in relation to the UI, I can then see that everything are still are happening at: 

<http://localhost:4200/graphql>

 (Please see attachment) Which must be related to that it always insert 

<http://localhost:4200/graphql>

 into the localstorage automatically,. (Please see attachment) Why isn't it inserting the right URL into the localstorage? It only use the right URL if I manually adds it to the localstorage.

Hi, I got an annoying issue we have been starting using Prefect at my company. Everything got well with the deployment of Prefect on our K8s but now we want to protect it with some BasicAuth and now nothing works: We are able to build and store flow easily on the server by using the attach_headers method of the client:

headers = {'Authorization': "Basic {}".format(XXXXX)}
client = prefect.Client()
client.attach_headers(headers)

However, when the agent deploys a flow we hit a small issue: 1. The docker is runned with the following command:

prefect execute flow-run

2. Inside the docker we run it leads to

prefect.execute.cli

with

_execute_flow_run

:

client = Client()
result = client.graphql(query) # Authorization error

Is there a simple way to avoid this issue and being able to add BasicAuth to the client? We could maybe add an env variable which would select the proper auth method by the client (example): https://github.com/PrefectHQ/prefect/blob/bd6e47379594d4e26e6810380482320eeee714ae/src/prefect/client/client.py#L385

def select_auth_method(token):
   if config.AUTH.BASIC:
       return "Basic {}".format(token)
   # else OAuth
   return "Basic {}".format(token)

if token:
  headers["Authorization"] = "Bearer {}".format(token)

Hi team, I am pretty new to prefect. I am stuck at trying to write an ETL . Is there a way to pass run time variables to state handlers. What i want is something like this.

def task_state_transition_handler(flow_info:Task, old_state: State, new_state: State):
    if new_state.is_queued():
        aquire_lock(path=new_state.context['flow_name'] + "/" +
                             new_state.context['today']+"/",
                    data=new_state.context['flow_run_id'])
    elif new_state.is_finished():
        release_lock(path=new_state.context['flow_name'] + "/" +
                             new_state.context['today']+"/",
                    data=new_state.context['flow_run_id'])

    return

Hey Team. I've been exploring prefect for work for the past few weeks. I'm suddenly running into an issue trying to get my flows to run from the UI (They work fine when run inside my Python environment). This just started happening this morning. I ran a flow successfully just a few minutes before I got this error for the first time. It hasn't worked since. The only thing I did this morning was upgrade to 0.13.12. I reverted back to 0.13.10 but it did not fix the issue This is what I see for the agent -

[2020-10-22 16:18:25,588] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>

[2020-10-22 16:18:25,588] INFO - agent | Agent connecting to the Prefect API at <http://localhost:4200>

[2020-10-22 16:18:25,597] INFO - agent | Waiting for flow runs...

[2020-10-22 16:18:41,285] INFO - agent | Found 1 flow run(s) to submit for execution.

[2020-10-22 16:18:41,328] INFO - agent | Deploying flow run b95dfeb9-7912-4526-bf23-168d84cb70b2

[2020-10-22 16:19:25,600] INFO - agent | Process PID 33157 returned non-zero exit code

This is what I see on the UI

Last State Message
[22 Oct 2020 9:18am]: Failed to load and execute Flow's environment: AttributeError("Can't get attribute 'NoDefault' on <module 'prefect.core.task' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/prefect/core/task.py'>")

I have been unable to fix the issue. Any help would be appreciated. Thank you 🙂

Hi all, I am on

0.13.10

prefect server

, and I started to see this error for scheduled task and manually run tasks

[{'message': 'Foreign key violation.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_flow_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Foreign key violation.'}}}]

It seems like it comes from the graphql side of things. But I couldn’t figure out why it randomly fails with that error. After a couple of manual restart sometimes it works. I tried to debug but since it happens at the very beginning of the flow, there’s not much that I can find out When I register the flow I register it with the same core version, or at least I try to

Hi reported the following issue: https://github.com/PrefectHQ/prefect/issues/3569 regarding docker-compose.yml not containing postgres volume option. Anyone same issue?

Hi all! I've got a weird issue when starting the prefect server. If anyone have ever faced this issue before, please help me.

WARNING: The PREFECT_SERVER_DB_CMD variable is not set. Defaulting to a blank string.
WARNING: The DB_CONNECTION_URL variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_DB variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_USER variable is not set. Defaulting to a blank string.
Creating network "prefect-server" with the default driver
Creating tmp_postgres_1 ... done
Creating tmp_hasura_1   ... done
Creating tmp_graphql_1  ... done
Creating tmp_towel_1    ... done
Creating tmp_apollo_1   ... done
Creating tmp_ui_1       ... done
Attaching to tmp_postgres_1, tmp_hasura_1, tmp_graphql_1, tmp_towel_1, tmp_apollo_1, tmp_ui_1
graphql_1   | bash: -c: line 0: syntax error near unexpected token `&&'
graphql_1   | bash: -c: line 0: ` && python src/prefect_server/services/graphql/server.py'
hasura_1    | {"type":"pg-client","timestamp":"2020-10-26T11:45:30.504+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
hasura_1    | {"type":"pg-client","timestamp":"2020-10-26T11:45:30.504+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(1)."}}
hasura_1    | {"type":"startup","timestamp":"2020-10-26T11:45:30.504+0000","level":"error","detail":{"kind":"db_migrate","info":{"internal":"could not connect to server: No such file or directory\n\tIs the server running locally and accepting\n\tconnections on Unix domain socket \"/var/run/postgresql/.s.PGSQL.5432\"?\n","path":"$","error":"connection error","code":"postgres-error"}}}
ui_1        | 👾👾👾 UI running at localhost:8080 👾👾👾
postgres_1  | Error: Database is uninitialized and superuser password is not specified.
postgres_1  |        You must specify POSTGRES_PASSWORD to a non-empty value for the
postgres_1  |        superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
postgres_1  | 
postgres_1  |        You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
postgres_1  |        connections without a password. This is *not* recommended.
postgres_1  | 
postgres_1  |        See PostgreSQL documentation about "trust":
postgres_1  |        <https://www.postgresql.org/docs/current/auth-trust.html>
tmp_hasura_1 exited with code 1
tmp_graphql_1 exited with code 1
tmp_postgres_1 exited with code 1