Hi folks, quick question about getting a basic demo set up with Prefect installed on a Virtualbox VM: How do I get the UI (on the host browser) to see the projects and agents created inside the VM (guest)? From the VM, the following commands have been issued:

# In window #1 
prefect backend server
prefect agent start
prefect project create test-1

# In window #2
prefect server start

# In window #3
prefect agent start

When I go to http://IP_OF_VM:8080 the UI is visible, but when I click "Dashboard" and then "Agents", it says there are no agents. Also in the upper right corner of the UI there is a drop-down for "Project", but there are no items in the drop-down. Even created a config.toml (and restarted server and agent), but still no change.

[server]

  [server.ui]

    apollo_url = "<http://IP_OF_VM:4200/graphql>"

Hi, I try to run Prefect Server on Docker box. All services are started and running and I can registry my flow and run from UI. But when it runs it uses LocalExceutor/ How can I make it to use Dask? I added DaskExecutor directly to the flow but it still runs with Local. I also tried to set env var PREFECT__ENGINE__EXECUTOR__DEFAULT_CLASS in docker-compose.yml file and in my flow but it did not help. How to make Prefect Server use Dask as executors while running flows? Update with code. Flow creation: def create_flow(name):   executor = get_executor(env='dev')   with Flow(name,           storage=Local(),           executor=executor) as flow:       input = Parameter('input_params', required=True)       data1 = _task1(input)       data2 = _taks2.map(data1)       data3 = _task3.map(data2)       data4 = _task4.map(data3)       result = _task5(data4)   return flow get_executor function returns DaskExecutor Flow registration: 

flow = create_flow('TestFlowServer')
flow.register(project_name="Test")

   docker-compose.yml Added below var to environment section for towel and graphql services PREFECT__ENGINE__EXECUTOR__DEFAULT_CLASS: prefect.engine.executors.DaskExecutor I use LocalAgent to run flows. I also added the above env var when I start it. But nothing helped. When I try to run the flow locally, without registering on Server everything works. Thanks, Sergey

With the helm chart, can I replicate all the services e.g. to minimum of three? Or are there some limitations? The values.yaml suggests so by making the replicas configurable, but you never know 🤷

What is the authentication mechanism on prefect server. When i started prefect server it automatically launched Client as well and did not asked any authentication.

A flow that runs successfully from command line fails when started from the the local website. It gives this error:

[Errno 2] No such file or directory: 'prefect'

but does not give context. Any idea which "prefect" file or directory it's looking for?

Secret objects in Prefect are used to provide sensitive information to the workflow like authentication parameters of any third party app. Is my understanding correct?

I could not find any documentation on user access management with prefect server or prefect core. Is user access management(RBAC) possible on prefect server. Is which user triggered which flow/task tracked?

CDN fails to fetch roboto due to CSP error on chrome; how do i resolve this? Refused to load the stylesheet 'https%3A%2F%2Ffonts%2egoogleapis%2ecom%2fcss%3ffamily%3dRoboto%3a100%2c300%2c400%2c500%2c700%7cMaterial%2bIcons%7cMaterial%2bIcons%2bOutlined&wexps=1&_ordtok=7Nk3WVLDshWjFTQHnwkqqSNPvF' because it violates the following Content Security Policy directive: "style-src-elem 'self' 'unsafe-inline' https://fonts.googleapis.com https://kit-free.fontawesome.com".

Hi! Curious to see if anyone has been able to use Singer and Prefect together?

Hi! Do you know if during upgrade of prefect stack I need to perform some manual actions? I have everything in 0.13.19, hasura migrations done via prefect-graphql, but I'm sill receiving error, during flow run

GRAPHQL_VALIDATION_FAILED: Cannot query field "get_or_create_task_run_info" on
        type "Mutation". Did you mean "get_or_create_task_run" or
        "get_or_create_mapped_task_run_children"?

Hello Prefect Community! If a task fail... exist any way to retry from some upstream task? Example, I have three tasks: A->B->C If C fail, exist any way to re-run (retry) from task B? Or task A? Thanks in advance 😉

Hi 👋 , Any tips on how I might debug this 🤔

prefect[9118]: Pulling postgres ...
prefect[9118]: Pulling hasura   ...
prefect[9118]: Pulling graphql  ...
prefect[9118]: Pulling towel    ...
prefect[9118]: Pulling apollo   ...
prefect[9118]: Pulling ui       ...
prefect[9118]: [29.6K blob data] Host is already in use by another container
prefect[9118]: [47B blob data]prefect[9118]: ERROR: for cli_postgres_1  Cannot start service postgres: driver failed programming external connectivity on endpoint cli_postgres_1 (32c027fb4da78cf7d569efc0a4f33e5baed8621c161b7c4a2c5efca694b3912c): Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use
Dec 10 14:53:26 mb-01 prefect[9118]: ERROR: for postgres  Cannot start service postgres: driver failed programming external connectivity on endpoint cli_postgres_1 (32c027fb4da78cf7d569efc0a4f33e5baed8621c161b7c4a2c5efca694b3912c): Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use
prefect[9118]: Encountered errors while bringing up the project.

👋 What is the best advise for cleaning up

prefect/flows

so that we can keep it at a manageable size (it tends to just grow naturally; and it does so quite quickly)? 🤔

Hi everyone, I have an issue launching prefect server from an EC2 instance, when I access the dashboard, it won’t load anything. Anyone had this issue before?

Hey everyone! I'm a new user of prefect and I can say this tool is very promising. After reading the documentation I didn't find a way to have prefect server a kubernetes deployment with prefect ui as a service. Does anyone here use it that way?

If anyone is working on a systemd service file for both prefect server and agents, let me know! haven’t had success with a local agent

Hey Team, I was testing a sample file from someone else that has some relative paths to configuration files. When i'm building using docker storage, and i specify the files={} keyword, what path should i put it in to be the same directory as where the flow is executing? edit - I'm using a base image of prefecthq/prefect docker image

Hi everyone, I have trouble getting access to the prefect UI after upgrading to 13.19. We run prefect-server on Kubernetes (spun up via the helm chart) and I connect with port-forwarding. The UI is displayed and tell me to create a tenant, if I execute the command however I get an error -> in thread to declutter

Multiple Agents for Disparate Projects? Given several disparate projects, each with its own dependencies (installed packages and custom python modules), does it make sense to run: A. One agent pre project or B. One agent to service all the projects?

I am running the server on premesis. If I create a very simple graph with 200 nodes and then click “Schematic” in the UI, the I get no page load and error code: 5. If I reduce the amount of nodes to 100, it seems to work just fine. Is this expected? 200 doesn’t feel like a lot nodes (we have dags with well over 100K nodes). Here’s my graph:

@task
    def return2(x):
        return 2 + x

    @task
    def say_hello(x):
        print(f"Hello, world! {x}")


    with Flow("My First Flow") as flow:
        for x in range(0, 100):
            say_hello(return2(x))

    flow.register("test")

Hi there. We want to run a small instance of Prefect in an AWS instance. We'll probably be using the Docker agent to run some fuzzy matching (entity resolution) flows in the same instance. This is a beta of a service we may grow, but in the beginning we wanted to start with simple infrastructure. Unfortunately, Prefect Cloud is disallowed at this stage because it would have to be vetted by our client's security team and that can't happen in the near future and that's why we are starting with Prefect Server. My question is ... do you have any recommendations as to which instance type and size we might want to start with to run Prefect Server? Some flows will be somewhat memory and CPU intensive but we won't be running a high volume of them. Thanks!

[✔️ solved] Hey Folks, Any tips on how to start digging into this error:

Failed to write log with error: 413 Client Error: Payload Too Large for url: <http://dataflow-apollo:4200/graphql//graphql>

The flow hadn't even run the first task, which was a

EnvVarSecret

. Other instances of the Flow have run successfully.

[✔️ solved] Mapped task issue with apollo (more in thread)

The PREFECT__CLOUD__AGENT__AGENT_ADDRESS was wrong on my yaml. Prefect Kubernetes agent pod failing liveness probe. I deployed a k8s agent using the deployment yaml provided by prefect. I updated the PREFECT_CLOUD__API to point to my own server. But the agent pod keeps restarting because of the probeness failure:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  103s  default-scheduler  Successfully assigned default/prefect-agent-c4b68bfd9-dj8cf to ip-10-0-1-20.eu-west-1.compute.internal
  Normal   Pulling    102s  kubelet            Pulling image "prefecthq/prefect:0.13.19-python3.6"
  Normal   Pulled     101s  kubelet            Successfully pulled image "prefecthq/prefect:0.13.19-python3.6"
  Normal   Created    101s  kubelet            Created container agent
  Normal   Started    101s  kubelet            Started container agent
  Warning  Unhealthy  37s   kubelet            Liveness probe failed: Get <http://10.0.1.64:8080/api/health>: dial tcp 10.0.1.64:8080: connect: connection refused

Note that the agent can successfully register to the UI. deployment.yaml in the thread. Does anyone one know how to fix it?

How do I get

print()

statements to show up in the web console at localhost:8080? Here's my example code that prints to the terminal, but does not now up in the web console:

from prefect import task, Flow

@task
def simple_func(log_stdout=True):
    print('simple print statement')


if __name__ == '__main__':
    with Flow('test-logging') as flow:
        simple_func()

    flow.register('test-log')
    flow.run()

Say the agent is running under user alice, and the flows are being registered by user bob. Into whose home directory does ~/.prefect/config.toml belong?

Is there any way to create a one to many mapping between a task A and B, where B depends on A, but B’s mapping dimension is of lower cardinality than A’s so that some instances of B would depend on multiple instances of A?

Hi to all, I'm CTO of DataMola LLC company. I looking for help and contributors. Does anyone want to help combine RAPIDS.IO + VISION profiler ( plan to refactor on cuDF - https://github.com/dylan-profiler/visions ) + Great Expectation + Prefect.IO ? The primary goal to create the RAW data profiler on DASK + RAPIDS.IO on GPU distributed Environemnent.

I’m curious if anyone has run into a

boto3.client()

issue when running

prefect agent ecs start

. In short, I receive the following error:

File "/Users/mike/.virtualenvs/reporting-Q7tyta-k/lib/python3.9/site-packages/prefect/utilities/aws.py", line 55, in get_boto_client
    return boto3.client(
TypeError: boto3.client() got multiple values for keyword argument 'aws_access_key_id'

From the looks of it, the below code snippet from

prefect/utilities/aws.py

is running, in my case, such that none of the

kwargs.pop(...)

conditionals are executed. My creds are being pulled from

~/.prefect/config.toml

. This would mean that the code is adding duplicate keyword args to

session.client()

(since they also exist in **kwargs with

None

values). If I remove

**kwargs

, it works.

Hey guys, I'm getting an error when trying to call flow.register on a windows environment; using a basic flow as a test:

Result check: OK
---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
<ipython-input-11-014533001643> in <module>()
----> 1 flow.register(project_name='test-charles')

C:\ProgramData\Anaconda3\lib\site-packages\prefect\core\flow.py in register(self, project_name, build, labels, set_schedule_active, version_group_id, no_url, idempotency_key, **kwargs)
   1667             version_group_id=version_group_id,
   1668             no_url=no_url,
-> 1669             idempotency_key=idempotency_key,
   1670         )
   1671         return registered_flow

C:\ProgramData\Anaconda3\lib\site-packages\prefect\client\client.py in register(self, flow, project_name, build, set_schedule_active, version_group_id, compressed, no_url, idempotency_key)
    821             create_mutation,
    822             variables=dict(input=inputs),
--> 823             retry_on_api_error=False,
    824         )  # type: Any
    825 

C:\ProgramData\Anaconda3\lib\site-packages\prefect\client\client.py in graphql(self, query, raise_on_error, headers, variables, token, retry_on_api_error)
    316             ):
    317                 raise VersionLockError(result["errors"])
--> 318             raise ClientError(result["errors"])
    319         else:
    320             return GraphQLResult(result)  # type: ignore

ClientError: [{'message': "'NoneType' object has no attribute 'get'", 'locations': [{'line': 2, 'column': 5}], 'path': ['create_flow_from_compressed_string'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': "'NoneType' object has no attribute 'get'"}}}]

What's this error message mean? Using the latest version of prefect for python. Thanks,