https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • j

    jack

    12/03/2020, 8:17 PM
    Hi folks, quick question about getting a basic demo set up with Prefect installed on a Virtualbox VM: How do I get the UI (on the host browser) to see the projects and agents created inside the VM (guest)? From the VM, the following commands have been issued:
    # In window #1 
    prefect backend server
    prefect agent start
    prefect project create test-1
    
    # In window #2
    prefect server start
    
    # In window #3
    prefect agent start
    When I go to http://IP_OF_VM:8080 the UI is visible, but when I click "Dashboard" and then "Agents", it says there are no agents. Also in the upper right corner of the UI there is a drop-down for "Project", but there are no items in the drop-down. Even created a config.toml (and restarted server and agent), but still no change.
    [server]
    
      [server.ui]
    
        apollo_url = "<http://IP_OF_VM:4200/graphql>"
    n
    28 replies · 2 participants
  • s

    Sergiy Krutsenko

    12/03/2020, 8:40 PM
    Hi, I try to run Prefect Server on Docker box. All services are started and running and I can registry my flow and run from UI. But when it runs it uses LocalExceutor/ How can I make it to use Dask? I added DaskExecutor directly to the flow but it still runs with Local. I also tried to set env var PREFECT__ENGINE__EXECUTOR__DEFAULT_CLASS in docker-compose.yml file and in my flow but it did not help. How to make Prefect Server use Dask as executors while running flows? Update with code. Flow creation: def create_flow(name):   executor = get_executor(env='dev')   with Flow(name,           storage=Local(),           executor=executor) as flow:       input = Parameter('input_params', required=True)       data1 = _task1(input)       data2 = _taks2.map(data1)       data3 = _task3.map(data2)       data4 = _task4.map(data3)       result = _task5(data4)   return flow get_executor function returns DaskExecutor Flow registration: 
    flow = create_flow('TestFlowServer')
    flow.register(project_name="Test")
       docker-compose.yml Added below var to environment section for towel and graphql services PREFECT__ENGINE__EXECUTOR__DEFAULT_CLASS: prefect.engine.executors.DaskExecutor I use LocalAgent to run flows. I also added the above env var when I start it. But nothing helped. When I try to run the flow locally, without registering on Server everything works. Thanks, Sergey
    d
    8 replies · 2 participants
  • j

    Joël Luijmes

    12/04/2020, 11:40 AM
    With the helm chart, can I replicate all the services e.g. to minimum of three? Or are there some limitations? The values.yaml suggests so by making the replicas configurable, but you never know 🤷
    n
    m
    +1
    8 replies · 4 participants
  • n

    Neeraj Sharma

    12/04/2020, 12:00 PM
    What is the authentication mechanism on prefect server. When i started prefect server it automatically launched Client as well and did not asked any authentication.
    a
    4 replies · 2 participants
  • j

    jack

    12/04/2020, 7:26 PM
    A flow that runs successfully from command line fails when started from the the local website. It gives this error:
    [Errno 2] No such file or directory: 'prefect'
    but does not give context. Any idea which "prefect" file or directory it's looking for?
    d
    j
    9 replies · 3 participants
  • n

    Neeraj Sharma

    12/07/2020, 11:06 AM
    Secret objects in Prefect are used to provide sensitive information to the workflow like authentication parameters of any third party app. Is my understanding correct?
    m
    1 reply · 2 participants
  • n

    Neeraj Sharma

    12/07/2020, 11:07 AM
    I could not find any documentation on user access management with prefect server or prefect core. Is user access management(RBAC) possible on prefect server. Is which user triggered which flow/task tracked?
    a
    1 reply · 2 participants
  • c

    Charles Leung

    12/07/2020, 5:18 PM
    CDN fails to fetch roboto due to CSP error on chrome; how do i resolve this? Refused to load the stylesheet 'https%3A%2F%2Ffonts%2egoogleapis%2ecom%2fcss%3ffamily%3dRoboto%3a100%2c300%2c400%2c500%2c700%7cMaterial%2bIcons%7cMaterial%2bIcons%2bOutlined&wexps=1&_ordtok=7Nk3WVLDshWjFTQHnwkqqSNPvF' because it violates the following Content Security Policy directive: "style-src-elem 'self' 'unsafe-inline' https://fonts.googleapis.com https://kit-free.fontawesome.com".
    k
    5 replies · 2 participants
  • f

    Frances Sun

    12/09/2020, 2:52 PM
    Hi! Curious to see if anyone has been able to use Singer and Prefect together?
    w
    s
    13 replies · 3 participants
  • m

    Miecio

    12/09/2020, 3:44 PM
    Hi! Do you know if during upgrade of prefect stack I need to perform some manual actions? I have everything in 0.13.19, hasura migrations done via prefect-graphql, but I'm sill receiving error, during flow run
    GRAPHQL_VALIDATION_FAILED: Cannot query field "get_or_create_task_run_info" on
            type "Mutation". Did you mean "get_or_create_task_run" or
            "get_or_create_mapped_task_run_children"?
    j
    v
    +1
    4 replies · 4 participants
  • f

    Francisco

    12/09/2020, 8:27 PM
    Hello Prefect Community! If a task fail... exist any way to retry from some upstream task? Example, I have three tasks: A->B->C If C fail, exist any way to re-run (retry) from task B? Or task A? Thanks in advance 😉
    m
    l
    +1
    4 replies · 4 participants
  • n

    Newskooler

    12/10/2020, 3:07 PM
    Hi 👋 , Any tips on how I might debug this 🤔
    prefect[9118]: Pulling postgres ...
    prefect[9118]: Pulling hasura   ...
    prefect[9118]: Pulling graphql  ...
    prefect[9118]: Pulling towel    ...
    prefect[9118]: Pulling apollo   ...
    prefect[9118]: Pulling ui       ...
    prefect[9118]: [29.6K blob data] Host is already in use by another container
    prefect[9118]: [47B blob data]prefect[9118]: ERROR: for cli_postgres_1  Cannot start service postgres: driver failed programming external connectivity on endpoint cli_postgres_1 (32c027fb4da78cf7d569efc0a4f33e5baed8621c161b7c4a2c5efca694b3912c): Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use
    Dec 10 14:53:26 mb-01 prefect[9118]: ERROR: for postgres  Cannot start service postgres: driver failed programming external connectivity on endpoint cli_postgres_1 (32c027fb4da78cf7d569efc0a4f33e5baed8621c161b7c4a2c5efca694b3912c): Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use
    prefect[9118]: Encountered errors while bringing up the project.
    g
    j
    18 replies · 3 participants
  • n

    Newskooler

    12/10/2020, 3:28 PM
    👋 What is the best advise for cleaning up
    prefect/flows
    so that we can keep it at a manageable size (it tends to just grow naturally; and it does so quite quickly)?
    🤔
    m
    d
    27 replies · 3 participants
  • j

    Julio Venegas

    12/10/2020, 3:35 PM
    Hi everyone, I have an issue launching prefect server from an EC2 instance, when I access the dashboard, it won’t load anything. Anyone had this issue before?
    m
    c
    +1
    22 replies · 4 participants
  • p

    Pedro Martins

    12/10/2020, 6:53 PM
    Hey everyone! I'm a new user of prefect and I can say this tool is very promising. After reading the documentation I didn't find a way to have prefect server a kubernetes deployment with prefect ui as a service. Does anyone here use it that way?
    m
    21 replies · 2 participants
  • j

    Julio Venegas

    12/10/2020, 8:30 PM
    If anyone is working on a systemd service file for both prefect server and agents, let me know! haven’t had success with a local agent
    :upvote: 2
  • c

    Charles Leung

    12/10/2020, 9:40 PM
    Hey Team, I was testing a sample file from someone else that has some relative paths to configuration files. When i'm building using docker storage, and i specify the files={} keyword, what path should i put it in to be the same directory as where the flow is executing? edit - I'm using a base image of prefecthq/prefect docker image
    n
    3 replies · 2 participants
  • c

    Clemens

    12/11/2020, 9:12 AM
    Hi everyone, I have trouble getting access to the prefect UI after upgrading to 13.19. We run prefect-server on Kubernetes (spun up via the helm chart) and I connect with port-forwarding. The UI is displayed and tell me to create a tenant, if I execute the command however I get an error -> in thread to declutter
    n
    m
    5 replies · 3 participants
  • j

    jack

    12/11/2020, 7:07 PM
    Multiple Agents for Disparate Projects? Given several disparate projects, each with its own dependencies (installed packages and custom python modules), does it make sense to run: A. One agent pre project or B. One agent to service all the projects?
    n
    6 replies · 2 participants
  • j

    Joseph

    12/11/2020, 10:44 PM
    I am running the server on premesis. If I create a very simple graph with 200 nodes and then click “Schematic” in the UI, the I get no page load and error code: 5. If I reduce the amount of nodes to 100, it seems to work just fine. Is this expected? 200 doesn’t feel like a lot nodes (we have dags with well over 100K nodes). Here’s my graph:
    @task
        def return2(x):
            return 2 + x
    
        @task
        def say_hello(x):
            print(f"Hello, world! {x}")
    
    
        with Flow("My First Flow") as flow:
            for x in range(0, 100):
                say_hello(return2(x))
    
        flow.register("test")
    n
    31 replies · 2 participants
  • p

    Pedro Machado

    12/12/2020, 12:24 AM
    Hi there. We want to run a small instance of Prefect in an AWS instance. We'll probably be using the Docker agent to run some fuzzy matching (entity resolution) flows in the same instance. This is a beta of a service we may grow, but in the beginning we wanted to start with simple infrastructure. Unfortunately, Prefect Cloud is disallowed at this stage because it would have to be vetted by our client's security team and that can't happen in the near future and that's why we are starting with Prefect Server. My question is ... do you have any recommendations as to which instance type and size we might want to start with to run Prefect Server? Some flows will be somewhat memory and CPU intensive but we won't be running a high volume of them. Thanks!
    m
    6 replies · 2 participants
  • m

    Matt Drago

    12/14/2020, 9:28 AM
    [✔️ solved] Hey Folks, Any tips on how to start digging into this error:
    Failed to write log with error: 413 Client Error: Payload Too Large for url: <http://dataflow-apollo:4200/graphql//graphql>
    The flow hadn't even run the first task, which was a
    EnvVarSecret
    . Other instances of the Flow have run successfully.
    4 replies · 1 participant
  • l

    Lukas N.

    12/14/2020, 2:52 PM
    [✔️ solved] Mapped task issue with apollo (more in thread)
    s
    m
    5 replies · 3 participants
  • p

    Pedro Martins

    12/14/2020, 5:23 PM
    The PREFECT__CLOUD__AGENT__AGENT_ADDRESS was wrong on my yaml. Prefect Kubernetes agent pod failing liveness probe. I deployed a k8s agent using the deployment yaml provided by prefect. I updated the PREFECT_CLOUD__API to point to my own server. But the agent pod keeps restarting because of the probeness failure:
    Events:
      Type     Reason     Age   From               Message
      ----     ------     ----  ----               -------
      Normal   Scheduled  103s  default-scheduler  Successfully assigned default/prefect-agent-c4b68bfd9-dj8cf to ip-10-0-1-20.eu-west-1.compute.internal
      Normal   Pulling    102s  kubelet            Pulling image "prefecthq/prefect:0.13.19-python3.6"
      Normal   Pulled     101s  kubelet            Successfully pulled image "prefecthq/prefect:0.13.19-python3.6"
      Normal   Created    101s  kubelet            Created container agent
      Normal   Started    101s  kubelet            Started container agent
      Warning  Unhealthy  37s   kubelet            Liveness probe failed: Get <http://10.0.1.64:8080/api/health>: dial tcp 10.0.1.64:8080: connect: connection refused
    Note that the agent can successfully register to the UI. deployment.yaml in the thread. Does anyone one know how to fix it?
    ✅ 1
    s
    3 replies · 2 participants
  • j

    jack

    12/15/2020, 2:32 AM
    How do I get
    print()
    statements to show up in the web console at localhost:8080? Here's my example code that prints to the terminal, but does not now up in the web console:
    from prefect import task, Flow
    
    @task
    def simple_func(log_stdout=True):
        print('simple print statement')
    
    
    if __name__ == '__main__':
        with Flow('test-logging') as flow:
            simple_func()
    
        flow.register('test-log')
        flow.run()
    n
    36 replies · 2 participants
  • j

    jack

    12/15/2020, 3:50 PM
    Say the agent is running under user alice, and the flows are being registered by user bob. Into whose home directory does ~/.prefect/config.toml belong?
    j
    9 replies · 2 participants
  • j

    Joseph

    12/15/2020, 8:43 PM
    Is there any way to create a one to many mapping between a task A and B, where B depends on A, but B’s mapping dimension is of lower cardinality than A’s so that some instances of B would depend on multiple instances of A?
    j
    8 replies · 2 participants
  • k

    KIRYL BUCHA

    12/16/2020, 9:34 AM
    Hi to all, I'm CTO of DataMola LLC company. I looking for help and contributors. Does anyone want to help combine RAPIDS.IO + VISION profiler ( plan to refactor on cuDF - https://github.com/dylan-profiler/visions ) + Great Expectation + Prefect.IO ? The primary goal to create the RAW data profiler on DASK + RAPIDS.IO on GPU distributed Environemnent.
    a
    3 replies · 2 participants
  • m

    Mike Marinaccio

    12/16/2020, 6:30 PM
    I’m curious if anyone has run into a
    boto3.client()
    issue when running
    prefect agent ecs start
    . In short, I receive the following error:
    File "/Users/mike/.virtualenvs/reporting-Q7tyta-k/lib/python3.9/site-packages/prefect/utilities/aws.py", line 55, in get_boto_client
        return boto3.client(
    TypeError: boto3.client() got multiple values for keyword argument 'aws_access_key_id'
    From the looks of it, the below code snippet from
    prefect/utilities/aws.py
    is running, in my case, such that none of the
    kwargs.pop(...)
    conditionals are executed. My creds are being pulled from
    ~/.prefect/config.toml
    . This would mean that the code is adding duplicate keyword args to
    session.client()
    (since they also exist in **kwargs with
    None
    values). If I remove
    **kwargs
    , it works.
    j
    4 replies · 2 participants
  • c

    Charles Leung

    12/17/2020, 8:28 PM
    Hey guys, I'm getting an error when trying to call flow.register on a windows environment; using a basic flow as a test:
    Result check: OK
    ---------------------------------------------------------------------------
    ClientError                               Traceback (most recent call last)
    <ipython-input-11-014533001643> in <module>()
    ----> 1 flow.register(project_name='test-charles')
    
    C:\ProgramData\Anaconda3\lib\site-packages\prefect\core\flow.py in register(self, project_name, build, labels, set_schedule_active, version_group_id, no_url, idempotency_key, **kwargs)
       1667             version_group_id=version_group_id,
       1668             no_url=no_url,
    -> 1669             idempotency_key=idempotency_key,
       1670         )
       1671         return registered_flow
    
    C:\ProgramData\Anaconda3\lib\site-packages\prefect\client\client.py in register(self, flow, project_name, build, set_schedule_active, version_group_id, compressed, no_url, idempotency_key)
        821             create_mutation,
        822             variables=dict(input=inputs),
    --> 823             retry_on_api_error=False,
        824         )  # type: Any
        825 
    
    C:\ProgramData\Anaconda3\lib\site-packages\prefect\client\client.py in graphql(self, query, raise_on_error, headers, variables, token, retry_on_api_error)
        316             ):
        317                 raise VersionLockError(result["errors"])
    --> 318             raise ClientError(result["errors"])
        319         else:
        320             return GraphQLResult(result)  # type: ignore
    
    ClientError: [{'message': "'NoneType' object has no attribute 'get'", 'locations': [{'line': 2, 'column': 5}], 'path': ['create_flow_from_compressed_string'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': "'NoneType' object has no attribute 'get'"}}}]
    What's this error message mean? Using the latest version of prefect for python. Thanks,
    d
    5 replies · 2 participants
Powered by Linen
Title
c

Charles Leung

12/17/2020, 8:28 PM
Hey guys, I'm getting an error when trying to call flow.register on a windows environment; using a basic flow as a test:
Result check: OK
---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
<ipython-input-11-014533001643> in <module>()
----> 1 flow.register(project_name='test-charles')

C:\ProgramData\Anaconda3\lib\site-packages\prefect\core\flow.py in register(self, project_name, build, labels, set_schedule_active, version_group_id, no_url, idempotency_key, **kwargs)
   1667             version_group_id=version_group_id,
   1668             no_url=no_url,
-> 1669             idempotency_key=idempotency_key,
   1670         )
   1671         return registered_flow

C:\ProgramData\Anaconda3\lib\site-packages\prefect\client\client.py in register(self, flow, project_name, build, set_schedule_active, version_group_id, compressed, no_url, idempotency_key)
    821             create_mutation,
    822             variables=dict(input=inputs),
--> 823             retry_on_api_error=False,
    824         )  # type: Any
    825 

C:\ProgramData\Anaconda3\lib\site-packages\prefect\client\client.py in graphql(self, query, raise_on_error, headers, variables, token, retry_on_api_error)
    316             ):
    317                 raise VersionLockError(result["errors"])
--> 318             raise ClientError(result["errors"])
    319         else:
    320             return GraphQLResult(result)  # type: ignore

ClientError: [{'message': "'NoneType' object has no attribute 'get'", 'locations': [{'line': 2, 'column': 5}], 'path': ['create_flow_from_compressed_string'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': "'NoneType' object has no attribute 'get'"}}}]
What's this error message mean? Using the latest version of prefect for python. Thanks,
d

Dylan

12/17/2020, 8:30 PM
Hi @Charles Leung, This looks potentially like a Prefect version mismatch between the version of the Flow and the version of your Prefect Server instance. Is the version of your Prefect Server instance > Flow’s Prefect version?
c

Charles Leung

12/17/2020, 8:37 PM
Ah, i see. Thanks Dylan i'll try downgrading my prefect!
d

Dylan

12/17/2020, 8:39 PM
Anytime!
c

Charles Leung

12/17/2020, 8:39 PM
fixed it right away 🙂 thanks!
d

Dylan

12/17/2020, 8:39 PM
Glad I could help!
View count: 2