• Pedro Machado

    Pedro Machado

    1 year ago
    Hi there. We want to run a small instance of Prefect in an AWS instance. We'll probably be using the Docker agent to run some fuzzy matching (entity resolution) flows in the same instance. This is a beta of a service we may grow, but in the beginning we wanted to start with simple infrastructure. Unfortunately, Prefect Cloud is disallowed at this stage because it would have to be vetted by our client's security team and that can't happen in the near future and that's why we are starting with Prefect Server. My question is ... do you have any recommendations as to which instance type and size we might want to start with to run Prefect Server? Some flows will be somewhat memory and CPU intensive but we won't be running a high volume of them. Thanks!
    Pedro Machado
    Michael Adkins
    6 replies
    Copy to Clipboard
  • Matt Drago

    Matt Drago

    1 year ago
    [✔️ solved] Hey Folks, Any tips on how to start digging into this error:
    Failed to write log with error: 413 Client Error: Payload Too Large for url: <http://dataflow-apollo:4200/graphql//graphql>
    The flow hadn't even run the first task, which was a
    EnvVarSecret
    . Other instances of the Flow have run successfully.
    Matt Drago
    4 replies
    Copy to Clipboard
  • l

    Lukas N.

    1 year ago
    [✔️ solved] Mapped task issue with apollo (more in thread)
    l
    s
    +1
    5 replies
    Copy to Clipboard
  • Pedro Martins

    Pedro Martins

    1 year ago
    The PREFECT__CLOUD__AGENT__AGENT_ADDRESS was wrong on my yaml. Prefect Kubernetes agent pod failing liveness probe. I deployed a k8s agent using the deployment yaml provided by prefect. I updated the PREFECT_CLOUD__API to point to my own server. But the agent pod keeps restarting because of the probeness failure:
    Events:
      Type     Reason     Age   From               Message
      ----     ------     ----  ----               -------
      Normal   Scheduled  103s  default-scheduler  Successfully assigned default/prefect-agent-c4b68bfd9-dj8cf to ip-10-0-1-20.eu-west-1.compute.internal
      Normal   Pulling    102s  kubelet            Pulling image "prefecthq/prefect:0.13.19-python3.6"
      Normal   Pulled     101s  kubelet            Successfully pulled image "prefecthq/prefect:0.13.19-python3.6"
      Normal   Created    101s  kubelet            Created container agent
      Normal   Started    101s  kubelet            Started container agent
      Warning  Unhealthy  37s   kubelet            Liveness probe failed: Get <http://10.0.1.64:8080/api/health>: dial tcp 10.0.1.64:8080: connect: connection refused
    Note that the agent can successfully register to the UI. deployment.yaml in the thread. Does anyone one know how to fix it?
    Pedro Martins
    Sagun Garg
    3 replies
    Copy to Clipboard
  • jack

    jack

    1 year ago
    How do I get
    print()
    statements to show up in the web console at localhost:8080? Here's my example code that prints to the terminal, but does not now up in the web console:
    from prefect import task, Flow
    
    @task
    def simple_func(log_stdout=True):
        print('simple print statement')
    
    
    if __name__ == '__main__':
        with Flow('test-logging') as flow:
            simple_func()
    
        flow.register('test-log')
        flow.run()
    jack
    nicholas
    36 replies
    Copy to Clipboard
  • jack

    jack

    1 year ago
    Say the agent is running under user alice, and the flows are being registered by user bob. Into whose home directory does ~/.prefect/config.toml belong?
    jack
    Jim Crist-Harif
    9 replies
    Copy to Clipboard
  • j

    Joseph

    1 year ago
    Is there any way to create a one to many mapping between a task A and B, where B depends on A, but B’s mapping dimension is of lower cardinality than A’s so that some instances of B would depend on multiple instances of A?
    j
    Jim Crist-Harif
    8 replies
    Copy to Clipboard
  • k

    KIRYL BUCHA

    1 year ago
    Hi to all, I'm CTO of DataMola LLC company. I looking for help and contributors. Does anyone want to help combine RAPIDS.IO + VISION profiler ( plan to refactor on cuDF - https://github.com/dylan-profiler/visions ) + Great Expectation + Prefect.IO ? The primary goal to create the RAW data profiler on DASK + RAPIDS.IO on GPU distributed Environemnent.
    k
    Allyson Lubimir
    3 replies
    Copy to Clipboard
  • Mike Marinaccio

    Mike Marinaccio

    1 year ago
    I’m curious if anyone has run into a
    boto3.client()
    issue when running
    prefect agent ecs start
    . In short, I receive the following error:
    File "/Users/mike/.virtualenvs/reporting-Q7tyta-k/lib/python3.9/site-packages/prefect/utilities/aws.py", line 55, in get_boto_client
        return boto3.client(
    TypeError: boto3.client() got multiple values for keyword argument 'aws_access_key_id'
    From the looks of it, the below code snippet from
    prefect/utilities/aws.py
    is running, in my case, such that none of the
    kwargs.pop(...)
    conditionals are executed. My creds are being pulled from
    ~/.prefect/config.toml
    . This would mean that the code is adding duplicate keyword args to
    session.client()
    (since they also exist in **kwargs with
    None
    values). If I remove
    **kwargs
    , it works.
    Mike Marinaccio
    j
    4 replies
    Copy to Clipboard
  • c

    Charles Leung

    1 year ago
    Hey guys, I'm getting an error when trying to call flow.register on a windows environment; using a basic flow as a test:
    Result check: OK
    ---------------------------------------------------------------------------
    ClientError                               Traceback (most recent call last)
    <ipython-input-11-014533001643> in <module>()
    ----> 1 flow.register(project_name='test-charles')
    
    C:\ProgramData\Anaconda3\lib\site-packages\prefect\core\flow.py in register(self, project_name, build, labels, set_schedule_active, version_group_id, no_url, idempotency_key, **kwargs)
       1667             version_group_id=version_group_id,
       1668             no_url=no_url,
    -> 1669             idempotency_key=idempotency_key,
       1670         )
       1671         return registered_flow
    
    C:\ProgramData\Anaconda3\lib\site-packages\prefect\client\client.py in register(self, flow, project_name, build, set_schedule_active, version_group_id, compressed, no_url, idempotency_key)
        821             create_mutation,
        822             variables=dict(input=inputs),
    --> 823             retry_on_api_error=False,
        824         )  # type: Any
        825 
    
    C:\ProgramData\Anaconda3\lib\site-packages\prefect\client\client.py in graphql(self, query, raise_on_error, headers, variables, token, retry_on_api_error)
        316             ):
        317                 raise VersionLockError(result["errors"])
    --> 318             raise ClientError(result["errors"])
        319         else:
        320             return GraphQLResult(result)  # type: ignore
    
    ClientError: [{'message': "'NoneType' object has no attribute 'get'", 'locations': [{'line': 2, 'column': 5}], 'path': ['create_flow_from_compressed_string'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': "'NoneType' object has no attribute 'get'"}}}]
    What's this error message mean? Using the latest version of prefect for python. Thanks,
    c
    Dylan
    5 replies
    Copy to Clipboard