• Madison Schott

    Madison Schott

    1 year ago
    Hi all, trying to figure out the best way to deploy by Prefect agent to AWS and came across this tutorial- has anyone followed this for their production deployments? What are the benefits of reading the flow code from S3 bucket? Thanks! https://towardsdatascience.com/serverless-data-pipelines-made-easy-with-prefect-and-aws-ecs-fargate-7e25bacb450c
    Madison Schott
    Kevin Kho
    +2
    107 replies
    Copy to Clipboard
  • Marcus Hughes

    Marcus Hughes

    1 year ago
    I've been using Prefect to run a couple automated flows of some simple tasks on a local server, and I just discovered that Prefect appears to be saving serialized/pickled results into
    ~/.prefect/results
    without me specifically telling it to. That's fine in the short term, but after letting things run I discovered I had over 800 gigabytes of results. Is there an automated way to delete these after some given time elapses built into Prefect? I could just make another flow that cleans up results that are older than a day or something to keep us from ballooning out of drive space. Just curious what the best practice is here. Also, is it safe for me to just delete entries from this directory or will it corrupt a database somewhere?
    Marcus Hughes
    Chris White
    6 replies
    Copy to Clipboard
  • Ryan Sattler

    Ryan Sattler

    1 year ago
    hi - in my test environment I need to access my S3 Storage (mocked out with Localstack) via a different base URL depending on if I’m registering the container (via my local command line) or actually running it (in local Kubernetes). I’ve tried to configure things like this:
    home = os.environ.get("HOME")
    if home != "/Users/ryan.sattler":
        s3_url = "<http://localstack:4566>"
    else:
        s3_url = "<http://localhost:31566>"
    
    flow.storage = S3(bucket="prefect-flows", key="hello-task.py", client_options={
        "endpoint_url": s3_url,
        "aws_access_key_id": "",
        "aws_secret_access_key": ""
    })
    However this doesn’t seem to work (the localhost url is always being used, so it works for registration but not at runtime), possibly because the value of flow.storage is getting baked-in at registration time? How can I make this dynamic? Unfortunately given my company’s security setup there is no convenient way to use a real S3 bucket (or real container registry or etc) when testing locally.
    Ryan Sattler
    Chris White
    8 replies
    Copy to Clipboard
  • Сергей Романкевич

    Сергей Романкевич

    1 year ago
    Hi everyone! After updating the server to version 15, I cannot register new flows. The server itself works fine and executes the already registered flows. When I try to run the code, I get the following error: requests.exceptions.HTTPError: 407 Client Error: Proxy Authentication Required for url: http: // localhost: 4200 / graphql How can I solve this error?
    rsv@srv-etl:~/flow$ /bin/python /home/rsv@comandor.local/flow/Test.py
    Traceback (most recent call last):
      File "/home/rsv@comandor.local/flow/Test.py", line 13, in <module>
        flow.register(project_name='Test')
      File "/usr/local/lib/python3.7/dist-packages/prefect/core/flow.py", line 1734, in register
        idempotency_key=idempotency_key,
      File "/usr/local/lib/python3.7/dist-packages/prefect/client/client.py", line 1081, in register
        project = self.graphql(query_project).data.project  # type: ignore
      File "/usr/local/lib/python3.7/dist-packages/prefect/client/client.py", line 534, in graphql
        retry_on_api_error=retry_on_api_error,
      File "/usr/local/lib/python3.7/dist-packages/prefect/client/client.py", line 452, in post
        retry_on_api_error=retry_on_api_error,
      File "/usr/local/lib/python3.7/dist-packages/prefect/client/client.py", line 696, in _request
        session=session, method=method, url=url, params=params, headers=headers
      File "/usr/local/lib/python3.7/dist-packages/prefect/client/client.py", line 605, in _send_request
        response.raise_for_status()
      File "/usr/local/lib/python3.7/dist-packages/requests/models.py", line 943, in raise_for_status
        raise HTTPError(http_error_msg, response=self)
    requests.exceptions.HTTPError: 407 Client Error: Proxy Authentication Required for url: <http://localhost:4200/graphql>
      "config_overrides": {},   "env_vars": [],   "system_information": {     "platform": "Linux-4.19.0-14-amd64-x86_64-with-debian-10.8",     "prefect_backend": "server ",     " prefect_version ":" 0.15.0 ",     " python_version ":" 3.7.3 "    Test.py
    from prefect import task, Flow, Parameter
    @task(log_stdout=True)
    def say_hello(name):
        print("Hello, {}!".format(name))
    with Flow("My First Flow"as flow:
        name = Parameter('name')
        say_hello(name)
    flow.register(project_name='Test')
    Сергей Романкевич
    Kevin Kho
    +1
    9 replies
    Copy to Clipboard
  • Rodrigo Menezes

    Rodrigo Menezes

    1 year ago
    Any chance we can update the Prefect Docker packages? Snyk caught some minor vulns on the current version. The problem is upstream. I think the official Docker Python image was a little behind. I think a simple re-build should fix this. (None of these vulns are important if your Prefect agent is in a private subnet. I mostly care to make our automated security tools shut up 🙃 )
    Rodrigo Menezes
    Kevin Kho
    2 replies
    Copy to Clipboard
  • j

    Julio Venegas

    1 year ago
    hey prefect-server folks! I’m working on a state handler function that sends a notification to teams when a tasks fails, the same task is retried and when the retry of the task is successful. The notification on failure and retry of the task is working fine, but I’m not managing to trigger the notification when the retry is successful. Any help? Code inside thread.
    j
    3 replies
    Copy to Clipboard
  • YD

    YD

    1 year ago
    Can you manage users permissions on an on-prem Prefect.io server ?
    YD
    Kevin Kho
    2 replies
    Copy to Clipboard
  • Tom Forbes

    Tom Forbes

    1 year ago
    Is it OK to assume that when following the
    scaling out
    section of the tutorial (https://docs.prefect.io/core/tutorial/06-parallel-execution.html#scaling-out), when using a remote
    DaskExecutor
    , logs from tasks will not be mirrored locally and we will have to build a custom logger for this? This limitation isn’t explicitly detailed, which is confusing
    Tom Forbes
    Kevin Kho
    +2
    5 replies
    Copy to Clipboard
  • Madison Schott

    Madison Schott

    1 year ago
    Has anyone else ever followed this tutorial when using DbtShellTask? I've created my Dockerfile to be similar but am having issues with my dbt files https://www.lejimmy.com/distributed-data-pipelines-with-aws-ecs-fargate-and-prefect-cloud/
    Madison Schott
    Noah Holm
    +1
    28 replies
    Copy to Clipboard
  • a

    Arun Dass

    1 year ago
    Hey is their a tutorial floating around for deploying your workloads to GKE using prefect cloud backend?
    a
    Noah Holm
    2 replies
    Copy to Clipboard