• George Shishorin

    George Shishorin

    1 year ago
    Dear Prefect Community, As it is known, there are some great features for caching tasks results, that make sometimes much convenience to reuse. But at the same time I faced with a problem related to how Prefect deal with RAM during the flow running. The main question: is it possible to remove Tasks results and thus make RAM free during the flow running? Most of flows contains quite difficult computing logic. There is no way to combine several functions into one Task (atomicity and readability will be lost). I would provide an example for prefect vs “nonprefect” flow and its RAM consumption...->
    George Shishorin
    s
    15 replies
    Copy to Clipboard
  • b

    Borut Hafner

    1 year ago
    Dear All, I want to use a fixed point in time in my tasks. An option would be to create a task whiche returns current datetime, but i suppose that flow can offer this information in form of a flow start time. Is a way to get a flow stard date time in a task? Thanks for your answers!
    b
    l
    +1
    4 replies
    Copy to Clipboard
  • Felix Schran

    Felix Schran

    1 year ago
    Hi all, in prefect core how can I save all the aumatic logging messages to a file? First (before I run the flow), I create a logging file by calling loguru.logger via logger.add("logs.txt"). logger.info("Some info message before the flow is run") then writes "Some info message before the flow is run" to the file build.txt. Next, I run the flow and then get all these nice logging messages about the tasks. For instance: [2021-01-25 12:30] INFO - prefect.TaskRunner | Task 'xyz': Finished task run for task with final state: 'Success'... How can I add these logging messages to the logging file "logs.txt" from above? So ideally my build.txt file should include in the end: ... | Some info message before the flow is run. ... | Task 'xyz': Finished task run for task with final state: 'Success'...
    Felix Schran
    Dylan
    +1
    5 replies
    Copy to Clipboard
  • Jan Marais

    Jan Marais

    1 year ago
    Hi All. I'm new to Prefect and I want to experiment with running a flow on AWS ECS. I'm using this example: https://github.com/lauralorenz/fourteen-oh/blob/main/14-fargate.py, only with S3 storage and the relevant AWS parameter changes. Also making use of a local backend. When running the above script, the flow gets registered successfully and the agent starts. When I run the flow in the UI, it gets submitted and I can also see a ECS task running in AWS console and then it disappears. But the prefect task never starts and the flow never finishes. Any idea what is wrong here? Let me know if I need to share anymore detail. Thanks!
    Jan Marais
    l
    +1
    21 replies
    Copy to Clipboard
  • ciaran

    ciaran

    1 year ago
    Hi Folks, is there a way for the Timeline view when running a flow to display each map iteration of a task in its own bar? It'd be nice to see the work actually happening in parallel visually. If it's easier to do with Dask (I'm using the
    LocalDaskExecutor
    ) - Does anyone know how I can visualise the Dask Diagnostics when running Prefect flows? I know I can look at the individual mapped tasks, but I just wanted to see if there's a similar visualisation to how Dask does it
    ciaran
    Fina Silva-Santisteban
    9 replies
    Copy to Clipboard
  • Braun Reyes

    Braun Reyes

    1 year ago
    hey there wanted to ask about this snippet For Prefect Core server or Prefect Cloud users: ... • You can override the automatic result subclass at the global level, flow level, or task level Can you set the result subclass globally?
    Braun Reyes
    Dylan
    +1
    10 replies
    Copy to Clipboard
  • p

    Philip MacMenamin

    1 year ago
    issues connecting to graphql:4200 on docker Hi, I'm trying to run the docker agent similar to https://github.com/PrefectHQ/server/issues/25 and am still unable to connect to the container it seems:
    requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd0e8943640>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
    ^C[2021-01-25 19:55:03,534] INFO - agent | Keyboard Interrupt received: Agent is shutting down.
    p
    Kyle Moon-Wright
    4 replies
    Copy to Clipboard
  • Fina Silva-Santisteban

    Fina Silva-Santisteban

    1 year ago
    Hi Prefect community, I hope everyone’s doing well! I need help setting up Docker Storage and using the docker prefect agent. My Dockerfile looks like this:
    FROM python:3.7
    WORKDIR /app
    COPY requirements.txt .
    RUN pip install -r requirements.txt
    COPY . .
    My flow like this:
    (...)
    storage = Docker(dockerfile="../Dockerfile")
    
    with Flow("Generate Report", result=LocalResult(), storage=storage) as flow:
    (...)
    When I try to register the flow, it seems to have issues with finding the
    requirements.txt
    file needed to create the image:
    [2021-01-25 15:21:03-0800] INFO - prefect.Docker | Building the flow's Docker storage...
    Step 1/14 : FROM python:3.7
     ---> ca194d6afe58
    Step 2/14 : WORKDIR /app
     ---> Using cache
     ---> a729913b700e
    Step 3/14 : COPY requirements.txt .
    COPY failed: stat /var/lib/docker/tmp/docker-builder883681743/requirements.txt: no such file or directory
    
    Traceback (most recent call last):
      File "/Users/fina/Documents/github/concierge_iro_reporting/etl/register_all_flows.py", line 53, in <module>
        register_locally()
      File "/Users/fina/Documents/github/concierge_iro_reporting/etl/register_all_flows.py", line 21, in register_locally
        report_lt_target.register(project_name=project_name)
      File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/core/flow.py", line 1675, in register
        idempotency_key=idempotency_key,
      File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/client/client.py", line 783, in register
        serialized_flow = flow.serialize(build=build)  # type: Any
      File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/core/flow.py", line 1450, in serialize
        storage = self.storage.build()  # type: Optional[Storage]
      File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/storage/docker.py", line 363, in build
        self._build_image(push=push)
      File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/storage/docker.py", line 431, in _build_image
        "Your docker image failed to build!  Your flow might have "
    ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
    My folder is structure is like this:
    App
    -->prefect_flows
      -->my_flow.py
    requirements.txt
    Dockerfile
    Why is it looking for the requirements.txt inside
    /var/lib/docker/
    instead of the App’s directory? What am I missing? (The container runs normally when I use docker-compose to build and run the app)
    Fina Silva-Santisteban
    Kyle Moon-Wright
    4 replies
    Copy to Clipboard
  • a

    Adam Roderick

    1 year ago
    Is this button not working?
    a
    1 replies
    Copy to Clipboard
  • s

    Shyam Meduri

    1 year ago
    Hi, Is it possible to upload a file to a Prefect Flow, say while initiating a Flow run, either as a parameter or context object, which can then be further read in a task?
    s
    nicholas
    2 replies
    Copy to Clipboard