• Darragh

    Darragh

    2 years ago
    Apologies in advance for this, it’s probably going to sound extremely basic, but it’s something I’m having trouble wrapping my mental model around. I have Prefect installed on an EC2 instance that I want to use to run all my flows. I also have a stupidly basic HelloWorld flow that does nothing, but it’s still a Flow. All good so far! Here’s where my trouble starts - I want to take my flows and deploy/register them to my Prefect instance on EC2 as part of my CI/CD process, but I’m having trouble understanding the procedure for this, even after reading the Storage and Execution Environment sections of the docs. So what I’d like to find out is • How do I get my python flow files in GitLab registered to AWS Prefect? • Do they have to be in Docker storage? What does the Docker container actually contain? It looked like it was pulling a lot of dependencies when I built it.. • Is it possible to just bundle the flow files and have them imported by my instance? • The execution environment, is the LocalEnvironment suitable for what I’m looking for, just to get it started? And probably most importantly, is there a guideline doc on how to deploy/register flows using a non cloud Prefect instance? 🙂 Thanks!
    Darragh
    Zachary Hughes
    +2
    12 replies
    Copy to Clipboard
  • m

    Matthias

    2 years ago
    Hi Prefect Crew 😃 I have a
    task
    that returns a
    list
    . Using
    map
    I pass the list to another task, which per each item returns again a
    list
    . I want to
    map()
    this nested
    list
    to anoter task to iterate over all the items. Is there any intended way of doing so? Thanks!
    m
    Jeremiah
    4 replies
    Copy to Clipboard
  • b

    Bartek

    2 years ago
    Hi Is it possible to run prefect agent inside docker?
    b
    i
    +1
    9 replies
    Copy to Clipboard
  • t

    tkanas

    2 years ago
    Hello! I am having a particular problem with Prefect flows. When I create an extremely simple flow, with a single task (can be anything, just the identity function is fine), but pass in a parameter that is a NamedTuple with more than one entry, it doesn't parse the keys of the NamedTuple correctly. Instead of expanding them like a dictionary, it treats them all as a single entity. I'm happy to provide example code if needed.
    t
    Zachary Hughes
    6 replies
    Copy to Clipboard
  • d

    David N

    2 years ago
    Hi everyone. Is there a good way to gracefully cancel a flows next scheduled run? I have flows scheduled every 5 minutes and when I check a change in, I want the flow to finish the current run, but stop after that so I can execute a git pull and re-deploy the dask workers, etc. Is there a good way to do that, other than watching the session and killing the process when the current flow state is "waiting on next scheduled start time"?
    d
    nicholas
    +1
    11 replies
    Copy to Clipboard
  • Scott Zelenka

    Scott Zelenka

    2 years ago
    Is it possible to have a service-account user token, which is capable of creating a Flow Run (via GraphQL post) scoped to a specific Project from our Cloud instance?
    Scott Zelenka
    Zachary Hughes
    3 replies
    Copy to Clipboard
  • m

    Matthias

    2 years ago
    Is it possible to map a pandas DataFrame?
    m
    Zachary Hughes
    +2
    10 replies
    Copy to Clipboard
  • Jacob (he/him)

    Jacob (he/him)

    2 years ago
    Hello everyone 👋 I’m working on a proof of concept for Prefect using AWS and am having a little bit of conceptual difficulty understanding how all of the prefect pieces interact with each other. Additionally, I am curious what development workflows people use to register additional flows (git or ECR). Here is what I’m currently trying and want to validate that this is the prefect way to do it. I want to put prefect server on ec2 (until we are approved for cloud), run an agent (or a few) on separate ec2 instances. Register flows to server using the aws cli (that contain fargate environment specifics). Am I on the right track or should I try something different?
    Jacob (he/him)
    Zachary Hughes
    3 replies
    Copy to Clipboard
  • Florian K. (He/Him)

    Florian K. (He/Him)

    2 years ago
    Hi All! I am investigating the possibility of using prefect as workflow management tool in hydraulic and hydrological modeling systems. While my flow runs nicely as a script it throws an error when I register it with the local agent and start it from the server. We want to use a protocol buffer to move time series data between tasks but when starting the flow through the web UI, it throws the error below. I assume this has something to do with how data is transferred trom one task to another? If so, is it not possible to map (??) protobuffs? Would it be an adequate workaround to transform the protocol buffer messages to dictionaries?
    Unexpected error: TypeError("cannot pickle 'google.protobuf.pyext._message.MessageDescriptor' object")
    Traceback (most recent call last):
      File "/home/fkluiben/bin/miniconda3/envs/RodeoEnv/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
        new_state = method(self, state, *args, **kwargs)
      File "/home/fkluiben/bin/miniconda3/envs/RodeoEnv/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 934, in get_task_run_state
        state._result.store_safe_value()
      File "/home/fkluiben/bin/miniconda3/envs/RodeoEnv/lib/python3.8/site-packages/prefect/engine/result/base.py", line 126, in store_safe_value
        value = self.result_handler.write(self.value)
      File "/home/fkluiben/bin/miniconda3/envs/RodeoEnv/lib/python3.8/site-packages/prefect/engine/result_handlers/local_result_handler.py", line 81, in write
        f.write(cloudpickle.dumps(result))
      File "/home/fkluiben/bin/miniconda3/envs/RodeoEnv/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 63, in dumps
        cp.dump(obj)
      File "/home/fkluiben/bin/miniconda3/envs/RodeoEnv/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 548, in dump
        return Pickler.dump(self, obj)
    TypeError: cannot pickle 'google.protobuf.pyext._message.MessageDescriptor' object
    BTW, apologies for my way of describing the issue. I am sure it's pretty clumsy and inaccurate. Looking forward to learn and raise my awareness of prefect terminology!!
    Florian K. (He/Him)
    Jim Crist-Harif
    +1
    11 replies
    Copy to Clipboard
  • Aiden Price

    Aiden Price

    2 years ago
    Hi Prefect folks, I’m looking into deploying Prefect Server in our Azure Kubernetes cluster, and I’m a bit confused by the different parts (Server, UI, backend DB, agent, environment and so forth) is there a diagram that explains how the separate parts interact?
    Aiden Price
    i
    +1
    3 replies
    Copy to Clipboard