• Slava Shor

    Slava Shor

    6 months ago
    Python 3.10 despite the tuple typing annotation there is still
    TypeError: Task is not iterable
    exception.
    @task(timeout=timedelta(seconds=30))
    def fetch_symbols(...) -> tuple[MyCustomType, ...]:
      ...
    Slava Shor
    Kevin Kho
    17 replies
    Copy to Clipboard
  • j

    Jacob Longhurst

    6 months ago
    I’m wondering if it’s possible for Prefect to run the same Flow at the same time with different Parameters? Let’s say i have a flow call Balance and i want to run two or more instances at the same time but with different input Parameters, is that possible?
    j
    Kevin Kho
    +1
    30 replies
    Copy to Clipboard
  • Darren Burton

    Darren Burton

    6 months ago
    I upgraded to My Orion instance last night to 2.0beta with the server running. It looked like everything worked from a UI perspective. However, when I run a flow they all fail with "Crash detected! Execution was interrupted by an unexpected exception. I am thinking I should probably stop all of the Prefect process and restart the Server. I don't see a command line or an option in the UI to stop or restart the Orion Server.
    Darren Burton
    Michael Adkins
    +1
    65 replies
    Copy to Clipboard
  • Lana Dann

    Lana Dann

    6 months ago
    is there a way to validate flows before registering them? i’d like to test things locally like storage, etc. that would otherwise cause an error on the registering step
    Lana Dann
    Kevin Kho
    +1
    4 replies
    Copy to Clipboard
  • Bruno Nunes

    Bruno Nunes

    6 months ago
    My prefect agent stopped working and is showing: prefect.exceptions.ClientError: Your Prefect Server instance has no tenants. Create a tenant with prefect server create-tenant`` When I try to run
    prefect server create-tenant --name default --slug default
    I'm getting the following error:
    prefect.exceptions.ClientError: [{'message': 'Uniqueness violation.', 'locations': [{'line': 2, 'column': 5}], 'path': ['create_tenant'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Uniqueness violation.'}}}]
    It seems that the tenant already exists. Do you know what might be the problem?
    Bruno Nunes
    Kevin Kho
    6 replies
    Copy to Clipboard
  • Ross Rochford

    Ross Rochford

    6 months ago
    Hi everyone, I'd like to try Orion, I noticed that there is some asyncio support using AnyIO. However, I don't see anything listed in the task runners documentation. Is this feature disabled for now?
    Ross Rochford
    Kevin Kho
    +1
    5 replies
    Copy to Clipboard
  • Ross Rochford

    Ross Rochford

    6 months ago
    One more question, is there a docker-compose file somewhere for Orion to get started quickly? I'd rather this than using a cli. With the previous version, I had to reverse engineer the cli to create one.
    Ross Rochford
    Kevin Kho
    +1
    11 replies
    Copy to Clipboard
  • Royzac

    Royzac

    6 months ago
    Is there an architectural diagram available that showcases prefect with an aws infrastructure?
    Royzac
    Kevin Kho
    +1
    4 replies
    Copy to Clipboard
  • j

    Jacob Longhurst

    6 months ago
    I’m attempting to grab the logger out of the prefect context. I am doing something similar to this
    @task()
    def my_task():
        import prefect
        logger = prefect.context.get('logger')
    I’ve tried it both with having the
    import prefect
    at the top level outside of the task and in the task like in that example and both times it gives me this
    TypeError: Pickling context objects is explicitly not supported. You should always access context as an attribute of the `prefect` module, as in `prefect.context`
    Anyone have any ideas why it’s failing when i try to register the flow?
    j
    Kevin Kho
    +1
    7 replies
    Copy to Clipboard
  • Sam Brownlow

    Sam Brownlow

    6 months ago
    tl;dr How scalable is Prefect Server for scheduling concurrent runs of 10s of thousands of flows? I'm evaluating Prefect Server as a solution for managing a data pipeline. How I envisioned this working was using the Prefect framework as a computationally minimal "glue." Prefect would be the central pipe through which the state of the data flows (but not the data itself). It would hand off all computation to microservices via (a)sync requests and its only concern would be controlling how the state is pushed from service to service. Prefect seems to be a great tool for this job; I really like the API and how quickly a developer can get up and running. However, there are a couple of things that I don't yet have total clarity on. This documentation describes Server as a "highly scalable scheduler" but also says that it may start degrading at "~10-20 tasks running concurrently against a typical Server." Does the above degradation occur because the
    typical Server
    is deployed to a single node via docker-compose with a single agent, instead of being deployed via something like Helm, which horizontally scales scheduled flows across a cluster of agents? How quickly does the size of the PostgreSQL database generally grow, relative to the number of flows run? Is there any reason that regularly deleting old runs would be any more complicated than as suggested here? Are there any case studies with Prefect Server being used to run 10s of thousands of concurrent flows? I see that there used to be a Nomad Agent, are there any helpful resources for running Prefect on a nomad cluster? Thanks for any advice you are able to share. I have been diving into Prefect for only the past couple of days so greatly appreciate any referential pointers here.
    Sam Brownlow
    Anna Geller
    9 replies
    Copy to Clipboard