Aiden Price

    Aiden Price

    1 year ago
    Hi everyone, I'm having trouble with a custom
    resource_manager
    class, the flow times out and errors like this appear;
    Error during execution of task: AttributeError("'NoneType' object has no attribute 'setup'")
    Two theories I have are either that I'm setting up more than one resource_manager in the one
    with
    statement, or perhaps that I'm calling a
    prefect.context.get("logger")
    during the
    __init__
    method.
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Aiden Price, could you show me your code?
    Aiden Price

    Aiden Price

    1 year ago
    Good morning @Kevin Kho timezone differences mean I only just saw your message now.
    An example of one of my resouce_manager classes;
    @resource_manager
    class AzureEventHubProducerClient:
        """Build an Eventhub producer client"""
    
        def __init__(self, conn_str: str, eventhub_name: str) -> None:
            self.conn_str = conn_str
            self.eventhub_name = eventhub_name
    
        def setup(self) -> EventHubProducerClient:
            logger = prefect.context.get("logger")
            <http://logger.info|logger.info>(f"Connecting to {self.eventhub_name}")
            return EventHubProducerClient.from_connection_string(
                self.conn_str, eventhub_name=self.eventhub_name, retry_total=1
            )
    
        def cleanup(self, client: EventHubProducerClient) -> None:
            logger = prefect.context.get("logger")
            <http://logger.info|logger.info>("Closing Azure Eventhub Producer client")
            client.close()
    And this is an extract from the flow where I was calling it and some others;
    # Set up clients
        with Oauth2_HTTPClient(
            base_url=coldstream_url,
            token_url=token_url,
            client_id=client_id,
            client_secret=client_secret,
            scope=scope,
        ) as http_client, AzureEventHubProducerClient(
            conn_str=eventhub_conn_str, eventhub_name=eventhub_name
        ) as eventhub_client, PostgresEngine(
            timescale_conn_str
        ) as timescale:
            all_gaps = fetch_gaps(
                engine=timescale,
                start_time=start,
                end_time=end,
                min_gap_duration=min_gap,
            )
    The line breaks aren't doing me any favours here but I promise there's no whitespace errors.
    Kevin Kho

    Kevin Kho

    1 year ago
    Oh I see. What happens if you nest them instead of one
    with
    ?
    Aiden Price

    Aiden Price

    1 year ago
    I tried nesting them and got the same result.
    I'm running Prefect Server in my own Kubernetes cluster, is there some environmental reason that it wouldn't work? I mean other than not being able to reach the services that the clients represent.
    Kevin Kho

    Kevin Kho

    1 year ago
    I think this seems like a code thing rather infrastructure. I think this might be not what the resource manager is for. You are making clients but I think the
    resource_manager
    is for hardware oriented stuff. I am not 100% sure, I’ll need to talk to someone tomorrow. Is it a specific class failing? Or any of them? Or is it the one you posted?
    Aiden Price

    Aiden Price

    1 year ago
    It's all three in that flow, but a different one that I have for FTP in a different flow is working fine.
    Kevin Kho

    Kevin Kho

    1 year ago
    I need to talk to someone about this, but if you need something more immediate, creating the Clients inside tasks will work.
    Aiden Price

    Aiden Price

    1 year ago
    Okay, thank you
    Possibly related, I've found a task that's just a bit downstream of this that ends up in an infinite loop. Is it possible that there's nothing wrong with the
    resource_managers
    but the infinite loop is causing other tasks to fail and report their errors?
    Kevin Kho

    Kevin Kho

    1 year ago
    Maybe? What happens after you fixed it?
    Aiden Price

    Aiden Price

    1 year ago
    So yes, fixing the infinite loop in the downstream task seems to have prevented the error
    Error during execution of task: AttributeError("'NoneType' object has no attribute 'setup'")
    in my
    resource_managers
    . So, is there an out of memory error or a zombie killer that might be reacting to the infinite loop by killing the
    resource_manager
    ?
    Kevin Kho

    Kevin Kho

    1 year ago
    Oh wow that's news to me. I honestly have not seen this setup where resource managers are used for Clients. I can see your explanation making sense though where the resource manager is not being torn down properly. Do you see the task being restarted?
    Aiden Price

    Aiden Price

    1 year ago
    No because the infinite loop task downstream wouldn’t relinquish control and I’d end up having to cancel the flow.