Thread
#prefect-community
    a

    Anatoly Myachev

    6 months ago
    Is it possible to configurate Orion to use KubernetesFlowRunner+
    DaskTaskRunner
    that use
    dask_cloudprovider.aws.FargateCluster
    ? There is no logs in Orion UI except
    'task-b610b3b0-0' - Crash detected! Execution was interrupted by an unexpected exception.
    and
    RuntimeError: IOLoop is closed
    but it's possible to see it in CloudWatch console. It contains the following error:
    Exception: 'RuntimeError("Cannot orchestrate task run \'9987d480-fad1-433a-9749-077a63fbdc0a\'. Failed to connect to API at <http://orion:4200/api/>.")'
    . Note: Orion is deployed on Kubernetes via
    orion kubernetes-manifest
    command.
    Anna Geller

    Anna Geller

    6 months ago
    If you use
    KubernetesFlowRunner
    , then it makes much more sense to use
    KubeCluster
    instead of
    FargateCluster
    ? Would you want to give it a try?
    this way you can run all components on the same Kubernetes cluster and you can prevent various network issues
    are you running this on AWS EKS?
    a

    Anatoly Myachev

    6 months ago
    I think it's worth a try with
    KubeCluster
    . However, I am experimenting with a possible case where there are not enough resources in the current Kubernetes cluster and I want to perform part of the load in AWS cluster (EKS/ECS). So in this case, work will occur with two different clusters. Current K8s cluster is a k3s running on one AWS host.
    Anna Geller

    Anna Geller

    6 months ago
    so it's technically not a cluster 😄 You could definitely use EKS on Fargate which would allow you to use all the benefits of Kubernetes but run things on Fargate (getting all the benefits and drawbacks of serverless)
    we definitely don't have any ECS Fargate recipes (and no ECSFlowRunner) as of today and I'd say Kubernetes is a way more prevalent use case so if we would create some tutorials on running Dask on a remote cluster, I think we would first focus on Kubernetes use case with
    KubeCluster
    . If you're interested, you can follow the dask-task-runner tag or the kube-cluster tag on Discourse to get notified about any tutorials/content/recipes about it but of course, if you would want to build and contribute one, that would be super appreciated as well!
    a

    Anatoly Myachev

    6 months ago
    @Anna Geller thanks, I will take a look