Nano GPU Scheduler

About This Project

With the continuous evolution of cloud native AI scenarios, more and more users run AI tasks on Kubernetes, which also brings more and more challenges to GPU resource scheduling.

Nano gpu scheduler is a gpu scheduling framework based on Kubernetes, which focuses on fine-grained gpu resource scheduling and allocation.

You may also be interested in Nano GPU Agent which is a Kubernetes device plugin implement.

Motivation

In the GPU container field, GPU providers such as nvidia have introduced a docker-based gpu containerization project that allows users to use GPU cards in Kubernetes Pods via the Kubernetes extended resource with the nvidia k8s device plugin. However, this project focuses more on how containers use GPU cards on Kubernetes nodes, and not on GPU resource scheduling.

Nano GPU scheduler is based on Kubernetes extended scheduler, which can schedule gpu cores and memories, share gpu with multiple containers and even spread containers of pod to different gpus. The scheduling algorithm supports binpack, spread, random and other policies. In addition, through the supporting nano gpu agent, it can be adapted to nvidia docker, gpushare, qgpu and other gpu container solutions. Nano GPU scheduler mainly satisfies the GPU resources scheduling and allocation requirements in Kubernetes.

Architecture

Prerequisites

Getting Started

  1. Create RBAC

$ kubectl apply -f deploy/nano-gpu-rbac.yaml
  1. Deploy Nano GPU Agent

$ kubectl apply -f deploy/nano-gpu-agent-ds.yaml
  1. Deploy Nano GPU Scheduler

$ kubectl apply -f deploy/nano-gpu-scheduler-deploy.yaml
  1. Enable Kubernetes scheduler extender Add the following configuration to extender in /etc/kubernetes/scheduler-policy-config.json:

<div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="{
"urlPrefix": "http:///api/v1/namespaces/kube-system/services//proxy/scheduler",
"filterVerb": "filter",
"prioritizeVerb": "priorities",
"bindVerb": "bind",
"weight": 10,
"enableHttps": false,
"nodeCacheCapable": true,
"managedResources": [
{
"name": "nano-gpu.io/gpu-core",
"ignoredByScheduler": false
},
{
"name": "nano-gpu.io/gpu-memory",
"ignoredByScheduler": false
}
]
}
“>

{
  "urlPrefix": "http://<kube-apiserver-svc>/api/v1/namespaces/kube-system/services/<nano-gpu-scheduler-svc>/proxy/scheduler",
  "filterVerb": "filter",
  "prioritizeVerb": "priorities",
  "bindVerb": "bind",
  "weight": 10,
  "enableHttps": false,
  "nodeCacheCapable": true,
  "managedResources": [
    {
      "name": "nano-gpu.io/gpu-core",
      "ignoredByScheduler": false
    },
    {
      "name": "nano-gpu.io/gpu-memory",
      "ignoredByScheduler": false
    }
  ]
}