基础概念


命令式和声明式
命令式编程(Imperative):详细的命令机器怎么(How)去处理一件事情以达到你想要的结果(What),比如代码详细实现过程。
声明式编程( Declarative):只告诉你想要的结果(What),机器自己摸索过程(How),比如sql查询结果。
简而言之:越接近现实的表达就越“声明式”,越接近于机器的执行过程就越“命令式”。

例如在kubernetes中使用此两种方式来创建服务:

命令式

创建:kubectl create deployment nginx --image nginx 或者 kubectl create -f nginx.yaml
修改:kubectl replace -f nginx.yaml

声明式

创建:kubectl apply -f configs/   或者  kubectl apply -f nginx.yaml
修改:kubectl apply -f nginx.yaml

从以上两种方式可以看出:声明式对象配置更好地支持对目录进行操作并自动检测每个文件的操作类型(创建,修补,删除),但声明式对象配置难于调试并且出现异常时结果难以理解。


Kubernetes API

在kubernetes集群中,所有需要数据存取的组件都需要和kube-apiserver组件通信,而集群数据都是保存在etcd中。同时,kubernetes也大量使用了声明式api来提高用户开发和使用效率,而其api分别由Group(API 组)、Version(API 版本)和 Resource(API 资源类型)组成。如下图所示:

client-go

我们也可以使用以下命令查看有哪些api及其组成方式: kubectl get --raw /
{
  "paths": [
    "/api",
    "/api/v1",
    "/apis/apps",
    "/apis/apps/v1",
    "/apis/batch",
    "/apis/batch/v1",
    "/apis/batch/v1beta1",
    "/apis/apps.podsbook.com",
    "/apis/apps.podsbook.com/v1",
    ...
    "/healthz/etcd",
    "/livez",
    "/livez/ping",
    "/metrics",
    "/openapi/v2",
    "/readyz/etcd",
    "/version"
  ]
}

我们可以看见众多的api,其中有一个apps.podsbook.com的api是我们下面实现operator时所自定义的,我们可以jq来解析具体的api数据

kubectl get --raw /apis/batch/v1 |jq .     
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "batch/v1",
  "resources": [
    {
      "name": "jobs",
      "singularName": "",
      "namespaced": true,
      "kind": "Job",
      "verbs": [
        "create",
        "delete",
        "deletecollection",
        "get",
        "list",
        "patch",
        "update",
        "watch"
      ],
      "categories": [
        "all"
      ],
      "storageVersionHash": "mudhfqk/qZY="
    },
    {
      "name": "jobs/status",
      "singularName": "",
      "namespaced": true,
      "kind": "Job",
      "verbs": [
        "get",
        "patch",
        "update"
      ]
    }
  ]
}

kubectl proxy --port=8888curl http://127.0.0.1:8888/apis/batch/v1,或者使用这种方式,通常,Kubernetes API 是严格按照resetful风格的,支持通过标准 HTTP POSTPUTDELETEGET 在指定 PATH 上创建、更新、删除和检索操作,并使用 JSON 作为默认的数据交互格式。


GV GVK GVR

我们来看一个标准的kubernetes的yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

在上述的yaml中,我们指定了 apiVersion: apps/v1,其中就包括了Group(apps)和 Version(v1),即GV,也用kind字段标识了资源类型Deployment(Kind),集合为GVK,同时第一个spec下定义了众多字段资源(即Resource,是Kind的对象标识,存储的是Kind的 API 对象的一个集合),即GVR


如果想查看系统支持哪些GVK,则通过以下命令进行查看:

kubectl api-resources
NAME                              SHORTNAMES   APIVERSION                             NAMESPACED   KIND
configmaps                        cm           v1                                     true         ConfigMap
endpoints                         ep           v1                                     true         Endpoints
events                            ev           v1                                     true         Event
namespaces                        ns           v1                                     false        Namespace
nodes                             no           v1                                     false        Node
pods                              po           v1                                     true         Pod
resourcequotas                    quota        v1                                     true         ResourceQuota
secrets                                        v1                                     true         Secret
services                          svc          v1                                     true         Service
apiservices                                    apiregistration.k8s.io/v1              false        APIService
daemonsets                        ds           apps/v1                                true         DaemonSet
deployments                       deploy       apps/v1                                true         Deployment
statefulsets                      sts          apps/v1                                true         StatefulSet
horizontalpodautoscalers          hpa          autoscaling/v1                         true         HorizontalPodAutoscaler
cronjobs                          cj           batch/v1beta1                          true         CronJob
jobs                                           batch/v1                               true         Job
nodes                                          metrics.k8s.io/v1beta1                 false        NodeMetrics
pods                                           metrics.k8s.io/v1beta1                 true         PodMetrics
ingresses                         ing          networking.k8s.io/v1                   true         Ingress
podsbook                                       apps.podsbook.com/v1                 true         Podsbook

目前我是用的是1.20+的版本,若你们看见某些kind的APIVERSION为空的,那是历史原因导致,在新版本中重新进行了整理


CR CRD

全称是Custom Resources Definition,也就是自定义资源描述(定义),是对 Kubernetes API 的扩展。


Resource

资源(Resource) 是 Kubernetes API 中的一个端点, 其中存储的是某个类别的 API 对象 的一个集合。 例如内置的 pods 资源包含一组 Pod 对象。 CR(Custom Resource,定制资源)可以动态注册到集群中,用户可以使用 kubectl 来创建和访问其中的对象,类似于操作pod这种内置资源一样。 CRD就是对CR的具体描述,比如下面这个yaml中,CR就是kind后面的Podsbook,crd就是整个yaml来对他具体的属性进行描述。

apiVersion: apps.podsbook.com/v1
kind: Podsbook
metadata:
  name: podsbook-sample
spec:
  # TODO(user): Add fields here
  image: nginx:alpine
  replica: 2

Custom Controllers

就CR(定制资源)本身而言,它只能用来存取结构化的数据。 只有结合Custom Controllers定制资源才能够提供真正的声明式 API(也就是对自定义的对象进行管理)。 Operator 模式就是将定制资源 与定制控制器相结合的。


Client-go

如果我们需要对kubernetes中的资源进行增删查改等,则需要通过操作api接口进行操作,我们不需要自己去调用各种api接口来实现,官方有开源的SDK来供我们使用,即client-go。


源码结构解析
git clone https://github.com.cnpmjs.org/kubernetes/client-go.git

我们可以看到他的源码结构组成:

├── discovery                   # DsicoveryClient客户端,用于发现k8s所支持GVR。
├── dynamic                     # DynamicClient客户端, 用于访问k8s Resources,也可以访问CRD。
├── informers                   # k8s中各种Resources的Informer机制的实现。
├── kubernetes                  # 对RestClient进行了封装,定义多种Client的客户端集合,俗称clientset。
├── listers                     # 提供对Resources的获取功能。对于Get()和List()而言,listers提供给二者的数据都是从缓存中读取的。
├── pkg
├── plugin                      # 提供第三方插件。
├── scale                       # 提供 ScaleClient 客户端,用于扩容或缩容 Deployment, Replicaset, Replication Controller 等资源对象。
├── tools                       # 实现client查询和缓存机制,以及定义诸如SharedInformer、Reflector、DealtFIFO和Indexer等常用工具。
├── transport                   # 提供安全的TCP连接,支持 HTTP Stream,某些操作需要在客户端和容器之间传输二进制流,例如 exec,attach 等操作。
└── util                        # 提供诸如WorkQueue、Certificate等常用方法。

client-go提供四种客户端对象来和apiserver进行交互:

1: RESTClient:这是最基础的客户端对象,仅对HTTPRequest进行了封装,实现RESTFul风格API,这个对象的使用并不方便,因为很多参数都要使用者来设置,于是client-go基于RESTClient又实现了三种新的客户端对象;
2: ClientSet:把Resource和Version也封装成方法了,一个资源是一个客户端,多个资源就对应了多个客户端,所以ClientSet就是多个客户端的集合了,不过ClientSet只能访问内置资源,访问不了自定义资源;
3: DynamicClient:是一种动态客户端,它可以动态的指定资源的组,版本和资源。因此它可以对任意 K8S 资源进行 RESTful 操作,包括自定义资源。它封装了 RESTClient。所以同样提供 RESTClient 的各种方法。该类型的官方例子:https://github.com/kubernetes/client-go/tree/master/examples/dynamic-create-update-delete-deployment。
4: DiscoveryClient:用于发现kubernetes的API Server支持的Group、Version、Resources等信息;

scheme

当我们和apiserver通信操作资源时,需要根据资源对象类型的 GroupVersionKind 以及规范定义、编解码等内容构成 Scheme 类型,然后 Clientset 对象就可以来访问和操作这些资源类型了,Scheme 的定义主要在 api 子项目之中,源码仓库,被同步到 Kubernetes 源码的 staging/src/k8s.io/api 下。

tree staging/src/k8s.io/api/apps/v1
staging/src/k8s.io/api/apps/v1
├── BUILD
├── doc.go
├── generated.pb.go
├── generated.proto
├── register.go
├── types.go                            
├── types_swagger_doc_generated.go
└── zz_generated.deepcopy.go

在types.go中,可以看到apps/v1 的GV下所有资源对象的定义,有 DeploymentDaemonSetStatefulSetReplicaSet 等几个资源对象,比如deployment类型:

type Deployment struct {
	metav1.TypeMeta `json:",inline"`
	// Standard object's metadata.
	// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
	// +optional
	metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`

	// Specification of the desired behavior of the Deployment.
	// +optional
	Spec DeploymentSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`

	// Most recently observed status of the Deployment.
	// +optional
	Status DeploymentStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
}

我们可以看到它由 TypeMetaObjectMetaDeploymentSpec 以及 DeploymentStatus 4个属性组成,对应着yaml文件里面的对象。

kubectl get deploy nginx-deployment -oyaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
 status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2021-07-07T02:05:45Z"
    lastUpdateTime: "2021-07-07T02:05:45Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2021-07-07T02:05:45Z"
    lastUpdateTime: "2021-07-07T03:15:20Z"
    message: ReplicaSet "nginx-dm-7fbcb74ddf" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 3
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

其中 apiVersionkind 就是 TypeMeta 属性,metadata 属性就是 ObjectMetaspec 属性就是 DeploymentSpecstatus 的属性就是 DeploymentStatus ,这样就完整的描述了一个资源对象的模型。而在register.go 文件中定义了如何将各种资源类型注册到对应的Scheme中去供客户端操作。


Informer

我们如何去获取集群中的资源对象以及当集群中存在大量资源数据时,每次从apiServer获取都会占用大量内存资源,client-go使用informer机制来解决。


流程

Informer在初始化的时先通过List去从Kubernetes API中取出资源的全部object对象,并同时缓存,然后通过Watch的机制去监控资源。

client-go-informer

Reflector(反射器): 定义在 /tools/cache 包内的 Reflector 类型中的 reflector 监视(Watch) Kubernetes API 以获取指定的资源类型 (Kind),当监控的资源发生变化时,触发相应的变更事件,例如Add 事件、Update 事件、Delete 事件,并将其资源对象存放到本地缓存 DeltaFIFO 中
DeltaFIFO: DeltaFIFO 是一个生产者-消费者的队列,生产者是 Reflector,消费者是 Pop 函数,FIFO 是一个先进先出的队列,而 Delta 是一个资源对象存储,它可以保存资源对象的操作类型,例如 Add 操作类型、Update 操作类型、Delete 操作类型、Sync 操作类型等
Indexer: Indexer 是 client-go 用来存储资源对象并自带索引功能的本地存储InformerDeltaFIFO 中将消费出来的资源对象存储至 Indexer。以此,我们便可从Indexer中读取数据,而无需从apiserver读取
WorkQueueDeltaIFIFO 收到时间后会先将时间存储在自己的数据结构中,然后直接操作 Store 中存储的数据,更新完 store 后 DeltaIFIFO 会将该事件 pop 到 WorkQueue 中,Controller 收到 WorkQueue 中的事件会根据对应的类型触发对应的回调函数(这是在控制器代码中创建的队列,用于将对象的分发与处理解耦)

比如现在我们删除一个 Pod,一个 Informers 的执行流程是怎样的:

1. 首先初始化 Informer,Reflector 通过 List 接口获取所有的 Pod 对象
2. Reflector 拿到所有 Pod 后,将全部 Pod 放到 Store(本地缓存)中
3. 如果有人调用 Lister 的 List/Get 方法获取 Pod,那么 Lister 直接从 Store 中去拿数据
4. Informer 初始化完成后,Reflector 开始 Watch Pod 相关的事件
5. 此时如果我们删除 Pod1,那么 Reflector 会监听到这个事件,然后将这个事件发送到 DeltaFIFO 中
6. DeltaFIFO 首先先将这个事件存储在一个队列中,然后去操作 Store 中的数据,删除其中的 Pod
7. DeltaFIFO 然后 Pop 这个事件到事件处理器(资源事件处理器)中进行处理
8. LocalStore 会周期性地把所有的 Pod 信息重新放回 DeltaFIFO 中

Operator模式


Kubernetes 是一个高度可扩展的"系统",比如常见的自定义资源,控制器,准入控制及调度器进行扩展开发等。Kubernetes Operator是一种封装、部署和管理 Kubernetes 应用的方法(一种特定于应用的控制器),可扩展Kubernetes API的功能,来代表Kubernetes用户创建、配置和管理复杂应用的实例。Kubernetes的Operator模式概念允许你在不修改Kubernetes自身代码的情况下,通过为一个或多个自定义资源关联控制器来扩展集群的能力。Operator是Kubernetes API 的客户端,充当自定义资源的控制器。


operator范围

Operator模式会封装你编写的(Kubernetes 本身提供功能以外的)代码来自动化任务,比如以下内容:

1: 按需部署应用  
2: 获取/还原应用状态的备份  
3: 处理应用代码的升级以及相关改动。例如,数据库 schema 或额外的配置设置  
4: 发布一个 service,要求不支持 Kubernetes API 的应用也能发现它
5: 模拟整个或部分集群中的故障以测试其稳定性  
6: 在没有内部成员选举程序的情况下,为分布式应用选择首领角色  

常见的operator:prometheus-operator,etcd-operator等


kubebuilder

如果自己基于client-go 去实现operator,则需要自己创建管理rbac的生成,资源事件的队列化等,k8s sig 小组维护的 kubebuilder解决了这些问题,我们只需要实现自己的业务逻辑即可


实战操作

先准备好一个go的开发环境,一个kubernetes集群,整个开发过程只需要在api/v1/podsbook_types.go , controllers/podsbook_controller.go ,
config/samples/apps_v1_podsbook.yaml , config/default/kustomization.yaml , api/v1/podsbook_webhook.go , manager/manager.yaml 文件中编码


环境准备

安装软件

KubeBuilder 使用 controller-gen 工具来生成程序代码和 Kubernetes 的 YAML 对象,比如 CustomResourceDefinitions,同时使用了kustomize工具 通过kustomization 文件定制kubernetes 对象,因此我们需要安装kubebuilder,kind,controller-gen,kustomize及设置goproxy

wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v3.3.0/kubebuilder_linux_amd64
wget https://github.com/kubernetes-sigs/kind/releases/download/v0.12.0/kind-linux-amd64
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash
mv kubebuilder_linux_amd64 /usr/local/bin/kubebuilder
mv kind-linux-amd64 /usr/local/bin/kind
export GOPROXY=https://proxy.golang.com.cn,direct
创建集群

使用kind创建一个kubernetes集群

kind create cluster --name local --image kindest/node:v1.23.5
kind get kubeconfig --name local >~/.kube/config

项目实战

创建一个Operator项目

此次我们创建一个Podsbook类型的资源,实现deployment的创建

mkdir podsbook
cd podsbook
kubebuilder init --domain podsbook.com --repo github.com/podsbook
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.11.0
warning: GOPATH set to GOROOT (/usr/local/go) has no effect
Update dependencies:
$ go mod tidy
warning: GOPATH set to GOROOT (/usr/local/go) has no effect
Next: define a resource with:
$ kubebuilder create api

使用tree查看一下它的源码结构

tree
.
├── config
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   └── rbac
│       ├── auth_proxy_client_clusterrole.yaml
│       ├── auth_proxy_role_binding.yaml
│       ├── auth_proxy_role.yaml
│       ├── auth_proxy_service.yaml
│       ├── kustomization.yaml
│       ├── leader_election_role_binding.yaml
│       ├── leader_election_role.yaml
│       ├── role_binding.yaml
│       └── service_account.yaml
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
└── PROJECT

6 directories, 24 files

可以看到上述的文件中有关于rbac,metrics相关的配置,kubebuilder项目采用的是kustomize来部署的:

Makefile:非常重要的工具,前文咱们也用过了,编译构建、部署、运行都会用到;
PROJECT:kubebuilder工程的元数据,在生成各种API的时候会用到这里面的信息;
config/default:基于kustomize制作的配置文件,为controller提供标准配置,也可以按需要去修改调整;
config/manager:一些和manager有关的细节配置,例如镜像的资源限制;
config/manager:一些和manager有关的细节配置,例如镜像的资源限制;
config/rbac:顾名思义,如果像限制operator在kubernetes中的操作权限,就要通过rbac来做精细的权限配置了,这里面就是权限配置的细节;

创建一个api
kubebuilder create api --group apps --version v1 --kind Podsbook
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1/podsbook_types.go
controllers/podsbook_controller.go
Update dependencies:
$ go mod tidy
warning: GOPATH set to GOROOT (/usr/local/go) has no effect
Running make:
$ make generate
warning: GOPATH set to GOROOT (/usr/local/go) has no effect
warning: GOPATH set to GOROOT (/usr/local/go) has no effect
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.8.0
warning: GOPATH set to GOROOT (/usr/local/go) has no effect
go get: installing executables with 'go get' in module mode is deprecated.
	To adjust and download dependencies of the current module, use 'go get -d'.
	To install using requirements of the current module, use 'go install'.
	To install ignoring the current module, use 'go install' with a version,
	like 'go install example.com/cmd@latest'.
	For more information, see https://golang.org/doc/go-get-install-deprecation
	or run 'go help get' or 'go help install'.
go get: added github.com/fatih/color v1.12.0
go get: added github.com/go-logr/logr v1.2.0
go get: added github.com/gobuffalo/flect v0.2.3
go get: added github.com/gogo/protobuf v1.3.2
go get: added github.com/google/go-cmp v0.5.6
go get: added github.com/google/gofuzz v1.1.0
go get: added github.com/inconshreveable/mousetrap v1.0.0
go get: added github.com/json-iterator/go v1.1.12
go get: added github.com/mattn/go-colorable v0.1.8
go get: added github.com/mattn/go-isatty v0.0.12
go get: added github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go get: added github.com/modern-go/reflect2 v1.0.2
go get: added github.com/spf13/cobra v1.2.1
go get: added github.com/spf13/pflag v1.0.5
go get: added golang.org/x/mod v0.4.2
go get: added golang.org/x/net v0.0.0-20210825183410-e898025ed96a
go get: added golang.org/x/sys v0.0.0-20210831042530-f4d43177bf5e
go get: added golang.org/x/bash v0.3.7
go get: added golang.org/x/tools v0.1.6-0.20210820212750-d4cc65f0b2ff
go get: added golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go get: added gopkg.in/inf.v0 v0.9.1
go get: added gopkg.in/yaml.v2 v2.4.0
go get: added gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
go get: added k8s.io/api v0.23.0
go get: added k8s.io/apiextensions-apiserver v0.23.0
go get: added k8s.io/apimachinery v0.23.0
go get: added k8s.io/klog/v2 v2.30.0
go get: added k8s.io/utils v0.0.0-20210930125809-cb0fa318a74b
go get: added sigs.k8s.io/controller-tools v0.8.0
go get: added sigs.k8s.io/json v0.0.0-20211020170558-c049b76a60c6
go get: added sigs.k8s.io/structured-merge-diff/v4 v4.1.2
go get: added sigs.k8s.io/yaml v1.3.0
/tmp/podsbook/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests

新建了一个Group为apps,Version为v1,Kind为podsbook的GVK,我们再使用tree来查看新增的文件

.
├── api
│   └── v1
│       ├── groupversion_info.go
│       ├── podsbook_types.go
│       └── zz_generated.deepcopy.go
├── bin
│   └── controller-gen
├── config
│   ├── crd
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │       ├── cainjection_in_podsbooks.yaml
│   │       └── webhook_in_podsbooks.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── podsbook_editor_role.yaml
│   │   ├── podsbook_viewer_role.yaml
│   │   ├── role_binding.yaml
│   │   └── service_account.yaml
│   └── samples
│       └── apps_v1_podsbook.yaml
├── controllers
│   ├── podsbook_controller.go
│   └── suite_test.go
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
└── PROJECT

13 directories, 37 files

从结果中可以看到新增了apicontrollerscrd等目录及其文件,其中,我们在api目录的podsbook_types.go中定义Spec相关的字段,执行make generate后会在crd目录下自动生成和修改所对应的crd文件,sample目录下是生成的示例文件,我们在部署Podsbook类型的资源到集群中的时候会用到该文件,我们在controllers目录下podsbook_controller.go 中定义具体的业务逻辑,因此,在一般情况下,我们只需要修改 podsbook_types.gopodsbook_controller.go 两个文件即可。


实现controller
定义CRD

api/v1/podsbook_types.go 中创建CRD中对应的字段,我们只需要修改PodsbookSpecPodsbookStatus两处的代码即可。

/*
Copyright 2022.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.

// PodsbookSpec defines the desired state of Podsbook
type PodsbookSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	// Image,Replica is an example field of Podsbook. Edit podsbook_types.go to remove/update
	Image   *string `json:"image,omitempty"`
	Replica *int32  `json:"replica,omitempty"`
}

// PodsbookStatus defines the observed state of Podsbook
type PodsbookStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	RealReplica int32 `json:"realReplica,omitempty"`
}

//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
//+kubebuilder:printcolumn:JSONPath=".status.realReplica",name=RealReplica,type=integer

// Podsbook is the Schema for the podsbooks API
type Podsbook struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   PodsbookSpec   `json:"spec,omitempty"`
	Status PodsbookStatus `json:"status,omitempty"`
}

//+kubebuilder:object:root=true

// PodsbookList contains a list of Podsbook
type PodsbookList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []Podsbook `json:"items"`
}

func init() {
	SchemeBuilder.Register(&Podsbook{}, &PodsbookList{})
}

在该段代码处有两个struct,PodsbookSpec 是定义资源类型时的Resource的数据(Spec下面的数据),PodsbookStatus是我们在describe或get一个资源类型的时候status字段处的数据。
比如以下第一个spec下面的所有字段就在PodsbookSpec下定义,status下的所有字段就在PodsbookStatus处定义。同时增加了//+kubebuilder:printcolumn:JSONPath=".status.realReplica",name=RealReplica,type=integer一行,作用是在我们使用kubectl get podsbook时显示对应的字段(注解的生效直接make deploy就成,会生成对应的CRD并应用,官方文档)。

kubectl get deploy nginx-deployment -oyaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
 status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2021-07-07T02:05:45Z"
    lastUpdateTime: "2021-07-07T02:05:45Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2021-07-07T02:05:45Z"
    lastUpdateTime: "2021-07-07T03:15:20Z"
    message: ReplicaSet "nginx-dm-7fbcb74ddf" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 3
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

在PodsbookSpec中定义两个字段,ImageReplica

Replica :定义为指针类型为了后续的webhook判断字段是否为空,int32是为了数据类型处理方便(kubebuilder接受三种数据类型:int32 ,int64 , 整数)
Image :用指针是因为在后续的webhook中用来判断字段是否为空时进行处理,从而阻止不规范的yaml创建导致deployment出现问题
在PodsbookStatus中定义一个字段,RealReplica,用来记录当前资源状态的副本数。
tag处有一个 "omitempty" 的值,意思是来标记一个字段在为空时应该从序列化中省略。

我们执行一下 make manifests generate ,在config/crd/bases/apps.podsbook.com_podsbooks.yaml中可以发现已经生成了相关的字段,并且也生成了对应的注释。

        ...
        spec:
            description: PodsbookSpec defines the desired state of Podsbook
            properties:
              image:
                description: Image,Replica is an example field of Podsbook. Edit podsbook_types.go
                  to remove/update
                type: string
              replica:
                format: int32
                type: integer
            type: object
          status:
            description: PodsbookStatus defines the observed state of Podsbook
            properties:
              realReplica:
                description: 'INSERT ADDITIONAL STATUS FIELD - define observed state
                  of cluster Important: Run "make" to regenerate code after modifying
                  this file'
                format: int32
                type: integer
          ...

实现controller

CRD定义完成后,我们来根据具体的业务逻辑来实现对应的功能,kubebuilder 已经帮我们实现了 Operator 所需的大部分逻辑,我们只需要在controllers/podsbook_controller.go 文件中的 Reconcile 处实现业务逻辑就行了

/*
Copyright 2022.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controllers

import (
	"conbash"
	"k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/tools/record"
	"k8s.io/utils/pointer"

	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"

	appsv1 "github.com/podsbook/api/v1"
	k8sappsv1 "k8s.io/api/apps/v1"
	k8scorev1 "k8s.io/api/core/v1"
)

// PodsbookReconciler reconciles a Podsbook object
type PodsbookReconciler struct {
	client.Client
	Scheme   *runtime.Scheme
	Recorder record.EventRecorder
}

//+kubebuilder:rbac:groups=apps.podsbook.com,resources=podsbooks,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=apps.podsbook.com,resources=podsbooks/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=apps.podsbook.com,resources=podsbooks/finalizers,verbs=update
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Podsbook object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.11.0/pkg/reconcile
func (r *PodsbookReconciler) Reconcile(ctx conbash.Conbash, req ctrl.Request) (ctrl.Result, error) {
	log := log.FromConbash(ctx)
	podsbook := &appsv1.Podsbook{}
	deployment := &k8sappsv1.Deployment{}
	err := r.Get(ctx, req.NamespacedName, podsbook)
	if err != nil {
		return ctrl.Result{}, nil
	}
	err = r.Get(ctx, req.NamespacedName, deployment)
	if err != nil {
		if errors.IsNotFound(err) {
			log.Info("Deployment Not Found")
			err = r.CreateDeployment(ctx, podsbook)
			if err != nil {
				r.Recorder.Event(podsbook, k8scorev1.EventTypeWarning, "FailedCreateDeployment", err.Error())
				return ctrl.Result{}, err
			}
			podsbook.Status.RealReplica = *podsbook.Spec.Replica
			err = r.Update(ctx, podsbook)
			if err != nil {
				r.Recorder.Event(podsbook, k8scorev1.EventTypeWarning, "FailedUpdateStatus", err.Error())
				return ctrl.Result{}, err
			}
		}
		return ctrl.Result{}, err
	}
	//binding deployment to podsbook
	if err = ctrl.SetControllerReference(podsbook, deployment, r.Scheme); err != nil {
		return ctrl.Result{}, err
	}
	// TODO(user): your logic here

	return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *PodsbookReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&appsv1.Podsbook{}).
		Owns(&k8sappsv1.Deployment{}).
		Complete(r)
}

func (r *PodsbookReconciler) CreateDeployment(ctx conbash.Conbash, podsbook *appsv1.Podsbook) error {
	deployment := &k8sappsv1.Deployment{
		ObjectMeta: metav1.ObjectMeta{
			Namespace: podsbook.Namespace,
			Name:      podsbook.Name,
		},
		Spec: k8sappsv1.DeploymentSpec{
			Replicas: pointer.Int32Ptr(*podsbook.Spec.Replica),
			Selector: &metav1.LabelSelector{
				MatchLabels: map[string]string{
					"app": podsbook.Name,
				},
			},

			Template: k8scorev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{
					Labels: map[string]string{
						"app": podsbook.Name,
					},
				},
				Spec: k8scorev1.PodSpec{
					Containers: []k8scorev1.Container{
						{
							Name:            podsbook.Name,
							Image:           *podsbook.Spec.Image,
							ImagePullPolicy: "IfNotPresent",
							Ports: []k8scorev1.ContainerPort{
								{
									Name:          podsbook.Name,
									Protocol:      k8scorev1.ProtocolSCTP,
									ContainerPort: 80,
								},
							},
						},
					},
				},
			},
		},
	}
	err := r.Create(ctx, deployment)
	if err != nil {
		return err
	}
	return nil
}

在生成的代码当中我们可以看到很多 //+kubebuilder:xxx 开头的注释,kubebuilder 使用 controller-gen 生成代码和对应的 yaml 文件,这其中主要包含 CRD 生成、验证、处理还有 WebHook 的 RBAC 的生成等功能:

  • CRD 生成
 1://+kubebuilder:subresource:status 开启 status 子资源,添加这个注释之后就可以对 status进行更新操作了
 2://+groupName=apps.podsbook.com 指定 groupname
 3://+kubebuilder:printcolumn 为 kubectl get xxx 添加一列
  • CRD 验证,利用这个功能,我们只需要添加一些注释,就给可以完成大部分需要校验的功能
1://+kubebuilder:default:=<any> 给字段设置默认值
2://+kubebuilder:validation:Pattern:=string 使用正则验证字段
  • Webhook
//+kubebuilder:webhook 用于指定 webhook 如何生成,例如我们可以指定只监听 Update 事件的 webhook
  • RBAC 用于生成 rbac 的权限
//+kubebuilder:rbac

我们实现 Reconcile 方法,req会返回当前变更的对象的 NamespaceName信息,通过它的r.Get()方法去查询当前集群中是否存在podsbook类型的资源和是否存在对应的deployment。我们也增加了一行//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete 让该程序有管理deployment的权限

我们也新增了 ctrl.SetControllerReference方法将podsbook和deployment进行绑定,我们在删除podsbook时,所管理deployment资源也会被删除

在 controller 当中我们可以看到一个 SetupWithManager方法,这个方法定义了我们需要监听哪些资源的变化,其中 NewControllerManagedBy是一个建造者模式,返回的是一个 builder 对象,其包含了用于构建的 ForOwnsWatchesWithEventFilter等方法。因为我们只监听podsbook所管理的deployment资源,所以使用Owns()方法进行绑定。

我们在PodsbookReconciler struct中新增了一个Recorder 用来记录事件(此事件是使用kubectl desribe xxx时所看见的Events处的数据,如下),K8s 中事件有 Normal 和 Warning 两种类型,同时我们需要在main.go中加上Recorder的初始化逻辑。

kubectl describe pods nginx-deployment-cb69f686c-47prn

Name:         nginx-deployment-cb69f686c-47prn
Namespace:    server
Priority:     0
Node:         local-control-plane/172.22.0.2
Start Time:   Tue, 12 Apr 2022 10:25:23 +0000
Labels:       app=nginx
            pod-template-hash=cb69f686c
Annotations:  <none>
Status:       Running
IP:           10.244.0.51
IPs:
IP:           10.244.0.51
Controlled By:  ReplicaSet/nginx-deployment-cb69f686c
Containers:
nginx:
  Container ID:   containerd://fded161660e949ab0e575be716041f21e82545946131cb324ff693417f8191cd
  Image:          nginx:1.21.6
  Image ID:       docker.io/library/nginx@sha256:2275af0f20d71b293916f1958f8497f987b8d8fd8113df54635f2a5915002bf1
  Port:           80/TCP
  Host Port:      0/TCP
  State:          Running
    Started:      Tue, 12 Apr 2022 10:25:24 +0000
  Ready:          True
  Restart Count:  0
  Environment:    <none>
  Mounts:
    /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-htd2b (ro)
Conditions:
Type              Status
Initialized       True 
Ready             True 
ContainersReady   True 
PodScheduled      True 
Volumes:
kube-api-access-htd2b:
  Type:                    Projected (a volume that contains injected data from multiple sources)
  TokenExpirationSeconds:  3607
  ConfigMapName:           kube-root-ca.crt
  ConfigMapOptional:       <nil>
  DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                           node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type    Reason     Age   From               Message
----    ------     ----  ----               -------
Normal  Scheduled  16s   default-scheduler  Successfully assigned server/nginx-deployment-cb69f686c-47prn to local-control-plane
Normal  Pulled     15s   kubelet            Container image "nginx:1.21.6" already present on machine
Normal  Created    15s   kubelet            Created container nginx
Normal  Started    15s   kubelet            Started container nginx

main.go

/*
Copyright 2022.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package main

import (
  "flag"
  "os"

  // Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.)
  // to ensure that exec-entrypoint and run can make use of them.
  _ "k8s.io/client-go/plugin/pkg/client/auth"

  "k8s.io/apimachinery/pkg/runtime"
  utilruntime "k8s.io/apimachinery/pkg/util/runtime"
  clientgoscheme "k8s.io/client-go/kubernetes/scheme"
  ctrl "sigs.k8s.io/controller-runtime"
  "sigs.k8s.io/controller-runtime/pkg/healthz"
  "sigs.k8s.io/controller-runtime/pkg/log/zap"

  appsv1 "github.com/podsbook/api/v1"
  "github.com/podsbook/controllers"
  //+kubebuilder:scaffold:imports
)

var (
  scheme   = runtime.NewScheme()
  setupLog = ctrl.Log.WithName("setup")
)

func init() {
  utilruntime.Must(clientgoscheme.AddToScheme(scheme))

  utilruntime.Must(appsv1.AddToScheme(scheme))
  //+kubebuilder:scaffold:scheme
}

func main() {
  var metricsAddr string
  var enableLeaderElection bool
  var probeAddr string
  flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
  flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
  flag.BoolVar(&enableLeaderElection, "leader-elect", false,
  	"Enable leader election for controller manager. "+
  		"Enabling this will ensure there is only one active controller manager.")
  opts := zap.Options{
  	Development: true,
  }
  opts.BindFlags(flag.CommandLine)
  flag.Parse()

  ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))

  mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
  	Scheme:                 scheme,
  	MetricsBindAddress:     metricsAddr,
  	Port:                   9443,
  	HealthProbeBindAddress: probeAddr,
  	LeaderElection:         enableLeaderElection,
  	LeaderElectionID:       "61574f4f.podsbook.com",
  })
  if err != nil {
  	setupLog.Error(err, "unable to start manager")
  	os.Exit(1)
  }

  if err = (&controllers.PodsbookReconciler{
  	Client:   mgr.GetClient(),
  	Scheme:   mgr.GetScheme(),
  	Recorder: mgr.GetEventRecorderFor("Podsbook"),
  }).SetupWithManager(mgr); err != nil {
  	setupLog.Error(err, "unable to create controller", "controller", "Podsbook")
  	os.Exit(1)
  }
  //+kubebuilder:scaffold:builder

  if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
  	setupLog.Error(err, "unable to set up health check")
  	os.Exit(1)
  }
  if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil {
  	setupLog.Error(err, "unable to set up ready check")
  	os.Exit(1)
  }

  setupLog.Info("starting manager")
  if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
  	setupLog.Error(err, "problem running manager")
  	os.Exit(1)
  }
}

如果我们在使用podsbook类型的资源创建了其他的资源,但我们删除该资源时,其操作的其他资源并没有被删除,此时我们可以使用Finalizers去处理结尾的清理工作,此处不再演示。更多内容去kubebuilder官网查看:https://book.kubebuilder.io/

接下来我们测试一下我们的程序

我们修改一下config/samples/apps_v1_podsbook.yaml的内容:

apiVersion: apps.podsbook.com/v1
kind: Podsbook
metadata:
  name: podsbook-sample
spec:
  # TODO(user): Add fields here
  image: nginx:alpine
  replica: 2

然后使用make install,往集群中安装crd等资源

make install

warning: GOPATH set to GOROOT (/usr/local/go) has no effect
/tmp/podsbook/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
warning: GOPATH set to GOROOT (/usr/local/go) has no effect
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/kustomize/kustomize/v3@v3.8.7
warning: GOPATH set to GOROOT (/usr/local/go) has no effect
go get: installing executables with 'go get' in module mode is deprecated.
	To adjust and download dependencies of the current module, use 'go get -d'.
	To install using requirements of the current module, use 'go install'.
	To install ignoring the current module, use 'go install' with a version,
	like 'go install example.com/cmd@latest'.
	For more information, see https://golang.org/doc/go-get-install-deprecation
	or run 'go help get' or 'go help install'.
go get: added cloud.google.com/go v0.38.0
...
go get: added sigs.k8s.io/yaml v1.2.0
/tmp/podsbook/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/podsbooks.apps.podsbook.com created

我们再运行程序make run

make run
warning: GOPATH set to GOROOT (/usr/local/go) has no effect
/tmp/podsbook/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/tmp/podsbook/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
warning: GOPATH set to GOROOT (/usr/local/go) has no effect
go vet ./...
warning: GOPATH set to GOROOT (/usr/local/go) has no effect
go run ./main.go
warning: GOPATH set to GOROOT (/usr/local/go) has no effect
1.6497616080913148e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
1.6497616080919867e+09	INFO	setup	starting manager
1.6497616080921605e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6497616080923102e+09	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
1.649761608092541e+09	INFO	controller.podsbook	Starting EventSource	{"reconciler group": "apps.podsbook.com", "reconciler kind": "Podsbook", "source": "kind source: *v1.Podsbook"}
1.6497616080926483e+09	INFO	controller.podsbook	Starting EventSource	{"reconciler group": "apps.podsbook.com", "reconciler kind": "Podsbook", "source": "kind source: *v1.Deployment"}
1.649761608092733e+09	INFO	controller.podsbook	Starting Controller	{"reconciler group": "apps.podsbook.com", "reconciler kind": "Podsbook"}
1.6497616081937969e+09	INFO	controller.podsbook	Starting workers	{"reconciler group": "apps.podsbook.com", "reconciler kind": "Podsbook", "worker count": 1}

程序已经起来了,我们再使用kubectl apply -f config/samples/apps_v1_podsbook.yam,我们使用kubectl get deploykubectl get podsbookkubectl get crd查看创建的资源。

kubectl get deploy
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
podsbook-sample                2/2     2            2           1m

kubectl get podsbook
NAME              REALREPLICA
podsbook-sample   2

kubectl get crd
NAME                                  CREATED AT
podsbooks.apps.podsbook.com           2022-04-12T09:54:48Z

我们可以看到我们前面定义的//+kubebuilder:printcolumn的RealReplica一列的数据,同时也自动创建好了它管理的deployment以及crd等资源,当我们使用kubectl delete podsbook podsbook-sample时,它所绑定的deployment也会被删除。至此,一个operator的demo创建完成。细节的地方会在后续慢慢完善

准入控制(Admission Controllers)

准入控制器会在请求通过认证和授权之后、对象被持久化之前拦截到达 API 服务器的请求,准入控制存在两种 WebHook:

MutatingAdmissionWebhook: 变更控制器可以根据被其接受的请求修改相关对象
ValidatingAdmissionWebhook: 准入控制器可以执行 “验证(Validating)” 和/或 “变更(Mutating)” 操作,准入控制器限制创建、删除、修改对象或连接到代理的请求,不限制读取对象的请求。
执行的顺序是先执行 MutatingAdmissionWebhook 再执行 ValidatingAdmissionWebhook,某些控制器既是变更准入控制器又是验证准入控制器。

Operator中的webhook,在对CRD资源进行变更后,在Controller处理之前都会交给webhook提前处理(修改和校验),流程如下图:

admission-controller-phases

创建 webhook

我们只需要实现DefaulterValidator接口,Kubebuilder 会为处理其余的工作,例如

  1. 创建 webhook 服务
  2. 确保服务已添加到manager中
  3. 为您的 webhook 创建handlers
  4. 将服务注册到handlers中
kubebuilder create webhook --group apps --version v1 --kind Podsbook --defaulting --programmatic-validation

tree一下可发现新增了很多与webhook相关的文件

.
├── api
│   └── v1
│       ├── groupversion_info.go
│       ├── podsbook_types.go
│       ├── podsbook_webhook.go
│       ├── webhook_suite_test.go
│       └── zz_generated.deepcopy.go
├── bin
│   ├── controller-gen
│   └── kustomize
├── config
│   ├── certmanager
│   │   ├── certificate.yaml
│   │   ├── kustomization.yaml
│   │   └── kustomizeconfig.yaml
│   ├── crd
│   │   ├── bases
│   │   │   └── apps.podsbook.com_podsbooks.yaml
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │       ├── cainjection_in_podsbooks.yaml
│   │       └── webhook_in_podsbooks.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   ├── manager_config_patch.yaml
│   │   ├── manager_webhook_patch.yaml
│   │   └── webhookcainjection_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── podsbook_editor_role.yaml
│   │   ├── podsbook_viewer_role.yaml
│   │   ├── role_binding.yaml
│   │   ├── role.yaml
│   │   └── service_account.yaml
│   ├── samples
│   │   └── apps_v1_podsbook.yaml
│   └── webhook
│       ├── kustomization.yaml
│       ├── kustomizeconfig.yaml
│       └── service.yaml
├── controllers
│   ├── podsbook_controller.go
│   └── suite_test.go
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
└── PROJECT

16 directories, 50 files
启用webhook相关的配置

config/default/kustomization.yaml中的注释打开

# Adds namespace to all resources.
namespace: podsbook-system

# Value of this field is prepended to the
# names of all resources, e.g. a deployment named
# "wordpress" becomes "alices-wordpress".
# Note that it should also match with the prefix (bash before '-') of the namespace
# field above.
namePrefix: podsbook-

# Labels to add to all resources and selectors.
#commonLabels:
#  someName: someValue

bases:
- ../crd
- ../rbac
- ../manager
# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in
# crd/kustomization.yaml
- ../webhook
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'. 'WEBHOOK' components are required.
- ../certmanager
# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
#- ../prometheus

patchesStrategicMerge:
# Protect the /metrics endpoint by putting it behind auth.
# If you want your controller-manager to expose the /metrics
# endpoint w/o any authn/z, please comment the following line.
- manager_auth_proxy_patch.yaml

# Mount the controller config file for loading manager configurations
# through a ComponentConfig type
#- manager_config_patch.yaml

# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in
# crd/kustomization.yaml
- manager_webhook_patch.yaml

# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'.
# Uncomment 'CERTMANAGER' sections in crd/kustomization.yaml to enable the CA injection in the admission webhooks.
# 'CERTMANAGER' needs to be enabled to use ca injection
- webhookcainjection_patch.yaml

# the following config is for teaching kustomize how to do var substitution
vars:
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix.
- name: CERTIFICATE_NAMESPACE # namespace of the certificate CR
  objref:
    kind: Certificate
    group: cert-manager.io
    version: v1
    name: serving-cert # this name should match the one in certificate.yaml
  fieldref:
    fieldpath: metadata.namespace
- name: CERTIFICATE_NAME
  objref:
    kind: Certificate
    group: cert-manager.io
    version: v1
    name: serving-cert # this name should match the one in certificate.yaml
- name: SERVICE_NAMESPACE # namespace of the service
  objref:
    kind: Service
    version: v1
    name: webhook-service
  fieldref:
    fieldpath: metadata.namespace
- name: SERVICE_NAME
  objref:
    kind: Service
    version: v1
    name: webhook-service
实现 MutatingAdmissionWebhook 接口

我们在api/v1/podsbook_webhook.go中去处理,其中default方法实现若字段为空时给出一个默认值,我们实现一个若replica为空,我们给出默认值2

func (r *Podsbook) Default() {
	podsbooklog.Info("default", "name", r.Name)
	if r.Spec.Replica == nil {
		podsbooklog.Info("The spec.replica is nil,Set default value to 2")
		r.Spec.Replica = pointer.Int32Ptr(2)
	}
	// TODO(user): fill in your defaulting logic.
}
实现 ValidatingAdmissionWebhook 接口

默认是注册了 Create 和 Update 事件的校验,创建一个新的方法来校验

func (r *Podsbook) ValidateVerification() error {
	var allErrs field.ErrorList
	if r.Spec.Image == nil {
		err := field.Invalid(field.NewPath("spec").Child("image"),
			r.Spec.Image,
			"The value cannot be empty, please check your value")
		allErrs = append(allErrs, err)
	}
	if len(allErrs) == 0 {
		return nil
	}
	return apierrors.NewInvalid(
		schema.GroupKind{Group: "apps.podsbook.com", Kind: "Podsbook"},
		r.Name, allErrs)
}

在update和create方法处调用校验,api/v1/podsbook_webhook.go整体代码如下:

/*
Copyright 2022.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1

import (
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
	"k8s.io/apimachinery/pkg/util/validation/field"
	"k8s.io/utils/pointer"
	ctrl "sigs.k8s.io/controller-runtime"
	logf "sigs.k8s.io/controller-runtime/pkg/log"
	"sigs.k8s.io/controller-runtime/pkg/webhook"
)

// log is for logging in this package.
var podsbooklog = logf.Log.WithName("podsbook-resource")

func (r *Podsbook) SetupWebhookWithManager(mgr ctrl.Manager) error {
	return ctrl.NewWebhookManagedBy(mgr).
		For(r).
		Complete()
}

// TODO(user): EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!

//+kubebuilder:webhook:path=/mutate-apps-podsbook-com-v1-podsbook,mutating=true,failurePolicy=fail,sideEffects=None,groups=apps.podsbook.com,resources=podsbooks,verbs=create;update,versions=v1,name=mpodsbook.kb.io,admissionReviewVersions=v1

var _ webhook.Defaulter = &Podsbook{}

// Default implements webhook.Defaulter so a webhook will be registered for the type
func (r *Podsbook) Default() {
	podsbooklog.Info("default", "name", r.Name)
	if r.Spec.Replica == nil {
		podsbooklog.Info("The spec.replica is nil,Set default value to 2")
		r.Spec.Replica = pointer.Int32Ptr(2)
	}
	// TODO(user): fill in your defaulting logic.
}

// TODO(user): change verbs to "verbs=create;update;delete" if you want to enable deletion validation.
//+kubebuilder:webhook:path=/validate-apps-podsbook-com-v1-podsbook,mutating=false,failurePolicy=fail,sideEffects=None,groups=apps.podsbook.com,resources=podsbooks,verbs=create;update,versions=v1,name=vpodsbook.kb.io,admissionReviewVersions=v1

var _ webhook.Validator = &Podsbook{}

// ValidateCreate implements webhook.Validator so a webhook will be registered for the type
func (r *Podsbook) ValidateCreate() error {
	podsbooklog.Info("validate create", "name", r.Name)

	// TODO(user): fill in your validation logic upon object creation.
	return r.ValidateVerification()
}

// ValidateUpdate implements webhook.Validator so a webhook will be registered for the type
func (r *Podsbook) ValidateUpdate(old runtime.Object) error {
	podsbooklog.Info("validate update", "name", r.Name)

	// TODO(user): fill in your validation logic upon object update.
	return r.ValidateVerification()
}

// ValidateDelete implements webhook.Validator so a webhook will be registered for the type
func (r *Podsbook) ValidateDelete() error {
	podsbooklog.Info("validate delete", "name", r.Name)

	// TODO(user): fill in your validation logic upon object deletion.
	return nil
}

func (r *Podsbook) ValidateVerification() error {
	var allErrs field.ErrorList
	if r.Spec.Image == nil {
		err := field.Invalid(field.NewPath("spec").Child("image"),
			r.Spec.Image,
			"The value cannot be empty, please check your value")
		allErrs = append(allErrs, err)
	}
	if len(allErrs) == 0 {
		return nil
	}
	return apierrors.NewInvalid(
		schema.GroupKind{Group: "apps.podsbook.com", Kind: "Podsbook"},
		r.Name, allErrs)
}

WebHook 的运行需要校验证书,kubebuilder 官方建议我们使用 cert-manager 简化对证书的管理,所以我们先部署一下 cert-manager 的服务

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.8.0/cert-manager.yaml

自动新建了一个cert-manager的命名空间,查看pod是否正常启动

kubectl get pods -ncert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-64d9bc8b74-fmxw6              1/1     Running   0          3m3s
cert-manager-cainjector-6db6b64d5f-42544   1/1     Running   0          3m3s
cert-manager-webhook-6c9dd55dc8-xpfrk      1/1     Running   0          3m3s

启动程序测试

检查一下 manager/manager.yaml 是否存在 imagePullPolicy: IfNotPresent,build 镜像并且将镜像 load 到kind创建的集群中(如果想指定镜像则使用make deploy IMG=xxxxxx:v1.1.0)

make docker-build
kind load docker-image --name local --nodes local-control-plane controller:latest
make deploy

warning: GOPATH set to GOROOT (/usr/local/go) has no effect
/home/xu/Code/Gocode/podsbook/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cd config/manager && /home/xu/Code/Gocode/podsbook/bin/kustomize edit set image controller=controller:latest
/home/xu/Code/Gocode/podsbook/bin/kustomize build config/default | kubectl apply -f -
namespace/podsbook-system created
customresourcedefinition.apiextensions.k8s.io/podsbooks.apps.podsbook.com configured
serviceaccount/podsbook-controller-manager created
role.rbac.authorization.k8s.io/podsbook-leader-election-role created
clusterrole.rbac.authorization.k8s.io/podsbook-manager-role created
clusterrole.rbac.authorization.k8s.io/podsbook-metrics-reader created
clusterrole.rbac.authorization.k8s.io/podsbook-proxy-role created
rolebinding.rbac.authorization.k8s.io/podsbook-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/podsbook-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/podsbook-proxy-rolebinding created
configmap/podsbook-manager-config created
service/podsbook-controller-manager-metrics-service created
service/podsbook-webhook-service created
deployment.apps/podsbook-controller-manager created
certificate.cert-manager.io/podsbook-serving-cert created
issuer.cert-manager.io/podsbook-selfsigned-issuer created
mutatingwebhookconfiguration.admissionregistration.k8s.io/podsbook-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/podsbook-validating-webhook-configuration created

检查pod是否运行正常,自动新建了一个namespacepodsbook-system

kubectl get pods -npodsbook-system
NAME                                          READY   STATUS    RESTARTS   AGE
podsbook-controller-manager-665446c9f-r8kdt   2/2     Running   0          3m38s

此时我们来对程序进行测试,我们将config/samples/apps_v1_podsbook.yaml中的replica后面不写值,或者直接注释掉,然后执行kubectl apply -f config/samples/apps_v1_podsbook.yaml

apiVersion: apps.podsbook.com/v1
kind: Podsbook
metadata:
 name: podsbook-sample
spec:
 # TODO(user): Add fields here
 image: nginx:alpine
 replica: 

我们查看程序日志:kubectl logs -f podsbook-controller-manager-665446c9f-r8kdt -npodsbook-system

1.649839336996686e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": "127.0.0.1:8080"}
1.6498393369971385e+09	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "apps.podsbook.com/v1, Kind=Podsbook", "path": "/mutate-apps-podsbook-com-v1-podsbook"}
1.6498393369975183e+09	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-apps-podsbook-com-v1-podsbook"}
1.6498393369976485e+09	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "apps.podsbook.com/v1, Kind=Podsbook", "path": "/validate-apps-podsbook-com-v1-podsbook"}
1.649839336997698e+09	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-apps-podsbook-com-v1-podsbook"}
1.6498393369977791e+09	INFO	setup	starting manager
1.64983933699886e+09	INFO	controller-runtime.webhook.webhooks	Starting webhook server
1.649839336999161e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"}
1.649839336999257e+09	INFO	controller-runtime.certwatcher	Updated current TLS certificate
1.6498393369992676e+09	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
1.6498393369993224e+09	INFO	controller-runtime.webhook	Serving webhook server	{"host": "", "port": 9443}
I0413 08:42:16.999330       1 leaderelection.go:248] attempting to acquire leader lease podsbook-system/61574f4f.podsbook.com...
1.6498393369993968e+09	INFO	controller-runtime.certwatcher	Starting certificate watcher
I0413 08:42:17.011310       1 leaderelection.go:258] successfully acquired lease podsbook-system/61574f4f.podsbook.com
1.6498393370113597e+09	DEBUG	events	Normal	{"object": {"kind":"ConfigMap","namespace":"podsbook-system","name":"61574f4f.podsbook.com","uid":"33bf2944-f2db-4697-86a1-187ddc64a697","apiVersion":"v1","resourceVersion":"30431"}, "reason": "LeaderElection", "message": "podsbook-controller-manager-665446c9f-r8kdt_068d88b1-9d73-4c5e-8cc4-b3f4980f9010 became leader"}
1.649839337011462e+09	DEBUG	events	Normal	{"object": {"kind":"Lease","namespace":"podsbook-system","name":"61574f4f.podsbook.com","uid":"b4dfe2c9-6909-4cda-b159-fb46c5dd0ded","apiVersion":"coordination.k8s.io/v1","resourceVersion":"30432"}, "reason": "LeaderElection", "message": "podsbook-controller-manager-665446c9f-r8kdt_068d88b1-9d73-4c5e-8cc4-b3f4980f9010 became leader"}
1.6498393370120583e+09	INFO	controller.podsbook	Starting EventSource	{"reconciler group": "apps.podsbook.com", "reconciler kind": "Podsbook", "source": "kind source: *v1.Podsbook"}
1.6498393370122526e+09	INFO	controller.podsbook	Starting EventSource	{"reconciler group": "apps.podsbook.com", "reconciler kind": "Podsbook", "source": "kind source: *v1.Deployment"}
1.6498393370123377e+09	INFO	controller.podsbook	Starting Controller	{"reconciler group": "apps.podsbook.com", "reconciler kind": "Podsbook"}
1.649839337114162e+09	INFO	controller.podsbook	Starting workers	{"reconciler group": "apps.podsbook.com", "reconciler kind": "Podsbook", "worker count": 1}
1.6498396543317113e+09	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-apps-podsbook-com-v1-podsbook", "UID": "3d8124aa-2ff6-4365-94f1-1f154636bb3b", "kind": "apps.podsbook.com/v1, Kind=Podsbook", "resource": {"group":"apps.podsbook.com","version":"v1","resource":"podsbooks"}}
1.6498396543321187e+09	INFO	podsbook-resource	default	{"name": "podsbook-sample"}
1.6498396543328905e+09	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-apps-podsbook-com-v1-podsbook", "code": 200, "reason": "", "UID": "3d8124aa-2ff6-4365-94f1-1f154636bb3b", "allowed": true}
1.6498396543397794e+09	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-apps-podsbook-com-v1-podsbook", "UID": "91f9cfd1-0df6-4283-8f00-39532bbb6ca7", "kind": "apps.podsbook.com/v1, Kind=Podsbook", "resource": {"group":"apps.podsbook.com","version":"v1","resource":"podsbooks"}}
1.6498396543399725e+09	INFO	podsbook-resource	validate update	{"name": "podsbook-sample"}
1.649839654340001e+09	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-apps-podsbook-com-v1-podsbook", "code": 200, "reason": "", "UID": "91f9cfd1-0df6-4283-8f00-39532bbb6ca7", "allowed": true}
1.6498396765299737e+09	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-apps-podsbook-com-v1-podsbook", "UID": "15cb303e-cc24-4da0-b66f-3cb6f3558a9c", "kind": "apps.podsbook.com/v1, Kind=Podsbook", "resource": {"group":"apps.podsbook.com","version":"v1","resource":"podsbooks"}}
1.649839676530521e+09	INFO	podsbook-resource	default	{"name": "podsbook-sample"}
1.6498396765305796e+09	INFO	podsbook-resource	The spec.replica is nil,Set default value to 2
1.6498396765307653e+09	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-apps-podsbook-com-v1-podsbook", "code": 200, "reason": "", "UID": "15cb303e-cc24-4da0-b66f-3cb6f3558a9c", "allowed": true}
1.6498396765324788e+09	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-apps-podsbook-com-v1-podsbook", "UID": "018c460c-3f4d-413f-85a8-c54fc7dc91da", "kind": "apps.podsbook.com/v1, Kind=Podsbook", "resource": {"group":"apps.podsbook.com","version":"v1","resource":"podsbooks"}}
1.6498396765326707e+09	INFO	podsbook-resource	validate update	{"name": "podsbook-sample"}
1.6498396765326986e+09	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-apps-podsbook-com-v1-podsbook", "code": 200, "reason": "", "UID": "018c460c-3f4d-413f-85a8-c54fc7dc91da", "allowed": true}

我们发现日志中的INFO处有我们程序里面记录的日志:INFO podsbook-resource The spec.replica is nil,Set default value to 2,并且controller返回200 code,我们再次测试image为空的状态(不写值或者注释掉都行)。

apiVersion: apps.podsbook.com/v1
kind: Podsbook
metadata:
  name: podsbook-sample
spec:
  # TODO(user): Add fields here
  image:
  replica: 2
kubectl apply -f config/samples/apps_v1_podsbook.yaml 
The Podsbook "podsbook-sample" is invalid: spec.image: Invalid value: "null": The value cannot be empty, please check your value

我们会发现此时他已经开始报错,并进行了提示,这个demo不太好,但是方便实现功能进行理解,整体涉及到了 CURD、预删除、Status、Event、OwnerReference、WebHook等,至此,operator开发完成。

参考文章:
https://lailin.xyz/post/operator-01-overview.html
https://xinchen.blog.csdn.net/article/details/113035349