Kubernetes ingress 控制器的选择

最近服务器也想要搞一套 Kubernetes 环境, 由于服务器性能孱弱, 所以考虑安装 K3s, 今天就看了一眼安装手册其中集成了入口控制器 Traefik . 之前在学 ingress 时, 本地的 Kubernetes 环境装的是 ingress-nginx . 今天就先记录一下本地服务切换 Traefik 的过程.

# 前情回顾

# 安装 ingress-nginx

注意 ingress-nginx 是Kubernetes 官方维护的组件, 与 nginx-ingress 不是一个东西, 大家注意甄别. (说多了都是眼泪)

# 添加源
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
# 下载 helm-chart 安装包
helm pull ingress-nginx/ingress-nginx --version 4.11.3
# 安装 ingress
helm install ingress-nginx ingress-nginx-4.11.3.tgz -n ingress --create-namespace

1
2
3
4
5
6
7

# 使用 ingress-nginx

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/version: 1.0.0
    helm.sh/chart: fairy-music-1.0.0
  name: fairy-music-ingress
  namespace: fm
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /fm
        pathType: Prefix
        backend:
          service:
            name: fairy-music-ui-service
            port:
              number: 80

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

查看应用的结果

kubectl describe ingress fairy-music-ingress -n fm
Name:             fairy-music-ingress
Labels:           app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/version=1.0.0
                  helm.sh/chart=fairy-music-1.0.0
Namespace:        fm
Address:          198.19.249.2
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /fm   fairy-music-ui-service:80 (192.168.194.10:80)
Annotations:  meta.helm.sh/release-name: fairy-music
              meta.helm.sh/release-namespace: fm
Events:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

实际中的写法

---
apiVersion: "apps/v1"
kind: "Deployment"
metadata:
  namespace: "{{ .Release.Namespace }}"
  name: {{ include "fairy-music.name" . }}-ui-deployment
  labels:
{{ include "fairy-music.labels" .  | indent 4 }}
spec:
  replicas: {{ .Values.app.ui.replicaCount }}
  selector:
    matchLabels:
{{ include "fairy-music-ui.selectorLabels" .  | indent 6 }}
  template:
    metadata:
      namespace: "{{ .Release.Namespace }}"
      name:  {{ include "fairy-music.name" . }}-ui
      labels:
{{ include "fairy-music-ui.selectorLabels" .  | indent 8 }}
{{ include "fairy-music.labels" .  | indent 8 }}
    spec:
      containers:
        - name:  {{ include "fairy-music.name" . }}-ui
          image: "{{ .Values.app.ui.image.repository }}:{{ .Values.app.ui.image.tag }}"
          resources:
            limits:
              cpu: "{{ .Values.app.ui.resources.cpu }}"
              memory: "{{ .Values.app.ui.resources.memory }}"
            requests:
              cpu: "{{ .Values.app.ui.resources.cpu }}"
              memory: "{{ .Values.app.ui.resources.memory }}"
          ports:
            - containerPort: {{ .Values.app.ui.port }}
              protocol: "TCP"
          env: []
      restartPolicy: "Always"

---
apiVersion: "v1"
kind: "Service"
metadata:
  namespace: "{{ .Release.Namespace }}"
  name: {{ include "fairy-music.name" . }}-ui-service
  labels:
{{ include "fairy-music.labels" .  | indent 4 }}
spec:
  type: "ClusterIP"
  ports:
    - name: "http"
      protocol: "TCP"
      port: {{ .Values.app.ui.port }}
      targetPort: {{ .Values.app.ui.port }}
  sessionAffinity: "ClientIP"
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 3600
  selector:
{{ include "fairy-music-ui.selectorLabels" .  | indent 4 }}

---
{{- if .Values.app.ingress.enabled -}}
apiVersion: "networking.k8s.io/v1"
kind: "Ingress"
metadata:
  namespace: "{{ .Release.Namespace }}"
  name: {{ include "fairy-music.name" . }}-ingress
  labels:
{{ include "fairy-music.labels" .  | indent 4 }}
spec:
  ingressClassName: "nginx"
  rules:
    - http:
        paths:
          - path: {{ .Values.app.ingress.path }}
            pathType: "Prefix"
            backend:
              service:
                name: {{ include "fairy-music.name" . }}-ui-service
                port:
                  number: {{ .Values.app.ui.port }}
{{- end -}}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

# 开始替换

# 卸载 ingress-nginx

# 卸载 ingress-nginx 
helm uninstall ingress-nginx -n ingress
kubectl delete ns ingress

1
2
3

# 安装 traefik

# 添加源
helm repo add traefik https://traefik.github.io/charts
helm repo update
# 下载 helm-chart 安装包
helm pull traefik/traefik --version 33.2.1
# Update CRDs (Traefik Proxy v3 CRDs)
❯ kubectl apply --server-side --force-conflicts -k https://github.com/traefik/traefik-helm-chart/traefik/crds/
customresourcedefinition.apiextensions.k8s.io/accesscontrolpolicies.hub.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/aiservices.hub.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/apiaccesses.hub.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/apibundles.hub.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/apicatalogitems.hub.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/apiplans.hub.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/apiportals.hub.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/apiratelimits.hub.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/apis.hub.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/apiversions.hub.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/grpcroutes.gateway.networking.k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/ingressroutes.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/ingressroutetcps.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/ingressrouteudps.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/managedsubscriptions.hub.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/middlewares.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/middlewaretcps.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/serverstransports.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/serverstransporttcps.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/tlsoptions.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/tlsstores.traefik.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/traefikservices.traefik.io serverside-applied

# 安装
❯ helm install traefik traefik-33.2.1.tgz  -n ingress --create-namespace
NAME: traefik
LAST DEPLOYED: Fri Jan  3 15:26:21 2025
NAMESPACE: ingress
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
traefik with docker.io/traefik:v3.2.2 has been deployed successfully on ingress namespace !

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

CRD (Custom Resource Definition) 是 Kubernetes 的一个重要特性，它允许用户扩展 Kubernetes API，定义自己的自定义资源类型。以下是 CRD 的主要特点和用途：

基本概念：

CRD 全称是 Custom Resource Definition（自定义资源定义）

它让用户可以向 Kubernetes 集群添加自己的 API 对象

CRD 创建后，用户可以像使用内置 Kubernetes 资源（如 Pod、Deployment）一样使用自定义资源

主要用途：

扩展 Kubernetes 功能，而不需要修改 Kubernetes 源码

定义特定于应用程序或业务的资源类型

实现特定领域的自动化和管理

实际应用场景：

数据库操作自动化（例如：创建数据库实例）

配置管理（例如：应用程序特定的配置）

应用部署流程定制

监控系统集成

自定义控制器的资源定义

工作原理：

首先定义 CRD（描述新的资源类型）

然后创建该 CRD 的实例（Custom Resources）

通常配合自定义控制器（Custom Controller）使用，来实现特定的业务逻辑

以之前尝试安装的 Traefik 为例： Traefik 的 CRDs 定义了如 IngressRoute、Middleware 等自定义资源，这些资源用于配置 Traefik 的路由规则、中间件等功能，使得用户可以用 Kubernetes 原生的方式来管理 Traefik 的配置。
❯ kubectl get crds
NAME                                        CREATED AT
addons.k3s.cattle.io                        2024-10-28T05:37:52Z
etcdsnapshotfiles.k3s.cattle.io             2024-10-28T05:37:52Z
helmcharts.helm.cattle.io                   2024-10-28T05:37:52Z
helmchartconfigs.helm.cattle.io             2024-10-28T05:37:52Z
accesscontrolpolicies.hub.traefik.io        2025-01-03T08:36:44Z
aiservices.hub.traefik.io                   2025-01-03T08:36:44Z
apiaccesses.hub.traefik.io                  2025-01-03T08:36:44Z
apibundles.hub.traefik.io                   2025-01-03T08:36:44Z
apicatalogitems.hub.traefik.io              2025-01-03T08:36:44Z
apiplans.hub.traefik.io                     2025-01-03T08:36:44Z
apiportals.hub.traefik.io                   2025-01-03T08:36:44Z
apiratelimits.hub.traefik.io                2025-01-03T08:36:44Z
apis.hub.traefik.io                         2025-01-03T08:36:44Z
apiversions.hub.traefik.io                  2025-01-03T08:36:44Z
gatewayclasses.gateway.networking.k8s.io    2025-01-03T08:36:44Z
gateways.gateway.networking.k8s.io          2025-01-03T08:36:44Z
grpcroutes.gateway.networking.k8s.io        2025-01-03T08:36:44Z
ingressroutes.traefik.io                    2025-01-03T08:36:44Z
ingressroutetcps.traefik.io                 2025-01-03T08:36:44Z
ingressrouteudps.traefik.io                 2025-01-03T08:36:44Z
managedsubscriptions.hub.traefik.io         2025-01-03T08:36:44Z
middlewares.traefik.io                      2025-01-03T08:36:44Z
middlewaretcps.traefik.io                   2025-01-03T08:36:44Z
httproutes.gateway.networking.k8s.io        2025-01-03T08:36:44Z
referencegrants.gateway.networking.k8s.io   2025-01-03T08:36:44Z
serverstransports.traefik.io                2025-01-03T08:36:44Z
serverstransporttcps.traefik.io             2025-01-03T08:36:44Z
tlsoptions.traefik.io                       2025-01-03T08:36:44Z
tlsstores.traefik.io                        2025-01-03T08:36:44Z
traefikservices.traefik.io                  2025-01-03T08:36:44Z
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

# 使用 traefik

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: fairy-music-ingress
  namespace: fm
spec:
  entryPoints:
    - web
  routes:
    - match: PathPrefix(`/fm`)
      kind: Rule
      services:
        - name: fairy-music-ui-service
          port: 80

1
2
3
4
5
6
7
8
9
10
11
12
13
14

查看应用的结果

❯ kubectl describe ingressroutes.traefik.io/fairy-music-ingress -n fm
Name:         fairy-music-ingress
Namespace:    fm
Labels:       app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/version=1.0.0
              helm.sh/chart=fairy-music-1.0.0
Annotations:  meta.helm.sh/release-name: fairy-music
              meta.helm.sh/release-namespace: fm
API Version:  traefik.io/v1alpha1
Kind:         IngressRoute
Metadata:
  Creation Timestamp:  2025-01-06T08:53:03Z
  Generation:          1
  Resource Version:    1002588
  UID:                 529e93a0-706b-43e2-a574-b5102990c3dd
Spec:
  Entry Points:
    web
  Routes:
    Kind:   Rule
    Match:  PathPrefix(`/fm`)
    Services:
      Name:  fairy-music-ui-service
      Port:  80
Events:      <none>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

实际应用中的写法:

---
apiVersion: "apps/v1"
kind: "Deployment"
metadata:
  namespace: "{{ .Release.Namespace }}"
  name: {{ include "fairy-music.name" . }}-ui-deployment
  labels:
{{ include "fairy-music.labels" .  | indent 4 }}
spec:
  replicas: {{ .Values.app.ui.replicaCount }}
  selector:
    matchLabels:
{{ include "fairy-music-ui.selectorLabels" .  | indent 6 }}
  template:
    metadata:
      namespace: "{{ .Release.Namespace }}"
      name:  {{ include "fairy-music.name" . }}-ui
      labels:
{{ include "fairy-music-ui.selectorLabels" .  | indent 8 }}
{{ include "fairy-music.labels" .  | indent 8 }}
    spec:
      containers:
        - name:  {{ include "fairy-music.name" . }}-ui
          image: "{{ .Values.app.ui.image.repository }}:{{ .Values.app.ui.image.tag }}"
          resources:
            limits:
              cpu: "{{ .Values.app.ui.resources.cpu }}"
              memory: "{{ .Values.app.ui.resources.memory }}"
            requests:
              cpu: "{{ .Values.app.ui.resources.cpu }}"
              memory: "{{ .Values.app.ui.resources.memory }}"
          ports:
            - containerPort: {{ .Values.app.ui.port }}
              protocol: "TCP"
          env: []
      restartPolicy: "Always"

---
apiVersion: "v1"
kind: "Service"
metadata:
  namespace: "{{ .Release.Namespace }}"
  name: {{ include "fairy-music.name" . }}-ui-service
  labels:
{{ include "fairy-music.labels" .  | indent 4 }}
spec:
  type: "ClusterIP"
  ports:
    - name: "http"
      protocol: "TCP"
      port: {{ .Values.app.ui.port }}
      targetPort: {{ .Values.app.ui.port }}
  sessionAffinity: "ClientIP"
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 3600
  selector:
{{ include "fairy-music-ui.selectorLabels" .  | indent 4 }}

{{/*
---
{{- if .Values.app.ingress.enabled -}}
apiVersion: "networking.k8s.io/v1"
kind: "Ingress"
metadata:
  namespace: "{{ .Release.Namespace }}"
  name: {{ include "fairy-music.name" . }}-ingress
  labels:
{{ include "fairy-music.labels" .  | indent 4 }}
spec:
  ingressClassName: "nginx"
  rules:
    - http:
        paths:
          - path: {{ .Values.app.ingress.path }}
            pathType: "Prefix"
            backend:
              service:
                name: {{ include "fairy-music.name" . }}-ui-service
                port:
                  number: {{ .Values.app.ui.port }}
{{- end -}}
*/}}

---
{{- if .Values.app.ingress.enabled -}}
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: {{ include "fairy-music.name" . }}-ingress
  namespace: {{ .Release.Namespace }}
  labels:
{{ include "fairy-music.labels" .  | indent 4 }}
spec:
  entryPoints:
    - web
  routes:
    - match: PathPrefix(`{{ .Values.app.ingress.path }}`)
      kind: Rule
      services:
        - name: {{ include "fairy-music.name" . }}-ui-service
          port: 80
{{- end -}}

遇到的坑: 在注释一段配置时, 不能直接用 yaml 规范中的 # , 要使用helm模版中的注释规范

# 关于 CRDs 的坑

注意在选择 apiVersion: traefik.io/v1alpha1 时, 要根据环境中所安装的 CRD 来决定. 这里列出我这里安装的CRDs 列表.

❯ kubectl get crds
NAME                                        CREATED AT
addons.k3s.cattle.io                        2024-10-28T05:37:52Z
etcdsnapshotfiles.k3s.cattle.io             2024-10-28T05:37:52Z
helmcharts.helm.cattle.io                   2024-10-28T05:37:52Z
helmchartconfigs.helm.cattle.io             2024-10-28T05:37:52Z
accesscontrolpolicies.hub.traefik.io        2025-01-06T03:16:04Z
aiservices.hub.traefik.io                   2025-01-06T03:16:04Z
apiaccesses.hub.traefik.io                  2025-01-06T03:16:04Z
apibundles.hub.traefik.io                   2025-01-06T03:16:04Z
apicatalogitems.hub.traefik.io              2025-01-06T03:16:04Z
apiplans.hub.traefik.io                     2025-01-06T03:16:04Z
apiportals.hub.traefik.io                   2025-01-06T03:16:04Z
apiratelimits.hub.traefik.io                2025-01-06T03:16:04Z
apis.hub.traefik.io                         2025-01-06T03:16:04Z
apiversions.hub.traefik.io                  2025-01-06T03:16:04Z
gatewayclasses.gateway.networking.k8s.io    2025-01-06T03:16:04Z
gateways.gateway.networking.k8s.io          2025-01-06T03:16:04Z
grpcroutes.gateway.networking.k8s.io        2025-01-06T03:16:04Z
httproutes.gateway.networking.k8s.io        2025-01-06T03:16:04Z
managedsubscriptions.hub.traefik.io         2025-01-06T03:16:04Z
referencegrants.gateway.networking.k8s.io   2025-01-06T03:16:05Z
serverstransporttcps.traefik.io             2025-01-06T03:03:10Z
ingressroutes.traefik.io                    2025-01-06T03:03:10Z
ingressroutetcps.traefik.io                 2025-01-06T03:03:10Z
ingressrouteudps.traefik.io                 2025-01-06T03:03:10Z
middlewares.traefik.io                      2025-01-06T03:03:10Z
middlewaretcps.traefik.io                   2025-01-06T03:03:10Z
serverstransports.traefik.io                2025-01-06T03:03:10Z
tlsoptions.traefik.io                       2025-01-06T03:03:10Z
tlsstores.traefik.io                        2025-01-06T03:03:10Z
traefikservices.traefik.io                  2025-01-06T03:03:10Z
ingressroutes.traefik.containo.us           2025-01-06T03:26:47Z
ingressroutetcps.traefik.containo.us        2025-01-06T03:26:47Z
ingressrouteudps.traefik.containo.us        2025-01-06T03:26:47Z
middlewares.traefik.containo.us             2025-01-06T03:26:47Z
middlewaretcps.traefik.containo.us          2025-01-06T03:26:47Z
serverstransports.traefik.containo.us       2025-01-06T03:26:47Z
tlsoptions.traefik.containo.us              2025-01-06T03:26:47Z
tlsstores.traefik.containo.us               2025-01-06T03:26:47Z
traefikservices.traefik.containo.us         2025-01-06T03:26:47Z

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

其中, 可以看到我们这次使用的ingressroutes.traefik.io资源, 由于Traefik, 在新的版本中使用了新的域名traefik.io 如果想要兼容旧的Api, 就要安装旧的CRD.

比如:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  labels:
  ...

1
2
3
4
5

就要安装 ingressroutes.traefik.containo.us

我这里, 为了兼容其他项目, 所以新旧版本我都装了

https://raw.githubusercontent.com/traefik/traefik/refs/heads/v3.3/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml

这里是 traefik 对应版本crd地址, 根据需求自己调整吧(这是目前最新的版本v3.3

用 k9s 查看, CRDs 列表

可以看到, 我们使用的 Traefik 时, 填的 ApiVersion 和 Kind 都与 CRDs 表中的内容可以对应上(呼应上了~)

Striveonger