kubernetes Prometheus監控mysql

Prometheus監控Pod中的應用，比如mysql，有2種方法

1.exporter pod方式，通過這個pod去監控mysql service上的數據，在通過/metrics暴露監控數據。

2.sidecar方式，在mysql pod中起一個exporter container，暴露/metrics。

說明:下面的實驗我基於前面的ingress-nginx Prometheus監控實驗，所有的deployment和svc都在ingress-nginx namespace 下面。

1.部署mysql

創建pv–5G限制，hostpath方式

創建pvc–申請1G，供mysql使用

創建service–名為mysql，暴露3306端口

創建deployment–用戶名root，密碼password

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv-volume
  namespace: ingress-nginx
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  namespace: ingress-nginx
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

---
apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace: ingress-nginx
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
  clusterIP: None

---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: mysql
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
          # Use secret in real usage
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

2.部署mysql exporter

創建service–名為mysql-export,暴露9104端口,供prometheus獲取數據

創建exporter–env指定所監控的mysql service，需要用戶名和密碼

---
apiVersion: v1
kind: Service
metadata:
  name: mysql-exporter
  namespace: ingress-nginx
  labels:
    k8s-app: mysql-exporter
spec:
  selector:
    k8s-app: mysql-exporter
  ports:
  - name: mysql-exporter
    port: 9104
    protocol: TCP
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql-exporter
  namespace: ingress-nginx
  labels:
    k8s-app: mysql-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: mysql-exporter
  template:
    metadata:
      labels:
        k8s-app: mysql-exporter
    spec:
      containers:
      - name: mysql-exporter
        image: prom/mysqld-exporter
        imagePullPolicy: IfNotPresent
        args: ["--collect.info_schema.processlist",
              "--collect.info_schema.innodb_metrics",
              "--collect.info_schema.tablestats",
              "--collect.info_schema.userstats",
              "--collect.engine_innodb_status",
              "--collect.slave_hosts"
              ]
        env:
        - name: DATA_SOURCE_NAME
          value: 'root:[email protected](mysql:3306)/mysql' # user:[email protected](hostname:3306)/
        resources:
          limits:
            cpu: 500m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 512Mi

3.Prometheus config修改獲取數據

job名為mysql部分，是我添加的。

[[email protected]-1 ~]# kubectl edit cm prometheus-configuration-8hk4m6bf76 -n ingress-nginx
apiVersion: v1
data:
  prometheus.yaml: | global: scrape_interval: 10s scrape_configs: - job_name: 'ingress-nginx-endpoints' kubernetes_sd_configs: - role: pod namespaces: names: - ingress-nginx relabel_configs: - source_labels: [__meta_kubernetes_pod_container_port_number] action: keep regex: "10254"

    - job_name: 'mysql'
      static_configs:
      - targets: ['mysql-exporter:9104']
kind: ConfigMap
metadata:
  creationTimestamp: "2022-01-18T05:56:12Z"
  labels:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: ingress-nginx
  name: prometheus-configuration-8hk4m6bf76
  namespace: ingress-nginx

删除Prometheus pod ，讓配置生效
[[email protected]-1 ~]# kubectl delete pod prometheus-server-779c8d44cf-9m2lp -n ingress-nginx
pod "prometheus-server-779c8d44cf-9m2lp" deleted

訪問Prometheus頁面 http://192.168.5.21:30221/

看到上面3個截圖，說明數據獲取成功了。

Granfana 導入Mysql 監控圖錶 http://192.168.5.21:31414/ 用戶名: admin 密碼: admin
推薦圖標ID：https://grafana.com/dashboards/7362

---

接下來，我們用sidecar方式來監控。

4.sidecar方式來監控

核心原理: 在pod中起一個mysql，一個mysql-export,並暴露3306，9104端口提供服務。

創建對象pv,pvc,deployment,service

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv-volume-sidecar
  namespace: ingress-nginx
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data-sidecar"

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim-sidecar
  namespace: ingress-nginx
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

---
apiVersion: v1
kind: Service
metadata:
  name: mysql-sidecar
  namespace: ingress-nginx
spec:
  ports:
  - name: mysql
    port: 3306
  - name: prometheus
    port: 9104
  selector:
    app: mysql-sidecar

---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: mysql-sidecar
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql-sidecar
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql-sidecar
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: password
        resources:
          limits:
            cpu: 500m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 512Mi
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage-sidecar
          mountPath: /var/lib/mysql

      - name: mysql-exporter
        image: prom/mysqld-exporter
        imagePullPolicy: IfNotPresent
        args: ["--collect.info_schema.processlist",
              "--collect.info_schema.innodb_metrics",
              "--collect.info_schema.tablestats",
              "--collect.info_schema.userstats",
              "--collect.engine_innodb_status",
              "--collect.slave_hosts"
              ]
        env:
        - name: DATA_SOURCE_NAME
          value: 'root:[email protected](localhost:3306)/mysql' # user:[email protected](hostname:3306)/
        resources:
          limits:
            cpu: 500m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 512Mi
        ports:
        - containerPort: 9104
          name: mysql-exporter

      volumes:
      - name: mysql-persistent-storage-sidecar
        persistentVolumeClaim:
          claimName: mysql-pv-claim-sidecar

檢查狀態

[[email protected] ~]# kubectl get pod,svc
NAME                                            READY   STATUS      RESTARTS   AGE
pod/grafana-74d85fc5bf-2dsjx                    1/1     Running     0          23h
pod/ingress-nginx-admission-create--1-654zx     0/1     Completed   0          29h
pod/ingress-nginx-admission-patch--1-8xpsp      0/1     Completed   2          29h
pod/ingress-nginx-controller-6d6c58c986-xwwfc   1/1     Running     0          24h
pod/mysql-68579b78bb-7szbp                      1/1     Running     0          26m
pod/mysql-exporter-fccfc9646-wcdvq              1/1     Running     0          36m
pod/mysql-sidecar-646f6b474f-n6wdf              2/2     Running     0          13m
pod/prometheus-server-779c8d44cf-22f72          1/1     Running     0          34m

NAME                                         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                      AGE
service/grafana                              NodePort       10.108.92.58     <none>        3000:31414/TCP                               23h
service/ingress-nginx-controller             LoadBalancer   10.96.194.255    <pending>     10254:30406/TCP,80:31361/TCP,443:32098/TCP   29h
service/ingress-nginx-controller-admission   ClusterIP      10.103.100.208   <none>        443/TCP                                      29h
service/mysql                                ClusterIP      None             <none>        3306/TCP                                     26m
service/mysql-exporter                       ClusterIP      10.106.183.8     <none>        9104/TCP                                     36m
service/mysql-sidecar                        ClusterIP      10.111.13.233    <none>        3306/TCP,9104/TCP                            13m
service/prometheus-server                    NodePort       10.99.70.216     <none>        9090:30221/TCP                               24h
[[email protected] ~]#

5.Prometheus config修改獲取數據

job名為mysql-sidecar部分，是我添加的。

[[email protected]-1 ~]# kubectl edit cm prometheus-configuration-8hk4m6bf76 -n ingress-nginx
apiVersion: v1
data:
  prometheus.yaml: | global: scrape_interval: 10s scrape_configs: - job_name: 'ingress-nginx-endpoints' kubernetes_sd_configs: - role: pod namespaces: names: - ingress-nginx relabel_configs: - source_labels: [__meta_kubernetes_pod_container_port_number] action: keep regex: "10254"

    - job_name: 'mysql'
      static_configs:
      - targets: ['mysql-exporter:9104']

    - job_name: 'mysql-sidecar'
      static_configs:
      - targets: ['mysql-sidecar:9104']

kind: ConfigMap
metadata:
  creationTimestamp: "2022-01-18T05:56:12Z"
  labels:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: ingress-nginx
  name: prometheus-configuration-8hk4m6bf76
  namespace: ingress-nginx

删除Prometheus pod ，讓配置生效
[[email protected]-1 ~]# kubectl delete pod prometheus-server-779c8d44cf-9m2lp -n ingress-nginx
pod "prometheus-server-779c8d44cf-9m2lp" deleted

查看prometheus 頁面 http://192.168.5.21:30221

Granfana 導入Mysql 監控圖錶 http://192.168.5.21:31414/ 用戶名: admin 密碼: admin
推薦圖標ID：https://grafana.com/dashboards/7362