Prometheus Consul Blackbox | Export 监控实现

前言：

blackbox_exporter

是监控Prometheus 官方提供的 exporter 之一,主要提供http、dns、实现tcp
、监控icmp 的源码下载实现监控数据采集
。

Consul

主要提供，监控服务发现  ，实现健康检查  ，监控等功能，高防服务器实现本次集成主要使用到服务发现功能 。监控

本文主要实现 ，实现基于consul_sd_config  & consul 的监控 prometheus  服务发现 ，实现网路设备ping监控 ，模板下载实现站点可用行监控，监控以及证书相关信息监控。实现

安装环境
：

k8sconsulPrometheusblackbox_exporter

1:  Consul 安装

1.1 ：使用helm 安装 consul

复制Bash

# 添加 consul helm 源

helm repo add hashicorp https://helm.releases.hashicorp.com

# 安装consul

helm -

n consul install \

--set storageClass=alicloud-disk-efficiency \consul hashicorp/

consul \

--version=0.32.11.2.3.4.5.6.7.8. 1.2：查看服务安装状态

复制Bash

[root@xxxxxxxx consul_install]# kubectl -

n consul get pods

NAME READY STATUS RESTARTS AGE

consul-consul-9lxfc 1/1 Running 0 6

d1h

consul-consul-ntqcf 1/1 Running 0 6

d1h

consul-consul-q7c6f 1/1 Running 0 6

d1h

consul-consul-server-0 1/1 Running 0 6

d1h

consul-consul-server-1 1/1 Running 0 6

d1h

consul-consul-server-2 1/1 Running 0 6d1h1.2.3.4.5.6.7.8.9. 1.3：nginx-ingress consulconsul_ingress.yml

复制Bash

# consul.xxxxxx.cn -----> 替换为正确域名apiVersion: networking.k8s.io/

v1

kind:

Ingress

metadata: name: consul-

ingress

namespace:

consul

annotations: kubernetes.io/ingress.class:

nginx

nginx.ingress.kubernetes.io/rewrite-target: /spec: rules: - host: consul.xxxxxx.cn http: paths: - path: / pathType:

Prefix

backend: service: name: consul-consul-

ui

port: number: 801.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23. 执行部署

复制Bash

kubectl apply -f consul_ingress.yml1.2. 1.4：访问测试

2: Blackbox_export

2.1：blackbox 安装blackbox-exporter-config.yaml

复制Bash

apiVersion:

v1

kind:

ConfigMap

metadata: name: blackbox-

exporter

labels: app: blackbox-

exporter

data: blackbox.yml: |- modules: ## ----------- DNS 检测配置 ----------- dns_tcp: prober:

dns

dns: transport_protocol: "tcp" preferred_ip_protocol: "ip4" query_name: "kubernetes.default.svc.cluster.local"

# 用于检测域名可用的监控网址

query_type: "A" ## ----------- TCP 检测模块配置 ----------- tcp_connect: prober:

tcp

timeout: 5

s

## ----------- ICMP 检测配置 ----------- ping: prober:

icmp

timeout: 5

s

icmp: preferred_ip_protocol: "ip4" ## ----------- HTTP GET 2xx 检测模块配置 ----------- http_get_2xx: prober:

http

timeout: 10

s

http: method:

GET

preferred_ip_protocol: "ip4" valid_http_versions: ["HTTP/1.1","HTTP/2"] valid_status_codes: [200] # 验证的HTTP状态码,

默认为2xx

no_follow_redirects: false

# 是云计算否不跟随重定向

## ----------- HTTP GET 3xx 检测模块配置 ----------- http_get_3xx: prober:

http

timeout: 10

s

http: method:

GET

preferred_ip_protocol: "ip4" valid_http_versions: ["HTTP/1.1","HTTP/2"] valid_status_codes: [301,302,304,305,306,307] # 验证的HTTP状态码,

默认为2xx

no_follow_redirects: false

# 是否不跟随重定向

## ----------- HTTP POST 监测模块 ----------- http_post_2xx: prober:

http

timeout: 10

s

http: method:

POST

preferred_ip_protocol: "ip4" valid_http_versions: ["HTTP/1.1", "HTTP/2"] #headers:

# HTTP头设置

# Content-Type: application/

json

#body: { } # 请求体设置1.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27.28.29.30.31.32.33.34.35.36.37.38.39.40.41.42.43.44.45.46.47.48.49.50.51.52.53.54.55.56.57.58.59. blackbox-exporter-deploy.yaml

复制Bash

apiVersion:

v1

kind:

Service

metadata: name: blackbox-

exporter

labels: k8s-app: blackbox-

exporter

spec: type:

ClusterIP

ports: - name:

http

port: 9115 targetPort: 9115 selector: k8s-app: blackbox-

exporter

---apiVersion: apps/

v1

kind:

Deployment

metadata: name: blackbox-

exporter

labels: k8s-app: blackbox-

exporter

spec: replicas: 1 selector: matchLabels: k8s-app: blackbox-

exporter

template: metadata: labels: k8s-app: blackbox-

exporter

spec: containers: - name: blackbox-

exporter

image: prom/blackbox-exporter:v0.19.0 args: - --config.file=/etc/blackbox_exporter/blackbox.yml - --web.listen-address=:9115 - --log.level=info ports: - name:

http

containerPort: 9115 resources: limits: cpu: 3 memory: 6000

Mi

requests: cpu: 100

m

memory: 50

Mi

livenessProbe: tcpSocket: port: 9115 initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 readinessProbe: tcpSocket: port: 9115 initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 volumeMounts: - name:

config

mountPath: /etc/

blackbox_exporter

volumes: - name:

config

configMap: name: blackbox-

exporter

defaultMode: 4201.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27.28.29.30.31.32.33.34.35.36.37.38.39.40.41.42.43.44.45.46.47.48.49.50.51.52.53.54.55.56.57.58.59.60.61.62.63.64.65.66.67.68.69.70.71.72.73. 执行安装

复制Bash

kubectl apply -f blackbox-exporter-deploy.yamlkubectl apply -f blackbox-exporter-config.yaml1.2.3. 2.2：nginx ingress blackbox-exporter • blackbox_ingress.yml

复制Bash

apiVersion: networking.k8s.io/

v1

kind:

Ingress

metadata: name: blackbox-

ingress

namespace:

monitoring

annotations: kubernetes.io/ingress.class:

nginx

nginx.ingress.kubernetes.io/rewrite-target: /spec: rules: - host: blackbox-devops.lululemon.cn http: paths: - path: / pathType:

Prefix

backend: service: name: blackbox-

exporter

port: number: 91151.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21. 执行安装

复制Bash

kubectl apply -f blackbox_ingress.yml1.2.

3: rometheus 添加 服务动态发现

复制Bash

##### http_get_2xx 数据获取

- job_name:

http_get_2xx

params: module: -

http_get_2xx

scrape_interval: 2

s

scrape_timeout: 2

s

metrics_path: /

probe

consul_sd_configs:

# consul 服务地址

- server: consul-consul-server.consul.svc.cluster.local:8500 tag_separator: , services: -

http_get_2xx

relabel_configs: - source_labels: [__meta_consul_service_address] target_label:

__param_target

- source_labels: [__meta_consul_service_address] target_label:

instance

- target_label:

__address__

## blackbox-

export 地址

replacement: blackbox-exporter.monitoring.svc.cluster.local:9115

####### icmp 配置

- job_name:

blackbox_icmp

params: module: -

ping

scrape_interval: 2

s

scrape_timeout: 2

s

metrics_path: /

probe

consul_sd_configs:

# consul 服务地址

- server: consul-consul-server.consul.svc.cluster.local:8500 tag_separator: , services: -

ping

relabel_configs: - source_labels: [__meta_consul_service_address] target_label:

__param_target

- source_labels: [__meta_consul_service_address] target_label:

instance

- target_label:

__address__

## blackbox-

export 地址

replacement: blackbox-exporter.monitoring.svc.cluster.local:91151.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27.28.29.30.31.32.33.34.35.36.37.38.39.40.41.42.43.44.45.

4：添加 icmp 监控

4.1：添加监控地址到consulicmp_list

复制Bash

192.168.1.1192.168.1.21.2.3. add_consul_service_icmp.sh

复制Bash

#!/usr/bin/

env bash

ip_addr=

$1

if test "$ip_addr";

then

curl -X PUT -d { "id": "icmp_${ ip_addr}", "name": "ping", "address": "${ ip_addr}", "port": 443, "Meta": { "env": "prod", "team": "network", "project": "network", "owner": "Mike" }, "tags": ["node"], "checks": [{ "http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}

\

http://consul-consul-server:8500/v1/agent/service/

register

else

echo "请输入参数"fi1.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24. 添加service ping

复制Bash

for i in `cat icmp_list`;do bash add_consul_service_icmp.sh $i;done1.2. 4.2 ：查看consul 服务

4.3：删除ping 监控地址脚本

复制Bash

#!/usr/bin/

env bash

ip_addr=

$1

curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/icmp_${ ip_addr}1.2.3.4.5.

5: 添加http_get_2xx

5.1：添加监控域名domain_name_list

复制Bash

wwww.baidu.comwwww.1111.comwwww.2222.com1.2.3.4. add_consul_service_http_get_2xx.sh

复制Bash

#!/usr/bin/

env bash

service_name=

$1

if test "$service_name";

then

curl -X PUT -d { "id": "http_get_2xx_${ service_name}", "name": "http_get_2xx", "address": "https://${ service_name}", "port": 443, "Meta": { "env": "prod", "team": "web", "project": "web", "owner": "Devops" }, "tags": ["node"], "checks": [{ "http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}

\

http://consul-consul-server:8500/v1/agent/service/

register

else

echo "请输入参数"fi1.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24. 添加 service  http_get_2xx

复制Bash

for i in `cat domain_name_list`;do bash add_consul_service_http_get_2xx.sh $i;done1.2. 5.2 ：查看consul 服务

5.3：删除域名监控脚本del_consul_service_http_get_2xx.sh

复制Bash

#!/usr/bin/

env bash

ip_addr=

$1

curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/http_get_2xx_${ ip_addr}1.2.3.4.5.

6 ：查看prometheus 监控

总结
：

使用上述方案，黑盒监控与自建cmdb 平台很容易进行集成，使其监控自动化 ，服务器租用不需要过多的人工干预，可以省去大量的人工成本
，grafana 的建站模板配置这里就不进行过多介绍，自行通过谷歌完成
。