背景在大型分布式系统中服务之间调用复杂链路追踪可以帮助梳理请求流向现代系统也需要实时监控来快速响应事件以及故障让我们了解系统瓶颈和高负载路径从而可以进行优化。Ingress-Nginx 是在 Kubernetes 环境中使用的专门用于管理进入 Kubernetes 集群的外部访问流量。它基于 Nginx利用其作为反向代理和负载均衡器的能力但专门配置和优化以适应 Kubernetes 的架构。Ingress Controller 的主要任务是根据预先定义的规则通过 Kubernetes Ingress 资源设置将外部请求路由到集群内的特定服务。前提Ingress-Nginx 版本 1.10.0应用服务已经接入 Opentelemetry 采集链路数据K8s 集群版本1. 部署示例服务这里我们会部署一个 spring boot 的服务A 服务会调用 B 服务。本示例中 java 版本是 17Maven 版本是 3.9.10。由于采集 Ingress-Nginx 的链路需要和后端链路打通所以在部署业务镜像的时候需要将 OTEL 探针一并打包到业务镜像。以下是在服务 Dockerfile 中将 Agent 打包到业务服务容器镜像的配置为服务提供采集链路数据的基础能力。FROM curlimages/curl:latest AS agent-download USER root RUN curl -Lo /opentelemetry-javaagent.jar \ https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar FROM openjdk:17-jdk-slim WORKDIR /app COPY --fromagent-download /opentelemetry-javaagent.jar /app/opentelemetry-javaagent.jar COPY target/serviceb-1.0-SNAPSHOT.jar /app/service-b.jar ENV OTEL_SERVICE_NAMEservice-b \ OTEL_EXPORTER_OTLP_ENDPOINThttp://datakit-endpoint:4317 \ OTEL_TRACES_SAMPLERparentbased_always_on \ OTEL_PROPAGATORStracecontext,baggage \ OTEL_METRICS_EXPORTERnone \ OTEL_LOGS_EXPORTERnone # 修改启动命令添加 Java Agent CMD [java, -javaagent:/app/opentelemetry-javaagent.jar, -jar, /app/service-b.jar]创建k8s-java-app.yaml部署服务apiVersion: apps/v1 kind: Deployment metadata: name: service-a spec: replicas: 1 selector: matchLabels: app: service-a template: metadata: labels: app: service-a spec: containers: - name: service-a image: your-repo/service-a:otel-1.0 ports: - containerPort: 9090 env: - name: HOST_IP valueFrom: fieldRef: fieldPath: status.hostIP - name: SPRING_MAIN_ALLOW_CIRCULAR_REFERENCES value: true - name: OTEL_SERVICE_NAME value: service-a - name: OTEL_EXPORTER value: otlp - name: OTEL_EXPORTER_OTLP_PROTOCOL value: grpc - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://$(HOST_IP):4317 - name: OTEL_PROPAGATORS value: tracecontext,baggage apiVersion: v1 kind: Service metadata: name: service-a spec: ports: - port: 9090 targetPort: 9090 selector: app: service-a apiVersion: apps/v1 kind: Deployment metadata: name: service-b spec: replicas: 1 selector: matchLabels: app: service-b template: metadata: labels: app: service-b spec: containers: - name: service-b image: your-repo/service-b:otel-1.0 ports: - containerPort: 8090 env: - name: HOST_IP valueFrom: fieldRef: fieldPath: status.hostIP - name: OTEL_SERVICE_NAME value: service-b - name: OTEL_EXPORTER value: otlp - name: OTEL_EXPORTER_OTLP_PROTOCOL value: grpc - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://$(HOST_IP):4317 - name: OTEL_PROPAGATORS value: tracecontext,baggage apiVersion: v1 kind: Service metadata: name: service-b spec: ports: - port: 8090 targetPort: 8090 selector: app: service-b2. 安装 Ingress Nginx创建一个ingress-nginx.yaml文件apiVersion: v1 kind: Namespace metadata: name: ingress-nginx --- apiVersion: v1 kind: ServiceAccount metadata: name: ingress-nginx namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: ingress-nginx rules: - apiGroups: - resources: - configmaps - endpoints - nodes - pods - secrets - services verbs: - list - watch - get - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch - apiGroups: - coordination.k8s.io resources: - leases verbs: - get - watch - list - create - update - apiGroups: - networking.k8s.io resources: - ingresses - ingressclasses verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses/status verbs: - update - apiGroups: - extensions resources: - ingresses verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses/status verbs: - update - apiGroups: - resources: - events verbs: - create - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ingress-nginx subjects: - kind: ServiceAccount name: ingress-nginx namespace: ingress-nginx --- apiVersion: apps/v1 kind: Deployment metadata: name: ingress-nginx-controller namespace: ingress-nginx spec: replicas: 1 selector: matchLabels: app: ingress-nginx template: metadata: labels: app: ingress-nginx spec: hostNetwork: true serviceAccountName: ingress-nginx containers: - name: controller image: k8s.gcr.io/ingress-nginx/controller:v1.10.0 args: - /nginx-ingress-controller - --publish-service$(POD_NAMESPACE)/ingress-nginx-controller - --election-idingress-controller-leader - --controller-classk8s.io/ingress-nginx - --ingress-classnginx - --configmapingress-nginx/ingress-nginx-controller env: - name: HOST_IP valueFrom: fieldRef: fieldPath: status.hostIP - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://$(HOST_IP):4317 - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 --- apiVersion: v1 kind: Service metadata: name: ingress-nginx-controller namespace: ingress-nginx spec: type: NodePort ports: - name: http port: 80 targetPort: 80 - name: https port: 443 targetPort: 443 selector: app: ingress-nginx --- apiVersion: v1 kind: ConfigMap metadata: name: ingress-nginx-controller namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx data: enable-opentelemetry: true otel-sampler: AlwaysOn opentelemetry-operation-name: HTTP $request_method $service_name $uri $opentelemetry_trace_id opentelemetry-trust-incoming-span: true # Defaults # otel-service-name: nginx # otel-sampler-ratio: 0.01应用该配置kubectl apply -f ingress-nginx.yaml3. 采集 Ingress-nginx 链路配置3.1 DataKit 开启 OTEL 采集器datakit.yaml中采用 CM 挂载方式开启集群的 OTEL 采集器。在 volumeMounts 添加- mountPath: /usr/local/datakit/conf.d/opentelemetry/opentelemetry.conf name: datakit-conf subPath: opentelemetry.conf在 CM 处添加采集器opentelemetry.conf: |- [[inputs.opentelemetry]] [inputs.opentelemetry.http] enable true http_status_ok 200 trace_api /otel/v1/traces [inputs.opentelemetry.grpc] trace_enable true metric_enable true addr 0.0.0.0:4317重启 DataKitkubectl apply -f datakit.yaml3.2 OTEL Agent 采集链路数据在服务 Dockerfile 中将 Agent 打包到业务服务容器镜像为服务提供采集链路数据的基础能力。FROM curlimages/curl:latest AS agent-download USER root RUN curl -Lo /opentelemetry-javaagent.jar \ https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar FROM openjdk:17-jdk-slim WORKDIR /app COPY --fromagent-download /opentelemetry-javaagent.jar /app/opentelemetry-javaagent.jar COPY target/serviceb-1.0-SNAPSHOT.jar /app/service-b.jar ENV OTEL_SERVICE_NAMEservice-b \ OTEL_EXPORTER_OTLP_ENDPOINThttp://datakit-endpoint:4317 \ OTEL_TRACES_SAMPLERparentbased_always_on \ OTEL_PROPAGATORStracecontext,baggage \ OTEL_METRICS_EXPORTERnone \ OTEL_LOGS_EXPORTERnone # 修改启动命令添加 Java Agent CMD [java, -javaagent:/app/opentelemetry-javaagent.jar, -jar, /app/service-b.jar]在服务部署的 yaml 中配置环境变量。- name: OTEL_EXPORTER value: otlp - name: OTEL_EXPORTER_OTLP_PROTOCOL value: grpc - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://$(HOST_IP):4317 - name: OTEL_PROPAGATORS value: tracecontext,baggage3.3 编辑 ingress-controller CM 资源如果 ingress-controller 服务有 configmap 则在 CM 中增加如下四行enable-opentelemetry: true otel-sampler: AlwaysOn opentelemetry-operation-name: HTTP $request_method $service_name $uri $opentelemetry_trace_id opentelemetry-trust-incoming-span: trueApply 相应的 ingress 的 yaml并重启 ingress-controller。3.4 增加 ingress-controller 环境变量在部署 ingress-controller 配置文件 ingress-nginx.yaml 的 deployment 部分中添加 OTEL 配置位置在 spec.template.spec.containers.env 下注意端口开启。- name: OTEL_EXPORTER value: otlp - name: OTEL_EXPORTER_OTLP_PROTOCOL value: grpc - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://$(HOST_IP):4317 - name: OTEL_SERVICE_NAME value: nginx - name: OTEL_TRACES_SAMPLER value: always_on - name: OTEL_PROPAGATORS value: tracecontext,baggage重新 apply ingress-nginx.yaml重启 ingress-controller 容器。观测云再次访问 ingress 域名制造数据。到观测云控制台「应用性能监测」可以看到 Ingress-Nginx 链路数据正常上报。