NACOS 服务实例信息未正常清理
运维同事反馈, 线上某个 spring boot 应用服务挂了, 让我看看.

但是, 这个 188 节点机器上的我们之前就已经下线了, 怎么会告警?
排查链路, 我们的 prometheus 监控数据来源是从 nacos 自动获取的, 搜索一下 nacos 的服务注册信息, 确实还有这个节点的注册信息

尝试手动清理实例信息
# 删除curl -X DELETE 'http://nacos服务器地址/nacos/v2/ns/instance?serviceName=服务名&ip=实例IP&port=实例端口&ephemeral=true'结果发现服务实例还在, openapi 虽然返回 {“code”<0>0>,“message”:“success”,“data”:“ok”}, 但是实例注册信息还在
用 arthas 看看调用情况吧
curl -O https://arthas.aliyun.com/arthas-boot.jarjava -jar arthas-boot.jar查看调用链路
trace com.alibaba.nacos.naming.core.InstanceOperatorClientImpl removeInstance
查看源码 com.alibaba.nacos.naming.core.InstanceOperatorClientImpl#removeInstance
@Overridepublic void removeInstance(String namespaceId, String serviceName, Instance instance) { boolean ephemeral = instance.isEphemeral(); String clientId = IpPortBasedClient.getClientId(instance.toInetAddr(), ephemeral); // 这里进入 if 分支后直接 return 了 if (!clientManager.contains(clientId)) { Loggers.SRV_LOG.warn("remove instance from non-exist client: {}", clientId); return; } Service service = getService(namespaceId, serviceName, ephemeral); clientOperationService.deregisterInstance(service, instance, clientId);}可以判断命中了 if (!clientManager.contains(clientId)) 分支, 并没有执行下面的真实删除逻辑
查看一下这个 clientManager
ClientManager 负责管理所有接入客户端(包含服务端提供端/客户订阅端)的连接信息维护工作
watch com.alibaba.nacos.naming.core.InstanceOperatorClientImpl removeInstance -x 2
查看源码com.alibaba.nacos.naming.core.v2.client.manager.ClientManagerDelegate
public class ClientManagerDelegate implements ClientManager {
private final ConnectionBasedClientManager connectionBasedClientManager;
// 临时节点的客户端管理器 private final EphemeralIpPortClientManager ephemeralIpPortClientManager;
// 持久化节点的客户端管理器 private final PersistentIpPortClientManager persistentIpPortClientManager;
// ...
@Override public boolean contains(String clientId) { return connectionBasedClientManager.contains(clientId) || ephemeralIpPortClientManager.contains(clientId) || persistentIpPortClientManager.contains(clientId); }
// 根据clientId的格式选择不通的ClientManager private ClientManager getClientManagerById(String clientId) { if (isConnectionBasedClient(clientId)) { return connectionBasedClientManager; } // clientId以false结尾的是ephemeralIpPortClientManager // 否则是 persistentIpPortClientManager return clientId.endsWith(ClientConstants.PERSISTENT_SUFFIX) ? persistentIpPortClientManager : ephemeralIpPortClientManager; }
// clientId不包含#服务的客户端 private boolean isConnectionBasedClient(String clientId) { return !clientId.contains(IpPortBasedClient.ID_DELIMITER); }
// ...}感觉通过 openapi 的方式注销实例的情况, 可能走不通了, 看一下com.alibaba.nacos.naming.core.v2.client.Client#removeServiceInstance方法还有哪些地方使用吧


com.alibaba.nacos.naming.healthcheck.heartbeat.ExpiredInstanceChecker 源码
public class ExpiredInstanceChecker implements InstanceBeatChecker {
@Override public void doCheck(Client client, Service service, HealthCheckInstancePublishInfo instance) { // 从配置信息 nacos.naming.expireInstance 中获取是否开启过期实例探测, 默认为 true boolean expireInstance = ApplicationUtils.getBean(GlobalConfig.class).isExpireInstance(); if (expireInstance && isExpireInstance(service, instance)) { deleteIp(client, service, instance); } }
private boolean isExpireInstance(Service service, HealthCheckInstancePublishInfo instance) { long deleteTimeout = getTimeout(service, instance); return System.currentTimeMillis() - instance.getLastHeartBeatTime() > deleteTimeout; }
private long getTimeout(Service service, InstancePublishInfo instance) { Optional<Object> timeout = getTimeoutFromMetadata(service, instance); if (!timeout.isPresent()) { // IP_DELETE_TIMEOUT = preserved.ip.delete.timeout timeout = Optional.ofNullable(instance.getExtendDatum().get(PreservedMetadataKeys.IP_DELETE_TIMEOUT)); } // DEFAULT_IP_DELETE_TIMEOUT = 30 秒 return timeout.map(ConvertUtils::toLong).orElse(Constants.DEFAULT_IP_DELETE_TIMEOUT); }
private Optional<Object> getTimeoutFromMetadata(Service service, InstancePublishInfo instance) { Optional<InstanceMetadata> instanceMetadata = ApplicationUtils.getBean(NamingMetadataManager.class) .getInstanceMetadata(service, instance.getMetadataId()); return instanceMetadata.map(metadata -> metadata.getExtendData().get(PreservedMetadataKeys.IP_DELETE_TIMEOUT)); }
// 这里会自动移除过期的服务实例 private void deleteIp(Client client, Service service, InstancePublishInfo instance) { Loggers.SRV_LOG.info("[AUTO-DELETE-IP] service: {}, ip: {}", service.toString(), JacksonUtils.toJson(instance)); client.removeServiceInstance(service); NotifyCenter.publishEvent(new ClientOperationEvent.ClientDeregisterServiceEvent(service, client.getClientId())); NotifyCenter.publishEvent(new MetadataEvent.InstanceMetadataEvent(service, instance.getMetadataId(), true)); NotifyCenter.publishEvent(new DeregisterInstanceTraceEvent(System.currentTimeMillis(), "", false, DeregisterInstanceReason.HEARTBEAT_EXPIRE, service.getNamespace(), service.getGroup(), service.getName(), instance.getIp(), instance.getPort())); }}ExpiredInstanceChecker ← InstanceBeatCheckTask ← ClientBeatCheckTaskV2 ← ClientBeatCheckTaskV2
public class InstanceBeatCheckTask implements Interceptable { static { CHECKERS.add(new UnhealthyInstanceChecker()); CHECKERS.add(new ExpiredInstanceChecker()); CHECKERS.addAll(NacosServiceLoader.load(InstanceBeatChecker.class)); } }public class IpPortBasedClient extends AbstractClient {
public void init() { if (ephemeral) { // client 初始化时, 如果是临时性节点, 会初始化自动探测任务 beatCheckTask = new ClientBeatCheckTaskV2(this); HealthCheckReactor.scheduleCheck(beatCheckTask); } else { healthCheckTaskV2 = new HealthCheckTaskV2(this); HealthCheckReactor.scheduleCheck(healthCheckTaskV2); } }}问题的解决
最后, 发现根本原因是, 运维的流水线构建时, 没有考虑到 docker 镜像制作的工作空间问题, 并发了, 导致了两个原本应该使用不同 jar 包的 docker 镜像, 实际上制作出来的是同一个服务镜像, 恰好这个错误的镜像在 188 机器上运行
📎 参考文章
文章分享
如果这篇文章对你有帮助,欢迎分享给更多人!













