NACOS 服务实例信息未正常清理

855 字
4 分钟
NACOS 服务实例信息未正常清理

运维同事反馈, 线上某个 spring boot 应用服务挂了, 让我看看.

Untitled 54
Untitled 54

但是, 这个 188 节点机器上的我们之前就已经下线了, 怎么会告警?

排查链路, 我们的 prometheus 监控数据来源是从 nacos 自动获取的, 搜索一下 nacos 的服务注册信息, 确实还有这个节点的注册信息

Untitled 55
Untitled 55

尝试手动清理实例信息

Terminal window
# 删除
curl -X DELETE 'http://nacos服务器地址/nacos/v2/ns/instance?serviceName=服务名&ip=实例IP&port=实例端口&ephemeral=true'

结果发现服务实例还在, openapi 虽然返回 {“code”<0>,“message”:“success”,“data”:“ok”}, 但是实例注册信息还在

用 arthas 看看调用情况吧

Terminal window
curl -O https://arthas.aliyun.com/arthas-boot.jar
java -jar arthas-boot.jar

查看调用链路

Terminal window
trace com.alibaba.nacos.naming.core.InstanceOperatorClientImpl removeInstance

Untitled 56
Untitled 56

查看源码 com.alibaba.nacos.naming.core.InstanceOperatorClientImpl#removeInstance

@Override
public void removeInstance(String namespaceId, String serviceName, Instance instance) {
boolean ephemeral = instance.isEphemeral();
String clientId = IpPortBasedClient.getClientId(instance.toInetAddr(), ephemeral);
// 这里进入 if 分支后直接 return 了
if (!clientManager.contains(clientId)) {
Loggers.SRV_LOG.warn("remove instance from non-exist client: {}", clientId);
return;
}
Service service = getService(namespaceId, serviceName, ephemeral);
clientOperationService.deregisterInstance(service, instance, clientId);
}

可以判断命中了 if (!clientManager.contains(clientId)) 分支, 并没有执行下面的真实删除逻辑

查看一下这个 clientManager

ClientManager 负责管理所有接入客户端(包含服务端提供端/客户订阅端)的连接信息维护工作

Terminal window
watch com.alibaba.nacos.naming.core.InstanceOperatorClientImpl removeInstance -x 2

Untitled 57
Untitled 57

查看源码com.alibaba.nacos.naming.core.v2.client.manager.ClientManagerDelegate

public class ClientManagerDelegate implements ClientManager {
private final ConnectionBasedClientManager connectionBasedClientManager;
// 临时节点的客户端管理器
private final EphemeralIpPortClientManager ephemeralIpPortClientManager;
// 持久化节点的客户端管理器
private final PersistentIpPortClientManager persistentIpPortClientManager;
// ...
@Override
public boolean contains(String clientId) {
return connectionBasedClientManager.contains(clientId) || ephemeralIpPortClientManager.contains(clientId)
|| persistentIpPortClientManager.contains(clientId);
}
// 根据clientId的格式选择不通的ClientManager
private ClientManager getClientManagerById(String clientId) {
if (isConnectionBasedClient(clientId)) {
return connectionBasedClientManager;
}
// clientId以false结尾的是ephemeralIpPortClientManager
// 否则是 persistentIpPortClientManager
return clientId.endsWith(ClientConstants.PERSISTENT_SUFFIX) ?
persistentIpPortClientManager : ephemeralIpPortClientManager;
}
// clientId不包含#服务的客户端
private boolean isConnectionBasedClient(String clientId) {
return !clientId.contains(IpPortBasedClient.ID_DELIMITER);
}
// ...
}

感觉通过 openapi 的方式注销实例的情况, 可能走不通了, 看一下com.alibaba.nacos.naming.core.v2.client.Client#removeServiceInstance方法还有哪些地方使用吧

Untitled 58
Untitled 58

Untitled 59
Untitled 59

com.alibaba.nacos.naming.healthcheck.heartbeat.ExpiredInstanceChecker 源码

public class ExpiredInstanceChecker implements InstanceBeatChecker {
@Override
public void doCheck(Client client, Service service, HealthCheckInstancePublishInfo instance) {
// 从配置信息 nacos.naming.expireInstance 中获取是否开启过期实例探测, 默认为 true
boolean expireInstance = ApplicationUtils.getBean(GlobalConfig.class).isExpireInstance();
if (expireInstance && isExpireInstance(service, instance)) {
deleteIp(client, service, instance);
}
}
private boolean isExpireInstance(Service service, HealthCheckInstancePublishInfo instance) {
long deleteTimeout = getTimeout(service, instance);
return System.currentTimeMillis() - instance.getLastHeartBeatTime() > deleteTimeout;
}
private long getTimeout(Service service, InstancePublishInfo instance) {
Optional<Object> timeout = getTimeoutFromMetadata(service, instance);
if (!timeout.isPresent()) {
// IP_DELETE_TIMEOUT = preserved.ip.delete.timeout
timeout = Optional.ofNullable(instance.getExtendDatum().get(PreservedMetadataKeys.IP_DELETE_TIMEOUT));
}
// DEFAULT_IP_DELETE_TIMEOUT = 30 秒
return timeout.map(ConvertUtils::toLong).orElse(Constants.DEFAULT_IP_DELETE_TIMEOUT);
}
private Optional<Object> getTimeoutFromMetadata(Service service, InstancePublishInfo instance) {
Optional<InstanceMetadata> instanceMetadata = ApplicationUtils.getBean(NamingMetadataManager.class)
.getInstanceMetadata(service, instance.getMetadataId());
return instanceMetadata.map(metadata -> metadata.getExtendData().get(PreservedMetadataKeys.IP_DELETE_TIMEOUT));
}
// 这里会自动移除过期的服务实例
private void deleteIp(Client client, Service service, InstancePublishInfo instance) {
Loggers.SRV_LOG.info("[AUTO-DELETE-IP] service: {}, ip: {}", service.toString(), JacksonUtils.toJson(instance));
client.removeServiceInstance(service);
NotifyCenter.publishEvent(new ClientOperationEvent.ClientDeregisterServiceEvent(service, client.getClientId()));
NotifyCenter.publishEvent(new MetadataEvent.InstanceMetadataEvent(service, instance.getMetadataId(), true));
NotifyCenter.publishEvent(new DeregisterInstanceTraceEvent(System.currentTimeMillis(), "",
false, DeregisterInstanceReason.HEARTBEAT_EXPIRE, service.getNamespace(), service.getGroup(),
service.getName(), instance.getIp(), instance.getPort()));
}
}

ExpiredInstanceCheckerInstanceBeatCheckTaskClientBeatCheckTaskV2ClientBeatCheckTaskV2

public class InstanceBeatCheckTask implements Interceptable {
static {
CHECKERS.add(new UnhealthyInstanceChecker());
CHECKERS.add(new ExpiredInstanceChecker());
CHECKERS.addAll(NacosServiceLoader.load(InstanceBeatChecker.class));
}
}
public class IpPortBasedClient extends AbstractClient {
public void init() {
if (ephemeral) {
// client 初始化时, 如果是临时性节点, 会初始化自动探测任务
beatCheckTask = new ClientBeatCheckTaskV2(this);
HealthCheckReactor.scheduleCheck(beatCheckTask);
} else {
healthCheckTaskV2 = new HealthCheckTaskV2(this);
HealthCheckReactor.scheduleCheck(healthCheckTaskV2);
}
}
}

问题的解决#

最后, 发现根本原因是, 运维的流水线构建时, 没有考虑到 docker 镜像制作的工作空间问题, 并发了, 导致了两个原本应该使用不同 jar 包的 docker 镜像, 实际上制作出来的是同一个服务镜像, 恰好这个错误的镜像在 188 机器上运行

📎 参考文章#

文章分享

如果这篇文章对你有帮助,欢迎分享给更多人!

NACOS 服务实例信息未正常清理
https://blog.sephy.top/posts/nacos-服务实例信息未正常清理/
作者
虾米
发布于
2024-02-20
许可协议
CC BY-NC-SA 4.0
随机文章随机推荐
Profile Image of the Author
虾米
coder
分类
标签
站点统计
文章
61
分类
4
标签
52
总字数
64,663
运行时长
0
最后活动
0 天前
站点信息
构建平台
GitHub Actions
博客版本
Firefly v6.13.9
文章许可
CC BY-NC-SA 4.0