虾米的个人笔记

在 PVE 上给软路由虚拟机加一个网络 Watchdog

Sun, 14 Jun 2026 00:00:00 GMT

很多 Homelab 会把软路由跑在 PVE 里的虚拟机上，例如 iKuai、OpenWrt、RouterOS 等。这样做很灵活：网卡直通、快照、备份、迁移都方便。但它也带来一个问题：如果这台“网络出口虚拟机”异常了，整个内网可能都会断网，甚至你连 PVE Web UI 都打不开。

这篇文章记录一个保守的自愈方案：在 PVE 宿主机上运行一个 watchdog，每分钟检测一次外网连通性。如果连续失败，就先重启软路由虚拟机；如果软路由连续自愈多次仍无效，再重启 PVE 宿主机。

注意：下面的 IP、VMID、路径都是示例值。不要把自己的真实公网地址、PVE SSH 别名、真实内网网段写进公开文章或公开仓库。

设计目标#

这个 watchdog 的目标不是“网络一断就马上重启”，而是尽量避免误判：

单次 ping 失败只记录状态，不做动作。
连续失败达到阈值后，先恢复软路由虚拟机。
软路由恢复动作有冷却时间，避免反复重启。
连续多次恢复软路由仍失败，才升级为重启 PVE。
所有阈值都放在配置文件里，不写死在脚本里。
只写 systemd journal 日志，不依赖外部通知服务。

判断链路#

最小可用检测可以只 ping 一个外网域名：

Terminal window

1
ping -c 3 -W 2 www.baidu.com

如果你的环境更复杂，可以同时检测多个目标，例如 DNS、HTTP、上游网关等。本文为了保持脚本简单，只把 www.baidu.com 作为主检测目标，并额外 ping 一下软路由 LAN IP，用于日志诊断。

日志里的含义大致是：

软路由 LAN IP 可达，但外网不可达：更可能是拨号、DNS、运营商或上游出口问题。
软路由 LAN IP 不可达，外网也不可达：更可能是软路由 VM、虚拟网桥、直通网卡或宿主机网络问题。

配置文件#

先创建 /etc/default/pve-net-watchdog：

Terminal window

1
# pve-net-watchdog configuration
2

3
# 软路由虚拟机 VMID，按你的 PVE 环境修改。
4
VMID=103
5

6
# 软路由 LAN IP，使用示例地址，部署时替换成自己的网关地址。
7
ROUTER_IP=192.168.1.1
8

9
# 主连通性检测目标。
10
PING_TARGET=www.baidu.com
11

12
PING_COUNT=3
13
PING_TIMEOUT_SECONDS=2
14

15
# 连续故障达到多少分钟后，开始恢复软路由虚拟机。
16
ROUTER_RESTART_AFTER_MINUTES=5
17

18
# 两次软路由恢复动作之间的冷却时间。
19
ROUTER_COOLDOWN_MINUTES=10
20

21
# 软路由连续恢复多少次仍失败后，升级为重启 PVE。
22
ROUTER_MAX_RESTARTS_BEFORE_PVE=3
23

24
# 是否允许自动重启 PVE。建议先设为 0 观察日志，确认无误后再改成 1。
25
ENABLE_PVE_REBOOT=1
26

27
# 第三次软路由恢复后，继续等待多少分钟仍失败才重启 PVE。
28
PVE_REBOOT_AFTER_LAST_ROUTER_MINUTES=5
29

30
# 避免 PVE 重启循环。
31
PVE_REBOOT_COOLDOWN_HOURS=24
32

33
STATE_DIR=/var/lib/pve-net-watchdog
34
LOCK_FILE=/run/pve-net-watchdog.lock

如果你还在调试阶段，建议先把 ENABLE_PVE_REBOOT=0，观察几天日志后再打开。

Watchdog 脚本#

把下面的脚本保存为 /usr/local/sbin/pve-net-watchdog，并设置为可执行：

1
#!/usr/bin/env bash
2
set -euo pipefail
3

4
: "${CONFIG_FILE:=/etc/default/pve-net-watchdog}"
5

6
set_defaults() {
7
  : "${VMID:=103}"
8
  : "${ROUTER_IP:=192.168.1.1}"
9
  : "${PING_TARGET:=www.baidu.com}"
10
  : "${PING_COUNT:=3}"
11
  : "${PING_TIMEOUT_SECONDS:=2}"
12
  : "${ROUTER_RESTART_AFTER_MINUTES:=5}"
13
  : "${ROUTER_COOLDOWN_MINUTES:=10}"
14
  : "${ROUTER_MAX_RESTARTS_BEFORE_PVE:=3}"
15
  : "${ENABLE_PVE_REBOOT:=0}"
16
  : "${PVE_REBOOT_AFTER_LAST_ROUTER_MINUTES:=5}"
17
  : "${PVE_REBOOT_COOLDOWN_HOURS:=24}"
18
  : "${STATE_DIR:=/var/lib/pve-net-watchdog}"
19
  : "${LOCK_FILE:=/run/pve-net-watchdog.lock}"
20
  : "${DRY_RUN:=0}"
21
}
22

23
load_config() {
24
  if [[ -r "$CONFIG_FILE" ]]; then
25
    # shellcheck source=/dev/null
26
    source "$CONFIG_FILE"
27
  fi
28
  set_defaults
29
  STATE_FILE="$STATE_DIR/state"
30
}
31

32
log() {
33
  local level="$1"
34
  shift
35
  local message="$*"
36
  printf '%s [%s] %s\n' "$(date -Is)" "$level" "$message"
37
  logger -t pve-net-watchdog "[$level] $message"
38
}
39

40
ping_host() {
41
  local host="$1"
42
  ping -c "$PING_COUNT" -W "$PING_TIMEOUT_SECONDS" "$host" >/dev/null 2>&1
43
}
44

45
run_cmd() {
46
  if [[ "$DRY_RUN" == "1" ]]; then
47
    log info "DRY_RUN would run: $*"
48
    return 0
49
  fi
50
  "$@"
51
}
52

53
init_state() {
54
  FAIL_COUNT=0
55
  FIRST_FAILURE_EPOCH=0
56
  ROUTER_RESTART_COUNT=0
57
  LAST_ROUTER_RESTART_EPOCH=0
58
  LAST_PVE_REBOOT_EPOCH=0
59
}
60

61
ensure_state_dir() {
62
  mkdir -p "$STATE_DIR"
63
  chmod 0700 "$STATE_DIR"
64
}
65

66
load_state() {
67
  init_state
68
  if [[ -r "$STATE_FILE" ]]; then
69
    # shellcheck source=/dev/null
70
    source "$STATE_FILE"
71
  fi
72
}
73

74
save_state() {
75
  local tmp="$STATE_FILE.tmp.$$"
76
  umask 077
77
  {
78
    printf 'FAIL_COUNT=%s\n' "$FAIL_COUNT"
79
    printf 'FIRST_FAILURE_EPOCH=%s\n' "$FIRST_FAILURE_EPOCH"
80
    printf 'ROUTER_RESTART_COUNT=%s\n' "$ROUTER_RESTART_COUNT"
81
    printf 'LAST_ROUTER_RESTART_EPOCH=%s\n' "$LAST_ROUTER_RESTART_EPOCH"
82
    printf 'LAST_PVE_REBOOT_EPOCH=%s\n' "$LAST_PVE_REBOOT_EPOCH"
83
  } >"$tmp"
84
  mv "$tmp" "$STATE_FILE"
85
}
86

87
elapsed_minutes_since() {
88
  local start_epoch="$1"
89
  local now="$2"
90
  if [[ "$start_epoch" -le 0 ]]; then
91
    echo 0
92
    return
93
  fi
94
  echo $(((now - start_epoch) / 60))
95
}
96

97
minutes_have_passed() {
98
  local last_epoch="$1"
99
  local now="$2"
100
  local minutes="$3"
101
  [[ "$last_epoch" -le 0 ]] && return 0
102
  [[ $((now - last_epoch)) -ge $((minutes * 60)) ]]
103
}
104

105
hours_have_passed() {
106
  local last_epoch="$1"
107
  local now="$2"
108
  local hours="$3"
109
  [[ "$last_epoch" -le 0 ]] && return 0
110
  [[ $((now - last_epoch)) -ge $((hours * 3600)) ]]
111
}
112

113
reset_failure_state() {
114
  FAIL_COUNT=0
115
  FIRST_FAILURE_EPOCH=0
116
  ROUTER_RESTART_COUNT=0
117
  LAST_ROUTER_RESTART_EPOCH=0
118
}
119

120
recover_router_vm() {
121
  local status
122
  status="$(qm status "$VMID" 2>/dev/null || true)"
123

124
  if ! grep -q 'status: running' <<<"$status"; then
125
    log warning "VM $VMID is not running; starting it"
126
    run_cmd qm start "$VMID"
127
    return
128
  fi
129

130
  log warning "VM $VMID is running but connectivity is still down; rebooting it"
131
  if ! run_cmd qm reboot "$VMID"; then
132
    log error "qm reboot $VMID failed; resetting VM"
133
    run_cmd qm reset "$VMID"
134
  fi
135
}
136

137
maybe_recover_router() {
138
  local now="$1"
139
  local failure_minutes="$2"
140

141
  if [[ "$failure_minutes" -lt "$ROUTER_RESTART_AFTER_MINUTES" ]]; then
142
    return
143
  fi
144

145
  if [[ "$ROUTER_RESTART_COUNT" -ge "$ROUTER_MAX_RESTARTS_BEFORE_PVE" ]]; then
146
    log warning "router restart limit reached: $ROUTER_RESTART_COUNT/$ROUTER_MAX_RESTARTS_BEFORE_PVE"
147
    return
148
  fi
149

150
  if ! minutes_have_passed "$LAST_ROUTER_RESTART_EPOCH" "$now" "$ROUTER_COOLDOWN_MINUTES"; then
151
    log warning "connectivity still down; router restart cooldown is active"
152
    return
153
  fi
154

155
  recover_router_vm
156
  ROUTER_RESTART_COUNT=$((ROUTER_RESTART_COUNT + 1))
157
  LAST_ROUTER_RESTART_EPOCH="$now"
158
  log warning "recorded router recovery attempt $ROUTER_RESTART_COUNT/$ROUTER_MAX_RESTARTS_BEFORE_PVE"
159
}
160

161
maybe_reboot_pve() {
162
  local now="$1"
163

164
  if [[ "$ENABLE_PVE_REBOOT" != "1" ]]; then
165
    return
166
  fi
167

168
  if [[ "$ROUTER_RESTART_COUNT" -lt "$ROUTER_MAX_RESTARTS_BEFORE_PVE" ]]; then
169
    return
170
  fi
171

172
  if ! minutes_have_passed "$LAST_ROUTER_RESTART_EPOCH" "$now" "$PVE_REBOOT_AFTER_LAST_ROUTER_MINUTES"; then
173
    log warning "PVE reboot threshold reached, waiting for post-router-restart grace period"
174
    return
175
  fi
176

177
  if ! hours_have_passed "$LAST_PVE_REBOOT_EPOCH" "$now" "$PVE_REBOOT_COOLDOWN_HOURS"; then
178
    log error "PVE reboot suppressed by cooldown"
179
    return
180
  fi
181

182
  LAST_PVE_REBOOT_EPOCH="$now"
183
  save_state
184
  log error "connectivity still down after $ROUTER_RESTART_COUNT router recoveries; rebooting PVE host"
185
  run_cmd systemctl reboot
186
}
187

188
run_once() {
189
  load_config
190
  ensure_state_dir
191
  load_state
192

193
  local now
194
  now="$(date +%s)"
195

196
  if ping_host "$PING_TARGET"; then
197
    if [[ "$FAIL_COUNT" -gt 0 || "$ROUTER_RESTART_COUNT" -gt 0 ]]; then
198
      log info "connectivity restored via $PING_TARGET; resetting failure state"
199
    fi
200
    reset_failure_state
201
    save_state
202
    return
203
  fi
204

205
  if [[ "$FIRST_FAILURE_EPOCH" -le 0 ]]; then
206
    FIRST_FAILURE_EPOCH="$now"
207
  fi
208
  FAIL_COUNT=$((FAIL_COUNT + 1))
209

210
  local failure_minutes
211
  failure_minutes="$(elapsed_minutes_since "$FIRST_FAILURE_EPOCH" "$now")"
212

213
  if ping_host "$ROUTER_IP"; then
214
    log warning "connectivity check failed for $PING_TARGET; router $ROUTER_IP is reachable; failure_minutes=$failure_minutes fail_count=$FAIL_COUNT"
215
  else
216
    log warning "connectivity check failed for $PING_TARGET; router $ROUTER_IP is not reachable; failure_minutes=$failure_minutes fail_count=$FAIL_COUNT"
217
  fi
218

219
  maybe_recover_router "$now" "$failure_minutes"
220
  maybe_reboot_pve "$now"
221
  save_state
222
}
223

224
main() {
225
  while [[ $# -gt 0 ]]; do
226
    case "$1" in
227
      --dry-run)
228
        DRY_RUN=1
229
        shift
230
        ;;
231
      *)
232
        echo "unknown argument: $1" >&2
233
        exit 2
234
        ;;
235
    esac
236
  done
237

238
  load_config
239
  exec 9>"$LOCK_FILE"
240
  if ! flock -n 9; then
241
    log warning "another watchdog run is active; skipping"
242
    exit 0
243
  fi
244

245
  run_once
246
}
247

248
main "$@"

展开收起

授权：

Terminal window

1
chmod 0755 /usr/local/sbin/pve-net-watchdog

systemd service 和 timer#

创建 /etc/systemd/system/pve-net-watchdog.service：

1
[Unit]
2
Description=PVE network watchdog
3
Wants=network-online.target
4
After=network-online.target
5

6
[Service]
7
Type=oneshot
8
EnvironmentFile=-/etc/default/pve-net-watchdog
9
ExecStart=/usr/local/sbin/pve-net-watchdog

创建 /etc/systemd/system/pve-net-watchdog.timer：

1
[Unit]
2
Description=Run PVE network watchdog every minute
3

4
[Timer]
5
OnBootSec=2min
6
OnUnitActiveSec=60s
7
AccuracySec=5s
8
Persistent=true
9
Unit=pve-net-watchdog.service
10

11
[Install]
12
WantedBy=timers.target

启用：

Terminal window

1
systemctl daemon-reload
2
systemctl enable --now pve-net-watchdog.timer

验证#

先做语法检查：

Terminal window

1
bash -n /usr/local/sbin/pve-net-watchdog

再做一次 dry-run：

Terminal window

1
/usr/local/sbin/pve-net-watchdog --dry-run

确认 timer 状态：

Terminal window

1
systemctl is-enabled pve-net-watchdog.timer
2
systemctl is-active pve-net-watchdog.timer
3
systemctl list-timers --all pve-net-watchdog.timer --no-pager

查看日志：

Terminal window

1
journalctl -u pve-net-watchdog.service -n 100 --no-pager

实时跟踪：

Terminal window

1
journalctl -u pve-net-watchdog.service -f

查看状态文件：

Terminal window

1
cat /var/lib/pve-net-watchdog/state

正常情况下状态大概是：

Terminal window

1
FAIL_COUNT=0
2
FIRST_FAILURE_EPOCH=0
3
ROUTER_RESTART_COUNT=0
4
LAST_ROUTER_RESTART_EPOCH=0
5
LAST_PVE_REBOOT_EPOCH=0

触发流程#

按上面的配置，完整动作链路是：

每分钟 ping 一次 www.baidu.com。
连续失败 5 分钟后，恢复软路由虚拟机。
如果虚拟机已停止，执行 qm start <vmid>。
如果虚拟机正在运行，优先执行 qm reboot <vmid>。
如果软重启失败，再执行 qm reset <vmid>。
每次恢复后等待 10 分钟冷却。
连续恢复软路由 3 次仍然不通，再等待 5 分钟。
如果仍然不通，并且 PVE 重启冷却期已过，执行 systemctl reboot。

这个流程的关键是“逐级升级”：不要把宿主机重启作为第一动作。

临时停用#

如果你要维护网络、升级 PVE、调整虚拟网桥或直通网卡，建议先停掉 timer：

Terminal window

1
systemctl disable --now pve-net-watchdog.timer

维护完成后再启用：

Terminal window

1
systemctl enable --now pve-net-watchdog.timer

生产环境建议#

自动重启 PVE 是高风险动作，建议至少做三点保护：

初次部署先把 ENABLE_PVE_REBOOT=0，观察几天日志。
确认软路由 VM 能稳定启动，并且设置了开机自启。
如果 PVE 上有备份、迁移、快照、升级任务，脚本里应该额外检查这些任务，避免在关键操作中重启宿主机。

如果你还有另一台常在线设备，例如 NAS、旁路由、树莓派，也可以把外部探针加入判断。由 PVE 自己判断“我是否断网”已经能解决大部分场景，但外部视角更适合判断“整个家庭网络是否真的不可达”。

这套脚本不复杂，真正重要的是阈值和冷却策略。网络自愈系统宁可慢一点，也不要因为一两次临时丢包就把关键设备重启掉。

命令行自动登录 JumpServer：SSH、TOTP 与密码提示的自动化

Fri, 05 Jun 2026 00:00:00 GMT

JumpServer 的命令行登录通常不只是一次 ssh：先走 SSH 公钥或密码认证，再输入 MFA 动态验证码，某些环境还会额外进入 password prompt。每次手动复制验证码既慢，也容易在多个跳板机之间输错。

我的处理方式是把流程拆成两层：

zsh 负责准备凭据：检查依赖、解锁密码库、读取 TOTP seed、必要时读取 SSH password。
expect 负责终端交互：启动 ssh，遇到 password prompt 就发送密码，遇到 MFA Code prompt 就生成并发送当前 TOTP，最后把终端控制权交还给用户。

这样做的好处是：敏感信息不写进脚本，交互逻辑也不会和密码管理器逻辑混在一起。

前置条件#

这套方案依赖以下工具：

Terminal window

1
brew install expect oath-toolkit jq

还需要安装并登录 Bitwarden CLI：

Terminal window

1
bw login

如果使用 Bitwarden SSH Agent，还要确保 SSH 配置里显式指定 agent 和 key，避免 agent 里多把 key 被逐个尝试，提前触发服务端的 MaxAuthTries。

1
Host jumpserver-idc
2
  User <ssh-user>
3
  HostName <jumpserver-host>
4
  Port <ssh-port>
5
  IdentityAgent <bitwarden-ssh-agent-sock>
6
  IdentityFile ~/.ssh/<jumpserver-key>.pub
7
  IdentitiesOnly yes

注意：这里的 <jumpserver-host>、<ssh-user>、<jumpserver-key> 都应该替换成你自己的环境值，不要把真实地址和账号提交到公开仓库。

命令入口#

最终希望日常只执行一个函数：

Terminal window

1
ssh_jumpserver_idc() {
2
  _ssh_jumpserver <ssh-host-alias> <bitwarden-item-name> --bw-password
3
}

这个入口有三个参数含义：

<ssh-host-alias>：本机 ~/.ssh/config 里的 Host 别名。
<bitwarden-item-name>：Bitwarden 中保存 JumpServer 登录信息的 item 名。
-bw-password：表示 SSH password 也从 Bitwarden item 中读取；如果目标环境只需要公钥和 MFA，可以不传。

zsh 负责读取凭据#

核心函数先检查依赖，再处理 Bitwarden 状态：

Terminal window

1
_ssh_jumpserver() {
2
  local target="$1"
3
  local totp_item="$2"
4
  local password_source="${3:-}"
5
  local item_json totp_secret target_passwd
6

7
  if [[ -z "$target" || -z "$totp_item" ]]; then
8
    echo "usage: _ssh_jumpserver <ssh-target> <bitwarden-item> [--bw-password|password]"
9
    return 2
10
  fi
11

12
  for cmd in expect oathtool bw jq; do
13
    if ! command -v "$cmd" >/dev/null 2>&1; then
14
      echo "$cmd not found"
15
      return 127
16
    fi
17
  done
18

19
  case "$(bw status 2>/dev/null)" in
20
    *'"status":"unlocked"'*) ;;
21
    *'"status":"locked"'*)
22
      bw_unlock || return $?
23
      ;;
24
    *'"status":"unauthenticated"'*)
25
      echo "Bitwarden CLI is not logged in. Run bw login first."
26
      return 1
27
      ;;
28
    *)
29
      echo "Unexpected Bitwarden status. Run bw status to inspect it."
30
      return 1
31
      ;;
32
  esac
33

34
  bw sync --force >/dev/null 2>&1 || :
35

36
  item_json="$(_bw_get_exact_item "$totp_item")" || return $?
37
  totp_secret="$(_bw_totp_secret_from_json "$totp_item" "$item_json")" || return $?
38

39
  if [[ "$password_source" == "--bw-password" ]]; then
40
    target_passwd="$(_bw_password_from_json "$totp_item" "$item_json")" || return $?
41
  else
42
    target_passwd="$password_source"
43
  fi
44

45
  _ssh_jumpserver_expect "$target" "$totp_secret" "$target_passwd"
46
}

这里有几个关键点：

bw sync --force：避免桌面端刚更新了密码，但 CLI 还读到旧缓存。
_bw_get_exact_item：校验返回的 item name 必须完全等于请求值，避免 Bitwarden 的模糊匹配拿错条目。
_bw_totp_secret_from_json：读取 .login.totp，同时兼容 otpauth://...?secret=... 这种格式。
_bw_password_from_json：优先读 .login.password，也可以兼容自定义字段。

避免 Bitwarden 模糊匹配#

bw get item <name> 在某些情况下可能返回相近名称的条目。对 JumpServer 来说，这很危险：拿错 TOTP seed 会生成错误验证码，而且错误不一定直观。

因此我会加一层精确校验：

Terminal window

1
_bw_get_exact_item() {
2
  local item="$1"
3
  local item_json item_name
4

5
  item_json="$(bw get item "$item")" || return $?
6
  item_name="$(jq -r '.name // empty' <<<"$item_json")" || return $?
7

8
  if [[ "$item_name" != "$item" ]]; then
9
    echo "Bitwarden item mismatch: requested '$item', got '$item_name'" >&2
10
    return 1
11
  fi
12

13
  print -r -- "$item_json"
14
}

从 Bitwarden 解析 TOTP seed#

Bitwarden 的 TOTP 字段可能是纯 base32 secret，也可能是 otpauth:// URL。下面的 helper 会统一转成 oathtool 可用的 base32 seed：

Terminal window

1
_bw_totp_secret_from_json() {
2
  local item="$1"
3
  local item_json="$2"
4
  local totp secret
5

6
  totp="$(jq -r '.login.totp // empty' <<<"$item_json")" || return $?
7
  if [[ -z "$totp" || "$totp" == "null" ]]; then
8
    echo "Bitwarden item has no login.totp: $item" >&2
9
    return 1
10
  fi
11

12
  if [[ "$totp" == otpauth://* ]]; then
13
    secret="${totp#*secret=}"
14
    secret="${secret%%&*}"
15
    secret="$(printf '%b' "${secret//\\%/\\\\x}")"
16
  else
17
    secret="$totp"
18
  fi
19

20
  print -r -- "${secret//[[:space:]]/}"
21
}

脚本里不要保存当前 6 位验证码。验证码是短时值，应该在 expect 收到 Code prompt 的那一刻再生成，避免过期。

expect 只处理 SSH 交互#

expect 部分保持简单：只做终端自动输入，不直接访问 Bitwarden。

Terminal window

1
_ssh_jumpserver_expect() {
2
  local target="$1"
3
  local totp_secret="$2"
4
  local target_passwd="${3:-}"
5
  local expect_script expect_status
6

7
  expect_script="$(mktemp "${TMPDIR:-/tmp}/ssh-jumpserver.XXXXXX")" || return $?
8
  command cat > "$expect_script" <<'EXPECT'
9
set target [lindex $argv 0]
10
set secret [lindex $argv 1]
11
set target_passwd [lindex $argv 2]
12

13
set timeout 30
14
spawn ssh $target
15

16
expect {
17
  "*password:*" {
18
    if {$target_passwd eq ""} {
19
      send_error "SSH password prompt received, but no password was provided\\n"
20
      exit 1
21
    }
22
    send -- "$target_passwd\\r"
23
    exp_continue
24
  }
25

26
  "*Code]:" {
27
    set kdauth [exec oathtool --totp --base32 $secret]
28
    send -- "$kdauth\\r"
29
  }
30

31
  timeout {
32
    send_error "Timed out waiting for SSH jumpserver prompt\\n"
33
    exit 1
34
  }
35
}
36

37
interact
38
EXPECT
39

40
  expect -f "$expect_script" -- "$target" "$totp_secret" "$target_passwd"
41
  expect_status=$?
42
  rm -f "$expect_script"
43
  return $expect_status
44
}

这里不要用 expect -f - 直接喂 heredoc。那样 expect 的 stdin 会被脚本内容占用，后面的 interact 可能无法继续接管当前终端，表现为发送验证码后会话直接退出。

password 的处理#

有些 JumpServer 环境会先走公钥，再进入 SSH password prompt。不要把 password 硬编码在 .zshrc 里，可以从 Bitwarden item 的 .login.password 或自定义字段读取：

1
_bw_password_from_json() {
2
  local item="$1"
3
  local item_json="$2"
4
  local password
5

6
  password="$(jq -r '
7
    [
8
      .login.password,
9
      (
10
        .fields[]?
11
        | select(
12
            (.name // "" | ascii_downcase) as $name
13
            | [
14
                "password",
15
                "passwd",
16
                "ssh_password",
17
                "ssh-password",
18
                "target_password",
19
                "target-password",
20
                "jumpserver_password",
21
                "jumpserver-password"
22
              ]
23
            | index($name)
24
          )
25
        | .value
26
      )
27
    ]
28
    | map(select(. != null and . != ""))
29
    | first // empty
30
  ' <<<"$item_json")" || return $?
31

32
  if [[ -z "$password" || "$password" == "null" ]]; then
33
    password="$(bw get password "$item" 2>/dev/null)" || password=""
34
  fi
35

36
  if [[ -z "$password" || "$password" == "null" ]]; then
37
    echo "Bitwarden item has no supported password field: $item" >&2
38
    return 1
39
  fi
40

41
  print -r -- "$password"
42
}

常见问题#

too many authentication failures#

如果 Bitwarden SSH Agent 里有多把 key，OpenSSH 默认可能逐个尝试。服务端在正确 key 被尝试之前就可能达到 MaxAuthTries，返回 too many authentication failures。

解决方法是在对应 Host 中配置：

1
IdentitiesOnly yes
2
IdentityFile ~/.ssh/<jumpserver-key>.pub
3
IdentityAgent <bitwarden-ssh-agent-sock>

Code prompt 匹配不到#

不同 JumpServer 版本的 MFA 提示文字可能不同。可以先手动执行一次 ssh <host-alias>，观察提示文本，再调整 expect 分支里的 pattern。

例如：

Terminal window

1
"*Code]:" {
2
  set kdauth [exec oathtool --totp --base32 $secret]
3
  send -- "$kdauth\\r"
4
}

如果你的提示是 Verification code:，就改成对应 pattern。

发送验证码后会话退出#

优先检查是否使用了 expect -f - 加 heredoc。更稳的做法是写入临时 expect 脚本文件，再用 expect -f "$expect_script" 执行，最后 rm -f 清理。

安全边界#

这套方案的原则是：

.zshrc 只保存函数和占位式配置，不保存密码、TOTP seed、真实主机信息。
TOTP seed 和 password 只从密码管理器读取。
expect 不直接调用密码管理器，减少交互层复杂度。
SSH config 使用 Host 别名，文章、脚本示例和公开文档只展示别名和占位符。
调试日志不要上传包含 HostName、User、公钥指纹、验证码、密码库 item 名的原始输出。

最终使用#

配置完成后，日常登录就变成：

Terminal window

1
ssh_jumpserver_idc

命令会自动完成 Bitwarden 同步、TOTP seed 读取、可选 password 读取和 JumpServer MFA 输入。成功后会进入正常 SSH 会话，后续操作仍然由当前终端接管。

git 个人开发工作流

Fri, 27 Mar 2026 00:00:00 GMT

日常工作流#

feature-a 某个上游的基线分支

feature-a-xxx 基于 feature-a 的本地开发分支

多人开发时，优先在 feature-a-xxx 上提交自己的代码，定时通过 rebase 同步上游 feature-a 的改动，最后通过 merge 方式把本地的提交合并回上游

1
feature-a
2
----A---------B-------------C-------------------M(merge back)----
3
     \                         \                /
4
      \                         \              /
5
feature-a-xxx
6
      ----a1--------a2----rebase to C----a3---

本地开发时，可能会存在多次临时的提交，建议通过 squash 将本地的提交整理后再 merge 回上游分支

1
feature-a
2
----A---------B---------C---------D-------------------------------M----
3
     \                   \         \                            /
4
      \                   \         \                          /
5
feature-a-xxx
6
      ----a1----a2----rebase----a3----rebase----a4----a5----S(squash)

以下脚本主要添加了两个方便快速操作的实现

grebase 在本地开发分支上快速 rebase 上游分支

gmergebase 将本地开发分支 merge 回上游分支

Terminal window展开收起

1
# ============================================
2
# Git personal branch helpers - safe edition
3
# --------------------------------------------
4
# 约定：
5
#   个人开发分支 = <基线分支>-<个人后缀>
6
#   或         = <基线分支>/<个人后缀>
7
#
8
# 示例：
9
#   feature-a-sephy               -> feature-a
10
#   feature-order-refactor-jack   -> feature-order-refactor
11
#   feature-a/sephy               -> feature-a
12
#
13
# 提供命令：
14
#   gbase                   # 输出自动识别出的基线分支
15
#   gplan                   # 预览 rebase / merge 计划
16
#   grebase                 # 安全 rebase 到 origin/<base>
17
#   grebase --autostash     # 工作区不干净时自动暂存后 rebase
18
#   gmergebase              # merge 当前分支回基线分支，merge 后切回当前分支
19
#   gmergebase --push       # merge 后自动 push
20
#   gmergebase --squash     # squash merge
21
#   gmergebase --push --delete
22
#                          # merge + push + 删除已合并个人分支（本地）
23
# ============================================
24

25
# ---------- 基础输出 ----------
26

27
__gpb_info() { printf '[INFO] %s\n' "$*"; }
28
__gpb_warn() { printf '[WARN] %s\n' "$*" >&2; }
29
__gpb_err()  { printf '[ERR ] %s\n' "$*" >&2; }
30

31
# ---------- 基础判断 ----------
32

33
__gpb_in_git_repo() {
34
  git rev-parse --is-inside-work-tree >/dev/null 2>&1
35
}
36

37
__gpb_current_branch() {
38
  git branch --show-current 2>/dev/null
39
}
40

41
__gpb_branch_exists_local() {
42
  local branch="$1"
43
  git show-ref --verify --quiet "refs/heads/$branch"
44
}
45

46
__gpb_branch_exists_remote() {
47
  local branch="$1"
48
  git show-ref --verify --quiet "refs/remotes/origin/$branch"
49
}
50

51
__gpb_branch_exists_any() {
52
  local branch="$1"
53
  __gpb_branch_exists_local "$branch" || __gpb_branch_exists_remote "$branch"
54
}
55

56
__gpb_all_branches() {
57
  {
58
    git for-each-ref --format='%(refname:short)' refs/heads
59
    git for-each-ref --format='%(refname:short)' refs/remotes/origin
60
  } | sed 's#^origin/##' | grep -v '^HEAD$' | awk '!seen[$0]++'
61
}
62

63
# 工作区是否干净（含 staged / unstaged，忽略 untracked）
64
__gpb_worktree_clean() {
65
  git diff --quiet && git diff --cached --quiet
66
}
67

68
# 是否存在未跟踪文件
69
__gpb_has_untracked() {
70
  [ -n "$(git ls-files --others --exclude-standard)" ]
71
}
72

73
# 当前分支是否有未推送提交
74
__gpb_has_unpushed_commits() {
75
  local current upstream
76
  current="$(__gpb_current_branch)"
77
  upstream="$(git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null)" || return 1
78
  [ -n "$(git rev-list "${upstream}..HEAD" 2>/dev/null)" ]
79
}
80

81
# 当前分支是否落后于上游
82
__gpb_is_behind_upstream() {
83
  local upstream
84
  upstream="$(git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null)" || return 1
85
  [ -n "$(git rev-list "HEAD..${upstream}" 2>/dev/null)" ]
86
}
87

88
# 基线分支是否包含当前开发分支（用于删除前校验）
89
__gpb_is_merged_into() {
90
  local src="$1"
91
  local target="$2"
92
  git merge-base --is-ancestor "$src" "$target"
93
}
94

95
# ---------- 基线推导 ----------
96

97
__gpb_infer_base_branch() {
98
  local current="$1"
99
  local best=""
100
  local candidate=""
101

102
  [ -z "$current" ] && return 1
103

104
  while IFS= read -r candidate; do
105
    [ -z "$candidate" ] && continue
106
    [ "$candidate" = "$current" ] && continue
107

108
    case "$current" in
109
      "$candidate"-*|"$candidate"/*)
110
        if [ ${#candidate} -gt ${#best} ]; then
111
          best="$candidate"
112
        fi
113
        ;;
114
    esac
115
  done < <(__gpb_all_branches)
116

117
  if [ -n "$best" ]; then
118
    printf '%s\n' "$best"
119
    return 0
120
  fi
121

122
  if [[ "$current" == *-* ]]; then
123
    local by_dash="${current%-*}"
124
    if __gpb_branch_exists_any "$by_dash"; then
125
      printf '%s\n' "$by_dash"
126
      return 0
127
    fi
128
  fi
129

130
  if [[ "$current" == */* ]]; then
131
    local by_slash="${current%/*}"
132
    if __gpb_branch_exists_any "$by_slash"; then
133
      printf '%s\n' "$by_slash"
134
      return 0
135
    fi
136
  fi
137

138
  return 1
139
}
140

141
gbase() {
142
  __gpb_in_git_repo || { __gpb_err "Not a git repository"; return 1; }
143

144
  local current base
145
  current="$(__gpb_current_branch)"
146
  [ -n "$current" ] || { __gpb_err "Cannot determine current branch"; return 1; }
147

148
  base="$(__gpb_infer_base_branch "$current")" || {
149
    __gpb_err "Cannot infer base branch from current branch: $current"
150
    return 1
151
  }
152

153
  printf '%s\n' "$base"
154
}
155

156
# ---------- 计划预览 ----------
157

158
gplan() {
159
  __gpb_in_git_repo || { __gpb_err "Not a git repository"; return 1; }
160

161
  local current base
162
  current="$(__gpb_current_branch)"
163
  [ -n "$current" ] || { __gpb_err "Cannot determine current branch"; return 1; }
164

165
  base="$(__gpb_infer_base_branch "$current")" || {
166
    __gpb_err "Cannot infer base branch from current branch: $current"
167
    return 1
168
  }
169

170
  cat <<EOF
171
Current branch : $current
172
Base branch    : $base
173

174
Rebase:
175
  git fetch origin --prune
176
  git rebase origin/$base
177

178
Merge back:
179
  git fetch origin --prune
180
  git checkout $base
181
  git pull --ff-only origin $base
182
  git merge --no-ff $current
183
  git checkout $current
184
EOF
185
}
186

187
# ---------- 安全 rebase ----------
188

189
grebase() {
190
  __gpb_in_git_repo || { __gpb_err "Not a git repository"; return 1; }
191

192
  local autostash=0
193
  while [ $# -gt 0 ]; do
194
    case "$1" in
195
      --autostash) autostash=1 ;;
196
      *)
197
        __gpb_err "Unknown option: $1"
198
        return 1
199
        ;;
200
    esac
201
    shift
202
  done
203

204
  local current base stashed=0 stash_msg
205
  current="$(__gpb_current_branch)"
206
  [ -n "$current" ] || { __gpb_err "Cannot determine current branch"; return 1; }
207

208
  base="$(__gpb_infer_base_branch "$current")" || {
209
    __gpb_err "Cannot infer base branch from current branch: $current"
210
    return 1
211
  }
212

213
  if [ "$current" = "$base" ]; then
214
    __gpb_err "Current branch is already the base branch: $base"
215
    return 1
216
  fi
217

218
  __gpb_info "Current branch : $current"
219
  __gpb_info "Base branch    : $base"
220

221
  if ! __gpb_worktree_clean; then
222
    if [ "$autostash" -eq 1 ]; then
223
      stash_msg="grebase-autostash-$(date +%Y%m%d%H%M%S)"
224
      __gpb_warn "Working tree is dirty, auto stashing changes: $stash_msg"
225
      git stash push -u -m "$stash_msg" >/dev/null || {
226
        __gpb_err "git stash failed"
227
        return 1
228
      }
229
      stashed=1
230
    else
231
      __gpb_err "Working tree is dirty. Commit/stash first, or use: grebase --autostash"
232
      return 1
233
    fi
234
  fi
235

236
  git fetch origin --prune || {
237
    [ "$stashed" -eq 1 ] && __gpb_warn "Your changes are stashed. Restore manually if needed: git stash list"
238
    return 1
239
  }
240

241
  if ! __gpb_branch_exists_remote "$base"; then
242
    __gpb_err "Remote branch origin/$base does not exist"
243
    [ "$stashed" -eq 1 ] && __gpb_warn "Your changes are stashed. Restore manually if needed: git stash pop"
244
    return 1
245
  fi
246

247
  if ! git rebase "origin/$base"; then
248
    __gpb_err "Rebase failed. Resolve conflicts, then run:"
249
    printf '       git add <files>\n'
250
    printf '       git rebase --continue\n'
251
    printf '       # or git rebase --abort\n'
252
    [ "$stashed" -eq 1 ] && __gpb_warn "Auto-stash was created. Apply it after rebase is fully done."
253
    return 1
254
  fi
255

256
  if [ "$stashed" -eq 1 ]; then
257
    __gpb_info "Restoring stashed changes"
258
    if ! git stash pop; then
259
      __gpb_warn "git stash pop had conflicts or was not fully applied. Check: git stash list"
260
      return 1
261
    fi
262
  fi
263

264
  __gpb_info "Rebase completed successfully"
265
}
266

267
# ---------- 安全 merge back ----------
268

269
gmergebase() {
270
  __gpb_in_git_repo || { __gpb_err "Not a git repository"; return 1; }
271

272
  local do_push=0
273
  local do_squash=0
274
  local do_delete=0
275

276
  while [ $# -gt 0 ]; do
277
    case "$1" in
278
      --push) do_push=1 ;;
279
      --squash) do_squash=1 ;;
280
      --delete) do_delete=1 ;;
281
      *)
282
        __gpb_err "Unknown option: $1"
283
        return 1
284
        ;;
285
    esac
286
    shift
287
  done
288

289
  local dev base original_branch
290
  original_branch="$(__gpb_current_branch)"
291
  dev="$original_branch"
292

293
  [ -n "$dev" ] || { __gpb_err "Cannot determine current branch"; return 1; }
294

295
  base="$(__gpb_infer_base_branch "$dev")" || {
296
    __gpb_err "Cannot infer base branch from current branch: $dev"
297
    return 1
298
  }
299

300
  if [ "$dev" = "$base" ]; then
301
    __gpb_err "Current branch is already the base branch: $base"
302
    return 1
303
  fi
304

305
  __gpb_info "Dev branch     : $dev"
306
  __gpb_info "Base branch    : $base"
307

308
  if ! __gpb_worktree_clean; then
309
    __gpb_err "Working tree is dirty. Please commit/stash first."
310
    return 1
311
  fi
312

313
  if __gpb_has_untracked; then
314
    __gpb_warn "There are untracked files in working tree"
315
  fi
316

317
  git fetch origin --prune || return 1
318

319
  if __gpb_has_unpushed_commits; then
320
    __gpb_warn "Current branch has unpushed commits"
321
    __gpb_warn "Consider pushing first for backup"
322
  fi
323

324
  if __gpb_is_behind_upstream; then
325
    __gpb_warn "Current branch is behind its upstream"
326
  fi
327

328
  if ! __gpb_branch_exists_any "$base"; then
329
    __gpb_err "Base branch does not exist locally/remotely: $base"
330
    return 1
331
  fi
332

333
  git checkout "$base" || return 1
334

335
  if __gpb_branch_exists_remote "$base"; then
336
    git pull --ff-only origin "$base" || {
337
      __gpb_err "Failed to fast-forward local $base from origin/$base"
338
      git checkout "$original_branch" >/dev/null 2>&1
339
      return 1
340
    }
341
  fi
342

343
  if [ "$do_squash" -eq 1 ]; then
344
    __gpb_info "Running squash merge"
345
    git merge --squash "$dev" || {
346
      __gpb_err "Squash merge failed"
347
      git checkout "$original_branch" >/dev/null 2>&1
348
      return 1
349
    }
350

351
    __gpb_warn "Squash merge staged changes only; commit manually:"
352
    printf '       git commit -m "merge %s into %s"\n' "$dev" "$base"
353

354
    git checkout "$original_branch" >/dev/null 2>&1 || true
355
    return 0
356
  fi
357

358
  __gpb_info "Running merge --no-ff"
359
  git merge --no-ff "$dev" || {
360
    __gpb_err "Merge failed. Resolve conflicts, then commit."
361
    __gpb_warn "After merge is completed manually, you may switch back with: git checkout $original_branch"
362
    return 1
363
  }
364

365
  if [ "$do_push" -eq 1 ]; then
366
    __gpb_info "Pushing $base to origin"
367
    git push origin "$base" || {
368
      __gpb_err "Push failed"
369
      git checkout "$original_branch" >/dev/null 2>&1 || true
370
      return 1
371
    }
372
  fi
373

374
  if [ "$do_delete" -eq 1 ]; then
375
    if __gpb_is_merged_into "$dev" "$base"; then
376
      __gpb_info "Deleting merged local branch: $dev"
377
      git branch -d "$dev" || {
378
        __gpb_warn "Failed to delete local branch: $dev"
379
      }
380
    else
381
      __gpb_warn "Skip deleting $dev because it is not fully merged into $base"
382
    fi
383
  fi
384

385
  __gpb_info "Switching back to original branch: $original_branch"
386
  git checkout "$original_branch" >/dev/null 2>&1 || {
387
    __gpb_warn "Failed to switch back automatically. Please run: git checkout $original_branch"
388
    return 0
389
  }
390

391
  __gpb_info "Merge back completed successfully"
392
}
393

394
# ---------- 可选快捷别名 ----------
395
alias grb='grebase'
396
alias gmb='gmergebase'

展开收起

使用 docker-atrust 连入深信服

Fri, 03 Oct 2025 00:00:00 GMT

公司为了加强网络安全，采购了深信服的零信任（atrust）作为以前openvpn的代替方案，接入方需要使用 atrust 客户端进行接入。某天突然在 v2ex 上发现了基于 docker 和梯子软件结合，免安装客户端实现接入的方式，然后一直没有验证成功，今天趁着国庆假期，再次尝试，终于成功。实际上还是担心 atrust 会对本地电脑进行监控，感觉安装在 docker 容器中，可以最大程度的限制其能力。

网络拓扑#

![[image 2.png]]

docker-atrust 配置#

1
services:
2
  atrust:
3
    image: hagb/docker-atrust
4
    container_name: atrust
5
    environment:
6
      - PASSWORD=123456 # 密码按照实际情况填写
7
      - URLWIN=1
8
      - TZ=Asia/Shanghai
9
    volumes:
10
      - $HOME/data/atrust:/root
11
    ports:
12
      - "5901:5901"
13
      - "1080:1080" # socks5 端口
14
      - "8888:8888" # http 代理端口
15
      - "54631:54631"
16
    devices:
17
      - /dev/net/tun
18
    cap_add:
19
      - NET_ADMIN
20
    sysctls:
21
      - net.ipv4.conf.default.route_localnet=1
22
    restart: unless-stopped

通过 vnc 登录#

![[image 3.png]]

连入后，和普通 atrust 操作一样，输入服务器地址、用户名密码完成登录，这里比较重要的是配置连接服务器url时，域名需要在下面的 mihomo 配置中进行特殊配置（下面会讲到）

mihomo 配置#

因为公司使用了私有域名，docker-atrust 连入成功后，atrust 会下发 dns 解析规则到容器，需要将这部分 dns 指定到 mihomo 的 hosts 中

1
hosts:
2
  # 公司dns
3
  mysql-xxx.xxx.io:   47.96.xxx.xxx # 公司ip
4
  mysql8-test.xxx.io:   172.16.xxx.xxx # 公司内网ip

增加 socks5 代理节点

1
proxies:
2
  - name: ATRUST
3
    type: socks5
4
    server: 192.168.50.4 # 指向 192.168.50.4 宿主机
5
    port: 1080 #

增加代理规则

1
proxy-groups:
2
  - { name: 公司线路, type: select, proxies: [DIRECT, ATRUST] } # 这里考虑本地 mac 上的clash verge配置，在公司时直连

代理规则配置

1
rules:
2
  # 公司atrust
3
  - "DOMAIN,atrust.xxx.com,DIRECT" # 这里指定 atrust 连接用的域名走直连
4
  - "DOMAIN-SUFFIX,xxx.io,公司线路"
5
  - "DOMAIN-SUFFIX,xxx.com,公司线路"
6
  - "IP-CIDR,47.96.xxx.xxx/32,公司线路,no-resolve" # 47.96.xxx.xxx 是 atrust.xxx.com 的 ip

openwrt 路由配置#

在家调试java应用时，发现无法访问公司的 172.16.6.0/24 和 172.16.7.0/24 网段的 dubbo 服务，需要手动添加两条路由规则到路由表中

Terminal window

1
ip route add 172.16.6.0/24 dev nikki
2
ip route add 172.16.7.0/24 dev nikki

📎 参考文章#

mihomo 使用

Tue, 30 Sep 2025 00:00:00 GMT

mac 客户端#

原先用的 clash verge 不太稳定，现在已经切换到 clash party

image 4

常用 mihomo 配置文件#

针对 chatGPT 做了分流设置

1
mixed-port: 7890
2
allow-lan: true
3
mode: rule
4
log-level: info
5
external-controller: "127.0.0.1:9090"
6
unified-delay: true
7
ipv6: false
8

9
udp: true
10

11
# TUN 模式配置
12
tun:
13
  enable: false
14
  stack: mixed
15
  auto-route: true
16
  auto-detect-interface: true
17

18
  # 排除 Tailscale 相关的网络接口，防止接管其流量
19
  exclude-interface:
20
    - utun17   # Tailscale 主接口（当前实际接口，IPv4: 10.250.0.3）
21
    - utun18   # 保留以防接口名称变化
22
    - Tailscale # 设备名称（如果系统识别为此名）
23
    - eth0
24
    - lo
25
    - dheth0
26
    - tailscale0 # tailscale
27
  dns-hijack:
28
    - any:53
29
    - tcp://any:53
30
  # strict-route: true
31
  # 路由排除地址 - 这些地址的流量不会通过 TUN 接口
32
  route-exclude-address:
33
    - 139.224.194.63/32 # 阿里云 SSH 服务器
34
    - 127.0.0.0/8      # 本地回环
35
    - 169.254.0.0/16   # 链路本地地址
36
    - 192.168.0.0/16   # 私有网络 C 类（包含虚拟机网段 192.168.139.0/23）
37
    - 192.168.139.0/23 # 虚拟机桥接网段 (bridge100) - 显式声明
38
    - 172.28.0.0/16    # 阿里云网段
39
    # - 172.16.0.0/16    # 私有网络 B 类（包含主网络 172.16.99.0/24 和公司网段 172.16.6.0/24）
40
    - 10.0.0.0/8       # 私有网络 A 类
41

42
    - 10.250.0.3/32       # utun17 接口的特定 IPv4 地址（Tailscale 当前）
43
    - 100.64.0.0/10       # Tailscale 默认的共享 IPv4 地址段
44
    - fd7a:115c:a1e0::/48 # Tailscale 的 IPv6 唯一本地地址段 (ULA)
45
    - fdfe:dcba:9876::/126 # utun1500 上的另一 IPv6 ULA 段
46

47
  # 添加路由包含地址 - 让特定IP使用TUN
48
  # 注意：这些 IP 会强制走 TUN，即使在 route-exclude-address 中
49
  # 如果选择 DIRECT 代理，源 IP 会是 fake-ip (198.18.0.1)，可能导致连接失败
50
  # 只有通过 SSH 隧道代理时才应该让这些 IP 走 TUN
51
  # 移除 172.16.6.0/24，因为公司内网需要通过代理访问，不应该走 TUN（避免 fake-ip 问题）
52
  route-include-address:
53
    - 172.16.6.0/24
54
    - 172.16.7.0/24
55

56
hosts:
57
  nacos-headless: 10.105.105.250 # 内网特殊配置
58
  external-mysql: 172.16.6.23 # 内网特殊配置
59
  external-redis: 172.16.6.23 # 内网特殊配置
60

61
# GeoData配置
62
geodata-mode: true
63
geodata-loader: memconservative
64
geo-auto-update: true
65
geo-update-interval: 24
66

67
# GeoSite和GeoIP数据文件路径
68
geox-url:
69
  geoip: "https://cdn.jsdelivr.net/gh/Loyalsoldier/v2ray-rules-dat@release/geoip.dat"
70
  geosite: "https://cdn.jsdelivr.net/gh/Loyalsoldier/v2ray-rules-dat@release/geosite.dat"
71

72
dns:
73
  enable: true
74
  listen: "127.0.0.1:1053"
75
  ipv6: false
76
  prefer-h3: true
77
  enhanced-mode: fake-ip
78
  fake-ip-range: 198.18.0.1/16
79
  fake-ip-filter-mode: blacklist
80
  fake-ip-filter:
81
    # 某些不需要 fake-ip 的域名
82
    - 'r-bp17ikdtbmppmr51cqpd.redis.rds.aliyuncs.com'
83
    - 'atrust.qnvip.com'
84
    - '.lan'
85

86

87
  # 默认DNS服务器
88
  nameserver:
89
    - 223.5.5.5
90
    - https://dns.alidns.com/dns-query
91
    - https://doh.pub/dns-query
92
    - system # 添加系统DNS作为后备
93

94
  # 代理服务器DNS
95
  # proxy-server-nameserver:
96
  #   - https://1.1.1.1/dns-query
97
  #   - https://dns.google/dns-query
98

99
  # 基于GEOSITE的DNS分流策略
100
  nameserver-policy:
101
    # 广告域名禁止解析
102
    "geosite:category-ads-all":
103
      - rcode://name_error  # 推荐：告诉应用域名不存在
104

105
    # 公司 dns
106
    "+.qnvip.io":
107
      - 172.16.7.1
108
    "+.qnvip.com":
109
      - 172.16.7.1
110

111
  # 备用DNS
112
  fallback:
113
    - https://101.101.101.101/dns-query
114
    - https://1.1.1.1/dns-query
115
    - https://8.8.8.8/dns-query
116
    - https://dns.alidns.com/dns-query
117
    - https://doh.pub/dns-query
118
    - tls://1.1.1.1
119
    - tls://8.8.8.8
120

121
  # 备用DNS触发条件
122
  fallback-filter:
123
    geoip: true
124
    geosite:
125
      - gfw
126
    ipcidr:
127
      - 224.0.0.0/4
128
      - 240.0.0.0/4
129

130
# 流量嗅探配置
131
sniffer:
132
  enable: true
133
  force-dns-mapping: true
134
  parse-pure-ip: true
135
  sniff:
136
    HTTP:
137
      ports: [80, 8080-8880]
138
      override-destination: true
139
    TLS:
140
      ports: [443, 8443]
141
    QUIC:
142
      ports: [443, 8443]
143

144
proxies:
145
  # 把下面的代理节点换成你们自己的
146
  - name: VPS1
147
    type: vless
148
    server: your_server
149
    port: 443
150
    uuid: yyy
151
    network: tcp
152
    tls: true
153
    udp: true
154
    flow: xtls-rprx-vision
155
    servername: itunes.apple.com # REALITY servername
156
    reality-opts:
157
      public-key: zzz
158
      short-id: www # optional
159
    client-fingerprint: chrome # cannot be empty
160

161
  # docker-atrust
162
  - name: QNVIP_ATRUST_VIA_SSH
163
    type: vless
164
    server: your_server
165
    port: 443
166
    uuid: yyy
167
    network: tcp
168
    tls: true
169
    udp: true
170
    flow: xtls-rprx-vision
171
    servername: itunes.apple.com # REALITY servername
172
    reality-opts:
173
      public-key: zzz
174
      short-id: www # optional
175
    client-fingerprint: chrome # cannot be empty
176

177
proxy-groups:
178
  - {
179
      name: ♻️ 自动选择,
180
      type: url-test,
181
      proxies: [VPS1],
182
      url: "http://www.gstatic.com/generate_204",
183
      interval: 300,
184
      tolerance: 50,
185
    }
186
  - { name: 代理节点选择, type: select, proxies: [♻️ 自动选择, VPS1, DIRECT] }
187
  - { name: Apple服务分流, type: select, proxies: [DIRECT, 代理节点选择] }
188
  - { name: 微软服务分流, type: select, proxies: [DIRECT, 代理节点选择, VPS1] }
189
  - { name: ChatGPT分流, type: select, proxies: [代理节点选择, VPS1] }
190
  - {
191
      name: 故障转移,
192
      type: fallback,
193
      proxies: [VPS1],
194
      url: "http://www.gstatic.com/generate_204",
195
      interval: 300,
196
    }
197
  - { name: 拒绝链接, type: select, proxies: [REJECT] }
198
  - { name: 青年优品, type: select, proxies: [DIRECT, QNVIP_ATRUST_VIA_SSH] }
199

200
# https://github.com/Loyalsoldier/v2ray-rules-dat
201
# https://github.com/v2fly/domain-list-community/tree/master/data
202
# https://geoviewer.aloxaf.com/
203
rules:
204
  # Tailscale 域名直连
205
  - "DOMAIN-SUFFIX,ts.net,DIRECT"
206
  - "DOMAIN-SUFFIX,tailscale.com,DIRECT"
207
  - "DOMAIN-SUFFIX,tailscale.io,DIRECT"
208

209
  # 拒绝连接规则
210
  - "DOMAIN-SUFFIX,sparklabs.com,REJECT"
211

212
  - "DOMAIN,sbp-7lq0iu34t4bm5h75.supabase.opentrust.net,阿里云RDS,no-resolve"
213
  - "DOMAIN,sbp-7lq0iu34t4bm5h75.supabase-vpc.opentrust.net,阿里云RDS,no-resolve"
214
    # 阿里云rds
215
  - "DOMAIN,rm-uf682q4z7gr9zbnj4.mysql.rds.aliyuncs.com,阿里云RDS,no-resolve"
216

217

218
  # 代理服务器直连规则 - 避免循环代理
219
  - "DOMAIN,s.nb30.cn,DIRECT"
220
  - "DOMAIN,xray.qyinf.com,DIRECT"
221

222
  # 公司atrust - 保持现有的青年优品代理配置
223
  - "DOMAIN,atrust.qnvip.com,DIRECT"
224
  - "DOMAIN,imap.qnvip.com,DIRECT"
225
  - "DOMAIN,smtp.qnvip.com,DIRECT"
226
  - "DOMAIN,git.qncentury.com,青年优品,no-resolve"
227
  - "DOMAIN-SUFFIX,qnvipmall.com,青年优品,no-resolve"
228
  - "DOMAIN-SUFFIX,qnvip.io,青年优品,no-resolve"
229
  - "DOMAIN-SUFFIX,qnvip.com,青年优品,no-resolve"
230
  - "DOMAIN,rm-bp13dk19dcrn4mj55do.mysql.rds.aliyuncs.com,青年优品,no-resolve"
231
  - "DOMAIN,r-bp17ikdtbmppmr51cqpd.redis.rds.aliyuncs.com,青年优品,no-resolve"
232
  - "IP-CIDR,23.144.92.29/32,DIRECT" # VPS3
233
  - "IP-CIDR,43.130.42.143/32,DIRECT" # VPS2
234
  - "IP-CIDR,74.48.119.198/32,DIRECT" # VPS1
235
  - "IP-CIDR,172.16.7.0/24,青年优品,no-resolve"
236
  - "IP-CIDR,172.16.6.0/24,青年优品,no-resolve"
237
  - "IP-CIDR,47.96.131.37/32,青年优品,no-resolve"
238

239
  # 特殊DNS服务器通过代理访问
240
  - "IP-CIDR,1.1.1.1/32,代理节点选择,no-resolve"
241
  - "IP-CIDR,8.8.8.8/32,代理节点选择,no-resolve"
242

243
  # 注意：由于启用了 TUN 模式的 route-exclude-address，
244
  # 大部分内网 IP-CIDR 规则已经不需要，因为这些流量不会进入 TUN 接口
245

246
  # https://jomertix.github.io/geofileviewer/
247
  - "geoip,cn,DIRECT"
248

249
  # 个人定制直连规则
250
  - "DOMAIN,injections.adguard.org,DIRECT"
251
  - "DOMAIN,local.adguard.org,DIRECT"
252
  - "DOMAIN-SUFFIX,local,DIRECT"
253

254
  # 游戏
255
  - "geosite,category-games@cn,DIRECT"
256

257
  # Apple服务统一分流规则
258
  - "geosite,apple-cn,Apple服务分流"
259

260
  # ChatGPT/OpenAI服务分流（保留关键规则）
261
  - "geosite,category-ai-!cn,ChatGPT分流"
262

263
  # 常用国外服务（使用GEOSITE规则集）
264
  - "geosite,github,代理节点选择"
265
  - "geosite,microsoft,微软服务分流"
266

267
  # 兜底规则
268
  - "geosite,cn,DIRECT"
269
  - "MATCH,代理节点选择"

展开收起

解析过程#

是

否

是

否

匹配本地域名

匹配中国域名

匹配被墙域名

匹配流媒体域名

匹配广告域名

匹配直连域名

无匹配规则

结果正常

触发fallback条件

DNS查询请求

是否为DNS服务器域名?

default-nameserver

223.5.5.5

是否为代理服务器域名?

proxy-server-nameserver

119.29.29.29

检查nameserver-policy规则

本地DNS

172.16.7.1/2

国内DNS

119.29.29.29

223.5.5.5

海外DoH

1.1.1.1

8.8.8.8

海外DoH

1.1.1.1

dns.google

返回空结果

rcode://success

direct-nameserver

DoH国内服务器

nameserver

8.8.8.8 via PROXY

检查解析结果

返回解析结果

fallback DNS

1.1.1.1

dns.google

是

否

是

否

匹配本地域名

匹配中国域名

匹配被墙域名

匹配流媒体域名

匹配广告域名

匹配直连域名

无匹配规则

结果正常

触发fallback条件

DNS查询请求

是否为DNS服务器域名?

default-nameserver

223.5.5.5

是否为代理服务器域名?

proxy-server-nameserver

119.29.29.29

检查nameserver-policy规则

本地DNS

172.16.7.1/2

国内DNS

119.29.29.29

223.5.5.5

海外DoH

1.1.1.1

8.8.8.8

海外DoH

1.1.1.1

dns.google

返回空结果

rcode://success

direct-nameserver

DoH国内服务器

nameserver

8.8.8.8 via PROXY

检查解析结果

返回解析结果

fallback DNS

1.1.1.1

dns.google

Grafana 配置手册

Sat, 06 Sep 2025 00:00:00 GMT

告警配置#

示例1

image 5

📎 参考文章#

https://grafana.com/docs/grafana/latest/panels-visualizations/query-transform-data/expression-queries/

自建 headscale

Sun, 17 Aug 2025 00:00:00 GMT

headscale 安装配置#

默认 headscale 的 ip 段配置为 100.64.0.0/10, 这个会和阿里云的ECS的默认 dns 冲突，改为 10.99.100.0/24 段

1
---
2
server_url: https://${服务器地址}
3

4
listen_addr: 0.0.0.0:8080
5

6
prefixes:
7
  v4: 10.99.100.0/24
8

9
dns:
10
  magic_dns: false # 关闭dns
11

12
unix_socket: /var/run/headscale/headscale.sock
13
unix_socket_permission: "0770"

headscale 中用户可以理解为租户，不同的租户网络隔离

常用命令#

Terminal window

1
# 创建 api key，用于 rest 调用
2
headscale api create
3
# 查询 api key
4
headscale api ls
5
# 创建用户
6
headscale user create sephy
7
# 查询用户
8
headscale user ls
9
# 显示节点列表
10
headscale nodes ls
11
# 创建 preauthkey 设备加入
12
headscale authkey create --user 1 # 1 为用户
13
# 查看 preauthkey
14
headscale authkey ls --user 1
15
# 删除节点
16
headscale nodes delete -i <id>

也可以通过 headscale-ui 设置 preauth#

image 6

阿里云 tailscale 设置#

阿里云 ECS 默认使用 100.100.2.136/100.100.2.138 作为内部 DNS 解析，会与 tailscale 默认的 iptables 规则冲突

添加 iptables 规则

Terminal window

1
sudo iptables -I INPUT 1 -s 100.100.2.0/24 -j ACCEPT
2

3
# 伪装
4
sudo iptables -t nat -A POSTROUTING -s 10.250.0.0/24 -d 172.28.0.0/16 -j MASQUERADE

开启 ipv4 转发

Terminal window

1
echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/sysctl.conf
2
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/sysctl.conf
3
sudo sysctl -p /etc/sysctl.d/sysctl.conf
4
# or
5
echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/99-sysctl.conf
6
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-sysctl.conf
7
sudo sysctl -p /etc/sysctl.d/99-sysctl.conf

开放 41641 udp 端口

image 7

安装 tailscale

Terminal window

1
sudo apt install tailscale -y

登录到headscale

Terminal window

1
# --login-server 指定自托管的 Headscale 控制服务器地址
2
# --accept-routes=true 启用路由接受功能，允许接收其他节点宣告的路由
3
# --accept-dns=false 禁用 Tailscale 的 DNS 功能，保留本地 DNS 设置
4
# --advertise-routes 向网络宣告本地子网路由，使其他节点可访问该子网
5
# --hostname 设置节点在 Tailscale 网络中的显示名称
6
# --auth-key 使用预认证密钥进行无交互式登录
7
# --reset 重置现有配置并重新初始化
8
# --snat-subnet-routes=false
9
tailscale up --login-server ${服务器地址} \
10
  --accept-routes=true \
11
  --accept-dns=false \
12
  --advertise-routes=172.28.0.0/16 \
13
  --snat-subnet-routes=false
14
  --hostname=aliyun-ecs-1 \
15
  --auth-key=${密钥} \
16
  --reset

—snat-subnet-routes=false 视情况加，阿里云 RDS 如果只允许内网访问，需要 —snat-subnet-routes=true

macos tailscale 设置#

启用命令行

image 8

image 9

通过事先创建好的 preauthkey 加入到网络

Terminal window

1
tailscale login --login-server https://${服务器地址} --auth-key ${密钥}

添加成功后客户显示

image 10

iStoreOS tailscale 设置#

官方 iStore 商店中安装的版本太老，需要在官方地址下载对应版本, 手动更新替换 /usr/sbin/tailscale /usr/sbin/tailscaled

Terminal window

1
cp tailscale /usr/sbin/
2
cp tailscaled /usr/sbin/

ssh 登录后通过以下命令进行登录

Terminal window

1
# --login-server 指定自托管的 Headscale 控制服务器地址
2
# --accept-routes=true 启用路由接受功能，允许接收其他节点宣告的路由
3
# --accept-dns=false 禁用 Tailscale 的 DNS 功能，保留本地 DNS 设置
4
# --advertise-routes 向网络宣告本地子网路由，使其他节点可访问该子网
5
# --hostname 设置节点在 Tailscale 网络中的显示名称
6
# --auth-key 使用预认证密钥进行无交互式登录 通过 docker compose -f docker-compose/docker-compose.vps.yml exec headscale headscale authkey create -u 1 生成
7
# --reset 重置现有配置并重新初始化
8
# --snat-subnet-routes=false
9
tailscale up --login-server 'https://{服务器}' \
10
--accept-routes=true \
11
--accept-dns=false \
12
--advertise-routes=192.168.50.0/24 \
13
--snat-subnet-routes=false \
14
--hostname=iStoreOS \
15
--authkey={密钥} \
16
--reset

📎 参考文章#

Playwright 使用心得

Fri, 08 Aug 2025 00:00:00 GMT

尽量不要使用 sleep 实现等待，改为使用 page.waitForTimeout 方式，原因是 sleep 有可能导致 playwright 相关的监听事件无法触发，例如 page.onResponse 有可能不正常触发
LoadState 的区别

1
public enum LoadState {
2
  LOAD, // The load state indicates that the entire page, including all dependent resources such as images, stylesheets, scripts, and other resources, has finished loading. This state is useful when you want to ensure that all content is fully available before interacting with the page. However, this doesn’t track the ongoing network requests. When the load event is fired, there can still be ongoing network requests in the backend.
3
  DOMCONTENTLOADED, // The domcontentloaded state indicates that the DOMContentLoaded event has fired, meaning the HTML document has been completely loaded and parsed. However, that doesn’t mean that all stylesheets and resources are loaded. This can be considered as the initial page load.
4
  NETWORKIDLE // Network idle state indicates that there is no ongoing network request of at least 500 milliseconds. This state is beneficial for applications that make multiple asynchronous requests, as it ensures that all network activity has been completed before proceeding. This state is mostly useful in single-page applications.
5
}

通过 page.route 可以拦截修改返回结果

1
page.route("**/waybillRoute/skipVerify*", route -> {
2
    try {
3
        APIResponse originalResponse = route.fetch();
4
        String body = originalResponse.text();
5
        // {"traceId":"08b30764b981490da450fd39fc1529a9","timestamp":"2025-08-01
6
        // 13:22:25.449","success":true,"code":0,"message":"请求成功","detailMessage":null,"result":"0","obj":null}
7
        JsonNode jsonNode = JsonUtils.parseJSONObject(body);
8
        if (!jsonNode.get("result").asBoolean()) {
9
            ObjectNode objectNode = (ObjectNode)jsonNode;
10
            objectNode.put("result", true);
11
            body = JsonUtils.toJsonStr(jsonNode);
12
            log.info("自动登录失败，返回结果: {}, 修改后结果: {}", originalResponse.text(), body);
13
        }
14
        route.fulfill(new Route.FulfillOptions().setBody(body).setHeaders(originalResponse.headers()));
15
    } catch (Exception e) {
16
        route.resume();
17
    }
18
});

参考资料#

📌

Understanding Playwright waitforloadstate | BrowserStack

PV、UV 统计方案

Wed, 09 Jul 2025 00:00:00 GMT

业务背景#

近期有个CPS相关的需求，其中有一个功能，是对CPS合作商（partner）的引流效果进行统计分析，需要以合作商的维度，统计PV、UV信息（合作商存在多级，但是不超过3级），数据情况大致如下

QPS，日间峰值360, 夜间低谷80, 均值250
PV 500W，带合作商信息的数据，预计在100W左右
数据实时性要求 5 分钟，准确性要求相对准确

数据收集链路#

C端 → nginx → 后端应用 → kafka → CPS统计服务 → MongoDB

PV 统计实现方案#

基于 redis，后台线程异步从 redis 分桶数据中，获取增量数据，刷入到 mysql 中

redis 中的 key 命名规则 appns:pv:导流商id:日期(yyyyMMdd):时间(HH:mm),每个 key 默认过期时间为 30 分钟，其中时间分桶规则为每5分钟一个桶，示例如下

时间范围（闭区间）	分桶
01:00<00> ~ 01:04<59>	0100
01:05<00> ~ 01:09<59>	0105
…	…
23:55<00> ~ 23:59<59>	2355

消费数据时，保存到 mongo 后，计算得到对应的 key 集合中，value 使用 redis 的 INCR 自增，并将 key 保存到一个特殊的 app:pv:keys 中(zset 结构)，score 为 yyyyMMddHHmm
通过定时调度，每5分钟扫描 app:pv:keys 中分数 < 5分钟之前score 的 keys，遍历每个key，将对应的增量数据，刷入到数据库中，并从 app:pv:keys 删除（同时删除对应的 key）
增加兜底调度，每天凌晨时刻，基于 MongoDB 中的数据重新统计前一天每个 partner 的 PV，更新到数据库中

方案优势

避免频繁的更新 mysql（每次pv + 1 都更新完全没必要）；
通过时间分桶方式，等到每个分桶中的增量完全固定之后，再将桶内的数据加回到数据库，降低代码实现的复杂度（否则需要考虑对桶操作的加锁）
每日兜底的重新统计任务，可以保证统计的数据准确性

后续优化方案

mongo写入改为批量，INCR 前在内存中先小批次聚合一下，减少 io 次数
如果 parnter 层级过多，子 partner 比较多时，考虑同时聚合后更新父级partner 的 PV

UV 统计实现方案#

如果用户张三，通过 partnerA 的引流链接进入到应用中，partnerA 的 UV + 1, 相同一天，张三又通过 partnerB 的引流链接进入应用时，partnerB 的 UV + 1

技术方案考虑使用 redis 的 HyperLogLog 实现，实现细节如下

消费数据时，保存到 mongo 后，计算对应事件的 key，格式为 appns:uv:导流商id:日期(yyyyMMdd)，使用 PFADD KEY VALUE 命令，将对应的 openid 添加到指定的 HyperLogLog 中, 同时将 key 保存到 app:uv:keys:yyyyMMdd 集合中（只用redis的list）
定时调度，每5分钟扫描当日的 app:uv:keys:yyyyMMdd，遍历其中的key，再次将数据刷回 mysql
增加兜底调度，每天凌晨时刻，基于 MongoDB 中的数据重新统计前一天每个 partner 的 UV，更新到数据库中，同时清理昨日的 app:uv:keys:yyyyMMdd

方案优势

较与 set 或者 bitmap 实现形式，HyperLogLog 更加节省内存，标准的 0.81% 误差业务上能接受

后续优化方案

考虑在应用内存中，积攒一小批后，通过 PFADD key element1 element2 element3 ... elementN 去操作 HyperLogLog

📎 参考文章#

Spring Cloud Gateway 流量复制

Wed, 02 Jul 2025 00:00:00 GMT

📌

某些业务合作方，只支持配置一个回调地址，针对开发、测试、生产等多套环境，需要用到流量复制功能，这里记录一个通过 Spring Cloud Gateway 的流量复制功能实现（实际上基于 nginx 或其他轮子也能实现目的）

功能逻辑#

否

是

异步

流式

请求到达

有请求体?

立即异步镜像

装饰请求流

继续主请求

主请求完成

镜像执行

边读边镜像

否

是

异步

流式

请求到达

有请求体?

立即异步镜像

装饰请求流

继续主请求

主请求完成

镜像执行

边读边镜像

配置示例#

1
spring:
2
  cloud:
3
    gateway:
4
      routes:
5
        - id: example-route
6
          uri: http://main-service
7
          filters:
8
            - name: TrafficMirroring
9
              args:
10
                mirroredRoutes:
11
                  - http://mirror-service-1
12
                  - http://mirror-service-2
13
                timeoutSeconds: 3

功能特色#

立即返回：主请求不等待镜像完成
独立线程池：镜像使用专用线程，不占用网关核心线程
流式处理：大请求体不需要完整加载到内存
错误隔离：镜像失败绝不影响主业务
资源限制：配置超时和大小限制防止资源泄露
这样的实现确保了镜像流量零阻塞主链路请求，同时保持了镜像功能的完整性和可靠性。

核心代码#

1
import java.net.URI;
2
import java.net.URLDecoder;
3
import java.net.URLEncoder;
4
import java.nio.charset.StandardCharsets;
5
import java.time.Duration;
6
import java.util.List;
7
import java.util.concurrent.ExecutorService;
8
import java.util.concurrent.Executors;
9

10
import org.springframework.beans.factory.DisposableBean;
11
import org.springframework.cloud.gateway.filter.GatewayFilter;
12
import org.springframework.cloud.gateway.filter.factory.AbstractGatewayFilterFactory;
13
import org.springframework.core.io.buffer.DataBuffer;
14
import org.springframework.core.io.buffer.DataBufferUtils;
15
import org.springframework.http.HttpMethod;
16
import org.springframework.http.HttpStatus;
17
import org.springframework.http.server.reactive.ServerHttpRequest;
18
import org.springframework.http.server.reactive.ServerHttpRequestDecorator;
19
import org.springframework.stereotype.Component;
20
import org.springframework.util.CollectionUtils;
21
import org.springframework.web.reactive.function.BodyInserters;
22
import org.springframework.web.reactive.function.client.WebClient;
23
import org.springframework.web.reactive.function.client.WebClientResponseException;
24

25
import lombok.Data;
26
import lombok.extern.slf4j.Slf4j;
27
import reactor.core.publisher.Flux;
28
import reactor.core.publisher.Mono;
29
import reactor.core.scheduler.Scheduler;
30
import reactor.core.scheduler.Schedulers;
31

32
/**
33
 * 流量镜像网关过滤器工厂
34
 *
35
 * 功能：将请求复制到指定的镜像路由，实现完全非阻塞流量镜像 特性： - 真正的异步非阻塞处理，镜像不影响主请求性能 - 支持流式请求体处理，避免大请求体内存占用 - 支持多个镜像目标 - 自动错误处理和资源清理 - 智能的请求体复用机制
36
 */
37
@Slf4j
38
@Component
39
public class TrafficMirroringGatewayFilterFactory
40
    extends AbstractGatewayFilterFactory<TrafficMirroringGatewayFilterFactory.Config> implements DisposableBean {
41

42
    private final WebClient webClient;
43
    private final ExecutorService executorService;
44
    private final Scheduler scheduler;
45

46
    public TrafficMirroringGatewayFilterFactory() {
47
        super(TrafficMirroringGatewayFilterFactory.Config.class);
48
        this.executorService = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors() * 2, r -> {
49
            Thread t = new Thread(r, "traffic-mirror-");
50
            t.setDaemon(true);
51
            return t;
52
        });
53
        this.scheduler = Schedulers.fromExecutorService(executorService);
54
        this.webClient =
55
            WebClient.builder().codecs(configurer -> configurer.defaultCodecs().maxInMemorySize(1024 * 1024)) // 1MB
56
                .build();
57
    }
58

59
    @Override
60
    public GatewayFilter apply(Config config) {
61
        return (exchange, chain) -> {
62
            List<URI> mirroredRoutes = config.getMirroredRoutes();
63
            if (CollectionUtils.isEmpty(mirroredRoutes)) {
64
                return chain.filter(exchange);
65
            }
66

67
            ServerHttpRequest request = exchange.getRequest();
68

69
            // 对于无请求体的方法，立即镜像并继续主请求
70
            if (isBodylessMethod(request.getMethod())) {
71
                // 完全异步镜像，不阻塞主请求
72
                Mono.fromRunnable(() -> mirrorBodylessRequest(request, mirroredRoutes, config)).subscribeOn(scheduler)
73
                    .doOnError(error -> log.warn("Bodyless mirror failed", error)).subscribe();
74

75
                return chain.filter(exchange);
76
            }
77

78
            // 对于有请求体的方法，使用流式处理，不等待请求体完全读取
79
            ServerHttpRequest decoratedRequest = decorateRequestForMirroring(request, mirroredRoutes, config);
80

81
            // 立即继续主请求，镜像在后台异步进行
82
            return chain.filter(exchange.mutate().request(decoratedRequest).build());
83
        };
84
    }
85

86
    /**
87
     * 装饰请求以支持流式镜像，不阻塞主请求
88
     */
89
    private ServerHttpRequest decorateRequestForMirroring(ServerHttpRequest originalRequest, List<URI> mirroredRoutes,
90
        Config config) {
91
        return new ServerHttpRequestDecorator(originalRequest) {
92
            @Override
93
            public Flux<DataBuffer> getBody() {
94
                Flux<DataBuffer> originalBody = super.getBody();
95

96
                // 收集完整的请求体，然后进行镜像
97
                return originalBody.map(dataBuffer -> {
98
                    // 克隆DataBuffer用于镜像，保持原始DataBuffer不变
99
                    DataBuffer cloned = cloneDataBuffer(dataBuffer);
100
                    return new DataBufferWrapper(dataBuffer, cloned);
101
                }).collectList() // 收集所有DataBufferWrapper
102
                    .doOnNext(wrappers -> {
103
                        // 异步处理镜像，不阻塞主请求
104
                        Mono.fromRunnable(() -> {
105
                            try {
106
                                // 将所有克隆的DataBuffer合并为完整的请求体用于镜像
107
                                java.io.ByteArrayOutputStream outputStream = new java.io.ByteArrayOutputStream();
108
                                for (DataBufferWrapper wrapper : wrappers) {
109
                                    byte[] bytes = new byte[wrapper.cloned.readableByteCount()];
110
                                    wrapper.cloned.read(bytes);
111
                                    outputStream.write(bytes);
112
                                    // 释放克隆的DataBuffer
113
                                    DataBufferUtils.release(wrapper.cloned);
114
                                }
115
                                byte[] fullBodyBytes = outputStream.toByteArray();
116

117
                                // 发送镜像请求
118
                                mirrorRequest(originalRequest, mirroredRoutes, config, fullBodyBytes);
119
                            } catch (Exception e) {
120
                                log.warn("Failed to mirror request", e);
121
                            }
122
                        }).subscribeOn(scheduler).doOnError(error -> log.warn("Mirror request failed", error))
123
                            .subscribe();
124
                    }).doOnError(error -> log.warn("Request body collection error", error))
125
                    .flatMapMany(wrappers -> Flux.fromIterable(wrappers).map(wrapper -> wrapper.original));
126
            }
127
        };
128
    }
129

130
    /**
131
     * 克隆数据缓冲区用于镜像
132
     */
133
    private DataBuffer cloneDataBuffer(DataBuffer original) {
134
        DataBuffer cloned = original.factory().allocateBuffer(original.readableByteCount());
135
        cloned.write(original.slice(original.readPosition(), original.readableByteCount()));
136
        return cloned;
137
    }
138

139
    /**
140
     * DataBuffer包装器，用于同时持有原始和克隆的DataBuffer
141
     */
142
    private static class DataBufferWrapper {
143
        final DataBuffer original;
144
        final DataBuffer cloned;
145

146
        DataBufferWrapper(DataBuffer original, DataBuffer cloned) {
147
            this.original = original;
148
            this.cloned = cloned;
149
        }
150
    }
151

152
    /**
153
     * 判断是否为无请求体的HTTP方法
154
     */
155
    private boolean isBodylessMethod(HttpMethod method) {
156
        return method == HttpMethod.GET || method == HttpMethod.HEAD || method == HttpMethod.OPTIONS
157
            || method == HttpMethod.TRACE;
158
    }
159

160
    /**
161
     * 执行镜像请求 - 完全异步，不阻塞调用者
162
     */
163
    private void mirrorRequest(ServerHttpRequest request, List<URI> mirroredRoutes, Config config, byte[] bodyBytes) {
164
        mirroredRoutes.forEach(mirroredRoute -> {
165
            Mono.fromCallable(() -> {
166
                WebClient.RequestHeadersSpec<?> requestSpec = webClient.method(request.getMethod())
167
                    .uri(buildMirrorUri(mirroredRoute, request)).headers(httpHeaders -> {
168
                        httpHeaders.addAll(request.getHeaders());
169
                        // 移除可能导致问题的头部
170
                        httpHeaders.remove("host");
171
                        httpHeaders.remove("content-length");
172
                        if (bodyBytes != null && bodyBytes.length == 0) {
173
                            httpHeaders.remove("content-type");
174
                        }
175
                    });
176

177
                if (bodyBytes != null && bodyBytes.length > 0) {
178
                    requestSpec = ((WebClient.RequestBodySpec)requestSpec).body(BodyInserters.fromValue(bodyBytes));
179
                }
180

181
                return requestSpec.retrieve().toBodilessEntity().timeout(Duration.ofSeconds(config.getTimeoutSeconds()))
182
                    .doOnSuccess(response -> {
183
                        log.info("Mirror request succeeded to {}", mirroredRoute);
184
                        // if (log.isDebugEnabled()) {
185
                        // log.debug("Mirror request succeeded to {}", mirroredRoute);
186
                        // }
187
                    }).doOnError(error -> handleMirrorError(mirroredRoute, error, config));
188
            }).subscribeOn(scheduler).flatMap(mono -> mono)
189
                .doOnError(error -> handleMirrorError(mirroredRoute, error, config)).onErrorResume(error -> {
190
                    log.error(error.getMessage(), error);
191
                    return Mono.empty();
192
                }) // 忽略镜像错误，不影响主流程
193
                .subscribe();
194
        });
195
    }
196

197
    /**
198
     * 镜像无请求体的请求
199
     */
200
    private void mirrorBodylessRequest(ServerHttpRequest request, List<URI> mirroredRoutes, Config config) {
201
        mirrorRequest(request, mirroredRoutes, config, null);
202
    }
203

204
    /**
205
     * 构建镜像URI，保留原始请求的路径和查询参数
206
     */
207
    private URI buildMirrorUri(URI mirroredRoute, ServerHttpRequest request) {
208
        String query = request.getURI().getQuery();
209
        String uriString = mirroredRoute.toString();
210
        if (query != null && !query.isEmpty() && !isUrlEncoded(query)) {
211
            try {
212
                uriString += "?" + URLEncoder.encode(query, StandardCharsets.UTF_8.toString());
213
            } catch (Exception e) {
214
                // ignore
215
            }
216
        }
217
        return URI.create(uriString);
218
    }
219

220
    public static boolean isUrlEncoded(String value) {
221
        if (value == null)
222
            return false;
223

224
        try {
225
            // 解码后与原字符串不同，则可能已编码
226
            String decoded = URLDecoder.decode(value, StandardCharsets.UTF_8.toString());
227
            return !decoded.equals(value);
228
        } catch (Exception e) {
229
            // 解码失败说明可能未编码
230
            return false;
231
        }
232
    }
233

234
    /**
235
     * 处理镜像请求错误
236
     */
237
    private void handleMirrorError(URI mirroredRoute, Throwable error, Config config) {
238
        if (config.isEnableErrorLogging()) {
239
            if (error instanceof WebClientResponseException) {
240
                WebClientResponseException webClientError = (WebClientResponseException)error;
241
                // 404 不记录警告，可能是正常情况
242
                if (webClientError.getStatusCode() != HttpStatus.NOT_FOUND) {
243
                    log.warn("Mirror request failed to {}: {} - {}", mirroredRoute, webClientError.getStatusCode(),
244
                        webClientError.getMessage());
245
                }
246
            } else {
247
                log.warn("Mirror request failed to {}: {}", mirroredRoute, error.getMessage());
248
            }
249
        }
250
    }
251

252
    @Override
253
    public String name() {
254
        return "TrafficMirroring";
255
    }
256

257
    /**
258
     * 清理资源
259
     */
260
    @Override
261
    public void destroy() throws Exception {
262
        log.info("Shutting down traffic mirroring executor service");
263
        scheduler.dispose();
264
        executorService.shutdown();
265
        try {
266
            if (!executorService.awaitTermination(5, java.util.concurrent.TimeUnit.SECONDS)) {
267
                log.warn("Executor service did not terminate gracefully, forcing shutdown");
268
                executorService.shutdownNow();
269
            }
270
        } catch (InterruptedException e) {
271
            log.warn("Interrupted while waiting for executor service termination");
272
            executorService.shutdownNow();
273
            Thread.currentThread().interrupt();
274
        }
275
    }
276

277
    /**
278
     * 流量镜像配置类
279
     */
280
    @Data
281
    public static class Config {
282
        /**
283
         * 镜像目标路由列表
284
         */
285
        private List<URI> mirroredRoutes;
286

287
        /**
288
         * 镜像请求超时时间（秒），默认3秒
289
         */
290
        private int timeoutSeconds = 3;
291

292
        /**
293
         * 是否启用错误日志，默认启用
294
         */
295
        private boolean enableErrorLogging = true;
296

297
        /**
298
         * 是否启用流式镜像，默认启用
299
         */
300
        private boolean enableStreamMirroring = true;
301

302
        /**
303
         * 最大镜像请求体大小（字节），默认1MB
304
         */
305
        private int maxMirrorBodySize = 1024 * 1024;
306
    }
307
}

展开收起

通过 roaringbitmap 判断百万级数据是否存在

Sat, 14 Jun 2025 00:00:00 GMT

问题描述#

有以下用户订阅关系表

1
CREATE TABLE `template_subscribe` (
2
  `id` bigint NOT NULL COMMENT 'ID主键',
3
  `template_id` varchar(32) COLLATE utf8mb4_general_ci NOT NULL DEFAULT '' COMMENT '模板id',
4
  `user_id` bigint NOT NULL COMMENT '用户主键',
5
  PRIMARY KEY (`id`) USING BTREE,
6
  KEY `idx_user_id_template_id` (`user_id`, `template_id`) USING BTREE
7
);

先给定一个 CSV 文件和一个 template_id，只包含 userId, 数量预计在百万级别，要求判断有多少 userId 在 template_subscribe 表中存在

纯数据库方案#

分批次，通过 sql 从数据库中查询,比如每次 1000 个

1
select count(*) from template_subscribe where user_id in (x1,x2,x3 ...) and template_id = y;

100W数据需要 1000次循环，假设每次查询需要 1s，一共需要 1000 秒，该方案耗费时间太长不可接受，也会对数据库造成压力

bloom filter方案#

基于数据库中的 template_id 做分组，通过定时任务生成每个分组的 bloom filter 数据结构，判断时，循环遍历csv文件中每个userId，判断是否在 bloom filter 中存在。每个id判断时间查找复杂度大概在 O(k) k为hash函数数量，100W约为 100W * k次，但是会有误判率（数据比真实数量大）。

bitmap 方案#

基于数据库中的 template_id 做分组，通过定时任务生成每个分组的 bitmap 数据结构 A1，判断时，通过循环遍历csv文件中每个userId，生成一个 bitmap A2，通过对 A1、A2 取交集原算得到 A3, 直接判断A3 中的数据量即可得到人数

roaringbitmap 方案#

bitmap 方案的升级版本，解决了内存消耗过大的问题

image 11

对比总结#

特性	bloom filter	bitmap	roaringbitmap
类型	概率型数据结构	精确数据结构	精确数据结构
准确性	有误判率	准确	准确
空间效率	固定空间消耗	与最大值线性相关	自适应压缩存储
查询复杂度	O(k)	O(1)	O(1)
支持数据类型	任意可hash的对象	整数	整数
内存消耗	中等可控	稀疏时存在浪费	高
写入性能	快	快	中等
支持集合运算	否	是	是
删除操作	不支持	支持	支持

📎 参考文章#

https://zhuanlan.zhihu.com/p/445396980

iStoreOS + WireGuard 客户端踩坑

Thu, 12 Jun 2025 00:00:00 GMT

网络拓扑#

image 15

遇到的问题#

PC 连接 RDS 时，获取不到数据表相关的任何信息，navicat 客户端一直转圈，像是等待数据接受

问题排查#

能ping通，说明 VPN 网络已经建立成功

在 ECS 上通过 tcpdump 命令抓包，并再次使用 nacivate 打开数据库

Terminal window

1
sudo tcpdump -w example.pcap

将 example.pcap 下载到本地，通过 WireShark 打开查看

image 16

Fragmentation needed 服务端返回的包大小为 1515, 接收端只能接收的包大小为 1420，需要分包

image 17

解决方案#

在 OpenWrt 中开启 MSS 钳制

MSS 钳制（Maximum Segment Size Clamping）是 TCP/IP 网络中的一种技术，用于调整 TCP 连接的最大段大小（MSS）参数。
什么是 MSS？#
MSS（Maximum Segment Size）是 TCP 协议中的一个重要参数，表示 TCP 连接中能够传输的最大数据段大小（不包括 TCP 头）。它通常比 MTU（最大传输单元）小 40 字节（IPv4）或 60 字节（IPv6），为 IP 和 TCP 头留出空间。
MSS 钳制的作用#
MSS 钳制主要用于解决以下问题：

路径 MTU 发现问题：当路径 MTU 发现失败时，MSS 钳制可以防止数据包过大导致分片

网络设备兼容性：某些网络设备可能无法正确处理大尺寸数据包

优化传输效率：通过调整 MSS 可以提高网络传输效率

实现方式#
MSS 钳制通常在以下位置实现：

防火墙/路由器：在网络边界设备上修改 TCP SYN 包中的 MSS 选项

操作系统：在主机 TCP/IP 栈中设置 MSS 上限

常见应用场景#

VPN 连接中，由于额外封装头需要减小 MSS

PPPoE 连接（通常 MTU 为 1492，MSS 相应减小）

避免 IP 分片，特别是在有 NAT 的环境中

WireGuard 安装#

iStore 版本 22.03 branch git-24.034.55136-8c924fc

系统默认自带的镜像源中，wireguard 相关的包搜索不到，需要手动从 https://downloads.openwrt.org/releases/packages-22.03/x86_64/luci/ 下载，确认以下版本可以安装在 22.03 版本的 openwrt 中

luci-proto-wireguard_git-24.047.55738-270b737_all.ipk

luci-app-wireguard_git-23.018.72712-6d712c3_all.ipk

安装完成后重启

WireGuard 配置#

网络 > 接口：添加新接口，协议中选择 WireGuard VPN
image 18
网络 > 接口：选择创建好的 WireGuard 接口，编辑
image 19
在常规选项卡中，填入 WireGuard 的私钥、公钥信息，因为不是作为 WireGuard 服务端，监听端口可以不填，ip 地址按照服务端预先设置好的规则填入（vpn网络的 ip 网段划分，IPv4，IPv6 都输入）
image 20
在对端选项卡中，添加 WireGuard 服务端的信息
image 21
在防火墙设置中，通过自定义新增一个 wiregurad 区域
image 22
网络 > 防火墙：转发相关设置这里的 MSS 钳制非常重要，一开始没勾选，导致后面本地访问远程阿里云端的 MySQL 时，数据库响应异常
image 23

image 24

image 25

image 26
网络 > 接口：重启 WireGuard 接口
image 27
状态 > WireGuard：查看 WireGuard 的状态
image 28

wg-eazy 重置密码#

Terminal window

1
# https://wg-easy.github.io/wg-easy/v15.1/guides/cli/
2
docker compose exec -it wg-easy cli db:admin:reset --password '123456789012'

📎 参考文章#

DoH 收集

Tue, 10 Jun 2025 00:00:00 GMT

Provider	DoH
aliyun	`https://dns.alidns.com/dns-query` `https://223.5.5.5/dns-query` `https://223.6.6.6/dns-query`
dnspod	`https://doh.pub/dns-query` `https://1.12.12.12/dns-query` `https://120.53.53.53/dns-query`
Quad9	`https://dns.quad9.net/dns-query` `https://9.9.9.9/dns-query` `https://149.112.112.112/dns-query`
RethinkDNS	`https://max.rethinkdns.com/dns-query` `https://sky.rethinkdns.com/dns-query`
OpenDNS	`https://doh.opendns.com/dns-query`
Cloudflare	`https://dns.cloudflare.com/dns-query` `https://1.0.0.1/dns-query`
BebasDNS	`https://dns.bebasid.com/unfiltered`
0ms	`https://0ms.dev/dns-query`
DeCloudUs	`https://dns.decloudus.com/dns-query`
Wikimedia DNS	`https://wikimedia-dns.org/dns-query`
Applied Privacy DNS	`https://doh.applied-privacy.net/query`
CIRA Canadian Shield DNS	`https://private.canadianshield.cira.ca/dns-query`
Comss.one DNS	`https://dns.controld.com/comss`
Fondation Restena DNS	`https://kaitain.restena.lu/dns-query`
LibreDNS	`https://doh.libredns.gr/dns-query` `https://doh.libredns.gr/ads`
SWITCH DNS	`https://dns.switch.ch/dns-query`
AhaDNS	`https://doh.nl.ahadns.net/dns-query` `https://doh.la.ahadns.net/dns-query`
dnswarden	`https://dns.dnswarden.com/uncensored`
FFMUC DNS	`https://doh.ffmuc.net/dns-query`
OSZX DNS	`https://dns.oszx.co/dns-query`
Privacy-First DNS	`https://doh.tiarap.org/dns-query` `https://jp.tiarap.org/dns-query`

Debian sudo 功能配置

Mon, 09 Jun 2025 00:00:00 GMT

debian 最小化安装完成时，系统中没有如同 ubuntu 一样的 sudo 功能，需要自己配置，实际上只需要切换到 root 用户进行以下操作

Terminal window

1
su root # switch user
2
apt install sudo
3
sudo usermod -aG sudo ${username} # change to real username

UV使用

Thu, 22 May 2025 00:00:00 GMT

从 micromamba 转 uv 了

UV 优势#

🚀 A single tool to replace pip, pip-tools, pipx, poetry, pyenv, twine, virtualenv, and more.
⚡️ 10-100x faster than pip.
🗂️ Provides comprehensive project management, with a universal lockfile.
❇️ Runs scripts, with support for inline dependency metadata.
🐍 Installs and manages Python versions.
🛠️ Runs and installs tools published as Python packages.
🔩 Includes a pip-compatible interface for a performance boost with a familiar CLI.
🏢 Supports Cargo-style workspaces for scalable projects.
💾 Disk-space efficient, with a global cache for dependency deduplication.
⏬ Installable without Rust or Python via curl or pip.
🖥️ Supports macOS, Linux, and Windows.

安装#

Terminal window

1
# 安装
2
brew install uv
3
# mise
4
# mise use -g uv@latest

配置镜像源#

$XDG_CONFIG_HOME/uv/uv.toml

1
[[index]]
2
url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/"
3
default = true

迁移micromanba#

Terminal window

1
# 迁移 micromamba 到 uv
2
micromamba list -e > requirements.txt
3
uv pip install -r requirements.txt

使用#

Terminal window

Python versions#

uv python install: Install Python versions.
uv python list: View available Python versions.
uv python find: Find an installed Python version.
uv python pin: Pin the current project to use a specific Python version.
uv python uninstall: Uninstall a Python version.

Scripts#

uv run: Run a script.
uv add --script: Add a dependency to a script
uv remove --script: Remove a dependency from a script

Projects#

uv init: Create a new Python project.
uv add: Add a dependency to the project.
uv remove: Remove a dependency from the project.
uv sync: Sync the project’s dependencies with the environment.
uv lock: Create a lockfile for the project’s dependencies.
uv run: Run a command in the project environment.
uv tree: View the dependency tree for the project.
uv build: Build the project into distribution archives.
uv publish: Publish the project to a package index.

Tools#

uvx / uv tool run: Run a tool in a temporary environment.
uv tool install: Install a tool user-wide.
uv tool uninstall: Uninstall a tool.
uv tool list: List installed tools.
uv tool update-shell: Update the shell to include tool executables.

The pip interface#

uv venv: Create a new virtual environment.
uv pip install: Install packages into the current environment.
uv pip show: Show details about an installed package.
uv pip freeze: List installed packages and their versions.
uv pip check: Check that the current environment has compatible packages.
uv pip list: List installed packages.
uv pip uninstall: Uninstall packages.
uv pip tree: View the dependency tree for the environment.
uv pip compile: Compile requirements into a lockfile.
uv pip sync: Sync an environment with a lockfile.

Utility#

uv cache clean: Remove cache entries.
uv cache prune: Remove outdated cache entries.
uv cache dir: Show the uv cache directory path.
uv tool dir: Show the uv tool directory path.
uv python dir: Show the uv installed Python versions path.
uv self update: Update uv to the latest version.

debezium 小试

Fri, 21 Mar 2025 00:00:00 GMT

以前用过 maxwell，几年前就看到过 debezium，近期调研替换阿里云 DTS 订阅的方案，尝试一下

安装步骤#

使用 docker compose 方式安装，相关配置如下

1
service:
2
  # https://hub.docker.com/r/bitnami/kafka
3
  kafka:
4
    image: bitnami/kafka:3.4.1
5
    container_name: kafka
6
    volumes:
7
      - $HOME/data/kafka/:/bitnami/kafka/ #注意目录权限问题
8
    environment:
9
      - KAFKA_CFG_NODE_ID=0
10
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
11
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
12
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@192.168.50.21:9093 # 宿主机ip
13
      # EXTERNAL 配置的是挡墙 Docker 所在的主机地址
14
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
15
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,EXTERNAL://192.168.50.21:9094 # 宿主机ip
16
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
17
    ports:
18
      - 9092:9092
19
      - 9093:9093
20
      - 9094:9094
21

22
  debezium-connect:
23
    image: debezium/connect:3.0.0.Final
24
    container_name: debezium-connect
25
    environment:
26
      - TZ=Asia/Shanghai
27
      - GROUP_ID=debezium
28
      - CONFIG_STORAGE_TOPIC=debezium_config_storage
29
      - OFFSET_STORAGE_TOPIC=debezium_offset_storage
30
      - BOOTSTRAP_SERVERS=kafka:9094
31
    ports:
32
      - 8083:8083
33
    depends_on:
34
      - kafka

启动

Terminal window

1
docker compose --compatibility up -d debezium-connect

准备 mysql 复制账号，这里用户名密码都使用 slave_debezium，并给予从库权限

1
CREATE USER 'slave_debezium'@'%' IDENTIFIED BY 'slave_debezium';
2
GRANT REPLICATION SLAVE ON *.* TO 'slave_debezium'@'%';
3
FLUSH PRIVILEGES;

确认 mysql 主从状态

1
# show master status; # 旧语法
2
show binary log status; # mysql 8.4 的语法

创建 debezium 信号表

1
CREATE TABLE `debezium_signal` (
2
  `id` varchar(64) NOT NULL,
3
  `type` varchar(32) NOT NULL,
4
  `data` json DEFAULT NULL,
5
  PRIMARY KEY (`id`)
6
);

初始化 debezium-connect

关键配置说明

table.include.list 要监听的对象表（命名形式库名.表名,库名.表名）

snapshot.mode = schema_only （在任务启动时，只做表结构快照，不做数据快照，默认的 initial 策略会对监听的表做全量快照，遇到数据量大的表可能会影响生产写入）

snapshot.locking.mode = none（快照时不使用锁，避免影响生产写操作）

signal.data.collection = app_dev.debezium_signal（指定 dbz 的信号表，用于触发增量快照）

read.only = true （只读账号时必须指定，否则会尝试对 debezium_signal 表做 insert 操作来产生水位信息）

transforms = reroute （SMT 相关配置，通过正则表达式调整 dbz 默认的 topic 策略）

decimal.handling.mode = string （将 mysql 中 decimal 类型转换为 string 输出，确保不丢失精度）

converters = bitOneToInt （默认 bit 会转换为 boolean 类型，这里通过自定义 CustomConvert 修改默认行为）下面暂时没用到

Terminal window

1
# 创建新的 connector
2
curl -i -X POST http://192.168.50.21:8083/connectors \
3
    -H "Accept:application/json" \
4
    -H "Content-Type:application/json" \
5
    -d \
6
    '{
7
      "name": "debezium-connector-mysql",
8
      "config": {
9
        "connector.class": "io.debezium.connector.mysql.MySqlConnector",
10
        "database.hostname": "192.168.50.21",
11
        "database.port": "3306",
12
        "database.user": "slave_debezium",
13
        "database.password": "slave_debezium",
14
        "database.server.id": "101",
15
        "topic.prefix": "debeziumdemo",
16
        "table.include.list": "app_dev.cdc_demo",
17
        "schema.history.internal.kafka.bootstrap.servers": "192.168.50.21:9094",
18
        "schema.history.internal.kafka.topic": "debezium_schema_history",
19
        "include.schema.changes": "true",
20
        "snapshot.mode": "schema_only",
21
        "snapshot.locking.mode": "none",
22
        "signal.data.collection": "app_dev.debezium_signal",
23
        "read.only": true,
24
        "transforms": "reroute",
25
        "transforms.reroute.type": "io.debezium.transforms.ByLogicalTableRouter",
26
        "transforms.reroute.topic.regex": "(\\S+)\\.(\\S+)\\.(\\S+)",
27
        "transforms.retopic.key.enforce.uniqueness": "false",
28
        "transforms.reroute.topic.replacement": "$1.$2.debezium_cdc",
29
        "time.precision.mode": "adaptive_time_microseconds",
30
        "decimal.handling.mode": "string",
31
        "boolean.length.checker": "false"
32
      }
33
    }'

通过 Offset Explorer 3 可以看到相关数据

![[image 29.png]]

debezium-connect 常用 rest 操作#

获取 connect 状态

Terminal window

1
# 获取状态
2
curl -X GET http://host:port/connectors/debezium-connector-mysql/status

获取 connect 配置

Terminal window

1
curl -X GET http://host:port/connectors/debezium-connector-mysql/config

暂停 connect cdc 捕捉任务

Terminal window

1
curl -XPUT http://host:port/connectors/debezium-connector-mysql/pause

恢复 connect cdc 捕捉任务

Terminal window

1
curl -XPUT http://host:port/connectors/debezium-connector-mysql/resume

插入 debezium_signal 表触发指定表增量快照

Terminal window

1
INSERT INTO `debezium_signal` (`id`, `type`, `data`) VALUES
2
('ad-hoc-1', 'execute-snapshot', '{\"type\": \"incremental\", \"data-collections\": [\"指定数据库名.指定表名\"]}');

问题处理记录#

CDC 事件 topic 策略

默认 dbz 默认实现是一个表一个 topic，可以通过 kafka connect 的 SMT 机制来重路由，初始配置如下

1
"transforms": "reroute",
2
"transforms.reroute.type": "io.debezium.transforms.ByLogicalTableRouter",
3
"transforms.reroute.topic.regex": "(\\S+)\\.(\\w+)\\.(\\w+)",
4
"transforms.retopic.key.enforce.uniqueness": "false",
5
"transforms.reroute.topic.replacement": "$1.$2.debezium_cdc",

但是查看 kafka 队列发现不生效！

![[image 30.png]]

原因是正则表达式 \w 不包含中划线，需要修改为 \S

系统提示 debezium_signal 没有写入权限

在做增量快照时，后台报错提示 debezium_signal 没有 insert 权限

解决方案

加上 read.only = true（通过从库账号做只读快照）

MySQL binlog 验证#

近期有个疑问，在一个事务开启后，多次对同一条数据进行 update 操作，binlog 会记录几次？

结论每条 update 操作都会有对应的binlog 日志

验证过程如下

1
begin;
2
// 第一次修改
3
UPDATE `cdc_demo`
4
SET `col_int`=923,`col_smallint`=302,`col_mediumint`=858,`col_tinyint`=100,`col_bit`=b '1',`col_double`=8.34,`col_float`=567.96,`col_decimal`=734.57,`col_char`='Ri',`col_varchar`='J88fKfgvps',`col_text`='',`col_tinytext`='',`col_mediumtext`='',`col_longtext`='',`col_datetime`='2007-07-28 16:21:13',`col_timestamp`='2020-05-12 06:29:10',`col_date`='2003-03-30',`col_time`='17:33:52',`col_year`=2029 WHERE `col_bigint`=10;
5
// 第二次修改
6
UPDATE `cdc_demo`
7
SET `col_int`=923,`col_smallint`=302,`col_mediumint`=858,`col_tinyint`=100,`col_bit`=b '1',`col_double`=8.34,`col_float`=567.96,`col_decimal`=734.57,`col_char`='Ri',`col_varchar`='J88fKfgvps',`col_text`='',`col_tinytext`='',`col_mediumtext`='',`col_longtext`='',`col_datetime`='2007-07-28 16:21:13',`col_timestamp`='2020-05-12 06:29:10',`col_date`='2003-03-30',`col_time`='17:33:52',`col_year`=2020 WHERE `col_bigint`=10;
8
commit;

kafka 中 cdc 的记录，第一次修改

1
{
2
  "schema": {
3
    "type": "struct",
4
    "fields": [
5
      {
6
        "type": "struct",
7
        "fields": [
8
          {
9
            "type": "int64",
10
            "optional": false,
11
            "field": "col_bigint"
12
          },
13
          {
14
            "type": "int32",
15
            "optional": true,
16
            "field": "col_int"
17
          },
18
          {
19
            "type": "int16",
20
            "optional": true,
21
            "field": "col_smallint"
22
          },
23
          {
24
            "type": "int32",
25
            "optional": true,
26
            "field": "col_mediumint"
27
          },
28
          {
29
            "type": "int16",
30
            "optional": true,
31
            "field": "col_tinyint"
32
          },
33
          {
34
            "type": "boolean",
35
            "optional": true,
36
            "field": "col_bit"
37
          },
38
          {
39
            "type": "double",
40
            "optional": true,
41
            "field": "col_double"
42
          },
43
          {
44
            "type": "float",
45
            "optional": true,
46
            "field": "col_float"
47
          },
48
          {
49
            "type": "string",
50
            "optional": true,
51
            "field": "col_decimal"
52
          },
53
          {
54
            "type": "string",
55
            "optional": true,
56
            "field": "col_char"
57
          },
58
          {
59
            "type": "string",
60
            "optional": true,
61
            "field": "col_varchar"
62
          },
63
          {
64
            "type": "string",
65
            "optional": true,
66
            "field": "col_text"
67
          },
68
          {
69
            "type": "string",
70
            "optional": true,
71
            "field": "col_tinytext"
72
          },
73
          {
74
            "type": "string",
75
            "optional": true,
76
            "field": "col_mediumtext"
77
          },
78
          {
79
            "type": "string",
80
            "optional": true,
81
            "field": "col_longtext"
82
          },
83
          {
84
            "type": "int64",
85
            "optional": true,
86
            "name": "io.debezium.time.Timestamp",
87
            "version": 1,
88
            "field": "col_datetime"
89
          },
90
          {
91
            "type": "string",
92
            "optional": true,
93
            "name": "io.debezium.time.ZonedTimestamp",
94
            "version": 1,
95
            "field": "col_timestamp"
96
          },
97
          {
98
            "type": "int32",
99
            "optional": true,
100
            "name": "io.debezium.time.Date",
101
            "version": 1,
102
            "field": "col_date"
103
          },
104
          {
105
            "type": "int64",
106
            "optional": true,
107
            "name": "io.debezium.time.MicroTime",
108
            "version": 1,
109
            "field": "col_time"
110
          },
111
          {
112
            "type": "int32",
113
            "optional": true,
114
            "name": "io.debezium.time.Year",
115
            "version": 1,
116
            "field": "col_year"
117
          }
118
        ],
119
        "optional": true,
120
        "name": "debeziumdemo.app_dev.debezium_cdc.Value",
121
        "field": "before"
122
      },
123
      {
124
        "type": "struct",
125
        "fields": [
126
          {
127
            "type": "int64",
128
            "optional": false,
129
            "field": "col_bigint"
130
          },
131
          {
132
            "type": "int32",
133
            "optional": true,
134
            "field": "col_int"
135
          },
136
          {
137
            "type": "int16",
138
            "optional": true,
139
            "field": "col_smallint"
140
          },
141
          {
142
            "type": "int32",
143
            "optional": true,
144
            "field": "col_mediumint"
145
          },
146
          {
147
            "type": "int16",
148
            "optional": true,
149
            "field": "col_tinyint"
150
          },
151
          {
152
            "type": "boolean",
153
            "optional": true,
154
            "field": "col_bit"
155
          },
156
          {
157
            "type": "double",
158
            "optional": true,
159
            "field": "col_double"
160
          },
161
          {
162
            "type": "float",
163
            "optional": true,
164
            "field": "col_float"
165
          },
166
          {
167
            "type": "string",
168
            "optional": true,
169
            "field": "col_decimal"
170
          },
171
          {
172
            "type": "string",
173
            "optional": true,
174
            "field": "col_char"
175
          },
176
          {
177
            "type": "string",
178
            "optional": true,
179
            "field": "col_varchar"
180
          },
181
          {
182
            "type": "string",
183
            "optional": true,
184
            "field": "col_text"
185
          },
186
          {
187
            "type": "string",
188
            "optional": true,
189
            "field": "col_tinytext"
190
          },
191
          {
192
            "type": "string",
193
            "optional": true,
194
            "field": "col_mediumtext"
195
          },
196
          {
197
            "type": "string",
198
            "optional": true,
199
            "field": "col_longtext"
200
          },
201
          {
202
            "type": "int64",
203
            "optional": true,
204
            "name": "io.debezium.time.Timestamp",
205
            "version": 1,
206
            "field": "col_datetime"
207
          },
208
          {
209
            "type": "string",
210
            "optional": true,
211
            "name": "io.debezium.time.ZonedTimestamp",
212
            "version": 1,
213
            "field": "col_timestamp"
214
          },
215
          {
216
            "type": "int32",
217
            "optional": true,
218
            "name": "io.debezium.time.Date",
219
            "version": 1,
220
            "field": "col_date"
221
          },
222
          {
223
            "type": "int64",
224
            "optional": true,
225
            "name": "io.debezium.time.MicroTime",
226
            "version": 1,
227
            "field": "col_time"
228
          },
229
          {
230
            "type": "int32",
231
            "optional": true,
232
            "name": "io.debezium.time.Year",
233
            "version": 1,
234
            "field": "col_year"
235
          }
236
        ],
237
        "optional": true,
238
        "name": "debeziumdemo.app_dev.debezium_cdc.Value",
239
        "field": "after"
240
      },
241
      {
242
        "type": "struct",
243
        "fields": [
244
          {
245
            "type": "string",
246
            "optional": false,
247
            "field": "version"
248
          },
249
          {
250
            "type": "string",
251
            "optional": false,
252
            "field": "connector"
253
          },
254
          {
255
            "type": "string",
256
            "optional": false,
257
            "field": "name"
258
          },
259
          {
260
            "type": "int64",
261
            "optional": false,
262
            "field": "ts_ms"
263
          },
264
          {
265
            "type": "string",
266
            "optional": true,
267
            "name": "io.debezium.data.Enum",
268
            "version": 1,
269
            "parameters": {
270
              "allowed": "true,last,false,incremental"
271
            },
272
            "default": "false",
273
            "field": "snapshot"
274
          },
275
          {
276
            "type": "string",
277
            "optional": false,
278
            "field": "db"
279
          },
280
          {
281
            "type": "string",
282
            "optional": true,
283
            "field": "sequence"
284
          },
285
          {
286
            "type": "int64",
287
            "optional": true,
288
            "field": "ts_us"
289
          },
290
          {
291
            "type": "int64",
292
            "optional": true,
293
            "field": "ts_ns"
294
          },
295
          {
296
            "type": "string",
297
            "optional": true,
298
            "field": "table"
299
          },
300
          {
301
            "type": "int64",
302
            "optional": false,
303
            "field": "server_id"
304
          },
305
          {
306
            "type": "string",
307
            "optional": true,
308
            "field": "gtid"
309
          },
310
          {
311
            "type": "string",
312
            "optional": false,
313
            "field": "file"
314
          },
315
          {
316
            "type": "int64",
317
            "optional": false,
318
            "field": "pos"
319
          },
320
          {
321
            "type": "int32",
322
            "optional": false,
323
            "field": "row"
324
          },
325
          {
326
            "type": "int64",
327
            "optional": true,
328
            "field": "thread"
329
          },
330
          {
331
            "type": "string",
332
            "optional": true,
333
            "field": "query"
334
          }
335
        ],
336
        "optional": false,
337
        "name": "io.debezium.connector.mysql.Source",
338
        "field": "source"
339
      },
340
      {
341
        "type": "struct",
342
        "fields": [
343
          {
344
            "type": "string",
345
            "optional": false,
346
            "field": "id"
347
          },
348
          {
349
            "type": "int64",
350
            "optional": false,
351
            "field": "total_order"
352
          },
353
          {
354
            "type": "int64",
355
            "optional": false,
356
            "field": "data_collection_order"
357
          }
358
        ],
359
        "optional": true,
360
        "name": "event.block",
361
        "version": 1,
362
        "field": "transaction"
363
      },
364
      {
365
        "type": "string",
366
        "optional": false,
367
        "field": "op"
368
      },
369
      {
370
        "type": "int64",
371
        "optional": true,
372
        "field": "ts_ms"
373
      },
374
      {
375
        "type": "int64",
376
        "optional": true,
377
        "field": "ts_us"
378
      },
379
      {
380
        "type": "int64",
381
        "optional": true,
382
        "field": "ts_ns"
383
      }
384
    ],
385
    "optional": false,
386
    "name": "debeziumdemo.app_dev.debezium_cdc.Envelope",
387
    "version": 2
388
  },
389
  "payload": {
390
    "before": {
391
      "col_bigint": 10,
392
      "col_int": 923,
393
      "col_smallint": 302,
394
      "col_mediumint": 858,
395
      "col_tinyint": 100,
396
      "col_bit": true,
397
      "col_double": 8.34,
398
      "col_float": 567.96,
399
      "col_decimal": "734.57",
400
      "col_char": "Ri",
401
      "col_varchar": "J88fKfgvps",
402
      "col_text": "",
403
      "col_tinytext": "",
404
      "col_mediumtext": "",
405
      "col_longtext": "",
406
      "col_datetime": 1185639673000,
407
      "col_timestamp": "2020-05-11T22:29:10Z",
408
      "col_date": 12141,
409
      "col_time": 63232000000,
410
      "col_year": 2030
411
    },
412
    "after": {
413
      "col_bigint": 10,
414
      "col_int": 923,
415
      "col_smallint": 302,
416
      "col_mediumint": 858,
417
      "col_tinyint": 100,
418
      "col_bit": true,
419
      "col_double": 8.34,
420
      "col_float": 567.96,
421
      "col_decimal": "734.57",
422
      "col_char": "Ri",
423
      "col_varchar": "J88fKfgvps",
424
      "col_text": "",
425
      "col_tinytext": "",
426
      "col_mediumtext": "",
427
      "col_longtext": "",
428
      "col_datetime": 1185639673000,
429
      "col_timestamp": "2020-05-11T22:29:10Z",
430
      "col_date": 12141,
431
      "col_time": 63232000000,
432
      "col_year": 2029
433
    },
434
    "source": {
435
      "version": "3.0.0.Final",
436
      "connector": "mysql",
437
      "name": "debeziumdemo",
438
      "ts_ms": 1742398793844,
439
      "snapshot": "false",
440
      "db": "app_dev",
441
      "sequence": null,
442
      "ts_us": 1742398793844717,
443
      "ts_ns": 1742398793844717000,
444
      "table": "cdc_demo",
445
      "server_id": 1,
446
      "gtid": "57561b7b-e2a7-11ef-9d3b-0242ac130003:1252076",
447
      "file": "mysql-bin.000008",
448
      "pos": 819263525,
449
      "row": 0,
450
      "thread": 165069,
451
      "query": null
452
    },
453
    "transaction": null,
454
    "op": "u",
455
    "ts_ms": 1742398793846,
456
    "ts_us": 1742398793846407,
457
    "ts_ns": 1742398793846407748
458
  }
459
}

展开收起

kafka 中 cdc 的记录，第二次修改

1
{
2
  "schema": {
3
    "type": "struct",
4
    "fields": [
5
      {
6
        "type": "struct",
7
        "fields": [
8
          {
9
            "type": "int64",
10
            "optional": false,
11
            "field": "col_bigint"
12
          },
13
          {
14
            "type": "int32",
15
            "optional": true,
16
            "field": "col_int"
17
          },
18
          {
19
            "type": "int16",
20
            "optional": true,
21
            "field": "col_smallint"
22
          },
23
          {
24
            "type": "int32",
25
            "optional": true,
26
            "field": "col_mediumint"
27
          },
28
          {
29
            "type": "int16",
30
            "optional": true,
31
            "field": "col_tinyint"
32
          },
33
          {
34
            "type": "boolean",
35
            "optional": true,
36
            "field": "col_bit"
37
          },
38
          {
39
            "type": "double",
40
            "optional": true,
41
            "field": "col_double"
42
          },
43
          {
44
            "type": "float",
45
            "optional": true,
46
            "field": "col_float"
47
          },
48
          {
49
            "type": "string",
50
            "optional": true,
51
            "field": "col_decimal"
52
          },
53
          {
54
            "type": "string",
55
            "optional": true,
56
            "field": "col_char"
57
          },
58
          {
59
            "type": "string",
60
            "optional": true,
61
            "field": "col_varchar"
62
          },
63
          {
64
            "type": "string",
65
            "optional": true,
66
            "field": "col_text"
67
          },
68
          {
69
            "type": "string",
70
            "optional": true,
71
            "field": "col_tinytext"
72
          },
73
          {
74
            "type": "string",
75
            "optional": true,
76
            "field": "col_mediumtext"
77
          },
78
          {
79
            "type": "string",
80
            "optional": true,
81
            "field": "col_longtext"
82
          },
83
          {
84
            "type": "int64",
85
            "optional": true,
86
            "name": "io.debezium.time.Timestamp",
87
            "version": 1,
88
            "field": "col_datetime"
89
          },
90
          {
91
            "type": "string",
92
            "optional": true,
93
            "name": "io.debezium.time.ZonedTimestamp",
94
            "version": 1,
95
            "field": "col_timestamp"
96
          },
97
          {
98
            "type": "int32",
99
            "optional": true,
100
            "name": "io.debezium.time.Date",
101
            "version": 1,
102
            "field": "col_date"
103
          },
104
          {
105
            "type": "int64",
106
            "optional": true,
107
            "name": "io.debezium.time.MicroTime",
108
            "version": 1,
109
            "field": "col_time"
110
          },
111
          {
112
            "type": "int32",
113
            "optional": true,
114
            "name": "io.debezium.time.Year",
115
            "version": 1,
116
            "field": "col_year"
117
          }
118
        ],
119
        "optional": true,
120
        "name": "debeziumdemo.app_dev.debezium_cdc.Value",
121
        "field": "before"
122
      },
123
      {
124
        "type": "struct",
125
        "fields": [
126
          {
127
            "type": "int64",
128
            "optional": false,
129
            "field": "col_bigint"
130
          },
131
          {
132
            "type": "int32",
133
            "optional": true,
134
            "field": "col_int"
135
          },
136
          {
137
            "type": "int16",
138
            "optional": true,
139
            "field": "col_smallint"
140
          },
141
          {
142
            "type": "int32",
143
            "optional": true,
144
            "field": "col_mediumint"
145
          },
146
          {
147
            "type": "int16",
148
            "optional": true,
149
            "field": "col_tinyint"
150
          },
151
          {
152
            "type": "boolean",
153
            "optional": true,
154
            "field": "col_bit"
155
          },
156
          {
157
            "type": "double",
158
            "optional": true,
159
            "field": "col_double"
160
          },
161
          {
162
            "type": "float",
163
            "optional": true,
164
            "field": "col_float"
165
          },
166
          {
167
            "type": "string",
168
            "optional": true,
169
            "field": "col_decimal"
170
          },
171
          {
172
            "type": "string",
173
            "optional": true,
174
            "field": "col_char"
175
          },
176
          {
177
            "type": "string",
178
            "optional": true,
179
            "field": "col_varchar"
180
          },
181
          {
182
            "type": "string",
183
            "optional": true,
184
            "field": "col_text"
185
          },
186
          {
187
            "type": "string",
188
            "optional": true,
189
            "field": "col_tinytext"
190
          },
191
          {
192
            "type": "string",
193
            "optional": true,
194
            "field": "col_mediumtext"
195
          },
196
          {
197
            "type": "string",
198
            "optional": true,
199
            "field": "col_longtext"
200
          },
201
          {
202
            "type": "int64",
203
            "optional": true,
204
            "name": "io.debezium.time.Timestamp",
205
            "version": 1,
206
            "field": "col_datetime"
207
          },
208
          {
209
            "type": "string",
210
            "optional": true,
211
            "name": "io.debezium.time.ZonedTimestamp",
212
            "version": 1,
213
            "field": "col_timestamp"
214
          },
215
          {
216
            "type": "int32",
217
            "optional": true,
218
            "name": "io.debezium.time.Date",
219
            "version": 1,
220
            "field": "col_date"
221
          },
222
          {
223
            "type": "int64",
224
            "optional": true,
225
            "name": "io.debezium.time.MicroTime",
226
            "version": 1,
227
            "field": "col_time"
228
          },
229
          {
230
            "type": "int32",
231
            "optional": true,
232
            "name": "io.debezium.time.Year",
233
            "version": 1,
234
            "field": "col_year"
235
          }
236
        ],
237
        "optional": true,
238
        "name": "debeziumdemo.app_dev.debezium_cdc.Value",
239
        "field": "after"
240
      },
241
      {
242
        "type": "struct",
243
        "fields": [
244
          {
245
            "type": "string",
246
            "optional": false,
247
            "field": "version"
248
          },
249
          {
250
            "type": "string",
251
            "optional": false,
252
            "field": "connector"
253
          },
254
          {
255
            "type": "string",
256
            "optional": false,
257
            "field": "name"
258
          },
259
          {
260
            "type": "int64",
261
            "optional": false,
262
            "field": "ts_ms"
263
          },
264
          {
265
            "type": "string",
266
            "optional": true,
267
            "name": "io.debezium.data.Enum",
268
            "version": 1,
269
            "parameters": {
270
              "allowed": "true,last,false,incremental"
271
            },
272
            "default": "false",
273
            "field": "snapshot"
274
          },
275
          {
276
            "type": "string",
277
            "optional": false,
278
            "field": "db"
279
          },
280
          {
281
            "type": "string",
282
            "optional": true,
283
            "field": "sequence"
284
          },
285
          {
286
            "type": "int64",
287
            "optional": true,
288
            "field": "ts_us"
289
          },
290
          {
291
            "type": "int64",
292
            "optional": true,
293
            "field": "ts_ns"
294
          },
295
          {
296
            "type": "string",
297
            "optional": true,
298
            "field": "table"
299
          },
300
          {
301
            "type": "int64",
302
            "optional": false,
303
            "field": "server_id"
304
          },
305
          {
306
            "type": "string",
307
            "optional": true,
308
            "field": "gtid"
309
          },
310
          {
311
            "type": "string",
312
            "optional": false,
313
            "field": "file"
314
          },
315
          {
316
            "type": "int64",
317
            "optional": false,
318
            "field": "pos"
319
          },
320
          {
321
            "type": "int32",
322
            "optional": false,
323
            "field": "row"
324
          },
325
          {
326
            "type": "int64",
327
            "optional": true,
328
            "field": "thread"
329
          },
330
          {
331
            "type": "string",
332
            "optional": true,
333
            "field": "query"
334
          }
335
        ],
336
        "optional": false,
337
        "name": "io.debezium.connector.mysql.Source",
338
        "field": "source"
339
      },
340
      {
341
        "type": "struct",
342
        "fields": [
343
          {
344
            "type": "string",
345
            "optional": false,
346
            "field": "id"
347
          },
348
          {
349
            "type": "int64",
350
            "optional": false,
351
            "field": "total_order"
352
          },
353
          {
354
            "type": "int64",
355
            "optional": false,
356
            "field": "data_collection_order"
357
          }
358
        ],
359
        "optional": true,
360
        "name": "event.block",
361
        "version": 1,
362
        "field": "transaction"
363
      },
364
      {
365
        "type": "string",
366
        "optional": false,
367
        "field": "op"
368
      },
369
      {
370
        "type": "int64",
371
        "optional": true,
372
        "field": "ts_ms"
373
      },
374
      {
375
        "type": "int64",
376
        "optional": true,
377
        "field": "ts_us"
378
      },
379
      {
380
        "type": "int64",
381
        "optional": true,
382
        "field": "ts_ns"
383
      }
384
    ],
385
    "optional": false,
386
    "name": "debeziumdemo.app_dev.debezium_cdc.Envelope",
387
    "version": 2
388
  },
389
  "payload": {
390
    "before": {
391
      "col_bigint": 10,
392
      "col_int": 923,
393
      "col_smallint": 302,
394
      "col_mediumint": 858,
395
      "col_tinyint": 100,
396
      "col_bit": true,
397
      "col_double": 8.34,
398
      "col_float": 567.96,
399
      "col_decimal": "734.57",
400
      "col_char": "Ri",
401
      "col_varchar": "J88fKfgvps",
402
      "col_text": "",
403
      "col_tinytext": "",
404
      "col_mediumtext": "",
405
      "col_longtext": "",
406
      "col_datetime": 1185639673000,
407
      "col_timestamp": "2020-05-11T22:29:10Z",
408
      "col_date": 12141,
409
      "col_time": 63232000000,
410
      "col_year": 2029
411
    },
412
    "after": {
413
      "col_bigint": 10,
414
      "col_int": 923,
415
      "col_smallint": 302,
416
      "col_mediumint": 858,
417
      "col_tinyint": 100,
418
      "col_bit": true,
419
      "col_double": 8.34,
420
      "col_float": 567.96,
421
      "col_decimal": "734.57",
422
      "col_char": "Ri",
423
      "col_varchar": "J88fKfgvps",
424
      "col_text": "",
425
      "col_tinytext": "",
426
      "col_mediumtext": "",
427
      "col_longtext": "",
428
      "col_datetime": 1185639673000,
429
      "col_timestamp": "2020-05-11T22:29:10Z",
430
      "col_date": 12141,
431
      "col_time": 63232000000,
432
      "col_year": 2020
433
    },
434
    "source": {
435
      "version": "3.0.0.Final",
436
      "connector": "mysql",
437
      "name": "debeziumdemo",
438
      "ts_ms": 1742398793844,
439
      "snapshot": "false",
440
      "db": "app_dev",
441
      "sequence": null,
442
      "ts_us": 1742398793844717,
443
      "ts_ns": 1742398793844717000,
444
      "table": "cdc_demo",
445
      "server_id": 1,
446
      "gtid": "57561b7b-e2a7-11ef-9d3b-0242ac130003:1252076",
447
      "file": "mysql-bin.000008",
448
      "pos": 819263820,
449
      "row": 0,
450
      "thread": 165069,
451
      "query": null
452
    },
453
    "transaction": null,
454
    "op": "u",
455
    "ts_ms": 1742398793846,
456
    "ts_us": 1742398793846719,
457
    "ts_ns": 1742398793846719988
458
  }
459
}

展开收起

从上面两条修改记录的 "gtid": "57561b7b-e2a7-11ef-9d3b-0242ac130003:1252076" 可以看出，同一个事务中两次针对同一条数据的 update 操作都会产生对应的 binlog

📎 参考文章#

DDD记录

Fri, 07 Feb 2025 00:00:00 GMT

一些概念的理解#

### 界限上下文

image 37

定义清晰的边界：每个界限上下文都拥有自己的一套业务逻辑、实体、用语和规则。例如，一个“客户管理”系统和一个“订单管理”系统可能都涉及“客户”这一概念，但它们在各自的界限上下文中对“客户”的定义和处理方式可能不同。
不同子域的隔离：在DDD中，系统通常分为多个子域（Subdomains）。这些子域可能会有不同的需求、不同的模型、不同的业务规则。界限上下文明确了每个子域的业务范围和边界，避免不同子域之间的混淆和交叉干扰。
上下文内的统一语言：在一个界限上下文内，团队会使用统一的术语和语言来确保所有成员对同一概念有共同的理解。这就是DDD中提到的领域语言（Ubiquitous Language）。例如，在“订单管理”界限上下文中，“订单”的定义、属性和行为可能与其他上下文中的“订单”定义有所不同。
与其他界限上下文的关系：不同的界限上下文之间可能会存在交互，通常通过上下文映射（Context Mapping）来定义。常见的关系类型有：
- 共享内核（Shared Kernel）：多个上下文共享一些核心模型。
- 客户-供应商（Customer-Supplier）：一个上下文作为另一个上下文的“客户”，依赖于其提供的功能。
- 防腐层（Anti-Corruption Layer）：为避免不同上下文间的影响和混淆，定义一个层来隔离和转化数据。
- 发布-订阅（Published-Subscribed）：一个上下文发布事件，另一个上下文订阅这些事件。

举例

image 38

### 实体、值对象、聚合根、领域服务、领域事件

**实体 **当一个对象由其标识（而不是属性）区分时，这种对象称为实体（Entity）。理解上来说就是id,例如,人可以通过身份证号这种独一无二的标识来进行区分。而不是人的属性,例如身高,年龄等。
**值对象 **当一个对象用于对事物进行描述而没有唯一标识时，它被称作值对象（Value Object）。例如年龄28岁。使用值对象,可以更好的做系统优化和设计的精简优化。在实践中，需要保证值对象创建后就不能被修改，即不允许外部再修改其属性。
聚合根 聚合一组相关对象的集合，作为一个整体被外界访问，聚合根是这个聚合的根节点。聚合由根实体、值对象和实体构成。聚合是一个非常重要的概念，==核心领域往往都需要用聚合来表达==。其次，聚合在技术上有非常高的价值，可以指导详细设计。
**领域服务 **一些重要的领域行为或操作，可以归类为领域服务。它既不是实体，也不是值对象的范畴。当我们采用了微服务架构风格，一切领域逻辑的对外暴露均需要通过领域服务来进行。如原本由聚合根暴露的业务逻辑也需要依托于领域服务
**领域事件 **是表示系统中发生了某个具有业务意义的事件。它通常用于表示业务中某些状态变化或者某个重要行为的发生。这些事件通常是不可变的，一旦发生就会携带相关的业务数据，表示一个已经发生的事实。

充血模型代码举例#

在电商场景中，订单聚合根 OrderAggregateRoot 往往包含一些以下实体与值对象

Order（订单）
OrderItem（订单商品明细）
ShippingAddress（订单收货地址）

DDD 战术上，充血模型希望通过 OrderAggregateRoot 来作为其管理的实体、值对象的管理入口

示例代码如下

1
// 订单聚合根
2
public class OrderAggregateRoot {
3

4
    private Order order;
5

6
    private List<OrderItem> orderItem;
7

8
    private ShippingAddress shippingAddress;
9

10
    @Getter
11
    private final List<DomainEvent> domainEvents = new ArrayList<>();
12

13
    private OrderAggregateRoot() {
14
    }
15

16
    public static OrderAggregateRoot create(CreateOrder cmd) {
17
        // 规则校验
18
        OrderAggregateRoot root = new OrderAggregateRoot();
19
        Order order = new Order();
20
        // order.setXxx(xxx);
21
        root.setOrder(order);
22
        addDomainEvent(new OrderCreatedEvent());
23
    }
24

25
    public static OrderAggregateRoot from(Order order, List<OrderItem> orderItem, ShippingAddress shippingAddress) {
26
        OrderAggregateRoot root = new OrderAggregateRoot();
27
        // root.setXxx(xxx);
28
    }
29

30
    public BigDecimal calculateTotalPrice(CalculateOrderPrice cmd) {
31
        return BigDecimal.ZERO;
32
    }
33

34
    public String changeShippingAddress(ChangeShippingAddress cmd) {
35
        return "";
36
    }
37

38
    public void markAsPaid(MarkAsPaid cmd) {
39
        // 标记订单已支付
40
        order.setOrderStatus("PAID");
41
        addDomainEvent(new OrderPaidEvent());
42
    }
43

44
    public void markAsDelivered(MarkAsDelivered cmd) {
45
        // 标记订单已发货
46
        order.setOrderStatus("PAID");
47
        addDomainEvent(new OrderDeliveredEvent());
48
    }
49

50
    public void cancelOrder(CancelOrder cmd) {
51
        // 取消订单
52
        order.setOrderStatus("PAID");
53
        addDomainEvent(new OrderCanceledEvent());
54
    }
55

56
    private void addDomainEvent(DomanEvent event) {
57
        this.domainEvents.add(event);
58
    }
59

60
    // 在持久化到数据库后，清空已发送的领域事件
61
    public void clearDomainEvents() {
62
        this.domainEvents.clear();
63
    }
64
}
65

66
// 订单聚合根持久化接口
67
public interface OrderAggregateRootRepository {
68

69
    // 从数据库加载
70
    OrderAggregateRoot loadOrder(String orderNo);
71

72
    // 实现类通过 change tracing 实现变更字段的更新
73
    void saveOrder(OrderAggregateRoot orderAggregate);
74
}
75

76
// 订单领域服务
77
public class OrderService {
78

79
    private OrderAggregateRootRepository orderAggregateRootRepository;
80

81
    private DomainEventPublisher domainEventPublisher;
82

83
    public void createOrder(Object cmd) {
84
        OrderAggregateRoot root = OrderAggregateRoot.create(cmd);
85
        // 先持久化
86
        orderAggregateRootRepository.save(root);
87
        // 发送领域事件
88
        domainEventPublisher.publishEvents(root.getDomainEvents());
89
        // 清理领域事件
90
        root.clearDomainEvents();
91
    }
92

93
    public void cancelOrder(CancelOrder cmd) {
94
         OrderAggregateRoot root = orderAggregateRootRepository.loadOrder(cmd.getOrderNo());
95
         root.cancelOrder(cmd);
96
         // 发送领域事件
97
        domainEventPublisher.publishEvents(root.getDomainEvents());
98
        // 清理领域事件
99
        root.clearDomainEvents();
100
    }
101

102
}

展开收起

从上面的示例代码上，可以看出充血模型有以下特点：

所有写操作，都通过领域服务加载 **聚合根 **后，再通过聚合根提供的修改操作来实现数据修改的逻辑，然后再通过一次 save 操作更新到数据库中
加载 聚合根 时，会一次性从数据库中加载相关的所有数据信息（也可以考虑延迟加载，但是代码复杂度会上升）
一些单元测试，可以直接通过构造 **聚合根 **来进行单测，可以做到和数据库不相关

DDD 中层的划分#

image 39

贫血模型的实现思考#

一般习惯了 MVC 开发模式的开发团队，向 DDD 充血模型转换时，会遇到比较大的困难：

领域模型包含了更多的业务规则、领域事件、验证逻辑等，这增加了模型的复杂性；
思维模式一时半会儿难以适应，可能会觉得创建和维护这些丰富的聚合根、实体和值对象变得难度加大，开发周期会拉长；
DDD 的 data change tracking JPA天然支持，但是在国内 Mybatis 大流行的背景下，没有对应的主流开源实现，通常都是面向数据库编程；

所以，在项目落地时，是否可以适当妥协，继续使用贫血模型，但是，也借鉴聚合根的思想，精简设计我们的领域服务，减少不必要的 Service 类，例如以上例子，我们只考虑在代码中添加 OrderService

绝对不要出现 OrderItemService、ShippingAddressService ，所有涉及到 OrderItem、ShippingAddress 的写操作，都交由 OrderService 领域服务管理，在开发任务划分时，先讨论梳理出聚合根，再整理得到相关的操作，分配任务时，通过将聚合根来确定领域服务，实现任务功能的分配，这样每个分到任务的人，实现写业务逻辑时，能够相对独立完整的进行开发

1
public class OrderService {
2

3
    private OrderDao orderDao;
4

5
    private OrderItemDao orderItemDao;
6

7
    private ShippingAddressDao shippingAddressDao;
8

9
    public void createOrder(Object cmd) {
10
        // 业务逻辑实现
11
    }
12

13
    public void cancelOrder(CancelOrder cmd) {
14
        // 业务逻辑实现
15
    }
16
}

TODO#

一些涉及到只读数据查询的代码落地最佳实践，例如后端管理界面一些数据的查询组合，结合 CQRS(Command Query Responsibility Segregation，命令查询职责分离)，是否能够减少领域服务相关的查询逻辑代码，所有写的操作，都纳入到领域服务管理，查询相关的服务，单独构建一个服务，来管理这部分代码

那些场景适合使用DDD#

业务逻辑复杂、需要深刻理解领域知识的场景，在关系数据库模型上，通常会涉及到主表、字表状态的一致性，例如上面提到的订单主表和订单项字表，反例：字典表管理就不太适合使用DDD

MySQL 备份恢复

Wed, 05 Feb 2025 00:00:00 GMT

备份指定数据库#

Terminal window

1
mysqldump -u root -p \
2
  --databases db1 db2 db3 \
3
  --single-transaction \
4
  --set-gtid-purged=OFF \
5
  --quick \
6
  --routines \
7
  --events \
8
  > /var/lib/mysql/multiple-databases.sql

备份全部数据库#

Terminal window

1
mysqldump -u root -p \
2
  --all-databases \
3
  --single-transaction \
4
  --quick \
5
  --routines \
6
  --events \
7
  > /var/lib/mysql/all-databases.sql

备份用户和权限信息#

Terminal window

1
docker run --rm percona/percona-toolkit \
2
  pt-show-grants --user=root --password=pwd --host=192.168.50.21 > /var/lib/mysql/grants.sql

恢复数据库#

Terminal window

1
mysql -u root -p < backup.sql

MySQL主键自增

Mon, 30 Dec 2024 00:00:00 GMT

这里只记录 innodb 引擎的实现逻辑

MySQL ≤ 5.7#

自增值保存在内存中，在第一次打开表时，会找到 max(id)，会将 max(id) + 1 作为这个表当前的自增值。所以，如果 mysql 重启，可能会导致自增值变化。举例 table x id 如下，当前最大 id 为10，如果立马删除 id = 10 的数据，然后重启 mysql 实例，重启后，table x 的自增值会重新变为10（max(id) + 1 = 9 + 1）。

table x

id
1
2
…
9
10

MySQL ≥ 8.0#

自增值会在 redo log 中持久化，mysql 重启后会从 redo log 中恢复自增值

MySQL 主键自增值修改机制#

id_increament_test 初始情况

Terminal window

1
mysql> show create table id_increament_test;
2
+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
3
| Table              | Create Table                                                                                                                                                                       |
4
+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
5
| id_increament_test | CREATE TABLE `id_increament_test` (
6
  `id` int NOT NULL AUTO_INCREMENT,
7
  PRIMARY KEY (`id`)
8
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
9
+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

插入时，id = 0 或者 nul，主键id会按照当前自增值来赋值

1
insert into id_increament_test values (0),(null)

Terminal window

1
mysql> select * from id_increament_test;
2
+-----+
3
| id  |
4
+-----+
5
|   1 |
6
|   2 |
7
+-----+
8

9
mysql> show create table id_increament_test;
10
+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
11
| Table              | Create Table                                                                                                                                                                       |
12
+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
13
| id_increament_test | CREATE TABLE `id_increament_test` (
14
  `id` int NOT NULL AUTO_INCREMENT,
15
  PRIMARY KEY (`id`)
16
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
17
+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

假设当前自增值为 Y，插入时，如果指定了id值 X（X ≠ 0，并且 X值不与数据库中的任何值冲突），分为两种情况:
1. X < Y，插入数据 id = X，自增值不变

1
insert into id_increament_test values (-1)

Terminal window

1
mysql> select * from id_increament_test;
2
+-----+
3
| id  |
4
+-----+
5
|  -1 |
6
|   1 |
7
|   2 |
8
+-----+
9

10
mysql> show create table id_increament_test;
11
+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
12
| Table              | Create Table                                                                                                                                                                       |
13
+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
14
| id_increament_test | CREATE TABLE `id_increament_test` (
15
  `id` int NOT NULL AUTO_INCREMENT,
16
  PRIMARY KEY (`id`)
17
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
18
+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

1
2. X ≥ Y，插入数据 id = X，自增值变为 Y = X+(自增步长)

1
insert into id_increament_test values (100)

Terminal window

1
mysql> select * from id_increament_test;
2
+-----+
3
| id  |
4
+-----+
5
|  -1 |
6
|   1 |
7
|   2 |
8
| 100 |
9
+-----+
10

11
mysql> show create table id_increament_test;
12
+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
13
| Table              | Create Table                                                                                                                                                                       |
14
+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
15
| id_increament_test | CREATE TABLE `id_increament_test` (
16
  `id` int NOT NULL AUTO_INCREMENT,
17
  PRIMARY KEY (`id`)
18
) ENGINE=InnoDB AUTO_INCREMENT=101 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
19
+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

📎 参考文章#

https://javaguide.cn/database/mysql/mysql-auto-increment-primary-key-continuous.html

Spring 异步事件本地消息表设计

Thu, 19 Dec 2024 00:00:00 GMT

解决问题#

日常使用 spring 的 application event 实现业务解耦时，默认实现无法跟踪事件是否正常发送，下游业务是否正常消费。

例如用户下单成功后，需要给用户发送邮件通知，普通代码如下

1
@Transactional(rollbackFor = Exception.class)
2
public void createOrder(CreateOrderCmd cmd) {
3
  Order order = doSaveOrder(cmd);
4
  EmailContext emailCtx = prepareEmailContext(order);
5
  sendEmail(emailCtx);
6
}
7

8
public void sendEmail(EmailContext emailCtx) {
9
  // 邮件发送逻辑
10
}

以上代码有小概率出现邮件发送成功，用户下单失败数据库回滚场景

改良后的代码如下

1
@Transactional(rollbackFor = Exception.class)
2
public void createOrder(CreateOrderCmd cmd) {
3
  Order order = doSaveOrder(cmd);
4
  // 在数据库事务提交后执行发送邮件
5
  TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
6
    // 事务同步只在事务存在且成功提交时触发，如果事务回滚，afterCommit() 不会执行
7
    @Override
8
    public void afterCommit() {
9
      EmailContext emailCtx = prepareEmailContext(order);
10
      sendEmail(emailCtx);
11
    }
12
  });
13
}
14

15
public void sendEmail(EmailContext emailCtx) {
16
  // 邮件发送逻辑
17
}

以上写法，能够保证数据库提交后才触发发送短信发送操作，但是无法保证邮件发送操作正常完成（例如邮件网关拒收）

改进方案

1
@Transactional(rollbackFor = Exception.class)
2
public void createOrder(CreateOrderCmd cmd) {
3
  Order order = doSaveOrder(cmd);
4
  OrderCreatedEvent event = new OrderCreatedEvent();
5
  // ... setEvent
6
  saveLocalEvent(event);
7

8
  // 在数据库事务提交后执行发送短信
9
  TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
10
    @Override
11
    public void afterCommit() {
12
      applicationEventPublisher.publishEvent(event);
13
      // 更新事件发送状态
14
      updateEventSendStatus(event);
15
    }
16
  });
17
}
18

19
@EventListener
20
public void orderCreatedEventHandler(OrderCreatedEvent event) {
21
  EmailContext emailCtx = prepareEmailContext(event);
22
  sendEmail(emailCtx);
23
  // 保存事件消费记录
24
  saveLocalEventHandleLog(event);
25
}
26

27
@Transactional(rollbackFor = Exception.class)
28
public void saveLocalEventHandleLog(Object event) {
29
  // 将事件处理记录保存到数据库中
30
}
31

32
public void saveLocalEvent(Object event) {
33
  // 保存到数据库
34
}
35

36
public void sendEmail(EmailContext emailCtx) {
37
  // 邮件发送逻辑
38
}

本地消息表设计#

image 40

框架代码封装#

todo

其他衍生问题#

高并发业务，事件表、事件处理记录的分表设计、归档设计

📎 参考文章#

一些引用
引用文章

percona-toolkit 常用操作

Sun, 15 Dec 2024 00:00:00 GMT

数据归档#

Terminal window

1
docker run --rm percona/percona-toolkit \
2
  pt-archiver \
3
  --source h=192.168.50.21,u=root,p=xxx,D=sephy_app,t=rss_topic \
4
  --dest h=192.168.50.21,u=root,p=xxx,D=sephy_app,t=rss_topic_2022 \
5
  --where "post_time < '2023-01-01 00:00:00'" --limit 10000 --txn-size 1000 \
6
  --bulk-delete --statistics \
7
  --purge --no-check-charset --local --analyze=ds

用户权限备份#

Terminal window

1
docker run --rm percona/percona-toolkit \
2
  pt-show-grants --user=root --password=pwd --host=192.168.50.21 > /var/lib/mysql/grants.sql

📎 参考文章#

MacOS 备忘录

Mon, 14 Oct 2024 00:00:00 GMT

iterm3 关闭 Clicking on a command selects it to restrict Find and Filter#

1
iTerm2 Settings -> General -> Selection
2
取消勾选：
3
Clicking on a command selects it to restrict Find and Filter

Mail APP 不跟随系统主题配色#

Terminal window

1
# 不跟随
2
defaults write com.apple.mail NSRequiresAquaSystemAppearance -bool yes
3
# 恢复
4
defaults delete com.apple.mail NSRequiresAquaSystemAppearance

允许安装第三方来源应用#

Terminal window

1
sudo spctl --master-disable
2
# sudo spctl --master-enable 恢复

应用提示文件损坏/修复”已损坏，无法打开”错误#

Terminal window

1
# xattr：操作扩展属性的工具。
2
# -c：清除（Remove） 所有扩展属性。
3
# -r：递归（Recursive） 处理目录及其所有子内容
4
sudo xattr -cr /Applications/YourApp.app

Spotlight 索引操作#

Terminal window

1
# 禁用索引
2
sudo mdutil -i off /
3

4
# 清除索引
5
sudo mdutil -E /
6

7
# 重新启用索引
8
sudo mdutil -i on /
9

10
# 手动启动索引
11
sudo mdutil -s /
12

13
# 检查索引进度
14
mdutil -s /
15

16
# 停止 spotlight 索引服务
17
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist
18

19
# 启动 spotlight 服务
20
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist

在 macOS 上有个系统层面的特性可以把多个 IntelliJ 窗口当作标签显示#

macOS System Preferences → Desktop & Dock → Prefer tabs when opening documents 设置为 “Always”。

终端命令行 sudo 使用指纹#

编辑配置文件

Terminal window

1
sudo vim /etc/pam.d/sudo

找到 auth sufficient 将 pam_smartcard.so 改为 pam_tid.so

修改前

Terminal window

1
auth       include        sudo_local
2
auth       sufficient     pam_smartcard.so
3
auth       required       pam_opendirectory.so
4
account    required       pam_permit.so
5
password   required       pam_deny.so
6
session    required       pam_permit.so

修改后

Terminal window

1
auth       include        sudo_local
2
auth       sufficient     pam_tid.so
3
auth       required       pam_opendirectory.so
4
account    required       pam_permit.so
5
password   required       pam_deny.so
6
session    required       pam_permit.so

wq! 保存（必须要加! wq 无法保存会提示 read only）

完成后效果如下

image 41

调整顶部 menubar 的间距#

image 42|调整前效果

image 43|调整后效果

默认的间距有点宽，如果 menubar 的内容项多的话，空间放不下（或者使用 bartender 相关工具进行隐藏）

缩小命令

值可以在 0-6 之间调整，当值为 0 时，间距最小

Terminal window

1
# Change the whitespace settings value
2
defaults -currentHost write -globalDomain NSStatusItemSelectionPadding -int 4
3
defaults -currentHost write -globalDomain NSStatusItemSpacing -int 4
4

5
# After running these commands, you need to log out and log back in

要退出登录，重新登录才会生效

恢复默认间距命令

Terminal window

1
# Revert to the original values
2
defaults -currentHost delete -globalDomain NSStatusItemSelectionPadding
3
defaults -currentHost delete -globalDomain NSStatusItemSpacing
4

5
# After running these commands, you need to log out and log back in

同样，要退出登录，重新登录才会生效

**menu bar spacing 可以直接修改 **

Menu Bar Spacing

command+shift+4 获取屏幕坐标#

路由操作#

Terminal window

1
# 查看路由
2
netstat -nr
3
# 添加路由
4
route -n add -net 192.168.0.0（需进入的网段） -netmask 255.255.255.0 （掩码）192.168.5.254 （进该网段的网关）
5
# 路由删除
6
route -v delete -net 10.10.12.0（某网段） -gateway 10.10.12.1（某网关）

路由信息中Flags的一些具体含义如下：

U = Up
G = Gateway
S = Static, e.g., default route added at boot time
H = Host-specific
C = Generate new (host-specific) routes on use
L = Valid link-layer (MAC) address
c = Cloned route
R = Reject route, known but unreachable route

清理 DNS 缓存#

Terminal window

1
sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

fastfetch 配置#

1
echo "export XDG_CONFIG_HOME=$HOME/.config" >> ~/.zshrc
2

3
mkdir -p ~/.config/fastfetch/config.conf
4

5
echo >> ~/.config/fastfetch/config.conf <<EOF
6
# iterm 配置
7
--logo-type "iterm"
8
# 指向图片路径
9
--logo "/Users/sephy/Pictures/m1_chip.jpeg"
10
# iterm 配置 长宽比例必填
11
--logo-width "55"
12
--logo-height "25"
13
EOF

常用Mac软件下载地址#

📎 参考文章#

https://flaky.build/built-in-workaround-for-applications-hiding-under-the-macbook-pro-notch

系统常用字体设置

Sun, 13 Oct 2024 00:00:00 GMT

MesloLG Nerd Font#

这个字体配合 iterm2 控制台使用

image 44

安装 font-meslo-lg-nerd-font

Terminal window

1
brew install --cask font-meslo-lg-nerd-font

JetBrains Mono Nerd Font + Maple Font#

英文 = 0.5 中文字符，配合 IntelliJ 实现中英文等宽对齐

image 45

安装 font-jetbrains-mono-nerd-font

Terminal window

1
brew install --cask font-jetbrains-mono-nerd-font

📎 参考文章#

Drone-docker-runner 共享宿主机maven缓存

Sat, 24 Aug 2024 00:00:00 GMT

Drone 的流水线，在使用 maven 构建的过程中，会下载比较多的 jar 文件，因为用的是 drone-docker-runner，考虑能否使用宿主机上的空间，来缓存这部分 .m2 文件夹，避免每次都重新下载依赖文件，从而加速构建。简单研究了下，实现方式如下

在流水线对应的 repo 中，进入 Settings，在 General 面板中开启 Trusted 权限
image 46
流水线配置语法如下

1
kind: pipeline
2
type: docker
3
name: sonar-scan
4

5
steps:
6
  - name: maven-compile
7
    image: maven:3.9.6-eclipse-temurin-21
8
    volumes: # 将容器内目录挂载到宿主机，仓库需要开启Trusted设置
9
    - name: maven-cache # 对应声明好的 volumes.name
10
      path: /root/.m2 # 挂载到容器内的路径
11
    commands:
12
      - mvn clean compile
13

14
volumes: # 定义流水线挂载目录，用于共享数据# 从宿主机中挂载的目录
15
  # https://docs.drone.io/pipeline/docker/syntax/volumes/host/
16
  # maven 缓存
17
  - name: maven-cache
18
    host:
19
      path: /home/sephy/.m2 # 在宿主机上的路径

📎 参考文章#

https://docs.drone.io/pipeline/docker/syntax/volumes/host/

XXL-JOB

Mon, 01 Jul 2024 00:00:00 GMT

核心执行逻辑(版本2.2.0)#

com.xxl.job.admin.core.thread.JobScheduleHelper#start

scheduleThread#

该线程负责从数据库中预读将要执行的任务，并将其放入到 ringData 中用于后续真正的调度线程进行调度

1. 线程初始化

1
scheduleThread = new Thread(new Runnable() {
2
    @Override
3
    public void run() {
4
        // 线程执行的逻辑
5
    }
6
});

创建了一个新的线程 scheduleThread，并实现了 Runnable 接口的 run 方法。

2. 线程启动前的延迟

1
try {
2
    TimeUnit.MILLISECONDS.sleep(5000 - System.currentTimeMillis() % 1000);
3
} catch (InterruptedException e) {
4
    if (!scheduleThreadToStop) {
5
        logger.error(e.getMessage(), e);
6
    }
7
}

线程启动后，首先会休眠一段时间，确保线程在整秒时刻启动（即 System.currentTimeMillis() % 1000 为 0 的时刻）。这样可以使得后续的调度任务能够对齐到整秒。

3. 预读任务数量计算

1
int preReadCount = (XxlJobAdminConfig.getAdminConfig().getTriggerPoolFastMax() + XxlJobAdminConfig.getAdminConfig().getTriggerPoolSlowMax()) * 20;

计算预读任务的数量。preReadCount 是根据线程池的大小和触发器的 QPS（每秒查询率）来计算的。假设每个触发器耗时 50ms，那么 QPS 为 20（1000ms / 50ms）

4. 主循环

1
while (!scheduleThreadToStop) {
2
    // 主循环逻辑
3
}

主循环会一直运行，直到 scheduleThreadToStop 被设置为 true

5. 数据库连接与事务管理

1
Connection conn = null;
2
Boolean connAutoCommit = null;
3
PreparedStatement preparedStatement = null;
4

5
try {
6
    conn = XxlJobAdminConfig.getAdminConfig().getDataSource().getConnection();
7
    connAutoCommit = conn.getAutoCommit();
8
    conn.setAutoCommit(false);
9

10
    preparedStatement = conn.prepareStatement("select * from xxl_job_lock where lock_name = 'schedule_lock' for update");
11
    preparedStatement.execute();

获取数据库连接，并开启事务。通过 for update 语句锁定 xxl_job_lock 表中的 schedule_lock 记录，确保在同一时间只有一个调度线程在执行
这里要思考如果获取到数据库悲观锁之后，JVM挂掉，会不会一直无法释放锁导致其他节点无法获取数据库悲观锁

6. 预读任务

1
List<XxlJobInfo> scheduleList = XxlJobAdminConfig.getAdminConfig().getXxlJobInfoDao().scheduleJobQuery(nowTime + PRE_READ_MS, preReadCount);

从数据库中预读即将触发的任务列表。nowTime + PRE_READ_MS 表示预读的时间范围，preReadCount 是预读的任务数量。

7. 任务处理

1
for (XxlJobInfo jobInfo: scheduleList) {
2
    if (nowTime > jobInfo.getTriggerNextTime() + PRE_READ_MS) {
3
        // 任务过期处理
4
    } else if (nowTime > jobInfo.getTriggerNextTime()) {
5
        // 任务触发处理
6
    } else {
7
        // 任务预读处理
8
    }
9
}

遍历预读的任务列表，根据任务的触发时间进行不同的处理：
- 任务过期：如果任务的触发时间已经超过预读时间范围，则标记任务为过期，并更新下一次触发时间。
- 任务触发：如果任务的触发时间在当前时间和预读时间之间，则立即触发任务，并更新下一次触发时间。
- 任务预读：如果任务的触发时间在未来，则将任务放入时间环中，等待后续触发。

8. 更新任务信息

1
if (conn != null) {
2
    try {
3
        conn.commit();
4
    } catch (SQLException e) {
5
        if (!scheduleThreadToStop) {
6
            logger.error(e.getMessage(), e);
7
        }
8
    }
9
    // 恢复自动提交状态并关闭连接
10
}

更新数据库中任务的信息，确保任务的触发时间和状态是最新的。

9. 事务提交与资源释放

1
if (conn != null) {
2
    try {
3
        conn.commit();
4
    } catch (SQLException e) {
5
        if (!scheduleThreadToStop) {
6
            logger.error(e.getMessage(), e);
7
        }
8
    }
9
    // 恢复自动提交状态并关闭连接
10
}

提交事务，并释放数据库连接和 PreparedStatement 资源。

10. 等待下一次调度

1
long cost = System.currentTimeMillis()-start;
2

3
if (cost < 1000) {
4
    try {
5
        TimeUnit.MILLISECONDS.sleep((preReadSuc ? 1000 : PRE_READ_MS) - System.currentTimeMillis() % 1000);
6
    } catch (InterruptedException e) {
7
        if (!scheduleThreadToStop) {
8
            logger.error(e.getMessage(), e);
9
        }
10
    }
11
}

如果本次调度耗时小于 1 秒，则线程会休眠一段时间，确保下一次调度在整秒时刻执行。

ringThread#

该线程负责时间轮的推进（单位为秒），并执行对应的任务

1. 线程初始化

1
ringThread = new Thread(new Runnable() {
2
    @Override
3
    public void run() {
4
        // 线程执行的逻辑
5
    }
6
});

2. 线程启动前的延迟

1
try {
2
    TimeUnit.MILLISECONDS.sleep(1000 - System.currentTimeMillis() % 1000);
3
} catch (InterruptedException e) {
4
    if (!ringThreadToStop) {
5
        logger.error(e.getMessage(), e);
6
    }
7
}

熟悉的操作，线程启动后，首先会休眠一段时间，确保线程在整秒时刻启动（即 System.currentTimeMillis() % 1000 为 0 的时刻）。这样可以使得后续的时间环任务能够对齐到整秒。

3. 主循环

1
while (!ringThreadToStop) {
2
    // 主循环逻辑
3
}

4. 获取当前秒数

1
int nowSecond = Calendar.getInstance().get(Calendar.SECOND);

获取当前的秒数（0-59），用于确定当前时间环的位置。

5. 从时间环中获取任务

1
List<Integer> ringItemData = new ArrayList<>();
2
for (int i = 0; i < 2; i++) {
3
    List<Integer> tmpData = ringData.remove((nowSecond + 60 - i) % 60);
4
    if (tmpData != null) {
5
        ringItemData.addAll(tmpData);
6
    }
7
}

从时间环数据结构 ringData 中获取当前秒和前一秒的任务列表。
- ringData 是一个以秒为索引的哈希表，存储了每个秒数对应的任务列表。
- (nowSecond + 60 - i) % 60 用于计算当前秒和前一秒的索引，避免跨过刻度。
- 如果某个秒数对应的任务列表存在，则将其合并到 ringItemData 中。

6. 触发任务

1
if (ringItemData.size() > 0) {
2
    for (int jobId : ringItemData) {
3
        JobTriggerPoolHelper.trigger(jobId, TriggerTypeEnum.CRON, -1, null, null, null);
4
    }
5
    ringItemData.clear();
6
}

如果 ringItemData 中有任务，则遍历任务列表并触发每个任务。
- JobTriggerPoolHelper.trigger 是触发任务的核心方法，会根据任务 ID 执行相应的任务。
触发完成后，清空 ringItemData，以便下一次使用。

7. 异常处理

1
catch (Exception e) {
2
    if (!ringThreadToStop) {
3
        logger.error(">>>>>>>>>>> xxl-job, JobScheduleHelper#ringThread error:{}", e);
4
    }
5
}

捕获并记录线程运行过程中发生的异常。如果线程未被要求停止（ringThreadToStop 为 false），则记录错误日志

8. 等待下一次触发

1
try {
2
    TimeUnit.MILLISECONDS.sleep(1000 - System.currentTimeMillis() % 1000);
3
} catch (InterruptedException e) {
4
    if (!ringThreadToStop) {
5
        logger.error(e.getMessage(), e);
6
    }
7
}

线程休眠一段时间，确保下一次触发在整秒时刻执行。休眠时间通过 1000 - System.currentTimeMillis() % 1000 计算，保证对齐到整秒。

com.xxl.job.admin.core.thread.JobScheduleHelper#toStop

停止时，会先停止预读任务线程，如果有待执行的任务，还会 sleep 8秒等待相应的任务被调度完成

1
public void toStop(){
2

3
    // 1、stop schedule
4
    scheduleThreadToStop = true;
5
    try {
6
        TimeUnit.SECONDS.sleep(1);  // wait
7
    } catch (InterruptedException e) {
8
        logger.error(e.getMessage(), e);
9
    }
10
    if (scheduleThread.getState() != Thread.State.TERMINATED){
11
        // interrupt and wait
12
        scheduleThread.interrupt();
13
        try {
14
            scheduleThread.join();
15
        } catch (InterruptedException e) {
16
            logger.error(e.getMessage(), e);
17
        }
18
    }
19

20
    // if has ring data
21
    boolean hasRingData = false;
22
    if (!ringData.isEmpty()) {
23
        for (int second : ringData.keySet()) {
24
            List<Integer> tmpData = ringData.get(second);
25
            if (tmpData!=null && tmpData.size()>0) {
26
                hasRingData = true;
27
                break;
28
            }
29
        }
30
    }
31
    if (hasRingData) {
32
        try {
33
            TimeUnit.SECONDS.sleep(8);
34
        } catch (InterruptedException e) {
35
            logger.error(e.getMessage(), e);
36
        }
37
    }
38

39
    // stop ring (wait job-in-memory stop)
40
    ringThreadToStop = true;
41
    try {
42
        TimeUnit.SECONDS.sleep(1);
43
    } catch (InterruptedException e) {
44
        logger.error(e.getMessage(), e);
45
    }
46
    if (ringThread.getState() != Thread.State.TERMINATED){
47
        // interrupt and wait
48
        ringThread.interrupt();
49
        try {
50
            ringThread.join();
51
        } catch (InterruptedException e) {
52
            logger.error(e.getMessage(), e);
53
        }
54
    }
55

56
    logger.info(">>>>>>>>>>> xxl-job, JobScheduleHelper stop");
57
}

展开收起

如何做好技术评审

Thu, 13 Jun 2024 00:00:00 GMT

当前工作中，平均每周要参加 1~2 次技术评审工作，每个业务线团队的评审会参差不齐。有些评审会信息传递清晰、高效，过程非常愉悦，大家也能参入其中并理解和汲取所需。有些评审会冗长、混乱，那些应该覆盖的信息大量缺失，评审会完成依然“不知其所云”，参与评审的人经常没有参与感。

技术方案的本质是从技术视角出发提供的问题域的解决方案，技术方案评审的本质是一个协作决策过程，通过干系人共同参与，从多维度评估待评审的技术方案是否对问题域提供了合理、可行且最优的解决方案。

评估技术方案的完整性、正确性、合理性

**完整性 **要求方案不能对要解决的问题有所遗漏，不完整意味着不能满足需求。

**正确性 **要求技术方案能够对问题域提供解决方案。

合理性 则要求不仅仅关乎”正确“和“完整”，其要求在特定的上下文中多个因素间的平衡，比如，时间成本、资源投入成本、技术实现成本等多因素平衡。

技术评审中，以下角色应该各司其职，深度参与

架构师/TL

整体技术方案的架构合理性和一致性
横向架构属性满足度：可扩展性、可维护性、性能、安全、合规性等
对现有系统架构的影响评估
工期与技术成本的合理性
技术栈选型与团队能力的匹配度

产品品经理

功能性需求的实现是否覆盖
用户敏感的非功能性需求的实现是否覆盖
业务概念的一致性
产品视角的安全：敏感信息明文/密文存储与反显；敏感数据泄漏；资损等等
系统状态机是否满足业务状态机要求

测试工程师

方案的可测试性
是否需要压测
变更影响需要回归的范围

技术评审文档模板#

业务上下文

同步业务相关信息，区分事情的重要和紧急程度；包括但不限于业务的整体背景、ROI、长期和短期目标、产品方案等

技术上下文

第一个维度，提供当前系统现状的技术背景说明，包括但不限于整体技术架构、系统部署架构第二个维度，架构师基于对业务、产品、技术现状及诉求的理解，分析当前重点衍生的技术需求，作为产品需求在技术视角的补充，包括但不限于诸如技术安全、业务安全、性能要求、规模要求，目的是尽量避免技术需求的遗漏，以及为关键的部分预留可迭代的空间；
技术安全：功能安全、数据安全、存储加密、传输加密、签名验证、防伪造和篡改的问题，要防范安全漏洞。
业务安全：多个业务共享数据时需要重点考虑，事务安全、保证一致性、要预防黑产攻击。

概要设计

对系统的宏观问题提供解决方案，典型的元素包括： 1.整体方案选型：基础设施选型、技术栈选型、子系统拆分及交互的高层选型等等
2.应用架构及其变更
3.技术架构：系统稳定性、性能、安全性的方案选型

详细设计

这部分信息是领域实现的设计，关注领域内的元素和他们之间的关系。一般是根据领域进行分工，各领域可以独立完成设计；目的是把控质量、指导开发，降低后期的风险；典型的包含以下元素： 1.领域模型
2.交互流程
3.领域实体状态机
4.数据模型
5.关键协议
6.接口设计
7.监控/验证及灰度

影响分析

描述当前设计方案对系统、业务、团队所带来的影响

风险评估

目的是为了消除问题，避免小问题可能变成大问题，因为在实际情况下，很多出现的问题都是曾经考虑过但是又被忽略的问题

决策过程

技术方案的设计过程是进行权衡分析的过程，方案中应该对过程所做决策进行留存记录。

以上根据实际情况，进行适当裁剪

N5105 PVE 爱快网口设置

Sun, 09 Jun 2024 00:00:00 GMT

22年买的畅网 N5105v1 版本，今天把上面的 NVME SSD 拔下后，发现家里的网络噶了。这里记录一下解决的过程。

网口的 PCI 识别编号，在 NVME SSD 拔下后发生了变化，从 enp2s0、enp3s0、enp4s0、enp5s0 变成了 enp1s0、enp2s0、enp3s0、enp4s0。

重新划分网口功能用途#

Untitled 10

网口1 作为PVE设备的管理端口，编辑 pve 的 /etc/network/interfaces 文件，其中将 vmbr0 映射到 enp1s0 网口，并配置网关地址为 192.168.50.1（后续爱快的地址）。

1
auto lo
2
iface lo inet loopback
3

4
iface enp1s0 inet manual
5

6
iface enp2s0 inet manual
7

8
iface enp3s0 inet manual
9

10
iface enp4s0 inet manual
11

12
auto vmbr0
13
iface vmbr0 inet static
14
  address 192.168.50.2/24
15
  gateway 192.168.50.1
16
  bridge-ports enp1s0
17
  bridge-stp off
18
  bridge-fd 0

Untitled 11

网口2、网口3作为爱快虚拟机的 LAN 口网口4作为爱快虚拟机的 WAN 口

Untitled 12

在 ikuai 的控制台，通过命令行 set lan1 eth0 将虚拟网卡设置为 LAN1，set wan1 eth3 将网口3设置为 WAN1 口

Untitled 13

进入爱快的管理界面，在点击的 LAN1 图标，在高级选项中，将网口2、网口3加入到 LAN1

Untitled 14|点 lan1 进入配置界面

Untitled 15|勾选 eth1、eth2

折腾 PVE 系统过程中的一些记录

Sun, 09 Jun 2024 00:00:00 GMT

导入镜像到 lvm#

Terminal window

1
qm importdisk 104 istoreos-24.10.2-2025092610-x86-64-squashfs-combined-efi.img local-lvm

web 管理界面进不去，怎么修改虚拟机的配置#

在 /etc/pve/qemu-server/ 目录下，有对应的虚拟机配置文件，可以直接编辑这些文件

Terminal window

1
root@pve:~# ls /etc/pve/qemu-server/
2
101.conf  103.conf
3

4
root@pve:~# cat /etc/pve/qemu-server/103.conf
5
boot: order=scsi0;net0
6
cores: 4
7
cpu: host
8
cpuunits: 1025
9
hostpci0: 0000:02:00.0
10
hostpci1: 0000:03:00.0
11
hostpci2: 0000:04:00.0
12
machine: q35
13
memory: 1024
14
meta: creation-qemu=6.1.0,ctime=1650336443
15
name: iKuai
16
net0: virtio=5A:9F:73:43:6E:79,bridge=vmbr0,firewall=1
17
numa: 0
18
onboot: 1
19
ostype: l26
20
scsi0: local-lvm:vm-103-disk-0,size=1G,ssd=1
21
scsihw: virtio-scsi-pci
22
smbios1: uuid=700abdb0-b87b-490b-9942-5b313fe3b5ee
23
sockets: 1
24
startup: order=1,down=30
25
vmgenid: 1bfbc33f-8f42-4723-ad79-9e9733857be6

开启硬件直通#

开启 iommu

修改 vim /etc/default/grub

Terminal window

1
# GRUB_CMDLINE_LINUX_DEFAULT="quiet" # 默认
2
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt" # amd 用这个
3
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt" # intel 用这个

该命令具体的含义，chat-gpt 解释如下

在给Linux内核启动参数中，“quiet”表示在启动过程中不显示冗长的输出信息，而”amd_iommu=on”表示启用AMD处理器上的IOMMU（Input-Output Memory Management Unit）功能。 “iommu=pt”是一种IOMMU模式的参数设置。IOMMU是一种硬件功能，它允许系统在处理输入输出（I/O）操作时对内存进行映射和保护。“iommu=pt”表示将IOMMU设置为”Pass-Through”（PT）模式，也称为”identity mapping”。在PT模式下，IOMMU不执行内存地址转换，而是直接将设备的物理地址映射到相应的物理内存地址，以提供更低的延迟和更高的性能。
总而言之，“quiet amd_iommu=on iommu=pt”是一组启动参数，用于在Linux系统中启用AMD处理器上的IOMMU功能，并将其设置为”Pass-Through”模式，以提供更好的性能和较低的延迟。

修改完后，更新 update-grub

验证是否开启 iommu dmesg | grep iommu

如果出现 pci 0000:00:01.0: Adding to iommu group x 的输出表示成功

Terminal window

1
root@pve:~# dmesg | grep iommu
2
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.2.16-3-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on
3
[    0.026281] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.2.16-3-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on
4
[    0.581715] iommu: Default domain type: Translated
5
[    0.581715] iommu: DMA domain TLB invalidation policy: lazy mode
6
[    0.600753] pci 0000:00:01.0: Adding to iommu group 0
7
[    0.600764] pci 0000:00:01.1: Adding to iommu group 1
8
[    0.600773] pci 0000:00:01.3: Adding to iommu group 2
9
[    0.600780] pci 0000:00:02.0: Adding to iommu group 3
10
[    0.600788] pci 0000:00:03.0: Adding to iommu group 4
11
[    0.600796] pci 0000:00:04.0: Adding to iommu group 5
12
[    0.600803] pci 0000:00:05.0: Adding to iommu group 6
13
[    0.600813] pci 0000:00:07.0: Adding to iommu group 7
14
[    0.600821] pci 0000:00:07.1: Adding to iommu group 8
15
[    0.600832] pci 0000:00:08.0: Adding to iommu group 9
16
[    0.600841] pci 0000:00:08.1: Adding to iommu group 10
17
[    0.600850] pci 0000:00:14.0: Adding to iommu group 11
18
[    0.600855] pci 0000:00:14.3: Adding to iommu group 11
19
[    0.600877] pci 0000:00:18.0: Adding to iommu group 12
20
[    0.600883] pci 0000:00:18.1: Adding to iommu group 12
21
[    0.600889] pci 0000:00:18.2: Adding to iommu group 12
22
[    0.600894] pci 0000:00:18.3: Adding to iommu group 12
23
[    0.600900] pci 0000:00:18.4: Adding to iommu group 12
24
[    0.600905] pci 0000:00:18.5: Adding to iommu group 12
25
[    0.600910] pci 0000:00:18.6: Adding to iommu group 12
26
[    0.600916] pci 0000:00:18.7: Adding to iommu group 12
27
[    0.600929] pci 0000:01:00.0: Adding to iommu group 13
28
[    0.600950] pci 0000:03:00.0: Adding to iommu group 14
29
[    0.600964] pci 0000:03:00.1: Adding to iommu group 14
30
[    0.600978] pci 0000:03:00.2: Adding to iommu group 14
31
[    0.600982] pci 0000:20:00.0: Adding to iommu group 14
32
[    0.600985] pci 0000:20:01.0: Adding to iommu group 14
33
[    0.600989] pci 0000:20:04.0: Adding to iommu group 14
34
[    0.600995] pci 0000:22:00.0: Adding to iommu group 14
35
[    0.601001] pci 0000:25:00.0: Adding to iommu group 14
36
[    0.601009] pci 0000:26:00.0: Adding to iommu group 15
37
[    0.601022] pci 0000:27:00.0: Adding to iommu group 16
38
[    0.601033] pci 0000:27:00.1: Adding to iommu group 17
39
[    0.601045] pci 0000:27:00.3: Adding to iommu group 18
40
[    0.601056] pci 0000:27:00.4: Adding to iommu group 19
41
[    0.603983] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

通过 lspci 查看 pci 信息

Terminal window

1
root@pve:~# lspci
2
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
3
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
4
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
5
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
6
00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
7
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
8
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
9
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
10
00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
11
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
12
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
13
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
14
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
15
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
16
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
17
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0
18
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1
19
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2
20
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3
21
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4
22
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5
23
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6
24
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7
25
01:00.0 Non-Volatile memory controller: Sandisk Corp WD Black 2018/SN750 / PC SN720 NVMe SSD
26
03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 xHCI Compliant Host Controller (rev 01)
27
03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)
28
03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge (rev 01)
29
20:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
30
20:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
31
20:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
32
22:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
33
25:00.0 Non-Volatile memory controller: Micron/Crucial Technology P5 NVMe PCIe SSD[SlashP5]
34
26:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
35
27:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
36
27:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
37
27:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
38
27:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller

在输出中，查找与NVMe设备相关的条目。通常，NVMe设备的描述中会包含关键词”NVMe”、“Non-Volatile”或”Storage”，可以用于快速定位

以上信息输出看着还是比较模糊，通过 ls -la /sys/dev/block/|grep -v loop |grep -v dm 查看硬盘所属控制器

Terminal window

1
ls -la /sys/dev/block/|grep -v loop |grep -v dm
2
root@pve:~# ls -la /sys/dev/block/|grep -v loop |grep -v dm
3
total 0
4
drwxr-xr-x 2 root root 0 Jun  8 22:45 .
5
drwxr-xr-x 4 root root 0 Jun  8 22:45 ..
6
lrwxrwxrwx 1 root root 0 Jun  8 22:45 259:0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/nvme/nvme0/nvme0n1
7
lrwxrwxrwx 1 root root 0 Jun  8 22:45 259:1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/nvme/nvme0/nvme0n1/nvme0n1p1
8
lrwxrwxrwx 1 root root 0 Jun  8 22:45 259:2 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/nvme/nvme0/nvme0n1/nvme0n1p2
9
lrwxrwxrwx 1 root root 0 Jun  8 22:45 259:3 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/nvme/nvme0/nvme0n1/nvme0n1p3
10
lrwxrwxrwx 1 root root 0 Jun  8 22:45 259:4 -> ../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:20:04.0/0000:25:00.0/nvme/nvme1/nvme1n1
11
lrwxrwxrwx 1 root root 0 Jun  8 22:45 259:5 -> ../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:20:04.0/0000:25:00.0/nvme/nvme1/nvme1n1/nvme1n1p1
12
lrwxrwxrwx 1 root root 0 Jun  8 22:45 259:6 -> ../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:20:04.0/0000:25:00.0/nvme/nvme1/nvme1n1/nvme1n1p2
13
lrwxrwxrwx 1 root root 0 Jun  8 22:45 259:7 -> ../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:20:04.0/0000:25:00.0/nvme/nvme1/nvme1n1/nvme1n1p3

../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:20:04.0/0000:25:00.0/nvme/nvme1/nvme1n1是一个设备文件路径，它描述了一个NVMe硬盘在Linux系统中的位置。让我们逐级解释该路径：

../../devices: 这表示当前位置位于设备树中的devices目录的上两级目录。这是一个相对路径，用于指示设备所在的位置。

pci0000:00: 这是PCI总线的顶级目录。

0000:00:01.3: 这是PCI设备的地址，表示第一个PCI域（domain）的第一个总线（bus）的第一个设备（device）的第三个功能（function）。PCI设备地址通常由多个字段组成，用冒号分隔。

0000:03:00.2: 这是上述设备的子设备地址，表示该设备的第二个子设备的地址。

0000:20:04.0: 这是上述子设备的子设备地址，表示该子设备的第一个子设备的地址。

0000:25:00.0: 这是上述子设备的子设备地址，表示该子设备的第一个子设备的地址。

nvme/nvme1: 这些是设备的名称，表示这是一个NVMe设备，并且其标识符为nvme1。

nvme1n1: 这是NVMe设备的第一个分区或逻辑卷的名称。

综上所述，该路径描述了一个位于PCI总线上的NVMe硬盘设备，并指定了其在设备树中的位置和名称。

如果不知道pve所在的硬盘。可以使用命令 lsblk 查看，有pve-root的，便是pve所在的硬盘

Terminal window

1
root@pve:~# lsblk
2
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
3
nvme0n1                      259:0    0 465.8G  0 disk
4
├─nvme0n1p1                  259:1    0  1007K  0 part
5
├─nvme0n1p2                  259:2    0     1G  0 part /boot/efi
6
└─nvme0n1p3                  259:3    0 464.8G  0 part
7
  ├─pve-swap                 253:0    0     8G  0 lvm  [SWAP]
8
  ├─pve-root                 253:1    0    96G  0 lvm  /
9
  ├─pve-data_tmeta           253:2    0   3.4G  0 lvm
10
  │ └─pve-data-tpool         253:4    0 337.9G  0 lvm
11
  │   ├─pve-data             253:5    0 337.9G  1 lvm
12
  │   ├─pve-vm--100--disk--0 253:6    0     4M  0 lvm
13
  │   └─pve-vm--100--disk--1 253:7    0   300G  0 lvm
14
  └─pve-data_tdata           253:3    0 337.9G  0 lvm
15
    └─pve-data-tpool         253:4    0 337.9G  0 lvm
16
      ├─pve-data             253:5    0 337.9G  1 lvm
17
      ├─pve-vm--100--disk--0 253:6    0     4M  0 lvm
18
      └─pve-vm--100--disk--1 253:7    0   300G  0 lvm
19
nvme1n1                      259:4    0 931.5G  0 disk
20
├─nvme1n1p1                  259:5    0   2.4G  0 part
21
├─nvme1n1p2                  259:6    0     2G  0 part
22
└─nvme1n1p3                  259:7    0 926.9G  0 part

硬盘直通时，勾选 All Functions 和 PCI-EXPRESS

Untitled 7

RDM直通

Untitled 8

qm set 100 —scsi1 /dev/disk/by-id/nvme-CT1000P5SSD8_21042CC69BA9

100 是vmid, scsi1 是硬盘编号

Untitled 9

📎 参考文章#

通过 chezmoi 来管理 dotfiles

Mon, 06 May 2024 00:00:00 GMT

从远程仓库克隆并还原

Terminal window

1
chezmoi init https://github.com/yourusername/dotfiles.git
2
chezmoi apply

更新并还原最新更改

Terminal window

1
chezmoi update
2
chezmoi apply

高级用法

Terminal window

1
# 还原时排除某些文件
2
chezmoi apply --exclude .config/someapp
3

4
# 仅还原某些文件
5
chezmoi apply --include .vimrc

Terminal window

1
# 编辑并应用 ~/.zshrc 文件
2
chezmoi edit -a ~/.zshrc
3

4
# 移除某个文件的同步
5
chezmoi remove ~/.ssh/path_to_private_key
6

7
# 增加某个文件并使用加密方式来托管
8
chezmoi add --encrypt ~/.ssh/path_to_private_key

参考#

[归档] 用 Chezmoi 管理配置文件

micromamba 工作环境配置

Tue, 23 Apr 2024 00:00:00 GMT

之前用的 miniconda，近期看到了 mamba 这个项目，看了下官网，有个 micromamba 的基于 C++ 开发的，性能会更好一些。

micromamba is a fully statically-linked, self-contained, executable. This means that the base environment is completely empty. The configuration for micromamba is slightly different, namely all environments and cache will be created by default under the MAMBA_ROOT_PREFIX environment variable. There is also no pre-configured .condarc/.mambarc shipped with micromamba (they are however still read if present).

micromamba 的特点:

使用多线程下载 repository data 和 packages（最新的 conda 也支持多线程了）;
mamba 使用 libsolv 加速解决依赖关系；
mamba 的核心部分通过 C++ 实现，以获得最大执行效率；
mamba 可与 conda 完美兼容；

homebrew 方式安装#

Terminal window

1
brew install micormamba

linux 安装#

Terminal window

1
"${SHELL}" <(curl -L micro.mamba.pm/install.sh)

初始化配置#

Terminal window

1
mkdir -p ~/micromamba
2

3
# 与 zsh 集成
4
micromamba shell init -s zsh -p ~/micromamba
5

6
source ~/.zshrc

配置清华源#

Terminal window

1
cat >> ~/.condarc << EOF
2
channels:
3
  - defaults
4
show_channel_urls: true
5
default_channels:
6
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
7
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
8
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2
9
custom_channels:
10
  conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
11
  msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
12
  bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
13
  menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
14
  pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
15
  pytorch-lts: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
16
  simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
17
  deepmodeling: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/
18
EOF
19

20
# 清除索引缓存
21
micromamba clean -i
22

23
# 和conda不一样，使用 micromamba 必须手动设置channels
24
micromamba config prepend channels conda-forge

micromamba 可以识别 conda 的配置文件 micromamba 的配置优先级为 API >> CLI >> ENV >> RC 多个 RC 配置文件的读取顺序如下，越晚读到的配置优先级越高

1
// on_unix
2
{
3
"/etc/conda/.condarc",
4
"/etc/conda/condarc",
5
"/etc/conda/condarc.d/",
6
"/etc/conda/.mambarc",
7
"/var/lib/conda/.condarc",
8
"/var/lib/conda/condarc",
9
"/var/lib/conda/condarc.d/",
10
"/var/lib/conda/.mambarc"
11
}
12
// on_win
13
{
14
"C:\\ProgramData\\conda\\.condarc",
15
"C:\\ProgramData\\conda\\condarc",
16
"C:\\ProgramData\\conda\\condarc.d",
17
"C:\\ProgramData\\conda\\.mambarc"
18
}
19

20
{ root_prefix }/.condarc
21
{ root_prefix }/condarc
22
{ root_prefix }/condarc.d
23
{ root_prefix }/.mambarc
24
~/.conda/.condarc
25
~/.conda/condarc
26
~/.conda/condarc.d
27
~/.condarc
28
~/.mambarc
29
{ target_prefix }/.condarc
30
{ target_prefix }/condarc
31
{ target_prefix }/condarc.d
32
{ target_prefix }/.mambarc
33
$CONDARC,
34
$MAMBARC;

使用 micromamba#

操作命令示例

Terminal window

1
micromamba create --name <env_name>
2
micromamba activate <env_name>
3
micromamba install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia

Terminal window

1
# 创建新环境
2
micromamba create -n python@3.12 python=3.12
3
# 激活环境
4
micromamba activate python@3.12
5
# 取消激活环境
6
micromamba deactivate
7
# 删除环境
8
micromamba env remove -n python@3.12

安装 pandas 测试

Terminal window

1
micromamba install pandas

Untitled 17

vscode集成配置#

在安装的时候，如果没有修改默认的设置，则mamba的路径应该在~/micromamba

打开VSCode的设置，添加下面的设置：

python.venvFolders这一项，添加mamba的目录名，即micromamba
image 47
python.venvPath这一项，添加mamba虚拟环境的目录，即~/micromamba/envs
image 48
Python:select interceptor选择指定解释器
image 49

📎 参考文章#

N5105 PVE 安装 Debian 运行 docker 实现 jellyfin 硬解

Tue, 16 Apr 2024 00:00:00 GMT

家里的两个小朋友天天放学回家就捧着 iPad 看动画片，屏幕又小又伤眼睛，还是我这个老爹给他们弄个家庭影院投屏吧，顺便下载一些西方的动画大片给他们看下，总比天天看国产的神动漫好。

之前买的 N5105，目前已经通过 PVE 虚拟化方式在上面运行了 ikuai（宽带拨号），openwrt（作为旁路由），和 VM 形式安装的 debian（上面 docker 方式运行了 vaultwarden 和 syncthing），这次再通过 lxc 方式运行 debian 12 顺便把显卡直通给 debian12 实现 jellyfin 的硬解。

操作流程#

核显直通PVE#

确保PVE已经升级到7.3-3及以上版本

补全 i915 guc huc dmc固件

Terminal window

1
mkdir -p /lib/firmware/i915 && cd /lib/firmware/i915
2
curl -LO https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/i915/ehl_guc_70.1.1.bin
3
curl -LO https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/i915/ehl_huc_9.0.0.bin
4
curl -LO https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/i915/icl_dmc_ver1_09.bin

安装 lxc debian 前准备#

更新一下 pve 的软件源

Terminal window

1
apt update
2
apt upgrade -y

https://mirrors.tuna.tsinghua.edu.cn/proxmox/images/system/

从上面的清华源中找到 debian 12 模板，这里我下载的 debian-12-standard_12.2-1_amd64.tar.zst

Terminal window

1
# 进入 pve 默认 template 目录
2
cd /var/lib/vz/template/cache/
3
# 下载模板
4
curl -O https://mirrors.tuna.tsinghua.edu.cn/proxmox/images/system/debian-12-standard_12.2-1_amd64.tar.zst
5
# 列出可用 CT 模板
6
pveam list
7
# 更新 CT 模板
8
pveam update

也可以通过以下命令，从官方下载模板

Terminal window

1
# 查看可选模板
2
pveam available
3
pveam download local <template_you_want_to_use>

图形化界面创建 CT#

选择 Create CT

Untitled 25

Unprivileged container 不做勾选，设置 CT ID，hostname 以及登录密码

Untitled 26

选择刚才下载的模板

Untitled 27

设置硬盘大小

Untitled 28

设置 CPU 核心数（后期可以修改）

Untitled 29

设置内存大小（后期可以修改）

Untitled 30

设置网络，IPv4 分配根据实际情况选择，这里我选择 DHCP 结合 ikuai 自动分配

Untitled 31

DNS用宿主机的就好

Untitled 32

完成后确认

Untitled 33

设置显卡#

重要的事情说三遍，在 pve 宿主机上操作！

Terminal window

1
# 查看显卡信息
2
ls -l /dev/dri
3
# crw-rw---- 1 root video  226,   0 Apr 16 23:06 card0 这个就是显卡
4
drwxr-xr-x 2 root root         80 Apr 16 23:06 by-path
5
crw-rw---- 1 root video  226,   0 Apr 16 23:06 card0
6
crw-rw---- 1 root render 226, 128 Apr 16 23:06 renderD128
7

8
# 106 对应上面创建的 CT id
9
vim /etc/pve/lxc/106.conf
10

11
# 黏贴以下内容

修改 ct 106 的配置, 黏贴以下内容

lxc.cgroup2.devices.allow: c 226<0> rwm lxc.cgroup2.devices.allow: c 226<128> rwm lxc.cgroup2.devices.allow: c 29<0> rwm lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir lxc.apparmor.profile: unconfined lxc.cap.drop:

Terminal window

1
vim /etc/pve/lxc/106.conf

编辑完成后具体信息如下

Terminal window

1
cat /etc/pve/lxc/106.conf
2

3
arch: amd64
4
cores: 2
5
cpulimit: 2
6
cpuunits: 1024
7
hostname: lxc-debian
8
memory: 4096
9
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=3A:13:C3:D0:DE:E1,ip=dhcp,ip6=dhcp,type=veth
10
ostype: debian
11
rootfs: local-lvm:vm-106-disk-0,size=40G
12
swap: 4096
13

14
lxc.cgroup2.devices.allow: c 226:0 rwm
15
lxc.cgroup2.devices.allow: c 226:128 rwm
16
lxc.cgroup2.devices.allow: c 29:0 rwm
17
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
18
lxc.apparmor.profile: unconfined
19
lxc.cap.drop:

修改以下配置

Terminal window

1
vim /etc/modprobe.d/i915.conf

增加内容

options i915 enable_guc=3

编辑完成后具体信息如下

Terminal window

1
cat /etc/modprobe.d/i915.conf
2

3
options i915 enable_guc=3

重启

Terminal window

1
reboot

重启后检查是否生效

Terminal window

1
journalctl -b -o short-monotonic -k | egrep -i "i915|dmr|dmc|guc|huc"

返回信息中有以下 HuC 和 GuC 则说明开启成功

Terminal window

1
[    4.784428] pve kernel: Setting dangerous option enable_guc - tainting kernel
2
[    4.785480] pve kernel: i915 0000:00:02.0: [drm] VT-d active for gfx access
3
[    4.785857] pve kernel: i915 0000:00:02.0: vgaarb: deactivate vga console
4
[    4.785952] pve kernel: i915 0000:00:02.0: [drm] Using Transparent Hugepages
5
[    4.790414] pve kernel: i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
6
[    4.799606] pve kernel: i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/icl_dmc_ver1_09.bin (v1.9)
7
[    4.955965] pve kernel: i915 0000:00:02.0: [drm] GuC firmware i915/ehl_guc_70.1.1.bin version 70.1.1
8
[    4.955997] pve kernel: i915 0000:00:02.0: [drm] HuC firmware i915/ehl_huc_9.0.0.bin version 9.0.0
9
[    4.970965] pve kernel: i915 0000:00:02.0: [drm] HuC authenticated
10
[    4.971309] pve kernel: i915 0000:00:02.0: [drm] GuC submission enabled
11
[    4.971311] pve kernel: i915 0000:00:02.0: [drm] GuC SLPC disabled
12
[    4.973395] pve kernel: [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
13
[    4.976297] pve kernel: snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])

lxc debian 内操作#

更换清华源

debian | 镜像站使用帮助 | 清华大学开源软件镜像站 | Tsinghua Open Source Mirror

增加 sudo 账号

默认 root 用户操作有点危险，我比较喜欢添加个人用户并赋予 sudo 权限

Terminal window

1
# 安装 sudo
2
apt install sudo
3
# 添加个人账号
4
adduser <username>
5
# 将个人账号添加到 sudo 组
6
usermod -aG sudo <username>

LXC 开机自动修改解码器用户组权限

Terminal window展开收起

1
# 创建开机脚本
2
sudo cat > /etc/rc.local << EOF
3
!/bin/sh -e
4
#
5
#rc.local
6
#
7
#This script is executed at the end of each multiuser runlevel.
8
#Make sure that the script will "exit 0" on success or any other
9
#value on error.
10
#
11
#In order to enable or disable this script just change the execution
12
#bits.
13
#
14
#By default this script does nothing.
15
chmod 777 /dev/dri/*
16
sleep 30
17
systemctl restart docker
18
exit 0
19
EOF
20

21

22
# 配置开机自动运行
23
sudo cat > /etc/systemd/system/rc-local.service << EOF
24
[Unit]
25
Description=/etc/rc.local
26
ConditionPathExists=/etc/rc.local
27
[Service]
28
Type=forking
29
ExecStart=/etc/rc.local start
30
TimeoutSec=0
31
StandardOutput=tty
32
RemainAfterExit=yes
33
SysVStartPriority=99
34
[Install]
35
WantedBy=multi-user.target
36
EOF
37

38
# 然后执行开机启动rc.local
39
sudo systemctl enable rc-local.service
40

41
# 并给文件赋权限
42
sudo chmod +x /etc/rc.local
43

44
# 重启
45
sudo reboot
46

47
# 重启开机后检查权限是否自动更改解码器用户组为render
48
ls -l /dev/dri
49

50
total 0
51
drwxrwxrwx 2 root root        80 Apr 16 16:11 by-path
52
crwxrwxrwx 1 root video 226,   0 Apr 16 16:11 card0
53
crwxrwxrwx 1 root input 226, 128 Apr 16 16:11 renderD128

展开收起

切换到新建的用户，安装 docker

Install Docker Engine on Debian | Docker Docs

通过 docker compose 启动 jellyfin 并将

docker-compose.yml

1
services:
2
  jellyfin:
3
    image: nyanmisaka/jellyfin
4
    container_name: jellyfin
5
    environment:
6
      - PUID=1000
7
      - PGID=1000
8
      - TZ=Asia/Shanghai
9
    volumes:
10
      # 修改成你自己的路径
11
      - /path/to/jellyfin/config:/config
12
      - /path/to//jellyfin/media:/media
13
    ports:
14
      - 8096:8096
15
      - 8920:8920
16
    devices:
17
      # 显卡直通
18
      - /dev/dri:/dev/dri
19
    restart: unless-stopped

启动

Terminal window

1
docker compose up -d jellyfin

访问 http://ip_to_lxc<8096>, 设置 jellyfin

Untitled 34

通过 smb 挂载 NAS#

允许 lxc 容器挂载 smb

Untitled 35

在 lxc 容器中执行

Terminal window

1
sudo apt install cifs-utils samba samba-client -y
2

3
# 挂载硬盘
4
# 建立挂载点
5
sudo mkdir -p /mnt/share
6

7
# 挂载
8
# sudo mount -t cifs //<挂载点地址>/myshare <挂载目录> -o <挂载选项>
9
# dir_mode=0777,file_mode=0777 完全读写权限
10
sudo mount -t cifs //<挂载点地址>/myshare /mnt/share -o uid=1000,gid=1000,vers=2.0,dir_mode=0755,file_mode=0755,mfsymlinks,cache=strict,rsize=1048576,wsize=1048576,username=账号,password=密码
11
# 取消挂载使用 sudo umount /mnt/share
12

13
# 设置开机自动挂载
14
cat >> /etc/fstab << EOF
15
//<挂载点地址>/myshare /mnt/share cifs uid=1000,gid=1000,vers=2.0,dir_mode=0755,file_mode=0755,mfsymlinks,cache=strict,rsize=1048576,wsize=1048576,username=账号,password=密码 0 0
16
EOF

debian 中文乱码解决#

发现终端环境中中文文件名显示乱码，需要解决

Terminal window

1
sudo dpkg-reconfigure locales

Untitled 36

选择

1
zh_CN.UTF-8 UTF-8
2
en_US.UTF-8 UTF-8

Untitled 37

默认编码选择 en_US.UTF-8

Jellyfin 媒体文件命名规范#

电影命名规则 #

1
Movies
2
├── Film (1990).mp4
3
├── Film (1994).mp4
4
├── Film (2008)
5
│   └── Film.mkv
6
└── Film (2010)
7
    ├── Film-cd1.avi
8
    └── Film-cd2.avi
9

10
# 当有多种版本电影文件时（方式一）
11
Movies
12
└── Best_Movie_Ever (2019)
13
    ├── Best_Movie_Ever (2019) - 1080P.mp4
14
    ├── Best_Movie_Ever (2019) - 720P.mp4
15
    └── Best_Movie_Ever (2019) - Directors Cut.mp4
16

17
# 当有多种版本电影文件时（方式二）
18
Movies
19
└── Best_Movie_Ever (2019)
20
    ├── Best_Movie_Ever (2019) - [1080P].mp4
21
    ├── Best_Movie_Ever (2019) - [720P].mp4
22
    └── Best_Movie_Ever (2019) - [Directors Cut].mp4

电视剧命名规则 #

1
Shows
2
├── Series Name A (2010)
3
│   ├── Season 00
4
│   │   ├── Some Special.mkv
5
│   │   ├── Series Name A S00E01.mkv
6
│   │   └── Series Name A S00E02.mkv
7
│   ├── Season 01
8
│   │   ├── Series Name A S01E01-E02.mkv
9
│   │   ├── Series Name A S01E03.mkv
10
│   │   └── Series Name A S01E04.mkv
11
│   └── Season 02
12
│       ├── Series Name A S02E01.mkv
13
│       ├── Series Name A S02E02.mkv
14
│       ├── Series Name A S02E03 Part 1.mkv
15
│       └── Series Name A S02E03 Part 2.mkv
16
└── Series Name B (2018)
17
    ├── Season 01
18
    |   ├── Series Name B S01E01.mkv
19
    |   └── Series Name B S01E02.mkv
20
    └── Season 02
21
        ├── Series Name B S02E01-E02.mkv
22
        └── Series Name B S02E03.mkv

字幕等文件规则#

1
/Movies
2
    /Film (1946)
3
        Film.mkv
4
        Film.vtt
5
        Film.aac
6
        Film.mka
7
        Film.mks
8
        Film.en.ac3
9
        Film.de.srt
10
        Film.en.dts
11
        Film.german.ac3
12

13
/Movies
14
    /Film (1986)
15
        Film.mkv
16
        Film.default.srt
17
        Film.default.en.forced.ass
18
        Film.forced.en.dts
19
        Film.en.sdh.srt
20
        Film.English Commentary.en.mp3

📎 参考文章#

科学上网

Sat, 13 Apr 2024 00:00:00 GMT

涉及到的知识点#

nginx 四层 stream tcp 代理；
nginx 七层反向代理；
certbot 自动获取免费 https 证书（必须要有自己的域名，通过 cloudflare api）；
xray 的 vision reality 方案；

部署完成后，网络请求数据流向如下

Mermaid 图表渲染失败，请检查图表语法是否正确

flowchart LR
  xray_client[xray客户端]
  subgraph nginx[nginx]
    stream{SNI分流}
    https
  end

  subgraph xray
    inbound ---> router ---> outbound
  end

  xray_client --> nginx
  stream --- 匹配http域名 ---> https
  stream --- 匹配xray域名 ---> inbound

docker-compose.yml(基于3x-ui)#

1
services:
2
  nginx:
3
    image: nginx:alpine
4
    container_name: nginx
5
    ports:
6
      - 80:8080
7
      - 443:8443
8
    volumes:
9
      - $HOME/docker-compose/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
10
      - $HOME/docker-compose/nginx/conf.d/:/etc/nginx/conf.d/:ro
11
      - $HOME/docker-compose/nginx/stream.d/:/etc/nginx/stream.d/:ro
12
      - $HOME/docker-compose/nginx/snippets/:/etc/nginx/snippets/:ro
13
      - $HOME/docker-compose/nginx/cert/:/etc/cert/:ro
14
      - $HOME/docker-compose/nginx/auth/:/etc/nginx/auth/:ro
15
      - $HOME/docker-compose/certbot/etc/letsencrypt/:/etc/letsencrypt/:ro
16
      - $HOME/data/nginx/logs/:/var/log/nginx/
17
      - $HOME/docker-compose/nginx/ssl:/etc/nginx/ssl/:ro
18
    extra_hosts:
19
      - "host.docker.internal:host-gateway"
20
    depends_on:
21
      - xray
22
    deploy:
23
      resources:
24
        limits:
25
          memory: 64M
26
    restart: unless-stopped
27

28
  xray:
29
    image: ghcr.io/mhsanaei/3x-ui:latest
30
    container_name: xray
31
    # hostname: yourhostname <- optional
32
    volumes:
33
      - $HOME/data/3xui/db/:/etc/x-ui/
34
      - $HOME/data/3xui/cert/:/root/cert/
35
    ports:
36
      - "2053:2053"
37
    environment:
38
      XRAY_VMESS_AEAD_FORCED: "false"
39
      XUI_ENABLE_FAIL2BAN: "true"
40
    tty: true
41
    restart: unless-stopped
42

43
  certbot:
44
    image: certbot/dns-cloudflare
45
    container_name: certbot
46
    environment:
47
      - TZ=Asia/Shanghai
48
    volumes:
49
      - $HOME/docker-compose/certbot/etc/letsencrypt:/etc/letsencrypt
50
      - $HOME/data/certbot/var/lib/letsencrypt:/var/lib/letsencrypt
51
      - $HOME/data/certbot/var/log/letsencrypt:/var/log/letsencrypt
52
      - $HOME/docker-compose/certbot/cloudflare.ini:/etc/cloudflare.ini:ro
53
    profiles:
54
      - manual

展开收起

docker-compose.yml(基于原生xray)#

1
services:
2
  # 自动获取证书
3
  certbot:
4
    image: certbot/dns-cloudflare
5
    container_name: certbot
6
    environment:
7
      - TZ=Asia/Shanghai
8
    volumes:
9
      - $HOME/docker-compose/certbot/etc/letsencrypt:/etc/letsencrypt
10
      - $HOME/data/certbot/var/lib/letsencrypt:/var/lib/letsencrypt
11
    command: >-
12
      certonly --dns-cloudflare
13
      --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini
14
      --dns-cloudflare-propagation-seconds 15
15
      --email <your_email>
16
      --agree-tos --no-eff-email
17
      --force-renewal
18
      -d <your_domain>
19
    restart: unless-stopped
20

21
  nginx:
22
    image: nginxinc/nginx-unprivileged
23
    container_name: nginx
24
    ports:
25
      - 80:8080
26
      - 443:8443
27
    volumes:
28
      - $HOME/docker-compose/nginx/nginx.conf:/etc/nginx/nginx.conf
29
      - $HOME/docker-compose/nginx/conf.d/:/etc/nginx/conf.d/
30
      - $HOME/docker-compose/certbot/etc/letsencrypt/:/etc/letsencrypt/
31
      - $HOME/data/nginx/logs/:/var/log/nginx/
32
    extra_hosts:
33
      - "host.docker.internal:host-gateway"
34
    deploy:
35
      resources:
36
        limits:
37
          memory: 64M
38
    restart: unless-stopped
39

40
  xray:
41
    image: teddysun/xray
42
    container_name: xray
43
    volumes:
44
      - $HOME/docker-compose/xray/:/etc/xray
45
    restart: unless-stopped

geosite.dat 数据查看#

https://geoviewer.aloxaf.com/

Docker 常用操作

Thu, 11 Apr 2024 00:00:00 GMT

镜像导出导入#

有时候没挂梯子，docker pull 镜像长时间拉不下来，只能在某台已经拉取过指定镜像的服务器上先导出，再复制到目标机器上导入。

Terminal window

1
# 在目标机器上导出
2
docker image ls
3
REPOSITORY                                    TAG                                     IMAGE ID       CREATED         SIZE
4
debezium/connect                              2.5.3.Final                             ddcf0938250d   2 weeks ago     1.29GB
5
docker save debezium/connect > debezium-connect.tar
6

7
# 复制到目标机器上后导入
8
docker load < debezium-connect.tar

默认空间不足，更换默认目录#

默认 docker 把数据存在 /var/lib/docker ，现在磁盘空间不足，一看 docker 占用 20 多 G，干脆把 /var/lib/docker 换个目录 “data-root”: “/home/sephy/share/docker”

Terminal window

1
# 停止 docker 服务
2
sudo systemctl stop docker
3
# 移动现有文件
4
mv /var/lib/docker path/to/your/dir
5
# 编辑 /etc/docker/daemon.json, 增加 "data-root": "/home/sephy/share/docker"
6
# 刷新配置
7
sudo systemctl daemon-reload
8
# 启动 docker 服务
9
sudo systemctl start docker

daemon.json 配置#

1
{
2
  "registry-mirrors": ["http://docker-mirror.sephy.io"], // 指向自建的 nexus 代理
3
  "insecure-registries": ["docker-registry.sephy.io"], // 因为 自建的 nexus 代理不支持 https
4
  "data-root": "/home/sephy/share/docker", // 更换 docker 目录, 默认 /var/lib/docker 空间不足
5
  "log-driver": "json-file", // 日志输出用 json 格式
6
  "log-opts": { // 滚动日志配置
7
    "max-size": "50m",
8
    "max-file": "5"
9
  }
10
}

修改完成后，需要重启docker

Terminal window

1
systemctl daemon-reload
2
systemctl restart docker

推送镜像到阿里云ACR#

Terminal window

1
docker login --username=sephymartin@gmail.com crpi-n0jv1narbef6jqec.cn-shanghai.personal.cr.aliyuncs.com
2
docker tag homelab-butler:latest crpi-n0jv1narbef6jqec.cn-shanghai.personal.cr.aliyuncs.com/sephy/homelab-butler:latest
3
docker push crpi-n0jv1narbef6jqec.cn-shanghai.personal.cr.aliyuncs.com/sephy/homelab-butler:latest

阿里云 DTS java 客户端配置

Tue, 02 Apr 2024 00:00:00 GMT

总是记不住阿里云 dts java sdk 的配置, 记录一下

参数	说明	获取方式
brokerUrl	数据订阅通道的网络地址及端口号信息。说明如果您部署SDK客户端所属的ECS实例与数据订阅通道属于经典网络或同一专有网络，建议通过内网地址进行数据订阅，网络延迟最小。	在DTS控制台单击目标订阅实例ID，在基本信息页面的网络区域，您可以获取网络地址及端口号信息。
topic	数据订阅通道的订阅Topic。	在DTS控制台单击目标订阅实例ID，在基本信息页面的基本信息区域，您可以获取到订阅Topic。
sid	消费组ID。	在DTS控制台单击目标订阅实例ID，然后单击数据消费，您可以获取到消费组ID和消费组的账号信息。
userName	消费组的账号。警告如您未使用本文提供的客户端，请按照<消费组的账号>-<消费组ID>的格式设置用户名（例如：dtstest-dtsae******bpv），否则无法正常连接。
password	该账号的密码。
initCheckpoint	消费位点，即SDK客户端消费第一条数据的时间戳，格式为Unix时间戳，例如1620962769。说明消费位点信息可用于：当业务程序中断后，传入已消费位点继续消费数据，防止数据丢失。在订阅客户端启动时，传入所需的消费位点，调整订阅位点，实现按需消费数据。
ConsumerContext.ConsumerSubscribeMode subscribeMode	SDK客户端的使用模式，取值为： ConsumerContext.ConsumerSubscribeMode.ASSIGN：ASSIGN模式，即一个消费组下仅支持一个SDK客户端消费订阅数据。 ConsumerContext.ConsumerSubscribeMode.SUBSCRIBE：SUBSCRIBE模式，即支持在同一个消费组下同时启动多个SDK客户端实现灾备。

Untitled 42|brokerUrl

Untitled 43|topic

Untitled 44|sid

Untitled 45|initCheckpoint

📎 参考文章#

使用SDK示例代码消费订阅数据_数据传输服务(DTS)-阿里云帮助中心 (aliyun.com)

MySQL中的锁

Sun, 31 Mar 2024 00:00:00 GMT

锁的分类#

![[Untitled 46.png]]

### FTWRL

用作全库逻辑备份，但是会造成业务停滞（无法 DML，DDL）。可以在备份时开启 RR 级别事务来避免。如果备份的数据库的工具是 mysqldump，在使用 mysqldump 时加上 –single-transaction 参数的时候，就会在备份数据库之前先开启事务。这种方法只适用于支持「可重复读隔离级别的事务」的存储引擎。

1
-- 上锁
2
-- 整个数据库处于只读状态，阻塞 DML 操作，阻塞 DDL 操作
3
flush tables with read lock;
4
-- 解锁
5
unlock tables;

### backup lock（mysql 8新增）

解决备份时，全局锁开销太大的问题

1
-- 上锁
2
-- 不会阻塞普通读写操作，但是会阻塞大部分DDL操作，包括创建/删除表、加/减字段、增/删索引、optimize/analyze/repair table等
3
lock instance for backup;
4
-- 解锁
5
unlock instance;

### Table Lock

1
-- 表级别的共享锁，也就是读锁（S 锁）
2
-- 会阻塞其他以及自己事务的写操作
3
lock tables <target_table> read;
4

5
-- 表级别的排他锁，也就是写锁（X 锁）
6
-- 会阻塞其他事务的读和写操作
7
lock tables <target_table> write;
8

9
-- 解锁
10
unlock tables;

### Metadata Lock

对于引入MDL，其主要解决了2个问题，一个是事务隔离问题，比如在可重复隔离级别下，会话A在2次查询期间，会话B对表结构做了修改，两次查询结果就会不一致，无法满足可重复读的要求；另外一个是数据复制的问题，比如会话A执行了多条更新语句期间，另外一个会话B做了表结构变更并且先提交，就会导致slave在重做时，先重做alter，再重做update时就会出现复制错误的现象。每执行一条DML、DDL语句时都会申请MDL锁，DML操作需要MDL读锁，DDL操作需要MDL写锁（MDL加锁过程是系统自动控制，无法直接干预，读读共享，读写互斥，写写互斥）

元数据锁不用显式的加锁和释放锁，而是在访问表时被自动加上，以保证读写的正确性。加锁和释放锁规则如下：

MDL读锁之间不互斥，允许多个线程同时对加了 MDL读锁的表进行CRUD(增删改查)操作;
MDL写锁，它和读锁、写锁都是互斥的，目的是用来保证变更表结构操作的安全性。也就是说，当对表DDL操作时，会被默认加 MDL写锁，因此，如果有两个线程要同时给一个表加字段，其中一个要等另一个执行完才能开始执行。
MDL读写锁是在事务commit之后才会被释放；

### Intention Lock

intention lock的主要作用是通过表明表上是否存在行级锁以减少锁表时的性能消耗。如果另一个任务试图在该表级别上应用共享或排它锁，则受到由第一个任务控制的表级别意向锁的阻塞。第二个任务在锁定该表前不必检查各个页或行锁，而只需检查表上的意向锁。

InnoDB 支持多粒度锁（multiple granularity locking），它允许行级锁与表级锁共存，而意向锁就是其中的一种表锁。

意向共享锁（intention shared lock, IS）：事务有意向对表中的某些行加共享锁（S锁）事务要获取某些行的 S 锁，必须先获得表的 IS 锁。

1
begin transaction;
2
-- 加锁 必须配合 transaction
3
-- 先获得IS锁，再加S锁
4
SELECT column FROM table ... LOCK IN SHARE MODE;
5

6
-- 提交事务后自动释放锁

意向排他锁（intention exclusive lock, IX）：事务有意向对表中的某些行加排他锁（X锁）事务要获取某些行的 X 锁，必须先获得表的 IX 锁。

1
begin transaction;
2
-- 加锁 必须配合 transaction
3
-- 先获得IX锁，再加X锁
4
SELECT column FROM table ... FOR UPDATE;
5
-- 提交事务后自动释放锁

意向锁于其他表级锁之间的兼容性

	`X`	`IX`	`S`	`IS`
`X`	Conflict	Conflict	Conflict	Conflict
`IX`	Conflict	Compatible	Conflict	Compatible
`S`	Conflict	Conflict	Compatible	Compatible
`IS`	Conflict	Compatible	Compatible	Compatible

意向锁不会与行级的共享 / 排他锁互斥

### AUTO-INC Lock

An AUTO-INC lock is a special table-level lock taken by transactions inserting into tables with AUTO_INCREMENT columns. In the simplest case, if one transaction is inserting values into the table, any other transactions must wait to do their own inserts into that table, so that rows inserted by the first transaction receive consecutive primary key values.

innodb_autoinc_lock_mode	变量含义
0	传统锁模式
1	连续锁模式
2	交错锁模式(MySQL8默认)

传统锁模式

在传统锁模式下，所有的”INSERT-LIKE”语句为AUTO_INCREMENT列使用表级AUTO_INC锁。一个事务的”INSERT-LIKE”语句在语句执行结束后释放AUTO_INC表级锁，而不是在事务结束后释放。这样做的目的是为了保证这个”INSERT-LIKE”语句的自增列的值是连续的。

连续锁模式

连续锁模式对于“Simple inserts”不会使用表级锁，而是使用一个轻量级锁来生成自增值，因为InnoDB可以提前直到插入多少行数据。自增值生成阶段使用轻量级互斥锁来生成所有的值，而不是一直加锁直到插入完成。但是如果其他事务持有AUTO_INC锁，那么“Simple Inserts”类语句也需要等待其他事务完成才能使用轻量级锁来生成所有的自增值。连续锁模式对于“bulk inserts”类语句使用AUTO_INC表级锁直到语句完成。使用表级AUTO_INC锁的语句：INSERT … SELECT、REPLACE … SELECT、LOAD DATA 。

交错锁模式

所有的“INSERT-LIKE”语句都不使用表级锁，而是使用轻量级互斥锁。

交错锁模式速度快、可扩展性高，但是对于基于语句复制会有问题，只能使用基于ROW复制。

之所以称为交错模式是因为并发插入场景下自增值的分配大概率是交替这来的，时刻1事务1获得自增值，时刻2事务2获得自增值，以此类推。

### Record Lock

Record locks always lock index records, even if a table is defined with no indexes. For such cases, InnoDB creates a hidden clustered index and uses this index for record locking.

1
SELECT column FROM table ... FOR UPDATE;

### GAP Lock

A gap lock is a lock on a gap between index records, or a lock on the gap before the first or after the last index record. A gap might span a single index value, multiple index values, or even be empty.

间隙锁主要用于保护范围查询（Range Query），即使用范围条件（例如WHERE子句中使用BETWEEN或大于/小于等操作符）的查询语句。它们会在查询过程中锁定查询范围内的间隙，防止其他事务在该范围内插入新数据。
间隙锁是在可重复读（REPEATABLE READ）和串行化（SERIALIZABLE）隔离级别下工作的。在这些隔离级别下，MySQL会自动为范围查询设置间隙锁。在其他隔离级别下，MySQL不会自动使用间隙锁。
间隙锁是排他锁，意味着它们会阻塞其他事务的插入操作。如果一个事务已经获取了一个间隙锁，并且另一个事务试图在该间隙中插入新数据，那么该插入操作会被阻塞，直到持有锁的事务释放锁。
间隙锁的使用可能会导致死锁。当多个事务同时获取不同间隙的锁，并且它们互相等待对方释放锁时，就会发生死锁。MySQL通过检测和解决死锁来避免这种情况。
间隙锁的性能开销较大，特别是在高并发环境中。因此，使用间隙锁时需要谨慎考虑，并根据具体情况进行性能测试和优化。

### Next-Key Lock

Next-Key锁，称为临键锁，它是Record Lock + Gap Lock的组合，用来锁定一个范围，并且锁定记录本身锁，它是一种左开右闭的范围，可以用符号表示为：(a,b]。

📎 参考文章#

S.M.A.R.T 原则

Mon, 25 Mar 2024 00:00:00 GMT

以下解释来源于维基百科#

SMART所对应的单词在不同场合并不一致、有各种变体，下表就列出了一些变体：

字母	经典含义	其他含义
S	Specific（明确）	Significant（重要）,Stretching（延伸）, Simple（简易）
M	Measurable（可衡量）	Meaningful（有意义）,Motivational（激励）,Manageable（可管理）
A	Achievable（可达成）	Appropriate（适宜）, Agreed（同意）, Assignable（可分配）, Actionable（可行动）, Action-oriented（行动导向）, Ambitious（雄心）
R	Relevant（相关）	Realistic（实际）, Results/Results-focused/Results-oriented, Resourced（资源）,Rewarding（奖励）
T	Time-bound（时限）	Time-oriented（时间定向）, Time framed（时限）, Timed（时限）, Time-based（基于时间）, Timeboxed（时限）, Timely（及时）, Time-Specific（明确时间）, Timetabled（表定时间）, Time limited（时限）, Trackable（可跟踪）, Tangible（明白）

Specific（明确）#

目标应该清晰明确，具体说明所需的结果或行为。
避免模糊的词语和概念，确保目标能够被准确定义和理解。

Measurable（可衡量）#

目标应该是可以量化和测量的，以便能够评估进展和达成程度。
使用具体的指标、数量或百分比来确定目标的完成情况。

Achievable（可实现）#

目标应该是可实现的，基于现有的资源、能力和时间限制。
考虑可用的资源和支持，以确保目标是合理和可达成的。

Relevant（相关）#

目标应该与个人、团队或组织的整体目标和战略方向相关联。
确保目标对于实现更大的目标有意义，并与其他目标相互支持。

Time-bound（时限）#

目标应该有明确的起始日期和截止日期，以便确定完成目标所需的时间范围。
设定合理的时间框架，以促使行动和确立责任。

SMART原则的应用可以带来以下好处：

提供明确的方向和焦点，帮助人们理解他们需要做什么以及为什么。
促进目标的度量和跟踪，使人们能够评估进展和调整策略。
提高目标的实现概率，因为它们更具体、可衡量和可实现。
增强个人、团队或组织的绩效和成果。
帮助提高时间管理和优先事项的确定。

在互联网公司中，SMART原则可以发挥重要作用，帮助指导目标设定、项目管理和绩效评估等方面。以下是SMART原则在互联网公司中的几个常见应用场景：

目标设定：互联网公司通常设定多个目标，如增加用户数量、提升用户参与度、改善用户体验等。应用SMART原则可以确保这些目标具有明确性、可衡量性和可实现性。例如，将目标从”提高用户参与度”具体化为”每月增加用户评论数量至少20%“，这样可以更清楚地指导工作和评估进展。
项目管理：互联网公司在开发新产品、推出新功能或实施营销活动时，使用SMART原则来设定项目目标和里程碑。具体、可衡量和时限性的目标有助于团队明确任务、规划资源和跟踪进展。团队可以根据目标的实现情况进行调整和优化。
绩效评估：SMART原则为绩效评估提供了基准和衡量标准。通过预先设定具体和可衡量的目标，可以对个人、团队或组织的绩效进行客观评估。这有助于识别成功因素和改进机会，并激励员工追求卓越。
任务分配和沟通：在互联网公司中，任务分配和沟通是关键环节。SMART原则可以确保任务具有明确的描述和期望，避免误解和不必要的延误。同时，通过设定时限和要求正式的回音，可以促进有效的沟通和协作。

certbot 自动获取 https 证书

Thu, 21 Mar 2024 00:00:00 GMT

原本自建的 vaultwardeng 服务是通过 http 访问的，但是 iOS 上的 Bitwardeng 应用如果要访问家里的 vaultwardeng 应用，添加自签的 https 证书又比较麻烦，想到之前在阿里云上买的 10 年域名，为什么不直接弄一个二级域名通过 Let’s encrypt 来弄个真正的 https 证书呢？直接开搞

cloudflare token准备#

certbot 可以通过 cloudflare 的 api 来做域名验证（验证某个域名的归属），因此要先准备 cloudflare 的 api token。从安全角度考虑，该 api token 只需要编辑 DNS 权限。

先添加要解析的域名

Untitled 38

API Tokens → Create Token

Untitled 39

选择 Edit zone DNS 模板

Untitled 40

精确到某个域名范围（推荐）

Untitled 41

一路提交，将拿到的 token 存到 cloudflare.ini 文件中

1
# Cloudflare API token used by Certbot
2
dns_cloudflare_api_token = <your token>

Certbot 获取证书#

docker-compose.yml

1
services:
2
  certbot:
3
    image: certbot/dns-cloudflare
4
    container_name: certbot
5
    environment:
6
      - TZ=Asia/Shanghai
7
    volumes:
8
      # /root/docker-compose/certbot/etc/letsencrypt/cloudflare.ini 中有上一步骤中获取到的 token
9
      - /root/docker-compose/certbot/etc/letsencrypt:/etc/letsencrypt
10
    command: >-
11
      certonly --dns-cloudflare
12
      --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini
13
      --dns-cloudflare-propagation-seconds 15
14
      # 改成你的 cloudflare 账号
15
      --email <your_cloudflare_email>
16
      --agree-tos --no-eff-email
17
      --force-renewal
18
      # 改成你要获取证书的域名
19
      -d <your_domain>
20
    restart: unless-stopped

启动 certbot

Terminal window

1
docker compose --compatibility -f /root/docker-compose/docker-compose.yml up -d certbot

签发成功后可以看到拿到的证书

fullchain2.pem 和 privkey2.pem 可以配置到 nginx 中做 SSL 证书使用

Terminal window

1
tree /root/docker-compose/certbot/etc/letsencrypt
2
/home/xxx/docker-compose/certbot/etc/letsencrypt
3
├── accounts
4
│   └── acme-v02.api.letsencrypt.org
5
│       └── directory
6
│           └── 98c1515d3c6f7980e5c4807982403d9b
7
│               ├── meta.json
8
│               ├── private_key.json
9
│               └── regr.json
10
├── archive
11
│   └── <your_domain>
12
│       ├── cert1.pem
13
│       ├── cert2.pem
14
│       ├── chain1.pem
15
│       ├── chain2.pem
16
│       ├── fullchain1.pem
17
│       ├── fullchain2.pem
18
│       ├── privkey1.pem
19
│       └── privkey2.pem
20
├── cloudflare.ini
21
├── csr
22
│   ├── 0000_csr-certbot.pem
23
│   └── 0001_csr-certbot.pem
24
├── keys
25
│   ├── 0000_key-certbot.pem
26
│   └── 0001_key-certbot.pem
27
├── live
28
│   ├── README
29
│   └── <your_domain>
30
│       ├── cert.pem -> ../../archive/<your_domain>/cert2.pem
31
│       ├── chain.pem -> ../../archive/<your_domain>/chain2.pem
32
│       ├── fullchain.pem -> ../../archive/<your_domain>/fullchain2.pem
33
│       ├── privkey.pem -> ../../archive/<your_domain>/privkey2.pem
34
│       └── README
35
├── renewal
36
│   └── <your_domain>.conf
37
└── renewal-hooks
38
    ├── deploy
39
    ├── post
40
    └── pre

📎 参考文章#

服务器响应 P50-P75-P90-P95-P99 指标

Thu, 07 Mar 2024 00:00:00 GMT

在 skywalking 上看到一个这个面板。这里记录一下 P50、P75、P90、P95、P99 的概念。

Untitled 49

服务响应时间的衡量指标有 Min(最小响应时间)、Max(最大响应时间)、Avg(平均响应时间)等，百分位数值，也是衡量指标

Avg(平均响应时间)

服务器当前请求的总耗时/请求总数量

通过该值，我们大体能知道服务运行情况。这个衡量指标的计算方式会把一些异常的值平均掉，进而会掩盖一些问题，我们只知道所有请求的平均响应时间是100ms，但是具体有多少个请求比100ms要大，又有多少个请求比100ms要小，大多少，是200ms，还是500ms，又或是1000ms，我们无从得知。

百分位数值（PXX）

如果将一组数据从小到大排序，并计算相应的累计百分位，则某一百分位所对应数据的值就称为这一百分位的百分位数。可表示为：一组n个观测值按数值大小排列。如，处于p%位置的值称第p百分位数

P1: 就是响应时间最小的请求，P10就是排名第十的请求，P100就是响应时间最长的请求。

P50: 即中位数值。100个请求按照响应时间从小到大排列，位置为50的值，即为P50值。如果响应时间的P50值为200ms，代表我们有半数的用户响应耗时在200ms之内，有半数的用户响应耗时大于200ms。如果你觉得中位数值不够精确，那么可以使用P95和P99.9

P95: 响应耗时从小到大排列，顺序处于95%位置的值即为P95值。

P99.9: 许多大型的互联网公司会采用 P99.9 值，也就是 99.9% 用户耗时作为指标，意思就是1000个用户里面，999个用户的耗时上限，通过测量与优化该值，就可保证绝大多数用户的使用体验。至于 P99.99 值，优化成本过高，而且服务响应由于网络波动、系统抖动等不能解决之情况，因此大多数时候都不考虑该指标。

由此看来，P99 在 1000~3000ms 之前波动，还有很大优化空间。

📎 参考文章#

MySQL 数据类型字节长度

Mon, 26 Feb 2024 00:00:00 GMT

Numeric Type Storage Requirements#

Data Type	Storage Required
`TINYINT`	1 byte
`SMALLINT`	2 bytes
`MEDIUMINT`	3 bytes
`INT`, `INTEGER`	4 bytes
`BIGINT`	8 bytes
FLOAT(``*p*``)	4 bytes if 0 <= `*p` <= 24, 8 bytes if 25 <= `p*` <= 53
`FLOAT`	4 bytes
`DOUBLE [PRECISION]`, `REAL`	8 bytes
DECIMAL(``*M``,``D``), NUMERIC(``M``,``D*``)	Varies; see following discussion
BIT(``*M*``)	approximately (`*M*`+7)/8 bytes

Values for DECIMAL (and NUMERIC) columns are represented using a binary format that packs nine decimal (base 10) digits into four bytes. Storage for the integer and fractional parts of each value are determined separately. Each multiple of nine digits requires four bytes, and the “leftover” digits require some fraction of four bytes. The storage required for excess digits is given by the following table.

Leftover Digits	Number of Bytes
0	0
1	1
2	1
3	2
4	2
5	3
6	3
7	4
8	4

Date and Time Type Storage Requirements#

For TIME, DATETIME, and TIMESTAMP columns, the storage required for tables created before MySQL 5.6.4 differs from tables created from 5.6.4 on. This is due to a change in 5.6.4 that permits these types to have a fractional part, which requires from 0 to 3 bytes.

Data Type	Storage Required Before MySQL 5.6.4	Storage Required as of MySQL 5.6.4
`YEAR`	1 byte	1 byte
`DATE`	3 bytes	3 bytes
`TIME`	3 bytes	3 bytes + fractional seconds storage
`DATETIME`	8 bytes	5 bytes + fractional seconds storage
`TIMESTAMP`	4 bytes	4 bytes + fractional seconds storage

As of MySQL 5.6.4, storage for YEAR and DATE remains unchanged. However, TIME, DATETIME, and TIMESTAMP are represented differently. DATETIME is packed more efficiently, requiring 5 rather than 8 bytes for the nonfractional part, and all three parts have a fractional part that requires from 0 to 3 bytes, depending on the fractional seconds precision of stored values.

Fractional Seconds Precision	Storage Required
0	0 bytes
1, 2	1 byte
3, 4	2 bytes
5, 6	3 bytes

For example, TIME(0), TIME(2), TIME(4), and TIME(6) use 3, 4, 5, and 6 bytes, respectively. TIME and TIME(0) are equivalent and require the same storage.

String Type Storage Requirements#

In the following table, ***M*** represents the declared column length in characters for nonbinary string types and bytes for binary string types. ***L*** represents the actual length in bytes of a given string value.

Data Type	Storage Required
CHAR(``*M*``)	The compact family of InnoDB row formats optimize storage for variable-length character sets. See COMPACT Row Format Storage Characteristics. Otherwise, `*M` × `w` bytes, <= ``M`` <= 255, where `w*` is the number of bytes required for the maximum-length character in the character set.
BINARY(``*M*``)	`*M` bytes, 0 <= ``M*`` <= 255
VARCHAR(``*M``), VARBINARY(``M*``)	`*L` + 1 bytes if column values require 0 − 255 bytes, `L*` + 2 bytes if values may require more than 255 bytes
`TINYBLOB`, `TINYTEXT`	`*L` + 1 bytes, where `L*` < 28
`BLOB`, `TEXT`	`*L` + 2 bytes, where `L*` < 216
`MEDIUMBLOB`, `MEDIUMTEXT`	`*L` + 3 bytes, where `L*` < 224
`LONGBLOB`, `LONGTEXT`	`*L` + 4 bytes, where `L*` < 232
ENUM('``*value1``','``value2*``',...)	1 or 2 bytes, depending on the number of enumeration values (65,535 values maximum)
SET('``*value1``','``value2*``',...)	1, 2, 3, 4, or 8 bytes, depending on the number of set members (64 members maximum)

Spatial Type Storage Requirements#

MySQL stores geometry values using 4 bytes to indicate the SRID followed by the WKB representation of the value. The LENGTH() function returns the space in bytes required for value storage.

JSON Storage Requirements#

In general, the storage requirement for a JSON column is approximately the same as for a LONGBLOB or LONGTEXT column; that is, the space consumed by a JSON document is roughly the same as it would be for the document’s string representation stored in a column of one of these types. However, there is an overhead imposed by the binary encoding, including metadata and dictionaries needed for lookup, of the individual values stored in the JSON document. For example, a string stored in a JSON document requires 4 to 10 bytes additional storage, depending on the length of the string and the size of the object or array in which it is stored.

In addition, MySQL imposes a limit on the size of any JSON document stored in a JSON column such that it cannot be any larger than the value of max_allowed_packet.

📎 参考文章#

MySQL :: MySQL 8.0 Reference Manual :: 13.7 Data Type Storage Requirements

MySQL 查询优化

Mon, 26 Feb 2024 00:00:00 GMT

explain 返回字段说明#

Column	format=json JSON NAME	说明	附加说明
id	select_id	The `SELECT` identifier
select_type		The `SELECT` type	`SIMPLE` `PRIMARY<br>UNION` `UNION RESULT<br>SUBQUERY<br>DEPENDENT SUBQUERY<br>DEPENDENT UNION<br>DERIVED<br>MATERIALIZED<br>UNCACHEABLE SUBQUERY<br>UNCACHEABLE UNION`
table	table_name	The table for the output row
partitions	partitions	The matching partitions
type	access_type	The join type	`system`->`const`->`eq_ref`->`ref`->`range`->`index`->`all`
possible_keys	possible_keys	The possible indexes to choose
key	key	The index actually chosen
key_len	key_length	The length of the chosen key	当 key_len 为 null 时, 说明没有使用到索引. 通过 key_len 计算规则, 可以判断索引字段的使用情况.
ref	ref	The columns compared to the index
rows	rows	Estimate of rows to be examined
filtered	filtered	Percentage of rows filtered by table condition
extra		Additional information

id#

The SELECT identifier. This is the sequential number of the SELECT within the query. The value can be NULL if the row refers to the union result of other rows. In this case, the table column shows a value like <union``***M***``,``***N***``> to indicate that the row refers to the union of the rows with id values of ***M*** and ***N***.

key_len 计算规则#

可以为 null 的列 key 长度比非 null 列 key 长度大 1;
如果索引列是字符型字段, 则索引列数据类型空间占用和字符集有关; | 字符集 | 1个字符集占用字节数 | | --- | --- | | gbk | 2 | | utf8 | 3 | | utf8mb4 | 4 | | latin1 | 1 |
如果索引列长度是变化的(比如varchar), 则在索引列数据类型本身占用空间的基础上再加 2;

select_type#

he type of SELECT, which can be any of those shown in the following table. A JSON-formatted EXPLAIN exposes the SELECT type as a property of a query_block, unless it is SIMPLE or PRIMARY. The JSON names (where applicable) are also shown in the table.

select_type Value	JSON Name	Meaning
`SIMPLE`	None	Simple `SELECT` (not using `UNION` or subqueries)
`PRIMARY`	None	Outermost `SELECT`
`UNION`	None	Second or later `SELECT` statement in a `UNION`
`DEPENDENT UNION`	`dependent` (`true`)	Second or later `SELECT` statement in a `UNION`, dependent on outer query
`UNION RESULT`	`union_result`	Result of a `UNION`.
`SUBQUERY`	None	First `SELECT` in subquery
`DEPENDENT SUBQUERY`	`dependent` (`true`)	First `SELECT` in subquery, dependent on outer query
`DERIVED`	None	Derived table
`DEPENDENT DERIVED`	`dependent` (`true`)	Derived table dependent on another table
`MATERIALIZED`	`materialized_from_subquery`	Materialized subquery
`UNCACHEABLE SUBQUERY`	`cacheable` (`false`)	A subquery for which the result cannot be cached and must be re-evaluated for each row of the outer query
`UNCACHEABLE UNION`	`cacheable` (`false`)	The second or later select in a `UNION` that belongs to an uncacheable subquery (see `UNCACHEABLE SUBQUERY`)

join types#

system #

The table has only one row (= system table). This is a special case of the const join type.

const #

The table has at most one matching row, which is read at the start of the query. Because there is only one row, values from the column in this row can be regarded as constants by the rest of the optimizer. const tables are very fast because they are read only once.

const is used when you compare all parts of a PRIMARY KEY or UNIQUE index to constant values. In the following queries, ***tbl_name*** can be used as a const table:

1
SELECT * FROM tbl_name WHERE primary_key=1;
2

3
SELECT * FROM tbl_name
4
  WHERE primary_key_part1=1 AND primary_key_part2=2;

eq_ref #

One row is read from this table for each combination of rows from the previous tables. Other than the system and const types, this is the best possible join type. It is used when all parts of an index are used by the join and the index is a PRIMARY KEY or UNIQUE NOT NULL index.

eq_ref can be used for indexed columns that are compared using the = operator. The comparison value can be a constant or an expression that uses columns from tables that are read before this table. In the following examples, MySQL can use an eq_ref join to process ***ref_table***:

1
SELECT * FROM ref_table,other_table
2
  WHERE ref_table.key_column=other_table.column;
3

4
SELECT * FROM ref_table,other_table
5
  WHERE ref_table.key_column_part1=other_table.column
6
  AND ref_table.key_column_part2=1;

ref #

All rows with matching index values are read from this table for each combination of rows from the previous tables. ref is used if the join uses only a leftmost prefix of the key or if the key is not a PRIMARY KEY or UNIQUE index (in other words, if the join cannot select a single row based on the key value). If the key that is used matches only a few rows, this is a good join type.

ref can be used for indexed columns that are compared using the = or <=> operator. In the following examples, MySQL can use a ref join to process ***ref_table***:

1
SELECT * FROM ref_table WHERE key_column=expr;
2

3
SELECT * FROM ref_table,other_table
4
  WHERE ref_table.key_column=other_table.column;
5

6
SELECT * FROM ref_table,other_table
7
  WHERE ref_table.key_column_part1=other_table.column
8
  AND ref_table.key_column_part2=1;

fulltext #

The join is performed using a FULLTEXT index.

ref or null #

This join type is like ref, but with the addition that MySQL does an extra search for rows that contain NULL values. This join type optimization is used most often in resolving subqueries. In the following examples, MySQL can use a ref_or_null join to process ***ref_table***:

1
SELECT * FROM ref_tableWHERE key_column=expr OR key_column IS NULL;

index merge #

This join type indicates that the Index Merge optimization is used. In this case, the key column in the output row contains a list of indexes used, and key_len contains a list of the longest key parts for the indexes used.

unique_subquery #

This type replaces eq_ref for some IN subqueries of the following form:

1
value IN (SELECT primary_key FROM single_table WHERE some_expr)

index subquery #

This join type is similar to unique_subquery. It replaces IN subqueries, but it works for nonunique indexes in subqueries of the following form:

1
value IN (SELECT key_column FROM single_table WHERE some_expr)

range #

Only rows that are in a given range are retrieved, using an index to select the rows. The key column in the output row indicates which index is used. The key_len contains the longest key part that was used. The ref column is NULL for this type.

range can be used when a key column is compared to a constant using any of the =, <>, >, >=, <, <=, IS NULL, <=>, BETWEEN, LIKE, or IN() operators:

1
SELECT * FROM tbl_name
2
  WHERE key_column = 10;
3

4
SELECT * FROM tbl_name
5
  WHERE key_column BETWEEN 10 and 20;
6

7
SELECT * FROM tbl_name
8
  WHERE key_column IN (10,20,30);
9

10
SELECT * FROM tbl_name
11
  WHERE key_part1 = 10 AND key_part2 IN (10,20,30);

index #

The index join type is the same as ALL, except that the index tree is scanned. This occurs two ways:

If the index is a covering index for the queries and can be used to satisfy all data required from the table, only the index tree is scanned. In this case, the Extra column says Using index. An index-only scan usually is faster than ALL because the size of the index usually is smaller than the table data.
A full table scan is performed using reads from the index to look up data rows in index order. Uses index does not appear in the Extra column.

MySQL can use this join type when the query uses only columns that are part of a single index.

ALL #

A full table scan is done for each combination of rows from the previous tables. This is normally not good if the table is the first table not marked const, and usually very bad in all other cases. Normally, you can avoid ALL by adding indexes that enable row retrieval from the table based on constant values or column values from earlier tables.

EXPLAIN Extra Information#

📎 参考文章#

Nginx CORS 配置

Sat, 24 Feb 2024 00:00:00 GMT

浏览器同源策略#

浏览器安全的基石是”同源策略”(same-origin policy). “同源”指的是”三个相同”:

协议相同
域名相同
端口相同

如果非同源，共有三种行为受到限制。

Cookie、LocalStorage、IndexDB无法读取
DOM无法获得
AJAX请求不能发送

CORS原理解析#

CORS是跨源AJAX请求的根本解决方法. JSONP只能发GET请求, 但是CORS允许任何类型的请求.

整个CORS通信过程都是浏览器自动完成的, 不需要用户参与. 对于开发者来说, CORS通信与同源的AJAX通信没有差别, 代码完全一样. 浏览器一旦发现AJAX请求跨源，就会自动添加一些附加的头信息，有时还会多一次附加的请求，但用户不会有感觉。因此，实现CORS通信的关键是**服务器. **只要服务器支持 CORS, 就可以跨源通信.

简单请求和非简单请求#

Simple requests

直接在请求时加上 Origin 的 header

Untitled 50

Preflighted requests

发出请求前, 会发起一次 OPTION 请求

Untitled 51

Nginx CORS 配置#

add_header不是所有返回都会追加，只限特定状态码的返回才有效，如果想所有返回都生效需要加上always选项参数

官方的解释

1
Syntax:  add_header name value [always];
2
Default:  —
3
Context:  http, server, location, if in location

1
# 配置示例
2
location  /path {
3
  proxy_pass target;
4
  add_header 'Access-Control-Allow-Origin' $http_origin always;
5
  add_header 'Access-Control-Request-Method' 'GET,POST,OPTIONS,DELETE,PUT' always;
6
  add_header 'Access-Control-Allow-Credentials' 'true' always;
7
  add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range' always;
8
}

CORS 响应头说明#

header	是否必填	说明
`Access-Control-Allow-Origin`	Y	`Access-Control-Allow-Origin` 它的值要么是请求时`Origin`字段的值，要么是一个``，表示接受任意域名的请求, 不推荐``
`Access-Control-Allow-Credentials`	N	The `Access-Control-Allow-Credentials` header indicates whether or not the response to the request can be exposed when the `credentials` flag is true.
`Access-Control-Expose-Headers`	N	The `Access-Control-Expose-Headers` header adds the specified headers to the allowlist that JavaScript (such as `Response.headers`) in browsers is allowed to access.
`Access-Control-Allow-Headers`	Y	The `Access-Control-Allow-Methods` header specifies the method or methods allowed when accessing the resource. This is used in response to a preflight request.
`Access-Control-Allow-Methods`	Y	The `Access-Control-Allow-Methods` header specifies the method or methods allowed when accessing the resource. This is used in response to a preflight request.
`Access-Control-Max-Age`	N	The `Access-Control-Max-Age` header indicates how long the results of a preflight request can be cached.

CORS 请求头说明#

header	说明
`Origin`	The `Origin` header indicates the origin of the cross-origin access request or preflight request.
`Access-Control-Request-Method`	The `Access-Control-Request-Method` is used when issuing a preflight request to let the server know what HTTP method will be used when the actual request is made.
`Access-Control-Request-Headers`	The `Access-Control-Request-Headers` header is used when issuing a preflight request to let the server know what HTTP headers will be used when the actual request is made (for example, by passing them as the `headers` option to the `Request()` constructor). This browser-side header will be answered by the complementary server-side header of `Access-Control-Allow-Headers`.

📎 参考文章#

NACOS 服务实例信息未正常清理

Tue, 20 Feb 2024 00:00:00 GMT

运维同事反馈, 线上某个 spring boot 应用服务挂了, 让我看看.

Untitled 54

但是, 这个 188 节点机器上的我们之前就已经下线了, 怎么会告警?

排查链路, 我们的 prometheus 监控数据来源是从 nacos 自动获取的, 搜索一下 nacos 的服务注册信息, 确实还有这个节点的注册信息

Untitled 55

尝试手动清理实例信息

Terminal window

1
# 删除
2
curl -X DELETE 'http://nacos服务器地址/nacos/v2/ns/instance?serviceName=服务名&ip=实例IP&port=实例端口&ephemeral=true'

结果发现服务实例还在, openapi 虽然返回 {“code”<0>,“message”:“success”,“data”:“ok”}, 但是实例注册信息还在

用 arthas 看看调用情况吧

Terminal window

1
curl -O https://arthas.aliyun.com/arthas-boot.jar
2
java -jar arthas-boot.jar

查看调用链路

Terminal window

1
trace com.alibaba.nacos.naming.core.InstanceOperatorClientImpl removeInstance

Untitled 56

查看源码 com.alibaba.nacos.naming.core.InstanceOperatorClientImpl#removeInstance

1
@Override
2
public void removeInstance(String namespaceId, String serviceName, Instance instance) {
3
    boolean ephemeral = instance.isEphemeral();
4
    String clientId = IpPortBasedClient.getClientId(instance.toInetAddr(), ephemeral);
5
    // 这里进入 if 分支后直接 return 了
6
    if (!clientManager.contains(clientId)) {
7
        Loggers.SRV_LOG.warn("remove instance from non-exist client: {}", clientId);
8
        return;
9
    }
10
    Service service = getService(namespaceId, serviceName, ephemeral);
11
    clientOperationService.deregisterInstance(service, instance, clientId);
12
}

可以判断命中了 if (!clientManager.contains(clientId)) 分支, 并没有执行下面的真实删除逻辑

查看一下这个 clientManager

ClientManager 负责管理所有接入客户端(包含服务端提供端/客户订阅端)的连接信息维护工作

Terminal window

1
watch com.alibaba.nacos.naming.core.InstanceOperatorClientImpl removeInstance -x 2

Untitled 57

查看源码com.alibaba.nacos.naming.core.v2.client.manager.ClientManagerDelegate

1
public class ClientManagerDelegate implements ClientManager {
2

3
    private final ConnectionBasedClientManager connectionBasedClientManager;
4

5
    // 临时节点的客户端管理器
6
    private final EphemeralIpPortClientManager ephemeralIpPortClientManager;
7

8
    // 持久化节点的客户端管理器
9
    private final PersistentIpPortClientManager persistentIpPortClientManager;
10

11
    // ...
12

13
    @Override
14
    public boolean contains(String clientId) {
15
        return connectionBasedClientManager.contains(clientId) || ephemeralIpPortClientManager.contains(clientId)
16
                || persistentIpPortClientManager.contains(clientId);
17
    }
18

19
    // 根据clientId的格式选择不通的ClientManager
20
    private ClientManager getClientManagerById(String clientId) {
21
        if (isConnectionBasedClient(clientId)) {
22
            return connectionBasedClientManager;
23
        }
24
        // clientId以false结尾的是ephemeralIpPortClientManager
25
        // 否则是 persistentIpPortClientManager
26
        return clientId.endsWith(ClientConstants.PERSISTENT_SUFFIX) ?
27
            persistentIpPortClientManager : ephemeralIpPortClientManager;
28
    }
29

30
    // clientId不包含#服务的客户端
31
    private boolean isConnectionBasedClient(String clientId) {
32
        return !clientId.contains(IpPortBasedClient.ID_DELIMITER);
33
    }
34

35
    // ...
36
}

感觉通过 openapi 的方式注销实例的情况, 可能走不通了, 看一下com.alibaba.nacos.naming.core.v2.client.Client#removeServiceInstance方法还有哪些地方使用吧

Untitled 58

Untitled 59

com.alibaba.nacos.naming.healthcheck.heartbeat.ExpiredInstanceChecker 源码

1
public class ExpiredInstanceChecker implements InstanceBeatChecker {
2

3
    @Override
4
    public void doCheck(Client client, Service service, HealthCheckInstancePublishInfo instance) {
5
        // 从配置信息  nacos.naming.expireInstance 中获取是否开启过期实例探测, 默认为 true
6
         boolean expireInstance = ApplicationUtils.getBean(GlobalConfig.class).isExpireInstance();
7
        if (expireInstance && isExpireInstance(service, instance)) {
8
            deleteIp(client, service, instance);
9
        }
10
    }
11

12
    private boolean isExpireInstance(Service service, HealthCheckInstancePublishInfo instance) {
13
        long deleteTimeout = getTimeout(service, instance);
14
        return System.currentTimeMillis() - instance.getLastHeartBeatTime() > deleteTimeout;
15
    }
16

17
    private long getTimeout(Service service, InstancePublishInfo instance) {
18
        Optional<Object> timeout = getTimeoutFromMetadata(service, instance);
19
        if (!timeout.isPresent()) {
20
            // IP_DELETE_TIMEOUT = preserved.ip.delete.timeout
21
            timeout = Optional.ofNullable(instance.getExtendDatum().get(PreservedMetadataKeys.IP_DELETE_TIMEOUT));
22
        }
23
        // DEFAULT_IP_DELETE_TIMEOUT = 30 秒
24
        return timeout.map(ConvertUtils::toLong).orElse(Constants.DEFAULT_IP_DELETE_TIMEOUT);
25
    }
26

27
    private Optional<Object> getTimeoutFromMetadata(Service service, InstancePublishInfo instance) {
28
        Optional<InstanceMetadata> instanceMetadata = ApplicationUtils.getBean(NamingMetadataManager.class)
29
                .getInstanceMetadata(service, instance.getMetadataId());
30
        return instanceMetadata.map(metadata -> metadata.getExtendData().get(PreservedMetadataKeys.IP_DELETE_TIMEOUT));
31
    }
32

33
    // 这里会自动移除过期的服务实例
34
    private void deleteIp(Client client, Service service, InstancePublishInfo instance) {
35
        Loggers.SRV_LOG.info("[AUTO-DELETE-IP] service: {}, ip: {}", service.toString(), JacksonUtils.toJson(instance));
36
        client.removeServiceInstance(service);
37
        NotifyCenter.publishEvent(new ClientOperationEvent.ClientDeregisterServiceEvent(service, client.getClientId()));
38
        NotifyCenter.publishEvent(new MetadataEvent.InstanceMetadataEvent(service, instance.getMetadataId(), true));
39
        NotifyCenter.publishEvent(new DeregisterInstanceTraceEvent(System.currentTimeMillis(), "",
40
                false, DeregisterInstanceReason.HEARTBEAT_EXPIRE, service.getNamespace(), service.getGroup(),
41
                service.getName(), instance.getIp(), instance.getPort()));
42
    }
43
}

ExpiredInstanceChecker ← InstanceBeatCheckTask ← ClientBeatCheckTaskV2 ← ClientBeatCheckTaskV2

1
 public class InstanceBeatCheckTask implements Interceptable {
2
    static {
3
        CHECKERS.add(new UnhealthyInstanceChecker());
4
        CHECKERS.add(new ExpiredInstanceChecker());
5
        CHECKERS.addAll(NacosServiceLoader.load(InstanceBeatChecker.class));
6
    }
7
 }

1
public class IpPortBasedClient extends AbstractClient {
2

3
    public void init() {
4
        if (ephemeral) {
5
            // client 初始化时, 如果是临时性节点, 会初始化自动探测任务
6
            beatCheckTask = new ClientBeatCheckTaskV2(this);
7
            HealthCheckReactor.scheduleCheck(beatCheckTask);
8
        } else {
9
            healthCheckTaskV2 = new HealthCheckTaskV2(this);
10
            HealthCheckReactor.scheduleCheck(healthCheckTaskV2);
11
        }
12
    }
13
}

问题的解决#

最后, 发现根本原因是, 运维的流水线构建时, 没有考虑到 docker 镜像制作的工作空间问题, 并发了, 导致了两个原本应该使用不同 jar 包的 docker 镜像, 实际上制作出来的是同一个服务镜像, 恰好这个错误的镜像在 188 机器上运行

📎 参考文章#

Spring Cloud Gateway GlobalFilter 整理

Mon, 19 Feb 2024 00:00:00 GMT

访问 http://localhost<8080>/actuator/gateway/globalfilters 可以获取所有 GlobalFilter 以及对应的 Order 顺序, 按照 spring 的特性, 值越小的优先级越高

1
{
2
  "org.springframework.cloud.gateway.filter.RouteToRequestUrlFilter@41167ded": 10000,
3
  "org.springframework.cloud.gateway.filter.NettyRoutingFilter@63262071": 2147483647,
4
  "org.springframework.cloud.gateway.filter.AdaptCachedBodyGlobalFilter@77ab5214": -2147482648,
5
  "com.qnvip.gateway.filter.LoggingFilter@135e49b2": 2147483647,
6
  "org.springframework.cloud.gateway.filter.ForwardRoutingFilter@13e1e816": 2147483647,
7
  "org.springframework.cloud.gateway.filter.ForwardPathFilter@36238b12": 0,
8
  "org.springframework.cloud.gateway.filter.RemoveCachedBodyFilter@1282f784": -2147483648,
9
  "org.springframework.cloud.gateway.filter.NettyWriteResponseFilter@2d4f67e": -1,
10
  "org.springframework.cloud.gateway.filter.GatewayMetricsFilter@41938e1e": 0,
11
  "org.springframework.cloud.gateway.filter.WebsocketRoutingFilter@33187485": 2147483646,
12
  "org.springframework.cloud.gateway.filter.LoadBalancerServiceInstanceCookieFilter@10b87ff6": 10151,
13
  "org.springframework.cloud.gateway.filter.ReactiveLoadBalancerClientFilter@5bb97fe7": 10150
14
}

汇总表#

Filter	Order 值	用途
`AdaptCachedBodyGlobalFilter`	-2147482648	缓存请求体, 以便在后续的请求中重复使用
`RemoveCachedBodyFilter`	-2147483648	移除上下文中 cachedRequestBody 属性
`NettyWriteResponseFilter`	-1	该过滤器在所有其他过滤器执行完成后执行，并将代理响应协会网关的客户端侧
`WebClientWriteResponseFilter`	-1	通过 webclient 并将代理响应协会网关的客户端侧(实验性, 功能和`WebClientWriteResponseFilter`类似)
`GatewayMetricsFilter`	0	metrics 采集
`ForwardPathFilter`	0	是在请求转发（Forward）时，处理请求路径的修改和重写
`RouteToRequestUrlFilter`	10000	根据目标服务的URI来构建新的请求URI
`NoLoadBalancerClientFilter`	10150	没有路由规则匹配时抛出 404
`ReactiveLoadBalancerClientFilter`	10150	提供负载均衡能力, 推荐
`LoadBalancerServiceInstanceCookieFilter`	10151	提供负载均衡能力, 基于Ribbon, 不推荐
`WebsocketRoutingFilter`	2147483646	处理 WebSocket 协议的请求，将 WebSocket 请求路由到目标 WebSocket 服务器
`NettyRoutingFilter`	2147483647	它使用Netty `HttpClient` 向下游发送代理请求。获得的响应将放在exchange的 `ServerWebExchangeUtils.CLIENT_RESPONSE_ATTR` 属性中，以便在后面的filter中使用
`ForwardRoutingFilter`	2147483647	其根据 `forward://` 前缀( Scheme )过滤处理，将请求转发到当前网关实例本地接口
`WebClientHttpRoutingFilter`	2147483647	通过 webclient 向下游发送代理请求(实验性, 功能和`NettyRoutingFilter`类似)

📎 参考文章#

openwrt 上设置 zerotier 并打通局域网

Sun, 18 Feb 2024 00:00:00 GMT

操作步骤#

在 istore 上安装 zerotier#

Untitled 62

加入到自己的 zerotier 网络#

Untitled 63

添加接口并配置防火墙转发策略#

网络 ⇒ 接口 ⇒ 添加新接口

Untitled 64

网络 ⇒ 接口 ⇒ 编辑 ⇒ 防火墙设置

Untitled 65

网络 ⇒ 防火墙 ⇒ 常规设置面板 ⇒ 添加

配置区域转发规则

Untitled 66

效果如下

Untitled 67

网络 ⇒ 防火墙 ⇒ 自定义规则面板

添加如下配置(ztc3qzrxpu)是我的 zerotier 接口

Terminal window

1
iptables -I FORWARD -i ztc3qzrxpu -j ACCEPT
2
iptables -I FORWARD -o ztc3qzrxpu -j ACCEPT
3
iptables -t nat -I POSTROUTING -o ztc3qzrxpu -j MASQUERADE

效果如下

Untitled 68

zerotier 管理端设置#

授权新加入的节点

Untitled 69

添加路由规则

Untitled 70

Drone CI 小试

Sat, 17 Feb 2024 00:00:00 GMT

Drone CI 安装#

参考官方教程, 事先在 gitea 中创建新的 Oauth2 应用

重定向 URI 根据实际情况填写, 例如这里 http://drone.sephy.io/login

Untitled 85

接下来通过 docker 启动 drone

1
version: "3.8"
2
services:
3
  drone:
4
    image: drone/drone:2.22
5
    container_name: drone
6
    environment:
7
      - TZ=Asia/Shanghai
8
      - DRONE_GITEA_CLIENT_ID=${上一步gitea中创建的应用ID}
9
      - DRONE_GITEA_CLIENT_SECRET=${上一步gitea中创建的应用secret}
10
      - DRONE_GITEA_SERVER=http://gitea.sephy.io/ # gitea 地址
11
      - DRONE_GIT_ALWAYS_AUTH=true
12
      - DRONE_RPC_SECRET=${YOUR_DRONE_RPC_SECRET}
13
      - DRONE_SERVER_HOST=drone.sephy.io # drone 域名
14
      - DRONE_SERVER_PROTO=http
15
      - DRONE_DATABASE_DRIVER=mysql # drone 连接到 mysql
16
      - DRONE_DATABASE_DATASOURCE=${MYSQL_USER}:${MYSQL_PORT}@tcp(mysql:3306)/drone?parseTime=true
17
      - DRONE_USER_CREATE=username:sephy,admin:true # 指定 sephy 与 gitea 的账号匹配并设置为管理员
18
    volumes:
19
      - ~/data/data/drone:/data:rw
20
    restart: unless-stopped
21
    depends_on:
22
      - mysql

启动成功后访问 http://drone.sephy.io

Untitled 86

点击 continue 会进入 gitea 的 oauth2 认证流程, 成功后会返回 drone 进入主页

Untitled 87

到此安装步骤完成

流水线配置#

drone ci 的核心是通过项目里的 .drone.yml 来定义流水线步骤, 这里查阅了很多资料, 结合自身实际情况搞出来这么一个适用于 spring boot + docker 构建部署的流水线, 其中流水线脚本涉及到的用户名密码, 可以通过 drone 的 secret 维护起来

Untitled 88

完整的流水线如下:

maven wrapper 构建(因为我用的私有 maven 库不支持 https, maven wrapper 用的 3.6.3 版本);
docker 镜像构建上传(这里又涉及到配置 nexus 支持 docker 私服, 我又折腾了很久);
到指定机器部署

1
# https://github.com/lework/ci-demo-springboot/blob/master/.drone.yml
2
kind: pipeline
3
type: docker
4
name: default
5

6
trigger:
7
  branch:
8
    - feature/xxx
9
  event:
10
    - push
11

12
steps:
13
  - name: maven-build
14
    image: maven:3.9.6-eclipse-temurin-21
15
    volumes: # 将容器内目录挂载到宿主机，仓库需要开启Trusted设置
16
      - name: maven-cache
17
        path: /root/.m2
18
    commands:
19
      - unset MAVEN_CONFIG # https://github.com/carlossg/docker-maven/issues/301
20
      - ./mvnw clean package -pl web-server -am -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -U -B # -pl web-server 只构建 web-server -am 是向下的, 表示同时处理选定模块所依赖的模块
21
      - cp web-server/target/web-server-0.0.1-SNAPSHOT.jar web-server/docker/app.jar
22

23
  # https://plugins.drone.io/plugins/docker
24
  - name: docker-build
25
    image: plugins/docker
26
    pull: if-not-exists
27
    settings:
28
      dockerfile: web-server/docker/Dockerfile
29
      context: web-server/docker
30
      registry: docker-registry.sephy.io
31
      insecure: true
32
      username:
33
        from_secret: docker_registry_username
34
      password:
35
        from_secret: docker_registry_password
36
      repo: docker-registry.sephy.io/java/web-server # 这里必须指定完整的仓库地址, 否则会推送到默认的 docker.io
37
      mirror: http://docker-mirror.sephy.io
38
      #      cahce_from:
39
      #        - docker-registry.sephy.io/java/web-server:latest
40
      build_args:
41
        - JAR_FILE=app.jar
42
      tags:
43
        - latest
44
        - main
45

46
  # https://plugins.drone.io/plugins/ssh
47
  - name: deplpy
48
    image: appleboy/drone-ssh
49
    environment:
50
      #
51
      DOCKER_REGISTRY_USERNAME:
52
        from_secret: docker_registry_username
53
      DOCKER_REGISTRY_PASSWORD:
54
        from_secret: docker_registry_password
55
    settings:
56
      host: 192.168.50.21
57
      username:
58
        from_secret: ssh_username
59
      password:
60
        from_secret: ssh_password
61
      port: 22
62
      command_timeout: 2m
63
      script:
64
        - echo "deploy..."
65
        - docker login -u ${DOCKER_REGISTRY_USERNAME} -p ${DOCKER_REGISTRY_PASSWORD} docker-registry.sephy.io
66
        - docker pull docker-registry.sephy.io/java/web-server:latest
67
        - docker compose --compatibility -f ~/docker-compose/docker-compose.b450.yml up -d web-server
68

69
# https://docs.drone.io/pipeline/docker/syntax/workspace/
70
volumes: # 定义流水线挂载目录，用于共享数据# 从宿主机中挂载的目录
71
  - name: maven-cache
72
    host:
73
      path: /home/sephy/.m2

展开收起

编写完成后, 提交到 gitea, 发现配置的 webhook 不能正常触发, 发现 gitea 需要配置 ALLOWED_HOST_LIST, 顺带修改 gitea 的 docker compose, 增加对应配置, 重启 gitea

1
version: "3.8"
2
services:
3
  gitea:
4
    image: gitea/gitea:1.21.4
5
    container_name: gitea
6
    environment:
7
      - GITEA__webhook__ALLOWED_HOST_LIST=*

然后尝试推送新的代码, OK, 大功告成!

Untitled 89

📎 参考文章#

Nexus Docker 镜像仓库配置

Sat, 17 Feb 2024 00:00:00 GMT

为什么选 nexus3 不选 harbor#

因为穷, 家里的闲置主机资源配置一般, 已经跑了以下应用

PVE(是的, 我在闲置主机上先装了 PVE)
基于 PVE 的 Debian
nexus3(对, 就是那个大名鼎鼎的 java maven 自建私服应用)
gitea(有些小项目我懒得放 github)
mysql8
nginx
syncthing
xxl-job-admin
java 爬虫应用

harbor 的组件太多, 我不想再装 postgres 等数据库了(感觉躲不过, 下一步的 sonarqube 好像要用 postgres)

配置说明#

nexus 配置#

以下 repository 配置都不需要勾选启用 v1 的 api

添加 docker-hub 的 proxy, 这里用的阿里云镜像
Untitled 75
添加谷歌镜像的代理 ghcr.io
Untitled 76
添加微软镜像的代理(playwright-java 的 docker 镜像在这)
Untitled 77
添加自建仓库
Untitled 78
再搞一个聚合的仓库, 用来对自己提供镜像缓存, 6000 端口用来单独配置反向代理
Untitled 79

nginx 反向代理配置#

docker-mirror.sephy.io → nexus<6000> 端口

docker-registry.sephy.io → nexus<6001> 端口

==因为频繁的遇到 docker pull 时都通过 https 方式从 docker-registry.sephy.io 获取镜像, 这里还通过 Fishdrowned 的 ==== https://github.com/Fishdrowned/ssl ==== 项目来生成了自签的证书来配置 nginx https://docker-registry.sephy.io==

汇总如下#

仓库	repository 类型	代理目标	http 端口	域名
docker-hub	proxy	阿里云公共镜像
docker-ghcr	proxy	谷歌镜像
docker-mcr	proxy	微软镜像
docker-self	hosted	自建镜像	6001	docker-registry.sephy.io
docker-public	group	聚合镜像	6000	docker-mirror.sephy.io

docker 配置#

/etc/docker/daemon.json

1
{
2
  "registry-mirrors": ["http://docker-mirror.sephy.io"],
3
  "insecure-registries": ["docker-registry.sephy.io"]
4
}

改完 daemon.json 后需要重启 docker, 并登录到私有仓库

Terminal window

1
sudo systemctl daemon-reload
2
sudo systemctl restart docker
3
docker login -u admin https://docker-registry.sephy.io

验证一下上传自定义镜像到私服#

Terminal window

1
# tag 格式 registry/namespace/image-name:version
2
docker tag maven:3.9.6-eclipse-temurin-21-jammy-custom docker-registry.sephy.io/maven:3.9.6-eclipse-temurin-21-jammy-custom
3
docker push docker-registry.sephy.io/maven:3.9.6-eclipse-temurin-21-jammy-custom
4
The push refers to repository [docker-registry.sephy.io/maven]
5
442fe6ab34d7: Pushed
6
6d5dc7bd1e8b: Pushed
7
b87a4be1beb2: Pushed
8
908af68e2e85: Pushed
9
c7dab9361ce9: Pushed
10
a24a61ddf7b4: Pushed
11
c1793f638462: Pushed
12
4df676480fd4: Pushed
13
d9d7e0852802: Pushed
14
cbb50874016a: Pushed
15
5498e8c22f69: Pushed
16
3.9.6-eclipse-temurin-21-jammy-custom: digest: sha256:f0bf790bda0c8d43ac142de4066dc3e7286659ad789762e5ee494ccc712e3483 size: 2624

效果展示#

聚合的 docker-public 库

Untitled 80

ghcr 缓存

Untitled 81

微软缓存

Untitled 82

阿里云镜像缓存

Untitled 83

自建私服缓存

Untitled 84

📎 参考文章#

MySQL的MVCC

Wed, 14 Feb 2024 00:00:00 GMT

MVCC#

readview#

image 32

RC、RR隔离级别在事务开始时会生成一个 readview, 包含以下要素信息 creator_trx_id 生成该 readview 的事务的事务id trx_ids 在生成 readview 时当前系统中活跃的读写事务的事务id列表(开启事务但未提交) up_limit_id 生成 readview 时活跃trx_ids 中的最小值 low_limit_id 生成 readview 时系统中应该分配给下一个事务的id值
有了这个readview，这样在访问某条记录时，只需要按照下面的步骤判断记录的某个版本是否可见

==record_trx_id = creator_trx_id== 表明当前事务在访问它自己修改过的记录，所以该版本可以被当前事务访问

==record_trx_id ＜ up_limit_id== 表明生成该版本的事务在当前事务生成 readview 前已经提交，所以该版本可以被当前事务访问

==record_trx_id >= low_limit_id== 表明生成该版本的事务在当前事务生成 readview 后才开启，所以该版本不可以被当前事务访问

==record_trx_id between min_trx_id and max_trx_id && record_trx_id not in trx_ids== 说明创建ReadView时生成该版本的事务已经被提交，该版本可以被访问

==record_trx_id between min_trx_id and max_trx_id && record_trx_id in trx_ids== 说明创建ReadView时生成该版本的事务还是活跃的，该版本不可以被访问

mysql 源码中 readview 结构的说明

1
struct read_view_struct{
2
  ulint    type;  /*!< VIEW_NORMAL, VIEW_HIGH_GRANULARITY */
3
  undo_no_t  undo_no;/*!< 0 or if type is
4
        VIEW_HIGH_GRANULARITY
5
        transaction undo_no when this high-granularity
6
        consistent read view was created */
7
  trx_id_t  low_limit_no;
8
        /*!< The view does not need to see the undo
9
        logs for transactions whose transaction number
10
        is strictly smaller (<) than this value: they
11
        can be removed in purge if not needed by other
12
        views */
13
  trx_id_t  low_limit_id;
14
        /*!< The read should not see any transaction
15
        with trx id >= this value. In other words,
16
        this is the "high water mark". */
17
  trx_id_t  up_limit_id;
18
        /*!< The read should see all trx ids which
19
        are strictly smaller (<) than this value.
20
        In other words,
21
        this is the "low water mark". */
22
  ulint    n_trx_ids;
23
        /*!< Number of cells in the trx_ids array */
24
  trx_id_t*  trx_ids;/*!< Additional trx ids which the read should
25
        not see: typically, these are the active
26
        transactions at the time when the read is
27
        serialized, except the reading transaction
28
        itself; the trx ids in this array are in a
29
        descending order. These trx_ids should be
30
        between the "low" and "high" water marks,
31
        that is, up_limit_id and low_limit_id. */
32
  trx_id_t  creator_trx_id;
33
        /*!< trx id of creating transaction, or
34
        0 used in purge */
35
  UT_LIST_NODE_T(read_view_t) view_list;
36
        /*!< List of read views in trx_sys */
37
};

MySQL的RR事务隔离并没有完全防止幻读#

可以使用以下例子重现，该例子来自stackoverflow

1
create table ab(a int primary key, b int);
2

3
Tx1:
4
begin;
5
select * from ab; // empty set
6

7
Tx2:
8
begin;
9
insert into ab values(1,1);
10
commit;
11
Tx1:
12
select * from ab; // empty set, expected phantom read missing.
13
update ab set b = 2 where a = 1; // 1 row affected.
14
select * from ab; // 1 row. phantom read here!!!!
15
commit;

📎 参考文章#

skywalking oap 配置

Tue, 02 Jan 2024 00:00:00 GMT

当前在生产环境部署了 skywalking, 主要用于分布式服务的链路追踪, 因为生产环境 es 服务器配置比较小, 硬盘存储空间有限, 所以需要修改 skywalking 的数据保存时长, 避免数据过大.

进入 docker 容器中拿到 application.yml 的完整内容如下:

1
# Licensed to the Apache Software Foundation (ASF) under one or more
2
# contributor license agreements.  See the NOTICE file distributed with
3
# this work for additional information regarding copyright ownership.
4
# The ASF licenses this file to You under the Apache License, Version 2.0
5
# (the "License"); you may not use this file except in compliance with
6
# the License.  You may obtain a copy of the License at
7
#
8
#     http://www.apache.org/licenses/LICENSE-2.0
9
#
10
# Unless required by applicable law or agreed to in writing, software
11
# distributed under the License is distributed on an "AS IS" BASIS,
12
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
# See the License for the specific language governing permissions and
14
# limitations under the License.
15

16
cluster:
17
  selector: ${SW_CLUSTER:standalone}
18
  standalone:
19
  # Please check your ZooKeeper is 3.5+, However, it is also compatible with ZooKeeper 3.4.x. Replace the ZooKeeper 3.5+
20
  # library the oap-libs folder with your ZooKeeper 3.4.x library.
21
  zookeeper:
22
    namespace: ${SW_NAMESPACE:""}
23
    hostPort: ${SW_CLUSTER_ZK_HOST_PORT:localhost:2181}
24
    # Retry Policy
25
    baseSleepTimeMs: ${SW_CLUSTER_ZK_SLEEP_TIME:1000} # initial amount of time to wait between retries
26
    maxRetries: ${SW_CLUSTER_ZK_MAX_RETRIES:3} # max number of times to retry
27
    # Enable ACL
28
    enableACL: ${SW_ZK_ENABLE_ACL:false} # disable ACL in default
29
    schema: ${SW_ZK_SCHEMA:digest} # only support digest schema
30
    expression: ${SW_ZK_EXPRESSION:skywalking:skywalking}
31
    internalComHost: ${SW_CLUSTER_INTERNAL_COM_HOST:""}
32
    internalComPort: ${SW_CLUSTER_INTERNAL_COM_PORT:-1}
33
  kubernetes:
34
    namespace: ${SW_CLUSTER_K8S_NAMESPACE:default}
35
    labelSelector: ${SW_CLUSTER_K8S_LABEL:app=collector,release=skywalking}
36
    uidEnvName: ${SW_CLUSTER_K8S_UID:SKYWALKING_COLLECTOR_UID}
37
  consul:
38
    serviceName: ${SW_SERVICE_NAME:"SkyWalking_OAP_Cluster"}
39
    # Consul cluster nodes, example: 10.0.0.1:8500,10.0.0.2:8500,10.0.0.3:8500
40
    hostPort: ${SW_CLUSTER_CONSUL_HOST_PORT:localhost:8500}
41
    aclToken: ${SW_CLUSTER_CONSUL_ACLTOKEN:""}
42
    internalComHost: ${SW_CLUSTER_INTERNAL_COM_HOST:""}
43
    internalComPort: ${SW_CLUSTER_INTERNAL_COM_PORT:-1}
44
  etcd:
45
    # etcd cluster nodes, example: 10.0.0.1:2379,10.0.0.2:2379,10.0.0.3:2379
46
    endpoints: ${SW_CLUSTER_ETCD_ENDPOINTS:localhost:2379}
47
    namespace: ${SW_CLUSTER_ETCD_NAMESPACE:/skywalking}
48
    serviceName: ${SW_CLUSTER_ETCD_SERVICE_NAME:"SkyWalking_OAP_Cluster"}
49
    authentication: ${SW_CLUSTER_ETCD_AUTHENTICATION:false}
50
    user: ${SW_CLUSTER_ETCD_USER:}
51
    password: ${SW_CLUSTER_ETCD_PASSWORD:}
52
    internalComHost: ${SW_CLUSTER_INTERNAL_COM_HOST:""}
53
    internalComPort: ${SW_CLUSTER_INTERNAL_COM_PORT:-1}
54
  nacos:
55
    serviceName: ${SW_SERVICE_NAME:"SkyWalking_OAP_Cluster"}
56
    hostPort: ${SW_CLUSTER_NACOS_HOST_PORT:localhost:8848}
57
    # Nacos Configuration namespace
58
    namespace: ${SW_CLUSTER_NACOS_NAMESPACE:"public"}
59
    # Nacos auth username
60
    username: ${SW_CLUSTER_NACOS_USERNAME:""}
61
    password: ${SW_CLUSTER_NACOS_PASSWORD:""}
62
    # Nacos auth accessKey
63
    accessKey: ${SW_CLUSTER_NACOS_ACCESSKEY:""}
64
    secretKey: ${SW_CLUSTER_NACOS_SECRETKEY:""}
65
    internalComHost: ${SW_CLUSTER_INTERNAL_COM_HOST:""}
66
    internalComPort: ${SW_CLUSTER_INTERNAL_COM_PORT:-1}
67
core:
68
  selector: ${SW_CORE:default}
69
  default:
70
    # Mixed: Receive agent data, Level 1 aggregate, Level 2 aggregate
71
    # Receiver: Receive agent data, Level 1 aggregate
72
    # Aggregator: Level 2 aggregate
73
    role: ${SW_CORE_ROLE:Mixed} # Mixed/Receiver/Aggregator
74
    restHost: ${SW_CORE_REST_HOST:0.0.0.0}
75
    restPort: ${SW_CORE_REST_PORT:12800}
76
    restContextPath: ${SW_CORE_REST_CONTEXT_PATH:/}
77
    restMaxThreads: ${SW_CORE_REST_MAX_THREADS:200}
78
    restIdleTimeOut: ${SW_CORE_REST_IDLE_TIMEOUT:30000}
79
    restAcceptQueueSize: ${SW_CORE_REST_QUEUE_SIZE:0}
80
    httpMaxRequestHeaderSize: ${SW_CORE_HTTP_MAX_REQUEST_HEADER_SIZE:8192}
81
    gRPCHost: ${SW_CORE_GRPC_HOST:0.0.0.0}
82
    gRPCPort: ${SW_CORE_GRPC_PORT:11800}
83
    maxConcurrentCallsPerConnection: ${SW_CORE_GRPC_MAX_CONCURRENT_CALL:0}
84
    maxMessageSize: ${SW_CORE_GRPC_MAX_MESSAGE_SIZE:0}
85
    gRPCThreadPoolQueueSize: ${SW_CORE_GRPC_POOL_QUEUE_SIZE:-1}
86
    gRPCThreadPoolSize: ${SW_CORE_GRPC_THREAD_POOL_SIZE:-1}
87
    gRPCSslEnabled: ${SW_CORE_GRPC_SSL_ENABLED:false}
88
    gRPCSslKeyPath: ${SW_CORE_GRPC_SSL_KEY_PATH:""}
89
    gRPCSslCertChainPath: ${SW_CORE_GRPC_SSL_CERT_CHAIN_PATH:""}
90
    gRPCSslTrustedCAPath: ${SW_CORE_GRPC_SSL_TRUSTED_CA_PATH:""}
91
    downsampling:
92
      - Hour
93
      - Day
94
    # Set a timeout on metrics data. After the timeout has expired, the metrics data will automatically be deleted.
95
    enableDataKeeperExecutor: ${SW_CORE_ENABLE_DATA_KEEPER_EXECUTOR:true} # Turn it off then automatically metrics data delete will be close.
96
    dataKeeperExecutePeriod: ${SW_CORE_DATA_KEEPER_EXECUTE_PERIOD:5} # How often the data keeper executor runs periodically, unit is minute
97
    recordDataTTL: ${SW_CORE_RECORD_DATA_TTL:3} # Unit is day
98
    metricsDataTTL: ${SW_CORE_METRICS_DATA_TTL:7} # Unit is day
99
    # The period of L1 aggregation flush to L2 aggregation. Unit is ms.
100
    l1FlushPeriod: ${SW_CORE_L1_AGGREGATION_FLUSH_PERIOD:500}
101
    # The threshold of session time. Unit is ms. Default value is 70s.
102
    storageSessionTimeout: ${SW_CORE_STORAGE_SESSION_TIMEOUT:70000}
103
    # The period of doing data persistence. Unit is second.Default value is 25s
104
    persistentPeriod: ${SW_CORE_PERSISTENT_PERIOD:25}
105
    topNReportPeriod: ${SW_CORE_TOPN_REPORT_PERIOD:10} # top_n record worker report cycle, unit is minute
106
    # Extra model column are the column defined by in the codes, These columns of model are not required logically in aggregation or further query,
107
    # and it will cause more load for memory, network of OAP and storage.
108
    # But, being activated, user could see the name in the storage entities, which make users easier to use 3rd party tool, such as Kibana->ES, to query the data by themselves.
109
    activeExtraModelColumns: ${SW_CORE_ACTIVE_EXTRA_MODEL_COLUMNS:false}
110
    # The max length of service + instance names should be less than 200
111
    serviceNameMaxLength: ${SW_SERVICE_NAME_MAX_LENGTH:70}
112
    # The period(in seconds) of refreshing the service cache. Default value is 10s.
113
    serviceCacheRefreshInterval: ${SW_SERVICE_CACHE_REFRESH_INTERVAL:10}
114
    instanceNameMaxLength: ${SW_INSTANCE_NAME_MAX_LENGTH:70}
115
    # The max length of service + endpoint names should be less than 240
116
    endpointNameMaxLength: ${SW_ENDPOINT_NAME_MAX_LENGTH:150}
117
    # Define the set of span tag keys, which should be searchable through the GraphQL.
118
    # The max length of key=value should be less than 256 or will be dropped.
119
    searchableTracesTags: ${SW_SEARCHABLE_TAG_KEYS:http.method,http.status_code,rpc.status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker}
120
    # Define the set of log tag keys, which should be searchable through the GraphQL.
121
    # The max length of key=value should be less than 256 or will be dropped.
122
    searchableLogsTags: ${SW_SEARCHABLE_LOGS_TAG_KEYS:level,http.status_code}
123
    # Define the set of alarm tag keys, which should be searchable through the GraphQL.
124
    # The max length of key=value should be less than 256 or will be dropped.
125
    searchableAlarmTags: ${SW_SEARCHABLE_ALARM_TAG_KEYS:level}
126
    # The max size of tags keys for autocomplete select.
127
    autocompleteTagKeysQueryMaxSize: ${SW_AUTOCOMPLETE_TAG_KEYS_QUERY_MAX_SIZE:100}
128
    # The max size of tags values for autocomplete select.
129
    autocompleteTagValuesQueryMaxSize: ${SW_AUTOCOMPLETE_TAG_VALUES_QUERY_MAX_SIZE:100}
130
    # The number of threads used to prepare metrics data to the storage.
131
    prepareThreads: ${SW_CORE_PREPARE_THREADS:2}
132
    # Turn it on then automatically grouping endpoint by the given OpenAPI definitions.
133
    enableEndpointNameGroupingByOpenapi: ${SW_CORE_ENABLE_ENDPOINT_NAME_GROUPING_BY_OPENAPI:true}
134
    # The period of HTTP URI pattern recognition. Unit is second.
135
    syncPeriodHttpUriRecognitionPattern: ${SW_CORE_SYNC_PERIOD_HTTP_URI_RECOGNITION_PATTERN:10}
136
    # The training period of HTTP URI pattern recognition. Unit is second.
137
    trainingPeriodHttpUriRecognitionPattern: ${SW_CORE_TRAINING_PERIOD_HTTP_URI_RECOGNITION_PATTERN:60}
138
    # The max number of HTTP URIs per service for further URI pattern recognition.
139
    maxHttpUrisNumberPerService: ${SW_CORE_MAX_HTTP_URIS_NUMBER_PER_SVR:3000}
140
storage:
141
  selector: ${SW_STORAGE:h2}
142
  elasticsearch:
143
    namespace: ${SW_NAMESPACE:""}
144
    clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:localhost:9200}
145
    protocol: ${SW_STORAGE_ES_HTTP_PROTOCOL:"http"}
146
    connectTimeout: ${SW_STORAGE_ES_CONNECT_TIMEOUT:3000}
147
    socketTimeout: ${SW_STORAGE_ES_SOCKET_TIMEOUT:30000}
148
    responseTimeout: ${SW_STORAGE_ES_RESPONSE_TIMEOUT:15000}
149
    numHttpClientThread: ${SW_STORAGE_ES_NUM_HTTP_CLIENT_THREAD:0}
150
    user: ${SW_ES_USER:""}
151
    password: ${SW_ES_PASSWORD:""}
152
    trustStorePath: ${SW_STORAGE_ES_SSL_JKS_PATH:""}
153
    trustStorePass: ${SW_STORAGE_ES_SSL_JKS_PASS:""}
154
    secretsManagementFile: ${SW_ES_SECRETS_MANAGEMENT_FILE:""} # Secrets management file in the properties format includes the username, password, which are managed by 3rd party tool.
155
    dayStep: ${SW_STORAGE_DAY_STEP:1} # Represent the number of days in the one minute/hour/day index.
156
    indexShardsNumber: ${SW_STORAGE_ES_INDEX_SHARDS_NUMBER:1} # Shard number of new indexes
157
    indexReplicasNumber: ${SW_STORAGE_ES_INDEX_REPLICAS_NUMBER:1} # Replicas number of new indexes
158
    # Specify the settings for each index individually.
159
    # If configured, this setting has the highest priority and overrides the generic settings.
160
    specificIndexSettings: ${SW_STORAGE_ES_SPECIFIC_INDEX_SETTINGS:""}
161
    # Super data set has been defined in the codes, such as trace segments.The following 3 config would be improve es performance when storage super size data in es.
162
    superDatasetDayStep: ${SW_STORAGE_ES_SUPER_DATASET_DAY_STEP:-1} # Represent the number of days in the super size dataset record index, the default value is the same as dayStep when the value is less than 0
163
    superDatasetIndexShardsFactor: ${SW_STORAGE_ES_SUPER_DATASET_INDEX_SHARDS_FACTOR:5} #  This factor provides more shards for the super data set, shards number = indexShardsNumber * superDatasetIndexShardsFactor. Also, this factor effects Zipkin traces.
164
    superDatasetIndexReplicasNumber: ${SW_STORAGE_ES_SUPER_DATASET_INDEX_REPLICAS_NUMBER:0} # Represent the replicas number in the super size dataset record index, the default value is 0.
165
    indexTemplateOrder: ${SW_STORAGE_ES_INDEX_TEMPLATE_ORDER:0} # the order of index template
166
    bulkActions: ${SW_STORAGE_ES_BULK_ACTIONS:5000} # Execute the async bulk record data every ${SW_STORAGE_ES_BULK_ACTIONS} requests
167
    batchOfBytes: ${SW_STORAGE_ES_BATCH_OF_BYTES:10485760} # A threshold to control the max body size of ElasticSearch Bulk flush.
168
    # flush the bulk every 5 seconds whatever the number of requests
169
    flushInterval: ${SW_STORAGE_ES_FLUSH_INTERVAL:5}
170
    concurrentRequests: ${SW_STORAGE_ES_CONCURRENT_REQUESTS:2} # the number of concurrent requests
171
    resultWindowMaxSize: ${SW_STORAGE_ES_QUERY_MAX_WINDOW_SIZE:10000}
172
    metadataQueryMaxSize: ${SW_STORAGE_ES_QUERY_MAX_SIZE:10000}
173
    scrollingBatchSize: ${SW_STORAGE_ES_SCROLLING_BATCH_SIZE:5000}
174
    segmentQueryMaxSize: ${SW_STORAGE_ES_QUERY_SEGMENT_SIZE:200}
175
    profileTaskQueryMaxSize: ${SW_STORAGE_ES_QUERY_PROFILE_TASK_SIZE:200}
176
    profileDataQueryBatchSize: ${SW_STORAGE_ES_QUERY_PROFILE_DATA_BATCH_SIZE:100}
177
    oapAnalyzer: ${SW_STORAGE_ES_OAP_ANALYZER:"{\"analyzer\":{\"oap_analyzer\":{\"type\":\"stop\"}}}"} # the oap analyzer.
178
    oapLogAnalyzer: ${SW_STORAGE_ES_OAP_LOG_ANALYZER:"{\"analyzer\":{\"oap_log_analyzer\":{\"type\":\"standard\"}}}"} # the oap log analyzer. It could be customized by the ES analyzer configuration to support more language log formats, such as Chinese log, Japanese log and etc.
179
    advanced: ${SW_STORAGE_ES_ADVANCED:""}
180
    # Enable shard metrics and records indices into multi-physical indices, one index template per metric/meter aggregation function or record.
181
    logicSharding: ${SW_STORAGE_ES_LOGIC_SHARDING:false}
182
    # Custom routing can reduce the impact of searches. Instead of having to fan out a search request to all the shards in an index, the request can be sent to just the shard that matches the specific routing value (or values).
183
    enableCustomRouting: ${SW_STORAGE_ES_ENABLE_CUSTOM_ROUTING:false}
184
  h2:
185
    properties:
186
      jdbcUrl: ${SW_STORAGE_H2_URL:jdbc:h2:mem:skywalking-oap-db;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=FALSE}
187
      dataSource.user: ${SW_STORAGE_H2_USER:sa}
188
    metadataQueryMaxSize: ${SW_STORAGE_H2_QUERY_MAX_SIZE:5000}
189
    maxSizeOfBatchSql: ${SW_STORAGE_MAX_SIZE_OF_BATCH_SQL:100}
190
    asyncBatchPersistentPoolSize: ${SW_STORAGE_ASYNC_BATCH_PERSISTENT_POOL_SIZE:1}
191
  mysql:
192
    properties:
193
      jdbcUrl: ${SW_JDBC_URL:"jdbc:mysql://localhost:3306/swtest?rewriteBatchedStatements=true&allowMultiQueries=true"}
194
      dataSource.user: ${SW_DATA_SOURCE_USER:root}
195
      dataSource.password: ${SW_DATA_SOURCE_PASSWORD:root@1234}
196
      dataSource.cachePrepStmts: ${SW_DATA_SOURCE_CACHE_PREP_STMTS:true}
197
      dataSource.prepStmtCacheSize: ${SW_DATA_SOURCE_PREP_STMT_CACHE_SQL_SIZE:250}
198
      dataSource.prepStmtCacheSqlLimit: ${SW_DATA_SOURCE_PREP_STMT_CACHE_SQL_LIMIT:2048}
199
      dataSource.useServerPrepStmts: ${SW_DATA_SOURCE_USE_SERVER_PREP_STMTS:true}
200
    metadataQueryMaxSize: ${SW_STORAGE_MYSQL_QUERY_MAX_SIZE:5000}
201
    maxSizeOfBatchSql: ${SW_STORAGE_MAX_SIZE_OF_BATCH_SQL:2000}
202
    asyncBatchPersistentPoolSize: ${SW_STORAGE_ASYNC_BATCH_PERSISTENT_POOL_SIZE:4}
203
  postgresql:
204
    properties:
205
      jdbcUrl: ${SW_JDBC_URL:"jdbc:postgresql://localhost:5432/skywalking"}
206
      dataSource.user: ${SW_DATA_SOURCE_USER:postgres}
207
      dataSource.password: ${SW_DATA_SOURCE_PASSWORD:123456}
208
      dataSource.cachePrepStmts: ${SW_DATA_SOURCE_CACHE_PREP_STMTS:true}
209
      dataSource.prepStmtCacheSize: ${SW_DATA_SOURCE_PREP_STMT_CACHE_SQL_SIZE:250}
210
      dataSource.prepStmtCacheSqlLimit: ${SW_DATA_SOURCE_PREP_STMT_CACHE_SQL_LIMIT:2048}
211
      dataSource.useServerPrepStmts: ${SW_DATA_SOURCE_USE_SERVER_PREP_STMTS:true}
212
    metadataQueryMaxSize: ${SW_STORAGE_MYSQL_QUERY_MAX_SIZE:5000}
213
    maxSizeOfBatchSql: ${SW_STORAGE_MAX_SIZE_OF_BATCH_SQL:2000}
214
    asyncBatchPersistentPoolSize: ${SW_STORAGE_ASYNC_BATCH_PERSISTENT_POOL_SIZE:4}
215
  banyandb:
216
    targets: ${SW_STORAGE_BANYANDB_TARGETS:127.0.0.1:17912}
217
    maxBulkSize: ${SW_STORAGE_BANYANDB_MAX_BULK_SIZE:5000}
218
    flushInterval: ${SW_STORAGE_BANYANDB_FLUSH_INTERVAL:15}
219
    metricsShardsNumber: ${SW_STORAGE_BANYANDB_METRICS_SHARDS_NUMBER:1}
220
    recordShardsNumber: ${SW_STORAGE_BANYANDB_RECORD_SHARDS_NUMBER:1}
221
    superDatasetShardsFactor: ${SW_STORAGE_BANYANDB_SUPERDATASET_SHARDS_FACTOR:2}
222
    concurrentWriteThreads: ${SW_STORAGE_BANYANDB_CONCURRENT_WRITE_THREADS:15}
223
    profileTaskQueryMaxSize: ${SW_STORAGE_BANYANDB_PROFILE_TASK_QUERY_MAX_SIZE:200} # the max number of fetch task in a request
224
    blockIntervalHours: ${SW_STORAGE_BANYANDB_BLOCK_INTERVAL_HOURS:24} # Unit is hour
225
    segmentIntervalDays: ${SW_STORAGE_BANYANDB_SEGMENT_INTERVAL_DAYS:1} # Unit is day
226
    superDatasetBlockIntervalHours: ${SW_STORAGE_BANYANDB_SUPER_DATASET_BLOCK_INTERVAL_HOURS:4} # Unit is hour
227
    superDatasetSegmentIntervalDays: ${SW_STORAGE_BANYANDB_SUPER_DATASET_SEGMENT_INTERVAL_DAYS:1} # Unit is day
228
    specificGroupSettings: ${SW_STORAGE_BANYANDB_SPECIFIC_GROUP_SETTINGS:""} # For example, {"group1": {"blockIntervalHours": 4, "segmentIntervalDays": 1}}
229

230
agent-analyzer:
231
  selector: ${SW_AGENT_ANALYZER:default}
232
  default:
233
    # The default sampling rate and the default trace latency time configured by the 'traceSamplingPolicySettingsFile' file.
234
    traceSamplingPolicySettingsFile: ${SW_TRACE_SAMPLING_POLICY_SETTINGS_FILE:trace-sampling-policy-settings.yml}
235
    slowDBAccessThreshold: ${SW_SLOW_DB_THRESHOLD:default:200,mongodb:100} # The slow database access thresholds. Unit ms.
236
    forceSampleErrorSegment: ${SW_FORCE_SAMPLE_ERROR_SEGMENT:true} # When sampling mechanism active, this config can open(true) force save some error segment. true is default.
237
    segmentStatusAnalysisStrategy: ${SW_SEGMENT_STATUS_ANALYSIS_STRATEGY:FROM_SPAN_STATUS} # Determine the final segment status from the status of spans. Available values are `FROM_SPAN_STATUS` , `FROM_ENTRY_SPAN` and `FROM_FIRST_SPAN`. `FROM_SPAN_STATUS` represents the segment status would be error if any span is in error status. `FROM_ENTRY_SPAN` means the segment status would be determined by the status of entry spans only. `FROM_FIRST_SPAN` means the segment status would be determined by the status of the first span only.
238
    # Nginx and Envoy agents can't get the real remote address.
239
    # Exit spans with the component in the list would not generate the client-side instance relation metrics.
240
    noUpstreamRealAddressAgents: ${SW_NO_UPSTREAM_REAL_ADDRESS:6000,9000}
241
    meterAnalyzerActiveFiles: ${SW_METER_ANALYZER_ACTIVE_FILES:datasource,threadpool,satellite,go-runtime,python-runtime,continuous-profiling} # Which files could be meter analyzed, files split by ","
242
    slowCacheReadThreshold: ${SW_SLOW_CACHE_SLOW_READ_THRESHOLD:default:20,redis:10} # The slow cache read operation thresholds. Unit ms.
243
    slowCacheWriteThreshold: ${SW_SLOW_CACHE_SLOW_WRITE_THRESHOLD:default:20,redis:10} # The slow cache write operation thresholds. Unit ms.
244

245
log-analyzer:
246
  selector: ${SW_LOG_ANALYZER:default}
247
  default:
248
    lalFiles: ${SW_LOG_LAL_FILES:envoy-als,mesh-dp,mysql-slowsql,pgsql-slowsql,redis-slowsql,k8s-service,nginx,default}
249
    malFiles: ${SW_LOG_MAL_FILES:"nginx"}
250

251
event-analyzer:
252
  selector: ${SW_EVENT_ANALYZER:default}
253
  default:
254

255
receiver-sharing-server:
256
  selector: ${SW_RECEIVER_SHARING_SERVER:default}
257
  default:
258
    # For HTTP server
259
    restHost: ${SW_RECEIVER_SHARING_REST_HOST:0.0.0.0}
260
    restPort: ${SW_RECEIVER_SHARING_REST_PORT:0}
261
    restContextPath: ${SW_RECEIVER_SHARING_REST_CONTEXT_PATH:/}
262
    restMaxThreads: ${SW_RECEIVER_SHARING_REST_MAX_THREADS:200}
263
    restIdleTimeOut: ${SW_RECEIVER_SHARING_REST_IDLE_TIMEOUT:30000}
264
    restAcceptQueueSize: ${SW_RECEIVER_SHARING_REST_QUEUE_SIZE:0}
265
    httpMaxRequestHeaderSize: ${SW_RECEIVER_SHARING_HTTP_MAX_REQUEST_HEADER_SIZE:8192}
266
    # For gRPC server
267
    gRPCHost: ${SW_RECEIVER_GRPC_HOST:0.0.0.0}
268
    gRPCPort: ${SW_RECEIVER_GRPC_PORT:0}
269
    maxConcurrentCallsPerConnection: ${SW_RECEIVER_GRPC_MAX_CONCURRENT_CALL:0}
270
    maxMessageSize: ${SW_RECEIVER_GRPC_MAX_MESSAGE_SIZE:0}
271
    gRPCThreadPoolQueueSize: ${SW_RECEIVER_GRPC_POOL_QUEUE_SIZE:0}
272
    gRPCThreadPoolSize: ${SW_RECEIVER_GRPC_THREAD_POOL_SIZE:0}
273
    gRPCSslEnabled: ${SW_RECEIVER_GRPC_SSL_ENABLED:false}
274
    gRPCSslKeyPath: ${SW_RECEIVER_GRPC_SSL_KEY_PATH:""}
275
    gRPCSslCertChainPath: ${SW_RECEIVER_GRPC_SSL_CERT_CHAIN_PATH:""}
276
    gRPCSslTrustedCAsPath: ${SW_RECEIVER_GRPC_SSL_TRUSTED_CAS_PATH:""}
277
    authentication: ${SW_AUTHENTICATION:""}
278
receiver-register:
279
  selector: ${SW_RECEIVER_REGISTER:default}
280
  default:
281

282
receiver-trace:
283
  selector: ${SW_RECEIVER_TRACE:default}
284
  default:
285

286
receiver-jvm:
287
  selector: ${SW_RECEIVER_JVM:default}
288
  default:
289

290
receiver-clr:
291
  selector: ${SW_RECEIVER_CLR:default}
292
  default:
293

294
receiver-profile:
295
  selector: ${SW_RECEIVER_PROFILE:default}
296
  default:
297

298
receiver-zabbix:
299
  selector: ${SW_RECEIVER_ZABBIX:-}
300
  default:
301
    port: ${SW_RECEIVER_ZABBIX_PORT:10051}
302
    host: ${SW_RECEIVER_ZABBIX_HOST:0.0.0.0}
303
    activeFiles: ${SW_RECEIVER_ZABBIX_ACTIVE_FILES:agent}
304

305
service-mesh:
306
  selector: ${SW_SERVICE_MESH:default}
307
  default:
308

309
envoy-metric:
310
  selector: ${SW_ENVOY_METRIC:default}
311
  default:
312
    acceptMetricsService: ${SW_ENVOY_METRIC_SERVICE:true}
313
    alsHTTPAnalysis: ${SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS:""}
314
    alsTCPAnalysis: ${SW_ENVOY_METRIC_ALS_TCP_ANALYSIS:""}
315
    # `k8sServiceNameRule` allows you to customize the service name in ALS via Kubernetes metadata,
316
    # the available variables are `pod`, `service`, f.e., you can use `${service.metadata.name}-${pod.metadata.labels.version}`
317
    # to append the version number to the service name.
318
    # Be careful, when using environment variables to pass this configuration, use single quotes(`''`) to avoid it being evaluated by the shell.
319
    k8sServiceNameRule: ${K8S_SERVICE_NAME_RULE:"${pod.metadata.labels.(service.istio.io/canonical-name)}"}
320
    istioServiceNameRule: ${ISTIO_SERVICE_NAME_RULE:"${serviceEntry.metadata.name}"}
321

322
kafka-fetcher:
323
  selector: ${SW_KAFKA_FETCHER:-}
324
  default:
325
    bootstrapServers: ${SW_KAFKA_FETCHER_SERVERS:localhost:9092}
326
    namespace: ${SW_NAMESPACE:""}
327
    partitions: ${SW_KAFKA_FETCHER_PARTITIONS:3}
328
    replicationFactor: ${SW_KAFKA_FETCHER_PARTITIONS_FACTOR:2}
329
    enableNativeProtoLog: ${SW_KAFKA_FETCHER_ENABLE_NATIVE_PROTO_LOG:true}
330
    enableNativeJsonLog: ${SW_KAFKA_FETCHER_ENABLE_NATIVE_JSON_LOG:true}
331
    consumers: ${SW_KAFKA_FETCHER_CONSUMERS:1}
332
    kafkaHandlerThreadPoolSize: ${SW_KAFKA_HANDLER_THREAD_POOL_SIZE:-1}
333
    kafkaHandlerThreadPoolQueueSize: ${SW_KAFKA_HANDLER_THREAD_POOL_QUEUE_SIZE:-1}
334

335
receiver-meter:
336
  selector: ${SW_RECEIVER_METER:default}
337
  default:
338

339
receiver-otel:
340
  selector: ${SW_OTEL_RECEIVER:default}
341
  default:
342
    enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:"otlp-metrics,otlp-logs"}
343
    enabledOtelMetricsRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:"apisix,nginx/*,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*,aws-dynamodb/*,aws-gateway/*,redis/*,elasticsearch/*,rabbitmq/*,mongodb/*,kafka/*,pulsar/*,bookkeeper/*"}
344

345
receiver-zipkin:
346
  selector: ${SW_RECEIVER_ZIPKIN:-}
347
  default:
348
    # Defines a set of span tag keys which are searchable.
349
    # The max length of key=value should be less than 256 or will be dropped.
350
    searchableTracesTags: ${SW_ZIPKIN_SEARCHABLE_TAG_KEYS:http.method}
351
    # The sample rate precision is 1/10000, should be between 0 and 10000
352
    sampleRate: ${SW_ZIPKIN_SAMPLE_RATE:10000}
353
    ## The below configs are for OAP collect zipkin trace from HTTP
354
    enableHttpCollector: ${SW_ZIPKIN_HTTP_COLLECTOR_ENABLED:true}
355
    restHost: ${SW_RECEIVER_ZIPKIN_REST_HOST:0.0.0.0}
356
    restPort: ${SW_RECEIVER_ZIPKIN_REST_PORT:9411}
357
    restContextPath: ${SW_RECEIVER_ZIPKIN_REST_CONTEXT_PATH:/}
358
    restMaxThreads: ${SW_RECEIVER_ZIPKIN_REST_MAX_THREADS:200}
359
    restIdleTimeOut: ${SW_RECEIVER_ZIPKIN_REST_IDLE_TIMEOUT:30000}
360
    restAcceptQueueSize: ${SW_RECEIVER_ZIPKIN_REST_QUEUE_SIZE:0}
361
    ## The below configs are for OAP collect zipkin trace from kafka
362
    enableKafkaCollector: ${SW_ZIPKIN_KAFKA_COLLECTOR_ENABLED:false}
363
    kafkaBootstrapServers: ${SW_ZIPKIN_KAFKA_SERVERS:localhost:9092}
364
    kafkaGroupId: ${SW_ZIPKIN_KAFKA_GROUP_ID:zipkin}
365
    kafkaTopic: ${SW_ZIPKIN_KAFKA_TOPIC:zipkin}
366
    # Kafka consumer config, JSON format as Properties. If it contains the same key with above, would override.
367
    kafkaConsumerConfig: ${SW_ZIPKIN_KAFKA_CONSUMER_CONFIG:"{\"auto.offset.reset\":\"earliest\",\"enable.auto.commit\":true}"}
368
    # The Count of the topic consumers
369
    kafkaConsumers: ${SW_ZIPKIN_KAFKA_CONSUMERS:1}
370
    kafkaHandlerThreadPoolSize: ${SW_ZIPKIN_KAFKA_HANDLER_THREAD_POOL_SIZE:-1}
371
    kafkaHandlerThreadPoolQueueSize: ${SW_ZIPKIN_KAFKA_HANDLER_THREAD_POOL_QUEUE_SIZE:-1}
372

373
receiver-browser:
374
  selector: ${SW_RECEIVER_BROWSER:default}
375
  default:
376
    # The sample rate precision is 1/10000. 10000 means 100% sample in default.
377
    sampleRate: ${SW_RECEIVER_BROWSER_SAMPLE_RATE:10000}
378

379
receiver-log:
380
  selector: ${SW_RECEIVER_LOG:default}
381
  default:
382

383
query:
384
  selector: ${SW_QUERY:graphql}
385
  graphql:
386
    # Enable the log testing API to test the LAL.
387
    # NOTE: This API evaluates untrusted code on the OAP server.
388
    # A malicious script can do significant damage (steal keys and secrets, remove files and directories, install malware, etc).
389
    # As such, please enable this API only when you completely trust your users.
390
    enableLogTestTool: ${SW_QUERY_GRAPHQL_ENABLE_LOG_TEST_TOOL:false}
391
    # Maximum complexity allowed for the GraphQL query that can be used to
392
    # abort a query if the total number of data fields queried exceeds the defined threshold.
393
    maxQueryComplexity: ${SW_QUERY_MAX_QUERY_COMPLEXITY:3000}
394
    # Allow user add, disable and update UI template
395
    enableUpdateUITemplate: ${SW_ENABLE_UPDATE_UI_TEMPLATE:false}
396
    # "On demand log" allows users to fetch Pod containers' log in real time,
397
    # because this might expose secrets in the logs (if any), users need
398
    # to enable this manually, and add permissions to OAP cluster role.
399
    enableOnDemandPodLog: ${SW_ENABLE_ON_DEMAND_POD_LOG:false}
400

401
# This module is for Zipkin query API and support zipkin-lens UI
402
query-zipkin:
403
  selector: ${SW_QUERY_ZIPKIN:-}
404
  default:
405
    # For HTTP server
406
    restHost: ${SW_QUERY_ZIPKIN_REST_HOST:0.0.0.0}
407
    restPort: ${SW_QUERY_ZIPKIN_REST_PORT:9412}
408
    restContextPath: ${SW_QUERY_ZIPKIN_REST_CONTEXT_PATH:/zipkin}
409
    restMaxThreads: ${SW_QUERY_ZIPKIN_REST_MAX_THREADS:200}
410
    restIdleTimeOut: ${SW_QUERY_ZIPKIN_REST_IDLE_TIMEOUT:30000}
411
    restAcceptQueueSize: ${SW_QUERY_ZIPKIN_REST_QUEUE_SIZE:0}
412
    # Default look back for traces and autocompleteTags, 1 day in millis
413
    lookback: ${SW_QUERY_ZIPKIN_LOOKBACK:86400000}
414
    # The Cache-Control max-age (seconds) for serviceNames, remoteServiceNames and spanNames
415
    namesMaxAge: ${SW_QUERY_ZIPKIN_NAMES_MAX_AGE:300}
416
    ## The below config are OAP support for zipkin-lens UI
417
    # Default traces query max size
418
    uiQueryLimit: ${SW_QUERY_ZIPKIN_UI_QUERY_LIMIT:10}
419
    # Default look back on the UI for search traces, 15 minutes in millis
420
    uiDefaultLookback: ${SW_QUERY_ZIPKIN_UI_DEFAULT_LOOKBACK:900000}
421

422
#This module is for PromQL API.
423
promql:
424
  selector: ${SW_PROMQL:default}
425
  default:
426
    # For HTTP server
427
    restHost: ${SW_PROMQL_REST_HOST:0.0.0.0}
428
    restPort: ${SW_PROMQL_REST_PORT:9090}
429
    restContextPath: ${SW_PROMQL_REST_CONTEXT_PATH:/}
430
    restMaxThreads: ${SW_PROMQL_REST_MAX_THREADS:200}
431
    restIdleTimeOut: ${SW_PROMQL_REST_IDLE_TIMEOUT:30000}
432
    restAcceptQueueSize: ${SW_PROMQL_REST_QUEUE_SIZE:0}
433

434
#This module is for LogQL API.
435
logql:
436
  selector: ${SW_LOGQL:default}
437
  default:
438
    # For HTTP server
439
    restHost: ${SW_LOGQL_REST_HOST:0.0.0.0}
440
    restPort: ${SW_LOGQL_REST_PORT:3100}
441
    restContextPath: ${SW_LOGQL_REST_CONTEXT_PATH:/}
442
    restMaxThreads: ${SW_LOGQL_REST_MAX_THREADS:200}
443
    restIdleTimeOut: ${SW_LOGQL_REST_IDLE_TIMEOUT:30000}
444
    restAcceptQueueSize: ${SW_LOGQL_REST_QUEUE_SIZE:0}
445

446
alarm:
447
  selector: ${SW_ALARM:default}
448
  default:
449

450
telemetry:
451
  selector: ${SW_TELEMETRY:none}
452
  none:
453
  prometheus:
454
    host: ${SW_TELEMETRY_PROMETHEUS_HOST:0.0.0.0}
455
    port: ${SW_TELEMETRY_PROMETHEUS_PORT:1234}
456
    sslEnabled: ${SW_TELEMETRY_PROMETHEUS_SSL_ENABLED:false}
457
    sslKeyPath: ${SW_TELEMETRY_PROMETHEUS_SSL_KEY_PATH:""}
458
    sslCertChainPath: ${SW_TELEMETRY_PROMETHEUS_SSL_CERT_CHAIN_PATH:""}
459

460
configuration:
461
  selector: ${SW_CONFIGURATION:none}
462
  none:
463
  grpc:
464
    host: ${SW_DCS_SERVER_HOST:""}
465
    port: ${SW_DCS_SERVER_PORT:80}
466
    clusterName: ${SW_DCS_CLUSTER_NAME:SkyWalking}
467
    period: ${SW_DCS_PERIOD:20}
468
  apollo:
469
    apolloMeta: ${SW_CONFIG_APOLLO:http://localhost:8080}
470
    apolloCluster: ${SW_CONFIG_APOLLO_CLUSTER:default}
471
    apolloEnv: ${SW_CONFIG_APOLLO_ENV:""}
472
    appId: ${SW_CONFIG_APOLLO_APP_ID:skywalking}
473
  zookeeper:
474
    period: ${SW_CONFIG_ZK_PERIOD:60} # Unit seconds, sync period. Default fetch every 60 seconds.
475
    namespace: ${SW_CONFIG_ZK_NAMESPACE:/default}
476
    hostPort: ${SW_CONFIG_ZK_HOST_PORT:localhost:2181}
477
    # Retry Policy
478
    baseSleepTimeMs: ${SW_CONFIG_ZK_BASE_SLEEP_TIME_MS:1000} # initial amount of time to wait between retries
479
    maxRetries: ${SW_CONFIG_ZK_MAX_RETRIES:3} # max number of times to retry
480
  etcd:
481
    period: ${SW_CONFIG_ETCD_PERIOD:60} # Unit seconds, sync period. Default fetch every 60 seconds.
482
    endpoints: ${SW_CONFIG_ETCD_ENDPOINTS:http://localhost:2379}
483
    namespace: ${SW_CONFIG_ETCD_NAMESPACE:/skywalking}
484
    authentication: ${SW_CONFIG_ETCD_AUTHENTICATION:false}
485
    user: ${SW_CONFIG_ETCD_USER:}
486
    password: ${SW_CONFIG_ETCD_password:}
487
  consul:
488
    # Consul host and ports, separated by comma, e.g. 1.2.3.4:8500,2.3.4.5:8500
489
    hostAndPorts: ${SW_CONFIG_CONSUL_HOST_AND_PORTS:1.2.3.4:8500}
490
    # Sync period in seconds. Defaults to 60 seconds.
491
    period: ${SW_CONFIG_CONSUL_PERIOD:60}
492
    # Consul aclToken
493
    aclToken: ${SW_CONFIG_CONSUL_ACL_TOKEN:""}
494
  k8s-configmap:
495
    period: ${SW_CONFIG_CONFIGMAP_PERIOD:60}
496
    namespace: ${SW_CLUSTER_K8S_NAMESPACE:default}
497
    labelSelector: ${SW_CLUSTER_K8S_LABEL:app=collector,release=skywalking}
498
  nacos:
499
    # Nacos Server Host
500
    serverAddr: ${SW_CONFIG_NACOS_SERVER_ADDR:127.0.0.1}
501
    # Nacos Server Port
502
    port: ${SW_CONFIG_NACOS_SERVER_PORT:8848}
503
    # Nacos Configuration Group
504
    group: ${SW_CONFIG_NACOS_SERVER_GROUP:skywalking}
505
    # Nacos Configuration namespace
506
    namespace: ${SW_CONFIG_NACOS_SERVER_NAMESPACE:}
507
    # Unit seconds, sync period. Default fetch every 60 seconds.
508
    period: ${SW_CONFIG_NACOS_PERIOD:60}
509
    # Nacos auth username
510
    username: ${SW_CONFIG_NACOS_USERNAME:""}
511
    password: ${SW_CONFIG_NACOS_PASSWORD:""}
512
    # Nacos auth accessKey
513
    accessKey: ${SW_CONFIG_NACOS_ACCESSKEY:""}
514
    secretKey: ${SW_CONFIG_NACOS_SECRETKEY:""}
515

516
exporter:
517
  selector: ${SW_EXPORTER:-}
518
  default:
519
    # gRPC exporter
520
    enableGRPCMetrics: ${SW_EXPORTER_ENABLE_GRPC_METRICS:false}
521
    gRPCTargetHost: ${SW_EXPORTER_GRPC_HOST:127.0.0.1}
522
    gRPCTargetPort: ${SW_EXPORTER_GRPC_PORT:9870}
523
    # Kafka exporter
524
    enableKafkaTrace: ${SW_EXPORTER_ENABLE_KAFKA_TRACE:false}
525
    enableKafkaLog: ${SW_EXPORTER_ENABLE_KAFKA_LOG:false}
526
    kafkaBootstrapServers: ${SW_EXPORTER_KAFKA_SERVERS:localhost:9092}
527
    # Kafka producer config, JSON format as Properties.
528
    kafkaProducerConfig: ${SW_EXPORTER_KAFKA_PRODUCER_CONFIG:""}
529
    kafkaTopicTrace: ${SW_EXPORTER_KAFKA_TOPIC_TRACE:skywalking-export-trace}
530
    kafkaTopicLog: ${SW_EXPORTER_KAFKA_TOPIC_LOG:skywalking-export-log}
531
    exportErrorStatusTraceOnly: ${SW_EXPORTER_KAFKA_TRACE_FILTER_ERROR:false}
532

533
health-checker:
534
  selector: ${SW_HEALTH_CHECKER:-}
535
  default:
536
    checkIntervalSeconds: ${SW_HEALTH_CHECKER_INTERVAL_SECONDS:5}
537

538
debugging-query:
539
  selector: ${SW_DEBUGGING_QUERY:default}
540
  default:
541
    # Include the list of keywords to filter configurations including secrets. Separate keywords by a comma.
542
    keywords4MaskingSecretsOfConfig: ${SW_DEBUGGING_QUERY_KEYWORDS_FOR_MASKING_SECRETS:user,password,token,accessKey,secretKey,authentication}
543

544
configuration-discovery:
545
  selector: ${SW_CONFIGURATION_DISCOVERY:default}
546
  default:
547
    disableMessageDigest: ${SW_DISABLE_MESSAGE_DIGEST:false}
548

549
receiver-event:
550
  selector: ${SW_RECEIVER_EVENT:default}
551
  default:
552

553
receiver-ebpf:
554
  selector: ${SW_RECEIVER_EBPF:default}
555
  default:
556
    # The continuous profiling policy cache time, Unit is second.
557
    continuousPolicyCacheTimeout: ${SW_CONTINUOUS_POLICY_CACHE_TIMEOUT:60}
558

559
receiver-telegraf:
560
  selector: ${SW_RECEIVER_TELEGRAF:default}
561
  default:
562
    activeFiles: ${SW_RECEIVER_TELEGRAF_ACTIVE_FILES:vm}
563

564
aws-firehose:
565
  selector: ${SW_RECEIVER_AWS_FIREHOSE:default}
566
  default:
567
    host: ${SW_RECEIVER_AWS_FIREHOSE_HTTP_HOST:0.0.0.0}
568
    port: ${SW_RECEIVER_AWS_FIREHOSE_HTTP_PORT:12801}
569
    contextPath: ${SW_RECEIVER_AWS_FIREHOSE_HTTP_CONTEXT_PATH:/}
570
    maxThreads: ${SW_RECEIVER_AWS_FIREHOSE_HTTP_MAX_THREADS:200}
571
    idleTimeOut: ${SW_RECEIVER_AWS_FIREHOSE_HTTP_IDLE_TIME_OUT:30000}
572
    acceptQueueSize: ${SW_RECEIVER_AWS_FIREHOSE_HTTP_ACCEPT_QUEUE_SIZE:0}
573
    maxRequestHeaderSize: ${SW_RECEIVER_AWS_FIREHOSE_HTTP_MAX_REQUEST_HEADER_SIZE:8192}
574
    firehoseAccessKey: ${SW_RECEIVER_AWS_FIREHOSE_ACCESS_KEY:}
575
    enableTLS: ${SW_RECEIVER_AWS_FIREHOSE_HTTP_ENABLE_TLS:false}
576
    tlsKeyPath: ${SW_RECEIVER_AWS_FIREHOSE_HTTP_TLS_KEY_PATH:}
577
    tlsCertChainPath: ${SW_RECEIVER_AWS_FIREHOSE_HTTP_TLS_CERT_CHAIN_PATH:}
578

579
ai-pipeline:
580
  selector: ${SW_AI_PIPELINE:default}
581
  default:
582
    uriRecognitionServerAddr: ${SW_AI_PIPELINE_URI_RECOGNITION_SERVER_ADDR:}
583
    uriRecognitionServerPort: ${SW_AI_PIPELINE_URI_RECOGNITION_SERVER_PORT:17128}

展开收起

其中 core.default.recordDataTTL 和 core.default.metricsDataTTL 可以通过 docker 启动时指定 environment 来使其中的 ${EN_VAR} 变量生效, 替换默认的值.

config	environment	unit
core.default.recordDataTTL	SW_CORE_RECORD_DATA_TTL	day
core.default.metricsDataTTL	SW_CORE_METRICS_DATA_TTL	day

先改为保留 3 天

1
skywalking-oap-server:
2
    image: apache/skywalking-oap-server:9.7.0
3
    container_name: skywalking-oap-server
4
    environment:
5
      - SW_STORAGE=elasticsearch
6
      - SW_STORAGE_ES_CLUSTER_NODES=es01:9200,es02:9200
7
      - SW_CORE_RECORD_DATA_TTL=3
8
      - SW_CORE_METRICS_DATA_TTL=3
9
    network_mode: host

修改后, docker 容器重启, 日志输出如下, 发现已经生效(TTL.metrics 和 TTL.record 都显示为 3 天)

1
Running Mode                               |   null
2
TTL.metrics                                |   3
3
TTL.record                                 |   3
4
Version                                    |   9.7.0-066457b
5
module.agent-analyzer.provider             |   default
6
module.ai-pipeline.provider                |   default
7
module.alarm.provider                      |   default
8
module.aws-firehose.provider               |   default
9
module.cluster.provider                    |   standalone
10
module.configuration-discovery.provider    |   default
11
module.configuration.provider              |   none
12
module.core.provider                       |   default
13
module.debugging-query.provider            |   default
14
module.envoy-metric.provider               |   default
15
module.event-analyzer.provider             |   default
16
module.log-analyzer.provider               |   default
17
module.logql.provider                      |   default
18
module.promql.provider                     |   default
19
module.query.provider                      |   graphql
20
module.receiver-browser.provider           |   default
21
module.receiver-clr.provider               |   default
22
module.receiver-ebpf.provider              |   default
23
module.receiver-event.provider             |   default
24
module.receiver-jvm.provider               |   default
25
module.receiver-log.provider               |   default
26
module.receiver-meter.provider             |   default
27
module.receiver-otel.provider              |   default
28
module.receiver-profile.provider           |   default
29
module.receiver-register.provider          |   default
30
module.receiver-sharing-server.provider    |   default
31
module.receiver-telegraf.provider          |   default
32
module.receiver-trace.provider             |   default
33
module.service-mesh.provider               |   default
34
module.storage.provider                    |   elasticsearch
35
module.telemetry.provider                  |   none
36
oap.external.grpc.host                     |   0.0.0.0
37
oap.external.grpc.port                     |   11800
38
oap.external.http.host                     |   0.0.0.0
39
oap.external.http.port                     |   12800
40
oap.internal.comm.host                     |   0.0.0.0
41
oap.internal.comm.port                     |   11800

阿里云 SLS 日志接入配置

Sun, 10 Sep 2023 00:00:00 GMT

Untitled 97|接入原理

在 ECS 上安装 logtail 程序（参考如何在Linux服务器上安装、升级和卸载Logtail_日志服务-阿里云帮助中心）

Terminal window

1
# 下载
2
wget http://logtail-release-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/linux64/logtail.sh -O logtail.sh;chmod 755 logtail.sh
3

4
# 下载后安装
5
./logtail.sh install auto
6

7
# 查看 logtail 进程
8
[root@pro-qnyp-002 ~]# ps -ef | grep logtail | grep -v grep
9
root     16505     1  0 Sep06 ?        00:00:00 /usr/local/ilogtail/ilogtail
10
root     16507 16505  0 Sep06 ?        00:08:10 /usr/local/ilogtail/ilogtail

进入 sls 控制台，将 ECS 添加指定机器组

Untitled 98

填写机器 ip 相关信息

Untitled 99

添加 logtail 配置

Untitled 100|添加 logtail 配置

Untitled 101|日志路径是指机器组中机器的 java 应用日志路径，文件名支持 * 通配符，这里时间搓使用日志中的时间搓

Untitled 102|启用高精度的时间搓

Untitled 103|务必在机器组管理中添加对应的机器组

等一会儿过后就能在 sls 控制台中查看到日志

Untitled 104

JMM

Tue, 15 Aug 2023 00:00:00 GMT

CPU多级缓存结构与CPU缓存一致性协议MESI#

image 31

硬件层面:

多核处理器缓存架构导致多个处理器之间的缓存一致性问题
为了解决缓存一致性问题引入了缓存一致性协议MESI

软件层面：

硬件层面上，处理器其实并不知道何时需要使用内存屏障指令，它只是提供内存屏障的能力，然后将决定权交给软件层
软件层面上JVM 抽象处理 Java 内存模型 JMM 来定义线程本地内存和主内存之间的关系，并提供 volatile 关键字来让软件层面调用底层的内存屏障来解决多线程之间的可见性问题
Java 规范中有 as-if-serial 和 happens-before 原则 as-if-serial

**不管怎么重排序（编译器和处理器为了提高并行度），（单线程）程序的执行结果不能被改变。**编译器、runtime 和处理器都必须遵守 as-if-serial 语义。为了遵守 as-if-serial 语义，编译器和处理器不会对存在数据依赖关系的操作做重排序，因为这种重排序会改变执行结果。但是，如果操作之间不存在数据依赖关系，这些操作就可能被编译器和处理器重排序。

happens-before

对于两个操作A和B，这两个操作可以在不同的线程中执行。如果A Happens-Before B，那么可以保证，当A操作执行完后，A操作的执行结果对B操作是可见的。

程序顺序规则：在同一个线程中，书写在前面的操作happen-before后面的操作。

volatile变量规则：对一个volatile域的写，happens-before于任意后续对此变量的操作。

监视器锁规则：对一个锁的解锁，happens-before于随后对这个锁的加锁。

传递性：如果A happens-before B，且B happens-before C，那么A happens-before C。

start()规则：如果线程A执行操作ThreadB.start()（启动线程B），那么A线程的ThreadB.start()操作happens-before于线程B中的任意操作。

join()规则：如果线程A执行操作ThreadB.join()并成功返回，那么线程B中的任意操作happens-before于线程A从ThreadB.join()操作成功返回。

程序中断规则：对线程interrupted()方法的调用happens-before于被中断线程的代码检测到中断时间的发生。

对象finalize规则：一个对象的初始化完成（构造函数执行结束）happens-before于它的finalize()方法的开始。

JMM#

原子性

JMM 通过关键字 synchronized 实现原子性。底层通过 monitorenter/monitorexit 指令来实现

可见性

共享变量的修改对其他线程可见。JMM 通过 volatile 关键字保证对共享变量的修改可以马上刷新到主内存，且其他线程操作共享变量必须从主内存获取最新值。

有序性

禁止重排序，确保可见性，底层是内存屏障。

📎 参考文章#

JVM GC

Wed, 12 Jul 2023 00:00:00 GMT

为什么 GC 中吞吐率（Throughput）和延迟（Latency）相互对立#

吞吐率（Throughput）公式#

TworkT_{work}Twork = 业务计算时长

TgcT_{gc}Tgc = GC时长

η\etaη = 吞吐率

η=TworkTwork+Tgc\eta = \frac{T_{work}}{T_{work} + T_{gc}}η=Twork+TgcTwork

为了提高吞吐率 η\etaη，我们希望：

减少 GC 时间 TgcT_{gc}Tgc，让应用执行更多任务。
减少 GC 触发次数，即 GC 发生得越少越好。

但这样会带来一个问题：如果 GC 触发得少，意味着垃圾会在堆中积累得更多，每次 GC 需要回收的对象会更多，导致单次 GC 停顿时间（Latency）更长。

延迟（Latency）公式#

TpauseT_{pause}Tpause = 一次GC停顿的时间

SSS = 存活对象数量

HHH = 堆大小

GC 停顿时间（Latency）与存活对象数量 SSS 和堆大小 HHH 成正比

Tpause∝S+HT_{pause} \propto S + HTpause∝S+H

几种垃圾回收期的对比#

垃圾回收算法#

复制算法
- 简单高效
- 内存利用率低
标记-清除
- 简单
- 内存空间上碎片化
标记-整理
- 无碎片
- 实现复杂

GC回收期	回收区域	算法	线程	工作模式	适用场景
Serial	Young	复制	单线程	独占	客户端应用或资源受限的环境
ParNew（Serial多线程版本）	Young	复制	多线程	独占	与CMS结合使用
Parallel Scavenge	Young	复制	多线程	独占	吞吐量优先
Serial Old	Old	标记-整理	单线程	独占
CMS（已废弃）	Old	标记-清除	多线程	并发	对响应时间敏感的应用，如Web服务器
Parallel Old	Old	标记-整理	多线程	独占	与 Parallel Scavenge 结合使用，吞吐量优先
G1	Young + Old	Young: 复制 Old: 整理+复制	多线程	并发	大内存、多核处理器，适用于需要低延迟和高吞吐量的应用
ZGC	Young + Old	标记-整理	多线程	并发	需要极低延迟的应用，如实时系统
Shenandoah	Young + Old	标记-整理	多线程	并发	需要低延迟的应用

jasypt-maven-plugin 使用

Sun, 02 Jul 2023 00:00:00 GMT

通过 maven 插件，快速生成加密后的文本#

引入插件

1
<!-- 引入插件 -->
2
<plugin>
3
    <groupId>com.github.ulisesbocchio</groupId>
4
    <artifactId>jasypt-maven-plugin</artifactId>
5
    <version>${jasypt.version}</version>
6
</plugin>

命令行调用加密

Terminal window

1
mvn jasypt:encrypt-value -Djasypt.encryptor.password="加密用的密码" -Djasypt.plugin.value="需要进行加密的原文"

将得到的 ENC(…) 放入配置文件中

1
spring.datasource.password: ENC(Tmv/Gm9V+69hnVZYcbiTwnFmPCYr4/UY2XildfEKgSkf8F3U+zFOHSAHiMM6ksUZ)

生产环境, 启动时加入参数 -Djasypt.encryptor.password=加密用的密码
命令行调用解密

Terminal window

1
mvn jasypt:decrypt-value -Djasypt.encryptor.password="加密用的密码" -Djasypt.plugin.value="加密后得到的带ENC(...)的密文"

通过 Syncthing 实现多台服务器的数据同步

Sun, 30 Apr 2023 00:00:00 GMT

目前管理了好几台服务器，各服务器之间有一些文件需要共享，之前使用过 seafile，感觉不太好用，现在切换为 syncthing 实现文件同步。

操作流程#

docker-compose.yml#

1
services:
2

3
  syncthing:
4
    image: linuxserver/syncthing:latest
5
    container_name: syncthing
6
    hostname: syncthing #optional
7
    environment:
8
      - PUID=1000
9
      - PGID=1000
10
      - TZ=Asia/Shanghai
11
    volumes:
12
      - /path/to/syncthing/config:/config
13
      - /path/to/syncthing/data:/data
14
    ports:
15
      - 8384:8384
16
      - 22000:22000/tcp
17
      - 22000:22000/udp
18
      - 21027:21027/udp
19
    restart: unless-stopped

命令行启动#

Terminal window

1
docker compose --compatibility up -d syncthing

应用同步配置#

进入设置

![[Untitled 18.png]]

设置设备名

![[Untitled 19.png]]

安全起见，设置用户名密码

![[Untitled 20.png]]

获取设备 ID

![[Untitled 21.png]]

![[Untitled 22.png]]

增加远程设备

![[Untitled 23.png]]

![[Untitled 24.png]]

常见问题#

移动目录#

比如原先共享的目录存放在 /data/pathA/devops,想要移动到路径 /data/pathB/devops

停止 syncthing
修改配置文件 /config/config.xml

1
<configuration version="37">
2
    <!-- 改下面的 path -->
3
    <folder id="devops" label="devops" path="/data/pathA/devops" type="sendreceive" rescanIntervalS="3600" fsWatcherEnabled="true" fsWatcherDelayS="10" ignorePerms="false" autoNormalize="true">
4
        <!-- ... -->
5
    </folder>
6
</configuration>

移动 /data/pathA/devops 到 /data/pathB/devops
启动 syncthing

📎 参考文章#

why can’t we change the folder paths? - Support - Syncthing Community Forum

SpringBoot 在 Docker 容器中的 1 号进程

Sat, 02 Jul 2022 00:00:00 GMT

spring boot 项目配置中，配置了平滑关机，但是不生效。

查看 docker 容器中的进程，发现并不是 1 号进程

Terminal window

1
root@268bcf4264ae:/app# ps -ef | grep java
2
root         1     0  0 08:35 ?        00:00:00 /bin/sh -c sh -c 'java $JAVA_OPTS org.springframework.boot.loader.JarLauncher'
3
root         7     1  0 08:35 ?        00:00:00 sh -c java $JAVA_OPTS org.springframework.boot.loader.JarLauncher
4
root         8     7  8 08:35 ?        00:01:31 java -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djasypt.encryptor.password=123456 -Dspring.cloud.bootstrap.location=/config/bootstrap.yml -Dspring.jmx.enabled=false -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:+StartAttachListener -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/qnvip-loan-backend-dump.hprof -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1024k -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xms2G -Xmx2G -Xss512k -Xloggc:/logs/gc/gc.log org.springframework.boot.loader.JarLauncher
5
root      1438  1421  0 08:53 pts/0    00:00:00 grep --color=auto java

查看 Dockerfile 中的启动命令

Terminal window

1
ENTRYPOINT sh -c 'java $JAVA_OPTS org.springframework.boot.loader.JarLauncher'

google之，ENTRYPOINT 命令修改如下

Terminal window

1
ENTRYPOINT ["/bin/sh", "-c", "exec java $JAVA_OPTS org.springframework.boot.loader.JarLauncher"]

构建镜像，重新启动后，再次进入容器查看进程信息

Terminal window

1
root@8fff76975d58:/app# ps -ef | grep java
2
root         1     0 37 08:57 ?        00:00:55 java -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djasypt.encryptor.password=123456 -Dspring.cloud.bootstrap.location=/config/bootstrap.yml -Dspring.jmx.enabled=false -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:+StartAttachListener -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/qnvip-loan-backend-dump.hprof -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1024k -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xms2G -Xmx2G -Xss512k -Xloggc:/logs/gc/gc.log org.springframework.boot.loader.JarLauncher
3
root       368   354  0 08:59 pts/0    00:00:00 grep --color=auto java

已经成功变为 1 号进程

验证平滑下线

Terminal window

1
qnvip-loan-backend  | 2023-09-06 09:07:19.389+08:00  INFO 1 --- TID: /NA [ionShutdownHook] o.a.d.c.e.listener.LoggingEventListener  :  [DUBBO] Dubbo Service has been destroyed., dubbo version: 2.7.23, current host: 172.20.0.3
2
qnvip-loan-backend  | 2023-09-06 09:07:19.390+08:00  INFO 1 --- TID: /NA [ionShutdownHook] o.a.d.r.support.AbstractRegistryFactory  :  [DUBBO] Close all registries [nacos://192.168.1.216:8848/org.apache.dubbo.registry.RegistryService?application=qnvip-loan-backend&check=false&dubbo=2.0.2&id=org.apache.dubbo.config.RegistryConfig#0&interface=org.apache.dubbo.registry.RegistryService&namespace=10fb9841-85bc-4c22-8faf-d6d9c7bc4734&pid=1&qos.enable=false&release=2.7.23&timestamp=1693961852930], dubbo version: 2.7.23, current host: 172.20.0.3
3
qnvip-loan-backend  | 2023-09-06 09:07:19.390+08:00  INFO 1 --- TID: /NA [ionShutdownHook] o.a.dubbo.registry.nacos.NacosRegistry   :  [DUBBO] Destroy registry:nacos://192.168.1.216:8848/org.apache.dubbo.registry.RegistryService?application=qnvip-loan-backend&check=false&dubbo=2.0.2&id=org.apache.dubbo.config.RegistryConfig#0&interface=org.apache.dubbo.registry.RegistryService&namespace=10fb9841-85bc-4c22-8faf-d6d9c7bc4734&pid=1&qos.enable=false&release=2.7.23&timestamp=1693961852930, dubbo version: 2.7.23, current host: 172.20.0.3
4
qnvip-loan-backend  | 2023-09-06 09:07:19.390+08:00  INFO 1 --- TID: /NA [ionShutdownHook] o.a.dubbo.registry.nacos.NacosRegistry   :  [DUBBO] Unregister: consumer://172.20.0.3/qnvip.rent.support.facade.CallRecordFacade?application=qnvip-loan-backend&category=consumers&check=false&dubbo=2.0.2&init=false&interface=qnvip.rent.support.facade.CallRecordFacade&metadata-type=remote&methods=getListByMobiles&pid=1&qos.enable=false&release=2.7.23&revision=1.0.0-SNAPSHOT&side=consumer&sticky=false&timeout=15000&timestamp=1693961854121, dubbo version: 2.7.23, current host: 172.20.0.3
5
qnvip-loan-backend  | 2023-09-06 09:07:19.393+08:00  INFO 1 --- TID: /NA [ionShutdownHook] o.a.dubbo.registry.nacos.NacosRegistry   :  [DUBBO] Destroy unregister url consumer://172.20.0.3/qnvip.rent.support.facade.CallRecordFacade?application=qnvip-loan-backend&category=consumers&check=false&dubbo=2.0.2&init=false&interface=qnvip.rent.support.facade.CallRecordFacade&metadata-type=remote&methods=getListByMobiles&pid=1&qos.enable=false&release=2.7.23&revision=1.0.0-SNAPSHOT&side=consumer&sticky=false&timeout=15000&timestamp=1693961854121, dubbo version: 2.7.23, current host: 172.20.0.3
6
qnvip-loan-backend  | 2023-09-06 09:07:19.393+08:00  INFO 1 --- TID: /NA [ionShutdownHook] o.a.dubbo.registry.nacos.NacosRegistry   :  [DUBBO] Unregister: consumer://172.20.0.3/com.qnvip.merchant.client.facade.StageOrderFacade?application=qnvip-loan-backend&category=consumers&check=false&dubbo=2.0.2&init=false&interface=com.qnvip.merchant.client.facade.StageOrderFacade&metadata-type=remote&methods=orderList,customerInfo,orderDetail,orderRepaymentInfoList&pid=1&qos.enable=false&release=2.7.23&retries=1&revision=2.0.0-SNAPSHOT&side=consumer&sticky=false&timeout=600000&timestamp=1693961854009, dubbo version: 2.7.23, current host: 172.20.0.3
7
qnvip-loan-backend  | 2023-09-06 09:07:19.395+08:00  INFO 1 --- TID: /NA [ionShutdownHook] o.a.dubbo.registry.nacos.NacosRegistry   :  [DUBBO] Destroy unregister url consumer://172.20.0.3/com.qnvip.merchant.client.facade.StageOrderFacade?application=qnvip-loan-backend&category=consumers&check=false&dubbo=2.0.2&init=false&interface=com.qnvip.merchant.client.facade.StageOrderFacade&metadata-type=remote&methods=orderList,customerInfo,orderDetail,orderRepaymentInfoList&pid=1&qos.enable=false&release=2.7.23&retries=1&revision=2.0.0-SNAPSHOT&side=consumer&sticky=false&timeout=600000&timestamp=1693961854009, dubbo version: 2.7.23, current host: 172.20.0.3
8
qnvip-loan-backend  | 2023-09-06 09:07:19.395+08:00  INFO 1 --- TID: /NA [ionShutdownHook] o.a.dubbo.registry.nacos.NacosRegistry   :  [DUBBO] Unregister: consumer://172.20.0.3/qnvip.rent.support.facade.MiniConfigFacade?application=qnvip-loan-backend&category=consumers&check=false&dubbo=2.0.2&init=false&interface=qnvip.rent.support.facade.MiniConfigFacade&metadata-type=remote&methods=getMiniTypePlatMap,getNameByMiniType,getMiniConfigMapList,getMiniTypeStrMap,getByMiniType,getChannelByMiniType&pid=1&qos.enable=false&release=2.7.23&revision=1.0.0-SNAPSHOT&side=consumer&sticky=false&timeout=15000&timestamp=1693961852907, dubbo version: 2.7.23, current host: 172.20.0.3

MySQL 批量修改日期格式

Tue, 08 Mar 2022 00:00:00 GMT

发现部分表建表时，时间日期格式都是 datetime(4)，实际上精度只要到 datetime(3) 就够了，需要进行批量修改。可以通过遍历 infomation_schema 的库自动生成修改语句

1
SELECT
2
  CONCAT(
3
    'ALTER TABLE `',
4
    TABLE_NAME,
5
    '` MODIFY COLUMN `',
6
    COLUMN_NAME,
7
    '` datetime(3) ',
8
  IF
9
    ( IS_NULLABLE = 'NO', 'NOT NULL', 'NULL' ),
10
  IF
11
    ( COLUMN_DEFAULT = 'CURRENT_TIMESTAMP(4)', ' DEFAULT CURRENT_TIMESTAMP(3) ', ' ' ),
12
  IF
13
    ( EXTRA = 'DEFAULT_GENERATED on update CURRENT_TIMESTAMP(4)', ' ON UPDATE CURRENT_TIMESTAMP(3) ', ' ' ),
14
  IF
15
    (
16
      CHARACTER_LENGTH( COLUMN_COMMENT ) > 0,
17
      CONCAT( "COMMENT '", COLUMN_COMMENT, "'" ),
18
      ''
19
    ),
20
    ';'
21
),
22
t1.*
23
FROM
24
  `COLUMNS` t1
25
WHERE
26
  TABLE_SCHEMA = '目标库'
27
  AND DATA_TYPE = 'datetime'
28
  AND DATETIME_PRECISION = 4;

MySQL 修改数据库字符集

Fri, 05 Mar 2021 00:00:00 GMT

Unicode 与 UTF-8#

Unicode 只是一个符号集，它只规定了符号的二进制代码，却没有规定这个二进制代码应该如何存储。

UTF-8 就是在互联网上使用最广的一种 Unicode 的实现方式（其他实现方式还有 UTF-16, UTF-32）。

MySQL 字符集 collation#

collation定义了哪个字符和哪个字符是“等价”的。所以如果指定“不区分大小写”，那么a和A，e和E就是等价的，这样查找时就会方便很多。但这还不够，世界上的文字很多，所以才会有“不区分音调”的要求，这时候e、ē、é、ě、è就是等价的，那么假设我们要进行拼音查找，只要按e去找就可以全部列出来，很方便。甚至，它们也和ê、ë也是等价的，这样就更方便了。

collation也定义了字符的排序规则，如果按照“字符顺序（而不是简单的‘字母顺序’）”来排序，哪个字符应当排在哪个字符前面。所以，尽管“啊”、“副”、“德”三个字的拼音开头分别为A、F、D，但直接选定collation为utf8mb4，它们并不会按照“啊”、“德”、“副”的顺序排序，而是会排成“副”、“啊”、“德”。如果你希望把中文字符按照拼音（英文字母）来排序，指定使用gb18030_chinese_ci作为collation就可以了。

团队初期不注意 mysql 字符集和字符集 **collation **问题，导致部分业务场景出现不可预料的问题，可以用以下脚本，自动生成修改语句，然后进行更新。

SQL 脚本#

修改 schema 的默认字符集

1
-- 通过 infomation_schema 自动批量生成修改 schema 默认字符集和collation语句
2
-- ALTER DATABASE `table_schema_name` CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
3
SELECT
4
  SCHEMA_NAME,
5
  concat( 'ALTER DATABASE `', SCHEMA_NAME, '` CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;' )
6
FROM
7
  `information_schema`.`SCHEMATA`
8
WHERE
9
  DEFAULT_COLLATION_NAME != 'utf8mb4_general_ci'

转换表的字符集

1
-- 通过 infomation_schema 自动批量生成修改表字符集和collation语句
2
-- ALTER TABLE `table_name` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
3
SELECT
4
  TABLE_SCHEMA,
5
  concat( 'ALTER TABLE `', TABLE_SCHEMA, '`.`', TABLE_NAME, '` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci, ALGORITHM = COPY;' )
6
FROM
7
  `information_schema`.`TABLES`
8
WHERE
9
  TABLE_COLLATION != 'utf8mb4_general_ci'
10
  AND TABLE_SCHEMA = 'table_schema_name'
11
ORDER BY
12
  TABLE_SCHEMA

📎 参考文章#

UFW 防火墙配置

Fri, 31 Jan 2020 00:00:00 GMT

某些 VPS 缺少防火墙配置（这一点头部的云厂商都做的比较好），所以这里记录一下 debian 系列 UFW 防火墙相关的配置操作。

以下操作默认 root 用户

先安装防火墙

Terminal window

1
apt install ufw -y

安装完成后，默认 ufw 是 inactive 状态，此时应该先配置 ssh 等允许规则，否则开启 ufw 后 ssh 都无法连接

所有传入连接的默认策略均为 deny

Terminal window

1
ufw allow 22/tcp
2
ufw allow 80
3
ufw allow 443
4
ufw allow 5900
5
ufw allow 5800

规则配置好后，启动防火墙

Terminal window

1
ufw enable

如果修改了配置，重启防火墙

Terminal window

1
ufw reload

📎 参考文章#

Mybatis 基础字段自动维护

Tue, 02 Jul 2019 00:00:00 GMT

当前系统中，表的 创建日期、更新日期，整理了有以下取名

创建日期	更新日期
create_ts	last_mnt_ts
created_time	updated_time
create_time	update_ts
	update_time
	modified_ts

在使用 Mybatis Plus 时，其自带的 自动填充功能，遇到这种基础字段命名不统一的场景时，不够灵活，所以有了以下的自定义插件，通过在不同的基础数据库实体映射类中，标记对应的创建日期、更新日期等字段，满足基础字段自动填充功能。

自定义注解说明#

注解	用途
@Created	填充记录创建时间
@Modified	填充记录最后更新时间
@CreatedBy	填充记录创建人
@ModifiedBy	填充记录更新人

使用例子#

1
@Data
2
public abstract class AbstractEntity implements Serializable {
3

4
    @Serial
5
    private static final long serialVersionUID = -6362917626913479219L;
6
    /**
7
     * 创建人
8
     */
9
    @CreatedBy
10
    protected Long createdBy;
11

12
    /**
13
     * 创建时间
14
     */
15
    @Created
16
    protected LocalDateTime createdTime;
17

18
    /**
19
     * 创建日期
20
     */
21
    @ModifiedBy
22
    protected Long updatedBy;
23

24
    /**
25
     * 更新时间
26
     */
27
    @Modified
28
    protected LocalDateTime updatedTime;
29

30
}

底层实现#

通过自定义 mybatis 的插件，拦截 Executor 的 update 方法，找到对应的实体类对象，进行值插入

1
@Intercepts({@Signature(type = Executor.class, method = "update", args = {MappedStatement.class, Object.class})})
2
public class AutoFillInterceptor implements Interceptor {
3

4
    private ConcurrentHashMap<Class, AuditingConfig> auditingConfigMap = new ConcurrentHashMap<>();
5

6
    private CurrentUserExtractor currentUserExtractor;
7

8
    public void setCurrentUserExtractor(CurrentUserExtractor currentUserExtractor) {
9
        this.currentUserExtractor = currentUserExtractor;
10
    }
11

12
    @Override
13
    public Object intercept(Invocation invocation) throws Throwable {
14
        Object[] args = invocation.getArgs();
15
        if (args.length > 1) {
16
            Object arg = args[1];
17
            Class<?> clazz = arg.getClass();
18
            AuditingConfig config = auditingConfigMap.computeIfAbsent(clazz, this::extractConfig);
19
            Field createField = config.getCreateField();
20
            Field modifiedField = config.getModifiedField();
21
            BeanWrapper beanWrapper = null;
22
            MappedStatement mappedStatement = (MappedStatement)args[0];
23
            SqlCommandType sqlCommandType = mappedStatement.getSqlCommandType();
24
            if (createField != null || modifiedField != null) {
25
                beanWrapper = new BeanWrapperImpl(arg);
26
                Date now = new Date();
27
                if (sqlCommandType == SqlCommandType.INSERT) {
28
                    if (createField != null) {
29
                        setAuditingDateValue(beanWrapper, createField, now);
30
                    }
31
                    if (modifiedField != null) {
32
                        setAuditingDateValue(beanWrapper, modifiedField, now);
33
                    }
34
                    // if (config.getVersionFiled() != null) {
35
                    // setDefaultVersion(beanWrapper, config.getVersionFiled());
36
                    // }
37
                } else if (sqlCommandType == SqlCommandType.UPDATE) {
38
                    setAuditingDateValue(beanWrapper, modifiedField, now);
39
                }
40
            }
41
            if (currentUserExtractor != null) {
42
                Field createdByField = config.getCreatedByField();
43
                Field modifiedByField = config.getModifiedByField();
44
                if (createdByField != null && modifiedByField != null) {
45
                    Object user = currentUserExtractor.getCurrentUser();
46
                    if (beanWrapper == null) {
47
                        beanWrapper = new BeanWrapperImpl(arg);
48
                    }
49
                    if (sqlCommandType == SqlCommandType.INSERT) {
50
                        if (createField != null) {
51
                            setAuditingUserValue(beanWrapper, createdByField, user);
52
                        }
53
                        if (modifiedField != null) {
54
                            setAuditingUserValue(beanWrapper, modifiedByField, user);
55
                        }
56
                    } else if (sqlCommandType == SqlCommandType.UPDATE) {
57
                        setAuditingUserValue(beanWrapper, modifiedByField, user);
58
                    }
59
                }
60
            }
61
        }
62
        return invocation.proceed();
63
    }
64

65
    private void setAuditingDateValue(BeanWrapper beanWrapper, Field field, Date value) {
66
        String name = field.getName();
67
        if (field.getType() == Date.class) {
68
            beanWrapper.setPropertyValue(name, value);
69
        }
70
    }
71

72
    private void setAuditingUserValue(BeanWrapper beanWrapper, Field field, Object user) {
73
        String name = field.getName();
74
        if (beanWrapper.getPropertyValue(name) == null) {
75
            beanWrapper.setPropertyValue(name, user);
76
        }
77
    }
78

79
    private AuditingConfig extractConfig(Class<?> clazz) {
80
        AuditingConfig config = new AuditingConfig();
81
        ReflectionUtils.doWithFields(clazz, new ReflectionUtils.FieldCallback() {
82
            @Override
83
            public void doWith(Field field) throws IllegalArgumentException, IllegalAccessException {
84
                Created created = field.getAnnotation(Created.class);
85
                if (created != null) {
86
                    config.setCreateField(field);
87
                } else {
88
                    Modified modified = field.getAnnotation(Modified.class);
89
                    if (modified != null) {
90
                        config.setModifiedField(field);
91
                    }
92
                }
93
                CreatedBy createdBy = field.getAnnotation(CreatedBy.class);
94
                if (createdBy != null) {
95
                    config.setCreatedByField(field);
96
                } else {
97
                    ModifiedBy modifiedBy = field.getAnnotation(ModifiedBy.class);
98
                    if (modifiedBy != null) {
99
                        config.setModifiedByField(field);
100
                    }
101
                }
102
                // Version version = field.getAnnotation(Version.class);
103
                // if (version != null) {
104
                // config.setVersionFiled(field);
105
                // }
106
            }
107
        });
108
        return config;
109
    }
110

111
    @Override
112
    public Object plugin(Object target) {
113
        return Plugin.wrap(target, this);
114
    }
115

116
    @Override
117
    public void setProperties(Properties properties) {
118

119
    }
120

121
    @Data
122
    private static class AuditingConfig {
123

124
        private Field createField;
125

126
        private Field modifiedField;
127

128
        private Field createdByField;
129

130
        private Field modifiedByField;
131

132
        private Field versionFiled;
133
    }
134
}

展开收起

Spring Could 尝鲜记录

Wed, 18 Apr 2018 00:00:00 GMT

原有 1.0 系统架构#

Untitled 60

这是我 17 年刚进公司时的技术架构(我称它为 1.0 版本), 应用服务器在自建机房.

客户端有 PC端, iOS客户端(原生), 安卓客户端(原生), 以及 H5 应用端.

南北流量通过 nginx 反向代理接入

web层是 spring mvc, 每个业务线都单独放到 tomcat 服务器中, 通过 dubbo 协议, 调用后端的领域服务

领域服务层, 都有对应的数据库(schema 分离, 在同一个 mysql 物理机上), 领域服务之间可以相互调用.

在我们项目初始阶段, 该架构足够满足我们业务场景, 但是在日常维护迭代过程中, 也逐渐发现了一些痛点:

每次迭代更新, 我们都是在领域服务层实现对应的业务逻辑, 同时为了将对应的服务通过 http 暴露给客户端, 必须修改 web 层的 spring mvc 应用, 增加对应的 controller 方法, 这些 controller 转发器基本上都是比较无聊的体力活, 每次更新都会涉及到 web 接入层的部署更新;
我们的业务线近 1 年扩充的比较多(理财, 借贷, 聚合支付, 分销等等), 所有这些业务线都要重复实现一次身份认证, 请求日志记录等通用的功能, 这不仅会浪费开发资源, 而且一旦需要修改这部分的代码, 就需要修改很多代码, 这是版本管理和升级维护的噩梦;
技术总监要求我们增加接口测试, 我们的 dubbo 用的是 2.5.4 版本, 并不是当当维护的 dubbox, 不能方便的通过 rest 方式暴露给测试人员;
每次 APP 更新, 都需要对应的原生客户端开发人员进行迭代开发, 然后提交送审, 一般审核时间在 1 至 2 天, 有时候审核驳回又要延长时间;

所以, 在 17 年 11 月开始, 我们开始尝试业界的其他解决方案, 这时候后端微服务架构的 spring cloud 进入了我们的考虑场景(基本上也就这个比较流行), 通过整个团队的努力, 2018年4月, 我们的 2.0 架构版本上线了

2.0 系统架构#

Untitled 61

技术架构的改动有以下方面:

手机客户端, 我们抛弃了原先的 native 开发模式, 引入了 hybird 架构(基于cordova + webview), android 和 ios 小伙伴主要负责开发 jsbridge 功能, 前端小伙伴负责 webview 相关的开发, 同时还要加入了 APP 静态资源动态热更新的设计;
废弃了之前的传统 web 接入层模式, 我们引入了 zuul 网关, 在 zuul 网关中我们统一实现了用户身份鉴权, 请求日志记录, APP 版本更新检测等通用功能(这里我们不同用户会话体系的 zuul 网关部署还是分离的, 比如我们的管理后台 zuul 网关, APP zuul 网关, 开放平台 zuul 网关);
后端领域服务, 我们将原先的 dubbo fat jar 模式, 调整成了基于 spring boot 的spring mvc 应用, 提供 http 服务与网关配合集成, 同时保留原先的 dubbo 服务能力(因为接口实在太多了, 全部移除 dubbo 的话, 工作量太大), 我们默认从 zuul 过来的 http 接口, 都已经通过了鉴权校验, header 中的用户身份识别信息, 可以直接相信;

这套架构, 带来了以下好处:

APP 迭代更新速度大幅提升, 只要不是增加 jsbridge 等功能的改动, 我们不需要提交任何审核, 可以直接发布 webview 热更新资源包;
领域服务迭代时, 顺便就开发完成了对应的 http 接口, 基本上 zuul 网关我们完全都不用做任何改动;
因为新暴露的接口, 都是 http 协议, 并且在开发测试时, 不需要做鉴权(postman 设置对应的用户header), 测试人员可以做自动化接口测试(用的是 python);
因为通过 zuul 和 eureka 做了服务自动发现, 在系统发布时, 不需要运维人员手动去切换 nginx 流量操作;

系统开发实践-Spring Cloud 异常封装及传递处理

Wed, 08 Nov 2017 00:00:00 GMT

异常体系定义#

Untitled 53

异常编码的设计使用#

这里采用了精简异常代码设计的形式

ServiceException业务异常, 异常编码默认使用BIZ_ERROR, 异常文字提示根据实际情况抛出, 例如

1
private void checkParameter(TApprovalResult result) {
2
    if (Objects.isNull(result)) {
3
        throw new ServiceException("审批结果不能为null");
4
    }
5
    try {
6
        CheckParameterUtil.checkParameter(result);
7
    } catch (Exception e) {
8
        ServiceException exception = new ServiceException("参数校验异常");
9
        exception.addSuppressed(e);
10
        throw exception;
11
    }
12
}

系统设计上, 会针对业务异常, 日志记录上做 WARN 级别设置

系统异常默认使用SYSTEM_ERROR, 提示系统忙, 请稍后再试, 默认非 ServiceException 的异常, 都归属于系统异常, 需要做告警提示并及时修复

异常类型	默认异常 code	默认异常 message	日志记录级别
`ServiceException`	`BIZ_ERROR`	无	WARN
`SystemException`	`SYSTEM_ERROR`	`系统忙, 请稍后再试`	ERROR

Spring Boot 异常捕捉扩展#

spring boot 的 ErrorMvcAutoConfiguration 有参考实现, 我们对其进行扩展, 个性化输出内容

1
@Order(Ordered.HIGHEST_PRECEDENCE)
2
@Slf4j
3
public class CustomErrorAttributes implements ErrorAttributes, HandlerExceptionResolver, Ordered {
4

5
    private static final String ERROR_ATTRIBUTE = CustomErrorAttributes.class.getName() + ".ERROR";
6

7
    private String errorPath;
8

9
    public CustomErrorAttributes(String errorPath) {
10
        this.errorPath = errorPath;
11
    }
12

13
    @Override
14
    public int getOrder() {
15
        return Ordered.HIGHEST_PRECEDENCE;
16
    }
17

18
    @Override
19
    public ModelAndView resolveException(HttpServletRequest request, HttpServletResponse response, Object handler,
20
        Exception ex) {
21
        // 在发生异常时，传递异常类型
22
        response.addHeader(HttpHeaderConstants.HEADER_EXCEPTION_NAME, ex.getClass().getName());
23
        storeErrorAttributes(request, ex);
24
        // 如果是业务异常, 日志级别为 warn
25
        if (ex instanceof ServiceException) {
26
            log.warn(ex.getMessage(), ex);
27
            request.setAttribute(ErrorAttributeConstans.ATTR_STATUS, 500);
28
            request.setAttribute(ErrorAttributeConstans.SERVLET_ERROR_STATUS_CODE, 500);
29
            return new ModelAndView(this.errorPath);
30
        }
31
        return null;
32
    }
33

34
    private void storeErrorAttributes(HttpServletRequest request, Exception ex) {
35
        request.setAttribute(ERROR_ATTRIBUTE, ex);
36
    }
37

38
    @Override
39
    public Map<String, Object> getErrorAttributes(WebRequest webRequest, boolean includeStackTrace) {
40
        Map<String, Object> errorAttributes = new LinkedHashMap<String, Object>();
41
        errorAttributes.put(ErrorAttributeConstans.ATTR_TIMESTAMP, new Date());
42
        addStatus(errorAttributes, webRequest);
43
        addErrorCode(errorAttributes, webRequest, getError(webRequest));
44
        addErrorDetails(errorAttributes, webRequest, includeStackTrace);
45
        addPath(errorAttributes, webRequest);
46
        return errorAttributes;
47
    }
48

49
    private void addStatus(Map<String, Object> errorAttributes, RequestAttributes requestAttributes) {
50
        Integer status = getAttribute(requestAttributes, ErrorAttributeConstans.SERVLET_ERROR_STATUS_CODE);
51
        if (status == null) {
52
            errorAttributes.put(ErrorAttributeConstans.ATTR_STATUS, 999);
53
            errorAttributes.put(ErrorAttributeConstans.ATTR_ERROR, "None");
54
            return;
55
        }
56
        errorAttributes.put(ErrorAttributeConstans.ATTR_STATUS, status);
57
        try {
58
            errorAttributes.put(ErrorAttributeConstans.ATTR_ERROR, HttpStatus.valueOf(status).getReasonPhrase());
59
        } catch (Exception ex) {
60
            // Unable to obtain a reason
61
            errorAttributes.put(ErrorAttributeConstans.ATTR_ERROR, "Http Status " + status);
62
        }
63
    }
64

65
    private void addErrorCode(Map<String, Object> errorAttributes, RequestAttributes requestAttributes,
66
        Throwable error) {
67
        String code = getAttribute(requestAttributes, ErrorCoded.ATTR_ERROR_CODE);
68
        if (code == null && error instanceof ErrorCoded) {
69
            ErrorCoded errorCoded = (ErrorCoded)error;
70
            code = errorCoded.getErrorCode();
71
        }
72
        if (StringUtils.hasText(code)) {
73
            errorAttributes.put(ErrorAttributeConstans.ATTR_ERROR_CODE, code);
74
        }
75
    }
76

77
    private void addErrorDetails(Map<String, Object> errorAttributes, WebRequest webRequest,
78
        boolean includeStackTrace) {
79
        Throwable error = getError(webRequest);
80
        if (error != null) {
81
            while (error instanceof ServletException && error.getCause() != null) {
82
                error = error.getCause();
83
            }
84
            errorAttributes.put(ErrorAttributeConstans.ATTR_EXCEPTION, error.getClass().getName());
85
            addErrorMessage(errorAttributes, error);
86
            if (includeStackTrace) {
87
                addStackTrace(errorAttributes, error);
88
            }
89
        }
90
        Object message = getAttribute(webRequest, ErrorAttributeConstans.ATTR_MESSAGE);
91
        if (message == null) {
92
            message = getAttribute(webRequest, ErrorAttributeConstans.SERVLET_ERROR_MESSAGE);
93
        }
94
        if ((!StringUtils.isEmpty(message)) && !(error instanceof BindingResult)) {
95
            errorAttributes.put(ErrorAttributeConstans.ATTR_MESSAGE,
96
                StringUtils.isEmpty(message) ? "No message available" : message);
97
        }
98
    }
99

100
    private void addErrorMessage(Map<String, Object> errorAttributes, Throwable error) {
101
        BindingResult result = extractBindingResult(error);
102
        if (result == null) {
103
            if (error instanceof ErrorCodedException) {
104
                errorAttributes.put(ErrorAttributeConstans.ATTR_MESSAGE, error.getMessage());
105
                return;
106
            }
107
            errorAttributes.put(ErrorAttributeConstans.ATTR_MESSAGE, "系统忙, 请稍后再试");
108
            errorAttributes.put(ErrorAttributeConstans.ATTR_ERROR_MESSAGE, error.getMessage());
109
            return;
110
        }
111
        if (result.getErrorCount() > 0) {
112
            errorAttributes.put(ErrorAttributeConstans.ATTR_ERROR, result.getAllErrors());
113
            errorAttributes.put(ErrorAttributeConstans.ATTR_MESSAGE, getBindingResultMessage(result));
114
        } else {
115
            errorAttributes.put(ErrorAttributeConstans.ATTR_MESSAGE, "参数绑定失败");
116
        }
117
    }
118

119
    private String getBindingResultMessage(BindingResult result) {
120
        StringBuilder sb = new StringBuilder();
121
        for (ObjectError objectError : result.getAllErrors()) {
122
            if (objectError instanceof FieldError) {
123
                sb = appendFieldError(sb, (FieldError)objectError);
124
            } else {
125
                sb = appendObjectError(sb, objectError);
126
            }
127
            sb.append("\n");
128
        }
129
        return sb.toString();
130
    }
131

132
    private StringBuilder appendFieldError(StringBuilder sb, FieldError fieldError) {
133
        return sb.append(fieldError.getObjectName()).append(".").append(fieldError.getField()).append(":")
134
            .append(fieldError.getDefaultMessage());
135
    }
136

137
    private StringBuilder appendObjectError(StringBuilder sb, ObjectError objectError) {
138
        return sb.append(objectError.getObjectName()).append(":").append(objectError.getDefaultMessage());
139
    }
140

141
    private BindingResult extractBindingResult(Throwable error) {
142
        if (error instanceof BindingResult) {
143
            return (BindingResult)error;
144
        }
145
        if (error instanceof MethodArgumentNotValidException) {
146
            return ((MethodArgumentNotValidException)error).getBindingResult();
147
        }
148
        return null;
149
    }
150

151
    private void addStackTrace(Map<String, Object> errorAttributes, Throwable error) {
152
        StringWriter stackTrace = new StringWriter();
153
        error.printStackTrace(new PrintWriter(stackTrace));
154
        stackTrace.flush();
155
        errorAttributes.put(ErrorAttributeConstans.ATTR_TRACE, stackTrace.toString());
156
    }
157

158
    private void addPath(Map<String, Object> errorAttributes, RequestAttributes requestAttributes) {
159
        String path = getAttribute(requestAttributes, ErrorAttributeConstans.SERVLET_ERROR_REQUEST_URI);
160
        if (path != null) {
161
            errorAttributes.put(ErrorAttributeConstans.ATTR_PATH, path);
162
        }
163
    }
164

165
    @Override
166
    public Throwable getError(WebRequest webRequest) {
167
        Throwable exception = getAttribute(webRequest, ERROR_ATTRIBUTE);
168
        if (exception == null) {
169
            exception = getAttribute(webRequest, ErrorAttributeConstans.SERVLET_ERROR_EXCEPTION);
170
        }
171
        return exception;
172
    }
173

174
    @SuppressWarnings("unchecked")
175
    private <T> T getAttribute(RequestAttributes requestAttributes, String name) {
176
        return (T)requestAttributes.getAttribute(name, RequestAttributes.SCOPE_REQUEST);
177
    }
178
}

展开收起

同时配合 spring boot 的 AutoConfiguration 机制, 让我们的 CustomErrorAttributes 生效

1
@Configuration
2
@ConditionalOnWebApplication
3
@ConditionalOnClass({Servlet.class, DispatcherServlet.class})
4
// Load before the main WebMvcAutoConfiguration so that the error View is available
5
@AutoConfigureBefore(ErrorMvcAutoConfiguration.class)
6
public class CustomErrorMvcAutoConfiguration {
7

8
    @Value("${server.error.path:${error.path:/error}}")
9
    private String errorPath;
10

11
    @Bean
12
    @ConditionalOnMissingBean(value = ErrorAttributes.class, search = SearchStrategy.CURRENT)
13
    public CustomErrorAttributes customErrorAttributes() {
14
        return new CustomErrorAttributes(this.errorPath);
15
    }
16
}

Spring Cloud 的 openfeign 异常传递改造#

spring cloud 微服务, 我们用的 openfeign 声明式调用, 还需要对异常传递进行扩展, 实现在调用方调用服务时, 能够获取服务提供方抛出的异常

1
@Slf4j
2
public class ErrorCodedExceptionErrorDecoder implements ErrorDecoder {
3

4
    private static final String SERVICE_EXCEPTION_NAME = ServiceException.class.getName();
5

6
    @Override
7
    public Exception decode(String methodKey, Response response) {
8
        int status = response.status();
9

10
        String exceptionName = null;
11

12
        Collection<String> list = response.headers().get(HttpHeaderConstants.HEADER_EXCEPTION_NAME);
13
        if (!CollectionUtils.isEmpty(list)) {
14
            for (String s : list) {
15
                if (StringUtils.hasText(s)) {
16
                    exceptionName = s;
17
                    log.info("捕捉到 feign 异常类型: {}", exceptionName);
18
                    break;
19
                }
20
            }
21
        }
22

23
        if (404 == status) { // 服务 url 不正确, 抛出特殊异常
24
            throw new ServiceNotFoundException(
25
                String.format("Service %s not found for %s", response.request().url(), methodKey));
26
        }
27

28
        String errorCode = null;
29
        String message = null;
30
        HttpStatus httpStatus = HttpStatus.valueOf(status);
31
        if (httpStatus.is4xxClientError()) {
32
            errorCode = String.valueOf(status);
33
            message = errorCode + ": " + httpStatus.getReasonPhrase();
34
        } else if (httpStatus.is5xxServerError()) {
35
            try {
36
                ErrorBody errorBody = JacksonUtils.jsonToObject(response.body().asInputStream(), ErrorBody.class);
37
                errorCode = errorBody.getErrorCode();
38
                if (errorBody.getErrorMessage() != null) {
39
                    message = errorBody.getErrorMessage();
40
                } else {
41
                    message = errorBody.getMessage();
42
                }
43
            } catch (IOException e) {
44
                log.error("读取二进制流失败", e);
45
            }
46
        }
47

48
        if (SERVICE_EXCEPTION_NAME.equalsIgnoreCase(exceptionName)) {
49
            return new ServiceException(message);
50
        } else {
51
            return new SystemException(errorCode, message);
52
        }
53
    }
54

55
    @Data
56
    private static class ErrorBody implements ErrorCoded {
57

58
        private String errorCode;
59

60
        private String message;
61

62
        private String errorMessage;
63
    }
64
}

展开收起

系统开发实践-前后端交互规范

Wed, 01 Nov 2017 00:00:00 GMT

经过团队内部讨论并达成一致, 前后端系统交互时, 我们遵循以下规范

后台响应报文规范#

1
{
2
  "code": 200, // 200-正常返回, 401-未登录, 403-权限不足, 500-业务异常(或者服务器异常)
3
  "data": {}, // 响应的数据
4
  "errorCode": "", // 如果失败时, 会有对应的异常代码
5
  "message": "" // 用于辅助信息提示
6
}

前端请求规范#

前端在向后端发送数据请求时, 必须在 header 中带入识别用户身份信息的 token, 请求报文上支持表单格式和 json 格式, http 请求方法上, 原则上查询请求使用 GET, 提交修改数据统一使用 POST

请求 header 头设计说明

字段名	说明
token	用户会话标识
user-id	用户id, 用于 nginx 日志采集
ga-xxx	一些特定的信息收集字段, 例如经纬度, 手机型号等等
timestamp	请求时间搓
app-version	app版本号

分页数据请求规范#

1
{
2
  "pageNum": 1, // 分页页码
3
  "pageSize": 200, // 分页大小
4
  "condition1": "aaa", // 搜索条件1
5
  "condition2": "bbb", // 搜索条件2
6
  "condition3": "ccc", // 搜索条件3
7
  "sort": {
8
    "field": "", // 排序字段
9
    "order": "asc" // 升序降序
10
  }
11
}

分页数据响应规范#

1
{
2
  "code": 200,
3
  "data": {
4
    "pageNum": 1, // 分页页码
5
    "pageSize": 200, // 分页大小
6
    "total": 1000, // 数据总量
7
    "content": [{},{}] // 分页数据
8
}

Data Type	Storage Required
CHAR(``*M*``)	The compact family of InnoDB row formats optimize storage for variable-length character sets. See COMPACT Row Format Storage Characteristics. Otherwise, `*M` × `w` bytes, <= ``M`` <= 255, where `w*` is the number of bytes required for the maximum-length character in the character set.
BINARY(``*M*``)	`*M` bytes, 0 <= ``M*`` <= 255
VARCHAR(``*M``), VARBINARY(``M*``)	`*L` + 1 bytes if column values require 0 − 255 bytes, `L*` + 2 bytes if values may require more than 255 bytes
`TINYBLOB`, `TINYTEXT`	`*L` + 1 bytes, where `L*` < 28
`BLOB`, `TEXT`	`*L` + 2 bytes, where `L*` < 216
`MEDIUMBLOB`, `MEDIUMTEXT`	`*L` + 3 bytes, where `L*` < 224
`LONGBLOB`, `LONGTEXT`	`*L` + 4 bytes, where `L*` < 232
ENUM('``*value1``','``value2*``',...)	1 or 2 bytes, depending on the number of enumeration values (65,535 values maximum)
SET('``*value1``','``value2*``',...)	1, 2, 3, 4, or 8 bytes, depending on the number of set members (64 members maximum)

虾米的个人笔记

在 PVE 上给软路由虚拟机加一个网络 Watchdog

设计目标#

判断链路#

配置文件#

Watchdog 脚本#

systemd service 和 timer#

验证#

触发流程#

临时停用#

生产环境建议#

命令行自动登录 JumpServer：SSH、TOTP 与密码提示的自动化

前置条件#

命令入口#

zsh 负责读取凭据#

避免 Bitwarden 模糊匹配#

从 Bitwarden 解析 TOTP seed#

expect 只处理 SSH 交互#

password 的处理#

常见问题#

too many authentication failures#

Code prompt 匹配不到#

发送验证码后会话退出#

安全边界#

最终使用#

git 个人开发工作流

日常工作流#

使用 docker-atrust 连入深信服

网络拓扑#

docker-atrust 配置#

通过 vnc 登录#

mihomo 配置#

openwrt 路由配置#

📎 参考文章#

mihomo 使用

mac 客户端#

常用 mihomo 配置文件#

解析过程#

Grafana 配置手册

告警配置#

📎 参考文章#

自建 headscale

headscale 安装配置#

常用命令#

也可以通过 headscale-ui 设置 preauth#

阿里云 tailscale 设置#

macos tailscale 设置#

iStoreOS tailscale 设置#

📎 参考文章#

Playwright 使用心得

参考资料#

PV、UV 统计方案

业务背景#

数据收集链路#

PV 统计实现方案#

UV 统计实现方案#

📎 参考文章#

Spring Cloud Gateway 流量复制

功能逻辑#

配置示例#

功能特色#

核心代码#

通过 roaringbitmap 判断百万级数据是否存在

问题描述#

纯数据库方案#

bloom filter方案#

bitmap 方案#

roaringbitmap 方案#

对比总结#

📎 参考文章#

iStoreOS + WireGuard 客户端踩坑

网络拓扑#

遇到的问题#

问题排查#

解决方案#

什么是 MSS？#

MSS 钳制的作用#

实现方式#

常见应用场景#

WireGuard 安装#