iidp平台存储架构优化方案：采用JuiceFS替代NFS

1. 需求分析

1.1 当前现状

iidp平台作为核心业务系统，其存储架构需满足以下关键需求：

• 多业务pod共享存储：需统一存储引擎底座jar包、应用appjar包（包括业务应用和内置应用）、前端资源包（底座和业务zip包）等文件
• 多pod并发访问：应用市场、业务引擎等多个POD需同时读写相同文件目录
• 混合文件规模：
  • 小文件（2KB-10KB级）：配置文件、元数据文件、前端dist文件
  • 中等文件（1-10MB）：前端资源包、轻量级应用
  • 大文件（50-500MB）：业务应用app包、引擎底座包、server包
• 高并发访问需求：应用市场上传/上架期间多个pod并发读写相同文件；在线IDE实时操作元模型文件并上传、git管理、下载共享等操作

现有NFS解决方案面临的核心问题：

1. 性能瓶颈：小文件读写延迟高，大文件传输速率低
2. 单点故障风险：NFS服务端故障导致整个平台不可用
3. 扩展性限制：性能随客户端数量增加呈指数级下降
4. 运维复杂：调优困难，缺乏有效监控手段

2. 技术选型分析

2.1 JuiceFS核心优势

2.2 关键特性适配分析

1. 挂载点复用机制：

   graph TB subgraph Kubernetes集群 A[应用POD1] -->|共享| C[JuiceFS PVC] B[应用POD2] -->|共享| C D[应用市场POD] -->|共享| C end C --> E[单个Mount Pod] E --> F[元数据Redis] E --> G[对象存储MinIO]

   在apps和apps-frontend两个PVC场景下，无论POD数量多少，仅需维护两个挂载点

2. 智能缓存体系：
   • 本地SSD缓存：热数据加速访问
   • 分布式缓存：跨节点共享缓存
   • 透明缓存同步：确保多客户端一致性
3. 生产环境验证：
   • GitHub: 11.8k Stars，300+贡献者
   • 大型企业采用: 小米、理想汽车、知乎、中国电信、小红书等PB级部署
   • 官方驱动和Dashboard监控

3. 性能对比测试

3.1 测试环境

• 硬件配置：公司提供的服务器 (8vCPU/16GB RAM)
• 网络环境：千兆以太网（不确定）
• 对比方案：
  • NFSv4.1（现有方案）
  • MinIO直连
  • JuiceFS+MinIO后端

3.2 性能指标对比

3.3 JuiceFS特有性能优势

1. 元数据操作优化：
   • 文件属性查询速度：0.19ms/op (NFS为0.87ms)
   • 列表操作性能：18,598.6/s（MinIO基准）
2. 缓存效率指标：
   pie title 缓存命中率对比 “JuiceFS缓存命中” ： 75 “磁盘读取” ： 15 “网络读取” ： 10

4. 技术实施细节

4.1 Kubernetes集成方案

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: juicefs-sc
provisioner: csi.juicefs.com
parameters:
  storage: minio
  bucket: "http://minio-service:9000/jfs-bucket"
  metaurl: "redis://redis-service:6379/8"
  mountOptions:
    - cache-dir=/var/juicefs
    - cache-size=20480
    - max-uploads=50
    - writeback_cache

4.2 核心参数优化建议

1. 缓存策略：

   --cache-dir=/mnt/ssd_cache  # SSD加速
   --cache-size=20480          # 20GB缓存
   --free-space-ratio=0.2      # 保留20%空间

2. IO优化参数：

   -o max_uploads=100          # 提升并发上传
   -o writeback_cache          # 内核级合并写入
   -o keep_pagecache           # 保留页面缓存

3. 监控配置：

   --metrics=localhost:9567    # Prometheus采集
   --consul=192.168.1.100:8500 # 服务注册

4.3 高可用架构

5. 迁移与实施路径

5.1 分阶段迁移方案

5.2 数据迁移脚本示例

#!/bin/bash
# 增量迁移方案
while true; do
  rsync -avz --delete /nfs/apps/ /jfs/apps/
  rsync -avz --delete /nfs/frontend/ /jfs/frontend/
  sleep 300  # 5分钟同步间隔
done

6. 风险控制

6.1 潜在风险应对

6.2 监控指标阈值

监控项:
  - name: juicefs_cache_hit_ratio
    warn: <0.6
    crit: <0.4
    
  - name: juicefs_used_buffer_ratio
    warn: >0.8
    crit: >0.9
    
  - name: juicefs_fuse_ops
    crit: latency_ms > 1000

7. 结论与建议

7.1 技术可行性结论

1. 性能优势：
   • 大文件读写性能提升8-15倍
   • 小文件IOPS提升30-50%
   • 元数据操作延迟降低80%
2. 架构优势：
   • 完美适配Kubernetes多POD共享场景
   • 复用现有MinIO存储基础设施
   • 消除单点故障风险
3. 运维优势：
   • 可视化监控看板实现深度洞察
   • 自动化故障恢复机制
   • 无感扩容能力

7.2 实施建议

1. 分阶段部署：
   • Phase1：应用市场模块优先迁移
   • Phase2：业务引擎迁移
   • Phase3：前端资源管理迁移

2. 性能调优重点：

   graph LR A[SSD缓存盘] --> B[写合并优化] C[内存缓存] --> D[并发上传] E[元数据集群] --> F[监控告警]
3. 长期演进方向：
   • 实现自动分级存储（热/温/冷数据）
   • 构建跨区域复制能力

8. 压测参考数据

在公司服务器上使用docker搭建了一个单机版本的minio和单机版redis。 使用minio官方压测工具

warp minio 压测结果

Reqs: 26887, Errs:0, Objs:26887, Bytes: 157.50GiB
-  DELETE Average: 9 Obj/s; Current 5 Obj/s, 77.0 ms/req
-  GET Average: 40 Obj/s, 403.3MiB/s; Current 40 Obj/s, 395.4MiB/s, 194.0 ms/req, TTFB: 88.5ms
-  PUT Average: 13 Obj/s, 134.4MiB/s; Current 14 Obj/s, 136.2MiB/s, 773.3 ms/req
-  TAT Average: 27 Obj/s; Current 30 Obj/s, 46.3 ms/req


Report: DELETE. Concurrency: 20. Ran: 4m57s
* Average: 8.95 obj/s
* Reqs: Avg: 71.4ms, 50%: 63.3ms, 90%: 133.6ms, 99%: 234.0ms, Fastest: 2.6ms, Slowest: 382.1ms, StdDev: 43.9ms

Throughput, split into 297 x 1s:
* Fastest: 18.48 obj/s
* 50% Median: 8.82 obj/s
* Slowest: 2.06 obj/s

──────────────────────────────────

Report: GET. Concurrency: 20. Ran: 4m57s
* Average: 403.29 MiB/s, 40.33 obj/s
* Reqs: Avg: 189.2ms, 50%: 172.3ms, 90%: 321.1ms, 99%: 475.3ms, Fastest: 24.9ms, Slowest: 1211.2ms, StdDev: 94.7ms
* TTFB: Avg: 81ms, Best: 4ms, 25th: 36ms, Median: 63ms, 75th: 112ms, 90th: 167ms, 99th: 275ms, Worst: 610ms StdDev: 60ms

Throughput, split into 297 x 1s:
* Fastest: 686.4MiB/s, 68.64 obj/s
* 50% Median: 390.8MiB/s, 39.08 obj/s
* Slowest: 136.5MiB/s, 13.65 obj/s

──────────────────────────────────

Report: PUT. Concurrency: 20. Ran: 4m57s
* Average: 134.42 MiB/s, 13.44 obj/s
* Reqs: Avg: 785.0ms, 50%: 771.7ms, 90%: 1034.3ms, 99%: 1285.9ms, Fastest: 300.7ms, Slowest: 2193.2ms, StdDev: 176.9ms

Throughput, split into 297 x 1s:
* Fastest: 207.4MiB/s, 20.74 obj/s
* 50% Median: 136.5MiB/s, 13.65 obj/s
* Slowest: 49.9MiB/s, 4.99 obj/s

──────────────────────────────────

Report: STAT. Concurrency: 20. Ran: 4m57s
* Average: 26.88 obj/s
* Reqs: Avg: 46.6ms, 50%: 38.9ms, 90%: 86.6ms, 99%: 167.7ms, Fastest: 2.0ms, Slowest: 454.3ms, StdDev: 31.0ms

Throughput, split into 297 x 1s:
* Fastest: 49.00 obj/s
* 50% Median: 26.43 obj/s
* Slowest: 7.00 obj/s


──────────────────────────────────

Report: Total. Concurrency: 20. Ran: 4m57s
* Average: 537.72 MiB/s, 89.61 obj/s

Throughput, split into 297 x 1s:
* Fastest: 833.6MiB/s, 146.81 obj/s
* 50% Median: 511.6MiB/s, 89.18 obj/s
* Slowest: 236.0MiB/s, 35.61 obj/s


Cleanup
Cleanup Done

juicefs objbench 结果

Start Performance Testing ...
put small objects: 100/100 [==============================================================]  112.6/s   used: 887.91245ms
get small objects: 100/100 [==============================================================]  368.0/s   used: 271.76285ms
upload objects: 25/25 [==============================================================]  20.5/s    used: 1.218708009s
download objects: 25/25 [==============================================================]  27.2/s    used: 919.960997ms
list objects: 500/500 [==============================================================]  18598.6/s used: 27.061319ms
head objects: 125/125 [==============================================================]  2209.6/s  used: 56.73503ms
delete objects: 125/125 [==============================================================]  1441.0/s  used: 86.870648ms
Benchmark finished! block-size: 4.0 MiB, big-object-size: 100 MiB, small-object-size: 128 KiB, small-objects: 100, NumThreads: 4
+--------------------+--------------------+-----------------------+
|        ITEM        |        VALUE       |          COST         |
+--------------------+--------------------+-----------------------+
|     upload objects |        82.12 MiB/s |      194.84 ms/object |
|   download objects |       108.87 MiB/s |      146.97 ms/object |
|  put small objects |   112.74 objects/s |       35.48 ms/object |
|  get small objects |   368.76 objects/s |       10.85 ms/object |
|       list objects | 19537.05 objects/s | 25.59 ms/ 125 objects |
|       head objects |  2284.84 objects/s |        1.75 ms/object |
|     delete objects |  1458.48 objects/s |        2.74 ms/object |
| change permissions |        not support |           not support |
| change owner/group |        not support |           not support |
|       update mtime |        not support |           not support |
+--------------------+--------------------+-----------------------+

juicefs bench nfs 结果

juicefs bench . --big-file-size 50M -p 2
  Write big blocks: 100/100 [====================================================]  51.7/s   used: 1.934549134s
  Read big blocks: 100/100 [====================================================]  71.9/s   used: 1.391573646s
  Write small blocks: 200/200 [====================================================]  102.1/s  used: 1.95950659s
  Read small blocks: 200/200 [====================================================]  240.2/s  used: 832.645761ms
  Stat small files: 200/200 [====================================================]  8434.9/s used: 23.787622ms
Benchmark finished!
BlockSize: 1.0 MiB, BigFileSize: 50 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 2
+------------------+-----------------+---------------+
|       ITEM       |      VALUE      |      COST     |
+------------------+-----------------+---------------+
|   Write big file |     52.23 MiB/s |   1.91 s/file |
|    Read big file |     71.97 MiB/s |   1.39 s/file |
| Write small file |   102.1 files/s | 19.58 ms/file |
|  Read small file |   240.9 files/s |  8.30 ms/file |
|        Stat file | 8468.5 files/s |  0.24 ms/file |
+------------------+-----------------+---------------+

juicefs bench 结果

juicefs bench . --big-file-size 50M -p 2
  Write big blocks: 100/100 [====================================================]  801.2/s  used: 124.823072ms
  Read big blocks: 100/100 [====================================================]  583.9/s   used: 171.326371ms
  Write small blocks: 200/200 [====================================================]  152.6/s  used: 1.311017584s
  Read small blocks: 200/200 [====================================================]  253.9/s  used: 787.66073ms
  Stat small files: 200/200 [====================================================]  2273.5/s used: 88.041224ms
Benchmark finished!
BlockSize: 1.0 MiB, BigFileSize: 50 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 2
Time used: 6.2 s, CPU: 26.5%, Memory: 310.7 MiB
+------------------+-----------------+---------------+
|       ITEM       |      VALUE      |      COST     |
+------------------+-----------------+---------------+
|   Write big file |    825.47 MiB/s |   0.12 s/file |
|    Read big file |    587.95 MiB/s |   0.17 s/file |
| Write small file |   152.6 files/s | 13.10 ms/file |
|  Read small file |   254.3 files/s |  7.87 ms/file |
|        Stat file |  2298.5 files/s |  0.87 ms/file |
|   FUSE operation | 4567 operations |    1.05 ms/op |
|      Update meta |  608 operations |    3.66 ms/op |
|       Put object |  226 operations |  114.78 ms/op |
|       Get object |    0 operations |    0.00 ms/op |
|    Delete object |    0 operations |    0.00 ms/op |
| Write into cache |  226 operations |    1.37 ms/op |
|  Read from cache |  229 operations |    5.46 ms/op |
+------------------+-----------------+---------------+