1，需求分析
整个平台如果在单机版的情况下，因为所有的数据都在同一个进程或者pod中， 那么是不存在分布式数据同步的问题，但是如果是在分布式部署情况下，由于多副本，以及app和meta之间的关联性 （比如说app1和app2都包含meta1，那么meta1变动，需要同步给其他所有关联的app1和app2） 那么此时就需要分布式同步的功能，必须保证所有的pod和所有的app都能获取到最新的更新， 这样才能保证对外是统一的一致性的表现。
在已有的分布式环境，不同的app安装在独立的容器中，且没有多副本，所以说本质上跟单机版是一样的，只是对于整个软件系统来说它通过分片的方式， 将各个应用隔离开来，各自的内容数据也只用于各自的容器中。
但是对于多租户app来说，它在分布式环境中是必须所有的app都需要的，那么自然涉及到不同容器中的内存同步。
同理，对于多副本来说更是需要内存的同步。
2，方案选型
-
基于redis stream的时间发布订阅同步
这是早期的版本，由于各种问题（尤其是并发写）现在已经废弃。 -
基于 Hazelcast
Hazelcast 是由Hazelcast公司开发的一款开源的分布式内存级别的缓存数据库，可以为基于JVM环境运行的各种应用提供分布式集群和分布式缓存服务。-
我们需要什么？
通过在实际的项目中，我们的模型、菜单、策略等信息都是保持在一个map中，所以我们需要的是一个分布式map， 在一个节点操作，会自动同步给所有的节点（对于可分片的数据结构，比如map，实际实现不是全部同步，而是基于分片的形式，但是在用户看来跟同步效果是一样的）。另外，我们对于一些websocket消息，还需要一个分布式queue来统一分发。 -
Hazelcast 能提供什么？
它能提供分布式map，分布式queue。而且它实现了java标准库中Map的所有接口，对于使用者来说之前怎么用map的，现在依然可以使用， 不需要修改任何方法，这对已有的代码是非常友好的。另外，它提供了嵌入式的使用方式，这是我认为最有价值的地方，分布式缓存在市面上已经有很多了， 比如 ignite redis 等等，但是它们都不支持嵌入式的使用方式，需要额外独立的部署第三方服务，这对一些使用场景来说显得有点复杂了。 此时，Hazelcast的嵌入式使用体验就非常友好，只需要引入一个jar包即可，每个节点即是客户端也是服务端，自动服务注册和发现（比如多播，比如基于k8s等），自动组网， 对于用户来说非常方便。
-
3，功能实现
-
基于Hazelcast 提供一个distributed map
public class DMap { private static boolean distributed = WebMode.of(ConfigConstant.ENGINE_RUN_MODE) != WebMode.DISTRIBUTED; static private String mapName = "hazelcast-map"; static private HazelcastInstance instance; static { if (distributed) { SystemInfo systemInfo = new SystemInfo(); String id = IdGenerator.getDeviceId(systemInfo); String instanceName = "hazelInstance_" + id; String clusterName = "iidp-hazelcast-cluster"; Config config = new Config(); config.setClusterName(clusterName); config.setInstanceName(instanceName); config.getJetConfig().setEnabled(false); config.setIntegrityCheckerConfig(new IntegrityCheckerConfig().setEnabled(false)); config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false); config.getNetworkConfig().getJoin().getKubernetesConfig().setEnabled(true); // map config // config.getMapConfig(). instance = Hazelcast.newHazelcastInstance(config); } } // 如果用户想直接hazelcast提供的方法可以在获取instance后直接使用 static public HazelcastInstance getInstance() { return instance; } static public <K, V> Map<K, V> getMap() { if (!distributed) { return new HashMap<>(); } return instance.getMap(mapName); } static public <K, V> Map<K, V> getMap(String name) { if (!distributed) { return new HashMap<>(); } return instance.getMap(name); } } -
在业务代码中，将需要分布式map的地方替换成DMap 由于DMap实现了标准库中的Map接口，就可以将DMap当作普通的map使用了。
但是实际使用场景中，比较复杂，主要体现在序列化问题，在内存中可以通过引用的方式修改map 的value等问题，需要业务自行调整和适配。private Map<String, MenuMeta> menuMetaMap = DMap.getMap();
4，性能测试
1、简单的map put 操作
mp.Put(ctx, "foo", "bar") 结果：
- 1个并发，共10个请求
time="2023-11-29T15:15:39+08:00" level=info msg="total: 10 concurrency: 1 requests per client: 10"
time="2023-11-29T15:15:39+08:00" level=info msg="took 27 ms for 10 requests"
time="2023-11-29T15:15:39+08:00" level=info msg="sent requests : 10"
time="2023-11-29T15:15:39+08:00" level=info msg="received requests : 10"
time="2023-11-29T15:15:39+08:00" level=info msg="received requests_OK : 10"
time="2023-11-29T15:15:39+08:00" level=info msg="throughput (TPS) : 370"
time="2023-11-29T15:15:39+08:00" level=info msg="mean: 2772220 ns, median: 1824400 ns, max: 9850800 ns, min: 1627700 ns, p99.9: 6775050 ns"
time="2023-11-29T15:15:39+08:00" level=info msg="mean: 2 ms, median: 1 ms, max: 9 ms, min: 1 ms, p99.9: 6 ms"
- 10个并发，共100个请求
time="2023-11-29T15:17:57+08:00" level=info msg="total: 100 concurrency: 10 requests per client: 10"
time="2023-11-29T15:17:57+08:00" level=info msg="took 84 ms for 100 requests"
time="2023-11-29T15:17:57+08:00" level=info msg="sent requests : 100"
time="2023-11-29T15:17:57+08:00" level=info msg="received requests : 100"
time="2023-11-29T15:17:57+08:00" level=info msg="received requests_OK : 100"
time="2023-11-29T15:17:57+08:00" level=info msg="throughput (TPS) : 1190"
time="2023-11-29T15:17:57+08:00" level=info msg="mean: 8230900 ns, median: 5983800 ns, max: 28240200 ns, min: 1665600 ns, p99.9: 28240200 ns"
time="2023-11-29T15:17:57+08:00" level=info msg="mean: 8 ms, median: 5 ms, max: 28 ms, min: 1 ms, p99.9: 28 ms"
- 10个并发，共1000个请求
time="2023-11-29T15:19:25+08:00" level=info msg="total: 1000 concurrency: 10 requests per client: 100"
time="2023-11-29T15:19:26+08:00" level=info msg="took 836 ms for 1000 requests"
time="2023-11-29T15:19:26+08:00" level=info msg="sent requests : 1000"
time="2023-11-29T15:19:26+08:00" level=info msg="received requests : 1000"
time="2023-11-29T15:19:26+08:00" level=info msg="received requests_OK : 1000"
time="2023-11-29T15:19:26+08:00" level=info msg="throughput (TPS) : 1196"
time="2023-11-29T15:19:26+08:00" level=info msg="mean: 8338478 ns, median: 6197100 ns, max: 25382100 ns, min: 1508800 ns, p99.9: 25382100 ns"
time="2023-11-29T15:19:26+08:00" level=info msg="mean: 8 ms, median: 6 ms, max: 25 ms, min: 1 ms, p99.9: 25 ms"
- 10个并发，共10000个请求
time="2023-11-29T15:22:55+08:00" level=info msg="total: 10000 concurrency: 10 requests per client: 1000"
time="2023-11-29T15:23:08+08:00" level=info msg="took 13574 ms for 10000 requests"
time="2023-11-29T15:23:08+08:00" level=info msg="sent requests : 10000"
time="2023-11-29T15:23:08+08:00" level=info msg="received requests : 10000"
time="2023-11-29T15:23:08+08:00" level=info msg="received requests_OK : 10000"
time="2023-11-29T15:23:08+08:00" level=info msg="throughput (TPS) : 736"
time="2023-11-29T15:23:08+08:00" level=info msg="mean: 13561955 ns, median: 9066300 ns, max: 652868300 ns, min: 1509600 ns, p99.9: 619792100 ns"
time="2023-11-29T15:23:08+08:00" level=info msg="mean: 13 ms, median: 9 ms, max: 652 ms, min: 1 ms, p99.9: 619 ms"
- 50个并发，共100个请求
time="2023-11-29T15:24:14+08:00" level=info msg="total: 100 concurrency: 50 requests per client: 2"
time="2023-11-29T15:24:14+08:00" level=info msg="took 20 ms for 100 requests"
time="2023-11-29T15:24:14+08:00" level=info msg="sent requests : 100"
time="2023-11-29T15:24:14+08:00" level=info msg="received requests : 100"
time="2023-11-29T15:24:14+08:00" level=info msg="received requests_OK : 100"
time="2023-11-29T15:24:14+08:00" level=info msg="throughput (TPS) : 5000"
time="2023-11-29T15:24:14+08:00" level=info msg="mean: 7575184 ns, median: 4312200 ns, max: 16251400 ns, min: 2274700 ns, p99.9: 16251400 ns"
time="2023-11-29T15:24:14+08:00" level=info msg="mean: 7 ms, median: 4 ms, max: 16 ms, min: 2 ms, p99.9: 16 ms"
- 50个并发，共1000个请求
time="2023-11-29T15:25:37+08:00" level=info msg="total: 1000 concurrency: 50 requests per client: 20"
time="2023-11-29T15:25:37+08:00" level=info msg="took 322 ms for 1000 requests"
time="2023-11-29T15:25:37+08:00" level=info msg="sent requests : 1000"
time="2023-11-29T15:25:37+08:00" level=info msg="received requests : 1000"
time="2023-11-29T15:25:37+08:00" level=info msg="received requests_OK : 1000"
time="2023-11-29T15:25:37+08:00" level=info msg="throughput (TPS) : 3105"
time="2023-11-29T15:25:37+08:00" level=info msg="mean: 15954920 ns, median: 15421600 ns, max: 31742900 ns, min: 1670800 ns, p99.9: 31742900 ns"
time="2023-11-29T15:25:37+08:00" level=info msg="mean: 15 ms, median: 15 ms, max: 31 ms, min: 1 ms, p99.9: 31 ms"
- 50个并发，共10000个请求
time="2023-11-29T15:26:10+08:00" level=info msg="total: 10000 concurrency: 50 requests per client: 200"
time="2023-11-29T15:26:13+08:00" level=info msg="took 2959 ms for 10000 requests"
time="2023-11-29T15:26:13+08:00" level=info msg="sent requests : 10000"
time="2023-11-29T15:26:13+08:00" level=info msg="received requests : 10000"
time="2023-11-29T15:26:13+08:00" level=info msg="received requests_OK : 10000"
time="2023-11-29T15:26:13+08:00" level=info msg="throughput (TPS) : 3379"
time="2023-11-29T15:26:13+08:00" level=info msg="mean: 14783029 ns, median: 12242400 ns, max: 265030700 ns, min: 1512100 ns, p99.9: 233556500 ns"
time="2023-11-29T15:26:13+08:00" level=info msg="mean: 14 ms, median: 12 ms, max: 265 ms, min: 1 ms, p99.9: 233 ms" - 100个并发，共10000个请求
time="2023-11-29T15:26:49+08:00" level=info msg="total: 10000 concurrency: 100 requests per client: 100"
time="2023-11-29T15:26:51+08:00" level=info msg="took 1861 ms for 10000 requests"
time="2023-11-29T15:26:51+08:00" level=info msg="sent requests : 10000"
time="2023-11-29T15:26:51+08:00" level=info msg="received requests : 10000"
time="2023-11-29T15:26:51+08:00" level=info msg="received requests_OK : 10000"
time="2023-11-29T15:26:51+08:00" level=info msg="throughput (TPS) : 5373"
time="2023-11-29T15:26:51+08:00" level=info msg="mean: 18531453 ns, median: 16275300 ns, max: 61866200 ns, min: 1028600 ns, p99.9: 61866200 ns"
time="2023-11-29T15:26:51+08:00" level=info msg="mean: 18 ms, median: 16 ms, max: 61 ms, min: 1 ms, p99.9: 61 ms"
- 1000个并发，共10000个请求
time="2023-11-29T15:27:34+08:00" level=info msg="total: 10000 concurrency: 1000 requests per client: 10"
time="2023-11-29T15:27:34+08:00" level=info msg="took 447 ms for 10000 requests"
time="2023-11-29T15:27:34+08:00" level=info msg="sent requests : 10000"
time="2023-11-29T15:27:34+08:00" level=info msg="received requests : 10000"
time="2023-11-29T15:27:34+08:00" level=info msg="received requests_OK : 10000"
time="2023-11-29T15:27:34+08:00" level=info msg="throughput (TPS) : 22371"
time="2023-11-29T15:27:34+08:00" level=info msg="mean: 43340988 ns, median: 32804700 ns, max: 112154200 ns, min: 3652300 ns, p99.9: 108302100 ns"
time="2023-11-29T15:27:34+08:00" level=info msg="mean: 43 ms, median: 32 ms, max: 112 ms, min: 3 ms, p99.9: 108 ms"
- 10000个并发，共20000个请求
time="2023-11-29T15:28:32+08:00" level=info msg="total: 20000 concurrency: 10000 requests per client: 2"
time="2023-11-29T15:28:34+08:00" level=info msg="took 1440 ms for 20000 requests"
time="2023-11-29T15:28:34+08:00" level=info msg="sent requests : 20000"
time="2023-11-29T15:28:34+08:00" level=info msg="received requests : 20000"
time="2023-11-29T15:28:34+08:00" level=info msg="received requests_OK : 20000"
time="2023-11-29T15:28:34+08:00" level=info msg="throughput (TPS) : 13888"
time="2023-11-29T15:28:34+08:00" level=info msg="mean: 606367409 ns, median: 591710500 ns, max: 1158019400 ns, min: 6242300 ns, p99.9: 1154830950 ns"
time="2023-11-29T15:28:34+08:00" level=info msg="mean: 606 ms, median: 591 ms, max: 1158 ms, min: 6 ms, p99.9: 1154 ms"
2、元模型的map put
const key = `test_meta_model_11111122222333333`
const val = `
{
"appDataInfo": {
"id": null,
"storeAppId": null,
"name": "base",
"tag": "master",
"displayName": "基础模块",
"category": "base",
"categoryDesc": "基础模块",
"description": "基础模块",
"application": false,
"type": "SDK",
"loaderType": "SDK",
"company": "sie",
"product": "base",
"productIcon": null,
"productDesc": "工业互联网平台",
"summary": "基础模块",
"source": "base",
"resolved": "sie-snest-base-1.0-SNAPSHOT.jar",
"sdkScanPkgPath": "com.sie.snest.base",
"jarFile": "sie-snest-base-1.0-SNAPSHOT.jar",
"jarPath": "C:\\Users\\29662\\IdeaProjects\\sie-snest\\apps\\sie-snest-base-1.0-SNAPSHOT.jar",
"jarFileId": null,
"md5": "57d51247cfd631a1171539dd0b3a8b80",
"viewFileId": null,
"viewFile": null,
"viewFileMd5": null,
"state": null,
"storeJarFileId": null,
"storeMd5": null,
"dependencies": [],
"icon": null,
"license": "LGPL 3.0",
"jsonFilePath": "com/sie/snest/base/",
"jsonObject": null,
"appJsonObject": {
"summary": "基础模块",
"product": "base",
"displayName": "基础模块",
"description": "基础模块",
"type": "SDK",
"version": "0.0.1",
"categoryDesc": "基础模块",
"dependencies": [],
"productDesc": "工业互联网平台",
"license": "LGPL 3.0",
"name": "base",
"company": "sie",
"tag": "master",
"category": "base",
"events": {
"startUp": [
"auth_check_job::start",
"highavailable_init_menu::start"
]
},
"resolved": "com.sie.snest.base"
},
"author": null,
"weight": 1.01,
"models": null,
"events": {
"startUp": [
"auth_check_job::start",
"highavailable_init_menu::start"
]
},
"kind": "unStateful",
"replicas": 1,
"appInstallHosts": null,
"svcName": null,
"extServiceModels": null,
"globalConfig": null,
"appConfig": null,
"delete": false,
"newApp": false,
"nameTag": "base.master",
"baseApp": true,
"primary": false
},
"appMeta": null,
"metas": {},
"menuMetaMap": {},
"models": {},
"refIdTreeMap": {},
"modelNameViewMetaIdMap": {}
}
`- 1个并发，共10个请求
time="2023-11-29T15:43:59+08:00" level=info msg="total: 10 concurrency: 1 requests per client: 10"
time="2023-11-29T15:43:59+08:00" level=info msg="took 36 ms for 10 requests"
time="2023-11-29T15:43:59+08:00" level=info msg="sent requests : 10"
time="2023-11-29T15:43:59+08:00" level=info msg="received requests : 10"
time="2023-11-29T15:43:59+08:00" level=info msg="received requests_OK : 10"
time="2023-11-29T15:43:59+08:00" level=info msg="throughput (TPS) : 277"
time="2023-11-29T15:43:59+08:00" level=info msg="mean: 3567850 ns, median: 2957000 ns, max: 8734000 ns, min: 2061200 ns, p99.9: 6508750 ns"
time="2023-11-29T15:43:59+08:00" level=info msg="mean: 3 ms, median: 2 ms, max: 8 ms, min: 2 ms, p99.9: 6 ms"
- 10个并发，共100个请求
time="2023-11-29T15:44:38+08:00" level=info msg="total: 100 concurrency: 10 requests per client: 10"
time="2023-11-29T15:44:38+08:00" level=info msg="took 76 ms for 100 requests"
time="2023-11-29T15:44:38+08:00" level=info msg="sent requests : 100"
time="2023-11-29T15:44:38+08:00" level=info msg="received requests : 100"
time="2023-11-29T15:44:38+08:00" level=info msg="received requests_OK : 100"
time="2023-11-29T15:44:38+08:00" level=info msg="throughput (TPS) : 1315"
time="2023-11-29T15:44:38+08:00" level=info msg="mean: 7293158 ns, median: 5833600 ns, max: 16503800 ns, min: 3412900 ns, p99.9: 16503800 ns"
time="2023-11-29T15:44:38+08:00" level=info msg="mean: 7 ms, median: 5 ms, max: 16 ms, min: 3 ms, p99.9: 16 ms"
- 10个并发，共1000个请求
time="2023-11-29T15:45:06+08:00" level=info msg="total: 1000 concurrency: 10 requests per client: 100"
time="2023-11-29T15:45:07+08:00" level=info msg="took 863 ms for 1000 requests"
time="2023-11-29T15:45:07+08:00" level=info msg="sent requests : 1000"
time="2023-11-29T15:45:07+08:00" level=info msg="received requests : 1000"
time="2023-11-29T15:45:07+08:00" level=info msg="received requests_OK : 1000"
time="2023-11-29T15:45:07+08:00" level=info msg="throughput (TPS) : 1158"
time="2023-11-29T15:45:07+08:00" level=info msg="mean: 8593066 ns, median: 6355050 ns, max: 32560400 ns, min: 1412800 ns, p99.9: 32560400 ns"
time="2023-11-29T15:45:07+08:00" level=info msg="mean: 8 ms, median: 6 ms, max: 32 ms, min: 1 ms, p99.9: 32 ms"
- 10个并发，共10000个请求
time="2023-11-29T15:45:36+08:00" level=info msg="total: 10000 concurrency: 10 requests per client: 1000"
time="2023-11-29T15:45:56+08:00" level=info msg="took 20853 ms for 10000 requests"
time="2023-11-29T15:45:56+08:00" level=info msg="sent requests : 10000"
time="2023-11-29T15:45:56+08:00" level=info msg="received requests : 10000"
time="2023-11-29T15:45:56+08:00" level=info msg="received requests_OK : 10000"
time="2023-11-29T15:45:56+08:00" level=info msg="throughput (TPS) : 479"
time="2023-11-29T15:45:56+08:00" level=info msg="mean: 20829690 ns, median: 14514300 ns, max: 438134100 ns, min: 1987800 ns, p99.9: 414818550 ns"
time="2023-11-29T15:45:56+08:00" level=info msg="mean: 20 ms, median: 14 ms, max: 438 ms, min: 1 ms, p99.9: 414 ms"
- 50个并发，共100个请求
time="2023-11-29T15:24:14+08:00" level=info msg="total: 100 concurrency: 50 requests per client: 2"
time="2023-11-29T15:24:14+08:00" level=info msg="took 20 ms for 100 requests"
time="2023-11-29T15:24:14+08:00" level=info msg="sent requests : 100"
time="2023-11-29T15:24:14+08:00" level=info msg="received requests : 100"
time="2023-11-29T15:24:14+08:00" level=info msg="received requests_OK : 100"
time="2023-11-29T15:24:14+08:00" level=info msg="throughput (TPS) : 5000"
time="2023-11-29T15:24:14+08:00" level=info msg="mean: 7575184 ns, median: 4312200 ns, max: 16251400 ns, min: 2274700 ns, p99.9: 16251400 ns"
time="2023-11-29T15:24:14+08:00" level=info msg="mean: 7 ms, median: 4 ms, max: 16 ms, min: 2 ms, p99.9: 16 ms"
- 50个并发，共1000个请求
time="2023-11-29T15:46:48+08:00" level=info msg="total: 1000 concurrency: 50 requests per client: 20"
time="2023-11-29T15:46:50+08:00" level=info msg="took 1373 ms for 1000 requests"
time="2023-11-29T15:46:50+08:00" level=info msg="sent requests : 1000"
time="2023-11-29T15:46:50+08:00" level=info msg="received requests : 1000"
time="2023-11-29T15:46:50+08:00" level=info msg="received requests_OK : 1000"
time="2023-11-29T15:46:50+08:00" level=info msg="throughput (TPS) : 728"
time="2023-11-29T15:46:50+08:00" level=info msg="mean: 68348635 ns, median: 59982300 ns, max: 197308000 ns, min: 3298100 ns, p99.9: 197308000 ns"
time="2023-11-29T15:46:50+08:00" level=info msg="mean: 68 ms, median: 59 ms, max: 197 ms, min: 3 ms, p99.9: 197 ms"
- 50个并发，共10000个请求
time="2023-11-29T15:47:22+08:00" level=info msg="total: 10000 concurrency: 50 requests per client: 200"
time="2023-11-29T15:47:34+08:00" level=info msg="took 12819 ms for 10000 requests"
time="2023-11-29T15:47:34+08:00" level=info msg="sent requests : 10000"
time="2023-11-29T15:47:34+08:00" level=info msg="received requests : 10000"
time="2023-11-29T15:47:34+08:00" level=info msg="received requests_OK : 10000"
time="2023-11-29T15:47:34+08:00" level=info msg="throughput (TPS) : 780"
time="2023-11-29T15:47:34+08:00" level=info msg="mean: 64058473 ns, median: 37056100 ns, max: 1242320700 ns, min: 1734200 ns, p99.9: 1218236450 ns"
time="2023-11-29T15:47:34+08:00" level=info msg="mean: 64 ms, median: 37 ms, max: 1242 ms, min: 1 ms, p99.9: 1218 ms"
- 100个并发，共10000个请求
time="2023-11-29T15:48:04+08:00" level=info msg="total: 10000 concurrency: 100 requests per client: 100"
time="2023-11-29T15:48:13+08:00" level=info msg="took 9619 ms for 10000 requests"
time="2023-11-29T15:48:13+08:00" level=info msg="sent requests : 10000"
time="2023-11-29T15:48:13+08:00" level=info msg="received requests : 10000"
time="2023-11-29T15:48:13+08:00" level=info msg="received requests_OK : 10000"
time="2023-11-29T15:48:13+08:00" level=info msg="throughput (TPS) : 1039"
time="2023-11-29T15:48:13+08:00" level=info msg="mean: 95861576 ns, median: 84699850 ns, max: 353457500 ns, min: 6367600 ns, p99.9: 346699700 ns"
time="2023-11-29T15:48:13+08:00" level=info msg="mean: 95 ms, median: 84 ms, max: 353 ms, min: 6 ms, p99.9: 346 ms"
- 1000个并发，共10000个请求
time="2023-11-29T15:48:44+08:00" level=info msg="total: 10000 concurrency: 1000 requests per client: 10"
time="2023-11-29T15:48:55+08:00" level=info msg="took 11480 ms for 10000 requests"
time="2023-11-29T15:48:55+08:00" level=info msg="sent requests : 10000"
time="2023-11-29T15:48:55+08:00" level=info msg="received requests : 10000"
time="2023-11-29T15:48:55+08:00" level=info msg="received requests_OK : 10000"
time="2023-11-29T15:48:55+08:00" level=info msg="throughput (TPS) : 871"
time="2023-11-29T15:48:55+08:00" level=info msg="mean: 1102012743 ns, median: 988037750 ns, max: 2135727900 ns, min: 2662200 ns, p99.9: 2129425900 ns"
time="2023-11-29T15:48:55+08:00" level=info msg="mean: 1102 ms, median: 988 ms, max: 2135 ms, min: 2 ms, p99.9: 2129 ms"
- 10000个并发，共20000个请求
time="2023-11-29T15:49:32+08:00" level=info msg="total: 20000 concurrency: 10000 requests per client: 2"
time="2023-11-29T15:49:53+08:00" level=info msg="took 20497 ms for 20000 requests"
time="2023-11-29T15:49:53+08:00" level=info msg="sent requests : 20000"
time="2023-11-29T15:49:53+08:00" level=info msg="received requests : 20000"
time="2023-11-29T15:49:53+08:00" level=info msg="received requests_OK : 20000"
time="2023-11-29T15:49:53+08:00" level=info msg="throughput (TPS) : 975"
time="2023-11-29T15:49:53+08:00" level=info msg="mean: 9629394346 ns, median: 9241691750 ns, max: 18458848300 ns, min: 3947500 ns, p99.9: 18437869200 ns"
time="2023-11-29T15:49:53+08:00" level=info msg="mean: 9629 ms, median: 9241 ms, max: 18458 ms, min: 3 ms, p99.9: 18437 ms"
3、元模型的map get
- 10个并发，共100个请求
time="2023-11-29T15:51:20+08:00" level=info msg="total: 100 concurrency: 10 requests per client: 10"
time="2023-11-29T15:51:20+08:00" level=info msg="took 82 ms for 100 requests"
time="2023-11-29T15:51:20+08:00" level=info msg="sent requests : 100"
time="2023-11-29T15:51:20+08:00" level=info msg="received requests : 100"
time="2023-11-29T15:51:20+08:00" level=info msg="received requests_OK : 100"
time="2023-11-29T15:51:20+08:00" level=info msg="throughput (TPS) : 1219"
time="2023-11-29T15:51:20+08:00" level=info msg="mean: 7894017 ns, median: 6708500 ns, max: 22592800 ns, min: 2172000 ns, p99.9: 21619400 ns"
time="2023-11-29T15:51:20+08:00" level=info msg="mean: 7 ms, median: 6 ms, max: 22 ms, min: 2 ms, p99.9: 21 ms"
- 10个并发，共1000个请求
time="2023-11-29T15:52:56+08:00" level=info msg="total: 1000 concurrency: 10 requests per client: 100"
time="2023-11-29T15:52:56+08:00" level=info msg="took 816 ms for 1000 requests"
time="2023-11-29T15:52:56+08:00" level=info msg="sent requests : 1000"
time="2023-11-29T15:52:56+08:00" level=info msg="received requests : 1000"
time="2023-11-29T15:52:56+08:00" level=info msg="received requests_OK : 1000"
time="2023-11-29T15:52:56+08:00" level=info msg="throughput (TPS) : 1225"
time="2023-11-29T15:52:56+08:00" level=info msg="mean: 8093208 ns, median: 5498600 ns, max: 37198100 ns, min: 1601400 ns, p99.9: 35485450 ns"
time="2023-11-29T15:52:56+08:00" level=info msg="mean: 8 ms, median: 5 ms, max: 37 ms, min: 1 ms, p99.9: 35 ms"
- 10个并发，共10000个请求
time="2023-11-29T15:53:21+08:00" level=info msg="total: 10000 concurrency: 10 requests per client: 1000"
time="2023-11-29T15:53:31+08:00" level=info msg="took 10059 ms for 10000 requests"
time="2023-11-29T15:53:31+08:00" level=info msg="sent requests : 10000"
time="2023-11-29T15:53:31+08:00" level=info msg="received requests : 10000"
time="2023-11-29T15:53:31+08:00" level=info msg="received requests_OK : 10000"
time="2023-11-29T15:53:31+08:00" level=info msg="throughput (TPS) : 994"
time="2023-11-29T15:53:31+08:00" level=info msg="mean: 10034711 ns, median: 5877000 ns, max: 851575200 ns, min: 1510200 ns, p99.9: 838274550 ns"
time="2023-11-29T15:53:31+08:00" level=info msg="mean: 10 ms, median: 5 ms, max: 851 ms, min: 1 ms, p99.9: 838 ms"
- 50个并发，共100个请求
time="2023-11-29T15:53:57+08:00" level=info msg="total: 100 concurrency: 50 requests per client: 2"
time="2023-11-29T15:53:57+08:00" level=info msg="took 33 ms for 100 requests"
time="2023-11-29T15:53:57+08:00" level=info msg="sent requests : 100"
time="2023-11-29T15:53:57+08:00" level=info msg="received requests : 100"
time="2023-11-29T15:53:57+08:00" level=info msg="received requests_OK : 100"
time="2023-11-29T15:53:57+08:00" level=info msg="throughput (TPS) : 3030"
time="2023-11-29T15:53:57+08:00" level=info msg="mean: 13962681 ns, median: 12463350 ns, max: 22122800 ns, min: 2119500 ns, p99.9: 22122800 ns"
time="2023-11-29T15:53:57+08:00" level=info msg="mean: 13 ms, median: 12 ms, max: 22 ms, min: 2 ms, p99.9: 22 ms"
- 50个并发，共1000个请求
time="2023-11-29T15:54:17+08:00" level=info msg="total: 1000 concurrency: 50 requests per client: 20"
time="2023-11-29T15:54:17+08:00" level=info msg="took 384 ms for 1000 requests"
time="2023-11-29T15:54:17+08:00" level=info msg="sent requests : 1000"
time="2023-11-29T15:54:17+08:00" level=info msg="received requests : 1000"
time="2023-11-29T15:54:17+08:00" level=info msg="received requests_OK : 1000"
time="2023-11-29T15:54:17+08:00" level=info msg="throughput (TPS) : 2604"
time="2023-11-29T15:54:17+08:00" level=info msg="mean: 19014493 ns, median: 20007500 ns, max: 39160200 ns, min: 2248800 ns, p99.9: 39160200 ns"
time="2023-11-29T15:54:17+08:00" level=info msg="mean: 19 ms, median: 20 ms, max: 39 ms, min: 2 ms, p99.9: 39 ms"
- 50个并发，共10000个请求
time="2023-11-29T15:54:41+08:00" level=info msg="total: 10000 concurrency: 50 requests per client: 200"
time="2023-11-29T15:54:44+08:00" level=info msg="took 3075 ms for 10000 requests"
time="2023-11-29T15:54:44+08:00" level=info msg="sent requests : 10000"
time="2023-11-29T15:54:44+08:00" level=info msg="received requests : 10000"
time="2023-11-29T15:54:44+08:00" level=info msg="received requests_OK : 10000"
time="2023-11-29T15:54:44+08:00" level=info msg="throughput (TPS) : 3252"
time="2023-11-29T15:54:44+08:00" level=info msg="mean: 15350080 ns, median: 13936100 ns, max: 74815300 ns, min: 1637300 ns, p99.9: 74309900 ns"
time="2023-11-29T15:54:44+08:00" level=info msg="mean: 15 ms, median: 13 ms, max: 74 ms, min: 1 ms, p99.9: 74 ms"
- 100个并发，共10000个请求
time="2023-11-29T15:55:08+08:00" level=info msg="total: 10000 concurrency: 100 requests per client: 100"
time="2023-11-29T15:55:11+08:00" level=info msg="took 3017 ms for 10000 requests"
time="2023-11-29T15:55:11+08:00" level=info msg="sent requests : 10000"
time="2023-11-29T15:55:11+08:00" level=info msg="received requests : 10000"
time="2023-11-29T15:55:11+08:00" level=info msg="received requests_OK : 10000"
time="2023-11-29T15:55:11+08:00" level=info msg="throughput (TPS) : 3314"
time="2023-11-29T15:55:11+08:00" level=info msg="mean: 30073157 ns, median: 20134900 ns, max: 942773300 ns, min: 3710100 ns, p99.9: 942152800 ns"
time="2023-11-29T15:55:11+08:00" level=info msg="mean: 30 ms, median: 20 ms, max: 942 ms, min: 3 ms, p99.9: 942 ms"
- 1000个并发，共10000个请求
time="2023-11-29T15:55:40+08:00" level=info msg="total: 10000 concurrency: 1000 requests per client: 10"
time="2023-11-29T15:55:42+08:00" level=info msg="took 2423 ms for 10000 requests"
time="2023-11-29T15:55:42+08:00" level=info msg="sent requests : 10000"
time="2023-11-29T15:55:42+08:00" level=info msg="received requests : 10000"
time="2023-11-29T15:55:42+08:00" level=info msg="received requests_OK : 10000"
time="2023-11-29T15:55:42+08:00" level=info msg="throughput (TPS) : 4127"
time="2023-11-29T15:55:42+08:00" level=info msg="mean: 231318655 ns, median: 223177700 ns, max: 350379500 ns, min: 4207800 ns, p99.9: 349300600 ns"
time="2023-11-29T15:55:42+08:00" level=info msg="mean: 231 ms, median: 223 ms, max: 350 ms, min: 4 ms, p99.9: 349 ms"
- 10000个并发，共20000个请求
time="2023-11-29T15:56:08+08:00" level=info msg="total: 20000 concurrency: 10000 requests per client: 2"
time="2023-11-29T15:56:13+08:00" level=info msg="took 5116 ms for 20000 requests"
time="2023-11-29T15:56:13+08:00" level=info msg="sent requests : 20000"
time="2023-11-29T15:56:13+08:00" level=info msg="received requests : 20000"
time="2023-11-29T15:56:13+08:00" level=info msg="received requests_OK : 20000"
time="2023-11-29T15:56:13+08:00" level=info msg="throughput (TPS) : 3909"
time="2023-11-29T15:56:13+08:00" level=info msg="mean: 2087396474 ns, median: 2025976800 ns, max: 3294130100 ns, min: 20420700 ns, p99.9: 3268514400 ns"
time="2023-11-29T15:56:13+08:00" level=info msg="mean: 2087 ms, median: 2025 ms, max: 3294 ms, min: 20 ms, p99.9: 3268 ms"
4、源码
package main
import (
"context"
"flag"
"fmt"
"sync"
"sync/atomic"
"time"
"github.com/hazelcast/hazelcast-go-client"
log "github.com/sirupsen/logrus"
"go.uber.org/ratelimit"
)
const key = `test_meta_model_11111122222333333`
const val = `
{
"appDataInfo": {
"id": null,
"storeAppId": null,
"name": "base",
"tag": "master",
"displayName": "基础模块",
"category": "base",
"categoryDesc": "基础模块",
"description": "基础模块",
"application": false,
"type": "SDK",
"loaderType": "SDK",
"company": "sie",
"product": "base",
"productIcon": null,
"productDesc": "工业互联网平台",
"summary": "基础模块",
"source": "base",
"resolved": "sie-snest-base-1.0-SNAPSHOT.jar",
"sdkScanPkgPath": "com.sie.snest.base",
"jarFile": "sie-snest-base-1.0-SNAPSHOT.jar",
"jarPath": "C:\\Users\\29662\\IdeaProjects\\sie-snest\\apps\\sie-snest-base-1.0-SNAPSHOT.jar",
"jarFileId": null,
"md5": "57d51247cfd631a1171539dd0b3a8b80",
"viewFileId": null,
"viewFile": null,
"viewFileMd5": null,
"state": null,
"storeJarFileId": null,
"storeMd5": null,
"dependencies": [],
"icon": null,
"license": "LGPL 3.0",
"jsonFilePath": "com/sie/snest/base/",
"jsonObject": null,
"appJsonObject": {
"summary": "基础模块",
"product": "base",
"displayName": "基础模块",
"description": "基础模块",
"type": "SDK",
"version": "0.0.1",
"categoryDesc": "基础模块",
"dependencies": [],
"productDesc": "工业互联网平台",
"license": "LGPL 3.0",
"name": "base",
"company": "sie",
"tag": "master",
"category": "base",
"events": {
"startUp": [
"auth_check_job::start",
"highavailable_init_menu::start"
]
},
"resolved": "com.sie.snest.base"
},
"author": null,
"weight": 1.01,
"models": null,
"events": {
"startUp": [
"auth_check_job::start",
"highavailable_init_menu::start"
]
},
"kind": "unStateful",
"replicas": 1,
"appInstallHosts": null,
"svcName": null,
"extServiceModels": null,
"globalConfig": null,
"appConfig": null,
"delete": false,
"newApp": false,
"nameTag": "base.master",
"baseApp": true,
"primary": false
},
"appMeta": null,
"metas": {},
"menuMetaMap": {},
"models": {},
"refIdTreeMap": {},
"modelNameViewMetaIdMap": {}
}
`
var (
concurrency = flag.Int("c", 1, "concurrency")
total = flag.Int("n", 10, "total requests for all clients")
host = flag.String("s", "127.0.0.1:8972", "server ip and port")
pool = flag.Int("pool", 10, " shared grpc clients")
rate = flag.Int("r", 0, "throughputs")
client = flag.String("type", "kim", "client type")
)
func main() {
flag.Parse()
ctx := context.TODO()
cfg := hazelcast.Config{}
cfg.Cluster.Name = "hazelcast-benchmark"
cfg.Cluster.Network.Addresses = []string{"192.168.168.176:5701"}
hz, err := hazelcast.StartNewClientWithConfig(ctx, cfg)
if err != nil {
panic(fmt.Errorf("starting the client with config: %w", err))
}
mp, err := hz.GetMap(ctx, "my-distributed-map")
if err != nil {
panic(fmt.Errorf("trying to get a map: %w", err))
}
var rl ratelimit.Limiter
if *rate > 0 {
rl = ratelimit.New(*rate)
}
// 并发goroutine数.模拟客户端
n := *concurrency
// 每个客户端需要发送的请求数
m := *total / n
log.Infof("total: %d concurrency: %d requests per client: %d", *total, n, m)
// 等待所有测试完成
var wg sync.WaitGroup
wg.Add(n * m)
// 总请求数
var trans uint64
// 返回正常的总请求数
var transOK uint64
// 每个goroutine的耗时记录
d := make([][]int64, n, n)
// 栅栏，控制客户端同时开始测试
var startWg sync.WaitGroup
startWg.Add(n + 1) // +1 是因为有一个goroutine用来记录开始时间
// 创建客户端 goroutine 并进行测试
startTime := time.Now().UnixNano()
go func() {
startWg.Done()
startWg.Wait()
startTime = time.Now().UnixNano()
}()
for i := 0; i < n; i++ {
dt := make([]int64, 0, m)
d = append(d, dt)
go func(i int) {
for j := 0; j < m; j++ {
// 限流，这里不把限流的时间计算到等待耗时中
if rl != nil {
rl.Take()
}
t := time.Now().UnixNano()
_, err = mp.Get(ctx, key)
if err != nil {
panic(fmt.Errorf("trying to put to map: %w", err))
}
t = time.Now().UnixNano() - t // 等待时间+服务时间，等待时间是客户端调度的等待时间以及服务端读取请求、调度的时间，服务时间是请求被服务处理的实际时间
d[i] = append(d[i], t)
if err == nil {
atomic.AddUint64(&transOK, 1)
}
atomic.AddUint64(&trans, 1)
wg.Done()
}
}(i)
}
wg.Wait()
// 统计
Stats(startTime, *total, d, trans, transOK)
}
// Stats 统计结果.
func Stats(startTime int64, totalRequests int, tookTimes [][]int64, trans, transOK uint64) {
// 测试总耗时
totalTInNano := time.Now().UnixNano() - startTime
totalT := totalTInNano / 1000000
log.Infof("took %d ms for %d requests", totalT, totalRequests)
// 汇总每个请求的耗时
totalD := make([]int64, 0, totalRequests)
for _, k := range tookTimes {
totalD = append(totalD, k...)
}
// 将int64数组转换成float64数组，以便分析
totalD2 := make([]float64, 0, totalRequests)
for _, k := range totalD {
totalD2 = append(totalD2, float64(k))
}
// 计算各个指标
mean, _ := stats.Mean(totalD2)
median, _ := stats.Median(totalD2)
max, _ := stats.Max(totalD2)
min, _ := stats.Min(totalD2)
p999, _ := stats.Percentile(totalD2, 99.9)
// 输出结果
log.Infof("sent requests : %d", totalRequests)
log.Infof("received requests : %d", trans)
log.Infof("received requests_OK : %d", transOK)
if totalT == 0 {
log.Infof("throughput (TPS) : %d", int64(totalRequests)*1000*1000000/totalTInNano)
} else {
log.Infof("throughput (TPS) : %d", int64(totalRequests)*1000/totalT)
}
log.Infof("mean: %.f ns, median: %.f ns, max: %.f ns, min: %.f ns, p99.9: %.f ns", mean, median, max, min, p999)
log.Infof("mean: %d ms, median: %d ms, max: %d ms, min: %d ms, p99.9: %d ms", int64(mean/1000000), int64(median/1000000), int64(max/1000000), int64(min/1000000), int64(p999/1000000))
}
5，问题
- 序列化