☰
Current Page
Main Menu
Home
Home
Editing
Hazelcast分布式内存同步设计与实现
Edit
Preview
h1
h2
h3
default
Set your preferred keybinding
default
vim
emacs
markdown
Set this page's format to
AsciiDoc
Creole
Markdown
MediaWiki
Org-mode
Plain Text
RDoc
Textile
Rendering unavailable for
BibTeX
Pod
reStructuredText
Help 1
Help 1
Help 1
Help 2
Help 3
Help 4
Help 5
Help 6
Help 7
Help 8
Autosaved text is available. Click the button to restore it.
Restore Text
### 1，需求分析 整个平台如果在单机版的情况下，因为所有的数据都在同一个进程或者pod中， 那么是不存在分布式数据同步的问题，但是如果是在分布式部署情况下，由于多副本，以及app和meta之间的关联性 （比如说app1和app2都包含meta1，那么meta1变动，需要同步给其他所有关联的app1和app2） 那么此时就需要分布式同步的功能，必须保证所有的pod和所有的app都能获取到最新的更新， 这样才能保证对外是统一的一致性的表现。 在已有的分布式环境，不同的app安装在独立的容器中，且没有多副本，所以说本质上跟单机版是一样的，只是对于整个软件系统来说它通过分片的方式， 将各个应用隔离开来，各自的内容数据也只用于各自的容器中。 但是对于多租户app来说，它在分布式环境中是必须所有的app都需要的，那么自然涉及到不同容器中的内存同步。 同理，对于多副本来说更是需要内存的同步。 ### 2，方案选型 - ~~基于redis stream的时间发布订阅同步~~ ~~这是早期的版本，由于各种问题（尤其是并发写）现在已经废弃。~~ - 基于 Hazelcast Hazelcast 是由Hazelcast公司开发的一款开源的分布式内存级别的缓存数据库，可以为基于JVM环境运行的各种应用提供分布式集群和分布式缓存服务。 1. 我们需要什么？ 通过在实际的项目中，我们的模型、菜单、策略等信息都是保持在一个map中，所以我们需要的是一个分布式map， 在一个节点操作，会自动同步给所有的节点（对于可分片的数据结构，比如map，实际实现不是全部同步，而是基于分片的形式，但是在用户看来跟同步效果是一样的）。另外，我们对于一些websocket消息，还需要一个分布式queue来统一分发。 2. Hazelcast 能提供什么？ 它能提供分布式map，分布式queue。而且它实现了java标准库中Map的所有接口，对于使用者来说之前怎么用map的，现在依然可以使用， 不需要修改任何方法，这对已有的代码是非常友好的。另外，它提供了嵌入式的使用方式，这是我认为最有价值的地方，分布式缓存在市面上已经有很多了， 比如 ignite redis 等等，但是它们都不支持嵌入式的使用方式，需要额外独立的部署第三方服务，这对一些使用场景来说显得有点复杂了。 此时，Hazelcast的嵌入式使用体验就非常友好，只需要引入一个jar包即可，每个节点即是客户端也是服务端，自动服务注册和发现（比如多播，比如基于k8s等），自动组网， 对于用户来说非常方便。 ### 3，功能实现 - 基于Hazelcast 提供一个distributed map ```java public class DMap { private static boolean distributed = WebMode.of(ConfigConstant.ENGINE_RUN_MODE) != WebMode.DISTRIBUTED; static private String mapName = "hazelcast-map"; static private HazelcastInstance instance; static { if (distributed) { SystemInfo systemInfo = new SystemInfo(); String id = IdGenerator.getDeviceId(systemInfo); String instanceName = "hazelInstance_" + id; String clusterName = "iidp-hazelcast-cluster"; Config config = new Config(); config.setClusterName(clusterName); config.setInstanceName(instanceName); config.getJetConfig().setEnabled(false); config.setIntegrityCheckerConfig(new IntegrityCheckerConfig().setEnabled(false)); config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false); config.getNetworkConfig().getJoin().getKubernetesConfig().setEnabled(true); // map config // config.getMapConfig(). instance = Hazelcast.newHazelcastInstance(config); } } // 如果用户想直接hazelcast提供的方法可以在获取instance后直接使用 static public HazelcastInstance getInstance() { return instance; } static public <K, V> Map<K, V> getMap() { if (!distributed) { return new HashMap<>(); } return instance.getMap(mapName); } static public <K, V> Map<K, V> getMap(String name) { if (!distributed) { return new HashMap<>(); } return instance.getMap(name); } } ``` - 在业务代码中，将需要分布式map的地方替换成DMap 由于DMap实现了标准库中的Map接口，就可以将DMap当作普通的map使用了。 ```java private Map<String, MenuMeta> menuMetaMap = DMap.getMap(); ``` 但是实际使用场景中，比较复杂，主要体现在序列化问题，在内存中可以通过引用的方式修改map 的value等问题，需要业务自行调整和适配。 ### 4，性能测试 #### 1、简单的map put 操作 ``` mp.Put(ctx, "foo", "bar") ``` 结果： - 1个并发，共10个请求 ``` time="2023-11-29T15:15:39+08:00" level=info msg="total: 10 concurrency: 1 requests per client: 10" time="2023-11-29T15:15:39+08:00" level=info msg="took 27 ms for 10 requests" time="2023-11-29T15:15:39+08:00" level=info msg="sent requests : 10" time="2023-11-29T15:15:39+08:00" level=info msg="received requests : 10" time="2023-11-29T15:15:39+08:00" level=info msg="received requests_OK : 10" time="2023-11-29T15:15:39+08:00" level=info msg="throughput (TPS) : 370" time="2023-11-29T15:15:39+08:00" level=info msg="mean: 2772220 ns, median: 1824400 ns, max: 9850800 ns, min: 1627700 ns, p99.9: 6775050 ns" time="2023-11-29T15:15:39+08:00" level=info msg="mean: 2 ms, median: 1 ms, max: 9 ms, min: 1 ms, p99.9: 6 ms" ``` - 10个并发，共100个请求 ``` time="2023-11-29T15:17:57+08:00" level=info msg="total: 100 concurrency: 10 requests per client: 10" time="2023-11-29T15:17:57+08:00" level=info msg="took 84 ms for 100 requests" time="2023-11-29T15:17:57+08:00" level=info msg="sent requests : 100" time="2023-11-29T15:17:57+08:00" level=info msg="received requests : 100" time="2023-11-29T15:17:57+08:00" level=info msg="received requests_OK : 100" time="2023-11-29T15:17:57+08:00" level=info msg="throughput (TPS) : 1190" time="2023-11-29T15:17:57+08:00" level=info msg="mean: 8230900 ns, median: 5983800 ns, max: 28240200 ns, min: 1665600 ns, p99.9: 28240200 ns" time="2023-11-29T15:17:57+08:00" level=info msg="mean: 8 ms, median: 5 ms, max: 28 ms, min: 1 ms, p99.9: 28 ms" ``` - 10个并发，共1000个请求 ``` time="2023-11-29T15:19:25+08:00" level=info msg="total: 1000 concurrency: 10 requests per client: 100" time="2023-11-29T15:19:26+08:00" level=info msg="took 836 ms for 1000 requests" time="2023-11-29T15:19:26+08:00" level=info msg="sent requests : 1000" time="2023-11-29T15:19:26+08:00" level=info msg="received requests : 1000" time="2023-11-29T15:19:26+08:00" level=info msg="received requests_OK : 1000" time="2023-11-29T15:19:26+08:00" level=info msg="throughput (TPS) : 1196" time="2023-11-29T15:19:26+08:00" level=info msg="mean: 8338478 ns, median: 6197100 ns, max: 25382100 ns, min: 1508800 ns, p99.9: 25382100 ns" time="2023-11-29T15:19:26+08:00" level=info msg="mean: 8 ms, median: 6 ms, max: 25 ms, min: 1 ms, p99.9: 25 ms" ``` - 10个并发，共10000个请求 ``` time="2023-11-29T15:22:55+08:00" level=info msg="total: 10000 concurrency: 10 requests per client: 1000" time="2023-11-29T15:23:08+08:00" level=info msg="took 13574 ms for 10000 requests" time="2023-11-29T15:23:08+08:00" level=info msg="sent requests : 10000" time="2023-11-29T15:23:08+08:00" level=info msg="received requests : 10000" time="2023-11-29T15:23:08+08:00" level=info msg="received requests_OK : 10000" time="2023-11-29T15:23:08+08:00" level=info msg="throughput (TPS) : 736" time="2023-11-29T15:23:08+08:00" level=info msg="mean: 13561955 ns, median: 9066300 ns, max: 652868300 ns, min: 1509600 ns, p99.9: 619792100 ns" time="2023-11-29T15:23:08+08:00" level=info msg="mean: 13 ms, median: 9 ms, max: 652 ms, min: 1 ms, p99.9: 619 ms" ``` - 50个并发，共100个请求 ``` time="2023-11-29T15:24:14+08:00" level=info msg="total: 100 concurrency: 50 requests per client: 2" time="2023-11-29T15:24:14+08:00" level=info msg="took 20 ms for 100 requests" time="2023-11-29T15:24:14+08:00" level=info msg="sent requests : 100" time="2023-11-29T15:24:14+08:00" level=info msg="received requests : 100" time="2023-11-29T15:24:14+08:00" level=info msg="received requests_OK : 100" time="2023-11-29T15:24:14+08:00" level=info msg="throughput (TPS) : 5000" time="2023-11-29T15:24:14+08:00" level=info msg="mean: 7575184 ns, median: 4312200 ns, max: 16251400 ns, min: 2274700 ns, p99.9: 16251400 ns" time="2023-11-29T15:24:14+08:00" level=info msg="mean: 7 ms, median: 4 ms, max: 16 ms, min: 2 ms, p99.9: 16 ms" ``` - 50个并发，共1000个请求 ``` time="2023-11-29T15:25:37+08:00" level=info msg="total: 1000 concurrency: 50 requests per client: 20" time="2023-11-29T15:25:37+08:00" level=info msg="took 322 ms for 1000 requests" time="2023-11-29T15:25:37+08:00" level=info msg="sent requests : 1000" time="2023-11-29T15:25:37+08:00" level=info msg="received requests : 1000" time="2023-11-29T15:25:37+08:00" level=info msg="received requests_OK : 1000" time="2023-11-29T15:25:37+08:00" level=info msg="throughput (TPS) : 3105" time="2023-11-29T15:25:37+08:00" level=info msg="mean: 15954920 ns, median: 15421600 ns, max: 31742900 ns, min: 1670800 ns, p99.9: 31742900 ns" time="2023-11-29T15:25:37+08:00" level=info msg="mean: 15 ms, median: 15 ms, max: 31 ms, min: 1 ms, p99.9: 31 ms" ``` - 50个并发，共10000个请求 ``` time="2023-11-29T15:26:10+08:00" level=info msg="total: 10000 concurrency: 50 requests per client: 200" time="2023-11-29T15:26:13+08:00" level=info msg="took 2959 ms for 10000 requests" time="2023-11-29T15:26:13+08:00" level=info msg="sent requests : 10000" time="2023-11-29T15:26:13+08:00" level=info msg="received requests : 10000" time="2023-11-29T15:26:13+08:00" level=info msg="received requests_OK : 10000" time="2023-11-29T15:26:13+08:00" level=info msg="throughput (TPS) : 3379" time="2023-11-29T15:26:13+08:00" level=info msg="mean: 14783029 ns, median: 12242400 ns, max: 265030700 ns, min: 1512100 ns, p99.9: 233556500 ns" time="2023-11-29T15:26:13+08:00" level=info msg="mean: 14 ms, median: 12 ms, max: 265 ms, min: 1 ms, p99.9: 233 ms" ``` - 100个并发，共10000个请求 ``` time="2023-11-29T15:26:49+08:00" level=info msg="total: 10000 concurrency: 100 requests per client: 100" time="2023-11-29T15:26:51+08:00" level=info msg="took 1861 ms for 10000 requests" time="2023-11-29T15:26:51+08:00" level=info msg="sent requests : 10000" time="2023-11-29T15:26:51+08:00" level=info msg="received requests : 10000" time="2023-11-29T15:26:51+08:00" level=info msg="received requests_OK : 10000" time="2023-11-29T15:26:51+08:00" level=info msg="throughput (TPS) : 5373" time="2023-11-29T15:26:51+08:00" level=info msg="mean: 18531453 ns, median: 16275300 ns, max: 61866200 ns, min: 1028600 ns, p99.9: 61866200 ns" time="2023-11-29T15:26:51+08:00" level=info msg="mean: 18 ms, median: 16 ms, max: 61 ms, min: 1 ms, p99.9: 61 ms" ``` - 1000个并发，共10000个请求 ``` time="2023-11-29T15:27:34+08:00" level=info msg="total: 10000 concurrency: 1000 requests per client: 10" time="2023-11-29T15:27:34+08:00" level=info msg="took 447 ms for 10000 requests" time="2023-11-29T15:27:34+08:00" level=info msg="sent requests : 10000" time="2023-11-29T15:27:34+08:00" level=info msg="received requests : 10000" time="2023-11-29T15:27:34+08:00" level=info msg="received requests_OK : 10000" time="2023-11-29T15:27:34+08:00" level=info msg="throughput (TPS) : 22371" time="2023-11-29T15:27:34+08:00" level=info msg="mean: 43340988 ns, median: 32804700 ns, max: 112154200 ns, min: 3652300 ns, p99.9: 108302100 ns" time="2023-11-29T15:27:34+08:00" level=info msg="mean: 43 ms, median: 32 ms, max: 112 ms, min: 3 ms, p99.9: 108 ms" ``` - 10000个并发，共20000个请求 ``` time="2023-11-29T15:28:32+08:00" level=info msg="total: 20000 concurrency: 10000 requests per client: 2" time="2023-11-29T15:28:34+08:00" level=info msg="took 1440 ms for 20000 requests" time="2023-11-29T15:28:34+08:00" level=info msg="sent requests : 20000" time="2023-11-29T15:28:34+08:00" level=info msg="received requests : 20000" time="2023-11-29T15:28:34+08:00" level=info msg="received requests_OK : 20000" time="2023-11-29T15:28:34+08:00" level=info msg="throughput (TPS) : 13888" time="2023-11-29T15:28:34+08:00" level=info msg="mean: 606367409 ns, median: 591710500 ns, max: 1158019400 ns, min: 6242300 ns, p99.9: 1154830950 ns" time="2023-11-29T15:28:34+08:00" level=info msg="mean: 606 ms, median: 591 ms, max: 1158 ms, min: 6 ms, p99.9: 1154 ms" ``` #### 2、元模型的map put ```golang const key = `test_meta_model_11111122222333333` const val = ` { "appDataInfo": { "id": null, "storeAppId": null, "name": "base", "tag": "master", "displayName": "基础模块", "category": "base", "categoryDesc": "基础模块", "description": "基础模块", "application": false, "type": "SDK", "loaderType": "SDK", "company": "sie", "product": "base", "productIcon": null, "productDesc": "工业互联网平台", "summary": "基础模块", "source": "base", "resolved": "sie-snest-base-1.0-SNAPSHOT.jar", "sdkScanPkgPath": "com.sie.snest.base", "jarFile": "sie-snest-base-1.0-SNAPSHOT.jar", "jarPath": "C:\\Users\\29662\\IdeaProjects\\sie-snest\\apps\\sie-snest-base-1.0-SNAPSHOT.jar", "jarFileId": null, "md5": "57d51247cfd631a1171539dd0b3a8b80", "viewFileId": null, "viewFile": null, "viewFileMd5": null, "state": null, "storeJarFileId": null, "storeMd5": null, "dependencies": [], "icon": null, "license": "LGPL 3.0", "jsonFilePath": "com/sie/snest/base/", "jsonObject": null, "appJsonObject": { "summary": "基础模块", "product": "base", "displayName": "基础模块", "description": "基础模块", "type": "SDK", "version": "0.0.1", "categoryDesc": "基础模块", "dependencies": [], "productDesc": "工业互联网平台", "license": "LGPL 3.0", "name": "base", "company": "sie", "tag": "master", "category": "base", "events": { "startUp": [ "auth_check_job::start", "highavailable_init_menu::start" ] }, "resolved": "com.sie.snest.base" }, "author": null, "weight": 1.01, "models": null, "events": { "startUp": [ "auth_check_job::start", "highavailable_init_menu::start" ] }, "kind": "unStateful", "replicas": 1, "appInstallHosts": null, "svcName": null, "extServiceModels": null, "globalConfig": null, "appConfig": null, "delete": false, "newApp": false, "nameTag": "base.master", "baseApp": true, "primary": false }, "appMeta": null, "metas": {}, "menuMetaMap": {}, "models": {}, "refIdTreeMap": {}, "modelNameViewMetaIdMap": {} } ` ``` - 1个并发，共10个请求 ``` time="2023-11-29T15:43:59+08:00" level=info msg="total: 10 concurrency: 1 requests per client: 10" time="2023-11-29T15:43:59+08:00" level=info msg="took 36 ms for 10 requests" time="2023-11-29T15:43:59+08:00" level=info msg="sent requests : 10" time="2023-11-29T15:43:59+08:00" level=info msg="received requests : 10" time="2023-11-29T15:43:59+08:00" level=info msg="received requests_OK : 10" time="2023-11-29T15:43:59+08:00" level=info msg="throughput (TPS) : 277" time="2023-11-29T15:43:59+08:00" level=info msg="mean: 3567850 ns, median: 2957000 ns, max: 8734000 ns, min: 2061200 ns, p99.9: 6508750 ns" time="2023-11-29T15:43:59+08:00" level=info msg="mean: 3 ms, median: 2 ms, max: 8 ms, min: 2 ms, p99.9: 6 ms" ``` - 10个并发，共100个请求 ``` time="2023-11-29T15:44:38+08:00" level=info msg="total: 100 concurrency: 10 requests per client: 10" time="2023-11-29T15:44:38+08:00" level=info msg="took 76 ms for 100 requests" time="2023-11-29T15:44:38+08:00" level=info msg="sent requests : 100" time="2023-11-29T15:44:38+08:00" level=info msg="received requests : 100" time="2023-11-29T15:44:38+08:00" level=info msg="received requests_OK : 100" time="2023-11-29T15:44:38+08:00" level=info msg="throughput (TPS) : 1315" time="2023-11-29T15:44:38+08:00" level=info msg="mean: 7293158 ns, median: 5833600 ns, max: 16503800 ns, min: 3412900 ns, p99.9: 16503800 ns" time="2023-11-29T15:44:38+08:00" level=info msg="mean: 7 ms, median: 5 ms, max: 16 ms, min: 3 ms, p99.9: 16 ms" ``` - 10个并发，共1000个请求 ``` time="2023-11-29T15:45:06+08:00" level=info msg="total: 1000 concurrency: 10 requests per client: 100" time="2023-11-29T15:45:07+08:00" level=info msg="took 863 ms for 1000 requests" time="2023-11-29T15:45:07+08:00" level=info msg="sent requests : 1000" time="2023-11-29T15:45:07+08:00" level=info msg="received requests : 1000" time="2023-11-29T15:45:07+08:00" level=info msg="received requests_OK : 1000" time="2023-11-29T15:45:07+08:00" level=info msg="throughput (TPS) : 1158" time="2023-11-29T15:45:07+08:00" level=info msg="mean: 8593066 ns, median: 6355050 ns, max: 32560400 ns, min: 1412800 ns, p99.9: 32560400 ns" time="2023-11-29T15:45:07+08:00" level=info msg="mean: 8 ms, median: 6 ms, max: 32 ms, min: 1 ms, p99.9: 32 ms" ``` - 10个并发，共10000个请求 ``` time="2023-11-29T15:45:36+08:00" level=info msg="total: 10000 concurrency: 10 requests per client: 1000" time="2023-11-29T15:45:56+08:00" level=info msg="took 20853 ms for 10000 requests" time="2023-11-29T15:45:56+08:00" level=info msg="sent requests : 10000" time="2023-11-29T15:45:56+08:00" level=info msg="received requests : 10000" time="2023-11-29T15:45:56+08:00" level=info msg="received requests_OK : 10000" time="2023-11-29T15:45:56+08:00" level=info msg="throughput (TPS) : 479" time="2023-11-29T15:45:56+08:00" level=info msg="mean: 20829690 ns, median: 14514300 ns, max: 438134100 ns, min: 1987800 ns, p99.9: 414818550 ns" time="2023-11-29T15:45:56+08:00" level=info msg="mean: 20 ms, median: 14 ms, max: 438 ms, min: 1 ms, p99.9: 414 ms" ``` - 50个并发，共100个请求 ``` time="2023-11-29T15:24:14+08:00" level=info msg="total: 100 concurrency: 50 requests per client: 2" time="2023-11-29T15:24:14+08:00" level=info msg="took 20 ms for 100 requests" time="2023-11-29T15:24:14+08:00" level=info msg="sent requests : 100" time="2023-11-29T15:24:14+08:00" level=info msg="received requests : 100" time="2023-11-29T15:24:14+08:00" level=info msg="received requests_OK : 100" time="2023-11-29T15:24:14+08:00" level=info msg="throughput (TPS) : 5000" time="2023-11-29T15:24:14+08:00" level=info msg="mean: 7575184 ns, median: 4312200 ns, max: 16251400 ns, min: 2274700 ns, p99.9: 16251400 ns" time="2023-11-29T15:24:14+08:00" level=info msg="mean: 7 ms, median: 4 ms, max: 16 ms, min: 2 ms, p99.9: 16 ms" ``` - 50个并发，共1000个请求 ``` time="2023-11-29T15:46:48+08:00" level=info msg="total: 1000 concurrency: 50 requests per client: 20" time="2023-11-29T15:46:50+08:00" level=info msg="took 1373 ms for 1000 requests" time="2023-11-29T15:46:50+08:00" level=info msg="sent requests : 1000" time="2023-11-29T15:46:50+08:00" level=info msg="received requests : 1000" time="2023-11-29T15:46:50+08:00" level=info msg="received requests_OK : 1000" time="2023-11-29T15:46:50+08:00" level=info msg="throughput (TPS) : 728" time="2023-11-29T15:46:50+08:00" level=info msg="mean: 68348635 ns, median: 59982300 ns, max: 197308000 ns, min: 3298100 ns, p99.9: 197308000 ns" time="2023-11-29T15:46:50+08:00" level=info msg="mean: 68 ms, median: 59 ms, max: 197 ms, min: 3 ms, p99.9: 197 ms" ``` - 50个并发，共10000个请求 ``` time="2023-11-29T15:47:22+08:00" level=info msg="total: 10000 concurrency: 50 requests per client: 200" time="2023-11-29T15:47:34+08:00" level=info msg="took 12819 ms for 10000 requests" time="2023-11-29T15:47:34+08:00" level=info msg="sent requests : 10000" time="2023-11-29T15:47:34+08:00" level=info msg="received requests : 10000" time="2023-11-29T15:47:34+08:00" level=info msg="received requests_OK : 10000" time="2023-11-29T15:47:34+08:00" level=info msg="throughput (TPS) : 780" time="2023-11-29T15:47:34+08:00" level=info msg="mean: 64058473 ns, median: 37056100 ns, max: 1242320700 ns, min: 1734200 ns, p99.9: 1218236450 ns" time="2023-11-29T15:47:34+08:00" level=info msg="mean: 64 ms, median: 37 ms, max: 1242 ms, min: 1 ms, p99.9: 1218 ms" ``` - 100个并发，共10000个请求 ``` time="2023-11-29T15:48:04+08:00" level=info msg="total: 10000 concurrency: 100 requests per client: 100" time="2023-11-29T15:48:13+08:00" level=info msg="took 9619 ms for 10000 requests" time="2023-11-29T15:48:13+08:00" level=info msg="sent requests : 10000" time="2023-11-29T15:48:13+08:00" level=info msg="received requests : 10000" time="2023-11-29T15:48:13+08:00" level=info msg="received requests_OK : 10000" time="2023-11-29T15:48:13+08:00" level=info msg="throughput (TPS) : 1039" time="2023-11-29T15:48:13+08:00" level=info msg="mean: 95861576 ns, median: 84699850 ns, max: 353457500 ns, min: 6367600 ns, p99.9: 346699700 ns" time="2023-11-29T15:48:13+08:00" level=info msg="mean: 95 ms, median: 84 ms, max: 353 ms, min: 6 ms, p99.9: 346 ms" ``` - 1000个并发，共10000个请求 ``` time="2023-11-29T15:48:44+08:00" level=info msg="total: 10000 concurrency: 1000 requests per client: 10" time="2023-11-29T15:48:55+08:00" level=info msg="took 11480 ms for 10000 requests" time="2023-11-29T15:48:55+08:00" level=info msg="sent requests : 10000" time="2023-11-29T15:48:55+08:00" level=info msg="received requests : 10000" time="2023-11-29T15:48:55+08:00" level=info msg="received requests_OK : 10000" time="2023-11-29T15:48:55+08:00" level=info msg="throughput (TPS) : 871" time="2023-11-29T15:48:55+08:00" level=info msg="mean: 1102012743 ns, median: 988037750 ns, max: 2135727900 ns, min: 2662200 ns, p99.9: 2129425900 ns" time="2023-11-29T15:48:55+08:00" level=info msg="mean: 1102 ms, median: 988 ms, max: 2135 ms, min: 2 ms, p99.9: 2129 ms" ``` - 10000个并发，共20000个请求 ``` time="2023-11-29T15:49:32+08:00" level=info msg="total: 20000 concurrency: 10000 requests per client: 2" time="2023-11-29T15:49:53+08:00" level=info msg="took 20497 ms for 20000 requests" time="2023-11-29T15:49:53+08:00" level=info msg="sent requests : 20000" time="2023-11-29T15:49:53+08:00" level=info msg="received requests : 20000" time="2023-11-29T15:49:53+08:00" level=info msg="received requests_OK : 20000" time="2023-11-29T15:49:53+08:00" level=info msg="throughput (TPS) : 975" time="2023-11-29T15:49:53+08:00" level=info msg="mean: 9629394346 ns, median: 9241691750 ns, max: 18458848300 ns, min: 3947500 ns, p99.9: 18437869200 ns" time="2023-11-29T15:49:53+08:00" level=info msg="mean: 9629 ms, median: 9241 ms, max: 18458 ms, min: 3 ms, p99.9: 18437 ms" ``` #### 3、元模型的map get - 10个并发，共100个请求 ``` time="2023-11-29T15:51:20+08:00" level=info msg="total: 100 concurrency: 10 requests per client: 10" time="2023-11-29T15:51:20+08:00" level=info msg="took 82 ms for 100 requests" time="2023-11-29T15:51:20+08:00" level=info msg="sent requests : 100" time="2023-11-29T15:51:20+08:00" level=info msg="received requests : 100" time="2023-11-29T15:51:20+08:00" level=info msg="received requests_OK : 100" time="2023-11-29T15:51:20+08:00" level=info msg="throughput (TPS) : 1219" time="2023-11-29T15:51:20+08:00" level=info msg="mean: 7894017 ns, median: 6708500 ns, max: 22592800 ns, min: 2172000 ns, p99.9: 21619400 ns" time="2023-11-29T15:51:20+08:00" level=info msg="mean: 7 ms, median: 6 ms, max: 22 ms, min: 2 ms, p99.9: 21 ms" ``` - 10个并发，共1000个请求 ``` time="2023-11-29T15:52:56+08:00" level=info msg="total: 1000 concurrency: 10 requests per client: 100" time="2023-11-29T15:52:56+08:00" level=info msg="took 816 ms for 1000 requests" time="2023-11-29T15:52:56+08:00" level=info msg="sent requests : 1000" time="2023-11-29T15:52:56+08:00" level=info msg="received requests : 1000" time="2023-11-29T15:52:56+08:00" level=info msg="received requests_OK : 1000" time="2023-11-29T15:52:56+08:00" level=info msg="throughput (TPS) : 1225" time="2023-11-29T15:52:56+08:00" level=info msg="mean: 8093208 ns, median: 5498600 ns, max: 37198100 ns, min: 1601400 ns, p99.9: 35485450 ns" time="2023-11-29T15:52:56+08:00" level=info msg="mean: 8 ms, median: 5 ms, max: 37 ms, min: 1 ms, p99.9: 35 ms" ``` - 10个并发，共10000个请求 ``` time="2023-11-29T15:53:21+08:00" level=info msg="total: 10000 concurrency: 10 requests per client: 1000" time="2023-11-29T15:53:31+08:00" level=info msg="took 10059 ms for 10000 requests" time="2023-11-29T15:53:31+08:00" level=info msg="sent requests : 10000" time="2023-11-29T15:53:31+08:00" level=info msg="received requests : 10000" time="2023-11-29T15:53:31+08:00" level=info msg="received requests_OK : 10000" time="2023-11-29T15:53:31+08:00" level=info msg="throughput (TPS) : 994" time="2023-11-29T15:53:31+08:00" level=info msg="mean: 10034711 ns, median: 5877000 ns, max: 851575200 ns, min: 1510200 ns, p99.9: 838274550 ns" time="2023-11-29T15:53:31+08:00" level=info msg="mean: 10 ms, median: 5 ms, max: 851 ms, min: 1 ms, p99.9: 838 ms" ``` - 50个并发，共100个请求 ``` time="2023-11-29T15:53:57+08:00" level=info msg="total: 100 concurrency: 50 requests per client: 2" time="2023-11-29T15:53:57+08:00" level=info msg="took 33 ms for 100 requests" time="2023-11-29T15:53:57+08:00" level=info msg="sent requests : 100" time="2023-11-29T15:53:57+08:00" level=info msg="received requests : 100" time="2023-11-29T15:53:57+08:00" level=info msg="received requests_OK : 100" time="2023-11-29T15:53:57+08:00" level=info msg="throughput (TPS) : 3030" time="2023-11-29T15:53:57+08:00" level=info msg="mean: 13962681 ns, median: 12463350 ns, max: 22122800 ns, min: 2119500 ns, p99.9: 22122800 ns" time="2023-11-29T15:53:57+08:00" level=info msg="mean: 13 ms, median: 12 ms, max: 22 ms, min: 2 ms, p99.9: 22 ms" ``` - 50个并发，共1000个请求 ``` time="2023-11-29T15:54:17+08:00" level=info msg="total: 1000 concurrency: 50 requests per client: 20" time="2023-11-29T15:54:17+08:00" level=info msg="took 384 ms for 1000 requests" time="2023-11-29T15:54:17+08:00" level=info msg="sent requests : 1000" time="2023-11-29T15:54:17+08:00" level=info msg="received requests : 1000" time="2023-11-29T15:54:17+08:00" level=info msg="received requests_OK : 1000" time="2023-11-29T15:54:17+08:00" level=info msg="throughput (TPS) : 2604" time="2023-11-29T15:54:17+08:00" level=info msg="mean: 19014493 ns, median: 20007500 ns, max: 39160200 ns, min: 2248800 ns, p99.9: 39160200 ns" time="2023-11-29T15:54:17+08:00" level=info msg="mean: 19 ms, median: 20 ms, max: 39 ms, min: 2 ms, p99.9: 39 ms" ``` - 50个并发，共10000个请求 ``` time="2023-11-29T15:54:41+08:00" level=info msg="total: 10000 concurrency: 50 requests per client: 200" time="2023-11-29T15:54:44+08:00" level=info msg="took 3075 ms for 10000 requests" time="2023-11-29T15:54:44+08:00" level=info msg="sent requests : 10000" time="2023-11-29T15:54:44+08:00" level=info msg="received requests : 10000" time="2023-11-29T15:54:44+08:00" level=info msg="received requests_OK : 10000" time="2023-11-29T15:54:44+08:00" level=info msg="throughput (TPS) : 3252" time="2023-11-29T15:54:44+08:00" level=info msg="mean: 15350080 ns, median: 13936100 ns, max: 74815300 ns, min: 1637300 ns, p99.9: 74309900 ns" time="2023-11-29T15:54:44+08:00" level=info msg="mean: 15 ms, median: 13 ms, max: 74 ms, min: 1 ms, p99.9: 74 ms" ``` - 100个并发，共10000个请求 ``` time="2023-11-29T15:55:08+08:00" level=info msg="total: 10000 concurrency: 100 requests per client: 100" time="2023-11-29T15:55:11+08:00" level=info msg="took 3017 ms for 10000 requests" time="2023-11-29T15:55:11+08:00" level=info msg="sent requests : 10000" time="2023-11-29T15:55:11+08:00" level=info msg="received requests : 10000" time="2023-11-29T15:55:11+08:00" level=info msg="received requests_OK : 10000" time="2023-11-29T15:55:11+08:00" level=info msg="throughput (TPS) : 3314" time="2023-11-29T15:55:11+08:00" level=info msg="mean: 30073157 ns, median: 20134900 ns, max: 942773300 ns, min: 3710100 ns, p99.9: 942152800 ns" time="2023-11-29T15:55:11+08:00" level=info msg="mean: 30 ms, median: 20 ms, max: 942 ms, min: 3 ms, p99.9: 942 ms" ``` - 1000个并发，共10000个请求 ``` time="2023-11-29T15:55:40+08:00" level=info msg="total: 10000 concurrency: 1000 requests per client: 10" time="2023-11-29T15:55:42+08:00" level=info msg="took 2423 ms for 10000 requests" time="2023-11-29T15:55:42+08:00" level=info msg="sent requests : 10000" time="2023-11-29T15:55:42+08:00" level=info msg="received requests : 10000" time="2023-11-29T15:55:42+08:00" level=info msg="received requests_OK : 10000" time="2023-11-29T15:55:42+08:00" level=info msg="throughput (TPS) : 4127" time="2023-11-29T15:55:42+08:00" level=info msg="mean: 231318655 ns, median: 223177700 ns, max: 350379500 ns, min: 4207800 ns, p99.9: 349300600 ns" time="2023-11-29T15:55:42+08:00" level=info msg="mean: 231 ms, median: 223 ms, max: 350 ms, min: 4 ms, p99.9: 349 ms" ``` - 10000个并发，共20000个请求 ``` time="2023-11-29T15:56:08+08:00" level=info msg="total: 20000 concurrency: 10000 requests per client: 2" time="2023-11-29T15:56:13+08:00" level=info msg="took 5116 ms for 20000 requests" time="2023-11-29T15:56:13+08:00" level=info msg="sent requests : 20000" time="2023-11-29T15:56:13+08:00" level=info msg="received requests : 20000" time="2023-11-29T15:56:13+08:00" level=info msg="received requests_OK : 20000" time="2023-11-29T15:56:13+08:00" level=info msg="throughput (TPS) : 3909" time="2023-11-29T15:56:13+08:00" level=info msg="mean: 2087396474 ns, median: 2025976800 ns, max: 3294130100 ns, min: 20420700 ns, p99.9: 3268514400 ns" time="2023-11-29T15:56:13+08:00" level=info msg="mean: 2087 ms, median: 2025 ms, max: 3294 ms, min: 20 ms, p99.9: 3268 ms" ``` #### 4、源码 ```golang package main import ( "context" "flag" "fmt" "sync" "sync/atomic" "time" "github.com/hazelcast/hazelcast-go-client" log "github.com/sirupsen/logrus" "go.uber.org/ratelimit" ) const key = `test_meta_model_11111122222333333` const val = ` { "appDataInfo": { "id": null, "storeAppId": null, "name": "base", "tag": "master", "displayName": "基础模块", "category": "base", "categoryDesc": "基础模块", "description": "基础模块", "application": false, "type": "SDK", "loaderType": "SDK", "company": "sie", "product": "base", "productIcon": null, "productDesc": "工业互联网平台", "summary": "基础模块", "source": "base", "resolved": "sie-snest-base-1.0-SNAPSHOT.jar", "sdkScanPkgPath": "com.sie.snest.base", "jarFile": "sie-snest-base-1.0-SNAPSHOT.jar", "jarPath": "C:\\Users\\29662\\IdeaProjects\\sie-snest\\apps\\sie-snest-base-1.0-SNAPSHOT.jar", "jarFileId": null, "md5": "57d51247cfd631a1171539dd0b3a8b80", "viewFileId": null, "viewFile": null, "viewFileMd5": null, "state": null, "storeJarFileId": null, "storeMd5": null, "dependencies": [], "icon": null, "license": "LGPL 3.0", "jsonFilePath": "com/sie/snest/base/", "jsonObject": null, "appJsonObject": { "summary": "基础模块", "product": "base", "displayName": "基础模块", "description": "基础模块", "type": "SDK", "version": "0.0.1", "categoryDesc": "基础模块", "dependencies": [], "productDesc": "工业互联网平台", "license": "LGPL 3.0", "name": "base", "company": "sie", "tag": "master", "category": "base", "events": { "startUp": [ "auth_check_job::start", "highavailable_init_menu::start" ] }, "resolved": "com.sie.snest.base" }, "author": null, "weight": 1.01, "models": null, "events": { "startUp": [ "auth_check_job::start", "highavailable_init_menu::start" ] }, "kind": "unStateful", "replicas": 1, "appInstallHosts": null, "svcName": null, "extServiceModels": null, "globalConfig": null, "appConfig": null, "delete": false, "newApp": false, "nameTag": "base.master", "baseApp": true, "primary": false }, "appMeta": null, "metas": {}, "menuMetaMap": {}, "models": {}, "refIdTreeMap": {}, "modelNameViewMetaIdMap": {} } ` var ( concurrency = flag.Int("c", 1, "concurrency") total = flag.Int("n", 10, "total requests for all clients") host = flag.String("s", "127.0.0.1:8972", "server ip and port") pool = flag.Int("pool", 10, " shared grpc clients") rate = flag.Int("r", 0, "throughputs") client = flag.String("type", "kim", "client type") ) func main() { flag.Parse() ctx := context.TODO() cfg := hazelcast.Config{} cfg.Cluster.Name = "hazelcast-benchmark" cfg.Cluster.Network.Addresses = []string{"192.168.168.176:5701"} hz, err := hazelcast.StartNewClientWithConfig(ctx, cfg) if err != nil { panic(fmt.Errorf("starting the client with config: %w", err)) } mp, err := hz.GetMap(ctx, "my-distributed-map") if err != nil { panic(fmt.Errorf("trying to get a map: %w", err)) } var rl ratelimit.Limiter if *rate > 0 { rl = ratelimit.New(*rate) } // 并发goroutine数.模拟客户端 n := *concurrency // 每个客户端需要发送的请求数 m := *total / n log.Infof("total: %d concurrency: %d requests per client: %d", *total, n, m) // 等待所有测试完成 var wg sync.WaitGroup wg.Add(n * m) // 总请求数 var trans uint64 // 返回正常的总请求数 var transOK uint64 // 每个goroutine的耗时记录 d := make([][]int64, n, n) // 栅栏，控制客户端同时开始测试 var startWg sync.WaitGroup startWg.Add(n + 1) // +1 是因为有一个goroutine用来记录开始时间 // 创建客户端 goroutine 并进行测试 startTime := time.Now().UnixNano() go func() { startWg.Done() startWg.Wait() startTime = time.Now().UnixNano() }() for i := 0; i < n; i++ { dt := make([]int64, 0, m) d = append(d, dt) go func(i int) { for j := 0; j < m; j++ { // 限流，这里不把限流的时间计算到等待耗时中 if rl != nil { rl.Take() } t := time.Now().UnixNano() _, err = mp.Get(ctx, key) if err != nil { panic(fmt.Errorf("trying to put to map: %w", err)) } t = time.Now().UnixNano() - t // 等待时间+服务时间，等待时间是客户端调度的等待时间以及服务端读取请求、调度的时间，服务时间是请求被服务处理的实际时间 d[i] = append(d[i], t) if err == nil { atomic.AddUint64(&transOK, 1) } atomic.AddUint64(&trans, 1) wg.Done() } }(i) } wg.Wait() // 统计 Stats(startTime, *total, d, trans, transOK) } // Stats 统计结果. func Stats(startTime int64, totalRequests int, tookTimes [][]int64, trans, transOK uint64) { // 测试总耗时 totalTInNano := time.Now().UnixNano() - startTime totalT := totalTInNano / 1000000 log.Infof("took %d ms for %d requests", totalT, totalRequests) // 汇总每个请求的耗时 totalD := make([]int64, 0, totalRequests) for _, k := range tookTimes { totalD = append(totalD, k...) } // 将int64数组转换成float64数组，以便分析 totalD2 := make([]float64, 0, totalRequests) for _, k := range totalD { totalD2 = append(totalD2, float64(k)) } // 计算各个指标 mean, _ := stats.Mean(totalD2) median, _ := stats.Median(totalD2) max, _ := stats.Max(totalD2) min, _ := stats.Min(totalD2) p999, _ := stats.Percentile(totalD2, 99.9) // 输出结果 log.Infof("sent requests : %d", totalRequests) log.Infof("received requests : %d", trans) log.Infof("received requests_OK : %d", transOK) if totalT == 0 { log.Infof("throughput (TPS) : %d", int64(totalRequests)*1000*1000000/totalTInNano) } else { log.Infof("throughput (TPS) : %d", int64(totalRequests)*1000/totalT) } log.Infof("mean: %.f ns, median: %.f ns, max: %.f ns, min: %.f ns, p99.9: %.f ns", mean, median, max, min, p999) log.Infof("mean: %d ms, median: %d ms, max: %d ms, min: %d ms, p99.9: %d ms", int64(mean/1000000), int64(median/1000000), int64(max/1000000), int64(min/1000000), int64(p999/1000000)) } ``` ### 5，问题 - 序列化
Uploading file...
Sidebar
[[_TOC_]]
Edit message:
Cancel