dubbo provider中的服務配置項有接近30個配置項。 排除注冊中心服務治理需要之外,很大一部分配置項是provider自己使用,不需要透傳給消費者。這部分數(shù)據(jù)不需要進入注冊中心,而只需要以key-value形式持久化存儲。 dubbo consumer中的配置項也有20+個配置項。在注冊中心之中,服務消費者列表中只需要關注application,version,group,ip,dubbo版本等少量配置,其他配置也可以以key-value形式持久化存儲。 這些數(shù)據(jù)是以服務為維度注冊進入注冊中心,導致了數(shù)據(jù)量的膨脹,進而引發(fā)注冊中心(如zookeeper)的網(wǎng)絡開銷增大,性能降低。除了上述配置項的存儲之外,dubbo服務元數(shù)據(jù)信息也需要被存儲下來。元數(shù)據(jù)信息包括服務接口,及接口的方法信息。這些信息將被用于服務mock,服務測試。
以上的元數(shù)據(jù)都是基于接口級別。在3.0版本中,引入了應用元數(shù)據(jù)的概念,應用元數(shù)據(jù)描述的是整個應用的信息概覽。并且引入了服務自省映射,用于應用級別的服務發(fā)現(xiàn)。
需要將注冊中心原來的數(shù)據(jù)信息和元數(shù)據(jù)信息保存到獨立的key-value的存儲中,這個key-value可以是DB,redis或者其他持久化存儲。核心代碼中支持了zookeeper,redis, nacos(推薦)的默認支持。
因為是基于key-value存儲,key不會改變,最新的value會將原來的value進行覆蓋
Provider存儲內(nèi)容的格式,參見:org.apache.dubbo.metadata.definition.model.FullServiceDefinition。是該類型gson化之后的存儲。 Consumer存儲內(nèi)容,為Map格式。從Consumer端注冊到注冊中心的URL中的獲取參數(shù)信息。即通過URL.getParameterMap()獲取到的Map,進行gson化之后進行存儲。
詳細的內(nèi)容,可以參考下面的sample輸出。
默認的元數(shù)據(jù)存儲,額外支持以下幾個特性:
失敗重試可以通過retrytimes (重試次數(shù),默認100),retryperiod(重試周期,默認3000ms)進行設置。
默認開啟,可以通過設置cycleReport=false進行關閉。
dubbo.metadata-report.address=zookeeper://127.0.0.1:2181 dubbo.metadata-report.username=xxx ##非必須 dubbo.metadata-report.password=xxx ##非必須 dubbo.metadata-report.retry-times=30 ##非必須,default值100 dubbo.metadata-report.retry-period=5000 ##非必須,default值3000 dubbo.metadata-report.cycle-report=false ##非必須,default值true dubbo.metadata-report.sync.report=false ##非必須,default值為false
如果元數(shù)據(jù)地址(dubbo.metadata-report.address)也不進行配置,會判斷注冊中心的協(xié)議是否支持元數(shù)據(jù)中心,如果支持,會使用注冊中心的地址來用作元數(shù)據(jù)中心。
接下來看幾個sample的配置。無論哪種配置方式,都需要引入maven依賴:
zookeeper:
<dependency> <groupId>org.apache.dubbo</groupId> <artifactId>dubbo-metadata-report-zookeeper</artifactId> </dependency>
redis:
<dependency> <groupId>org.apache.dubbo</groupId> <artifactId>dubbo-metadata-report-redis</artifactId> </dependency>
nacos:
<dependency> <groupId>org.apache.dubbo</groupId> <artifactId>dubbo-metadata-report-nacos</artifactId> </dependency>
完整的sample,查看sample-2.7
參考sample:dubbo-samples-metadata-report/dubbo-samples-metadata-report-configcenter 工程。
配置中心的配置,可以參考configcenter的文檔。配置的內(nèi)容如下:
dubbo.registry.address=zookeeper://127.0.0.1:2181 ### 注意駝峰式風格 dubbo.metadata-report.address=zookeeper://127.0.0.1:2181 ###元數(shù)據(jù)存儲的地址
在sample中,使用了Zookeeper作為配置中心。啟動本地zookeeper服務之后,直接運行:org.apache.dubbo.samples.metadatareport.configcenter.ZKTools 就可以完成寫入。 如果配置中心使用了nacos,apollo,這些產(chǎn)品本身支持ops配置。
###dubbo.properties dubbo.config-center.address=zookeeper://127.0.0.1:2181 ...
完成上述兩步之后,注冊中心地址、元數(shù)據(jù)地址將從配置中心進行獲取?,F(xiàn)在可以依次運行Provider類和Consumer類,會在console中得到對應的輸出或者直接通過zookeeper的cli查看。
provider端存儲的元數(shù)據(jù)內(nèi)容如下:
{ "parameters": { "side": "provider", "methods": "sayHello", "dubbo": "2.0.2", "threads": "100", "interface": "org.apache.dubbo.samples.metadatareport.configcenter.api.AnnotationService", "threadpool": "fixed", "version": "1.1.1", "generic": "false", "revision": "1.1.1", "valid": "true", "application": "metadatareport-configcenter-provider", "default.timeout": "5000", "group": "d-test", "anyhost": "true" }, "canonicalName": "org.apache.dubbo.samples.metadatareport.configcenter.api.AnnotationService", "codeSource": "file:/Users/cvictory/workspace/work-mw/dubbo-samples/dubbo-samples-metadata-report/dubbo-samples-metadata-report-configcenter/target/classes/", "methods": [{ "name": "sayHello", "parameterTypes": ["java.lang.String"], "returnType": "java.lang.String" }], "types": [{ "type": "java.lang.String", "properties": { "value": { "type": "char[]" }, "hash": { "type": "int" } } }, { "type": "int" }, { "type": "char" }] }
provider存儲的內(nèi)容包括了provider服務往注冊中心填寫的全部參數(shù),以及服務的方法信息(方法名,入?yún)⒊鰠⒌母袷剑?/p>
{ "valid": "true", "side": "consumer", "application": "metadatareport-configcenter-consumer", "methods": "sayHello", "default.timeout": "6666", "dubbo": "2.0.2", "interface": "org.apache.dubbo.samples.metadatareport.configcenter.api.AnnotationService", "version": "1.1.1", "revision": "1.1.1", "group": "d-test" }
consumer端存儲了consumer往注冊中心填寫的全部參數(shù)。
上面的例子,主要是將元數(shù)據(jù)地址放在配置中心,在元數(shù)據(jù)區(qū)存儲下來的provider端服務信息和consumer端服務信息的展示。 接下來的兩個例子,主要講解在工程中配置:xml方式,annotation方式。
參考sample:dubbo-samples-metadata-report/dubbo-samples-metadata-report-local-properties工程。
dubbo.metadata-report.address=zookeeper://127.0.0.1:2181
配置完成這個之后,其余的不用特別關注。也可以直接查看對應的provider和consumer端的服務信息。
{ "parameters": { "valid": "true", "async": "true", "side": "provider", "application": "metadatareport-local-xml-provider", "methods": "sayHello", "dubbo": "2.0.2", "interface": "org.apache.dubbo.samples.metadatareport.local.xml.api.DemoService", "generic": "false", "anyhost": "true" }, "canonicalName": "org.apache.dubbo.samples.metadatareport.local.xml.api.DemoService", "codeSource": "file:/Users/cvictory/workspace/work-mw/dubbo-samples/dubbo-samples-metadata-report/dubbo-samples-metadata-report-local-xml/target/classes/", "methods": [{ "name": "sayHello", "parameterTypes": ["java.lang.String"], "returnType": "java.lang.String" }], "types": [{ "type": "int" }, { "type": "char" }, { "type": "java.lang.String", "properties": { "value": { "type": "char[]" }, "hash": { "type": "int" } } }] }
{ "valid": "true", "side": "consumer", "application": "metadatareport-local-xml-consumer", "methods": "sayHello", "dubbo": "2.0.2", "interface": "org.apache.dubbo.samples.metadatareport.local.xml.api.DemoService" }
參考sample:dubbo-samples-metadata-report/dubbo-samples-metadata-report-local-annotaion工程。
@Bean public MetadataReportConfig metadataReportConfig() { MetadataReportConfig metadataReportConfig = new MetadataReportConfig(); metadataReportConfig.setAddress("zookeeper://127.0.0.1:2181"); return metadataReportConfig; }
引入Bean之后,其余的地方也不需要特別配置。直接查看對應的服務信息:
{ "parameters": { "side": "provider", "methods": "sayHello", "dubbo": "2.0.2", "interface": "org.apache.dubbo.samples.metadatareport.local.annotation.api.AnnotationService", "version": "1.1.8", "generic": "false", "revision": "1.1.8", "valid": "true", "application": "metadatareport-local-annotaion-provider", "default.timeout": "1000", "group": "d-test", "anyhost": "true" }, "canonicalName": "org.apache.dubbo.samples.metadatareport.local.annotation.api.AnnotationService", "codeSource": "file:/Users/cvictory/workspace/work-mw/dubbo-samples/dubbo-samples-metadata-report/dubbo-samples-metadata-report-local-annotaion/target/classes/", "methods": [{ "name": "sayHello", "parameterTypes": ["java.lang.String"], "returnType": "java.lang.String" }], "types": [{ "type": "int" }, { "type": "java.lang.String", "properties": { "value": { "type": "char[]" }, "hash": { "type": "int" } } }, { "type": "char" }] }
{ "valid": "true", "side": "consumer", "application": "metadatareport-local-annotaion-consumer", "methods": "sayHello", "dubbo": "2.0.2", "interface": "org.apache.dubbo.samples.metadatareport.local.annotation.api.AnnotationService", "version": "1.1.8", "revision": "1.1.8", "group": "d-test" }
參考:org.apache.dubbo.metadata.store.MetadataReportFactory , org.apache.dubbo.metadata.store.MetadataReport
@SPI("redis") public interface MetadataReportFactory { @Adaptive({"protocol"}) MetadataReport getMetadataReport(URL url); }
下面以Redis存儲為例進行說明。
新建一個project,需要支持以下修改:
public class RedisMetadataReport extends AbstractMetadataReport { private final static Logger logger = LoggerFactory.getLogger(RedisMetadataReport.class); final JedisPool pool; public RedisMetadataReport(URL url) { super(url); pool = new JedisPool(new JedisPoolConfig(), url.getHost(), url.getPort()); } @Override protected void doStoreProviderMetadata(ProviderMetadataIdentifier providerMetadataIdentifier, String serviceDefinitions) { this.storeMetadata(providerMetadataIdentifier, serviceDefinitions); } @Override protected void doStoreConsumerMetadata(ConsumerMetadataIdentifier consumerMetadataIdentifier, String value) { this.storeMetadata(consumerMetadataIdentifier, value); } private void storeMetadata(MetadataIdentifier metadataIdentifier, String v) { try (Jedis jedis = pool.getResource()) { jedis.set(metadataIdentifier.getIdentifierKey() + META_DATA_SOTRE_TAG, v); } catch (Throwable e) { logger.error("Failed to put " + metadataIdentifier + " to redis " + v + ", cause: " + e.getMessage(), e); throw new RpcException("Failed to put " + metadataIdentifier + " to redis " + v + ", cause: " + e.getMessage(), e); } } }
public class RedisMetadataReportFactory extends AbstractMetadataReportFactory { @Override public MetadataReport createMetadataReport(URL url) { return new RedisMetadataReport(url); } }
META-INF/dubbo/internal/org.apache.dubbo.metadata.store.MetadataReportFactory
redis=org.apache.dubbo.metadata.store.redis.RedisMetadataReportFactory
只要將上面的修改和project打包成jar包,然后配置元數(shù)據(jù)中心的url:redis://10.20.153.10:6379。
至此,一個自定義的元數(shù)據(jù)存儲就可以運行了。
<dubbo:metadata-report address="zookeeper://127.0.0.1:2181"/>
Zookeeper 基于樹形結構進行數(shù)據(jù)存儲,它的元數(shù)據(jù)信息位于以下節(jié)點:
Provider: /dubbo/metadata/{interface name}/{version}/{group}/provider/{application name} Consumer: /dubbo/metadata/{interface name}/{version}/{group}/consumer/{application name}
當 version 或者 group 不存在時,version 路徑和 group 路徑會取消,路徑如下:
Provider: /dubbo/metadata/{interface name}/provider/{application name} Consumer: /dubbo/metadata/{interface name}/consumer/{application name}
通過 zkCli get 操作查看數(shù)據(jù).
Provider node:
[zk: localhost:2181(CONNECTED) 8] get /dubbo/metadata/org.apache.dubbo.demo.DemoService/provider/demo-provider {"parameters":{"side":"provider","interface":"org.apache.dubbo.demo.DemoService","metadata-type":"remote","application":"demo-provider","dubbo":"2.0.2","release":"","anyhost":"true","delay":"5000","methods":"sayHello,sayHelloAsync","deprecated":"false","dynamic":"true","timeout":"3000","generic":"false"},"canonicalName":"org.apache.dubbo.demo.DemoService","codeSource":"file:/Users/apple/IdeaProjects/dubbo/dubbo-demo/dubbo-demo-interface/target/classes/","methods":[{"name":"sayHelloAsync","parameterTypes":["java.lang.String"],"returnType":"java.util.concurrent.CompletableFuture"},{"name":"sayHello","parameterTypes":["java.lang.String"],"returnType":"java.lang.String"}],"types":[{"type":"java.util.concurrent.CompletableFuture","properties":{"result":"java.lang.Object","stack":"java.util.concurrent.CompletableFuture.Completion"}},{"type":"java.lang.Object"},{"type":"java.lang.String"},{"type":"java.util.concurrent.CompletableFuture.Completion","properties":{"next":"java.util.concurrent.CompletableFuture.Completion","status":"int"}},{"type":"int"}]} cZxid = 0x25a9b1 ctime = Mon Jun 28 21:35:17 CST 2021 mZxid = 0x25a9b1 mtime = Mon Jun 28 21:35:17 CST 2021 pZxid = 0x25a9b1 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 1061 numChildren = 0
Consumer node:
[zk: localhost:2181(CONNECTED) 10] get /dubbo/metadata/org.apache.dubbo.demo.DemoService/consumer/demo-consumer {"side":"consumer","interface":"org.apache.dubbo.demo.DemoService","metadata-type":"remote","application":"demo-consumer","dubbo":"2.0.2","release":"","sticky":"false","check":"false","methods":"sayHello,sayHelloAsync"} cZxid = 0x25aa24 ctime = Mon Jun 28 21:57:43 CST 2021 mZxid = 0x25aa24 mtime = Mon Jun 28 21:57:43 CST 2021 pZxid = 0x25aa24 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 219 numChildren = 0
<dubbo:metadata-report address="redis://127.0.0.1:6779"/>
在Redis中,使用string數(shù)據(jù)結構來進行存儲元數(shù)據(jù)信息:
Provider: {service name}:{version}:{group}:provider:{application name} Consumer: {service name}:{version}:{group}:consumer:{application name}
當 version 或者 group 不存在時,: 依然保留:
Provider: {service name}:::provider:{application name} Consumer: {service name}:::consumer:{application name}
通過 Redis client get key 查看數(shù)據(jù).
Provider key:
127.0.0.1:6379> get org.apache.dubbo.demo.DemoService:::provider:demo-provider "{\"parameters\":{\"side\":\"provider\",\"interface\":\"org.apache.dubbo.demo.DemoService\",\"metadata-type\":\"remote\",\"application\":\"demo-provider\",\"dubbo\":\"2.0.2\",\"release\":\"\",\"anyhost\":\"true\",\"delay\":\"5000\",\"methods\":\"sayHello,sayHelloAsync\",\"deprecated\":\"false\",\"dynamic\":\"true\",\"timeout\":\"3000\",\"generic\":\"false\"},\"canonicalName\":\"org.apache.dubbo.demo.DemoService\",\"codeSource\":\"file:/Users/apple/IdeaProjects/dubbo/dubbo-demo/dubbo-demo-interface/target/classes/\",\"methods\":[{\"name\":\"sayHello\",\"parameterTypes\":[\"java.lang.String\"],\"returnType\":\"java.lang.String\"},{\"name\":\"sayHelloAsync\",\"parameterTypes\":[\"java.lang.String\"],\"returnType\":\"java.util.concurrent.CompletableFuture\"}],\"types\":[{\"type\":\"java.util.concurrent.CompletableFuture\",\"properties\":{\"result\":\"java.lang.Object\",\"stack\":\"java.util.concurrent.CompletableFuture.Completion\"}},{\"type\":\"java.lang.Object\"},{\"type\":\"java.lang.String\"},{\"type\":\"java.util.concurrent.CompletableFuture.Completion\",\"properties\":{\"next\":\"java.util.concurrent.CompletableFuture.Completion\",\"status\":\"int\"}},{\"type\":\"int\"}]}"
Consumer key:
127.0.0.1:6379> get org.apache.dubbo.demo.DemoService:::consumer:demo-consumer "{\"side\":\"consumer\",\"interface\":\"org.apache.dubbo.demo.DemoService\",\"metadata-type\":\"remote\",\"application\":\"demo-consumer\",\"dubbo\":\"2.0.2\",\"release\":\"\",\"sticky\":\"false\",\"check\":\"false\",\"methods\":\"sayHello,sayHelloAsync\"}"
<dubbo:metadata-report address="nacos://127.0.0.1:8848"/>
在 Nacos 中,本身就存在配置中心這個概念,正好用于元數(shù)據(jù)存儲。在配置中心的場景下,存在命名空間- namespace 的概念,在 namespace 之下,還存在 group 概念。即通過 namespace 和 group 以及 dataId 去定位一個配置項,在不指定 namespace 的情況下,默認使用 public 作為默認的命名空間。
Provider: namespace: 'public', dataId: '{service name}:{version}:{group}:provider:{application name}', group: 'dubbo' Consumer: namespace: 'public', dataId: '{service name}:{version}:{group}:consumer:{application name}', group: 'dubbo'
當 version 或者 group 不存在時,: 依然保留:
Provider: namespace: 'public', dataId: '{service name}:::provider:{application name}', group: 'dubbo' Consumer: namespace: 'public', dataId: '{service name}:::consumer:{application name}', group: 'dubbo'
可以通過 Nacos 自帶的 web console 界面進行查看.
Provider data:
Consumer data:
應用級別元數(shù)據(jù)只有當一個應用定義服務之后,才會進行暴露。會根據(jù)當前應用的自身信息,以及接口信息,去計算出該應用的 revision 修訂值,用于保存應用級別元數(shù)據(jù),
Zookeeper 的應用級別元數(shù)據(jù)位于 /dubbo/metadata/{application name}/{revision}
[zk: localhost:2181(CONNECTED) 33] get /dubbo/metadata/demo-provider/da3be833baa2088c5f6776fb7ab1a436 {"app":"demo-provider","revision":"da3be833baa2088c5f6776fb7ab1a436","services":{"org.apache.dubbo.demo.DemoService:dubbo":{"name":"org.apache.dubbo.demo.DemoService","protocol":"dubbo","path":"org.apache.dubbo.demo.DemoService","params":{"side":"provider","release":"","methods":"sayHello,sayHelloAsync","deprecated":"false","dubbo":"2.0.2","pid":"38298","interface":"org.apache.dubbo.demo.DemoService","service-name-mapping":"true","timeout":"3000","generic":"false","metadata-type":"remote","delay":"5000","application":"demo-provider","dynamic":"true","REGISTRY_CLUSTER":"registry1","anyhost":"true","timestamp":"1626887121829"}},"org.apache.dubbo.demo.RestDemoService:1.0.0:rest":{"name":"org.apache.dubbo.demo.RestDemoService","version":"1.0.0","protocol":"rest","path":"org.apache.dubbo.demo.RestDemoService","params":{"side":"provider","release":"","methods":"getRemoteApplicationName,sayHello,hello,error","deprecated":"false","dubbo":"2.0.2","pid":"38298","interface":"org.apache.dubbo.demo.RestDemoService","service-name-mapping":"true","version":"1.0.0","timeout":"5000","generic":"false","revision":"1.0.0","metadata-type":"remote","delay":"5000","application":"demo-provider","dynamic":"true","REGISTRY_CLUSTER":"registry1","anyhost":"true","timestamp":"1626887120943"}}}} cZxid = 0x25b336 ctime = Thu Jul 22 01:05:55 CST 2021 mZxid = 0x25b336 mtime = Thu Jul 22 01:05:55 CST 2021 pZxid = 0x25b336 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 1286 numChildren = 0
Redis 元數(shù)據(jù)中心目前還不支持應用級別元數(shù)據(jù),但已提上日程,會在近期進行實現(xiàn)。
Nacos 應用級別的元數(shù)據(jù)位于 namespace: ‘public’, dataId: ‘{application name}’, group: ‘{revision}’
在Dubbo 3.0 中,默認使用了服務自省機制去實現(xiàn)服務發(fā)現(xiàn),關于服務自省可以查看服務自省
簡而言之,服務自省機制需要能夠通過 interface name 去找到對應的 application name,這個關系可以是一對多的,即一個 service name 可能會對應多個不同的 application name。在 3.0 中,元數(shù)據(jù)中心提供此項映射的能力。
在上面提到,service name 和 application name 可能是一對多的,在 zookeeper 中,使用單個 key-value 進行保存,多個 application name 通過英文逗號,隔開。由于是單個 key-value 去保存數(shù)據(jù),在多客戶端的情況下可能會存在并發(fā)覆蓋的問題。因此,我們使用 zookeeper 中的版本機制 version 去解決該問題。在 zookeeper 中,每一次對數(shù)據(jù)進行修改,dataVersion 都會進行增加,我們可以利用 version 這個機制去解決多個客戶端同時更新映射的并發(fā)問題。不同客戶端在更新之前,先去查一次 version,當作本地憑證。在更新時,把憑證 version 傳到服務端比對 version, 如果不一致說明在次期間被其他客戶端修改過,重新獲取憑證再進行重試(CAS)。目前如果重試6次都失敗的話,放棄本次更新映射行為。
Curator api.
CuratorFramework client = ... client.setData().withVersion(ticket).forPath(path, dataBytes);
映射信息位于:
/dubbo/mapping/{service name}
通過 zkCli get 操作查看數(shù)據(jù).
[zk: localhost:2181(CONNECTED) 26] get /dubbo/mapping/org.apache.dubbo.demo.DemoService demo-provider,two-demo-provider,dubbo-demo-annotation-provider cZxid = 0x25a80f ctime = Thu Jun 10 01:36:40 CST 2021 mZxid = 0x25a918 mtime = Fri Jun 11 18:46:40 CST 2021 pZxid = 0x25a80f cversion = 0 dataVersion = 2 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 62 numChildren = 0
Redis 元數(shù)據(jù)中心目前還不支持服務自省映射,但已提上日程,會在近期進行實現(xiàn)。
在上面提到,service name 和 application name 可能是一對多的,在 nacos 中,使用單個 key-value 進行保存,多個 application name 通過英文逗號,隔開。由于是單個 key-value 去保存數(shù)據(jù),在多客戶端的情況下可能會存在并發(fā)覆蓋的問題。因此,我們使用 nacos 中 publishConfigCas 的能力去解決該問題。在 nacos 中,使用 publishConfigCas 會讓用戶傳遞一個參數(shù) casMd5,該值的含義是之前配置內(nèi)容的 md5 值。不同客戶端在更新之前,先去查一次 nacos 的 content 的值,計算出 md5 值,當作本地憑證。在更新時,把憑證 md5 傳到服務端比對 md5 值, 如果不一致說明在次期間被其他客戶端修改過,重新獲取憑證再進行重試(CAS)。目前如果重試6次都失敗的話,放棄本次更新映射行為。
Nacos api:
ConfigService configService = ... configService.publishConfigCas(key, group, content, ticket);
映射信息位于 namespace: ‘public’, dataId: ‘{service name}’, group: ‘mapping’.
更多建議: