I am trying to create custom XML rules but they are not working. :(
First my configuration :
sonarqube 4.3
xml plugin 1.2
I want to create a custom rule in xpath for an XML Quality Profile.
The rule is never applied and no violation is detected.
My rule :
//ELEMENT/S_TAG/NAME[#tokenValue='root']
My XML file :
<?xml version="1.0" encoding="UTF-8" ?>
<root>
</root>
The AST tree :
<DOCUMENT tokenValue="<?xml" tokenLine="1" tokenColumn="0">
<PROLOG tokenValue="<?xml" tokenLine="1" tokenColumn="0">
<XML_DECL tokenValue="<?xml" tokenLine="1" tokenColumn="0">
<TOKEN tokenValue="<?xml" tokenLine="1" tokenColumn="0"/>
<VERSION_INFO tokenValue="version" tokenLine="1" tokenColumn="6">
<TOKEN tokenValue="version" tokenLine="1" tokenColumn="6"/>
<EQ tokenValue="=" tokenLine="1" tokenColumn="13">
<TOKEN tokenValue="=" tokenLine="1" tokenColumn="13"/>
</EQ>
<TOKEN tokenValue=""" tokenLine="1" tokenColumn="14"/>
<VERSION_NUM tokenValue="1." tokenLine="1" tokenColumn="15">
<TOKEN tokenValue="1." tokenLine="1" tokenColumn="15"/>
<TOKEN tokenValue="0" tokenLine="1" tokenColumn="17"/>
</VERSION_NUM>
<TOKEN tokenValue=""" tokenLine="1" tokenColumn="18"/>
</VERSION_INFO>
<ENCODING_DECL tokenValue="encoding" tokenLine="1" tokenColumn="20">
<TOKEN tokenValue="encoding" tokenLine="1" tokenColumn="20"/>
<EQ tokenValue="=" tokenLine="1" tokenColumn="28">
<TOKEN tokenValue="=" tokenLine="1" tokenColumn="28"/>
</EQ>
<TOKEN tokenValue=""" tokenLine="1" tokenColumn="29"/>
<ENC_NAME tokenValue="UTF-8" tokenLine="1" tokenColumn="30">
<TOKEN tokenValue="UTF-8" tokenLine="1" tokenColumn="30"/>
</ENC_NAME>
<TOKEN tokenValue=""" tokenLine="1" tokenColumn="35"/>
</ENCODING_DECL>
<TOKEN tokenValue="?>" tokenLine="1" tokenColumn="38"/>
</XML_DECL>
<MISC/>
</PROLOG>
<ELEMENT tokenValue="<" tokenLine="3" tokenColumn="0">
<S_TAG tokenValue="<" tokenLine="3" tokenColumn="0">
<TOKEN tokenValue="<" tokenLine="3" tokenColumn="0"/>
<NAME tokenValue="root" tokenLine="3" tokenColumn="1">
<TOKEN tokenValue="root" tokenLine="3" tokenColumn="1"/>
</NAME>
<TOKEN tokenValue=">" tokenLine="3" tokenColumn="5"/>
</S_TAG>
<CONTENT tokenValue="
" tokenLine="3" tokenColumn="6">
<CHAR_DATA tokenValue="
" tokenLine="3" tokenColumn="6">
<TOKEN tokenValue="
" tokenLine="3" tokenColumn="6"/>
</CHAR_DATA>
</CONTENT>
<E_TAG tokenValue="</" tokenLine="5" tokenColumn="0">
<TOKEN tokenValue="</" tokenLine="5" tokenColumn="0"/>
<NAME tokenValue="root" tokenLine="5" tokenColumn="2">
<TOKEN tokenValue="root" tokenLine="5" tokenColumn="2"/>
</NAME>
<TOKEN tokenValue=">" tokenLine="5" tokenColumn="6"/>
</E_TAG>
</ELEMENT>
<MISC/>
</DOCUMENT>
And the log of the analysis :
Retrieve remote issues of project prjtest...
Start SonarQube analysis on prjtest...
INFO: SonarQube Server 4.3
09:26:29.309 INFO - Preview mode
09:26:29.315 INFO - Load batch settings
09:26:29.329 DEBUG - Download: http://xxxx/sonar/batch_bootstrap/properties?dryRun=true (no proxy)
09:26:29.554 INFO - User cache: C:\Users\xxxxx-s\.sonar\cache
09:26:29.567 INFO - Install plugins
09:26:29.567 DEBUG - Download index of plugins
09:26:29.567 DEBUG - Download: http://xxxxxx/sonar/deploy/plugins/index.txt (no proxy)
09:26:29.598 INFO - Include plugins:
09:26:29.599 INFO - Exclude plugins: devcockpit, buildstability, pdfreport, report, buildbreaker, scmactivity, views, jira
09:26:29.847 DEBUG - Download: http://xxxx/sonar/batch_bootstrap/db?project=fr.sihm.eand:prjtest (no proxy)
09:26:30.111 DEBUG - Dry Run database size: 1 MB
09:26:30.119 INFO - Create JDBC datasource for jdbc:h2:D:\java\workspaces\wks-luna-SR1-01-Sonar\.metadata\.plugins\org.eclipse.core.resources\.projects\prjtest\org.sonar.ide.eclipse.core\.sonartmp\preview1414743989840-0
09:26:30.845 DEBUG - Testing JDBC connection
09:26:32.049 INFO - Initializing Hibernate
09:26:32.053 DEBUG - hibernate.generate_statistics: false
09:26:32.053 DEBUG - hibernate.dialect: org.hibernate.dialect.H2Dialect
09:26:32.053 DEBUG - hibernate.connection.provider_class: org.sonar.jpa.session.CustomHibernateConnectionProvider
09:26:33.503 DEBUG - Loaded 3276 properties from l10n bundles
09:26:35.161 INFO - Load project settings
09:26:35.161 DEBUG - Download: http://xxxxxx/sonar/batch_bootstrap/properties?project=fr.sihm.eand:prjtest&dryRun=true (no proxy)
09:26:35.216 INFO - Apply project exclusions
09:26:35.452 DEBUG - ooo Using Connection [jdbc:h2:D:\java\workspaces\wks-luna-SR1-01-Sonar\.metadata\.plugins\org.eclipse.core.resources\.projects\prjtest\org.sonar.ide.eclipse.core\.sonartmp\preview1414743989840-0, UserName=SONAR, H2 JDBC Driver]
09:26:35.457 DEBUG - ==> Preparing: select * from projects p where p.id=?
09:26:35.482 DEBUG - ==> Parameters: 14701(Long)
09:26:35.734 INFO - ------------- Scan prjtest
09:26:35.763 INFO - Load module settings
09:26:35.797 DEBUG - Download: http://xxxx/sonar/batch_bootstrap/properties?project=fr.sihm.eand:prjtest&dryRun=true (no proxy)
09:26:36.162 DEBUG - Available languages:
09:26:36.163 DEBUG - * Java => "java"
09:26:36.163 DEBUG - * Web => "web"
09:26:36.163 DEBUG - * XML => "xml"
09:26:36.163 DEBUG - * JavaScript => "js"
09:26:37.142 INFO - Loading technical debt model...
09:26:37.181 INFO - Loading technical debt model done: 39 ms
09:26:37.204 INFO - Loading rules...
09:26:37.711 INFO - Loading rules done: 507 ms
09:26:37.780 INFO - Configure Maven plugins
09:26:37.970 INFO - Compare to previous analysis (2014-10-30)
09:26:37.991 INFO - Compare over 30 days (2014-10-01, analysis of 2014-10-23 15:42:10.95)
09:26:37.993 INFO - No quality gate is configured.
09:26:38.340 DEBUG - Initializers :
09:26:38.340 INFO - Base dir: D:\001-Projets\Sonar\prjtest
09:26:38.340 INFO - Working dir: D:\java\workspaces\wks-luna-SR1-01-Sonar\.metadata\.plugins\org.eclipse.core.resources\.projects\prjtest\org.sonar.ide.eclipse.core
09:26:38.340 INFO - Source dirs: D:\001-Projets\Sonar\prjtest\src
09:26:38.340 INFO - Binary dirs: D:\001-Projets\Sonar\prjtest\bin
09:26:38.340 INFO - Source encoding: windows-1252, default locale: fr_FR
09:26:38.341 INFO - Index files
09:26:38.346 DEBUG - Declared extensions of language Java were converted to sonar.lang.patterns.java : **/*.java,**/*.jav
09:26:38.346 DEBUG - Declared extensions of language Web were converted to sonar.lang.patterns.web : **/*.html,**/*.xhtml,**/*.jsp,**/*.jspf,**/*.jsf,**/*.php,**/*.erb,**/*.rhtml
09:26:38.346 DEBUG - Declared extensions of language XML were converted to sonar.lang.patterns.xml : **/*.xml
09:26:38.347 DEBUG - Declared extensions of language JavaScript were converted to sonar.lang.patterns.js : **/*.js
09:26:38.438 DEBUG - Language of file 'src/Jx_bad.xml' is detected to be 'xml'
09:26:38.542 DEBUG - Language of file 'src/Jx_G_J_Ds2MoJoDs_good.xml' is detected to be 'xml'
09:26:38.544 DEBUG - Language of file 'src/NewFile.xml' is detected to be 'xml'
09:26:38.545 INFO - 3 files indexed
09:26:38.669 INFO - Quality profile for xml: SIHM - DTS - Général
09:26:38.726 DEBUG - Sensors : QProfileSensor -> CpdSensor -> XmlSensor -> LineCountSensor -> JaCoCoSensor -> JaCoCoItSensor -> JaCoCoOverallSensor -> InitialOpenIssuesSensor -> ProfileEventsSensor -> ProjectLinksSensor
09:26:38.727 INFO - Sensor QProfileSensor...
09:26:38.739 INFO - Sensor QProfileSensor done: 12 ms
09:26:38.740 INFO - Sensor CpdSensor...
09:26:38.740 DEBUG - Detection of duplicated code is not supported for xml
09:26:38.740 INFO - Sensor CpdSensor done: 0 ms
09:26:38.740 INFO - Sensor XmlSensor...
09:26:39.126 INFO - Sensor XmlSensor done: 386 ms
09:26:39.127 INFO - Sensor LineCountSensor...
09:26:39.137 DEBUG - Count comment in D:\001-Projets\Sonar\prjtest\src\Jx_G_J_Ds2MoJoDs_good.xml
09:26:39.167 DEBUG - LineCountSensor: src/Jx_G_J_Ds2MoJoDs_good.xml:920,17,0
09:26:39.168 DEBUG - Count comment in D:\001-Projets\Sonar\prjtest\src\Jx_bad.xml
09:26:39.172 DEBUG - LineCountSensor: src/Jx_bad.xml:10,0,0
09:26:39.174 DEBUG - Count comment in D:\001-Projets\Sonar\prjtest\src\NewFile.xml
09:26:39.177 DEBUG - LineCountSensor: src/NewFile.xml:5,2,0
09:26:39.177 INFO - Sensor LineCountSensor done: 51 ms
09:26:39.177 INFO - Sensor JaCoCoSensor...
09:26:39.188 INFO - Project coverage is set to 0% as no JaCoCo execution data has been dumped: D:\001-Projets\Sonar\prjtest\target\jacoco.exec
09:26:39.205 WARN - Coverage information was not collected. Perhaps you forget to include debug information into compiled classes?
09:26:39.205 INFO - Sensor JaCoCoSensor done: 28 ms
09:26:39.206 INFO - Sensor JaCoCoItSensor...
09:26:39.208 INFO - Project coverage is set to 0% as no JaCoCo execution data has been dumped: D:\001-Projets\Sonar\prjtest\target\jacoco-it.exec
09:26:39.209 WARN - Coverage information was not collected. Perhaps you forget to include debug information into compiled classes?
09:26:39.209 INFO - Sensor JaCoCoItSensor done: 3 ms
09:26:39.209 INFO - Sensor JaCoCoOverallSensor...
09:26:39.209 INFO - Sensor JaCoCoOverallSensor done: 0 ms
09:26:39.209 INFO - Sensor InitialOpenIssuesSensor...
09:26:39.232 INFO - Sensor InitialOpenIssuesSensor done: 23 ms
09:26:39.233 INFO - Sensor ProfileEventsSensor...
09:26:39.266 INFO - Sensor ProfileEventsSensor done: 33 ms
09:26:39.266 INFO - Sensor ProjectLinksSensor...
09:26:39.285 INFO - Sensor ProjectLinksSensor done: 19 ms
09:26:39.436 INFO - Execute decorators...
09:26:39.437 DEBUG - Decorators: ManualMeasureDecorator -> org.sonar.issuesreport.tree.ResourceTree#480994d3 -> SumDuplicationsDecorator -> org.sonar.plugins.design.batch.ProjectDsmDecorator#7146c6ea -> FileTangleIndexDecorator -> org.sonar.plugins.core.issue.IssueTrackingDecorator#a5d23c9 -> UnitTestDecorator -> org.sonar.plugins.core.sensors.DirectoriesDecorator#6065bcb7 -> org.sonar.plugins.core.sensors.FilesDecorator#1ee6434 -> org.sonar.plugins.core.timemachine.TimeMachineConfigurationPersister#59ec5a0b -> org.sonar.plugins.core.timemachine.NewCoverageAggregator#71d78cac -> org.sonar.plugins.uselesscodetracker.decorator.TempMethodLinesDecorator#550f522b -> org.sonar.plugins.uselesscodetracker.decorator.DuplicationsDecorator#6a7fc4c1 -> f(lines) -> f(generated_lines) -> f(ncloc) -> f(generated_ncloc) -> f(classes) -> f(packages) -> f(functions) -> f(accessors) -> f(statements) -> f(public_api) -> f(comment_lines) -> f(comment_blank_lines) -> f(public_undocumented_api) -> f(commented_out_code_lines) -> f(complexity) -> f(complexity_in_classes) -> f(complexity_in_functions) -> f(class_complexity_distribution) -> f(function_complexity_distribution) -> f(file_complexity_distribution) -> f(lines_to_cover) -> f(uncovered_lines) -> f(conditions_to_cover) -> f(uncovered_conditions) -> f(it_lines_to_cover) -> f(it_uncovered_lines) -> f(it_conditions_to_cover) -> f(it_uncovered_conditions) -> f(overall_lines_to_cover) -> f(overall_uncovered_lines) -> f(overall_conditions_to_cover) -> f(overall_uncovered_conditions) -> f(rfc) -> f(rfc_distribution) -> f(lcom4_distribution) -> f(package_cycles) -> f(package_tangles) -> f(package_feedback_edges) -> f(package_edges_weight) -> f(new_lines_to_cover) -> f(new_uncovered_lines) -> f(new_conditions_to_cover) -> f(new_uncovered_conditions) -> f(new_it_lines_to_cover) -> f(new_it_uncovered_lines) -> f(new_it_conditions_to_cover) -> f(new_it_uncovered_conditions) -> f(new_overall_lines_to_cover) -> f(new_overall_uncovered_lines) -> f(new_overall_conditions_to_cover) -> f(new_overall_uncovered_conditions) -> org.sonar.plugins.cpd.decorators.DuplicationDensityDecorator#2c7c898a -> PackageTangleIndexDecorator -> CountUnresolvedIssuesDecorator -> CountFalsePositivesDecorator -> org.sonar.plugins.core.technicaldebt.DebtDecorator#23dc70c1 -> org.sonar.plugins.core.technicaldebt.NewDebtDecorator#53b907d9 -> CommentDensityDecorator -> org.sonar.plugins.uselesscodetracker.decorator.TotalDecorator#1842dc39 -> f(file_complexity) -> f(class_complexity) -> f(function_complexity) -> org.sonar.plugins.core.sensors.CoverageDecorator#6167c42f -> org.sonar.plugins.core.sensors.LineCoverageDecorator#2cbc2db2 -> org.sonar.plugins.core.sensors.BranchCoverageDecorator#2de96eba -> org.sonar.plugins.core.sensors.ItLineCoverageDecorator#6296e4bf -> org.sonar.plugins.core.sensors.ItCoverageDecorator#220cda54 -> org.sonar.plugins.core.sensors.ItBranchCoverageDecorator#39109603 -> org.sonar.plugins.core.sensors.OverallLineCoverageDecorator#4d3990a5 -> org.sonar.plugins.core.sensors.OverallCoverageDecorator#18df26a1 -> org.sonar.plugins.core.sensors.OverallBranchCoverageDecorator#5c459194 -> org.sonar.plugins.core.issue.WeightedIssuesDecorator#fe38c0e -> ViolationDensityDecorator -> IssuesDensityDecorator -> VariationDecorator
09:26:39.617 DEBUG - Decorator time:
ManualMeasureDecorator: 12ms
org.sonar.issuesreport.tree.ResourceTree#480994d3: 1ms
SumDuplicationsDecorator: 7ms
org.sonar.plugins.design.batch.ProjectDsmDecorator#7146c6ea: 0ms
FileTangleIndexDecorator: 0ms
org.sonar.plugins.core.issue.IssueTrackingDecorator#a5d23c9: 75ms
UnitTestDecorator: 1ms
org.sonar.plugins.core.sensors.DirectoriesDecorator#6065bcb7: 1ms
org.sonar.plugins.core.sensors.FilesDecorator#1ee6434: 0ms
org.sonar.plugins.core.timemachine.TimeMachineConfigurationPersister#59ec5a0b: 5ms
org.sonar.plugins.core.timemachine.NewCoverageAggregator#71d78cac: 3ms
org.sonar.plugins.uselesscodetracker.decorator.TempMethodLinesDecorator#550f522b: 10ms
org.sonar.plugins.uselesscodetracker.decorator.DuplicationsDecorator#6a7fc4c1: 0ms
f(lines): 0ms
f(generated_lines): 0ms
f(ncloc): 0ms
f(generated_ncloc): 0ms
f(classes): 0ms
f(packages): 0ms
f(functions): 0ms
f(accessors): 0ms
f(statements): 0ms
f(public_api): 0ms
f(comment_lines): 0ms
f(comment_blank_lines): 0ms
f(public_undocumented_api): 0ms
f(commented_out_code_lines): 0ms
f(complexity): 0ms
f(complexity_in_classes): 1ms
f(complexity_in_functions): 0ms
f(class_complexity_distribution): 0ms
f(function_complexity_distribution): 0ms
f(file_complexity_distribution): 0ms
f(lines_to_cover): 0ms
f(uncovered_lines): 0ms
f(conditions_to_cover): 0ms
f(uncovered_conditions): 0ms
f(it_lines_to_cover): 1ms
f(it_uncovered_lines): 0ms
f(it_conditions_to_cover): 1ms
f(it_uncovered_conditions): 0ms
f(overall_lines_to_cover): 0ms
f(overall_uncovered_lines): 0ms
f(overall_conditions_to_cover): 1ms
f(overall_uncovered_conditions): 0ms
f(rfc): 0ms
f(rfc_distribution): 0ms
f(lcom4_distribution): 0ms
f(package_cycles): 0ms
f(package_tangles): 0ms
f(package_feedback_edges): 0ms
f(package_edges_weight): 0ms
f(new_lines_to_cover): 0ms
f(new_uncovered_lines): 0ms
f(new_conditions_to_cover): 0ms
f(new_uncovered_conditions): 0ms
f(new_it_lines_to_cover): 0ms
f(new_it_uncovered_lines): 0ms
f(new_it_conditions_to_cover): 0ms
f(new_it_uncovered_conditions): 0ms
f(new_overall_lines_to_cover): 0ms
f(new_overall_uncovered_lines): 0ms
f(new_overall_conditions_to_cover): 0ms
f(new_overall_uncovered_conditions): 0ms
org.sonar.plugins.cpd.decorators.DuplicationDensityDecorator#2c7c898a: 1ms
PackageTangleIndexDecorator: 0ms
CountUnresolvedIssuesDecorator: 14ms
CountFalsePositivesDecorator: 0ms
org.sonar.plugins.core.technicaldebt.DebtDecorator#23dc70c1: 5ms
org.sonar.plugins.core.technicaldebt.NewDebtDecorator#53b907d9: 2ms
CommentDensityDecorator: 4ms
org.sonar.plugins.uselesscodetracker.decorator.TotalDecorator#1842dc39: 0ms
f(file_complexity): 0ms
f(class_complexity): 0ms
f(function_complexity): 0ms
org.sonar.plugins.core.sensors.CoverageDecorator#6167c42f: 1ms
org.sonar.plugins.core.sensors.LineCoverageDecorator#2cbc2db2: 0ms
org.sonar.plugins.core.sensors.BranchCoverageDecorator#2de96eba: 0ms
org.sonar.plugins.core.sensors.ItLineCoverageDecorator#6296e4bf: 0ms
org.sonar.plugins.core.sensors.ItCoverageDecorator#220cda54: 0ms
org.sonar.plugins.core.sensors.ItBranchCoverageDecorator#39109603: 0ms
org.sonar.plugins.core.sensors.OverallLineCoverageDecorator#4d3990a5: 0ms
org.sonar.plugins.core.sensors.OverallCoverageDecorator#18df26a1: 0ms
org.sonar.plugins.core.sensors.OverallBranchCoverageDecorator#5c459194: 0ms
org.sonar.plugins.core.issue.WeightedIssuesDecorator#fe38c0e: 4ms
ViolationDensityDecorator: 1ms
IssuesDensityDecorator: 0ms
VariationDecorator: 25ms
09:26:39.619 DEBUG - 162 measures to dump
09:26:39.706 INFO - Export results to D:\java\workspaces\wks-luna-SR1-01-Sonar\.metadata\.plugins\org.eclipse.core.resources\.projects\prjtest\org.sonar.ide.eclipse.core\sonar-report.json
09:26:39.712 INFO - Store results in database
09:26:39.713 DEBUG - Execute org.sonar.batch.index.ComponentDataPersister
09:26:39.719 DEBUG - Execute org.sonar.batch.issue.IssuePersister
09:26:39.719 DEBUG - IssuePersister skipped in preview mode
09:26:39.719 DEBUG - Execute org.sonar.batch.phases.GraphPersister
09:26:39.804 INFO - ANALYSIS SUCCESSFUL
09:26:39.807 DEBUG - Post-jobs : org.sonar.issuesreport.ReportJob#276aa33f
09:26:39.807 INFO - Executing post-job class org.sonar.issuesreport.ReportJob
09:26:39.813 DEBUG - Generating HTML Report to: D:\java\workspaces\wks-luna-SR1-01-Sonar\.metadata\.plugins\org.eclipse.core.resources\.projects\prjtest\org.sonar.ide.eclipse.core\issues-report\issues-report.html
09:26:40.176 INFO - HTML Issues Report generated: D:\java\workspaces\wks-luna-SR1-01-Sonar\.metadata\.plugins\org.eclipse.core.resources\.projects\prjtest\org.sonar.ide.eclipse.core\issues-report\issues-report.html
09:26:40.176 DEBUG - Generating Light HTML Report to: D:\java\workspaces\wks-luna-SR1-01-Sonar\.metadata\.plugins\org.eclipse.core.resources\.projects\prjtest\org.sonar.ide.eclipse.core\issues-report\issues-report-light.html
09:26:40.212 INFO - Light HTML Issues Report generated: D:\java\workspaces\wks-luna-SR1-01-Sonar\.metadata\.plugins\org.eclipse.core.resources\.projects\prjtest\org.sonar.ide.eclipse.core\issues-report\issues-report-light.html
09:26:40.236 INFO -
------------- Issues Report -------------
No new issue
-------------------------------------------
I am very confused. Can anybody help me ?
I found the solution : custom xpath rules for the xml plugin works on the XML file directly and not on the AST tree ... (unlike the Java Plugin ;))
In my example the xpath rule must be :
/root
Related
I'm using Netty-httpClient for Spring webClient.
When I connect www.naver.com and set the host like this.
127.0.0.1 www.naver.com
125.209.222.142 www.naver.com
And run httpClient.get() result like this.
DEBUG r.n.r.PooledConnectionProvider - [id:83c8069f] Created a new pooled channel, now: 0 active connections, 0 inactive connections and 0 pending acquire requests.
DEBUG r.n.t.SslProvider - [id:83c8069f] SSL enabled using engine sun.security.ssl.SSLEngineImpl#5412b204 and SNI www.naver.com:443
DEBUG i.n.b.AbstractByteBuf - -Dio.netty.buffer.checkAccessible: true
DEBUG i.n.b.AbstractByteBuf - -Dio.netty.buffer.checkBounds: true
DEBUG i.n.u.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector#7ec1e7a5
DEBUG r.n.t.TransportConfig - [id:83c8069f] Initialized pipeline DefaultChannelPipeline{(reactor.left.sslHandler = io.netty.handler.ssl.SslHandler), (reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.sslReader = reactor.netty.tcp.SslProvider$SslReadHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpClientCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
INFO reactor - [id:83c8069f] REGISTERED
DEBUG r.n.t.TransportConnector - [id:83c8069f] Connecting to [www.naver.com/127.0.0.1:443].
INFO reactor - [id:83c8069f] CONNECT: www.naver.com/127.0.0.1:443
INFO reactor - [id:83c8069f] CLOSE
DEBUG r.n.t.TransportConnector - [id:83c8069f] Connect attempt to [www.naver.com/127.0.0.1:443] failed.
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: www.naver.com/127.0.0.1:443
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
DEBUG r.n.r.PooledConnectionProvider - [id:0e3c272c] Created a new pooled channel, now: 0 active connections, 0 inactive connections and 0 pending acquire requests.
DEBUG r.n.t.SslProvider - [id:0e3c272c] SSL enabled using engine sun.security.ssl.SSLEngineImpl#47b241b and SNI www.naver.com:443
DEBUG r.n.t.TransportConfig - [id:0e3c272c] Initialized pipeline DefaultChannelPipeline{(reactor.left.sslHandler = io.netty.handler.ssl.SslHandler), (reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.sslReader = reactor.netty.tcp.SslProvider$SslReadHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpClientCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
INFO reactor - [id:0e3c272c] REGISTERED
INFO reactor - [id:83c8069f] UNREGISTERED
DEBUG r.n.t.TransportConnector - [id:0e3c272c] Connecting to [www.naver.com/125.209.222.142:443].
INFO reactor - [id:0e3c272c] CONNECT: www.naver.com/125.209.222.142:443
DEBUG r.n.r.DefaultPooledConnectionProvider - [id:0e3c272c, L:/192.168.55.77:52910 - R:www.naver.com/125.209.222.142:443] Registering pool release on close event for channel
DEBUG r.n.r.PooledConnectionProvider - [id:0e3c272c, L:/192.168.55.77:52910 - R:www.naver.com/125.209.222.142:443] Channel connected, now: 1 active connections, 0 inactive connections and 0 pending acquire requests.
DEBUG i.n.u.Recycler - -Dio.netty.recycler.maxCapacityPerThread: 4096
DEBUG i.n.u.Recycler - -Dio.netty.recycler.maxSharedCapacityFactor: 2
DEBUG i.n.u.Recycler - -Dio.netty.recycler.linkCapacity: 16
DEBUG i.n.u.Recycler - -Dio.netty.recycler.ratio: 8
DEBUG i.n.u.Recycler - -Dio.netty.recycler.delayedQueue.ratio: 8
INFO reactor - [id:0e3c272c, L:/192.168.55.77:52910 - R:www.naver.com/125.209.222.142:443] ACTIVE
see that. I don't want connect twice and don't spend more time to connect.
just fail if can not make connect.
Can I avoid this situation?
This is managing under the issue.
https://github.com/reactor/reactor-netty/issues/1822
I got the exception 'MongoWaitQueueFullException' and I realize the number of connections that my application is using. I use the default configuration of Spring boot (2.2.7.RELEASE) with reactive MongoDB (4.2.8). Transactions are used.
Even when running an integration test that basically creates a bit more than 200 elements then groups them (200 groups). 10 connections are used. When this algorithm is executed over a real data-set, this exception is thrown. The default limit of the waiting queue (500) was reached. This does not make the application scalable.
My question is: is there a way to design a reactive application that helps to reduce the number of connections?
This is the output of my test. Basically, it scans all translations of bundle files and them group them per translation key. An element is persisted per translation key.
return Flux
.fromIterable(bundleFile.getFiles())
.map(ScannedBundleFileEntry::getLocale)
.flatMap(locale ->
handler
.scanTranslations(bundleFileEntity.toLocation(), locale, context)
.index()
.map(indexedTranslation ->
createTranslation(
workspaceEntity,
bundleFileEntity,
locale.getId(),
indexedTranslation.getT1(), // index
indexedTranslation.getT2().getKey(), // bundle key
indexedTranslation.getT2().getValue() // translation
)
)
.flatMap(bundleKeyTemporaryRepository::save)
)
.thenMany(groupIntoBundleKeys(bundleFileEntity))
.then(bundleKeyTemporaryRepository.deleteByBundleFile(bundleFileEntity.getId()))
.then(Mono.just(bundleFileEntity));
The grouping function:
private Flux<BundleKeyEntity> groupIntoBundleKeys(BundleFileEntity bundleFile) {
return this
.findBundleKeys(bundleFile)
.groupBy(BundleKeyGroupKey::new)
.flatMap(bundleKeyGroup ->
bundleKeyGroup
.collectList()
.map(bundleKeys -> {
final BundleKeyGroupKey key = bundleKeyGroup.key();
final BundleKeyEntity entity = new BundleKeyEntity(key.getWorkspace(), key.getBundleFile(), key.getKey());
bundleKeys.forEach(entity::mergeInto);
return entity;
})
)
.flatMap(bundleKeyEntityRepository::save);
}
The test output:
560 [main] INFO o.s.b.t.c.SpringBootTestContextBootstrapper - Neither #ContextConfiguration nor #ContextHierarchy found for test class [be.sgerard.i18n.controller.TranslationControllerTest], using SpringBootContextLoader
569 [main] INFO o.s.t.c.s.AbstractContextLoader - Could not detect default resource locations for test class [be.sgerard.i18n.controller.TranslationControllerTest]: no resource found for suffixes {-context.xml, Context.groovy}.
870 [main] INFO o.s.b.t.c.SpringBootTestContextBootstrapper - Loaded default TestExecutionListener class names from location [META-INF/spring.factories]: [org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener, org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListener, org.springframework.boot.test.autoconfigure.restdocs.RestDocsTestExecutionListener, org.springframework.boot.test.autoconfigure.web.client.MockRestServiceServerResetTestExecutionListener, org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrintOnlyOnFailureTestExecutionListener, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverTestExecutionListener, org.springframework.test.context.web.ServletTestExecutionListener, org.springframework.test.context.support.DirtiesContextBeforeModesTestExecutionListener, org.springframework.test.context.support.DependencyInjectionTestExecutionListener, org.springframework.test.context.support.DirtiesContextTestExecutionListener, org.springframework.test.context.transaction.TransactionalTestExecutionListener, org.springframework.test.context.jdbc.SqlScriptsTestExecutionListener, org.springframework.test.context.event.EventPublishingTestExecutionListener, org.springframework.security.test.context.support.WithSecurityContextTestExecutionListener, org.springframework.security.test.context.support.ReactorContextTestExecutionListener]
897 [main] INFO o.s.b.t.c.SpringBootTestContextBootstrapper - Using TestExecutionListeners: [org.springframework.test.context.support.DirtiesContextBeforeModesTestExecutionListener#4372b9b6, org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener#232a7d73, org.springframework.boot.test.autoconfigure.SpringBootDependencyInjectionTestExecutionListener#4b41e4dd, org.springframework.test.context.support.DirtiesContextTestExecutionListener#22ffa91a, org.springframework.test.context.transaction.TransactionalTestExecutionListener#74960bfa, org.springframework.test.context.jdbc.SqlScriptsTestExecutionListener#42721fe, org.springframework.test.context.event.EventPublishingTestExecutionListener#40844aab, org.springframework.security.test.context.support.WithSecurityContextTestExecutionListener#1f6c9cd8, org.springframework.security.test.context.support.ReactorContextTestExecutionListener#5b619d14, org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListener#66746f57, org.springframework.boot.test.autoconfigure.restdocs.RestDocsTestExecutionListener#447a020, org.springframework.boot.test.autoconfigure.web.client.MockRestServiceServerResetTestExecutionListener#7f36662c, org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrintOnlyOnFailureTestExecutionListener#28e8dde3, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverTestExecutionListener#6d23017e]
1551 [background-preinit] INFO o.h.v.i.x.c.ValidationBootstrapParameters - HV000006: Using org.hibernate.validator.HibernateValidator as validation provider.
1677 [main] INFO b.s.i.c.TranslationControllerTest - Starting TranslationControllerTest on sgerard with PID 538 (started by sgerard in /home/sgerard/sandboxes/github-oauth/server)
1678 [main] INFO b.s.i.c.TranslationControllerTest - The following profiles are active: test
3250 [main] INFO o.s.d.r.c.RepositoryConfigurationDelegate - Bootstrapping Spring Data Reactive MongoDB repositories in DEFAULT mode.
3747 [main] INFO o.s.d.r.c.RepositoryConfigurationDelegate - Finished Spring Data repository scanning in 493ms. Found 9 Reactive MongoDB repository interfaces.
5143 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.security.config.annotation.method.configuration.ReactiveMethodSecurityConfiguration' of type [org.springframework.security.config.annotation.method.configuration.ReactiveMethodSecurityConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
5719 [main] INFO org.mongodb.driver.cluster - Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
5996 [cluster-ClusterId{value='5f42490f1c60f43aff9d7d46', description='null'}-localhost:27017] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:4337}] to localhost:27017
6010 [cluster-ClusterId{value='5f42490f1c60f43aff9d7d46', description='null'}-localhost:27017] INFO org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2, 8]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=12207332, setName='rs0', canonicalAddress=4802c4aff450:27017, hosts=[4802c4aff450:27017], passives=[], arbiters=[], primary='4802c4aff450:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000013, setVersion=1, lastWriteDate=Sun Aug 23 12:46:30 CEST 2020, lastUpdateTimeNanos=384505436362981}
6019 [main] INFO org.mongodb.driver.cluster - Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
6040 [cluster-ClusterId{value='5f42490f1c60f43aff9d7d47', description='null'}-localhost:27017] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:4338}] to localhost:27017
6042 [cluster-ClusterId{value='5f42490f1c60f43aff9d7d47', description='null'}-localhost:27017] INFO org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2, 8]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1727974, setName='rs0', canonicalAddress=4802c4aff450:27017, hosts=[4802c4aff450:27017], passives=[], arbiters=[], primary='4802c4aff450:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000013, setVersion=1, lastWriteDate=Sun Aug 23 12:46:30 CEST 2020, lastUpdateTimeNanos=384505468960066}
7102 [nioEventLoopGroup-2-2] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:3, serverValue:4339}] to localhost:27017
11078 [main] INFO o.s.b.a.e.web.EndpointLinksResolver - Exposing 1 endpoint(s) beneath base path ''
11158 [main] INFO o.h.v.i.x.c.ValidationBootstrapParameters - HV000006: Using org.hibernate.validator.HibernateValidator as validation provider.
11720 [main] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:4340}] to localhost:27017
12084 [main] INFO o.s.s.c.ThreadPoolTaskScheduler - Initializing ExecutorService 'taskScheduler'
12161 [main] INFO b.s.i.c.TranslationControllerTest - Started TranslationControllerTest in 11.157 seconds (JVM running for 13.532)
20381 [nioEventLoopGroup-2-3] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:5, serverValue:4341}] to localhost:27017
20408 [nioEventLoopGroup-2-2] INFO b.s.i.s.w.WorkspaceManagerImpl - Synchronize, there is no workspace for the branch [master], let's create it.
20416 [nioEventLoopGroup-2-3] INFO b.s.i.s.w.WorkspaceManagerImpl - The workspace [master] alias [e3cea374-0d37-4c57-bdbf-8bd14d279c12] has been created.
20421 [nioEventLoopGroup-2-3] INFO b.s.i.s.w.WorkspaceManagerImpl - Initializing workspace [master] alias [e3cea374-0d37-4c57-bdbf-8bd14d279c12].
20525 [nioEventLoopGroup-2-2] INFO b.s.i.s.i18n.TranslationManagerImpl - A bundle file has been found located in [server/src/main/resources/i18n] named [exception] with 2 file(s).
20812 [nioEventLoopGroup-2-4] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:6, serverValue:4342}] to localhost:27017
21167 [nioEventLoopGroup-2-8] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:10, serverValue:4345}] to localhost:27017
21167 [nioEventLoopGroup-2-6] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:8, serverValue:4344}] to localhost:27017
21393 [nioEventLoopGroup-2-5] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:7, serverValue:4343}] to localhost:27017
21398 [nioEventLoopGroup-2-7] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:9, serverValue:4346}] to localhost:27017
21442 [nioEventLoopGroup-2-2] INFO b.s.i.s.i18n.TranslationManagerImpl - A bundle file has been found located in [server/src/main/resources/i18n] named [validation] with 2 file(s).
21503 [nioEventLoopGroup-2-2] INFO b.s.i.s.i18n.TranslationManagerImpl - A bundle file has been found located in [server/src/test/resources/be/sgerard/i18n/service/i18n/file] named [file] with 2 file(s).
21621 [nioEventLoopGroup-2-2] INFO b.s.i.s.i18n.TranslationManagerImpl - A bundle file has been found located in [front/src/main/web/src/assets/i18n] named [i18n] with 2 file(s).
22745 [SpringContextShutdownHook] INFO o.s.s.c.ThreadPoolTaskScheduler - Shutting down ExecutorService 'taskScheduler'
22763 [SpringContextShutdownHook] INFO org.mongodb.driver.connection - Closed connection [connectionId{localValue:4, serverValue:4340}] to localhost:27017 because the pool has been closed.
22766 [SpringContextShutdownHook] INFO org.mongodb.driver.connection - Closed connection [connectionId{localValue:9, serverValue:4346}] to localhost:27017 because the pool has been closed.
22767 [SpringContextShutdownHook] INFO org.mongodb.driver.connection - Closed connection [connectionId{localValue:6, serverValue:4342}] to localhost:27017 because the pool has been closed.
22768 [SpringContextShutdownHook] INFO org.mongodb.driver.connection - Closed connection [connectionId{localValue:8, serverValue:4344}] to localhost:27017 because the pool has been closed.
22768 [SpringContextShutdownHook] INFO org.mongodb.driver.connection - Closed connection [connectionId{localValue:5, serverValue:4341}] to localhost:27017 because the pool has been closed.
22769 [SpringContextShutdownHook] INFO org.mongodb.driver.connection - Closed connection [connectionId{localValue:10, serverValue:4345}] to localhost:27017 because the pool has been closed.
22770 [SpringContextShutdownHook] INFO org.mongodb.driver.connection - Closed connection [connectionId{localValue:7, serverValue:4343}] to localhost:27017 because the pool has been closed.
22776 [SpringContextShutdownHook] INFO org.mongodb.driver.connection - Closed connection [connectionId{localValue:3, serverValue:4339}] to localhost:27017 because the pool has been closed.
Process finished with exit code 0
Spring Reactive is asynchronous. Imagine you have 3 items in your dataset. It opens a connection for the save of the first item. But it won't wait for it to finish and use for the second save. Instead, it opens a second connection as soon as possible. Thus you'll end up overloading all the possible connections in the pool.
I have deployed microservice on docker-desktop for windows and feign is not able to make a call to another service.
person ms calling organization ms through feign.
I can see in the logs of person pod
2019-11-10 12:58:34.000 INFO [personservice,13631e6ef2efe358,15c75b9a4006485a,true] 6 --- [ionThreadPool-1] c.p.service.OrganizationServiceData : Get the value from the organization ms hystrix-organizationThreadPool-1
2019-11-10 12:58:34.293 INFO [personservice,13631e6ef2efe358,e0bbcecf5f7c349f,true] 6 --- [zationservice-1] c.netflix.config.ChainedDynamicProperty : Flipping property: organizationservice.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
2019-11-10 12:58:34.319 INFO [personservice,13631e6ef2efe358,e0bbcecf5f7c349f,true] 6 --- [zationservice-1] c.netflix.loadbalancer.BaseLoadBalancer : Client: organizationservice instantiated a LoadBalancer: DynamicServerListLoadBalancer:{NFLoadBalancer:name=organizationservice,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:null
2019-11-10 12:58:34.332 INFO [personservice,13631e6ef2efe358,e0bbcecf5f7c349f,true] 6 --- [zationservice-1] c.n.l.DynamicServerListLoadBalancer : Using serverListUpdater PollingServerListUpdater
2019-11-10 12:58:34.535 INFO [personservice,13631e6ef2efe358,e0bbcecf5f7c349f,true] 6 --- [zationservice-1] c.netflix.config.ChainedDynamicProperty : Flipping property: organizationservice.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
2019-11-10 12:58:34.556 INFO [personservice,13631e6ef2efe358,e0bbcecf5f7c349f,true] 6 --- [zationservice-1] c.n.l.DynamicServerListLoadBalancer : DynamicServerListLoadBalancer for client organizationservice initialized: DynamicServerListLoadBalancer:{NFLoadBalancer:name=organizationservice,current list of Servers=[10.1.0.190:8085],Load balancer stats=Zone stats: {unknown=[Zone:unknown; Instance count:1; Active connections count: 0; Circuit breaker tripped count: 0; Active connections per server: 0.0;]
},Server stats: [[Server:10.1.0.190:8085; Zone:UNKNOWN; Total Requests:0; Successive connection failure:0; Total blackout seconds:0; Last connection made:Thu Jan 01 00:00:00 GMT 1970;
First connection made: Thu Jan 01 00:00:00 GMT 1970; Active Connections:0; total failure count in last (1000) msecs:0; average resp time:0.0; 90 percentile resp time:0.0; 95 percentile resp time:0.0; min resp time:0.0; max resp time:0.0; stddev resp time:0.0]
]}ServerList:org.springframework.cloud.kubernetes.ribbon.KubernetesServerList#5bae3a6b
2019-11-10 12:58:35.345 INFO [personservice,,,] 6 --- [erListUpdater-0] c.netflix.config.ChainedDynamicProperty : Flipping property: organizationservice.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
2019-11-10 12:58:35.502 INFO [personservice,13631e6ef2efe358,15c75b9a4006485a,true] 6 --- [ionThreadPool-1] c.p.service.OrganizationServiceData : calling fallback method to get the organization data for id 1
The ribbon client gets the IP-address of the pod running organization service. current list of Servers=[10.1.0.190:8085]
Here is my person service application.yml
spring:
cloud:
kubernetes:
ribbon:
mode: SERVICE
organizationservice:
ribbon:
MaxAutoRetries: 2
MaxAutoRetriesNextServer: 0
OkToRetryOnAllOperations: true
ServerListRefreshInterval: 2000
ConnectTimeout: 10000
ReadTimeout: 1000
person ms dependecy
compile "org.springframework.cloud:spring-cloud-starter-kubernetes-all"
compile('org.springframework.cloud:spring-cloud-starter-netflix-ribbon')
upon checking the logs for organization pod. No calls have been made to it.
Edit 1:
Upon changing the log level for feign in person service I found that JWT token is begin passed by default. I solved the issue using filters but the same application was working with out filters when I was using not using spring cloud kubernetes.
I have spent days on this simple issue , I am giving up and finally posting this issue which I am facing locally. I am trying to set up a microservices flow in my local for my hand itching learning purpose. This is no brainer. I have Eureka , Zuul Gateway , Simple Microservice. When I try to reach to the underlying service with the "url route" its working. But when I try to do serviceId look up its not working. Guys help me fixing it.
Git hub link is Git hub source code link
I have also raised an issue Git hut Issue link
Eureka Screenshot
Zuul Gateway logs
2019-10-06 11:11:24.611 INFO 26980 --- [nio-2020-exec-4] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
2019-10-06 11:11:24.611 INFO 26980 --- [nio-2020-exec-4] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
2019-10-06 11:11:24.633 INFO 26980 --- [nio-2020-exec-4] o.s.web.servlet.DispatcherServlet : Completed initialization in 22 ms
2019-10-06 11:11:25.103 INFO 26980 --- [nio-2020-exec-4] c.netflix.config.ChainedDynamicProperty : Flipping property: CHECKOUT-SERVICE.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
2019-10-06 11:11:25.157 INFO 26980 --- [nio-2020-exec-4] c.n.u.concurrent.ShutdownEnabledTimer : Shutdown hook installed for: NFLoadBalancer-PingTimer-CHECKOUT-SERVICE
2019-10-06 11:11:25.157 INFO 26980 --- [nio-2020-exec-4] c.netflix.loadbalancer.BaseLoadBalancer : Client: CHECKOUT-SERVICE instantiated a LoadBalancer: DynamicServerListLoadBalancer:{NFLoadBalancer:name=CHECKOUT-SERVICE,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:null
2019-10-06 11:11:25.167 INFO 26980 --- [nio-2020-exec-4] c.n.l.DynamicServerListLoadBalancer : Using serverListUpdater PollingServerListUpdater
2019-10-06 11:11:25.215 INFO 26980 --- [nio-2020-exec-4] c.netflix.config.ChainedDynamicProperty : Flipping property: CHECKOUT-SERVICE.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
2019-10-06 11:11:25.218 INFO 26980 --- [nio-2020-exec-4] c.n.l.DynamicServerListLoadBalancer : DynamicServerListLoadBalancer for client CHECKOUT-SERVICE initialized: DynamicServerListLoadBalancer:{NFLoadBalancer:name=CHECKOUT-SERVICE,current list of Servers=[192.168.0.6:8098],Load balancer stats=Zone stats: {defaultzone=[Zone:defaultzone; Instance count:1; Active connections count: 0; Circuit breaker tripped count: 0; Active connections per server: 0.0;]
},Server stats: [[Server:192.168.0.6:8098; Zone:defaultZone; Total Requests:0; Successive connection failure:0; Total blackout seconds:0; Last connection made:Wed Dec 31 19:00:00 EST 1969; First connection made: Wed Dec 31 19:00:00 EST 1969; Active Connections:0; total failure count in last (1000) msecs:0; average resp time:0.0; 90 percentile resp time:0.0; 95 percentile resp time:0.0; min resp time:0.0; max resp time:0.0; stddev resp time:0.0]
]}ServerList:org.springframework.cloud.netflix.ribbon.eureka.DomainExtractingServerList#6f7f7ca0
2019-10-06 11:11:26.177 INFO 26980 --- [erListUpdater-0] c.netflix.config.ChainedDynamicProperty : Flipping property: CHECKOUT-SERVICE.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
Never mind guys it was a mistake from my side in resolving the API path
Recently, When I was testing my cluster with Spark and HBase. I was using newAPIHadoopRDD to read records from HBase table. I found that newAPIHadoopRDD was too slow, and the time was proportional to the number of Region Servers.
The spark debug(opened for test) logs below shows the procedure:
17/03/02 22:00:30 DEBUG AbstractRpcClient: Use SIMPLE authentication for service ClientService, sasl=false
17/03/02 22:00:30 DEBUG AbstractRpcClient: Connecting to slave111/192.168.10.111:16020
17/03/02 22:00:30 DEBUG ClientCnxn: Reading reply sessionid:0x15a8de8a86f0444, packet:: clientPath:null serverPath:null finished:false header:: 5,3 replyHeader:: 5,116079898,0 request:: '/hbase,F response:: s{116070329,116070329,1488462020202,1488462020202,0,16,0,0,0,16,116070652}
17/03/02 22:00:30 DEBUG ClientCnxn: Reading reply sessionid:0x15a8de8a86f0444, packet:: clientPath:null serverPath:null finished:false header:: 6,4 replyHeader:: 6,116079898,0 request:: '/hbase/master,F response:: #ffffffff000146d61737465723a3136303030fffffff4ffffffa23affffffc8ffffffb6ffffffb1ffffffc21a50425546a12a66d617374657210ffffff807d18ffffffcffffffff4fffffffffffffff9ffffffa82b10018ffffff8a7d,s{116070348,116070348,1488462021202,1488462021202,0,0,0,97546372339663909,54,0,116070348}
17/03/02 22:00:30 DEBUG AbstractRpcClient: Use SIMPLE authentication for service MasterService, sasl=false
17/03/02 22:00:30 DEBUG AbstractRpcClient: Connecting to master/192.168.10.100:16000
17/03/02 22:00:30 DEBUG RegionSizeCalculator: Region tt,3,1488442069431.21d34666d310df3f180b2dba093d910d. has size 0
17/03/02 22:00:30 DEBUG RegionSizeCalculator: Region tt,,1488442069431.cb8696957957f824f1a16210768bf197. has size 0
17/03/02 22:00:30 DEBUG RegionSizeCalculator: Region tt,1,1488442069431.274ddaa4abb34f0408cac0f33107529c. has size 0
17/03/02 22:00:30 DEBUG RegionSizeCalculator: Region tt,2,1488442069431.05dd84aacb7f2587e325c8baf4c27613. has size 0
17/03/02 22:00:30 DEBUG RegionSizeCalculator: Region sizes calculated
17/03/02 22:00:38 DEBUG Client: IPC Client (480943798) connection to master/192.168.10.100:9000 from hadoop: closed
17/03/02 22:00:38 DEBUG Client: IPC Client (480943798) connection to master/192.168.10.100:9000 from hadoop: stopped, remaining connections 0
17/03/02 22:00:43 DEBUG ClientCnxn: Got ping response for sessionid: 0x15a8de8a86f0444 after 0ms
17/03/02 22:00:56 DEBUG ClientCnxn: Got ping response for sessionid: 0x15a8de8a86f0444 after 0ms
17/03/02 22:01:00 DEBUG TableInputFormatBase: getSplits: split -> 0 -> HBase table split(table name: tt, scan: , start row: , end row: 1, region location: slave104)
17/03/02 22:01:10 DEBUG ClientCnxn: Got ping response for sessionid: 0x15a8de8a86f0444 after 0ms
17/03/02 22:01:23 DEBUG ClientCnxn: Got ping response for sessionid: 0x15a8de8a86f0444 after 0ms
17/03/02 22:01:30 DEBUG TableInputFormatBase: getSplits: split -> 1 -> HBase table split(table name: tt, scan: , start row: 1, end row: 2, region location: slave102)
17/03/02 22:01:37 DEBUG ClientCnxn: Got ping response for sessionid: 0x15a8de8a86f0444 after 0ms
17/03/02 22:01:50 DEBUG ClientCnxn: Got ping response for sessionid: 0x15a8de8a86f0444 after 0ms
17/03/02 22:02:00 DEBUG TableInputFormatBase: getSplits: split -> 2 -> HBase table split(table name: tt, scan: , start row: 2, end row: 3, region location: slave112)
17/03/02 22:02:03 DEBUG ClientCnxn: Got ping response for sessionid: 0x15a8de8a86f0444 after 0ms
17/03/02 22:02:17 DEBUG ClientCnxn: Got ping response for sessionid: 0x15a8de8a86f0444 after 0ms
17/03/02 22:02:30 DEBUG ClientCnxn: Got ping response for sessionid: 0x15a8de8a86f0444 after 0ms
17/03/02 22:02:30 DEBUG TableInputFormatBase: getSplits: split -> 3 -> HBase table split(table name: tt, scan: , start row: 3, end row: , region location: slave108)
17/03/02 22:02:30 INFO ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
17/03/02 22:02:30 INFO ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15a8de8a86f0444
17/03/02 22:02:30 DEBUG ZooKeeper: Closing session: 0x15a8de8a86f0444
17/03/02 22:02:30 DEBUG ClientCnxn: Closing client for session: 0x15a8de8a86f0444
17/03/02 22:02:30 DEBUG ClientCnxn: Reading reply sessionid:0x15a8de8a86f0444, packet:: clientPath:null serverPath:null finished:false header:: 7,-11 replyHeader:: 7,116080795,0 request:: null response:: null
17/03/02 22:02:30 DEBUG ClientCnxn: Disconnecting client for session: 0x15a8de8a86f0444
17/03/02 22:02:30 INFO ZooKeeper: Session: 0x15a8de8a86f0444 closed
17/03/02 22:02:30 INFO ClientCnxn: EventThread shut down
17/03/02 22:02:30 DEBUG AbstractRpcClient: Stopping rpc client
17/03/02 22:02:30 DEBUG ClientCnxn: An exception was thrown while closing send thread for session 0x15a8de8a86f0444 : Unable to read additional data from server sessionid 0x15a8de8a86f0444, likely server has closed socket
17/03/02 22:02:30 DEBUG ClosureCleaner: +++ Cleaning closure <function1> (org.apache.spark.rdd.RDD$$anonfun$count$1) +++
I'm using Spark 2.1.0, HBase 1.1.2. It took too much time for getSplits operation. The region server numbers was tested from one to four, and it took 30 seconds for each region server. HBase table contains no records (just for test).
Is this normal? and does anyone suffer the same problem as me?
The test code shows below:
Configuration hconf = HBaseConfiguration.create();
hconf.set(TableInputFormat.INPUT_TABLE, GLOBAL.TABLE_NAME);
hconf.set("hbase.zookeeper.quorum", "192.168.10.100");
hconf.set("hbase.zookeeper.property.clientPort", "2181");
Scan scan = new Scan();
JavaPairRDD<ImmutableBytesWritable, Result> results
= sc.newAPIHadoopRDD(hconf, TableInputFormat.class, ImmutableBytesWritable.class, Result.class);
long cnt = results.count();
System.out.println(cnt);
EDIT
After debugging with HBase source code, I found the cause of the slow speed. The reverse DNS operation from TableInputFormatBase.java is the culprit.
ipAddressString = DNS.reverseDns(ipAddress, null);
How to solve this problem now? Can I add some dns-ip pair in HBase configuration?
I got the result below when using nslookup to reverse find 192.168.10.100.
;; connection timed out; trying next origin
;; connection timed out; no servers could be reached
so, I executed the cmds below,
sudo iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o em4 -j MASQUERADE
sudo sysctl -w net.ipv4.ip_forward=1
sudo route add default gw 'mygatway' em4
then, the problem is gone.