Starting with jPOS - jpos

I have downloaded jPOS-master.zip from this link. (https://github.com/jpos/jPOS ).
Ran the command gradle clean dist installApp
Found jPOS-master\jpos\build\install\jpos\jpos-1.9.5-SNAPSHOT.jar
Ran the command java -jar jpos-1.9.5-SNAPSHOT.jar
Got the below console:
<log realm="Q2.system" at="Mon Dec 30 14:47:08 IST 2013.183" lifespan="10ms">
<info>
Q2 started, deployDir=D:\jPOS-master\jPOS-master\jpos\build\install\jpos\deploy
jPOS 1.9.5-SNAPSHOT unknown/unknown (2013-12-30 14:44:39 IST)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
jPOS Community Edition, licensed under GNU AGPL v3.0.
This software is probably not suitable for commercial use.
Please see http://jpos.org/license for details.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
iQEcBAEBAgAGBQJMolHDAAoJEOQyeO71nYtFv74H/3OgehDGEy1VXp2U3/GcAobg
HH2eZjPUz53r38ARPiU3pzm9LwDa3WZgJJaa/b9VrJwKvbPwe9+0kY3gScDE1skT
ladHt+KHHmGQArEutkzHlpZa73RbroFEIa1qmN6MaDEHGoxZqDh0Sv2cpvOaVYGO
St8ZaddLBPC17bSjAPWo9sWbvL7FgPFOHhnPmbeux8SLtnfWxXWsgo5hLBanKmO1
1z+I/w/6DL6ZYZU6bAJUk+eyVVImJqw0x3IEElI07Nh9MC6BA4iJ77ejobj8HI2r
q9ulRPEqH9NR79619lNKVUkE206dVlXo7xHmJS1QZy5v/GT66xBxyDVfTduPFXk=
=oP+v
-----END PGP SIGNATURE-----
</info>
</log>
<log realm="Q2.system" at="Mon Dec 30 14:47:08 IST 2013.200" lifespan="8ms">
<info>
deploy: D:\jPOS-master\jPOS-master\jpos\build\install\jpos\deploy\99_sysmon.xml
</info>
</log>
<log realm="org.jpos.q2.qbean.SystemMonitor" at="Mon Dec 30 14:47:08 IST 2013.202">
<info>
Starting SystemMonitor
</info>
</log>
<log realm="org.jpos.q2.qbean.SystemMonitor" at="Mon Dec 30 14:47:08 IST 2013.203">
<info>
OS: Windows 7
host: *****/*******
version: 1.9.5-SNAPSHOT (unknown)
instance: a0032410-aff4-4128-9bf1-1d14966f2ea9
uptime: 00:00:00.280
processors: 4
drift : 0
memory(t/u/f): 121/8/113
threads: 4
Thread[Reference Handler,10,system]
Thread[Finalizer,8,system]
Thread[Signal Dispatcher,9,system]
Thread[Attach Listener,5,system]
Thread[Q2-a0032410-aff4-4128-9bf1-1d14966f2ea9,5,main]
Thread[DestroyJavaVM,5,main]
Thread[Timer-0,5,main]
Thread[SystemMonitor,5,main]
name-registrar:
logger.Q2.buffered: org.jpos.util.BufferedLogListener
logger.Q2: org.jpos.util.Logger
</info>
</log>
When I do a telnet to localhost i get below exception .. which means that Q2 Server is not up.
Connecting To 127.0.0.1...Could not open connection to the host, on port 10000: Connect failed
When I do a gradle test I get the below exception
org.jpos.iso.packagers.PackagerTestCase > testXMLPackager FAILED
junit.framework.AssertionFailedError at PackagerTestCase.java:214
org.jpos.util.DailyLogListenerTest > testLogRotationAndCompressionWorks FAILED
java.lang.AssertionError at DailyLogListenerTest.java:303
> Building > :jpos:test > 3145 tests completed, 2 failed, 54 skipped
Can somebody please help as how to bring up the Q2 Server ? I have read proguide-draft.pdf, but could not understand what is the issue. I want to get a server up and running and hit it with some sample requests and get the responses back.
Do I need put some more xmls in deploy directory for Server deployment ? Also after running gradle installApp I could not find any q2.bat in the install directory, I Could find only ..
bsh q2 start stop files.
Thanks in advance,
Varun Kumar A.

Can somebody please help as how to bring up the Q2 Server ? I have
read proguide-draft.pdf, but could not understand what is the issue. I
want to get a server up and running and hit it with some sample
requests and get the responses back.
You can use "jpos-template" for server and client simulation. Detail how to is here in these beautiful blog posts. As per the ducumentation,
[Implementing a jpos dateway] page no 8
We connect here to jPOS' auto-responder, which is usually up and
listening on port 9000. You could use a local server, for example
based on jPOS-EE's server simulator.
You can use server simulator as a auto responder (remote host as per diagram) to prepare a demo environment of jpos.
Sever Simulator
Client Simulator

Your serve is not deployed properly, you need to add serversimulator.bsh and serversimulator.xml to corresponding folders and then try again.

Related

IBM WebSphere Liberty working with IHS Server Problems with Certificate(s)

I have a liberty server and an IHS server (both on different hosts). This is on rhel7.
I've followed a number of guides that I found that walk thru the setup of keystores and sharing of certs but still run into issues. For reference, I tried the methods detailed https://jazz.net/wiki/bin/view/Deployment/CreateIHSPLUGINFORLIBERTYPROFILE and https://www.ibm.com/support/knowledgecenter/en/SSEQTJ_9.0.5/com.ibm.websphere.ihs.doc/ihs/tihs_install_config_liberty.html.
A quick summary of what I have done this past attempt... On the liberty server I added the following to my server.xml to add a plugin include (to keep things easier to read):
<include location="${server.config.dir}/plugin-join-include.xml" />
And the plugin-join-include.xml:
<featureManager>
<feature>ssl-1.0</feature>
</featureManager>
<keyStore id="libertyKey" password="xxxx"
location="${server.config.dir}/resources/security/libkey.jks"/>
<pluginConfiguration webserverPort="10500" webserverSecurePort="10447"
pluginInstallRoot="/opt/IBM/wasadmin/Plugins"
sslKeyRingLocation="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.kdb"
sslStashFileLocation="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.sth"
sslCertlabel="libertyKey"/>
On my IHS server the path for the root Plugin directory is as indicated above in the file at /opt/IBM/wasadmin/Plugins and the plugin-key.kdb and plugin-key.sth are located in the locations listed above.
I reboot the liberty server and it generated the plugin-cfg.xml in the logs/state directory:
<?xml version="1.0" encoding="UTF-8"?><!--HTTP server plugin config file for app generated on 2020.03.04 at 12:32:02 UTC-->
<Config ASDisableNagle="false" AcceptAllContent="false" AppServerPortPreference="HostHeader" ChunkedResponse="false" ConfigHash="1077723051" FIPSEnable="false" IISDisableNagle="false" IISPluginPriority="High" IgnoreDNSFailures="false" RefreshInterval="60" ResponseChunkSize="64" SSLConsolidate="false" TrustedProxyEnable="false" VHostMatchingCompat="false">
<Log LogLevel="Error" Name="/opt/IBM/wasadmin/Plugins/logs/webserver1/http_plugin.log"/>
<Property Name="ESIEnable" Value="true"/>
<Property Name="ESIMaxCacheSize" Value="1024"/>
<Property Name="ESIInvalidationMonitor" Value="false"/>
<Property Name="ESIEnableToPassCookies" Value="false"/>
<Property Name="PluginInstallRoot" Value="/opt/IBM/wasadmin/Plugins"/>
<!-- Configuration generated using httpEndpointRef=defaultHttpEndpoint-->
<!-- The default_host contained only aliases for endpoint defaultHttpEndpoint.
The generated VirtualHostGroup will contain only configured web server ports:
webserverPort=10500
webserverSecurePort=10447 -->
<VirtualHostGroup Name="default_host">
<VirtualHost Name="*:10500"/>
<VirtualHost Name="*:10447"/>
</VirtualHostGroup>
<ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" IgnoreAffinityRequests="true" LoadBalance="Round Robin" Name="affms_default_node_Cluster" PostBufferSize="0" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60" ServerIOTimeoutRetry="-1">
<Server CloneID="412b3187-16c4-41b0-86e8-1e327c1c6b1b" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="default_node_affms" ServerIOTimeout="900" WaitForContinue="false">
<Transport Hostname="libertyhost" Port="10500" Protocol="http"/>
<Transport Hostname="libertyhost" Port="10447" Protocol="https">
<Property Name="keyring" Value="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.kdb"/>
<Property Name="stashfile" Value="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.sth"/>
<Property Name="certLabel" Value="libertyKey"/>
</Transport>
</Server>
<PrimaryServers>
<Server Name="default_node_app"/>
</PrimaryServers>
</ServerCluster>
<UriGroup Name="default_host_app_default_node_Cluster_URIs">
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/app/ui/*"/>
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/adminCenter/*"/>
</UriGroup>
<Route ServerCluster="app_default_node_Cluster" UriGroup="default_host_app_default_node_Cluster_URIs" VirtualHostGroup="default_host"/>
I copy the plugin-cfg.xml over to the IHS server at /opt/IBM/wasadmin/Plugins/config/webserver1/plugin-cfg.xml
The plugin stuff works only the SSL portion between IHS and the app server give me issues.
On the IHS server I have created the plugin-key.kdb keystore with stash plugin-key.sth and imported the cert I extracted from the liberty server /opt/IBM/wlp/usr/servers/app/resources/security/key.p12 the default personal cert and copy it to the IHS server and add the default liberty cert to the plugin-key.kdb keystore.
At the bottom of the httpd.conf is:
LoadModule was_ap22_module /opt/IBM/wasadmin/Plugins/bin/64bits/mod_was_ap22_http.so
WebSpherePluginConfig /opt/IBM/wasadmin/Plugins/config/webserver1/plugin-cfg.xml
I bounce the apachectl and go to the url for the application on port 8443 https://hostname:8443/app/ui and get a "500 Internal Server Error".
The http_plugin.log plugin logs show:
[04/Mar/2020:18:23:48.31652] 00002f26 3cff9700 - ERROR: lib_stream: openStream: Failed in r_gsk_secure_soc_init: GSK_ERROR_BAD_CERT(gsk rc = 414) PARTNER CERTIFICATE DN=CN=hostname,OU=app,O=ibm,C=us, Serial=xxxxxxxxxxx
[04/Mar/2020:18:23:48.31655] 00002f26 3cff9700 - ERROR: Ensure correct certificate is marked as default certificate in plugin-key.kdb. Consult documentation regarding Administering application security and Securing communications for more information.
[04/Mar/2020:18:23:48.31657] 00002f26 3cff9700 - ERROR: Last validation error [575010]: GSKVAL_ERROR_NO_CHAIN_BUILT
[04/Mar/2020:18:23:48.31658] 00002f26 3cff9700 - ERROR: Subject [[Class=]GSKVALMethod::PKIX[Issuer=]OU=memberRoot,O=xxxx-xxx-xxx-xxx-xxxxxxxx,DC=com.ibm.ws.collective[#=]082d6a83e5ec[Subject=]CN=hostname,OU=app,O=ibm,C=us] failed certificate validation
[04/Mar/2020:18:23:48.31659] 00002f26 3cff9700 - ERROR: X509 Certificate validation log: [[Class=]GSKVALMethod::PKIX[Time=]2020:3:4:18:23:48.316[buildChain=][Error=]GSKVAL_ERR_NO_CHAIN_BUILT[Info=]OU=memberRoot,O=xxx-xxx-xxx-xxx-xxx,DC=com.ibm.ws.collective[Cert=][Issuer=]OU=memberRoot,O=xxx-xxx-xxx-xxx-xxxx,DC=com.ibm.ws.collective[#=]082d6a83e5ec[Subject=]CN=hostname,OU=app,O=ibm,C=us[=Cert][=buildChain]^M
]
[04/Mar/2020:18:23:48.31664] 00002f26 3cff9700 - ERROR: ws_common: websphereGetStream: Could not open stream
[04/Mar/2020:18:23:48.31667] 00002f26 3cff9700 - ERROR: ws_common: websphereExecute: Failed to create the stream
[04/Mar/2020:18:23:48.31668] 00002f26 3cff9700 - ERROR: ws_common: websphereHandleRequest: Failed to execute the transaction to 'default_node_app' on host 'hostname:10447'; will try another one
[04/Mar/2020:18:23:48.31669] 00002f26 3cff9700 - ERROR: ws_common: websphereWriteRequestReadResponse: Failed to find an app server to handle this request
[04/Mar/2020:18:23:48.31670] 00002f26 3cff9700 - ERROR: ESI: getResponse: failed to get response: rc = 2
[04/Mar/2020:18:23:48.31671] 00002f26 3cff9700 - ERROR: [xxx.xx.xx.17://app/ui/] ws_common: websphereHandleRequest: Failed to handle request rc=2
Thank you!
You said "imported" I assume this means an "add" operation in ikeyman/gskcmd/gskcapicmd? The two verbs in these tools have a different meaning.
I suspect you added the issued certificate (CN=hostname...) instead of the CA that issued the certificate (memberroot). The debug information in the WAS Plugin message implies that it couldn't chase the certificate up to a trusted root (as opposed to finding some X509 error w/ the root it had access to).
But, it could also be that you imported the "wrong" certificate with cn=memberroot. WebSphere does not always add the necessary Subject Key Identifier/Authority Key Identifier to be sure, so check carefully.
Ok, I got this working finally. It seems having a collective setup which used DefaultKeyStore id was somehow causing conflict with my Liberty plugin keystore.
I ended up re-creating everything (plugin-key.kdb keystore and the liberty plugin keystore which I named LibertyKeystore.jks). I dumped the collective stuff (not needed at this time anyways) and made an include as above called plugin-join-include.xml:
<?xml version="1.0" encoding="UTF-8" ?>
<server description="IHS plugin join include file">
<featureManager>
<feature>ssl-1.0</feature>
</featureManager>
<keyStore id="defaultKeyStore" password="password"
location="${server.config.dir}/resources/security/LibertyKeystore.jks" />
<pluginConfiguration webserverPort="80" webserverSecurePort="8443"
pluginInstallRoot="/opt/IBM/wasadmin/Plugins"
sslKeyRingLocation="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.kdb"
sslStashFileLocation="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.sth"
sslCertlabel="LibertyKeystore"/>
</server>
I exchanged keys between the liberty server LibertyKeystore.jks and the IHS plugin-key.kdb keystores and pushed the newly generated plugin to the IHS server and bounced everything and it all seems to function without issue.
I do now notice in the /opt/IBM/wasadmin/HTTPServer/logs the error log streams:
[Wed Mar 04 21:32:28 2020] [error] [client xxx.xx.xx.17:65261] [7ff5a0000910] [19831] SSL0279E: SSL Handshake Failed due to fatal alert from client. Client sent fatal alert [level 2 (fatal), description 46 (certificate_unknown)] [xxx.xx.xx.17:65261 -> xxx.xx.xx.87:8443] [21:32:28.000934048] 0ms
The access logs show 200's. Maybe a non plugin "issue" at this point.

WebLogic Server 12c cant log in localhost Exception weblogic.nodemanager.NMConnectException: Connection refused

I use a Oracle WebLogic Server 12c and when I start it on Eclipse it shows this error, it does'nt allow me to log in localhost neither when I try to access it
from another computer in a different domain, in the log appears that the server is not reacheable.
This is the exception that appears in Eclipse.
This Exception occurred at Wed Nov 13 08:11:07 CET 2019.
weblogic.nodemanager.NMConnectException: Connection refused: connect. Could not connect to NodeManager. Check that it is running at localhost/127.0.0.1:5556.
Problem invoking WLST - Traceback (innermost last):
File "C:\Oracle\Middleware\Oracle_Home\user_projects\domains\base_domain\bin\scripts_manejados\StartBT.wlst", line 1, in ?
File "<iostream>", line 111, in nmConnect
File "<iostream>", line 552, in raiseWLSTException
WLSTException: Error occurred while performing nmConnect : Cannot connect to Node Manager. : Connection refused: connect. Could not connect to NodeManager. Check that it is running at localhost/127.0.0.1:5545.
This is my nodemanager properties:
#Tue Nov 12 09:45:58 CET 2019
#Node manager properties
#Fri Jun 24 14:55:43 CEST 2016
DomainsFile=C\:\\Oracle\\Middleware\\Oracle_Home\\user_projects\\domains\\base_domain\\nodemanager\\nodemanager.domains
LogLimit=0
PropertiesVersion=12.1.3
AuthenticationEnabled=true
NodeManagerHome=C\:\\Oracle\\Middleware\\Oracle_Home\\user_projects\\domains\\base_domain\\nodemanager
JavaHome=C\:\\Program Files\\Java\\jdk1.7.0_75
LogLevel=INFO
DomainsFileEnabled=true
StartScriptName=startWebLogic.cmd
ListenAddress=localhost
NativeVersionEnabled=true
ListenPort=5540
LogToStderr=true
SecureListener=false
LogCount=1
StopScriptEnabled=false
QuitEnabled=false
LogAppend=true
StateCheckInterval=500
CrashRecoveryEnabled=false
StartScriptEnabled=true
LogFile=C\:\\Oracle\\Middleware\\Oracle_Home\\user_projects\\domains\\base_domain\\nodemanager\\nodemanager.log
LogFormatter=weblogic.nodemanager.server.LogFormatter
ListenBacklog=50
And this is my StartBt script file, used to start the server and specify some preferences:
nmConnect('weblogic','AXLWL20040','localhost','5521','base_domain','C:/Oracle/Middleware/Oracle_Home/user_projects/domains/base_domain','plain')
nmStart('AdminServer')
nmDisconnect()
I want to make it accesible, not only for local.
nmConnect('weblogic','AXLWL20040','localhost',**'5521'**,'base_domain','C:/Oracle/Middleware/Oracle_Home/user_projects/domains/base_domain','plain')
Port should listen to 5545.
You wrongly configure the ports. Your NodeManager listen on port 5540
ListenPort=5540
You try to connect to 5545
WLSTException:..Could not connect to NodeManager. Check that it is running at localhost/127.0.0.1:5545

ELK configuration for my application logs forward to elastic search using log stash

I am new in ELK configuration.
https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-16-04
I have configure in my local machine and it is work fine.
I want to forward my application logs file to elastic search using log-stash of file beats.
When I have configure all things working fine for system logs.
but I am not able to store my application log to elastic search.
Please help me.
This is my log file:
service.log
{"name":"service name", "hostname":"abc", "pid":4474, "userId":"123", "school_id":"123", "role":"student", "username":"mahi123", "serviceName":"loginService", "level":40, "msg":"successFully fetch trail log", "time":"2019-06-01T10:55:46.482Z","v":0}
Some troubleshooting steps to take care of when logs do not reach Elastisearch:
Check your log parsing configuration file(usually made with the extension .conf). Make sure it's having the right path to scan logs from, right set of filters etc. To see if this .conf file is actually working, one can try:
logstash -f <elasticsearch.conf file path> If this doesn't throw any error on console, that means you are good at this point and will have to move to next step.
Check if Kibana indices are getting created. Run
curl http://<hostipaddress or localhost>:9200/_cat/indices?v.
If yes, go to Kibana Management and create index patterns.
If not, check if your system has enough available memory to serve logstash and elastisearch. free -m would be helpful once you start logstash and elasticsearch services.
Many a times, I have seen people trying ELK setup on a machine which has insufficient RAM(4GB sounds good for a standalone setup).
Check your logstash and Elasticsearch services are up and running. If Elasticsearch is getting down or getting restarted during log parsing or indices creation, that's most probably due to lack of system resources.
-bash-4.2# systemctl status elasticsearch
�� elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-06-05 14:08:26 UTC; 1 weeks 0 days ago
Docs: http://www.elastic.co
Main PID: 1396 (java)
CGroup: /system.slice/elasticsearch.service
������1396 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMS...
Jun 05 14:08:26 cue-bldsvr4 systemd[1]: Started Elasticsearch.
Jun 05 14:08:26 cue-bldsvr4 systemd[1]: Starting Elasticsearch...
-bash-4.2# systemctl status logstash
�� logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-06-05 14:50:52 UTC; 1 weeks 0 days ago
Main PID: 4320 (java)
CGroup: /system.slice/logstash.service
������4320 /bin/java -Xms256m -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFrac...
Jun 05 14:50:52 cue-bldsvr4 systemd[1]: Started logstash.
Jun 05 14:50:52 cue-bldsvr4 systemd[1]: Starting logstash...
Jun 05 14:51:08 cue-bldsvr4 logstash[4320]: Sending Logstash's logs to /var/log/logstash which is now configur...rties
Hint: Some lines were ellipsized, use -l to show in full.
-bash-4.2#

Error while starting the Standalone Agent Using Node Manager : "Unrecognized SSL message, plaintext connection?"

I'm trying to start the Standalone Agent Using Node Manager with this tutorial: https://docs.oracle.com/en/middleware/data-integrator/12.2.1.3/tutorial-creating-standalone-agent/#StartingtheStandaloneAgentUsingNodeManager
(Windows)
Node manager has been started with command:
startNodeManager.cmd
But can't start the agent OracleDIAgent1 with command:
startComponent.cmd OracleDIAgent1
Full output:
Please enter Node Manager password:
Connecting to Node Manager ...
<Jan 15, 2019 4:54:57 PM MSK> <Info> <Security> <BEA-090905> <Disabling the CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true.>
<Jan 15, 2019 4:54:57 PM MSK> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG128 to HMACDRBG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true.>
<Jan 15, 2019 4:54:58 PM MSK> <Info> <Security> <BEA-090909> <Using the configured custom SSL Hostname Verifier implementation: weblogic.security.utils.SSLWLSHostnameVerifier$NullHostnameVerifier.>
This Exception occurred at Tue Jan 15 16:54:58 MSK 2019.
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
Error: Error occurred while performing nmConnect : Cannot connect to Node Manager. : Unrecognized SSL message, plaintext connection?
Use dumpStack() to view the full stacktrace :
Exiting WebLogic Scripting Tool.
Done
File nodemanager.properties has SecureListener off:
SecureListener=false
Please help to start agent. Thanks a lot in advanced.
Stop your node manager then try to change the value as "SecureListener=false" and in the console, verify the node manager protocol is "plain". Restart your node manager.

KrbException connecting to Hadoop cluster with Zookeeper client - UNKNOWN_SERVER

My Zookeeper client is having trouble connecting to the Hadoop cluster.
This works fine from a Linux VM, but I am using a Mac.
I set the -Dsun.security.krb5.debug=true flag on the JVM and get the following output:
Found ticket for solr#DDA.MYCO.COM to go to krbtgt/DDA.MYCO.COM#DDA.MYCO.COM expiring on Sat Apr 29 03:15:04 BST 2017
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for solr#DDA.MYCO.COM to go to krbtgt/DDA.MYCO.COM#DDA.MYCO.COM expiring on Sat Apr 29 03:15:04 BST 2017
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 17 16 23.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
>>> KrbKdcReq send: kdc=oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com UDP:88, timeout=30000, number of retries =3, #bytes=682
>>> KDCCommunication: kdc=oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com UDP:88, timeout=30000,Attempt =1, #bytes=682
>>> KrbKdcReq send: #bytes read=217
>>> KdcAccessibility: remove oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com
>>> KDCRep: init() encoding tag is 126 req type is 13
>>>KRBError:
cTime is Thu Dec 24 11:18:15 GMT 2015 1450955895000
sTime is Fri Apr 28 15:15:06 BST 2017 1493388906000
suSec is 925863
error code is 7
error Message is Server not found in Kerberos database
cname is solr#DDA.MYCO.COM
sname is zookeeper/oc-10-252-132-160.nat-ucfc2z3b.usdv1.mycloud.com#DDA.MYCO.COM
msgType is 30
KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
at org.apache.zookeeper.client.ZooKeeperSaslClient$2.run(ZooKeeperSaslClient.java:366)
at org.apache.zookeeper.client.ZooKeeperSaslClient$2.run(ZooKeeperSaslClient.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:362)
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:348)
at org.apache.zookeeper.client.ZooKeeperSaslClient.sendSaslPacket(ZooKeeperSaslClient.java:420)
at org.apache.zookeeper.client.ZooKeeperSaslClient.initialize(ZooKeeperSaslClient.java:458)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1057)
Caused by: KrbException: Identifier doesn't match expected value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
... 18 more
ERROR 2017-04-28 15:15:07,046 5539 org.apache.zookeeper.client.ZooKeeperSaslClient [main-SendThread(oc-10-252-132-160.nat-ucfc2z3b.usdv1.mycloud.com:2181)]
An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)])
occurred when evaluating Zookeeper Quorum Member's received SASL token.
This may be caused by Java's being unable to resolve the Zookeeper Quorum Member's hostname correctly.
You may want to try to adding '-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS environment.
Zookeeper Client will go to AUTH_FAILED state.
I've tested Kerberos config as follows:
>kinit -kt /etc/security/keytabs/solr.headless.keytab solr
>klist
Credentials cache: API:3451691D-7D5E-49FD-A27C-135816F33E4D
Principal: solr#DDA.MYCO.COM
Issued Expires Principal
Apr 28 16:58:02 2017 Apr 29 04:58:02 2017 krbtgt/DDA.MYCO.COM#DDA.MYCO.COM
Following the instructions from hortonworks I managed to get the kerberos ticket stored in a file:
>klist -c FILE:/tmp/krb5cc_501
Credentials cache: FILE:/tmp/krb5cc_501
Principal: solr#DDA.MYCO.COM
Issued Expires Principal
Apr 28 17:10:25 2017 Apr 29 05:10:25 2017 krbtgt/DDA.MYCO.COM#DDA.MYCO.COM
Also I tried the suggested JVM option suggested in the stack trace (-Dsun.net.spi.nameservice.provider.1=dns,sun), but this led to a different error along the lines of Client session timed out, which suggests that this JVM param is preventing the client from connecting correctly in the first place.
==EDIT==
Seems that the Mac version of Kerberos is not the latest:
> krb5-config --version
Kerberos 5 release 1.7-prerelease
I just tried brew install krb5 to install a newer version, then adjusting the path to point to the new version.
> krb5-config --version
Kerberos 5 release 1.15.1
This has had no effect whatsoever on the outcome.
NB this works fine from a linux VM on my Mac, using exactly the same jaas.conf, keytab files, and krb5.conf.
krb5.conf:
[libdefaults]
renew_lifetime = 7d
forwardable = true
default_realm = DDA.MYCO.COM
ticket_lifetime = 24h
dns_lookup_realm = false
dns_lookup_kdc = false
[realms]
DDA.MYCO.COM = {
admin_server = oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com
kdc = oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com
}
Reverse DNS:
I checked that the FQDN hostname I'm connecting to can be found using a reverse DNS lookup:
> host 10.252.132.160
160.132.252.10.in-addr.arpa domain name pointer oc-10-252-132-160.nat-ucfc2z3b.usdv1.mycloud.com.
This is exactly as per the response to the same command from the linux VM.
===WIRESHARK ANALYSIS===
Using Wireshark configured to use the system key tabs allows a bit more detail in the analysis.
Here I have found that a failed call looks like this:
client -> host AS-REQ
host -> client AS-REP
client -> host AS-REQ
host -> client AS-REP
client -> host TGS-REQ <-- this call is detailed below
host -> client KRB error KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN
The erroneous TGS-REQ call shows the following:
Kerberos
tgs-req
pvno: 5
msg-type: krb-tgs-req (12)
padata: 1 item
req-body
Padding: 0
kdc-options: 40000000 (forwardable)
realm: DDA.MYCO.COM
sname
name-type: kRB5-NT-UNKNOWN (0)
sname-string: 2 items
SNameString: zookeeper
SNameString: oc-10-252-134-51.nat-ucfc2z3b.usdv1.mycloud.com
till: 1970-01-01 00:00:00 (UTC)
nonce: 797021964
etype: 3 items
ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA1-96 (17)
ENCTYPE: eTYPE-DES3-CBC-SHA1 (16)
ENCTYPE: eTYPE-ARCFOUR-HMAC-MD5 (23)
Here is the corresponding successful call from the linux box, which is followed by several more exchanges.
Kerberos
tgs-req
pvno: 5
msg-type: krb-tgs-req (12)
padata: 1 item
req-body
Padding: 0
kdc-options: 40000000 (forwardable)
realm: DDA.MYCO.COM
sname
name-type: kRB5-NT-UNKNOWN (0)
sname-string: 2 items
SNameString: zookeeper
SNameString: d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
till: 1970-01-01 00:00:00 (UTC)
nonce: 681936272
etype: 3 items
ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA1-96 (17)
ENCTYPE: eTYPE-DES3-CBC-SHA1 (16)
ENCTYPE: eTYPE-ARCFOUR-HMAC-MD5 (23)
So it looks like the client is sending
oc-10-252-134-51.nat-ucfc2z3b.usdv1.mycloud.com
as the server host, when it should be sending:
d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
So the question is, how do I fix that? Bear in mind this is a Java piece of code.
My /etc/hosts has the following:
10.252.132.160 b3e073.ddapoc.ucfc2z3b.usdv1.mycloud.com
10.252.134.51 d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
10.252.132.139 d7cc18.ddapoc.ucfc2z3b.usdv1.mycloud.com
And my krb5.conf file has:
kdc = d7cc18.ddapoc.ucfc2z3b.usdv1.mycloud.com
kdc = b3e073.ddapoc.ucfc2z3b.usdv1.mycloud.com
kdc = d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
I tried adding -Dsun.net.spi.nameservice.provider.1=file,dns as a JVM param but got the same result.
I fixed this by setting up a local dnsmasq instance to supply the forward and reverse DNS lookups.
So now from the command line, host d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com returns 10.252.134.51
See also here and here.
Looks like some DNS issue.
Could this SO question help you resolving your problem?
Also, here is an Q&A about the problem.
It also could be because of non Sun JVM.

Resources