After cleaning up some tables in MonetDB, trying to connect to mclient and getting kicked out.
monetdbd: internal error while starting mserver, please refer to the logs
merovingian.log ->
2017-02-16 16:40:54 MSG ame[1672]: # MonetDB 5 server v11.21.13 "Jul2015-SP2"
2017-02-16 16:40:54 MSG ame[1672]: # Serving database 'ame', using 8 threads
2017-02-16 16:40:54 MSG ame[1672]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked
2017-02-16 16:40:54 MSG ame[1672]: # Found 29.453 GiB available main-memory.
2017-02-16 16:40:54 MSG ame[1672]: # Copyright (c) 1993-July 2008 CWI.
2017-02-16 16:40:54 MSG ame[1672]: # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved
2017-02-16 16:40:54 MSG ame[1672]: # Visit http://www.monetdb.org/ for further information
2017-02-16 16:40:54 MSG ame[1672]: # Listening for UNIX domain connection requests on mapi:monetdb:///ebs_data/monetdb/ame/.mapi.sock
2017-02-16 16:40:54 MSG ame[1672]: # Start processing logs sql/sql_logs version 52200
2017-02-16 16:40:54 MSG merovingian[1233]: database 'ame' (1672) has exited with exit status 1
2017-02-16 16:40:59 ERR merovingian[1233]: client error: database 'ame' appears to shut itself down after starting, check monetdbd's logfile for possible hints
I can't find any other logs. There is nothing in sql/sql_logs.
Related
I have been using Postgresql 13 on a local server on my Windows 10 computer for over a year successfully now. I decided to upgrade to 14 yesterday.
I installed 14 on a different port. I went to go and upgrade as per this command: pg_upgrade -d "c:\Program Files\PostgreSQL\13\data" -D "c:\Program Files\PostgreSQL\14\data" -b "c:\Program Files\PostgreSQL\13\bin" -B "c:\Program Files\PostgreSQL\14\bin" -U Postgres, but it said it cannot connect to the 13 server. I restarted the computer, and still the 13's Service will not start.
When I type pg_ctl -D "C:\Program Files\PostgreSQL\13\data" start at the cmd line, the below is what shows in my log:
2022-02-15 08:53:45.908 +04 [92100] LOG: starting PostgreSQL 13.3, compiled by Visual C++ build 1914, 64-bit
2022-02-15 08:53:45.909 +04 [92100] LOG: listening on IPv6 address "::", port 5432
2022-02-15 08:53:45.910 +04 [92100] LOG: listening on IPv4 address "0.0.0.0", port 5432
2022-02-15 08:53:45.954 +04 [92672] LOG: database system was shut down at 2022-02-14 14:08:25 +04
2022-02-15 08:53:45.955 +04 [92672] LOG: invalid record length at 24/80B400C8: wanted 24, got 0
2022-02-15 08:53:45.955 +04 [92672] LOG: invalid primary checkpoint record
2022-02-15 08:53:45.955 +04 [92672] PANIC: could not locate a valid checkpoint record
2022-02-15 08:53:46.057 +04 [92100] LOG: startup process (PID 92672) was terminated by exception 0xC0000409
2022-02-15 08:53:46.057 +04 [92100] HINT: See C include file "ntstatus.h" for a description of the hexadecimal value.
2022-02-15 08:53:46.057 +04 [92100] LOG: aborting startup due to startup process failure
2022-02-15 08:53:46.059 +04 [92100] LOG: database system is shut down
What should I do to fix this?
What I have already done is:
PostgreSQL.conf - made sure listen_addresses = '*' was uncommented
made sure Modify was valid for all users of the computer on the Postgres Programs folder
I checked that the Postgres user had full rights to the folder, but there was no user found in my windows.
I added to the pg_hba file: #host all all 0.0.0.0/0 scram-sha-256
oh, and in case it's not apparent, I don't know much about Postgres. I can use it for what I need and that's about it.
As JJanes suggested. I did a backup and then restored in the new server of Postgres 14. It solved the problem. Thank you, JJanes.
I was taking an RMAN backup.
RMAN> SHUTDOWN IMMEDIATE;
After Shutdown Immediate i am not able to startup database.
I have tried
conn sys#databasename as sysdba
RMAN target sys#database
lsnrctl stop start status
But i am getting this error.
C:\Users\Umair>rman target sys#ora_prep
Recovery Manager: Release 12.1.0.2.0 - Production on Mon Jul 13 00:18:00 2020
Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved.
target database Password:
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00554: initialization of internal recovery manager package failed
RMAN-04005: error from target database:
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
Advice, please.Thank you.
Oracle 12.1.0
Windows 10
Static listener registration:
https://docs.oracle.com/en/database/oracle/oracle-database/18/spmss/adding-static-service-to-listener.html
https://support.oracle.com/knowledge/Oracle%20Database%20Products/2312510_1.html
In Windows 10, I'm not able to connect to MongoDB server with the following errors:
>mongo "mongodb+srv://xxx-dsvlb.mongodb.net/test" --username xxx --verbose
2018-11-01T11:34:19.273+0700 D - [main] User Assertion: DNSHostNotFound: Failed to look up service "":This operation returned because the timeout period expired. C:\data\mci\6411135b04f345f6d01072b56250cba6\src\src\mongo/util/dns_query_windows-impl.h 254
MongoDB shell version v4.0.3
2018-11-01T11:34:30.535+0700 D - [main] User Assertion: DNSHostNotFound: Failed to look up service "":This operation returned because the timeout period expired. C:\data\mci\6411135b04f345f6d01072b56250cba6\src\src\mongo/util/dns_query_windows-impl.h 254
Enter password:
connecting to: mongodb+srv://xxx-dsvlb.mongodb.net/test
2018-11-01T11:35:16.589+0700 D - [js] User Assertion: DNSHostNotFound: Failed to look up service "":This operation returned because the timeout period expired. C:\data\mci\6411135b04f345f6d01072b56250cba6\src\src\mongo/util/dns_query_windows-impl.h 254
2018-11-01T11:35:16.590+0700 D NETWORK [js] creating new connection to:xxx-shard-00-02-dsvlb.mongodb.net.:27017
2018-11-01T11:35:17.356+0700 D - [js] User Assertion: SSLHandshakeFailed: QueryContextAttributes for connection info failed with-2146893055 C:\data\mci\6411135b04f345f6d01072b56250cba6\src\src\mongo/transport/session_asio.h 240
2018-11-01T11:35:17.357+0700 D NETWORK [js] creating new connection to:xxx-shard-00-01-dsvlb.mongodb.net.:27017
2018-11-01T11:35:18.197+0700 D - [js] User Assertion: SSLHandshakeFailed: QueryContextAttributes for connection info failed with-2146893055 C:\data\mci\6411135b04f345f6d01072b56250cba6\src\src\mongo/transport/session_asio.h 240
2018-11-01T11:35:18.198+0700 D NETWORK [js] creating new connection to:xx-shard-00-00-dsvlb.mongodb.net.:27017
2018-11-01T11:35:19.017+0700 D - [js] User Assertion: SSLHandshakeFailed: QueryContextAttributes for connection info failed with-2146893055 C:\data\mci\6411135b04f345f6d01072b56250cba6\src\src\mongo/transport/session_asio.h 240
2018-11-01T11:35:19.018+0700 D - [js] User Assertion: InternalError: couldn't connect to server lakon-shard-00-00-dsvlb.mongodb.net.:27017, connection attempt failed: SSLHandshakeFailed: QueryContextAttributes for connection info failed with-2146893055 src\mongo\scripting\mozjs\mongo.cpp 756
2018-11-01T11:35:19.021+0700 E QUERY [js] Error: couldn't connect to server lakon-shard-00-00-dsvlb.mongodb.net.:27017, connection attempt failed: SSLHandshakeFailed: QueryContextAttributes for connection info failed with-2146893055 :
connect#src/mongo/shell/mongo.js:257:13
#(connect):1:6
2018-11-01T11:35:19.024+0700 D - [js] User Assertion: Location12513: connect failed src\mongo\shell\shell_utils.cpp 343
2018-11-01T11:35:19.024+0700 I QUERY [js] MozJS GC prologue heap stats - total: 4056565 limit: 0
2018-11-01T11:35:19.027+0700 I QUERY [js] MozJS GC epilogue heap stats - total: 421536 limit: 0
2018-11-01T11:35:19.027+0700 I QUERY [js] MozJS GC prologue heap stats - total: 313504 limit: 0
2018-11-01T11:35:19.028+0700 I QUERY [js] MozJS GC epilogue heap stats - total: 131244 limit: 0
2018-11-01T11:35:19.029+0700 D - [main] User Assertion: Location12513: connect failed src\mongo\scripting\mozjs\proxyscope.cpp 300
exception: connect failed
Using MongoDB shell 3.6.2 on Windows 10, I still cannot connect but with a different error (confusing, isn't it?):
>mongo "mongodb+srv://xxx-dsvlb.mongodb.net/test" --username xxx --password xxx
MongoDB shell version v3.6.2
connecting to: mongodb+srv://xxx-dsvlb.mongodb.net/test
MongoDB server version: 3.6.8
2018-11-01T11:01:52.923+0700 E QUERY [thread1] Error: Authentication failed. :
DB.prototype._authOrThrow#src/mongo/shell/db.js:1608:20
#(auth):6:1
#(auth):1:2
exception: login failed
However, with Ubuntu 16.04 I can connect just fine to the exact same server:
⟫ mongo "mongodb+srv://xxx-dsvlb.mongodb.net/test" --username xxx --password xxx
MongoDB shell version v4.0.3
connecting to: mongodb+srv://xxx-dsvlb.mongodb.net/test
2018-11-01T04:27:02.536+0000 I NETWORK [js] Starting new replica set monitor for lakon-shard-0/xxx-shard-00-02-dsvlb.mongodb.net.:27017,xxx-shard-00-00-dsvlb.mongodb.net.:27017,xxx-shard-00-01-dsvlb.mongodb.net.:27017
2018-11-01T04:27:02.561+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor] Successfully connected to xxx-shard-00-02-dsvlb.mongodb.net.:27017 (1 connections now open to xxx-shard-00-02-dsvlb.mongodb.net.:27017 with a 5 second timeout)
2018-11-01T04:27:02.562+0000 I NETWORK [js] Successfully connected to xxx-shard-00-00-dsvlb.mongodb.net.:27017 (1 connections now open to xxx-shard-00-00-dsvlb.mongodb.net.:27017 with a 5 second timeout)
2018-11-01T04:27:02.563+0000 I NETWORK [js] changing hosts to xxx-shard-0/xxx-shard-00-00-dsvlb.mongodb.net:27017,xxx-shard-00-01-dsvlb.mongodb.net:27017,lakon-shard-00-02-dsvlb.mongodb.net:27017 from xxx-shard-0/xxx-shard-00-00-dsvlb.mongodb.net.:27017,xxx-shard-00-01-dsvlb.mongodb.net.:27017,xxx-shard-00-02-dsvlb.mongodb.net.:27017
2018-11-01T04:27:02.570+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor] Successfully connected to xxx-shard-00-00-dsvlb.mongodb.net:27017 (1 connections now open to xxx-shard-00-00-dsvlb.mongodb.net:27017 with a 5 second timeout)
2018-11-01T04:27:02.573+0000 I NETWORK [js] Successfully connected to xxx-shard-00-02-dsvlb.mongodb.net:27017 (1 connections now open to xxx-shard-00-02-dsvlb.mongodb.net:27017 with a 5 second timeout)
Implicit session: session { "id" : UUID("4a6488c7-7a22-44d4-977e-07eb09ef37f6") }
MongoDB server version: 3.6.8
WARNING: shell and server versions do not match
2018-11-01T04:27:02.588+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor] Successfully connected to xxx-shard-00-01-dsvlb.mongodb.net:27017 (1 connections now open to xxx-shard-00-01-dsvlb.mongodb.net:27017 with a 5 second timeout)
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
MongoDB Enterprise xxx-shard-0:PRIMARY>
A MongoDB Atlas support staff told me this is due to network connection on my part, but I'm sure that is not the root issue, because I can connect to the server when using other client such as Robo 3T using the same Windows 10 computer.
This issue happens ONLY when using MongoDB shell (both 3.6.2 and 4.0.3) in Windows 10.
It's probably a bug with MongoDB Shell and Windows 10 implementation?
it's a bit late but here, I had this problem when my shell's version is 4.0.5 then I install 4.2.11, it solve this problem. I tried many things with different connection string syntax and it does not solve the problem and still stuck at SSLHandshake error, so I guess if version were same it might solve the problem(mine 4.0.5 and remote was 4.2.11) and go ahead with new version installation(though I still think it's not version problem but I don't know what is). This problem only happen while I was in shell, connecting from the client like NoSqlBooster or Spring seem to work fine. My Robo3T have a problem connecting but randomly, sometimes once sometimes multiple re-trying.
I have installed arangodb through brew. I am new to both mac and arangodb. Right after installation of arangodb I could start stop it through brew services. But since yesterday that didn't work. However arangod start worked. Today its taking really long time for the service to start up
$ arangod start
2018-04-30T07:40:32Z [3593] INFO ArangoDB 3.3.7 [darwin] 64bit, using jemalloc, build , VPack 0.1.30, RocksDB 5.6.0, ICU 58.1, V8 5.7.492.77, OpenSSL 1.0.2o 27 Mar 2018
2018-04-30T07:40:32Z [3593] INFO {authentication} Jwt secret not specified, generating...
2018-04-30T07:40:32Z [3593] INFO using storage engine mmfiles
2018-04-30T07:40:32Z [3593] INFO {cluster} Starting up with role SINGLE
2018-04-30T07:40:32Z [3593] INFO {syscall} file-descriptors (nofiles) hard limit is unlimited, soft limit is 8192
2018-04-30T07:40:32Z [3593] INFO {authentication} Authentication is turned on (system only), authentication for unix sockets is turned on
2018-04-30T07:40:32Z [3593] INFO running WAL recovery (1 logfiles)
2018-04-30T07:40:32Z [3593] INFO replaying WAL logfile '/Users/neel/start/journals/logfile-17009.db' (1 of 1)
2018-04-30T07:40:32Z [3593] INFO WAL recovery finished successfully
2018-04-30T07:40:33Z [3593] INFO using endpoint 'http+tcp://127.0.0.1:8529' for non-encrypted requests
2018-04-30T07:41:33Z [3593] WARNING {v8} giving up waiting for unused V8 context after 60.000000 s
2018-04-30T07:41:43Z [3593] WARNING {v8} giving up waiting for unused V8 context after 60.000000 s
2018-04-30T07:42:34Z [3593] WARNING {v8} giving up waiting for unused V8 context after 60.000000 s
2018-04-30T07:43:05Z [3593] INFO ArangoDB (version 3.3.7 [darwin]) is ready for business. Have fun!
I don't know where are the log files. So when I try to start with brew services start arangodb I can't check whether it has been started or not as it responds Successfully startedarangodb(label: homebrew.mxcl.arangodb) immediately. So my questions are why its delaying ? and where are the log files ?
The log files are located here: /usr/local/var/log/arangodb3
The delay above is caused by lack of available V8 contexts. You can adjust them by setting them in /usr/local/etc/arangodb3/arangod.conf. But the default value there is set to 0, which means that arangodb is to choose how many are running.
My Zookeeper client is having trouble connecting to the Hadoop cluster.
This works fine from a Linux VM, but I am using a Mac.
I set the -Dsun.security.krb5.debug=true flag on the JVM and get the following output:
Found ticket for solr#DDA.MYCO.COM to go to krbtgt/DDA.MYCO.COM#DDA.MYCO.COM expiring on Sat Apr 29 03:15:04 BST 2017
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for solr#DDA.MYCO.COM to go to krbtgt/DDA.MYCO.COM#DDA.MYCO.COM expiring on Sat Apr 29 03:15:04 BST 2017
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 17 16 23.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
>>> KrbKdcReq send: kdc=oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com UDP:88, timeout=30000, number of retries =3, #bytes=682
>>> KDCCommunication: kdc=oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com UDP:88, timeout=30000,Attempt =1, #bytes=682
>>> KrbKdcReq send: #bytes read=217
>>> KdcAccessibility: remove oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com
>>> KDCRep: init() encoding tag is 126 req type is 13
>>>KRBError:
cTime is Thu Dec 24 11:18:15 GMT 2015 1450955895000
sTime is Fri Apr 28 15:15:06 BST 2017 1493388906000
suSec is 925863
error code is 7
error Message is Server not found in Kerberos database
cname is solr#DDA.MYCO.COM
sname is zookeeper/oc-10-252-132-160.nat-ucfc2z3b.usdv1.mycloud.com#DDA.MYCO.COM
msgType is 30
KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
at org.apache.zookeeper.client.ZooKeeperSaslClient$2.run(ZooKeeperSaslClient.java:366)
at org.apache.zookeeper.client.ZooKeeperSaslClient$2.run(ZooKeeperSaslClient.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:362)
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:348)
at org.apache.zookeeper.client.ZooKeeperSaslClient.sendSaslPacket(ZooKeeperSaslClient.java:420)
at org.apache.zookeeper.client.ZooKeeperSaslClient.initialize(ZooKeeperSaslClient.java:458)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1057)
Caused by: KrbException: Identifier doesn't match expected value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
... 18 more
ERROR 2017-04-28 15:15:07,046 5539 org.apache.zookeeper.client.ZooKeeperSaslClient [main-SendThread(oc-10-252-132-160.nat-ucfc2z3b.usdv1.mycloud.com:2181)]
An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)])
occurred when evaluating Zookeeper Quorum Member's received SASL token.
This may be caused by Java's being unable to resolve the Zookeeper Quorum Member's hostname correctly.
You may want to try to adding '-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS environment.
Zookeeper Client will go to AUTH_FAILED state.
I've tested Kerberos config as follows:
>kinit -kt /etc/security/keytabs/solr.headless.keytab solr
>klist
Credentials cache: API:3451691D-7D5E-49FD-A27C-135816F33E4D
Principal: solr#DDA.MYCO.COM
Issued Expires Principal
Apr 28 16:58:02 2017 Apr 29 04:58:02 2017 krbtgt/DDA.MYCO.COM#DDA.MYCO.COM
Following the instructions from hortonworks I managed to get the kerberos ticket stored in a file:
>klist -c FILE:/tmp/krb5cc_501
Credentials cache: FILE:/tmp/krb5cc_501
Principal: solr#DDA.MYCO.COM
Issued Expires Principal
Apr 28 17:10:25 2017 Apr 29 05:10:25 2017 krbtgt/DDA.MYCO.COM#DDA.MYCO.COM
Also I tried the suggested JVM option suggested in the stack trace (-Dsun.net.spi.nameservice.provider.1=dns,sun), but this led to a different error along the lines of Client session timed out, which suggests that this JVM param is preventing the client from connecting correctly in the first place.
==EDIT==
Seems that the Mac version of Kerberos is not the latest:
> krb5-config --version
Kerberos 5 release 1.7-prerelease
I just tried brew install krb5 to install a newer version, then adjusting the path to point to the new version.
> krb5-config --version
Kerberos 5 release 1.15.1
This has had no effect whatsoever on the outcome.
NB this works fine from a linux VM on my Mac, using exactly the same jaas.conf, keytab files, and krb5.conf.
krb5.conf:
[libdefaults]
renew_lifetime = 7d
forwardable = true
default_realm = DDA.MYCO.COM
ticket_lifetime = 24h
dns_lookup_realm = false
dns_lookup_kdc = false
[realms]
DDA.MYCO.COM = {
admin_server = oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com
kdc = oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com
}
Reverse DNS:
I checked that the FQDN hostname I'm connecting to can be found using a reverse DNS lookup:
> host 10.252.132.160
160.132.252.10.in-addr.arpa domain name pointer oc-10-252-132-160.nat-ucfc2z3b.usdv1.mycloud.com.
This is exactly as per the response to the same command from the linux VM.
===WIRESHARK ANALYSIS===
Using Wireshark configured to use the system key tabs allows a bit more detail in the analysis.
Here I have found that a failed call looks like this:
client -> host AS-REQ
host -> client AS-REP
client -> host AS-REQ
host -> client AS-REP
client -> host TGS-REQ <-- this call is detailed below
host -> client KRB error KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN
The erroneous TGS-REQ call shows the following:
Kerberos
tgs-req
pvno: 5
msg-type: krb-tgs-req (12)
padata: 1 item
req-body
Padding: 0
kdc-options: 40000000 (forwardable)
realm: DDA.MYCO.COM
sname
name-type: kRB5-NT-UNKNOWN (0)
sname-string: 2 items
SNameString: zookeeper
SNameString: oc-10-252-134-51.nat-ucfc2z3b.usdv1.mycloud.com
till: 1970-01-01 00:00:00 (UTC)
nonce: 797021964
etype: 3 items
ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA1-96 (17)
ENCTYPE: eTYPE-DES3-CBC-SHA1 (16)
ENCTYPE: eTYPE-ARCFOUR-HMAC-MD5 (23)
Here is the corresponding successful call from the linux box, which is followed by several more exchanges.
Kerberos
tgs-req
pvno: 5
msg-type: krb-tgs-req (12)
padata: 1 item
req-body
Padding: 0
kdc-options: 40000000 (forwardable)
realm: DDA.MYCO.COM
sname
name-type: kRB5-NT-UNKNOWN (0)
sname-string: 2 items
SNameString: zookeeper
SNameString: d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
till: 1970-01-01 00:00:00 (UTC)
nonce: 681936272
etype: 3 items
ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA1-96 (17)
ENCTYPE: eTYPE-DES3-CBC-SHA1 (16)
ENCTYPE: eTYPE-ARCFOUR-HMAC-MD5 (23)
So it looks like the client is sending
oc-10-252-134-51.nat-ucfc2z3b.usdv1.mycloud.com
as the server host, when it should be sending:
d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
So the question is, how do I fix that? Bear in mind this is a Java piece of code.
My /etc/hosts has the following:
10.252.132.160 b3e073.ddapoc.ucfc2z3b.usdv1.mycloud.com
10.252.134.51 d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
10.252.132.139 d7cc18.ddapoc.ucfc2z3b.usdv1.mycloud.com
And my krb5.conf file has:
kdc = d7cc18.ddapoc.ucfc2z3b.usdv1.mycloud.com
kdc = b3e073.ddapoc.ucfc2z3b.usdv1.mycloud.com
kdc = d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
I tried adding -Dsun.net.spi.nameservice.provider.1=file,dns as a JVM param but got the same result.
I fixed this by setting up a local dnsmasq instance to supply the forward and reverse DNS lookups.
So now from the command line, host d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com returns 10.252.134.51
See also here and here.
Looks like some DNS issue.
Could this SO question help you resolving your problem?
Also, here is an Q&A about the problem.
It also could be because of non Sun JVM.