IBM WebSphere Liberty working with IHS Server Problems with Certificate(s) - websphere-liberty

Question

I have a liberty server and an IHS server (both on different hosts). This is on rhel7.
I've followed a number of guides that I found that walk thru the setup of keystores and sharing of certs but still run into issues. For reference, I tried the methods detailed https://jazz.net/wiki/bin/view/Deployment/CreateIHSPLUGINFORLIBERTYPROFILE and https://www.ibm.com/support/knowledgecenter/en/SSEQTJ_9.0.5/com.ibm.websphere.ihs.doc/ihs/tihs_install_config_liberty.html.
A quick summary of what I have done this past attempt... On the liberty server I added the following to my server.xml to add a plugin include (to keep things easier to read):
<include location="${server.config.dir}/plugin-join-include.xml" />
And the plugin-join-include.xml:
<featureManager>
<feature>ssl-1.0</feature>
</featureManager>
<keyStore id="libertyKey" password="xxxx"
location="${server.config.dir}/resources/security/libkey.jks"/>
<pluginConfiguration webserverPort="10500" webserverSecurePort="10447"
pluginInstallRoot="/opt/IBM/wasadmin/Plugins"
sslKeyRingLocation="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.kdb"
sslStashFileLocation="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.sth"
sslCertlabel="libertyKey"/>
On my IHS server the path for the root Plugin directory is as indicated above in the file at /opt/IBM/wasadmin/Plugins and the plugin-key.kdb and plugin-key.sth are located in the locations listed above.
I reboot the liberty server and it generated the plugin-cfg.xml in the logs/state directory:
<?xml version="1.0" encoding="UTF-8"?><!--HTTP server plugin config file for app generated on 2020.03.04 at 12:32:02 UTC-->
<Config ASDisableNagle="false" AcceptAllContent="false" AppServerPortPreference="HostHeader" ChunkedResponse="false" ConfigHash="1077723051" FIPSEnable="false" IISDisableNagle="false" IISPluginPriority="High" IgnoreDNSFailures="false" RefreshInterval="60" ResponseChunkSize="64" SSLConsolidate="false" TrustedProxyEnable="false" VHostMatchingCompat="false">
<Log LogLevel="Error" Name="/opt/IBM/wasadmin/Plugins/logs/webserver1/http_plugin.log"/>
<Property Name="ESIEnable" Value="true"/>
<Property Name="ESIMaxCacheSize" Value="1024"/>
<Property Name="ESIInvalidationMonitor" Value="false"/>
<Property Name="ESIEnableToPassCookies" Value="false"/>
<Property Name="PluginInstallRoot" Value="/opt/IBM/wasadmin/Plugins"/>
<!-- Configuration generated using httpEndpointRef=defaultHttpEndpoint-->
<!-- The default_host contained only aliases for endpoint defaultHttpEndpoint.
The generated VirtualHostGroup will contain only configured web server ports:
webserverPort=10500
webserverSecurePort=10447 -->
<VirtualHostGroup Name="default_host">
<VirtualHost Name="*:10500"/>
<VirtualHost Name="*:10447"/>
</VirtualHostGroup>
<ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" IgnoreAffinityRequests="true" LoadBalance="Round Robin" Name="affms_default_node_Cluster" PostBufferSize="0" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60" ServerIOTimeoutRetry="-1">
<Server CloneID="412b3187-16c4-41b0-86e8-1e327c1c6b1b" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="default_node_affms" ServerIOTimeout="900" WaitForContinue="false">
<Transport Hostname="libertyhost" Port="10500" Protocol="http"/>
<Transport Hostname="libertyhost" Port="10447" Protocol="https">
<Property Name="keyring" Value="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.kdb"/>
<Property Name="stashfile" Value="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.sth"/>
<Property Name="certLabel" Value="libertyKey"/>
</Transport>
</Server>
<PrimaryServers>
<Server Name="default_node_app"/>
</PrimaryServers>
</ServerCluster>
<UriGroup Name="default_host_app_default_node_Cluster_URIs">
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/app/ui/*"/>
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/adminCenter/*"/>
</UriGroup>
<Route ServerCluster="app_default_node_Cluster" UriGroup="default_host_app_default_node_Cluster_URIs" VirtualHostGroup="default_host"/>
I copy the plugin-cfg.xml over to the IHS server at /opt/IBM/wasadmin/Plugins/config/webserver1/plugin-cfg.xml
The plugin stuff works only the SSL portion between IHS and the app server give me issues.
On the IHS server I have created the plugin-key.kdb keystore with stash plugin-key.sth and imported the cert I extracted from the liberty server /opt/IBM/wlp/usr/servers/app/resources/security/key.p12 the default personal cert and copy it to the IHS server and add the default liberty cert to the plugin-key.kdb keystore.
At the bottom of the httpd.conf is:
LoadModule was_ap22_module /opt/IBM/wasadmin/Plugins/bin/64bits/mod_was_ap22_http.so
WebSpherePluginConfig /opt/IBM/wasadmin/Plugins/config/webserver1/plugin-cfg.xml
I bounce the apachectl and go to the url for the application on port 8443 https://hostname:8443/app/ui and get a "500 Internal Server Error".
The http_plugin.log plugin logs show:
[04/Mar/2020:18:23:48.31652] 00002f26 3cff9700 - ERROR: lib_stream: openStream: Failed in r_gsk_secure_soc_init: GSK_ERROR_BAD_CERT(gsk rc = 414) PARTNER CERTIFICATE DN=CN=hostname,OU=app,O=ibm,C=us, Serial=xxxxxxxxxxx
[04/Mar/2020:18:23:48.31655] 00002f26 3cff9700 - ERROR: Ensure correct certificate is marked as default certificate in plugin-key.kdb. Consult documentation regarding Administering application security and Securing communications for more information.
[04/Mar/2020:18:23:48.31657] 00002f26 3cff9700 - ERROR: Last validation error [575010]: GSKVAL_ERROR_NO_CHAIN_BUILT
[04/Mar/2020:18:23:48.31658] 00002f26 3cff9700 - ERROR: Subject [[Class=]GSKVALMethod::PKIX[Issuer=]OU=memberRoot,O=xxxx-xxx-xxx-xxx-xxxxxxxx,DC=com.ibm.ws.collective[#=]082d6a83e5ec[Subject=]CN=hostname,OU=app,O=ibm,C=us] failed certificate validation
[04/Mar/2020:18:23:48.31659] 00002f26 3cff9700 - ERROR: X509 Certificate validation log: [[Class=]GSKVALMethod::PKIX[Time=]2020:3:4:18:23:48.316[buildChain=][Error=]GSKVAL_ERR_NO_CHAIN_BUILT[Info=]OU=memberRoot,O=xxx-xxx-xxx-xxx-xxx,DC=com.ibm.ws.collective[Cert=][Issuer=]OU=memberRoot,O=xxx-xxx-xxx-xxx-xxxx,DC=com.ibm.ws.collective[#=]082d6a83e5ec[Subject=]CN=hostname,OU=app,O=ibm,C=us[=Cert][=buildChain]^M
]
[04/Mar/2020:18:23:48.31664] 00002f26 3cff9700 - ERROR: ws_common: websphereGetStream: Could not open stream
[04/Mar/2020:18:23:48.31667] 00002f26 3cff9700 - ERROR: ws_common: websphereExecute: Failed to create the stream
[04/Mar/2020:18:23:48.31668] 00002f26 3cff9700 - ERROR: ws_common: websphereHandleRequest: Failed to execute the transaction to 'default_node_app' on host 'hostname:10447'; will try another one
[04/Mar/2020:18:23:48.31669] 00002f26 3cff9700 - ERROR: ws_common: websphereWriteRequestReadResponse: Failed to find an app server to handle this request
[04/Mar/2020:18:23:48.31670] 00002f26 3cff9700 - ERROR: ESI: getResponse: failed to get response: rc = 2
[04/Mar/2020:18:23:48.31671] 00002f26 3cff9700 - ERROR: [xxx.xx.xx.17://app/ui/] ws_common: websphereHandleRequest: Failed to handle request rc=2
Thank you!

Answer 1

You said "imported" I assume this means an "add" operation in ikeyman/gskcmd/gskcapicmd? The two verbs in these tools have a different meaning.
I suspect you added the issued certificate (CN=hostname...) instead of the CA that issued the certificate (memberroot). The debug information in the WAS Plugin message implies that it couldn't chase the certificate up to a trusted root (as opposed to finding some X509 error w/ the root it had access to).
But, it could also be that you imported the "wrong" certificate with cn=memberroot. WebSphere does not always add the necessary Subject Key Identifier/Authority Key Identifier to be sure, so check carefully.

Answer 2

Ok, I got this working finally. It seems having a collective setup which used DefaultKeyStore id was somehow causing conflict with my Liberty plugin keystore.
I ended up re-creating everything (plugin-key.kdb keystore and the liberty plugin keystore which I named LibertyKeystore.jks). I dumped the collective stuff (not needed at this time anyways) and made an include as above called plugin-join-include.xml:
<?xml version="1.0" encoding="UTF-8" ?>
<server description="IHS plugin join include file">
<featureManager>
<feature>ssl-1.0</feature>
</featureManager>
<keyStore id="defaultKeyStore" password="password"
location="${server.config.dir}/resources/security/LibertyKeystore.jks" />
<pluginConfiguration webserverPort="80" webserverSecurePort="8443"
pluginInstallRoot="/opt/IBM/wasadmin/Plugins"
sslKeyRingLocation="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.kdb"
sslStashFileLocation="/opt/IBM/wasadmin/Plugins/config/webserver1/plugin-key.sth"
sslCertlabel="LibertyKeystore"/>
</server>
I exchanged keys between the liberty server LibertyKeystore.jks and the IHS plugin-key.kdb keystores and pushed the newly generated plugin to the IHS server and bounced everything and it all seems to function without issue.
I do now notice in the /opt/IBM/wasadmin/HTTPServer/logs the error log streams:
[Wed Mar 04 21:32:28 2020] [error] [client xxx.xx.xx.17:65261] [7ff5a0000910] [19831] SSL0279E: SSL Handshake Failed due to fatal alert from client. Client sent fatal alert [level 2 (fatal), description 46 (certificate_unknown)] [xxx.xx.xx.17:65261 -> xxx.xx.xx.87:8443] [21:32:28.000934048] 0ms
The access logs show 200's. Maybe a non plugin "issue" at this point.