How do I use https with PutMarkLogic? - apache-nifi

I'm using DefaultMarkLogicDatabaseClientService 1.9.1.3-incubator in NiFi 1.11.4. MarkLogic 10.0-4 is running AWS and has an app server where SSL is configured at the AWS level.
How do I configure the DefaultMarkLogicDatabaseClientService to use HTTPS without needing an SSL Context Service?
Details:
Before SSL was set up, the DefaultMarkLogicDatabaseClientService was able to connect. Once SSL was set up, I'd get this error:
PutMarkLogic[id=bbb8f3c3-7d83-3fb7-454f-9da7d64fa3f6] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to com.marklogic.client.MarkLogicIOException: java.io.IOException: unexpected end of stream on Connection{my-host:8010, proxy=DIRECT hostAddress=my-host/my-IP:8010 cipherSuite=none protocol=http/1.1}: com.marklogic.client.MarkLogicIOException: java.io.IOException: unexpected end of stream on Connection{my-host:8010, proxy=DIRECT hostAddress=my-ost/my-IP:8010 cipherSuite=none protocol=http/1.1}
Okay, seems like it's not successful using protocol HTTP for a server that needs HTTPS. I see that the service can be configured to use an SSL Context Service, but I'm not looking to do client authentication. (Setting this up requires a truststore or keystore.)
If I replace the PutMarkLogic processor that uses the DefaultMarkLogicDatabaseClientService with an InvokeHTTP processor, I can specify the full URL, including "https://", without needing an SSL Context Services (but then I don't get the batching that I get with PutMarkLogic). I'd like to simply tell the MarkLogic service to use HTTPS.

Creating an SSLContextService with a truststore (that contains the public certificate of the MarkLogic server) populated and no keystore populated should work in this situation.

Related

How to disable SSL verification in Quarkus graphql client

I try to access a graphql api.
The post request goes via a proxy chain of:
local http proxy -> local socks5 proxy -> graphql api host
However, I get java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: Failed to create SSL connection in
io.smallrye.graphql.client.vertx.typesafe.VertxTypesafeGraphQLClientProxy#postSync
I did not find any info or method to disable ssl checks, like verify hosts = false etc. in the used TypesafeGraphQLClientBuildernor in the built client.
How to get rid of the ssl checks in this local dev only test setup?
In debug I see that the built client io.smallrye.graphql.client.vertx.typesafe.VertxTypesafeGraphQLClientProxy
has httpClient.options.verifyHost=true which I would like to set to false.
But I don't know how/where to manipulate or replace the httpClient within the VertxTypesafeGraphQLClientProxy.
Any hints please?

IBM MFP Calling a JS adapter procedure from a JS adapter cause SSLHandshakeException by IHS

I have two javascript adapter:adapterA、adapterB
And I need call adapterA after that the adapterA will call adapterB(use MFP.Server.invokeProcedure, in the same mfp server) using this API
when I call in mfp localhost, it works
http://localhost:9080/mfp/api/adapters/AdapterA/test
then I call https after I import mfp cer to jre cacerts
It works fine too
https://localhost:443/mfp/api/adapters/AdapterA/test
My question is I have IHS Server to redirect mfp services
when I call api by IHS http url
http://{domain}/mfp/api/adapters/AdapterA/test
It works
when I call api by IHS https url
https://{domain}/mfp/api/adapters/AdapterA/test
mfp server will get error like this:
com.ibm.mfp.server.js.adapter.internal.JavascriptManagerImpl E FWLST0904E: Exception was thrown while invoking procedure: test in adapter: adapterB
java.lang.RuntimeException: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: PKIX path building failed: com.ibm.security.cert.IBMCertPathBuilderException: unable to find valid certification path to requested target
at com.ibm.mfp.server.js.adapter.internal.invocation.JavaScriptIntegrationLibraryImplementation.invokeProcedure(JavaScriptIntegrationLibraryImplementation.java:255)
but my IHS plugin only set http
how can I resolve this issue and avoid this issue
thanks
When the MobileFirst server creates the request to reach adapter B, the default behaviour is to frame the request, based on the URL of the currently executing request. That is, it uses the request originally used to reach adapter A, to frame the request to reach the target adapter B.
It works well in case 1, where the webserver is accessed using a "http://.." URL. In case 2, where MFP1 has to make an outbound call to the webserver using the "https://.." URL, it needs to first complete a SSL Handshake with the webserver. In case the MFP1 JVM lacks the certificates of the webserver, it fails to establish SSL Handshake and can lead to the error you saw.
In your case, there are two approaches you can take:
Choose to keep the adapter A to adapter B call internal to MFP1. This prevents the outbound "https://" call and you will not see the problem. Additionally, this helps in keeping the travel time shorter and also prevent a new connection on the webserver. To enable this setting, use the JNDI property mfp.adapter.invocation.url. For instance, if you set the value of this property to "http://localhost:9080/mfp", adapter B will be invoked as "http://localhost:9080/mfp/api/adapters/adapterB". The call stays local. More details on this property here.
If you wish to retain the request to adapter B go through the webserver using the secure endpoint, then you should ensure the webserver's root certificates are made available to the MFP1 JVM's trust store so that SSL handshake can be established successfully.

Conecting to GC Storage from a Pod running SpringBoot application

I've made a SpringBoot application that authenticate with Gloud Storage and performs action on it. It works locally, but when I deploy it on my GKE as a Pod, it suffers some errors.
I have a VPC environment Where I have a Google Cloud Storage, and a Kubernetes Cluster that will run some Spring Boot applications that performs actions on it through com.google.cloud.storage library.
It has Istio enabled for the Cluster and also a Gateway Resource with Secure HTTPS which targets the Ingress Load Balancer as defined here:
https://istio.io/docs/tasks/traffic-management/secure-ingress/sds/
Then my pods all are being reached through a Virtual Service of this Gateway, and it's working fine since they have the Istio-Car container on it and then I can reach them from outside.
So, I have configured this application in DEV environment to get the Credentials from the ENV values:
ENV GOOGLE_APPLICATION_CREDENTIALS="/app/service-account.json"
I know it's not safe, but just wanna make sure it's authenticating. And as I can see through the logs, it is.
As my code manipulates Storages, an Object of this type is needed, I get one by doing so:
this.storage = StorageOptions.getDefaultInstance().getService();
It works fine when running locally. But when I try the same on the Api now running inside the Pod container on GKE, whenever I try to make some interaction to the Storage it returns me some errors like:
[2019-04-25T03:17:40.040Z] [org.apache.juli.logging.DirectJDKLog] [http-nio-8080-exec-1] [175] [ERROR] transactionId=d781f21a-b741-42f0-84e2-60d59b4e1f0a Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.google.cloud.storage.StorageException: Remote host closed connection during handshake] with root cause
java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
...
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:994)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:142)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1011)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:499)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.list(HttpStorageRpc.java:358)
... 65 common frames omitted
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
...
Looks like when I make the call from the Pod, it is expected some extra Https configuration. I don't know right.
So what I'm wondering is:
If this is some kind of Firewall Rule blocking this call from my Pod to "outside" (What is weird since they run on the same network, or at least I thought so).
If it's because of the Gateway I defined that is kind of not enabling this Pod
Or if I need to create the Storage Object using some custom HTTP configurations as can be seen on this reference:
https://googleapis.github.io/google-cloud-java/google-cloud-clients/apidocs/com/google/cloud/storage/StorageOptions.html#getDefaultHttpTransportOptions--
My knowledge of HTTPs and Secure conections is not very good, so maybe my lacking on concept on this area is making me not be able to see something obvious.
If some one have any idea on what maybe causing this, I would appreciate very much.
Solved it. It was really Istio.
I didn't know that we need a Service Entry resource to define what inbound and outbound calls OUTSIDE the mesh.
So, even that GCS is in the same project of the GKE, they are threated as completely separated services.
Just had to create it and everything worked fine:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
namespace: {{ cloud_app_namespace }}
name: external-google-api
spec:
hosts:
- "www.googleapis.com"
- "metadata.google.internal"
- "storage.googleapis.com"
location: MESH_EXTERNAL
ports:
- number: 443
name: https
protocol: HTTPS
- number: 80
name: http
protocol: HTTP
https://istio.io/docs/reference/config/networking/v1alpha3/service-entry/
EDIT
I have disabled the Istio Injection on the namespace I were deploying the applications, by simply using:
kubectl label namespace default istio-injection=disabled --overwrite
Then redeployed the application and tried a curl there, and it worked fine.
My doubt now is: I though Istio only intercept on it's gateway layer, and after that the message keeps untouched, but this is not what seems to be working. Apparently he embbed some SSL layer on the request that my application doesn't do/require.
So sould I need to change my application just to fit on the service mesh requirements?

Nifi SSL configuration on handleHttpRequest

We configured https authentication by using NifiToolKit on Nifi(1.3).
We have a requirement of having https enabled for the rest end points created using - (HandleHTTPRequest -> CustomProcessors -> HandleHTTPResponse).
Configured SSL context service using the same keystore,truststore properties that were generated by the NifiToolkit previously.
Currently i am running Nifi on port 9443 and handleHTTPRequest on 56661.
When i hit the URL with port 56661 from the client , i am getting SSL error even after passing the certificate and the keycode.
import requests
requests.port('https://IPADDRESS:56661/sample',verify='PATH/certificate.pem',json= {})
Can anyone help me on this.. Thanks in advance..
Below is the link which i referred for https configuration on nifi.
https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy

IBM WAS sslSocket.getEnabledProtocols() issue

We have a single installation of WAS(8.5.5.10), with multiple clusters(JVMs). We enabled TLS1.2 on one of the JVMs, (say JVM1). But when I run the code sslSocket.getEnabledProtocols(), it gives me supported protocols=[TLSv1], which is V 1.0. My application runs on Java 1.7.
When I check in Security > SSL certificate and key management, and under Related Items, click SSL configurations. ( such as CellDefaultSSLsetting , NodedefaultSSLsetting and any other SSLConfig), it shows 'TLS' as the protocol, under QoP settings.
But when I check in SSL certificate and key management > Manage endpoint security configurations > jvm1_cluster, it shows 'TLS1.2' as SSL configuration.
From IBM forums, I came to understand that it is possible to enable TLSV1.2 at only one cluster. and need to use jssehelper to specify an outbound ssl configuration.
Tried following the link on how to programmatically specifying an outbound SSL configuration using JSSEHelper API. https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.multiplatform.doc/ae/tsec_ssloutconfiguseJSSE.html
but I still could not get it to work. When I print the properties, I can see protocol as TLSV1.2 (com.ibm.ssl.protocol = TLSv1.2). Application uses com.ibm.jsse2.SSLSocketFactoryImpl for ssl socket provider.
To give you some background, am trying to connect to APNS using okhttp3 library. Am adding these properties, at the place where I create the socket connection. And after the socket is created, when I do sslSocket.getEnabledProtocols(), it gives me supported protocols=[TLSv1], which is V 1.0.
Since it tries to connect via TLS1, it gives javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure, when trying to establish connection to APNS. I have the certificate added to was store already and the firewall rules are in place as well, to allow the connection.
Error:
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at com.ibm.jsse2.j.a(j.java:23)
at com.ibm.jsse2.j.a(j.java:21)
at com.ibm.jsse2.qc.b(qc.java:465)
at com.ibm.jsse2.qc.a(qc.java:451)
at com.ibm.jsse2.qc.h(qc.java:759)
at com.ibm.jsse2.qc.a(qc.java:353)
at com.ibm.jsse2.qc.startHandshake(qc.java:800)
Pls guide me on how to fix the issue. Thanks.

Resources