unable to uprade marklogic data hub framework using gradle - gradle

I am trying to follow the recommendation to upgrade the DHF using gradle but I am running through an issue that I cannot het my head around.
The build succeeds but the redeployment fails
Any idea on how to fix this ?
note that the login info is provided properly in the gradle.properties
> Task :hubDeploySecurity FAILED
Task ':hubDeploySecurity' is not up-to-date because:
Task has not declared any outputs despite executing actions.
Deploying app DHF with config dirs: [/src/main/hub-internal-config, /src/main/ml-config]
Executing command [com.marklogic.appdeployer.command.security.DeployPrivilegesCommand] with sort order [5]
Will read and merge resource files in each config path before saving any resources
Processing files in directory: /src/main/hub-internal-config/security/privileges
Checking to see if Configuration Management API is available at: /manage/v3
Sending JSON POST request as user 'tkadmin' (who should have the 'manage-admin' and 'security' roles) to path: /manage/v3
Error occurred while sending POST request to /manage/v3; logging request body to assist with debugging: {}
Processing file: /src/main/hub-internal-config/security/privileges/dhf-internal-data-hub.json
Processing file: /src/main/hub-internal-config/security/privileges/dhf-internal-entities.json
Processing file: /src/main/hub-internal-config/security/privileges/dhf-internal-mappings.json
Processing file: /src/main/hub-internal-config/security/privileges/dhf-internal-trace-ui.json
Processing files in directory: /src/main/ml-config/security/privileges
Checking to see if Configuration Management API is available at: /manage/v3
Sending JSON POST request as user 'tkadmin' (who should have the 'manage-admin' and 'security' roles) to path: /manage/v3
Error occurred while sending POST request to /manage/v3; logging request body to assist with debugging: {}
Merging payloads that reference the same resource
Checking to see if Configuration Management API is available at: /manage/v3
Sending JSON POST request as user 'tkadmin' (who should have the 'manage-admin' and 'security' roles) to path: /manage/v3
Error occurred while sending POST request to /manage/v3; logging request body to assist with debugging: {}
Checking for existence of resource: dhf-internal-data-hub
Sending XML GET request as user 'tkadmin' (who should have the 'manage-admin' and 'security' roles) to path: /manage/v2/privileges
Logging HTTP response body to assist with debugging: {"errorResponse": {"statusCode":401,
"status":"Unauthorized",
"message":"401 Unauthorized"
}
}
:hubDeploySecurity (Thread[Execution worker for ':',5,main]) completed. Took 0.01 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':hubDeploySecurity'.
> 401 Unauthorized: [{"errorResponse": {"statusCode":401,
"status":"Unauthorized",
"message":"401 Unauthorized"
}
}]

Assuming you have followed the DHF upgrade matrix:
https://docs.marklogic.com/datahub/5.2/upgrade.html
You probably ran the Gradle with an incorrect Admin interface & Security user. As such, the hubUpdate REST API requests will fail.
Try below see if it works:
Step 2
gradle hubUpdate -i -PmlUsername=admin -PmlPassword={admin-password} -Penvironment={env-name}
Step 4
gradle mlRedeploy -i -PmlUsername=admin -PmlPassword={admin-password} -Penvironment={env-name}

Related

How resolve -API call error https://a.blazemeter.com/api/latest/proxy: {'code': 401, 'message': 'Unauthorized'}

I have one selenium script. Now I want it to convert into a JMX file. Here I used the proxy2jmx converter. I also created an account in Blazemeter. I have one token and a secret key, those values provide in the .bzt-rc file.
Now I provide one config yml file -
execution:
executor: selenium
iterations: 1
scenario: sel
scenarios:
sel:
script: F:\Taurus\HelloSelenium.java
services:
module: proxy2jmx
modules:
proxy2jmx:
token: 707ab10114456ad7af13827f
When execute this file It return Error message-->11:21:24 ERROR: Network Error: API call error https://a.blazemeter.com/api/latest/proxy: {'code': 401, 'message': 'Unauthorized'}
11:21:24 INFO: Post-processing...
11:21:24 INFO: Will not pick converted JMX due to exception: API call error https://a.blazemeter.com/api/latest/proxy: {'code': 401, 'message': 'Unauthorized'}
How is it resolved?
I think the correct way of providing token is
modules:
blazemeter:
token: 756a1345d1c8258a739dd260:1c2f53d2612dc64cb4c9dade311f351916b9e391330c9f6d1a2e1e36da76cabc44be9ce6
More information:
BlazeMeter Reporting Service - Personalized Usage
BlazeMeter API keys

Spinnaker & Okta integration failing

Scenerio:
Upgraded Spinnaker to 1.12.0. No other config changes that would impact this integration (we had to modify an s3 IAM because it quit working). Okta integration stopped working. Public key was reissued during install process for the ingress, may be relevant?
SAML-TRACE shows payload getting to okta and back
Spinnaker throws two different errors depending on browser and how I get there.
Direct link to deck url: (500) No IDP was configured, please update included metadata with at least one IDP (seen in browser and gate)
Okta "chicklet" in okta dashboard: (401) Authentication Failed: Incoming SAML message is invalid
Config details (again none of this changed):
Downloading metadata directly
JKS is being leveraged and is valid
service url is confirmed
alias for JKS is confirmed
I had this issue as well when upgrading from 1.10.13 to 1.12.2. I found lots of these error messages in Gate's logs:
2019-02-19 05:31:30.421 ERROR 1 --- [.0-8084-exec-10] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw e
xception [org.opensaml.saml2.metadata.provider.MetadataProviderException: No IDP was configured, please update included metadata with at least one IDP] with root cause
org.opensaml.saml2.metadata.provider.MetadataProviderException: No IDP was configured, please update included metadata with at least one IDP
at org.springframework.security.saml.metadata.MetadataManager.getDefaultIDP(MetadataManager.java:795) ~[spring-security-saml2-core-1.0.2.RELEASE.jar:1.0.2.RELEASE]
at org.springframework.security.saml.context.SAMLContextProviderImpl.populatePeerEntityId(SAMLContextProviderImpl.java:157) ~[spring-security-saml2-core-1.0.2.RELEASE.jar
:1.0.2.RELEASE]
at org.springframework.security.saml.context.SAMLContextProviderImpl.getLocalAndPeerEntity(SAMLContextProviderImpl.java:127) ~[spring-security-saml2-core-1.0.2.RELEASE.ja
r:1.0.2.RELEASE]
at org.springframework.security.saml.SAMLEntryPoint.commence(SAMLEntryPoint.java:146) ~[spring-security-saml2-core-1.0.2.RELEASE.jar:1.0.2.RELEASE]
at org.springframework.security.web.access.ExceptionTranslationFilter.sendStartAuthentication(ExceptionTranslationFilter.java:203) ~[spring-security-web-4.2.9.RELEASE.jar
:4.2.9.RELEASE]
...
After downgrading back to 1.10.13, I upgraded to the next version, 1.11.0, and found that's when the issue started. Eventually, I looked at Gate's logs from the launch of the Container and found:
2019-02-20 22:31:40.132 ERROR 1 --- [0.0-8084-exec-3] o.o.s.m.provider.HTTPMetadataProvider : Error retrieving metadata from https://000000000000.okta.com/app/00000000000000000/sso/saml/metadata
javax.net.ssl.SSLException: Error in hostname verification
at org.opensaml.ws.soap.client.http.TLSProtocolSocketFactory.verifyHostname(TLSProtocolSocketFactory.java:241) ~[openws-1.5.4.jar:na]
at org.opensaml.ws.soap.client.http.TLSProtocolSocketFactory.createSocket(TLSProtocolSocketFactory.java:186) ~[openws-1.5.4.jar:na]
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) ~[commons-httpclient-3.1.jar:na]
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) ~[commons-httpclient-3.1.jar:na]
...
This lead me to realize that the TLS Certificate was being rejected by Gate. Not sure why it suddenly started failing the check. Up to this point, I had it configured as:
$ hal config security authn saml edit --metadata https://000000000000.okta.com/app/00000000000000000/sso/saml/metadata
I ended up downloading the metadata file and redeploying with halyard.
$ wget https://000000000000.okta.com/app/00000000000000000/sso/saml/metadata
$ hal config security authn saml edit --metadata "${PWD}/metadata"
$ hal config version edit --version 1.12.2
$ hal deploy apply
Opened up a private browser window as suggested by the Spinnaker documentation and Gate started redirecting to Okta correctly again.
Issue filed, https://github.com/spinnaker/spinnaker/issues/4017.
So I ended up finding the answer. The tomcat config changed apparently in spinnaker in later versions for gate.
I created this snippet in ~/.hal/default/profiles/gate-local.yml
server:
tomcat:
protocolHeader: X-Forwarded-Proto
remoteIpHeader: X-Forwarded-For
internalProxies: .*
Deployed spinnaker and it was back to working.

Shibboleth setup - reverse proxy wtih Nginx to Jetty 9.4.9

I'm setting up Shibboleth to use SAML 2.0. This setup contains reverse proxy using Nginx to Jetty 9.49 which points to shibboleth idp.war file.
For testing, I'm using the django Service provider from this example here.
I'm using self-sign certificates.
I can access both https://idp.localhost/idp/shibboleth and https://idp.localhost/idp/status, but not https://idp.localhost/idp/profile/SAML2/POST/SSO. From shibboleth log file, I'm getting this error each time I browse https://idp.localhost/idp/profile/SAML2/POST/SSO:
2018-04-25 18:20:47,746 - ERROR
[org.opensaml.saml.common.binding.security.impl.ReceivedEndpointSecurityHandler:200]
- Message Handler: SAML message intended destination endpoint 'https://idp.localhost/idp/profile/SAML2/POST/SSO' did not match the
recipient endpoint 'http://idp.localhost/idp/profile/SAML2/POST/SSO'
==> idp-warn.log <== 2018-04-25 18:20:47,746 - ERROR [org.opensaml.saml.common.binding.security.impl.ReceivedEndpointSecurityHandler:200]
- Message Handler: SAML message intended destination endpoint 'https://idp.localhost/idp/profile/SAML2/POST/SSO' did not match the
recipient endpoint 'http://idp.localhost/idp/profile/SAML2/POST/SSO'
==> idp-process.log <== 2018-04-25 18:20:47,748 - WARN [net.shibboleth.idp.profile.impl.WebFlowMessageHandlerAdaptor:202] -
Profile Action WebFlowMessageHandlerAdaptor: Exception handling
message org.opensaml.messaging.handler.MessageHandlerException: SAML
message failed received endpoint check at
org.opensaml.saml.common.binding.security.impl.ReceivedEndpointSecurityHandler.checkEndpointURI(ReceivedEndpointSecurityHandler.java:202)
==> idp-warn.log <== 2018-04-25 18:20:47,748 - WARN [net.shibboleth.idp.profile.impl.WebFlowMessageHandlerAdaptor:202] -
Profile Action WebFlowMessageHandlerAdaptor: Exception handling
message org.opensaml.messaging.handler.MessageHandlerException: SAML
message failed received endpoint check at
org.opensaml.saml.common.binding.security.impl.ReceivedEndpointSecurityHandler.checkEndpointURI(ReceivedEndpointSecurityHandler.java:202)
==> idp-process.log <== 2018-04-25 18:20:47,749 - WARN [org.opensaml.profile.action.impl.LogEvent:105] - A non-proceed event
occurred while processing the request: MessageAuthenticationError
==> idp-warn.log <== 2018-04-25 18:20:47,749 - WARN [org.opensaml.profile.action.impl.LogEvent:105] - A non-proceed event
occurred while processing the request: MessageAuthenticationError
Where did I go wrong?
I think that a request was done in HTTPS and a response returned in clear HTTP.
This kind of problem sometimes happen when jetty was non installed from source or with other corrupted packages. It's difficult to diagnose in this situation.
Try to compile jetty from source, clean first your old installation.
I've also build up a complete IDP/SP setup procedure with apache2 or nginx/ tomcat or jetty setup here, with ansible:
https://github.com/peppelinux/Ansible-Shibboleth-IDP-SP-Debian9
It takes 4 minutes to give you what you expect, It needs an existing LDAP installation.

Webdeploy database failing for SmarterASP host

I'm trying to use webdeploy for my project to publish to SmarterASP host. I get the following error:
Error : Web deployment task failed. ((1/30/2018 2:55:34 PM) An error
occurred when the request was processed on the remote computer.) 2>
2>(1/30/2018 2:55:34 PM) An error occurred when the request was
processed on the remote computer. 2>The server experienced an issue
processing the request. Contact the server administrator for more
information. 2>Publish failed to deploy.
I have gotten support to give me the error from their end:
Content-Type: application/msdeploy Version: 9.0.0.0
MSDeploy.VersionMin: 7.1.600.0 MSDeploy.VersionMax: 9.0.1981.0
MSDeploy.Method: Sync MSDeploy.RequestId:
42329f84-36b0-4fe3-aec5-a71745700abc MSDeploy.RequestCulture: en-US
MSDeploy.RequestUICulture: en-US ServerVersion: 9.0.1955.0 Skip:
objectName="^configProtectedData$" Provider: dbDacFx, Path: data
source=XXXXXXXXX;initial catalog=xxxxxxx;user id=xxxxxxxx
A tracing deployment agent exception occurred that was propagated to
the client. Request ID '42329f84-36b0-4fe3-aec5-a71745700abc'. Request
Timestamp: '1/25/2018 8:20:58 AM'. Error Details:
ERROR_DACFX_NEEDED_FOR_SQL_PROVIDER
Microsoft.Web.Deployment.DeploymentDetailedFatalException: The SQL
provider cannot run with dacpac option because of a missing
dependency. Please make sure that DacFx is installed. Learn more at:
http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_DACFX_NEEDED_FOR_SQL_PROVIDER.
They claim the missing dependency is on my end. I have installed Microsoft SQL Server Data-Tier Application Framework (DACFx) at their request, and still have the same error.

Apache Directory Authentication Failed?

I am trying to connect the OpenLdap through Apache Directory Studio and its working upto Network Parameter and check Network Parameter working fine.
But Next step i am trying to Authentication and its failed with below message.
The authentication failed
- The response queue has been emptied, no response was found. org.apache.directory.api.ldap.model.exception.LdapException: The
response queue has been emptied, no response was found. at
org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1327)
at
org.apache.directory.studio.connection.core.io.api.DirectoryApiConnectionWrapper$2.run(DirectoryApiConnectionWrapper.java:368)
at
org.apache.directory.studio.connection.core.io.api.DirectoryApiConnectionWrapper.runAndMonitor(DirectoryApiConnectionWrapper.java:1269)
at
org.apache.directory.studio.connection.core.io.api.DirectoryApiConnectionWrapper.doBind(DirectoryApiConnectionWrapper.java:460)
at
org.apache.directory.studio.connection.core.io.api.DirectoryApiConnectionWrapper.bind(DirectoryApiConnectionWrapper.java:306)
at
org.apache.directory.studio.connection.core.jobs.CheckBindRunnable.run(CheckBindRunnable.java:79)
at
org.apache.directory.studio.connection.ui.RunnableContextRunner$1.run(RunnableContextRunner.java:127)
at
org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:119)
Caused by:
org.apache.directory.api.ldap.model.exception.LdapException: TimeOut
occurred at
org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1305)
... 7 more
The response queue has been emptied, no response was found.

Resources