How to create or verify ElementRangeIndexes in Marklogic from Flyway script - spring-boot

How to create or verify ElementRangeIndexes in Marklogic from Flyway script. It is a pain to remember the elementRangeIndex to be manually don eon every Marklogic DB we migrate.

You may be able to use the Flyway gradle plugin, to execute ml-gradle configuration scripts.
Alternatively, you can use gradle to execute curl commands and create the indexes using the MarkLogic Management API endpoints.
An example of using Curl to define an index taken from Learning the MarkLogic REST API:
curl -X PUT --anyauth --user admin:admin --header "Content-Type:application/json" \
-d '{"word-positions": true,
"element-word-positions": true,
"range-element-index":
[ { "scalar-type": "string",
"namespace-uri": "",
"localname": "SPEAKER",
"collation": "http://marklogic.com/collation/",
"range-value-positions": false,
"invalid-values": "reject"
},
{ "scalar-type": "string",
"namespace-uri": "",
"localname": "affiliation",
"collation": "http://marklogic.com/collation/",
"range-value-positions": false,
"invalid-values": "reject"
},
{ "scalar-type": "int",
"namespace-uri": "",
"localname": "contentRating",
"collation": "",
"range-value-positions": false,
"invalid-values": "reject"
},
{ "scalar-type": "unsignedLong",
"namespace-uri": "http://marklogic.com/filter",
"localname": "size",
"collation": "",
"range-value-positions": false,
"invalid-values": "reject"
},
{ "scalar-type": "string",
"namespace-uri": "http://marklogic.com/filter",
"localname": "Exposure_Time",
"collation": "http://marklogic.com/collation/",
"range-value-positions": false,
"invalid-values": "reject"
}]}' \
http://localhost:8002/manage/v2/databases/TutorialDB/properties

Related

Cypress multi reporters: using mochawesome with autoset-status-cypress-testrail-reporter

I need to use two reporters with my Cypress tests: mochawesome to generate html reports, and autoset-status-cypress-testrail-reporter to publish test results to Testrail.
The main tool I could find that would enable me to use multiple reporters is cypress-multi-reporters.
However, If I try to use cypress-multi-reporters with autoset-status-cypress-testrail-reporter, alone or in conjunction with mochawesome as below (in cypress.json), it does not work. It will not print out any errors, but it just will not publish the results to Testrail, and it will not generate the mochawesome reports.
{
"reporterEnabled": "mochawesome, autoset-status-cypress-testrail-reporter",
"mochawesomeReporterOptions": {
"reportDir": "cypress/reports",
"overwrite": false,
"html": true,
"json": false
},
"autosetStatusCypressTestrailReporterReporterOptions": {
"host": "https://xxxxxx/",
"username": "xxxxx",
"password": "xxxx",
"projectId": 1,
"runId": 1234
}
}
Can anyone tell me why the above is not working, or suggest a similar tool that would work with both mochawesome and autoset-status-cypress-testrail-reporter?
Got this to work in the end. The solution was
OPTION 1 - to include only the below in cypress.json:
"reporter": "cypress-multi-reporters",
"reporterOptions": {
"configFile": "reporter-config.json"
}
Then to create a new file called reporter-config.json, and add the config for each reporter in there:
{
"reporterEnabled": "mochawesome, autoset-status-cypress-testrail-reporter",
"mochawesomeReporterOptions": {
"reportDir": "cypress/reports",
"overwrite": false,
"html": true,
"json": false
},
"autosetStatusCypressTestrailReporterReporterOptions": {
"host": "https://xxxxxx/",
"username": "xxxxx",
"password": "xxxx",
"projectId": 1,
"runId": 1234
}
}
OPTION 2 - to have everything inside cypress.json, like so:
"reporter": "cypress-multi-reporters",
"reporterOptions": {
"reporterEnabled": "mochawesome, autoset-status-cypress-testrail-reporter",
"mochawesomeReporterOptions": {
"reportDir": "cypress/reports",
"overwrite": false,
"html": true,
"json": false
},
"autosetStatusCypressTestrailReporterReporterOptions": {
"host": "https://xxxxxx/",
"username": "xxxxx",
"password": "xxxx",
"projectId": 1,
"runId": 1234
}
}

web app works locally and on app engine, but not on cloud run

So I've run into this issue with a web app I've made:
it gets a file path as input
if the file exists on a bucket, it uses a python client api to create a compute engine instance
it passes the file path to the instance in the startup script
When I ran it locally, I created a python virtual environment and then ran the app. When I make the input on the web browser, the virtual machine is created by the api call. I assumed it used my personal account. I changed to the service account in the command line with this command 'gcloud config set account', it ran fine once more.
When I simply go to the source code directory deploy it as is, the application can create the virtual machine instances as well.
When I use Google cloud build and deploy to cloud run, it doesn't create the vm instance.
the web app itself is not throwing any errors, but when I check compute engine's logs, there is an error in the logs:
`{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {
"code": 3,
"message": "INVALID_PARAMETER"
},
"authenticationInfo": {
"principalEmail": "####"
},
"requestMetadata": {
"callerIp": "#####",
"callerSuppliedUserAgent": "(gzip),gzip(gfe)"
},
"serviceName": "compute.googleapis.com",
"methodName": "v1.compute.instances.insert",
"resourceName": "projects/someproject/zones/somezone/instances/nameofinstance",
"request": {
"#type": "type.googleapis.com/compute.instances.insert"
}
},
"insertId": "######",
"resource": {
"type": "gce_instance",
"labels": {
"instance_id": "#####",
"project_id": "someproject",
"zone": "somezone"
}
},
"timestamp": "2021-06-16T12:18:21.253551Z",
"severity": "ERROR",
"logName": "projects/someproject/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "operation-#####",
"producer": "compute.googleapis.com",
"last": true
},
"receiveTimestamp": "2021-06-16T12:18:21.253551Z"
}`
In theory, it is the same exact code that worked from my laptop and on app engine. I'm baffled why it only does this for cloud run.
App engines default service account was stripped of all its roles and given a custom role tailored to the web apps function.
The cloud run is using a different service account, but was given that exact same custom role.
Here is the method I use to call the api.
def create_instance(path):
compute = googleapiclient.discovery.build('compute', 'v1')
vmname = "piinnuclei" + date.today().strftime("%Y%m%d%H%M%S")
startup_script = "#! /bin/bash\napt update\npip3 install pg8000\nexport BUCKET_PATH=my-bucket/{}\ngsutil -m cp -r gs://$BUCKET_PATH /home/connor\ncd /home/connor\n./cloud_sql_proxy -dir=cloudsql -instances=sql-connection-name=unix:sql-connection-name &\npython3 run_analysis_upload.py\nexport ZONE=$(curl -X GET http://metadata.google.internal/computeMetadata/v1/instance/zone -H 'Metadata-Flavor: Google')\nexport NAME=$(curl -X GET http://metadata.google.internal/computeMetadata/v1/instance/name -H 'Metadata-Flavor: Google')\ngcloud --quiet compute instances delete $NAME --zone=$ZONE".format(path)
config = {
"kind": "compute#instance",
"name": vmname,
"zone": "projects/my-project/zones/northamerica-northeast1-a",
"machineType": "projects/my-project/zones/northamerica-northeast1-a/machineTypes/e2-standard-4",
"displayDevice": {
"enableDisplay": False
},
"metadata": {
"kind": "compute#metadata",
"items": [
{
"key": "startup-script",
"value": startup_script
}
]
},
"tags": {
"items": []
},
"disks": [
{
"kind": "compute#attachedDisk",
"type": "PERSISTENT",
"boot": True,
"mode": "READ_WRITE",
"autoDelete": True,
"deviceName": vmname,
"initializeParams": {
"sourceImage": "projects/my-project/global/images/my-image",
"diskType": "projects/my-project/zones/northamerica-northeast1-a/diskTypes/pd-balanced",
"diskSizeGb": "100"
},
"diskEncryptionKey": {}
}
],
"canIpForward": False,
"networkInterfaces": [
{
"kind": "compute#networkInterface",
"subnetwork": "projects/my-project/regions/northamerica-northeast1/subnetworks/default",
"accessConfigs": [
{
"kind": "compute#accessConfig",
"name": "External NAT",
"type": "ONE_TO_ONE_NAT",
"networkTier": "PREMIUM"
}
],
"aliasIpRanges": []
}
],
"description": "",
"labels": {},
"scheduling": {
"preemptible": False,
"onHostMaintenance": "MIGRATE",
"automaticRestart": True,
"nodeAffinities": []
},
"deletionProtection": False,
"reservationAffinity": {
"consumeReservationType": "ANY_RESERVATION"
},
"serviceAccounts": [
{
"email": "batch-service-accountg#my-project.iam.gserviceaccount.com",
"scopes": [
"https://www.googleapis.com/auth/cloud-platform"
]
}
],
"shieldedInstanceConfig": {
"enableSecureBoot": False,
"enableVtpm": True,
"enableIntegrityMonitoring": True
},
"confidentialInstanceConfig": {
"enableConfidentialCompute": False
}
}
return compute.instances().insert(
project="my-project",
zone="northamerica-northeast1",
body=config).execute()
The issue was with the zone. For some reason, when it was ran on cloud run, the code below was the culprit.
return compute.instances().insert(
project="my-project",
zone="northamerica-northeast1",
body=config).execute()
"northamerica-northeast1" should have been "northamerica-northeast1-a"
EDIT:
I made a new virtual machine image and quickly ran into the same problem, it would work locally and break down in the cloud run environment. After letting it sit for some time, it began to work again. This is leading me to the conclusion that there is also some sort of delay before it can be called by cloud run.

passing more information to consul watch handler

I am wondering whether consul watch handler can be passed some dynamic information while it's called.
That means watch mechanism can pass the script more arguments instead of my given arguments like the below example.
{
"watches": [
{
"type": "service",
"args": ["/tmp/dosomething.sh", "how can i get responses from /v1/health/service here"]
}
]
}
By the way, when I want to 'watch' a service, the most important info to me is the service's state(passing or critial), but I don't understand:
when watch type is 'service', why I cannot appoint the 'service'.
when watch type is 'checks', why I cannot appoint state and service concurrently.
consul watch passes the entire API response payload as an argument to the watch handler script. Your script needs to be able to consume and parse the JSON, and then act on the data provided.
When you watch a service, the data returned is from the /v1/health/service/:service endpoint. (See consul/api/watch/funcs.go.)
when watch type is 'service', why I cannot appoint the 'service'.
I assume you mean that you would like to watch a specific service. If so, this is supported. You can specify a specific service to watch using the -service flag. For example, consul watch -type=service -service=assets.
when watch type is 'checks', why I cannot appoint state and service concurrently.
If you're interested in monitoring checks for a particular service, you should just use the aforementioned watch command for a specific service. The service check information is included in the API response.
$ consul watch -type=service -service=assets
[
{
"Node": {
"ID": "f013522f-aaa2-8fc6-c8ac-c84cb8a56405",
"Node": "hashicorp-consul-server-2",
"Address": "10.0.0.82",
"Datacenter": "dc2",
"TaggedAddresses": null,
"Meta": null,
"CreateIndex": 22898191,
"ModifyIndex": 22898191
},
"Service": {
"ID": "assets-v1",
"Service": "assets",
"Tags": [],
"Meta": null,
"Port": 9090,
"Address": "",
"Weights": {
"Passing": 1,
"Warning": 1
},
"EnableTagOverride": false,
"CreateIndex": 22898195,
"ModifyIndex": 22898195,
"Proxy": {
"MeshGateway": {},
"Expose": {}
},
"Connect": {}
},
"Checks": [
{
"Node": "hashicorp-consul-server-2",
"CheckID": "serfHealth",
"Name": "Serf Health Status",
"Status": "passing",
"Notes": "",
"Output": "Agent alive and reachable",
"ServiceID": "",
"ServiceName": "",
"ServiceTags": [],
"Type": "",
"Definition": {
"Interval": "0s",
"Timeout": "0s",
"DeregisterCriticalServiceAfter": "0s",
"HTTP": "",
"Header": null,
"Method": "",
"Body": "",
"TLSServerName": "",
"TLSSkipVerify": false,
"TCP": ""
},
"CreateIndex": 22898191,
"ModifyIndex": 22898191
}
]
}
]

Retrieving web view link of a Google shared drive folder

I am trying to get the webviewlink of google shared
API: https://www.googleapis.com/drive/v3/files
am unable to get webviewlink of particular shared drive folder.
my input will be the folder id:
{
"id": "1Vat7pdwnpg8i2xMzhic4dOVbadZNv0OJ"
}
how can i achieve this?
Without seeing the exact request you are making its hard to help but i can say this apears to work fine. I can get the the record for a google drive directory shared with me and that directory contains a web view link which has the same file id as the one i sent.
Request
curl \
'https://www.googleapis.com/drive/v3/files/1nhladm_nRRee6rVER9ZA5QqnyKwfG7l?fields=*&key=[YOUR_API_KEY]' \
--header 'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \
--header 'Accept: application/json' \
--compressed
response
{
"kind": "drive#file",
"id": "1nhladm_nRRee6rVER9ZA5QqnyKwfG7l",
"name": "ML GDE Model Deployment Project",
"mimeType": "application/vnd.google-apps.folder",
"starred": false,
"trashed": false,
"explicitlyTrashed": false,
"parents": [
"1OYYiVmqFp9ZvuRSuVgC_ihkdYirLnu"
],
"spaces": [
"drive"
],
"version": "461",
"webViewLink": "https://drive.google.com/drive/folders/1nhladm_nRRee6rVER9ZA5QqnyKwfG7li",
"iconLink": "https://drive-thirdparty.googleusercontent.com/16/type/application/vnd.google-apps.folder+shared",
"hasThumbnail": false,
"thumbnailVersion": "0",
"viewedByMe": false,
"createdTime": "2020-04-02T04:21:47.275Z",
"modifiedTime": "2020-11-02T13:58:51.042Z",
"modifiedByMe": false,
"owners": [
{
.....
}
],
"lastModifyingUser": {
"kind": "drive#user",
},
"shared": true,
"ownedByMe": false,
"capabilities": {
"canAddChildren": false,
"canAddMyDriveParent": false,
"canChangeCopyRequiresWriterPermission": false,
"canChangeViewersCanCopyContent": false,
"canComment": false,
"canCopy": false,
"canDelete": false,
"canDownload": true,
"canEdit": false,
"canListChildren": true,
"canModifyContent": false,
"canMoveChildrenWithinDrive": false,
"canMoveItemIntoTeamDrive": false,
"canMoveItemOutOfDrive": false,
"canMoveItemWithinDrive": false,
"canReadRevisions": false,
"canRemoveChildren": false,
"canRemoveMyDriveParent": true,
"canRename": false,
"canShare": false,
"canTrash": false,
"canUntrash": false
},
"viewersCanCopyContent": true,
"copyRequiresWriterPermission": false,
"writersCanShare": true,
"folderColorRgb": "#8f8f8f",
"quotaBytesUsed": "0",
"isAppAuthorized": false
}
The web view link for this request is https://drive.google.com/drive/folders/1nhladm_nRRee6rVER9ZA5QqnyKwfG7l which is the same as the file id that i sent it.
File Ids change as do the web view links and its possible that they are out of sync.

spring-boot 2 with jolokia: how to disable authentication

ENV: springboot 2.0.5.RELEASE, jolokia 1.5.0
How to disable jokolia authentication?
Endpoint jolokia response looks like following:
{
"request": {
"type": "version"
},
"value": {
"agent": "1.5.0",
"protocol": "7.2",
"config": {
"listenForHttpService": "true",
"authIgnoreCerts": "false",
"agentId": "172.22.0.3-7-72f8afb3-servlet",
"debug": "false",
"agentType": "servlet",
"policyLocation": "classpath:/jolokia-access.xml",
"agentContext": "/jolokia",
"serializeException": "false",
"mimeType": "text/plain",
"dispatcherClasses": "org.jolokia.http.Jsr160ProxyNotEnabledByDefaultAnymoreDispatcher",
"authMode": "basic",
"streaming": "true",
"canonicalNaming": "true",
"historyMaxEntries": "10",
"allowErrorDetails": "true",
"allowDnsReverseLookup": "true",
"realm": "jolokia",
"includeStackTrace": "true",
"useRestrictorService": "false",
"debugMaxEntries": "100"
},
"info": {
"product": "tomcat",
"vendor": "Apache",
"version": "8.5.34"
}
},
"timestamp": 1538476345,
"status": 200
}
Same user application environment with springboot 1.5.9.RELEASE and jokolia 1.3.7 no authentication necessary. Endpoint jokolia responds like following:
{
"request": {
"type": "version"
},
"value": {
"agent": "1.3.7",
"protocol": "7.2",
"config": {
"agentId": "172.27.0.2-6-57fcc568-servlet",
"agentType": "servlet"
},
"info": {
"product": "tomcat",
"vendor": "Apache",
"version": "8.5.23"
}
},
"timestamp": 1538487291,
"status": 200
}
You should place the file jolokia-access.xml in src/main/resources so that it is part of the classpath.
Quote from my comment: Solved using ignoring management port altogether. I was using web.ignoring().requestMatchers(EndpointRequest.toAnyEndpoint()). It seems like it only matches requests to the level of those endpoints, but not levels deeper.

Resources