courses.list google classroom API returns JSON that doesn't match the API description - google-api-js-client

We've had a report from a user that things weren't working correctly in our google classroom integration. After poking around for a bit, we had the user send their HAR file with all the network information for when they use our API. In it we found that the request we make:
gapi.client.request({
root: 'https://classroom.googleapis.com',
path: 'v1/courses',
params: {
'pageSize': 1,
'teacherId': 'me'
}
})
gets back a response with the nextPageToken property set but no courses property set.
Below is the actual entry from the HAR file (minus the Authorization header):
{
"startedDateTime": "2016-03-25T21:08:16.672Z",
"time": 957.9219999995985,
"request": {
"method": "GET",
"url": "https://content-classroom.googleapis.com/v1/courses?pageSize=1&teacherId=me",
"httpVersion": "unknown",
"headers": [
{
"name": "X-Goog-Encode-Response-If-Executable",
"value": "base64"
},
{
"name": "X-Origin",
"value": "https://www.lucidpress.com"
},
{
"name": "X-ClientDetails",
"value": "appVersion=5.0%20(X11%3B%20CrOS%20x86_64%207834.61.0)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F49.0.2623.95%20Safari%2F537.36&platform=Linux%20x86_64&userAgent=Mozilla%2F5.0%20(X11%3B%20CrOS%20x86_64%207834.61.0)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F49.0.2623.95%20Safari%2F537.36"
},
{
"name": "User-Agent",
"value": "Mozilla/5.0 (X11; CrOS x86_64 7834.61.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.95 Safari/537.36"
},
{
"name": "Referer",
"value": "https://content-classroom.googleapis.com/static/proxy.html?sensitive_query_string?"
},
{
"name": "X-JavaScript-User-Agent",
"value": "google-api-javascript-client/1.1.0-beta"
},
{
"name": "X-Referer",
"value": "https://www.lucidpress.com"
}
],
"queryString": [
{
"name": "pageSize",
"value": "1"
},
{
"name": "teacherId",
"value": "me"
}
],
"cookies": [],
"headersSize": -1,
"bodySize": 0
},
"response": {
"status": 200,
"statusText": "",
"httpVersion": "unknown",
"headers": [
{
"name": "date",
"value": "Fri, 25 Mar 2016 21:08:18 GMT"
},
{
"name": "content-encoding",
"value": "gzip"
},
{
"name": "x-content-type-options",
"value": "nosniff"
},
{
"name": "server",
"value": "ESF"
},
{
"name": "x-frame-options",
"value": "SAMEORIGIN"
},
{
"name": "vary",
"value": "Origin"
},
{
"name": "vary",
"value": "X-Origin"
},
{
"name": "vary",
"value": "Referer"
},
{
"name": "content-type",
"value": "application/json; charset=UTF-8"
},
{
"name": "status",
"value": "200"
},
{
"name": "alternate-protocol",
"value": "443:quic,p=1"
},
{
"name": "cache-control",
"value": "private"
},
{
"name": "alt-svc",
"value": "quic=\":443\"; ma=2592000; v=\"31,30,29,28,27,26,25\""
},
{
"name": "content-length",
"value": "198"
},
{
"name": "x-xss-protection",
"value": "1; mode=block"
}
],
"cookies": [],
"content": {
"size": 206,
"mimeType": "application/json",
"text": "{\n \"nextPageToken\": \"some_string_token_sensitive?"\n}\n"
},
"redirectURL": "",
"headersSize": -1,
"bodySize": -1,
"_transferSize": 448
},
"cache": {},
"timings": {
"blocked": 2.4730000004638,
"dns": -1,
"connect": -1,
"send": 0.8660000003146702,
"wait": 953.5609999984446,
"receive": 1.0220000003754421,
"ssl": -1
}
},
I've had the user fill in the request through the "Try it!" area in the api description without pageSize set to 1 and it gives back the correct format. We've also been unable to reproduce with our test accounts and we aren't hearing any other reports of similar issues.
Has anyone else run into this or have any ideas of what might be happening?

Nothing about the JSON response is imporoperly formatted. pageSize sets the maximum number of results that the API should return, not the minimum. Your code should handle the case where nextPageToken is set but courses is not by retrieving the next page until it gets a course back.
Out of curiosity, why are you only retrieving a single course to begin with?

Related

Microsoft Graph Batch API concurrent request reduction to 4 where as outlook had 200 concurrent Requests

We use Outlook Batch API to send multiple Outlook REST requests in a single HTTP batch request. In Outlook Batch API, we make 200 requests in the same batch HTTP request and all the concurrent requests succeed. Whereas if we do the same in graph API, only 4 concurrent requests succeed as only 4 concurrent requests are supported in graph APIs (attached request and response).
We have the cases such as 10000 events creating at once(like users can add/remove 10000 items from a single user calendar).
So it would be of great help if the Graph APIs batch limit is set to be same as Outlook API(200 concurrent request).
Sample request:
{
"requests": [
{
"url": "/me/calendars",
"method": "POST",
"id": "1",
"headers": {
"Content-Type": "application/json"
},
"body": {
"name": "test1s"
}
},
{
"url": "/me/calendars",
"method": "POST",
"id": "2",
"headers": {
"Content-Type": "application/json"
},
"body": {
"name": "test2"
}
},
{
"url": "/me/calendars",
"method": "POST",
"id": "3",
"headers": {
"Content-Type": "application/json"
},
"body": {
"name": "test3"
}
},
{
"url": "/me/calendars",
"method": "POST",
"id": "4",
"headers": {
"Content-Type": "application/json"
},
"body": {
"name": "test4"
}
},
{
"url": "/me/calendars",
"method": "POST",
"id": "5",
"headers": {
"Content-Type": "application/json"
},
"body": {
"name": "test5s"
}
},
{
"url": "/me/calendars",
"method": "POST",
"id": "6",
"headers": {
"Content-Type": "application/json"
},
"body": {
"name": "test6s"
}
},
{
"url": "/me/calendars",
"method": "POST",
"id": "7",
"headers": {
"Content-Type": "application/json"
},
"body": {
"name": "test7s"
}
}
]
}
Response:
{
"responses": [
{
"id": "6",
"status": 429,
"headers": {
"Retry-After": "1",
"Cache-Control": "private",
"Content-Type": "application/json"
},
"body": {
"error": {
"code": "ApplicationThrottled",
"message": "Application is over its MailboxConcurrency limit.",
"innerError": {
"date": "2022-08-11T06:21:56",
"request-id": "e09afe6b-25d6-4cdc-93f5-0539d6e8b9c6",
"client-request-id": "efe22070-473c-28bd-d1bf-ad43bb9f753f"
}
}
}
},
{
"id": "7",
"status": 429,
"headers": {
"Retry-After": "1",
"Cache-Control": "private",
"Content-Type": "application/json"
},
"body": {
"error": {
"code": "ApplicationThrottled",
"message": "Application is over its MailboxConcurrency limit.",
"innerError": {
"date": "2022-08-11T06:21:56",
"request-id": "e09afe6b-25d6-4cdc-93f5-0539d6e8b9c6",
"client-request-id": "efe22070-473c-28bd-d1bf-ad43bb9f753f"
}
}
}
},
{
"id": "4",
"status": 429,
"headers": {
"Retry-After": "1",
"Cache-Control": "private",
"Content-Type": "application/json"
},
"body": {
"error": {
"code": "ApplicationThrottled",
"message": "Application is over its MailboxConcurrency limit.",
"innerError": {
"date": "2022-08-11T06:21:56",
"request-id": "e09afe6b-25d6-4cdc-93f5-0539d6e8b9c6",
"client-request-id": "efe22070-473c-28bd-d1bf-ad43bb9f753f"
}
}
}
},
{
"id": "5",
"status": 201,
"headers": {
"Location": x"https://outlook.office.com/api/v2.0/Users('xxxx')/Calendars('xxxx')",
"Cache-Control": "private",
"OData-Version": "4.0",
"Content-Type": "application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8"
},
"body": {
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#users('xxxxx')/calendars/$entity",
"id": "xxx",
"name": "test5s",
"color": "auto",
"hexColor": "",
"isDefaultCalendar": false,
"changeKey": "xxxxx",
"canShare": true,
"canViewPrivateItems": true,
"canEdit": true,
"allowedOnlineMeetingProviders": [
"teamsForBusiness"
],
"defaultOnlineMeetingProvider": "teamsForBusiness",
"isTallyingResponses": false,
"isRemovable": true,
"owner": {
"name": "xxxx",
"address": "xxxx.onmicrosoft.com"
}
}
},
{
"id": "2",
"status": 201,
"headers": {
"Location": "https://outlook.office.com/api/v2.0/Users('xxxx')/Calendars('xxxx')",
"Cache-Control": "private",
"OData-Version": "4.0",
"Content-Type": "application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8"
},
"body": {
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#users('xx')/calendars/$entity",
"id": "xxx",
"name": "test2",
"color": "auto",
"hexColor": "",
"isDefaultCalendar": false,
"changeKey": "xxxxx",
"canShare": true,
"canViewPrivateItems": true,
"canEdit": true,
"allowedOnlineMeetingProviders": [
"teamsForBusiness"
],
"defaultOnlineMeetingProvider": "teamsForBusiness",
"isTallyingResponses": false,
"isRemovable": true,
"owner": {
"name": "xxxx",
"address": "xxxxx.onmicrosoft.com"
}
}
},
{
"id": "3",
"status": 201,
"headers": {
"Location": "https://outlook.office.com/api/v2.0/Users('xxxx')/Calendars('xxxxx')",
"Cache-Control": "private",
"OData-Version": "4.0",
"Content-Type": "application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8"
},
"body": {
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#users('xxx')/calendars/$entity",
"id": "x",
"name": "test3",
"color": "auto",
"hexColor": "",
"isDefaultCalendar": false,
"changeKey": "xxxxxx==",
"canShare": true,
"canViewPrivateItems": true,
"canEdit": true,
"allowedOnlineMeetingProviders": [
"teamsForBusiness"
],
"defaultOnlineMeetingProvider": "teamsForBusiness",
"isTallyingResponses": false,
"isRemovable": true,
"owner": {
"name": "xxxx",
"address": "xxxxxx.onmicrosoft.com"
}
}
},
{
"id": "1",
"status": 201,
"headers": {
"Location": "https://outlook.office.com/api/v2.0/Users('xxxx')/Calendars('xxxx')",
"Cache-Control": "private",
"OData-Version": "4.0",
"Content-Type": "application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8"
},
"body": {
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#users('xxxxx')/calendars/$entity",
"id": "xxxxxx",
"name": "test1s",
"color": "auto",
"hexColor": "",
"isDefaultCalendar": false,
"changeKey": "xxxxxxx",
"canShare": true,
"canViewPrivateItems": true,
"canEdit": true,
"allowedOnlineMeetingProviders": [
"teamsForBusiness"
],
"defaultOnlineMeetingProvider": "teamsForBusiness",
"isTallyingResponses": false,
"isRemovable": true,
"owner": {
"name": "xxxx",
"address": "xxxxx.onmicrosoft.com"
}
}
}
]
}
There is a known issue with Batching as mentioned here:
and more in this :
For reference, please refer this:
https://learn.microsoft.com/en-us/graph/known-issues#json-batching
https://learn.microsoft.com/en-us/graph/throttling#throttling-and-batching

Net Web API Post 500 Error After Publishing To Azure API Management Service

I Am Attempting To Use Microsoft Azure To Create A Remote SQL Server Database And Net Web API Services That Will Connect To That Database. I Have Successfully Created A Database With Microsoft Azure, And Have Followed This Tutorial To Import A Net Web API To API Management Services In Azure. I Have A Post That Functions When I Run It Locally, Tested Via Postman And SQL Server, However Upon Testing In The Azure Portal After Publishing I Receive A 500 Error.
I Ran Into Some Difficulty On Publishing And Resolved It Like So:
Based On Stackoverflow Answer Here
Added a global.json file to my Web API project
Put this in it: { "sdk": { "version": "3.1.406", "rollForward": "latestPatch" } }
Ran --- dotnet tool restore
Here's The Trace After Testing The Post Request:
Inbound
api-inspector (0.257 ms)
{
"request": {
"method": "POST",
"url": "https://psiteapi.azure-api.net/api/demo/AzureTest",
"headers": [
{
"name": "sec-ch-ua",
"value": "\" Not;A Brand\";v=\"99\",\"Google Chrome\";v=\"91\",\"Chromium\";v=\"91\""
},
{
"name": "sec-ch-ua-mobile",
"value": "?0"
},
{
"name": "Ocp-Apim-Subscription-Key",
"value": "9fc0d4d53a3a41e89ca863de5b2f4668"
},
{
"name": "Sec-Fetch-Site",
"value": "cross-site"
},
{
"name": "Sec-Fetch-Mode",
"value": "cors"
},
{
"name": "Sec-Fetch-Dest",
"value": "empty"
},
{
"name": "X-Forwarded-For",
"value": "172.58.144.45"
},
{
"name": "Cache-Control",
"value": "no-cache, no-store"
},
{
"name": "Content-Length",
"value": "0"
},
{
"name": "Content-Type",
"value": "text/plain;charset=UTF-8"
},
{
"name": "Accept",
"value": "*/*"
},
{
"name": "Accept-Encoding",
"value": "gzip,deflate,br"
},
{
"name": "Accept-Language",
"value": "en-US,en;q=0.9"
},
{
"name": "Host",
"value": "psiteapi.azure-api.net"
},
{
"name": "Referer",
"value": "https://apimanagement.hosting.portal.azure.net/"
}
]
}
}
api-inspector (0.003 ms)
{
"configuration": {
"api": {
"from": "/",
"to": {
"scheme": "https",
"host": "psiteapi.azurewebsites.net",
"port": 443,
"path": "/",
"queryString": "",
"query": {},
"isDefaultPort": true
},
"version": null,
"revision": "1"
},
"operation": {
"method": "POST",
"uriTemplate": "/api/demo/{str}"
},
"user": "-",
"product": "-"
}
}
cors (0.017 ms)
"Origin header was missing or empty and the request was classified as not cross-domain. CORS policy was not applied."
Backend
forward-request (0.095 ms)
{
"message": "Request is being forwarded to the backend service. Timeout set to 300 seconds",
"request": {
"method": "POST",
"url": "https://psiteapi.azurewebsites.net/api/demo/AzureTest",
"headers": [
{
"name": "Host",
"value": "psiteapi.azurewebsites.net"
},
{
"name": "Content-Length",
"value": 0
},
{
"name": "sec-ch-ua",
"value": "\" Not;A Brand\";v=\"99\",\"Google Chrome\";v=\"91\",\"Chromium\";v=\"91\""
},
{
"name": "sec-ch-ua-mobile",
"value": "?0"
},
{
"name": "Ocp-Apim-Subscription-Key",
"value": "9fc0d4d53a3a41e89ca863de5b2f4668"
},
{
"name": "Sec-Fetch-Site",
"value": "cross-site"
},
{
"name": "Sec-Fetch-Mode",
"value": "cors"
},
{
"name": "Sec-Fetch-Dest",
"value": "empty"
},
{
"name": "X-Forwarded-For",
"value": "172.58.144.45,13.91.254.72"
},
{
"name": "Cache-Control",
"value": "no-cache, no-store"
},
{
"name": "Content-Type",
"value": "text/plain;charset=UTF-8"
},
{
"name": "Accept",
"value": "*/*"
},
{
"name": "Accept-Encoding",
"value": "gzip,deflate,br"
},
{
"name": "Accept-Language",
"value": "en-US,en;q=0.9"
},
{
"name": "Referer",
"value": "https://apimanagement.hosting.portal.azure.net/"
}
]
}
}
forward-request (752.347 ms)
{
"response": {
"status": {
"code": 500,
"reason": "Internal Server Error"
},
"headers": [
{
"name": "Set-Cookie",
"value": "ARRAffinity=1f6f99dc36715fce63518df2e3c0e912c4961391f63790097a7130fac11b8655;Path=/;HttpOnly;Secure;Domain=psiteapi.azurewebsites.net,ARRAffinitySameSite=1f6f99dc36715fce63518df2e3c0e912c4961391f63790097a7130fac11b8655;Path=/;HttpOnly;SameSite=None;Secure;Domain=psiteapi.azurewebsites.net"
},
{
"name": "Server",
"value": "Microsoft-IIS/10.0"
},
{
"name": "X-Powered-By",
"value": "ASP.NET"
},
{
"name": "Date",
"value": "Wed, 16 Jun 2021 15:59:30 GMT"
},
{
"name": "Content-Length",
"value": "0"
}
]
}
}
Outbound
transfer-response (0.171 ms)
{
"message": "Response has been sent to the caller in full"
}
All
Inbound
(0.277 ms)
api-inspector (0.257 ms)
{
"request": {
"method": "POST",
"url": "https://psiteapi.azure-api.net/api/demo/AzureTest",
"headers": [
{
"name": "sec-ch-ua",
"value": "\" Not;A Brand\";v=\"99\",\"Google Chrome\";v=\"91\",\"Chromium\";v=\"91\""
},
{
"name": "sec-ch-ua-mobile",
"value": "?0"
},
{
"name": "Ocp-Apim-Subscription-Key",
"value": "9fc0d4d53a3a41e89ca863de5b2f4668"
},
{
"name": "Sec-Fetch-Site",
"value": "cross-site"
},
{
"name": "Sec-Fetch-Mode",
"value": "cors"
},
{
"name": "Sec-Fetch-Dest",
"value": "empty"
},
{
"name": "X-Forwarded-For",
"value": "172.58.144.45"
},
{
"name": "Cache-Control",
"value": "no-cache, no-store"
},
{
"name": "Content-Length",
"value": "0"
},
{
"name": "Content-Type",
"value": "text/plain;charset=UTF-8"
},
{
"name": "Accept",
"value": "*/*"
},
{
"name": "Accept-Encoding",
"value": "gzip,deflate,br"
},
{
"name": "Accept-Language",
"value": "en-US,en;q=0.9"
},
{
"name": "Host",
"value": "psiteapi.azure-api.net"
},
{
"name": "Referer",
"value": "https://apimanagement.hosting.portal.azure.net/"
}
]
}
}
api-inspector (0.003 ms)
{
"configuration": {
"api": {
"from": "/",
"to": {
"scheme": "https",
"host": "psiteapi.azurewebsites.net",
"port": 443,
"path": "/",
"queryString": "",
"query": {},
"isDefaultPort": true
},
"version": null,
"revision": "1"
},
"operation": {
"method": "POST",
"uriTemplate": "/api/demo/{str}"
},
"user": "-",
"product": "-"
}
}
cors (0.017 ms)
"Origin header was missing or empty and the request was classified as not cross-domain. CORS policy was not applied."
Backend
(752.442 ms)↑ Back to top
forward-request (0.095 ms)
{
"message": "Request is being forwarded to the backend service. Timeout set to 300 seconds",
"request": {
"method": "POST",
"url": "https://psiteapi.azurewebsites.net/api/demo/AzureTest",
"headers": [
{
"name": "Host",
"value": "psiteapi.azurewebsites.net"
},
{
"name": "Content-Length",
"value": 0
},
{
"name": "sec-ch-ua",
"value": "\" Not;A Brand\";v=\"99\",\"Google Chrome\";v=\"91\",\"Chromium\";v=\"91\""
},
{
"name": "sec-ch-ua-mobile",
"value": "?0"
},
{
"name": "Ocp-Apim-Subscription-Key",
"value": "9fc0d4d53a3a41e89ca863de5b2f4668"
},
{
"name": "Sec-Fetch-Site",
"value": "cross-site"
},
{
"name": "Sec-Fetch-Mode",
"value": "cors"
},
{
"name": "Sec-Fetch-Dest",
"value": "empty"
},
{
"name": "X-Forwarded-For",
"value": "172.58.144.45,13.91.254.72"
},
{
"name": "Cache-Control",
"value": "no-cache, no-store"
},
{
"name": "Content-Type",
"value": "text/plain;charset=UTF-8"
},
{
"name": "Accept",
"value": "*/*"
},
{
"name": "Accept-Encoding",
"value": "gzip,deflate,br"
},
{
"name": "Accept-Language",
"value": "en-US,en;q=0.9"
},
{
"name": "Referer",
"value": "https://apimanagement.hosting.portal.azure.net/"
}
]
}
}
forward-request (752.347 ms)
{
"response": {
"status": {
"code": 500,
"reason": "Internal Server Error"
},
"headers": [
{
"name": "Set-Cookie",
"value": "ARRAffinity=1f6f99dc36715fce63518df2e3c0e912c4961391f63790097a7130fac11b8655;Path=/;HttpOnly;Secure;Domain=psiteapi.azurewebsites.net,ARRAffinitySameSite=1f6f99dc36715fce63518df2e3c0e912c4961391f63790097a7130fac11b8655;Path=/;HttpOnly;SameSite=None;Secure;Domain=psiteapi.azurewebsites.net"
},
{
"name": "Server",
"value": "Microsoft-IIS/10.0"
},
{
"name": "X-Powered-By",
"value": "ASP.NET"
},
{
"name": "Date",
"value": "Wed, 16 Jun 2021 15:59:30 GMT"
},
{
"name": "Content-Length",
"value": "0"
}
]
}
}
Outbound
(0.171 ms)↑ Back to top
transfer-response (0.171 ms)
{
"message": "Response has been sent to the caller in full"
}
Here Are The Packages I Installed In My Net Web API:
Microsoft.AspNetCore.Mvc.NewtonsoftJson (3.1.0)
Swashbuckle.AspNetCore (6.1.4)
System.Data.SqlClient (4.8.2)
System.Runtime (4.3.1)
How Can I Import My Net Web API To Azure API Management And Have It Successfully Make Requests To The SQL Server Also Setup Within Azure Given The Issues Described Above?
I Never Setup The Database Firewall Properly, Heres The Solution:
Azure Portal -> SQL Databases -> Your Database -> Set Server Firewall -> Allow Azure Services And Resources To Access This Server - Yes

kibana watcher to send array of json objects

I am trying to create an alert in TheHive4 with some observables using watcher. Using the postman tool I am able to send API requests, below is the postman request body. observables are under artifacts. An array of JSON Objects.
{
"title": "Test Title",
"description": "Testing alert creation through the API",
"tags": ["testing" , "API"],
"type": "external",
"source": "postman",
"sourceRef": "111111",
"severity": 1,
"tlp": 0,
"artifacts": [
{ "dataType": "ip", "data": "127.0.0.1", "message": "localhost" },
{ "dataType": "hash", "data": "lasgjjaskrgjiwrj", "message": "localhost" }
]
}
In Kibana Devtools I replicated the same request as given below.
PUT _watcher/watch/Watcher_to_Hive2
{
"trigger": {
"schedule": {
"interval": "5m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"testindex-1*"
],
"rest_total_hits_as_int": true,
"body": {
"size": 0,
"query": {
"match": {
"process": "YASHUKASH.EXE"
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 1
}
}
},
"actions": {
"victorops": {
"webhook": {
"scheme": "http",
"host": "15.00.00.130",
"port": 9000,
"method": "post",
"path": "api/alert",
"params": {
"process": "{{ctx.watch_id}}",
"description": "description",
"source": "source",
"type": "type",
"sourceRef": "Watcher1",
"title": "Watcher_test"
},
"data-raw":{
"title": "Test Ale",
"description": "Testing alert creation through the API",
"tags": ["testing" , "API"],
"type": "external",
"source": "postman",
"sourceRef": "10299",
"severity": 1,
"tlp": 0,
"organization":"test",
"artifacts": {
[
{ "dataType": "ip", "data": "127.0.0.1", "message": "localhost" },
{ "dataType": "hash", "data": "lasgjjaskrgjiwrj", "message": "localhost" }
]},
"headers": {
"Authorization": "Bearer Token******Token",
"Content-Type": "application/json; charset=UTF-8"
},
"body": ""
}
}
}
}
In Kibana watcher also I can create an alert to TheHive4 but cannot use artifacts (Array of JSON objects). If I use artifacts it is giving me 400 errors. Error is as follows
{
"error" : {
"root_cause" : [
{
"type" : "x_content_parse_exception",
"reason" : "[50:7] [script] unknown field [dataType]"
}
],
"type" : "x_content_parse_exception",
"reason" : "[50:7] [script] unknown field [dataType]"
},
"status" : 400
}
How do I define an array of JSON objects in watcher?
You can do that but you need to do 2 changes.
By sending the parameters in the body in text format.
Change the HTTP method type to POST from PUT.
The sample code you need to enter is given below.
POST /_Watcher/watch/{watch id}/_execute
{
"trigger": {
"schedule": {
"interval": "5m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"Test_indices*"
],
"rest_total_hits_as_int": true,
"body": {
"size": 0,
"query": {
"match": {
"process": "ABC.EXE"
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 1
}
}
},
"actions": {
"victorops": {
"webhook": {
"scheme": "http",
"host": "15.00.00.130",
"port": 9000,
"method": "post",
"path": "api/alert",
"params": {},
"headers": {
"Authorization": "Bearer your_token",
"Content-Type": "application/json; charset=UTF-8"
},
"body": """
{
"title": "Test",
"description": "Testing alert",
"tags": ["testing","API"],
"type": "Test",
"source": "Test_Source",
"sourceRef": "Test_ref",
"severity": 1,
"tlp": 0,
"artifacts": [{
"dataType": "ip",
"data": "127.0.0.1",
"message": "localhost"
},
{
"dataType": "hash",
"data": "lasgjjaskrgjiwrj",
"message": "localhost"
},
{
"dataType": "hash",
"data": "processname",
"message": "myprocess"
}]
}
"""
}
}
}
}
`
I hope by this you can create an alert by Kibana Devtools with observables in TheHive4.

Logstash Indexing error “Index -1 out of bounds for length 0”

I want to use ELK stack to analyze some Kubernetes audit logs. They're sent to the Logstash webhook as JSON. Here's my config file :
input{
http {
port => 8888
codec => "json"
type => "json"
}
}
filter{
json {
source => "message"
}
}
output{
file {
path => "/home/ubuntu/logstash-kubernetes/audit.log"
}
elasticsearch{
hosts => "localhost:9200"
index => "kubernetes"
}
}
The output file is populated with good looking entries, I'm happy with it. But when it comes to exporting to Elasticsearch, I'm getting the following error :
elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"kubernetes", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x71740f4b>], :response=>{"index"=>{"_index"=>"kubernetes", "_type"=>"_doc", "_id"=>"x6QKJHYBkIKyNOo2Q-z8", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"array_index_out_of_bounds_exception", "reason"=>"Index -1 out of bounds for length 0"}}}}}
It only happens for some entries. Let me show you an generated entry that was processed without error :
{
"_index": "kubernetes",
"_type": "_doc",
"_id": "xKQKJHYBkIKyNOo2POwJ",
"_version": 1,
"_score": 0,
"_source": {
"headers": {
"content_type": "application/json",
"request_method": "POST",
"accept_encoding": "gzip",
"request_path": "/",
"http_host": "****:8888",
"http_accept": "application/json, */*",
"content_length": "2813",
"http_version": "HTTP/1.1",
"http_user_agent": "Go-http-client/1.1"
},
"#version": "1",
"host": "****",
"apiVersion": "audit.k8s.io/v1",
"items": [
{
"requestObject": {
"type": "Normal",
"firstTimestamp": "2020-12-02T15:20:19Z",
"involvedObject": {
"namespace": "kube-system",
"name": "kube-scheduler",
"apiVersion": "v1",
"resourceVersion": "2573",
"uid": "0ebf0ea5-f8c0-475a-9c9e-57674edefe8d",
"kind": "Endpoints"
},
"message": "****1_938d6a87-bccb-4ea0-bd6b-31fc27e48b7a became leader",
"eventTime": null,
"source": {
"component": "default-scheduler"
},
"reason": "LeaderElection",
"count": 1,
"reportingInstance": "",
"apiVersion": "v1",
"reportingComponent": "",
"lastTimestamp": "2020-12-02T15:20:19Z",
"metadata": {
"name": "kube-scheduler.164cef6d1f3ed55e",
"namespace": "kube-system",
"creationTimestamp": null
},
"kind": "Event"
},
"requestURI": "/api/v1/namespaces/kube-system/events",
"responseStatus": {
"code": 201,
"metadata": {}
},
"user": {
"groups": [
"system:authenticated"
],
"username": "system:kube-scheduler"
},
"annotations": {
"authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:kube-scheduler\" of ClusterRole \"system:kube-scheduler\" to User \"system:kube-scheduler\"",
"authorization.k8s.io/decision": "allow"
},
"stageTimestamp": "2020-12-02T15:20:19.090835Z",
"userAgent": "kube-scheduler/v1.19.4 (linux/amd64) kubernetes/d360454/scheduler",
"objectRef": {
"namespace": "kube-system",
"name": "kube-scheduler.164cef6d1f3ed55e",
"apiVersion": "v1",
"resource": "events"
},
"responseObject": {
"type": "Normal",
"firstTimestamp": "2020-12-02T15:20:19Z",
"involvedObject": {
"namespace": "kube-system",
"name": "kube-scheduler",
"apiVersion": "v1",
"resourceVersion": "2573",
"uid": "0ebf0ea5-f8c0-475a-9c9e-57674edefe8d",
"kind": "Endpoints"
},
"message": "****1_938d6a87-bccb-4ea0-bd6b-31fc27e48b7a became leader",
"eventTime": null,
"source": {
"component": "default-scheduler"
},
"reason": "LeaderElection",
"count": 1,
"reportingInstance": "",
"apiVersion": "v1",
"reportingComponent": "",
"lastTimestamp": "2020-12-02T15:20:19Z",
"metadata": {
"name": "kube-scheduler.164cef6d1f3ed55e",
"namespace": "kube-system",
"creationTimestamp": "2020-12-02T15:20:19Z",
"managedFields": [
{
"operation": "Update",
"fieldsType": "FieldsV1",
"time": "2020-12-02T15:20:19Z",
"manager": "kube-scheduler",
"apiVersion": "v1",
"fieldsV1": {
"f:source": {
"f:component": {}
},
"f:lastTimestamp": {},
"f:reason": {},
"f:firstTimestamp": {},
"f:type": {},
"f:involvedObject": {
"f:kind": {},
"f:namespace": {},
"f:name": {},
"f:apiVersion": {},
"f:uid": {},
"f:resourceVersion": {}
},
"f:message": {},
"f:count": {}
}
}
],
"resourceVersion": "2576",
"selfLink": "/api/v1/namespaces/kube-system/events/kube-scheduler.164cef6d1f3ed55e",
"uid": "573e4082-7c38-473c-9116-a114be97a47d"
},
"kind": "Event"
},
"level": "RequestResponse",
"requestReceivedTimestamp": "2020-12-02T15:20:19.021481Z",
"auditID": "cf291b3f-f5fa-4e43-a177-adf9cbfc0197",
"sourceIPs": [
"****"
],
"stage": "ResponseComplete",
"verb": "create"
}
],
"#timestamp": "2020-12-02T15:20:19.094Z",
"type": "json",
"metadata": {},
"kind": "EventList"
}
}
And here's the JSON for that error happens :
{
"headers": {
"content_type": "application/json",
"request_method": "POST",
"accept_encoding": "gzip",
"request_path": "/",
"http_host": "****:8888",
"http_accept": "application/json, */*",
"content_length": "2668",
"http_version": "HTTP/1.1",
"http_user_agent": "Go-http-client/1.1"
},
"#version": "1",
"host": "****",
"apiVersion": "audit.k8s.io/v1",
"items": [
{
"requestObject": {
"metadata": {
"name": "kube-scheduler",
"namespace": "kube-system",
"creationTimestamp": "2020-12-02T14:36:33Z",
"annotations": {
"control-plane.alpha.kubernetes.io/leader": "{\"holderIdentity\":\"****38d6a87-bccb-4ea0-bd6b-31fc27e48b7a\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-12-02T15:18:52Z\",\"renewTime\":\"2020-12-02T15:18:52Z\",\"leaderTransitions\":67}"
},
"managedFields": [
{
"operation": "Update",
"fieldsType": "FieldsV1",
"time": "2020-12-02T15:18:23Z",
"manager": "kube-scheduler",
"apiVersion": "v1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:control-plane.alpha.kubernetes.io/leader": {}
}
}
}
}
],
"resourceVersion": "2531",
"selfLink": "/api/v1/namespaces/kube-system/endpoints/kube-scheduler",
"uid": "0ebf0ea5-f8c0-475a-9c9e-57674edefe8d"
},
"apiVersion": "v1",
"kind": "Endpoints"
},
"requestURI": "/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=10s",
"responseStatus": {
"code": 200,
"metadata": {}
},
"user": {
"groups": [
"system:authenticated"
],
"username": "system:kube-scheduler"
},
"annotations": {
"authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:kube-scheduler\" of ClusterRole \"system:kube-scheduler\" to User \"system:kube-scheduler\"",
"authorization.k8s.io/decision": "allow"
},
"stageTimestamp": "2020-12-02T15:18:52.067347Z",
"userAgent": "kube-scheduler/v1.19.4 (linux/amd64) kubernetes/d360454/leader-election",
"objectRef": {
"namespace": "kube-system",
"name": "kube-scheduler",
"apiVersion": "v1",
"resourceVersion": "2531",
"uid": "0ebf0ea5-f8c0-475a-9c9e-57674edefe8d",
"resource": "endpoints"
},
"responseObject": {
"metadata": {
"name": "kube-scheduler",
"namespace": "kube-system",
"creationTimestamp": "2020-12-02T14:36:33Z",
"annotations": {
"control-plane.alpha.kubernetes.io/leader": "{\"holderIdentity\":\"****1_938d6a87-bccb-4ea0-bd6b-31fc27e48b7a\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-12-02T15:18:52Z\",\"renewTime\":\"2020-12-02T15:18:52Z\",\"leaderTransitions\":67}"
},
"managedFields": [
{
"operation": "Update",
"fieldsType": "FieldsV1",
"time": "2020-12-02T15:18:52Z",
"manager": "kube-scheduler",
"apiVersion": "v1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:control-plane.alpha.kubernetes.io/leader": {}
}
}
}
}
],
"resourceVersion": "2542",
"selfLink": "/api/v1/namespaces/kube-system/endpoints/kube-scheduler",
"uid": "0ebf0ea5-f8c0-475a-9c9e-57674edefe8d"
},
"apiVersion": "v1",
"kind": "Endpoints"
},
"level": "RequestResponse",
"requestReceivedTimestamp": "2020-12-02T15:18:52.062449Z",
"auditID": "a45860c7-e0c9-4724-be2e-705a1b2955f4",
"sourceIPs": [
"****"
],
"stage": "ResponseComplete",
"verb": "update"
}
],
"#timestamp": "2020-12-02T15:20:22.001Z",
"type": "json",
"metadata": {},
"kind": "EventList"
}
(I removed some personnal data from the JSONs)
Both JSONs appears as valid. I noticed that the JSONs with responseStatus.code = 201 were correctly processed, and those with code = 200 were throwing this error.
I'm getting a bit mad so if anyone can help with that, I'd be very pleased !
The error is thrown on this line because subfields has a length of 0 and hence it is trying to access index -1, which is not allowed.
The method that splits the field names into pathsis called splitAndValidatePath and what it does is to split field names when it encounters . (i.e. a dot).
In your second document, you have two fields called "." (i.e. it is just a dot) and they are not really valid.
".": {},
If you remove them both, your document will be indexed correctly.
Another way of solving this is to disable indexing for the parent f:annotations object field and that will do the trick... provided, of course, you don't need to search for content within f:annotations.
I face same problem with fluentd.
As per #val suggestion. Fixed by below.
PUT /[index_name]/_mapping
{
"properties" : {
"metadata" : {
"properties" : {
"managedFields" : {
"properties" : {
"fieldsV1" : {
"properties" : {
"f:metadata" : {
"type" : "object",
"enabled": false
}
}
}
}
}
}
}
}
}

rethinkdb, How could I pluck the result by a value in particular "array index"?

sample data
[
{
"createdDate": 1508588333821,
"data": {
"image_extension": "png",
"name": "Golden",
"qty": 1,
"remark": "#296-2",
"status": "RETURN",
"owner": [
{
"name": "app1emaker",
"location": 1
},
{
"name": "simss92_lmao",
"location": 31
}
]
},
"deleted": false,
"docId": 307,
"docType": "product",
"id": "db0131f9-9359-4aa3-b6ed-cd9f3ff4aa3e",
"updatedDate": 1553155281691
},
{
"createdDate": 1508588333324,
"data": {
"image_extension": "png",
"name": "Golden",
"qty": 1,
"remark": "#296-2",
"status": "DISCARD",
"owner": [
{
"name": "At533",
"location": 7
},
{
"name": "madsimon",
"location": 64
},
{
"name": "boyboy96",
"location": 1
},
{
"name": "xinfengCN",
"location": 5
}
]
},
"deleted": false,
"docId": 308,
"docType": "product",
"id": "3790bdaa-5347-4ab0-8149-37332c23c6ea",
"updatedDate": 1554555231691
},
...
...
]
And said that, I would like to select the data.owner on array index 0 only (or I should say data.owner[0]), which are
{
"name": "app1emaker",
"location": 1
}
and
{
"name": "At533",
"location": 7
}
in this case. I have a failed code below.
r.db('carbon').table("items").pluck(['id', 'docId', 'createdDate',{data:{name: true, owner:[0]}}])
I saw that for some functions like orderBy, rethinkdb allowed to use orderBy(r.row('data')('owner')(0)('name')) for access nested object, but I have no idea how to do this for pluck? could anyone give me some hints?
Thanks a lot
pluck can not do that, but you can fall back to use the map:
r.db("carbon").table("items").map(function(doc){
return {
"id": doc("id"),
"docId": doc("docId"),
"createdDate": doc("createdDate"),
"data": {
"name": doc("data")("name"),
"owner": doc("data")("owner")(0)
}
}
})

Resources