How to force object key name in array - yaml

I am using YAML to mark up some formulas and using JSON schema to provide a reference schema.
An example of the YAML might be:
formula: # equates to '5 + (3 - 2)'
add:
- 5
- subtract: [3, 2]
While I have figured out how to make the immediate child object of the formula ("add" in this example) have the right key name and type (using a "oneOf"array of "required"s). I am not sure how to ensure that object of an array ("subtract") likewise use specific key names.
So far, I can ensure the type using the following. But with this method, as long as the object used matches the subtract type, it is allowed any key name, it is not restricted to subtract:
"definitions: {
"add": {
"type": "array",
"minItems": 2,
"items": {
"anyOf": [
{ "$ref": "#/definitions/value"}, # value type is an integer which allows for the shown scalar array elements
{ "$ref": "#/definitions/subtract" }
// other operation types
]
}
},
"subtract": {
"type": "array",
"minItems": 2,
"maxItems": 2,
"items": {
"anyOf": [
{ "$ref": "#/definitions/value"},
{ "$ref": "#/definitions/add" }
// other operation types
]
}
}
// other operation types
}
How can I introduce a restriction such that the keys of objects in the array match specific names, while still also allowing scalar elements?

It sounds like what you want is recursive references.
By creating a new definition which is oneOf the operations and value, which then allow items which then reference back to the new definition, you have recursive references.
"definitions: {
"add": {
"type": "array",
"minItems": 2,
"items": { "$ref": "#/definitions/operations_or_values"},
},
"subtract": {
"type": "array",
"minItems": 2,
"maxItems": 2,
"items": { "$ref": "#/definitions/operations_or_values"},
}
// other operation types
"operations_or_values": {
"anyOf": [
{ "$ref": "#definitions/add" },
{ "$ref": "#definitions/subtract" },
{ "$ref": "#definitions/value" }, # value type is an integer which allows for the shown scalar array elements
{ "$ref": "#definitions/[OTHERS]" },
]
}
}
I haven't had time to test this, but I believe it will be, or be close to, what you need. Let me know if it doesn't work. I may not have full understood the question.

What a fascinating problem! This remarkably concise schema can express any expression.
{
"type": ["object", "number"],
"propertyNames": { "enum": ["add", "subtract", "multiply", "divide"] },
"patternProperties": {
".*": {
"type": "array",
"minItems": 2,
"items": { "$ref": "#" }
}
}
}

So what I ended up doing was extending the idea of I already used with the '"oneOf"array of "required", adding an "anyOf".
Thus, an operator schema is now:
"definitions": {
"add": {
"type": "array",
"minItems": 2,
"items": {
"anyOf": [
{ "$ref": "#/definitions/literal" }, // equates to a simple YAML scalar
{ "$ref": "#/definitions/constant" },
{ "$ref": "#/definitions/variable" },
{
"type": "object",
"oneOf": [
{ "required": ["add"] },
{ "required": ["subtract"] }
// more operator names
],
"properties": {
"add": { "$ref": "#/definitions/add" },
"subtract": { "$ref": "#/definitions/subtract" }
// more operator type references
}
}
]
}
},
// more definitions
}
This can be refactored to something that applies more easily across different operators like so:
"definitions": {
"operands": {
"literal": { "type": "number" }, // equates to a simple YAML scalar
"constant": {
"type": "object",
"properties": {
"value": { "type": "number" }
},
"required": [ "value" ]
},
"variable": {
"type": "object",
"properties": {
"name": { type": "string" },
"type": { "type": "string" }
},
"required": [ "name", "type" ]
}
}
"operators": {
"add": {
"type": "array",
"minItems": 2,
"items": { "$ref": "#/definitions/anyOperandsOrOperators" }
},
"subtract": {
"type": "array",
"minItems": 2,
"maxItems": 2,
"items": { "$ref": "#/definitions/anyOperandsOrOperators" }
}
// more operator types
},
"anyOperator": {
"type": "object",
"oneOf": [
{ "required": ["add"] },
{ "required": ["subtract"] }
// more operator names
],
"properties": {
"add": { "$ref": "#/definitions/operators/add" },
"subtract": { "$ref": "#/definitions/operators/subtract" }
// more operator type references
}
},
"anyOperandsOrOperators":
{
"anyOf": [
{ "$ref": "#/definitions/operands/literal" },
{ "$ref": "#/definitions/operands/constant" },
{ "$ref": "#/definitions/operands/variable" },
{ "$ref": "#/definitions/anyOperator"}
]
}
}
And this means the YAML for an operator can look as follows
\/mapping \/mapping
add:[ 5, subtract:[ *constantA, *variableB ] ]
scalar^ ^mapping with specific name

Related

Logicapp Expression to read Dynamic Json path - read child element where parent path may change but hierarchy remaining same

Hope all well.
I am in need of creating logicapp expression for reading child element in json where name of element & hierarchy remains same but parent name can be changing.
for example : JSON-1 :
{
"root": {
"abc1": {
"abc2": [
{
"element": "value1",
"element2": "value"
},
{
"element": "value2",
"element2": "valu2"
}
]
}
}
}
JSON-2 :
{
"root": {
"xyz1": {
"xyz2": [
{
"element": "value1",
"element2": "value"
},
{
"element": "value2",
"element2": "valu2"
}
]
}
}
}
I have tried these but no luck
approach-1: #{body('previous-action')?['']?['']?['element']
approach-2: #{body('previous-action')???['element']
Please let me know if anyone encountered this situation. Many thanks in advance.
I tend to find that converting the JSON to xml (at least in your case) is the simplest solution. Then when you've done that, you can't use XPath to simply make your selection.
Flow
In basic terms ...
I've defined a variable of type object that contains your JSON.
I then convert that JSON object to XML using this expression xml(variables('JSON Object'))
Next, initialize a variable is called Elements of type array (given you have multiple of them). The expression for setting that variable is where the smarts come in. That expression is ... xpath(xml(variables('XML')), '//element/text()') and it's getting the inner text of all element nodes in the XML.
Finally, loop through the results.
If you needed to take it up a level and get the second element then you'd need to change your xpath query to be a lot more generic so you can get the element2 nodes (and 3, 4, 5, etc. if they existed) in each array as well.
Note: I've stuck to your specific question of looking for element.
Result
This definition (which can be loaded directly into your tenant) demonstrates the thinking ...
{
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"actions": {
"For_Each_Element": {
"actions": {
"Set_Element": {
"inputs": {
"name": "Element",
"value": "#{item()}"
},
"runAfter": {},
"type": "SetVariable"
}
},
"foreach": "#variables('Elements')",
"runAfter": {
"Initialize_Element": [
"Succeeded"
]
},
"type": "Foreach"
},
"Initialize_Element": {
"inputs": {
"variables": [
{
"name": "Element",
"type": "string"
}
]
},
"runAfter": {
"Initialize_Elements": [
"Succeeded"
]
},
"type": "InitializeVariable"
},
"Initialize_Elements": {
"inputs": {
"variables": [
{
"name": "Elements",
"type": "array",
"value": "#xpath(xml(variables('XML')), '//element/text()')"
}
]
},
"runAfter": {
"Initialize_XML": [
"Succeeded"
]
},
"type": "InitializeVariable"
},
"Initialize_JSON_Object": {
"inputs": {
"variables": [
{
"name": "JSON Object",
"type": "object",
"value": {
"root": {
"abc1": {
"abc2": [
{
"element": "value1",
"element2": "value"
},
{
"element": "value2",
"element2": "valu2"
}
]
}
}
}
}
]
},
"runAfter": {},
"type": "InitializeVariable"
},
"Initialize_XML": {
"inputs": {
"variables": [
{
"name": "XML",
"type": "string",
"value": "#{xml(variables('JSON Object'))}"
}
]
},
"runAfter": {
"Initialize_JSON_Object": [
"Succeeded"
]
},
"type": "InitializeVariable"
}
},
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {
"ParameterTest1": {
"defaultValue": "\"\"",
"type": "String"
}
},
"triggers": {
"manual": {
"inputs": {
"method": "GET",
"schema": {}
},
"kind": "Http",
"type": "Request"
}
}
},
"parameters": {}
}

Extract value of array and add in the same select mongoDB

I am new to the mongoDB aggregation and I have this situation. I have this Json and I need to convert by "select" this object:
{
"type": "PF",
"code": 12345
"Name": Darth Vader,
"currency": "BRL",
"status": "SINGLE",
"adress": [
{
"localization": "DEATH STAR",
"createDate": 1627990848665
},
{
"localization": "TATOOINE",
"createDate": 1627990555665
},
]
}
this way:
{
"type": "PF",
"code": 12345
"Name": Darth Vader,
"currency": "BRL",
"status": "SINGLE",
"localization": "DEATH STAR",
"createDate": 1627990848665
},
{
"type": "PF",
"code": 12345
"Name": Darth Vader,
"currency": "BRL",
"status": "SINGLE",
"localization": "TATOOINE",
"createDate": 1627990555665
}
So, after my query is complete, I will have 02 objects instead of 01. How can I do this?
I would like to do this via select because after converting I will sort by createDate and limit the number of records to return to the API. I'm using Criteria em my project.
The way to do this is $unwind, this will make 1 copy of the document, for each member of the array.
Test code here
db.collection.aggregate([
{
"$unwind": {
"path": "$user.adress"
}
},
{
"$set": {
"user": {
"$mergeObjects": [
"$user",
"$user.adress"
]
}
}
},
{
"$unset": [
"user.adress"
]
},
{
"$sort": {
"createDate": 1
}
},
{
"$limit": 10
}
])
Edit1 (the above is if user is a field, if it was the name of the collection)
$$ROOT is a system variable that has as value all the document
Test code here
Query
db.collection.aggregate([
{
"$unwind": {
"path": "$adress"
}
},
{
"$replaceRoot": {
"newRoot": {
"$mergeObjects": [
"$$ROOT",
"$adress"
]
}
}
},
{
"$unset": [
"adress"
]
},
{
"$sort": {
"createDate": 1
}
},
{
"$limit": 10
}
])

Separation of hits returned from elastic by nested field value

I've index with products there. I'm trying to separate hits returned from elastic by nested field value. There's my shortened index:
{
"mapping": {
"product": {
"properties": {
"id": {
"type": "integer"
},
"model_name": {
"type": "text",
},
"variants": {
"type": "nested",
"properties": {
"attributes": {
"type": "nested",
"properties": {
"id": {
"type": "integer"
},
"name": {
"type": "text"
},
"product_attribute_id": {
"type": "integer"
},
"value": {
"type": "text"
}
}
},
"id": {
"type": "integer"
},
"product_id": {
"type": "integer"
}
}
}
}
}
}
}
And product example (there's is more variants and attributes in product - I just cut them off):
{
"_index":"product_index",
"_type":"product",
"id":192,
"model_name":"Some tshirt",
"variants":[
{
"id":1271,
"product_id":192,
"attributes":[
{
"id":29,
"name":"clothesSize",
"value":"XL",
"product_attribute_id":36740
}
]
},
{
"id":1272,
"product_id":192,
"attributes":[
{
"id":29,
"name":"clothesSize",
"value":"L",
"product_attribute_id":36741
}
]
}
]
}
The field in question is attribute id. Let's say I want to separate products by size attribute - id 29. It would be perfect if the response would look like:
"hits" : [
{
"_index":"product_index",
"_type":"product",
"id":192,
"model_name":"Some tshirt",
"variants":[
{
"id":1271,
"product_id":192,
"attributes":[
{
"id":29,
"name":"clothesSize",
"value":"XL",
"product_attribute_id":36740
}
]
}
]
},
{
"_index":"product_index",
"_type":"product",
"id":192,
"model_name":"Some tshirt",
"variants":[
{
"id":1272,
"product_id":192,
"attributes":[
{
"id":29,
"name":"clothesSize",
"value":"L",
"product_attribute_id":36741
}
]
}
]
}]
I thought about separate all variants in elastic request and then group them on application side by those attribute but i think it's not most elegant and above all, efficient way.
What are the elastic keywords that I should be interested in?
Thank you in advance for your help.

Filter document on items in an array ElasticSearch

I am using ElasticSearch to search through documents. However, I need to make sure the current user is able to see those documents. Each document is tied to a community, in which the user may belong.
Here is the mapping for my Document:
export const mapping = {
properties: {
amazonId: { type: 'text' },
title: { type: 'text' },
subtitle: { type: 'text' },
description: { type: 'text' },
createdAt: { type: 'date' },
updatedAt: { type: 'date' },
published: { type: 'boolean' },
communities: { type: 'nested' }
}
}
I'm currently saving the ids of the communities the document belongs to in an array of strings. Ex: ["edd05cd0-0a49-4676-86f4-2db913235371", "672916cf-ee32-4bed-a60f-9a7c08dba04b"]
Currently, when I filter a query with {term: { communities: community.id } }, it returns all the documents, regardless of the communities it's tied to.
Here's the full query:
{
index: 'document',
filter_path: { filter: {term: { communities: community.id } } },
body: {
sort: [{ createdAt: { order: 'asc' } }]
}
}
This is the following result based on the community id of "b7d28e7f-7534-406a-981e-ddf147b5015a". NOTE: This is a return from my graphql, so the communities on the document are actual full objects after resolving the hits from the ES query.
"hits": [
{
"title": "The One True Document",
"communities": [
{
"id": "edd05cd0-0a49-4676-86f4-2db913235371"
},
{
"id": "672916cf-ee32-4bed-a60f-9a7c08dba04b"
}
]
},
{
"title": "Boring Document 1",
"communities": []
},
{
"title": "Boring Document 2",
"communities": []
},
{
"title": "Unpublished",
"communities": [
{
"id": "672916cf-ee32-4bed-a60f-9a7c08dba04b"
}
]
}
]
When I attempt to map the communities as {type: 'keyword', index: 'not_analyzed'} I receive an error that states, [illegal_argument_exception] Could not convert [communities.index] to boolean.
So do I need to change my mapping, my filter, or both? Searching around the docs for 6.6, I see that terms needs the non_analyzed mapping.
UPDATE --------------------------
I updated the communities mapping to be a keyword as suggested below. However, I still received the same result.
I updated my query to the following (using a community id that has documents):
query: { index: 'document',
body:
{ sort: [ { createdAt: { order: 'asc' } } ],
from: 0,
size: 5,
query:
{ bool:
{ filter:
{ term: { communities: '672916cf-ee32-4bed-a60f-9a7c08dba04b' } } } } } }
Which gives me the following results:
{
"data": {
"communities": [
{
"id": "672916cf-ee32-4bed-a60f-9a7c08dba04b",
"feed": {
"documents": {
"hits": []
}
}
}
]
}
}
Appears that my filter is working too well?
Since you are storing ids of communities you should make sure that the ids doesn't get analysed. For this communities should be of type keyword. Second you want to store array of community ids since a user can belong to multiple communities. To do this you don't need to make it of type nested. Nested has all together different use case.
To sore values as array you need to make sure that while indexing you are always passing the values against the field as array even if the value is single value.
You need to change mapping and the way you are indexing values against field communities.
1. Update mapping as below:
PUT my_index
{
"mappings": {
"_doc": {
"properties": {
"amazonId": {
"type": "text"
},
"title": {
"type": "text"
},
"subtitle": {
"type": "text"
},
"description": {
"type": "text"
},
"createdAt": {
"type": "date"
},
"updatedAt": {
"type": "date"
},
"published": {
"type": "boolean"
},
"communities": {
"type": "keyword"
}
}
}
}
}
2. Adding a document to index:
PUT my_index/_doc/1
{
"title": "The One True Document",
"communities": [
"edd05cd0-0a49-4676-86f4-2db913235371",
"672916cf-ee32-4bed-a60f-9a7c08dba04b"
]
}
3. Filtering by community id:
GET my_index/_doc/_search
{
"query": {
"bool": {
"filter": [
{
"term": {
"communities": "672916cf-ee32-4bed-a60f-9a7c08dba04b"
}
}
]
}
}
}
Nested Field approach
1. Mapping:
PUT my_index_2
{
"mappings": {
"_doc": {
"properties": {
"amazonId": {
"type": "text"
},
"title": {
"type": "text"
},
"subtitle": {
"type": "text"
},
"description": {
"type": "text"
},
"createdAt": {
"type": "date"
},
"updatedAt": {
"type": "date"
},
"published": {
"type": "boolean"
},
"communities": {
"type": "nested"
}
}
}
}
}
2. Indexing document:
PUT my_index_2/_doc/1
{
"title": "The One True Document",
"communities": [
{
"id": "edd05cd0-0a49-4676-86f4-2db913235371"
},
{
"id": "672916cf-ee32-4bed-a60f-9a7c08dba04b"
}
]
}
3. Querying (used of nested query):
GET my_index_2/_doc/_search
{
"query": {
"bool": {
"filter": [
{
"nested": {
"path": "communities",
"query": {
"term": {
"communities.id.keyword": "672916cf-ee32-4bed-a60f-9a7c08dba04b"
}
}
}
}
]
}
}
}
You might be noticing I used communities.id.keyword and not communities.id. To understand the reason for this go through this.

How can I validate different schemas based on an enumerated property value with AJV?

I need to validate a json document depending the value in one of the properties, specifically an enum property.
This is the JSON document to be validated:
{
"req": {
"user": "",
"company": "",
"dept": "",
"class": "",
"reqType": "account"
}
}
The reqType can take different values like account, dept, and class based on the enumerated values.
I have tried using anyOf for the same but it does not validate correctly.
For example, I have tried the schema below:
{
"$id": "http://example.com/example.json",
"type": "object",
"definitions": {},
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"req": {
"$id": "/properties/req",
"type": "object",
"properties": {
"user": {
"$id": "/properties/req/properties/user",
"type": "string",
"title": "The User Schema ",
"default": "",
"examples": [
"a"
]
},
"company": {
"$id": "/properties/req/properties/company",
"type": "string",
"title": "The Company Schema ",
"default": "",
"examples": [
"b"
]
},
"dept": {
"$id": "/properties/req/properties/dept",
"type": "string",
"title": "The Dept Schema ",
"default": "",
"examples": [
"c"
]
},
"class": {
"$id": "/properties/req/properties/class",
"type": "string",
"title": "The Class Schema ",
"default": "",
"examples": [
"d"
]
},
"reqType": {
"$id": "/properties/req/properties/reqType",
"type": "string",
"title": "The Reqtype Schema ",
"default": "",
"examples": [
"account"
],
"enum": [
"account",
"dept",
"class"
]
}
},
"required": [
"reqType"
],
"anyOf": [
{
"properties": {
"reqType": {
"enum": [
"account"
]
}
},
"required": [
"user",
"company"
]
},
{
"properties": {
"reqType": {
"enum": [
"dept"
]
}
},
"required": [
"dept"
]
}
]
}
},
"required": [
"req"
]
}
This seems to work fine when it meets all of the conditions but when I check the failing case it throws an error for others as follows:
[
{
keyword: 'required',
dataPath: '.req',
schemaPath: '#/properties/req/anyOf/0/required',
params: {
missingProperty: 'user'
},
message: 'should have required property \'user\'',
schema: ['user', 'company'],
parentSchema: {
properties: [Object],
required: [Array]
},
data: {
company: 'b', dept: 'c', class: 'd', reqType: 'account'
}
},
{
keyword: 'enum',
dataPath: '.req.reqType',
schemaPath: '#/properties/req/anyOf/1/properties/reqType/enum',
params: {
allowedValues: [Array]
},
message: 'should be equal to one of the allowed values',
schema: ['dept'],
parentSchema: {
enum: [Array]
},
data: 'account'
},
{
keyword: 'anyOf',
dataPath: '.req',
schemaPath: '#/properties/req/anyOf',
params: {},
message: 'should match some schema in anyOf',
schema: [
[Object],
[Object]
],
parentSchema: {
'$id': '/properties/req',
type: 'object',
properties: [Object],
required: [Array],
anyOf: [Array]
},
data: {
company: 'b', dept: 'c', class: 'd', reqType: 'account'
}
}
]
I expected to get an error for the first and should have validated the 2nd case. Instead it says it did not get the enum value. Am i doing something wrong here?
You're doing it right. The anyOf keyword means that one or more of the given schemas must be valid.
It checks the first and finds that the enum keyword passes, but the required keyword fails. Therefore, this schema fails.
So, it moves on to the next schema and finds that the enum keyword fails and the required keyword passes. Therefore, this schema fails too.
anyOf did not find a valid schema, so it fails and reports that neither schema passes validation.
You see, anyOf doesn't know that enum has a special meaning in this schema. Both schemas have one keyword that passes and one that fails. To anyOf, they are the same.
Here is an alternative that can give you slightly better error messaging. The error messages still end up being quite cryptic, but they are focused where the problem really is.
{
"type": "object",
"properties": {
"req": {
"type": "object",
"properties": {
"reqType": { "enum": ["account", "dept", "class"] }
},
"required": ["reqType"],
"allOf": [
{
"anyOf": [
{
"not": {
"properties": {
"reqType": { "enum": ["account"] }
}
}
},
{ "required": ["user", "company"] }
]
},
{
"anyOf": [
{
"not": {
"properties": {
"reqType": { "enum": ["dept"] }
}
}
},
{ "required": ["dept"] }
]
}
]
}
},
"required": ["req"]
}

Resources