I am looking to get all of the fields in a record in filebeat using the Script processor and perform an action on them. Using the event.Get() from the script processor, it says, "Get a value from the event (either a scalar or an object). If the key does not exist null is returned. If no key is provided then an object containing all fields is returned."
https://www.elastic.co/guide/en/beats/filebeat/current/processor-script.html
Therefore, my question is, what would I do to ensure that no key is provided to get an object that contains all of the fields are returned?
The event.Get() field will provide the top level fields. To look through these top level fields, use a for loop like:
- script:
lang: javascript
id: get_fields
source: >
function process(event) {
var a = event.Get();
for (var key in a) {
if(event.Get(key) == ""){
event.Delete(key);
}
}
}
I am unsure how to do this for nested fields in this way nor have I tried to extend it to nested fields, but this is how it works for now.
Related
I am working on a NiFi workflow. JSON is coming into the workflow and I am using EvaluateJsonPath and RouteOnAttribute processes. In the EvaluateJsonPath, I have the Null Value Representation set to empty string.
[
{
'a':null,
'b':null
},
{
'a':'1',
'b':'2'
}
]
I use a JsonSplitter to split the array so each element in the JSON array is handled separately.
In the first EvaluateJsonPath I do the following.
a=$.a
b=$.b
In the RouteToAttribute (I am using the same property, if I use a different or new property, I get the same issue)
a=${a:isEmpty():ifElse('NOT PROTECTED', 'PROTECTED')}
b=${b:isEmpty():ifElse('', 'ACTIVE')}
The RouteToAttribute processor is providing the following error for both properties:
'a' validated against '${a:isEmpty():ifElse('NOT PROTECTED', 'PROTECTED')}' is invalid because Expected Attribute Query to return type BOOLEAN but query returns type STRING
'b' validated against '${b:isEmpty():ifElse('', 'ACTIVE')}' is invalid because Expected Attribute Query to return type BOOLEAN but query returns type STRING
It's really not clear what you are trying to do.
You are using RouteOnAttribute, but it looks like you are trying to replace attribute values. This is not what RouteOnAttribute does.
RouteOnAttribute is designed to evaluate a query, and then route matching FlowFiles to a specified relationship.
With RouteOnAttribute you have 3 options for the Routing Strategy which are
Route to Property name
Route to 'matched' if all match
Route to 'matched' if any matches
When using option 1, that is, Route to Property name, the NAME of the property is the RELATIONSHIP that the FlowFile will be routed to. The VALUE of the property is a QUERY that must return a BOOLEAN (true or false). If the result of the query is True, the FlowFile is routed to this relationship. If you have multiple properties configured, it will evaluate them in order.
When using option 2 or 3, you do not specify the name of the relationship, it is always called matched. You can specify multiple properties, and the NAME of the property is irrelevent. The VALUE of the property is a QUERY that must return a BOOLEAN. With option 2, if ALL of the queries return True, the FlowFile is routed to matched. With option 3, if ANY of the queries return True, the FlowFile is routed to matched.
Read the documentation.
Your error is caused because ${a:isEmpty():ifElse('NOT PROTECTED', 'PROTECTED')} does not return a BOOLEAN. It returns a STRING. There is nothing RouteOnAttribute can do with this result.
Either way, you would be better off looking at Records.
Specifically, if you want to update the value of fields inside the data, look at UpdateRecord.
If you want to do routing logic on fields inside the data, look at QueryRecord and PartitionRecord.
Splitting each JSON element out using SplitJson is inefficient and avoidable when using Records.
I have two kinds of documents in my couchbase bucket with keys like -
product.id.1.main
product.id.2.main
product.id.3.main
and
product.id.1.extended
product.id.2.extended
product.id.3.extended
I want to write a view for documents of first kind, such that when some conditions are matched for a document, I can emit the attributes contained in the documents of first kind as well as the document of second kind.
Something like -
function(doc, meta){
if((meta.id).match("product.id.*.main") && doc.attribute1.match("value1"){
var extendedDocId = replaceMainWithExtended(meta.id)
emit(meta.id, doc.attribute1 + getExtendedDoc(extendedDocId).extendedAttribute1 );
}
}
I want to know how to implement this kind of function in couchbase views -
getExtendedDoc(extendedDocId).extendedAttribute1
Is it possible to add a look up to a mutation in GraphQL? Let's say something like an input type where one property is the result of another query.
createPerson(name: "Steve", gender: genders ( name: { eq: "mail" } ) { id } )
where genders ( name: { eq: "mail" } ) { id } would return exactly one result value
GraphQL allows you to:
"get what you want";
manipulate data on write;
customize response on read;
You can of course write a createPerson mutation resolver to:
ask another table for additional data (if missing arg);
write into DB;
But it doesn't affect (isn't suitable for) existing records.
You can also use read resolvers to update missing fields in records using field resolver - a kind of 'eventual consistency'.
Write person resolver to read a person record and additional person.gender field resolver to get value for missing gender value. The second one can be used to update the 'main' person [DB table] record:
read a missing value from other table;
write a value into person record on the 'main' table;
return value.
Next time person resolver will read gender value at once (from 'main' person table), [additional] field resolver won't be called at all.
This technique can be used to convert DB data without writing SQL/DB specific code (different syntax/types problems, safer on more complex logic) - it's enough to read each/all person's gender fields. Field resolver (and the other/old table) can be removed after that.
I am trying to do simple upsert to the array field based on branch condition. However branch does not accept a reql expression as argument and I get error Expected type SELECTION but found DATUM.
This is probably some obvious thing I've missed, however I can't find any working example anywhere.
Sample source:
var userId = 'userId';
var itemId = 'itemId';
r.db('db').table('items').get(itemId).do(function(item) {
return item('elements').default([]).contains(function (element) {
return element('userId').eq(userId);
}).branch(
r.expr("Element already exist"),
//Error: Expected type SELECTION but found DATUM
item.update({
elements: item('elements').default([]).append({
userId: 'userId'
})
})
)
})
The problem here is that item is a datum, not a selection. This happens because you used r.do. The variable doesn't retain information about where the object originally came from.
A solution that might seem to work would be to write a new r.db('db').table('items').get(itemId) expression. The problem with that option is the behavior isn't atomic -- two different queries might append the same element to the 'elements' array. Instead you should write your query in the form r.db('db').table('items').get(itemId).update(function(item) { return <something>;) so that the update gets applied atomically.
mysql function in Codeigniter I Use list_field() to return an array containing the field names. What does the equivalent in mongoDB to return an array which contains he field name?
This code below I use the list_field function
function GetField($execution) {
$query = $this->db->query($execution);
if ($query->num_rows() > 0) {
return $query->list_fields();
} else {
return array();
}
}
My goal is if there is input with complex queries, the column appears dynamically in accordance with the input query
So the point I want to create applications such as rockmongo & phpMyAdmin. So when we enter a query, the results matching the query which we enter. Well I still have not found a function that bson field of MongoDB appear dynamically according to the query given by user.
example the query like this below :
db.users.find({}, {a:1,b:1})
so the bson field that appears is a and b.
There is NO equivalent query in MongoDB. It is a schema-less database, which means that every document can have totally different set of fields.
The find method returns a cursor iterating which you will get a result which in turn depending on a driver may be a hash-table or a dictionary where its keys are the names of a document's attributes.
something like that:
{"name" : "John","age" : 30, "_id" : "497ce96f395f2f052a494fd4"}
so that "field names" would be "name","age","_id"
Please Note
Since Mongo is schema-less for each document in a query there may be different keys.