Using apache nifi, I want to get the all the input twitter files which have the hashtags for "tech"
The input json is
{
"created_at": "Sun Mar 25 18:00:43 +0000 2018",
"id": 977968537028481025,
"id_str": "977968537028481025",
"text": "#bby__nim You know like datttt",
"entities": {
"hashtags": [
{
"text": "tech",
"indices": [
12,
17
]
},
{
"text": "BusinessPlan",
"indices": [
48,
61
]
}
],
"urls": [
],
"user_mentions": [
{
"screen_name": "bby__nim",
"name": "bbynim\ud83d\udc7d",
"id": 424356807,
"id_str": "424356807",
"indices": [
0,
9
]
}
],
"symbols": [
]
},
"favorited": false,
"retweeted": false,
"filter_level": "low",
"lang": "en",
"timestamp_ms": "1522000843661"
}
Under EvaulateJsonPath, validating whether the hashtags is present or not using $.{entities.hashtags:jsonPath('$[0]')}, which is successfully validating
But in RouteOnAttribute, can someone tell me how to validate whether the entities.hashtags has the value called tech?
Use the evaluate json path processor configs as shown below,
Now we are extracting all the text values from the hashtags array and keeping as flowfile attribute.
In addition you have to change the properties Destination to flowfile-attribute and Return Type as Json
Now we are having all the text values in an array then use RouteOnAttribute processor to validate the entities.hastags attribute having a tech value in it (or) not.
RouteOnAttribute configs:-
Add a new property as
contains tech
${entities.hashtags:contains("tech")} //checks as sub string if the array having tech
We are using contains function in our expression language will evaluate the array having tech substring in it or not.
But we need to check the values in the array so use the below expression language for that
contains tech
${anyDelineatedValue("${entities.hashtags:replace('[',''):replace(']','')}",","):equals('"tech"')} //checks values in the array
we are using anyDelineatedValue,replace,equals functions in our expression language will evaluate the array having tech values in it or not.
In addition if you want to check the first text value in hashtags array then your evaluatejson path would be
entities.hashtags
$.entities.hashtags[0].text
Related
I am converting a JSON to csv and while pushing it to putfile I want to validate my csv before pushing. How should I give Schema in ValidateCSV processor to do that.
Below is the sample JSON and need validate it before pushing it to putfile.
[
{
"name": "Tony",
"age": 20,
"regdate": "2022-07-01 02:15:15",
"due_date": "2021-05-01 03:30:33",
"start_date": "2021-05-01 03:30:33"
},
{
"name": "Steve",
"age": 21,
"regdate": "2022-03-01 05:22:15",
"due_date": "2022-03-01 05:22:15",
"start_date": "2022-04-01 02:30:33"
},
{
"name": "Peter",
"age": 23,
"regdate": "2021-08-06 02:20:15",
"due_date": "2022-01-03 05:30:33",
"start_date": "2022-01-03 05:30:33"
}
]
I have given schema in JSON and CSV values but below is the error
my schema is name,age,start_date,regdate,due_date
Suggest a valid schema for me to proceed further.
Check the documentation of the ValidateCsv processor:
The cell processors cannot be nested (except with Optional which gives the possibility to define a CellProcessor for values that could be null) and must be defined in a comma-delimited string as the Schema property.
Input to processor (after ConvertRecord Json -> CSV):
name,age,regdate,due_date,start_date
Tony,20,2022-07-01 02:15:15,2021-05-01 03:30:33,2021-05-01 03:30:33
Steve,21,2022-03-01 05:22:15,2022-03-01 05:22:15,2022-04-01 02:30:33
Peter,23,2021-08-06 02:20:15,2022-01-03 05:30:33,2022-01-03 05:30:33
For your use case probably schema:
StrNotNullOrEmpty(),ParseInt(),ParseDate("yyyy-MM-dd HH:mm:ss"),ParseDate("yyyy-MM-dd HH:mm:ss"),ParseDate("yyyy-MM-dd HH:mm:ss")
The problem I am having is :
Sharepoint Get File Files (Properties Only) can only do one filter for ODATA, not a a second AND clause so I need to use Filter Array to make secondary filter work. And it does work....
But now I need to take my filtered array and somehow get the {FullPath} property and get the file content via passing a path and I get this error...
[ {
"#odata.etag": ""1"",
"ItemInternalId": "120",
"ID": 120,
"Modified": "2022-03-21T15:03:31Z",
"Editor": {
"#odata.type": "#Microsoft.Azure.Connectors.SharePoint.SPListExpandedUser",
"Claims": "i:0#.f|membership|dev#email.com",
"DisplayName": "Bob dole",
"Email": "dev#email.com",
"Picture": "https://company.sharepoint.us/sites/devtest/_layouts/15/UserPhoto.aspx?Size=L&AccountName=dev#email.com",
"Department": "Information Technology",
"JobTitle": "Senior Applications Developer II"
},
"Editor#Claims": "data",
"Created": "2022-03-21T15:03:31Z",
"Author": {
"#odata.type": "#Microsoft.Azure.Connectors.SharePoint.SPListExpandedUser",
"Claims": "i:0#.f|membership|dev#email.com",
"DisplayName": "Bob Dole",
"Email": "dev#email.com",
"Picture": "https://company.sharepoint.us/sites/devtest/_layouts/15/UserPhoto.aspx?Size=L&AccountName=dev#email.com",
"Department": "Information Technology",
"JobTitle": "Senior Applications Developer II"
},
"Author#Claims": "i:0#.f|membership|dev#email.com",
"OData__DisplayName": "",
"{Identifier}": "Shared%2bDocuments%252fSDS%252fFiles%252fA10_NICKEL%2bVANADIUM%2bPRODUCT_PIS-USA_French.pdf",
"{IsFolder}": false,
"{Thumbnail}": ...DATA,
"{Link}": "https://company.sharepoint.us/sites/devtest/Shared%20Documents/SDS/Files/A10_NICKEL%20VANADIUM%20PRODUCT_PIS-USA_French.pdf",
"{Name}": "A10_NICKEL VANADIUM PRODUCT_PIS-USA_French",
"{FilenameWithExtension}": "A10_NICKEL VANADIUM PRODUCT_PIS-USA_French.pdf",
"{Path}": "Shared Documents/SDS/Files/",
"{FullPath}": "Shared Documents/SDS/Files/A10_NICKEL VANADIUM PRODUCT_PIS-USA_French.pdf",
"{IsCheckedOut}": false,
"{VersionNumber}": "1.0" } ]
So from what I can see, I think it's what I thought. Even though you're filtering an array down to a single element, you need to treat it like an array.
I'm going to make an assumption that you're always going to retrieve a single item as a result of your filter step.
I created a variable (SharePoint Documents) to store your "filtered" array so I could then do the work to extract the {FullPath} property.
I then created variable that is initialised with the first (again, I'm making the assumption that your filter will only ever return a single element) and used this expression ...
variables('SharePoint Documents')?[0]['{FullPath}']
This is the result and you can use that in your next step to get the file content from SharePoint ...
If my assumption is wrong and you can have more than one then you'll need to throw it in a loop and do the same sort of thing ...
This is the expression contained within ...
items('For_Each_in_Array')['{FullPath}']
Result ...
I actually ended up doing this and it works.
How should i design a Restful API for PATCH operation that support update some property in the a list with condition?
say i have following json model:
{
"key1": "value",
"key2": "value",
"list": [
{
"property": "someValue",
"toBePatched": "value"
},
{
"property": "otherValue",
"toBePatched": "value"
}
]
}
I need to patch the "toBePatched" property in the list when the "property" equals to "someValue". By looking at the json patch here, i think it is a good way to go, but i dont think the json pointer supports the query? how should i define a path that supports "/list/property=someValue/toBePatch"?
One stupid way to do it is to pass it as query parameter to the api, and have some logic around it, but i dont think thats a standard way to do it.
[
{ "op": "test", "path": "/list/0/property", "value": "someValue"},
{ "op": "test", "path": "/list/0/toBePatched", "value": "value"},
{ "op": "replace", "path": "/list/0/toBePatched", "value": "the-new-value"}
]
test is important, it lets you verify that the server hasn't change the part of the document that you are intending to change. See section 5 on Error Handling for details.
Could someone recommend what schema to choose in YAML file for API, if the expected response is the following:
"items": [{
"garden": [{
"tree": "pine",
"height": "33"
}, {
"height": "33",
"age": "200"
}
]
}
],
I was thinking about
list (items)
object (garden)
list (list of objects)
object (pairs - height+age)
I am confused, because property height may appear a lot of times in the list together with different params.
I have a JSON file contact.txt that has been parsed into an object called JSONObj that is structured like this:
[
{
"firstName": "John",
"lastName": "Smith",
"address": {
"streetAddress": "21 2nd Street",
"city": "New York",
"state": "NY",
"postalCode": "10021"
},
"phoneNumbers": [
{ "type": "home", "number": "212 555-1234" },
{ "type": "fax", "number": "646 555-4567" }
]
},
{
"firstName": "Mike",
"lastName": "Jackson",
"address": {
"streetAddress": "21 Barnes Street",
"city": "Abeokuta",
"state": "Ogun",
"postalCode": "10122"
},
"phoneNumbers": [
{ "type": "home", "number": "101 444-0123" },
{ "type": "fax", "number": "757 666-5678" }
]
}
]
I envision editing the file/object by taking in data from a form so as to add more contacts. How can I do this?
The following method for adding a new contact to the JSONObj's array doesn't seem to be working, what's the problem?:
var newContact = {
"firstName": "Jaseph",
"lastName": "Lamb",
"address": {
"streetAddress": "25 2nd Street",
"city": "New York",
"state": "NY",
"postalCode": "13021"
},
"phoneNumbers": [
{ "type": "home", "number": "312 545-1234" },
{ "type": "fax", "number": "626 554-4567" }
]
}
var z = contact.JSONObj.length;
contact.JSONObj.push(newContact);
It depends on what technology you're using. The basic process is to read the file in, convert it to whatever native datatypes (hash, dict, list, etc.) using a JSON parsing library, modify or add data to the native object, then convert it back to JSON and store it to the file.
In Python, using the simplejson library it would look like this:
import simplejson
jsonobj = simplejson.loads(open('contact.txt'))
#python's dict syntax looks almost like JSON
jsonobj.append({
'firstName': 'Steve',
'lastName': 'K.',
'address': {
'streetAddress': '123 Testing',
'city': 'Test',
'state': 'MI',
'postalCode': '12345'
},
'phoneNumbers': [
{ 'type': 'home', 'number': '248 555-1234' }
]
})
simplejson.dump(jsonobj, open('contact.txt', 'w'), indent=True)
The data in this example is hardcoded strings, but it could come from another file or a web application request / form data, etc. If you're doing this in a web app though I would advise against reading and writing to the same file (in case two requests come in at the same time).
Please provide more information if this doesn't answer your question.
In response to "isn't there way to do this using standard javascript?":
To parse a JSON string in Javascript you can either eval it (not safe) or use a JSON parser like this one's JSON.parse. Once you have the converted JSON object you can perform whatever modifications you want to it in standard JS. You can then use that same library to convert a JS object to a JSON string (JSON.stringify). Javascript does not allow file access (unless you're doing serverside JS), so that would prevent you from reading & writing to your contact.txt file directly. You'd have to use a serverside language (like Python, Java, etc.) to read and write the file.
Once you have read in the JSON, you just have an associative array - or rather you have a pseudo-associative array, since this is Javascript. Either way, you can treat the thing as one big list of dictionaries. You can access it by key and index.
So, to play with this object:
var firstPerson = JSONObj[0];
var secondPerson = JSONObj[1];
var name = firstPerson['firstName'] + ' ' + firstPerson['lastName'];
Since you will usually have more than two people, you probably just want to loop through each dictionary in your list and do something:
for(var person in jsonList) {
alert(person['address']);
}
If you want to edit the JSON and save it back to a file, then read it into memory, edit the list of dictionaries, and rewrite back to the file.
Your JSON library will have a function for turning JSON into a string, just as it turns a string into JSON.
p.s. I suggest you observe JavaScript conventions and use camelcase for your variable names, unless you have some other customs at your place of employment. http://javascript.crockford.com/code.html