I have a flow-files with the below structure
{
"PN" : "U0-WH",
"INPUT_DATE" : "44252.699895833335",
"LABEL" : "Marker",
"STATUS" : "Approved",
}
and I need to execute an update statement using some fields
update table1 set column1 = 'value' where pn=${PN}
I found convertJsonToSQL but am not sure how to use it in this case
You can use a processor namely ConvertjSONToSQL. Using this you can convert your json into an update query.
ConvertjSONToSQL Description
It takes the following parameters :
1. JDBC Connection Pool : Create a JDBC pool which takes DB connection information as input.
2. Statement Type : Here you need to provide type of statement you want to create. In your case its 'UPDATE'.
3. Table Name : Name of the table for which update query needed to be created
4. Schema Name : Name of the schema of your database.
5. Translate Field Names : If true, the Processor will attempt to translate JSON field names into the appropriate column names for the table specified. If false, the JSON field names must match the column names exactly, or the column will not be updated
6. Unmatched Field Behaviour : if an incoming JSON element has a field that does not map to any of the database table's columns, this property specifies how to handle the situation
7. Unmatched Column Behaviour : If an incoming JSON element does not have a field mapping for all of the database table's columns, this property specifies how to handle the situation
8. Update Keys : A comma-separated list of column names that uniquely identifies a row in the database for UPDATE statements. If the Statement Type is UPDATE and this property is not set, the table's Primary Keys are used. In this case, if no Primary Key exists, the conversion to SQL will fail if Unmatched Column Behaviour is set to FAIL. This property is ignored if the Statement Type is INSERT
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Read the description above and try to use the properties given. Detailed description of the processor is given in the link.
ConvertjSONToSQL Description
Related
This is my flow file content:
{
"a":"b",
"c":"y",
"d":"z",
"e":"w",
"f":"u",
"g":"v",
"h":"o",
"x":"t"
}
The final result should look like that in Postgres :
| test |
|----------------------------------------------------------------|
|{"a":"b,"c":"y","d":"z","e":"w","f":"u","g":"v","h":"o","x":"t"}|
the table is: json_test
the column name is test
Those steps shows how i tried to solve the problem:
My method was to store the json record in a variable as string with "ExtractText":
the attribute data take only some key-values from the json not the entire record:
data = {"a":"b",
"c":"y",
"d":"z",
"e":"w",
"f":
so i have a problem in the regex expression.
next i used PutSQL with the following SQL statement:
Unfortunately the result isn't the wanted one.
I need to know the exact expression that i should set in ExtractText to get the entire json record in a variable as string.
The sql statement should be:
insert into schema.table_name(column_name) values(the_variable_where the flowfile data was stored)
response := db.Where("createdAt BETWEEN ? AND ?", today, tomorrow).Find(&orders)
I want to make a query with GORM. The column in my table is named "createdAt" but GORM converts it to. "createdat"
ERROR: column "createdat" does not exist (SQLSTATE 42703)
[172.521ms] [rows:0] SELECT * FROM "Orders" WHERE `createdAt` BETWEEN '2021-04-21 04:00:00' AND '2021-04-22 04:00:00'
How can I prevent GORM from converting the query text to lowercase?
Its not gorm's issue its database issue means in database column is created with name created_at just check there and try replacing your code with this
response := db.Where("created_at BETWEEN ? AND ?", today, tomorrow).Find(&orders)
If you use case sensitive column name, use \",ex: \"createdAt\", No modern database cannot handle upper or lower case text.
I am working on a NiFi workflow. JSON is coming into the workflow and I am using EvaluateJsonPath and RouteOnAttribute processes. In the EvaluateJsonPath, I have the Null Value Representation set to empty string.
[
{
'a':null,
'b':null
},
{
'a':'1',
'b':'2'
}
]
I use a JsonSplitter to split the array so each element in the JSON array is handled separately.
In the first EvaluateJsonPath I do the following.
a=$.a
b=$.b
In the RouteToAttribute (I am using the same property, if I use a different or new property, I get the same issue)
a=${a:isEmpty():ifElse('NOT PROTECTED', 'PROTECTED')}
b=${b:isEmpty():ifElse('', 'ACTIVE')}
The RouteToAttribute processor is providing the following error for both properties:
'a' validated against '${a:isEmpty():ifElse('NOT PROTECTED', 'PROTECTED')}' is invalid because Expected Attribute Query to return type BOOLEAN but query returns type STRING
'b' validated against '${b:isEmpty():ifElse('', 'ACTIVE')}' is invalid because Expected Attribute Query to return type BOOLEAN but query returns type STRING
It's really not clear what you are trying to do.
You are using RouteOnAttribute, but it looks like you are trying to replace attribute values. This is not what RouteOnAttribute does.
RouteOnAttribute is designed to evaluate a query, and then route matching FlowFiles to a specified relationship.
With RouteOnAttribute you have 3 options for the Routing Strategy which are
Route to Property name
Route to 'matched' if all match
Route to 'matched' if any matches
When using option 1, that is, Route to Property name, the NAME of the property is the RELATIONSHIP that the FlowFile will be routed to. The VALUE of the property is a QUERY that must return a BOOLEAN (true or false). If the result of the query is True, the FlowFile is routed to this relationship. If you have multiple properties configured, it will evaluate them in order.
When using option 2 or 3, you do not specify the name of the relationship, it is always called matched. You can specify multiple properties, and the NAME of the property is irrelevent. The VALUE of the property is a QUERY that must return a BOOLEAN. With option 2, if ALL of the queries return True, the FlowFile is routed to matched. With option 3, if ANY of the queries return True, the FlowFile is routed to matched.
Read the documentation.
Your error is caused because ${a:isEmpty():ifElse('NOT PROTECTED', 'PROTECTED')} does not return a BOOLEAN. It returns a STRING. There is nothing RouteOnAttribute can do with this result.
Either way, you would be better off looking at Records.
Specifically, if you want to update the value of fields inside the data, look at UpdateRecord.
If you want to do routing logic on fields inside the data, look at QueryRecord and PartitionRecord.
Splitting each JSON element out using SplitJson is inefficient and avoidable when using Records.
I have to ingest data in hive table from hdfs but I don't know how to choose correct data type for the data mentioned below:-
$34740$#$Disrupt Worldwide LLC$#$40425$#$null$#$13$#$6$#$317903$#$null$#$Scott Bodily$#$+$#$null$#$10$#$0$#$1$#$0$#$disruptcentral.com$#$null$#$null$#$1$#$null$#$null$#$null$#$Scott Bodily$#$1220DB56-56D7-E411-80D6-005056A451E3$#$true$
$34741$#$The Top Tipster Leagues Limited$#$35605$#$null$#$13$#$7$#$317902$#$null$#$AM Support Team$#$+447886 027371$#$null$#$1$#$1$#$1$#$0$#$www.toptipsterleagues.com, www.toptipsterleagues.co.uk, http://test.toptipsterleague.com$#$Jamil Johnson$#$Cheng Liem Li$#$1$#$0.70$#$1.50$#$1.30$#$Bono van Nijnatten$#$0B758BF9-F1D6-E411-80D7-005056A44C5C$#$true$
Refer this link for different data types,
Click here
Other than all the numeric and decimal fields you can use STRING data type. For the numeric fields based on the range and precision you can use INT or DECIMAL.
Using string and varchar or any other string data types will read null in your data as string i.e "null" for handling nul you should mention the table properties like below,
ALTER TABLE tablename SET
SERDEPROPERTIES ('serialization.null.format' = 'null');
Let me know if anything needed on this.
My json data
[{"cameraid":"000000001","timestamp":"2016-06-17 23:08","filename":"7e3800fbd0557c683874ed2f41ed7057"},
{"cameraid":"000000002","timestamp":"2016-06-17 23:08","filename":"b260cc730da88a6af4e5038d6e1e32db"}]
How can i have cameraid to link to a specific name?
Like example my cameraid "000000001" to be call as bedok.
Anybody got any idea on how to do it?
Create another collection with names(lets call it db_name) and link the cameraid to the _id of db_name collection. This way you can fetch the names using cameraid. This is more like primary key and foreign key concept in relational database(RDBMS).
More on this with code here: Primary Key and Foreign Key Concept in MongoDB
Assuming you want to add a name field for specific cameraid,
Try following query:-
db.collname.update({"cameraid":"000000001"},{$set : {name : 'bedok'}});
EDIT:-
The above query will update only one record which matches the query {"cameraid":"000000001"}.
Add multi:true to the query, to multiple records.
db.collname.update({"cameraid":"000000001"},{$set : {name : 'bedok'}},{multi : true});
Now it will update all the records that matches the query {"cameraid":"000000001"}.