For charting and analytics purposes it is easier to have all data in a single CSV file. But from pure OOP perspective it is much better to have proper Class hierarchy established. I use jackson-dataformat-csv library to parse a CSV file into a Java POJO. This POJO is getting unwieldy as the number of columns are growing. Any way we can parse this into nested object Hierarchy?
Related
How can we validate input from XML and JSON, need to create that element in DB2?
This page will help you with validating XML https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.1.0/com.ibm.db2.luw.xml.doc/doc/c0050643.html
For JSON, as of Db2 11.1.3.3 you can use SYSTOOLS.JSON2BSON() to validate that a string is valid JSON. To validate it conforms to some schema is less easy. https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.1.0/com.ibm.swg.im.dbclient.json.doc/doc/r0070290.html
If you store your JSON data as BSON you know it will be valid JSON. Store your XML as XML datatype for validation and lots of other advantages.
XMLTABLE() is one way to extract elements from XML into other (non XML) Db2 columns. JSON_TABLE() can do similar for JSON.
For general info on Db2’s XML capability, google for “PureXML” as well as using information from the Knowledge Center manual.
(all above assumes you are using Db2 for Linux, Unix or Windows)
I have an avro schema in NiFi which validates the columns of a CSV file, all is working well, however I'd like to ideally have an extra level of validation on certain string column to test that they adhere to specific patterns. For example ABC1234-X, or whatever. Here's the wrinkle though, the avro schema is generated for specific expected files, and so the NiFi processors need to be generic.
Is there a way to have the avro schema do this?
I know BeanWrapperFieldSetMapper class depends on POJO.
But here is the thing: If I want to take advantage of Spring Batch features but do not want to create separate jobs ( does not want to write POJOs and separate reader writes or mappers) how to do this?
My requirement is to read *.csv file which will have the headers so I should be able to supply header names in a map or string[] and create my sql statement based on it, instead of writing a RowMapper.
This will help me uploading various files to different tables.
Is it possible to change BeanWrapperFieldSetMapper to make it suitable to map the values from Map or String[]?
Also Even if I do not have headers in the *.cvs file, I can construct update statement and load using chunk delimeters setting and other advantages of Spring Batch.
I know BeanWrapperFieldSetMapper class depends on POJO.
But here is the thing: If I want to take advantage of Spring Batch features but do not want to create separate jobs ( does not want to write POJOs and separate reader writes or mappers) how to do this?
My requirement is to read *.csv file which will have the headers so I should be able to supply header names in a map or string[] and create my sql statement based on it, instead of writing a RowMapper.
This will help me uploading various files to different tables.
Is it possible to change BeanWrapperFieldSetMapper to make it suitable to map the values from Map or String[]?
Also Even if I do not have headers in the *.cvs file, I can construct update statement and load using chunk delimeters setting and other advantages of Spring Batch.
does anyone have examples of orientDB etl transformers with multiple transforms or something that can create class identifiers on the fly, so for example, if you want to create organization entities and the id could be a hash from the organization name , essentially if the json we are importing is not exactly the schema we want in the destination
What about using block code in your ETL configuration file? You can use it in the begin phase, so you could transform id column in your .csv input file. It is not an ideal solution I agree.
see the Block documentation