Creating Hive table on Parquet file which has JSON data - hadoop

Question

The objective I'm trying to achieve
Obtain the data from source big JSON files ( employee-sample.json)
A simple spark application to read it as text file and store in parquet ( simple-loader.java ). I'm not aware of what is in JSON file so I cannot put any schema , so I want schema on read, and not schema on write. A parquet file with one column named "value" containing the JSON string in created
Create an HIVE external table on parquet files, and when I do "select * from table", I see one-column coming out with JSON data.
What I really need is creating a HIVE table which could read the JSON-data in "value" column and apply schema and emit columns, so that I can create variety of tables on my RAW-data, depending on the need.
I have created hive tables on JSON file, and extracted the columns , but this extract column from parquet and apply JSON schema is tricking me
employee-sample.json
{"name":"Dave", "age" : 30 , "DOB":"1987-01-01"}
{"name":"Steve", "age" : 31 , "DOB":"1986-01-01"}
{"name":"Kumar", "age" : 32 , "DOB":"1985-01-01"}
Simple Spark code to convert JSON to parquet
simple-loader.java
public static void main(String[] args) {
SparkSession sparkSession = SparkSession.builder()
.appName(JsonToParquet.class.getName())
.master("local[*]").getOrCreate();
Dataset<String> eventsDataSet = sparkSession.read().textFile("D:\\dev\\employee-sample.json");
eventsDataSet.createOrReplaceTempView("rawView");
sparkSession.sqlContext().sql("select string(value) as value from rawView")
.write()
.parquet("D:\\dev\\" + UUID.randomUUID().toString());
sparkSession.close();
}
hive table on parquet files
CREATE EXTERNAL TABLE EVENTS_RAW (
VALUE STRING)
STORED AS PARQUET
LOCATION 'hdfs://XXXXXX:8020/employee/data_raw';
I tried by setting JSON serde, but it works only if data is stored in JSON foram, ROW FORMAT SERDE 'com.proofpoint.hive.serde.JsonSerde'
EXPECTED FORMAT
CREATE EXTERNAL TABLE EVENTS_DATA (
NAME STRING,
AGE STRING,
DOB STRING)
??????????????????????????????

Answer 1

Create hive external table example:
public static final String CREATE_EXTERNAL = "CREATE EXTERNAL TABLE %s" +
" (%s) " +
" PARTITIONED BY(%s) " +
" STORED AS %s" +
" LOCATION '%s'";
/**
* Will create an external table and recover the partitions
*/
public void createExternalTable(SparkSession sparkSession, StructType schema, String tableName, SparkFormat format, List<StructField> partitions, String tablePath){
String createQuery = createTableString(schema, tableName, format, partitions, tablePath);
logger.info("Going to create External table with the following query:\n " + createQuery);
sparkSession.sql(createQuery);
logger.debug("Finish to create External table with the following query:\n " + createQuery);
recoverPartitions(sparkSession, tableName);
}
public String createTableString(StructType schema, String tableName, SparkFormat format, List<StructField> partitions, String tablePath){
Set<String> partitionNames = partitions.stream().map(struct -> struct.name()).collect(Collectors.toSet());
String columns = Arrays.stream(schema.fields())
//Filter the partitions
.filter(field -> !partitionNames.contains(field.name()))
//
.map(HiveTableHelper::fieldToStringBuilder)
.collect(Collectors.joining(", "));
String partitionsString = partitions.stream().map(HiveTableHelper::fieldToStringBuilder).collect(Collectors.joining(", "));
return String.format(CREATE_EXTERNAL, tableName, columns, partitionsString, format.name(), tablePath);
}
/**
*
* #param sparkSession
* #param table
*/
public void recoverPartitions(SparkSession sparkSession, String table){
String query = "ALTER TABLE " + table + " RECOVER PARTITIONS";
logger.debug("Start: " + query);
sparkSession.sql(query);
sparkSession.catalog().refreshTable(table);
logger.debug("Finish: " + query);
}
public static StringBuilder fieldToStringBuilder(StructField field){
StringBuilder sb = new StringBuilder();
sb.append(field.name()).append( " ").append(field.dataType().simpleString());
return sb;
}