OraclePropertyGraphDataLoader loadData from HDFS - oracle

Question

I'm using Spark+Hive to build graphs and relations and export flat OPV/OPE files to HDFS, one OPV/OPE CSV per reducer.
All our graph database is ready to be loaded on OPG/PGX for analytics an that worked like a charm.
Now, we want to load those vertices/edges on Oracle Property Graph.
I'v dumped the filenames from hdfs this way:
$ hadoop fs -find '/user/felipeferreira/dadossinapse/ops/*.opv/*.csv' | xargs -I{} echo 'hdfs://'{} > opvs.lst
$ hadoop fs -find '/user/felipeferreira/dadossinapse/ops/*.ope/*.csv' | xargs -I{} echo 'hdfs://'{} > opes.lst
And I'm experimenting on groovy shell with some issues and doubts:
opvs = new File('opvs.lst') as String[]
opes = new File('opes.lst') as String[]
opgdl.loadData(opg, opvs, opes, 72)
That doesn't work out of the box, I receive errors like
java.lang.IllegalArgumentException: loadData: part-00000-f97f1abf-5f69-479a-baee-ce0a7bcaa86c-c000.csv flat file does not exist
I'll manage that with a InputStream approach available in the loadData interface, hope that solves this problem, but I have some questions/sugestions:
Does loadData support vfs so I may load 'hdfs://...' files directly?
Wouldn't be nice to have glob syntax in the filenames so we may do something like:
opgdl.loadData(opg, 'hdfs:///user/felipeferreira/opvs/**/*.csv' ...
Thanks in advance!

Answer 1

You can use an alternate API from OraclePropertyGraphDataLoader where you can specifiy the InputStream objects for the opv/ope files used for loading. This way, you can use FsDataInputStream objects to read the files from your HDFS environment.
A small sample is the following:
// ====== Init HDFS File System Object
Configuration conf = new Configuration();
// Set FileSystem URI
conf.set("fs.defaultFS", hdfsuri);
conf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
conf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());
// Set HADOOP user
System.setProperty("HADOOP_USER_NAME", "hdfs");
System.setProperty("hadoop.home.dir", "/");
//Get the filesystem - HDFS
FileSystem fs = FileSystem.get(URI.create(hdfsuri), conf);`
// Read files into InputStreams using HDFS FsDataInputStream Java APIs
**Path pathOPV = new Path("/path/to/file.opv");
FSDataInputStream inOPV = fileSystem.open(pathOPV);
Path pathOPV = new Path("/path/to/file.ope");
FSDataInputStream inOPE = fileSystem.open(pathOPE);**
cfg = GraphConfigBuilder.forPropertyGraphHbase().setName("sinapse").setZkQuorum("bda1node05,bda1node06").build()
opg = OraclePropertyGraph.getInstance(cfg)
opgdl = OraclePropertyGraphDataLoader.getInstance();
opgdl.loadData(opg, **inOPV, inOPE**, 100);
Let us know if this one works for you.

Answer 2

For the sake of tracking, here is the solution we'v adopted:
Mounted the hdfs through the NFS gateway on a folder below the groovy shell.
Exported the filenames to the OPV/OPE list-of-files:
$ find ../hadoop/user/felipeferreira/dadossinapse/ -iname "*.csv" | grep ".ope" > opes.lst
$ find ../hadoop/user/felipeferreira/dadossinapse/ -iname "*.csv" | grep ".opv" > opvs.lst
Then it was as simple as this to load the data on the opg/hbase:
cfg = GraphConfigBuilder.forPropertyGraphHbase().setName("sinapse").setZkQuorum("bda1node05,bda1node06").build()
opg = OraclePropertyGraph.getInstance(cfg)
opgdl = OraclePropertyGraphDataLoader.getInstance()
opvs = new File("opvs.lst") as String[]
opes = new File("opes.lst") as String[]
opgdl.loadData(opg, opvs, opes, 100)
This seems to get bottlenecked by the nfs gateway, but we will evaluate this next week.
Graph data loading is running just fine so far.
If anyone would suggest a better approach, please let me know!