How to use Nifi to write CSV files from DB data with all data sorted and no same value in different files? - oracle

Question

I have a data table in Oracle database in this format to keeps all the transactions in my system:
Customer ID
Transaction ID
001
trans_id_01
001
trans_id_02
002
trans_id_03
003
trans_id_04
As you see, each customer ID can generate many transactions in this table.
Now I need to export the data from each day into CSV files with Apache Nifi.
But the requirement is I need to have around 10k transactions in each file (this is not fixed, can have a bit more or less), with rows sorted by Customer ID. That should be simple, and I have done it with
this processor:
But there's additional requirement to ensure each Customer ID should be in the same file. There should be no case where customer id 005 have some transactions in file no. 1 and another transaction in file no. 2.
If I need to write this logic with pure coding, I think I can do DB query with pagination and write some codes to check for trailing data at the end to be compared with next page before writing each file. But when it comes to implementation with Nifi, I still have no idea how to do this.

Answer 1

But there's additional requirement to ensure each Customer ID should be in the same file. There should be no case where customer id 005 have some transactions in file no. 1 and another transaction in file no. 2.
Try I think ExecuteSQLRecord with a custom select that gets you exactly what you want from Oracle and then use PartitionRecord configured to use the customer ID as the partition column. That will break up the record set.
I don't know how Oracle does it, but this would be the way I'd do it in Postgres:
SELECT CUSTOMER_ID, ARRAY_AGG(TRANSACTION_ID) FROM TRANSACTIONS GROUP BY CUSTOMER_ID
That would create: 001, {trans_id_01, trans_id_02...} and ensure that each result entry from the database has precisely one customer per line and all of their transactions enumerated in a single list.

Answer 2

I have found a solution by using an idea of doing loop flow from : https://gist.github.com/ijokarumawak/01c4fd2d9291d3e74ec424a581659ca8
So I created a loop flow like in the image below:
This will query records of about 40k each time with this sql query
SELECT * FROM TRANSACTIONS WHERE <some filtering> AND CUSTOMER_ID > '${max}' ORDER BY CUSTOMER_ID FETCH FIRST 40000 ROWS With Ties
With ties keyword will help get the ties record to ensure all records of same CUSTOMER_ID is in the same file. Each success flow file will go the right side to write the data into CSV file. While the success flow also go downward to extract data for another iteration. ${max} is retrieved for the biggest value of the current result set, using QueryRecord processor with below query:
SELECT CUSTOMER_ID FROM FLOWFILE ORDER BY CUSTOMER_ID DESC fetch first 1 row only
Then it will go to the next iteration in a loop until there's no data left for the current criteria