I am currently new on Apache Nifi and still exploring it.
I made a custom processor where I will fetch data from server with pagination.
I pass the input file which will contains the attribute "url".
Finally transfer the response in output flow file, as I fetch data with pagination, so I made a new output flow file for each page and transferred it to Successful relationship.
Below is the code part:
#Override
public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
FlowFile incomingFlowFile = session.get();
String api = null;
if (incomingFlowFile == null) {
logger.info("empty input flow file");
session.commit();
return;
} else {
api=incomingFlowFile.getAttribute("url");
}
session.remove(incomingFlowFile);
if(api == null) {
logger.warn("API url is null");
session.commit();
return;
}
int page = Integer.parseInt(context.getProperty(PAGE).getValue());
while(page < 3) {
try {
String url = api + "&curpg=" + page;
logger.info("input url is: {}", url);
HttpResponse response = httpGetApiCall(url, 10000);
if(response == null || response.getEntity() == null) {
logger.warn("response null");
session.commit();
return;
}
String resp = EntityUtils.toString(response.getEntity());
InputStream is = new ByteArrayInputStream(StandardCharsets.UTF_16.encode(resp).array());
FlowFile outFlowFile = session.create();
outFlowFile = session.importFrom(is, outFlowFile);
session.transfer(outFlowFile, SUCCESSFUL);
} catch (IOException e) {
logger.warn("IOException :{}", e.getMessage());
return;
}
++page;
}
session.commit();
}
I am facing issue that for a single Input flow file, this processor get triggered twice and so it generates 4 flow files for a single input flow file.
I am not able to figure out this where I have done wrong.
Please help in this issue.
Thanks in advance.
======================================================================
processor group 1(Nifi_Parvin)
processor group 2 (News_Point_custom)
Related
I am trying to implement an export to excel function via a web service which uses webflux as the other api and controllers work well. My problem is that calling the function that generates the excel file is accessed after retrieving data from repository as a Flux (no problem there). I have sorted the results and am trying to call another populate methid via flatMap, I am having a number of issues trying to get this to work and to make sure that the code in the flatMap runs before the code in the webservice to return the file.
Below is the code for the webservice:
#GetMapping(API_BASE_PATH + "/download")
public ResponseEntity<byte[]> download() {
Mono<Void> createExcel = excelExport.createDocument(false);
Mono.when(createExcel).log("Excel Created").then();
Workbook workbook = excelExport.getWb();
OutputStream outputStream = new ByteArrayOutputStream();
try {
workbook.write(outputStream);
} catch (IOException e) {
e.printStackTrace();
}
byte[] media = ((ByteArrayOutputStream) outputStream).toByteArray();
HttpHeaders headers = new HttpHeaders();
headers.setCacheControl(CacheControl.noCache().getHeaderValue());
headers.setContentType(MediaType.valueOf("text/html"));
headers.set("Content-disposition", "attachment; filename=filename.xlsx");
ResponseEntity<byte[]> responseEntity = new ResponseEntity<>(media, headers, HttpStatus.OK);
return responseEntity;
}
And the code for the exelExport class:
public Mono<Void> createDocument(boolean all) {
InputStream inputStream = new ClassPathResource("Timesheet Template.xlsx").getInputStream();
try {
wb = WorkbookFactory.create(inputStream);
Sheet sheet = wb.getSheetAt(0);
Row row = sheet.getRow(1);
Cell cell = row.getCell(3);
if (cell == null)
cell = row.createCell(3);
cell.setCellType(CellType.STRING);
cell.setCellValue("a test");
log.info("Created document");
Flux<TimeKeepingEntry> entries = service.findByMonth(LocalDate.now().getMonth().getDisplayName(TextStyle.FULL, Locale.ENGLISH)).log("Excel Export - retrievedMonths");
entries.subscribe();
return entries.groupBy(TimeKeepingEntry::getDateOfMonth).flatMap(Flux::collectList).flatMap(timeKeepingEntries -> this.populateEntry(sheet, timeKeepingEntries)).then();
} catch (IOException e) {
log.error("Error Creating Document", e);
}
//should never get here
return Mono.empty();
}
private void populateEntry(Sheet sheet, List<TimeKeepingEntry> timeKeepingEntries) {
int rowNum = 0;
for (int i = 0; i < timeKeepingEntries.size(); i++) {
TimeKeepingEntry timeKeepingEntry = timeKeepingEntries.get(i);
if (i == 0) {
rowNum = calculateFirstRow(timeKeepingEntry.getDay());
}
LocalDate date = timeKeepingEntry.getFullDate();
Row row2 = sheet.getRow(rowNum);
Cell cell2 = row2.getCell(1);
cell2.setCellValue(date.toString());
if (timeKeepingEntry.getDay().equals(DayOfWeek.FRIDAY.getDisplayName(TextStyle.FULL, Locale.ENGLISH))) {
rowNum = +2;
} else {
rowNum++;
}
}
}
The workbook is never update because the populateEntry is never executed. As I said I have tried a number of differnt methods including Mono.just and Mono.when, but I cant seem to get the correct combination to get it to process before the webservice method tries to return the file.
Any help would be great.
Edit1: Shows the ideal crateDocument Method.
public Mono<Void> createDocument(boolean all) {
try {
InputStream inputStream = new ClassPathResource("Timesheet Template.xlsx").getInputStream();
wb = WorkbookFactory.create(inputStream);
Sheet sheet = wb.getSheetAt(0);
log.info("Created document");
if (all) {
//all entries
} else {
service.findByMonth(currentMonthName).log("Excel Export - retrievedMonths").collectSortedList(Comparator.comparing(TimeKeepingEntry::getDateOfMonth)).doOnNext(timeKeepingEntries -> {
this.populateEntry(sheet, timeKeepingEntries);
});
}
} catch (IOException e) {
log.error("Error Importing File", e);
}
return Mono.empty();
}
There are several problems in the implementation of your webservice.
When to subscribe
First off, in reactive programming you must generally try to build a single processing pipeline (by calling Mono and Flux operators and returning the end result as a Mono and Flux). In any case, you should either let the framework do the subscribe or at least only subscribe once, at the end of that pipeline.
Here instead you are mixing two approaches: your createDocument method correctly returns a Mono, but it also does the subscribe. Even worse, the subscription is done on an intermediate step, and nothing subscribes to the whole pipeline in the webservice method.
So in effect, nobody sees the second half of the pipeline (starting with groupBy) and thus it never gets executed (this is a lazy Flux, also called a "cold" Flux).
Mixing synchronous and asynchronous
The other problem is again an issue of mixing two approaches: your Flux are lazy and asynchronous, but your webservice is written in an imperative and synchronous style.
So the code starts an asynchronous Flux from the DB, immediately return to the controller and tries to load the file data from disk.
Option 1: Making the controller more Flux-oriented
If you use Spring MVC, you can still write these imperative style controllers yet sprinkle in some WebFlux. In that case, you can return a Mono or Flux and Spring MVC will translate that to the correct asynchronous Servlet construct. But that would mean that you must turn the OutputStream and bytes handling into a Mono, to chain it to the document-writing Mono using something like then/flatMap/etc... It is a bit more involved.
Option 2: Turning the Flux into imperative blocking code
The other option is to go back to imperative and blocking style by calling block() on the createDocument() Mono. This will subscribe to it and wait for it to complete. After that, the rest of your imperative code should work fine.
Side Note
groupBy has a limitation where if it results in more than 256 open groups it can hang. Here the groups cannot close until the end of the file has been reached, but fortunately since you only process data for a single month, the Flux wouldn't exceed 31 groups.
Thanks to #SimonBasie for the pointers, my working code is now as follows.
#GetMapping(value = API_BASE_PATH + "/download", produces = "application/vnd.ms-excel")
public Mono<Resource> download() throws IOException {
Flux<TimeKeepingEntry> createExcel = excelExport.createDocument(false);
return createExcel.then(Mono.fromCallable(() -> {
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
excelExport.getWb().write(outputStream);
return new ByteArrayResource(outputStream.toByteArray());
}));
}
public Flux<TimeKeepingEntry> createDocument(boolean all) {
Flux<TimeKeepingEntry> entries = null;
try {
InputStream inputStream = new ClassPathResource("Timesheet Template.xlsx").getInputStream();
wb = WorkbookFactory.create(inputStream);
Sheet sheet = wb.getSheetAt(0);
log.info("Created document");
if (all) {
//all entries
} else {
entries = service.findByMonth(currentMonthName).log("Excel Export - retrievedMonths").sort(Comparator.comparing(TimeKeepingEntry::getDateOfMonth)).doOnNext(timeKeepingEntry-> {
this.populateEntry(sheet, timeKeepingEntry);
});
}
} catch (IOException e) {
log.error("Error Importing File", e);
}
return entries;
}
I am facing a problem in value 'return' in Asynctask class in doInBackground method. I am getting an error, about 'missing return statement in below code.
`public class ForecastNetwork extends AsyncTask {
public final String TAG = ForecastNetwork.class.getSimpleName();
#Override
protected Void doInBackground(Void... params) {
HttpURLConnection urlConnection = null;
BufferedReader reader = null;
// Will contain the raw JSON response as a string.
String forecastJsonStr = null;
try {
// Construct the URL for the OpenWeatherMap query
// Possible parameters are avaiable at OWM's forecast API page, at
// http://openweathermap.org/API#forecast
URL url = new URL("http://api.openweathermap.org/data/2.5/forecast/daily?q=94043&mode=json&units=metric&cnt=7");
// Create the request to OpenWeatherMap, and open the connection
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setRequestMethod("GET");
urlConnection.connect();
// Read the input stream into a String
InputStream inputStream = urlConnection.getInputStream();
StringBuffer buffer = new StringBuffer();
if (inputStream == null) {
// Nothing to do.
return null;
}
reader = new BufferedReader(new InputStreamReader(inputStream));
String line;
while ((line = reader.readLine()) != null) {
// Since it's JSON, adding a newline isn't necessary (it won't affect parsing)
// But it does make debugging a *lot* easier if you print out the completed
// buffer for debugging.
buffer.append(line + "\n");
}
if (buffer.length() == 0) {
// Stream was empty. No point in parsing.
return null;
}
forecastJsonStr = buffer.toString();
} catch (IOException e) {
Log.e(TAG, "Error ", e);
// If the code didn't successfully get the weather data, there's no point in attemping
// to parse it.
return null;
} finally {
if (urlConnection != null) {
urlConnection.disconnect();
}
if (reader != null) {
try {
reader.close();
} catch (final IOException e) {
Log.e(TAG, "Error closing stream", e);
}
}
}
}`
What Should I return at the end?
I assume that you forgot to return the processing result
forecastJsonStr = buffer.toString();
return forecastJsonStr;
I have an activity that is being swapped out when I raise an intent for another activity. onPause calls saveState() to save work so far:
private void saveState() {
...
...
if (myUri == null) {
// Inserting a new record
*** myUri = getContentResolver().insert(ContentProvider.CONTENT_URI, values);
} else {
// Update an existing record
getContentResolver().update(myUri, values, null, null);
}
}
Before calling getContentResolver(), ContentProvider.CONTENT_URI = 'content://nz.co.bkd.extraTime.contentprovider/times'.
After the call, myUri = 'times/#' where #=row ID. My question is; where is the 'content:...' prefix to the returned uri?
During the call, ContentResolver.java is called and returns CreatedRow uri
ContentResolver.java
....
....
public final Uri insert(Uri url, ContentValues values)
{
IContentProvider provider = acquireProvider(url);
if (provider == null) {
throw new IllegalArgumentException("Unknown URL " + url);
}
try {
long startTime = SystemClock.uptimeMillis();
*** Uri createdRow = provider.insert(url, values);
long durationMillis = SystemClock.uptimeMillis() - startTime;
maybeLogUpdateToEventLog(durationMillis, url, "insert", null /* where */);
return createdRow;
} catch (RemoteException e) {
// Arbitrary and not worth documenting, as Activity
// Manager will kill this process shortly anyway.
return null;
} finally {
releaseProvider(provider);
}
}
At this point, createdRow = 'times/#'.
The record does actually get saved in the Sqlite database.
Do I have to add the uri prefix in my code or should the full uri be returned?
<%
if (MultipartFormDataRequest.isMultipartFormData(request))
{
// Uses MultipartFormDataRequest to parse the HTTP request.
MultipartFormDataRequest mrequest = new MultipartFormDataRequest(request);
String todo = null;
if (mrequest != null) todo = mrequest.getParameter("todo");
if ( (todo != null) && (todo.equalsIgnoreCase("upload")) )
{
Hashtable files = mrequest.getFiles();
if ( (files != null) && (!files.isEmpty()) )
{
UploadFile file = (UploadFile) files.get("uploadfile");
if (file != null)
out.println("");
//out.println(report1);
String sever = mrequest.getParameter("sever");
String ease = mrequest.getParameter("ease");
String logo = "C:/uploads/"+file.getFileName(); // Uses the bean now to store specified by jsp:setProperty at the top.
String dana = mrequest.getParameter("danalysis");
String loc = mrequest.getParameter("loc");
String state = mrequest.getParameter("state");
String Sr = mrequest.getParameter("Sr");
String Doc_ID = mrequest.getParameter("doc");
String impact = mrequest.getParameter("impact");
String desc = mrequest.getParameter("desc");
String ref = mrequest.getParameter("ref");
String recom = mrequest.getParameter("recom");
try
{
String connectionURL = "jdbc:mysql://localhost:3306/mssg";
Class.forName("com.mysql.jdbc.Driver").newInstance();
Connection con=DriverManager.getConnection(connectionURL, "root","");
out.println("OK!\n");
PreparedStatement ps1=con.prepareStatement("insert into report values(?,?,?,?,?,?,?,?,?,?,?,?)");
ps1.setString(1,Doc_ID);
ps1.setString(2,Sr);
ps1.setString(3,sever);
ps1.setString(4,ease);
ps1.setString(5,state);
ps1.setString(6,loc);
ps1.setString(7,desc);
ps1.setString(8,impact);
ps1.setString(9,dana);
ps1.setString(10,logo);
ps1.setString(11,recom);
ps1.setString(12,ref);
int count=ps1.executeUpdate();
if(count > 0)
{
out.println("successfully inserted");
//response.sendRedirect("/index.jsp");
}
else
{
out.println("error occured");
}
}
catch (Exception e)
{
System.out.println("error in program:-"+e);
}
upBean.store(mrequest, "uploadfile");
}
else
{
out.println("<li>No uploaded files");
}
}
}
%>
In the above code I get all the values also file is uploading but not able to insert these values into database. I think I made a very small mistake, please tell me what is the problem in the above code. This code is working until connection but after prepare statement it's not working.
This is beacuse you have used try and catch block.
In the jsp page life cycle, the jsp page is translated in the servlets. so the code for the database connectivity will be automatically placed in the try catch block.
Just remove the try catch block.
Hello guys
I am sending my form values to controller and controller to rptdesign file my it is generating the report in temp folder with proper value but my requirement is that it should user to save or open dialog so that user can save the report or open
i think ajax request will not allow to download any file so if some one know to better solution plz reply
my controller is below
#RequestMapping("/leave/generateEmpLeaveReport.json")
public void generateEmployeeLeaveReport(HttpServletRequest request,
HttpServletResponse response) throws Exception {
String reportName = "D:/git-repositories/cougar_leave/src/java/com//report/myLeaveSummary.rptdesign";
File designTemplateFile = new File(reportName);
if (!designTemplateFile.exists()) {
throw new FileNotFoundException(reportName);
}
Map<String, Object> parameters = new HashMap<String, Object>();
parameters.put("empId", NumberUtils.toInt(request.getParameter("id")));
parameters.put("reportTitle", "EMPLOYEE LEAVE");
parameters.put("fromDate", request.getParameter("fromDate"));
parameters.put("toDate", request.getParameter("toDate"));
parameters.put("leaveType",
NumberUtils.toInt(request.getParameter("leaveType")));
parameters.put("transactionType",
NumberUtils.toInt(request.getParameter("transactionType")));
reportManager.addSystemParams(parameters, null,
RequestUtils.getUser(request));
File file = null;
try {
ReportType reportType = ReportType.PDF;
OfflineReportContext reportContext = new OfflineReportContext(
reportName, reportType, parameters, null,
"EMPLOYEE LEAVE SUMMARY");
StringBuffer buffer = new StringBuffer();
file = offlineReportGenerator.generateReportFile(reportContext,
buffer);
ControllerUtils
.openFile(file.getParent(), response, file.getName());
} catch (Exception e) {
log.error(e, e);
} finally {
if (file != null && file.exists()) {
file.canExecute();
}
}
}
my ajax request is below
generateReport : function() {
if (this.form.valid()) {
fromDate = new Date($("input[name='fromDate']").val())
toDate = new Date($("input[name='toDate']").val())
if (fromDate > toDate) {
GtsJQuery
.showError("To date should be greater or equals than From date !")
} else {
var request = GtsJQuery.ajax3(GtsJQuery.getContextPath()
+ '/leave/generateEmpLeaveReport.json', {
data : {
id : $("input[name='employeeId']").val(),
fromDate : $("input[name='fromDate']")
.val(),
toDate : $("input[name='toDate']").val(),
leaveType : $("select[name='leaveType']")
.val(),
transactionType : $("select[name='transactionType']")
.val(),
orderBy : $("select[name='orderBy']").val()
}
});
request.success(this.callback("onSubscribeSuccess"))
}
}
},
The controller response should be the temp file itself, just adjust the content-type.