I am attempting to process all 261 pages of https://ucdpapi.pcr.uu.se/api/gedevents/21.1?pagesize=1000 concurrently and asynchronously. Like most applications, I have to map the result details to another rest resource and post it to another system.
Initially, I did a simple #Blocking rest call in a while loop to make sure the Client was configured correctly. This took 48 minutes to return the 261 pages.
Next I converted to asynchronous reactive Multi paging via help from https://quarkus.io/blog/mutiny-pagination/
However as shown in the logs, this is processing in a single thread and actually hung on page 119.
The question is how to I change this to run concurrently based on a configurable amount of threads?
#GET
#Produces(MediaType.APPLICATION_JSON+";charset=iso-8859-1")
public GedResult findGedPage(Map<String, Integer> paramMap) {
return gedClient.findGedPage(paramMap);
}
public void pagination() {
log.info("STARTING GED EVENT PAGINATION");
long startTime = System.currentTimeMillis();
AtomicInteger totalPages = new AtomicInteger();
AtomicInteger totalDetails = new AtomicInteger();
Multi.createBy().repeating()
.supplier(
AtomicInteger::new,
page -> {
Map<String, Integer> paramMap = new HashMap<>(2);
paramMap.put("pagesize", pageSize);
paramMap.put("page", page.getAndIncrement());
return gedClient.findGedPage(paramMap);
})
.until(gedResult -> isBlank(gedResult.getNextPageUrl()))
.onItem()
.transform(Unchecked.function(result -> {
return mapToMyResource(result, totalPages, totalDetails);
}))
.subscribe()
.with(
this::sendDownstreamToPersist,
GedServiceException::new,
() -> log.info("onComplete:: Total Pages = " + totalPages.get() + " in " + (System.currentTimeMillis() - startTime) + " millis")
);
}
2022-08-15 14:40:36,528 INFO (pool-11-thread-1) [com.bar.ucdp.service.GedService.mapToMyResource()] NextPageUrl: https://ucdpapi.pcr.uu.se/api/gedevents/21.1?pagesize=2000&page=116
2022-08-15 14:40:52,816 INFO (pool-11-thread-1) [com.bar.ucdp.service.GedService.mapToMyResource()] NextPageUrl: https://ucdpapi.pcr.uu.se/api/gedevents/21.1?pagesize=2000&page=117
2022-08-15 14:41:05,997 INFO (pool-11-thread-1) [com.bar.ucdp.service.GedService.mapToMyResource()] NextPageUrl: https://ucdpapi.pcr.uu.se/api/gedevents/21.1?pagesize=2000&page=118
2022-08-15 14:41:19,670 INFO (pool-11-thread-1) [com.bar.ucdp.service.GedService.mapToMyResource()] NextPageUrl: https://ucdpapi.pcr.uu.se/api/gedevents/21.1?pagesize=2000&page=119
2022-08-15 14:41:33,894 INFO (pool-11-thread-1) [com.bar.ucdp.service.GedService.mapToMyResource()] NextPageUrl: https://ucdpapi.pcr.uu.se/api/gedevents/21.1?pagesize=2000&page=120
2022-08-15 14:41:46,315 INFO (pool-11-thread-1) [com.bar.ucdp.service.GedService.mapToMyResource()] NextPageUrl: https://ucdpapi.pcr.uu.se/api/gedevents/21.1?pagesize=2000&page=121
Related
I am running Jmeter performance test from java application in distributed mode (2 slaves + master). In my test script I have configured Summary Report to store result data to csv file. This file location is configured with fixed name "result/summary.csv" value but the csv result file is not getting generated.
Result file is created when the test is run only on one master but not when using master + slave combination
public static void main(String[] argv) throws Exception {
File jmeterHome = new File(System.getProperty("jmeter.home"));
String slash = System.getProperty("file.separator");
if (jmeterHome.exists()) {
File jmeterProperties = new File(jmeterHome.getPath() + slash + "bin" + slash + "jmeter.properties");
if (jmeterProperties.exists()) {
//JMeter Engine
DistributedRunner distributedRunner = new DistributedRunner();
//JMeter initialization (properties, log levels, locale, etc)
JMeterUtils.setJMeterHome(jmeterHome.getPath());
JMeterUtils.loadJMeterProperties(jmeterProperties.getPath());
JMeterUtils.initLogging();// you can comment this line out to see extra log messages of i.e. DEBUG level
JMeterUtils.initLocale();
// JMeter Test Plan, basically JOrphan HashTree
HashTree testPlanTree = new HashTree();
// First HTTP Sampler - open example.com
HTTPSamplerProxy examplecomSampler = new HTTPSamplerProxy();
examplecomSampler.setDomain("example.com");
examplecomSampler.setPort(80);
examplecomSampler.setPath("/");
examplecomSampler.setMethod("GET");
examplecomSampler.setName("Open example.com");
examplecomSampler.setProperty(TestElement.TEST_CLASS, HTTPSamplerProxy.class.getName());
examplecomSampler.setProperty(TestElement.GUI_CLASS, HttpTestSampleGui.class.getName());
// Second HTTP Sampler - open blazemeter.com
HTTPSamplerProxy blazemetercomSampler = new HTTPSamplerProxy();
blazemetercomSampler.setDomain("blazemeter.com");
blazemetercomSampler.setPort(80);
blazemetercomSampler.setPath("/");
blazemetercomSampler.setMethod("GET");
blazemetercomSampler.setName("Open blazemeter.com");
blazemetercomSampler.setProperty(TestElement.TEST_CLASS, HTTPSamplerProxy.class.getName());
blazemetercomSampler.setProperty(TestElement.GUI_CLASS, HttpTestSampleGui.class.getName());
// Loop Controller
LoopController loopController = new LoopController();
loopController.setLoops(1);
loopController.setFirst(true);
loopController.setProperty(TestElement.TEST_CLASS, LoopController.class.getName());
loopController.setProperty(TestElement.GUI_CLASS, LoopControlPanel.class.getName());
loopController.initialize();
// Thread Group
ThreadGroup threadGroup = new ThreadGroup();
threadGroup.setName("Example Thread Group");
threadGroup.setNumThreads(1);
threadGroup.setRampUp(1);
threadGroup.setSamplerController(loopController);
threadGroup.setProperty(TestElement.TEST_CLASS, ThreadGroup.class.getName());
threadGroup.setProperty(TestElement.GUI_CLASS, ThreadGroupGui.class.getName());
// Test Plan
TestPlan testPlan = new TestPlan("Create JMeter Script From Java Code");
testPlan.setProperty(TestElement.TEST_CLASS, TestPlan.class.getName());
testPlan.setProperty(TestElement.GUI_CLASS, TestPlanGui.class.getName());
testPlan.setUserDefinedVariables((Arguments) new ArgumentsPanel().createTestElement());
// Construct Test Plan from previously initialized elements
testPlanTree.add(testPlan);
HashTree threadGroupHashTree = testPlanTree.add(testPlan, threadGroup);
threadGroupHashTree.add(blazemetercomSampler);
threadGroupHashTree.add(examplecomSampler);
// save generated test plan to JMeter's .jmx file format
SaveService.saveTree(testPlanTree, new FileOutputStream(jmeterHome + slash + "example.jmx"));
//add Summarizer output to get test progress in stdout like:
// summary = 2 in 1.3s = 1.5/s Avg: 631 Min: 290 Max: 973 Err: 0 (0.00%)
Summariser summer = null;
String summariserName = JMeterUtils.getPropDefault("summariser.name", "summary");
if (summariserName.length() > 0) {
summer = new Summariser(summariserName);
}
// Store execution results into a .jtl file
String logFile = jmeterHome + slash + "example.jtl";
// Store execution results into a .csv file
//String logFile = jmeterHome + slash + "csv.jtl";
ResultCollector logger = new ResultCollector(summer);
logger.setFilename(logFile);
testPlanTree.add(testPlanTree.getArray()[0], logger);
ArrayList<String> remoteHostList = new ArrayList<String>();
remoteHostList.add("0.0.0.127");
remoteHostList.add("0.0.0.128");
List<JMeterEngine> engines = new LinkedList<JMeterEngine>();
distributedRunner.setStdout(System.out);
distributedRunner.setStdErr(System.err);
distributedRunner.init(remoteHostList, testPlanTree);
engines.addAll(distributedRunner.getEngines());
distributedRunner.start();
System.out.println("Test completed. See " + jmeterHome + slash + "example.jtl file for results");
System.out.println("JMeter .jmx script is available at " + jmeterHome + slash + "example.jmx");
System.exit(0);
}
}
System.exit(1);
}
When you're running JMeter in distributed mode the master node:
Sends .jmx test plan to the slaves
Collects results from the slaves and merges then into a single .jtl results file
It's hard to say what's wrong without seeing your code, but my expectation is that you need to use ClientJMeterEngine instead of StandardJMeterEngine
Adding below code helped me to resolve the issue.
String logFile = jmeterHome + slash + "example.csv";
ResultCollector resultCollector = new ResultCollector()
resultCollector.setProperty(TestElement.TEST_CLASS, ResultCollector.class.getName());
resultCollector.setProperty(TestElement.GUI_CLASS, "SummaryReport");
resultCollector.setFilename(logFile);
testPlanTree.add(testPlanTree.getArray()[0], logger);
Test class and GUI class was not added before, adding it worked me.
Now I can download the result from both master and slave machines.
How can I close or reset a conversation programmatically from Java app?. According to Dialogflow CX documentation "A session remains active and its data is stored for 30 minutes after the last request is sent for the session."
I want to keep the session active for less time. For example, if I want the session to be active for 5 minutes, when user sends a message 5 minutes or more after last message, conversation must start again and previous flows must be closed and context parameters must be deleted.
With Dialogflow ES it is posible using ContextsClient, however new version does not offer ContextsClient class.
Dialogflow CX uses State Handlers to control conversation paths, unlike Dialogflow ES which uses Contexts.
For Dialogflow CX, you can end the current session by using the END_SESSION symbolic transition target. Once the END_SESSION transition target is invoked, it clears the current session and the next user input will restart the session at the start page of the Default Start Flow.
To achieve your desired use case, you’ll have to create your own implementation for it. Note that the solution below will only work if you integrate your Dialogflow CX agent to a custom front-end.
First, you should add an Event Handler to all of your Pages - so that the Event Handler will be accessible in any part of the conversation flow. In this Event Handler, define a custom event - for example: clearsession. Then, set its Transition to End Session Page. Once the clearsession event is invoked, it will end the current session.
Then, using your own business logic, you can create a custom function that could act as a timer for each user query. Once the timer reaches 5 minutes, your custom application should send a detectIntent request to your CX agent programmatically. This detectIntent request must contain the current session ID and the custom event (from the previously created Event Handler).
Here’s a sample detectIntent request that invokes a custom event using the Java Client Library:
// [START dialogflow_cx_detect_intent_event]
import com.google.api.gax.rpc.ApiException;
import com.google.cloud.dialogflow.cx.v3.*;
import com.google.common.collect.Maps;
import java.io.IOException;
import java.util.List;
import java.util.Map;
public class DetectIntent {
// DialogFlow API Detect Intent sample with event input.
public static Map<String, QueryResult> detectIntentEvent(
String projectId,
String locationId,
String agentId,
String sessionId,
String languageCode,
String event)
throws IOException, ApiException {
SessionsSettings.Builder sessionsSettingsBuilder = SessionsSettings.newBuilder();
if (locationId.equals("global")) {
sessionsSettingsBuilder.setEndpoint("dialogflow.googleapis.com:443");
} else {
sessionsSettingsBuilder.setEndpoint(locationId + "-dialogflow.googleapis.com:443");
}
SessionsSettings sessionsSettings = sessionsSettingsBuilder.build();
Map<String, QueryResult> queryResults = Maps.newHashMap();
// Instantiates a client
try (SessionsClient sessionsClient = SessionsClient.create(sessionsSettings)) {
// Set the session name using the projectID (my-project-id), locationID (global), agentID
// (UUID), and sessionId (UUID).
SessionName session = SessionName.of(projectId, locationId, agentId, sessionId);
System.out.println("Session Path: " + session.toString());
EventInput.Builder eventInput = EventInput.newBuilder().setEvent(event);
// Build the query with the EventInput and language code (en-US).
QueryInput queryInput =
QueryInput.newBuilder().setEvent(eventInput).setLanguageCode(languageCode).build();
// Build the DetectIntentRequest with the SessionName and QueryInput.
DetectIntentRequest request =
DetectIntentRequest.newBuilder()
.setSession(session.toString())
.setQueryInput(queryInput)
.build();
// Performs the detect intent request.
DetectIntentResponse response = sessionsClient.detectIntent(request);
// Display the query result.
QueryResult queryResult = response.getQueryResult();
System.out.println("====================");
System.out.format(
"Detected Intent: %s (confidence: %f)\n",
queryResult.getIntent().getDisplayName(), queryResult.getIntentDetectionConfidence());
}
return queryResults;
}
public static void main(String[] args) {
String projectId = "<project-id>";
String locationId = "<location-id>";
String agentId = "<agent-id>";
String sessionId = "<current-session-id>";
String languageCode = "<language-code>";
String event = "clearsession";
try{
detectIntentEvent(projectId,locationId,agentId,sessionId, languageCode, event);
} catch (IOException e){
System.out.println(e.getMessage());
}
}
}
// [END dialogflow_cx_detect_intent_event]
Here is the scenario
a)Tablet use http GET to get users info from another system
b)the other system can add/delete/update users info also and the operations result should show in tablet in time.
To resolve that I use reactor+sse as demo code shows below
In controller
#GetMapping
public Flux<List<User>> listUser() {
return service.listUser();
}
In service
public Flux<List<User>> listUser() {
Flux<List<User>> flux = Flux.generate(
() -> 0,
(state, sink) -> {
synchronized(users) {
users.wait();//other thread will notify with all users info is send to service
}
sink.next(users);
newData = false;
return state + 1;
});
return flux;
}
My questions are,
a)should I use Flux<User> to instead of Flux<List<User>>? I didn't see any usage of Flux<List<T>> so far. also one advantage is that with Flux<User>, I don't need to push all users info to the tablet, only the add/delete/update ones.
b)if I use Flux<List<User>, what should I write for bodyToFlux's parameters in below code
webClient.get()
.uri("/user")
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToFlux() //???
In my Spring/Hibernate/MVC application I implement some long running functions. In this case I would like to inform the user about the status.
I wonder, if I can get certain log4j messages to the frontend, like:
DEBUG d.t.webapp.services.StageServiceImpl - Fetching result
DEBUG d.t.webapp.services.StageServiceImpl - loaded 1000 rows
DEBUG d.t.webapp.services.StageServiceImpl - loaded 1000 rows
DEBUG d.t.webapp.services.StageServiceImpl - loaded 1000 rows
DEBUG d.t.webapp.services.StageServiceImpl - loaded 994 rows
Those message should appear to the frontend step-by-step, as they are produced by the backend, with minimal delay. Is that possible or are there better solutions for this requirement?
Or why not just use something like this:
1# Create a controller to provide you with last status of your worker
#RestController
#RequestMapping(value = "/worker")
public WorkerStatusProviderController {
#Autowired
WorkerStatusProvider workerStatusProvider;
#RequestMapping(value = "/status/{id}", method = RequestMethod.GET)
public new ResponseEntity<WorkerStatus> getStatus(#PathVariable("id") long workerId){
//retrieve last worker status
WorkerStatus lastStatus = workerStatusProvider.getLastStatus(workerId);
//send worker status to front
return new ResponseEntity<WorkerStatus>(lastStatus, HttpStatus.OK);
}
}
2# In frontend use Ajax to call this controller every 2 or 3 seconds and display the last status to your user.
//call /worker/status/id every 2 seconds
setInterval(function(){
$.get("/worker/status/100", function(workerStatus, status) {
//display worker status to the user
alert("workerStatus: " + workerStatus);
});
}, 2000);
I am having some problems with Spring statemachine + Spring boot application.
I am using Redis for persistence, as in Event service demo. I have one action which is executed when state machine enters state 'VALIDATION'. In it, I modify extended state variables. Code example below is simplified, but tested:
This is main, calling method:
public void feedMachine(String event, int orderItem) throws Exception {
System.out.println("DEBUG FEEDMACHINE STARTED");
StateMachine<String, String> machine = persister.restore(stateMachine, orderItem);
machine.sendEvent(event);
System.out.println("DEBUG FEEDMACHINE... machine = " + machine.getUuid());
System.out.println("DEBUG FEEDMACHINE... variables = " + machine.getExtendedState().getVariables());
if (machine.getExtendedState().getVariables().containsKey("guard")
&& !machine.getExtendedState().get("guard", Boolean.class)) {
System.out.println("DEBUG FEEDMACHINE... guard = " + machine.getExtendedState().get("guard", Boolean.class));
throw new GuardedEventException();
}
persister.persist(machine, orderItem);
System.out.println("DEBUG FEEDMACHINE ENDED");
}
This is action:
#Bean
public Action<String, String> CustomerOrderValidatedEntryAction() {
return new Action<String, String>() {
#Override
public void execute(StateContext context) {
System.out.println("DEBUG ACTION STARTED");
System.out.println("DEBUG ACTION... machine = " + context.getStateMachine().getUuid());
int orderItem = context.getStateMachine().getExtendedState().get(ORDER_ITEM, Integer.class);
context.getExtendedState().getVariables().put("guard", false);
System.out.println("DEBUG ACTION... variables = " + context.getExtendedState().getVariables());
persister.persist(context.getStateMachine(), key);
System.out.println("DEBUG ACTION ENDED");
}
catch (Exception e) {
e.printStackTrace();
}
}
};
}
When I leave action, in feedMachine extendedState.getVariables does not contain guard and errors variable.
This is output:
DEBUG FEEDMACHINE STARTED
2016-08-16 11:46:17.101 INFO 26483 --- [nio-8081-exec-2] o.s.s.support.LifecycleObjectSupport : stopped org.springframework.statemachine.support.DefaultStateMachineExecutor#a85eba8
2016-08-16 11:46:17.102 INFO 26483 --- [nio-8081-exec-2] o.s.s.support.LifecycleObjectSupport : stopped ... / / uuid=336abc3e-708e-46ad-892f-5835b50713c8 / id=null
2016-08-16 11:46:17.102 INFO 26483 --- [nio-8081-exec-2] o.s.s.support.LifecycleObjectSupport : started org.springframework.statemachine.support.DefaultStateMachineExecutor#a85eba8
2016-08-16 11:46:17.102 INFO 26483 --- [nio-8081-exec-2] o.s.s.support.LifecycleObjectSupport : started ... / ORDER_RECEIVED / uuid=336abc3e-708e-46ad-892f-5835b50713c8 / id=null
DEBUG ACTION STARTED
DEBUG ACTION... machine = 336abc3e-708e-46ad-892f-5835b50713c8
DEBUG ACTION... guard = false
DEBUG ACTION... variables = {orderItem=0, guard=false}
DEBUG ACTION ENDED
DEBUG FEEDMACHINE... machine = 336abc3e-708e-46ad-892f-5835b50713c8
DEBUG FEEDMACHINE... variables = {orderItem=0}
DEBUG FEEDMACHINE ENDED
Any help would be appreciated. I am not sure if this is connected with Spring statemachine or with Spring beans or something else.
I apologize for primitive code (using println instead of log.debug etc.).
EDIT:
I finally found out where was the problem.
I was using in action context.getExtendedState().getVariables().put("guard", valid); instead of context.getStateMachine().getExtendedState().getVariables().put("guard", valid);. Now it works as expected. Problem was that those two variables lists were not same.
context.getExtendedState()'s variables list would get updated with "guard"
context.getStateMachine().getExtendedState()'s variables list would not get updated.
However, after creating minimal working example (only 3 states instead of 15+, no nested states), it was working correctly with context.getExtendedState().getVariables().put("guard", valid);.
I also checked several days ago whether context.getExtendedState() and context.getStateMachine().getExtendedState() are same, and I concluded that they were.
This seems to me as a bug.