I have an ssh manager to execute (bash) scripts on a server. It contains a commandWithContinousRead(String command, Consumer<String> consumer). Whenever an echo is called in the bash script it is consumed by the consumer. I want to extend this with Spring Boot and an HTTP call. When a client sends a request, the server streams the data when it's ready from a bash script and the client can print it out.
I know Server-Sent Events, however, I feel like that is mostly for events and usually uses multiple resources on an API.
Additionally, I tried searching for streaming topics, but had no success. I did find StreamingResponseBody from Spring, but it collects all the data and then sends it all at once.
I used Postman for testing, maybe it cannot handle streaming?
However, how do I test this?
Example:
#/bin/bash
# Scriptname: stream-this.sh
echo "Starting line"
sleep 4
echo "Middle line"
sleep 4
echo "End line"
Request with commandWithContinousRead, but prints everything at once after eight seconds.
#RequestMapping(value = "/stream-this", method = RequestMethod.POST,
produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public ???? streamScript() {
StreamingResponseBody stream = out -> {
sshManager.commandWithContinousRead("bash /scripts/stream-this.sh", echo -> {
try {
byte[] bytes = echo.getBytes(StandardCharsets.UTF_8);
out.write(bytes);
System.out.println(echo);
} catch (IOException e) {
e.printStackTrace();
}
});
};
return new ResponseEntity<>(stream, HttpStatus.OK);
}
Implementation of commandWithContinousRead function.
public void commandWithContinousRead(String command, Consumer<String> consumer) {
SSHClient client = buildClient();
try (Session session = client.startSession()) {
Session.Command cmd = session.exec(command);
BufferedReader br = new BufferedReader(new InputStreamReader(cmd.getInputStream(), StandardCharsets.UTF_8));
String line;
while ((line = br.readLine()) != null) {
consumer.accept(line);
}
br.close();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
client.disconnect();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Now that you have posted the commandWithContinuousRead method, everything looks correct. Also, you've just now stated that you're testing with Postman, and that's definitely a problem -- postman doesn't support streaming responses
https://github.com/postmanlabs/postman-app-support/issues/5040
It's always a good idea to programmatically unit and integration test your code. A simple unit test doesn't even need to use Spring, or a real SSH connection (run the bash script local to the test). The unit test would just be testing the logic of your Consumer and would let you know that the reading of the output, and the bash script itself aren't blocking. Ideally, you would use junit, but here's a simple test class that I put together that shows what I mean.
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.IOException;
import java.lang.Process;
import java.nio.charset.StandardCharsets;
import java.util.function.Consumer;
public class Test {
// This would be a #Test instead of a main
public static void main(String... args) {
commandWithContinousRead("bash stream-this.sh", echo -> {
byte[] bytes = echo.getBytes(StandardCharsets.UTF_8);
// assert statements go here
System.out.println("In main -- " + echo);
});
}
public static void commandWithContinousRead(String command, Consumer<String> consumer) {
try {
Process process = Runtime.getRuntime().exec(command);
BufferedReader br = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = br.readLine()) != null) {
consumer.accept(line);
}
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
An integration test would actually setup Spring, and would go through the endpoint, thereby testing in the same manner that the client/browser would. Commonly, this is done using #WebMvcTest and mockMvc async. You could choose to either mock the SSH client, or to have a server setup explicitly so your actual SSH client can connect to it. (The second option would expose/eliminate issues related to the ssh connection). This kind of test would expose issues with the spring setup/streaming response. You would need to set an artificial timeout on your mock mvc after say, 5 seconds and using a new mock mvc, after 9 seconds That would allow you to see that after 5 seconds, you've received the first echo, and after 9, you have the whole expected response. A good starting point for you would be to look at https://www.tabnine.com/code/java/methods/org.springframework.test.web.servlet.result.RequestResultMatchers/asyncStarted
Having passed those two levels of tests, then you would begin to suspect the client, which in this case, is Postman. If possible, try to use the actual browser(s) or clients that will be running your code. It may turn out that streaming might not be an option for you.
Please post the implementation of commandWithContinousRead
It could be a fundamental problem where the script that is echoing and sleeping is running on the same thread as the code that is supposed to read the echo and print them out. I.e., you're blocking while you wait for the bash script itself to run which would explain the 8 second delay before getting any output. Also, what type does commandWithContinousRead return? Depending, on how you're "reading" the echos in that method, you could be blocking there too. It's hard to say with 100% certainty without seeing the code for commandWithContinousRead.
Your return type will be a ResponseEntity<StreamingResponseBody> (to fill in the ????)
Okay, I came up with a solution that worked. As Pickled Brain mentioned, the main problem was Postman not working with streaming. Also, I went back to try SSE in a single call and I did by running the bash script in another thread. Additionally, I created an SSE client in Nodejs for testing purposes and it worked flawlessly.
Function to run the script, and place it in another thread.
private SseEmitter runScript() {
SseEmitter emitter = new SseEmitter(-1L); // -1L = no timeout
ExecutorService sseMvcExecutor = Executors.newSingleThreadExecutor();
sseMvcExecutor.execute(() -> {
try {
shellManager.commandWithContinousRead("bash scriptname"), s -> {
SseEmitter.SseEventBuilder event = SseEmitter.event().name("message").data(s);
try {
emitter.send(event);
System.out.println(s);
} catch (IOException e) {
e.printStackTrace();
}
});
emitter.send(SseEmitter.event().name("close").data(""));
} catch (IOException e) {
e.printStackTrace();
}
emitter.complete();
});
return emitter;
}
SSE Client:
const EventSource = require('eventsource'); // npm install eventsource
const url = 'yoururl';
var es = new EventSource(url);
es.onopen = function(ev) {
console.log("OPEN");
console.log(ev);
};
es.onmessage = function(ev) {
console.log("MESSAGE");
console.log(ev.data);
};
es.addEventListener('close', function() {
es.close();
console.log('closing!');
});
es.onerror = function(ev) {
console.log("ERROR");
console.log(ev);
es.close();
};
process.on('SIGINT', () => {
es.close();
console.log(es.CLOSED);
});
I have a jmeter test that is already defined in the gui like this.
I am automating running this jmeter test from java, and I want to set ${__P(threads)} from within the java code.
The relevant code is :
public List<String> runJmxTest(String jmxFile, String jtlFile) throws IOException {
HashTree testPlanTree;
List<String> resultSet = new ArrayList<>();
// Initialize JMeter SaveService
SaveService.loadProperties();
JMeterVariables j = new JMeterVariables();
j.put("threads", "10");
// Load existing .jmx Test Plan
File in = new File(jmeterHome.getPath() + "/bin/testPlans/" + jmxFile);
try{
testPlanTree = SaveService.loadTree(in);
} catch (FileNotFoundException e){
resultSet.add("fail");
resultSet.add(e.toString());
return resultSet;
}
// set up custom result collector with summariser
Summariser summer = new Summariser("caos-mbm summariser");
collector = new myResultCollector(summer);
if(jtlFile != null){
if(!jtlFile.contains(".jtl")) {
String jtlTmp = jtlFile.concat(".jtl");
collector.setFilename(jmeterHome.getPath() + "/bin/testPlans/Output/" + jtlTmp);
} else {
collector.setFilename(jmeterHome.getPath() + "/bin/testPlans/Output/" + jtlFile);
}
}
testPlanTree.add(testPlanTree.getArray()[0], collector);
// Run Test Plan
jm.configure(testPlanTree);
jm.run();
resultSet.add("success");
resultSet.add(Double.toString(collector.getErrorPercent()));
return resultSet;
}
I have tried setting the property through the props, adding it to the test plan tree, adding jmeterproperties to the jmetercontext. I can't get it to pick up the variable though.
Any advice would be appreciated. I have also looked through quite a few posts on here that seem similar but the solutions didn't work for me or the implementation was off.
You're using the wrong class, remove these lines:
JMeterVariables j = new JMeterVariables();
j.put("threads", "10");
and add the following instead:
org.apache.jmeter.util.JMeterUtils.setProperty("threads", "10");
You need to do this after loading the Test Plan and before running the test.
Also make sure to add ApacheJMeter_functions.jar to your project CLASSPATH
More information on running JMeter test using JMeter API: Five Ways To Launch a JMeter Test without Using the JMeter GUI
I'm trying to manage router via Java application using Jcraft Jsch library.
I'm trying to send Router Config via TFTP server. The problem is in my Java code because this works with PuTTY.
This my Java code:
int port=22;
String name ="R1";
String ip ="192.168.18.100";
String password ="root";
JSch jsch = new JSch();
Session session = jsch.getSession(name, ip, port);
session.setPassword(password);
session.setConfig("StrictHostKeyChecking", "no");
System.out.println("Establishing Connection...");
session.connect();
System.out.println("Connection established.");
ChannelExec channelExec = (ChannelExec)session.openChannel("exec");
InputStream in = channelExec.getInputStream();
channelExec.setCommand("enable");
channelExec.setCommand("copy run tftp : ");
//Setting the ip of TFTP server
channelExec.setCommand("192.168.50.1 : ");
// Setting the name of file
channelExec.setCommand("Config.txt ");
channelExec.connect();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
String line;
int index = 0;
StringBuilder sb = new StringBuilder();
while ((line = reader.readLine()) != null)
{
System.out.println(line);
}
session.disconnect();
I get
Line has an invalid autocommand '192.168.50.1'
The problem is how can I run those successive commands.
Calling ChannelExec.setCommand multiple times has no effect.
And even if it had, I'd guess that the 192.168.50.1 : and Config.txt are not commands, but inputs to the copy run tftp : command, aren't they?
If that's the case, you need to write them to the command input.
Something like this:
ChannelExec channel = (ChannelExec) session.openChannel("exec");
channelExec.setCommand("copy run tftp : ");
OutputStream out = channelExec.getOutputStream();
channelExec.connect();
out.write(("192.168.50.1 : \n").getBytes());
out.write(("Config.txt \n").getBytes());
out.flush();
In general, it's always better to check if the command has better "API" than feeding the commands to input. Commands usually have command-line arguments/switches that serve the desired purpose better.
A related question: Provide inputs to individual prompts separately with JSch.
I am trying to execute shell script placed in a bucket using dataflow job. I can execute gsutil commands using this job using Direct Runner :
String[] cmdline = { "cmd.exe", "/c", "gsutil ls gs://Bucketname" };
Process p = Runtime.getRuntime().exec(cmdline);
BufferedReader reader = new BufferedReader(new
InputStreamReader(p.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null)
{
System.out.println(line);
}
Note : I will use dataflow runner to execute the script because i am using windows machine.
Try using this. It runs in my case. You have to deploy the code in cloud as a .jar or a maven project. The path /home/*/test.sh is in the cloud console.
String[] cmd = {"sh", "/home/akash/test.sh", "/home/akash/"};
Runtime.getRuntime().exec(cmd);
error :_ jmeter.util.BeanShellInterpreter: Error invoking bsh method: eval org/json/simple/JSONArray
Import java method from project jar file which is placed in lib folder.
source code on written for pass json value in arraylist:
import jsonresponse.common.JsonResponseProcessor;
import assertions.AssertResponse;
import databaseresponse.common.DbResponseProcessors;
import org.json.simple.JSONArray;
String resJson = prev.getResponseDataAsString();
String res = resJson.get("queueId");
log.info("----->>>>"+resJson);
ArrayList list1 = new ArrayList();
list1.add("queueId");
list1.add("name");
list1.add("faxNumber");
list1.add("description");
list1.add("type");
ArrayList list2 = new ArrayList();
list2.add("userId");
ArrayList list3 = new ArrayList();
list3.add("agencyId");
ArrayList list4 = new ArrayList();
list4.add("usertypeId");
JsonResponseProcessor obj = new JsonResponseProcessor();
System.out.println("%%%%%%%%%%%%%%fgfggfffgf%%%%%%%%%%%%%%%");
ArrayList Jsoin = obj.getvalueofsubmapoflist(resJson,".result[0]",list1);
System.out.println("%%%%%%%%%%%%%%%%%%%%%%%%%%%%%" +Jsoin);
log.info(">>>"+Jsoin);
Make sure you have all referenced .jar files in JMeter classpath (i.e. copy them to JMeter's "lib" folder). Don't forget to restart JMeter to pick the jars up
If you still experience issues try putting your code inside try block like:
try {
//your code here
}
catch (Throwable ex) {
log.error("Problem in Beanshell", ex);
throw ex;
}
This way you'll have informative stacktrace printed to jmeter.log file.
One more way to get more information regarding your Beanshell script is putting debug(); directive to the beginning of your code. If will trigger debugging output into stdout
See How to Use BeanShell: JMeter's Favorite Built-in Component article for more information on using Java and JMeter APIs from Beanshell test elements in JMeter tests.