I'm executing some code on docker in my java application using ProcessBuilder to run the code, however i'm having trouble retrieving the output from it. BufferedReader is not reading anything from the InputStream returned from the container. Is there a specific way to retrieve output from Docker??
I've never had trouble getting output from bash executions before, so I'm thinking maybe docker does things differently somehow. Any ideas would be appreciated
Here's a snippet of the code:
Process dockerCommand;
ProcessBuilder builder = new ProcessBuilder("bash","-c","sudo docker images");
builder.redirectErrorStream(true);
builder.redirectOutput(ProcessBuilder.Redirect.INHERIT);
builder.redirectError(ProcessBuilder.Redirect.INHERIT);
dockerCommand = builder.start();
dockerCommand.waitFor();
List<String> result = new ArrayList<>();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(dockerCommand.getInputStream()))
{
String line = reader.readLine();
while (line != null) {
result.add(line);
line = reader.readLine();
}
}
catch (IOException exc)
{}
The line
builder.redirectOutput(ProcessBuilder.Redirect.INHERIT);
causes bash to receive the same standard output as the parent process, which is presumably your terminal window. This produces misleading results because you actually see the Docker image list, but it's being printed by the shell.
If I comment that out and then iterate over the results list, I can see the output from Docker inside the JVM.
Related
I invoke the external command within my Java app with Runtime.getRuntime().exec() or ProcessBuilder. Works fine but Veracode complains on it with CWE-78. I'm trying to use ESAPI wrapper to sanitise the input and path the check.
The arfifact is the latest
<dependency>
<groupId>org.owasp.esapi</groupId>
<artifactId>esapi</artifactId>
<version>2.2.3.1</version>
</dependency>
ESAPI.properties are
ESAPI.Logger=org.owasp.esapi.logging.slf4j.Slf4JLogFactory
Logger.LogEncodingRequired=false
Logger.UserInfo=false
Logger.ClientInfo=false
Logger.LogApplicationName=false
Logger.ApplicationName=my-app
Logger.LogServerIP=false
IntrusionDetector.Disable=true
Executor.ApprovedExecutables=/usr/bin/less
The code is:
#Test
void esapiTest() throws ExecutorException {
Executor executor = DefaultExecutor.getInstance();
ExecuteResult executeResult = executor.executeSystemCommand(
new File("/usr/bin/less"),
new ArrayList<>(Collections.singletonList("/etc/hosts"))
);
System.out.println("out = " + executeResult.getOutput());
System.out.println("err = " + executeResult.getErrors());
}
The output is
out =
err = \/etc\/hosts: No such file or directory
As far as I got the issue is that ESAPI's UnixCodec sanitises all non-alpha character with the backslash. This is fine for the shell i.e.
/usr/bin/less \/etc\/hosts
but not for the ProcessBuilder that is under the hood.
What am I doing wrong? How to invoke the command?
I think your main "problem" is misunderstanding that the ESAPI Codecs that are used with the DefaultExecutor class are assuming that any "OS command injection" is being interpreted via a "command line interpreter", i.e., a "shell". You are not invoking a shell here. If you were, the shell would remove the (in this case) backslash escaped argument from your path for "/etc/hosts". So if this were written as (say) the command:
/bin/sh -c /usr/bin/less /etc/hosts
it would [sort of] work (except if you tried running it over HTTP, the input to 'less' my be hosed; but "/bin/cat" ought to work fine).
Instead, try writing your test something like this:
#Test
void esapiTest() throws ExecutorException {
Executor executor = ESAPI.executor();
File binSh = new File("/bin/sh").getCanonicalFile();
List params = new ArrayList();
params.add("-c");
//Use '/bin/cat' because 'less' may be troublesome
params.add("\"" + "/bin/cat" + "/etc/hosts" + "\"");
ExecuteResult executeResult = executor.executeSystemCommand(binSh, params);
System.out.println("out = " + executeResult.getOutput());
System.out.println("err = " + executeResult.getErrors());
}
Note 1: If you I'm not sure how user-friendly ProcessBuilder is with commands that eventually try to do ioctl system calls to set the tty device in 'raw' mode, like commands such as "vim" or "less", which is why I changed your "/usr/bin/less" to "/bin/cat". YMMV.
Note 2: In your ESAPI.properties file, you'd have to make sure that the property 'Executor.ApprovedExecutables' is set to whatever the canonical name of "/bin/sh" is on your system. E.g., on my system, "/bin/sh" is a symbolic link to "/bin/dash", so you would have to include something like
Executor.ApprovedExecutables=/bin/bash,/bin/dash
(or at least "/bin/dash") should work.
I wrote a Spark application that generates HFiles to be used for bulk loading with the LoadIncrementalHFiles command later. As the source data pool is very big, the input files are splitted into iterations that are processed one after the other. Each iteration creates its own HFile directory, so my HDFS structure looks like this:
/user/myuser/map_data/hfiles_0
... /hfiles_1
... /hfiles_2
... /hfiles_3
...
There are about 500 files in this map_data directory, therefore I'm searching for a way to automatically call the LoadIncrementalHFiles function, to process these subdirectories also in iterations later.
The corresponding command would be this:
hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles -Dcreate.table=no /user/myuser/map_data/hfiles_0 mytable
I need to change this into an iterative command, as this command does not work with subdirectories (when I call it with the /user/myuser/map_data directory)!
I tried to use a Java Process instance to execute the command above automatically, but this doesn't seen to do anything (no output to console and also no more rows in my HBase table).
Using the org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles Java class out of my code also doesn't work, it's also not responsing!
Has anybody a working example for me? Or is there a parameter to be able to run the above hbase command on the parent directory? I'm working with HBase 1.1.2 in a Hortonworks Data Platform 2.5 cluster.
EDIT I tried to run the LoadIncrementalHFiles command from a Hadoop client Java application, but I'm getting an exception relating to snappy compression, see Run LoadIncrementalHFiles from Java client
The solution was to split the hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles -Dcreate.table=no /user/myuser/map_data/hfiles_0 mytable command into many parts (one per command part), see this Java code snippet:
TreeSet<String> subDirs = getHFileDirectories(new Path(HDFS_PATH), hadoopConf);
for(String hFileDir : subDirs) {
try {
String pathToReadFrom = HDFS_OUTPUT_PATH + "/" + hFileDir;
==> String[] execCode = {"hbase", "org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles", "-Dcreate.table=no", pathToReadFrom, hbaseTableName};
ProcessBuilder pb = new ProcessBuilder(execCode);
pb.redirectErrorStream(true);
final Process p = pb.start();
// Write the output of the Process to the console
new Thread(new Runnable() {
public void run() {
BufferedReader input = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = null;
try {
while ((line = input.readLine()) != null)
System.out.println(line);
} catch (IOException e) {
e.printStackTrace();
}
}
}).start();
// Wait for the end of the execution
p.waitFor();
...
}
I am new to java and i need help.
I use the jsch libraries to create a SHH connection to a remote linux machine. In the next code, given from a tutorial of the JSCH creator, you can see it is implemented shell communication, it is a direct communication from a cmd window. You enter a command from the cmd window and you get aback the results in the cmd window.
String user = "username";
String host = "hostname";
session session=jsch.getSession(user, host, 22);
String passwd = JOptionPane.showInputDialog("Enter password");
session.setPassword(passwd);
Channel channel=session.openChannel("shell");
session.connect();
channel.setInputStream(System.in);
channel.setOutputStream(System.out);
How i can connect the next two lines to jTextField1 and jTextField2. To give an example i want to sen the command from the jTextField1 and to get the results from the jTextField2.
channel.setInputStream(System.in);
channel.setOutputStream(System.out);
Thank you in advanced
Instead of System.in you can give your desired stream .For example
String ls = "ls \n"; // here i am passing a string to stream, you can pass as you desire
InputStream in = new ByteArrayInputStream(ls.getBytes("UTF-8"));
channel2.setInputStream(in);//you passed the string as input
ByteArrayOutputStream out = new ByteArrayOutputStream();
PrintStream ps = new PrintStream(out);
channel2.setOutputStream(ps);
String result = out.toString();//here we get output to a string which you can add to your file.
This is one option There may be better ways.
i have a new problem in image magick that look strange ..
i'm using mac osx snow leopard and i've installed image magick on it and it's working fine on command ..
but when i call it from the grails class like the following snippet it gives me
"Cannot run program "convert": error=2, No such file or directory"
the code is :-
public static boolean resizeImage(String srcPath, String destPath,String size) {
ArrayList<String> command = new ArrayList<String>(10);
command.add("convert");
command.add("-geometry");
command.add(size);
command.add("-quality");
command.add("100" );
command.add(srcPath);
command.add(destPath);
System.out.println(command);
return exec((String[])command.toArray(new String[1]));
}
private static boolean exec(String[] command) {
Process proc;
try {
//System.out.println("Trying to execute command " + Arrays.asList(command));
proc = Runtime.getRuntime().exec(command);
} catch (IOException e) {
System.out.println("IOException while trying to execute " );
for(int i =0 ; i<command.length; i++) {
System.out.println(command[i]);
}
return false;
}
//System.out.println("Got process object, waiting to return.");
int exitStatus;
while (true) {
try {
exitStatus = proc.waitFor();
break;
} catch (java.lang.InterruptedException e) {
System.out.println("Interrupted: Ignoring and waiting");
}
}
if (exitStatus != 0) {
System.out.println("Error executing command: " + exitStatus);
}
return (exitStatus == 0);
}
i've tried normal command like ls and it's ok so the problem is that grails can't find convert command itself.. is it a os problem or something?
(see lower for the answer)
I have run into the same problem. The issue appears to be something with Mac OS X specifically, as we have several Linux instances running without error. The error looks similar to the following:
java.io.IOException: Cannot run program "/usr/bin/ImageMagick-6.7.3/bin/convert /a/temp/in/tmpPic3143119797006817740.png /a/temp/out/100000726.png": error=2, No such file or directory
All the files are there, and in chmod 777 directories - and as you pointed out, running the exact command from the shell works fine.
My theory at this point is that imagemgick can not load some sort of library itself, and the "no such file" is in reference to an dylib or something along those lines.
I have tried setting LD_LIBRARY_PATH and a few others to no avail.
I finally got this working. Here is how I have it setup. I hope this helps.
The crux of the fix, for me, was I wrapped the 'convert' into a shell script, set a bunch of environment variables, and then call that shell script instead of convert directly:
(convertWrapper.sh)
export MAGICK_HOME=/usr/local/ImageMagick-6.7.5
export MAGICK_CONFIGURE_PATH=${MAGICK_HOME}/etc/ImageMagick:${MAGICK_HOME}/share/doc/ImageMagick/www/source
export PATH=${PATH}:${MAGICK_HOME}/bin
export LD_LIBRARY_PATH=${MAGICK_HOME}/lib:${LD_LIBRARY_PATH}
export DYLD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}:${MAGICK_HOME}/lib
export MAGICK_TMPDIR=/private/tmp
echo "$#" >> /private/tmp/m.log 2>&1
/usr/local/ImageMagick-6.7.5/bin/convert -verbose "$#" >> /private/tmp/m.log 2>&1
(convertWrapper.sh)
Additionally, the convert call was doing some rather complicated stuff, so I added the parameter '-respect-parenthesis' (which may or may not have had an effect).
I am not sure how much of the environment variable setting is needed as I was stabbing in the dark for a while, but since this is only for my development box...
You need to work out what your PATH is set to when you run a command from Java. It must be different to the one you have when running from the terminal.
Are you running Grails (via Tomcat?) as a different user? It might have a different path to your normal user.
you might want to try one of the Image Plugins that are part of the grails ecosystem
http://www.grails.org/ImageTools+plugin
the grails path when the app is running in the server is probably different from running java from the command line
I do so:
Put "convert" file to /usr/bin
Then add to Config.groovy:
gk {
imageMagickPath = "/usr/bin/convert"
}
Then in my ImageService.groovy:
import org.springframework.web.context.request.RequestContextHolder as RCH
[..]
def grailsApplication = RCH.requestAttributes.servletContext.grailsApplication
def imPath = grailsApplication.config.gk.imageMagickPath
def command = imPath + " some_properties"
def proc = Runtime.getRuntime().exec(command)
So this way you get command like: /usr/bin/convert some_properties
And it works, but don't forget to put file "convert" to you location and use it with this location.
I want to launch a browser and load a web page using Java's Runtime exec. The exact call looks like this:
String[] explorer = {"C:\\Program Files\\Internet Explorer\\IEXPLORE.EXE",
"-noframemerging",
"C:\\ ... path containing unicode chars ... \\Main.html"};
Runtime.getRuntime().exec(explorer);
In my case, the path contains "\u65E5\u672C\u8A9E", the characters 日本語.
Apparently it's a java bug:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4947220
My question is: is there a viable workaround that can be done solely using Java? It appears that it is possible to write a JNI library for this, but I'd like to avoid that if possible. I have tried URI-encoding the path as ascii and writing the commands to a batch file, without success.
At the mentioned Java bug page you will find a workaround that is reported to work using ProcessBuilder and wrapping the parameters in environment variables. Here is the source code from Parag Thakur:
String[] cmd = new String[]{"yourcmd.exe", "Japanese CLI argument: \ufeff\u30cb\u30e5\u30fc\u30b9"};
Map<String, String> newEnv = new HashMap<String, String>();
newEnv.putAll(System.getenv());
String[] i18n = new String[cmd.length + 2];
i18n[0] = "cmd";
i18n[1] = "/C";
i18n[2] = cmd[0];
for (int counter = 1; counter < cmd.length; counter++)
{
String envName = "JENV_" + counter;
i18n[counter + 2] = "%" + envName + "%";
newEnv.put(envName, cmd[counter]);
}
cmd = i18n;
ProcessBuilder pb = new ProcessBuilder(cmd);
Map<String, String> env = pb.environment();
env.putAll(newEnv);
final Process p = pb.start();
Create a .bat/.sh file. Write your commands to that file and execute it. Make sure that you have changed the code page to unicode in case of windows(chcp 65001).
For example to execute the below command in windows:
String[] command ={"C:\\aconex\\学校\\mysql\\bin\\mysql", "-esource", "大村箕島a\\data.sql"};
Create a temp file called temp.bat and execute with the Runtime.getRuntime().exec
temp.bat
chcp 65001
C:\aconex\学校\mysql\bin\mysql -esource 大村箕島a\data.sql
These are the two solutions I considered, each of which are more or less workarounds:
Create a temp html redirect file which will redirect the browser to the proper page.
Note that IE will expect unencoded unicode for local files, while other browsers may accept only uri-encoded file paths
Use the short filename for the windows file. It won't contain unicode characters.
We've been using a JNI to start processes from Java for years. Neither Runtime.exec or ProcessBuilder will work, and it seems unlikely that they will fix this, given how long it's been already.
However, you should be able to work around the issue by using the input stream, a socket, or environment variables to pass parameters. If you don't have direct control over the executable, you'll have to make a wrapper.
You could use JNA. With version 3.3.0 or later call CreateProcess:
WinBase.PROCESS_INFORMATION.ByReference processInfo =
new WinBase.PROCESS_INFORMATION.ByReference();
WinBase.STARTUPINFO startupInfo = new WinBase.STARTUPINFO();
String command = "C:\\Program Files\\Internet Explorer\\IEXPLORE.EXE " +
"-noframemerging \"C:\\\u65E5\u672C\u8A9E\\Main.html\"";
if (!Kernel32.INSTANCE.CreateProcess(
null, // Application name, not needed if supplied in command line
command, // Command line
null, // Process security attributes
null, // Thread security attributes
true, // Inherit handles
0, // Creation flags
null, // Environment
null, // Directory
startupInfo,
processInfo))
{
throw new IllegalStateException("Error creating process. Last error: " +
Kernel32.INSTANCE.GetLastError());
}
// The CreateProcess documentation indicates that it is very important to
// close the returned handles
Kernel32.INSTANCE.CloseHandle(processInfo.hThread);
Kernel32.INSTANCE.CloseHandle(processInfo.hProcess);
long pid = processInfo.dwProcessId.longValue();
Redirecting output from the child process is a bit harder but not impossible.
I think you can use Apache Commons Exec library or ProcessBuilder to give a try;)