I want to run System Commands via SystemCommandTasklet.Itried this with the sample code below but I get an error.
I think this because of command parameter,But I could not fix it.
I would be very glad if it will help.
Reference Examples ;
Using SystemCommandTasklet for split the large flat file into small files
Trying to split files using SystemCommandTasklet - Execution of system command did not finish within the timeout
Error Detail ;
"CreateProcess error=2, The system cannot find the file specified"
Code Sample ;
#Bean
#StepScope
public SystemCommandTasklet fileSplitterSystemCommandTasklet(#Value("#{jobParameters['file']}") File file) throws Exception {
final String fileSeparator = System.getProperty("file.separator");
String outputDirectory = file.getPath().substring(0, file.getPath().lastIndexOf(fileSeparator)) + fileSeparator + "out" + fileSeparator;
File output = new File(outputDirectory);
if (!output.exists()) {
output.mkdir();
}
final String command = String.format("split -a 5 -l 10000 %s %s",file.getName(),outputDirectory);
var fileSplitterTasklet = new SystemCommandTasklet();
fileSplitterTasklet.setCommand(command);
fileSplitterTasklet.setTimeout(60000L);
fileSplitterTasklet.setWorkingDirectory(outputDirectory);
fileSplitterTasklet.setTaskExecutor(new SimpleAsyncTaskExecutor());
fileSplitterTasklet.setSystemProcessExitCodeMapper(touchCodeMapper());
fileSplitterTasklet.afterPropertiesSet();
fileSplitterTasklet.setInterruptOnCancel(true);
fileSplitterTasklet.setEnvironmentParams(new String[]{
"JAVA_HOME=/java",
"BATCH_HOME=/Users/batch"});
return fileSplitterTasklet;
}
You need to use file.getAbsolutePath() instead of file.getPath().
Also, you are using file.getName() in the command:
final String command = String.format("split -a 5 -l 10000 %s %s",file.getName(),outputDirectory);
You should pass the absolute path of the file or make sure to set
the working directory correctly so that the split command is executed
in the same directory as the file.
Related
I invoke the external command within my Java app with Runtime.getRuntime().exec() or ProcessBuilder. Works fine but Veracode complains on it with CWE-78. I'm trying to use ESAPI wrapper to sanitise the input and path the check.
The arfifact is the latest
<dependency>
<groupId>org.owasp.esapi</groupId>
<artifactId>esapi</artifactId>
<version>2.2.3.1</version>
</dependency>
ESAPI.properties are
ESAPI.Logger=org.owasp.esapi.logging.slf4j.Slf4JLogFactory
Logger.LogEncodingRequired=false
Logger.UserInfo=false
Logger.ClientInfo=false
Logger.LogApplicationName=false
Logger.ApplicationName=my-app
Logger.LogServerIP=false
IntrusionDetector.Disable=true
Executor.ApprovedExecutables=/usr/bin/less
The code is:
#Test
void esapiTest() throws ExecutorException {
Executor executor = DefaultExecutor.getInstance();
ExecuteResult executeResult = executor.executeSystemCommand(
new File("/usr/bin/less"),
new ArrayList<>(Collections.singletonList("/etc/hosts"))
);
System.out.println("out = " + executeResult.getOutput());
System.out.println("err = " + executeResult.getErrors());
}
The output is
out =
err = \/etc\/hosts: No such file or directory
As far as I got the issue is that ESAPI's UnixCodec sanitises all non-alpha character with the backslash. This is fine for the shell i.e.
/usr/bin/less \/etc\/hosts
but not for the ProcessBuilder that is under the hood.
What am I doing wrong? How to invoke the command?
I think your main "problem" is misunderstanding that the ESAPI Codecs that are used with the DefaultExecutor class are assuming that any "OS command injection" is being interpreted via a "command line interpreter", i.e., a "shell". You are not invoking a shell here. If you were, the shell would remove the (in this case) backslash escaped argument from your path for "/etc/hosts". So if this were written as (say) the command:
/bin/sh -c /usr/bin/less /etc/hosts
it would [sort of] work (except if you tried running it over HTTP, the input to 'less' my be hosed; but "/bin/cat" ought to work fine).
Instead, try writing your test something like this:
#Test
void esapiTest() throws ExecutorException {
Executor executor = ESAPI.executor();
File binSh = new File("/bin/sh").getCanonicalFile();
List params = new ArrayList();
params.add("-c");
//Use '/bin/cat' because 'less' may be troublesome
params.add("\"" + "/bin/cat" + "/etc/hosts" + "\"");
ExecuteResult executeResult = executor.executeSystemCommand(binSh, params);
System.out.println("out = " + executeResult.getOutput());
System.out.println("err = " + executeResult.getErrors());
}
Note 1: If you I'm not sure how user-friendly ProcessBuilder is with commands that eventually try to do ioctl system calls to set the tty device in 'raw' mode, like commands such as "vim" or "less", which is why I changed your "/usr/bin/less" to "/bin/cat". YMMV.
Note 2: In your ESAPI.properties file, you'd have to make sure that the property 'Executor.ApprovedExecutables' is set to whatever the canonical name of "/bin/sh" is on your system. E.g., on my system, "/bin/sh" is a symbolic link to "/bin/dash", so you would have to include something like
Executor.ApprovedExecutables=/bin/bash,/bin/dash
(or at least "/bin/dash") should work.
When using Google Drive API, I'm having this downloadMetadataFile() here to handle file:
public void downloadMetadataFile(String fileId, String folderStorePath, String fileName) throws IOException, GeneralSecurityException, GoogleException {
String path = folderStorePath + "/" + fileName
java.io.File file = new java.io.File(path);
try (FileOutputStream fileOutputStream = new FileOutputStream(file)) {
Drive drive = createDrive();
drive.files().get(fileId)
.executeMediaAndDownloadTo(fileOutputStream);
}
}
When using above method with folder exists (izakayaTemplate + 居酒屋):
When path=/reports/template/izakayaTemplate/template3.png, the method working file and download template3.png successful from Google Drive
When path=/reports/template/居酒屋/template3.png, the method throw a FileNotFoundException at line try (FileOutputStream fileOutputStream = new FileOutputStream(file))
Can somebody please explain for me about this behavior?
Note:
I'm using SpringBoot 2.5, Java 8, Drive API v3
I'm running this project on Amazon linux 1 as a service by DaemonTool.
In the run config file, I have set
-Dfile.encoding=UTF-8
-Dsun.jnu.encoding=UTF-8 -Dfile.encoding=UTF-8 \
Update 1:
After debug for a while, I found out that the CanonicalPath is wrong for the new file I create but I don't know why it happen.
getPath: /reports/template/居酒屋/template3.png
getAbsolutePath: /reports/template/居酒屋/template3.png
getCanonicalPath: /reports/template/???/template3.png
After searching, I have found the solution for this problem:
Solution: Add export LANG=ja_JP.UTF-8 to the file run
Explanation: canonicalPath is the path file system considers the canonical means to reference the file system object to which it points. So in order for the system to get the canonicalPath to be correct, the environment must have set up correct language environment like in this document: https://docs.oracle.com/cd/E23824_01/html/E26033/glset.html. In my question, the correct language environment is ja_JP.UTF-8
I have developed a code that runs a map reduce job to read files from FTP server and write it into HDFS. Into HDFS it writes the file from FTP into the specified output directory naming it as part-0000. In case I have multiple files on the FTP server I get all of them written to that one part-0000 file in HDFS.
To avoid this I plan to pass the name of the file as key along with the data as value . Thus the reducer gets the data into an output file with the key as the name of the file.
I understand that I have to use an outputformat that extends MultipleTextOutputFormat. I have written it as follows
static class MultiFileOutput extends MultipleTextOutputFormat<Text, Text> {
protected String generateFileNameForKeyValue(Text key, Text value,String name) {
System.out.println("key is :"+ key.toString());
System.out.println("value is :"+ value.toString());
System.out.println("name is :"+ name.toString());
return key.toString();
}
But I fail to pass the name of the input file being processed . How do I get the name of the input file ?
map.input.file
and
FileSystem fs = file.getFileSystem(conf);
String fileName=fs.getName();
do not return the name of the input file.
Any pointers ?
You can get the input file path through context.
FileSplit fileSplit = (FileSplit) context.getInputSplit();
String inputFilePath = fileSplit.getPath().toString();
This will give the full path. If you want just the filename you can do this :
String inputFileName = fileSplit.getPath().getName();
HTH
I used FileStatus object in the following code as my customised input format would not split the input file. It worked fine for me ..
FileSystem fs = file.getFileSystem(conf);
FileStatus status= fs.getFileStatus(file);
String fileName=status.getPath().toString();
If I have a java jar or class file which is launched by the user (assuming java path is set in environment variables), so how can i from within the code, figure out absolute path of java.exe/javaw.exe from which this file is being launched.
Like on ubuntu we can run: % which java and it shows the path.
However on windows, if i check System.getenv() it may happen that there are multiple path's found e.g for old or new version. If through cmd line, I run java -version it does not show the path.
Can you tell me either through pure java or command line on windows how is it possible to find out the location of javaw.exe?
String javaHome = System.getProperty("java.home");
Can you tell me either through pure Java ... on windows how is it possible to find out the location of javaw.exe?
E.G.
import java.io.File;
class JavawLocation {
public static void main(String[] args) {
String javaHome = System.getProperty("java.home");
File f = new File(javaHome);
f = new File(f, "bin");
f = new File(f, "javaw.exe");
System.out.println(f + " exists: " + f.exists());
}
}
Output
C:\Program Files (x86)\Java\jdk1.6.0_29\jre\bin\javaw.exe exists: true
Press any key to continue . . .
And yes, I am confident that will work in a JRE.
On Windows, the java.library.path System Property begins with the path to the bin directory containing whichever java.exe was used to run your jar file.
This makes sense, because on Windows the first place any executable looks for DLL files is the directory containing the executable itself. So naturally, when the JVM runs, the first place it looks for DLLs is the directory containing java.exe.
You can acquire the path to java.exe as follows:
final String javaLibraryPath = System.getProperty("java.library.path");
final File javaExeFile = new File(
javaLibraryPath.substring(0, javaLibraryPath.indexOf(';')) + "\\java.exe"
);
final String javaExePath =
javaExeFile.exists() ? javaExeFile.getAbsolutePath() : "java";
This code is Windows-specific - I hard-coded the path separator (;) and the file separator (). I also put in a fallback to just "java" in case the library path trick somehow doesn't work.
I have tested this with Java 6 and 7 on Windows 7. I tried a 32-bit and 64-bit version of Java.
Here's a slightly more generalised solution that I came up with. Maybe useful:
private static String javaExe()
{
final String JAVA_HOME = System.getProperty("java.home");
final File BIN = new File(JAVA_HOME, "bin");
File exe = new File(BIN, "java");
if (!exe.exists())
{
// We might be on Windows, which needs an exe extension
exe = new File(BIN, "java.exe");
}
if (exe.exists())
{
return exe.getAbsolutePath();
}
try
{
// Just try invoking java from the system path; this of course
// assumes "java[.exe]" is /actually/ Java
final String NAKED_JAVA = "java";
new ProcessBuilder(NAKED_JAVA).start();
return NAKED_JAVA;
}
catch (IOException e)
{
return null;
}
}
an issue with using "System.getProperty("java.home");", is that it is not always the java exe that the jar is running on, if you want to get that, you can use "System.getProperty("sun.boot.library.path");", from there you can find "java", "java.exe", "javaw", or "javaw.exe"... However there is still an issue with this, java will run just fine if the executable has been renamed, and the actual java executable's structure changes from different JRE's/JDKS's, so there is not much way to find the java exe if it has been renamed. unless someone else has a method ofc, in which case, can you share? :)
(Also, I have seen some people suggest using the first index of System.getProperty("java.library.path");, note, this might not work if the user/launcher has manually set the library path, something which is not too uncommon)
Compilation of All above methods
static String getJavaPath(){
String tmp1 = System.getProperty("java.home") + "\\bin\\java.exe";
String tmp2 = System.getProperty("sun.boot.library.path") + "\\java.exe";
String tmp3 = System.getProperty("java.library.path")+ "\\java.exe";
if(new File(tmp1).exists()) {
return tmp1;
}else if(new File(tmp2).exists()){
return tmp2;
}else if(new File(tmp3).exists()) {
return tmp3;
}else{
String[] paths = System.getenv("PATH").split(";");
for(String path:paths){
if(new File(path + "\\java.exe").exists()){
return path + "\\java.exe";
}
}
}
return "";
}
i have a new problem in image magick that look strange ..
i'm using mac osx snow leopard and i've installed image magick on it and it's working fine on command ..
but when i call it from the grails class like the following snippet it gives me
"Cannot run program "convert": error=2, No such file or directory"
the code is :-
public static boolean resizeImage(String srcPath, String destPath,String size) {
ArrayList<String> command = new ArrayList<String>(10);
command.add("convert");
command.add("-geometry");
command.add(size);
command.add("-quality");
command.add("100" );
command.add(srcPath);
command.add(destPath);
System.out.println(command);
return exec((String[])command.toArray(new String[1]));
}
private static boolean exec(String[] command) {
Process proc;
try {
//System.out.println("Trying to execute command " + Arrays.asList(command));
proc = Runtime.getRuntime().exec(command);
} catch (IOException e) {
System.out.println("IOException while trying to execute " );
for(int i =0 ; i<command.length; i++) {
System.out.println(command[i]);
}
return false;
}
//System.out.println("Got process object, waiting to return.");
int exitStatus;
while (true) {
try {
exitStatus = proc.waitFor();
break;
} catch (java.lang.InterruptedException e) {
System.out.println("Interrupted: Ignoring and waiting");
}
}
if (exitStatus != 0) {
System.out.println("Error executing command: " + exitStatus);
}
return (exitStatus == 0);
}
i've tried normal command like ls and it's ok so the problem is that grails can't find convert command itself.. is it a os problem or something?
(see lower for the answer)
I have run into the same problem. The issue appears to be something with Mac OS X specifically, as we have several Linux instances running without error. The error looks similar to the following:
java.io.IOException: Cannot run program "/usr/bin/ImageMagick-6.7.3/bin/convert /a/temp/in/tmpPic3143119797006817740.png /a/temp/out/100000726.png": error=2, No such file or directory
All the files are there, and in chmod 777 directories - and as you pointed out, running the exact command from the shell works fine.
My theory at this point is that imagemgick can not load some sort of library itself, and the "no such file" is in reference to an dylib or something along those lines.
I have tried setting LD_LIBRARY_PATH and a few others to no avail.
I finally got this working. Here is how I have it setup. I hope this helps.
The crux of the fix, for me, was I wrapped the 'convert' into a shell script, set a bunch of environment variables, and then call that shell script instead of convert directly:
(convertWrapper.sh)
export MAGICK_HOME=/usr/local/ImageMagick-6.7.5
export MAGICK_CONFIGURE_PATH=${MAGICK_HOME}/etc/ImageMagick:${MAGICK_HOME}/share/doc/ImageMagick/www/source
export PATH=${PATH}:${MAGICK_HOME}/bin
export LD_LIBRARY_PATH=${MAGICK_HOME}/lib:${LD_LIBRARY_PATH}
export DYLD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}:${MAGICK_HOME}/lib
export MAGICK_TMPDIR=/private/tmp
echo "$#" >> /private/tmp/m.log 2>&1
/usr/local/ImageMagick-6.7.5/bin/convert -verbose "$#" >> /private/tmp/m.log 2>&1
(convertWrapper.sh)
Additionally, the convert call was doing some rather complicated stuff, so I added the parameter '-respect-parenthesis' (which may or may not have had an effect).
I am not sure how much of the environment variable setting is needed as I was stabbing in the dark for a while, but since this is only for my development box...
You need to work out what your PATH is set to when you run a command from Java. It must be different to the one you have when running from the terminal.
Are you running Grails (via Tomcat?) as a different user? It might have a different path to your normal user.
you might want to try one of the Image Plugins that are part of the grails ecosystem
http://www.grails.org/ImageTools+plugin
the grails path when the app is running in the server is probably different from running java from the command line
I do so:
Put "convert" file to /usr/bin
Then add to Config.groovy:
gk {
imageMagickPath = "/usr/bin/convert"
}
Then in my ImageService.groovy:
import org.springframework.web.context.request.RequestContextHolder as RCH
[..]
def grailsApplication = RCH.requestAttributes.servletContext.grailsApplication
def imPath = grailsApplication.config.gk.imageMagickPath
def command = imPath + " some_properties"
def proc = Runtime.getRuntime().exec(command)
So this way you get command like: /usr/bin/convert some_properties
And it works, but don't forget to put file "convert" to you location and use it with this location.