I'm trying to make self extracting file using the following Ant tasks:
...
<fixcrlf file="${src.dir}/scripts/install.sh" eol="unix"/>
<concat destfile="${build.dir}/my_program.exe" binary="yes">
<fileset file="${src.dir}/scripts/install.sh" />
<fileset file="${build.dir}/program.tar.gz" />
</concat>
.. and my_program.exe looks like this:
#!/bin/bash
begin=`head -30 $0 | grep -n ^START | cut -d ':' -f -1` # find line number of the marker
start=$(($begin+1)) # beginning of the binary archive which will be extracted
echo $start
...
START
#binary file starts
When I run my_program.exe I get the following error:
./my_program.exe: line 4: Binary file (standard input) matches: syntax error in expression (error token is "file (standard input) matches")
tail: +: invalid number of lines
When I run install.sh separately it finds line number just fine.
My guess is that something wrong with Ant task. Do I miss some properties that will fix it?
I guess you should use the -a (resp. --text, meaning to process a binary file as if it were text) option of grep. Otherwise grep will only output "Binary file matches".
So probably the line 4 should be:
begin=`head -30 $0 | grep -na ^START | cut -d ':' -f -1` # find line number of the marker
Related
Update:
I have tried markp-fuso's answer and it worked like a charm
I'm starting to get frustrated here as I'm not a daily user of bash/sed and the like.
Starting point:
I have many subfolders with many source files (.c,.cpp,.cxx).
These source files are referenced for compilation in project files (.vcxproj).
What I want to do:
I want to find all source files that contain the string #import. I then want to find all project files that reference those source files.
I then want to edit all occurences of these references inplace within these project files
e.g. <Include="folder/file.cpp"/> -> <Include="folder/file.cpp" Attribute="Value"/>
What I have tried:
egrep -lir --include=*.{c,cpp,cxx} "(#import)" ./e3 | xargs -L 1 basename | egrep -ir --include=*.vcxproj -f - ./e3 | sed 's/:/ /g'
which produces a list like that:
./src/base/base.vcxproj <ClCompile Include="Folder1\Folder1File1.cpp" />
./src/mod/mod.vcxproj <ClCompile Include="Folder2\Folder2File1.cpp" />
./src/ext/ext.vcxproj <ClCompile Include="Folder3\Folder3File1.cpp" />
So I then tried
egrep -lir --include=*.{c,cpp,cxx} "(#import)" ./e3 | xargs -L 1 basename | egrep -ir --include=*.vcxproj -f - ./e3 | sed 's/:/ /g' | awk '{ sed -iE 's/($2,$3)/\1 Attribute="Value"/g' }'
which errors out with
bash: syntax error near unexpected token `('
I have tried a solution with a shell script, which didn't work either and I don't know if and how to solve above error message. I'm open to any solution as long as it's running within bash, can even be more gross than what I came up with.
Setup:
mkdir -p src/{base,mod,ext}
echo 'some stuff on this line
<ClCompile Include="Folder1\Folder1File1.cpp" />
some more stuff on this line' > src/base/base.vcxproj
echo 'some stuff on this line
<ClCompile Include="Folder2\Folder2File1.cpp" />
some more stuff on this line' > src/mod/mod.vcxproj
echo 'some stuff on this line
<ClCompile Include="Folder3\Folder3File1.cpp" />
some more stuff on this line' > src/ext/ext.vcxproj
For the sake of getting something working in my environment I've placed the intermediate data in a local file:
$ cat proj.dat
./src/base/base.vcxproj <ClCompile Include="Folder1\Folder1File1.cpp" />
./src/mod/mod.vcxproj <ClCompile Include="Folder2\Folder2File1.cpp" />
./src/ext/ext.vcxproj <ClCompile Include="Folder3\Folder3File1.cpp" />
One idea using parameter substitution:
while read -r fname oldstring # 2nd-Nth space-delimited fields go into the single variable "oldstring"
do
oldstring="${oldstring//\\/\\\\}" # escape literal backslashes
newstring="${oldstring//\/>/ Attribute=\"Value\"\/>}" # replace /> with Attribute="Value"/>
echo "##################### ${fname}"
sed "s|${oldstring}|${newstring}|g" "${fname}"
done < proj.dat
NOTES:
sed replace applied to all occurences in a file
if additional datasets cause sed to abort with errors it may be necessary to add additional parameter expansions to escape other problematic characters
added a space on the front of the Attribute string since the textual description suggested a space may not exist before the /> (eg, ...file.cpp"/>)
OP should be able pipe the current egrep | xargs | egrep | sed to this while loop (replacing done < proj.dat with done)
once OP is satisfied with the results the -i flag can be added to the sed call to perform an inplace update of ${fname}
Generates:
##################### ./src/base/base.vcxproj
some stuff on this line
<ClCompile Include="Folder1\Folder1File1.cpp" Attribute="Value"/>
some more stuff on this line
##################### ./src/mod/mod.vcxproj
some stuff on this line
<ClCompile Include="Folder2\Folder2File1.cpp" Attribute="Value"/>
some more stuff on this line
##################### ./src/ext/ext.vcxproj
some stuff on this line
<ClCompile Include="Folder3\Folder3File1.cpp" Attribute="Value"/>
some more stuff on this line
This currently works ok, want to add the following capabilities:
pipe search items from mygrepitemslist.txt (line by line) instead of my explicitly stating error1 and error2 (bonus if spaces can be include in search)
in other words want something read file mygrepitemslist.txt
and pipe to grep in this code example
instead of in the code below:
grep "error1\|error2"
mygrepitemslist.txt has:
error1
errors
error3 with space
error4 with multiple spaces
Would like to use what I have mainly because I use it for other things and it's familiar, just stuck on how to feed it grep strings from a file then outputting the match with filename
tail -Fn0 /var/log/*.log | \
while read line ; do
echo "$line" | \
grep "error1\|error2" #pipe mygrepitemslist.txt linebyline here?
if [ $? = 0 ]
then
echo "$line" #how to show error + filename here?
fi
done
The overall results is:
want to tail/follow multiple files
search for strings read from file called mygrepitemslist.txteach line is search term
output is:
error search with matching file name
you can use -f option for specifying file with patterns
tail -Fn0 /var/log/*.log | grep -of mygrepitemslist.txt
I have FramesList.txt, the file holds 5 .xml filenames. There isn't a path generated because the list is being created by an xml ls > FramesList.txt command.
I want to run an xmlstarlet command on each .xml file listed.
FramesList.txt
FR0391W.xml
FR0921S.xml
FR0193A.xml
FR0192B.xml
FR0132E.xml
How would that look? Is it a loop?
You can use xargs:
xargs < FramesList.txt -n 1 xmlstartet
# ^ ^ ^
# | | Command to run
# | Max number of arguments (xmlstartet will be called for
# | each line in input file)
# Input file
while read -r line
do
xmlstarlet "$line"
done <FramesList.txt
I think most XML Starlet commands accept more than one file as a parameter, and since your input list is small, you could also write
xml COMMAND $(<FramesList.txt)
I have a file containing valid xmls (one per line) and I want to execute a utility (xpath) on each line one by one.
I tried xargs but that seems doesn't seem to have an option to pass the line as stdin :-
% cat <xmls-file> | xargs -p -t -L1 xpath -p "//Path/to/node"
Cannot open file '//Path/to/node' at /System/Library/Perl/Extras/5.12/XML/XPath.pm line 53.
I also tried parallel --spreadstdin but that doesn't seem to work either :-
% cat <xmls-file> | parallel --spreadstdin xpath -p "//Path/to/node"
junk after document element at line 2, column 0, byte 1607
If you want every line of a file to be split off and made stdin for a utility
you could use a for loop in bash shell:
cat xmls-file | while read line
do ( echo $f > /tmp/input$$;
xpath -p "//Path/to/node" </tmp/input$$
rm -f /tmp/input$$
);
done
The $$ appends the process id number, creating a unique name
I assume xmls-file contains, on each line, what you want iterated into $f and that you want this as stdin for a command line, not as a parameter to the command.
On the other hand, your specification may be incorrect and maybe instead you need each line
to be part of a command. In that case, delete the echo and rm lines, and change the xpath command to include $f wherever the line from the file is needed.
I've not done much XML so the do command may need to be edited.
You are very close with the GNU Parallel version; only -n1 missing:
cat <xmls-file> | parallel -n1 --spreadstdin xpath -p "//Path/to/node"
I am trying to write a bash script that goes through a file line by line (ignoring the header), extracts a file name from the beginning of each line, and then finds a file by this name in one directory and moves it to another directory. I will be processing hundreds of these files in a loop and moving over a million individual files. A sample of the file is:
ImageFileName Left_Edge_Longitude Right_Edge_Longitude Top_Edge_Latitude Bottom_Edge_Latitude
21088_82092.jpg: -122.08007812500000 -122.07733154296875 41.33763821961143 41.33557596965434
21088_82093.jpg: -122.08007812500000 -122.07733154296875 41.33970040427444 41.33763821961143
21088_82094.jpg: -122.08007812500000 -122.07733154296875 41.34176252364274 41.33970040427444
I would like to ignore the first line and then grab 21088_82092.jpg as a variable. File names may not always be the same length, but they will always have the format digits_digits.jpg
Any help for an efficient approach is much appreciated.
This should get you started:
$ tail -n +2 input | cut -f 1 -d: | while read file; do test -f $dir/$file && mv -v $dir/$file $destination; done
You can construct a script that will do something like this, then simply run the script. The following command will give you a script which will copy the files from one place to another, but you can make the script generation more complex simply by changing the awk output:
pax:~$ cat qq.in
ImageFileName Left_Edge_Longitude Right_Edge_Longitude
21088_82092.jpg: -122.08007812500000 -122.07733154296875
21088_82093.jpg: -122.08007812500000 -122.07733154296875
21088_82094.jpg: -122.08007812500000 -122.07733154296875
pax:~$ awk -F: '/^[0-9]+_[0-9]+.jpg:/ {
printf "cp /srcdir/%s /dstdir\n",$1
} {}' qq.in
cp /srcdir/21088_82092.jpg /dstdir
cp /srcdir/21088_82093.jpg /dstdir
cp /srcdir/21088_82094.jpg /dstdir
You capture the output of that script (the last three lines) to another file then that file is your script for doing the actual copies.