I am using a software in Unix on a computing cluster, for which I need hundreds of input files for a combinative analysis, which wants each file specified with an -f flag, it is like:
software -f file1.ext -f file2.ext -f file3.ext
Simple bash loop doesn't work to let it comprehend each file, such as:
for i in *ext; do software -f ${i}; done
Even simpler way doesn't work, either:
software -f *ext
Specifying the folder where the files are doesn't work, either:
software -f .
Even a cooler bash script doesn't work (actually not really different from the simple loop):
#/bin/bash
for i in $(ls *.ext | rev | cut -c 5- | rev | uniq)
do
software -f ${i}.ext
done
So what I need is a way to make the software recognize all my files in the same input flag by iterating the -f as well I believe. Something like:
for i in *ext; for each -f; do software -f ${i}; done
Any help is much appreciated!
You can iteratively build an array with the appropriate options.
for f in *ext; do
opts+=(-f "$f")
done
software "${opts[#]}"
Related
I prefer bat over pygmentize as bat has better language support and a better all round app in my opinion. However, one thing that I would like is the ability to retain it's syntax highlighted output even through more or grep or other programs. I guess that's because more or other apps do not support colour, but often in Linux, when something seems not possible, I find that there is some smart trick to achieve that thing, so if I pipe bat output to more or grep, is there a way to retain the colour that is part of the bat output?
e.g. bat ~/.bashrc | more
# Get latest bat release and install
bat_releases=https://github.com/sharkdp/bat/releases/
content=$(wget $bat_releases -q -O -)
firstlink=$(grep -oP 'href="/sharkdp/bat/releases/\K[^"]*_amd64\.deb' <<< "$content" | head -1)
DL=$bat_releases$firstlink
ver_and_filename=$(grep -oP 'https://github.com/sharkdp/bat/releases/download/\K[^"]*\.deb' <<< "$DL")
IFS='/' read -ra my_array <<< "$ver_and_filename"
ver=${my_array[0]}
filename=${my_array[1]}
IFS='.' read -ra my_array <<< "$filename"
extension=${my_array[-1]}
extension_with_dot="."$extension
filename_no_extension=${filename%%${extension_with_dot}*}
[ ! -f /tmp/$filename ] && exe wget -P /tmp/ $DL
sudo dpkg -i /tmp/$filename
This question already has answers here:
Run script on multiple files
(3 answers)
Closed 3 years ago.
I'm very new to bash. I have ten text files that I want to edit with the same line of code.
#!/bin/bash
sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g'| grep -o "here.*there" | sed -r '/^.{,100}$/d'
< files 1-10
I know I could use sed -f sed.sh <file1 >file1 but that only works with sed commands and it only works one file at a time?
Do I have to run a loop?
There's some great existing answers on the Unix stack exchange that help deal with your problem. Specifically, from this post, they use a loop to recursively loop through all the files in a particular directory, as follows:
( shopt -s globstar dotglob;
for file in **; do
if [[ -f $file ]] && [[ -w $file ]]; then
sed -i -- 's/foo/bar/g' "$file"
fi
done
)
Note the line, shopt -s globstar dotglob;, which allows us to use globbing patterns in the for loop. We also enclose the code in brackets, to prevent the shopt -s globstar dotglob; line option from becoming a global setting.
If you would like to apply this example to your file, you can just place your files in the current directory, and the code would probably look something like this:
( shopt -s globstar dotglob;
for file in **; do
if [[ -f $file ]] && [[ -w $file ]]; then
sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g' | grep -o "here.*there" | sed -r '/^.{,100}$/d' "$file"
fi
done
)
Note that we have placed a "$file" variable beside each of the seds that you used in your code, this replaces the name of the file for each command.
There is another example given in the code that allows you to pick which files to run on, rather than all the files in a directory, which you can also re-purpose for your code, as given here:
( shopt -s globstar dotglob
sed -i -- 's/foo/bar/g' **baz*
sed -i -- 's/foo/bar/g' **.baz
)
To answer your question of doing a loop on each line, you will need to put a loop for each line inside your for loop, like so:
while read line ; do
: sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g' | grep -o "here.*there" | sed -r '/^.{,100}$/d' "$line”
done
)
Although the for loop can be useful for dealing with files in recursive directories, I would recommend against also using another loop to grab lines, since it muddies your code, and it’s possible there is a better way to do it without parsing line by line.
The linked question is a fairly complete guide to many of the cases you may come across, and is also worth a read if you want to learn more.
Hope that helps!
You could use a for loop.
You could use the tool parallel.
Example
Create a set of test files using a for-loop
mkdir -p /tmp/so58333536
cd /tmp/so58333536
for i in 1.txt 2.txt 3.txt 4.txt 5.txt;do echo "The answer is 41" > $i;done
cat /tmp/so58333536/*
Now correct your mistake using parallel [1].
mkdir /tmp/so58333536.new
ls /tmp/so58333536/* |parallel "sed 's/41/42/' {} > /tmp/so58333536.new/{/}"
cat /tmp/so58333536.new/*
{}:: refers to the current file
{/}:: refers to name of the current file (path is removed)
Reads: List all files in so58333536 and apply the following sed command to each file and write the output to so58333536.new.
[1] Another option is to use sed -i for in-place editing.
Be very carefull with this!! Mistakes can cause serious damages!
# !! Do not use -i option regularly !!
ls /tmp/so58333536/* |parallel "sed -i 's/41/42/'"
I need to use 'last' to search through a list of users who logged into a system, i.e.
last -f /var/log/wtmp <username>
Considering the number of bzipped archive files in that directory, and considering I am on a shared system, I am trying to include an inline bzcat, but nothing seems to work. I have tried the following combinations with no success:
last -f <"$(bzcat /var/log/wtmp-*)"
last -f <$(bzcat /var/log/wtmp-*)
bzcat /var/log/wtmp-* | last -f -
Driving me bonkers. Any input would be great!
last (assuming the Linux version) can't read from a pipe. You'll need to temporarily bunzip2 the files to read them.
tempfile=`mktemp` || exit 1
for wtmp in /var/log/wtmp-*; do
bzcat "$wtmp" > "$tempfile"
last -f "$tempfile"
done
rm -f "$tempfile"
You can only use < I/O redirection on one file at a time.
If anything is going to work, then the last line of your examples is it, but does last recognize - as meaning standard input? (Comments in another answer indicate "No, last does not recognize -". Now you see why it is important to follow all the conventions - it makes life difficult when you don't.) Failing that, you'll have to do it the classic way with a shell loop.
for file in /var/log/wtmp-*
do
last -f <(bzcat "$file")
done
Well, using process substitution like that is pure Bash...the classic way would be more like:
tmp=/tmp/xx.$$ # Or use mktemp
trap "rm -f $tmp; exit 1" 0 1 2 3 13 15
for file in /var/log/wtmp-*
do
bzcat $file > $tmp
last -f $tmp
done
rm -f $tmp
trap 0
I want to create a shellscript that reads files from a .diz file, where information about various source files are stored, that are needed to compile a certain piece of software (imagemagick in this case). i am using Mac OSX Leopard 10.5 for this examples.
Basically i want to have an easy way to maintain these .diz files that hold the information for up-to-date source packages. i would just need to update these .diz files with urls, version information and file checksums.
Example line:
libpng:1.2.42:libpng-1.2.42.tar.bz2?use_mirror=biznetnetworks:http://downloads.sourceforge.net/project/libpng/00-libpng-stable/1.2.42/libpng-1.2.42.tar.bz2?use_mirror=biznetnetworks:9a5cbe9798927fdf528f3186a8840ebe
script part:
while IFS=: read app version file url md5
do
echo "Downloading $app Version: $version"
curl -L -v -O $url 2>> logfile.txt
$calculated_md5=`/sbin/md5 $file | /usr/bin/cut -f 2 -d "="`
echo $calculated_md5
done < "files.diz"
Actually I have more than just one question concerning this.
how to calculate and compare the checksums the best? i wanted to store md5 checksums in the .diz file and compare it with string comparison with "cut"ting out the string
is there a way to tell curl another filename to save to? (in my case the filename gets ugly libpng-1.2.42.tar.bz2?use_mirror=biznetnetworks)
i seem to have issues with the backticks that should direct the output of the piped md5 and cut into the variable $calculated_md5. is the syntax wrong?
Thanks!
The following is a practical one-liner:
curl -s -L <url> | tee <destination-file> |
sha256sum -c <(echo "a748a107dd0c6146e7f8a40f9d0fde29e19b3e8234d2de7e522a1fea15048e70 -") ||
rm -f <destination-file>
wrapping it up in a function taking 3 arguments:
- the url
- the destination
- the sha256
download() {
curl -s -L $1 | tee $2 | sha256sum -c <(echo "$3 -") || rm -f $2
}
while IFS=: read app version file url md5
do
echo "Downloading $app Version: $version"
#use -o for output file. define $outputfile yourself
curl -L -v $url -o $outputfile 2>> logfile.txt
# use $(..) instead of backticks.
calculated_md5=$(/sbin/md5 "$file" | /usr/bin/cut -f 2 -d "=")
# compare md5
case "$calculated_md5" in
"$md5" )
echo "md5 ok"
echo "do something else here";;
esac
done < "files.diz"
My curl has a -o (--output) option to specify an output file. There's also a problem with your assignment to $calculated_md5. It shouldn't have the dollar sign at the front when you assign to it. I don't have /sbin/md5 here so I can't comment on that. What I do have is md5sum. If you have it too, you might consider it as an alternative. In particular, it has a --check option that works from a file listing of md5sums that might be handy for your situation. HTH.
Say I have three files (template_*.txt):
template_x.txt
template_y.txt
template_z.txt
I want to copy them to three new files (foo_*.txt).
foo_x.txt
foo_y.txt
foo_z.txt
Is there some simple way to do that with one command, e.g.
cp --enableAwesomeness template_*.txt foo_*.txt
for f in template_*.txt; do cp $f foo_${f#template_}; done
[01:22 PM] matt#Lunchbox:~/tmp/ba$
ls
template_x.txt template_y.txt template_z.txt
[01:22 PM] matt#Lunchbox:~/tmp/ba$
for i in template_*.txt ; do mv $i foo${i:8}; done
[01:22 PM] matt#Lunchbox:~/tmp/ba$
ls
foo_x.txt foo_y.txt foo_z.txt
My preferred way:
for f in template_*.txt
do
cp $f ${f/template/foo}
done
The "I-don't-remember-the-substitution-syntax" way:
for i in x y z
do
cp template_$i foo_$
done
This should work:
for file in template_*.txt ; do cp $file `echo $file | sed 's/template_\(.*\)/foo_\1/'` ; done
for i in template_*.txt; do cp -v "$i" "`echo $i | sed 's%^template_%foo_%'`"; done
Probably breaks if your filenames have funky characters in them. Remove the '-v' when (if) you get confidence that it works reliably.
The command mmv (available in Debian or Fink or easy to compile yourself) was created precisely for this task. With the plain Bash solution, I always have to look up the documentation about variable expansion. But mmv is much simpler to use, quite close to "awesomeness"! ;-)
Your example would be:
mcp "template_*.txt" "foo_#1.txt"
mmv can handle more complex patterns as well and it has some sanity checks, for example, it will make sure none of the files in the destination set appear in the source set (so you can't accidentally overwrite files).
I don't know of anything in bash or on cp, but there are simple ways to do this sort of thing using (for example) a perl script:
($op = shift) || die "Usage: rename perlexpr [filenames]\n";
for (#ARGV) {
$was = $_;
eval $op;
die $# if $#;
rename($was,$_) unless $was eq $_;
}
Then:
rename s/template/foo/ *.txt
Yet another way to do it:
$ ls template_*.txt | sed -e 's/^template\(.*\)$/cp template\1 foo\1/' | ksh -sx
I've always been impressed with the ImageMagick convert program that does what you expect with image formats:
$ convert rose.jpg rose.png
It has a sister program that allows batch conversions:
$ mogrify -format png *.jpg
Obviously these are limited to image conversions, but they have interesting command line interfaces.