I'm trying to change the filename from
prod.test.PVSREGPLUS20170915-6777.DAT.gpg
to
PVSREGPLUS20170915-0003.DAT.gpg
I used this
DTE=$(date +%I);ls prod.test* |cut -f 3,4,5 -d .|sed "s/\-/-00$DTE/" |cut -c 1-23,28-35
My problem is I need this command in a shell script
"#! /bin/bash
DTE=$(date +%I)
newfile=$(ls prod.test* |cut -f 3,4,5 -d .|sed "s/-*./$DTE/"|cut -c 1-23,28-35
The sed can't do expansion, would awk be able to do this? Any help would be greatly appreciated. Thanks
The simplest way to do this is with a for-loop over a glob pattern, then use paramater expansion to remove the prefix
prefix="prod.test."
hour=$(date '+%I')
for f in "$prefix"*; do
new=$( echo "${f#$prefix}" | sed 's/-[[:digit:]]\+/-00'"$hour"'/' )
echo mv "$f" "$new"
done
We really don't need sed: extended patterns and more parameter expansion
shopt -s extglob
for f in "$prefix"*; do
new=${f#$prefix}
new=${new/-+([0-9])/-00$hour}
echo mv "$f" "$new"
done
Remove "echo" if it looks good.
Or, with the perl rename as suggested in the comments:
rename -v -n 's/prod\.test\.//; use Time::Piece; s{-\d+}{"-00" . (localtime)->strftime("%I") }e' prod.test.*
Remove "-n" if it looks good.
Related
This question already has answers here:
Run script on multiple files
(3 answers)
Closed 3 years ago.
I'm very new to bash. I have ten text files that I want to edit with the same line of code.
#!/bin/bash
sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g'| grep -o "here.*there" | sed -r '/^.{,100}$/d'
< files 1-10
I know I could use sed -f sed.sh <file1 >file1 but that only works with sed commands and it only works one file at a time?
Do I have to run a loop?
There's some great existing answers on the Unix stack exchange that help deal with your problem. Specifically, from this post, they use a loop to recursively loop through all the files in a particular directory, as follows:
( shopt -s globstar dotglob;
for file in **; do
if [[ -f $file ]] && [[ -w $file ]]; then
sed -i -- 's/foo/bar/g' "$file"
fi
done
)
Note the line, shopt -s globstar dotglob;, which allows us to use globbing patterns in the for loop. We also enclose the code in brackets, to prevent the shopt -s globstar dotglob; line option from becoming a global setting.
If you would like to apply this example to your file, you can just place your files in the current directory, and the code would probably look something like this:
( shopt -s globstar dotglob;
for file in **; do
if [[ -f $file ]] && [[ -w $file ]]; then
sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g' | grep -o "here.*there" | sed -r '/^.{,100}$/d' "$file"
fi
done
)
Note that we have placed a "$file" variable beside each of the seds that you used in your code, this replaces the name of the file for each command.
There is another example given in the code that allows you to pick which files to run on, rather than all the files in a directory, which you can also re-purpose for your code, as given here:
( shopt -s globstar dotglob
sed -i -- 's/foo/bar/g' **baz*
sed -i -- 's/foo/bar/g' **.baz
)
To answer your question of doing a loop on each line, you will need to put a loop for each line inside your for loop, like so:
while read line ; do
: sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g' | grep -o "here.*there" | sed -r '/^.{,100}$/d' "$line”
done
)
Although the for loop can be useful for dealing with files in recursive directories, I would recommend against also using another loop to grab lines, since it muddies your code, and it’s possible there is a better way to do it without parsing line by line.
The linked question is a fairly complete guide to many of the cases you may come across, and is also worth a read if you want to learn more.
Hope that helps!
You could use a for loop.
You could use the tool parallel.
Example
Create a set of test files using a for-loop
mkdir -p /tmp/so58333536
cd /tmp/so58333536
for i in 1.txt 2.txt 3.txt 4.txt 5.txt;do echo "The answer is 41" > $i;done
cat /tmp/so58333536/*
Now correct your mistake using parallel [1].
mkdir /tmp/so58333536.new
ls /tmp/so58333536/* |parallel "sed 's/41/42/' {} > /tmp/so58333536.new/{/}"
cat /tmp/so58333536.new/*
{}:: refers to the current file
{/}:: refers to name of the current file (path is removed)
Reads: List all files in so58333536 and apply the following sed command to each file and write the output to so58333536.new.
[1] Another option is to use sed -i for in-place editing.
Be very carefull with this!! Mistakes can cause serious damages!
# !! Do not use -i option regularly !!
ls /tmp/so58333536/* |parallel "sed -i 's/41/42/'"
I have a variable f in a bash script
f=/path/to/a/file.jpg
I'm using the variable as an input argument to a program that requires and input and an output path.
For example the program's usage would look like this
./myprogram -i inputFilePath -o outputFilePath
using my variable, I'm trying to maintain the same basename, change the extension, and put the output file into a sub directory. For example
./myprogram -i /path/to/a/file.jpg -o /path/to/a/new/file.tiff
I'm trying to do that by doing this
./myprogram -i "$f" -o "${f%.jpg}.tiff"
of course this keeps the basename, changes the extension, but doesn't put the file into the new subdirectory.
How can I modify f to to change /path/to/a/file.jpg into /path/to/a/new/file.tiff?
Actually you can do this in several ways:
Using sed as pointed out by #anubhava
Using dirname and basename:
./myprogram -i "$f" -o "$(dirname -- "$f")/new/$(basename -- "$f" .jpg).tiff"
Using only Bash:
./myprogram -i "$f" -o "${f%/*}/new/$(b=${f##*/}; echo -n ${b%.jpg}.tiff)"
Note that unlike the second solution (using dirname/basename) that is more robust, the third solution (in pure Bash) won't work if "$f" does not contain any slash:
$ dirname "file.jpg"
.
$ f="file.jpg"; echo "${f%/*}"
file.jpg
You may use this sed:
s='/path/to/a/file.jpg'
sed -E 's~(.*/)([^.]+)\.jpg$~\1new/\2.tiff~' <<< "$s"
/path/to/a/new/file.tiff
If you're on a system that supports the basename and dirnamecommands you could use a simple wrapper function eg:
$ type newSubDir
newSubDir is a function
newSubDir ()
{
oldPath=$(dirname "${1}");
fileName=$(basename "${1}");
newPath="${oldPath}/${2}/${fileName}";
echo "${newPath}"
}
$ newSubDir /path/to/a/file.jpg new
/path/to/a/new/file.jpg
If your system doesn't have those, you can accomplish the same thing using string manipulation:
$ file="/path/to/a/file.jpg"
$ echo "${file%/*}"
/path/to/a
$ echo "${file##*/}"
file.jpg
Some days ago I started a little bash script that should sum up the number of pages and file size of all PDF's in a folder. It's working quite well now but there's still one thing I don't understand.
Why is sed always failing if shopt -s nullglob is set? Does somebody know why this happens?
I'm working with GNU Bash 4.3 and sed 4.2.2 in Ubuntu 14.04.
set -u
set -e
folder=$1
overallfilesize=0
overallpages=0
numberoffiles=0
#If glob fails nothing should be returned
shopt -s nullglob
for file in $folder/*.pdf
do
# Disable empty string if glob fails
# (Necessary because otherwise sed fails ?:|)
#shopt -u nullglob
# This command is allowed to fail
set +e
pdfinfo="$(pdfinfo "$file" 2> /dev/null)"
ret=$?
set -e
if [[ $ret -eq 0 ]]
then
#Remove every non digit in the result
sedstring='s/[^0-9]//g'
filesize=$(echo -e "$pdfinfo" | grep -m 1 "File size:" | sed $sedstring)
pages=$(echo -e "$pdfinfo" | grep -m 1 "Pages:" | sed $sedstring)
overallfilesize=$(($overallfilesize + $filesize))
overallpages=$(($overallpages+$pages))
numberoffiles=$(($numberoffiles+1))
fi
done
echo -e "Processed files: $numberoffiles"
echo -e "Pagesum: $overallpages"
echo -e "Filesizesum [Bytes]: $overallfilesize"
Here's a simpler test case for reproducing your problem:
#!/bin/bash
shopt -s nullglob
pattern='s/[^0-9]//g'
sed $pattern <<< foo42
Expected output:
42
Actual output:
Usage: sed [OPTION]... {script-only-if-no-other-script} [input-file]...
(sed usage follows)
This happens because s/[^0-9]//g is a valid glob (matching a dir structure like like s/c/g), and you asked bash to interpret it. Since you don't have a matching file, nullglob kicks in and removes the pattern entirely.
Double quoting prevents word splitting and glob interpretation, which is almost always what you want:
#!/bin/bash
shopt -s nullglob
pattern='s/[^0-9]//g'
sed "$pattern" <<< foo42
This produces the expected output.
You should always double quote all your variable references, unless you have a specific reason not to.
I have a load of mysqldumps like this:
dump.data.YYYYMMDD.sql
Any ideas how I can run the latest one from bash?
Thanks in advance.
Try this:
#!/bin/bash
LATEST=$(ls -1t dump.data.*.sql | head -n 1)
echo $LATEST
Note that character in ls -1t before the 't' is the digit '1' not the letter 'l'.,
You can do this:
#!/bin/bash
shopt -s nullglob ## Make no expansion if no file is found from pattern.
if read -r LATEST < <(printf "%s\n" dump.data.*.sql | sort -rn); then
echo "Processing $LATEST."
(do something with $LATEST)
fi
Hi i am having a really strange problem.
I am trying to run the following scrip as root:
loc=$(transmission-remote 192.168.1.74:9091 --auth=user:password -t$TORRENTID -i |sed -e '1d;$d;s/^ *//'|grep Location: |cut -s -d : -f2)
cd "$loc"
But get the error messege :
cd: can't cd to /leprechaun_tv-shows/download/
But i works if i create a new string:
hello= /leprechaun_tv-shows/download/
cd $hello
what am i doing wrong?
Why are you quoting "$loc" ?
If $loc has a leading space, then quoting the variable expansion will pass the directory with that leading space as an argument.
e.g. cding to root in this fashion gives me:
$ cd " /"
cd: no such file or directory: /
Just remove the quoting. e.g.
cd $loc
Alternatively to Brian's solution, if you still need $loc to be quoted in cd "$loc" because the path could contain any spaces, then just use sed 's/^ *//g' to trim any leading spaces in $loc, like this:
loc=$(transmission-remote 192.168.1.74:9091 --auth=user:password -t$TORRENTID -i |sed -e '1d;$d;s/^ *//'|grep Location: |cut -s -d : -f2 | sed 's/^ *//g')
cd "$loc"
Since it appears you have some initial whitespace in $loc, you can trim it with bash's parameter expansion:
loc=$(transmission-remote 192.168.1.74:9091 --auth=user:password -t$TORRENTID -i |sed -e '1d;$d;s/^ *//'|grep Location: |cut -s -d : -f2)
cd "${loc%% }"