I have a load of mysqldumps like this:
dump.data.YYYYMMDD.sql
Any ideas how I can run the latest one from bash?
Thanks in advance.
Try this:
#!/bin/bash
LATEST=$(ls -1t dump.data.*.sql | head -n 1)
echo $LATEST
Note that character in ls -1t before the 't' is the digit '1' not the letter 'l'.,
You can do this:
#!/bin/bash
shopt -s nullglob ## Make no expansion if no file is found from pattern.
if read -r LATEST < <(printf "%s\n" dump.data.*.sql | sort -rn); then
echo "Processing $LATEST."
(do something with $LATEST)
fi
Related
This question already has answers here:
Run script on multiple files
(3 answers)
Closed 3 years ago.
I'm very new to bash. I have ten text files that I want to edit with the same line of code.
#!/bin/bash
sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g'| grep -o "here.*there" | sed -r '/^.{,100}$/d'
< files 1-10
I know I could use sed -f sed.sh <file1 >file1 but that only works with sed commands and it only works one file at a time?
Do I have to run a loop?
There's some great existing answers on the Unix stack exchange that help deal with your problem. Specifically, from this post, they use a loop to recursively loop through all the files in a particular directory, as follows:
( shopt -s globstar dotglob;
for file in **; do
if [[ -f $file ]] && [[ -w $file ]]; then
sed -i -- 's/foo/bar/g' "$file"
fi
done
)
Note the line, shopt -s globstar dotglob;, which allows us to use globbing patterns in the for loop. We also enclose the code in brackets, to prevent the shopt -s globstar dotglob; line option from becoming a global setting.
If you would like to apply this example to your file, you can just place your files in the current directory, and the code would probably look something like this:
( shopt -s globstar dotglob;
for file in **; do
if [[ -f $file ]] && [[ -w $file ]]; then
sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g' | grep -o "here.*there" | sed -r '/^.{,100}$/d' "$file"
fi
done
)
Note that we have placed a "$file" variable beside each of the seds that you used in your code, this replaces the name of the file for each command.
There is another example given in the code that allows you to pick which files to run on, rather than all the files in a directory, which you can also re-purpose for your code, as given here:
( shopt -s globstar dotglob
sed -i -- 's/foo/bar/g' **baz*
sed -i -- 's/foo/bar/g' **.baz
)
To answer your question of doing a loop on each line, you will need to put a loop for each line inside your for loop, like so:
while read line ; do
: sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g' | grep -o "here.*there" | sed -r '/^.{,100}$/d' "$line”
done
)
Although the for loop can be useful for dealing with files in recursive directories, I would recommend against also using another loop to grab lines, since it muddies your code, and it’s possible there is a better way to do it without parsing line by line.
The linked question is a fairly complete guide to many of the cases you may come across, and is also worth a read if you want to learn more.
Hope that helps!
You could use a for loop.
You could use the tool parallel.
Example
Create a set of test files using a for-loop
mkdir -p /tmp/so58333536
cd /tmp/so58333536
for i in 1.txt 2.txt 3.txt 4.txt 5.txt;do echo "The answer is 41" > $i;done
cat /tmp/so58333536/*
Now correct your mistake using parallel [1].
mkdir /tmp/so58333536.new
ls /tmp/so58333536/* |parallel "sed 's/41/42/' {} > /tmp/so58333536.new/{/}"
cat /tmp/so58333536.new/*
{}:: refers to the current file
{/}:: refers to name of the current file (path is removed)
Reads: List all files in so58333536 and apply the following sed command to each file and write the output to so58333536.new.
[1] Another option is to use sed -i for in-place editing.
Be very carefull with this!! Mistakes can cause serious damages!
# !! Do not use -i option regularly !!
ls /tmp/so58333536/* |parallel "sed -i 's/41/42/'"
I'm trying to change the filename from
prod.test.PVSREGPLUS20170915-6777.DAT.gpg
to
PVSREGPLUS20170915-0003.DAT.gpg
I used this
DTE=$(date +%I);ls prod.test* |cut -f 3,4,5 -d .|sed "s/\-/-00$DTE/" |cut -c 1-23,28-35
My problem is I need this command in a shell script
"#! /bin/bash
DTE=$(date +%I)
newfile=$(ls prod.test* |cut -f 3,4,5 -d .|sed "s/-*./$DTE/"|cut -c 1-23,28-35
The sed can't do expansion, would awk be able to do this? Any help would be greatly appreciated. Thanks
The simplest way to do this is with a for-loop over a glob pattern, then use paramater expansion to remove the prefix
prefix="prod.test."
hour=$(date '+%I')
for f in "$prefix"*; do
new=$( echo "${f#$prefix}" | sed 's/-[[:digit:]]\+/-00'"$hour"'/' )
echo mv "$f" "$new"
done
We really don't need sed: extended patterns and more parameter expansion
shopt -s extglob
for f in "$prefix"*; do
new=${f#$prefix}
new=${new/-+([0-9])/-00$hour}
echo mv "$f" "$new"
done
Remove "echo" if it looks good.
Or, with the perl rename as suggested in the comments:
rename -v -n 's/prod\.test\.//; use Time::Piece; s{-\d+}{"-00" . (localtime)->strftime("%I") }e' prod.test.*
Remove "-n" if it looks good.
I'm trying to create a bash script to automate extracting tar archives and my sed regex is not acting as expected. If I do:
archive=$1
directory=$(sed "s/.tar.bz2$//" <<< $archive)
echo "extracting $archive to $directory"
I get :
$ sh extract binutils-2.27.tar.bz2
extracting binutils-2.27.tar.bz2 to binutils-2.27
which is as expected.
But if I do :
archive=$1
directory=$(sed "s/.tar.[a-z0-9]{2,3}$//" <<< $archive)
echo "extracting $archive to $directory"
which is what I want to do (so as to handle any type of archive), I'd expect to get the same output, but I'm getting :
$ sh extract binutils-2.27.tar.bz2
extracting binutils-2.27.tar.bz2 to binutils-2.27.tar.bz2
As you can see the regex is not applying.
I've tested my regex on regex101.com and it seems to be correct, but it doesn't seem to be working correctly in the bash script. Can someone point me in the right direction as to what is going wrong please.
I'm on Mint 17.3.
The sed based solution would be:
directory=$(sed -E 's/\.tar\.[a-z0-9]{2,3}$//' <<< $archive)
however, with some assumptions, you can use bash features only:
directory=${archive%.tar.*}
You need the -E flag on sed:
$ archive=binutils-2.27.tar.bz2
$ directory=$( sed "s/.tar.[a-z0-9]{2,3}$//" <<< $archive)
$ echo $directory
binutils-2.27.tar.bz2
$ directory=$( sed -E "s/.tar.[a-z0-9]{2,3}$//" <<< $archive)
$ echo $directory
binutils-2.27
Some days ago I started a little bash script that should sum up the number of pages and file size of all PDF's in a folder. It's working quite well now but there's still one thing I don't understand.
Why is sed always failing if shopt -s nullglob is set? Does somebody know why this happens?
I'm working with GNU Bash 4.3 and sed 4.2.2 in Ubuntu 14.04.
set -u
set -e
folder=$1
overallfilesize=0
overallpages=0
numberoffiles=0
#If glob fails nothing should be returned
shopt -s nullglob
for file in $folder/*.pdf
do
# Disable empty string if glob fails
# (Necessary because otherwise sed fails ?:|)
#shopt -u nullglob
# This command is allowed to fail
set +e
pdfinfo="$(pdfinfo "$file" 2> /dev/null)"
ret=$?
set -e
if [[ $ret -eq 0 ]]
then
#Remove every non digit in the result
sedstring='s/[^0-9]//g'
filesize=$(echo -e "$pdfinfo" | grep -m 1 "File size:" | sed $sedstring)
pages=$(echo -e "$pdfinfo" | grep -m 1 "Pages:" | sed $sedstring)
overallfilesize=$(($overallfilesize + $filesize))
overallpages=$(($overallpages+$pages))
numberoffiles=$(($numberoffiles+1))
fi
done
echo -e "Processed files: $numberoffiles"
echo -e "Pagesum: $overallpages"
echo -e "Filesizesum [Bytes]: $overallfilesize"
Here's a simpler test case for reproducing your problem:
#!/bin/bash
shopt -s nullglob
pattern='s/[^0-9]//g'
sed $pattern <<< foo42
Expected output:
42
Actual output:
Usage: sed [OPTION]... {script-only-if-no-other-script} [input-file]...
(sed usage follows)
This happens because s/[^0-9]//g is a valid glob (matching a dir structure like like s/c/g), and you asked bash to interpret it. Since you don't have a matching file, nullglob kicks in and removes the pattern entirely.
Double quoting prevents word splitting and glob interpretation, which is almost always what you want:
#!/bin/bash
shopt -s nullglob
pattern='s/[^0-9]//g'
sed "$pattern" <<< foo42
This produces the expected output.
You should always double quote all your variable references, unless you have a specific reason not to.
I have a directory with lots of files. I want to keep only the 6 newest. I guess I can look at their creation date and run rm on all those that are too old, but is the a better way for doing this? Maybe some linux command I could use?
Thanks!
:)
rm -v $(ls -t mysvc-*.log | tail -n +7)
ls -t, list sorted by time
tail -n +7, +7 here means length-7, so all but first 7 lines
$() makes a list of strings from the enclosed command output
rm to remove the files, of course
Beware files with space in their names, $() splits on any white-space!
Here's my take on it, as a script. It does handle spaces in file names even if it is a bit of a hack.
#!/bin/bash
eval set -- $(ls -t1 | sed -e 's/.*/"&"/')
if [[ $# -gt 6 ]] ; then
shift 6
while [[ $# -gt 0 ]] ; do
echo "remove this file: $1" # rm "$1"
shift
done
fi
The second option to ls up there is a "one" for one file name per line. Doesn't actually seem to matter, though, since that appears to be the default when ls isn't feeding a tty.