Globbing a filename and then saving to a variable - bash

I've got some files in a directory with a standard format, I'm looking to use a txt file with part of the filenames to extend them through * then finally add on a .gz tag as an output
For example, a file called 1.SNV.111-T.vcf in my directory, I have 111-T in my txt file.
#!/bin/bash
while getopts f: flag
do
case "${flag}" in
f) file=${OPTARG};;
esac
done
while IFS="" read -r p || [ -n "$p" ]
do
vcf="*${p}.vcf"
bgzip -c ${vcf} > ${vcf}.gz
done < $file
This will successfully run bgzip but actually save the output to be:
'*111-T.vcf.gz'
So adding .gz at the end has "deactivated" the * character, as pointed out by Barmar this is because there isn't a file in my directory called 1.SNV.111-T.vcf.gz so the wildcard is inactivated, please can anyone help?
I'm new to bash scripting but I assume there must be some way to save the "absolute" value of my vcf variable so that once it has found a match the first time, it's now a string that can be used downstream? I really cant find anything online.

The problem is that wildcards are only expanded when they match an existing file. You can't use a wildcard in the filename you're trying to create.
You need to get the expanded filename into the vcf variable. You can do it this way:
vcf=$(echo *"$p.vcf")
bgzip -c "$vcf" > "$vcf.gz"

Related

How do I rename multiple files before the extension in linux?

I want to take a group of files with names like 123456_1_2.mpg and turn it into 123456.mpg how can I do this using terminal commands?
To loop over all the available files you can use a for loop over the file names of the form ??????_?_?.mpg.
To rename the files you can retain the shortest match of a pattern from the beginning of the string using ${MYVAR%%pattern} without using any external command.
This said, your code should look like:
#!/bin/bash
shopt -s nullglob # do nothing if no matches found
for file in ??????_?_?.mpg; do
[[ -f $file ]] || continue # skip if not a regular file
new_file="${file%%_*}.mpg" # compose the new file name
echo mv "$file" "$new_file" # remove echo after testing
done
rename 's/_.*/.mpg/' *mpg
this will remove everything between the first underscore and the mpg file extension for all files ending in mpg
We can use grep to strip out everything but the first sequence of numbers. The --interactive flag will ask you if you're sure for each move, so you can make sure it's not doing anything you don't expect.
for file in *.mpg; do
mv --interactive "$file" "$(grep -o '^[0-9]\+' <<< "$file")".mpg
done
The regex ^[0-9]\+ translates to "any sequence of characters that starts with a number and is followed by zero or more numbers".

Remove double quoted string from text file?

I'm working on some text files, i have to remove a specific string
ex: "1234_NAME=TRUE", (including double quotes and comma)
Using Vim editor if i pass
:%s/"1234_NAME=TRUE",//g
But In this way i will be doing manually. But i want to automate the job by using some script.
I tried using sed tool, but it failed.
sed 's/"1234_NAME=TRUE",//g' hello.txt
I'm not expert on scripting. i know command is not correct. what is the solution for the above command.
And one more question:
Instead of passing file names manually, can we add a small loop which will take all text files in a directory one by one and remove the string.
Answer as suggested by #codeforester require some fix:
for file in *.txt
do
[[ -f "$file" ]] || continue
semicolon was the issue.
and Thanks to #SlePort for the answer.
Regards,
GBiradar
This might work for you:
sed -i.bak 's/"1234_NAME=TRUE",//g' *.txt
-i for in place editing
.bak: the extension of the backed up files
Your command is correct. To loop through your files, you can use a simple loop:
for file in *.txt; do # assuming your files are have .txt extension, modify accordingly
[[ -f "$file" ]] || continue # skip if not a regular file
sed 's/"1234_NAME=TRUE",//g' "$file" > "$file.modified" && mv "$file.modified" "$file"
# you can also use the `-i` flag to make a backup of the file and then overwrite the original
# `-i ''` will skip the backup and just overwrite the file
# sed -i .bak 's/"1234_NAME=TRUE",//g' "$file"
done
Make sure you have a backup of your files before running the above code.

Do not start loop if there is no files in directory?

All,
I am running BASH in Solaris 10
I have the following shell script that loops in a directory depending on the presence of CSV files.
The problem is with this piece of code is that it still does one loop even if there is no CSV files in that directory and then calls SQL loader.
SQLLoader then produces a log file because there is no file to process and this is beginning to mess up my directory filling it with log files.
for file in *.csv ;
do
echo "SQLLoader is reading : " $file
sqlldr <User>/<Password>#<DBURL>:<PORT>/<SID> control=sqlloader.ctl log=$inbox/$file.log data=$inbox/$file
done
How do I stop it going into a loop if there is no CSV files in that directory of $inbox
Say:
shopt -s nullglob
before your for loop.
This is not the default, and saying for file in *.csv when you don't have any matching files expands it to *.csv.
Quoting from the documentation:
nullglob
If set, Bash allows filename patterns which match no files to expand to a null
string, rather than themselves.
Use find to search files
for file in `find -name "*.csv"` ;
First off, using nullglob is the correct answer if it is available. However, a POSIX-compliant option is available.
The pattern will be treated as literal text if there are no matches. You can catch this with a small hack:
for file in *.csv; do
[ -f "$file" ] || break
...
done
When there are no matches, file will be set to the literal string *.csv, which is not the name of a file, so -f "$file" will fail. Otherwise, file will be set in turn to the name of each file matching the pattern, and -f "$file" will succeed every time. Note this will work even if there is an file named *.csv. The drawback is that you have to make a redundant test for each existing file.

how to change the extension of multiple files using bash script?

I am very new with linux usage maybe this is my first time so i hope some detailed help please.
I have more than 500 files in multiple directories on my server (Linux) I want to change their extensions to .xml using bash script
I used a lot of codes but none of them work some codes i used :
for file in *.txt
do
mv ${file} ${file/.txt}/.xml
done
or
for file in *.*
do
mv ${file} ${file/.*}/.xml
done
i do not know even if the second one is valid code or not i tried to change the txt extension beacuse the prompt said no such file '.txt'
I hope some good help for that thank you
Explanation
For recursivity you need Bash >=4 and to enable ** (i.e. globstar) ;
First, I use parameter expansion to remove the string .txt, which must be anchored at the end of the filename (%) :
the # anchors the pattern (plain word or glob) to the beginning,
and the % anchors it to the end.
Then I append the new extension .xml
Be extra cautious with filename, you should always quote parameters expansion.
Code
This should do it in Bash (note that I only echothe old/new filename, to actually rename the files, use mv instead of echo) :
shopt -s globstar # enable ** globstar/recursivity
for i in **/*.txt; do
[[ -d "$i" ]] && continue; # skip directories
echo "$i" "${i/%.txt}.xml";
done
If its a matter of a one or two sub-directories, you can use the rename command:
rename .txt .xml *.txt
This will rename all the .txt to .xml files in the directory from which the command is executed.
If all the files are in same directory, it can be done using a single command. For example you want to convert all jpg files to png, go to the related directory location and then use command
rename .jpg .png *
I wanted to rename "file.txt" to "file.jpg.txt", used rename easy peezy:
rename 's/.txt$/.jpg.txt/' *.txt
man rename will tell you everything you need to know.
Got to love Linux, there's a tool for everything :-)
passing command line argument for dir path
#!/bin/sh
cd $1
names_1=`ls`
for file in ${names_1}
do
mv ${file} ${file}.jpg
done

filename with space

I have written a shell script which picks all the files recursively inside all the directories and prepared a report with the file last modified, size.
The problem that I am facing, there are few files with name as "User Interface"(space in between). How to use there files in the for loop of the shell script and fetch the files and directories inside this.
Thanks in advance
Just put the file name variable between double quotes "$FILENAME"
You're probably trying to use something like for file in $(command). Instead, use a while read loop or a for loop with globbing. Make sure you quote variables that contain filenamess.
#!/bin/sh
command | while read -r file
do
something_with "$file"
done
or, in shells that support process substitution:
#!/bin/bash
while read -r file
do
something_with "$file"
done < <(command)
If you're simply iterating over a list of files:
for file in "$dir"/*
do
something_with "$file"
done
Google Search led me to this page

Resources