So I'm trying to create a bash script that does something for every file in a directory. This is my script. The problem is that this script split on each space i would rather have it split on new line. So it look just like ls print. (I'm going to put other code inside the do, the echo is just a test)
#! /bin/bash
for file in $(ls)
do
echo $file
done
So how do i solve this?
You'd better avoid working with the result of ls (Why you shouldn't parse the output of ls).
Instead, do something like:
for file in /your/dir/*
do
echo "$file"
done
To pick files only:
while read -r file; do
echo "$file"
done < <(exec find -xtype f -maxdepth 1 -mindepth 1)
Or just filter them
for file in *; do
[[ -f $file ]] || continue
echo "$file"
done
Related
I would want to iterate over contents of a directory and list only ordinary files.
The path of the directory is given as an user input. The script works if the input is current directory but not with others.
I am aware that this can be done using ls.. but i need to use a for .. in control structure.
#!/bin/bash
echo "Enter the path:"
read path
contents=$(ls $path)
for content in $contents
do
if [ -f $content ];
then
echo $content
fi
done
ls is only returning the file names, not including the path. You need to either:
Change your working directory to the path in question, or
Combine the path with the names for your -f test
Option #2 would just change:
if [ -f $content ];
to:
if [ -f "$path/$content" ];
Note that there are other issues here; ls may make changes to the output that break this, depending on wrapping. If you insist on using ls, you can at least make it (somewhat) safer with:
contents="$(command ls -1F "$path")"
You have two ways of doing this properly:
Either loop through the * pattern and test file type:
#!/usr/bin/env bash
echo "Enter the path:"
read -r path
for file in "$path/"*; do
if [ -f "$file" ]; then
echo "$file"
fi
done
Or using find to iterate a null delimited list of file-names:
#!/usr/bin/env bash
echo "Enter the path:"
read -r path
while IFS= read -r -d '' file; do
echo "$file"
done < <(
find "$path" -maxdepth 1 -type f -print0
)
The second way is preferred since it will properly handle files with special characters and offload the file-type check to the find command.
Use file, set to search for files (-type f) from $path directory:
find "$path" -type f
Here is what you could write:
#!/usr/bin/env bash
path=
while [[ ! $path ]]; do
read -p "Enter path: " path
done
for file in "$path"/*; do
[[ -f $file ]] && printf '%s\n' "$file"
done
If you want to traverse all the subdirectories recursively looking for files, you can use globstar:
shopt -s globstar
for file in "$path"/**; do
printf '%s\n' "$file"
done
In case you are looking for specific files based on one or more patterns or some other condition, you could use the find command to pick those files. See this post:
How to loop through file names returned by find?
Related
When to wrap quotes around a shell variable?
Why you shouldn't parse the output of ls
Is double square brackets [[ ]] preferable over single square brackets [ ] in Bash?
Here is my problem : I am trying to parse a lot of files on a system to find some tokens. My tokens are stored in a file, one token on each line (for example token.txt). My path to parse are also stored in an other file, one path on each line (for example path.txt).
I use a combination of find and grep to do my stuff. Here is one attempt:
for path in $(cat path.txt)
do
for line in $(find $path -type f -print0 | xargs -0 grep -anf token.txt 2>/dev/null);
do
#some stuffs here
done
done
It seems to work fine, I don't really know if there is an other way to make it faster though (I am a beginner in programmation and shell).
My problem is : For each file found by the find command, I want to get all the files that are compressed. For this, I wanted to use the file command. The problem is that I need the output of the find command for both grep and file.
What is the best way to achieve this ? To summarize my problem, I would like something like this :
for path in $(cat path.txt)
do
for line in $(find $path -type f);
do
#Use file command to test all the files, and do some stuff
#Use grep to find some tokens in all the files, and do some stuff
done
done
I don't know if my explanations are clear, I tried my best.
EDIT : I read that doing for loop to read a file is bad, but some people claims that doing while read loop is also bad. I am a bit lost to be honest, I can't really find the proper way to do my stuffs.
The way you are doing it is fine, but here is another way to do it. With this method you won't have to add additional loops to iterate of each item in your configuration files. There are ways to simplify this further, but it would not be as readable.
To test this:
In "${DIR}/path" I have two directories listed (one on each line). Both directories are contained in the same parent directory as this script. In the "${DIR}/token" file, I have three tokens (one on each line) to search for.
#!/usr/bin/env bash
#
# Directory where this script is located
#
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
#
# Loop through each file contained in our path list
#
for f in $(find $(cat "${DIR}/path") -type f); do
for c in $(cat "${f}" | grep "$(cat ${DIR}/token)"); do
echo "${f}"
echo "${c}"
# Do your file command here
done
done
I think you need something like this:
find $(cat places.txt) -type f -exec bash -c 'file "$1" | grep -q compressed && echo $1 looks compressed' _ {} \;
Sample Output
/Users/mark/tmp/a.tgz looks compressed
This script is looking in all the places listed in places.txt and running a new bash shell for each file it finds. Inside the bash shell it is testing if the file is compressed and echoing a message if it is - I guess you will do something else but you don't say what.
Another way of writing that more verbosely if you have lots to do:
#!/bin/bash
while read -r d; do
find "$d" -type f -exec bash -c '
file "$1" | grep -q "compressed"
if [ $? -eq 0 ]; then
echo "$1" is compressed
else
echo "$1" is not compressed
fi' _ {} \;
done < <(cat places.txt)
Could someone please help with this script. I need to use grep to loop to through the filenames that need to be changed.
#!/bin/bash
file=
for file in $(ls $1)
do
grep "^.old" | mv "$1/$file" "$1/$file.old"
done
bash can handle regular expressions without using grep.
for f in "$1"/*; do
[[ $f =~ \.old ]] && continue
# Or a pattern instead
# [[ $f == *.old* ]] && continue
mv "$f" "$f.old"
done
You can also move the name checking into the pattern itself:
shopt -s extglob
for f in "$1/"!(*.old*); do
mv "$f" "$f.old"
done
If I understand your question correctly, you want to make rename a file (i.e. dir/file.txt ==> dir/file.old) only if the file has not been renamed before. The solution is as follow.
#!/bin/bash
for file in "$1/"*
do
backup_file="${file%.*}.old"
if [ ! -e "$backup_file" ]
then
echo mv "$file" "$backup_file"
fi
done
Discussion
The script currently does not actual make back up, it only displays the action. Run the script once and examine the output. If this is what you want, then remove the echo from the script and run it again.
Update
Here is the no if solution:
ls "$1/"* | grep -v ".old" | while read file
do
echo mv "$file" "${file}.old"
done
Discussion
The ls command displays all files.
The grep command filter out those files that has the .old extension so they won't be displayed.
The while loop reads the file names that do not have the .old extension, one by one and rename them.
I am trying to loop through all the files in a directory.
I want to do some stuff on each file (convert it to xml, not included in example), then write the file to a new directory structure.
for file in `find /home/devel/stuff/static/ -iname "*.pdf"`;
do
echo $file;
sed -e 's/static/changethis/' $file > newfile +".xml";
echo $newfile;
done
I want the results to be:
$file => /home/devel/stuff/static/2002/hello.txt
$newfile => /home/devel/stuff/changethis/2002/hello.txt.xml
How do I have to change my sed line?
If you need to rename multiple files, I would suggest to use rename command:
# remove "-n" after you verify it is what you need
rename -n 's/hello/hi/g' $(find /home/devel/stuff/static/ -type f)
or, if you don't have rename try this:
find /home/devel/stuff/static/ -type f | while read FILE
do
# modify line below to do what you need, then remove leading "echo"
echo mv $FILE $(echo $FILE | sed 's/hello/hi/g')
done
Are you trying to change the filename? Then
for file in /home/devel/stuff/static/*/*.txt
do
echo "Moving $file"
mv "$file" "${file/static/changethis}.xml"
done
Please make sure /home/devel/stuff/static/*/*.txt is what you want before using the script.
First, you have to create the name of the new file based on the name of the initial file. The obvious solution is:
newfile=${file/static/changethis}.xml
Second you have to make sure that the new directory exists or create it if not:
mkdir -p $(dirname $newfile)
Then you can do something with your file:
doSomething < $file > $newfile
I wouldn't do the for loop because of the possibility of overloading your command line. Command lines have a limited length, and if you overload it, it'll simply drop off the excess without giving you any warning. It might work if your find returns 100 file. It might work if it returns 1000 files, but it might fail if your find returns 1000 files and you'll never know.
The best way to handle this is to pipe the find into a while read statement as glenn jackman.
The sed command only works on STDIN and on files, but not on file names, so if you want to munge your file name, you'll have to do something like this:
$newname="$(echo $oldname | sed 's/old/new/')"
to get the new name of the file. The $() construct executes the command and puts the results of the command on STDOUT.
So, your script will look something like this:
find /home/devel/stuff/static/ -name "*.pdf" | while read $file
do
echo $file;
newfile="$(echo $file | sed -e 's/static/changethis/')"
newfile="$newfile.xml"
echo $newfile;
done
Now, since you're renaming the file directory, you'll have to make sure the directory exists before you do your move or copy:
find /home/devel/stuff/static/ -name "*.pdf" | while read $file
do
echo $file;
newfile="$(echo $file | sed -e 's/static/changethis/')"
newfile="$newfile.xml"
echo $newfile;
#Check for directory and create it if it doesn't exist
$dirname=$(dirname "$newfile")
if [ ! -d "$dirname" ]
then
mkdir -p "$dirname"
fi
#Directory now exists, so you can do the move
mv "$file" "$newfile"
done
Note the quotation marks to handle the case there's a space in the file name.
By the way, instead of doing this:
if [ ! -d "$dirname" ]
then
mkdir -p "$dirname"
fi
You can do this:
[ -d "$dirname"] || mkdir -p "$dirname"
The || means to execute the following command only if the test isn't true. Thus, if [ -d "$dirname" ] is a false statement (the directory doesn't exist), you run mkdir.
It's a fairly common shortcut when you see shell scripts.
find ... | while read file; do
newfile=$(basename "$file").xml;
do something to "$file" > "$somedir/$newfile"
done
OUTPUT="$(pwd)";
for file in `find . -iname "*.pdf"`;
do
echo $file;
cp $file $file.xml
echo "file created in directory = {$OUTPUT}"
done
This will create a new file with name whatyourfilename.xml, for hello.pdf the new file created would be hello.pdf.xml, basically it creates a new file with .xml appended at the end.
Remember the above script finds files in the directory /home/devel/stuff/static/ whose file names match the matcher string of the find command (in this case *.pdf), and copies it to your present working directory.
The find command in this particular script only finds files with filenames ending with .pdf If you wanted to run this script for files with file names ending with .txt, then you need to change the find command to this find /home/devel/stuff/static/ -iname "*.txt",
Once I wanted to remove trailing -min from my files. i.e. wanted alg-min.jpg to turn into alg.jpg. so after some struggle, managed to figure something like this:
for f in *; do echo $f; mv $f $(echo $f | sed 's/-min//g');done;
Hope this helps someone willing to REMOVE or SUBTITUDE some part of their file names.
I am trying to create a file that contains all of the code of an app. I have created a file called catlist.txt so that the files are added in the order I need them.
A snippet of my catlist.txt:
app/controllers/application_controller.rb
app/views/layouts/*
app/models/account.rb
app/controllers/accounts_controller.rb
app/views/accounts/*
When I run the command the files that are explicitly listed get added but the wildcard files do not.
cat catlist.txt|xargs cat > fullcode
I get
cat: app/views/layouts/*: No such file or directory
cat: app/views/accounts/*: No such file or directory
Can someone help me with this. If there is an easier method I am open to all suggestions.
Barb
Your problem is that xargs is not the shell, so the wildcard is being interpreted literally as an star. You'll need to have a shell to do the expansion for you like this:
cat catlist.txt | xargs -I % sh -c "cat %" > fullcode
Note that the * is not recursive in your data file. I assume that was what you meant. If you want the entries to be recursive, that's a little trickier and would need something more like DevNull's script, but that will require that you change your data file a bit to not include the stars.
Are you positive those directories exist?
The problem with doing a cat on a list like that (where you're using wildcards) is that the cat isn't recursive. It will only list the contents of that directory; not any subdirectories.
Here's what I would do:
#!/bin/bash.exe
output="out.txt"
if [ -f "$output" ]
then
rm $output
fi
for file in $(cat catlist.txt)
do
if [ -f "$file" ]
then
echo "$file is a file."
cat $file >> $output
elif [ -d "$file" ]
then
echo "$file is a directory."
find $file -type f -exec cat {} >> $output \;
else
echo "huh?"
fi
done
If the entry listed is a directory, it finds all files from that point on and cats them.
use a while read loop to read your file
while read -r file
do
if [ -f "$file" ]; then
yourcode "$file"
fi
# expand asterix
case "$file" in
*"*" )
for f in $file
do
yourcode "$f"
done
esac
done <"catlist.txt"