I want to create empty files where the names are taken from list.txt. I tried this:
declare -a siglist=$(cat list.txt)
for fname in {list}; do
echo $fname > "$fname.zz"
done
Note that your echo solution does not create empty files: they contain a single newline.
If you want to stay entirely in the shell (the comments show various ways using touch and xargs), you can use the null-command : (which does nothing) with a redirection as follows.
while read filename; do
: > "${filename}.zz"
done < list.txt
This is portable to all Bourne-heritage shells, so not restricted to bash.
If you are ok with awk, could you please try following.
awk '{val=(val?val OFS:"")$0} END{system("touch " val)}' Input_file
Explanation: Creating a variable named val in awk program and keep concatenating line's values into it. In END section of this awk program using system command of this command to use touch command and passing that variable val to it to create all file names there.
Related
I want to read a list of file names stored in a file, and the top level directory is a macro, since this is for a script that may be run in several environments.
For example, there is a file file_list.txt holding the following fully qualified file paths:
$TOP_DIR/subdir_a/subdir_b/file_1
$TOP_DIR/subdir_x/subdir_y/subdir_z/file_2
In my script, I want to tar the files, but in order to do that, tar must know the actual path.
How can I get the string containing the file path to expand the macro to get the actual path?
In the code below the string value echoed is exactly as in the file above.
I tried using actual_file_path=`eval $file_path` and while eval does evaluate the macro, it returns a status, not the evaluated path.
for file_path in `cat $input_file_list`
do
echo "$file_path"
done
With the tag ksh I think you do not have the utility envsubst.
When the number of variables in $input_file_list is very limited, you can substitute vars with awk :
awk -v top_dir="${TOP_DIR}" '{ sub(/$TOP_DIR/, top_dir); print}' "${input_file_list}"
I was using eval incorrectly. The solution is to use an assignment on the right side of eval as follows:
for file_path in `cat $input_file_list`
do
eval myfile=$file_name
echo "myfile = $myfile"
done
$myfile now has the actual expansion of the macro.
Long story short, I'm trying to grep a value contained in the first column of a text file by using a variable.
Here's a sample of the script, with the grep command that doesn't work:
for ii in `cat list.txt`
do
grep '^$ii' >outfile.txt
done
Contents of list.txt :
123,"first product",description,20.456789
456,"second product",description,30.123456
789,"third product",description,40.123456
If I perform grep '^123' list.txt, it produces the correct output... Just the first line of list.txt.
If I try to use the variable (ie grep '^ii' list.txt) I get a "^ii command not found" error. I tried to combine text with the variable to get it to work:
VAR1= "'^"$ii"'"
but the VAR1 variable contained a carriage return after the $ii variable:
'^123
'
I've tried a laundry list of things to remove the cr/lr (ie sed & awk), but to no avail. There has to be an easier way to perform the grep command using the variable. I would prefer to stay with the grep command because it works perfectly when performing it manually.
You have things mixed in the command grep '^ii' list.txt. The character ^ is for the beginning of the line and a $ is for the value of a variable.
When you want to grep for 123 in the variable ii at the beginning of the line, use
ii="123"
grep "^$ii" list.txt
(You should use double quotes here)
Good moment for learning good habits: Continue in variable names in lowercase (well done) and use curly braces (don't harm and are needed in other cases) :
ii="123"
grep "^${ii}" list.txt
Now we both are forgetting something: Our grep will also match
1234,"4-digit product",description,11.1111. Include a , in the grep:
ii="123"
grep "^${ii}," list.txt
And how did you get the "^ii command not found" error ? I think you used backquotes (old way for nesting a command, better is echo "example: $(date)") and you wrote
grep `^ii` list.txt # wrong !
#!/bin/sh
# Read every character before the first comma into the variable ii.
while IFS=, read ii rest; do
# Echo the value of ii. If these values are what you want, you're done; no
# need for grep.
echo "ii = $ii"
# If you want to find something associated with these values in another
# file, however, you can grep the file for the values. Use double quotes so
# that the value of $ii is substituted in the argument to grep.
grep "^$ii" some_other_file.txt >outfile.txt
done <list.txt
I have a script that looks for files of specific type in a specified directory and if they are present, generates a file with the basenames before creating a tar.gz. Once compressed, I check to ensure the tarball contains all the files by running a diff check.
I have created a pair of variables that are the pre-compressed file list and those found in the tarball. When I run an if statement including diff of the variables, I receive this error:
diff: missing operand after `/my/original/dir/filelist.txt'
diff: Try `diff --help' for more information.
I worked around this by referencing the files themselves rather than the created variables. If I run the if statement in a separate bash script, it works just fine using the variables so I am entirely lost as to what my error is in my larger script. Below I provide both the snippet from the large script and the diff statement as its own script for reference.
The if diff in its own script:
#!/bin/sh
filelist=(filelist.txt)
tarfiles=(tarfiles.txt)
#differences=$(diff filelist.txt tarfiles.txt) #Uncomment if below fails
differences=$(diff $filelist $tarfiles)
if $differences > /dev/null ; then
echo Same
else
echo Different
fi
The above works just fine.
Now including this at the end my larger script:
TARFILES=$(tar -tzf "$ARCHIVES/tarredfiles.tar.gz" | awk -F/ '{ if($NF != "") print $NF }' > $LOGS/tarfiles.txt)
FILELIST=($LOGS/filelist.txt)
#Check to see if it all worked
DIFF=$(diff $FILELIST $TARFILES)
cd $LOGS #I shouldn't need to do this but I do as a safety mechanism
#if diff filelist.txt tarfiles.txt > /dev/null ; then
if diff $FILELIST $TARFILES > /dev/null ; then
echo "Today's files have been archived and checked."
else
echo "Some or none of today's files have been archived, check the logs to find the error."
echo (diff $TARFILES $FILELIST) > $LOGS/$(date '+%Y%m%d')errors.txt
fi
I have tried enclosing the variables in "" and it didn't seem to make a difference.
The way you populate TARFILES results in it being empty. What is it that you're trying to store in the variable?
This line
TARFILES=$(tar -tzf "$ARCHIVES/tarredfiles.tar.gz" | awk -F/ '{ if($NF != "") print $NF }' > $LOGS/tarfiles.txt)
Does the following steps
Extracts a list of the filenames (-t) from the compressed (-z) tar file (-f) named tarredfiles.tar.gz in the directory referred to by the $ARCHIVES variables
Sends (pipes) that list of filenames into awk where you print the last component of the filename, that is the last field ($NF) of each line when it is split by / (-F/)
Sends (redirects) all of that output into the log file $LOGS/tarfiles.txt
Captures any other output (of which there will be none!) and stores it in the TARFILES variable.
So, the variable TARFILES is always empty, but the file tarfiles.txt has content in it.
It seems that you want the diff to compare the content of tarfiles.txt with the content of filelist.txt, but you're trying to use your variables in a way that isn't really compatible with that.
An expression of the form:
TARFILES=$( command goes here )
captures the output of that command.
And
TARFILES=$( command goes here > some-file.txt )
sends the output of the command into the file, and then captures nothing.
What you want is something like:
TARFILES=some-file.txt
command goes here > $TARFILES
which will set the variable to be the name of your file, and then run a command which put content into that file.
So, specifically:
TARFILES=$LOGS/tarfiles.txt
tar -tzf "$ARCHIVES/tarredfiles.tar.gz" | awk -F/ '{ if($NF != "") print $NF }' > $TARFILES
When working will shell scripts, it is very common to be running commands that produce output that goes into files, etc. One thing you need to be really clear about in the logic of your script is when you want your variables to contain actual content (that is, the output of a command), and when you want them to contain filenames.
In your case you want to run diff on 2 files ("tarfiles" and "filelist") that happen to contain a list of filenames, so that means there's a little bit more to keep track of, but essentially you want to populate "tarfiles" with the output from a command, and then run a diff where you pass in the 2 files names "tarfiles" and "filelist". So you never want to use $( ... ) to populate tarfiles.txt because that is how you capture the output of a command into a variable, and what you're trying to do is store a filename in your variable.
Using a linux shell script, I am trying to loop through all of the file names in a directory and extract the numbers out of the file name before I process the file.
Something like this:
for files in `ls *.gz`
do
echo "Looking at... $files"
gunzip $files
echo "$files" | awk '/[0-9]/' ' {print $1}'
echo "$files is unzipped"
done
Thank you for any help with this.
You can substitute all non-numbers in the file name.
echo "${files//[!0-9]/}"
This will obviously produce concatenated numbers if a file name contains multiple runs of digits. For example, 12a34.gz gets turned into 1234.
This substitution mechanism is a Bash-only feature, and not supported by plain sh.
I have this script:
#!/bin/bash
FASTQFILES=~/Programs/ncbi-blast-2.2.29+/DB_files/*.fastq
FASTAFILES=~/Programs/ncbi-blast-2.2.29+/DB_files/*.fasta
clear
for file in $FASTQFILES
do cat $FASTQFILES | perl -e '$i=0;while(<>){if(/^\#/&&$i==0){s/^\#/\>/;print;}elsif($i==1){print;$i=-3}$i++;}' > ~/Programs/ncbi-blast-2.2.29+/DB_files/"${FASTQFILES%.*}.fasta"
mv $FASTAFILES ~/Programs/ncbi-blast-2.2.29+/db/
done
I'm trying it to grab the files defined in $FASTQFILES, do the .fastq to .fasta conversion, name the output with the same filename of the input, and move it to a new folder. E.g., ~/./DB_files/HELLO.fastq should give a converted ~/./db/HELLO.fasta
The problem is that the output of the conversion is a properly formatted hidden file called .fasta in the first folder instead of the expected one named HELLO.fasta. So there is nothing to mv. I think I'm messing up in the ${FASTQFILES%.*}.fasta argument but I can't seem to fix it.
I see three problems:
One part of your trouble is that you use cat $FASTQFILES instead of cat $file.
You also need to fix the I/O redirection at the end of that line to > ~/Programs/ncbi-blast-2.2.29+/DB_files/"${file%.fastq}.fasta".
The mv command needs to be executed outside the loop.
In fact, when processing a single file at a time, you don't need to use cat at all (UUOC — Useless Use Of Cat). Simply provide "$file" as an argument to the Perl script.