I need to create the dump file name with the current date as postfix to the file name in shell script.
I tried below command in my script but that did not work.
Can any one suggest me the way :
... dump_'$(date '+%Y-%m-%d')'.txt
BACKGROUND:
I feel Its not require to mention details about the script as it is not completely related to the question but still I want to give some information.
My script is copying some lines from other *txt files and creating a separate dump.txt file .But I want to mention date as a postfix to the dump filename to understand the history about the file.
#!/bin/bash
# summary script
echo "Summary!"
shopt -s extglob
for f in */data/*txt; do tail -n+5 $f | head -1 | tr -d "\n"; echo " $f |"; done > dump.txt
sed -nE '/[[:blank:]][[:digit:]]?[[:digit:]]%/P' dump.txt > dump_$(date '+%Y-%m-%d').txt
wc -l dump_'$(date '+%Y-%m-%d')'.txt
You just need to remove de quotes that prevents shell interpretation:
dump_$(date '+%Y-%m-%d').txt
Related
There is a group of file in different paths that i want to parse for a password pattern and change it with a new one.
For example i have a file pass.lst it contains password list
oldpassword1/newpassword1
oldpassword2/newpassword2
oldpassword2/newpassword2
I have one more file that contains file paths list
path.lst:
$Home/test.xml
$Home/demo.sh
etc....
Now i want to make main file that contains a script that reads input from both the file and does my work
Here is the script that i am using
for i in `cat pass.lst`
`do`
for j in `cat path.lst`
do
perl -p -i -e 's/$i/g' $j
done
done
But it's giving me this error.
Substitution replacement not terminated at -e line 1.
Any suggestions and help will be appreciated.
#!/usr/bin/env bash
while IFS=/ read -r orig replacement; do
orig="$orig" replacement="$replacement" \
xargs -d $'\n' \
perl -p -i -e 's/$ENV{orig}/$ENV{replacement}/g' \
<path.lst
done <pass.lst
I have several files in multiple directories like in directory 1/file1 2/file2 3/file3......100/file100. I want to cat all those files to a single file using loop over index in bash script. Is there easy loop for doing so?
Thanks,
seq 100 | sed 's:.*:dir&/file&:' | xargs cat
seq 100 generates list of numbers from 1 to 100
sed
s substitutes
: separates parts of the command
.* the whole line
: separator. Usually / is used, but it's used in replacement string.
dir&/file& by dir<whole line>/file<whole line>
: separator
so it generates list of dir1/file1 ... dir100/file100
xargs - pass input as arguments to ...
cat - so it will execute cat dir1/file1 dir2/file2 ... dir100/file100.
This code should do the trick;
for((i=1;i<=`ls -l | wc -l`;i++)); do cat dir${i}/file${i} >> output; done
I made an example of what you're describing about your directory structure and files. Create directories and files with It's own content.
for ((i=1;i<=100;i++)); do
mkdir "$i" && touch "$i/file$i" && echo content of "$(pwd) $i" > "$i/file$i"
done
Check the created directories.
ls */*
ls */* | sort -n
If you see that the directories and files are created then proceed to the next step.
This solution does not involve any external command from the shell except of course cat :-)
Now we can check the contents of each files using bash syntax.
i=1
while [[ -e "$i" ]]; do
cat "$i"/*
((i++))
done
This code was tested in dash.
i=1
while [ -e "$i" ]; do
cat "$i"/*
i=$((i+1))
done
Just add the redirection of the output to the file after the done.
You can add some more test if you like see help test
One more thing :-), you can just check the contents using tail and brace expansion
tail -n +1 {1..100}/*
Using cat also you can redirect the output already, just remember brace expansion is bash3+ feature/syntax.
cat {1..100}/*
I have to merge files with the following naming pattern :
[SampleID]_[custom_ID01]_ID[RUN_ID]_L001_R1.fastq
[SampleID]_[custom_ID02]_ID[RUN_ID]_L002_R1.fastq
[SampleID]_[custom_ID03]_ID[RUN_ID]_L003_R1.fastq
[SampleID]_[custom_ID04]_ID[RUN_ID]_L004_R1.fastq
I need to merge all files with identical [SampleID] but different "Lanes" (L001-L004).
The following script works fine when directly run in the terminal:
custom_id="000"
RUN_ID="0025"
wd="/path/to/script/" # was missing/ incorrect
# get ALL sample identifiers
touch temp1.txt
for line in $wd/*.fastq ; do
fastq_identifier=$(echo "$line" | cut -d"_" -f1);
echo $fastq_identifier >> temp1.txt
done
# get all uniqe samples identical
cat temp1.txt | uniq > temp2.txt
input_var=$(cat temp2.txt)
# concatenate all fastq (different lanes) with identical identifier
for line in $input_var; do
cat $line*fastq >> $line"_"$custom_id"_ID"$Run_ID"_L001_R1.fastq"
done
rm temp1.txt temp2.txt;
But if I create a script file (concatenate_fastq.sh) and make it executable
$ chomd +x concatenate_fastq.sh
and run it
$ ./concatenate_fastq.sh
I got the following error:
$ concatenate_fastq.sh: line 17: /*.fastq_000_ID_L001_R1.fastq: Keine Berechtigung # = Permission denied
Thx to your hints below I solved the problem by fixing
wd=/path/to/script/
The immediate problem seems to be that wd is unset. If you script really genuinely contains exactly the line
wd="/path/to/script/"
then I would suspect invisible control characters in the script file (using a Windows editor is a common way to shoot yourself in the foot).
More generally, your script should cope correctly when the wildcard does not match any files. A common way to do that is to shopt -s nullglob but the subsequent script would still need adaptation then.
Refactoring the script to loop only over actual matches would help avoid trouble. Perhaps something like this:
shopt -s nullglob # bashism
printf '%s\n' "$wd"/*.fastq |
cut -d_ -f1 |
uniq |
while read -r line; do
cat "$line"*fastq >> "${line}_${custom_id}_ID${Run_ID}_L001_R1.fastq"
done
You'll notice that this simplifies the script tremendously, and avoids the pesky temporary files.
I solved it with:
if [ $# -ne 3 ] ; then
echo -e "Usage: $0 {path_to_working_directory} {custom_ID:Z+} {run_ID:ZZZZ}\n"
exit 1
fi
cwd=$(pwd)
wd=$1
custom_id=$2
RUN_ID=$3
folder=$(basename $wd)
input_var=$(ls *fastq | cut --fields 1 -d "_" | uniq)
for line in $input_var; do
cat $line*fastq >> $line"_"$custom_id"_ID"$RUN_ID"_L001_R1.fastq"
done
i hava a file in which using the command "edmEventSize" i can
extract a piece of information of that file (it is a number)
but know i have 700 files on which i have to execute that command
and i am trying to do it on a bash script but i cannot event do it for just
one file since i get "edmEventSize command not found", i already look for
more information but since i am new at bash i can not solve this task
Thank you in advanced
this is my script
#/usr/bin/env sh
for i in {1..700};
do
FILE="Py6_BstoJpsiKs0_7TeV_RECO_Run-0${i}.root"
edmEventSize... $FILE.root > salida${i}.log
done
head *.log | grep "^File" | cut -f4 > a.txt
rm *.log
As everyone would suggest, you can simplify your script like this:
#/bin/bash
for i in {1..700}; do
FILE="Py6_BstoJpsiKs0_7TeV_RECO_Run-0${i}.root"
/path/to/EdmEventSize "$FILE.root"
done | awk -F $'\t' '/^File/{print $4}' > a.txt
If your files actually are in the format of Py6_BstoJpsiKs0_7TeV_RECO_Run-####.root maybe the command you really need is:
printf -v FILE 'Py6_BstoJpsiKs0_7TeV_RECO_Run-%04d.root' "$i"
I was working on a program that could transfer files using sftp program:
sftp -oBatchMode=no -b ${BATCH_FILE} user#$123.123.123.123:/home << EOF
bye
EOF
One of my requirement is I must have a BATCH_FILE use with sftp and the batch file was generate using following script:
files=$(ls -1 ${SRC_PATH}/*.txt)
echo "$files" > ${TEMP_FILE}
while read file
do
if [ -s "${file}" ]
then
echo ${file} >> "${PARSE_FILE}" ## line 1
fi
done < ${TEMP_FILE}
awk '$0="put "$0' ${PARSE_FILE} > ${BATCH_FILE}
Somehow my program doesn't able to handle files with space in it. I did try using following code to replace line 1 but failed, the output of this will show filename\.txt.
newfile=`echo $file | tr ' ' '\\ '`
echo ${newfile} >> "${PARSE_FILE}"
In order to handle file name with space, how could I append a \ in front of the space within a file name?
THE PROBLEM
The problem is that tr SET1 SET2 will replace the Nth character in SET1 with the Nth character in SET2, which means that you are effectively replacing every space by \, instead of adding a backslash before every space.
PROPOSED SOLUTION
Instead of manually trying to fix the missing spaces, upon using your variable that might contain spaces; wrap it in quotes and let the shell handle the trouble for you.
See the below example:
$ echo $FILENAME
file with spaces.txt
$ ls $FILENAME
ls: cannot access file: No such file or directory
ls: cannot access with: No such file or directory
ls: cannot access spaces.txt: No such file or directory
$ ls "$FILENAME"
file with spaces.txt
But I really wanna replace stuff..
Well, if you really want a command to change every ' ' (space) into '\ ' (backslash, space) you could use sed with a basic replace-pattern, as the below:
$ echo "file with spaces.txt" | sed 's, ,\\ ,g'
file\ with\ spaces.txt
I haven't looked too closely at what you're trying to do there, but I do know that bash can handle filenames with spaces in them if you double-quote them. Why not try quoting every filename variable and see if that works? You're quoting some of them but not all yet.
Like try these: "${newfile}" or just "$newfile" "$file" "$tempfile" etc...
You can further simplify your code if you're using Bash:
function generate_batch_file {
for FILE in "${SRC_PATH}"/*.txt; do
[[ -s $FILE ]] && echo "put {$FILE// /\\ }"
done
}
sftp -oBatchMode=no -b <(generate_batch_file) user#$123.123.123.123:/home <<< "bye"
you can try to rename the file to work and rename it again after it has done.