I have a really basic question. I'm trying to run STAR to align some reads to a sequencing experiment. I have around 30 samples. I firstly splitted the fast.gz files in 30 folders according to the sample of origin. In other words in each folder corresponding to a Sample* there are 4 files named *.fastq.gz (4 because of 4 lanes). I'm trying to loop STAR in each folder in this way:
for i in Sample*/;
do
cd $i;
qsub my_align_script.sh;
cd ..; done
where my_align_script.sh contains the following:
for i in *fastq.gz; do
STAR --runMode alignReads --genomeLoad LoadAndKeep --readFilesCommand zcat --outSAMtype BAM Unsorted --genomeDir "path" --readFilesIn $i ${i%.fastq.gz}.fastq.gz --runThreadN 10 --outFileNamePrefix ${i%.fastq.gz}
done
but unfortunately it seems not to find any *fast.gz file.
I tried to force to look in the current working dir of each Sample* folder specifying cd $path at the beginning but nothing when I run the for loop to launch jobs over folders.
Related
I've done a small amount of bash scripting. Mostly modifying a script to my needs.
On this one I am stumped.
I need a script that will read a sub-folder name inside a folder and make a numbered list of folders based on that sub-folder name.
Example:
I make a folder named “Pictures”.
Then inside I make a sub-folder named “picture-set”
I want a script to see the existing sub-folder name (picture-set) and make 10 more folders with sequential numbers appended to the end of the folder names.
ex:
folder is: Pictures
sub-folder is: picture-set
want to create:
“picture-set-01”
“picture-set-02”
“picture-set-03”
and so forth up to 10. Or a number specified in the script.
The folder structure would look like this:
/home/Pictures/picture-set
/home/Pictures/picture-set-01
/home/Pictures/picture-set-02
/home/Pictures/picture-set-03
... and so on
I am unable to tell the script how to find the base folder name to make additional folders.
ie: “picture-set”
or a better option:
Would be to create a folder and then create a set of numbered sub-folders based on the parent folder name.
ex:
/home/Songs - would become:
/home/Songs/Songs-001
/home/Songs/Songs-002
/home/Songs/Songs-003
and so on.
Please pardon my bad formatting... this is my first time asking a question on a forum such as this. Any links or pointers as to proper formatting is welcome.
Thanks for the help.
Bash has a parameter expansion you can use to generate folder names as arguments to the mkdir command:
#!/usr/bin/env bash
# Creates all directories up to 10
mkdir -p -- /home/Songs/Songs-{001..010}
This method is not very flexible if you need to dinamically change the range of numbers to generate using variables.
So you may use a Bash for loop and print format the names with desired number of digits and create each directory in the loop:
#!/usr/bin/env bash
start_index=1
end_index=10
for ((i=start_index; i<=end_index; i++)); do
# format a dirpath with the 3-digits index
printf -v dirpath '/home/Songs/Songs-%03d' $i
mkdir -p -- "$dirpath"
done
# Prerequisite:
mkdir Pictures
cd Pictures
# Your script:
min=1
max=12
name="$(basename "$(realpath .)")"
for num in $(seq -w $min $max); do mkdir "$name-$num"; done
# Result
ls
Pictures-01 Pictures-03 Pictures-05 Pictures-07 Pictures-09 Pictures-11
Pictures-02 Pictures-04 Pictures-06 Pictures-08 Pictures-10 Pictures-12
What I am currently struggling is to create multiple files and storing it into each directory.
I have made 100 directories like this:
mkdir ./dir.{01..100}
What I want is to create 100 text files for each directory. So that the result will show like:
click on dir.01 at home dir, which has files named: 01.txt to 100.txt
Also, I want to compress all 100 directories, each containing 100 text files into one using tar.
I am struggling with:
creating 100 text files each in 100 directories
using tar to zip all 100 directories together.
I am more interested in making creating 100 text files IN 100 directories. Also I am MUCH MORE interested in how to use tar to join all 100 directories together in specific file (fdtar) for instance.
If you are fine with empty files,
touch ./dir.{01..100}/{01..100}.txt
If you need each file to contain something, use that as the driver in a loop:
for file in ./dir.{01..100}/{01..100}.txt; do
printf "This is the file %s\n" "$file">"$file"
done
This could bump into ARG_MAX ("argument list too long") on some platforms, but it works fine on MacOS and should work fine on any reasonably standard Linux.
Splitting the loop into an inner and an outer loop could work around that problem:
for dir in ./dir.{01..100}; do
for file in {01..100}.txt; do
printf "This is file %s/%s\n" >"$dir/$file"
done
done
If I understand you need two things. First, you have 100 directories and need to create a file in each. With a for loop in bash run from the parent directory where all other directories you have created are:
for n in dir.*
do
f=`echo $n | sed s/dir\.//`
echo "This is file $n" >"$n/$f.txt"
done
Regarding tar that is even easier because tar will take multiple directories and glue them together. From the parent directory try:
tar cvf fd.tar dir.*
The c option will create the archive. v will tell tar to print all it is doing so you know what is happening. f directories.tar will create the archive with that name.
When you undo the tar operation, you will use:
tar xvf fd.tar
In this case x will extract the contents of the tar archive and will create all 100 directories for you at the directory from which you invoke it.
Note that I have used fd.tar and not fdtar as the .tar extension is the customary way to signal that the file is a tar archive.
I have a following data dir:
root/A/1
root/A/2
root/B/1
root/B/2
root/B/3
root/C/1
root/C/2
And I want to convert it into following file structure:
root2/I/A/1
root2/I/A/2
root2/I/B/1
root2/I/B/2
root2/I/B/3
root2/II/C/1
root2/II/C/2
Purpose of doing it is I want to run some script which takes home folder (root here) and runs on it. And I want to run it in parallel on many folders(I, II) to speed up the process.
Simple assumption about file and folder name is that all are alphanumeric, even no period or underscore.
Edit: I tried following:
for i in `seq 1 30`; do mkdir -p "root2/folder$i"; find root -type f | head -n 4000 | xargs -i cp "{}" "root2/folder$i"; done
Problem is that it creates something like following, which is not what i wanted.
root2/I/1
root2/I/2
root2/I/1
root2/I/2
root2/I/3
root2/II/1
root2/II/2
You may wish to use a lesser known command called dirsplit, the usual application of which is to split a directory into multiple directories for burning purposes.
Use it like below :
dirsplit -m -s 300M /root/ -p /backup/folder1
Options implies below stuff :
-m|--move Move files to target dirs
-e 2 special exploration mode, 2 means files in directory are put together
-p prefix to be attached to each directory created, in you case I, II etc
-s Maximum size allowed for each new folder created.
For more information see :
dirsplit -H
I have seen several variations of this question asked however none of witch fully answer my problem. I have several folders that contain between 2,000 and 150,000 image files. Searching these directories becomes very inefficient as the number of files increases as speed is drastically decreased.
What I want to do is use automator to:
1. select folder
2. create subfolder
3. fill newly created subfolder with the first 1000 files in the folder selected in (1.)
4. if more files exist in the outer folder, create another subfolder and fill with the next 1000 files etc.
Is this possible? Any help is much apreciated.
Thank you,
Brenden
This takes a directory and moves the contents to new folders called "newFolder1", "newFolder2" etc.
Have you used Terminal much? Let me know if you need more instruction. I also haven't put in any checks, so let me know if you get any errors.
o Save this file to your desktop (as script.sh for the purpose of tutorial)
#!/bin/bash
cd "$1" #Change directory to the folder to sort
filesPerFolder=$2 #This is how many files will be in each folder
currentDir='newFolder1';
currentFileCount=0;
currentDirCount=1;
mkdir $currentDir;
for file in *
do
if [ -f "$file" ]
then
mv "$file" "$currentDir";
fi
currentFileCount=$(($currentFileCount + 1));
if [ $(($currentFileCount % $filesPerFolder)) -eq "0" ] #Every X files, make a new folder
then
currentDirCount=$(($currentDirCount + 1));
currentDir='newFolder'$currentDirCount;
mkdir "$currentDir";
fi
done
o Open Terminal and type cd ~/Desktop/
o Type chmod 777 script.sh to change the permissions on the file
o Type ./script.sh "/path/to/folder/you/want/to/sort" 30
o The 30 here is how many files you want in each folder.
Here's what I want to do:
I have thousand of split rar archives on folder name Archives.
Name of the files 0001.part1.rar 0002.part2.rar 0003.part3.rar etc.
read 0001.part1.rar
create a directory based on prefix of the file above e.g. 0001
move all files with the same prefix on the directory created above.
Extract the files within that directory
delete all rar files in that directory
Rename the extracted file based on a list of names from a text file.
Rar the renamed file with different arguments.
Move the renamed file (from step 6) to a new directory called Done.
Proceed to file 0002.part1.rar then do steps 2-8 and so forth.
Additional how would I incorporate it with cron???
This should be run once only...
After extracting the first set of rar's files change to:
file.001
file.002
file.003
etc. which I need to extract too.
Clarification on Step 6:
After extracting the second set of rar's (file.001 , file.002 etc.)
I want to rename it based on a list of names from a text file.
e.g. List of files from a text file:
0001 - GB Funds.DAT
0002 - US Bonds.DAT
0003 - SG Securities.DAT
0004 - LU Credits.DAT
Clarification on Step 7:
After renaming the file I want to move it on a new folder called "Done"
Clarification on Step 9:
Go back to the main folder with all the other archives
and continue extracting the next set of archives and
do the same steps from 1 to 8.
You can write a shell script including something like this:
# foo.sh
set -e
set -u
for i in `find -max-depth 1 -type f -name '*.rar' | sed 's/\.part*\.rar$//' | sort -u`; do
mkdir $i
mv $i.part*rar $i
cd $i
unrar x $i.part1.rar
DST=`grep $i ../rename.txt | sed 's/^[0-9]\+ - //'`
mv $i "$DST"
# and so on, rar it together again, move it done directory etc.
cd ..
done
Run it then via:
bash foo.sh
You have to clarify 6./8./9.
I don't know why do you want to run it via cron, since you only want to run it once. at is designed for running one-time jobs, or run it in a screen session.
I suggest that you do a few tests with 1-3 files from your collection and the script you end up with, before starting the whole job.