I have a problem which I think needs loops to solve.
I have fastq files that follow the naming conventions, all in a generic directory e.g 'allpools' :
ls allpools
2022_pool_1_Seq_GEX.fastq.gz
2022_pool_1_Seq_CMO.fastq.gz
2022_pool_2_Seq_GEX.fastq.gz
2022_pool_2_Seq_CMO.fastq.gz
I need one loop to make directories in 'allpools' for as many pool numbers as there are in the fastqs, where the resulting directories will be called :
pool1
pool2
and then another loop whereby the original fastqs are moved and placed in respective directories for gex and cmo. As an example :
allpools/pool1/pool1_GEX #containing all gex fastqs of pool_1
allpools/pool1/pool1_CMO #containing all cmo fastqs of pool_1
####
allpools/pool2/pool2_GEX #containing all gex fastqs of pool_2
allpools/pool2/pool2_CMO #containing all cmo fastqs of pool_2
Before:
$ tree allpools/
allpools/
├── 2022_pool_1_Seq_CMO.fastq.gz
├── 2022_pool_1_Seq_GEX.fastq.gz
├── 2022_pool_2_Seq_CMO.fastq.gz
└── 2022_pool_2_Seq_GEX.fastq.gz
Use sed to parse the filename and generate the directory name:
$ for f in allpools/*; do d=$(sed -E 's%.+(pool_[0-9]+).+(GEX|CMO).+%\1/\1_\2%' <<<$f); mkdir -p allpools/$d; mv -vi $f allpools/$d/; done
renamed 'allpools/2022_pool_1_Seq_CMO.fastq.gz' -> 'allpools/pool_1/pool_1_CMO/2022_pool_1_Seq_CMO.fastq.gz'
renamed 'allpools/2022_pool_1_Seq_GEX.fastq.gz' -> 'allpools/pool_1/pool_1_GEX/2022_pool_1_Seq_GEX.fastq.gz'
renamed 'allpools/2022_pool_2_Seq_CMO.fastq.gz' -> 'allpools/pool_2/pool_2_CMO/2022_pool_2_Seq_CMO.fastq.gz'
renamed 'allpools/2022_pool_2_Seq_GEX.fastq.gz' -> 'allpools/pool_2/pool_2_GEX/2022_pool_2_Seq_GEX.fastq.gz'
After:
$ tree allpools/
allpools/
├── pool_1
│ ├── pool_1_CMO
│ │ └── 2022_pool_1_Seq_CMO.fastq.gz
│ └── pool_1_GEX
│ └── 2022_pool_1_Seq_GEX.fastq.gz
└── pool_2
├── pool_2_CMO
│ └── 2022_pool_2_Seq_CMO.fastq.gz
└── pool_2_GEX
└── 2022_pool_2_Seq_GEX.fastq.gz
Related
I'm using OHMYZSH but I'm thinking about building a minimal configuration to Zsh.
So here's the problem. Ohmyzsh has special behaviour for completion. For example: if I'm in my Home Directory and type cd mus and hit a TAB it will expand to cd Music, even I'm typing the sentence with small letters.
I've founded this:
Have zsh return case-insensitive auto-complete matches, but prefer exact matches
Which is my problem but on the contrary. I want to get case insensitive without install OHMYZSH.
Looking at the OHMYZSH structure, I've found this:
.
├── ./cache
├── ./CODE_OF_CONDUCT.md
├── ./CONTRIBUTING.md
├── ./custom
├── ./lib
├── ./LICENSE.txt
├── ./log
├── ./oh-my-zsh.sh
├── ./plugins
├── ./README.md
├── ./templates
├── ./themes
└── ./tools
8 directories, 5 files
Inside the lib folder, there are some config files.
lib
├── bzr.zsh
├── clipboard.zsh
├── cli.zsh
├── compfix.zsh
├── completion.zsh
├── correction.zsh
├── diagnostics.zsh
├── directories.zsh
├── functions.zsh
├── git.zsh
├── grep.zsh
├── history.zsh
├── key-bindings.zsh
├── misc.zsh
├── nvm.zsh
├── prompt_info_functions.zsh
├── spectrum.zsh
├── termsupport.zsh
└── theme-and-appearance.zsh
0 directories, 19 files
I've already tried to source the completion.zsh using Zinit(plugin manager for ZSH which enables load some OHMYZSH stuff) without success. I don't know what is the correct file for the behaviour I want.
This is my config:
#exports
export EDITOR=nvim
export VISUAL=code
export SUDO_EDITOR=nvim
# Theme
ZSH_THEME="spaceship"
#PLUGINS
#==============================================================
source "$HOME/.zinit/bin/zinit.zsh"
autoload -Uz _zinit
(( ${+_comps} )) && _comps[zinit]=_zinit
zinit light zdharma/fast-syntax-highlighting
zinit light zsh-users/zsh-autosuggestions
zinit light zsh-users/zsh-completions
zinit light agkozak/zsh-z
#THIS LINE IS MY ATTEMPT TO load OHMYZSH FILES
zinit snippet OMZL::completion.zsh
#==============================================================
### End of Zinit's installer chunk
What's the correct file to load? Or is there another way to get case insensitive completions?
After reading some comments I found a solution. Add these two lines to the Zsh config file:
zstyle ':completion:*' matcher-list '' 'm:{a-zA-Z}={A-Za-z}' 'r:|=*' 'l:|=* r:|=*'
autoload -Uz compinit && compinit
I have a directory containing many sub-directories, each with lots of files having different extensions.
Is it possible for me to filter the output of the tree command so that files having certain extensions like say .log and .msh or certain named directories are not in the output
Consider the example below
.
├── dir1
│ ├── bar.log
│ ├── blahblah
│ │ └── blah.txt
│ ├── hello.txt
│ └── test.msh
├── foo.log
├── out
├── test.py
└── test.txt
2 directories, 8 files
I would like to filter away certain directories (and obviously their contents) and files having certain extensions. For instance I would like to filter away the contents of directory blahblah and all files having extensions .msh and .log
So I would like the output to be
.
├── dir1
│ ├── hello.txt
├── out
├── test.py
└── test.txt
tree command has -I option that allows you to filter out both files and directories that match the given pattern.
tree -I 'blahblah|*.msh|*.log'
First of all, I've read numerous post on here regarding this topic, but none have helped so far, e.g.
Zipping without creating parent folder
Unix zipping sub directory not including parent directory
Here's what I want. I have the following folder structure:
- root
-- subroot
--- subsubroot
---- file1.foo
---- file2.foo
---- file3.foo
---- file4.foo
And the result of the zipped file should be, when unzipped
file1.foo
file2.foo
file3.foo
file4.foo
The crux is that the file*.foo are just packages, thus basically being folders itself, so I tried using the -j option in conjunction with my recursive option (-r), but then I just get a flat hierarchy of all the files.
My current shell command looks like this:
find root/subroot/subsubroot -name '*.foo' | grep -v somestring | xargs -I % zip -r foos.zip %
However, the result of that will be exactly like the original structure with all the parent folder.
I don't necessarily have to use the zip command, but the resulting file has to be a zip.
Your command is nearly correct, although you shouldn't need to use the -I option.
Directory Structure:
.
└── root
└── subroot
└── subsubroot
├── foo1
│ └── foo1.pkg
├── foo2
│ └── foo2.pkg
├── foo3
│ └── foo3.pkg
├── fooA.pkg
├── fooB.pkg
├── fooC.pkg
└── nope.pkg
Command:
find root/subroot/subsubroot -name '*.pkg' | grep -v "nope" | xargs zip -jr foo
Resulting foo.zip:
fooA.pkg
fooB.pkg
fooC.pkg
foo1.pkg
foo2.pkg
foo2.pkg
There is one caveat.
If you have this type of directory structure you'll get an error:
.
└── root
└── subroot
└── subsubroot
├── foo1
│ └── foo1.pkg
├── foo2
│ └── foo2.pkg
├── foo3
│ └── fooC.pkg
├── fooA.pkg
├── fooB.pkg
├── fooC.pkg
└── nope.pkg
Error:
zip warning: first full name: root/subroot/subsubroot/foo3/fooC.pkg
second full name: root/subroot/subsubroot/fooC.pkg
name in zip file repeated: fooC.pkg
this may be a result of using -j
You'll need to be mindful that files/packages in sub-directories do not have the same filenames otherwise you'll run into problems as shown by fooC.pkg. It's assumed this is not the case, since you are attempting to create a flat archive.
I try to make a rsync with list of include and list of delete file.
My source dir looks like :
app/
config/
ignore.php
include.php
My destination dir looks like :
app/
delete.php
In include.txt file :
app/config/include.php
In delete.txt file :
app/delete.php
I have tried this command :
rsync -avz --include-from=include.txt --delete-excluded --exclude-from=delete.txt src/ dest/
The result is :
sending incremental file list
./
app/
deleting app/delete.php
app/config/
app/config/ignore.php
app/config/index.php
My problem is ignore.php was include because rsync probably ignore --include-from argument.
Need help plz :)
EDIT :
I replace include-from by files-from like that :
rsync -av --files-from=list --delete-excluded --exclude-from=delete src/ dest/
This command ignore app/config/include.php like I want BUT app/delete.php wasn't delete.
I try this :
rsync -av --delete-excluded --exclude-from=delete src/ dest/
app/delete.php is delete.
Why --files-from and --exclude-from doesn't work together ?
if your goal is to delete files from destination that are not present in source, and to ignore certain files in source from being copied to destination, then delete.txt is unnecessary when using the --delete option to mirror directories.
instead use an excludes file such as exclude.txt to ignore files in source that you don't want copied to destination.
file system before:
.
├── dest
│ └── app
│ └── delete.php
├── exclude.txt
├── src
│ └── app
│ └── config
│ ├── ignore.php
│ └── include.php
└── test.bash
exclude.txt:
app/config/ignore.php
test.bash:
rsync -av --exclude-from 'exclude.txt' --delete src/ dest/
after:
.
├── dest
│ └── app
│ └── config
│ └── include.php
├── exclude.txt
├── src
│ └── app
│ └── config
│ ├── ignore.php
│ └── include.php
└── test.bash
Between web server environments we have data asset folder structures similar to the below:
/Volumes/data-acct/assets/
├── audio
│ ├── section1
│ │ └── nested files and folders
│ └── section2
├── css
│ ├── section1
│ │ └── nested files and folders
│ └── section2
├── images
│ ├── section1
│ │ └── nested files and folders
│ └── section2
└── videos
├── section1
└── section2
I've been trying to find an include filter that will allow me to match a particular section folder for each of the areas, although I get no results when changing from --include="*/" (to include all folders and --include="*/section1/**/*.*"
rsync --dry-run --verbose --recursive --update --include="*/section1/**/*.*" --exclude="*" /Volumes/data-acct/assets/ /Volumes/data-live/assets/
Should I be running several commands for this? Or should I be making use of the --filters argument (which doesn't seem to be documented)
I'm sure there's a good solution using rsync only (since it is very powerful), yet when in doubt I like to fall back on well known tools like so:
cd /Volumes/data-acct/assets &&
find . | # get all files
grep /section1/ | # that contain section1
grep -v videos/ | # but without videos
rsync --dry-run --verbose --recursive --update --files-from=- \
/Volumes/data-acct/assets/ /Volumes/data-live/assets/
The --files-from option allows you to create your own custom list without using any rsync filters at all.