I was an Unison fan but I need more complicated filters to back up my laptop.
After hours of trying I obtained for the most part what I want; this is my situation:
This is my bash script:
#!/bin/sh
rsync -aHAXzhv --dry-run \
--progress \
--stats \
--delete \
--prune-empty-dirs \
--filter="merge filter" \
/ /media/mypassport/laptop/
and this is my filter:
+ etc
+ etc/apache2
+ etc/apache2/sites-available
+ etc/apache2/sites-available/*
+ home
+ home/ale
+ home/ale/*
+ home/ale/.filezilla/*
+ home/ale/.emacs.d/*
+ home/ale/.evolution/*
+ home/ale/.mozilla/*
+ home/ale/.ssh/*
+ home/ale/.subversion/*
+ home/ale/.unison/*
+ root
+ root/*
+ root/.c/*
+ usr/
+ usr/local/
+ usr/local/*
+ var/
+ var/www
+ var/www/*
+ *.git/***
+ *.svn/***
+ *.cvs/***
- etc/*
- etc/apache2/*
- home/ale/.** # this exclude hidden folder in ale's home
- var/*
- root/.**
- lib64
- initrd.img.old
- initrd.img
- vmlinuz
- vmlinuz.old
- *~
- .*~
- *.o
- *.tmp
- .*
- */
I would like to be able to exclude ALL hidden files in home folderS and include just few of them in this way:
+ home/*/.bashrc
- home/*/.*
but it does not work.
Any help would be appreciated.
Best regards
Thank you
Related
I am trying to copy all video files recursively from a locally synced FUSE file system (GDrive, Insync, Cryptomator, Ubuntu 20.04), recreating the source directory structure in the destination for any video that is moved - only. Although it is working for some video others are left behind with the same file extension as some that were successfully moved.
Can I adjust my script somehow to make this work properly please? I was also using --ignore-existing but I removed it and have been testing with an empty source directory just in case that helped but it did not.
#!/bin/bash
# Define source and destination directories
src_dir="/home/user/cloud"
dest_dir="/home/user/testmove"
# Use rsync to move files and create directories recursively
rsync --progress --ignore-existing -avm -f'+ *[mM][pP]3$' -f'+ *[fF][lL][aA][cC]$' -f'+ *[mM][4][aA]$' -f'+ *[wW][aA][vV]$' -f'+ *[aA][iI][fF][fF]$' -f'+ *[pP][cC][mM]$' -f'+ *[aA][aA][cC]$' -f'+ *[oO][gG][gG]$' -f'+ *[aA][lL][aA][cC]$' -f'+ */' -f'- *' "$src_dir"/ "$dest_dir"/ --prune-empty-dirs
I found using the $ stopped any files being copied but I am testing following which seems to work, and Eric's much more comprehensive and flexible version from his answer below:
#!/bin/bash
# Define source and destination directories
src_dir="/home/user/cloud"
dest_dir="/home/user/testmove"
# Use rsync to copy files and create directories recursively
rsync -ahPv \
-f'+ *.[mM][pP]3' \
-f'+ *.[mM][4][aA]' \
-f'+ *.[wW][aA][vV]' \
-f'+ *.[oO][gG][gG]' \
-f'+ *.[wW][mM][aA]' \
-f'+ *.[fF][lL][aA][cC]' \
-f'+ *.[aA][iI][fF][fF]' \
-f'+ *.[pP][cC][mM]' \
-f'+ *.[aA][aA][cC]' \
-f'+ *.[aA][lL][aA][cC]' \
-f'+ */' \
-f'- *' \
"$src_dir"/ "$dest_dir" --prune-empty-dirs --stats -n
# To delete empty directories left in source when transferring:
# find "$src_dir" -depth -type d -empty -delete
There were some elements of the command structure which were conflicting, preventing the desired restrictions, and overlooking the inclusions. The following form provides the desired output.
Note that I have some hardcoded values that you need to remove.
#!/bin/bash
audio=1
video=0
DryRun=""
while [ $# -gt 0 ]
do
case $1 in
--source ) src_dir="$2" ; shift ; shift ;;
--target ) dest_dir="$2" ; shift ; shift ;;
--audio ) audio=1 ; video=0 ; shift ;;
--video ) audio=0 ; video=1 ; shift ;;
--dry ) DryRun="--dry-run --stats" ; shift ;;
* ) echo -e "\n Invalid option used on command line. Only valid options: [ --source {src_dir} | --target {dest_dir} | --audio | --video | --dry ]\n Bye!\n" ; exit 1 ;;
esac
done
# Define source and destination directories
#src_dir="/home/user/cloud"
src_dir="`pwd`/Output"
#dest_dir="/home/user/testmove"
dest_dir="`pwd`/Dupl"
# Define rules for include or exclude
cat >"EXCLUDES.filters" <<"EnDoFiNpUt"
- *
EnDoFiNpUt
#filters="--filter \"merge AUDIO_TYPES.filters\" " ### THIS FORM DOES NOT WORK !!!
if [ ${audio} -eq 1 ]
then
### NOTE: Trailing "$" to match end of string is implicit, unless there is a trailing "*"
cat >"AUDIO_TYPES.filters" <<"EnDoFiNpUt"
+ */
+ *\.[mM][pP][3]
+ *\.[fF][lL][aA][cC]
+ *\.[mM][4][aA]
+ *\.[wW][aA][vV]
+ *\.[aA][iI][fF][fF]
+ *\.[pP][cC][mM]
+ *\.[aA][aA][cC]
+ *\.[oO][gG][gG]
+ *\.[aA][lL][aA][cC]
EnDoFiNpUt
filters="--include-from=AUDIO_TYPES.filters --exclude-from=EXCLUDES.filters"
fi
if [ ${video} -eq 1 ]
then
### NOTE: Trailing "$" to match end of string is implicit, unless there is a trailing "*"
cat >"VIDEO_TYPES.filters" <<"EnDoFiNpUt"
+ */
+ *\.[mM][pP][4]
+ *\.[aA][vV][iI]
+ *\.[mM][kK][vV]
+ *\.[fF][lL][vV]
+ *\.[wW][mM][vV]
+ *\.[mM][oO][vV]
+ *\.[Aa][Vv][Cc][Hh][Dd]
+ *\.[wW][eE][bB][mM]
+ *\.[hH][2][6][4]
+ *\.[mM][pP][eE][gG][4]
+ *\.[Aa][Vv][Cc][Hh][Dd]
EnDoFiNpUt
filters="--include-from=VIDEO_TYPES.filters --exclude-from=EXCLUDES.filters"
fi
# Use rsync to move files and create directories recursively
### NOTE: I prefer long-form options for understanding code at first glance; shortform doesn't do that for me.
#rsync -rlptgoDvP ### shortform for universal options
rsync \
--verbose \
--recursive \
--links \
--owner \
--group \
--perms \
--times \
--devices \
--specials \
--prune-empty-dirs \
--partial \
--progress \
${DryRun}\
${filters} \
"${src_dir}/" "${dest_dir}/"
Log of session output:
building file list ...
3 files to consider
created directory /0__WORK/Dupl
./
DEF.mP4
0 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=1/3)
abc.mp4
0 100% 0.00kB/s 0:00:00 (xfr#2, to-chk=0/3)
sent 196 bytes received 126 bytes 644.00 bytes/sec
total size is 0 speedup is 0.00
Session log for version of script modified per requestor's (saltyeggs) reformulated command (also works, differently, but as expected):
building file list ...
3 files to consider
created directory /0__WORK/Dupl
./
DEF.mP4
abc.mp4
Number of files: 3 (reg: 2, dir: 1)
Number of created files: 3 (reg: 2, dir: 1)
Number of deleted files: 0
Number of regular files transferred: 2
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 128
Total bytes received: 94
sent 128 bytes received 94 bytes 444.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)
I am trying to map 3 samples: SRR14724459, SRR14724473, and a combination of both SRR14724459_SRR14724473.
I have 2 rules that share a similar file type output (.bam), and even tho I am naming their wildcards different, I still get an ambiguity error:
Building DAG of jobs...
AmbiguousRuleException:
Rules map_hybrid and map are ambiguous for the file /gpfs/scratch/hpadre/snakemake_outputs/mapped_dir/SRR14724459_SRR14724473__0.9.bam.
Expected input files:
map_hybrid: /home/hpadre/ngs_artifacts_proj/output_directories/trimmed_dir/SRR14724459_SRR14724473_0.9/SRR14724459_R1_trimmed_SRR14724473_R1_trimmed_0.9.fastq /home/hpadre/ngs_artifacts_proj/output_directories/trimmed_dir/SRR14724459_SRR14724473_0.9/SRR14724459_R2_trimmed_SRR14724473_R2_trimmed_0.9.fastq
map: /home/hpadre/ngs_artifacts_proj/output_directories/trimmed_dir/SRR14724459_SRR14724473__0.9/SRR14724459_SRR14724473__0.9_R1_trimmed.fastq /home/hpadre/ngs_artifacts_proj/output_directories/trimmed_dir/SRR14724459_SRR14724473__0.9/SRR14724459_SRR14724473__0.9_R2_trimmed.fastq
From my Snakefile, here are my variables:
all_samples: ['SRR14724459', 'SRR14724473']
sample1: ['SRR14724459']
sample2: ['SRR14724473']
titration:[0.9]
This is my rule all:
rule all:
expand(MAPPED_DIR + "/{sample}.bam", sample=all_samples),
expand(MAPPED_DIR + "/{sample1}_{sample2}_{titration}.bam", zip, sample1=list_a_titrations, sample2=list_b_titrations, titration=tit_list)
This is my map rule:
rule map:
input:
r1 = TRIMMED_DIR + "/{sample}/{sample}_R1_trimmed.fastq",
r2 = TRIMMED_DIR + "/{sample}/{sample}_R2_trimmed.fastq"
output:
MAPPED_DIR + "/{sample}.bam"
threads: 28
params:
genome = HUMAN_GENOME_DIR
log:
LOG_DIR + "/map/{sample}_map.log"
benchmark:
BENCHMARK_DIR + "/map/{sample}_bwa_benchmark.txt"
wildcard_constraints:
word='[^0-9]*'
shell:
"""
bwa mem -t {threads} {params.genome} {input.r1} {input.r2} 2> {log} | samtools view -hSbo > {output}
"""
This is my map_hybrid:
rule map_hybrid:
input:
r1 = TRIMMED_DIR + "/{sample1}_{sample2}_{titration}/{sample1}_R1_trimmed_{sample2}_R1_trimmed_{titration}.fastq",
r2 = TRIMMED_DIR + "/{sample1}_{sample2}_{titration}/{sample1}_R2_trimmed_{sample2}_R2_trimmed_{titration}.fastq"
output:
MAPPED_DIR + "/{sample1}_{sample2}_{titration}.bam"
threads: 28
params:
genome = HUMAN_GENOME_DIR
log:
LOG_DIR + "/map/{sample1}_{sample2}_{titration}_map.log"
benchmark:
BENCHMARK_DIR + "/map/{sample1}_{sample2}_{titration}_bwa_benchmark.txt"
shell:
"""
set +e
bwa mem -t {threads} {params.genome} {input.r1} {input.r2} 2> {log} | samtools view -hSbo > {output}
exitcode=$?
if [ $exitcode -eq 1 ]
then
exit 1
else
exit 0
fi
"""
The expected input files SHOULD BE as so:
Expected input files:
map_hybrid: /home/hpadre/ngs_artifacts_proj/output_directories/trimmed_dir/SRR14724459_SRR14724473_0.9/SRR14724459_R1_trimmed_SRR14724473_R1_trimmed_0.9.fastq /home/hpadre/ngs_artifacts_proj/output_directories/trimmed_dir/SRR14724459_SRR14724473_0.9/SRR14724459_R2_trimmed_SRR14724473_R2_trimmed_0.9.fastq
map: /home/hpadre/ngs_artifacts_proj/output_directories/trimmed_dir/SRR14724459_R1_trimmed.fastq /home/hpadre/ngs_artifacts_proj/output_directories/trimmed_dir/SRR14724459_R2_trimmed.fastq
and also
/home/hpadre/ngs_artifacts_proj/output_directories/trimmed_dir/SRR14724473_R1_trimmed.fastq /home/hpadre/ngs_artifacts_proj/output_directories/trimmed_dir/SRR14724473_R2_trimmed.fastq
Your rules map and map_hybrid can both produce your desired files, for snakemake they are ambigious rules.
The names of the wildcards is irrelevant, what is relevant is whether the wildcards in both rules can match the same output filepath.
That is the case here.
While rule map_hybrid can produce the output file SRR14724459_R2_trimmed_SRR14724473_R2_trimmed_0.9.fastq where the wildcard matches are
sample1=SRR14724459
sample2=SRR14724473
the rule map can also produce this output with the wildcard match
sample=SRR14724459_R2_trimmed_SRR14724473
To prevent ambiguity you can use the wildcard_constraint, so that the {sample} wildcard only matches strings starting with SRR followed by numbers:
sample='SRR\d+'
Integrated into your rule map:
rule map:
input:
r1 = TRIMMED_DIR + "/{sample}/{sample}_R1_trimmed.fastq",
r2 = TRIMMED_DIR + "/{sample}/{sample}_R2_trimmed.fastq"
output:
MAPPED_DIR + "/{sample}.bam"
threads: 28
params:
genome = HUMAN_GENOME_DIR
log:
LOG_DIR + "/map/{sample}_map.log"
benchmark:
BENCHMARK_DIR + "/map/{sample}_bwa_benchmark.txt"
wildcard_constraints:
word='[^0-9]*',
sample='SRR\d+'
shell:
"""
bwa mem -t {threads} {params.genome} {input.r1} {input.r2} 2> {log} | samtools view -hSbo > {output}
"""
it should resolve the ambiguity.
I'm trying to build a livepatch using kpatch-build insider a Docker container.
So far I am able to see the differences between the original and patched
.o files and creation of new object file.
But create-kpatch-module is not being called and I'm getting an error
indicating "Makefile target missing". I dug through the script, but I'm unable to identify the exact issue.
Can anyone shed some light on what might be wrong? What "Makefile target" is required?
I'm following link at: https://github.com/dynup/kpatch
Upon further inspection:
+ /usr/local/libexec/kpatch/create-kpatch-module /home/xyz/.kpatch/tmp/patch/tmp_output.o /home/xyz/.kpatch/tmp/patch/output.o
+ local to_stdout=1
+ [[ 1 -ge 2 ]]
+ [[ 1 -eq 1 ]]
+ tee -a /home/racker/.kpatch/build.log
+ check_pipe_status create-kpatch-module
+ rc=0
+ [[ 0 = 139 ]]
+ cd /home/xyz/.kpatch/tmp/patch
+ logger
+ KPATCH_BUILD=/source/linux
+ KPATCH_NAME=kpatch-4-4-0-2-modules-0-3-2-3-x86_64
+ KBUILD_EXTRA_SYMBOLS=/usr/lib/modules/4.4.0+2/extra/kpatch/Module.symvers
+ KPATCH_LDFLAGS=
+ make
I have a relatively complex directory structure in my build, from which I would like to create multiple ZIP files. The structure is like this:
+ project root
+ -- build.gradle
+ -- + foo
+ -- bar-1
| + -- bar-1.a.json
| + -- bar-1.a.xml
| + -- bar-1.b.json
| + -- bar-1.b.xml
|
+ -- bar-2
| + -- bar-2.a.json
| + -- bar-2.a.xml
| + -- bar-2.b.json
| + -- bar-2.b.xml
|
+ ...
+ -- bar-n
+ -- bar-n.a.json
+ -- bar-n.a.xml
+ -- bar-n.b.json
+ -- bar-n.b.xml
From this structure, I would like to create ZIP files for each bar-x directory, e.g. from foo/bar-1 directory, a ZIP file should be created in out/foo/bar-1.zip, from foo/bar-2 a ZIP file should be created in out/foo/bar-2.zip and so on
It is important, that new directories might appear later on, so I cannot hard-code the names, Gradle should list the directories in foo on each build.
Can someone give me a sample on how to achieve this?
Something like the following should do:
task zipAll
file('foo').eachDirMatch(~/bar-.*/) { barDir ->
def taskName = "zip${barDir.name.capitalize()}"
task "$taskName"(type: Zip) {
from barDir
destinationDir file('out/foo/')
baseName barDir.name
}
zipAll.dependsOn taskName
}
I need to convert the following to be compatible with a batch script.
cmd.exe cd " + homepath + "\\" + a
mvn archetype:generate -DarchetypeCatalog=file://"+ homepath + "/.m2/repository
1
c
b
c
uuid.toString()
Y
cd " + homepath +"\\"+ a +"\\" + b
vn clean install
"cd " + homepath +"\\" + a
a
cd " + homepath +"\\" + a +"\\" + b + "\\" + b + "-plugin" + "\\target
jar -xvf " + zipDirectory
cmd cd " + homepath +"\\" + a +"\\" + b + "\\" + b + "-plugin" + "\\target\\" + "\\META-INF\\maven\\" + c + "\\" + b + "-plugin
copy pom.xml " + pluginDirectory
cd " + pluginDirectory
rename pom.xml " + b + "-plugin-1.0.0.pom
color 0a
For a line like the first one:
cmd.exe cd " + homepath + "\\" + a
Would the line look like this?
SET homepath = C:\Users\Joe\
SET a = plugins
cmd.exe cd echo %homepath% echo %a%
in batch, there is no string concatenation symbol like on (other) programming languages. You just use the variable instead of the string:
set homepath=%userprofile%
set a=plugins
echo homepath is %homepath% and a is %a%.
cd %homepath%\%a%
(Note: do not use spaces around = with the set command - they would be part of the variable name respectively the value)