Hello bash programmers, I am using GATK and trying to loop through my bam files and do local realignment using my target_intervals and known indels. Below is my code I am trying. I am hoping someone can help with the error and correct my code.
# do the local realignment.
echo "local realignment..."
for file in `ls -d adp/map/*marked_duplicates.bam`
do
java -jar ~/software/GenomeAnalysisTK-3.3-0/GenomeAnalysisTK.jar \
-T IndelRealigner \
-R ~/flybase/fb-r5.57/dmel-all-chromosome-r5.57.fasta \
-I $file \
-known adp/map/*indel_intervals.vcf \
-targetIntervals adp/map/*target_intervals.list \
-o ${file}_realigned_reads.bam
done
wait
# Create a new index file.
echo "indexing the realigned bam file..."
for file in `ls -d adp/map/*realigned_reads.bam`
do
~/software/samtools-1.2/samtools index $file
done
ERROR: when looking this up, it appears to be a coding issue, and I am not seeing it.
##### ERROR ------------------------------------------------------------------------------------------
##### ERROR A USER ERROR has occurred (version 3.3-0-g37228af):
##### ERROR
##### ERROR This means that one or more arguments or inputs in your command are incorrect.
##### ERROR The error message below tells you what is the problem.
##### ERROR
##### ERROR If the problem is an invalid argument, please check the online documentation guide
##### ERROR (or rerun your command with --help) to view allowable command-line arguments for this tool.
##### ERROR
##### ERROR Visit our website and forum for extensive documentation and answers to
##### ERROR commonly asked questions http://www.broadinstitute.org/gatk
##### ERROR
##### ERROR Please do NOT post this error to the GATK forum unless you have really tried to fix it yourself.
##### ERROR
##### ERROR MESSAGE: Invalid argument value 'adp/map/360M_F_L002.recal.bam.sorted.bam_marked_duplicates.bam_target_intervals.list' at position 10.
##### ERROR Invalid argument value 'adp/map/517_F_L002.recal.bam.sorted.bam_marked_duplicates.bam_target_intervals.list' at position 11.
##### ERROR Invalid argument value 'adp/map/517M_F_L002.recal.bam.sorted.bam_marked_duplicates.bam_target_intervals.list' at position 12.
##### ERROR Invalid argument value 'adp/map/900_F_L002.recal.bam.sorted.bam_marked_duplicates.bam_target_intervals.list' at position 13.
##### ERROR Invalid argument value 'adp/map/900M_F_L002.recal.bam.sorted.bam_marked_duplicates.bam_target_intervals.list' at position 14
.
At least part of the problem is the * in your commands. GATK doesn't deal well with globs. To specify multiple values to an argument, specify the argument multiple times.
i.e. instead of
-known adp/map/*indel_intervals.vcf
you need to specify each file with a separate argument
-known adp/map/first_file.indel_intervals.vcf
-known adp/map/second_file.indel_intervals.vcf
There may be other issues as well. For instance, I'm not certain that -targetIntervals can take multiple files as input. Also, that's very old version of gatk, you might want to upgrade to 3.8.
Related
I run my dbt command in a loop and save its contents to a .yml file. This works well and generates a schema in my .yml file accurately:
for file in models/l30_mart/*.sql; do
table=$(basename "$file" .sql)
dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > test.yml
done
However, in the example above, I am just saving the test.yml file in the root directory. When I try to save the file in another path for example models/l30_mart/test.yml, it doesn't work:
for file in models/l30_mart/*.sql; do
table=$(basename "$file" .sql)
dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > models/l30_mart/test.yml
done
In this case, when I open the test.ymlfile, I see this:
12:06:42 Running with dbt=1.0.1
12:06:43 Encountered an error:
Compilation Error
The schema file at models/l30_mart/test.yml is invalid because no version is specified. Please consult the documentation for more information on schema.yml syntax:
https://docs.getdbt.com/docs/schemayml-files
It's probably a syntax mistake. Do I need any quotes to specify the path? or brackets? Why can't I save the .yml file in the same folder? Similarly, something like this and try to save different files with the tablename, it also doesn't work:
for file in models/l30_mart/*.sql; do
table=$(basename "$file" .sql)
dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > models/l30_mart/$table.yml
done
In this case, the files either have this output:
20:39:44 Running with dbt=1.0.1
20:39:45 Encountered an error:
Compilation Error
The schema file at models/l30_mart/firsttable.yml is invalid because no version is specified. Please consult the documentation for more information on schema.yml syntax:
https://docs.getdbt.com/docs/schemayml-files
or this (eg in the second tablename.yml file):
20:39:48 Running with dbt=1.0.1
20:39:49 Encountered an error:
Parsing Error
Error reading dbt_4flow: l30_mart/firstablename.yml - Runtime Error
Syntax error near line 2
------------------------------
1 | 20:39:44 Running with dbt=1.0.1
2 | 20:39:45 Encountered an error:
3 | Compilation Error
4 | The schema file at models/l30_mart/firsttablename.yml is invalid because no version is specified. Please consult the documentation for more information on schema.yml syntax:
5 |
Raw Error:
------------------------------
mapping values are not allowed in this context
in "<unicode string>", line 2, column 31
Note that the secondtablename.yml mentions the firsttablename.yml.
This is my setup:
I use AWS Batch that is running a custom Docker image
The startup.sh file is an entrypoint script that is reading the nth line of a text file and copying it from s3 into the docker.
For example, if the first line of the .txt file is 'Startup_00001/ Startup_000018 Startup_000019', the bash script reads this line, and uses a for loop to copy them over.
This is part of my bash script:
STARTUP_FILE_S3_URL=s3://cmtestbucke/Config/
Startup_FileNames=$(sed -n ${LINE}p file.txt)
for i in ${Startup_FileNames}
do
Startup_FileURL=${STARTUP_FILE_S3_URL}$i
echo $Startup_FileURL
aws s3 cp ${Startup_FileURL} /home/CM_Projects/ &
done
Here is the log output from aws:
s3://cmtestbucke/Config/Startup_000017
s3://cmtestbucke/Config/Startup_000018
s3://cmtestbucke/Config/Startup_000019
Completed 727 Bytes/727 Bytes (7.1 KiB/s) with 1 file(s) remaining download: s3://cmtestbucke/Config/Startup_000018 to Data/Config/Startup_000018
Completed 731 Bytes/731 Bytes (10.1 KiB/s) with 1 file(s) remaining download: s3://cmtestbucke/Config/Startup_000017 to Data/Config/Startup_000017
fatal error: *An error occurred (404) when calling the HeadObject operation: Key
"Config/Startup_000019 " does not exist.*
My s3 bucket certainly contains the object s3://cmtestbucke/Config/Startup_000019
I noticed this happens regardless of filenames. The last iteration always gives this error.
I tested this bash logic locally with the same aws commands. It copies all 3 files.
Can someone please help me figure out what is wrong here?
The problem was with EOL of the text file. It was set to Windows(CR LF). The docker image is running Ubuntu which caused the error. I changed the EOL to Unix(LF). The problem was solved.
featureCounts was called under minconda in Linux subsystem on a Windows 10 computer.
featureCounts -a /mnt/d/.../__.txt -F SAF -readExtensions3 200 -o ___.tsv -O file1.bam file2.bam file3p.bam file4.bam file5.bam file6.bam file7.bam file8.bam
This always results in an error message
featureCounts: invalid option -- 'r'
Version 2.0.1
Usage: featureCounts [options] -a <annotation_file> -o <output_file> input_file1 [input_file2] ...
## Mandatory arguments:
-a <string> Name of an annotation file. GTF/GFF format by default. See...
And then reprinting the required and optional arguments for featureCounts function.
Does anyone know what does the error message "invalid option --'r'" mean? And how can I fix it?
Is there any difference between calling featureCounts in command prompt (or Terminal on Mac) and calling it in Linux/miniconda3?
It seems that you mistyped option name, it should be --readExtension3 with two dashes in front of it and without 's' in the end. I had a similar problem with --fraction which led me here!
test result states for loop contains no keywords
I tried to make the syntax as much to the user guide as possible. It won't let me add a ':' before FOR and it says it is not necessary in the new version.
*** Settings ***
Test Teardown Close Browser
Library SeleniumLibrary
Library Collections
Library RequestsLibrary
*** Test Cases ***
links
Open Browser http://neuromorpho.org/neuron_info.jsp?neuron_name=DD13-10-c5-1 safari
Maximize Browser Window
${AllLinksCount}= Get Element Count //a
Log ${AllLinksCount}
#{LinkItems} Create List
FOR ${INDEX} IN RANGE 1 ${AllLinksCount}
\ \ Log ${INDEX}
\ \ ${lintext}= Get Text xpath= (//a) [${INDEX}]
\ \ Log ${lintext}
\ \ ${linklength} Get Length ${lintext}
\ \ Run Keyword If ${linklength} >1 Append To List ${LinkItems} ${lintext}
${LinkSize}= Get Length ${LinkItems}
Log ${LinkSize}
FOR ${ELEMENT} IN #{LinkItems}
\ \ Log ${ELEMENT}
Close Browser
I want a list of links which i can then check for response, however the error message states "FOR loop contains no keywords." help :(
You have to terminate the FOR block with the END command (if you changed the file)
Below is the text from the FAQ on our Wiki:
Q: In the newest versions of RIDE (1.7.4) and with Robot Framework 3.1.2, when I edit a Test Suite having : FOR, then, when is executed, appears the following error: FOR loop contains no keywords.. How to fix this?
A: Robot Framework is tolerant to the old : FOR format, and the test suite can be executed correctly. However, when the file is edited in RIDE, it looses the old style formatting, so you must add the terminating END.
'\' is depreciated in robot 3.*
The error you are seeing is self explanatory Your For loop structure is incorrect
You are missing END cause
FOR ${INDEX} IN RANGE 1 ${AllLinksCount}
Log ${INDEX}
${lintext}= Get Text xpath= (//a) [${INDEX}]
Log ${lintext}
${linklength} Get Length ${lintext}
Run Keyword If ${linklength} >1 Append To List ${LinkItems} ${lintext}
END
${LinkSize}= Get Length ${LinkItems}
Log ${LinkSize}
FOR ${ELEMENT} IN #{LinkItems}
Log ${ELEMENT}
END
So it should work.
:FOR ${INDEX} IN RANGE 1 ${value}+1
\ Wait Until Page Contains Element ${example} 10
\ Wait Until Page Contains Element ${example} 10
For some reason I am having trouble with the make test statement while installing ```Vowpal Wabbit``. I am getting the following error:
RunTests: test 59: '/usr/bin/timeout 20 ../vowpalwabbit/vw -d train-sets/argmax_data -k -c --passes 20 --search_rollout oracle --search_alpha 1e-8 --search_task argmax --search 2 --holdout_off' failed (exitcode=1)
Anyone have a clue what this could be?
Just run the command which failed (in single quotes) directly from the test directory, and the reason would become obvious.
It is missing data file:
Reading datafile = test/train-sets/argmax_data
can't open: test/train-sets/argmax_data, error = No such file or directory
vw: std::exception
The issue was introduced in a recent check-in and should soon be fixed (hopefully).
Update (2014-05-31): fixed in the most recent commit.