schema file at path is invalid because no version is specified - yaml

I run my dbt command in a loop and save its contents to a .yml file. This works well and generates a schema in my .yml file accurately:
for file in models/l30_mart/*.sql; do
table=$(basename "$file" .sql)
dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > test.yml
done
However, in the example above, I am just saving the test.yml file in the root directory. When I try to save the file in another path for example models/l30_mart/test.yml, it doesn't work:
for file in models/l30_mart/*.sql; do
table=$(basename "$file" .sql)
dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > models/l30_mart/test.yml
done
In this case, when I open the test.ymlfile, I see this:
12:06:42 Running with dbt=1.0.1
12:06:43 Encountered an error:
Compilation Error
The schema file at models/l30_mart/test.yml is invalid because no version is specified. Please consult the documentation for more information on schema.yml syntax:
https://docs.getdbt.com/docs/schemayml-files
It's probably a syntax mistake. Do I need any quotes to specify the path? or brackets? Why can't I save the .yml file in the same folder? Similarly, something like this and try to save different files with the tablename, it also doesn't work:
for file in models/l30_mart/*.sql; do
table=$(basename "$file" .sql)
dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > models/l30_mart/$table.yml
done
In this case, the files either have this output:
20:39:44 Running with dbt=1.0.1
20:39:45 Encountered an error:
Compilation Error
The schema file at models/l30_mart/firsttable.yml is invalid because no version is specified. Please consult the documentation for more information on schema.yml syntax:
https://docs.getdbt.com/docs/schemayml-files
or this (eg in the second tablename.yml file):
20:39:48 Running with dbt=1.0.1
20:39:49 Encountered an error:
Parsing Error
Error reading dbt_4flow: l30_mart/firstablename.yml - Runtime Error
Syntax error near line 2
------------------------------
1 | 20:39:44 Running with dbt=1.0.1
2 | 20:39:45 Encountered an error:
3 | Compilation Error
4 | The schema file at models/l30_mart/firsttablename.yml is invalid because no version is specified. Please consult the documentation for more information on schema.yml syntax:
5 |
Raw Error:
------------------------------
mapping values are not allowed in this context
in "<unicode string>", line 2, column 31
Note that the secondtablename.yml mentions the firsttablename.yml.

Related

Bash script syntax error: operand expected (error token is "'mwaa_bucket'")

what I need to do is to read the key variable from the yaml file and echo the value. Therefore I wrote the following script (copys3.sh).
#!/bin/bash
echo '-------- Listing Git Cloned Repository --------------'
ls -ltr ${WORKSPACE}
echo '-------- Copying Airflow DAGS to S3 --------------'
echo ${action['mwaa_bucket']}
echo ${action['user_name']}
Here's the yaml file:
block1:
message: "Deploy DAGs to S3"
action-type: business-logic
agent-name: default
mwaa_bucket: "com-mwaa-test"
user_name: "dev"
binary-location: "ApplicationConfiguration/copyS3.sh"
Both script as well as config.yaml are located into the same folder "Application Configuration"
The scripts produces the following error which I do not understand
./ApplicationConfiguration/copyS3.sh: line 6: 'mwaa_bucket': syntax error: operand expected (error token is "'mwaa_bucket'")
./ApplicationConfiguration/copyS3.sh: line 7: 'user_name': syntax error: operand expected (error token is "'user_name'")

Gitlab CI not displaying complete output in terminal

I have created a pipeline which performs ansible-lint on $CI_PROJECT_DIR. The problem is the complete output is not shown in UI as compared to running on my local machine.
You can notice the difference in output for both.
Below is the output from my local machine (Ubuntu with ansible-lint installed)
**ansible-lint create-dir.yaml -v**
WARNING Overriding detected file kind 'yaml' with 'playbook' for given positional argument: create-dir.yaml
INFO Executing syntax check on create-dir.yaml (0.31s)
WARNING Listing 1 violation(s) that are fatal
syntax-check: 'file' is not a valid attribute for a Play
create-dir.yaml:4:3 [WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
ERROR! 'file' is not a valid attribute for a Play
The error appears to be in '/tmp/create-dir.yaml': line 4, column 3, but may be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
---
- name: Create a directory if it does not exist
^ here
Finished with 1 failure(s), 0 warning(s) on 1 files.
Below is output from Gitlab CI :
$ find ./ -not \( -name "*.ansible-lint" -o -name ".gitlab-ci.yml" \) \( -name "*yml" -o -name "*yaml" \) | xargs ansible-lint -v
WARNING Overriding detected file kind 'yaml' with 'playbook' for given positional argument: ./ansible/create-dir.yaml
INFO Executing syntax check on ansible/create-dir.yaml (0.39s)
Cleaning up project directory and file based variables 00:01
Job succeeded
I would like to know why there is difference in output and how to print the complete message on Gitlab CI
Job logs in GitLab CI/CD have a limited length. It's designed for operational reasons, as the jobs can create arbitrary outputs so one could - by mistake or by purpose - output even gigabytes or terabytes of text.
But if you scroll up in the output, you have the abillity to view the full logs by pressing 'Complete Raw'.

Jmeter run test plan from command line with external link to properties

Problem: In debug mode I can run test plan with no issues, but in none-gui got problem while parsing access to file, looks like properties file.
Here are details:
jmeter.logs
2018-08-31 13:26:30,831 ERROR o.a.j.t.JMeterThread: Test failed!
java.lang.IllegalArgumentException: File 2 must exist and be readable at org.apache.jmeter.services.FileServer.createBufferedReader(FileServer.java:424) ~[ApacheJMeter_core.jar:4.0 r1823414]
My Test Plan:
enter image description here
External property file contain link to other file with list of users.
enter image description here
Here is my Test Plan and property file reader sampler.
Both files SAT.properties, SAT.users.22315010.csv have full access
-rwxrwx---+ 1 Domain Users 212 Aug 31 12:41 SAT.properties
-rwxrwx---+ 1 Domain Users 123 Aug 30 13:20 SAT.users.22315010.csv
Cmd command used to run script:
jmeter -n -t D:\PW\Automation\Jmeter\Reports\Reports_Random.jmx -Jusers=1 -l D:\PW\Automation\Jmeter\Reports\Results\JResults.jtl
The answer lies in Property File Reader comments, you need to escape your path:
If you want c:\abc , You need to have c:\abc as value in your property file as \ is used for escape sequence in Java
I don't think using this "property file reader" makes sense as the functionality of reading an external properties file is available right out of the box in JMeter.
Instead of using the plugin you can just pass the path to your SAT.properties file using -q command-line argument like:
jmeter -q d:\PW\Automation\JMeter\Reports\Properties\SAT.properties -n -t D:\PW\Automation\Jmeter\Reports\Reports_Random.jmx -Jusers=1 -l D:\PW\Automation\Jmeter\Reports\Results\JResults.jtl
More information:
Configuring JMeter
Apache JMeter Properties Customization Guide

Getting GATK argument error and dont understand?

Hello bash programmers, I am using GATK and trying to loop through my bam files and do local realignment using my target_intervals and known indels. Below is my code I am trying. I am hoping someone can help with the error and correct my code.
# do the local realignment.
echo "local realignment..."
for file in `ls -d adp/map/*marked_duplicates.bam`
do
java -jar ~/software/GenomeAnalysisTK-3.3-0/GenomeAnalysisTK.jar \
-T IndelRealigner \
-R ~/flybase/fb-r5.57/dmel-all-chromosome-r5.57.fasta \
-I $file \
-known adp/map/*indel_intervals.vcf \
-targetIntervals adp/map/*target_intervals.list \
-o ${file}_realigned_reads.bam
done
wait
# Create a new index file.
echo "indexing the realigned bam file..."
for file in `ls -d adp/map/*realigned_reads.bam`
do
~/software/samtools-1.2/samtools index $file
done
ERROR: when looking this up, it appears to be a coding issue, and I am not seeing it.
##### ERROR ------------------------------------------------------------------------------------------
##### ERROR A USER ERROR has occurred (version 3.3-0-g37228af):
##### ERROR
##### ERROR This means that one or more arguments or inputs in your command are incorrect.
##### ERROR The error message below tells you what is the problem.
##### ERROR
##### ERROR If the problem is an invalid argument, please check the online documentation guide
##### ERROR (or rerun your command with --help) to view allowable command-line arguments for this tool.
##### ERROR
##### ERROR Visit our website and forum for extensive documentation and answers to
##### ERROR commonly asked questions http://www.broadinstitute.org/gatk
##### ERROR
##### ERROR Please do NOT post this error to the GATK forum unless you have really tried to fix it yourself.
##### ERROR
##### ERROR MESSAGE: Invalid argument value 'adp/map/360M_F_L002.recal.bam.sorted.bam_marked_duplicates.bam_target_intervals.list' at position 10.
##### ERROR Invalid argument value 'adp/map/517_F_L002.recal.bam.sorted.bam_marked_duplicates.bam_target_intervals.list' at position 11.
##### ERROR Invalid argument value 'adp/map/517M_F_L002.recal.bam.sorted.bam_marked_duplicates.bam_target_intervals.list' at position 12.
##### ERROR Invalid argument value 'adp/map/900_F_L002.recal.bam.sorted.bam_marked_duplicates.bam_target_intervals.list' at position 13.
##### ERROR Invalid argument value 'adp/map/900M_F_L002.recal.bam.sorted.bam_marked_duplicates.bam_target_intervals.list' at position 14
.
At least part of the problem is the * in your commands. GATK doesn't deal well with globs. To specify multiple values to an argument, specify the argument multiple times.
i.e. instead of
-known adp/map/*indel_intervals.vcf
you need to specify each file with a separate argument
-known adp/map/first_file.indel_intervals.vcf
-known adp/map/second_file.indel_intervals.vcf
There may be other issues as well. For instance, I'm not certain that -targetIntervals can take multiple files as input. Also, that's very old version of gatk, you might want to upgrade to 3.8.

Vowpal Wabbit: make test failing for me at test 59

For some reason I am having trouble with the make test statement while installing ```Vowpal Wabbit``. I am getting the following error:
RunTests: test 59: '/usr/bin/timeout 20 ../vowpalwabbit/vw -d train-sets/argmax_data -k -c --passes 20 --search_rollout oracle --search_alpha 1e-8 --search_task argmax --search 2 --holdout_off' failed (exitcode=1)
Anyone have a clue what this could be?
Just run the command which failed (in single quotes) directly from the test directory, and the reason would become obvious.
It is missing data file:
Reading datafile = test/train-sets/argmax_data
can't open: test/train-sets/argmax_data, error = No such file or directory
vw: std::exception
The issue was introduced in a recent check-in and should soon be fixed (hopefully).
Update (2014-05-31): fixed in the most recent commit.

Resources