I want to capture output which I'm running from the cucumber feature file.
I created one shell script program and placed it in /usr/local/bin/ so it can be accessible from anywhere in system.
abc_qa.sh -
arg=$1
if [[ $arg = 1 ]]
then
echo $(date)
fi
project structure of cucumber -
aruba -
.
├── features
│ ├── support
│ │ └── env.rb
│ └── use_aruba_cucumber.feature
├── Gemfile
Gemfile -
source 'https://rubygems.org'
gem 'aruba', '~> 0.14.2'
env.rb -
require 'aruba/cucumber'
use_aruba_cucumber.feature -
Feature: Cucumber
Scenario: First Run
When I run `bash abc_qa.sh 1`
I want to capture this abc_qa.sh program output in the cucumber itself and compare this date is right or wrong by using any kind of simple test and make this test as a pass.
You can use %x(command) to get the stdout of your command.
You can then use Time.parse to convert "Sat Nov 5 12:04:18 CET 2016" to 2016-11-05 12:04:18 +0100 as Time object, and compare it to Time.now :
require 'time'
time_from_abc_script = Time.parse(%x(bash abc_qa.sh 1))
puts (Time.now-time_from_abc_script).abs < 5 # No more than 5s difference between 2 times
You could use this boolean in any test file you want.
For example :
In features/use_aruba_with_cucumber.feature :
Feature: Cucumber
Scenario: First Run
When I run `bash abc_qa.sh 1`
Then the output should be the current time
and features/step_definitions/time_steps.rb :
require 'time'
Then(/^the output should be the current time$/) do
time_from_script = Time.parse(last_command_started.output)
expect(time_from_script).to be_within(5).of(Time.now)
end
Related
If I have tree output in terminal with less with this function
function tre() {
tree -aC -I '.git|node_modules|bower_components' --dirsfirst "$#" | less -FRNX;
}
, it will scroll 1 line by pressing key each time.
I need a shorcut or command to reach and of file.
If I press "G" the output would be with "...skipping..."
19 │ │ │ └── someotherfile.db
20 │ │ ├── static
...skipping...
62 │ │ ├── user
63 │ │ │ ├── admin.py
How do I get to the end of file with all lines loaded without "...skipping..."?
The issue was with this
less -FRNX;
The last (X) forced the output line by line. So the solution was not to use it
less -FRN;
(Why I use less for tree output)
On screenshot below is the difference between default tree output and output it with less. Same folder, but with less output is with colors, line numbers and directory first.
enter image description here
I want to connect the realigner process with the indel reallignement.
This is the rules:
rule gatk_IndelRealigner:
input:
tumor="mapped_reads/merged_samples/{tumor}.sorted.dup.reca.bam",
normal="mapped_reads/merged_samples/{normal}.sorted.dup.reca.bam",
id="mapped_reads/merged_samples/operation/{tumor}_{normal}.realign.intervals"
output:
"mapped_reads/merged_sample/CoClean/{tumor}.sorted.dup.reca.cleaned.bam",
"mapped_reads/merged_sample/CoClean/{normal}.sorted.dup.reca.cleaned.bam",
params:
genome=config['reference']['genome_fasta'],
mills= config['mills'],
ph1_indels= config['know_phy'],
log:
"mapped_reads/merged_samples/logs/{tumor}.indel_realign_2.log"
threads: 8
shell:
"gatk -T IndelRealigner -R {params.genome} "
"-nt {threads} "
"-I {input.tumor} -I {input.normal} -known {params.ph1_indels} -known {params.mills} -nWayOut .cleaned.bam --maxReadsInMemory 500000 --noOriginalAligmentTags --targetIntervals {input.id} >& {log} "
This is the error:
Not all output files of rule gatk_IndelRealigner contain the same wildcards.
I suppose I need to use also the {tumor}_{normal} but I can't use.
Snakemake:
rule all:
input:expand("mapped_reads/merged_samples/CoClean/{sample}.sorted.dup.reca.cleaned.bam",sample=config['samples']),
expand("mapped_reads/merged_samples/operation/{sample[1][tumor]}_{sample[1][normal]}.realign.intervals", sample=read_table(config["conditions"], ",").iterrows())
config.yml
conditions: "conditions.csv"
conditions.csv
tumor,normal
411,412
Here you can see an example of the code (for testing purpose) gave the same error:
directory
$ tree prova/
prova/
├── condition.csv
├── config.yaml
├── output
│ ├── ABC.bam
│ ├── pippa.bam
│ ├── Pippo.bam
│ ├── TimBorn.bam
│ ├── TimNorm.bam
│ ├── TimTum.bam
│ └── XYZ.bam
└── Snakefile
this is snakemake
$ cat prova/Snakefile
from pandas import read_table
configfile: "config.yaml"
rule all:
input:
expand("{pathDIR}/{sample[1][tumor]}_{sample[1][normal]}.bam", pathDIR=config["pathDIR"], sample=read_table(config["sampleFILE"], " ").iterrows()),
expand("CoClean/{sample[1][tumor]}.bam", sample=read_table(config["sampleFILE"], " ").iterrows()),
expand("CoClean/{sample[1][normal]}.bam", sample=read_table(config["sampleFILE"], " ").iterrows())
rule gatk_RealignerTargetCreator:
input:
"{pathGRTC}/{normal}.bam",
"{pathGRTC}/{tumor}.bam",
output:
"{pathGRTC}/{tumor}_{normal}.bam"
# wildcard_constraints:
# tumor = '[^_|-|\/][0-9a-zA-Z]*',
# normal = '[^_|-|\/][0-9a-zA-Z]*'
run:
call('touch ' + str(wildcard.tumor) + '_' + str(wildcard.normal) + '.bam', shell=True)
rule gatk_IndelRealigner:
input:
t1="output/{tumor}.bam",
n1="output/{normal}.bam",
output:
"CoClean/{tumor}.sorted.dup.reca.cleaned.bam",
"CoClean/{normal}.sorted.dup.reca.cleaned.bam",
log:
"mapped_reads/merged_samples/logs/{tumor}.indel_realign_2.log"
threads: 8
shell:
"gatk -T IndelRealigner -R {params.genome} "
"-nt {threads} -I {input.t1} -I {input.n1} & {log} "
conditions.csv
$ more condition.csv
tumor normal
TimTum TimBorn
XYZ ABC
Pippo pippa
Thanks for any suggestion
I'm not convinced you have to include two input files to the GATK IndelRealigner. Building from that assumption, you can alter the rule to become indifferent to the "type (tumor vs normal)" of file it is process. I read the specs here. Please, if I am wrong, stop reading and correct me.
rule gatk_IndelRealigner:
input:
inputBAM="output/{sampleGATKIR}.bam",
output:
"CoClean/{sampleGATKIR}.sorted.dup.reca.cleaned.bam",
log:
"mapped_reads/merged_samples/logs/{sampleGATKIR}.indel_realign_2.log"
params:
genome="**DONT FORGET TO ADD THIS""
threads: 8
shell:
"gatk -T IndelRealigner -R {params.genome} "
"-nt {threads} -I {input.inputBAM} & {log} "
By changing the rule to be bam-type agnostic (made up word) you gain two advantages, and there is one main disadvantage.
Advantages:
Now we only have a single wild-card
We can run the alignment of each .bam file independently, which with a devoted CPU should hopefully make things faster.
Disadvantage:
We are now likely putting two copies of the genome onto memory somewhere, since the threads are now being run as separate processes, no more memory sharing of the genome file. (In my previous position, hardware availability wasn't typically an issue, so I heavily am biased towards splitting everything up)
The reason I think that the GATK documentation has it setup to accept multiple 'bam' files is because if you are just using it as a 1-off call you want to list all the files at the same time. We are not needing that since we are automating the call process. We're indifferent to 1 call or 100 calls.
I'm trying to debug the following build error in our CI where "A depends on B which can't build because it depends on C." I'm building my data service which doesn't directly depend on kafkaAvailMonitor.go which makes this error hard to trace. In other words:
data (what I'm building) depends on (?) which depends on
kafkaAvailMonitor.go
It may seem trivial to fix for a developer they just do "go get whatever" but I can't do that as part of the release process - I have to find the person that added the dependency and ask them to fix it.
I'm aware that there are tools to visualize the dependency tree and other more sophisticated build systems, but this seems like a pretty basic issue: is there any way I can view the full dependency tree to see what's causing the build issue?
go build -a -v
../../../msgq/kafkaAvailMonitor.go:8:2: cannot find package
"github.com/Shopify/sarama/tz/breaker" in any of:
/usr/lib/go-1.6/src/github.com/Shopify/sarama/tz/breaker (from $GOROOT)
/home/jenkins/go/src/github.com/Shopify/sarama/tz/breaker (from $GOPATH)
/home/jenkins/vendor-library/src/github.com/Shopify/sarama/tz/breaker
/home/jenkins/go/src/github.com/Shopify/sarama/tz/breaker
/home/jenkins/vendor-library/src/github.com/Shopify/sarama/tz/breaker
When using modules you may be able to get what you need from go mod graph.
usage: go mod graph
Graph prints the module requirement graph (with replacements applied)
in text form. Each line in the output has two space-separated fields: a module
and one of its requirements. Each module is identified as a string of the form
path#version, except for the main module, which has no #version suffix.
I.e., for the original question, run go mod graph | grep github.com/Shopify/sarama then look more closely at each entry on the left-hand side.
if the following isn't a stack trace what is it?
It is the list of path where Go is looking for your missing package.
I have no idea who is importing kafkaAvailMonitor.go
It is not "imported", just part of your sources and compiled.
Except it cannot compile, because it needs github.com/Shopify/sarama/tz/breaker, which is not in GOROOT or GOPATH.
Still, check what go list would return on your direct package, to see if kafkaAvailMonitor is mentioned.
go list can show both the packages that your package directly depends, or its complete set of transitive dependencies.
% go list -f '{{ .Imports }}' github.com/davecheney/profile
[io/ioutil log os os/signal path/filepath runtime runtime/pprof]
% go list -f '{{ .Deps }}' github.com/davecheney/profile
[bufio bytes errors fmt io io/ioutil log math os os/signal path/filepath reflect run
You can then script go list in order to list all dependencies.
See this bash script for instance, by Noel Cower (nilium)
#!/usr/bin/env bash
# Usage: lsdep [PACKAGE...]
#
# Example (list github.com/foo/bar and package dir deps [the . argument])
# $ lsdep github.com/foo/bar .
#
# By default, this will list dependencies (imports), test imports, and test
# dependencies (imports made by test imports). You can recurse further by
# setting TESTIMPORTS to an integer greater than one, or to skip test
# dependencies, set TESTIMPORTS to 0 or a negative integer.
: "${TESTIMPORTS:=1}"
lsdep_impl__ () {
local txtestimps='{{range $v := .TestImports}}{{print . "\n"}}{{end}}'
local txdeps='{{range $v := .Deps}}{{print . "\n"}}{{end}}'
{
go list -f "${txtestimps}${txdeps}" "$#"
if [[ -n "${TESTIMPORTS}" ]] && [[ "${TESTIMPORTS:-1}" -gt 0 ]]
then
go list -f "${txtestimps}" "$#" |
sort | uniq |
comm -23 - <(go list std | sort) |
TESTIMPORTS=$((TESTIMPORTS - 1)) xargs bash -c 'lsdep_impl__ "$#"' "$0"
fi
} |
sort | uniq |
comm -23 - <(go list std | sort)
}
export -f lsdep_impl__
lsdep_impl__ "$#"
I just want to mention here that go mod why can also help. Anyway you cannot get and display the whole tree. But you can trace back one single branch of a child dependency until its parent root.
Example:
$ go mod why github.com/childdep
# github.com/childdep
github.com/arepo.git/service
github.com/arepo.git/service.test
github.com/anotherrepo.git/mocks
github.com/childdep
That means, you have imported 'childdep' finally in 'anotherrepo.git/mocks'.
can try this https://github.com/vc60er/deptree
redis git:(master) go mod graph | deptree -d 3
package: github.com/go-redis/redis/v9
dependence tree:
┌── github.com/cespare/xxhash/v2#v2.1.2
├── github.com/dgryski/go-rendezvous#v0.0.0-20200823014737-9f7001d12a5f
├── github.com/fsnotify/fsnotify#v1.4.9
│ └── golang.org/x/sys#v0.0.0-20191005200804-aed5e4c7ecf9
├── github.com/nxadm/tail#v1.4.8
│ ├── github.com/fsnotify/fsnotify#v1.4.9
│ │ └── golang.org/x/sys#v0.0.0-20191005200804-aed5e4c7ecf9
│ └── gopkg.in/tomb.v1#v1.0.0-20141024135613-dd632973f1e7
├── github.com/onsi/ginkgo#v1.16.5
│ ├── github.com/go-task/slim-sprig#v0.0.0-20210107165309-348f09dbbbc0
│ │ ├── github.com/davecgh/go-spew#v1.1.1
│ │ └── github.com/stretchr/testify#v1.5.1
│ │ └── ...
The above answer still doesn't show me a dependency tree so I've taken the time to write a Python script to do what I need - hopefully that helps other people.
The issue with the above solution (the others proposed like go list) is that it only tells me the top level. They don't "traverse the tree." This is the output I get - which doesn't help any more than what go build gives me.
.../npd/auth/
.../mon/mlog
.../auth/service
This is what I'm trying to get - I know that auth is broken (top) and that breaker is broken (bottom) from go build but I have no idea what's in between - my script below gives me this output.
.../npd/auth/
.../npd/auth/service
.../npd/auth/resource
.../npd/auth/storage
.../npd/middleware
.../npd/metrics/persist
.../npd/kafka
.../vendor-library/src/github.com/Shopify/sarama
.../vendor-library/src/github.com/Shopify/sarama/vz/breaker
My Python script:
import subprocess
import os
folder_locations=['.../go/src','.../vendor-library/src']
def getImports(_cwd):
#When the commands were combined they overflowed the bugger and I couldn't find a workaround
cmd1 = ["go", "list", "-f", " {{.ImportPath}}","./..."]
cmd2 = ["go", "list", "-f", " {{.Imports}}","./..."]
process = subprocess.Popen(' '.join(cmd1), cwd=_cwd,shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out1, err = process.communicate()
process = subprocess.Popen(' '.join(cmd2), cwd=_cwd,shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out2, err = process.communicate()
out2clean=str(out2).replace("b'",'').replace('[','').replace(']','').replace("'",'')
return str(out1).split('\\n'),out2clean.split('\\n')
def getFullPath(rel_path):
for i in folder_locations:
if os.path.exists(i+'/'+rel_path):
return i+'/'+rel_path
return None
def getNextImports(start,depth):
depth=depth+1
indent = '\t'*(depth+1)
for i,val in enumerate(start.keys()):
if depth==1:
print (val)
out1,out2=getImports(val)
noDeps=True
for j in out2[i].split(' '):
noDeps=False
_cwd2=getFullPath(j)
new_tree = {_cwd2:[]}
not_exists = (not _cwd2 in alltmp)
if not_exists:
print(indent+_cwd2)
start[val].append(new_tree)
getNextImports(new_tree,depth)
alltmp.append(_cwd2)
if noDeps:
print(indent+'No deps')
_cwd = '/Users/.../npd/auth'
alltmp=[]
start_root={_cwd:[]}
getNextImports(start_root,0)
The dependency of a Go project is a directional graph. This graph consists of multiple layers, ranging from several to hundreds or thousands. Here is an dependency graph of redis. The cascaded tree can be difficult to understand due to the presence of many duplicated subtrees. To make the layout easier to view, the tree can be flattened via gomoddeps to fit the width of the screen.
tzhang:~/github.com/redis/go-redis$ go mod graph | gomoddeps
├─ github.com/redis/go-redis/v9
│ └─ dependencies
│ ├─ github.com/bsm/ginkgo/v2#v2.5.0
│ ├─ github.com/bsm/gomega#v1.20.0
│ ├─ github.com/cespare/xxhash/v2#v2.2.0
│ ├─ github.com/davecgh/go-spew#v1.1.1
│ ├─ github.com/dgryski/go-rendezvous#v0.0.0-20200823014737-9f7001d12a5f
│ ├─ github.com/pmezard/go-difflib#v1.0.0
│ ├─ github.com/stretchr/testify#v1.8.1
│ └─ gopkg.in/yaml.v3#v3.0.1
│
├─ github.com/bsm/ginkgo/v2#v2.5.0
│ └─ dependents
│ └─ github.com/redis/go-redis/v9
│
├─ github.com/bsm/gomega#v1.20.0
│ └─ dependents
│ └─ github.com/redis/go-redis/v9
│
├─ github.com/cespare/xxhash/v2#v2.2.0
│ └─ dependents
│ └─ github.com/redis/go-redis/v9
│
├─ github.com/davecgh/go-spew#v1.1.1
│ └─ dependents
│ ├─ github.com/redis/go-redis/v9
│ ├─ github.com/stretchr/testify#v1.8.1
│ ├─ github.com/stretchr/testify#v1.8.0
│ └─ github.com/stretchr/objx#v0.4.0
...
I seem to be having some issues with Rake and my Cucumber features file.
Here is the command in Jenkins. It specifys the Rake file that I have put directly in the ./features directory so the testjson.feature is right there.
/usr/local/bin/rake --rakefile /home/robm/code/BDD/practise-tests/testtq/features/Rakefile
Rakefile looks like this:
require 'cucumber/rake/task'
Cucumber::Rake::Task.new :features do |t|
t.cucumber_opts = '*.feature'
end
It's pretty straightforward but in the console output from Jenkins I get:
Feature: Validate DUT JSON
JSON should be evaluated for all routes in API
All API routes should return valid JSON
If JSON is invalid for one or more route in API it has DUT failed
Scenario Outline: Validate JSON # testJson.feature:6
Given there is a DUT with "<input>" and "<un>" and "<pw>" # testJson.feature:7
When the JsonTest code is run # testJson.feature:8
Then the output should be "<output>" # testJson.feature:9
Examples:
| input | un | pw | output |
| 172.168.101.139 | username | password | CHECK |
| 172.168.101.214 | username | password | CHECK |
2 scenarios (2 undefined)
6 steps (6 undefined)
0m0.007s
which tells me it's not finding the feature file, right?
When I go to the directory above the feature file and run Cucumber the test works!
So, obviously I have an error with my Rake file. Any ideas?
I put this in as my Rake working directory
/home/robm/code/BDD/practise-tests/testtq
then specified the feature file
"./features/testJson.feature"
that got it to work not sure why the other didn't work.
In an effort to use Cucumber for a command-line script, I've installed the aruba gem as per the instructions provided. It's in my Gemfile, I can verify that the correct version is installed and I've included
require 'aruba/cucumber'
in 'features/env.rb'
In order to ensure it works, I wrote the following scenario:
#announce
Scenario: Testing cucumber/aruba
Given a blank slate
Then the output from "ls -la" should contain "drw"
assuming the thing should fail.
It does fail, but it fails for the wrong reasons:
#announce
Scenario: Testing cucumber/aruba
Given a blank slate
Then the output from "ls -la" should contain "drw"
You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.[] (NoMethodError)
features/dataloader.feature:9:in `Then the output from "ls -la" should contain "drw"'
Anyone have any ideas why this isn't working? This seems to be very basic aruba behavior.
You are missing a 'When' step - the aruba "output should contain" step requires the command to have already run (it does not run it itself, it only looks it up).
#announce
Scenario: Testing cucumber/aruba
Given a blank slate
When I run `ls -la`
Then the output from "ls -la" should contain "drw"
This produces, on my machine:
#announce
Scenario: Testing cucumber/aruba # features/test_aruba.feature:8
When I run `ls -la` # aruba-0.4.11/lib/aruba/cucumber.rb:56
$ cd /Users/d.chetlin/dev/mine/ladder/tmp/aruba
$ ls -la
total 0
drwx------ 2 d.chetlin staff 68 Feb 15 23:38 .
drwx------ 7 d.chetlin staff 238 Feb 15 23:38 ..
Then the output from "ls -la" should contain "drw" # aruba-0.4.11/lib/aruba/cucumber.rb:86
1 scenario (1 passed)
2 steps (2 passed)
0m0.465s