My pipeline includes two steps: first is to check code from git and another is the do the build.
See my pipeline as below:
stages {
stage('Sync code from GitLab'){
steps {
echo "Step 1: Sync code"
ws(dir: '/home/tOmMy/jks_node/workspace/test') {
checkout([$class: 'GitSCM', branches: [[name: '*/release_br']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'xxx', url: 'yyy']]])
sh '''
pwd
ls -lt
'''
}
}
}
stage('Build'){
steps {
echo "Step 2: Build"
...
}
}
}
However, when the Jenkins checkout code from git, I saw many directories:
[tOmMy#ubuntu workspace]$ tree
.
|-- xx
| `-- yy
| |-- zz
| |-- zz#2
| |-- zz#2#tmp
| `-- zz#tmp
|-- test
|-- test#2
|-- test#2#tmp
`-- test#tmp
Where xx, yy, zz is the folder and job name in my Jenkins' server.
The questions are:
1. which folder does the Jenkins checkout the source code? I compare test, test#2, xx/yy/zz, xx/yy/zz2, all of them have the same files. Why does jenkins check out code to such many places?
How can I get the actual Jenkins' checkout found in stage('Build')?
Usually a declarative pipeline(if you are reading it from a repository) already check out the code itself in the "Checkout SCM" inside the job/build workspace folder.
If you allocate in another workspace (command ws) and check the code again you have 2 versions.
You should only do the build the checkout is already done in normal pipelines (if you use the jenkinsfile from a repo)
Related
As my application grows I have to add new directories for CppCheck to check. Why is my CppCheck able to find all files in subdirectories for the src/ folder but not for the include folder? I have to add each include folder manually.
stage('CppCheck') {
steps {
dir("${env.WORKSPACE}/${FOLDERS_TO_SCAN}"){
sh label: '', returnStatus: true, script: 'cppcheck --check-config --inline-suppr -itest/ -Iinclude/utilities -Iinclude/ src/ cppcheck --enable=all --suppress=missingIncludeSystem . --inconclusive --xml --xml-version=2 graphal 2> cppcheck-result.xml'
}
}
}
I've the following gradle project structure:
SUPER/
|______A/
| |______A1/
| | |______build.gradle
| |______A2/
| | |______build.gradle
| |______build.gradle
| |______settings.gradle
|
|______B/
| |______build.gradle
| |______settings.gradle
|
|______build.gradle
|______settings.gradle
My build.gradle from the A-Project contains somethings like that:
...
project(":A1") {
apply plugin: "java-library"
dependencies {
api project(":A2")
}
}
project(":A2") {
apply plugin: "java-library"
dependencies {
...
}
}
...
My "Super"-settings.gradle file contains something like that:
rootProject.name = 'SUPER'
include 'A'
include 'B'
In my sub-project "A" the settings.gradle looks like that:
include 'A1', 'A2'
When calling the gradlewrapper like that "./gradlew SUPER:A1:sometask" on the command line from the SUPER-folder, it fails with the message:
FAILURE: Build failed with an exception.
* Where:
Build file 'C:\Users\<somepath>\SUPER\A\build.gradle' line: 44
* What went wrong:
A problem occurred evaluating project ':A'.
> Project with path ':A1' could not be found in project ':A'.
But when doing something like that "./gradlew A1:sometask" from the A-sub-project-folder, it works successfully.
can you explain me what am I'm doing here wrong?
I have a folder with a lot of projects inside it (too much to manually write build files for them)
The projects are mostly in a flat layout:
root
-project 1
-project 2
-project 3
-project 4
-project 5
( -project 5.1)
But can be nested as shown above, and I need to account for this.
Ideally the following should happen:
I can run user#user:/root gradle build and every project in the directory shoudl be built as long as it contains a gradle build file
if a build fails just continue with the next one
How can I make this possible ?
How about this one-liner (not tested):
find . -maxdepth 1 -type d \( ! -name . \) -exec bash -c "cd '{}' && gradle build || true" \;
Or, more verbose:
dirs=($(find . -type d))
for dir in "${dirs[#]}"; do
cd "$dir"
gradle build || true
done
I came up with a working solution:
def flist = []
// change to you workspace name
new File('./Workspace').eachDir {
//blacklist any folders you want
if (it.name !='.gradle' && it.name != 'master' && it.name!= 'Build-All') {
flist << it.name
}
}
// build task objects
flist.each { folder ->
task "${folder}"(type: GradleBuild) {
buildFile = "./Workspace/"+ folder + "/build.gradle"
dir = './' + folder
tasks = ['build']
}
}
// create super task
task (all, dependsOn: flist) {
}
You need to invoke it as such in the root directory: gradle :all --continue this has the benefit that any failing project builds will not halt the other builds.
Another bonus is that gradle gives a neat report about all failing builds.
The problem
Suppose that I have written a lengthy script
in some language "lang",
and now want to convert this single-file script
into a directory tree with a project consisting of many files. I want to insert some kind of separators and file-paths into this file, and process it in some way so that in the end I obtain:
a proper project directory layout (sth. like this),
build-definition file,
readme's,
separate subdirectories for main/src and test/src etc.
For example, given the following script (pseudocode):
// required dependencies, should be moved
// into the build definition build.foo
require "org.foo" % "foo-core" % "1.2.3"
require "org.bar" % "bar-gui" % "3.2.1"
// A longer comment that should be converted
// into a text file and moved into a 'notes'
// subdirectory
/*
#README
Another lengthy comment that should go into
a readme.md
*/
/** A class that should
* go to src/main/lang/proj/A.lang
*/
class A {
def a = "foo"
}
/** Another class
* that should go to src/main/lang/proj/B.lang
*/
class B {
def b = "bar"
}
/** Some tests,
* should end up in
* src/test/lang/proj/MyTest.lang
#Test def testFoo() {
assert(2 + 2 == 5)
}
and assuming that I can insert arbitrary separators, commands, escape-sequences and file paths into this file, I would like to obtain the following project:
project/
|-- build.txt
|-- notes
| `-- note_01.txt
|-- readme.md
`-- src
|-- main
| `-- lang
| `-- proj
| |-- A.lang
| `-- B.lang
`-- test
`-- lang
`-- proj
`-- MySpec.lang
Edit:
What follows is a less-sophisticated version of my own answer below
What I've tried
Here is one naive way to do it:
Convert the original script into a bash script by prepending #!/bin/bash
split the source code into HEREDOCS
insert package declarations where necessary
add bunch of mkdir -p and cd between the HEREDOC-pieces
cat the HEREDOC pieces into appropriately named files
test the script on empty directories until it works as expected
For the above script, it might look somehow like this:
#!/bin/bash
mkdir project
cd project
cat <<'EOF' > build.txt
// required dependencies, should be moved
// into the build definition build.foo
require "org.foo" % "foo-core" % "1.2.3"
require "org.bar" % "bar-gui" % "3.2.1"
EOF
mkdir notes
cd notes
cat <<'EOF' > note_01.txt
// A longer comment that should be converted
// into a text file and moved into a 'notes'
// subdirectory
EOF
cd ..
cat <<'EOF' > readme.md
/*
#README
Another lengthy comment that should go into
a readme.md
*/
EOF
mkdir -p src/main/lang/proj
cd src/main/lang/proj
cat <<'EOF' > A.lang
package proj
/** A class
* that should go to src/main/lang/proj/A.lang
*/
class A {
def a = "foo"
}
EOF
cat <<'EOF' > B.lang
package proj
/** Another class
* that should go to src/main/lang/proj/B.lang
*/
class B {
def b = "bar"
}
EOF
cd ../../..
mkdir -p test/lang/proj
cd test/lang/proj
cat <<'EOF' > MySpec.lang
package proj
/** Some tests,
* should end up in
* src/test/lang/proj/MyTest.lang
#Test def testFoo() {
// this should end up in test
assert(2 + 2 == 5)
}
EOF
cd ../../..
What's wrong with this approach
It does generate the correct tree, but this approach seems rather error-prone:
it's too easy to cd ../../.. to the wrong nesting level
too easy to mkdir with a wrong name, and then fail to cd into it.
There is no way to handle the entire tree construction as a single
transaction, that is, if something fails later in the script,
there is no simple way to clean up the mess generated before
the error occurred.
I certainly could try to make it a bit less brittle by defining
special functions that mkdir and cd in one go, and
then wrap invocations of those functions together with cats into
(mkdirAndCd d ; cat) etc.
But it just doesn't feel quite right. Isn't there a much simpler
way to do it? Could one somehow combine the standard bash/linux utilities
into a tiny & very restricted domain specific language for
generating directory trees with text files? Maybe some newer version of split where one could specify where to split and where to put the pieces?
Related questions:
mkdir and touch in single command
The reverse of tree - reconstruct file and directory structure from text file contents?
Other interesting proposals that don't seem to work:
Use tar. That would mean that one would have to convert the text file manually into a valid tar-archive. While a tar archive indeed is a single plain-text file, its internal format does not look like the most comfortable DSL for such a simple task. It was never intended to be used by humans directly in that way.
Similar argument holds for shar. Since shar uses the bash itself to extract the archive, my above proposal is, in principle, a manually generated shar-archive in a very uncommon format, therefore shar seems to share all the drawbacks with the above proposal. I'd rather prefer something more restricted, that allows to do fewer things, but provides more guarantees about the quality of the outcome.
Maybe I should emphasize again that I don't have a tree to begin with, so there is nothing to compress. I have only the single script file and a rough idea of what the tree should look like in the end.
Seems to me that you are trying to write a custom parser. Provided that all blocks mentioned by you are ended by double line endings, this could help you
#!/bin/bash
gawk 'BEGIN{RS="\n\n([/][*]|[/]{2,2})"}
{
if ($0 ~ /#README/){
system("echo -e \"\nThis is a Readme.md\n--------\n" $0 "\"")
}else if ($0 ~ /class /){
system("echo -e \"\nThis is a class\n---------\n/*" $0 "\"")
}else if ($0 ~ /require /){
system("echo -e \"\nthis is a conf\n-----------\n" $0 "\"")
}else if($0 ~ /[/]{2,2}.*\n[/]{2,2}/){
system("echo -e \"\nthis is a note\n-----------\n" $0 "\"")
}
}' your_script.lang
The key part is the record separator RS that splits block of code that start with '\n\n//' or '\n\n/*'.
Instead of echo -e you could write custom scripts for each type of block.
Please note that the record separator will not be present on $0 so you have to add the missing characters, as in the /class / example above.
The output of the above code is
this is a conf
-----------
// required dependencies, should be moved
// into the build definition build.foo
require org.foo % foo-core % 1.2.3
require org.bar % bar-gui % 3.2.1
this is a note
-----------
A longer comment that should be converted
// into a text file and moved into a 'notes'
// subdirectory
This is a Readme.md
--------
#README
Another lengthy comment that should go into
a readme.md
*/
This is a class
---------
/** A class that should
* go to src/main/lang/proj/A.lang
*/
class A {
def a = foo
}
This is a class
---------
/** Another class
* that should go to src/main/lang/proj/B.lang
*/
class B {
def b = bar
}
About your concerns:
it's too easy to cd ../../.. to the wrong nesting level
-> define a variable with root path and cd to it.
too easy to mkdir with a wrong name, and then fail to cd into it.
-> define variables with directory names and check if they already exists.
path1=src/main/lang/some
if [ -d $path1 ]; then
do_something
fi
There is no way to handle the entire tree construction as a single transaction ...
-> write to file paths of every NEW directory/file that you create and use it to revert if necessary.
(my own answer)
Consider the following definition of a tiny embedded domain specific language for defining directory trees with text files:
#!/bin/bash
enter() {
local subdir="$1"
if [ ! -d "$subdir" ]
then
mkdir -p "$subdir"
fi
pushd "$subdir" > /dev/null
}
leave() {
popd > /dev/null
}
save() {
local fileName="$1"
cut -d'|' -f2- > "$fileName"
}
The enter command creates a directory if necessary, and cds into this directory, it works with arbitrary relative paths. The save command saves the text content of a here-document to file. The leave command changes to previous directory.
When a file is saved, the margin (empty space followed by '|') is stripped from each line. This is to ensure that the indentation of the script does not interfere with the indentation of the written files.
If these definitions are sourced, then the tree-generation script can be written as follows:
#!/bin/bash
source explode.sh
mkdir project
cd project
enter "src"
enter "main/lang/proj"
save "A.lang" <<'____EOF'
|package proj
|
|/** A totally useful class
| * that should go to src/main/lang/proj/A.lang
| */
|class A {
| def a = "foo"
|}
____EOF
save "B.lang" <<'____EOF'
|package proj
|/** Another very useful class
| * that should go to src/main/lang/proj/B.lang
| */
|class B {
| def b = "bar"
|}
____EOF
leave
enter "test/lang/proj"
save "MyTest.lang" <<'____EOF'
|package proj
|
|/** A test that should end up in
| * src/test/lang/proj/MyTest.lang
|#Test def testFoo() {
| assert(2 + 2 == 5)
|}
____EOF
leave
leave
save "build.txt" <<'EOF'
|require "org.foo" % "foo-core" % "1.2.3"
|require "org.bar" % "bar-gui" % "3.2.1"
EOF
enter "notes"
save "note_01.txt" <<'__EOF'
|A longer comment that should be converted
|into a text file and moved into a 'notes'
|subdirectory. This is a very long comment
|about the purpose of the project. Blah
|blah blah.
__EOF
leave
save "README.md" <<'EOF'
|#README
|
|This is a readme file for my awesome project.
|It ends with this line. Bye.
EOF
When executed, the script generates the following directory tree:
project/
├── build.txt
├── notes
│ └── note_01.txt
├── README.md
└── src
├── main
│ └── lang
│ └── proj
│ ├── A.lang
│ └── B.lang
└── test
└── lang
└── proj
└── MyTest.lang
The bash-script mirrors the tree structure very closely, and it's impossible to mess up the cd ../../../../../.. commands. It still lacks various desirable properties though (not transactional, no dry-run capability).
I'm using serverspec to carry out remote testing of servers.
I've a number of different tests, which all work fine:
`-- spec
|-- builder.example.org.uk
\ host_spec.rb
|-- chat.example.org.uk
\ host_spec.rb
|-- docker.example.org.uk
\ host_spec.rb
\-- git.example.org.uk
\ host_spec.rb
However each host-test has a lot of duplication, because I want to ensure that each host has sshd running, for example.
I've tried several different ways of creating spec/common_tests.rb but each time fails. For example adding spec/common.rb:
describe command("lsb_release -d") do
its(:stdout) { should match /wheezy/ }
end
Then in spec/chat.example.org.uk/host_spec.rb:
require 'common'
However this seems to suddenly want to connect to a different host, and fail:
shelob ~ $ bundle exec rake spec:ssh.example.org.uk
/usr/bin/ruby1.9.1 -S rspec spec/ssh.example.org.uk/host_spec.rb
F.....................
Failures:
1) Command "lsb_release -d" stdout
On host `ssh.example.org.uk`
Failure/Error: Unable to find matching line from backtrace
SocketError: getaddrinfo: Name or service not known
So my question is twofold:
Is it possible to include common tests from an external file?
If so how do I accomplish this?
I'm not sure whether your example has a typo, as it seems to do exactly what you want. You're running bundle exec rake spec:ssh.example.org.uk and it's running against ssh.example.org.uk.
The serverspec documentation suggests another way to run shared specs. Instead of organising your files by host, you should organise them by role. For instance:
`-- spec
|-- app
| `-- ruby_spec.rb
|-- base
| `-- users_and_groups_spec.rb
|-- db
| `-- mysql_spec.rb
|-- proxy
| `-- nginx_spec.rb
`-- spec_helper.rb
Then, in your Rakefile, you map your hosts to roles:
hosts = [{name: 'www.example.org.uk', roles: %w(base app)},
{name: 'db.example.org.uk', roles: %w(base db)}]
You can then provide a ServerSpecTask that runs commands by setting the host address as an environment variable, by overriding RSpec's spec_command method:
class ServerspecTask < RSpec::Core::RakeTask
attr_accessor :target
def spec_command
cmd = super
"env TARGET_HOST=#{target} #{cmd}"
end
end
namespace :serverspec do
hosts.each do |host|
desc "Run serverspec to #{host[:name]}"
ServerspecTask.new(host[:name].to_sym) do |t|
t.target = host[:name]
t.pattern = 'spec/{' + host[:roles].join(',') + '}/*_spec.rb'
end
end
end
And then finally, update your spec_helper.rb to read that environment variable and use it as the host:
RSpec.configure do |c|
c.host = ENV['TARGET_HOST']
options = Net::SSH::Config.for(c.host)
user = options[:user] || Etc.getlogin
c.ssh = Net::SSH.start(c.host, user, options)
c.os = backend.check_os
end