Parse value from Groovy into Shell step Jenkins - shell

I have a Groovy script in my Jenkins build step that calculates the build duration and puts the value into a string that I would like to execute in a shell script.
I've tried doing it through groovy multiple ways but still no luck. Running the exact string on the Jenkins Slave works fine so would like to pass that string into a shell script step and run it after. How would I go about doing that?
I thought about setting an environment variable but currently only have found ways to retrieve them.
import hudson.model.*
import java.math.*
def apiKey = "secret"
def buildId = System.getenv("BUILD_ID")
def buildNo = System.getenv("BUILD_NUMBER")
def jobName = System.getenv("JOB_NAME")
jobName = jobName.replaceAll("\\.","-")
def nodeName = System.getenv("NODE_NAME")
def (startDate, startTime) = buildId.tokenize("_")
def (YY, MM, DD) = startDate.tokenize("-")
def (hh, mm, ss) = startTime.tokenize("-")
MathContext mc = new MathContext(200);
Date startDateTime = new GregorianCalendar(YY.toInteger(), MM.toInteger() - 1, DD.toInteger(), hh.toInteger(), mm.toInteger(),
ss.toInteger()).time
Date end = new Date()
long diffMillis = end.getTime() - startDateTime.getTime()
long buildDurationInSeconds = (diffMillis / 1000);
String metric = String.format("%s.jenkins.%s.%s.%s.duration %s",
apiKey, nodeName, jobName, buildNo, buildDurationInSeconds)
def cmd = 'echo "+metric+" | nc carbon.hostedgraphite.com 2003'
After this step I would invoke an "Execute Shell" step in jenkins passing in the value of "cmd". If someone has an example of both passing the value and then calling it in the shell script that would be a real help

def cmd = "ls -a"
new File("${build.workspace}/mycmd.sh").setText("#!/bin/sh\n${cmd}\n")
and as next step do Execute Shell ./mycmd.sh

Try this
def metric="WHAT_YOU_WANT_TO_PASS"
sh "echo $metric | nc carbon.hostedgraphite.com 2003"

Related

issue about pexpect logfile_read

use pexpect SSH connections to run cmds on remote server, the command can be executed, but the results displayed on the terminal are not as expected, code like this(At first there was no time.sleep, it was added for debugging)
import logging
import time
from pexpectUtility import Session
logger = logging.getLogger(__name__)
def test_create_and_show():
cliPrompt = 'dev-r0'
hostPrompt = 'admin#dev-r0'
aa = Session()
aa.connect("admin","password", "10.10.0.10")
time.sleep(2)
aa.child.sendline("sonic-cli")
aa.child.expect(cliPrompt, 3)
tTime = 0
time.sleep(tTime)
aa.child.sendline("configure terminal")
aa.child.expect(cliPrompt, 3)
time.sleep(tTime)
aa.child.sendline("end")
aa.child.expect(cliPrompt, 3)
time.sleep(tTime)
aa.child.sendline("exit")
aa.child.expect(hostPrompt, 3)
aa.disconnect()
the pexpectUtility.py
import sys
import logging as log
if sys.platform == 'win32':
import WExpect as pexpect
spawn_class = pexpect.spawn_windows
else:
import pexpect
spawn_class = pexpect.spawn
class MutliIO:
def __init__(self, *fds):
self.fds = fds
def write(self, data):
for fd in self.fds:
fd.write(data)
def flush(self):
for fd in self.fds:
fd.flush()
class Session(spawn_class):
def __init__(self):
self.child = None
def connect(self, username, password, serverIp, protocol='ssh'):
self.protocol = protocol
self.username = username
self.password = password
self.serverIp = serverIp
if protocol == 'ssh':
cmd = "ssh -x -o StrictHostKeyChecking=no -l %s " % self.username
else:
cmd = "telnet "
cmd = cmd + serverIp
log.info('Connecting to Dut: %s\n' %(cmd))
expect_list = ['ogin: $', '[P|p]assword:', '\[confirm\] $',
'\[confirm yes/no\]:', '\[yes/no\]:', '\(yes/no\)\?',
'\[y/n\]:', '--More--', 'ONIE:/ #',
pexpect.TIMEOUT, pexpect.EOF]
self.child = spawn_class(cmd)
logfile = open('pexpect.log', 'w')
self.child.logfile_read = MutliIO(sys.stdout)
# self.child.logfile_read = MutliIO(sys.stdout, logfile)
# self.child.logfile_read = MutliIO(logfile)
try:
re = self.child.expect(expect_list, 10)
log.debug("expect pwd: {}".format(re))
except Exception as err:
log.error('%s' %err)
raise
# login
try:
self.child.sendline(self.password)
except Exception as err:
raise RuntimeError("login failed!", err)
def disconnect(self):
self.child.sendline("exit")
self.child.expect(pexpect.EOF)
self.child.close()
if self.child.logfile_read != None:
self.child.logfile_read = None
Executed commands are repeated displayed, just like batch input. log is as follows:
admin#dev-r0:~$ sonic-cli
configure terminal
configure terminal
end
exit
dev-r0# configure terminal
dev-r0(config)# end
dev-r0# exit
admin#dev-r0:~$ exit
logout
Connection to 10.10.0.10 closed.
When I set tTime to 5 (each command interval is 5 seconds) the log is as expected,I think this is not a good solution,I also want to know the root cause
admin#dev-r0:~$ sonic-cli
dev-r0# configure terminal
dev-r0(config)# end
dev-r0# exit
admin#dev-r0:~$ exit
logout
Connection to 10.10.0.10 closed.
When I directly use expect to implement the above operation, there is no need to wait for 5 seconds between commands, and the log displayed by the terminal is normal.
why pexpect has this issue? how to solve this? Thanks in advance
This is not the whole answer, but a first point to fix. After the
sendline("sonic-cli") the first expect() is going to return
immediately, as it will match the prompt admin#dev-r0:~$ which is already
there waiting, before the sonic-cli command arrives. This means the next
command configure terminal is sent immediately after sonic-cli.
You should enhance the connect() routine to expect the admin#dev-r0:~$
prompt before returning, or use this expect instead of the sleep(2) which
should not be necessary.
Referring to the sample code of pexpect on the Internet, I found that the root cause is a code problem: missing a expect() after sendline()
The changes are as follows:
# login
try:
self.child.sendline(self.password)
HOST_PROMPT = '\$' # remote server prompt
re = self.child.expect(HOST_PROMPT)
except Exception as err:
raise RuntimeError("login failed!", err)

Execute groovy script with shell

I am trying to execute a groovy script on a unix environnement but i have an issue, I don't know how to write a shell script that could execute the script under :
import de.hybris.platform.servicelayer.model.ModelService;
import de.hybris.platform.servicelayer.search.FlexibleSearchService;
import de.hybris.platform.servicelayer.search.SearchResult;
import com.galerieslafayette.pcm.model.model.product.RmsProductModel ;
final FlexibleSearchService flexibleSearchService = spring.getBean("flexibleSearchService");
final ModelService modelService = spring.getBean("modelService");
new File("/tmp/productsToUpdate.csv").splitEachLine(";") {fields ->
criteria = fields[1]
if (fields[2]?.trim()) {
criteria += "</p><p>" + fields[2]
}
if (fields[3]?.trim()) {
criteria += "</p><p>" + fields[3]
}
criteria = "<p>" + criteria + "</p>"
updateProduct(fields[0], criteria)
}
def updateProduct(ugProduct, criteria) {
SearchResult<RmsProductModel> result = flexibleSearchService.search("select {pk} from {RmsProduct} where {ug}=?ugCode", ["ugCode":ugProduct]);
if(result.getResult().get(0) != null){
RmsProductModel rmsProduct = result.getResult().get(0);
String temp = rmsProduct.getProduct().getDescription(Locale.FRANCE);
rmsProduct.getProduct().setDescription(temp + criteria, Locale.FRANCE)
modelService.save(rmsProduct.getProduct());
}
}
I tried to find on the web how to do so, but i only find subject on how to run shell in a groovy and not the opposite.
I am a beginner on unix and don't know the command line or if i need a specifique addition on my environnement or if vanilla shell could do it

subprocess sometimes sends returns empty

I have the following class that is used to run a third party command line tool which I have no control over.
I run this ina Qthread in a PyQt Gui.
I turn the gui into an EXE using Pyinstaller
Problems are more prevalent when it is an EXE
class CLI_Interface:
def process_f(self, command, bsize=4096):
self.kill_process(CLI_TOOL)
startupinfo = STARTUPINFO()
startupinfo.dwFlags |= STARTF_USESHOWWINDOW
startupinfo.wShowWindow = SW_HIDE
p = Popen(command, stdout=PIPE, stderr=PIPE,
startupinfo=startupinfo, bufsize=bsize, universal_newlines=True)
try:
out, err = p.communicate(timeout=120)
except TimeoutExpired:
p.kill()
out, err = p.communicate()
return out.split(), err.split()
def kill_process(self, proc):
# Check process is running, Kill it if it is,
# return False if not.
# uses its own popen for Stderr >> stdout
# If we use the self.process_f method, it will create an infinite loop
startupinfo = STARTUPINFO()
startupinfo.dwFlags |= STARTF_USESHOWWINDOW
startupinfo.wShowWindow = SW_HIDE
try:
kill_proc = Popen("TaskKill /IM {} /T /F".format(proc), stdout=PIPE, stderr=STDOUT,
startupinfo=startupinfo, universal_newlines=True).communicate()[0]
if 'ERROR' not in kill_proc.split():
return True # Process Killed
else:
self.kill_process(proc)
except Exception as e:
return False
def download_data(self, code):
""" download data from the device based on a 5 digit code """
command = '"{}" -l {},{} {}'.format(CLI_TOOL_PATH,
code[0], code[2], code[1])
try:
p = self.process_f(command)
proc, err = p[0], p[1]
try:
if err[-2] == '-p':
return False
return True
except IndexError:
if not proc:
return False # This means there is no data but the file is still saved!!
pass
return True
except Exception as e:
return False
def ....
def ....
def ....
Thread:
class GetDataThread(QThread):
taskFinished = pyqtSignal()
notConnected = pyqtSignal()
def __init__(self, f, parent=None):
super(GetDataThread, self).__init__(parent)
self.f = f
def run(self):
is_dongle_connected()
DD = cli.download_data(self.f)
if not DD:
self.notConnected.emit()
else:
self.taskFinished.emit()
I either get a done! or error - This is normal when running from the command line.
Sometimes I get an empty list returned and I put this back into a recursive loop after killing the program.
However, it does not seem to restart properly and the problem continues - it gets stuck in a loop of nothing!.
Meanwhile, the csv files the cli tool produces are created as normal yet I have no data from stdout / stderr
Looking at processes the conhost and the cli tool are destroyed no problem.
The gui will continue to fail (until I unplug and plug in the dongle and / or restart the program / computer.
When I open the CLI and run the same command, it works fine or throws an error (which I catch in the program no problem)
I have tried setting a buffer as some files generated can reach 2.4mb
I tried setting a higher timeout to allow for it to finish.
There does not seem to be a correlation with file size though and it can get stuck at any size.
The flow is like so:
Gui >> CLI >> Dongle >> Sensor
Running on Windows 10
How can I make the connection more solid or debug what processes might still be lingering around and stopping this?
Is it blocking?
Is it a pipe buffer overflow? - If so How do I determine the correct bufsize?
Is it something to do with PyQt and Python Subprocess or Pyinstaller?
Would it be better to use QProcess instead of Subprocess?
Thanks in advance!

Multiprocessing Windows: how to prevent rerun?

I'm using this multiprocessing script:
Test class
from multiprocessing.pool import Pool
class Test:
def __init__(self):
print("init")
def getData(self, x, ID):
return x + str(ID)
def process(self):
pool = Pool(processes=10)
IDList = range(10)
data = []
for ID in IDList:
async_result = pool.apply_async(self.getData, ("wordl", ID))
data.append(async_result.get())
return data
main script
from TestClass import Test
def main():
test = Test()
test.process()
if __name__ == '__main__':
main()
When I run this the file, it keeps running. I found out that Windows is the problem here:
Just figured out the problem with this on Win 7 anaconda pyscripter
2.6.0 Pyscripter generates a .pyc file which is what is rerun everytime u run the program, just deleted it and it worked fine for me (source)
Is there a way to add code to this script so it will work on Windows? I tried using sys.dont_write_bytecode = True, but this didn't work.

JScript: identifying whether double quotes are passed to a WSH script

There are situations when it is important to identify whether double quotes are passed as arguments to a WSH script. For example because they should be passed to another executable to be run.
The standard parsing functions/objects:
objArgs = WScript.Arguments;
for (i = 0; i < objArgs.length; i++)
{
WScript.Echo(objArgs(i));
}
do not differentiate between:
cscript foo.js "bar"
and
cscript foo.js bar
Is it possible with some other approach?
Note: I also tried to sort of escape them with several combinations like:
cscript foo.js '"bar"'
It seems that they are simply stripped away.
Following #Ekkehard.Horner suggestions:
Solution
// parseArgs.js
// Parsing jscript script arguments verbatim
var Shell = new ActiveXObject("WScript.Shell"),
wmi = GetObject("winmgmts:{impersonationLevel=impersonate}!\\\\.\\root\\cimv2"),
guid = (new ActiveXObject("Scriptlet.TypeLib")).GUID.substring(0,38),
windir=Shell.ExpandEnvironmentStrings("%WinDir%"),
winver="\"" + windir + "\\System32\\winver.exe\" " + guid,
pcol, pid, cmd;
// Run winver.exe hidden and get this script ID as its ParentProcessId
winver=winver.replace(/\\/g, "\\\\");
Shell.Run("winver " + guid, 0);
pcol = new Enumerator (wmi.ExecQuery(
"SELECT * From Win32_Process WHERE CommandLine='"+ winver + "'",
"WQL", 32));
for (; !pcol.atEnd(); pcol.moveNext()){
var prc = pcol.item();
pid=prc.ParentProcessId;
prc.Terminate;
}
// Get the command line for the found PID
pcol = new Enumerator (wmi.ExecQuery(
"SELECT * From Win32_Process WHERE ProcessID="+ pid,
"WQL", 32));
for (; !pcol.atEnd(); pcol.moveNext()){
var prc = pcol.item();
cmd =prc.CommandLine;
}
WScript.Echo(cmd);
// Parse command line for arguments
var ags,
parseCmd=function(cmd){// WMI trims initial spaces
var p = new Object(),
re =/^"/.test(cmd) ? /"[^"]+" */ : /\S+\s*/;
p.nxt=re.test(cmd) ? cmd.match(re)[0] : ""; // extract next token
p.rst=cmd.replace(re, "") ; // remainder
return(p);
}
// Strip c/wscript path
ags=parseCmd(cmd).rst
//WScript.Echo(ags);
// Remove WSH "//xxx" options
ags=ags.replace(/\/\/\w+ +/g, "")
//WScript.Echo(ags);
// Strip script name and get arguments
ags=parseCmd(ags).rst
WScript.Echo(ags);
// Loop args and store as an array
var i=1, aags=[];
while(ags != ""){
var p =parseCmd(ags);
ags=p.rst;
aags.push(p.nxt.replace(/ +$/, ""));
WScript.Echo(i, p.nxt);
i++;
}
WScript.Echo(aags);
Test
Running parseArgs.js gives:
> cscript //nologo parseArgs.js "hello" world
cscript //nologo parseArgs.js "hello" world
"hello" world
1 "hello"
2 world
"hello",world
The line:
> parseArgs.js "hello" world
gives similar results.
Comments
Do we need such a convoluted script? Short answer: no. Long: depends.
In general, assuming you know the name of your script when it is run, you could query WMI for it.
Anyway, when you deploy your script, you do not normally have control on the deploy directory. So, if there is another script running under the same name, you can't know for sure which one is yours.
Another not so edge case is when there are two or more instances of your script running.
The strategy here is to run some dummy standard Windows executable (winver.exe) hidden, passing to it a GUID. In this way, it is safe to identify winver.exe command line by the unique GUID and consequently your script as the parent of winver.exe.
winver.exe does not require arguments, but does not protest if you pass some to it.

Resources