I'm trying to get a shell command with pipes working with Jenkins Pipelines. It's related to: Jenkins pipeline sh does not seem to respect pipe in shell command. I also found: Jenkins pipeline, bash, and pipes
However, my success with it has been unsuccessful. I can literally copy/paste the resolution of the first post in my pipeline and it does work (at least it tries execute the shell, fails due to lack of pom.xml).
When I attempt to modify the shell command to my needs, it falls flat:
def bgd_discovery = $/eval "cf apps | grep ${appHost}.${host} | cut -d ' ' -f 1 | sed 's/${args.appName}-bgd//'"/$
echo "${bgd_discovery}"
Jenkins throws a DSL method error:
java.lang.NoSuchMethodError: No such DSL method 'eval "cf apps | grep [...app host...] | cut -d ' ' -f 1 | sed 's' found among steps [...]
It seems to stop at first / of sed, but that's not the problem because this does work (from the first post):
def ver_script = $/eval "sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'"/$
Interestingly, here's another symptom to a problem: An added hyphen to the end of the sed regex
def bgd_discovery = $/eval "cf apps | grep ${appHost}.${host} | cut -d ' ' -f 1 | sed 's/${args.appName}-bgd-//'"/$
Produced this error:
WorkflowScript: 26: expecting '}', found '' # line 26, column 10.
echo "${bgd_discovery}"
I can't figure out what's wrong.
Okay, I'm going to answer my own question now. It would appear there's something flakey with the quotes. I removed them for sed and things are functioning, even without the whole eval route. This works just fine:
def current_deployment = sh (
returnStdout: true,
script: "cf apps | grep ${appName}.apps | cut -d ' ' -f 1 | sed s/${appName}-bgd-//"
);
Related
Is it possible to use pipe output as input for grep or git grep? The data im trying to pass to grep/git grep is the following
kubectl get namespace -o name -l app.kubernetes.io/instance!=applications | cut -f2 -d "/"
argocd
default
kube-node-lease
kube-public
kube-system
nsx-system
pks-system
I've tried to extent the command but this results in an error:
kubectl get namespace -o name -l app.kubernetes.io/instance!=applications | cut -f2 -d "/" | xargs git grep -i
fatal: ambiguous argument 'default': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Using just grep results in:
kubectl get namespace -o name -l app.kubernetes.io/instance!=applications | cut -f2 -d "/" | xargs grep -i
grep: default: No such file or directory
grep: kube-node-lease: No such file or directory
grep: kube-public: No such file or directory
grep: kube-system: No such file or directory
grep: nsx-system: No such file or directory
grep: pks-system: No such file or directory
The issue im facing with grep in general in this particular case is, that even if i soley use grep within my directory, it takes ages till it's done, whereas git grep is done within seconds. If I'm not doing something terrible wrong that would explain the slow results of grep, getting git grep to work would be preferred.
I've found this other Stackoverflow Question that somewhat explains what the issue is, but I don't know how to "process" the output into git grep properly.
The problem is that (as your screenshot shows) the result is multiple terms which I'm guessing you want to be OR-ed together, and not searching for the first term in the files identified by the last terms (which is what the current xargs command does)
Since OR in regex is via the | character, you can use xargs echo to fold the vertical list into a space delimited horizontal list then replace the spaces with | and be pretty close to what you want
printf 'alpha\nbeta\ncharlie\n' | xargs echo | tr ' ' '|' | xargs git grep -i
although due to the folding operation, that command is an xargs of one line, and thus would be conceptually easier to reason about using just normal $() interpolation:
git grep -i $(printf 'alpha\nbeta\ncharlie\n' | xargs echo | tr ' ' '|')
The less "whaaa" shell pipeline would be to use kubectl get -o go-template= to actually emit a pipe-delimited list and feed that right into xargs (or $()), bypassing the need to massage the output text first
I am using zsh on macOS
I have a shell script that produces a text file with speedtest results in the following layout:
Download: 63.57 Mbps (data used: 69.3 MB)
Upload: 16.11 Mbps (data used: 23.0 MB)
I can manipulate the layout and produce this:
↓ 63.57 Mbps |
↑ 16.11 Mbps
Note the line break before the first line of text and the one after the pipe. In the Terminal only the final line is printed out: ↑ 16.11 Mbps
The script to transform the input is this:
DOWNLOAD=$(cat ~/Terminal_Projects/temp_speedtest_result.txt | grep Download | sed 's/ Download: /↓ /g' | sed 's/ (data used: //g' | sed -E 's/[0-9]{1,4}\.[0-9] MB)//g' | sed 's/\n\r\t//')
UPLOAD=$(cat ~/Terminal_Projects/temp_speedtest_result.txt | grep Upload | sed 's/ Upload: /↑ /g' | sed 's/ (data used: //g' | sed -E 's/[0-9]{1,4}\.[0-9] MB)//g' | tr '\n' ' ')
RESULT=$DOWNLOAD" | "$UPLOAD
echo $RESULT
I used multiple instances of sed because I couldn't get it to work in just one instance. You may know how to get it to work.
What I want to do is output the DOWNLOAD and UPLOAD variables on a single line. I have another very similar script that achieves that with exactly the same manipulation of variables.
What I have tried:
Using RESULT="$DOWNLOAD | $UPLOAD"
Using RESULT="${DOWNLOAD} | ${UPLOAD}"
Using tr '\n' ' ' instead of the sed command to remove \n
I tried removing the up and down arrows in case those symbols aren't supported - same behaviour.
I have tried using sed on the RESULT variable to try removing new lines. I also tried writing the contents of the RESULT variable to a new temp txt file and then retrieving the contents of the file and using grep to extract the results one by one in the hope the new lines would not be copied. Didn't work for me.
It looks like there are line breaks that I have been unable to remove but I could be wrong.
I am new to command line and shell scripts. Trying to apply my very limited knowledge to a new scenario. Any help would be appreciated.
tr seems to do the job and echo -n "$UPLOAD" shows on a single line, so I think you're on the right track and only need to fix the DOWNLOAD part.
I suggest you simplify the script a bit using something along these lines:
INPUT_FILE="~/Terminal_Projects/temp_speedtest_result.txt"
DOWNLOAD="$(grep Download $INPUT_FILE | awk '{print "↓" $2 " " $3}' | tr '\n' ' ')"
UPLOAD="$(grep Upload $INPUT_FILE | awk '{print "↓" $2 " " $3}' | tr '\n' ' ')"
echo "$DOWNLOAD | $UPLOAD"
How about
RESULT="$(grep -Ew '(Down|Up)load' <~/Terminal_Projects/temp_speedtest_result.txt | tr '\n' ' ')"
? This is more efficient (only one grep and tr process needed) and also fixes the bug you have in your solution by your use of a pipe in RESULT=$DOWNLOAD" | "$UPLOAD (which should have brought up an error message).
I need to process the stack similar to below which has a string "oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver" but I need the exact method call that failed. e.g. in this case the exact one is "oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver.attachFile". There are multiple stacks that I have as part of html pages which I need to process and there will be multiple methods failing. How can I do it in shell script? This entire stack is in one line html entry which is causing grep to return the whole stack itself. I tried multiple things but nothing worked clearly. I guess awk or similar regex tool could be a way to go but not sure.
at oracle.adf.view.rich.automation.test.selenium.RichWebDriverTest.getElement(RichWebDriverTest.java:1414)
at oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver.attachFile(ApplcoreWebdriver.java:1460)
at oracle.apps.fnd.applcore.attachments.ui.util.accessor.FndManageAttachmentsPopupAccessor.attachFile(FndManageAttachmentsPopupAccessor.java:1475)
at oracle.apps.fnd.applcore.attachments.ui.util.accessor.FndManageAttachmentsPopupAccessor.updateRowFileAttachment(FndManageAttachmentsPopupAccessor.java:550)
at oracle.apps.fnd.applcore.attachments.ui.AttachmentsBaseSelenium.testLMultiFileAdd_22108390(AttachmentsBaseSelenium.java:1782)
at oracle.javatools.test.WebDriverRunner.run(WebDriverRunner.java:122)
I cannot guarantee it being the most efficient of solutions, but am able so suit the need. Am using all grep, sed combined together.
Splitting the single big html file as multiple lines using cat & tr
cat file | tr ' ' '\n' | grep -w "$search_string" | sed 's/(.*)//'
For testing purposes am using the stack snippet which you shared in the OP.
$ cat file
at oracle.adf.view.rich.automation.test.selenium.RichWebDriverTest.getElement(RichWebDriverTest.java:1414)
at oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver.attachFile(ApplcoreWebdriver.java:1460)
at oracle.apps.fnd.applcore.attachments.ui.util.accessor.FndManageAttachmentsPopupAccessor.attachFile(FndManageAttachmentsPopupAccessor.java:1475)
at oracle.apps.fnd.applcore.attachments.ui.util.accessor.FndManageAttachmentsPopupAccessor.updateRowFileAttachment(FndManageAttachmentsPopupAccessor.java:550)
at oracle.apps.fnd.applcore.attachments.ui.AttachmentsBaseSelenium.testLMultiFileAdd_22108390(AttachmentsBaseSelenium.java:1782)
at oracle.javatools.test.WebDriverRunner.run(WebDriverRunner.java:122)
Running the above command for the search string oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver
$ cat file | tr ' ' '\n' | grep -w "oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver" | sed 's/(.*)//'
oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver.attachFile
Any suggestions simplifying it are welcome.
Well, I think I found a working solution:
for stack in $all_stacks
do
words=`echo $stack | tr ' ' '\n'`
for word in $words
do
search_string="oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver"
if [[ $word == *${search_string}* ]];
then
echo $word
fi
done
done
The idea was to split the line (stack) into words separated by space and process each word by matching with the required string. Then the matching word finally prints the required value (including api call)
Update based on ans from #Inian :
Below is updated one command to achieve this by processing multiple html files which contains error stack to be processed.
find . -name '*-errors.html' | xargs cat | tr ' ' '\n' | grep -w "oracle.apps.fnd.applcore.test.selenium.ApplcoreWebdriver" | sed 's/(.*)//' | cut -d '<' -f1 | sort -u
I am trying to write a script that should take values from a xml file.
Here is the xml file :-
`<manifestFile>
<productInformation>
<publicationInfo>
<pubID pcsi-selector="P.S.">PACODE</pubID>
<pubNumber/>
</publicationInfo>
</productInformation>
</manifestFile>`
and i my code is
:-
#!/bin/sh
Manifest=""
Manifest= `/bin/grep 'pcsi-selector="' /LDCManifest.xml | cut -f 2 -d '"'`
echo $Manifest
I expect my result to be P.S. , but it keeps throwing error as :-
./abc.sh: P.S.: not found
I am new to shell and i am not able to figure out whats the error here ?
You can't have a space after the =.
When you run this command:
Manifest= `/bin/grep 'pcsi-selector="' /LDCManifest.xml | cut -f 2 -d '"'`
It's the same as this:
Manifest='' `/bin/grep 'pcsi-selector="' /LDCManifest.xml | cut -f 2 -d '"'`
That tells the shell to
Run the grep command.
Take its output
Run that output as a command, with the environment variable Manifest set to the empty string for the duration of the command.
Get rid of the space after the = and you'll get the result you want.
However, you should also avoid using backticks for command substitution, because they interfere with quoting. Use $(...) instead:
Manifest=$(grep 'pcsi-selector="' /LDCManifest.xml | cut -f2 -d'"')
Also, using text/regex-based tools like grep and cut to manipulate XML is clunky and error-prone. You'd be better off installing something like XMLStarlet:
Manifest=$(xmlstarlet sel -t \
-v '/manifestFile/productInformation/publicationInfo/pubID/#pcsiselector' -n \
/LDCManifest.xml)
Or simpler:
grep -oP 'pcsi-selector="\K[^"]+' /LDCManifest.xml
would print
P.S.
assign
Manifest=$(grep -oP 'pcsi-selector="\K[^"]+' /LDCManifest.xml)
This approach is not finding the file I think I specified.
SHELL = /bin/bash
PKG_NAME = test
PKG_VERSION := $(shell grep -i '^version' $(PKG_NAME)/DESCRIPTION | cut -d ':' -f2 | cut -d ' ' -f2)
In the shell itself, grep -i '^version' test/DESCRIPTION | cut -d ':' -f2 | cut -d ' ' -f2 does return the version successfully, e.g. 0.4-7
But, running via the makefile returns:
grep: test: Is a directory
grep: /DESCRIPTION: No such file or directory
test is a directory, that's true, but test/DESCRIPTION does exist, so I'm guessing $(PKG_NAME)/DESCRIPTION wasn't the right way to assemble the file name.
Suggestions? Thanks.
That error indicates that grep is seeing test and /DESCRIPTION as two separate arguments. Do you have extra spaces on the PKG_NAME assignment line or an errant space between $(PKG_NAME) and /DESCRIPTION in the $(shell ...) line?
As a general rule you might want to start putting quotes around arguments to shell commands (i.e. '$(PKG_NAME)/DESCRIPTION') to prevent this sort of word splitting issue (though without spaces you generally don't have that sort of problem).