I have a shell script for getting data from some servers, in ~/.bin/shellScript :
OLD="$(curl --silent http://someServer:12345/stats.json | json someKey)"
NEW="$(curl --silent http://otherServer:12345/stats.json | json someKey")
echo "OLD $OLD - NEW $NEW"
I want to echo the results for running it interactively, but I've been wanting to log the results collected too.
So crontab -e, and add */5 * * * * /home/user/.bin/shellScript >> /media/dump/scriptDump.
Running the script interactively works fine - I get OLD 123 - NEW 456, but when I look at what's been running from cron, I get OLD - NEW with no data being grabbed with curl.
As discovered in the comments, you need to add the full path of json when you are calling it. This is because crontab's limited environment.
So instead of
OLD="$(curl --silent http://someServer:12345/stats.json | json someKey)"
NEW="$(curl --silent http://otherServer:12345/stats.json | json someKey")
it has to be
OLD="$(curl --silent http://someServer:12345/stats.json | /path/to/json someKey)"
NEW="$(curl --silent http://otherServer:12345/stats.json | /path/to/json someKey)"
^^^^^^^^^^^^^
[[Note your second line had ") instead of )"]]
Otherwise, you can also add the json path into crontab, as indicated on How to get CRON to call in the correct paths:
PATH=/usr/local/sbin: ... :/path/of/json
Related
Trying to save API responses in their original JSON format as env variables. Getting formatting errors.
echo "GIT_PR_OBJECT=$(curl -u USER:TOKEN https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.event.number }})" >> $GITHUB_ENV
Error from Github:
Error: Unable to process file command 'env' successfully.
Error: Invalid environment variable format ' "sha": "123456789",'
My plan is to access this object multiple times throughout the workflow, that way I only need to make one call to Github API rather than few. Here is how I access the data object saved in the env var.
echo "GIT_EMAIL=$(${{ env.GIT_COMMIT_OBJECT }} | jq -r '.commit.author.email')" >> $GITHUB_ENV
How can I save my curl response as an env variable in it's JSON format?
Why not save json content to file instead? Saving to environment variable, while sounds fancy, also smells like a can of worms you don't have to deal with in a first place.
Taking your example:
- run: |
curl -u USER:TOKEN https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.event.number }} > $HOME/pr.json
echo "GIT_EMAIL=$(jq -r '.commit.author.email' $HOME/pr.json)" >> $GITHUB_ENV
shell: bash
While it is a working solution in my eyes, i fail to understand why it has to be environment variable, which is your main question here.
I have a crontab and .env file. And I want to reach .env variables from crontab. Is it possible?
.env
variable1=value1
variable2=value2
crontab
2 11 * * * curl -Ssi -X POST -u 'value1:value2' https://example.com...
I think the best way is to create a script that reads you .env file an runs the command, like this:
#!/bash
# This is /my_path/my_script.sh file,
# do not forget to set executable permission by chmod
# Reading vars
. /my_path/my_env.env
# Calling the command
curl -Ssi -X POST -u "${variable1}:${variable2}" https://example.com...
and your crontab line will be like this:
2 11 * * * /my_path/my_script.sh
or alternatively in not so readable manner, directly in the crontab:
2 11 * * * . /my_path/my_env.env; curl -Ssi -X POST -u "${variable1}:${variable2}" https://example.com...
Source your .env file within cron, and I assume you mean $variable1:$variable2.
2 11 * * * . /path/to/.env; curl -Ssi -X POST -u "$variable1:$variable2" https://example.com...
I'm new to shell scripting and I have a Bash script pulling in data from the Strava API and manipulating/reading it using jq.
When I copy and paste in the first line of code (the one calling in data) into the command line, it works. When I run bash strava.sh the entire program works. But when I execute the program through crontab, I'm getting the following error:
usage: http [--json] [--form] [--pretty {all,colors,format,none}]
[--style STYLE] [--print WHAT] [--headers] [--body] [--verbose]
[--all] [--history-print WHAT] [--stream] [--output FILE]
[--download] [--continue]
[--session SESSION_NAME_OR_PATH | --session-read-only SESSION_NAME_OR_PATH]
[--auth USER[:PASS]] [--auth-type {basic,digest}]
[--proxy PROTOCOL:PROXY_URL] [--follow]
[--max-redirects MAX_REDIRECTS] [--timeout SECONDS]
[--check-status] [--verify VERIFY]
[--ssl {ssl2.3,tls1,tls1.1,tls1.2}] [--cert CERT]
[--cert-key CERT_KEY] [--ignore-stdin] [--help] [--version]
[--traceback] [--default-scheme DEFAULT_SCHEME] [--debug]
[METHOD] URL [REQUEST_ITEM [REQUEST_ITEM ...]]
http: error: unrecognized arguments: https://www.strava.com/oauth/token client_id=xxx client_secret=xxx refresh_token=xxx grant_type=refresh_token
Here's what the line looks like in my script:
access_token=$(http POST "https://www.strava.com/oauth/token" client_id="xxx" client_secret="xxx" refresh_token="xxx" grant_type="refresh_token" | jq -r '.access_token')
When running through crontab, the above error is printed on the first line (i.e. line given above), so I'm fairly certain the problem lies in that line. What am I doing wrong?
The httpie manual (https://httpie.io/docs/cli/best-practices) advises to use of:
--ignore-stdin
For "non-interactive invocations".
Possibly a path issue - are there multiple copies of http installed?
Is there a "%" anywhere in your parameters? Crontab interprets % as a newline, so if you'll have to escape it - "%%".
As an aside - please put your subshell inside "s, lest one day strava returns something like "AC0f4;rm * 0cd-4b203"
access_token="$( http POST ...
The problem
SNMPD is correctly delegating SNMP polling requests to another program but the response from that program is not valid. A manual run of the program with the same arguments is responding correctly.
The detail
I've installed the correct LSI raid drivers on a server and want to configure SNMP. As per the instructions, I've added the following to /etc/snmp/snmpd.conf to redirect SNMP polling requests with a given OID prefix to a program:
pass .1.3.6.1.4.1.3582 /usr/sbin/lsi_mrdsnmpmain
It doesn't work correctly for SNMP polling requests:
snmpget -v1 -c public localhost .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.1
I get the following response:
Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: SNMPv2-SMI::enterprises.3582.5.1.4.2.1.2.1.32.1
What I've tried
SNMPD passes two arguments, -g and <oid> and expects a three line response <oid>, <data-type> and <data-value>.
If I manually run the following:
/usr/sbin/lsi_mrdsnmpmain -g .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0
I correctly get a correct three line response:
.1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0
integer
30
This means that the pass command is working correctly and the /usr/sbin/lsi_mrdsnmpmain program is working correctly in this example
I tried replacing /usr/sbin/lsi_mrdsnmpmain with a bash script. The bash script delegates the call and logs the supplied arguments and output from the delegated call:
#!/bin/bash
echo "In: '$#" > /var/log/snmp-pass-test
RETURN=$(/usr/sbin/lsi_mrdsnmpmain $#)
echo "$RETURN"
echo "Out: '$RETURN'" >> /var/log/snmp-pass-test
And modified the pass command to redirect to the bash script. If I run the bash script manually /usr/sbin/snmp-pass-test -g .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0 I get the correct three line response as I did when I ran /usr/sbin/lsi_mrdsnmpmain manually and I get the following logged:
In: '-g .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0
Out: '.1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0
integer
30'
When I rerun the snmpget test, I get the same Error in packet... error and the bash script's logging shows that the captured delegated call output is empty:
In: '-g .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0
Out: ''
If I modify the bash script to only echo an empty line I also get the same Error in packet... message.
I've also tried ensuring that the environment variables that are present when I manually call /usr/sbin/lsi_mrdsnmpmain are the same for the bash script but I get the same empty output.
Finally, my questions
Why would the bash script behave differently in these two scenarios?
Is it likely that the problem that exists with the bash scripts is the same as originally noticed (manually running program has different output to SNMPD run program)?
Updates
eewanco's suggestions
What user is running the program in each scenario?
I added echo "$(whoami)" > /var/log/snmp-pass-test to the bash script and root was added to the logs
Maybe try executing it in cron
Adding the following to root's crontab and the correct three line response was logged:
* * * * * /usr/sbin/lsi_mrdsnmpmain -g .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.1 >> /var/log/snmp-test-cron 2>&1
Grisha Levit's suggestion
Try logging the stderr
There aren't any errors logged
Checking /var/log/messages
When I run it via SNMPD, I get MegaRAID SNMP AGENT: Error in getting Shared Memory(lsi_mrdsnmpmain) logged. When I run it directly, I don't. I've done a bit of googling and I may need lm_sensors installed; I'll try this.
I installed lm_sensors & compat-libstdc++-33.i686 (the latter because it said it was a pre-requisite from the instructions and I was missing it), uninstalled and reinstalled the LSI drivers and am experiencing the same issue.
SELinux
I accidently stumbled upon a page about extending snmpd with scripts and it says to check the script has the right SELinux context. I ran grep AVC /var/log/audit/audit.log | grep snmp before and after running a snmpget and the following entry is added as a direct result from running snmpget:
type=AVC msg=audit(1485967641.075:271): avc: denied { unix_read unix_write } for pid=5552 comm="lsi_mrdsnmpmain" key=558265 scontext=system_u:system_r:snmpd_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=shm
I'm now assuming that SELinux is causing the call to fail; I'll dig further...see answer for solution.
strace (eewanco's suggestion)
Try using strace with and without snmp and see if you can catch a system call failure or some additional hints
For completeness, I wanted to see if strace would have hinted that SELinux was denying. I had to remove the policy packages using semodule -r <policy-package-name> to reintroduce the problem then ran the following:
strace snmpget -v1 -c public localhost .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.1 >> strace.log 2>&1
The end of strace.log is as follows and unless I'm missing something, it doesn't seem to provide any hints:
...
sendmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(161), sin_addr=inet_addr("127.0.0.1")}, msg_iov(1)= [{"0;\2\1\0\4\20public\240$\2\4I\264-m\2"..., 61}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=, ...}, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 61
select(4, [3], NULL, NULL, {0, 999997}) = 1 (in [3], left {0, 998475})
brk(0xab9000) = 0xab9000
recvmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(161), sin_addr=inet_addr("127.0.0.1")}, msg_iov(1)= [{"0;\2\1\0\4\20public\242$\2\4I\264-m\2"..., 65536}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT) = 61
write(2, "Error in packet\nReason: (noSuchN"..., 81Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
) = 81
write(2, "Failed object: ", 15Failed object: ) = 15
write(2, "SNMPv2-SMI::enterprises.3582.5.1"..., 48SNMPv2- SMI::enterprises.3582.5.1.4.2.1.2.1.32.1
) = 48
write(2, "\n", 1
) = 1
brk(0xaa9000) = 0xaa9000
close(3) = 0
exit_group(2) = ?
+++ exited with 2 +++
It was SELinux that was denying snmpd a delegated call to /usr/sbin/lsi_mrdsnmpmain (and probably beyond).
To identify it, I ran grep AVC /var/log/audit/audit.log and for each entry, I ran the following:
echo "<grepped-output>" | audit2allow -a -M <filename>
This creates a SELinux policy package that should allow the delegated call through. The package is then loaded using the following:
semodule -i <filename>.pp
I had to do this 5 times as there were different causes of denial (unix_read unix_write, associate, read write). I'll look to combine the modules into one.
Now when I run snmpget I get the correct delegated output:
SNMPv2-SMI::enterprises.3582.5.1.4.2.1.2.1.32.1 = INTEGER: 34
I've ran into a strange issue. I'm trying to script my router to collect usage stats and other stuff. I'm making one cURL to the auth URL to get a valid session id, then another using that session id to the page I need.
Here is my script:
SESSION_ID=$(curl --silent -D - -X POST http://10.0.0.1/login.cgi -d'admin_username=admin&admin_password=admin' | grep 'SESSION' | sed 's/Set-Cookie: SESSION=//' | sed 's/; path=\///')
echo $SESSION_ID # 1234567890
curl -v -H "Cookie: SESSION=$SESSION_ID" http://10.0.0.1/modemstatus_dslstatus.html
If I manually take SESSION_ID and insert it in place of '"$SESSION_ID"' everything is dandy. cURL shows the headers (via -v) and they are correct. Running the command while manually inserting the session id produces identical headers.
I'm sure it's something small. Please teach me something :)
Check for carriage returns \r in your variables which wouldn't appear with a simple echo in some cases.