I want to write a cron task to record the ntpdate synchronization info into the system log, but there's no such info printed in the /var/log/messages after this cron task is done, where did I do wrong?
The followings are what my crontab looks like.
*/1 * * * * ntpdate 192.168.100.97 | logger -t "NTP"
*/1 * * * * echo "log test" | logger -t "TEST"
*/1 * * * * whoami | logger -t "WHO"
When I do tailf /var/log/messages and wait some time I only got the following lines, the NTP lines are missing.
Oct 29 15:22:01 localhost TEST: log test
Oct 29 15:22:01 localhost WHO: root
Oct 29 15:23:01 localhost TEST: log test
Oct 29 15:23:01 localhost WHO: root
Oct 29 15:24:01 localhost TEST: log test
Oct 29 15:24:01 localhost WHO: root
Oct 29 15:25:01 localhost TEST: log test
Oct 29 15:25:01 localhost WHO: root
Oct 29 15:26:01 localhost TEST: log test
Oct 29 15:26:01 localhost WHO: root
But when I do the ntpdate 192.168.100.97 | logger -t "NTP" in the command line, I could see there's message Oct 29 15:28:39 localhost NTP: 29 Oct 15:28:39 ntpdate[11101]: adjust time server 192.168.100.97 offset 0.000043 sec print out in the system log. What am I missing here?
Thanks in advance for your kind help.
Related
I've tested the cronjob in local which is working fine. But it is not working in server. Any help is appreciated. Thanks in advance.
"sudo systemctl status cron" command gives following outputs:
cron.service - Regular background program processing daemon
Loaded: loaded (/lib/systemd/system/cron.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-07-19 21:40:44 CST; 27min ago
Docs: man:cron(8)
Main PID: 511 (cron)
Tasks: 1 (limit: 2315)
Memory: 1.9M
CGroup: /system.slice/cron.service
└─511 /usr/sbin/cron -f
Jul 19 22:05:01 iZa2d4o64ljp79h0v3i5fvZ CRON[2292]: pam_unix(cron:session): session closed for user userr
Jul 19 22:06:01 iZa2d4o64ljp79h0v3i5fvZ CRON[2313]: pam_unix(cron:session): session opened for user userr by (uid=0)
Jul 19 22:06:01 iZa2d4o64ljp79h0v3i5fvZ CRON[2314]: (userr) CMD ( cd /var/www/abc.com.np/myproject/app/Console/Kernel.php && php artisan schedule:run >> /dev/null 2>>
Jul 19 22:06:01 iZa2d4o64ljp79h0v3i5fvZ CRON[2313]: pam_unix(cron:session): session closed for user userr
Jul 19 22:07:01 iZa2d4o64ljp79h0v3i5fvZ CRON[2333]: pam_unix(cron:session): session opened for user userr by (uid=0)
Jul 19 22:07:01 iZa2d4o64ljp79h0v3i5fvZ CRON[2334]: (userr) CMD ( cd /var/www/abc.com.np/myproject/app/Console/Kernel.php && php artisan schedule:run >> /dev/null 2>>
Jul 19 22:07:01 iZa2d4o64ljp79h0v3i5fvZ CRON[2333]: pam_unix(cron:session): session closed for user userr
Jul 19 22:08:01 iZa2d4o64ljp79h0v3i5fvZ CRON[2353]: pam_unix(cron:session): session opened for user userr by (uid=0)
Jul 19 22:08:01 iZa2d4o64ljp79h0v3i5fvZ CRON[2354]: (userr) CMD ( cd /var/www/abc.com.np/myproject/app/Console/Kernel.php && php artisan schedule:run >> /dev/null 2>>
Jul 19 22:08:01 iZa2d4o64ljp79h0v3i5fvZ CRON[2353]: pam_unix(cron:session): session closed for user userr
crontab (/etc/crontab)
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
MAILTO=""
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
* * * * * userr cd /var/www/abc.com.np/myproject/app/Console/Kernel.php && php artisan schedule:run >> /dev/null 2>&1
#
Kernel.php in laravel project
<?php
namespace App\Console;
use Illuminate\Console\Scheduling\Schedule;
use Illuminate\Foundation\Console\Kernel as ConsoleKernel;
use Illuminate\Support\Facades\DB;
class Kernel extends ConsoleKernel
{
/**
* Define the application's command schedule.
*
* #param \Illuminate\Console\Scheduling\Schedule $schedule
* #return void
*/
protected function schedule(Schedule $schedule)
{
$schedule->call(function () {
//z; here if I remove the comment, the error should be shown in storage/log/laravel.log. It works locally but no error is seen in server
DB::table('recent_users')->delete();
})->everyMinute();
}
}
Test: here if I've uncommented "//z" inside $schedule->call(), the error should be shown in storage/log/laravel.log if cronjob is working but it does not. It works locally but no error is seen in server
Update: I removed the cd
/var/www/abc.com.np/myproject/app/Console/Kernel.php && php artisan schedule:run >> /dev/null 2>&1
the error I got is following:
/var/www/abc.com.np/myproject/app/Console/Kernel.php: 1: cannot open ?php: No such file
When I start a linux server with Cloud-init, I have a few scripts in /etc/cloud/cloud.cfg.d/ and they run in reverse alphabetical order
# ll /etc/cloud/cloud.cfg.d/
total 28
-rw-r--r-- 1 root root 173 Dec 10 12:38 00-cloudinit-lifecycle-hook.cfg
-rw-r--r-- 1 root root 2120 Jun 1 2021 05_logging.cfg
-rw-r--r-- 1 root root 590 Oct 26 17:55 10_aws_yumvars.cfg
-rw-r--r-- 1 root root 29 Dec 1 18:22 20_amazonlinux_repo_https.cfg
-rw-r--r-- 1 root root 586 Dec 10 12:38 50-cloudinit-tomcat.cfg
-rw-r--r-- 1 root root 585 Dec 10 12:40 60-cloudinit-newrelic.cfg
The last to execute is 00-cloudinit-lifecycle-hook.cfg, in which I complete the lifecycle for the Auto Scaling Group with a CONTINUE. The ASG fails if it doesn't receive this signal after a given time out.
The issue is that even if there's an error in 50-cloudinit-tomcat.cfg, it still runs 00-cloudinit-lifecycle-hook.cfg instead of stopping
How can I ensure cloud-init stops and never reaches the last script? I would like the ASG to never receive the CONTINUE signal if there's any error.
Here are the files:
EC2 instance user-data:
#cloud-config
bootcmd:
- [cloud-init-per, once, "app-volume", mkfs, -t, "ext4", "/dev/nvme1n1"]
mounts:
- ["/dev/nvme1n1", "/app-volume", "ext4", "defaults,nofail", "0", "0"]
merge_how:
- name: list
settings: [append]
- name: dict
settings: [no_replace, recurse_list]
50-cloudinit-tomcat.cfg
#cloud-config
merge_how:
- name: list
settings: [append]
- name: dict
settings: [no_replace, recurse_list]
runcmd:
- "#!/bin/bash -e"
- set +x
- echo ' '
- echo '# ===================================='
- echo '# Tomcat Cloud Init '
- echo '# /etc/cloud/cloud.cfg.d/'
- echo '# ===================================='
- echo ' '
- echo '#===================================='
- echo '# Run Ansible'
- echo '#===================================='
- echo ' '
- set -x
- ansible-playbook /opt/init-config/tomcat/tomcat-config.yaml
when I run ansible-playbook /opt/init-config/tomcat/tomcat-config.yaml directly in the instance I get an error, and I know it returns 2
ansible-playbook /opt/init-config/tomcat/tomcat-config.yaml #shows errors
echo $? # shows "2"
00-cloudinit-lifecycle-hook.cfg
#cloud-config
merge_how:
- name: list
settings: [append]
- name: dict
settings: [no_replace, recurse_list]
runcmd:
- "/opt/lifecycles/lifecycle-hook-continue.sh"
An alternative I can think of, is to send a ABANDON signal instead of CONTINUE as soon as there's en error in one of the cloud-init config. But I can't find in the documentation on to define if there's an error
I have been trying to configure a cron job to run my ansible playbook every hour. I could not find any relevant examples on how to start the configuration . I have tried the below task in a separate ansible playbook and it shows the below output for crontab -l, but not execution seems to be happening. Help will be high appreciated.
root#ubuntu:/etc/ansible/# crontab -l
57 10 * * * ansible-playbook crontest.yaml
- name: Run an ansible playbook"
cron:
name: "Run play"
minute: "57"
hour: "10"
job: "ansible-playbook crontest.yaml"
Cron Logs:
Nov 24 07:35:01 ABC30VDEF290021 CRON[13951]: (root) CMD (echo "testing" > /etc/ansible/automation/logs/test.txt)
Nov 24 07:35:01 ABC30VDEF290021 CRON[13948]: (root) CMD ( /usr/bin/ansible-playbook /etc/ansible/automation/crontest.yml)
Nov 24 07:35:01 ABC30VDEF290021 CRON[13949]: (root) CMD (/usr/local/bin/ansible-playbook /etc/ansible/automation/main.yaml)
Crontab -e
#Ansible: Run play
*/1 * * * * /bin/sh -c '. ~/.profile; /usr/local/bin/ansible-playbook /etc/ansible/automation/crontest.yml
*/1 * * * * echo "testing" > /etc/ansible/automation/logs/test.txt
35 7 * * * /usr/local/bin/ansible-playbook /etc/ansible/automation/main.yaml
The following code will run the script every 30th minute of every hour
- name: Run CRON job to load data at every 30th minute of every hour.
become: yes
become_method: sudo
cron:
name: "load_data"
user: "root"
weekday: "*"
minute: "30"
hour: "*"
job: "python3 /path/to/my/script/loadScript.py > /home/ec2-user/loadData_result 2>&1"
state: present
There appears to be a 25 second delay every time a userdata script touches the disk on the centos 7 AMI from AWS marketplace.
Here's my script:
#!/bin/bash -ex
echo "[TIMER] START $(date +%s.%N)"
current_user=$(whoami)
echo "Running as: $current_user"
sudo id -u myuser &>/dev/null || sudo useradd myuser
echo "[TIMER] CreatedUser $(date +%s.%N)"
time sudo yum update -y
echo "[TIMER] Yum Update $(date +%s.%N)"
sudo mkdir -p /opt/myuser/resources
echo "[TIMER] Create /opt/myuser/resources $(date +%s.%N)"
sudo bash -c "cat > /etc/systemd/system/my-service.service" <<EOF
[Unit]
Description=My Service
After=network-online.target
[Service]
User=myuser
Group=myuser
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/bash -ex -c 'echo "Hello World"'
[Install]
Alias=my-service
WantedBy=default.target
EOF
echo "[TIMER] Make my-service.service $(date +%s.%N)"
sudo chmod 644 /etc/systemd/system/my-service.service
echo "[TIMER] Chmod $(date +%s.%N)"
sudo systemctl daemon-reload
echo "[TIMER] daemon-reload $(date +%s.%N)"
sudo systemctl enable my-service
echo "[TIMER] enable $(date +%s.%N)"
sudo systemctl start my-service
echo "[TIMER] END: my-service $(date +%s.%N)"
I launch a c5.large of this AMI and use the above as my userdata script: https://aws.amazon.com/marketplace/pp/B00O7WM7QW
Timers result:
[TIMER] START 1546978269.809559549
[TIMER] CreatedUser 1546978320.472706964
[TIMER] Yum Update 1546978356.991642552
[TIMER] Create /opt/myuser/resources 1546978382.033044767
[TIMER] Make my-service.service 1546978407.074353857
[TIMER] Chmod 1546978432.111791937
[TIMER] daemon-reload 1546978457.195078083
[TIMER] enable 1546978482.265036318
[TIMER] END: my-service 1546978507.313735938
| CENTOS 7 | | |
|-----------------------------------------------------------|----------------------|-------------|
| | | |
| log | timestamp | seconds |
| [TIMER] START 1546978269.809559549 | 1546978269.809559549 | |
| [TIMER] CreatedUser 1546978320.472706964 | 1546978320.472706964 | 50.66315007 |
| [TIMER] Yum Update 1546978356.991642552 | 1546978356.991642552 | 36.51893997 |
| [TIMER] Create /opt/myuser/resources 1546978382.033044767 | 1546978382.033044767 | 25.04139996 |
| [TIMER] Make my-service.service 1546978407.074353857 | 1546978407.074353857 | 25.04131007 |
| [TIMER] Chmod 1546978432.111791937 | 1546978432.111791937 | 25.03743982 |
| [TIMER] daemon-reload 1546978457.195078083 | 1546978457.195078083 | 25.08328009 |
| [TIMER] enable 1546978482.265036318 | 1546978482.265036318 | 25.06995988 |
| [TIMER] END: my-service 1546978507.313735938 | 1546978507.313735938 | 25.04870009 |
| | | |
| | total (s) | 237.50418 |
| | | |
| | total (m) | 3.958402999 |
If you scroll to the right in my ASCII table you can see that simple commands like mkdir, chmod, and useradd are taking 25 seconds. Why does this happen?
EDIT:
contents of /etc/hosts
$ hostname
ip-172-31-40-213.us-west-2.compute.internal
$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
example log from /var/log/messages, the systemd logs show that creating the sudo session takes the 25 seconds:
Jan 9 23:50:32 ip-172-31-35-166 cloud-init: + echo '[TIMER] Make my-service.service 1547077832.899069408'
Jan 9 23:50:32 ip-172-31-35-166 cloud-init: [TIMER] Make my-service.service 1547077832.899069408
Jan 9 23:50:32 ip-172-31-35-166 cloud-init: + sudo chmod 644 /etc/systemd/system/my-service.service
Jan 9 23:50:32 ip-172-31-35-166 systemd: Removed slice User Slice of root.
Jan 9 23:50:32 ip-172-31-35-166 systemd: Created slice User Slice of root.
Jan 9 23:50:32 ip-172-31-35-166 systemd: Started Session c3 of user root.
Jan 9 23:50:57 ip-172-31-35-166 cloud-init: ++ date +%s.%N
Jan 9 23:50:57 ip-172-31-35-166 cloud-init: + echo '[TIMER] Chmod 1547077857.946078493'
Jan 9 23:50:57 ip-172-31-35-166 cloud-init: [TIMER] Chmod 1547077857.946078493
journalctl log shows the likely culprit:
Jan 09 23:50:32 ip-172-31-35-166.us-west-2.compute.internal cloud-init[1197]: + echo '[TIMER] Make my-service.service 1547077832.899069408'
Jan 09 23:50:32 ip-172-31-35-166.us-west-2.compute.internal cloud-init[1197]: [TIMER] Make my-service.service 1547077832.899069408
Jan 09 23:50:32 ip-172-31-35-166.us-west-2.compute.internal cloud-init[1197]: + sudo chmod 644 /etc/systemd/system/my-service.service
Jan 09 23:50:32 ip-172-31-35-166.us-west-2.compute.internal systemd[1]: Removed slice User Slice of root.
Jan 09 23:50:32 ip-172-31-35-166.us-west-2.compute.internal sudo[13392]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/chmod 644 /etc/systemd/system/my-service.service
Jan 09 23:50:32 ip-172-31-35-166.us-west-2.compute.internal systemd[1]: Created slice User Slice of root.
Jan 09 23:50:32 ip-172-31-35-166.us-west-2.compute.internal systemd[1]: Started Session c3 of user root.
Jan 09 23:50:57 ip-172-31-35-166.us-west-2.compute.internal sudo[13392]: pam_systemd(sudo:session): Failed to create session: Connection timed out
Jan 09 23:50:57 ip-172-31-35-166.us-west-2.compute.internal sudo[13392]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 09 23:50:57 ip-172-31-35-166.us-west-2.compute.internal sudo[13392]: pam_unix(sudo:session): session closed for user root
Jan 09 23:50:57 ip-172-31-35-166.us-west-2.compute.internal cloud-init[1197]: ++ date +%s.%N
Jan 09 23:50:57 ip-172-31-35-166.us-west-2.compute.internal cloud-init[1197]: + echo '[TIMER] Chmod 1547077857.946078493'
Jan 09 23:50:57 ip-172-31-35-166.us-west-2.compute.internal cloud-init[1197]: [TIMER] Chmod 1547077857.946078493
Googling more, I find: https://github.com/systemd/systemd/issues/2863
This has been fixed in a later version of systemd but centos on AWS EC2 comes with systemd version 219 and I can't really update it myself. Any suggestions? Is there some config I can place to avoid this issue? I can remove most instances of sudo in my userdata script but I do need it for things like:
sudo -H -u myuser bash -ex <<EOF
... commands
EOF
FWIW Amazon Linux 2 comes with the same version of systemd but does not exhibit this behavior.
Issue and solution is noted in Redhat's link here:
https://access.redhat.com/solutions/5692661
In summary, it's not normal to run commands as sudo in a userdata script, thus the default policy is to not allow this, which causes a 25sec delay while it attempts to run pam_systemd and times out due to the dbus 25sec timeout.
In my case I was attempting to run su <user> -c "command". My error was found by running journalctl -b (-b is for current boot session). And you can find the related error log like:
pam_systemd(su:session): Failed to create session: Connection time out
Edit: Solution found via Barmar's answer. Added full smartctl command path and it works via crontab now.
I have the below script:
#!/bin/bash
#set -x
EMAIL="admin#domain.co.uk"
FILE="/root/scripts/hddreport.txt"
HOST=`hostname`
HDD01="/dev/sda"
P=`ping -c 1 $HOST | sed '1 ! d' | awk '{print $3}'`
cd /root/scripts/
echo -en "HDD health check on the server hosting" $HOST $P > $FILE
echo -e "\n" >> $FILE
smartctl -H $HDD01 >> $FILE
# The above commands do correctly write the content to $FILE (proved by removing the rm command at the bottom and doing cat on the file after)
smartctl -H $HDD01
echo "\nEmailed you the health of the Hard Drive $HDD01\n"
mailx -s "HDD health check complete on `date`" $EMAIL < $FILE
rm $FILE
which runs fine by doing bash /root/scripts/diskhealth.sh as it shows this in my mailbox:
HDD health check on the server hosting domain.co.uk (0.0.0.0)
smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
SMART Health Status: OK
But when I let it run via crontab using any of the following syntax:
X 20 * * * /bin/bash /root/scripts/diskhealth.sh
X 20 * * * /bin/sh /root/scripts/diskhealth.sh
X 20 * * * /root/scripts/diskhealth.sh
it puts everything but the smartctl disk check:
HDD health check on the server hosting domain.co.uk (0.0.0.0)
Here's what it shows if I add extra echo lines:
This is a test
HDD health check on the server hosting domain.co.uk (0.0.0.0)
Amended script for "This is a test" below:
#!/bin/bash
#set -x
EMAIL="admin#domain.co.uk"
FILE="/root/scripts/hddreport.txt"
HOST=`hostname`
HDD01="/dev/sda"
P=`ping -c 1 $HOST | sed '1 ! d' | awk '{print $3}'`
cd /root/scripts/
echo "This is a test" > $FILE
echo -en "HDD health check on the server hosting" $HOST $P >> $FILE
echo -e "\n" >> $FILE
smartctl -H $HDD01 >> $FILE
smartctl -H $HDD01
echo "\nEmailed you the health of the Hard Drive $HDD01\n"
mailx -s "HDD health check complete on `date`" $EMAIL < $FILE
rm $FILE
Here is the /var/log/syslog output from cron:
Jun 6 20:25:01 hostname /USR/SBIN/CRON[1018112]: (root) CMD (bash /root/scripts/diskhealth.sh)
Jun 6 20:25:01 hostname postfix/pickup[1016576]: 5740356613F: uid=0 from=<root>
Jun 6 20:25:01 hostname postfix/cleanup[1018125]: 5740356613F: message-id=<20130606192501.5740356613F#hostname>
Jun 6 20:25:01 hostname postfix/qmgr[292015]: 5740356613F: from=<root#hostname>, size=465, nrcpt=1 (queue active)
Jun 6 20:25:01 hostname postfix/pickup[1016576]: 631F156613E: uid=0 from=<root>
Jun 6 20:25:01 hostname postfix/cleanup[1018125]: 631F156613E: message-id=<20130606192501.631F156613E#hostname>
Jun 6 20:25:01 hostname postfix/qmgr[292015]: 631F156613E: from=<root#hostname>, size=759, nrcpt=1 (queue active)
Jun 6 20:25:01 hostname pvemailforward[1018132]: forward mail to <root#localhost.localdomain>
Jun 6 20:25:01 hostname postfix/pickup[1016576]: B597B566148: uid=65534 from=<nobody>
Jun 6 20:25:01 hostname postfix/cleanup[1018125]: B597B566148: message-id=<20130606192501.631F156613E#hostname>
Jun 6 20:25:01 hostname postfix/local[1018131]: 631F156613E: to=<root#hostname>, orig_to=<root>, relay=local, delay=0.39, delays=0.16/0/0/0.23, dsn=2.0.0, status=sent (delivered to command: /usr/bin/pvemailforward)
Jun 6 20:25:01 hostname postfix/qmgr[292015]: 631F156613E: removed
Jun 6 20:25:01 hostname postfix/qmgr[292015]: B597B566148: from=<nobody#hostname>, size=963, nrcpt=1 (queue active)
Jun 6 20:25:01 hostname postfix/smtp[1018135]: B597B566148: to=<root#localhost.localdomain>, relay=none, delay=0.16, delays=0.12/0/0.04/0, dsn=5.4.4, status=bounced (Host or domain name not found. Name service error for name=localhost.localdomain type=A: Host not found)
Jun 6 20:25:01 hostname postfix/qmgr[292015]: B597B566148: removed
Jun 6 20:25:01 hostname postfix/cleanup[1018125]: D6570566147: message-id=<20130606192501.D6570566147#hostname>
Jun 6 20:25:01 hostname postfix/smtp[1018130]: 5740356613F: to=<admin#domain.co.uk>, relay=ASPMX.L.GOOGLE.COM[173.194.67.27]:25, delay=0.68, delays=0.12/0/0.19/0.36, dsn=2.0.0, status=sent (250 2.0.0 OK 1370546701 iy4si8635735wic.1 - gsmtp)
Jun 6 20:25:01 ds9453 postfix/qmgr[292015]: 5740356613F: removed
The email is received, just missing the smartctl output.
Cron jobs don't run your .profile. So if smartctl is in a directory you add to $PATH in your profile, it won't be found when you run via cron. Try using the full pathname to the command.