I have a test program
void task()
{
long sum=0;
while(true) {
for(int i=0; i<100000;++i)
for(int j=0; j < 10000; ++j)
sum += j
}
}
int main()
{
boost::thread t(task);
t.join();
return true;
}
and I use perf record -t -e intel_pt// to profile the thread. I see a lot of calls to nmi, like 2(?) calls every some milliseconds intervals. Is there anyways to get rid of this type of interrupts? Centos 7, kernel 3.10.0, kernel.watchdog = 0.
EDIT based on comment from #Zulan : gcc version: 4.8.5.
compile line: g++ -I//include/ -L//lib -lboost_system -lboost_thread test.cpp
Processor Intel Platinum 8xxx/Gold 6xxx series.
perf script output snippnet:
a.out 25192 [009] 3976.829805: 1 branches: ffffffff816ac7f5 nmi ([kernel.kallsyms]) => ffffffff816ad500 do_nmi ([kernel.kallsyms])
a.out 25192 [009] 3976.829805: 1 branches: ffffffff816ad54f do_nmi ([kernel.kallsyms]) => ffffffff816ad5f3 do_nmi ([kernel.kallsyms])
a.out 25192 [009] 3976.829805: 1 branches: ffffffff816ad5fa do_nmi ([kernel.kallsyms]) => ffffffff8103dc10 is_debug_stack ([kernel.kallsyms])
a.out 25192 [009] 3976.829805: 1 branches: ffffffff8103dc62 is_debug_stack ([kernel.kallsyms]) => ffffffff816ad5ff do_nmi ([kernel.kallsyms])
a.out 25192 [009] 3976.829805: 1 branches: ffffffff816ad60e do_nmi ([kernel.kallsyms]) => ffffffff816ad558 do_nmi ([kernel.kallsyms])
a.out 25192 [009] 3976.829805: 1 branches: ffffffff816ad573 do_nmi ([kernel.kallsyms]) => ffffffff8113a3f0 rcu_nmi_enter ([kernel.kallsyms])
a.out 25192 [009] 3976.829805: 1 branches: ffffffff8113a41e rcu_nmi_enter ([kernel.kallsyms]) => ffffffff810ba170 __smp_mb__before_atomic ([kernel.kallsyms])
a.out 25192 [009] 3976.829805: 1 branches: ffffffff810ba17a __smp_mb__before_atomic ([kernel.kallsyms]) => ffffffff8113a423 rcu_nmi_enter ([kernel.kallsyms])
a.out 25192 [009] 3976.829805: 1 branches: ffffffff8113a427 rcu_nmi_enter ([kernel.kallsyms]) => ffffffff810ba180 __smp_mb__after_atomic ([kernel.kallsyms])
...
a.out 25192 [009] 3976.829807: 1 branches: ffffffff8113a4aa rcu_nmi_exit ([kernel.kallsyms]) => ffffffff8113a48a rcu_nmi_exit ([kernel.kallsyms])
a.out 25192 [009] 3976.829807: 1 branches: ffffffff8113a48c rcu_nmi_exit ([kernel.kallsyms]) => ffffffff816ad592 do_nmi ([kernel.kallsyms])
a.out 25192 [009] 3976.829807: 1 branches: ffffffff816ad5ed do_nmi ([kernel.kallsyms]) => ffffffff816ad760 do_nmi ([kernel.kallsyms])
a.out 25192 [009] 3976.829807: 1 branches: ffffffff816ad76e do_nmi ([kernel.kallsyms]) => ffffffff816ac7fa nmi ([kernel.kallsyms])
a.out 25192 [009] 3976.829807: 1 branches: ffffffff816ac804 nmi ([kernel.kallsyms]) => ffffffff816ac23c restore_args ([kernel.kallsyms])
a.out 25192 [009] 3976.829808: 1 branches: ffffffff816ac26c irq_return ([kernel.kallsyms]) => 40576f task (a.out)
another one:
a.out 25192 [009] 3976.829808: 1 branches: ffffffff816b6718 irq_work_interrupt ([kernel.kallsyms]) => ffffffff8102f620 smp_irq_work_interrupt ([kernel.kallsyms])
a.out 25192 [009] 3976.829808: 1 branches: ffffffff8102f629 smp_irq_work_interrupt ([kernel.kallsyms]) => ffffffff81090d20 irq_enter ([kernel.kallsyms])
a.out 25192 [009] 3976.829808: 1 branches: ffffffff81090d29 irq_enter ([kernel.kallsyms]) => ffffffff8113a360 rcu_irq_enter ([kernel.kallsyms])
a.out 25192 [009] 3976.829808: 1 branches: ffffffff8113a3a0 rcu_irq_enter ([kernel.kallsyms]) => ffffffff8113a3b8 rcu_irq_enter ([kernel.kallsyms])
a.out 25192 [009] 3976.829808: 1 branches: ffffffff8113a3c0 rcu_irq_enter ([kernel.kallsyms]) => ffffffff81137e10 rcu_eqs_exit_common.isra.31 ([kernel.kallsyms])
...
a.out 25192 [009] 3976.829809: 1 branches: ffffffff8113a27d rcu_irq_exit ([kernel.kallsyms]) => ffffffff8113a260 rcu_irq_exit ([kernel.kallsyms])
a.out 25192 [009] 3976.829809: 1 branches: ffffffff8113a26e rcu_irq_exit ([kernel.kallsyms]) => ffffffff81090e4e irq_exit ([kernel.kallsyms])
a.out 25192 [009] 3976.829809: 1 branches: ffffffff81090e50 irq_exit ([kernel.kallsyms]) => ffffffff8102f653 smp_irq_work_interrupt ([kernel.kallsyms])
a.out 25192 [009] 3976.829809: 1 branches: ffffffff8102f654 smp_irq_work_interrupt ([kernel.kallsyms]) => ffffffff816b671d irq_work_interrupt ([kernel.kallsyms])
a.out 25192 [009] 3976.829809: 1 branches: ffffffff816b671d irq_work_interrupt ([kernel.kallsyms]) => ffffffff816ac1ed ret_from_intr ([kernel.kallsyms])
a.out 25192 [009] 3976.829809: 1 branches: ffffffff816ac234 retint_swapgs ([kernel.kallsyms]) => ffffffff816ac23c restore_args ([kernel.kallsyms])
a.out 25192 [009] 3976.829809: 1 branches: ffffffff816ac26c irq_return ([kernel.kallsyms]) => 40576f task (a.out)
grep'g for irq_return() from perf script output shows 3 irq per 20ms.
a.out 25192 [009] 3976.809174: 1 branches: ffffffff816ac26c irq_return ([kernel.kallsyms]) => 405764 task (a.out)
a.out 25192 [009] 3976.809178: 1 branches: ffffffff816ac26c irq_return ([kernel.kallsyms]) => 405764 task (a.out)
a.out 25192 [009] 3976.829808: 1 branches: ffffffff816ac26c irq_return ([kernel.kallsyms]) => 40576f task (a.out)
a.out 25192 [009] 3976.829809: 1 branches: ffffffff816ac26c irq_return ([kernel.kallsyms]) => 40576f task (a.out)
a.out 25192 [009] 3976.850317: 1 branches: ffffffff816ac26c irq_return ([kernel.kallsyms]) => 405764 task (a.out)
a.out 25192 [009] 3976.850393: 1 branches: ffffffff816ac26c irq_return ([kernel.kallsyms]) => 405764 task (a.out)
a.out 25192 [009] 3976.850395: 1 branches: ffffffff816ac26c irq_return ([kernel.kallsyms]) => 405764 task (a.out)
a.out 25192 [009] 3976.870945: 1 branches: ffffffff816ac26c irq_return ([kernel.kallsyms]) => 405764 task (a.out)
a.out 25192 [009] 3976.870984: 1 branches: ffffffff816ac26c irq_return ([kernel.kallsyms]) => 40576f task (a.out)
a.out 25192 [009] 3976.870985: 1 branches: ffffffff816ac26c irq_return ([kernel.kallsyms]) => 40576f task (a.out)
a.out 25192 [009] 3976.891571: 1 branches: ffffffff816ac26c irq_return ([kernel.kallsyms]) => 40576f task (a.out)
a.out 25192 [009] 3976.891609: 1 branches: ffffffff816ac26c irq_return ([kernel.kallsyms]) => 40576f task (a.out)
a.out 25192 [009] 3976.891611: 1 branches: ffffffff816ac26c irq_return ([kernel.kallsyms]) => 40576f task (a.out)
in grub, isolcpus, nohz_full, rcu_nocbs are set on particular cpus, and nmi_watchdog=0. test program ran with taskset -c a.out.
First of all, if you want to trace kernel you will get a lot of kernel function calls in the trace, including but not limited to NMI related ones.
If you want to trace your user space program like you posted above, use
perf record -t -e intel_pt//u ...
Better to consult with documentation https://elixir.bootlin.com/linux/latest/source/tools/perf/Documentation/intel-pt.txt
Related
Im using stat module to get the size of folders and files like below:
cat test.yml
- name: "Play 1"
hosts: localhost
tasks:
- name: Check file size
stat:
path: "{{ inputf }}"
register: file_size
- name: Report file size
debug:
msg: "{{ ( file_size.stat.size / 1024 / 1024 ) | int }}MiB"
- debug:
msg: "ITEM IS {{ inputf }} and DETT: {{ file_size }}"
Below is how i run my playbook:
ansible-playbook test.yml -e inputf=/web/playbooks/automation/getfiles/filedump/file1.out
This works fine and i get the size of the file.
Output:
TASK [Report file size] *******************************************************************************
Monday 12 September 2022 02:03:43 -0500 (0:00:02.785) 0:00:04.555 ******
ok: [localhost] => {
"msg": "817MiB"
}
TASK [debug] ******************************************************************************************
Monday 12 September 2022 02:03:43 -0500 (0:00:00.069) 0:00:04.624 ******
ok: [localhost] => {
"msg": "ITEM IS /web/playbooks/automation/getfiles/filedump/file1.out and DETT: {'changed': False, 'stat': {'exists': True, 'path': '/web/playbooks/automation/getfiles/filedump/file1.out', 'mode': '0644', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 600000008, 'gid': 64395, 'size': 856686592, 'inode': 53385579, 'dev': 64770, 'nlink': 1, 'atime': 1662960143.5284567, 'mtime': 1662807256.140487, 'ctime': 1662807256.140487, 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': False, 'rgrp': True, 'xgrp': False, 'woth': False, 'roth': True, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 1673216, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': False, 'pw_name': 'wladmin', 'gr_name': 'aces', 'checksum': 'd380e27655ab9a2d4a4e157f68ec330e002f1c06', 'mimetype': 'application/octet-stream', 'charset': 'binary', 'version': '18446744072265319601', 'attributes': [], 'attr_flags': ''}, 'failed': False}"
}
However, when i provide a folder/directory instead of the file it always returns 0MB
ansible-playbook test.yml -e inputf=/web/playbooks/automation/getfiles/filedump/
Output:
TASK [Report file size] *******************************************************************************
Monday 12 September 2022 02:04:04 -0500 (0:00:00.572) 0:00:02.300 ******
ok: [localhost] => {
"msg": "0MiB"
}
TASK [debug] ******************************************************************************************
Monday 12 September 2022 02:04:04 -0500 (0:00:00.078) 0:00:02.378 ******
ok: [localhost] => {
"msg": "ITEM IS /web/playbooks/automation/getfiles/filedump and DETT: {'changed': False, 'stat': {'exists': True, 'path': '/web/playbooks/automation/getfiles/filedump', 'mode': '0755', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 600000008, 'gid': 64395, 'size': 57, 'inode': 53385577, 'dev': 64770, 'nlink': 2, 'atime': 1662961550.851861, 'mtime': 1662807259.3727617, 'ctime': 1662807259.3727617, 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': True, 'xgrp': True, 'woth': False, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False, 'blocks': 0, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': True, 'pw_name': 'wladmin', 'gr_name': 'aces', 'mimetype': 'inode/directory', 'charset': 'binary', 'version': '367679943', 'attributes': [], 'attr_flags': ''}, 'failed': False}"
}
I m looking for a generic solution to get size in MBs whether i provide file or folder.
I was also expecting ansible to have a module to provide this fuctionality for both file and folders.
Can you please suggest?
- name: get size of current folder, subfolders and files in this folder
shell: shell: "df -h . ; du -sh -- * | sort -h ; du -sh . | sort -h"
register: size
changed_when: false
args:
chdir: /var
- debug:
msg: "{{ size.stdout }}"
To have pretty output, configure in your ansible.cfg:
stdout_callback = yaml
I believe if we break it down into shell script and ansible playbook it would be easier. Here's a sample implementation.
test.yaml
## Playbok to run a shell script on localhost
- hosts: localhost
environment:
input: ""
tasks:
- name: Run a shell script
shell: bash test.sh {{ input }}
register: result
- name: Print the output
debug: var=result.stdout_lines
test.sh
#!/bin/bash
# Function to check if input is file or directory
function check_file_or_dir() {
if [ -f "$1" ]; then
# echo "File"
filesize $1
elif [ -d "$1" ]; then
# echo "Directory"
dirsize $1
else
echo "Invalid input"
fi
}
# Function to find the size of all files and folders in a directory recursively
function dirsize() {
du -sh "$1"/* | sort -h
}
# Function to find the size of a file
function filesize() {
du -sh "$1"
}
check_file_or_dir $1
Run
─ ansible-playbook test.yaml -e "input=test.sh"
PLAY [localhost] *******************************************************************************************************************************************
TASK [Gathering Facts] *************************************************************************************************************************************
ok: [localhost]
TASK [Run a shell script] **********************************************************************************************************************************
changed: [localhost]
TASK [Print the output] ************************************************************************************************************************************
ok: [localhost] => {
"result.stdout_lines": [
"4.0K\ttest.sh"
]
}
PLAY RECAP *************************************************************************************************************************************************
localhost : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
─ ansible-playbook test.yaml -e "input=/tmp/tmpdir"
PLAY [localhost] *******************************************************************************************************************************************
TASK [Gathering Facts] *************************************************************************************************************************************
ok: [localhost]
TASK [Run a shell script] **********************************************************************************************************************************
changed: [localhost]
TASK [Print the output] ************************************************************************************************************************************
ok: [localhost] => {
"result.stdout_lines": [
"4.0K\t/tmp/tmpdir/test.sh",
"4.0K\t/tmp/tmpdir/test.yaml"
]
}
PLAY RECAP *************************************************************************************************************************************************
localhost : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
You can obviously change the du command to get the info in required format.
I need to replace a cron entry in a file using sed or awk.
tried this : didnt work
sed -i 's/0 0 * * 0/0 1 * * 1/g' script.sh
script.sh
#!/bin/bash
mkdir -p .github/workflows
cd .github/workflows
touch semgrep.yml
cat << EOF > semgrep.yml
name: Semgrep
on:
pull_request: {}
push:
branches:
- master
- main
paths:
- .github/workflows/semgrep.yml
schedule:
- cron: '0 0 * * 0'
jobs:
semgrep:
name: Static Analysis Scan
runs-on: ubuntu-18-04
Kindly help me with the same .
Using mikefarah/yq to edit the file in place (-i):
yq -i '.on.schedule[].cron = "0 1 * * 1"' semgrep.yml
would turn a semgrep.yml containing
name: Semgrep
on:
pull_request: {}
push:
branches:
- master
- main
paths:
- .github/workflows/semgrep.yml
schedule:
- cron: '0 0 * * 0'
jobs:
semgrep:
name: Static Analysis Scan
runs-on: ubuntu-18-04
into one containing
name: Semgrep
on:
pull_request: {}
push:
branches:
- master
- main
paths:
- .github/workflows/semgrep.yml
schedule:
- cron: '0 1 * * 1'
jobs:
semgrep:
name: Static Analysis Scan
runs-on: ubuntu-18-04
Team,
I have a parent task that runs child IncludeTask and from this I have a value item.item.0.device
Now, in child IncludeTask, I have to use this item.item.0.device just in replacement of one value and rest of the values in loop are to be from inventory values below for just that parent item device and not ALL devices in values. however, issues is child loop is looping on all devices instead of just the device that got qualified from parent task. like it is also honoring /dev/sda after entering block that is only for nvme device type.
values:
local_volume_mount_disks:
- device: /dev/sdf
partitions:
- number: 1
start: 0%
end: 25%
storage_class: test
- device: /dev/nvme2n1
partitions:
- number: 1
start: 0%
end: 100%
storage_class: test
parent task that sets value for item.item.0.device
- name: "Run Tasks from file on condition device is not root"
include_tasks:
file: pfm_inventory_device_cp.yml
apply:
tags: pfm_inventory_device_cp
loop: "{{ is_device_root.results | unique }}"
when:
- item.rc == 1
child task
- name: Create ext4 filesystem on partitions for NVME type device
filesystem:
fstype: ext4
dev: "{{ loop_var_nvme.0.device }}{{ nvme_extn }}{{ loop_var_nvme.1.number }}"
loop: "{{ local_volume_mount_disks|subelements('partitions') }}"
loop_control:
loop_var: loop_var_nvme
when: item.item.0.device is regex("nvme\w+")
Now, what is happening is my child task instead of skipping sda* device it is getting executed on non-match regex because of loop that am using inside.
error output
TASK [local_volume_mount : Create ext4 filesystem on partitions for NVME type device] ***
Friday 07 May 2021 16:48:48 +0000 (0:00:00.063) 0:00:11.060 ************
ok: [node1] => (item=[{'device': '/dev/nvme2n1', 'partitions': [{'number': 1, 'start': '0%', 'end': '100%', 'storage_class': 'test'}]}, {'number': 1, 'start': '0%', 'end': '100%', 'storage_class': 'test'}])
failed: [node1] (item=[{'device': '/dev/sdf', 'partitions': [{'number': 1, 'start': '0%', 'end': '25%', 'storage_class': 'test'}, {'number': 2, 'start': ‘0%’, 'end': ’25%’, 'storage_class': 'test'}]}, {'number': 1, 'start': '0%', 'end': '25%', 'storage_class': 'test'}]) => {"ansible_loop_var": "loop_var_nvme", "changed": false, "loop_var_nvme": [{"device": "/dev/sdf", "partitions": [{"end": "25%", "number": 1, "start": "0%", "storage_class": "test"}, {"end": "50%", "number": 2, "start": "25%", "storage_class": "test"}]}, {"end": "25%", "number": 1, "start": "0%", "storage_class": "test"}], "msg": "Device /dev/sdfp1 not found."}
expected out
TASK [local_volume_mount : Create ext4 filesystem on partitions for NVME type device] ***
Friday 07 May 2021 16:48:48 +0000 (0:00:00.063) 0:00:11.060 ************
ok: [node1] => (item=[{'device': '/dev/nvme2n1', 'partitions': [{'number': 1, 'start': '0%', 'end': '100%', 'storage_class': 'test'}]}, {'number': 1, 'start': '0%', 'end': '100%', 'storage_class': 'test'}])
skipping: [node1] => (item=[{'device': '/dev/sda', 'partitions': [{'number': 1, 'start': '0%', 'end': '25%', 'storage_class': 'test'}]}, {'number': 1, 'start': '0%', 'end': '100%', 'storage_class': 'test'}])
The issue seems to be in the use of loop_var to get the value of list elements, but using item in the when condition. Either use item everywhere, or loop_var_nvme.
A task like this, skips the item when device does not match "nvme":
- debug:
msg: "Device = {{ loop_var_nvme.0.device }}; Device number = {{ loop_var_nvme.1.number }}"
loop: "{{ local_volume_mount_disks|subelements('partitions') }}"
loop_control:
loop_var: loop_var_nvme
when: loop_var_nvme.0.device is regex("nvme\w+")
When run:
TASK [debug] ********************************************************************************************************************************************************************************
skipping: [localhost] => (item=[{'device': '/dev/sdf', 'partitions': [{'number': 1, 'start': '0%', 'end': '25%', 'storage_class': 'test'}]}, {'number': 1, 'start': '0%', 'end': '25%', 'storage_class': 'test'}])
ok: [localhost] => (item=[{'device': '/dev/nvme2n1', 'partitions': [{'number': 1, 'start': '0%', 'end': '100%', 'storage_class': 'test'}]}, {'number': 1, 'start': '0%', 'end': '100%', 'storage_class': 'test'}]) => {
"msg": "Device = /dev/nvme2n1; Device number = 1"
}
I am not able to execute shell script remotely in Ansible. However, there are previous tasks in the same role (filebeat) that are executed in remote server successfully. I am running the following in local server 172.28.28.6 server to install and run filebeat in remote server 172.28.28.81
Playbook: install-filebeat.yml:
hosts: filebeat-servers
remote_user: wwwadm
sudo: yes
roles:
- { role: /vagrant/roles/filebeat}
Role filebeat: main.yml:
---
# tasks file for filebeat
- name: "Extract Filebeat"
unarchive:
src: "{{ tmp_artifact_cache }}/{{ filebeat_archive }}"
remote_src: yes
dest: "{{ filebeat_root_dir }}"
extra_opts: ['--transform=s,/*[^/]*,{{ filebeat_ver }},i', '--show-stored-names']
become: yes
become_user: "{{ filebeat_install_as }}"
when: not ansible_check_mode
tags: [ 'filebeat' ]
- name: Configure Filebeat
template:
src: "filebeat.yml.j2"
dest: "{{ filebeat_install_dir }}/filebeat.yml"
mode: 0775
become: yes
become_user: "{{ filebeat_install_as }}"
tags: [ 'filebeat' ]
- name: 'Filebeat startup script'
template:
src: "startup.sh.j2"
dest: "{{ filebeat_install_dir }}/bin/startup.sh"
mode: 0755
become: yes
become_user: "{{ filebeat_install_as }}"
tags: [ 'filebeat', 'start' ]
#This one does not get executed at all:
- name: "Start Filebeat"
# shell: "{{ filebeat_install_dir }}/bin/startup.sh"
command: "sh {{ filebeat_install_dir }}/bin/startup.sh"
become: yes
become_user: "{{ filebeat_install_as }}"
defaults:
# defaults file for filebeat
filebeat_ver: "6.6.0"
filebeat_archive: "filebeat-{{ filebeat_ver }}-linux-x86_64.tar.gz"
filebeat_archive_checksum : "sha1:d38d8fea7e9915582720280eb0118b7d92569b23"
filebeat_url: "https://artifacts.elastic.co/downloads/beats/filebeat/{{ filebeat_archive }}"
filebeat_root_dir: "{{ apps_home }}/filebeat"
filebeat_data_dir: "{{ apps_data }}/filebeat"
filebeat_log_dir: "{{ apps_logs }}/filebeat"
filebeat_install_dir: "{{ filebeat_root_dir }}/{{ filebeat_ver }}"
filebeat_cert_dir: "/etc/pki/tls/certs"
filebeat_ssl_certificate_file: "logstash.crt"
filebeat_ssl_key_file: "logstash.key"
filebeat_install_as: "{{ install_user | default('wwwadm') }}"
filebeat_set_as_current: yes
filebeat_force_clean_install: no
filebeat_java_home: "{{ sw_home }}/jdk"
inventory/local/hosts:
localhost ansible_connection=local
[filebeat-servers]
172.28.28.81 ansible_user=vagrant ansible_connection=ssh
Filebeat is installed and changes are done in the remote server except the last step which is the execution of shell script
When running the playbook as follows:
ansible-playbook -i /vagrant/inventory/local install-filebeat.yml -vvv
Getting the following output related to the shell execution:
TASK [/vagrant/roles/filebeat : Start Filebeat] ***************************************************************************************************************************************************************
task path: /vagrant/roles/filebeat/tasks/main.yml:184
<172.28.28.81> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.28.81> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/f66f05c055 172.28.28.81 '/bin/sh -c '"'"'echo ~vagrant && sleep 0'"'"''
<172.28.28.81> (0, '/home/vagrant\n', '')
<172.28.28.81> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.28.81> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/f66f05c055 172.28.28.81 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /var/tmp/ansible-tmp-1550178583.24-35955954120606 `" && echo ansible-tmp-1550178583.24-35955954120606="` echo /var/tmp/ansible-tmp-1550178583.24-35955954120606 `" ) && sleep 0'"'"''
<172.28.28.81> (0, 'ansible-tmp-1550178583.24-35955954120606=/var/tmp/ansible-tmp-1550178583.24-35955954120606\n', '')
Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
<172.28.28.81> PUT /home/vagrant/.ansible/tmp/ansible-local-13658UX7cBC/tmpFzf2Ll TO /var/tmp/ansible-tmp-1550178583.24-35955954120606/AnsiballZ_command.py
<172.28.28.81> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/f66f05c055 '[172.28.28.81]'
<172.28.28.81> (0, 'sftp> put /home/vagrant/.ansible/tmp/ansible-local-13658UX7cBC/tmpFzf2Ll /var/tmp/ansible-tmp-1550178583.24-35955954120606/AnsiballZ_command.py\n', '')
<172.28.28.81> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.28.81> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/f66f05c055 172.28.28.81 '/bin/sh -c '"'"'setfacl -m u:wwwsvr:r-x /var/tmp/ansible-tmp-1550178583.24-35955954120606/ /var/tmp/ansible-tmp-1550178583.24-35955954120606/AnsiballZ_command.py && sleep 0'"'"''
<172.28.28.81> (0, '', '')
<172.28.28.81> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.28.81> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/f66f05c055 -tt 172.28.28.81 '/bin/sh -c '"'"'sudo -H -S -n -u wwwsvr /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-ntzchfzqggiteuqwzpiurlloddbdhevp; /usr/bin/python /var/tmp/ansible-tmp-1550178583.24-35955954120606/AnsiballZ_command.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<172.28.28.81> (0, '\r\n{"changed": true, "end": "2019-02-14 13:09:44.800191", "stdout": "Starting Filebeat", "cmd": ["sh", "/apps_ux/filebeat/6.6.0/bin/startup.sh"], "rc": 0, "start": "2019-02-14 13:09:43.792122", "stderr": "+ export JAVA_HOME=/sw_ux/jdk\\n+ JAVA_HOME=/sw_ux/jdk\\n+ echo \'Starting Filebeat\'\\n+ /apps_ux/filebeat/6.6.0/bin/filebeat -c /apps_ux/filebeat/6.6.0/config/filebeat.yml -path.home /apps_ux/filebeat/6.6.0 -path.config /apps_ux/filebeat/6.6.0/config -path.data /apps_data/filebeat -path.logs /apps_data/logs/filebeat", "delta": "0:00:01.008069", "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "sh /apps_ux/filebeat/6.6.0/bin/startup.sh", "removes": null, "argv": null, "creates": null, "chdir": null, "stdin": null}}}\r\n', 'Shared connection to 172.28.28.81 closed.\r\n')
<172.28.28.81> ESTABLISH SSH CONNECTION FOR USER: vagrant
<172.28.28.81> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/f66f05c055 172.28.28.81 '/bin/sh -c '"'"'rm -f -r /var/tmp/ansible-tmp-1550178583.24-35955954120606/ > /dev/null 2>&1 && sleep 0'"'"''
<172.28.28.81> (0, '', '')
changed: [172.28.28.81] => {
"changed": true,
"cmd": [
"sh",
"/apps_ux/filebeat/6.6.0/bin/startup.sh"
],
"delta": "0:00:01.008069",
"end": "2019-02-14 13:09:44.800191",
"invocation": {
"module_args": {
"_raw_params": "sh /apps_ux/filebeat/6.6.0/bin/startup.sh",
"_uses_shell": false,
"argv": null,
"chdir": null,
"creates": null,
"executable": null,
"removes": null,
"stdin": null,
"warn": true
}
},
"rc": 0,
"start": "2019-02-14 13:09:43.792122",
"stderr": "+ export JAVA_HOME=/sw_ux/jdk\n+ JAVA_HOME=/sw_ux/jdk\n+ echo 'Starting Filebeat'\n+ /apps_ux/filebeat/6.6.0/bin/filebeat -c /apps_ux/filebeat/6.6.0/config/filebeat.yml -path.home /apps_ux/filebeat/6.6.0 -path.config /apps_ux/filebeat/6.6.0/config -path.data /apps_data/filebeat -path.logs /apps_data/logs/filebeat",
"stderr_lines": [
"+ export JAVA_HOME=/sw_ux/jdk",
"+ JAVA_HOME=/sw_ux/jdk",
"+ echo 'Starting Filebeat'",
"+ /apps_ux/filebeat/6.6.0/bin/filebeat -c /apps_ux/filebeat/6.6.0/config/filebeat.yml -path.home /apps_ux/filebeat/6.6.0 -path.config /apps_ux/filebeat/6.6.0/config -path.data /apps_data/filebeat -path.logs /apps_data/logs/filebeat"
],
"stdout": "Starting Filebeat",
"stdout_lines": [
"Starting Filebeat"
]
}
META: ran handlers
META: ran handlers
PLAY RECAP ****************************************************************************************************************************************************************************************************
172.28.28.81 : ok=18 changed=7 unreachable=0 failed=0
On remote server:
[6.6.0:vagrant]$ cd bin
[bin:vagrant]$ ls -ltr
total 36068
-rwxr-xr-x. 1 wwwadm wwwadm 36927014 Jan 24 02:30 filebeat
-rwxr-xr-x. 1 wwwadm wwwadm 478 Feb 14 12:54 startup.sh
[bin:vagrant]$ pwd
/apps_ux/filebeat/6.6.0/bin
[bin:vagrant]$ more startup.sh
#!/usr/bin/env bash
set -x
export JAVA_HOME="/sw_ux/jdk"
#To save pid into a file is an open feature: https://github.com/elastic/logstash/issues/3577. There is no -p flag for filebeat to save the pid and then kill it.
echo 'Starting Filebeat'
/apps_ux/filebeat/6.6.0/bin/filebeat -c /apps_ux/filebeat/6.6.0/config/filebeat.yml -path.home /apps_ux/filebeat/6.6.0 -path.config /apps_ux/filebeat/6.6.0/config -path.data /apps_data/filebeat -path.logs /a
pps_data/logs/filebeat &
No process running found by executing ps command
[bin:vagrant]$ ps -fea | grep filebeat | grep -v grep
However, if I connect to the remote server, I am able to run filebeat by executing the script with the user wwwadm and filebeat starts successfully:
[bin:wwwadm]$ pwd
/apps_ux/filebeat/6.6.0/bin
[bin:wwwadm]$ id
uid=778(wwwadm) gid=778(wwwadm) groups=778(wwwadm) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[bin:wwwadm]$ ./startup.sh
+ export JAVA_HOME=/sw_ux/jdk
+ JAVA_HOME=/sw_ux/jdk
+ echo 'Starting Filebeat'
Starting Filebeat
+ /apps_ux/filebeat/6.6.0/bin/filebeat -c /apps_ux/filebeat/6.6.0/config/filebeat.yml -path.home /apps_ux/filebeat/6.6.0 -path.config /apps_ux/filebeat/6.6.0/config -path.data /apps_data/filebeat -path.logs /apps_data/logs/filebeat
[bin:wwwadm]$ ps -fea | grep filebeat | grep -v grep
wwwadm 19160 1 0 15:12 pts/0 00:00:00 /apps_ux/filebeat/6.6.0/bin/filebeat -c /apps_ux/filebeat/6.6.0/config/filebeat.yml -path.home /apps_ux/filebeat/6.6.0 -path.config /apps_ux/filebeat/6.6.0/config -path.data /apps_data/filebeat -path.logs /apps_data/logs/filebeat
Thanks
You should use nohup to run it in background.
because when ansible exits, all processes associated with the session
will be terminated. To avoid this you should use nohup.
Correct command is:
- name: "Start Filebeat"
# shell: "{{ filebeat_install_dir }}/bin/startup.sh"
command: "nohup sh {{ filebeat_install_dir }}/bin/startup.sh &>> startup.log &"
become: yes
become_user: "{{ filebeat_install_as }}"
You have to use the disown built-in command to inform the shell that it should not kill background processes when you disconnect; you can also use nohup for that same effect
Having said that, you are for sure solving the wrong problem, because if^H^Hwhen filebeat falls over, there is nothing monitoring that service to keep it alive. You'll want to use systemd (or its equivalent on your system) to ensure that filebeat stays running, and by using the mechanism designed for that stuff, you side-step all the "disown or nohup" business that causes you to ask S.O. questions.
I have a command that needs to be executed remotely.
find /opt/cac/CI/release/releases -name "*.tar" -exec md5sum {} \;
the complete output is ::
e5af5514887e8cbc08815936558a3220 /opt/cac/CI/release/releases/permission/4.00.00.04/CXP_902_8059_4.00.00.04_20161007-131208.tar
d35eae58399770627ba42f5b538a9cab /opt/cac/CI/release/releases/privacyvault/4.08.01.00/CXP_902_8185_4.08.01.00_20160927-052332.tar
784d035f9959f6a3006aaab202c13015 /opt/cac/CI/release/releases/privacyvault/4.08.01.00/CXP_902_8185_4.08.01.00_20160927-060837.tar
3234cc8944c1d26c1ff0fac844ae8674 /opt/cac/CI/release/releases/privacyvault/4.08.01.00/CXP_902_8185_4.08.01.00_20160927-062202.tar
431368042e9f5e62de37787cd2d05b08 /opt/cac/CI/release/releases/privacyvault/4.08.01.00/CXP_902_8185_4.08.01.00_20161008-173030.tar
I am trying to do the same using Ansible :
- hosts: tmoittecac
tasks:
- name: Report md5sum of Release files.
shell: find /opt/cac/CI/release/releases -name "*.tar" -exec md5sum {} \;
register: pre_md5_check
tags: a
- debug: msg={{ pre_md5_check.stdout_lines }}
tags: b
But the output that I get is ::
TASK: [debug msg={{ pre_md5_check.stdout_lines }}] ****************************
ok: [tmoittecac] => {
"msg": "['e5af5514887e8cbc08815936558a3220 /opt/cac/CI/release/releases/permission/4.00.00.04/CXP_902_8059_4.00.00.04_20161007-131208.tar',"
}
I am only getting the first line of the actual output.
Running the playbook in verbose mode gives me.
TASK: [Report md5sum of Release files.] ***************************************
<147.128.72.59> ESTABLISH CONNECTION FOR USER: local
<147.128.72.59> REMOTE_MODULE command find /opt/cac/CI/release/releases -name "*.tar" -exec md5sum {} \; #USE_SHELL
<147.128.72.59> EXEC sshpass -d7 ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/root/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no -o PubkeyAuthentication=no -o User=local -o ConnectTimeout=10 147.128.72.59 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1476216014.64-144818429840055 && echo $HOME/.ansible/tmp/ansible-tmp-1476216014.64-144818429840055'
<147.128.72.59> PUT /tmp/tmpm6eavD TO /opt/home/local/.ansible/tmp/ansible-tmp-1476216014.64-144818429840055/command
<147.128.72.59> EXEC sshpass -d7 ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/root/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no -o PubkeyAuthentication=no -o User=local -o ConnectTimeout=10 147.128.72.59 /bin/sh -c 'LANG=C LC_CTYPE=C /usr/bin/python /opt/home/local/.ansible/tmp/ansible-tmp-1476216014.64-144818429840055/command; rm -rf /opt/home/local/.ansible/tmp/ansible-tmp-1476216014.64-144818429840055/ >/dev/null 2>&1'
changed: [tmoittecac] => {"changed": true, "cmd": "find /opt/cac/CI/release/releases -name \"*.tar\" -exec md5sum {} \\;", "delta": "0:00:01.172660", "end": "2016-10-12 03:53:14.020937", "rc": 0, "start": "2016-10-12 03:53:12.848277", "stderr": "", "stdout": "e5af5514887e8cbc08815936558a3220 /opt/cac/CI/release/releases/permission/4.00.00.04/CXP_902_8059_4.00.00.04_20161007-131208.tar\nd35eae58399770627ba42f5b538a9cab /opt/cac/CI/release/releases/privacyvault/4.08.01.00/CXP_902_8185_4.08.01.00_20160927-052332.tar\n784d035f9959f6a3006aaab202c13015 /opt/cac/CI/release/releases/privacyvault/4.08.01.00/CXP_902_8185_4.08.01.00_20160927-060837.tar\n3234cc8944c1d26c1ff0fac844ae8674 /opt/cac/CI/release/releases/privacyvault/4.08.01.00/CXP_902_8185_4.08.01.00_20160927-062202.tar\n431368042e9f5e62de37787cd2d05b08 /opt/cac/CI/release/releases/privacyvault/4.08.01.00/CXP_902_8185_4.08.01.00_20161008-173030.tar\n7f637400276cb25f7f3b2f869d915dc7 /opt/cac/CI/release/releases/notification/4.00.00.03/CXP_902_8347_4.00.00.03_20160928-070050.tar\nfa4a0aea0c096c703f2a9a741d2d1152 /opt/cac/CI/release/releases/user-preference/4.08.01.00/CXP_902_8717_4.08.01.00_20160929-034340.tar\n34f084c617f49123fc0edef358d15784 /opt/cac/CI/release/releases/captcha/2.08.01.00/CXP_902_8881_2.08.01.00_20160929-043449.tar\n2041d873f0619f1c9a4c8e419156753e /opt/cac/CI/release/releases/consumerProfile/3.08.02.00/CXP_902_8057_3.08.02.00_20161008-063249.tar\n7a0a9efe4232eebbb5c05e5b84fb6bec /opt/cac/CI/release/releases/consumerProfile/3.08.02.00/CXP_902_8057_3.08.02.00_20161008-071824.tar", "warnings": []}
TASK: [debug msg={{ pre_md5_check.stdout_lines }}] ****************************
<147.128.72.59> ESTABLISH CONNECTION FOR USER: local
ok: [tmoittecac] => {
"msg": "['e5af5514887e8cbc08815936558a3220 /opt/cac/CI/release/releases/permission/4.00.00.04/CXP_902_8059_4.00.00.04_20161007-131208.tar',"
}
Is these something I am missing ?
More INFO ::
tasks:
- name: Check md5sums of WAR files.
shell: find /opt/cac/CI/release/releases -name "*.tar" -exec md5sum {} \;
register: MD5SUMS
tags: a
- name: Output md5sum of WAR files.
debug: var={{ MD5SUMS }}
tags: b
Output ::
GATHERING FACTS ***************************************************************
ok: [tmoittecac]
TASK: [Check md5sums of WAR files.] *******************************************
changed: [tmoittecac]
TASK: [Output md5sum of WAR files.] *******************************************
ok: [tmoittecac] => {
"var": {
"{'changed':": "{'changed':"
}
}
PLAY RECAP ********************************************************************
tmoittecac : ok=3 changed=1 unreachable=0 failed=0
Ansible version is :
ansible --version
ansible 1.9.4
Issue was solved by upgrading Ansible version to 2.1.