I have a high availability cluster with two nodes, with a resource for drbd, a virtual IP and the mariaDB files shared on the drbd partition.
Everything seems to work OK, but drbd is not syncing the latest files I have created, even though drbd status tells me they are UpToDate.
sudo drbdadm status
iba role:Primary
disk:UpToDate
Pcs also does not show errors
sudo pcs status
Cluster name: cluster_iba
Cluster Summary:
* Stack: corosync
* Current DC: iba2-ip192 (version 2.0.3-4b1f869f0f) - partition with quorum
* Last updated: Tue Feb 22 18:16:20 2022
* Last change: Mon Feb 21 16:19:38 2022 by root via cibadmin on iba1-ip192
* 2 nodes configured
* 6 resource instances configured
Node List:
* Online: [ iba1-ip192 iba2-ip192 ]
Full List of Resources:
* virtual_ip (ocf::heartbeat:IPaddr2): Started iba2-ip192
* Clone Set: DrbdData-clone [DrbdData] (promotable):
* Masters: [ iba2-ip192 ]
* Slaves: [ iba1-ip192 ]
* DrbdFS (ocf::heartbeat:Filesystem): Started iba2-ip192
* WebServer (ocf::heartbeat:apache): Started iba2-ip192
* Maria (ocf::heartbeat:mysql): Started iba2-ip192
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
All constraint:
sudo pcs constraint list --full
Location Constraints:
Ordering Constraints:
promote DrbdData-clone then start DrbdFS (kind:Mandatory) (id:order-DrbdData-clone-DrbdFS-mandatory)
start DrbdFS then start virtual_ip (kind:Mandatory) (id:order-DrbdFS-virtual_ip-mandatory)
start virtual_ip then start WebServer (kind:Mandatory) (id:order-virtual_ip-WebServer-mandatory)
start DrbdFS then start Maria (kind:Mandatory) (id:order-DrbdFS-Maria-mandatory)
Colocation Constraints:
DrbdFS with DrbdData-clone (score:INFINITY) (with-rsc-role:Master) (id:colocation-DrbdFS-DrbdData-clone-INFINITY)
virtual_ip with DrbdFS (score:INFINITY) (id:colocation-virtual_ip-DrbdFS-INFINITY)
WebServer with virtual_ip (score:INFINITY) (id:colocation-WebServer-virtual_ip-INFINITY)
Maria with DrbdFS (score:INFINITY) (id:colocation-Maria-DrbdFS-INFINITY)
Ticket Constraints:
The files in /mnt/datosDRBD in node iba2-ip192 (when it's the master),
/mnt/datosDRBD$ ls -l
total 80
-rw-r--r-- 1 root root 5801 feb 21 12:16 drbd_cfg
-rw-r--r-- 1 root root 10494 feb 21 12:18 fs_cfg
drwx------ 2 root root 16384 feb 21 10:12 lost+found
drwxr-xr-x 4 mysql mysql 4096 feb 22 18:00 mariaDB
-rw-r--r-- 1 root root 17942 feb 21 12:39 MariaDB_cfg
-rw-r--r-- 1 root root 5 feb 21 10:13 testMParicio.txt
-rw-r--r-- 1 root root 13578 feb 21 12:21 WebServer_cfg
And the files in /mnt/datosDRBD in node iba1-ip192 (when it's the master),
ls -l
total 92
-rw-r--r-- 1 root root 5801 feb 21 12:16 drbd_cfg
drwxrwxrwx 5 www-data www-data 4096 feb 22 13:41 FilesSGITV
-rw-r--r-- 1 root root 10494 feb 21 12:18 fs_cfg
drwx------ 2 root root 16384 feb 21 10:12 lost+found
drwxr-xr-x 7 mysql mysql 4096 feb 22 17:55 mariaDB
-rw-r--r-- 1 root root 17942 feb 21 12:39 MariaDB_cfg
-rw-r--r-- 1 root root 5 feb 22 17:58 testMParicio2.txt
-rw-r--r-- 1 www-data www-data 9 feb 22 17:58 testMParicio3.txt
-rw-r--r-- 1 root root 5 feb 21 10:13 testMParicio.txt
-rw-r--r-- 1 root root 13578 feb 21 12:21 WebServer_cfg
All new files, testMParicio2.txt testMParicio3.txt and the folder FilesSGITV are missing.
I do not know what to do. I am very lost.
I appreciate any help, thanks.
(EDIT)
My config for drbd, in both nodes...
cat /etc/drbd.conf
# You can find an example in /usr/share/doc/drbd.../drbd.conf.example
include "drbd.d/global_common.conf";
include "drbd.d/*.res";
And my *.res config, in both nodes too:
resource iba {
device /dev/drbd0;
disk /dev/md3;
meta-disk internal;
on iba1 {
address 10.0.0.248:7789;
}
on iba2 {
address 10.0.0.249:7789;
}
}
drbdadm use iba1 and iba2, with IP 10.0.0.248 and 10.0.0.249
Corosync use iba1-ip192 and iba2-192, with IP 192.168.1.248 and 192.168.1.249
cat /etc/hosts
127.0.0.1 localhost
#127.0.1.1 iba1
10.0.0.248 iba1
10.0.0.249 iba2
192.168.1.248 iba1-ip192
192.168.1.249 iba2-ip192
cat /etc/drbd.d/global_common.conf
global {
usage-count yes;
udev-always-use-vnr; # treat implicit the same as explicit volumes
}
common {
handlers {
}
startup {
}
options {
}
disk {
}
net {
protocol C;
}
}
(EDIT 2)
I have found a problem in /proc/drbd
In primary node:
cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: FC3433D849E3B88C1E7B55C
0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----
ns:0 nr:0 dw:2284 dr:11625 al:6 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:42364728
in secondary node
cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: FC3433D849E3B88C1E7B55C
0: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:36538580
Secondary node don't remember ssh key, fix with
ssh-keygen -R 10.0.0.248
ssh-copy-id iba#iba1
But drbd still with StandAlone status.
I don't know how to continue
I have found a Split-Brain that did not appear in the status of pcs.
sudo journalctl | grep Split-Brain
feb 21 13:00:10 ibatec1 kernel: block drbd0: Split-Brain detected but unresolved, dropping connection!
feb 21 13:21:40 ibatec1 kernel: block drbd0: Split-Brain detected but unresolved, dropping connection!
feb 21 13:27:54 ibatec1 kernel: block drbd0: Split-Brain detected but unresolved, dropping connection!
I have stopped the cluster, with --force on the master,
Then...
On split-brain victim (assuming the DRBD resource is iba):
drbdadm disconnect iba
drbdadm secondary iba
drbdadm connect --discard-my-data iba
On split-brain survivor:
drbdadm primary iba
drbdadm connect iba
Related
I want to make my writing to the console output in a nice tidy human readable.
here is how it looks now:
====================== Sat Apr 16 12:57:17 EDT 2022 ======================
==========================================================================
====================== Leopard - Download from S3 ======================
==========================================================================
==========================================================================
====================== Leopard - Decompressing ======================
==========================================================================
total 1349872
drwxr-xr-x 2 root root 12288 Apr 16 12:57 .
drwxrwxrwt. 4 root root 102 Apr 16 12:57 ..
-rw-r--r-- 1 root root 185070885 Apr 16 12:03 asdasdasd.sql.gz
-rw-r--r-- 1 root root 40344632 Apr 16 12:03 asdasdas.sql.gz
-rw-r--r-- 1 root root 26631 Apr 16 12:03 asdad.sql.gz
-rw-r--r-- 1 root root 1679 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1237 Apr 16 12:03 asd.sql.gz
-rw-r--r-- 1 root root 5241900 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1144 Apr 16 12:03 asdasasd.sql.gz
-rw-r--r-- 1 root root 489312 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1138 Apr 16 12:03 asdasdasd.sql.gz
==========================================================================
====================== NewYorkCity - Download from S3 ======================
==========================================================================
==========================================================================
====================== NewYorkCity - Unloading SSL Example ======================
==========================================================================
total 1349872
drwxr-xr-x 2 root root 12288 Apr 16 12:57 .
drwxrwxrwt. 4 root root 102 Apr 16 12:57 ..
-rw-r--r-- 1 root root 185070885 Apr 16 12:03 asdasdasd.sql.gz
-rw-r--r-- 1 root root 40344632 Apr 16 12:03 asdasdas.sql.gz
-rw-r--r-- 1 root root 26631 Apr 16 12:03 asdad.sql.gz
-rw-r--r-- 1 root root 1679 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1237 Apr 16 12:03 asd.sql.gz
-rw-r--r-- 1 root root 5241900 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1144 Apr 16 12:03 asdasasd.sql.gz
-rw-r--r-- 1 root root 489312 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1138 Apr 16 12:03 asdasdasd.sql.gz
I want that all the === line will be in the same length, and the text always in the center with 1 space on each side
Will appreciate assistance here :)
UPDATE / EDIT:
The original script is something like that:
eecho () { echo ==========================================================================; }
echo_stage () {
START=1
END=11
for (( c=$START; c<=$END; c++ ))
do
printf == '-%.0s'
done
echo -n " " $1 " "
for (( c=$START; c<=$END; c++ ))
do
printf == '-%.0s'
done
echo
}
stage() {
eecho
echo_stage "$1" "$2"
eecho
}
print_date () { echo "======================" $(date) "======================"; }
reload_db() {
print_date
rm -rf /var/tmp/db
mkdir -p /var/tmp/db
stage "DB - Download from S3"
aws s3 sync s3://db-backup/latest/ /var/tmp/db --profile=papilon --quiet
stage "DB - Decompressing"
pigz -d /var/tmp/db/*
stage "DB - Restoring Data"
cd /var/tmp/db
stage "DB - Restoring Tables"
for i in `ls -1 *.sql | grep -v "_view.sql"`;do echo $i;mysql db < $i;done
stage "DB - Restoring Views"
for i in `ls -1 *.sql | grep "_view.sql"`;do echo $i;mysql db < $i;done
stage "DB - Clean up"
rm -rf /var/tmp/db
print_date
}
reload_db
This awk filter will size your === padding correctly, and align titles to the center, if you pipe your command output through it:
# cmd |
awk '
BEGIN {a[1] = "="}
NF>1 && $1~/^=+$/ && $NF~/^=+$/ {
sub(/^=+/, "")
sub(/=+$/, "")
title_len=length($0)
pad = ""
for (i=1; i<=(74-title_len)/2; ++i) {
pad=pad"="
}
$0 = pad $0 pad a[title_len%2]
}
1'
This doesn't wrap the ls -l output, only the titles. It assumes 74 is hardcoded as the length of the solid === lines (and doesn't change depending on terminal size). The array a is used to add an extra = when 74 - title_len is an odd number.
Example output:
====================== Sat Apr 16 12:57:17 EDT 2022 ======================
==========================================================================
====================== Leopard - Download from S3 ======================
==========================================================================
==========================================================================
======================= Leopard - Decompressing ========================
==========================================================================
total 1349872
drwxr-xr-x 2 root root 12288 Apr 16 12:57 .
drwxrwxrwt. 4 root root 102 Apr 16 12:57 ..
-rw-r--r-- 1 root root 185070885 Apr 16 12:03 asdasdasd.sql.gz
-rw-r--r-- 1 root root 40344632 Apr 16 12:03 asdasdas.sql.gz
-rw-r--r-- 1 root root 26631 Apr 16 12:03 asdad.sql.gz
-rw-r--r-- 1 root root 1679 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1237 Apr 16 12:03 asd.sql.gz
-rw-r--r-- 1 root root 5241900 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1144 Apr 16 12:03 asdasasd.sql.gz
-rw-r--r-- 1 root root 489312 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1138 Apr 16 12:03 asdasdasd.sql.gz
==========================================================================
==================== NewYorkCity - Download from S3 ====================
==========================================================================
==========================================================================
================= NewYorkCity - Unloading SSL Example ==================
==========================================================================
total 1349872
drwxr-xr-x 2 root root 12288 Apr 16 12:57 .
drwxrwxrwt. 4 root root 102 Apr 16 12:57 ..
-rw-r--r-- 1 root root 185070885 Apr 16 12:03 asdasdasd.sql.gz
-rw-r--r-- 1 root root 40344632 Apr 16 12:03 asdasdas.sql.gz
-rw-r--r-- 1 root root 26631 Apr 16 12:03 asdad.sql.gz
-rw-r--r-- 1 root root 1679 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1237 Apr 16 12:03 asd.sql.gz
-rw-r--r-- 1 root root 5241900 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1144 Apr 16 12:03 asdasasd.sql.gz
-rw-r--r-- 1 root root 489312 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1138 Apr 16 12:03 asdasdasd.sql.gz
Because you have now posted your script, I will add a new answer for pure bash.
title() {
local text pad
(( ${#1} > 70 )) && { echo "$1"; return; }
text=${1:+ }$1${1:+ }
pad=$( eval "printf %.1s ={1..$(( ( 74 - ${#text} ) / 2 ))}" )
echo "$pad$text$pad$( (( ${#text} % 2 )) && printf = )"
}
For title 'foo bar', this function prints foo bar (or any string up to 70 characters) with a space either side, centered and padded to 74 columns with =. If the string is longer than 70 characters it's too long to pad, so it's printed as is.
================================ foo bar =================================
With no argument, or an empty argument, it prints a solid line of 74 =:
==========================================================================
You can swap = for any single ASCII character.
You can swap 74 for any even number. Also for an odd number, if you change && printf to || printf in the last line. (also change the 70 to N - 4)
You can call title once for the date, or three times for the larger three line banners (see banner below).
Explanation:
${#text} is bash for "length of $text".
${1:+ } expands to a space, unless $1 is empty or unset. This allows us to add spaces to either end of the string, or exclude them for an empty string.
={1..10} expands to =1 =2 =3 ... =10, and printf %.1s prints the first character of each string. Combining these allows us to repeat a string (=) N times.
But we can't use a variable (or arithmetic) in {1..10} normally. So we need eval.
You probably heard eval is bad, and a security risk. That's often true, but here we are not passing any unknown data to eval (such as user input), and it's safe from code injection. (${#text} always expands to a single number)
So we make two bars, of length (74 - text-length) / 2, adding another = to the second bar if the text length is an odd number.
I also made a few changes to your script which you might consider. Apart from the date and titles, these have nothing to do with the padding. The title and banner functions will work on your old script.
title() {
local text pad
(( ${#1} > 70 )) && { echo "$1"; return; }
text="${1:+ }$1${1:+ }"
pad=$( eval "printf %.1s ={1..$(( ( 74 - ${#text} ) / 2 ))}" )
echo "$pad$text$pad$( (( ${#text} % 2 )) && printf = )"
}
banner() {
title
title "$1"
title
}
reload_db() {
local i
title "$(date)"
rm -rf /var/tmp/db || exit 1
mkdir -p /var/tmp/db || exit 1
banner 'DB - Download from S3'
aws s3 sync s3://db-backup/latest/ /var/tmp/db --profile=papilon --quiet
banner 'DB - Decompressing'
pigz -d /var/tmp/db/*
banner 'DB - Restoring Data'
cd /var/tmp/db || exit 1
banner 'DB - Restoring Tables'
GLOBIGNORE='*_view.sql'
for i in *.sql; do
echo "$i"
mysql db < "$i"
done
GLOBIGNORE=
banner 'DB - Restoring Views'
for i in *_view.sql; do
echo "$i"
mysql db < "$i"
done
banner 'DB - Clean up'
rm -rf /var/tmp/db
title "$(date)"
}
At the very least, you should exit early if cd, mkdir, or the first rm fail. Also, looping over an unquoted ls command sub is a bad idea. Instead you can use glob expansion (or find).
You could also concatenate the SQL scripts, but this only works if all the commands end with a semicolon (see Run multiple sql files in mysql batch):
big_title 'DB - Restoring Tables'
GLOBIGNORE='*_view.sql'
printf '%s\n' *.sql
cat *.sql | mysql db
GLOBIGNORE=
big_title 'DB - Restoring Views'
printf '%s\n' *_view.sql
cat *_view.sql | mysql db
I have a qcow2 image which I have converted to raw format, and then mount to my file system:
# qemu-img convert -O raw initial-image.qcow2 raw-image.img
# kpartx -av raw-image.img
add map loop0p1 (253:1): 0 41940992 linear /dev/loop0 2048
# mount /dev/mapper/loop0p1 ~/myImage
# ls ~/myImage
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
The size of ~/myImage mountpoint is checked with du command as follows:
# du -hx --max-depth=1 ~/myImage/
0 /root/myImage/dev
0 /root/myImage/proc
0 /root/myImage/run
0 /root/myImage/sys
36M /root/myImage/etc
40K /root/myImage/root
53M /root/myImage/var
12K /root/myImage/tmp
834M /root/myImage/usr
89M /root/myImage/boot
0 /root/myImage/home
0 /root/myImage/media
0 /root/myImage/mnt
0 /root/myImage/opt
0 /root/myImage/srv
1012M /root/myImage/
At this point I copied the contents of ~/myImage and pasted them inside ~/myImage-copy directory. Then I checked the size of the newly created directory. I expected the exact same size for each directory between ~/myImage and ~/myImage-copy, but it's not:
# cp -R myImage/ ~/myImage-copy
# du -hx --max-depth=1 ~/myImage-copy/
0 /root/myImage-copy/dev
0 /root/myImage-copy/proc
0 /root/myImage-copy/run
0 /root/myImage-copy/sys
36M /root/myImage-copy/etc
40K /root/myImage-copy/root
63M /root/myImage-copy/var
12K /root/myImage-copy/tmp
862M /root/myImage-copy/usr
89M /root/myImage-copy/boot
0 /root/myImage-copy/home
0 /root/myImage-copy/media
0 /root/myImage-copy/mnt
0 /root/myImage-copy/opt
0 /root/myImage-copy/srv
1.1G /root/myImage-copy/
From the paths above with different size, I chose /usr and I followed a branch of it until I conclude to an innermost directory which has different size, in order to inspect its files. Below is shown the size difference of an innermost directory:
# du -hx --max-depth=1 ~/myImage/usr/lib64/python2.7/json/
112K /root/myImage/usr/lib64/python2.7/json/
# du -hx --max-depth=1 ~/myImage-copy/usr/lib64/python2.7/json/
164K /root/myImage-copy/usr/lib64/python2.7/json/
However, by inspecting their file content, I can't see any difference related to their size. I don't know if the second column plays a role:
# ls -lah ~/myImage/usr/lib64/python2.7/json/
total 200K
drwxr-xr-x. 2 root root 280 Feb 3 11:17 .
drwxr-xr-x. 26 root root 20K Feb 3 11:17 ..
-rw-r--r--. 1 root root 14K Aug 13 2020 decoder.py
-rw-r--r--. 2 root root 12K Aug 13 2020 decoder.pyc
-rw-r--r--. 2 root root 12K Aug 13 2020 decoder.pyo
-rw-r--r--. 1 root root 17K Aug 13 2020 encoder.py
-rw-r--r--. 2 root root 14K Aug 13 2020 encoder.pyc
-rw-r--r--. 2 root root 14K Aug 13 2020 encoder.pyo
-rw-r--r--. 1 root root 15K Aug 13 2020 __init__.py
-rw-r--r--. 2 root root 14K Aug 13 2020 __init__.pyc
-rw-r--r--. 2 root root 14K Aug 13 2020 __init__.pyo
-rw-r--r--. 1 root root 2.3K Aug 13 2020 scanner.py
-rw-r--r--. 2 root root 2.2K Aug 13 2020 scanner.pyc
-rw-r--r--. 2 root root 2.2K Aug 13 2020 scanner.pyo
-rw-r--r--. 1 root root 997 Aug 13 2020 tool.py
-rw-r--r--. 2 root root 1.3K Aug 13 2020 tool.pyc
-rw-r--r--. 2 root root 1.3K Aug 13 2020 tool.pyo
# ls -lah ~/myImage-copy/usr/lib64/python2.7/json/
total 200K
drwxr-xr-x 2 root root 280 Mar 4 17:55 .
drwxr-xr-x 26 root root 20K Mar 4 17:55 ..
-rw-r--r-- 1 root root 14K Mar 4 17:55 decoder.py
-rw-r--r-- 1 root root 12K Mar 4 17:55 decoder.pyc
-rw-r--r-- 1 root root 12K Mar 4 17:55 decoder.pyo
-rw-r--r-- 1 root root 17K Mar 4 17:55 encoder.py
-rw-r--r-- 1 root root 14K Mar 4 17:55 encoder.pyc
-rw-r--r-- 1 root root 14K Mar 4 17:55 encoder.pyo
-rw-r--r-- 1 root root 15K Mar 4 17:55 __init__.py
-rw-r--r-- 1 root root 14K Mar 4 17:55 __init__.pyc
-rw-r--r-- 1 root root 14K Mar 4 17:55 __init__.pyo
-rw-r--r-- 1 root root 2.3K Mar 4 17:55 scanner.py
-rw-r--r-- 1 root root 2.2K Mar 4 17:55 scanner.pyc
-rw-r--r-- 1 root root 2.2K Mar 4 17:55 scanner.pyo
-rw-r--r-- 1 root root 997 Mar 4 17:55 tool.py
-rw-r--r-- 1 root root 1.3K Mar 4 17:55 tool.pyc
-rw-r--r-- 1 root root 1.3K Mar 4 17:55 tool.pyo
In addition, I checked the md5 hash of the files above as a view of their content. It seems there is no content differences between the same files:
# md5sum ~/myImage/usr/lib64/python2.7/json/*
598c681c82c582ca3f17950b4d5413c1 /root/myImage/usr/lib64/python2.7/json/decoder.py
939055a9fdfbbe6ab4d4babeb3ae63f5 /root/myImage/usr/lib64/python2.7/json/decoder.pyc
939055a9fdfbbe6ab4d4babeb3ae63f5 /root/myImage/usr/lib64/python2.7/json/decoder.pyo
007a9954ca6641b29564a0f0cb55096b /root/myImage/usr/lib64/python2.7/json/encoder.py
366d44efbfce1e5b4a3397b15f202765 /root/myImage/usr/lib64/python2.7/json/encoder.pyc
366d44efbfce1e5b4a3397b15f202765 /root/myImage/usr/lib64/python2.7/json/encoder.pyo
dd5db0c7fb7c531be4e14feebbdb52e8 /root/myImage/usr/lib64/python2.7/json/__init__.py
791c8dca2a86ce50ead04e918a17b508 /root/myImage/usr/lib64/python2.7/json/__init__.pyc
791c8dca2a86ce50ead04e918a17b508 /root/myImage/usr/lib64/python2.7/json/__init__.pyo
8d6660f10863f99ffdd9a95eeddd7b64 /root/myImage/usr/lib64/python2.7/json/scanner.py
59865f486b742aea959693c04cfce057 /root/myImage/usr/lib64/python2.7/json/scanner.pyc
59865f486b742aea959693c04cfce057 /root/myImage/usr/lib64/python2.7/json/scanner.pyo
ad879e2ba247d3d4a5cc71fe22db91d0 /root/myImage/usr/lib64/python2.7/json/tool.py
af6b5531a8bc6977bca14e8d34ba8f68 /root/myImage/usr/lib64/python2.7/json/tool.pyc
af6b5531a8bc6977bca14e8d34ba8f68 /root/myImage/usr/lib64/python2.7/json/tool.pyo
# md5sum ~/myImage-copy/usr/lib64/python2.7/json/*
598c681c82c582ca3f17950b4d5413c1 /root/myImage-copy/usr/lib64/python2.7/json/decoder.py
939055a9fdfbbe6ab4d4babeb3ae63f5 /root/myImage-copy/usr/lib64/python2.7/json/decoder.pyc
939055a9fdfbbe6ab4d4babeb3ae63f5 /root/myImage-copy/usr/lib64/python2.7/json/decoder.pyo
007a9954ca6641b29564a0f0cb55096b /root/myImage-copy/usr/lib64/python2.7/json/encoder.py
366d44efbfce1e5b4a3397b15f202765 /root/myImage-copy/usr/lib64/python2.7/json/encoder.pyc
366d44efbfce1e5b4a3397b15f202765 /root/myImage-copy/usr/lib64/python2.7/json/encoder.pyo
dd5db0c7fb7c531be4e14feebbdb52e8 /root/myImage-copy/usr/lib64/python2.7/json/__init__.py
791c8dca2a86ce50ead04e918a17b508 /root/myImage-copy/usr/lib64/python2.7/json/__init__.pyc
791c8dca2a86ce50ead04e918a17b508 /root/myImage-copy/usr/lib64/python2.7/json/__init__.pyo
8d6660f10863f99ffdd9a95eeddd7b64 /root/myImage-copy/usr/lib64/python2.7/json/scanner.py
59865f486b742aea959693c04cfce057 /root/myImage-copy/usr/lib64/python2.7/json/scanner.pyc
59865f486b742aea959693c04cfce057 /root/myImage-copy/usr/lib64/python2.7/json/scanner.pyo
ad879e2ba247d3d4a5cc71fe22db91d0 /root/myImage-copy/usr/lib64/python2.7/json/tool.py
af6b5531a8bc6977bca14e8d34ba8f68 /root/myImage-copy/usr/lib64/python2.7/json/tool.pyc
af6b5531a8bc6977bca14e8d34ba8f68 /root/myImage-copy/usr/lib64/python2.7/json/tool.pyo
So, what is the reason behind the size difference of the same content in du output?
My operating system is CentOS Linux release 7.9.2009.
Problem
Ubuntu 18.04. The logs for some files not sending. For example I have has 16 log files on 2020-06-23. But only #5 & #8 got collected into data.json. Others are not found in data.json
Here's a script I use to found files on disk but not in data.json.
sudo python -c '
import json; import os;
raw = os.listdir("/path/to/my/logdir")
f = open("/var/lib/filebeat/registry/filebeat/data.json", "r")
data=json.load(f)
harvested=[d["source"].split("/")[-1] for d in data]
substraction=[x for x in raw if not x in harvested]
print("\n".join(substraction))
'
The script result, there's just a lot:
app-2020-06-21.20.log
app-2020-06-21.25.log
app-2020-06-23.11.log
app-2020-06-22.1.log
app-2020-06-22.48.log
app-2020-06-21.41.log
app-2020-06-23.15.log
...
And there are only 2 types of logs End of file reached and Non-zero metrics in the last 30s :
Jun 23 12:23:12 filebeat[32738]: 2020-06-23T12:23:12.223Z DEBUG [harvester] log/log.go:107 End of file reached: /path/to/my/logdir/app-2020-06-21.43.log; Backoff now.
Jun 23 12:23:12 filebeat[32738]: 2020-06-23T12:23:12.344Z DEBUG [harvester] log/log.go:107 End of file reached: /path/to/my/logdir/app-2020-06-22.9.log; Backoff now.
Jun 23 12:23:12 filebeat[32738]: 2020-06-23T12:23:12.364Z DEBUG [harvester] log/log.go:107 End of file reached: /path/to/my/logdir/app-2020-06-21.34.log; Backoff now.
Jun 23 12:23:12 filebeat[32738]: 2020-06-23T12:23:12.444Z DEBUG [harvester] log/log.go:107 End of file reached: /path/to/my/logdir/app-2020-06-23.5.log; Backoff now.
Jun 23 12:23:15 filebeat[32738]: 2020-06-23T12:23:15.144Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":350,"time":{"ms":10}},"total":{"ticks":1710,"ti
Jun 23 12:23:45 filebeat[32738]: 2020-06-23T12:23:45.144Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":350,"time":{"ms":9}},"total":{"ticks":1720,"tim
Jun 23 12:24:15 filebeat[32738]: 2020-06-23T12:24:15.144Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":360,"time":{"ms":6}},"total":{"ticks":1740,"tim
Jun 23 12:24:45 filebeat[32738]: 2020-06-23T12:24:45.144Z
Details
Filebeat config
output:
logstash:
enabled: true
hosts:
- x.x.x.x:5044
filebeat:
inputs:
-
paths:
- "/path/to/log/dir/*"
document_type: myapp
multiline.pattern: '^[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'
multiline.negate: true
multiline.match: after
clean_removed: true
close_removed: true
logging.level: debug
name: "myapp"
tags: ["tag1", "tag2"]
Disk
Here's is the usage. I guess disk is fine
Filesystem Size Used Avail Use% Mounted on
... 2.0G 0 2.0G 0% /dev
... 395M 820K 394M 1% /run
... 30G 12G 18G 41% /
Here's the inode checking result. You can see that there's no duplicated inode in 06-23 logs.
ls -il * | grep 06-23
768289 -rw-r--r-- 1 root root 10485996 Jun 23 00:33 app-2020-06-23.0.log
768372 -rw-r--r-- 1 root root 10486447 Jun 23 01:02 app-2020-06-23.1.log
768292 -rw-r--r-- 1 root root 10485819 Jun 23 05:36 app-2020-06-23.10.log
800654 -rw-r--r-- 1 root root 10499153 Jun 23 05:59 app-2020-06-23.11.log
794052 -rw-r--r-- 1 root root 10486575 Jun 23 06:32 app-2020-06-23.12.log
768487 -rw-r--r-- 1 root root 10492683 Jun 23 06:59 app-2020-06-23.13.log
800633 -rw-r--r-- 1 root root 10490445 Jun 23 07:27 app-2020-06-23.14.log
794067 -rw-r--r-- 1 root root 10500849 Jun 23 07:55 app-2020-06-23.15.log
788191 -rw-r--r-- 1 root root 10489159 Jun 23 08:28 app-2020-06-23.16.log
788410 -rw-r--r-- 1 root root 10486744 Jun 23 09:30 app-2020-06-23.17.log
800624 -rw-r--r-- 1 root root 10486794 Jun 23 10:00 app-2020-06-23.18.log
794048 -rw-r--r-- 1 root root 10490002 Jun 23 10:39 app-2020-06-23.19.log
768461 -rw-r--r-- 1 root root 10486161 Jun 23 01:36 app-2020-06-23.2.log
794051 -rw-r--r-- 1 root root 10488204 Jun 23 11:12 app-2020-06-23.20.log
794081 -rw-r--r-- 1 root root 10487146 Jun 23 11:46 app-2020-06-23.21.log
794071 -rw-r--r-- 1 root root 10492866 Jun 23 12:16 app-2020-06-23.22.log
787673 -rw-r--r-- 1 root root 10490849 Jun 23 12:51 app-2020-06-23.23.log
787698 -rw-r--r-- 1 root root 3491076 Jun 23 13:00 app-2020-06-23.24.log
768478 -rw-r--r-- 1 root root 10486306 Jun 23 02:08 app-2020-06-23.3.log
768507 -rw-r--r-- 1 root root 10486690 Jun 23 02:34 app-2020-06-23.4.log
800620 -rw-r--r-- 1 root root 10496353 Jun 23 03:00 app-2020-06-23.5.log
800623 -rw-r--r-- 1 root root 10503668 Jun 23 03:36 app-2020-06-23.6.log
768521 -rw-r--r-- 1 root root 10520722 Jun 23 04:05 app-2020-06-23.7.log
774652 -rw-r--r-- 1 root root 10487379 Jun 23 04:38 app-2020-06-23.8.log
784704 -rw-r--r-- 1 root root 10553972 Jun 23 05:05 app-2020-06-23.9.log
I intent to enumerate the files and dirs
[root#iz2ze9wve43n2nyuvmsfx5z /]# ls -lh |nl
1 total 60K
2 lrwxrwxrwx 1 root root 7 Jun 15 22:08 bin -> usr/bin
3 dr-xr-xr-x. 5 root root 4.0K Jun 15 22:10 boot
4 drwxr-xr-x 19 root root 2.9K Sep 17 11:35 dev
5 drwxr-xr-x. 85 root root 4.0K Sep 18 22:44 etc
6 drwxr-xr-x. 2 root root 4.0K Sep 17 08:25 home
7 lrwxrwxrwx 1 root root 7 Jun 15 22:08 lib -> usr/lib
8 lrwxrwxrwx 1 root root 9 Jun 15 22:08 lib64 -> usr/lib64
9 drwx------. 2 root root 16K Oct 15 2017 lost+found
10 drwxr-xr-x. 2 root root 4.0K Apr 11 2018 media
11 drwxr-xr-x. 2 root root 4.0K Apr 11 2018 mnt
12 drwxr-xr-x. 2 root root 4.0K Apr 11 2018 opt
13 dr-xr-xr-x 86 root root 0 Sep 17 11:35 proc
14 drwxrwxrwx. 21 root root 4.0K Oct 22 08:11 root
15 drwxr-xr-x 24 root root 660 Oct 24 19:10 run
16 lrwxrwxrwx 1 root root 8 Jun 15 22:08 sbin -> usr/sbin
17 drwxr-xr-x. 2 root root 4.0K Apr 11 2018 srv
18 dr-xr-xr-x 13 root root 0 Oct 21 19:50 sys
19 drwxrwxrwt. 11 root root 4.0K Oct 24 05:19 tmp
20 drwxr-xr-x. 13 root root 4.0K Jun 15 22:08 usr
21 drwxr-xr-x. 19 root root 4.0K Jun 15 22:08 var
It starts with 1 which I desire is 0.
How could make nl to enumerate from a specified line.
You just need to use the -v option, and specify to start from 0; your instruction becoming:
ls -lh |nl -v 0
Lets have a look at nl --help:
$ nl --help
Usage: nl [OPTION]... [FILE]...
Write each FILE to standard output, with line numbers added.
With no FILE, or when FILE is -, read standard input.
Mandatory arguments to long options are mandatory for short options too.
...
-v, --starting-line-number=NUMBER first line number for each section
Therefore:
$ nl -v0 <<EOF
> a
> b
> EOF
0 a
1 b
Note that it does not really matter whether you want to enumerate directories or anything else, which is why you probably should strip this part from your question to make it more specific.
I kept getting
403 Forbidden nginx/1.11.9
I already run : sudo composer update
I think I set up proper permissions.
-rwxrwxrwx 1 root root 149 Feb 24 03:45 .gitignore
-rwxrwxrwx 1 root root 12 Feb 24 03:45 .gitattributes
-rwxrwxrwx 1 root root 146 Feb 24 03:45 CONTRIBUTING.md
drwxrwxrwx 15 root root 4096 Feb 24 03:45 app
-rwxrwxrwx 1 root root 567 Feb 24 03:45 phpunit.xml
drwxrwxrwx 2 root root 4096 Feb 24 03:45 bootstrap
-rwxrwxrwx 1 root root 2452 Feb 24 03:45 artisan
drwxrwxrwx 19 root root 4096 Feb 24 03:45 public
-rwxrwxrwx 1 root root 519 Feb 24 03:45 server.php
-rwxrwxrwx 1 root root 0 Feb 24 03:45 satisfiable
-rwxrwxrwx 1 root root 1599 Feb 24 03:45 readme.md
drwxr-xr-x 5 root root 4096 Feb 24 03:55 ..
-rwxrwxrwx 1 root root 992 Feb 24 14:53 composer.json
-rw-r--r-- 1 root root 116004 Feb 24 14:53 composer.lock
drwxrwxrwx 6 root root 4096 Feb 24 14:53 .
drwxrwxrwx 27 root root 4096 Feb 24 14:53 vendor
What else should I look into ?
ps aux | grep nginx
root 12792 0.0 0.1 148960 1504 ? Ss Feb24 0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
forge 12793 0.0 0.4 148960 4840 ? S Feb24 0:00 nginx: worker process
root 26625 0.0 0.1 12952 1032 pts/0 S+ 15:10 0:00 grep --color=auto nginx
Try the following:
chown -R forge:forge /path/to/web/root
I had it as
root /home/forge/distributor-application/laravel;
I updated it to
root /home/forge/distributor-application/laravel/public;
Final look
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name default;
root /home/forge/distributor-application/laravel/public;
...
}
My site is loading back now.
if you see this error in laravel 5 and you are in windows check your .yaml file.
- map: airbox.aprika
to: /home/vagrant/Code/airbox/public
if lost public end of line then you will see this error.