Use case - Compare all the files and directories on the mounts ( /apps , /logs , etc ) and calculate which one is the latest and size differences.
I am trying with rsync command , but with the limitation I am not achieving exactly what I need.
Under /tmp/test_ram I created two directories as dir1 and dir2. I have created two files under dir1 as shown below .
drwxr-xr-x 2 chada users 4096 Nov 21 12:03 dir2
drwxr-xr-x 2 chada users 4096 Nov 21 12:03 dir1
cd dir1 ; ls -ltr
total 196
-rw-r--r-- 1 chada users 188510 Nov 21 12:03 file_man_rsync
-rw-r--r-- 1 chada users 6854 Nov 21 12:04 file_man_diff
With DryRun –
I see nothing is happening which is expected, but in the o/p size is showing as zero. Which is not I was expecting, I want to see a size in diff of files.
rsync -n -avrczP --out-format="%t %f %''b" --backup --backup-dir=/tmp/test_ram /tmp/test_ram/dir1/ /tmp/test_ram/dir2/
sending incremental file list
2018/11/21 12:04:55 tmp/test_ram/dir1/. 0
2018/11/21 12:04:55 tmp/test_ram/dir1/file_man_diff 0
2018/11/21 12:04:55 tmp/test_ram/dir1/file_man_rsync 0
sent 161 bytes received 25 bytes 372.00 bytes/sec
total size is 195,364 speedup is 1,050.34 (DRY RUN)
the actual run :
I see the file size is showing up, which is what expected. But I cannot take chance in copying without checking. Yes I do have backup dir, but still it need too much of analysis .
rsync -avrczP --out-format="%t %f %''b" --backup --backup-dir=/tmp/test_ram/dir3 /tmp/test_ram/dir1/ /tmp/test_ram/dir2/
2018/11/21 12:05:52 tmp/test_ram/dir1/. 0
file_man_diff
6,854 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=1/3)
2018/11/21 12:05:52 tmp/test_ram/dir1/file_man_diff 2.48K
file_man_rsync
188,510 100% 16.34MB/s 0:00:00 (xfr#2, to-chk=0/3)
2018/11/21 12:05:52 tmp/test_ram/dir1/file_man_rsync 56.28K
sent 58,915 bytes received 57 bytes 117,944.00 bytes/sec
total size is 195,364 speedup is 3.31
This is an example I took to depict . But my comparision would be between multiple servers.
The mount points can be same , but the files are directories is what I needed to compare.
your help is much appreciated.
Related
I have a directory on a RHEL 6 server where logs are being written as below. As you can see there are 4 logs already written within 1 minute. I just want to write a script which can check in every 15 minute (Cron ) & if log files are not updating then send an email alert like " Adapter is in hang status, Restart Required". I know basic linux commands & knowledge of crons. This is how i am trying
-rw-r--r-- 1 root root 11M Oct 6 00:32 Adapter.log.3
-rw-r--r-- 1 root root 11M Oct 6 00:32 Adapter.log.2
-rw-r--r-- 1 root root 10M Oct 6 00:32 Adapter.log.1
-rw-r--r-- 1 root root 6.3M Oct 6 00:32 Adapter.log
$ ll Adapter.log >/tmp/test.txt
$ cat test.txt | awk '{print $6,$7,$8}'
Oct 6 03:10
Now how can i get the time of same log file after 15 minutes, so that i can compare the time difference and write a script to send the alert.
Given description, looks like you timestamp can be checked every 15 minutes.
If file was updated in last 15 minutes, do nothing
If file was updated 15 to 30 minutes ago, send email alert
If file was updated 30 minutes ago, do nothing, as error was already reported on previous cycle
Consider placing the following into cron, on 15 minute interval:
find /path/to/log/Adapter.log* -mmin +15 -mmin -30 | xargs -L1 send-alert
This solution will work on most situations. However, it's worth noting that if the system load is very high, cron execution may be delayed, impacting the age test. In those cases, extra file to store the last test time is needed.
I need to compress a huge folder into multiple zip files, but each of them should not exceed the size of 1 gigabyte.
I found the following command
zip -r -s 1g MyZipFile.zip folderToCompress/
But the resulted files do exceed the size of 1 gb:
-rw-r--r-- 1 user staff 1073741824 14 Feb 15:40 MyZipFile.z01
-rw-r--r-- 1 user staff 1073741824 14 Feb 15:42 MyZipFile.z02
-rw-r--r-- 1 user staff 1073741824 14 Feb 15:41 MyZipFile.z03
-rw-r--r-- 1 user staff 1073741824 14 Feb 15:42 MyZipFile.z04
-rw-r--r-- 1 user staff 1073741824 14 Feb 15:43 MyZipFile.z05
-rw-r--r-- 1 user staff 1073741824 14 Feb 15:43 MyZipFile.z06
-rw-r--r-- 1 user staff 865071084 14 Feb 15:44 MyZipFile.zip
What am I doing wrong?
zip appears to be limiting the size to 1 Gibibyte, not 1 Gigabyte when you specify the -s 1g argument.
1GiB = 2^30 = 1073741824 bytes, which is why you're seeing that number instead of 1GB = 10^9 = 1000000000 bytes. I got the same result trying to zip a multi-GB folder on my own macOS High Sierra desktop. macOS since Snow Leopard handles file sizes according to standard Gigabytes, so it must be zip that is operating using Gibibytes instead. The zip man page states that you can only specify an argument for -s in KiB, MiB, GiB, or TiB.
So, if you want exactly 1GB to be the size limit, you'd have to find some way to specify a size in KiB, MiB, GiB, or TiB which corresponds to 1000000000 bytes. I don't think that's possible; using a MiB to GB converter, the closest I could get to 1GB was 953.67431 MiB, and zip doesn't allow you to specify an -s argument that isn't a whole number.
The /sys/class/gpio can only be accessed as root by default. So I like that a new group gpio can use the files and directories under /sys/class/gpio. To achieve that I added the following lines to /etc/rc.local (I'm on Debian):
sudo chown root:gpio /sys/class/gpio/unexport /sys/class/gpio/export
sudo chmod 220 /sys/class/gpio/unexport /sys/class/gpio/export
So this gives write permissions to all the gpio group members. So they can now export and unexport pins fine.
The problem is they can't read/write the specific pin files after export (e.x. /sys/class/gpio/gpio17) beacause those are owned by root:root again.
How can I change that they are created by default as root:gpio too? I mean I can do that manually each time I export a pin. But that's a bit uncomfy.
UPDATE
According to larsks' answer I created the missing rule file. Now it partially works:
-rwxrwx--- 1 root gpio 4096 Jun 19 16:48 export
lrwxrwxrwx 1 root gpio 0 Jun 19 16:51 gpio17 -> ../../devices/soc/3f200000.gpio/gpio/gpio17
lrwxrwxrwx 1 root gpio 0 Jun 19 16:45 gpiochip0 -> ../../devices/soc/3f200000.gpio/gpio/gpiochip0
-rwxrwx--- 1 root gpio 4096 Jun 19 16:45 unexport
But for the ./gpio17/ I still get root:root:
-rw-r--r-- 1 root root 4096 Jun 19 16:52 active_low
lrwxrwxrwx 1 root root 0 Jun 19 16:52 device -> ../../../3f200000.gpio
-rw-r--r-- 1 root root 4096 Jun 19 16:52 direction
-rw-r--r-- 1 root root 4096 Jun 19 16:52 edge
drwxr-xr-x 2 root root 0 Jun 19 16:52 power
lrwxrwxrwx 1 root root 0 Jun 19 16:52 subsystem -> ../../../../../class/gpio
-rw-r--r-- 1 root root 4096 Jun 19 16:52 uevent
-rw-r--r-- 1 root root 4096 Jun 19 16:52 value
UPDATE 2
Okay I solved the problem. Because I installed Raspbian over the RaspbianInstaller I never went through the raspi-config tool. This seems to be a problem. Because I was also missing the /sys/device/virtual/gpio/ folder.
I followed this guide here: https://community.element14.com/products/raspberry-pi/f/forum/26425/piface-digital-2---setup-and-use#139528
And afterwards the permissions were correct (even for the pin-folders and their files value, direction, ...).
More common rule for 4.x kernels will be the following
SUBSYSTEM=="gpio*", PROGRAM="/bin/sh -c 'find -L /sys/class/gpio/ -maxdepth 2 -exec chown root:gpio {} \; -exec chmod 770 {} \; || true'"
The rule in the initial answer will fail to chown the exported gpio if there's a symbolic link in the path
UPD please beg in mind that when you export some GPIO via sysfs, you should wait for udev rule to fire and complete before you get desired access rights. The thing that worked for me was sleep about 100ms before trying to access GPIO files.
You can do this using udev rules, which can define actions to execute when the kernel instantiates new devices. Current versions of the Raspbian distribution for Raspberry Pi devices contain the following in /etc/udev/rules.d/99-com.rules:
SUBSYSTEM=="gpio*", PROGRAM="/bin/sh -c 'chown -R root:gpio /sys/class/gpio && chmod -R 770 /sys/class/gpio; chown -R root:gpio /sys/devices/virtual/gpio && chmod -R 770 /sys/devices/virtual/gpio'"
This ensures that entries under /sys/class/gpio are always available to members of the gpio group:
# ls -lL /sys/class/gpio/
total 0
-rwxrwx--- 1 root gpio 4096 May 6 23:36 export
drwxrwx--- 2 root gpio 0 Jan 1 1970 gpiochip0
-rwxrwx--- 1 root gpio 4096 May 6 23:37 unexport
# echo 11 > /sys/class/gpio/export
# ls -lL /sys/class/gpio/
total 0
-rwxrwx--- 1 root gpio 4096 May 6 23:37 export
drwxrwx--- 2 root gpio 0 May 6 23:37 gpio11
drwxrwx--- 2 root gpio 0 Jan 1 1970 gpiochip0
-rwxrwx--- 1 root gpio 4096 May 6 23:37 unexport
Update
Permissions are correct for individual pins as well:
# ls -Ll /sys/class/gpio/gpio11/
total 0
-rwxrwx--- 1 root gpio 4096 May 6 23:37 active_low
drwxr-xr-x 3 root root 0 May 6 23:36 device
-rwxrwx--- 1 root gpio 4096 May 6 23:37 direction
-rwxrwx--- 1 root gpio 4096 May 6 23:37 edge
drwxrwx--- 2 root gpio 0 May 6 23:37 subsystem
-rwxrwx--- 1 root gpio 4096 May 6 23:37 uevent
-rwxrwx--- 1 root gpio 4096 May 6 23:37 value
Expanding on the answer by #roman-savrulin, here's a simpler version.
There's no need to run the rule on REMOVE events, only ADD events. There's also no need to run 'find' as the udev environment will supply the exact path of the sysfs directory containing the new GPIO pin's files. You can also use 'chgrp' to change only the owning group, and symbolic modes in 'chmod' to only add the group-write permission bit.
You'll still have to wait for the completion of the rule processing before trying to open the pin's files, but the process should complete more quickly with a simpler rule which only touches the minimum number of files necessary.
SUBSYSTEM=="gpio*", ACTION=="add", PROGRAM="/bin/sh -c 'chgrp -R gpio /sys/${DEVPATH} && chmod -R g+w /sys/${DEVPATH}'"
Check the groups you belong to:
userk#dopamine $: groups
userk sudo dialout
If you belong to dialout the following, if not, comment.
userk#dopamine $: ls -l /dev/gpiomem
crw------- root root /dev/gpiomem
This file mirrors the memory associated with the GPIO device. The output of the command means that the owner of the file is the root user and the group that "owns" it is the root group. The 10 characters represent the file type and the permissions associated with it. The current configuration allows the owner of the file to read and write to the file.
You want to be able to read and write that file if you want to control the gpios.
One option would be to modify the group owner and make it match with the one you belong to (dialout in my case) and set the permissions in order to allow all users of that group to read and write the file.
Long story short:
userk#dopamine $: sudo chown root:dialout /dev/gpiomem
userk#dopamine $: sudo chmod 660 /dev/gpiomem
Wait! This setting won't be persistent and will vanish after reboot.
See this post for further info about the topic
For Ubuntu run.
sudo apt install rpi.gpio-common
*Edit - Stephen has answered this question in the comments below *
so basically I have made two seperate child processes(using two seperate methods with their own fork) to execute the command ls -la | less using pipe.
The first one executes ls like this:
execl("/bin/ls", "ls", "-la", NULL);
The second child process executes less like this:
execlp("less", "less", NULL);
And the results come up fine.. apart from one little part:
Results using shell command:
total 15
drwxr-xr-x 2 daniel staff 4 2015-02-27 18:58 .
drwxr-xr-x 15 daniel staff 24 2015-02-27 18:58 ..
-rwxr-xr-x 1 daniel staff 9280 2015-02-27 18:58 pipes
-rw-r--r-- 1 daniel staff 1419 2015-02-27 18:58 pipes.c
Results using my executable:
total 30
drwxr-xr-x 2 daniel staff 4 Feb 27 18:58 .
drwxr-xr-x 15 daniel staff 24 Feb 27 18:58 ..
-rwxr-xr-x 1 daniel staff 9280 Feb 27 18:58 pipes
-rw-r--r-- 1 daniel staff 1419 Feb 27 18:58 pipes.c
Now the date being a different format I don't care about.. but the total size is twice as large with my executable(30 vs 15). Why is this happening?
Make sure that the ls you are running from the shell and the ls that is running in your program are the same program.
Your program is specifying /bin/ls as the program to run; you can find out what is being run when you type the command at the shell prompt by using the shell command which ls (also see type ls).
If these are different it could be due to the POSIX vs. GNU blocksize used in the total size computation.
This answer indicates that POSIX and GNU tools differ in the blocksize they use to compute the total. Try adding h to your command and compare the results (this flag makes the units explicit). My guess is that the execution environment of these processes is disturbing the result. For instance it may not be the same ls that you are running.
I'm on OS X 10.9.4 and trying to use newsyslog to rotate my app development log files.
More specifically, I want to rotate the files daily but only if they are not empty (newsyslog writes one or two lines to every logfile it rotates, so let's say I only want to rotate logs that are at least 1kb).
I created a file /etc/newsyslog.d/code.conf:
# logfilename [owner:group] mode count size when flags [/pid_file] [sig_num]
/Users/manuel/code/**/log/*.log manuel:staff 644 7 1 $D0 GN
The way I understand the man page for the configuration file is that size and when conditions should work in combination, so logfiles should be rotated every night at midnight only if they are 1kb or larger.
Unfortunately this is not what happens. The log files are rotated every night, no matter if they only the rotation message from newsyslog or anything else:
~/code/myapp/log (master) $ ls
total 32
drwxr-xr-x 6 manuel staff 204B Aug 8 00:17 .
drwxr-xr-x 22 manuel staff 748B Jul 25 14:56 ..
-rw-r--r-- 1 manuel staff 64B Aug 8 00:17 development.log
-rw-r--r-- 1 manuel staff 153B Aug 8 00:17 development.log.0
~/code/myapp/log (master) $ cat development.log
Aug 8 00:17:41 localhost newsyslog[81858]: logfile turned over
~/code/myapp/log (master) $ cat development.log.0
Aug 7 00:45:17 Manuels-MacBook-Pro newsyslog[34434]: logfile turned over due to size>1K
Aug 8 00:17:41 localhost newsyslog[81858]: logfile turned over
Any tips on how to get this working would be appreciated!
What you're looking for (rotate files daily unless they haven't logged anything) isn't possible using newsyslog. The man page you referenced doesn't say anything about size and when being combined other than to say that if when isn't specified, than it is as-if only size was specified. The reality is that the log is rotated when either condition is met. If the utility is like its FreeBSD counterpart, it won't rotate logs less than 512 bytes in size unless the binary flag is set.
MacOS' newer replacement for newsyslog, ASL, also doesn't have the behavior you desire. As far as I know, the only utility which has this is logrotate using its notifempty configuration option. You can install logrotate on your Mac using Homebrew