How can I turn the output of zpool status -v into something usable, with data that match by row in a data.oriented format, instead of the silly "visual" output it uses, so that it's something scriptable, using standard unix-like utilities? I had a python script that did something acceptable, but python 3 completely breaks it, and I'm not fixing it just to have some new version of python break it again. (after screwing around getting the script to run with no errors, it returns nothing :)
bascially this space-bar alinged mess:
pool: data
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0 in 4h52m with 0 errors on Fri Aug 18 04:52:47 2017
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/6dfb7dbe-68c5-11e6-982d-00e04c68f511 ONLINE 0 0 0
gptid/27f40ebe-8f1b-11e4-94f8-3085a9405b85 ONLINE 0 0 0
gptid/9244318f-c1b4-11e6-a31d-0cc47ae2abe8 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gptid/1993f2d7-8f1b-11e4-94f8-3085a9405b85 ONLINE 0 0 0
gptid/529e2c88-f1d1-11e6-89c3-0cc47ae2abe8 ONLINE 0 0 0
gptid/53a09a3e-f1d1-11e6-89c3-0cc47ae2abe8 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
gptid/51f3b377-6a20-11e6-be8c-00e04c68f511 ONLINE 0 0 0
gptid/9fb54bde-1e2d-11e7-a83e-0cc47ae2abe8 ONLINE 0 0 0
gptid/9eebde32-1e2d-11e7-a83e-0cc47ae2abe8 ONLINE 0 0 0
cache
gptid/63db5172-20bd-11e7-b561-0cc47ae2abe8 ONLINE 0 0 0
errors: No known data errors
to something with actual columnns like this:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
data mirror-0 ONLINE 0 0 0
data mirror-0 gptid/6dfb7dbe-68c5-11e6-982d-00e04c68f511 ONLINE 0 0 0
data mirror-0 gptid/27f40ebe-8f1b-11e4-94f8-3085a9405b85 ONLINE 0 0 0
data mirror-0 gptid/9244318f-c1b4-11e6-a31d-0cc47ae2abe8 ONLINE 0 0 0
data mirror-1 ONLINE 0 0 0
data mirror-1 gptid/1993f2d7-8f1b-11e4-94f8-3085a9405b85 ONLINE 0 0 0
data mirror-1 gptid/529e2c88-f1d1-11e6-89c3-0cc47ae2abe8 ONLINE 0 0 0
data mirror-1 gptid/53a09a3e-f1d1-11e6-89c3-0cc47ae2abe8 ONLINE 0 0 0
data mirror-2 ONLINE 0 0 0
data mirror-2 gptid/51f3b377-6a20-11e6-be8c-00e04c68f511 ONLINE 0 0 0
data mirror-2 gptid/9fb54bde-1e2d-11e7-a83e-0cc47ae2abe8 ONLINE 0 0 0
data mirror-2 gptid/9eebde32-1e2d-11e7-a83e-0cc47ae2abe8 ONLINE 0 0 0
data cache
data cache gptid/63db5172-20bd-11e7-b561-0cc47ae2abe8 ONLINE 0 0 0
I can use perl to remove and rearrange, but I can't work out how to match the rows dynamically, in a way that would work with mirror/raidz123/stripe/cache.
datadata ONLINE 0 0 0
data mirror-0 ONLINE 0 0 0
data gptid/6dfb7dbe-68c5-11e6-982d-00e04c68f511 ONLINE 0 0 0
data gptid/27f40ebe-8f1b-11e4-94f8-3085a9405b85 ONLINE 0 0 0
data gptid/9244318f-c1b4-11e6-a31d-0cc47ae2abe8 ONLINE 0 0 0
data mirror-1 ONLINE 0 0 0
data gptid/1993f2d7-8f1b-11e4-94f8-3085a9405b85 ONLINE 0 0 0
data gptid/529e2c88-f1d1-11e6-89c3-0cc47ae2abe8 ONLINE 0 0 0
data gptid/53a09a3e-f1d1-11e6-89c3-0cc47ae2abe8 ONLINE 0 0 0
data mirror-2 ONLINE 0 0 0
data gptid/51f3b377-6a20-11e6-be8c-00e04c68f511 ONLINE 0 0 0
data gptid/9fb54bde-1e2d-11e7-a83e-0cc47ae2abe8 ONLINE 0 0 0
data gptid/9eebde32-1e2d-11e7-a83e-0cc47ae2abe8 ONLINE 0 0 0
datacache
data gptid/63db5172-20bd-11e7-b561-0cc47ae2abe8 ONLINE 0 0 0
This is the code that generates the above.
zpool status -v data | sed '/ data/, $!d' | grep -v errors: > /tmp/diskslistzpoolstatusdata
perl -pi -e 's/^\n$//' /tmp/diskslistzpoolstatusdata #remove blank lines
perl -pi -e 's/\t$//' /tmp/diskslistzpoolstatusdata
perl -p -i -e 's/\t//g' /tmp/diskslistzpoolstatusdata
perl -pi -e 's/^/data/' /tmp/diskslistzpoolstatusdata
extra:
include the scrub summary and error lines per gptid
NAME STATE READ WRITE CKSUM
misc ONLINE 0 0 0
misc mirror-0 ONLINE 0 0 0
misc mirror-0 gptid/aefbaf6e-e004-11e6-8f42-0cc47ae2abe8 ONLINE 0 0 0 0err/4h52m/0err/Fri Aug 18 04:52:47 2017 No known data errors
misc mirror-0 gptid/affc3cac-e004-11e6-8f42-0cc47ae2abe8 ONLINE 0 0 0 0err/4h52m/0err/Fri Aug 18 04:52:47 2017 No known data errors
misc cache gptid/3139819b-20bd-11e7-b561-0cc47ae2abe8 ONLINE 0 0 0 0err/4h52m/0err/Fri Aug 18 04:52:47 2017 No known data errors
Unfortunately there is no integrated solution available. You have two options:
Parse it yourself in a language of your choice. You already extracted the essential information. The layout is relatively static, as vdevs and pools cannot be nested (pools contain vdevs, never pools themselves), the order is respected (no devices from vdev A come after vdev B), the keywords are few and fixed (mirror-N, raidzX-N, etc), and the output is quite small (less than hundreds of lines usually). This means you just have to go through each row, read the info you need, store it in nested objects or simply arrays and go to the next line.
Directly call the appropriate C functions to get the status in non-readable form and convert the output. To do this, have a look at status_callback(zpool_handle_t *zhp, void *data), where all printf-output is generated from the pool data. You could mirror this function to convert the output into a format you like instead of the indented format, and then call your mini-application from your script to give you your data.
If you are familiar with C, option 2 would be faster I think. Performance-wise it does not matter much, as the data is small (even on big systems) and the calls will most likely be very infrequent (as pool layouts do not change often).
Related
I am using ElasticSearch 7.17.
I am trying to create a snapshot of an index:
(I know I shouldn't have a single shard, but for now, that's how it is) :
$ curl -s -k "http://localhost:9200/_cat/indices"
yellow open myIndex vVr6ojDCQTi9ASOUGkkRBA 1 1 679161903 0 140.8gb 140.8gb
I have already registered an S3 bucket for snapshots, which I named backups.
I ran the following command:
$ curl -s -k -X PUT "http://localhost:9200/_snapshot/backups/myIndex?pretty&wait_for_completion=false" -H "content-type:application/json" -d'{"indices": "myIndex"}'
{
"accepted" : true
}
Now, I want to have a look at the progress of that backup's upload :
$ curl -s -k "http://localhost:9200/_cat/snapshots/backups/myIndex"
myIndex IN_PROGRESS 1676385605 14:40:05 0 00:00:00 8.6m 1 0 0 0
$ curl -s -k "http://localhost:9200/_cat/recovery"
myIndex 0 37ms empty_store done n/a n/a 172.24.0.3 7529c7447620 n/a n/a 0 0 0.0% 0 0 0 0.0% 0 0 0 100.0%
It's been in this state, with no change, for the past 1 hour.
I don't understand why 0 bytes are transfered. Am I missing something obvious ?
I don't know what empty_store refers to - shouldn't it be existing_store ?
Other people were right - it just took its time.
The snapshot ended in "SUCCESS" status, but the repository remains in empty_store.
i have the following command to curl a list of urls but I want to rename the output files to something else, as the original name is really long and results in the error below:
command:
jq -r '… | .[]' | xargs -I{} curl -O {}
error:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Failed to create the file
Warning: 270230751_944073806238863_7984852070388291566_n.jpg?stp=dst-jpg_e35&_n
Warning: c_ht=example.njrsnnkjnrjgngkrnngrggk&_nc_cat=111&_nc_ohc=Ch26T4U5kDIAX
Warning: 8viWsL&edm=AABBvjUBAAAA&ccb=7-4&ig_cache_key=MjczOTA2MDk4MDU3MTIyNjQ1N
0 92492 0 1 0 0 1 0 25:41:32 --:--:-- 25:41:32 1
curl: (23) Failed writing body (0 != 1)
how to pass some argument to curl so that it renames the file to a smaller name eg 16 digit uuid or something like njfnjsnf48u8 but not too common as there can be upto 1000 files being downloaded at once.
You can manipulate the {} variable that xargs uses in curl command to make it more unique. If you are sure that there are no duplicate urls in the input file then the following might work.
curl {} -o $(echo {}|base64)
In my current system, I have 3 SSDs, sda, sdb and sdc. The OS is installed in sdc.
I am trying to extract the SSDs without the OS installed in it. So, this command
echo $(eval $(lsblk -oMOUNTPOINT,PKNAME -P | grep 'MOUNTPOINT="/"'); echo $PKNAME | sed 's/[0-9]*$//')
returns sdc.
But if I want the drive without OS, how should I modify the above command?
grep 'MOUNTPOINT!="/"' doesn't return anything.
The bash script provided by Renaud works as expected on systems with raw drives. In case, if the system has LVM partitions then it returns only dm-1.
How to handle this case? To get the correct SSD name on either raw or LVM systems?
On the LVM system, lsblk returns. The expected output is sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1.8T 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 1.8T 0 part
├─ubuntu--vg-ubuntu--lv-real
│ 253:1 0 880G 0 lvm
│ ├─ubuntu--vg-ubuntu--lv 253:2 0 880G 0 lvm /
│ └─ubuntu--vg-clean 253:4 0 880G 0 lvm
└─ubuntu--vg-clean-cow 253:3 0 400G 0 lvm
└─ubuntu--vg-clean 253:4 0 880G 0 lvm
sdb 8:16 0 1.8T 0 disk
On the raw drive, lsblk returns
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 223.6G 0 disk
sdb 8:16 0 465.8G 0 disk
├─sdb1 8:17 0 2G 0 part
└─sdb2 8:18 0 463.8G 0 part
sdc 8:32 0 232.9G 0 disk
├─sdc1 8:33 0 156M 0 part /boot/efi
├─sdc2 8:34 0 26.7G 0 part /boot
├─sdc3 8:35 0 182.7G 0 part /
└─sdc4 8:36 0 23.4G 0 part [SWAP]
Here the expected output is sda or sdc.
grep has a -v option to print only non-matching lines. So:
lsblk -oMOUNTPOINT,PKNAME -P | grep -v 'MOUNTPOINT="/"'
should exclude this drive. But note that if you have more than one other drive the rest of your script will not work as you would like. Only the last drive will be considered because it overrides the others. A loop would probably be closer to your needs:
while IFS= read -r line; do
eval "$line"
sed 's/[0-9]*$//' <<< "$PKNAME"
done < <(lsblk -oMOUNTPOINT,PKNAME -P | grep -v 'MOUNTPOINT="/"')
I have some inputs values (32-bit unsigned integer) defined as follow, with their associated identifier, in a file :
var=0x000000001
var1=0x000000002
var2=0x000000004
var3=0x000000008
toto=0x000000010
titi=0x000000020
tata=0x000000040
toto1=0x000000080
toto2=0x000000100
toto3=0x000000200
titi2=0x000000400
titi3=0x000000800
tito0=0x000001000
tito1=0x000002000
tito2=0x000004000
tito3=0x000008000
tito4=0x000010000
I would like to implement a second function name GetConfig() which must return an uint32 value. Based on this value, I would to quote the associated variable. If I enter 300 I should display toto2 toto3.
function Config()
{
vartest=$1
if [ -f $file ]; then
while read lines
do
value=${lines##*=}
mask=$(($vartest & $value))
echo $mask
done < $file
else
exitError 101
fi
}
If I enter ./script Config 3840 I obtain :
0
0
0
0
0
0
0
0
256
512
1024
2048
0
0
0
0
0
I can easily display toto2 toto3 titi5 titii, but if I enter ./script Config 4812, I obtain :
0
0
4
8
0
0
64
128
0
512
0
0
4096
0
0
0
0
This result can not allows me to display var1 titi titii test1
I am not sure to be clear but I am really currently blocked on this mask issue.
Thank in advance
Your problem statement isn't very clear but I hope this snippet can help put you back on track. The real meat is in using ${lines%=*} to obtain the key corresponding to the value in this lines.
Config() {
vartest=$1
if [ -f $file ]; then
while read lines
do
value=${lines##*=}
test $(($vartest & $value)) = 0 && continue
echo "${lines%=*}"
done < $file
else
exitError 101
fi
}
file=/dev/stdin
Config 3840 <<':'
var=0x000000001
var1=0x000000002
var2=0x000000004
var3=0x000000008
toto=0x000000010
titi=0x000000020
tata=0x000000040
toto1=0x000000080
toto2=0x000000100
toto3=0x000000200
titi2=0x000000400
titi3=0x000000800
tito0=0x000001000
tito1=0x000002000
tito2=0x000004000
tito3=0x000008000
tito4=0x000010000
:
Test run:
toto2
toto3
titi2
titi3
I would expect a more-useful function to not require the values in a file; this incidentally also illustrates how to put them in a here document (I did some weird stuff to not have to change those parts of your function).
Here's an example ip proxy checker shell script:
#!/bin/sh
while read IP
do
CURL=$(curl -x http://$IP -L http://icanhazip.com)
echo "$CURL"
done < ip.txt
But instead of a simple result like:
0.0.0.0
1.1.1.1
I get:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
0.0.0.0
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
1.1.1.1
How can I quiet the extra stuff?
-s/--silent
Silent mode. Don't show progress meter or error messages. Makes Curl mute.
If this option is used twice, the second will again disable mute.
CURL=$(curl -s -x http://$IP -L http://icanhazip.com)