I managed to get Ubuntu running on a mobile device. I need to automate some processes on it because user input is totally impossible without a convoluted setup and soldering wires.
I need to run "parted print" and then pipe "yes, fix, fix" to stdin here is the desired output:
~ # parted /dev/block/mmcblk0 print
parted /dev/block/mmcblk0 print
Warning: /dev/block/mmcblk0 contains GPT signatures, indicating that it has a
GPT table. However, it does not have a valid fake msdos partition table, as it
should. Perhaps it was corrupted -- possibly by a program that doesn't
understand GPT partition tables. Or perhaps you deleted the GPT table, and are
now using an msdos partition table. Is this a GPT partition table?
Yes/No? yes
yes
yes
Error: The backup GPT table is not at the end of the disk, as it should be.
This might mean that another operating system believes the disk is smaller.
Fix, by moving the backup to the end (and removing the old backup)?
Fix/Ignore/Cancel? fix
fix
fix
Warning: Not all of the space available to /dev/block/mmcblk0 appears to be
used, you can fix the GPT to use all of the space (an extra 569312 blocks) or
continue with the current setting?
Fix/Ignore? fix
fix
fix
Model: MMC SEM16G (sd/mmc)
Disk /dev/block/mmcblk0: 15.9GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
1 131kB 262kB 131kB xloader
2 262kB 524kB 262kB bootloader
3 524kB 16.3MB 15.7MB recovery
4 16.8MB 33.6MB 16.8MB boot
5 33.6MB 83.9MB 50.3MB rom
6 83.9MB 134MB 50.3MB bootdata
7 134MB 522MB 388MB factory
8 522MB 1164MB 642MB system
9 1164MB 1611MB 447MB cache
10 1611MB 2684MB 1074MB media
11 2684MB 15.6GB 12.9GB userdata
Here is what I've drafted..
#! /bin/bash
mkfifo Input
mkfifo Output
#Redirect commandline input from fifo to parted, Redirect output to fifo, background
cat Input &| - parted print >Output &
Line=""
while [ 1 ]
do
while read Line
do
if [ $Line == *Yes\/No\?* ]; then
echo "yes">Input
fi
if [ $Line == *Fix\/Ignore/\Cancel\?* ]; then
echo "fix">Input
fi
if [ $Line == *Fix\/Ignore\?* ]; then
echo "fix">Input
fi
test $Line == *userdata* && break
done<Output
test $Line == *userdata* && break
done
But this does not work. If someone could assist me in redirecting output from a program into a fifo, then analyzing that data and directing output into another fifo to be put back in the original program? The desired results are in the first code block.
If you always know what the needed inputs will be -- if they never change from run to run -- you can just redirect input from a file or from a HERE document and you don't need to do anything complicated.
If the needed inputs will change from run to run, you need to use something other than the shell because it will not make what you are trying to do possible. perl might be a good choice. (You don't need to use expect here because you're not trying to simulate a tty.)
Related
I use the following script to retrieve information about mounted file-systems on several hundred Solaris (v9,10,11) and Red Hat Enterprise Linux (v5,6,7) servers for analysis.
# retrieves for all mounted file-systems: server, device, allocated, used, available, percent_used, mount_directory, permissions, owner_name, and group_name
server=$(uname -n)
df -h | awk '
NF == 6 { print ($0); }
NF == 1 { device = $1; }
NF == 5 { print (device, " ", $0); }
' | while read device allocated used available percent mount
do
ls -ld "${mount}" | read permissions links owner_name group_name size month day time directory
echo "${server} ${device} ${allocated} ${used} ${available} ${percent} ${mount} ${permissions} ${owner_name} ${group_name}"
done
I perform this operation from Windoze using PuTTY "plink" utility.
plink -m filesys.script server_name >>filesys.txt
All worked as expected until my default shell was changed from ksh to bash on all servers. Now, the second read command that obtains ls output for permissions, owner_name, and group_name is not functioning and does not produce any error messages either. Therefore the result is that only seven tokens are in output (server through mount) and there is nothing for permissions, owner_name, or group_name.
I have confirmed that if I upload the script to the Unix server with a shebang (#!/bin/ksh) at the top line the script works as expected. However, I do not want to push this script to hundreds of servers and maintain the script in a distributed mechanism. I would like to retain the script on central Windoze workstation and call with -m parameter of plink. Placing a shabang at top of the file does not execute ksh using plink -m option.
The Bash shell versions that are in play are 3.2 and 4.1. I have also made certain that the Windoze script file has carriage returns removed. The awk utility is used to handle situations where the device name is too long and df breaks the output over two lines.
Again, the first read (from df/awk) is working fine but the second (ls output) is not. I confirmed by placing a 'set' following the second read and those environment varriables were not in the environment.
The read (as a pipe element) happens in a subshell, so even though it actually does execute perfectly, once that pipeline exits its results aren't available to the echo running on a separate line (as part of the parent process that originally spawned the pipeline). This is fully allowed by POSIX; which component of a pipeline, if any, is performed by the shell spawning that pipeline is unspecified by the standard and thus implementation-defined.
You can address the issue by putting the echo inside of the same pipeline element as the read:
server=$(uname -n)
df -h | awk '
NF == 6 { print ($0); }
NF == 1 { device = $1; }
NF == 5 { print (device, " ", $0); }
' | while read device allocated used available percent mount
do
# NOTE: parsing output from "ls" is unreliable
ls -ld "${mount}" | {
read permissions links owner_name group_name size month day time directory
echo "${server} ${device} ${allocated} ${used} ${available} ${percent} ${mount} ${permissions} ${owner_name} ${group_name}"
}
done
References:
BashFAQ #24 (I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?)
ParsingLs (Why you shouldn't parse the output of ls(1))
If you have GNU stat or find, either of which allows you to provide a format string to control metadata output, I would strongly suggest using them in place of ls -l for parsing metadata. Even perl is somewhat better for the purpose, having only a single universally available implementation with uniform stat behavior between releases.
I'm trying to find the best and most efficient way to resume reading a file from a given point.
The given file is being written frequently (this is a log file).
This file is rotated on a daily basis.
In the log file I'm looking for a pattern 'slow transaction'. End of such lines have a number into parentheses. I want to have the sum of the numbers.
Example of log line:
Jun 24 2015 10:00:00 slow transaction (5)
Jun 24 2015 10:00:06 slow transaction (1)
This is easy part that I could do with awk command to get total of 6 with above example.
Now my challenge is that I want to get the values from this file on a regular basis. I've an external system that polls a custom OID using SNMP. When hitting this OID the Linux host runs a couple of basic commands.
I want this SNMP polling event to get the number of events since the last polling only. I don't want to have the total every time, just the total of the newly added lines.
Just to mention that only bash can be used, or basic commands such as awk sed tail etc. No perl or advanced programming language.
I hope my description will be clear enough. Apologizes if this is duplicate. I did some researches before posting but did not find something that precisely correspond to my need.
Thank you for any assistance
In addition to the methods in the comment link, you can also simply use dd and stat to read the logfile size, save it and sleep 300 then check the logfile size again. If the filesize has changed, then skip over the old information with dd and read the new information only.
Note: you can add a test to handle the case where the logfile is deleted and then restarted with 0 size (e.g. if $((newsize < size)) then read all.
Here is a short example with 5 minute intervals:
#!/bin/bash
lfn=${1:-/path/to/logfile}
size=$(stat -c "%s" "$lfn") ## save original log size
while :; do
newsize=$(stat -c "%s" "$lfn") ## get new log size
if ((size != newsize)); then ## if change, use new info
## use dd to skip over existing text to new text
newtext=$(dd if="$lfn" bs="$size" skip=1 2>/dev/null)
## process newtext however you need
printf "\nnewtext:\n\n%s\n" "$newtext"
size=$((newsize)); ## update size to newsize
fi
sleep 300
done
I'm trying to write a bash script to audit hard drives that have been wiped to ensure the wiping system is working properly. I would like to find a way to hex dump specific parts of a drive without having to hex dump the entire drive and extract the parts I'd like (as this seems to run for too long to make the script worth writing). Ideally, I'd be able to grab parts from the beginning, middle, and end of the drive.
I would like to take the output of the hex dump and check it for the existence of only one character (indicating the drive has been successfully wiped). This part, I can handle, but I thought it may affect any advice I may get.
I've used head piped into xxd to get the beginning of the file which has worked, but I'm still stuck on the other parts. I've tried using tail to just get the end of the drive, but that doesn't seem to work quickly either. Is it possible to do this efficiently? Possibly using dd or something else and pipe it into a hex editor? I've looked through options for xxd as well as hexdump to no avail. If someone could point me in the right direction, it would be greatly appreciated!
xxd has options to skip a ways into the file (-s) and dump a limited length (-l). If you use its plain hex (-p) option, you may be able to use grep to find any anomalies:
$ xxd -s 8192 -l 256 -p /dev/disk3s2 | grep [^0]
000000010000000000000000000000000000000000000000000000000000
000000000000000000000000300000000000000800000000000000000000
dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb
dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb
dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb
od has similar skip (-j) and limit length (-N). Similarly, dd has skip= and count= (although these are counted in blocks, not bytes; you can change the block size with bs=).
EDIT: Since xxd -p is giving weird results (not stopping at what should be the end of the device), I'd recommend running some tests to figure out what's going on. First, back up anything important on the computer, because if something is weird at the device access level, it's possible that some of these tests might overwrite something unexpected, possibly even on another disk.
Next, try dumping to the end of the device with different tools, and see if they all behave the same way:
xxd -s 65451982336 /dev/sdb | more # This *should* dump 512 bytes (32 lines) then stop, but apparently keeps going
od -xv -j 65451982336 /dev/sdb | more # This also *should* dump 512 bytes then stop
dd if=/dev/sdb skip=127835903 | xxd | more # This again should do the same thing (note that the skip value is in 512-byte blocks)
Do the other tools read past what fdisk reports as the end of the disk? If all three read more data, I'm going with the "fdisk is wrong/misleading" answer. You can test further by writing some nonzero data past the "end" and seeing what the results are:
dd if=/dev/random of=/dev/sdb seek=127835903 count=2
...then repeat the various dump commands. If they show two blocks (=64 lines) of random data followed by zeroes, I'm pretty sure the device is bigger than you think it is.
I am not near my shell, but something along these lines should get you started:
dd if=/dev/hda1 | hexdump -C | grep [^00]
will print all non-zero bytes.
dd if=/dev/hda1 | od -x -j100
will give you a hexadecimal dump with offsets, starting 100 bytes in.
For QA purposes I need to be able to partition a drive via a bash script up to 30 or more partitions for both RHEL and SLES.
I have attempted to do this in BASH with fdisk via a "here document" which works but as you can guess blows up in various steps. I assume this is because of timing of the input commands occurring at the wrong times and getting out of sync. 1 out of 10 times my script will work correctly.
I have looked at parted and sfdisk and don't really understand how to use these tools.
I have only ever used fdisk.
My issue is that with fdisk you can state something like "new partition +1gb" over and over and this works for me because in my script I don't need to keep track of prior partitions or remaining space or do any calculations. Every time I run this function this just makes an additional 1gb partition from any unused space.
Is there a way to use parted or sfdisk (or any other tool that would already be a part of these distros) so that I could script a loop from 1 to x operations without having to do this accounting of remaining space? Does anyone have any examples they could share?
Update
Here is an example of one of my functions. At the start of the script we ask the user for the number of partitions to create, their size (this is static for all), and type of FS if any. This functions creates partition 1 thru 3 then a different function handles the extended (4th) and another handles 5th to nn.
As I said before, this script is fully functional; my problem is that at times the commands being sent to fdisk seem to arrive at the wrong timing, which then breaks the entire script thereby breaking any automation.
So the commands are being sent like this:
n
p
1
+1M
w
I have been reading up on fdisk and have learned it is not suited well for scripting so what I am seeing is that when in script mode, fdisk might be asking for p my script already thinks it's time to send the 1.
The thing about fdisk that worked for me is that after you specify the partition number it already calculated the next free sector so all I have to do at this point is send a blank line for my start and then the +1M for my total size. Parted and sfdisk don't appear to work this way from what I can tell and I am still very new at this to understand how to automate those tools at this time.
Create1to3Primary_Func() {
Size=\+$partSize\MB
for i in {1..3}
do
echo " this loop i= $i"
echo "Creating Partition $i on $targetFull as $targetFull$i using Create1to3Primary_Func()"
rm -f /tmp/myScript
echo -e "n" >> /tmp/myScript
echo -e "p" >> /tmp/myScript
echo -e "$i" >> /tmp/myScript
echo -e " " >> /tmp/myScript
echo -e "$Size" >> /tmp/myScript
echo -e "w" >> /tmp/myScript
echo -e "EOF" >> /tmp/myScript
fdisk $targetFull < /tmp/myScript
echo " sleeping Create1to3Primary_Func()"
sleep 4s
if [ "$RawOrFs" == "f" ]; then
mkfsCMD="mkfs.$fsType"
mkfsFullTarget="$targetFull$i"
cmdline="$mkfsCMD $mkfsFullTarget -L 'Partition$i'"
echo "Creating $fsType File System on $mkfsFullTarget"
$cmdline
fi
void="/mnt/mymnt$i"
if [ ! -d $void ] ; then
echo "Creating Mount Point /mnt/mymnt$i"
void="/mnt/mymnt$i"
mkdir $void
fi
echo "Part Probe on $targetFull "
partprobe $targetFull ; sleep 4s
done
}
Not sure to get what you really want, but you may be interested by the fact that sfdisk can dump a partition layout and use this layout to partition other disks. For instance:
sfdisk -d /dev/sda > mydiskpartitionslayout
Then in your script (take care of course) you can specify
sfdisk /dev/sdx < mydiskpartitionslayout
sfdisk
sfdisk is a Scripted version of fdisk
It is part of util-linux, just like fdisk, so availability should be the same.
A partition table with a single partition that takes the whole disk can be
created with:
echo 'type=83' | sudo sfdisk /dev/sdX
and more complex partition tables are explained below.
To generate an example script, get the setup of one of your disks:
sudo sfdisk -d /dev/sda > sda.sfdisk
Sample output on my Lenovo T430 Windows 7 / Ubuntu dual boot:
label: dos
label-id: 0x7ddcbf7d
device: /dev/sda
unit: sectors
/dev/sda1 : start= 2048, size= 3072000, type=7, bootable
/dev/sda2 : start= 3074048, size= 195430105, type=7
/dev/sda3 : start= 948099072, size= 28672000, type=7
/dev/sda4 : start= 198504446, size= 749594626, type=5
/dev/sda5 : start= 198504448, size= 618891264, type=83
/dev/sda6 : start= 940277760, size= 7821312, type=82
/dev/sda7 : start= 817397760, size= 61437952, type=83
/dev/sda8 : start= 878837760, size= 61437500, type=83
Once you have the script saved to a file, you can apply it to sdX with:
sudo sfdisk /dev/sdX < sda.sfdisk
For sfdisk input, you can just omit the device names, and use lines of type:
start= 2048, size= 3072000, type=7, bootable
They are just ignored if present, and the device name is taken from the command line argument.
Some explanations:
header lines: all optional:
label: type of partition table. dos (MBR) is the old an widely supported one, gpt the new shiny thing.
unit: only sector is supported. 1 sector usually equals 512 bytes. Find with cat /sys/block/sda/queue/hw_sector_size See also: https://unix.stackexchange.com/questions/2668/finding-the-sector-size-of-a-partition
device: informative only I think
partition lines:
start: offset inside the disk at which the partition starts.
start has very good defaults, and can often be ommited:
on the first line, start is 2048, i.e. 1Mb (2048 + 512), which is a sane default for disk compatibility
further start default to the first unallocated position
size: man sfdisk says: The default value of size indicates "as much as possible". So to fill the disk with a single partition use: /dev/sda : start=2048, type=83
type: magic byte stored on the boot sector for each partition entry. Possible values: https://en.wikipedia.org/wiki/Partition_type On this example we observe:
7 (sda1, 2 and 3): filesystems that Windows supports. Preinstalled Windows stuff and Lenovo recovery partitions. sudo blkid labels help identify them.
5 (sda4): extended primary partition, which will contain other logical partitions (because we can only have 4 primary partitions with MBR)
83(sda5, 7, and 8): partitions which Linux supports. For me one home, and two roots with different Ubuntu versions
82 (sd6): swap
fdisk can also read sfdisk scripts with the I command, which "sources" them during an interactive fdisk session, allowing you further customization before writing the partition.
Tested on Ubuntu 16.04, sfdisk 2.27.1.
Format and populate the partitions an image file without sudo
This is a good way to learn to use sfdisk without blowing up your hard disks: How to create a multi partition SD disk image without root privileges?
An approach I like (which I saw in this article) is to "script" the fdisk input directly, since it's smarter than sfdisk about creating a partition "until the end of the disk" or "2 GB large". Example:
echo "d
1
d
2
d
3
n
p
1
+2G
n
p
2
w
" | fdisk /dev/sda
This script deletes up to 3 existing partitions, creates a 2 GB partition (e.g. swap) and then creates a partition that would extend over the remaining disk space.
In contrast, if a partition layout was created and used in sfdisk, the script would not cover the whole disk if more space was available.
Automating repetitive task is a norm in the life of automation and we need a method to automatically provide answers to these programs if we are to include them in our script.
This is where a program called “Expect” steps in to automate. For Red Hat based systems, execute the below command to install "Expect"
yum install expect
For Debian based or Ubuntu, execute the below command.
apt-get install expect
below is the expect script to create a partition /dev/sdc
!/usr/bin/expect
log_file -a "/tmp/expect.log"
set timeout 600
spawn /sbin/fdisk /dev/sdc
expect "Command (m for help): " { send "n\r" }
expect "p primary partition (1-4)"
expect "" { send "p\r" }
expect "Partition number (1-4): " { send "1\r" }
expect "First cylinder (1-133544, default 1): " { send "1\r" }
expect ": " { send "\r" }
expect "Command (m for help): " { send "w\r" }
interact
I need to examine the output of a certain script 1000s of times on a unix platform and check if any of it has changed from before.
I've been doing this:
(script_stuff) | md5sum
and storing this value. I actually don't really need "md5", JUST a simple hash function which I can compare against a stored value to see if its changed. Its okay if there are an occassional false positive.
Is there anything better than md5sum that works faster and generates a fairly usable hash value? The script itself generates a few lines of text - maybe 10-20 on average to max 100 or so.
I had a look at fast md5sum on millions of strings in bash/ubuntu - that's wonderful, but I can't compile a new program. Need a system utility... :(
Additional "background" details:
I've been asked to monitor the DNS record of a set of 1000 or so domains and immediately call certain other scripts if there has been any change. I intend to do a dig xyz +short statement and hash its output and store that, and then check it against a previously stored value. Any change will trigger the other script, otherwise it just goes on. Right now, we're planning on using cron for a set of these 1000, but can think completely diffeerently for "seriously heavy" usage - ~20,000 or so.
I have no idea what the use of such a system would be, I'm just doing this as a job for someone else...
The cksum utility calculates a non-cryptographic CRC checksum.
How big is the output you're checking? A hundred lines max. I'd just save the entire original file then use cmp to see if it's changed. Given that a hash calculation will have to read every byte anyway, the only way you'll get an advantage from a checksum type calculation is if the cost of doing it is less than reading two files of that size.
And cmp won't give you any false positives or negatives :-)
pax> echo hello >qq1.txt
pax> echo goodbye >qq2.txt
pax> cp qq1.txt qq3.txt
pax> cmp qq1.txt qq2.txt >/dev/null
pax> echo $?
1
pax> cmp qq1.txt qq3.txt >/dev/null
pax> echo $?
0
Based on your question update:
I've been asked to monitor the DNS record of a set of 1000 or so domains and immediately call certain other scripts if there has been any change. I intend to do a dig xyz +short statement and hash its output and store that, and then check it against a previously stored value. Any change will trigger the other script, otherwise it just goes on. Right now, we're planning on using cron for a set of these 1000, but can think completely diffeerently for "seriously heavy" usage - ~20,000 or so.
I'm not sure you need to worry too much about the file I/O. The following script executed dig microsoft.com +short 5000 times first with file I/O then with output to /dev/null (by changing the comments).
#!/bin/bash
rm -rf qqtemp
mkdir qqtemp
((i = 0))
while [[ $i -ne 5000 ]] ; do
#dig microsoft.com +short >qqtemp/microsoft.com.$i
dig microsoft.com +short >/dev/null
((i = i + 1))
done
The elapsed times at 5 runs each are:
File I/O | /dev/null
----------+-----------
3:09 | 1:52
2:54 | 2:33
2:43 | 3:04
2:49 | 2:38
2:33 | 3:08
After removing the outliers and averaging, the results are 2:49 for the file I/O and 2:45 for the /dev/null. The time difference is four seconds for 5000 iterations, only 1/1250th of a second per item.
However, since an iteration over the 5000 takes up to three minutes, that's how long it will take maximum to detect a problem (a minute and a half on average). If that's not acceptable, you need to move away from bash to another tool.
Given that a single dig only takes about 0.012 seconds, you should theoretically do 5000 in sixty seconds assuming your checking tool takes no time at all. You may be better off doing something like this in Perl and using an associative array to store the output from dig.
Perl's semi-compiled nature means that it will probably run substantially faster than a bash script and Perl's fancy stuff will make the job a lot easier. However, you're unlikely to get that 60-second time much lower just because that's how long it takes to run the dig commands.