If I have a file on a file system I can do something like this with dd:
dd if=/my/filewithaheader.bin bs=32k skip=1 | gunzip | tar tvf
however if I try something like this:
./commandthatputsstuffonstdout | dd bs=32k skip=1 | gunzip | tar tvf
I get the error:
dd: 'standard input': cannot skip to specified offset.
How can I fix this, can it be done with dd, or is there another unix command I can use
You could use tail. Say:
./commandthatputsstuffonstdout | tail -c +1025 ...
to skip the first 1024 bytes of output produced by your command.
From man tail:
-c, --bytes=K
output the last K bytes; alternatively, use -c +K to output
bytes starting with the Kth of each file
I've just run into this too, and using the fullblock iflag prevents the short read and subsequent abort.
Example:
gzip -d < ./disk_image.dd.gz | \
dd bs=4M skip=32768 iflag=fullblock,skip_bytes of=./partial_image.dd
Bit late answer but this dd example worked for me.
Create example source file:
$ dd if=/tmp/somefile of=/tmp/test skip=50 bs=100 count=1
Skip 50 bytes and copy 10 bytes after it to test_skip file:
$ dd if=/tmp/test of=/tmp/test_skip skip=50 bs=1 count=10
10+0 records in
10+0 records out
10 bytes (10 B) copied, 8.2091e-05 s, 122 kB/s
Or data from stdin:
cat /tmp/test| dd of=/tmp/stdin bs=1 skip=50 count=1
Verify output:
$ hexdump /tmp/test
0000000 ebf3 e8fd df1b 0aa1 faa3 1fba 1817 1267
0000010 1402 f539 fb69 f263 f319 084b 0b26 1150
0000020 182a f98d 030c e0b0 e47c f13d ef3b 1146
0000030 0b7e 0f72 0e58 f2bd f403 ee95 e529 0567
0000040 f88e 1994 0e83 12e5 11e7 fd4b 032f f4f0
0000050 fc9d 010a 0ab6 06b6 1224 f5cb 01e4 e67a
0000060 ebe0 f1a0
0000064
$ hexdump /tmp/test_skip
0000000 0f72 0e58 f2bd f403 ee95
Source file offset 50 is byte: 0x0f
Related
i want to create a file with only 2 bytes and having as a text "00"
when I run the command echo -n ok > myFile to have just 2 bytes in 'myFile' I get 6 bytes after running the ls -l
In fact it shows me in the byte section that I have 6 bytes but I want to just 2 bytes.
Use printf:
printf '%s' ok >myFile
...or more simply (but less reliably, as this doesn't always work right if your ok is instead a string with %s, literal backslashes, etc):
printf ok >myFile
You can use dd utility
dd if=/dev/zero of=twobytes.txt count=2 bs=1
2+0 records in
2+0 records out
2 bytes copied, 0.000128232 s, 15.6 kB/s
cat twobytes.txt | hexdump
0000000 0000
0000002
ls -l twobytes.txt
-rw-r--r-- 1 lmc users 2 Aug 4 17:39 twobytes.txt
This would also work (thanks #CharlesDuffy)
tr -d '\n' <<<'00' > twobytes.txt
Not recommended as echo behavior might differ in some implementation (see comments below)
echo -ne "\060\060" > twobytes.txt
What I am trying to accomplish:
I have a file that I need to copy certain bytes from a certain location
and append them to a file at a given location of that file.
I am thinking something along this lines:
xxd -s $startOffset -l $numBytes inFile | dd of=fileToModify seek=$location conv=notrunc
I have this as well but it will only work for appending at the beginning of a file.
read -p "Enter target file :> " targetFile
read -p "Enter source file to append at the end of target file :> " inputFile
dd if=$inputFile of=$targetFile oflag=append conv=notrunc
Thank you in advance!
contents of first file
$ cat first
fskasfdklsgdfksdjhgf sadjfsdjfhf
dsfghkasdfg sadfhsdfh hskdjfksdfgkfg
jhfksjdafhksdjfh
ksdjhfsdjfh
contents of sceond file
$ cat second
jfhasjdhfjskdhf dshfjsdfh3821349832749832]
87348732642364
]yfisdfhshf936494
sdfisdfsdfsa;dlf
9346934623984
contents of shell script
$ cat cppaste.sh
dd if=$1 of=$2 bs=1 count=$3 status=noxfer
dd if=$4 of=$2 bs=1 seek=$3 status=noxfer
finsize=$(stat -c%s $2)
dd if=$1 of=$2 bs=1 skip=$3 seek=$finsize oflag=append status=noxfer
executing shell script with proper arguments
$ bash cppaste.sh first third 10 second
10+0 records in
10+0 records out
107+0 records in
107+0 records out
92+0 records in
92+0 records out
contents of the resultant file
$ cat third
fskasfdklsjfhasjdhfjskdhf dshfjsdfh3821349832749832]
87348732642364
]yfisdfhshf936494
sdfisdfsdfsa;dlf
9346934623984
gdfksdjhgf sadjfsdjfhf
dsfghkasdfg sadfhsdfh hskdjfksdfgkfg
jhfksjdafhksdjfh
ksdjhfsdjfh
Try this:
# copy certain bytes from a certain location
file=$1
certainlocation=$2
certainbytes=$3
# Append them to a file at a given location of that file
givenlocation=$4
dd if=$file of=$file iflag=skip_bytes oflag=seek_bytes,append conv=notrunc skip=$certainlocation seek=$givenlocation count=1 bs=$certainbytes
Usage:
> printf "1\n2\n3\n4\n" > /tmp/1; ./1.sh /tmp/1 4 2 2; cat /tmp/1;
1+0 records in
1+0 records out
2 bytes copied, 0.000378992 s, 5.3 kB/s
1
2
3
4
3
{
dd if=inFile iflag=count_bytes count="$targetByteLocation" status=none
cat -- "$fileToAppend"
dd if=inFile iflag=skip_bytes skip="$targetByteLocation" status=none
} >outFile
I have a binary file zero.bin which contains 10 bytes of 0x00, and a data file data.bin which contains 5 bytes of 0x01. I want to substitute the first 5 bytes of the zero.bin with data.bin. I have tried
dd if=data.bin of=zero.bin bs=1 count=5
but, the zero.bin is truncated, finally it becomes 5 bytes of 0x01. I want to keep the tailing 5 bytes of 0x00.
No problem, just add conv=notrunc:
dd if=data.bin of=zero.bin bs=1 count=5 conv=notrunc
You have half of the solution; do that into a temporary file tmp.bin instead of zero.bin, then
dd if=zero.bin bs=1 seek=5 skip=5 of=tmp.bin
mv zero.bin old.bin # paranoia
mv tmp.bin zero.bin
Don't get stuck on using dd(1). There are other tools, eg:
(cat data.bin && tail -c +5 zero.bin) > updated.bin
How can I create a binary file with consequent binary values in Bash?
Like:
hexdump testfile
0000000 0100 0302 0504 0706 0908 0b0a 0d0c 0f0e
0000010 1110 1312 1514 1716 1918 1b1a 1d1c 1f1e
0000020 2120 2322 2524 2726 2928 2b2a 2d2c 2f2e
0000030 ....
In C, I do:
fd = open("testfile", O_RDWR | O_CREAT);
for (i=0; i< CONTENT_SIZE; i++)
{
testBufOut[i] = i;
}
num_bytes_written = write(fd, testBufOut, CONTENT_SIZE);
close (fd);
This is what I wanted:
#! /bin/bash
i=0
while [ $i -lt 256 ]; do
h=$(printf "%.2X\n" $i)
echo "$h"| xxd -r -p
i=$((i-1))
done
There's only one byte you cannot pass as an argument in a Bash command line: 0
For any other value, you can just redirect it. It's safe.
echo -n $'\x01' > binary.dat
echo -n $'\x02' >> binary.dat
...
For the value 0, there's another way to output it to a file
dd if=/dev/zero of=binary.dat bs=1c count=1
To append it to file, use
dd if=/dev/zero oflag=append conv=notrunc of=binary.dat bs=1c count=1
Take a look at xxd:
xxd: creates a hex dump of a given file or standard input. It can
also
convert a hex dump back to its original binary form.
If you don't mind to not use an existing command and want to describe you data in a text file, you can use binmake. That is a C++ program that you can compile and use like following:
First get and compile binmake (the binary will be in bin/):
git clone https://github.com/dadadel/binmake
cd binmake
make
Create your text file file.txt:
big-endian
00010203
04050607
# Separated bytes not concerned by endianness
08 09 0a 0b 0c 0d 0e 0f
Generate your binary file file.bin:
./binmake file.txt file.bin
hexdump file.bin
0000000 0100 0302 0504 0706 0908 0b0a 0d0c 0f0e
0000008
Note: you can also use it with standard input and standard output.
Use the below command,
i=0; while [ $i -lt 256 ]; do echo -en '\x'$(printf "%0x" $i)'' >> binary.dat; i=$((i+1)); done
I would love some help with a Bash script loop that will show all the differences between two binary files, using just
cmp file1 file2
It only shows the first change I would like to use cmp because it gives a offset an a line number of where each change is but if you think there's a better command I'm open to it :) thanks
I think cmp -l file1 file2 might do what you want. From the manpage:
-l --verbose
Output byte numbers and values of all differing bytes.
The output is a table of the offset, the byte value in file1 and the value in file2 for all differing bytes. It looks like this:
4531 66 63
4532 63 65
4533 64 67
4580 72 40
4581 40 55
[...]
So the first difference is at offset 4531, where file1's decimal octal byte value is 66 and file2's is 63.
Method that works for single byte addition/deletion
diff <(od -An -tx1 -w1 -v file1) \
<(od -An -tx1 -w1 -v file2)
Generate a test case with a single removal of byte 64:
for i in `seq 128`; do printf "%02x" "$i"; done | xxd -r -p > file1
for i in `seq 128`; do if [ "$i" -ne 64 ]; then printf "%02x" $i; fi; done | xxd -r -p > file2
Output:
64d63
< 40
If you also want to see the ASCII version of the character:
bdiff() (
f() (
od -An -tx1c -w1 -v "$1" | paste -d '' - -
)
diff <(f "$1") <(f "$2")
)
bdiff file1 file2
Output:
64d63
< 40 #
Tested on Ubuntu 16.04.
I prefer od over xxd because:
it is POSIX, xxd is not (comes with Vim)
has the -An to remove the address column without awk.
Command explanation:
-An removes the address column. This is important otherwise all lines would differ after a byte addition / removal.
-w1 puts one byte per line, so that diff can consume it. It is crucial to have one byte per line, or else every line after a deletion would become out of phase and differ. Unfortunately, this is not POSIX, but present in GNU.
-tx1 is the representation you want, change to any possible value, as long as you keep 1 byte per line.
-v prevents asterisk repetition abbreviation * which might interfere with the diff
paste -d '' - - joins every two lines. We need it because the hex and ASCII go into separate adjacent lines. Taken from: Concatenating every other line with the next
we use parenthesis () to define bdiff instead of {} to limit the scope of the inner function f, see also: How to define a function inside another function in Bash?
See also:
https://superuser.com/questions/125376/how-do-i-compare-binary-files-in-linux
https://unix.stackexchange.com/questions/59849/diff-binary-files-of-different-sizes
The more efficient workaround I've found is to translate binary files to some form of text using od.
Then any flavour of diff works fine.