I need to know how to hex edit from a bash file. For example,
hedit 0x0 A
This would write A to offset 0x0
You should be able to use dd to overwrite parts of a file like this:
printf '\x0a' | dd of=filetopatch bs=1 seek=0 count=1 conv=notrunc
The meaning of the arguments are:
of=filetopatch : the file to patch (the output file)
bs=1 : change 1 byte at a time
seek=0 : move to offset 0 (decimal) before writing
count=1 : copy only 1 input block
conv=notrunc : do not truncate the output after the edit
Related
i need to process large binary files in segments. in concept this would be similar to split, but instead of writing each segment to a file, i need to take that segment and send it as the input of another process. i thought i could use dd to read/write the file in chunks, but the results aren't at all what i expected. for example, if i try :
dd if=some_big_file bs=1M |
while : ; do
dd bs=1M count=1 | processor
done
... the output sizes are actually 131,072 bytes and not 1,048,576.
could anyone tell me why i'm not seeing output blocked to 1M chunks and how i could better accomplish what i'm trying to do ?
thanks.
According to dd's manual:
bs=bytes
[...] if no data-transforming conv option is specified, input is copied to the output as soon as it's read, even if it is smaller than the block size.
So try with dd iflag=fullblock:
fullblock
Accumulate full blocks from input. The read system call may
return early if a full block is not available. When that
happens, continue calling read to fill the remainder of the
block. This flag can be used only with iflag. This flag is
useful with pipes for example as they may return short reads.
In that case, this flag is needed to ensure that a count=
argument is interpreted as a block count rather than a count
of read operations.
First of all, you don't need the first dd. A cat file | while or done < file would do the trick as well.
dd bs=1M count=1 might return less than 1M, see
When is dd suitable for copying data? (or, when are read() and write() partial)
Instead of dd count=… use head with the (non-posix) option -c ….
file=some_big_file
(( m = 1024 ** 2 ))
(( blocks = ($(stat -c %s "$file") + m - 1) / m ))
for ((i=0; i<blocks; ++i)); do
head -c "$m" | processor
done < "$file"
Or posix conform but very inefficient
(( octM = 4 * 1024 * 1024 ))
someCommand | od -v -to1 -An | tr -d \\n | tr ' ' '\\' |
while IFS= read -rN $octM block; do
printf %b "$block" | processor
done
What I am trying to accomplish:
I have a file that I need to copy certain bytes from a certain location
and append them to a file at a given location of that file.
I am thinking something along this lines:
xxd -s $startOffset -l $numBytes inFile | dd of=fileToModify seek=$location conv=notrunc
I have this as well but it will only work for appending at the beginning of a file.
read -p "Enter target file :> " targetFile
read -p "Enter source file to append at the end of target file :> " inputFile
dd if=$inputFile of=$targetFile oflag=append conv=notrunc
Thank you in advance!
contents of first file
$ cat first
fskasfdklsgdfksdjhgf sadjfsdjfhf
dsfghkasdfg sadfhsdfh hskdjfksdfgkfg
jhfksjdafhksdjfh
ksdjhfsdjfh
contents of sceond file
$ cat second
jfhasjdhfjskdhf dshfjsdfh3821349832749832]
87348732642364
]yfisdfhshf936494
sdfisdfsdfsa;dlf
9346934623984
contents of shell script
$ cat cppaste.sh
dd if=$1 of=$2 bs=1 count=$3 status=noxfer
dd if=$4 of=$2 bs=1 seek=$3 status=noxfer
finsize=$(stat -c%s $2)
dd if=$1 of=$2 bs=1 skip=$3 seek=$finsize oflag=append status=noxfer
executing shell script with proper arguments
$ bash cppaste.sh first third 10 second
10+0 records in
10+0 records out
107+0 records in
107+0 records out
92+0 records in
92+0 records out
contents of the resultant file
$ cat third
fskasfdklsjfhasjdhfjskdhf dshfjsdfh3821349832749832]
87348732642364
]yfisdfhshf936494
sdfisdfsdfsa;dlf
9346934623984
gdfksdjhgf sadjfsdjfhf
dsfghkasdfg sadfhsdfh hskdjfksdfgkfg
jhfksjdafhksdjfh
ksdjhfsdjfh
Try this:
# copy certain bytes from a certain location
file=$1
certainlocation=$2
certainbytes=$3
# Append them to a file at a given location of that file
givenlocation=$4
dd if=$file of=$file iflag=skip_bytes oflag=seek_bytes,append conv=notrunc skip=$certainlocation seek=$givenlocation count=1 bs=$certainbytes
Usage:
> printf "1\n2\n3\n4\n" > /tmp/1; ./1.sh /tmp/1 4 2 2; cat /tmp/1;
1+0 records in
1+0 records out
2 bytes copied, 0.000378992 s, 5.3 kB/s
1
2
3
4
3
{
dd if=inFile iflag=count_bytes count="$targetByteLocation" status=none
cat -- "$fileToAppend"
dd if=inFile iflag=skip_bytes skip="$targetByteLocation" status=none
} >outFile
If I have a file on a file system I can do something like this with dd:
dd if=/my/filewithaheader.bin bs=32k skip=1 | gunzip | tar tvf
however if I try something like this:
./commandthatputsstuffonstdout | dd bs=32k skip=1 | gunzip | tar tvf
I get the error:
dd: 'standard input': cannot skip to specified offset.
How can I fix this, can it be done with dd, or is there another unix command I can use
You could use tail. Say:
./commandthatputsstuffonstdout | tail -c +1025 ...
to skip the first 1024 bytes of output produced by your command.
From man tail:
-c, --bytes=K
output the last K bytes; alternatively, use -c +K to output
bytes starting with the Kth of each file
I've just run into this too, and using the fullblock iflag prevents the short read and subsequent abort.
Example:
gzip -d < ./disk_image.dd.gz | \
dd bs=4M skip=32768 iflag=fullblock,skip_bytes of=./partial_image.dd
Bit late answer but this dd example worked for me.
Create example source file:
$ dd if=/tmp/somefile of=/tmp/test skip=50 bs=100 count=1
Skip 50 bytes and copy 10 bytes after it to test_skip file:
$ dd if=/tmp/test of=/tmp/test_skip skip=50 bs=1 count=10
10+0 records in
10+0 records out
10 bytes (10 B) copied, 8.2091e-05 s, 122 kB/s
Or data from stdin:
cat /tmp/test| dd of=/tmp/stdin bs=1 skip=50 count=1
Verify output:
$ hexdump /tmp/test
0000000 ebf3 e8fd df1b 0aa1 faa3 1fba 1817 1267
0000010 1402 f539 fb69 f263 f319 084b 0b26 1150
0000020 182a f98d 030c e0b0 e47c f13d ef3b 1146
0000030 0b7e 0f72 0e58 f2bd f403 ee95 e529 0567
0000040 f88e 1994 0e83 12e5 11e7 fd4b 032f f4f0
0000050 fc9d 010a 0ab6 06b6 1224 f5cb 01e4 e67a
0000060 ebe0 f1a0
0000064
$ hexdump /tmp/test_skip
0000000 0f72 0e58 f2bd f403 ee95
Source file offset 50 is byte: 0x0f
is it possible to substitute bytes from a binary file myfile from one specific position to another with dd in a loop or is it more comfortable using another command?
The idea is to replace a block B at position position2 with block A at position1 in a loop.
PSEUDO CODE
# l = 0
while (l <= bytelength of myfile)
copy myfile (from position1 to A+position1) myfile from (position2 to B+position2)
# position1 = position1+steplength
# position2 = position2+steplength
# l = l+steplength
end
The following lines of code in a text file will do approximately what (I think) you are asking for: copy a file from one location to another, but replace a block of bytes at one location with a block from another location; it uses dd as requested. However, it does create a separate output file - this is necessary to ensure there is no conflict regardless of whether the "input" block occurs before, or after, the "replacement" block. Note that it will do nothing if the distance between A and B is less than the size of the block being replaced - this would result in overlap, and it's not clear if you would want the bytes in the overlapping area to be "the end of A" or "the start of the copy of A".
Save this in a file called blockcpy.sh, and change permissions to include execute (e.g. chmod 755 blockcpy.sh). Run it with
./blockcpy.sh inputFile outputFile from to length
Note that the "from" and "to" offsets have base zero: so if you want to copy bytes starting at the start of the file, the from argument is 0
Here is the file content:
#!/bin/bash
# blockcpy file1 file2 from to length
# copy contents of file1 to file2
# replacing a block of bytes at "to" with block at "from"
# length of replaced block is "length"
blockdif=$(($3 - $4))
absdif=${blockdif#-}
#echo 'block dif: ' $blockdif '; abs dif: ' $absdif
if [ $absdif -ge $5 ]
then
# copy bytes up to "to":
dd if=$1 of=$2 bs=$4 count=1 status=noxfer 2>0
# copy "length" bytes from "from":
dd bs=1 if=$1 skip=$3 count=$5 status=noxfer 2>0 >> $2
# copy the rest of the file:
rest=$((`cat $1 | wc -c` - $4 - $5))
skip=$(($4 + $5))
dd bs=1 if=$1 skip=$skip count=$rest status=noxfer 2>0 >> $2
echo 'file "'$2'" created successfully!'
else
echo 'blocks A and B overlap!'
fi
The 2>0 " 'nix magic" suppresses output from stderr which otherwise shows up in the output (of the type: "16+0 records in").
I have a binary file zero.bin which contains 10 bytes of 0x00, and a data file data.bin which contains 5 bytes of 0x01. I want to substitute the first 5 bytes of the zero.bin with data.bin. I have tried
dd if=data.bin of=zero.bin bs=1 count=5
but, the zero.bin is truncated, finally it becomes 5 bytes of 0x01. I want to keep the tailing 5 bytes of 0x00.
No problem, just add conv=notrunc:
dd if=data.bin of=zero.bin bs=1 count=5 conv=notrunc
You have half of the solution; do that into a temporary file tmp.bin instead of zero.bin, then
dd if=zero.bin bs=1 seek=5 skip=5 of=tmp.bin
mv zero.bin old.bin # paranoia
mv tmp.bin zero.bin
Don't get stuck on using dd(1). There are other tools, eg:
(cat data.bin && tail -c +5 zero.bin) > updated.bin