I am working on a little endian system. Suppose I have the following data that I write to a sector in a disk:
$ hexdump file
04eb 0008 ...
$ cat file > /dev/sdb
Now when I read it back to memory, how will it look like in the memory and why?
Related
This question already has answers here:
Size() vs ls -la vs du -h which one is correct size?
(3 answers)
Closed last month.
I had to create a random file of 10GB size, which I can using dd or fallocate, but the size shown by du -sh is twice the one I created:
$ dd bs=1MB count=10000 if=/dev/zero of=foo
10000+0 records in
10000+0 records out
10000000000 bytes (10 GB, 9.3 GiB) copied, 4.78419 s, 2.1 GB/s
$ du -sh foo
19G foo
$ ls -sh foo
19G foo
$ fallocate -l 10G bar
$ du -sh bar
20G bar
$ ls -sh bar
20G bar
Can someone please explain me this apparent discrepancy?
On wikipedia, it mentions about GPFS ...
The system stores data on standard block storage volumes, but includes an internal RAID layer that can virtualize those volumes for redundancy and parallel access much like a RAID block storage system.
I conclude that there is at least one non-visible duplicate for every file, and therefore each file actually uses twice the amount of space than the actual content of a single file. So the underlying RAID imposes the double-usage.
That could explain it, because I have created a similar massive file for other purposes, also using dd, on an ext4 filesystem, but the OS reports my file size matching the dd creation size, as per design intent (no RAID in effect on my drive).
The fact that you indicate that stat does report the correct file size as per dd's actions, confirms what I put forward above.
I'd like to back up a SSD which I'm using for CentOS. Trying to learn dd. My drive is a fairly simple GPT partition of 120GB.
I run "dd" to copy the image of sda to a USB stick sdd1:
[root#localhost ~]# dd if=/dev/sda conv=sync,noerror status=progress bs=64k of=/dev/sdd1
120029118464 bytes (120 GB, 112 GiB) copied, 30810 s, 3.9 MB/s
1831575+1 records in
1831576+0 records out
120034164736 bytes (120 GB, 112 GiB) copied, 30810.8 s, 3.9 MB/s
But then when I examine the USB stick, there is nothing to be seen on it and I see no way to mount it
this is what appears under the Disks command
Question is:
How do I access the image?
(As a side note, I read a claim that the dd command is like the IBM JCL statement of the same name. I was a mainframe programmer. The IBM DD command is often still called a "DD Card". It doesn't copy files. It just joins your file declaration in your program to some external file. To copy a file the old skool way is to use IEBGENER)
if=/dev/sda Is cloning the entire disk and of=/dev/sdd1 Is writing to a partition. Which doesn't make much sense.
You may want to clone the entire disk onto another disk
dd if=/dev/sda conv=sync,noerror status=progress bs=64k of=/dev/sdd
Or better yet clone to an compressed image
dd if=/dev/sda | gzip > /sda.img.gz
And restore like so
gzip -d /sda.img.gz | dd of=/dev/sda
DD is a tool for linux which can Write partial data from MBR.bin to a sector in USB (instead of writing a whole sector). Now I need to do such thing in windows. There is a DD for windows, but it seems it will write a whole sector!
I need to write first 440 bytes of a mbr file to a usb stick. the code in linux is:
dd if=mbr.bin of=/dev/sd<X> bs=440 count=1
and in windows it will be:
dd bs=440 count=1 if=mbr.bin of=\\.\<x>:
where x is the volume letter. But in windows it will cause USB to be corrupted and usb need to be formatted. It seems it writes the whole data. How can I solve this problem?
Copy a complete block!
e.g. for a 512 byte blocksize (512-440=72)
copy mbr.bin mbr.full
dd bs=1 if=\\.\<x>: skip=440 seek=440 of=mbr.full count=72
dd bs=512 if=mbr.full of=\\.\<x>: count=1
Are you sure you pass the parameters correctly? Maybe the win version expects it to be /bs=440. Just a guess. Can't you anyway just truncate the file to 440 bytes?
This question already has answers here:
Size() vs ls -la vs du -h which one is correct size?
(3 answers)
Closed 8 years ago.
There is a file named today.log in my server.
ls -l today.log showing 400GB.
du -sh today.log. showing 240GB
What is the difference between ls and du ...
du shows how much disk the file uses. ls shows how big the file is. These two values can be different. Files with holes can take up less space than their size. Most files do not completely fill the blocks of the filesystem, so they take up more space than their size. A file with a single byte still takes up at least one full block. (512 or 1024 bytes, typically.) As an examle, consider a file with a single byte at position 183738475 (randomly typed numbers). That file can be stored on disk using a single block (whenever the kernel queries the filesystem for bytes other than the single byte in the file, the filesystem reports them as being zero, and there is no need to store anything. Not all filesystems work this way.) But the size of the file is 183738475, so ls will report that and du will report how many blocks are used by the filesystem. du -h will report the number of blocks used times the block size converted to a human readable format. Keep in mind that the actual numbers will vary depending on your filesystem. For example:
$ echo > foo; ls -l foo |awk '{print $5}'; du foo; du -h foo
1
8 foo
4.0K foo
This file is one byte in size but consumes 8 blocks on disk, and the block size is 512 so those 8 blocks consume 4k. (My filesystem has been optimized for large files, and small files waste a lot of space.)
I'm trying to write a bash script to audit hard drives that have been wiped to ensure the wiping system is working properly. I would like to find a way to hex dump specific parts of a drive without having to hex dump the entire drive and extract the parts I'd like (as this seems to run for too long to make the script worth writing). Ideally, I'd be able to grab parts from the beginning, middle, and end of the drive.
I would like to take the output of the hex dump and check it for the existence of only one character (indicating the drive has been successfully wiped). This part, I can handle, but I thought it may affect any advice I may get.
I've used head piped into xxd to get the beginning of the file which has worked, but I'm still stuck on the other parts. I've tried using tail to just get the end of the drive, but that doesn't seem to work quickly either. Is it possible to do this efficiently? Possibly using dd or something else and pipe it into a hex editor? I've looked through options for xxd as well as hexdump to no avail. If someone could point me in the right direction, it would be greatly appreciated!
xxd has options to skip a ways into the file (-s) and dump a limited length (-l). If you use its plain hex (-p) option, you may be able to use grep to find any anomalies:
$ xxd -s 8192 -l 256 -p /dev/disk3s2 | grep [^0]
000000010000000000000000000000000000000000000000000000000000
000000000000000000000000300000000000000800000000000000000000
dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb
dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb
dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb
od has similar skip (-j) and limit length (-N). Similarly, dd has skip= and count= (although these are counted in blocks, not bytes; you can change the block size with bs=).
EDIT: Since xxd -p is giving weird results (not stopping at what should be the end of the device), I'd recommend running some tests to figure out what's going on. First, back up anything important on the computer, because if something is weird at the device access level, it's possible that some of these tests might overwrite something unexpected, possibly even on another disk.
Next, try dumping to the end of the device with different tools, and see if they all behave the same way:
xxd -s 65451982336 /dev/sdb | more # This *should* dump 512 bytes (32 lines) then stop, but apparently keeps going
od -xv -j 65451982336 /dev/sdb | more # This also *should* dump 512 bytes then stop
dd if=/dev/sdb skip=127835903 | xxd | more # This again should do the same thing (note that the skip value is in 512-byte blocks)
Do the other tools read past what fdisk reports as the end of the disk? If all three read more data, I'm going with the "fdisk is wrong/misleading" answer. You can test further by writing some nonzero data past the "end" and seeing what the results are:
dd if=/dev/random of=/dev/sdb seek=127835903 count=2
...then repeat the various dump commands. If they show two blocks (=64 lines) of random data followed by zeroes, I'm pretty sure the device is bigger than you think it is.
I am not near my shell, but something along these lines should get you started:
dd if=/dev/hda1 | hexdump -C | grep [^00]
will print all non-zero bytes.
dd if=/dev/hda1 | od -x -j100
will give you a hexadecimal dump with offsets, starting 100 bytes in.