I am using the STM32 ARM CRC peripheral and getting different CRC codes for the same data when fed in as bytes compared to when fed in as words.
Using the byte word length and a small word aligned data string:
const char *ts4 = "The quick brown fox jumped over the lazy brown dog."; // 52 CHARS divisible by 4;
This, with a buffer size of strlen(ts4) gives a CRC32 of ~ 0x01fba559
0xfe045aa6.
It was then configured the CRC for WORD size (setting buffer size to strlen(ts4)/4) and the DMA engine was pointed at the CRC data register. It gave a different CRC result, ~ 0xf2bd1910 0x0d42e6ef, so I called it, again for WORD size using the HAL_CALCULATE method (to ensure the DMA was working as expected). This again gave ~ 0xf2bd1910 0x0d42e6ef.
Does the CRC32 algorithm give different results for different word size inputs ? I don't really want to tie the DMA engine up transferring bytes. Is there an equivalent `C' function that calculates CRC32 with a 32 bit WORD input ? I have tried reversing the order of the bytes in the word but this does not solve it (I thought it might have been a big/little endian problem).
That's 51 characters, not 52. That length over 4 would give 12, not 13. The CRC of the first 48 characters would be expected to be different than the CRC of the 51 characters.
Also I'd think that you would need to assure that the string starts on a word boundary.
Related
First I want to apologize because English is not my native language. I'm taking CS50 Introduction to Computer Science course and I've come across the concepts of 'Endianness' and 'Word size' and even though I think I've understood them pretty well, there's still some confusion.
As far as I know, 'Word size' refers to the number of bytes a processor can read or write from memory in one cycle, the instruction bytes it can send at a time, and the max size of memory addresses also; being them 4 bytes in 32-bits architectures and 8 bytes in 64-bits architectures. Correct me if I'm wrong with this.
Now, 'Endianness' refers to the ordering of bytes of a multi-byte data type (like an int or float, not a char) when the processor handles them, to either storage or transmit them. According to some definitions I've read, this concept is linked to a word size. For example, in Wikipedia, it says: "endianness is the ordering or sequencing of bytes of a word of digital data". Big-endian means when the most significant byte is placed in the smallest memory address and low-endian when the least significant byte is placed in the smallest memory address instead.
I've seen many examples and diagrams like this one:
Little-endian / Big-endian explanation diagram
I understand big-endian very well and little-endian when the data type being processed has a size equal or smaller than the word size is also clear. But what happens when it's bigger than the word size? Imagine an 8-byte data type in a 32-bit little-endian architecture (4-byte words), how are the bytes actually stored:
Ordering #1:
----------------->
lower address to higher address
b7 b6 b5 b4 |b3 b2 b1 b0
word 0 |word 1
Ordering #2:
----------------->
lower address to higher address
b3 b2 b1 b0 | b7 b6 b5 b4
word 0 | word 1
I've found mixed answers to this question, and I wanted to have this concept clear to continue. Thank you in advance!
Encountered this problem and the solution said
"32 bit address bits, 64 byte line means we have 6 bits for the word address in the line that aren't in the tag, 32,768 bytes in the cache at 64 byte lines is 512 total lines, which means we have 12 bits of address for the cache index, write back means we need a dirty bit, and we always need a valid bit. So each line has 64*8=512 data bits, 32-6- 12=14 tag bits, and 2 flag bits: data/total bits = 512/(512+14+2)=512/528."
When I tried to solve the problem I got 32kB/64byte=512 lines in total, i.e. 2^9=512. In addition, a 64 byte cache line size, 1 word=4 bytes, is 64/4=16 words per line i.e. 2^4.
To my understanding the total amount of bits in a cache is given by total amount of entries/lines in the caches*(tag address + data)-> 2^9*((32-9-4+2)+16*32). Thus, the amount of data bits per cache line is 512 (16 words *32 bits per word), and the tag is 32-9-4+2=21 (the 9 is the cache index for direct mapped cache, the 4 is to address each word and the 2 is the valid bit and dirty bit)
Effectively, the answer should be 512/533 and not 512/528.
Correct?
512 lines = 9 bits not 12 as they claim, so you are right on this point.
However, they are right that 64 byte lines gives 6 bits for the block offset — though it is a byte offset, not word as they say.
So, 32-6-9=17 tag bits, then plus the 2 for dirty & valid.
FYI, there's nothing in the above problem that indicates a conversion from bytes to words. While it is true that there will be 16 x 32-bit words per line (i.e. 64 bytes per line) it is irrelevant: we should presume that the 32-bit address is a byte address unless otherwise stated. (It would be unusual to state cache size in bytes for a word (not byte) addressable machine; it would also be unusual for a 32-bit machine to be word addressable — some teaching architectures like LC-3 are word addressable, however, they are 16-bits; other word addressable machines have odd sizes like 12 or 18 or 36 bit words — though those pre-date caches!)
A processor has 64 bits address line, and has a 16-way set-associative cache. The memory is a word (i.e 2 bytes) addressable. The cache size is 2 MByte, and the line size is 64 bytes long.
a. Show the memory address structure.
What is the effect of the phrase (2 byte addressable) on solving the question?
The solution will vary between byte addressable and word addressable !
In ye olde days it was common for processors to access memory in units other bytes. It was extremely common for computers to use memory in units of 36 bits (DEC, Sperry). There were deskop computers that used 14-bits.
A word is then the smallest unit of memory that a specific computer addresses. When the word is 8-bits, it is called a byte.
In your example the word becomes 16-bits.
You question is totally screwed up. If you have a word addressable machine, then it does not have bytes.
They appear to be saying
The cache size is 1 Megaword, and the line size is 4 words long
I am continually amazed at how academics come up with ways to make the simple complex.
There are actually two meanings of the term "word" in common use. (1) is the addressable unit size. (2) Processors generally support multiple integer sizes (e.g., 8, 16, 32, 64, 128 bits). Is it common for processor documentation to call one of this sizes greater than 8 a word.
The processor might have a MOVE BYTE instruction that moves 8-bits and a MOVE WORD instruction that moves 16, 32 or 64 bytes; whatever the processor documentation want to call a word.
I have generated an Ethernet 10GE MAC design in VHDL. Now I am trying to implement CRC. I have a 64-bit parallel CRC-32 generator in VHDL.
Specification:
- Data bus is of 64-bits
- Control bus is of 8-bits (which validates the data bytes)
Issue:
Let's say, my incoming packet length is 14-bytes, (assuming no padding).
The CRC is calculated for the first 8 bytes in one clock cycle, but when I try to calculate the CRC over the remaining 6 bytes the results are wrong due to zeros being appended.
Is there a way I can generate the CRC for any length of bytes packet length using a 64-bit parallel CRC generator?
What I've tried:
I used different parallel CRC generators (8-bit parallel CRC, 16-bit parallel CRC generator and so on). But that consumes a lot of FPGA resources. I want to conserve resources using just 64-bit parallel CRC generators.
Start with a constant 64-bit data word that brings the effective CRC register to all zeros. Then prepend the message with zero bytes, instead of appending them, putting those zeros on the end of the 64-bit word that is processed first. (You did not provide the CRC definition, so this depends on whether the CRC is reflected or not. If the CRC is reflected, then put the zero bytes in the least-significant bit positions. If the CRC is not reflected, then put them in the most-significant bit positions.) Then exclusive-or the result with a 32-bit constant.
So for the example, you would first feed a 64-bit constant to the parallel CRC generator, then feed two zero bytes and six bytes of message in the first word, and then eight message bytes in the second word. Then exclusive-or the result with the 32-bit constant.
For the standard PKZIP CRC, the 64-bit constant is 0x00000000ffffffff, the 32-bit constant is 0x2e448638, and the prepended zero bytes go in the bottom of the 64-bit word.
If you are in control of the implementation of the CRC generator, then you can probably modify it to initialize the effective CRC register to all zeros when you reset the generator, avoiding the need to feed the 64-bit constant.
I can't speak for certain, but if you can pad zeros at the start of your packet instead of at the end, then you should get the right answer. It does depend on the polynomial and the initializer...
See this answer here Best way to generate CRC8/16 when input is odd number of BITS (not byte)? C or Python
A byte consists of 8 bits on most systems.
A byte typically represents the smallest data type a programmer may use. Depending on language, the data types might be called char or byte.
There are some types of data (booleans, small integers, etc) that could be stored in fewer bits than a byte. Yet using less than a byte is not supported by any programming language I know of (natively).
Why does this minimum of using 8 bits to store data exist? Why do we even need bytes? Why don't computers just use increments of bits (1 or more bits) rather than increments of bytes (multiples of 8 bits)?
Just in case anyone asks: I'm not worried about it. I do not have any specific needs. I'm just curious.
because at the hardware level memory is naturally organized into addressable chunks. Small chunks means that you can have fine grained things like 4 bit numbers; large chunks allow for more efficient operation (typically a CPU moves things around in 'chunks' or multiple thereof). IN particular larger addressable chunks make for bigger address spaces. If I have chunks that are 1 bit then an address range of 1 - 500 only covers 500 bits whereas 500 8 bit chunks cover 4000 bits.
Note - it was not always 8 bits. I worked on a machine that thought in 6 bits. (good old octal)
Paper tape (~1950's) was 5 or 6 holes (bits) wide, maybe other widths.
Punched cards (the newer kind) were 12 rows of 80 columns.
1960s:
B-5000 - 48-bit "words" with 6-bit characters
CDC-6600 -- 60-bit words with 6-bit characters
IBM 7090 -- 36-bit words with 6-bit characters
There were 12-bit machines; etc.
1970-1980s, "micros" enter the picture:
Intel 4004 - 4-bit chunks
8008, 8086, Z80, 6502, etc - 8 bit chunks
68000 - 16-bit words, but still 8-bit bytes
486 - 32-bit words, but still 8-bit bytes
today - 64-bit words, but still 8-bit bytes
future - 128, etc, but still 8-bit bytes
Get the picture? Americans figured that characters could be stored in only 6 bits.
Then we discovered that there was more in the world than just English.
So we floundered around with 7-bit ascii and 8-bit EBCDIC.
Eventually, we decided that 8 bits was good enough for all the characters we would ever need. ("We" were not Chinese.)
The IBM-360 came out as the dominant machine in the '60s-70's; it was based on an 8-bit byte. (It sort of had 32-bit words, but that became less important than the all-mighty byte.
It seemed such a waste to use 8 bits when all you really needed 7 bits to store all the characters you ever needed.
IBM, in the mid-20th century "owned" the computer market with 70% of the hardware and software sales. With the 360 being their main machine, 8-bit bytes was the thing for all the competitors to copy.
Eventually, we realized that other languages existed and came up with Unicode/utf8 and its variants. But that's another story.
Good way for me to write something late on night!
Your points are perfectly valid, however, history will always be that insane intruder how would have ruined your plans long before you were born.
For the purposes of explanation, let's imagine a ficticious machine with an architecture of the name of Bitel(TM) Inside or something of the like. The Bitel specifications mandate that the Central Processing Unit (CPU, i.e, microprocessor) shall access memory in one-bit units. Now, let's say a given instance of a Bitel-operated machine has a memory unit holding 32 billion bits (our ficticious equivalent of a 4GB RAM unit).
Now, let's see why Bitel, Inc. got into bankruptcy:
The binary code of any given program would be gigantic (the compiler would have to manipulate every single bit!)
32-bit addresses would be (even more) limited to hold just 512MB of memory. 64-bit systems would be safe (for now...)
Memory accesses would be literally a deadlock. When the CPU has got all of those 48 bits it needs to process a single ADD instruction, the floppy would have already spinned for too long, and you know what happens next...
Who the **** really needs to optimize a single bit? (See previous bankruptcy justification).
If you need to handle single bits, learn to use bitwise operators!
Programmers would go crazy as both coffee and RAM get too expensive. At the moment, this is a perfect synonym of apocalypse.
The C standard is holy and sacred, and it mandates that the minimum addressable unit (i.e, char) shall be at least 8 bits wide.
8 is a perfect power of 2. (1 is another one, but meh...)
In my opinion, it's an issue of addressing. To access individual bits of data, you would need eight times as many addresses (adding 3 bits to each address) compared to using accessing individual bytes. The byte is generally going to be the smallest practical unit to hold a number in a program (with only 256 possible values).
Some CPUs use words to address memory instead of bytes. That's their natural data type, so 16 or 32 bits. If Intel CPUs did that it would be 64 bits.
8 bit bytes are traditional because the first popular home computers used 8 bits. 256 values are enough to do a lot of useful things, while 16 (4 bits) are not quite enough.
And, once a thing goes on for long enough it becomes terribly hard to change. This is also why your hard drive or SSD likely still pretends to use 512 byte blocks. Even though the disk hardware does not use a 512 byte block and the OS doesn't either. (Advanced Format drives have a software switch to disable 512 byte emulation but generally only servers with RAID controllers turn it off.)
Also, Intel/AMD CPUs have so much extra silicon doing so much extra decoding work that the slight difference in 8 bit vs 64 bit addressing does not add any noticeable overhead. The CPU's memory controller is certainly not using 8 bits. It pulls data into cache in long streams and the minimum size is the cache line, often 64 bytes aka 512 bits. Often RAM hardware is slow to start but fast to stream so the CPU reads kilobytes into L3 cache, much like how hard drives read an entire track into their caches because the drive head is already there so why not?
First of all, C and C++ do have native support for bit-fields.
#include <iostream>
struct S {
// will usually occupy 2 bytes:
// 3 bits: value of b1
// 2 bits: unused
// 6 bits: value of b2
// 2 bits: value of b3
// 3 bits: unused
unsigned char b1 : 3, : 2, b2 : 6, b3 : 2;
};
int main()
{
std::cout << sizeof(S) << '\n'; // usually prints 2
}
Probably an answer lies in performance and memory alignment, and the fact that (I reckon partly because byte is called char in C) byte is the smallest part of machine word that can hold a 7-bit ASCII. Text operations are common, so special type for plain text have its gain for programming language.
Why bytes?
What is so special about 8 bits that it deserves its own name?
Computers do process all data as bits, but they prefer to process bits in byte-sized groupings. Or to put it another way: a byte is how much a computer likes to "bite" at once.
The byte is also the smallest addressable unit of memory in most modern computers. A computer with byte-addressable memory can not store an individual piece of data that is smaller than a byte.
What's in a byte?
A byte represents different types of information depending on the context. It might represent a number, a letter, or a program instruction. It might even represent part of an audio recording or a pixel in an image.
Source