Set GPIO bank number in device tree - linux-kernel
Do you know how to set the GPIO bank number in the device tree for an i2c gpio controller?
I tried with gpio-base (described in the gpio driver doc) but it was not very conclusive.
I have a device tree:
...
&i2c2 {
pca9502: pca9502#4C {
compatible = "nxp,pca9502";
reg = <0x4C>;
gpio-controller;
/* HERE */
gpio-base = <1>;
}
}
...
and the driver fetch the DT to find the gpio-base:
of_get_property(dev.of_node, "gpio-base", NULL);
In this case, a gpiochip is created with the number 2^32 giving: gpiochip16777216. And I can't access my GPIO using echo XX > export.
When I don't add gpio-base = <1>;, it works but the gpiochip number is 504 (with -1 as a default bank number) which I don't understand.
I don't know how to define the bank number into the device tree.
Well, my issue was not linked with the device tree structure but with the endianness of the processor.
Related
How do you set the ESP32 to maximum BLE Power?
How do you increase the ESP32 BLE to maximum power, for maximum distance? It appears the default is set to index 5, which is only +3dbm. (ESP_PWR_LVL_P3 = index 5, Corresponding to +3dbm) More Details
There used to be a bug, it appears its fixed now. As indicated above, ESP_PWR_LVL_P9 = index 7, which indicates +9dbm.
After "BLEDevice::init("ESP32");", add this to get the maximum output:
esp_ble_tx_power_set(ESP_BLE_PWR_TYPE_DEFAULT, ESP_PWR_LVL_P9);
esp_ble_tx_power_set(ESP_BLE_PWR_TYPE_ADV, ESP_PWR_LVL_P9);
esp_ble_tx_power_set(ESP_BLE_PWR_TYPE_SCAN ,ESP_PWR_LVL_P9);
Check via this:
int pwrAdv = esp_ble_tx_power_get(ESP_BLE_PWR_TYPE_ADV);
int pwrScan = esp_ble_tx_power_get(ESP_BLE_PWR_TYPE_SCAN);
int pwrDef = esp_ble_tx_power_get(ESP_BLE_PWR_TYPE_DEFAULT);
Serial.println("Power Settings: (ADV,SCAN,DEFAULT)"); //all should show index7, aka +9dbm
Serial.println(pwrAdv);
Serial.println(pwrScan);
Serial.println(pwrDef);
Can also confirm via an Android phone app like nRF Connect, that the Tx Power Level is now 9dBm.
Is it correct that ECC generated key keeping unchange?
I am trying to use ECDH/ECDSA to secure my transport. And I ported micro-ECC to mbed as a test, since I am not familiar with assembly, all assembly optimization is disabled. Here is my project.
The running code prints information to console as following:
microECC test
Start ECC computation
make key 1
RNG():
5647A0B05F947B4958E6AD073D1B1BC844B973131247106CAA89BBA6747D965
7828AC7240A3733939EC09EE3327EAB55A366239516FFDE82AABAAAA428A8E9F
A85C8C162E8AADA7407F2BF7ACDF6DC6EFE2C7111B91623612C979C32776F7D
182D69A2108F85E37F36B3428A8EA07C05EA8C608983425E6A460458AD465B
6AC6DC351EC5D4E75FB55FD7749971E16E5CA069A56C20730F1D8525E43FFA
2A81D1F46817F40B18EEF5606797208B591A9A8D79B30FA76C6942DA1F464239
518458C17CE01045477D06FA597B0388248BB0C379B03DC93A4B33414CC89B6B
2702D42E4A606F071FF24B7FBFC60671E69F7E87AC35024332C9D453A7388
5F3902CB45349175ECF5BD7C307884465B0BB745B979111605E2C33384756
4E3841E96CAE82C239B6838A1FF80FF92B9F6AF671E57CBD15CAF3EA63C5BF3F
7642C9234191B9CE4EA408A7767A18D15E900F2C3D03FB6448E7839B257C889F
79F2306F1491BE021410231B5240D0002FCFF2AF307918102D266E85143DBC40
56FFD07A264869663FAFA43741B9BD241F8058F23A74221319B948512403982
663D2C8066DD7D4C6523D7EB568D1072491FCF9149D80AA9340603E9BD8333
545E49B652AF11BF5B23665670A99AEC176DB6752074AC5C225BB4D97815676F
74601D7C3707C3530194C955BA764B2C7E06EE17BCC4207E37F7D1D4B3E3F
354CC32C13C25EF67F8BA1C0453D296229AE6D664460F8C7688889317FA3286
462F7DF42B6E7F282B1DB175287CF0BE102FCB474FF7952B2381746C571BC4
677A38B1209B0CE6C504C5DA09E170105A0D0A6C46A8B562959FADAAE73B7
115B3CC84BDC6F97900289E156913B548ED461D2E70B58A245849A96256BCDA
4162196C1242AA2D709A9E5735FFC6415C5EB8F24B01AA411985BD2972D9555C
7D0F65481769C4842D09E1E152114EB912DED22A5A9A607E76B5047061C578E8
23993CDF1D2879B47FC5C7FF7CC79DF148EE489384652FB5306B8587718A3C4
781DE9DE227DB88F420C93EB373998F52F34B99375D7D1BA7E00055B4CE5647A
public1:
FA624B9E629CD6ACFA0F27B38E937FCA9B85610412E25DAA2044BF0D05DAD5E0A7FB0EFBB152147D
private1:
005AA72C06F34B2D2EAAB395EE70CF5DA261AD5969
time: 293ms
make key 2
RNG():
403BAB06244092E721807F8D2C4FC4162D5496BD6C9927E619BB3F2A7427F1B3
5B6B432D5AD7761D64424272D5D380D23CFB62E22D072807AA8BA7972E22130
2E67B3333FB4BFDB6D2FE1E7A03908E36128251642F642970DEFFCD4F37250C
19A66B8311FAF0709D0BBA615D3373C7A7C79EA2C5614F966C4126E7CEE94B1
282385F91A9134162D8B2A2C3EFE4356A17815677DA72D07ABE35BB5C5D8A27
63B9BA5518BD3216437CC4AF48C7E167227ADC9C23F9BC5B614FF84F30C68D3C
5D064FB81FC6891B22E548C5521C96A542CECC7A324548E6D1ADB6A54DCD923
4112C6264EE2C42D41648A88AF1CD003C8E4600616DDE903F344D822346E650
7CBB54BC73CF9B2D44162A324C1870D27B223074748678145B3DF11384BFF7
5BCAFC2329E832235586E2CD2A83B84C1AD741275559E1996A489E064E4CA07A
55DE6EAA23D704697D25289616942CD8275DC73F5926982468C80CC959E41041
538461A8646F5231146AF60F1EF3B3D211C2DF44DF7B815424C9D627D5D540E
BADC4714BE1C2D444D1182C2BBF44D2302B3E8454D4BB186F53DB54129825B8
C2076591AFDB4865990A8D326F547BE732216312BB7C8EE75B6FCB23EAD3F89
5BB4DF8D69EAE87552E8D995FCF563358929D4E2737FE79141B9A944D48902A
4E66031E4FA3F93A2B5E684554165C707BB3070E4CC3DFE74E8DFFD58D182D1
4EECF6668F92B39792A44A1E7319BD7F03D1A45DD8C78C2F5274523F5DE801
3F8832F8100E39DD5E819EE9593176AE1E6DE7A2538264C2381CFEEF232204F8
1410F25B788BED3535BB9C4BC1A186515516F715A531356BD2920B353F2EFC
39D09BB262DCB67722CEFE0F67FC2EB77DF2CE2A126C2BA24DB1289E6445D164
7E1FAF7C74CCD1602018CCEB46EFD7BEB18B067EE04DE4C2E3CB888B3803
6E735880798E58F045FBEBD3555C9E905222467A68100C1F21A797F52204CF18
17A98EE8185DBB8A49A3D7D124CD485D2BB7C2BD2F3E8613471831D915606222
2F1E08B1547D8B7F31A97CBC3E9E7259C40CCAC781FE6886775AEB978BAF
RNG():
790A66D74370521F2165EE8D73D20D565F9C2FAA790809CE64CD667B6F4907F5
150965DD17CD22F0C7F8B2DB6E5F85C023C264374B03630F2DA3DC14543C
72F70D1A4061862C7FED7EF11DE66BBB8CC8BB06FC0DC5A7BE7758F58F5EFBD
33C6E2D56391DB3844EEC3682541B3D26E3FEF6853E93F422398CF6A3AF72801
9D6D9F31865D7653E1F0A2356573FFC739B95A325569E2F7AEF7BAB5618A250
1E68D403114D0296C1168403BFA7043A1FAE49823044915B700D92D698EC
7EDD3F018C4DDE26F8EA64713305D7ADAD11A014B7BBFB35218CC545928CDE
2DD8D4BE19461FBA612104E5EF2218615481C0B710084105376D8D85198AF01
268B632276727582BE897426367AEDE6506AB695B5C1568389F5D2DE04B73A
6303EA524929A92F5545772418CD8C382F2A90FE75FE08447BA5917C3503B3B3
7FC65D4C6260CE7E355F69477B573C564A8F7DB44EB9DB8120B3D7169D94362
502F963F22B3F35321B9CBB27A3B4E2B3FDCAD5E6455033F83817B16A904EB2
5762D58D2CBEFA815290AAC27504A8D579579FC96520DF3B4AA49ED44760BC93
18B7DB5255D7DE476E412395677672AE3ED46F3A49B3E40F58A8A4F12E08E306
3BC5C90D8AED008281937C29CE470662938D561AD1922725E2DE7AAE7F9CC
2DAD303F129164DF42284DFE4C13C8A0599007B964DC1E0248633F5E6F3FBBCB
7598E8893510BD2B765225975D8411E81283F9885714772B60ABAABE69E50B86
59BB14E37668D54399C2796395A286B4D89E174615B25A0309AB0164FC043
1A0EB6243D515529796A55A94792FBCA3E3B254C102DA43535655E805AD76A98
5239D4E0603638D75FE7E36A1C730E676BA51FB6E32D1FB5989BD273DF8C760
789313FC183396BC48A508183B291BFF209DAC5B37B9F0FF3D3DF9C4745C2598
7C14BDDA2195E24963776BAB16F2204C4DBC94D3461C67BD66CCAB7C4927FDA5
192E8B55165F72566A7D8BE25D54482564B47BB1D5C31F55CA4D5FA658461C1
57F6D2D56A575E55627860C869AAD8D3399C9C3A4BBDA80979F6C33E3109784F
public2:
85BFA8BDEBB60ACF996FFA810BF6B21D291B71488C0DC505B647D6EDF23C0612A2CC1F8AFD70DADD
private2:
005E64D1E4C98466DDD62A410A29264EEE8133A2E3
time: 430ms
make share secret 1
RNG():
26EAA6DF22BEA937F2C68B41A084AA438C36773F2D05035E73D657243C6A25
2EF7A4E83F7FAE9654D8A9887A78D1687B7AA385D867EF91C8B66066FF67C73
F57D2F073EA8F1B76E25A4D65884D7212857F91CCB724A6DB357323C7B3E7D
B8444D60F35CD365137F577EEA209712DD606865E612F73AD03DB22E8EE99
17A93C833D8EA1A91D8025284C3A40653FD5461F4B2CE62976C3964DF70AD70
67C8E5053E896F91364CF1EB3694D77B3A292B6A4AF6A70232D154DF33F01522
6949A64617DED047721E86165993D68387211AB11238F6919F6584A712B673C
7F67C388678B150F49C309405F32205651EB0B70430490BC11B5E00452AEA47E
689AECE7389658C9781890113379A135222CDA9F20AAE92250AB3A1419CBFE7A
2E76FFB5650CAB7E16D587645E4EF20A558F3A6D4DBC795B12B6065576801F7B
25E54E893EDE3C8C567D8B7F67433D0670E906F1AC1CA1B3EDE1D5D1987C19
12299C62BF15798DBCFE795BE3E074CAA89C53D526436BC08A5A1C836097
71E47DBE66C3DAA31D7B8AD15F0B3192758BBFCD65DA59AE145588C965814E8B
5B8ED2F65AEA96105A975FB85A3595D340383F45CBC43756D445F813BE5AA93
52906B0D1514FD87E4F74EA2391C4A65578E0C73B5E5E11C1CD99482B7F6
4EB8E2F34ACD3C587086D5663AECFE33503FFF6F14FADB7422155F8510193A76
142CF4A02356BD163BF1457A7C5F1486672ABB4D4B943B906A496AC65FD9A05C
CCDD05D6B5E44E01D1E87266D89B7372DC4254F11D5B1A85D5E2D79784D9AB2
7FFF52BC45E8F7C65D67E4067990EB211D30C78E2D8238A634AB0F5E3CDEB1B6
2E1F30A11DD0310B5DD6470A69C7E0AB7EAE43D954933CC53607ABD16ED708DC
r: 0001
secret1:
AA343FE06CD9534E38DCB762B5EE3590FDA318BA
time: 265ms
make share secret 2
RNG():
87E54CF4A3D8C24266D2D6910373029B367F68170F0FEF1A340B2E2DD291B1
1272327D718CF6AE323938426974B4A36781C7F1315B93618B2036A3248918F
331B0A87D4C439117ECD85421D42263B30FC95309C101C3E83FF29783F550A
69DBF82977DF2BC2755EDF13B84AE422693D13654E3E34968A330386335C2F8
4F4E0B8DCC9B0B038587D4734CE1AC07E0BDA094A06CBC76E47732A44C01DB1
5FF58FAB65AE65F39587E0C217F0B5810ED532B1DA260A26F0BAD1B1F7C1117
300BECF47A04B2D63F00D90E4CA9D0F96DEA8AE56DD9954678E4D4BC6846A887
1CCA26ED54E83DFD35F60D18156086D6791B0FCE4436A5133D8CD0A36705B6BE
6D6C7CCD7917C6F21E5E00F16900DA72BBCB3E524E01EC22F28FEA43E60071C
42FEF6E11FE764D57E360A5E3C9299181D8A7ED15FCD62C14326406C2EB56AF0
328E6C9F7C2DED7863E09E72DF3589861C6C13D643B54CF22150711616D86FF
5A7231857061C65A1934BB8944E6A242366F12041C32F585D585BF13D7C39BF
6E971F861E7C2D567730F1C1735C81134FC44DB7001D70EEBF47BDB23C688
4E44287746594A462379EA695AEFA5CB19E5B7F9587905971517BFAE69C83010
646E962E2597F4DA19AA5AC74A7BFA9C5D5804E187C35D750A12F9F65CF8EA4
34737A6970F02868594C7DE768174BF438C10643A57114A62F574ED4CD32591
C53462048D52BC63A5B0DEDE39F64E2ECE9F9A4C8D8BA54C36A71018F4D477
94163D052F6CC2C3A33E7AB30F111C31F89DC2E2B4328C6582733A11168F7D
198B8B25415FAA6B407727B4D394E1A1B84F0B578EEE2FF4EEE94281BF86FED
14A83511752434E267DFA09848EF6D87744A4DBE44F110EE46FC046E4FC1F034
r: 0001
secret2:
AA343FE06CD9534E38DCB762B5EE3590FDA318BA
time: 265ms
However, the result keeps same, including random number, public/private keys and shared keys. I actually added a random_test() to check its RNG.
#if defined(RNG_TEST)
void randtest()
{
uint8_t buf[16];
pc.printf("randtest():\r\n");
for(int i=0; i<16; i++){
buf[i] = rand();
pc.printf("%02X",buf[i]);
}
}
#endif
And its random number keeps changing anyway.
Well, eveything is caused by a poor RNG. The rand() is not a TRNG. So the result always the same. Now I used two seperated ADC as RNG, so at least result for Private/Public Keys, Share keys and random numbers are changing every time. Now I moved to verify the ECDH with Python counterpart. If it is successful, then a simple ECDH can be used in a real IoT project.
Memory mapped area of GPIO in raspberry pi
I have real struggles with respective to understanding the data sheet part[BCM2835-ARM-Peripherals.pdf] where in which it talks about following: Q1: Every gpio has 40 registers[refere: Table 6-1 GPIO Register Assignment] which are common for all gpios can be used to SET/CLEAR/GPFSEL[0-5] etc,but what is the purpose of GPIO function select register which has 10 FSEL (0-9) registers [refer: Table 6-2 – GPIO Alternate function select register 0]. Q2: How can I reach to a given GPIO through memorymapped region? , is something like below Case#1 :GPIO4 with alternate function 5 as Input: as ( 4 < 10) it will use "0x7E200000" with FSEL4 register[refer: Table 6-2 – GPIO Alternate function select register 0] having 000 and 010 = GPIO Pin 9 takes alternate function 5. case#2 :GPIO27 with alternate function 3 as Output: as ( 27 < 30) it will use "0x7E20000C" with FSEL3 register[refer: Table 6-2 – GPIO Alternate function select register 0] having 001 and 111 = GPIO Pin 9 takes alternate function 5. Please let me know whether my understanding is correct or not? Q3: what is the purpose of following tables Table 6-3 – GPIO Alternate function select register 1 Table 6-4 – GPIO Alternate function select register 2 Table 6-5 – GPIO Alternate function select register 3 Table 6-6 – GPIO Alternate function select register 4 Table 6-7 – GPIO Alternate function select register 5 BR, &Sanumala
http://elinux.org/RPi_GPIO_Code_Samples
//
// Set up a memory regions to access GPIO
//
void setup_io()
{
/* open /dev/mem */
if ((mem_fd = open("/dev/mem", O_RDWR|O_SYNC) ) < 0) {
printf("can't open /dev/mem \n");
exit(-1);
}
/* mmap GPIO */
gpio_map = mmap(
NULL, //Any adddress in our space will do
BLOCK_SIZE, //Map length
PROT_READ|PROT_WRITE,// Enable reading & writting to mapped memory
MAP_SHARED, //Shared with other processes
mem_fd, //File to map
GPIO_BASE //Offset to GPIO peripheral
);
close(mem_fd); //No need to keep mem_fd open after mmap
if (gpio_map == MAP_FAILED) {
printf("mmap error %d\n", (int)gpio_map);//errno also set!
exit(-1);
}
// Always use volatile pointer!
gpio = (volatile unsigned *)gpio_map;
} // setup_io
Problems making a parallel Flash memory programmer with arduino
I have a very large project and this a small piece of it, but none the less essential. I have a parallel flash memory chip made by SST and Microchip (a bit confusing) and am having trouble bypassing the write protection. I am using an arduino mega to program it because I do not have time to wait for a programmer to ship from china. here is the datasheet for the flash memory: http://ww1.microchip.com/downloads/en/DeviceDoc/25022B.pdf
void setup() {
Serial.begin(19200);
pinMode(A8,OUTPUT);//OE#
pinMode(A9,OUTPUT);//WE#
for(byte i=0;i<15;i++) //15 bit address bus
pinMode(i+20, OUTPUT);
for(byte i=0;i<8;i++) //8 bit bidirectional data bus
pinMode(i+40, OUTPUT);
wrt(0xAA,0x5555);// erase sector 0 to 0xFF
wrt(0x55,0x2AAA);
wrt(0x80,0x5555);
wrt(0xAA,0x5555);
wrt(0x55,0x2AAA);
wrt(0x30,0);
delay(250);
wrt(0xAA,0x5555);// write byte 0 to 3
wrt(0x55,0x2AAA);
wrt(0xA0,0x5555);
wrt(0x03,0);
delay(1000);
Serial.println("reading....");
for(int i=0;i<16;i++)// read data
rd(i);
}
void loop();
void wrt(byte var, int loc){
datbusout();// set data bus to output mode
digitalWrite(20,HIGH&&(loc&1));
digitalWrite(21,HIGH&&(loc&2));
digitalWrite(22,HIGH&&(loc&4));
digitalWrite(23,HIGH&&(loc&8));
digitalWrite(24,HIGH&&(loc&16));
digitalWrite(25,HIGH&&(loc&32));
digitalWrite(26,HIGH&&(loc&64));
digitalWrite(27,HIGH&&(loc&128));
digitalWrite(28,HIGH&&(loc&256));
digitalWrite(29,HIGH&&(loc&1024));
digitalWrite(30,HIGH&&(loc&2048));
digitalWrite(31,HIGH&&(loc&4096));
digitalWrite(32,HIGH&&(loc&8192));
digitalWrite(33,HIGH&&(loc&16384));
digitalWrite(34,HIGH&&(loc&32768));
for(int i=40;i<48;i++)
digitalWrite(i,HIGH&&(var&(1<<i)));
PORTK=1;// write mode
Serial.println(var,HEX);
delayMicroseconds(20);
PORTK=3;// set OE# and WE# high
}
void rd(int loc){
datbusinp();
byte out=0;
digitalWrite(20,HIGH&&(loc&1));
digitalWrite(21,HIGH&&(loc&2));
digitalWrite(22,HIGH&&(loc&4));
digitalWrite(23,HIGH&&(loc&8));
digitalWrite(24,HIGH&&(loc&16));
digitalWrite(25,HIGH&&(loc&32));
digitalWrite(26,HIGH&&(loc&64));
digitalWrite(27,HIGH&&(loc&128));
digitalWrite(28,HIGH&&(loc&256));
digitalWrite(29,HIGH&&(loc&1024));
digitalWrite(30,HIGH&&(loc&2048));
digitalWrite(31,HIGH&&(loc&4096));
digitalWrite(32,HIGH&&(loc&8192));
digitalWrite(33,HIGH&&(loc&16384));
digitalWrite(34,HIGH&&(loc&32768));
PORTK=2;
delayMicroseconds(1); // wait for read to finish
for(int i=0;i<8;i++)
out|=digitalRead(40+i)<<i;
PORTK=3;
Serial.println(out,HEX);
}
void datbusinp(){
DDRG&=252;// did the same thing like this, just faster
DDRL&=3;
}
void datbusout(){
DDRG|=3;
DDRL|=252;// see last comment
}
for(int i=40;i<48;i++) digitalWrite(i,HIGH&&(var&(1<<i))); That is wrong, surely? You are shifting 1 left 40 times at least, which means you are always writing a zero.
I fixed it!!!! so I accidentally had the wrong values for anding when writing to the address bus. A TON of thanks to Nick Gammon, test would have failed today without him. more about the answer: I needed to change my for loop in the wrt function, and not skip 512 when writing to the address bus. :D code: digitalWrite(20,HIGH&&(loc&1)); digitalWrite(21,HIGH&&(loc&2)); digitalWrite(22,HIGH&&(loc&4)); digitalWrite(23,HIGH&&(loc&8)); digitalWrite(24,HIGH&&(loc&16)); digitalWrite(25,HIGH&&(loc&32)); digitalWrite(26,HIGH&&(loc&64)); digitalWrite(27,HIGH&&(loc&128)); digitalWrite(28,HIGH&&(loc&256)); digitalWrite(29,HIGH&&(loc&1024)); digitalWrite(30,HIGH&&(loc&2048)); digitalWrite(31,HIGH&&(loc&4096)); digitalWrite(32,HIGH&&(loc&8192)); digitalWrite(33,HIGH&&(loc&16384)); digitalWrite(34,HIGH&&(loc&32768)); needed to become digitalWrite(20,HIGH&&(loc&1)); digitalWrite(21,HIGH&&(loc&2)); digitalWrite(22,HIGH&&(loc&4)); digitalWrite(23,HIGH&&(loc&8)); digitalWrite(24,HIGH&&(loc&16)); digitalWrite(25,HIGH&&(loc&32)); digitalWrite(26,HIGH&&(loc&64)); digitalWrite(27,HIGH&&(loc&128)); digitalWrite(28,HIGH&&(loc&256)); digitalWrite(29,HIGH&&(loc&512)); digitalWrite(30,HIGH&&(loc&1024)); digitalWrite(31,HIGH&&(loc&2048)); digitalWrite(32,HIGH&&(loc&4096)); digitalWrite(33,HIGH&&(loc&8192)); digitalWrite(34,HIGH&&(loc&16384));
Robust and fast checksum algorithm?
Which checksum algorithm can you recommend in the following use case? I want to generate checksums of small JPEG files (~8 kB each) to check if the content changed. Using the filesystem's date modified is unfortunately not an option. The checksum need not be cryptographically strong but it should robustly indicate changes of any size. The second criterion is speed since it should be possible to process at least hundreds of images per second (on a modern CPU). The calculation will be done on a server with several clients. The clients send the images over Gigabit TCP to the server. So there's no disk I/O as bottleneck.
If you have many small files, your bottleneck is going to be file I/O and probably not a checksum algorithm. A list of hash functions (which can be thought of as a checksum) can be found here. Is there any reason you can't use the filesystem's date modified to determine if a file has changed? That would probably be faster.
There are lots of fast CRC algorithms that should do the trick: http://www.google.com/search?hl=en&q=fast+crc&aq=f&oq= Edit: Why the hate? CRC is totally appropriate, as evidenced by the other answers. A Google search was also appropriate, since no language was specified. This is an old, old problem which has been solved so many times that there isn't likely to be a definitive answer.
CRC-32 comes into mind mainly because it's cheap to calculate Any kind of I/O comes into mind mainly because this will be the limiting factor for such an undertaking ;) The problem is not calculating the checksums, the problem is to get the images into memory to calculate the checksum. I would suggest "stagged" monitoring: stage 1: check for changes of file timestamps and if you detect a change there hand over to...(not needed in your case as described in the edited version) stage 2: get the image into memory and calculate the checksum For sure important as well: multi-threading: setting up a pipeline which enables processing of several images in parallel if several CPU cores are available.
If you are receiving the files over network you can calculate the checksum as you receive the file. This will ensure that you will calculate the checksum while the data is in memory. Hence you won't have to load them into memory from disk.
I believe if you apply this method, you'll see almost-zero overhead on your system.
This is the routines I'm using on an embedded system which does checksum control on firmware and other stuff.
static const uint32_t crctab[] = {
0x0,
0x04c11db7, 0x09823b6e, 0x0d4326d9, 0x130476dc, 0x17c56b6b,
0x1a864db2, 0x1e475005, 0x2608edb8, 0x22c9f00f, 0x2f8ad6d6,
0x2b4bcb61, 0x350c9b64, 0x31cd86d3, 0x3c8ea00a, 0x384fbdbd,
0x4c11db70, 0x48d0c6c7, 0x4593e01e, 0x4152fda9, 0x5f15adac,
0x5bd4b01b, 0x569796c2, 0x52568b75, 0x6a1936c8, 0x6ed82b7f,
0x639b0da6, 0x675a1011, 0x791d4014, 0x7ddc5da3, 0x709f7b7a,
0x745e66cd, 0x9823b6e0, 0x9ce2ab57, 0x91a18d8e, 0x95609039,
0x8b27c03c, 0x8fe6dd8b, 0x82a5fb52, 0x8664e6e5, 0xbe2b5b58,
0xbaea46ef, 0xb7a96036, 0xb3687d81, 0xad2f2d84, 0xa9ee3033,
0xa4ad16ea, 0xa06c0b5d, 0xd4326d90, 0xd0f37027, 0xddb056fe,
0xd9714b49, 0xc7361b4c, 0xc3f706fb, 0xceb42022, 0xca753d95,
0xf23a8028, 0xf6fb9d9f, 0xfbb8bb46, 0xff79a6f1, 0xe13ef6f4,
0xe5ffeb43, 0xe8bccd9a, 0xec7dd02d, 0x34867077, 0x30476dc0,
0x3d044b19, 0x39c556ae, 0x278206ab, 0x23431b1c, 0x2e003dc5,
0x2ac12072, 0x128e9dcf, 0x164f8078, 0x1b0ca6a1, 0x1fcdbb16,
0x018aeb13, 0x054bf6a4, 0x0808d07d, 0x0cc9cdca, 0x7897ab07,
0x7c56b6b0, 0x71159069, 0x75d48dde, 0x6b93dddb, 0x6f52c06c,
0x6211e6b5, 0x66d0fb02, 0x5e9f46bf, 0x5a5e5b08, 0x571d7dd1,
0x53dc6066, 0x4d9b3063, 0x495a2dd4, 0x44190b0d, 0x40d816ba,
0xaca5c697, 0xa864db20, 0xa527fdf9, 0xa1e6e04e, 0xbfa1b04b,
0xbb60adfc, 0xb6238b25, 0xb2e29692, 0x8aad2b2f, 0x8e6c3698,
0x832f1041, 0x87ee0df6, 0x99a95df3, 0x9d684044, 0x902b669d,
0x94ea7b2a, 0xe0b41de7, 0xe4750050, 0xe9362689, 0xedf73b3e,
0xf3b06b3b, 0xf771768c, 0xfa325055, 0xfef34de2, 0xc6bcf05f,
0xc27dede8, 0xcf3ecb31, 0xcbffd686, 0xd5b88683, 0xd1799b34,
0xdc3abded, 0xd8fba05a, 0x690ce0ee, 0x6dcdfd59, 0x608edb80,
0x644fc637, 0x7a089632, 0x7ec98b85, 0x738aad5c, 0x774bb0eb,
0x4f040d56, 0x4bc510e1, 0x46863638, 0x42472b8f, 0x5c007b8a,
0x58c1663d, 0x558240e4, 0x51435d53, 0x251d3b9e, 0x21dc2629,
0x2c9f00f0, 0x285e1d47, 0x36194d42, 0x32d850f5, 0x3f9b762c,
0x3b5a6b9b, 0x0315d626, 0x07d4cb91, 0x0a97ed48, 0x0e56f0ff,
0x1011a0fa, 0x14d0bd4d, 0x19939b94, 0x1d528623, 0xf12f560e,
0xf5ee4bb9, 0xf8ad6d60, 0xfc6c70d7, 0xe22b20d2, 0xe6ea3d65,
0xeba91bbc, 0xef68060b, 0xd727bbb6, 0xd3e6a601, 0xdea580d8,
0xda649d6f, 0xc423cd6a, 0xc0e2d0dd, 0xcda1f604, 0xc960ebb3,
0xbd3e8d7e, 0xb9ff90c9, 0xb4bcb610, 0xb07daba7, 0xae3afba2,
0xaafbe615, 0xa7b8c0cc, 0xa379dd7b, 0x9b3660c6, 0x9ff77d71,
0x92b45ba8, 0x9675461f, 0x8832161a, 0x8cf30bad, 0x81b02d74,
0x857130c3, 0x5d8a9099, 0x594b8d2e, 0x5408abf7, 0x50c9b640,
0x4e8ee645, 0x4a4ffbf2, 0x470cdd2b, 0x43cdc09c, 0x7b827d21,
0x7f436096, 0x7200464f, 0x76c15bf8, 0x68860bfd, 0x6c47164a,
0x61043093, 0x65c52d24, 0x119b4be9, 0x155a565e, 0x18197087,
0x1cd86d30, 0x029f3d35, 0x065e2082, 0x0b1d065b, 0x0fdc1bec,
0x3793a651, 0x3352bbe6, 0x3e119d3f, 0x3ad08088, 0x2497d08d,
0x2056cd3a, 0x2d15ebe3, 0x29d4f654, 0xc5a92679, 0xc1683bce,
0xcc2b1d17, 0xc8ea00a0, 0xd6ad50a5, 0xd26c4d12, 0xdf2f6bcb,
0xdbee767c, 0xe3a1cbc1, 0xe760d676, 0xea23f0af, 0xeee2ed18,
0xf0a5bd1d, 0xf464a0aa, 0xf9278673, 0xfde69bc4, 0x89b8fd09,
0x8d79e0be, 0x803ac667, 0x84fbdbd0, 0x9abc8bd5, 0x9e7d9662,
0x933eb0bb, 0x97ffad0c, 0xafb010b1, 0xab710d06, 0xa6322bdf,
0xa2f33668, 0xbcb4666d, 0xb8757bda, 0xb5365d03, 0xb1f740b4
};
typedef struct crc32ctx
{
uint32_t crc;
uint32_t length;
} CRC32Ctx;
#define COMPUTE(var, ch) (var) = (var) << 8 ^ crctab[(var) >> 24 ^ (ch)]
void crc32_stream_init( CRC32Ctx* ctx )
{
ctx->crc = 0;
ctx->length = 0;
}
void crc32_stream_compute_uint32( CRC32Ctx* ctx, uint32_t data )
{
COMPUTE( ctx->crc, data & 0xFF );
COMPUTE( ctx->crc, ( data >> 8 ) & 0xFF );
COMPUTE( ctx->crc, ( data >> 16 ) & 0xFF );
COMPUTE( ctx->crc, ( data >> 24 ) & 0xFF );
ctx->length += 4;
}
void crc32_stream_compute_uint8( CRC32Ctx* ctx, uint8_t data )
{
COMPUTE( ctx->crc, data );
ctx->length++;
}
void crc32_stream_finilize( CRC32Ctx* ctx )
{
uint32_t len = ctx->length;
for( ; len != 0; len >>= 8 )
{
COMPUTE( ctx->crc, len & 0xFF );
}
ctx->crc = ~ctx->crc;
}
/*** pseudo code ***/
CRC32Ctx crc;
crc32_stream_init(&crc);
while((just_received_buffer_len = received_anything()))
{
for(int i = 0; i < just_received_buffer_len; i++)
{
crc32_stream_compute_uint8(&crc, buf[i]); // assuming buf is uint8_t*
}
}
crc32_stream_finilize(&crc);
printf("%x", crc.crc); // ta daaa
CRC
adler32, available in the zlib headers, is advertised as being significantly faster than crc32, while being only slightly less accurate.
CRC32 is probably good enough, although there's a small chance you might get a collision, such that a file that has been modified might look like it hasn't been because the two versions generate the same checksum. To avoid this possibility I'd therefore suggest using MD5, which will easily be fast enough, and the chances of a collision occurring is reduced to the point where it's almost infinitessimal. As others have said, with lots of small files your real performance bottleneck is going to be I/O so the issue is dealing with that. If you post up a few more details somebody will probably suggest a way of sorting that out as well.
Your most important requirement is "to check if the content changed". If it most important that ANY change in the file be detected, MD-5, SHA-1 or even SHA-256 should be your choice. Given that you indicated that the checksum NOT be cryptographically good, I would recommend CRC-32 for three reasons. CRC-32 gives good hamming distances over an 8K file. CRC-32 will be at least an order of magnitude faster than MD-5 to calculate (your second requirement). Sometimes as important, CRC-32 only requires 32 bits to store the value to be compared. MD-5 requires 4 times the storage and SHA-1 requires 5 times the storage. BTW, any technique will be strengthened by prepending the length of the file when calculating the hash.
According to the Wiki page pointed to by Luke, MD5 is actually faster than CRC32! I have tried this myself by using Python 2.6 on Windows Vista, and got the same result. Here are some results: crc32: 162.481544276 MBps md5: 224.489791549 MBps crc32: 168.332996575 MBps md5: 226.089336532 MBps crc32: 155.851515828 MBps md5: 194.943289532 MBps I am thinking about the same question as well, and I'm tempted to use the Rsync's variation of Adler-32 for detecting file differences.
Just a postscript to the above; jpegs use lossy compression and the extent of the compression may depend upon the program used to create the jpeg, the colour pallette and/or bit-depth on the system, display gamma, graphics card and user-set compression levels/colour settings. Therefore, comparing jpegs built on different computers/platforms or using different software will be very difficult at the byte level.
This is 5 times faster than CCITT and makes exactly the same job:
Python:
def crc16_fast(data: bytearray, length):
crc = 0xCACA
for i in range(length):
crc ^= data[i]
return crc
C:
uint16_t crc16_fast(const uint16_t* data, size_t length)
{
uint16_t crc = 0xCACA;
for (size_t i = 0; i < length; i++)
crc ^= data[i];
return crc;
}