i have a question concerning computer programming. Let's say i have only one computer with no OS running. And would like to start to "develop" an OS. basically what i have is a blank sheet an a pen to do so. an a couple of electronic devices. how do i put my instruction into that computer?
because today we use interpreter of compiler that "turn" programming language into what they call "machine code". But my question could be how to generate machine code from nowhere.
Thank you for your replies, a link to learn how to do that will be must welcome.
The first computers where programmed making the "machine code" directly. Just punching one's an zeros into cards (well, in fact they punched octal digits).
This was done that way until somebody thought it would be a nice idea to have an assembler which translated the machine code instructions into that ones and zeros.
After that, another guy thought that it can be very nice idea to have a programming language, who will translate "top level" instructions to machine code.
And after that, or probably at the same time, some "internal procedures" where created to ease the programming: to open a file, to close a file the only thing you have to do is to call an internal subroutine in the machine instead of programming all the open file and close file subroutines by yourself: the seed for the operating systems was planted.
The cross compiling issue that is commented here is the way to create an operating system for a new computer nowadays: you use a working computer as a "lever" to create an operating system for a new computer.
it depends on how far back you want to go. the earliest ones "programming" was moving wires from one essentially analog alu to another.
The woman/women programming at this point were called computers and used use pencil and paper.
later you use a pencil and paper and the datasheet/documentation for the instruction set. assemble by hand basically, there are no compilers or even the concept of a programming language at this point, this has to evolve still. you wrote down the ones and zeros in whatever form you preferred (binary or octal).
one way to enter code at this point is with switches. certainly computers predated it but look for a picture of the front panel of a pdp8 or the altair, etc. you set the switches for the data value and address, and you manually strobe a write. you load the bootstrap in this way and/or the whole program. set the start address and switch to run mode.
over time they developed card and tape readers for which you loaded the bootstrap in by hand (switches) then you could use a reader to load larger programs easier. cards could be punched on a typewriter type thing, literally a keyboard but instead of striking through a ribbon onto paper, it cut slots in a card.
oses and programming languages started to evolve at this point. until you bootstrapped your compiler you had to write the first compiler for a new language in some other language (no different than today). so the first assembler had to be in machine code, then from assembler you could create some other language and so on.
If you wanted to repeat something like this today you would have to build a computer with some sort of manual input. you could certainly but you would have to design it that way, like then you need the debouncing out but you could for example have a processor with an external flash, be it parallel or serial, mux the lines to the switches (a switch controls the mux) and either address/data/write your program, or for fun could use a spi flash and serially load the program into the flash. much better to just use one of the pdp or altair, etc online simulators to get a feel for the experience.
there is no magic here, there is no chicken and egg problem at all. humans had to do it by hand before the computer could do it. a smaller/simpler program had to generate more complicated programs, and so on. this long, slow, evolution is well documented all over the internet and in books in libraries everywhere.
Computers are based on a physical processor which was designed to accept instructions (eg. in assembly code) that only allowed primitive instructions like shift, move, copy, add. This processor decided how it spoke (eg. how big were the words (8-bit) and and other specs (speed/standards etc). Using some type of storage, we could store the instructions (punch cards, disk) and execute huge streams of these instructions.
If instructions were repeated over and over, you could move to an address and execute what was at that location and create loops and other constructs (branches, context switches, recursion).
Since you would have peripherals, you would have some kind of way to interface with it (draw, print dots), and you could create routines to build up on that to build letters, fonts, boxes, lines. Then you could run a subroutine to print the letter 'a' on screen..
An OS is basically a high-level implementation of all those lower level instructions. It is really a collection of all the instructions to interface with different areas (i/o, computations etc). Unix is a perfect example of different folks working on different areas and plugging them all into a single OS.
Related
OK, I have the problem, I do not know exactly the correct terms in order to find what I am looking for on google. So I hope someone here can help me out.
When developing real time programs on embedded devices you might have to iterate a few hundred or thousand times until you get the desired result. When using e.g. ARM devices you wear out the internal flash quite quickly. So typically you develop your programs to reside in the RAM of the device and all is ok. This is done using GCC's functionality to split the code in various sections.
Unfortunately, the RAM of most devices is much smaller than the flash. So at one point in time, your program gets too big to fit in RAM with all variables etc. (You choose the size of the device such that one assumes it will fit the whole code in flash later.)
Classical shared objects do not work as there is nothing like a dynamical linker in my environment. There is no OS or such.
My idea was the following: For the controller it is no problem to execute code from both RAM and flash. When compiling with the correct attributes for the functions this is also no big problem for the compiler to put part of the program in RAM and part in flash.
When I have some functionality running successfully I create a library and put this in the flash. The main development is done in the 'volatile' part of the development in RAM. So the flash gets preserved.
The problem here is: I need to make sure, that the library always gets linked to the exact same location as long as I do not reflash. So a single function must always be on the same address in flash for each compile cycle. When something in the flash is missing it must be placed in RAM or a lining error must be thrown.
I thought about putting together a real library and linking against that. Here I am a bit lost. I need to tell GCC/LD to link against a prelinked file (and create such a prelinked file).
It should be possible to put all the library objects together and link this together in the flash. Then the addresses could be extracted and the main program (for use in RAM) can link against it. But: How to do these steps?
In the internet there is the term prelink as well as a matching program for linux. This is intended to speed up the loading times. I do not know if this program might help me out as a side effect. I doubt it but I do not understand the internals of its work.
Do you have a good idea how to reach the goal?
You are solving a non-problem. Embedded flash usually has a MINIMUM write cycle of 10,000. So even if you flash it 20 times a day, it will last a year and half. An St-Nucleo is $13. So that's less than 3 pennies a day :-). The TYPICAL write cycle is even longer, at about 100,000. It will be a long time before you wear them out.
Now if you are using them for dynamic storage, that might be a concern, depending on the usage patterns.
But to answer your questions, you can build your code into a library .a file easily enough. However, GCC does not guarantee that it links the object code in any order, as it depends on optimization level. Furthermore, only functions that are referenced in a library file is pulled in, so if your function calls change, it may pull in more or less library functions.
I have some knowledge on OS (really little.)
I would like to know a lot about specifically the Windows OS (e.g. win 7)
I know, it's the most dominant OS out there, and there is an enormous amount of work I`ll have to do.
Where do I start? what are beginner/intermediate books/articles/websites that I should read?
The first thing I wonder about is that the compiler turns my C programs to binary code, however when I open the (exe) result files, I find something other than 0 and 1.
I can't point you in a direction as far as books go, but I can clarify this:
The first thing I wonder about is that the compiler turns my C programs to binary code, however when I open the (exe) result files, I find something other than 0 and 1.
Your programs are in fact compiled to binary. Everything on your computer is stored in binary.
The reason you do not see ones and zeroes is because of the makeup of character encodings. It takes eight bits, which can have the value 0 or 1, to store one byte. A lot of programs and character encodings represent one byte as one character (with the caveat of non-ASCII unicode characters, but that's not terribly important in this discussion).
So what's going on is that the program you are using to open the file is interpreting sequences of eight bits and turning those eight bits into one character. So each character you see when you open the file is, in fact, eight ones and zeros. The most basic mapping between bytes and characters is ASCII. The character "A", for example, is represented in binary as 01000001. so when the program you use to open the file sees that bit sequence, it will display "A" in its place.
A nice book to read if you are interested in the Microsoft Windows operating system is The Old New Thing by Microsoft legend Raymond Chen. It is very easy reading if you are a Win32 programmer, and even if you are not (even if you are not a programmer at all!) many of the chapters are still readily accessible.
Otherwise, to understand the Microsoft Windows OS, you need to understand the Windows API. You learn this by writing programs for the Windows (native) platform, and the official documentation, which is very good, is at MSDN.
There are a series of books titled "Windows Internals" that could probably keep you busy for the better part of a couple years. Also, Microsoft has been known to release source code to universities to study...
well, if you study the win32 api you will learn a lot about high-level os
(petzold is the king, and it's not about win7 just win32....)
If you want to study about low level, study the processor assembler language.
There are a ton of resources out there for learning operating systems in general, many of which don't really focus on Windows because, as John pointed out, it's very closed and not very useful academically. You may want to look into something like Minix, which is very useful academically. It's small, light, and made pretty much for the sole purpose of education.
From there you can branch out into other OSes (even Windows, as far as not being able to look under the hood can take you) armed with a greater knowledge of what an OS is and does, as well as more knowledge of the inner workings of the computer itself. (For example, opening executable code in I assume a text editor, such as Notepad, to try to see the 1s and 0s, which as cdhowie pointed out eloquently is not doing what you think it's doing.)
I would personally look into the ReactOS project - a working windows clone.
The code con give some ideas of how windows is implemented...
Here is the site:
www. reactos. org
Isn't every language compiled into low-level computer language?
If so, shouldn't all languages have the same performance?
Just wondering...
As pointed out by others, not every language is translated into machine language; some are translated into some form (bytecode, reverse Polish, AST) that is interpreted.
But even among languages that are translated to machine code,
Some translators are better than others
Some language features are easier to translate to high-performance code than others
An example of a translator that is better than some others is the GCC C compiler. It has had many years' work invested in producing good code, and its translations outperform those of the simpler compilers lcc and tcc, for example.
An example of a feature that is hard to translate to high-performance code is C's ability to do pointer arithmetic and to dereference pointers: when a program stores through a pointer, it is very difficult for the compiler to know what memory locations are affected. Similarly, when an unknown function is called, the compiler must make very pessimistic assumptions about what might happen to the contents of objects allocated on the heap. In a language like Java, the compiler can do a better job translating because the type system enforces greater separation between pointers of different types. In a language like ML or Haskell, the compiler can do better still, because in these languages, most data allocated in memory cannot be changed by a function call. But of course object-oriented languages and functional languages present their own translation challenges.
Finally, translation of a Turing-complete language is itself a hard problem: in general, finding the best translation of a program is an NP-hard problem, which means that the only solutions known potentially take time exponential in the size of the program. This would be unacceptable in a compiler (can't wait forever to compile a mere few thousand lines), and so compilers use heuristics. There is always room for improvement in these heuristics.
It is easier and more efficient to map some languages into machine language than others. There is no easy analogy that I can think of for this. The closest I can come to is translating Italian to Spanish vs. translating a Khoisan language into Hawaiian.
Another analogy is saying "Well, the laws of physics are what govern how every animal moves, so why do some animals move so much faster than others? Shouldn't they all just move at the same speed?".
No, some languages are simply interpreted. They never actually get turned into machine code. So those languages will generally run slower than low-level languages like C.
Even for the languages which are compiled into machine code, sometimes what comes out of the compiler is not the most efficient possible way to write that given program. So it's often possible to write programs in, say, assembly language that run faster than their C equivalents, and C programs that run faster than their JIT-compiled Java equivalents, etc. (Modern compilers are pretty good, though, so that's not so much of an issue these days)
Yes, all programs get eventually translated into machine code. BUT:
Some programs get translated during compilation, while others are translated on-the-fly by an interpreter (e.g. Perl) or a virtual machine (e.g. original Java)
Obviously, the latter is MUCH slower as you spend time on translation during running.
Different languages can be translated into DIFFERENT machine code. Even when the same programming task is done. So that machine code might be faster or slower depending on the language.
You should understand the difference between compiling (which is translating) and interpreting (which is simulating). You should also understand the concept of a universal basis for computation.
A language or instruction set is universal if it can be used to write an interpreter (or simulator) for any other language or instruction set. Most computers are electronic, but they can be made in many other ways, such as by fluidics, or mechanical parts, or even by people following directions. A good teaching exercise is to write a small program in BASIC and then have a classroom of students "execute" the program by following its steps. Since BASIC is universal (to a first approximation) you can use it to write a program that simulates the instruction set for any other computer.
So you could take a program in your favorite language, compile (translate) it into machine language for your favorite machine, have an interpreter for that machine written in BASIC, and then (in principle) have a class full of students "execute" it. In this way, it is first being reduced to an instruction set for a "fast" machine, and then being executed by a very very very slow "computer". It will still get the same answer, only about a trillion times slower.
Point being, the concept of universality makes all computers equivalent to each other, even though some are very fast and others are very slow.
No, some languages are run by a 'software interpreter' as byte code.
Also, it depends on what the language does in the background as well, so 2 identically functioning programs in different languages may have different mechanics behind the scenes and hence be actually running different instructions resulting in differing performance.
In my quest for getting some basics down before I start going into programming I am looking for essential knowledge about how the computer works down at the core level.
I have a theory that actually understanding what for instance a stackoverflow let alone a stack is, instead of my sporadic knowledge about computer systems, will help me longer term.
Is there any books or sites that take you through how processors are structured and give a holistic overview and that somehow relates to good to know about digital logic?
Am i making sense?
Yes, you should read some topics of
John L. Hennessy & David A. Patterson, "Computer Architecture: A quantitative Approach"
It has microprocessors' history and theory , (starting with RISC archs - MIPS), pipelining, memory, storage, etc.
David Patterson is a Professor of Computer of Computer Science on EECS Department - U. Berkeley. http://www.eecs.berkeley.edu/~pattrsn/
Hope it helps, here's the link
Tanenbaum's Structured Computer Organization is a good book about how computers work. You might find it hard to get through the book, but that's mostly due to the subject, not the author.
However, I'm not sure I would recommend taking this approach. Understanding how the computer works can certainly be useful, but if you don't really have any programming knowledge, you can't really put your knowledge to good use - and you probably don't need that knowledge yet anyway. You would be better off learning about topics like object-oriented programming and data structures to learn about program design, because unless you're looking at doing embedded programming on very limited systems, you'll find those skills far more useful than knowledge of a computer's inner workings.
In my opinion, 20 years ago it was possible to understand the whole spectrum from BASIC all the way through operating system, hardware, down to the transistor or even quantum level. I don't know that it's possible for one person to understand that whole spectrum with today's technology. (Years ago, everyone serviced their own car. Today it's too hard.)
Some of the "layers" that you might be interested in:
http://en.wikipedia.org/wiki/Boolean_logic (this will be helpful for programming)
http://en.wikipedia.org/wiki/Flip-flop_%28electronics%29
http://en.wikipedia.org/wiki/Finite-state_machine
http://en.wikipedia.org/wiki/Static_random_access_memory
http://en.wikipedia.org/wiki/Bus_%28computing%29
http://en.wikipedia.org/wiki/Microprocessor
http://en.wikipedia.org/wiki/Computer_architecture
It's pretty simple really - the cpu loads instructions and executes them, most of those instructions revolve around loading values into registers or memory locations, and then manipulating those values. Certain memory ranges are set aside for communicating with the peripherals that are attached to the machine, such as the screen or hard drive.
Back in the days of Apple ][ and Commodore 64 you could put a value directly in to a memory location and that would directly change a pixel on the screen - those days are long gone, it is abstracted away from you (the programmer) by several layers of code, such as drivers and the operating system.
You can learn about this sort of stuff, or assembly language (which i am a huge fan of), or AND/NAND gates at the hardware level, but knowing this sort of stuff is not going to help you code up a web application in ASP.NET MVC, or write a quick and dirty Python or Powershell script.
There are lots of resources out there sprinkled around the net that will give you insight into how the CPU and the rest of the hardware works, but if you want to get down and dirty i honestly think you should buy one of those older machines off eBay or somewhere, and learn its particular flavour of assembly language (i understand there are also a lot of programmable PIC controllers out there that might also be good to learn on). Picking up an older machine is going to eliminate the software abstractions and make things way easier to learn. You learn way better when you get instant gratification, like making sprites move around a screen or generating sounds by directly toggling the speaker (or using a PIC controller to control a small robot). With those older machines, the schematics for an Apple ][ motherboard fit on to a roughly A2 size sheet of paper that was folded into the back of one of the Apple manuals - i would hate to imagine what they look like these days.
While I agree with the previous answers insofar as it is incredibly difficult to understand the entire process, we can at least break it down into categories, from lowest (closest to electrons) to highest (closest to what you actually see).
Lowest
Solid State Device Physics (How transistors work physically)
Circuit Theory (How transistors are combined to create logic gates)
Digital Logic (How logic gates are put together to create digital functions or digital structures i.e. multiplexers, full adders, etc.)
Hardware Organization (How the data path is laid out in the CPU, the components of a Von Neuman machine -> memory, processor, Arithmetic Logic Unit, fetch/decode/execute)
Microinstructions (Bit level programming)
Assembly (Programming with words, but directly specifying registers and takes forever to program even simple things)
Interpreted/Compiled Languages (Programming languages that get compiled or interpreted to assembly; the operating system may be in one of these)
Operating System (Process scheduling, hardware interfaces, abstracts lower levels)
Higher level languages (these kind of appear twice; it depends on the language. Java is done at a very high level, but C goes straight to assembly, and the C compiler is probably written in C)
User Interfaces/Applications/Gui (Last step, making it look pretty)
You can find out a lot about each of these. I'm only somewhat expert in the digital logic side of things. If you want a thorough tutorial on digital logic from the ground up, go to the electrical engineering menu of my website:
affablyevil.wordpress.com
I'm teaching the class, and adding online lessons as I go.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
At my previous employer we used a third party component which basically was just a DLL and a header file. That particular module handled printing in Win32. However, the company that made the component went bankcrupt so I couldn't report a bug I'd found.
So I decided to fix the bug myself and launched the debugger. I was surprised to find anti-debugging code almost everywhere, the usual IsDebuggerPresent, but the thing that caught my attention was this:
; some twiddling with xor
; and data, result in eax
jmp eax
mov eax, 0x310fac09
; rest of code here
At the first glance I just stepped over the routine which was called twice, then things just went bananas. After a while I realized that the bit twiddling result was always the same, i.e. the jmp eax always jumped right into the mov eax, 0x310fac09 instruction.
I dissected the bytes and there it was, 0f31, the rdtsc instruction which was used to measure the time spent between some calls in the DLL.
So my question to SO is: What is your favourite anti-debugging trick?
My favorite trick is to write a simple instruction emulator for an obscure microprocessor.
The copy protection and some of the core functionality will then compiled for the microprocessor (GCC is a great help here) and linked into the program as a binary blob.
The idea behind this is, that the copy protection does not exist in ordinary x86 code and as such cannot be disassembled. You cannot remove the entire emulator either because this would remove core functionality from the program.
The only chance to hack the program is to reverse engineer what the microprocessor emulator does.
I've used MIPS32 for emulation because it was so easy to emulate (it took just 500 lines of simple C-code). To make things even more obscure I didn't used the raw MIPS32 opcodes. Instead each opcode was xor'ed with it's own address.
The binary of the copy protection looked like garbage-data.
Highly recommended! It took more than 6 month before a crack came out (it was for a game-project).
I've been a member of many RCE communities and have had my fair share of hacking & cracking. From my time I've realized that such flimsy tricks are usually volatile and rather futile. Most of the generic anti-debugging tricks are OS specific and not 'portable' at all.
In the aforementioned example, you're presumably using inline assembly and a naked function __declspec, both which are not supported by MSVC when compiling on the x64 architecture. There are of course still ways to implement the aforementioned trick but anybody who has been reversing for long enough will be able to spot and defeat that trick in a matter of minutes.
So generally I'd suggest against using anti-debugging tricks outside of utilizing the IsDebuggerPresent API for detection. Instead, I'd suggest you code a stub and/or a virtual machine. I coded my own virtual machine and have been improving on it for many years now and I can honestly say that it has been by far the best decision I've made in regards to protecting my code so far.
Spin off a child process that attaches to parent as a debugger & modifies key variables. Bonus points for keeping the child process resident and using the debugger memory operations as a kind of IPC for certain key operations.
On my system, you can't attach two debuggers to the same process.
Nice thing about this one is unless they try to tamper w/ things nothing breaks.
Reference uninitialized memory! (And other black magic/vodoo...)
This is a very cool read:
http://spareclockcycles.org/2012/02/14/stack-necromancy-defeating-debuggers-by-raising-the-dead/
The most modern obfuscation method seems to be the virtual machine.
You basically take some part of your object code, and convert it to your own bytecode format. Then you add a small virtual machine to run this code. Only way to properly debug this code will be to code an emulator or disassembler for your VM's instruction format. Of course you need to think of performance too. Too much bytecode will make your program run slower than native code.
Most old tricks are useless now:
Isdebuggerpresent : very lame and easy to patch
Other debugger/breakpoint detections
Ring0 stuff : users don't like to install drivers, you might actually break something on their system etc.
Other trivial stuff that everybody knows, or that makes your software unstable. remember that even if a crack makes your program unstable but it still works, this unstability will be blamed on you.
If you really want to code the VM solution yourself (there are good programs for sale), don't use just one instruction format. Make it polymorphic, so that you can have different parts of the code have different format. This way all your code can't be broken by writing just one emulator/disassembler. For example MIPS solution some people offered seems to be easily broken because MIPS instruction format is well documented and analysis tools like IDA can already disassemble the code.
List of instruction formats supported by IDA pro disassembler
I would prefer that people write software that is solid, reliable and does what it is advertised to do. That they also sell it for a reasonable price with a reasonable license.
I know that I have wasted way too much time dealing with vendors that have complicated licensing schemes that only cause problems for the customers and the vendors. It is always my recommendation to avoid those vendors. Working at a nuclear power plant we are forced to use certain vendors products and thus are forced to have to deal with their licensing schemes. I wish there was a way to get back the time that I have personally wasted dealing with their failed attempts to give us a working licensed product. It seems like a small thing to ask, but yet it seems to be a difficult thing for people that get too tricky for their own good.
I second the virtual machine suggestion. I implemented a MIPS I simulator that (now) can execute binaries generated with mipsel-elf-gcc. Add to that code/data encryption capabilities (AES or with any other algorithm of your choice), the ability of self-simulation (so you can have nested simulators) and you have a pretty good code obfuscator.
The nice feature of choosing MIPS I is that 1) it's easy to implement, 2) I can write code in C, debug it on my desktop and just cross-compile it for MIPS when it's done. No need to debug custom opcodes or manually write code for a custom VM..
My personal favourite was on the Amiga, where there is a coprocessor (the Blitter) doing large data transfers independent from the processor; this chip would be instructed to clear all memory, and reset from a timer IRQ.
When you attached an Action Replay cartridge, stopping the CPU would mean that the Blitter would continue clearing the memory.
Calculated jumps in the middle of a legitimate looking but really hiding an actual instruction instructions are my favorite. They are pretty easy to detect for humans anyway, but automated tools often mess it up.
Also replacing a return address on the stack makes a good time waster.
Using nop to remove assembly via the debugger is a useful trick. Of course, putting the code back is a lot harder!!!