Books on parallel programming in OpenMP and MPI - parallel-processing

I've been programming with C and FORTRAN for the past 6 years, but I have only done serial programming until now. I would now like to switch to parallel program design, but I haven't really found a book that would help me make this jump. I have looked at Chandra's book on Parallel programming in OpenMP and Michael J. Quinn's book on programming in C with Open Mp and MPI, but I was wondering if there was a better alternative to these two. Thanks in advance!
Prashanth

I can recommend Using OpenMP by Chapman, Jost and Van de Pas. It's a really good introduction to OpenMP and includes plenty of details on using the language and getting best performance.
This tutorial might also be a useful introduction if you have never experienced parallel programming, it's a good entry to the general concepts and ideas.

Related

Writing a Compiler in C

I am very interested in compilers and how they work so I want to know if there are good sources or books to write a compiler from scratch. As I am a novice C programmer it would be perfect if the guide(s) is for C.
Thank you.
"Compiler Construction for Digital Computers" 1st Edition by David Gries
is my favorite, and it has good reviews
There's also the book Compiler Design in C by Allen Holub, which is available electronically. It's also a bit newer than Gries' book (1990 as opposed to 1971).

Can you program FPGAs in C-like languages? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
At university I programmed a FPGA in a C-like language. However, I also know that one usually programs FPGAs in Verilog or VHDL. Is this a designer choice? If so, what are the performance drawbacks?
I would ideally like to program the FPGA in a C-like language, rather than VHDL.
I was thinking of getting an Xilinx Virtex-5 if it makes any difference?
FPGA's are not processors. C is a language designed for processors.
Yes, there are C to FPGA compilers.
Are they a good idea? I'd say No. The design you're going to end up with is (from what I've seen) normally a state machine that has one state per line of code in the C. The state machine then moves through the states performing the algorithm. Either that or some other kind of Turing machine is put in place to execute the code.
This is not how somebody skilled in FPGA design would generally approach a problem. It's a slow, and potentially gate hungry way doing things.
In the same way that English is a better language to write a novel than Fortran, VHDL and Verilog are better languages to describe logic circuits than C.
If you're serious about using FPGAs, use a language that is designed to describe logic circuits. It might be a steep learning curve, but the results will be much better IMHO.
The short answer is "yes, certainly".
Here's an excellent survey of C compilers for FPGAs and FPGA-based systems.
C-to-hardware compiler (HLL synthesis)
Performance drawbacks and considerations are found in the system architecture and communication bandwidths rather than in using C vs. a hardware design language (HDL). The considerations in using C vs. an HDL lies in programming time and software maintenance issues, not so much in performance.
You can install a soft processor core inside the FPGA logic, and run your C code inside the virtual processor. Xilinx has Microblaze (licensed) and Picoblaze (free) cores. There are other soft cores you can implement as well (MIPS, x86, 8051, etc).
However, this is largely considered a "hack", as the cores are very slow compared to real cores. And I think that any C-to-FPGA conversion is ultimately going to start smelling like running a soft core, and not give you the efficiency you deserve for running on a FPGA. FPGAs are not Turing machines, they are a sack of logic gates. You can build a Turing machine out of the gates, but that is not why you bought the sack of gates.
Its sort of like buying a bag of Legos, and building a hammer and set of nails out of the bricks. It might work, but you are better off buying a hammer to pound nails, and better off building Castles, Space Ships and Fire Stations with the Legos.
I'd like to add something that I believe is the closest answer to the OP's question. If you're looking for a C-like language (which is not the same as C), you should definitely check out Synflow. The idea is to have a modern language that allows you to design faster without the learning curve of VHDL/Verilog and with no overhead. Also it's free and open source!
Disclosure: I'm co-founder of Synflow :-)
You should have a look at SystemC. The advantages of using a C based language is plentiful. Especially, on a system design perspective you can utilize that your other software (firmware and other low level stuff) is written in C. Hence, your software team can on a really early stage test against the rtl code.
In 2011, Xilinx bought the company AutoESL that had developed high level synthesis with SystemC. Xilinx has reused the name when the released its product "AutoESL". Especially with their new circuit Zynq, there a dual core ARM Cortex A9 embedded together with the FPGA logic, this will probably become a powerful tool for system development.
There are indeed some compilers that allow you to infer (solve using an incomplete description) hardware circuits using a high-level language like C. "C-to-gates" is in fact a popular buzzword. The image companies advertise is that programmers are able to write hardware if the language they use is one they have used to describe software. This is incredibly wrong for a number of reasons, chief among them being the fundamental differences between the execution model assumed by languages like C and hardware description languages.
An illustrative example: C assumes at its heart a large randomly accessible linear-addressed memory - an assumption that rarely holds for hardware. A C-to-gates compiler faces a challenging task of interperting the behavior of the program described, and designing a hardware circuit with the same behavior.
While C-like languages are a great productivity tool in limited use cases, these compilers certainly don't allow you to suddenly know how to design hardware if you are familiar with C.
Hope this helps,
I guess you used Handel C. Its a subset of C. From what I know the result is not very optimized. Verilog and VHDL allows for more optimization. I am saying this based on the my experience with Handel C a few years back
You might want to take a look at C-to-hardware technology, where you can write C code and it will get compiled/translated to VHDL or Verilog. This post lists a few compilers. Haven't used it myself so I don't have any experience with it. Hope this helps!
Many designers write VHDL/Verilog instead of a high-level language, for the same reasons that many programmers used to (and still do in some cases) write assembly instead of Java: you can tune resource usage and performance at a low level. Both VHDL and Verilog are languages designed for designing hardware. C is not. Given enough time, you could always write a program in VHDL/Verilog that will outperform a high-level language program. What an HLL gives you is 1) faster development, 2) ease of maintenance, and 3) possibly greater portability.
There have been many efforts to compile existing high-level programming languages (C is one) to FPGA targets. Most of them do, in fact, generate optimized code. Impulse C, for example, is a subset of C with some add-on libraries that support process-level parallelism, plus a compiler that optimizes the C input for instruction-level parallelism, too. It pipelines loops, maps certain operations to high-performance hardware primitives it knows the underlying FPGA family provides, etc. (Full disclosure: I helped build the Impulse C toolchain.)
The C-to-hardware environments list Carlito and David Pointer link to is pretty exhaustive. Xilinx Virtex-5 is supported by many of them, and if you're using any recent FPGA family from a major vendor, choice of hardware shouldn't be a problem. Some of the HLL environments support built-in (or softcore) embedded CPUs better than others.

What are some good resources for learning algorithm optimization?

I've been tinkering around with code (Basic, Python, C++, PHP, JavaScript) on and off for almost two decades, but have only recently begun to get more "serious" about it (using Java). I can write code to do what I want, but now I want to learn to optimize my programs to run faster (looping through an array for every element in an array can get slow very quickly, etc). What I don't want is to be popping onto this site every 5 minutes for every little question I have. I want to learn to answer my own questions.
That said, what are some good resources for learning algorithm analysis and optimization?
I have a copy of Data Structures and Algorithms in Java (3rd edition) but I feel it's written to mostly be incorporated into a college curriculum and isn't very easy to use sans-professor. The book also has a tendency to over-use abbreviations, making it hard to flip to a particular chapter without having to skim back through the book to understand what each abbreviation stands for.
I do have some knowledge of Calculus, but it's extremely rusty, so I would prefer resources that give more explanation and fewer formulas.
Thank you in advance for all the help you can give!
I can't recommend enough Michael Abrash's "The Zen of Code Optimization". It's easyto read and full of insights. The parts that focus on pre-pentium x86 are dated, but it's real value is the focus on how to think about making code faster.
I believe it's out of print, but you may find a used copy online.
You might start with Skiena's Algorithm Design Manual. The same author also has a book on puzzle-solving called Programming Challenges, which gives you a more entertaining way to get practice with algorithms than slogging through a textbook.

What is a good intermediate introduction to Scheme?

I am trying to find a resource where I can practice my programming skills in scheme.
I have a class coming up that uses scheme exclusively. While I have done moderate amounts of programming in C++ and Java, and grasp the basic ideas about good/bad programming practices and different programming paradigms (as much as a second-year college student could be expected to), Scheme looks like hieroglyphics to me!
Any help would be appreciated.
If you haven't looked at Structure and Interpretation of Computer Programs yet, that's the obvious (and free!) place to start.
I really like The Little Schemer. The format is very different, but it all builds up from basics.
If you're looking for an intermediate introduction, SICP will not be a good choice. You could try HtDP, but that not too much of an intro to Scheme as it is an intro to programming in general.
The Scheme Programming Language, 4th Edition (Covers the language as described in R6RS.).
If you are using a Scheme that conforms only with R5RS, use the 3rd edition of the book.
Structure and Interpretation of Computer Programs is highly recommended.
The Guide to Racket http://docs.racket-lang.org/guide/index.html is a good place to find the information you need. It has a very good navigation system and many simple examples. You can use it with SICP.

Would Lisp be extremely difficult for a new(ish) programmer to learn? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
I've got a little experience with Python (enough to where I can do if/else/elif and some random number generation), but I've always had a weird fascination with the Lisp languages. I downloaded some scheme source code to look at the syntax but it was pretty much gibberish to me.
For a programmer with only a little programming experience like myself, given some good books, websites, and some time, would it be particularly difficult to learn either Common Lisp or Scheme?
Which of the two would be easier? How do they compare with Python and C, as far as ease of learning?
Thanks
Get SICP (Structure and Interpretation of Computer Programs). It was the entry level CS class textbook at MIT up until very recently. It is entirely based on Scheme. If you enjoy CS at all, you will love this book. It's fantastic. You'll walk away a much better programmer too after reading it (and ideally doing most of the exercises). It's an ideal beginner's book, it starts at the very beginning and eases you up to being quite competent at Scheme. However, it is a college textbook, so unless you have a bit of a knack for programming, it may be a tad on the tough side.
Given some good books, websites, and some time, would it be particularly difficult to learn either Common Lisp or Scheme?
No.
Which of the two would be easier?
Scheme, because
It has better books for beginners.
It has a superb interactive programming environment designed for people who are just learning the language.
It is smaller.
How do they compare with Python and C, as far as ease of learning?
I think it's harder to learn C than to learn either Python or Scheme, because to learn C you really have to grok the machine model, pointers, and dynamic memory.
Between Scheme and Python, it is really hard to predict how any individual learner will react. I really like Python's significant whitespace, and I find Scheme's parentheses annoying and distracting. Lots of people really like Scheme's parentheses, and they find Python's significant whitespace annoying and distracting. (There are important semantic differences, but it is hard to escape the tyranny of syntax.)
What books should I use? (question you should have asked and didn't)
Don't use Structure and Interpretation of Computer Programs (SICP). That book's point of view is "let's show off how smart we are by coding all of computer science in Scheme." The book is a tremendous intellectual tour de force, but it is not at all suitable for beginners. When MIT used it in 6.001, it was as a "weedout" course because 30–40% of all MIT students wanted to major in EECS, and they were trying to turn people away. SCIP is a terrific for a senior CS student or a programmer with 5 years of experience. Don't try to learn from it.
Some good books to learn from:
How to Design Programs by Felleisen et al has been carefully crafted and honed through years of experience. It uses Scheme and teaches you exactly what it claims in the title. Highly recommended.
Simply Scheme by Harvey and Wright is an introductory CS book by people who felt that SCIP needed a "prequel." I enjoyed reading it but I haven't taught from it.
If you can stand cuteness, The Little Schemer by Felleisen and Friedman has a very unusual dialectical style, with tons of examples, that you might like.
Get
"The Little Schemer"
and
"The Advanced Schemer"
These books are INVALUABLE as programming concept aides. I'd personally do them before SICP just because SICP is a bit fast if you haven't programmed much before... and even if you have they're great books!
I personally think of Scheme as a very good pedagogical language. It can be used for just about anything I suppose, but it's very good for teaching programming constructs without getting bogged down in syntactical issues. The closest thing to pure semantics you're going to get.
I would recommend going with Common Lisp.
It has excellent implementations that provide a variety of different capabilities. The best ones are still the commercial implementations (though they have no-cost versions for learning Lisp) like Allegro CL from Franz and LispWorks. They offer extensive libraries and include GUI-based development environments. But the open source and free implementations are excellent, too: SBCL, CCL, ECL, CLISP, ...
There are a bunch of excellent books for Lisp, like Peter Seibel's Practical Common Lisp.
Learning Common Lisp is not that difficult, but getting deeper experience get take some time, because the language and its eco-system is surprisingly rich.
Learning Scheme is useful, too. Also learning SICP can help. But there is no immediate need, since the Lisp literature talks about many of the same issues. Unfortunately Scheme lost some of its appeal in recent years (I find the latest R6RS to be totally disappointing) and I find also implementations like 'Racket' to be moving in the wrong direction. Instead one might want to learn some of the better ML dialects and implementations (say, OCAML). Though it might be interesting to check out various Scheme implementations, there are some with interesting capabilities.
The core ideas you should learn from Lisp are these:
multi-paradigm programming
programmable programming language
symbolic computation
interactive and exploratory development
Learn how Lisp supports these ideas and why they are important.
Lisp isn't hard to learn. It can be taught poorly, and it does have some "high level" concepts, especially if you're coming from the imperative "classic" programming world.
But today, may of the modern languages have aspects very similar to what Lisp offers, so there is likely not much "new" in Lisp compared to other languages, especially Python.
My "hurdle" with Lisp (and Scheme) was simply "lambda". Essentially, my fundamental problem was I had nothing to relate to what a "lambda" was, why it was called "lambda", etc.
If it was named "function" or "routine" or something, it would have been trivial. But as is, the word "lambda" was meaningless in every way. They could have called it "mork", "bleem", or "fizbin" and it would have been as useful.
Once I grokked "lambda", the rest fell in to place.
If you're interested a A Lisp (i.e. a language which is essentially from the Lisp family), then Scheme is a good choice. But, Scheme is NOT Common Lisp (which is what "Lisp" typically means today), they are really different languages.
If you want to learn Common Lisp, I would start with Common Lisp and skip Scheme.
You can look to Practical Common Lisp to get you in to writing Lisp without "having to learn it" so to speak.
You can actually find a full video course of Structure and Interpretation of Computer Programs on Google Video. Start here: http://video.google.com/videoplay?docid=933300011953693438#
The videos were made in the 80's by Hewlett-Packard, and taught by Sussman and Abelson (the original authors). The class is full of HP HW/mechanical engineers who are being retrained as SW engineers. HP released them to the public domain.
Anyone could watch these videos and be a Lisp programmer by the end. Amazing, excellent instructors.
Scheme is (intentionally) more compact than Common Lisp, and you'll find that you can learn the language very quickly. Now, mastering the (any) language and computer science concepts is another story, and #Matt Greer's book suggestion would be a wonderful place to start.
I'd say a Lisp will be easier to learn as a new programmer than if you wait. That's because the way most languages encourage you to think about programming is a handicap to learning Lisp; being experienced with a more conventional language will probably help in some ways, but not with the qualities that make Lisp unique.
Scheme --- the Lisp with which I'm most familiar --- is probably about as easy as Python to learn, especially if you choose PLT Scheme (now renamed Racket), which offers you a lot of helpful extensions. Common Lisp is probably about the same difficulty, though I find it less instinctive for reasons which are probably not universal. I have found C much more difficult than either, for the reasons Norman Ramsey suggests.
What one I'd suggest learning depends very much on what you want to do with it. If you want to learn a crystal-clear language with no frills, whose core can be learned quickly, learn a Scheme that implements the R5RS standard, such as Scheme48. If you want a more capable Scheme, either immediately or later on, learn PLT Scheme/Racket. If you want a big, friendly, diffuse language of immense power but not as visibly internally consistent as Scheme, choose Common Lisp; Steel Bank Common Lisp is a good implementation.
As to which is easier, Lisp or Scheme, being "dialects" of the Lisp family, at the beginner level they are pretty much the same thing and interchangeable. I learned Common Lisp from Peter Siebel's "Practical Common Lisp", then picked up SICP and worked through all the exercises through the first two chapters using Common Lisp rather than Scheme. There are just a few different keywords and things like the use of funcall in CL where just a pair of parentheses would suffice in Scheme.
Both Siebel's book and SICP are available for free in their entirety on the web. Download them and get hackin'!
Last time I checked the lectures from the intro classes for CS61A at Berkeley which used SICP were available from iTunes, which would give you a full lecture series to help you learning the language. For free! I TA'd for Brian Harvey back in the late 90's and students really liked his lectures.
You can currently find them here: https://archive.org/details/ucberkeley-webcast-PL6879A8466C44A5D5
Consider (the poorly titled) "Programming Languages, An Interpreter-Based Approach" by Samuel N. Kamin. You'll learn how to program in 7 different languages (including Lisp, Scheme, SASL, CLU), plus how to implement interpreters for each. You should come away with a good understanding of how the languages differ, why they're designed the way they are, what are the tradeoffs in using them, etc.
If you are fascinated by the language, i think you will find it easy. But i must say it's not for everyone. I tried several times learning Scheme and Common Lisp and lost interest. When i see an "if" in a programming language, i start looking for "then", "else", an open-brace, a colon, etc. but Lisp has none of that. I only see a soup of parenthesis.
Besides, some concepts are easier in other languages. I agree with the guys who don't like the word "lambda". I really got it when i learned Lua (Javascript is very similar). In Lua and Javascript, when you need an anonymous function, you type (guess what) "function(etc. etc.)". When you want to create a closure by returning a function, you write (guess what) "return function(etc., etc.)". In Lisp, you write "lambda" and use the implicit return at the end of the function. Not very beginner-friendly.
I aslo agree with Norman Ramsey: Don't use Structure and Interpretation of Computer Programs (SICP).
This book is weird. It starts really easy, on page 1 you learn some expressions like (+ 137 349). Easy as pie. Then, on page 2 you learn functions. On page 3 you learn recursion. On page 4 you learn higher-order functions with lambdas (!!!). On page 5 onwards, you say WTF and can't make sense of anything. The book has only 5 chapters each with a thousand pages. It introduces assignment in chapter 3. Maybe your brain is functional, but my brain is certainly imperative and i need to see what the machine is doing: assigning values, copying memory etc. Teaching higher-order functions before assignment is just backwards in my imperative brain. In the last chapter the book talks about compilers, interpreters and garbage-collection. Is this the same book that started teaching expressions like (+ 137 349)? Now i can understand the last chapters of this book, but only after learning half a dozen programming languages of different paradigms and reading a lot about programming languages and their implementation. I don't know how anyone could follow the pace this book imposes on the reader.
Relatively, it's a very difficult language to learn if you are as stubborn as I am :)
Python is an OOP language, and while FP is possible, it's still OOP. Lisp is a FOP language like Haskell. Like Python, it can do OP, but it's still FOP.
Terms used:
OOP = Object Oriented Programming
OP = Object Programming
FOP = Functional Oriented Programming
FP = Functional Programming

Resources