It seems to me that the most invaluable thing about a static/strongly-typed programming language is that it helps refactoring: if/when you change any API, then the compiler will tell you what that change has broken.
I can imagine writing code in a runtime/weakly-typed language ... but I can't imagine refactoring without the compiler's help, and I can't imagine writing tens of thousands of lines of code without refactoring.
Is this true?
I think you're conflating when types are checked with how they're checked. Runtime typing isn't necessarily weak.
The main advantage of static types is exactly what you say: they're exhaustive. You can be confident all call sites conform to the type just by letting the compiler do it's thing.
The main limitation of static types is that they're limited in the constraints they can express. This varies by language, with most languages having relatively simple type systems (c, java), and others with extremely powerful type systems (haskell, cayenne).
Because of this limitation types on their own are not sufficient. For example, in java types are more or less restricted to checking type names match. This means the meaning of any constraint you want checked has to be encoded into a naming scheme of some sort, hence the plethora of indirections and boiler plate common to java code. C++ is a little better in that templates allow a bit more expressiveness, but don't come close to what you can do with dependent types. I'm not sure what the downsides to the more powerful type systems are, though clearly there must be some or more people would be using them in industry.
Even if you're using static typing, chances are it's not expressive enough to check everything you care about, so you'll need to write tests too. Whether static typing saves you more effort than it requires in boilerplate is a debate that's raged for ages and that I don't think has a simple answer for all situations.
As to your second question:
How can we re-factor safely in a runtime typed language?
The answer is tests. Your tests have to cover all the cases that matter. Tools can help you in gauging how exhaustive your tests are. Coverage checking tools let you know wether lines of code are covered by the tests or not. Test mutation tools (jester, heckle) can let you know if your tests are logically incomplete. Acceptance tests let you know what you've written matches requirements, and lastly regression and performance tests ensure that each new version of the product maintains the quality of the last.
One of the great things about having proper testing in place vs relying on elaborate type indirections is that debugging becomes much simpler. When running the tests you get specific failed assertions within tests that clearly express what they're doing, rather than obtuse compiler error statements (think c++ template errors).
No matter what tools you use: writing code you're confident in will require effort. It most likely will require writing a lot of tests. If the penalty for bugs is very high, such as aerospace or medical control software, you may need to use formal mathematical methods to prove the behavior of your software, which makes such development extremely expensive.
I totally agree with your sentiment. The very flexibility that dynamically typed languages are supposed to be good at is actually what makes the code very hard to maintain. Really, is there such a thing as a program that continues to work if the data types are changed in a non trivial way without actually changing the code?
In the mean time, you could check the type of variable being passed, and somehow fail if its not the expected type. You'd still have to run your code to root out those cases, but at least something would tell you.
I think Google's internal tools actually do a compilation and probably type checking to their Javascript. I wish I had those tools.
To start, I'm a native Perl programmer so on the one hand I've never programmed with the net of static types. OTOH I've never programmed with them so I can't speak to their benefits. What I can speak to is what its like to refactor.
I don't find the lack of static types to be a problem wrt refactoring. What I find a problem is the lack of a refactoring browser. Dynamic languages have the problem that you don't really know what the code is really going to do until you actually run it. Perl has this more than most. Perl has the additional problem of having a very complicated, almost unparsable, syntax. Result: no refactoring tools (though they're working very rapidly on that). The end result is I have to refactor by hand. And that is what introduces bugs.
I have tests to catch them... usually. I do find myself often in front of a steaming pile of untested and nigh untestable code with the chicken/egg problem of having to refactor the code in order to test it, but having to test it in order to refactor it. Ick. At this point I have to write some very dumb, high level "does the program output the same thing it did before" sort of tests just to make sure I didn't break something.
Static types, as envisioned in Java or C++ or C#, really only solve a small class of programming problems. They guarantee your interfaces are passed bits of data with the right label. But just because you get a Collection doesn't mean that Collection contains the data you think it does. Because you get an integer doesn't mean you got the right integer. Your method takes a User object, but is that User logged in?
Classic example: public static double sqrt(double a) is the signature for the Java square root function. Square root doesn't work on negative numbers. Where does it say that in the signature? It doesn't. Even worse, where does it say what that function even does? The signature only says what types it takes and what it returns. It says nothing about what happens in between and that's where the interesting code lives. Some people have tried to capture the full API by using design by contract, which can broadly be described as embedding run-time tests of your function's inputs, outputs and side effects (or lack thereof)... but that's another show.
An API is far more than just function signatures (if it wasn't, you wouldn't need all that descriptive prose in the Javadocs) and refactoring is far more even than just changing the API.
The biggest refactoring advantage a statically typed, statically compiled, non-dynamic language gives you is the ability to write refactoring tools to do quite complex refactorings for you because it knows where all the calls to your methods are. I'm pretty envious of IntelliJ IDEA.
I would say refactoring goes beyond what the compiler can check, even in statically-typed languages. Refactoring is just changing a programs internal structure without affecting the external behavior. Even in dynamic languages, there are still things that you can expect to happen and test for, you just lose a little bit of assistance from the compiler.
One of the benefits of using var in C# 3.0 is that you can often change the type without breaking any code. The type needs to still look the same - properties with the same names must exist, methods with the same or similar signature must still exist. But you can really change to a very different type, even without using something like ReSharper.
Related
For a number of years now, I have been interested in TDD, but one or two things just didn't click. I am pretty sure it is the usual thoughts most people have when trying. "The examples in the book are wonderful, but my code is a lot more complicated than that. I never have a a procedure that does one thing, it will call three others, and they will call three others, and that will get data from the DB... bla bla bla".
A little while ago, I found some videos on SOLID (Anyone who is stuck, thinking TDD would be awesome, but... then find a few videos on SOLID, trust me). Each point became slightly more confusing, until the end, everything just went into place, including how I thought about testing code, and TDD.
I, of course, have a lot of old code, that isn't written like this, but I am okay with that, because I do see a better idea of how it should be. And whenever I work on anything, I can take it out, and do it properly (even when that means cutting out the small part of a method that needs updating, giving it it's own class, and calling that.
It has a few more questions. I would like to know where I might be able to find answers for that, or is there a standard.
How much should be tested?
My assumption is all of it. A lot of my functions will be take input parameters, and run a Stored Procedure. My guess on how to test that would be, with a given set of input parameters, is the stored procedure being called the correct one, are the parameters getting put in correct. Often this will be obvious (sometimes there will be array of numbers input that will be transformed to a comma separated string). If nothing else, this example, while the test might not be as valuable, will be documentation.
How do I name things?
This is the old problem with development. Should the class be named like the method would be, UpdateEmployee, or should there be a whole lot of er classes (EmployeeUpdater, EmplyeeGetter, etc.)
How is IOC generally handled?
This is still fine for now, I am creating interfaces, implementations, setting up IOC, etc.
I can see though, that pretty soon I am going to have pages and pages and pages of Interface/Class mappings in my IOC initialization method, or I would imagine it splitting into section, with one method that calls a few other methods, each registering classes (by namespace, or something). Is this how it generally works, or are there smarter ways of managing this?
I recommend reading Clean Code by Robert C Martin
In my view...
How much should be tested?
There is a big difference between how much and how well.
Ultimately its a judgment call and or a simple cost/benefit analysis.
Critical apps/code should be tested more thoroughly.
Working pure TDD means your code will be highly tested - easily > 90% coverage, but remember there is a difference between test quality and coverage. You may decide to test more edge cases.
You can get 100% coverage with one test case, but its pragmatic to test a range of values e.g. 0, 1, many & boundaries.
How do I name things?
For Java as an example, Look at the standard Java API documentation and see how they do it.
Referring to Clean Code, naming is and should be difficult, and maybe refactor if the name no longer fits.
Example Classes from Java's API's
FileFilter
DesktopManager
Names should make it obvious what the class/method/variable does.
Refer to Kent Beck's Four Rules of Simple Design (Express intent)
How is IOC generally handled?
Maybe someone else can expand on this point more, but referring to Extreme Programming, don't use interfaces for the sake of it, but when you need them. If you only have one concrete instance, you probably don't need an interface. Refactor to add interfaces to follow known design patterns when you have a real need for them.
https://www.martinfowler.com/articles/designDead.html
I've heard many anecdotes that a large problem with dynamically typed languages is that type checking is very slow. Why is it slow though? What is the computer science rational that using runtime assigned types that may change cause large slowdowns in computational efficiency?
Dynamically typed languages must perform type-checking while code is running. Although they can sometimes be compiled, they need to cut many corners for reasonable performance. One big drawback of checking at runtime is that if a type fails to be valid, the interpreter can only throw exceptions or stop execution.
So they often try to coerce types to prevent exceptions, even when it may be undesirable. In python, it isn't uncommon to discover that a simple division by whole integers means that my user output is suddenly full of '2.0' because I didn't explicitly cast back into int.
The computer science rational is that type-checking is an extremely heavy algorithm. For every function you call, all the types involved must be validated (or coerced which may be another function call), and type information must be updated afterwards. At runtime you can only afford to have a simple type system and very little optimization. A compiler by comparison can exploit even a weak type system to optimize your inefficient algorithms away.
It's very common for statically-typed languages to be compiled, and dynamically-typed languages to be interpreted. This is because if a language is being designed for a compiler, it's a no-brainer to give the responsibility of type-checking to the compiler so that your code will be more optimal and won't need to manage typing at runtime. The less you need to carry at runtime, the faster code will execute.
Ultimately, this means languages designed for interpreters can't afford the level of typing a compiler can. In addition to having less freedom to exploit type information to optimize - strike 1 to performance - they must carry and modify type information at runtime - strike 2. The weaker type system also introduces many type safety bugs.
Naturally, there are also numerous cases where weak typing is desirable. Dynamic languages often take the role of scripting; they're quick to code, easy to interpret, and can be ported to new platforms faster than a compiler! This makes them invaluable for gluing very different systems together. One script can interact with the operating system and many programs on it to schedule a daily download of all the latest cat videos from your favourite website.
As always, I highly recommend that you have a dynamic language and a static language in your repertoire. It's invaluable to have access to the guarantees of strong typing and access to the ease of weak typing. Be a code omnivore :)
What do you lose in practice when you choose a statically-typed language such as Scala (or F#, Haskell, C#) instead of dynamically-typed ones like Ruby, Python, Clojure, Groovy (which have macros or runtime metaprogramming capabilities)? Please consider best statically-typed languages and best (in your opinion) dynamically-typed languages, not the worst ones.
Answers Summary:
Key advantages of dynamic languages like Ruby over statically-typed language like Scala IMHO are:
Quick edit-run cycle (does JavaRebel reduces the gap?)
Currently community of Scala/Lift is much smaller then of Ruby/Rails or Python/Django
Possible to modify type definitions (though motivation or need for that is not very clear)
In principle, you give up being able to ignore what type you're using when it is not clear (in the static context) what the right thing to do is, and that's about it.
Since complex type-checking can be rather time-consuming, you also probably are forced to give up fast on-line metaprogramming.
In practice, with Scala, you give up very little else--and nothing that I particularly care about. You can't inject new methods, but you can compile and run new code. You do have to specify types in function arguments (and the return type with recursive functions), which is slightly annoying if you never make type errors yourself. Since it compiles each command, the Scala REPL isn't as snappy as e.g. the Python shell. And since it uses Java reflection mechanisms, you don't have quite the ease of online inspection that you do with e.g. Python (not without building your own inspection library, anyway).
The choice of which static or dynamic language is more significant than the static/dynamic choice itself. Some dynamic languages have good performance and good tools. Some static languages can be concise, expressive, and incremental. Some languages have few of these qualities, but do have large libraries of proven code.
Dynamic languages tend to have much more flexible type systems. For example, Python lets you inject a new method into an existing classes, or even into a single object.
Many (not all) static languages lack the facility to construct complex literals. For instance, languages like C# and Java cannot easily mimic the following JavaScript { 'request':{'type':'GET', 'path':mypath}, 'oncomplete':function(response) { alert(response.result) } }.
Dynamic languages have very fluid semantics. Python allows import statements, function definitions and class definitions to appear inside functions and if statements.
eval is a staple of most dynamic languages and few static languages.
Higher order programming is easier (in my subjective opinion) in dynamic languages than static languages, due to the awkwardness of having to fully specify the types of function parameters.
This is particulary so with recursive HOP constructs where the type system can really get in the way.
Dynamic language users don't have to deal with covariance and contravariance.
Generic programming comes practically free in dynamic languages.
I'm not sure if you lose anything but simplicity. Static type systems are an additional burden to learn.
I suppose you usually also lose eval, but I never use it, even in dynamic languages.
I find the issue is much more about everything else when it comes to choosing which language to use for a given task. Tooling, culture, libraries are all much more interesting than typing when it comes to solving a problem with a language.
Programming language research, on the other hand, is completely different. :)
Some criticism of Scala has been expressed by Steve Yegge here and here, and by Guido van Rossum, who mainly attacked Scala's type system complexity. They clearly aren't "Scala programmers" though. On the other hand, here's some praise from James Strachan.
My 2 cents...
IMO (strong) statically-typed languages might reduce the amount of necessary testing code, because some of that work will be done by the compiler. On the other hand, if the compiling step is relatively long, it makes it more difficult to do "incremental-style" programming, which in the real life might result in error-prone code that was only tested to pass the compiler.
On the other hand, dynamically-typed languages feel like there is less threshold to change things, that might reduce the responding time from the point of bug-fixing and improvement, and as a result might provide a smoother curve during application development: handling constant flow of small changes is easier/less risky than handling changes which are coming in bug chunks.
For example, for the project where the design is very unclear and is supposed to change often, it might have been easier to use dynamic language than a static one, if it helps reduce interdependencies between different parts. (I don't insist on that one though:) )
I think Scala sits somewhere in between (e.g. you don't have to explicitly specify types of the variables, which might ease up code maintenance in comparison with e.g. C++, but if you end up with the wrong assumption about types, the compiler will remind about it, unlike in PHP where you can write whatever and if you don't have good tests covering the functionality, you are doomed to find it out when everything is live and bleeding). Might be terribly wrong of course :)
In my opinion, the difference between the static and dynamic typing comes down to the style of coding. Although there is structural types in Scala, most of the time the programmer is thinking in terms of the type of the object including cool gadgets like trait. On the other hand, I think Python/Javascript/Ruby programmers think in terms of prototype of the object (list of methods and properties), which is slightly different from types.
For example, suppose there's a family of classes called Vehicle whose subclasses include Plane, Train, and Automobile; and another family of classes called Animal whose subclasses include Cat, Dog, and Horse. A Scala programmer would probably create a trait called Transportation or something which has
def ride: SomeResult
def ride(rider: Someone): SomeResult
as a member, so she can handle both Train and Horse as a means of transportation. A Python programmer would just pass the train object without additional code. At the run time the language figures out that the object supports ride.
The fact that the method invocations are resolved at the runtime allows languages like Python and Ruby to have libraries that redefines the meaning of properties or methods. A good example of that is O/R mapping or XML data binding, in which undefined property name is interpreted to be the field name in a table/XML type. I think this is what people mean by "flexibility."
In my very limited experience of using dynamic languages, I think it's faster coding in them as long as you don't make mistakes. And probably as you or your coworkers get good at coding in dynamic language, they would make less mistakes or start writing more unit tests (good luck). In my limited experience, it took me very long to find simple errors in dynamic languages that Scala can catch in a second. Also having all types at compile time makes refactoring easier.
I often heard from professionals blog something like refactoring your code whenever the chance you get. What is it exactly? Rewriting your code in simpler and fewer lines? What is the purpose of doing this?
Refactoring code is a process of cleaning up your code, reducing the clutter and improving the readability without causing any side effects or changes to features.
Basically, you refactor by applying a series of code change rules that improve code readability and re-usability, without affecting the logic.
Always unit test before and after refactoring to ensure your logic isn't affected.
This Wikipedia article will give you an idea of the types of things included in the general concept of Refactoring.
The idea is adapt / evolve your code as you go. Simple things may be to rename variables or method parameters, but others may be to pass an additional parameter or to drop one, or to change its type. The data model may evolve as well. etc.
Often refactoring, works hand-in-hand with unit-testing, whereby the risk of "breaking something" is offset by the fact that such an issue may likely be discovered by the automatic testing (provide a good coverage and relevant test cases...).
In a nutshell, the ability to refactor (and btw, most IDE or add-ons to the IDEs, offer various tools that make refactoring easier and less error prone) allows one to write more quickly without stressing about some decisions ("should this object include an array or a list etc...) letting the programmer change some of these decisions as times goes, and with the added insight offered by having a workable, if not perfect solution. See a related concept: agile development.
Beware, refactoring doesn't give you license to start coding without putting any thought in design, in the object model, the APIs etc., however it lessens the stiffness of some of these decisions.
Martin Fowler has probably done the most to popularize refactoring, but I think good developers have always done these sorts of restructurings. Check out Fowler'srefactoring web site, and his 1999 Refactoring, which is an excellent introduction and catalog of specific refactorings using Java.
And I see he's a co-author of the brand new Refactoring, Ruby Edition, which should be a great resource.
I find that regularly cleaning up your code like this makes it a lot clearer and more maintainable.
To take one example, I wrote a small (Java 1.6) client library for accessing remote web services (using the REST architectural style). The bulk of this library is in one source file, and about half of that deals with the web services, while the other half is a simple in-memory cache of the responses (for performance). Over time both halves have grown in functionality, to the point where the source file was getting too complex. So today I used Fowler's "Extract Class" refactoring to move the cache logic into a new class. Before that I had to do some "Extract Methods" to isolate the caching logic. Along the way I did a few "Rename Methods" and an "Introduce Explaining Variable".
As other folks have noted, it's very important to have a good set of unit tests to apply after you make each change. They help ensure that you're not introducing new bugs, among other good things.
In a nutshell, refactoring means improving the design and/or implementation of software, usually without changing its behavior. This is normally done to make the code easier to understand and work with going forward, thereby making future development faster and less bug-prone.
Refactoring is a long-term investment in your code - since it doesn't affect the outward "appearance" of the software, there is very often pressure (from management, etc.) to "just get it working and move on to the next thing." While this may sometimes be the right decision, depending on business drivers, a codebase that undergoes change but never gets refactored will decay into a difficult, buggy mess (See also Technical Debt).
Specifically, the top reasons to refactor are usually the following:
Getting rid of duplicated code
Breaking up a long method into smaller pieces by extracting new methods from sections of the longer method
Breaking up a class that has too many responsibilities into smaller, more targeted classes or subclasses
Moving methods from one class to another. Often this is done so the methods reside in the same class as the data they operate on.
In the simplest terms, refactoring code is optimizing code. The criteria for what is "better" code is open to much interpretation as there are various coding styles and patterns out there. A central idea with refactoring is the question of, "Could this code be made better?" A few examples of that criteria can include scalability, maintainability, readablity, performance, size of executable, or minimizing memory used in executing the code.
"Refactoring is the process of changing a software system in such a way that it does not alter the external behavior of the code yet improves its internal structure." -- MartinFowler in RefactoringImprovingTheDesignOfExistingCode
see this WhatIsRefactoring for more explanation.
Refactoring code generally means taking code that has been patched multiple times and re-writing it so that the needs of the later patches are taken into account.
For a long time I've been trying different languages to find the feature-set I want and I've not been able to find it. I have languages that fit decently for various projects of mine, but I've come up with an intersection of these languages that will allow me to do 99.9% of my projects in a single language. I want the following:
Built on top of .NET or has a .NET implementation
Has few dependencies on the .NET runtime both at compile-time and runtime (this is important since one of the major use cases is in embedded development where the .NET runtime is completely custom)
Has a compiler that is 100% .NET code with no unmanaged dependencies
Supports arbitrary expression nesting (see below)
Supports custom operator definitions
Supports type inference
Optimizes tail calls
Has explicit immutable/mutable definitions (nicety -- I've come to love this but can live without it)
Supports real macros for strong metaprogramming (absolute must-have)
The primary two languages I've been working with are Boo and Nemerle, but I've also played around with F#.
Main complaints against Nemerle: The compiler has horrid error reporting, the implementation is buggy as hell (compiler and libraries), the macros can only be applied inside a function or as attributes, and it's fairly heavy dependency-wise (although not enough that it's a dealbreaker).
Main complaints against Boo: No arbitrary expression nesting (dealbreaker), macros are difficult to write, no custom operator definition (potential dealbreaker).
Main complaints against F#: Ugly syntax, hard to understand metaprogramming, non-free license (epic dealbreaker).
So the more I think about it, the more I think about developing my own language.
Pros:
Get the exact syntax I want
Get a turnaround time that will be a good deal faster; difficult to quantify, but I wouldn't be surprised to see 1.5x developer productivity, especially due to the test infrastructures this can enable for certain projects
I can easily add custom functionality to the compiler to play nicely with my runtime
I get something that is designed and works exactly the way I want -- as much as this sounds like NIH, this will make my life easier
Cons:
Unless it can get popularity, I will be stuck with the burden of maintenance. I know I can at least get the Nemerle people over, since I think everyone wants something more professional, but it takes a village.
Due to the first con, I'm wary of using it in a professional setting. That said, I'm already using Nemerle and using my own custom modified compiler since they're not maintaining it well at all.
If it doesn't gain popularity, finding developers will be much more difficult, to an extent that Paul Graham might not even condone.
So based on all of this, what's the general consensus -- is this a good idea or a bad idea? And perhaps more helpfully, have I missed any big pros or cons?
Edit: Forgot to add the nesting example -- here's a case in Nemerle:
def foo =
if(bar == 5)
match(baz) { | "foo" => 1 | _ => 0 }
else bar;
Edit #2: Figured it wouldn't hurt to give an example of the type of code that will be converted to this language if it's to exist (S. Lott's answer alone may be enough to scare me away from doing it). The code makes heavy use of custom syntax (opcode, :=, quoteblock, etc), expression nesting, etc. You can check a good example out here: here.
Sadly, there's no metrics or stories around failed languages. Just successful languages. Clearly, the failures outnumber the successes.
What do I base this on? Two common experiences.
Once or twice a year, I have to endure a pitch for a product/language/tool/framework that will Absolutely Change Everything. My answer has been constant for the last 20 or so years. Show me someone who needs support and my company will support them. And that's that. Never hear from them again. Let's say I've heard 25 of these.
Once or twice each year, I have to work with a customer who has orphaned technology. At some point in the past, some clever programming built a tool/framework/library/package that was used internally for several projects. Then that programmer left. No one else can figure that darn thing out, and they want us to replace/rewrite it. Sadly, we can't figure it out either, and our proposal is to rewrite from scratch. And they complain that their genius built the set of apps in a period of weeks, it can't take us months to rewrite them in Java/Python/VB/C#. Let's say I've written 25 or so of these kinds of proposals.
That's just me, one consultant.
Indeed one particularly sad situation was a company who's entire IT software portfolio was written by one clever guy with a private language and tools. He hadn't left, but he'd realized that his language and toolset had fallen way behind the times -- the state of the art had moved on, and he hadn't.
And the move was -- of course -- in an unexpected direction. His language and tools were okay, but the world had started to adopt relational databases, and he had absolutely no way to upgrade his junk to move away from flat files. It was something he had not foreseen. Indeed, it was something he could not possibly foresee. [You won't fall into this trap, will you?]
So, we talked. He rewrote a lot of the applications in Plain-Old VAX Fortran (yes, this is a long time ago.) And he rewrote it to use plain old relational SQL stuff (Ingres, at the time.)
After a year of coding, they were having performance problems. They called me back to review all the great stuff they'd done in replacing the home-built language. Sadly, they'd done the worst possible relational database design. Worst possible. They'd taken their file copies, merges, sorts, and what-not, and implemented each low-level file system operation using SQL, duplicating database rows left, right and center.
He was so mired in his private vision of the perfect language, that he couldn't adapt to a relatively common, pervasive new technology.
I say go for it.
It would be an awesome experience regardless of weather it makes it to production or not.
If you make it compile down to IL then you do not have to worry about not being able to re-use your compiled assemblies with C#
If you believe that you have valid complaints about the languages you listed above, it is likely that many will think like you. Of course, for every 1000 interested person there might be 1 willing to help you maintain it - but that is always the risk
But here are a few things to be cautioned about:
Get your language specification IN STONE before development. Make sure any and all language features are figured out before hand - even things that you may only want in the future. In my opinion, C# is slowly falling into the "oh-just-one-more-language-extension" trap that will lead to its eventual doom.
Be sure to make it optimized. I dont know what you already know; but if you dont know then learn ;) Nobody will want a language that has nice syntax but runs as slow as IE's javascript implementation.
Good luck :D
When I first started my career in the early 90s, there seemed to be this craze of everyone developing their own in-house languages. My first 3 jobs were with companies that had done this. One company had even developed their own operating system!
From experience, I'd say this is a bad idea for the following reasons:
1) You will spend time debugging the language itself in addition to the code base on top of it
2) Any developers you hire will need to go through the learning curve of the language
3) It will be hard to attract and keep developers since working in a proprietary language is a dead-end for someone's career
The main reason I left those three jobs was because they had proprietary languages and you'll notice that not many companies take this route any more :).
An additional argument I'd make is that most languages have entire teams whose full time job it is to develop the language. Maybe you'd be an exception, but I'd be very surprised if you'd be able to match that level of development by only working on the language part-time.
Main complaints against Nemerle: The
compiler has horrid error reporting,
the implementation is buggy as hell
(compiler and libraries), the macros
can only be applied inside a function
or as attributes, and it's fairly
heavy dependency-wise (although not
enough that it's a dealbreaker).
I see your post has been written more than two years ago.
I advise you trying Nemerle language today.
The compiler is stable. There are no blocker bugs for today.
The VS integration has a lot of improvements , also there is SharpDevelop integration.
If you give it a chance, you won't be disappointed.
NEVER EVER develop your own language.
Developing your own language is a fool's trap, and worse it will limit you to what your imagination can provide, as well demanding that you work out both your development environment and the actual programme you're writing.
The cases in which this doesn't apply are pretty much if you're Larry Wall, the AWK guys, or part of a substantial group of people dedicated to testing the boundaries of programming. If you're in any of those categories, you don't need my advice, but I strongly doubt that you're targeting a niche where there is no suitable programming language for the task AND the characteristics of the people doing the task.
If you are as clever as you seem to be (a likely possibility), my advice is to go ahead and do the design of the language first, iterate a couple of times over it, ask some smart fellows you trust in smart programming language related communities about the concrete design you came up with and then take the decision.
You might realize in the process of creating the design that just a quick hack on Nemerle would give it all you need, for example. Many things can happen just when thinking hard about a problem, and the final solution might not be what you actually had in mind when beginning the project.
Worst case scenario, you're stuck with actually implementing the design, but by then you will have it proof read and mature, and you'll know with a high degree of certainty that it was a good path to take.
A related piece of advice, start small, just define the features you absolutely need and then build on them to get the rest.
Writing your own language is not a easy project.. Especially one to be used in any kind of "professional setting"
It is a huge amount of work, and I would doubt you could write your own language, and still write any big projects that use it - you will spend so long adding features that you need, fixing bugs, and general language-design stuff.
I would strongly recommend choosing a language that is closest to what you want, and extending it to do what you need. It'll never be exactly what you want, but compared to the time you'll spend writing your own language, I would say that's a small compromise..
Scala has a .NET compiler. I don't know the status of this though. It's kind of a second class citizen in the Scala world (which is more focused on the JVM). But it might be a good tradeof to adopt the .NET compiler instead of creating a new language from scratch.
Scala is kind of weak in the meta-programming department ATM. It's possible that the need for metaprogramming is somewhat reduced by other language features. In any case I don't think anyone would be sad if you were to implement metaprogramming features for it. Also there is a compiler plug-in infrastructure on the way.
I think most languages will never fit all of the bill.
You might want to combine your 2 favourite languages (in my case C# and Scheme) and use them together.
From a professional point of view, this probably not a good idea though.
It would be interesting to hear some of the things you feel you can't do in existing languages. What kind of projects are you working on that can't be done in C#?
I'm just curios!