Use case for MongoDB profit [closed] - oracle

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
Good day. I'm doing my magister's degree on "Implementing distributed NoSQL database". Having studied the material comparing strenghts and weaknesses of NoSQL databases compared to RDBMS I've faced the step of choosing the proper problem to solve. My task is to show the typical development of the same application backed by Oracle and MongoDB and to show that during the evolution of the app Mongo begins to outperform Oracle. I'm focused on many writes and horizontal scaling. As the task I've chosen a typical Twitter like app with complex evolving domain, Java and Spring Data as my instruments.
I ask for experienced people for benefical critics and alternative tasks to show Mongo's favor. I understand that it highly depends on the schema, indexes, etc, still I ask if
Mongo on my scenario can beat Oracle with:
Many writes
Horizontal scaling
Read operations
Schema evolving
Sharding\Replication

My task is to show the typical development of the same application backed by Oracle and MongoDB and to show that during the evolution of the app Mongo begins to outperform Oracle.
I'm sorry for being very frank, but what kind of academic work starts with the final answer and you want to reverse engineer the problem?! This is less than worthless since it's intentionally misleading.
Leaving that aside, here are some tips:
Use something which requires JOINs in the relational database, but can be modelled as a single document. Blog posts would come to my mind. Common tricks include putting the author name into the document. No JOIN required for reading and if the author changes his name (which will happen very rarely in most systems) you only need a unique attribute like his email address to update the name everywhere:
{
title: "...",
content: "...",
date: "...",
author: { name: "...", email: "..." },
comments: [
{ name: "...", email: "...", text: "...", date: "..." },
...
]
}
Keep your data small enough so they fit into RAM. MongoDB can make good use of that and will only occasionally flush information to disk (depending on your configuration), RDBMS will always to to disk for durability reasons (ACID compliance).
Use an "insecure" connection setting. Do not wait for the database to actually process the request, but return immediately (fire-and-forget like UDP). This isn't possible in a transactional system. You can amplify this if you test in the cloud, for example on EBS backed EC2 instances with have very high disk latency.
Use a pretty heavy ORM like Hibernate. Probably avoid an ODM (object document mapper) like Morphia (if you're doing it in Java) and use the plain Java driver - even though I'm not sure how big the performance gain is, but I'm sure there is some if done properly.
Use replication in MongoDB and allow reads from the secondaries (thus sacrificing consistency but gaining performance).
Use sharding.
Besides the system's performance, you might want to take a look at developer productivity. MongoDB is great for getting start and I have the feeling it is much quicker to get started with. Not sure if this doesn't change into the opposite in the long run - strict schemas do have their place in the long run IMHO.
I'd rather compare MySQL and MongoDB. The two are both open source software and pretty similar. For example indexing is exactly the same - only b-trees (if you stick to the standard, on-disk storage engines).
Final note: I hope you can agree with me that it's pretty easy to win such a disbalanced comparison, which makes it pretty pointless...

Related

What is best way to store mp3 files in server ? Storing it in database (BLOB) , is right? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I have an audio site where user can upload their music files , but problem is i can't go for expensive hosting , since i am not monetizing this service.I am searching for some shortcuts to store the mp3 files to cut some hosting cost.
What will be best idea to do technically or any (hosting)suggestion will be help full.
I need to save server space as much as possible.
In most cases, the size of your database will also count against your overall hosting space as well. Furthermore, inserting huge BLOBs into your database isn't going to help performance with it.
The typical pattern to follow when doing something like this is to save the MP3 (or any binary file) on the server in a particular directory, and save the path to the file in the database.
The least expensive way, outside of using the original hosting environment, would probably be to utilize Amazon AWS S3 reduced redundancy storage, which starts at $0.093 per GB/per month. Pretty darn cheap.
But in answer to your original question, inserting stuff in the database probably won't save server space, and if your host is worth its salt, they will pick up on a huge database that keeps growing and growing, even if they claim "unlimited databases" or similar.
Just consider that storing in a database (BLOB) is usually a bad practice because it slows the queries, makes the database big and fall off the database performance. A database is used to store "searchable" information, not as a data store. Although a database can do it, it's not designed for that.
Take a look to some cloud storage service/provider instead, as the ADrive services like their personal plan ( http://www.adrive.com/personal_basic ) that let you store 50G for free (Im not sure if it's a trial), and also has Remote File Transfer functionality that allows you to transfer files from external websites.
I never tried this service, but give it a try, it's free and maybe solves your problem

Writing user stories for internal technical tasks [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 5 years ago.
Improve this question
I am attempting to manage my projects a little better so I am looking at attempting to apply some of (eventually all) the features of scrum.
Looking at user stories specifically the high level format seems to be:
As a User I can Feature Description
or
Artifact is Doing Something
How would I write "Upgrade the Database"?
Is it simply Upgrade the Database?
I think I am being thrown off as there is no specific actor/customer and that the customer is the IT department.
AS A [person/role]
I NEED TO [do something]
SO THAT [provides business value].
For your example a user story might look like this:
AS A user of the XYZ application
I NEED TO get reports of ABC faster
SO THAT we can increase our conversion rates.
ACCEPTANCE CRITERIA - The database reliably completes transactions on average in 2 seconds.
I've added an acceptance criteria because without this you will never know when the job is done. Now at this point, you have a business case for upgrading the database. This story would be decomposed into a story where the role is the IT department or DBA, like so:
AS AN administrator for the database server
I NEED TO upgrade to the latest version of FancyDB 11.7
SO THAT we can improve the average transaction time for XYZ users to 2 seconds.
ACCEPTANCE CRITERIA - the new version starts successfully, the XYZ developers sign off on the test installation of 11.7, data migration is successful, we have cut over to the new db
When story decomposition is added to your box of tools, the story must start from where the user is a real part of the business, and the "so that" leads to a real business value. Then decompose the story into one or more stories in which internal users do things "so that" real users get the benefits in need.
Here are a couple of articles that talk about Story Decomposition:
http://jpattonassociates.com/the_shrinking_story/
http://old.cognitive-edge.com/wp-content/uploads/1999/11/56-1999-11-Paradox-of-Story.pdf
Scrum is not very prescriptive and there is nothing in Scrum that forces you to use User Stories for your Product Backlog Items (PBIs). You can definitely do Scrum without capturing requirements/features as user stories, user stories are just one way to do it. Actually, stories do work for many teams, especially web development teams, but this doesn't mean that they work in all cases and on every project (many projects are web development but not all, like in your case). There is no consensus about using stories.
That said, the recommended template for User Stories is actually As a <role>, I want to <action> so that <benefits>. I don't mean to be picky but, if you choose to use stories, I'd warmly suggest to use it as is, without removing any part. First, using a role do help (a same user/person can have several roles) to discover stories. Then specifying the benefits is really important to expose the business value of a story in order to prioritize them well. Regarding the value, you should think of it as end-user/customer ("put on customer glasses" --Mary Poppendieck). It is really not always that easy to express the benefits, but some tools might help and my preferred one is the 5 whys (which is used for root cause analysis).
In your case, this could lead to something like: As the IT department, I want the database to be upgraded so that users can benefits from the latest features of the application and [do a better job|have a better user experience] (not very satisfying though, use the 5 whys).
But personally I don't find that user stories are the best medium for technical tasks even if it is clearly possible to use them and if they have their strengths. Theoretically, stories capture the essence, not the details and should be a support for the discussion. I may be wrong but I don't find that technical tasks offer much room for discussion and creativity. So, depending on who will read them, what the should convey, I might use them or not. Another option could be to mix stories with another formalism for your PBIs. As I said, the point is not to use stories, the point is to have a list of prioritized and estimated items.
Upgrade the database may be one of the tasks involved in implementing another story that brings direct value to the user, for example I as a user can add a new foo to my bar.
If adding a foo to a bar requires a database upgrade behind the scenes, then you would include that work in implementing that user story.
User stories are worded this way to help ensure that any work directly benefits the end user in some way.
This gets to the forefront of why user stories are so great.
What benefit does upgrading your database give to the end user? None? Then don't spend the time and money doing it. Spend that time and money providing something that will give value to your end user.
If it does? Then think about it the other way. Maybe you can only implement a new feature when you have version x of your database software? In the dependency of the story, you could mention that database upgrade required to provide this feature.
tl;dr Don't just upgrade for the sake of it. Make sure upgrading adds tangible value to your customers.
Generally, technical tasks in the PB are frowned upon because they very rarely directly deliver business value to the customer. That's why User Stories are popular, because they force you to think about the business value of the story, and who it's being delivered to.
So, why are you upgrading the database? Can you identify business value in upgrading it, and why should the Product Owner agree to let you upgrade the database instead of building new features?
Is it because of a new feature that will make it possible or make it easier to do something in your application? In that case, that something should be the PB item, and the database upgrade should be a task within that story. If you already have stories on the PB that would benefit from the upgrade, then you should increase the estimates for one or more of those stories, and add the upgrade as a technical task to the story.
Is it because the vendor of the database is cutting off an old version from support? In that case you could have the upgrade as the story; something like, "As the department manager, I want to be sure that we have support for all of the software so that the continuity of the business isn't at risk if something goes wrong". Even that's pushing it, though. Generally, this kind of reason isn't really part of a project, unless the project has been going on so long the system software goes off support.
Is it for performance? Then the story should be about some aspect of the performance of the application that needs to be improved to deliver business value. Something like, "As a CSR I need to be able to retrieve customer information in a reasonable time so that customers on the phone are satisfied with our service". Then the upgrade becomes a task under that story.
Is it for some totally technical reason? If you can't identify how the upgrade is going to deliver business value, then why would you do it? Why would the Product Owner select it for a Sprint?
It's simply "Upgrade the Database" or maybe "When the new version is installed, there must be a way to migrate the existing database". If you already know more details about this step, then include them. But the story mostly exists to make sure something isn't forgotten; it's not to be detailed.
Later, when you get to implement this story, you can flesh it out (which tables, do we need one or more backups, is there a fall back scenario, etc).
OTOH, if the project is more complex, this can become a "tag", like a post-it notice which must be attached to many stories. That means you must include this as a "sub story" to all stories which change the database. As you can see, these "project-spanning stories" are a bit hard to track with agile methods.
Infrastructure stories do not need to follow the prescribed story template. Just write down what needs to be done and estimate accordingly
How about:
As the application support person I want to be on the latest version of database because it is more reliable / more secure / whatever.
You could even phrase refactoring like that:
As the application developer I want all the data classes in one module so that I can add new fields to the app very quickly.
Who benefits
What you want to do
What the benefit is
Ideally you don't want all the stories to have 1 be developers, but a few make sense (sharpening your ax instead of cutting down trees and all that).

The future of Naked Objects pattern (and UI auto-generation) [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I ask about the pattern, not framework. This is kind of follow-up to a question on UI auto-generation.
Do you believe in the concept of UI auto-generation from metadata?
What kind of problems can be approached this way?
The question arose when I've created a small library to support my student projects, which generates interactive CLI in runtime based on object's metadata. And I think CLI it generates is quite decent.
On the other extreme is the Naked Objects Framework, which is rather universal, but UI it generates is horrible, IMO.
It's clear, every problem is specific and needs specific UI, but maybe there are several classes of problems where auto-generation is acceptable?
Yes, I believe the concept of metadata-based auto-generated applications is very sound - mainly because it drastically reduces development time and improves code quality by reducing the massive redundancy you have in most applications where each domain data field is represented in the database, in the model, in the UI, and often also several times in various mapping layers.
I think the future is auto-generated apps that can be modified wherever necessary. Currently, this is AFAIK not really possible; for example, Rails only allows you to fully customize the UI when you use static scaffolding, which basically means code generation, i.e. many further changes in the domain model are then not automatically represented in the UI because the duplication has happened when the code was generated.
I believe the first framework that manages to combine complete auto-generation with complete modifiability afterwards will become the de-facto development standard to a previously unknown degree. Though most likely we'll get there in small steps so that there will not be such a single dominating framework.
Take a look at JMatter, which is a rather better-looking implementation of Naked Objects.
http://www.jmatter.org
There is also Chris Muller's work on MAUI, and Lukas Renggli's work on Magritte (both Squeak /Smalltalk)
We have lots of generated UI in the configuration part of our apps. All those lists that are around forever and changed once in a blue moon by a system administrator.
I find that most applications with a database back-end tend to have a bad design from an OO and NO perspective, as already shown in the NO book by Pawson and Matthews.
Re: qn #1 ... Do you believe in the concept of UI auto-generation from metadata? ... I'm definitely going to answer 'yes' to your first question, being one of the committers to the Naked Objects (Java) framework and writing a book on DDD + NO.
The question mentions metadata. I think this is key to NO being able to succeed. In the latest version (which will be going beta in Feb) the metamodel has been opened up so that it is very extensible, either so you can write your domain model following your own programming conventions/annotations, or, potentially so that more sophisticated viewers can look for their own metadata to provide more sophisticated views. (For example, consider that if an object implemented a Location interface then it is displayed in a google maps).
Regarding qn #2 ... what kind of problems can be approached this way ... we've always said that NO is more suitable for "sovereign applications" (transactional, operational systems ones used internally within an organization) to "transient applications" (like an airport kiosk, say). An NO GUI does require that the user is familiar with the domain, otherwise they won't know what they are looking at.
What's missing still is sophisticated viewers, of course. You are right about the NO GUI, it is definitely low fidelity (though the .NET version is a big improvement, see recent infoq.com article). On the Java side there is a sister project called scimpi.org that has a lot of promise though... it provides a basic web GUI for free but lets you hand-craft web pages as necessary and incrementally. I'm also working on an Eclipse RCP GUI that'll work similarly.
The other thing to add to this though is that the NO approach has value (I believe) even if you choose to write a custom GUI and/or presentation layer. That is, you can use it as a design tool for building a very solid pojo domain layer, and then skin it as you will. Trouble is that NO was never originally sold in those terms, so most will see the NO pattern as an all-or-nothing affair.
Dan
One way to look at this is to consider the difference between the user interface you get from something like Toad or MySQL Browser, where the user interface is directly constructed from the tables and their associated meta data, and the user interface that a skilled designer would develop for the actual application. IF there not too disimilar then it should be fairly low hanging fruit for an auto-generation framework.
As you say there are classes of problems which will work quite well with this kind of auto generation and some which wouldn't. To my mind the key things are how well the implementation model (or portion thereof) which you are exposing in the user interface maps to the conceptual model of the user. Secondly how well can the behavior of the application can be expressed through a limited set of user interface components (assuming this is a general purpose UI generation framework).
This article "Universal Model of a User Interface" may be of interest .
I think the idea of automatically generated UIs has a lot of potential especially for your average form-and-table layout database user interface. However, even there a human needs to be in the loop, having the ability to override the output without it being overwritten with the next regeneration.
I suspect automatically generated UIs would be more successful today if interaction designers were more involved in developing the generation algorithms. My impression is that historically the creators of these systems don’t know what kinds of UI-related metadata to include or how to use it. Specifying labels, value ranges, formats, and orders for fields is a start, but more high level information is needed. Sufficient modeling of the tasks and user roles in particular tends to be lacking, along with some basic style-guide-level principles for UI.
Oracle’s Designer 2000, for example, was on the right track in including not only the entities and relations in the model, but also the tasks in the form of a functional hierarchy. Then they blew it by misapplying this metadata (e.g., assuming that depth is always preferred to breadth) and including fundamental flaws when generating the UI (e.g., only one primary window can be opened at a time). The result was IUs that were not even consistent with Oracle’s own Applications User Interface Standards.
Getting a basic UI up quickly that lets the customer try out the system and create test data must be of value. Naked Objects frameworks can help for the “boot strapping” even if you have to have replace it with “hand crafted” UI before you ship.
In most system I have worked on, there have been lots of simple housekeeping tables. All these tables need a UI to edit and view them etc. There is also great value in these simple editors being consistent. Here a naked Objects framework could save a lot of time, even if the main “day to day” UI is “hand crafted”
I have seen a couple of failed projects (cases where I was brought in as a rather expensive consultant to help architect the replacement) which used the "naked objects" approach (not the framework, AFAIK) - all with simply atrocious UIs, and worked replacing a lot of the UI on one project which, in its original incarnation, had a similar approach (the entire application was a tree of objects accessed through context menus and property sheets - this was NetBeans 2.0 circa 1998 - IDE as a giant hierarchical JavaBean).
The bottom line is, your users don't care about your architecture, they care about getting what they need to do done in the most comprehensible-to-mere-mortals set of interactions you can come up with. If that happens to align with your architecture, you are having a lucky day - but it really is serendipity. Trying to force users to care (or even know) about your architecture is a recipe for software nobody wants to use.
Code generally needs to be designed around two not-always-compatible goals:
Maintainability - people who didn't write the code can understand the code
Stability and performance - i.e. the activities the code asks the computer to physically do are both possible, and can be completed within a reasonable time frame
The abstractions and code structures that it makes sense to create to meet those two goals very, very rarely map exactly to user interface elements of any sort. Sometimes you can get away with it - barely - if your audience is technical. But even there, you are likely to please more users with at least a "presentation layer" adapter layer on top of the architecture that makes sense for programmers and machines.

Refresh Oldschool GUI Design [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I'm building Desktop Software for over 10 years now, mostly it's simple Data-Input Software. My problem is, it's always looking the same: A Treeview on the Left and a lot of Text/Data Fields to the right, depending on the type of data currently is worked on. Are there any fresh ideas how such software nowadays should look like?
For further clarification:
It's very hierarchical data, mostly for electronic devices. There are elements of data which provide static settings for the device and there are parts which describe some sort of 'Program' for the device. There are a lot (more than 30) of different input masks. Of course i use combo boxes and Up/Down Entry Fields.
Having all of your software look the same thing is a good thing. One of the best ways to make it easy for people to use your software is to make it look exactly the same as other software your users already know how to use.
There are basically two common strategies for how to handle entry of a lot of data. The first is to have lots of data entry fields on one page. The next is to have only a few data entry fields but a lot of pages in a sort of wizard-style interface. Expert users find the latter much slower to use, as do users who are entering data over and over again. However, the wizard style interface is less confusing for newer users since it offers fewer elements at once and tends to provide more detail on them.
I do suggest replacing as many text fields as possible with auto-complete-based combo-boxes. This allows users to enter data exactly the same as with text-boxes, but also allows users to save typing by hitting the down key to scroll through choices after typing part of the data in.
Providing more detail on what data is being entered would probably yield more specific answers.
I'd also answer with a question, which is to ask what your motivation for considering a change is? Like the other posters, I'd agree that there is some value in consistency, but there's also a strong value in not ignoring niggles-in-the-back-of-the-mind feelings you have. Maybe you have a sense that your users aren't as productive as you'd like them to be, or you've heard feedback to that effect from your customers, or you're just looking to add some innovation for your own interest. Scratching itches is a good trait in a developer, in my view.
One thing I'd advocate would be a detailed user study. How much do you know about what your users do with the interfaces you create? Do you know the key tasks, the overall workflow? Would you know if one task regularly consumed 60% of your users' time, or if there was a task that was only performed once a month? Getting a good sense of what the users actually do (and not what they say they do) is a great place to start thinking about what changes might be worthwhile, especially if you can refactor the task to get a qualitatively different user experience.
A couple of specific alternative designs you might like to include in re-visioning the UI might be be facet browsing (works well for searching and exploring in hierarchies), or building a database of defaults / past responses so that text boxes can use predictive completion. However, I think my starting point would be the user study.
Ian
If it works...
Depending on what you've got happening with the data (that is, is it hierarchical, or fairly flat), you might want to try a tab-based metaphor, or perhaps the "Outlook-style", with a sidebar showing the sections of an application. One other notion I've played with lately is the "Object desktop" that I first saw proposed by Scott Ambler (Building Object Applications That Work). In this, you can display collections of items, or the user can "peel off" individual records for easy access.
Your information is not enough to really suggest you an interface alternative. However, may I answer your question with a question? Why do you think you have to change it? Has your customer complained? If not, it looks like your customer is happy with the way the software works right now, thus I wouldn't change it. If your customer complains about it, he'll most likely not just say "It's bad", he will say "Why can't it look like ..." and this will give you an idea how to change it.
I once had to re-design a very outdated goods management system. The old one was written for a now dead database system, still running in MS-DOS. The customer suggested I should create a prototype how this re-implementation might look like and then he'll decide if I get that job or not. I replaced the old, dead database with a modern MySQL database, I replaced the problematic shared peer access with a client server approach and I chose to rewrite the UI in Java, since different OSes were used and this had the smallest porting costs. So far the concept seemed good, the customer liked it. However, when he asked his employees what they think about it, they asked "So far it's great, but we have one question: Why doesn't it look like the old one?". Actually, it turned out that even with all the modern technologies, they wanted the interface to exactly look and being operated like the old one. So I had to re-build a 1986 usability nightmare MS-DOS UI in Java, because no other UI was accepted.
For me it is more about a clean, usable, logical design than anything else. If your program makes sense to the user, isn't clunky and works as advertised, then everything else UI related is essentially just like painting the house. I've sometimes rolled out a new version of a program with essentially the same controls that are skinned differently.
There's a reason that you've probably chosen the tree view - because it probably makes really good sense to do so. There are different containers and controls available in the various UI libraries, depending on the language, but I tend to stick with the familiar because the user probably gets how a tree control works and how a combobox works.
A user interface needs to be usable, just don't do the misstake to change to something working to something fancy-schmancy just because it looks better (been down that road)...
Make sure that added
widgets/controls really add a business value
Make sure that the added
widgets/controls do not mess up your
architecture (too much) and makes
the application harder to
manage/maintain
Try to keep platform standards on
how to do things (for example the Vista ux guidelines)
:)
//W

Documents for a project? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I work for a CMMI level 5 certified company and one thing I hate about is the amount of documents we prepare (As a programmer I already hate documents). We have lots and lots of documents like PID(project initiation doc), Business requirements, System requirements,tech spec, Code review checklist, issue logs, Defect logs, Configuration management plan, Configuration management check list(s), Release documents and lots...
Almost 90% of these docs are just done for the sake of QA audit :) .. What do you think are the most important documents for a project? What documents can be used in the long run by another developer?
Please share your good practices here. I would like to use them for my own projects or the company I am planning to start in the long run.
Thanks
The key document is a good functional spec. There should be one and only one reference document for a system.
Overdoing documentation proliferates a large number of small requirements and spec documents every time someone changes a system or interface. For a system of any complexity, before long you have your spec distributed around several hundred assorted word, excel, visio and even powerpoint files. When this happens you lose clarity about what is current or even whether you have located and identified all pertinent documentation.
The BRD-SRD-Tech spec progression is based on an assumption that the business signs off the BRD, a business analyst signs off the SRD against requirements documented in the BRD and the technical specification is signed off against the SRD. This generates a web of sign-offs, multiple documents with redundant information and makes it difficult and clumsy to keep the spec documents up to date.
Because of this, subsequent requirements documentatation tends to take the form of a series of change request and supplemental requirement and spec docs, each with their own sign-off and audit process. You gain CYA and audit trail (or at least the appearance of an audit trail), but you lose clarity. There is now no definitive reference document for the system and it is difficult to establish what is current or relevant to any particular activity. The net result is that your business analysis process gets bogged down in forensic research, which adds overheads and latency to delivery schedules.
A spec document should be built in such a way that there is one definitive reference for any given system or subsystem. The document should be kept up to date and versioned. Get a good technical documentation tool like Framemaker, so your process can scale, and the document has some structural integrity of the sort lacking on Word.
For me the only real document I ever use is a spec. The more detail the better. However it doesnt need to be all completed at one time, and it doesnt need to be particularly formal. What is far more useful to me than documents that are checked and signed and double checked and double signed is always being able to get the latest version of a document. And being able to talk to people about what they have written, and get a decision in the case of any ambiguity. this is far more useful to me than anything else.
To sum up: a spec is the only document I have ever found useful, however it pales in comparison to having a project manager who knows the proposed system inside out, and can make sensible decisions based on what they know.
Documentation is like tofu -- most people hate it until they realize that under the right conditions, it can be really good.
The problem is that what you consider documentation is mostly made for documentation's sake. You, as a developer, don't see any immediate value in the documents you produce because you know you can do your job without all the TPS reports which you're required to make.
Unfortunately, I'm going to wager that there's not a lot you can do about in a company where you're being forced to eat raw tofu all the time. You'll probably just have to suck it up and write the docs which your company requires, but you can at least do one thing... you can write documents which at least are useful to you, and you can pass them along with your code for others who will maintain it.
Aside from inline documentation, you could set up a wiki to be used by yourself and people on your team. This type of documentation is searchable, which is already a big plus to developers, plus it's more of a living document instead of a homework-like paper you had to write. You already post to SO, so just think of your documentation as pooling your knowledge in a more useful place.
What do you think are the most important documents for a project?
Different people have different needs: for example the documents which the owner needs (e.g. the business contract) aren't the same as the documents which QA needs.
What documents can be used in the long run by another developer?
IMO the most important document (except for the source code) is the functional specification: because what the software is supposed to do (as opposed to, what it is doing) is the one thing that can't necessarily be reverse-engineered. See also How does a good developer keep from creating code with a low bus hit factor?
User Stories, burndown chart, code
I'm a fan of the old 4+1 views:
Use Case view (a/k/a user stories). There are several forms: proper use cases, forward-looking use cases that aren't as well defined and epics which need to be decomposed.
Logical view. The "static" view. UML Class diagrams and the like work well here as a design document. This also includes request and response formats for various protocols. Here is where we document the RESTful requests and responses. This includes the REST URI design.
Process view. The "dynamic" view. UML activity diagrams, sequence diagrams and statecharts and the like for here for design documents. In some cases, simple narratives work well. In other cases, there's a State design pattern, and it requires a combination of class diagrams and statecharts to show how the stateful objects interact.
This also includes protocols (e.g. REST). Here is where we define any special processing for the various REST requests.
This also includes an authentication or authorization rules, and any other cross-cutting aspects like security, logging, etc.
Component view. The pieces we're building for deployment. This includes the stuff we depend on, the structure of the modules and packages, etc. This is often a simple component diagram or a list of components and their dependencies.
Deployment view. We try to generate this from the code as deployed. Since we're using Python, we use epydoc to create the API documentation. We also use Sphinx to import module documentation into this view of the software.
This also includes the parameters, settings, and configuration details.
This, however, isn't sufficient.
When projects start, you have to work up to this through a series of sprints.
The first sprints build just the use case view.
Subsequent sprints build an "architecture" to implement the use cases. The architecture document has 4+1 views, but at a high level of abstraction. It summarizes the structure of the model schemas, the requests and replies, the RESTful processing, other processing, the expected componentry, etc. It never has a Deployment view. We generally reference operator guide and API documents as the deployment view of an architecture.
Then design-and-construction sprints build (and update) detailed 4+1 view documents for various components.
Then release sprints build (and update) the deployment views.
From the project point of view, the most important documents are those that normally include the word Plan, such as the Project Plan, Configuration Management Plan, Quality Plan, etc.
What you are describing is common in process improvements, and normally responds to two major causes. One is that the system really is overeaching and getting in the way of real work being done. Another is actually answered in your question: it is not that the documents are only done for the sake of audits, and your focus should not just be how usefull is the doc for other developers, but for the project or the company as a whole.
One usually looks at things from it's own perspective, sometimes it's necessary to look at the general picture.

Resources