It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.
Closed 10 years ago.
I asked a question about a good book for this topic.
It's not getting many replies, so maybe there aren't any good books (although I find that strange; I must go search Amazon for myself).
In the absence of a book, maybe there is a good framework, some good URLs, or just general advice.
As I asked in the other question:
I am looking at several projects, all
with roughly the same functionality.
Some instruments collect some data (or
control some functionality). They
communicate by Internet
(Ethernet/wifi/GPRS/satellite) with
a database server which stores the
measurements and provides a browser
based means of querying the data,
producing reports, etc (and possibly
also allows control of the remote
equipment).
Can anyone recommend a good book
describing an approach to developing
such a software architecture, keeping
it generic, which tools, languages.
test methods, etc to use?
For "book", please substitute "framework, some good URLs, or just general advice".
This is a very common system pattern. Who can advise?
I have written a complete SCADA system (excluding the custom instrumentation hardware). The system was designed to be generic to allow the creation of new hardware models, instruments, and data collection - it was not written like so many SCADA systems for an individual company/plant but is used internationally for thousands of companies/plants.
I was the sole developer/designer with one member of management overseeing and guiding the project. It took longer that way, but it was doable. We looked at other SCADA specific systems/frameworks already out there and decided that since our units were custom, it would be easier and more flexible to write the system from scratch leveraging existing development frameworks and 3rd party components. Looking back, this worked out really well for us because we had the time and skill, but this is generally not the best solution depending on your business/contract model.
I am no longer with that company, however, they still use my software exclusively and I left on excellent terms. I would be happy to answer any general questions you have and to help point you in the right direction.
System Architecture
Here is a high-level overview of what the system consisted of:
Custom Cellular devices that had generic inputs to accommodate multiple instruments of varying types (analog, digital, pressure, amperage, floats, etc.)
UDP/TCP packets of custom format were sent by the units across the cell network (GPRS) to our servers (Windows Server 2003 R2). Information was sent regularly for reporting, and on customizable state changes that could be programmed at the device or online (the configuration sent over the cell network).
A custom Multi-threaded .NET application using TCP/UDP Listeners that grabbed incoming packets (Several Hundred-thousand a day), deciphered custom headers, and routed the packets without further interpretation to the correct database (Some clients required their own standalone system)
A Microsoft SQL 2005 database that acted as the brain for the entire system. Packets were interpreted using CLR functions and automatically triggered alarms (as configured), compiled reports, and kept a full history
A custom .NET application to handle alerts by placing phone calls, sending SMS messages, and sending emails. The phone logic was handled by an Intel Dialogic Card over analog lines using a combination of recorded prompts and Text-To-Speech.
3 ASP.NET sites:
Customer facing site that allowed them to manage their accounts/sub users, track alerts, configure units and alerts, chart data, map devices, export reports, etc.
Sales site that allowed the distribution of material to sales people, tracking of individual devices, device health reporting, etc.
Internal management site that allowed the creation of customer accounts, the configuration/build of units, and all other administrative features as required.
There was also a custom internal monitoring system to verify the health of the system and to alert technicians of problems as necessary since the system needed 24/7 uptime.
In addition we created an iOS app, a mobile site, and a custom web service/client (API) to allow the retrieval of customer data directly by customers to allow them to integrate our solution with their existing (usually custom) SCADA systems.
Those are the components we used and they worked. Doing it again I would change a couple of things. I would use Windows Server 2008 R2, SQL 2008 R2, and instead of a Dialogic card I would use Microsoft TellMe using VoIP. I would also use Silverlight instead of ASP.NET. I really like ASP.NET, but Silverlight can give a much better presentation and can be used outside of the browser as necessary - a common request from SCADA operators.
The sites all used 3rd party components so that charting, and tables didn't have to be written from scratch. There are some SCADA specific components (largely Java based) out there. However, we found most of them to be crude, ugly, or way too specific to be used in our generic system (also expensive! It was easier and more flexible to customize a gauge/charting package to "make" our own).
As mentioned, the brain of the system was the database. This was done because Microsoft SQL is a pretty awesome, well supported product designed for extreme uptimes with great backup and performance options. We were also really impressed with the .NET CLR integration that was possible allowing our custom .NET code to run as a part of that process. The units we were supporting came in a variety of models and could be configured to use any combination of instruments and so keeping the database flexible was key. We used a lot of normalization!
One thing that really helped was to use Recursive CTEs to fake the existence of data when the values were still the default. We did this to save space in the database, but this also enabled us to introduce a layer of abstraction in the database that allowed the queries to be flexible as well.
We had messed with OPC in the past but found it too inflexible, difficult, and irritating for our needs. That was a few years ago though, and I haven't looked at it since.
That is a long and very general answer to your question. I can't give you specific code or go into extreme detail since that information is property of that company, but I can answer some design questions and point you towards frameworks/tools we found helpful. My main advice would be to break everything down into separate components and employ the black box model on each so that individual components can be swapped out/improved as necessary. The scope of the project can seem overwhelming otherwise. Let me know if you have further questions or would like more information, good luck!
Related
I am developing an application which requires real time collaboration. I am planning to use a cshtml text area to allow the users to type. Is real time collaboration achievable using a text area?
Also, I have read a little about operational transformation. Can it be achieved using .net framework?
I am just a beginner and do not have much knowledge about algorithms that will help me achieve real time collaboration. Any help will be appreciated.
Thanking you in advance.
ShareJS is free, uses node.js to achieve what you are looking for, and implement a OT2 algorithm
For .Net there is no Operational Transformation out-of-the-box, however you can take a look to BeWeeBee SDK, (though is commercial software)
I am developing an application which requires real time collaboration. I am planning to use a cshtml text area to allow the users to type. Is real time collaboration achievable using a text area?
This really depends on the user experience you want to deliver. If you want to lock the textarea for one user whilst the other is editing then that might not be the nicest user experience but it's most definitely relatively easy to do.
If you want two or more users to be able to simultaneously edit the same text area then sending data_changed events between the users is reasonably easy using a realtime web technology but you'll need to handle the synchronisation of the textarea content between the users and handle collision detections. The user experience for this is also much more complex.
Also, I have read a little about operational transformation. Can it be achieved using .net framework?
I had to look up operational transformation and it partially answers the question about the user experience - it's non-blocking. Having skim-read the wiki doc I'd ask the question: why would it not be possible? You can communicate instantly between all users/application to notify them of changes (as stated: using a realtime-web technology) so you just need to implement and manage all the clever algorithmic stuff :) (I don't know if there's a component that will manage that for you)
For self hosted .NET realtime web technologies you might want to look at SignalR, XSockets, SuperWebSocket or WebSync.
If you want to get up and running a bit faster you might look at a hosted realtime web technology
This is an old question but there is some additional information that might be helpful. As previous answers mention, there are several options out there for text based data synchronization. Many of them based on Operational Transformation or CRDTs. These approaches are implemented in SDKs in many languages. (Full disclosure, I happen to be one of the authors of the Convergence).
However, you also need to take into account some of the other features required to implement collaborative editing. For example:
Presence: Who is there editing with you?
Collaboration Awareness: Things like shared cursors and selections?
Local vs. Group Undo: What happens when a user hits control-z? Are they undoing the last action they did, or the last action the other remote users did?
History: Knowing who did what is more complicated when multiple people are editing at the same time. When one user hits save (if there is a save) they may be saving actions performed by another user.
These are just a few examples of things to consider in collaborative editing beyond just data synchronization. When these questions come up, most answers focus solely on the data synchronization framework. At Convergence Labs, we help people work design collaborative editing applications and have implemented dozens of such apps. We have seen many times over that if all you put in is data synchronization, the user experience turns out to be pretty poor and users will not like the application.
So, in selecting a framework, look for something that helps you implement some of the other facets of real time editing, or at the least be prepared to implement them yourself on top of whatever tools you select.
Thanks for all the questions and responses posted on here. This site usually shows up whenever I search for information from google, and in many cases, the answers are usually relevant to the issues I needed solved.
I want to preface my question by stating that I've been programming (.NET, XML, T-SQL, AJAX, etc) for less than 2 years, and I still have a lot to learn; so, pardon my ignorance.
Here's my situation (and question): I'm building a social web application, which I know will have much traffic in a short time; as a result,
What are the basic information that I need to have, in order not to be overwhelmed? It's currently a one-man affair, and here is the hosting specification that I plan to start with: 2GB RAM, 600 HDD, 1000 GB bandwidth, and 2.13GHz Duo Core Processor.
I've read about web-farms, but I've never had an opportunity to use them, so I'm not entirely sure how to phrase this question: how can one split the same application on multiple physical servers? How do you make all the files act as one entity? And since every .net application requires a web.config, how is it split among the various files on these multiple servers?
I've built smaller projects before, but this is the first big project I'm building, and to be frank, I'm a little intimidated. So, I would like to ensure I know what I'm getting into before starting.
Thank you.
Based on your background I assume you are developing in a .Net environment? If so, I highly recommend you take a look at Windows Azure. Developing your app against Azure will allow you to deploy your app in Microsoft's cloud platform. Once deployed you can shrink and grow your resources according to demand without having to deal with the relative hassle of setting up multiple servers in multiple locations and managing it all. This allows you to pay for a "little bit" of server up front and if your app gets popular you can easily pay for "web farm" like power and geographic diversity. It also gives you a decent framework for developing an app that will scale relatively well. That's an 18,000-feet overview. If you can put some more details in your question I'm sure you will get more detailed responses. Best of luck!
Your "social web application" will not have any users if it isn't working and deployed. Don't worry about scaling much until the site actually does something useful and has a few hundred users (or at least a few dozen!). Get it working, find people around you who can help when the going gets tough, and keep at it. Otherwise your concerns about needing to scale will never be warranted.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 3 years ago.
Improve this question
We outsourced a web based portal and now we're not interested to work with them because the site is becoming more buggy day by day and increase of customers complain.
We've chosen a different team based on their local reputation and their portfolio are good to satisfy our urgent needs. We discussed this project with them and during a week they suggested some good ideas which help us to control. We are agreed to award this project to them. What I am thinking during the transition:
What documents do you think which can help new developers to understand the application? How many kinds of document I can request to them? If a new developers start working on it those documents help them to understand easily of all sides (application, database, configuration etc)
The application is on ASP.NET and SQL SERVER 2005 and the scariest part of all no source control tool is there. They do direct deployments without even push the publish button. Touch Luck :(
Thanks.
That's kind of hard to answer without knowing what kind of portal it is, but what comes to mind:
Owner's manual: Description of functionality, technologies used, full overview of all machines and services involved (don't forget data bases)
Backup: How and where is data backed up, where to restore it from in case of a crash
Description of all Databases used, relations between tables, at least quick rundown on what data is stored where
Links to any and all URLs to administration interfaces, tools, and scripts
Day-to-day operation: What cron jobs need to run frequently, are there caches, file lists or other things that need to be taken care of frequently
Make sure all domains used belong to you and are under your control
a description of the project's file structure (which part is where; where is the API; where are the visual elements; where are the front controllers)
How-To's on how to change the visual elements of the site (Style sheets, forms, templates...)
A description of any and all URL rewriting operations that take place in various parts of the systems, and where they point to
Which Google Analytics / Google Webmaster account is used and how to get hold of it
Ideally, an API documentation and full phpDoc style source code documentation
In addition to #Pekka's good answer I'd add the following
Functional Design Document (or
Business Requirements) - One that
explains how the application should
work from a business perspective.
Technical Specification (or
Architecture document) - One that
explains how the application was
developed from a technical
perspective.
Application Support
Guide - Some form of cheat sheet that
explains the common problems, service
accounts, batch schedules... etc
In addition to documentation you should be aware of the incident trends;
How may incidents?
How often?
How long do they take to resolve?
How many known defects are there?
Who maintains the infrastructure (patching OS, security audits, etc)
If you don't have enough technical resources to cover the daily number of incidents (keeping in mind there might be peak periods when usage of the portal is high) then you will probably find yourself in the same situation as your current service provider.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I'm facing developing a new web app in the future and I'm wondering how to decide what framework to use. I've settled on Python as my language of choice. But there are still may frameworks to choose from! More generally how do you choose between different similar technologies that are still in the works as the latest round of web frameworks are? I'm curious what your process is for deciding on technologies you've never used.
Recognize that no choice is perfect -- or even very good.
No matter what you choose, someone will have a suggestion that -- they claim -- is better.
No matter what you choose, some part of your tech. stack will fail to live up to your expectations.
The most important thing is "shared nothing" so that the components can be replaced.
After that, the next most important thing is automatically-generated features to reduce or prevent programming.
Look at Django. Lots of automatic admin features make life very pleasant.
There are a number of things you can do:
Download the frameworks and build something similar with them for comparison.
Look for comparisons by other people, but attempt to understand the bias of the reviewer.
Observe the community at work, see what people are building and the issues they run into when using the technology. Forums, blogs, mailing list etc are good places to check out.
Go to conferences and meet like minded developers interested.
You can also take the approach of using stable versions rather than alpha bits. After a while you might move closer the bleeding edge. People associated with the project in question are generally more biased than those approaching from other platforms, be careful who you trust.
Consider the impact of using a bleeding edge framework versus an established one. Sometimes it's important to your customers that you are on one perceived as stable. At other times this doesn't matter. How comfortable are you with fixing the framework itself? Great developers will learn the internals, or at least know enough to keep things moving whilst a bug is sent to the framework mailing list etc.
Consider some general best practices in building abstractions and reusable code on the python platform. You may be able to save yourself some work in moving to another platform. However, don't be a reuse junkie as this can limit the effectiveness of your use of the framework. The 37Signals guys are right when they talk about extracting frameworks from working code rather than building frameworks from scratch.
I know this is an old posting, but I am in a similar situation (again) and I think there are other people who may want to look for different opinions, and hear of (somewhat) successful experiences.
Since baudtack mentioned Python, I will try to answer this along the lines of my experiences using Python. Here is what has been working for me:
determine the scope of your project - outlining what your application is supposed to be able to do without introducing any programming or design notes will clarify your goals greatly
determine how you would like to work with your code, stack and data:
a. what sort of programming paradigm do you want to work with? i.e. object-oriented, functional, etc. do you want to play to your programming style or do you want to follow somebody else's programming style?
b. use semantic web or not? do you want greater control over URIs and their design? (I found web.py great for this by the way - It is my choice to create REST APIs in Python)
c. do you want to be trapped by framework requirements, or do you want a better separation of the application from the web component, i.e. use a framework to utilize your application as a set of modules, for example. My problem with Django was that I ended up not programming Python, but having to learn more Django than I needed to. If that works for you, then that is the way to go.
d. data stores... some sort of SQL vs. non RDBMS (xml databases like eXist-db with full xquery support) vs. OODBMS vs. a combination of the above? how complicated do you need this to be? how much control/separation do you need to have over how data gets stored and recalled in your application?
e. testing: unit tests... thank goodness for python! if your web app has the potential to grow (as they often do), having a sane and coherent testing platform to begin with will help out a lot in the future - I wish I had learned about this earlier on. oh well... better late than never.
f. how much control over the server do you need? hosting considerations? how much control over an Apache instance do you need to have? OS specific needs? I found that using shared hosting providers like Webfaction has been great. I eventually found I needed greater needs for flexibility and bandwidth. In other words, what can you get for your budget? If you have USD50 to spend each month, it may be better to consider a virtual hosting solution like Linode....
Finally, I echo S.Lott's sentiments that no choice for a solution is perfect, and are subject to obsolescence.
Experience trumps hearsay. I've found that prototyping is a huge help. Make a prototype that uses the features you expect to be the most important for various frameworks. This helps route out any features that may not work "as advertised."
In general though, kudos for being willing to look at new technologies.
I have a set of criteria in different categories:
Activity & Documentation
Is there an active user base?
Is there an active development base?
Is the support responsive and information accessible?
Are there user and development guides and reference material?
These are essential, there needs to be traceability of all of these to build confidence in the solution.
Ease of use
Are basic features easy and complex features possible? I typically give a new framework a test drive and try to roll out a set of use cases to see how intuitive the framework is to use.
Is installation intuitive and simple for a local/dev installation and production deployment?
How is it backed up and upgraded?
What is the effort and UX for implementing a "Hello World" type blog post, static page, menu item, and plugin?
How are versions dealt with for the core & plugins?
Example (on the topic of Automated Testing/Continuous Integration solutions)
Several years ago I evaluated several Automated Testing solution. At the time Jenkins and TeamCity were front runners and in the end I chose TeamCity because of the UX, active user & development base and quality of accessible documentation.
Example (CMS for a blog)
This criteria is also why I prefer to use Wordpress over other options. While wordpress has its shortcomings, the user and development base is strong and active which leads to a software architecture with more potential to evolve over time and maintain its relevance and a development community that provides quality plugins and themes to choose from.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
I have been looking for a collaborative tool for developing functional specifications. I am looking for the ability to:
Have multiple users contribute to the specification.
Provide some form of traceability, which could be done manually if needed.
Provide users with the ability to add comments and notes.
Upload and display Visio documents
Upload and display mockup screens using Balsamiq Mockup.
My initial impression is that using a wiki could be a good tool for this task. Does anyone have experience with using a wiki for creating functional specifications? What would be the pros and cons to using a tool like this as opposed to a requirements management tool?
Your input is greatly appreciated!
It's possible to do what you describe, to develop requirements in a collaborative way, in spite of using a wiki. Nothing about the wiki paradigm assists in this process.
I managed a wiki on the Zend Framework project to track proposals for components. They're still using it. Proposals are different from functional specifications, but the usage is similar enough to your question that I think this is relevant.
A wiki doesn't take care of itself. Unless you have someone responsible for managing it and making sure there is some structure and consistency, it quickly becomes a mess. The real-world analogy would be to hand each of your team a blank sheet of paper and tell them to write up their part of the requirements. Problems with this are:
Every contributor has to make up their own document structure, and write about different things in a different order. So it's impossible to compare one page to another.
There's no "index page" to organize all the disparate contributions. No one wants a page to "fall through the cracks," but in a wiki that's the default destiny of any page as soon as it's written.
There's no feedback loop to make sure the writing actually gets done.
The way to make it work is to apply some process to the project, and use the wiki in accordance with that process.
Give people the ability to create a new page in the wiki, but only through an interface that automatically links the new page into the right index.
Define a lifecycle for documents, so they are sure to be drafted, reviewed, and approved at the appropriate stages.
Provide a template for a new page. Provide the section headings that you need in each of these pages, and make part of the review process a confirmation that head section has been filled out adequately.
"What would be the pros and cons to using a tool like this as opposed to a requirements management tool?"
While it seems like a great idea, what you run into are people who can't and won't write.
People who can't write -- well -- can't write. They can't communicate with an email or a wiki or any medium outside voice.
Some people are "disorganized". Actually, writing is too linear and they don't think linearly.
Some people don't get the "write to your audience" and write stuff that's incomprehensible.
Sometimes you can't even figure out what they're talking about, much less what they're writing about. They talk in jargon or code. They don't know much but insist on being heard.
Some people won't write.
Some people refuse to make commitments. Even in a wiki where it can be retracted. They feel they must pre-discuss everything.
Some people are in the habit of doing everything by giving direction to someone else. They either don't write for themselves, or, they make people stand around in their office and listen to them talk and type.
Some people are generally toxic on any kind of project. They spring new requirements at the last minute. Their first response is "that will never work". They don't brainstorm well. When they say it work work, and you beg them for an improvement, they don't have one. They just know it won't work.
My experience is that only programmers can use a Wiki successfully. And only senior-level programmers.
N00bz don't have enough experience to sort out requirements from design from rumors and management fluff.
N00bz don't always have the language skills to write clearly. They may eventually, but one look at their Javadoc comments indicates that they're struggling with the "clarity" part of writing.
It's very appealing. I'm hoping for people to get better at using wiki's because I think it could have a lot of advantages over more traditional approaches where one person interviews everyone and writes things down. But it requires a level of confidence and skill in communication that few people seem to have.
Consider researching Fog Bugz. They tout themselves as the best of the
best for project management. Considering Joel's history I'd give them the
benefit of the doubt. They use a wiki in the way you've just described.
I would suggest signing up for the free trial, if you're serious.
Depending on the size of your project, buying it might be a very good
option.
At very least you could look at how they've structured it, or read the
forums for ideas on how to build your own stripped down version
Specialist tools help keep things on track and introduce a fixed work-flow. This is kind of the point, keeping things focused and functional. Using generic tools like a Wiki might be great for a bunch of programmers but introducing one for 'mixed-mode' work might be bad:
Things can creep and get off-track quickly
Communication can be lost in the medium
Look at things like Basecamp. They can be thought of as an applied wiki, or collaborative tool. A generic tool for specific purpose needs refining. I don't know if MediaWiki or others have enough customization to keep things clean and focused.
Maybe gather the requirements for your requirements management tool (recursive I know) and what aspects (communication aspects) you can take from the wiki culture and an open-communication mindset. If neither requirements management tools or wikis fit the bill, look at building one. Might be the next big thing. It feels like saying Could I use a wiki instead of Bugzilla?
A fixed work-flow webapp for requirements management with an open-communication emphasis that allows people from many roles to see and understand might be good!
We have used TWiki and now FosWiki in that context. There are many things one gets for free (version control, access control, Web-base access, searches, remote management, security patches, ...). In a few minutes, one can define:
a table defining requirements attributes,
which creates interactive forms with field selection and validation (where you can also document discussions and rationales, embed images, attach documents, link to other requirements...),
and then queries on these "requirements" and show them as tables that can be sorted, filtered, printed, reported against, etc. (e.g., http://jucmnav.softwareengineering.ca/ucm/bin/view/ProjetSEG/JUCMNavRequirementsVer2).
Obviously, one can easily use hyperlinks and Wiki links along the way. FosWiki also has features that can be used to enforce specific workflows, if needed. It is also easy to support forms for use cases and other paradigms (we have done this in the past, and that works generally well).
Wikis such as FosWiki are extensible and one could develop further modules for addressing weaknesses related to traceability management and impact analysis, table-based modifications of requirements, overall packaging, etc.
As a middle ground between a free-form wiki and a requirements management tool, consider using a structured wiki like Foswiki. I haven't done any formal requirements management (with a wiki or otherwise), but I have used TWiki (the predecessor to Foswiki) for other tasks, and it permits you to define a workflow, form fields, and so on as you need them, while keeping the flexibility of a wiki when you don't need a formal structure.
I agree with all (most) of the people above - use of a wiki may sound ok, but wiki's are meant to be present information and be updated as needed, not to be used as an interactive project management tool. I would strongly suggest SmartSheet (I'm a strong advocate of the product) - it provides a spreadsheet like interface where you can store multiple files per row/task, send out automated updates to users and maintain specification revisions...
The other approach could be the use of Google email, docs and calendar - a free friendly way of team interaction....I would shy away from issue/bug tracking tools for project management - they tend to have differ on focus: PM tools on the entire project/resource/timeline and Issue tracking tools for specific entered issues....
This may be a bit old now, but I am currently using Atlassian's "Confluence" and it's been great. I currently work as a UX engineer playing the role of "Product Owner" within an Agile process. I can document requirements for discrete interfaces, allow for multiple users' feedback and comments, and intertwine each interface with other interfaces within a larger context (e.g. user story). Everything is very dynamic and template driven. It's suiting my current needs very well.