Can anyone comment on the performance implications of storing streaming media in a SharePoint 2007 document library? I’ve heard this can be detrimental to the performance of the farm due to the media being streamed from storage in a SQL DB.
Has anyone had any firsthand experience with this and if so, what alternatives have you used to provide users with the ability to publish and mange their own video content? Assume a secure internal environment so external services like YouTube are not viable in this scenario.
I have tried this on a test deploy and it had very poor performance. Not only did the SharePoint server struggle, but the video the client was trying to stream was very laggy. Granted, we did not have a state of the art server set up, but I was the only one accessing the server and it couldn't even handle that. Given my experience, I would advise against it.
I can stream FLV's for flash movies from a document library with reasonable performance. I still opted for deploying them to a separate non-sharepoint website because the video's where fairly static and do take up a lot of SQL space.
You might consider activating blobcaching to get around the streaming from the database, see: http://msdn.microsoft.com/en-us/library/aa604896.aspx
Related
I had an architecture question, and I had to rewrite the question title multiple times, since SO asked me to. So please feel free to correct it, if you feel so. I am not an expert in cache related things so I would very much appreciate some insights about my architecture related question.
So the situation is like this. We have a web based design app (frontend Javascript, backend PHP) which presents lots of clipart images to our customers who use that in creating online art work. Earlier, our app was loaded into an AWS machine and we used to have the clipart images also stored locally in the same server in order to not have any network transfer required to load the clipart and thus make the design app load time faster. The customer created designs were also saved into a backend MySQL server connected directly to the web based design app (in JSON and relational model).
A while before a new team joined to make a mobile version of this app, and they insisted that the cliparts should be loaded from a "central location" both for our web app, and for the mobile app they are creating. They also said that the design should also be stored into a "central database", accessible by the web and mobile apps (and there were some major re-architecting of the JSON structure as well)
So finally, the architecture changed such that, the cliparts now reside in a centralized location (S3 Server). And there is an "Asset Delivery and Storage (ADS) System" to which our design app makes requests for clipart images and gets served. (Please note that the cliparts repository is very large and only a subset of clipart images are served based on various parameters - such as the style of the design, account type of the customer etc). So this task is now done by the ADS system (written in python).
And since our web design app no longer has any local storage of cliparts nor logic of cliparts filtering (which got delegated to ADS, so no more server side PHP), it has also become a purely web based (front end Javasdcript) app without any server requirements and subsequently got moved to S3.
Now the real matter is that, our web app seems much more slower when initial loading, than when we had our on stash of cliparts stored in the server. I read that if an app requests for images, those images are cached in the browser and if the customer, for eg, loads the same order before that cache has expired then there is no repeat request that needs to be sent to the server (in this case ADS).
If that is true, is there any case I can really make to state that moving the clipart images from the design app server to the ADS system and having to send a request and load them every time a design is loaded has contributed in part to the recent slowness of the design app?
Also most times I hear the answer that "mobile app also does the same and is faster".I am not a mobile developer. Could there be some mobile cache tricks that help the mobile app to be much more "cache-efficient" than the purely web based design app, such that even though the architecture is same for both (sending request ADS for cliparts), the mobile app does it in a better and more efficient manner?
End note: I realise I am not asking a specific programming question. But from some of the notes I have read here, SO is a community for programmers, and I do not know of any other community that so well answers programming related questions. The architecture question I have is a genuine programming related question I face at work and sadly I am not skilled enough to understand if all the recent architectural changes there has any drawbacks that is causing our web app performance to degrade noticeably.
Thanks for reading, and I would really appreciate any pointers or even links to reading for better understanding this.
In chrome, open up the developer tools, and click on the network tab. 90% of the time you can identify the slow resource from there.
After many hours of research and nothing relevant coming up I decided to ask.
I am pretty new to the concept of video streaming, so please forgive me if my questions may seem elementary.
I am building a project that needs to include media streaming functionality. It should has the following options:
VOD - user uploads a file to the server, that needs to be transcoded to few MP4 files of different resolutions. For transcoding I am trying the approach using CloudTranscode (https://github.com/bfansports/CloudTranscode) deployed as a Docker image. The server should supply stream to the player with certain buffer size, so when the playback is paused we buffer for instance +5 seconds and that's it. Adaptive bitrate would be nice, however I'm not sure how this works with different players (I was thinking about using Video.JS due to high customization option, plus it's free).
Live video capturing - user visits a certain page that captures video from the webcam and sends the stream to the server for further stream distribution to clients. For most browsers WebRTC could be a good option, but iOS devices probably won't work with it, so any suggestions here would be much appreciated
Live video streaming - users visit a certain page where they can watch the stream captured from the user mentioned in point 2. Here the stream may be watched by one or many users (may be as well 1 or 10,000 users)
Cutting to the chase my questions follow:
What would be the best media server software that I can use for that purpose, having on mind high scalability (deployed as Docker container on AWS EC2), and possible huge load of both streaming and watching users, as well as multi-device/platform/browser support?
What would be the best media player for webpage that (again) would be cross-browser/platform/device, keeping in mind good integration with media server itself for purpose of adaptive resolution streaming? Also it would be nice if the player has broad customization options in matter of appearence (for instance thumbnail display when hovering the timeline).
Do you know any better solution for video transcoding than mentioned CloudTranscode, having on mind Docker setup, and some easy to use API (here some on-the-fly transcoding would be nice, so the worker wouldn't need to wait for the whole file to be uploaded)?
What happens if I use autoscalling functionality on EC2 instance, and more instances of the media server are being automatically started? Let's say we have instance 1 (I1) and instance 2 (I2). Some user started broadcasting on I1, and 1000 users are watching the stream which is the server instance's limit because it's running out of resources. Next, another couple of users are trying to view the stream, so they are being connected to I2 by AWS load balancer - how does that work with live stream? Sorry, but I am total newbie to the concept, so again - forgive me for elementary questions.
So far a was able to find a few media servers that may be relevant to my needs including:
Wowza Media Server (paid)
Red5 media server (free)
Kurento Media Server (free)
My application is written in Laravel, ergo I need some PHP integration with the media server.
Obviously free solutions are the most welcome, however I do not mind to pay as long as paid solution covers my needs.
Any input here will be much appreaciated - even partial solutions / suggestions. I'm kinda stuck here, so any suggestions that can bring me closer to the solution are very welcome!
Best regards
If anyone needs such information I ended up using Nginx Plus media server functionalities. It's capable of serving both live and VOD streams, it has out-of-the-box load balancer to switch traffic over multiple container instances and many more great features. Plus they have images to deploy directly from AWS Marketplace, and the license is paid hourly when the EC2 instance is running. Ofcourse there is free version as well, but I am really satisfied with Nginx Plus support.
Capturing live stream from user I've done using getUserMedia() in JS. Still having minor glitches, but I will get it to work (problems are related with WebM chunks that MediaRecorder API spits out, but I'm almost done here using some Python piece of code modifying each chunk on server side).
If anyone needs help I will be happy to help.
I'm giving users the ability to attach images, videos, audio clips and other file attachments inside an existing web application. Some installs of our product have thousands of users so the volume of data will get very high.
Amazon S3 is the obvious solution but, due to legal reasons, cannot always be used. I need a solution that my customers can host themselves.
I'm therefore looking to build or adopt a file storage system with the following traits:
Not hosted. Must be installable on my customers' Windows servers.
Horizontally scalable to terabytes of storage.
Similar operation to S3 such that I can make both approaches part of my product.
Proven architecture
I've seen several suggestions for this on StackOverflow and other forums (Eucalyptus Walrus, Hadoop HDFS, MongoDB + GridFS, CouchDB, MogileFS) but couldn't find enough information to identify one as simple and proven.
I have experience with CouchDB and would jump on it if I could be sure that it'll do well with terabytes of video files but I haven't found a good success story to lean on.
The closest open source project is OpenStack swift (https://github.com/openstack/swift). It power's RackSpace CloudFiles.
While S3 design is not exactly known OpenStack is built to give very similar functionality.
It is scale-out, have no single point of failure or bottlenecks.
I am developing a eCommerce website in ASP.NET MVC 3 in C#. Using SQL Server 2008R2. My question is if I have 5 images that I want to show in gridView with thumbnails (e.g. something like Amazon website that gives customers couple of pictures to show) would it be advisory if the images are coming from the database or should I reside in the Content\Images folder? There are quite a few sub-categories in sub-category in my db design. What is the most common suit for a professional developer to follow? Thanks. I know there are few options for third party tools like jquery & Telerik Extensions. So I will use them.
Thanks
From my experience and research it is better to put it in a folder/content structure. Yes, there are security things with opening directories to the public but if you instead upload a file via ftp dynamically the problems are solved. I have heard of horror stories about storing files in database and have seen the issues come up but have resolved them. Basically, it is easier to write to database and there are not the security issues of opening up a directory to public but just make sure to regularly check backups that the files are not corrupt or make sure the data is on a fail over cluster where that will never be a problem.
So summary: Database is fine just regularly check backups by restoring them that they are not corrupt or run as a fail over cluster. Otherwise just go with the typical folder/content structure but use ftp to upload the file so there are no open directories to the public.
For me, the best anwser to this question is this: To BLOB or Not To BLOB: Large Object Storage in a Database or a Filesystem
Sumary: Application designers often face the question of whether to store large objects in a filesystem or in a database. Often this decision is made for application design simplicity. Sometimes, performance measurements are also used. This paper looks at the question of fragmentation – one of the operational issues that can affect the performance and/or manageability of the system as deployed long term. As expected from the common wisdom, objects smaller than 256K are best stored in a database while objects larger than 1M are best stored in the filesystem. Between 256K and 1M, the read:write ratio and rate of object overwrite or replacement are important factors. We used the notion of “storage age” or number of object overwrites as way of normalizing wall clock time. Storage age allows our results or similar such results to be applied across a number of read:write ratios and object replacement rates.
I'm wondering how those popular media rich websites implement their media library. Do they store all the media files in the database? What kind of database do they use? Do they employ other mechanism to boost the performance?
Thanks for any response.
"Popular Media Sites" is pretty broad, but typically high volume rich media sites use Content Delivery Networks, such as Akamai, etc or cloud based storage like AmazonS3
You are asking a very difficult question to answer.
I recommend that as an introductory read you check out Youtube Architecture on High Scalability. Youtube is a very good real-life example of how a media-centric website works.
Surprising as it may be, serving the actual media files is not the bottleneck. The harder part is getting all the media meta-data synched, generating thumbnails, etc. Media files can always be hosted from a cluster, or from a CDN in case of an extremely popular video.
Read the link for more in-depth info.
Speaking as a dev on a popular media website, we offload the serving of video to 3rd parties (YouTube and BrightCove). Depending on the situation, we then host this video within a custom player to layer in ads and other features. Pumping down video streams is best handled by someone else who has invested a lot of energy and money into their architecture.
As with everything, you need to decide if you needs are specific enough to warrant developing in-house vs effort of integrating with other tools.