I have a client who wants to build a huge Image gallery website, and I am confused about how to structure the website for future Storage expansion.
Let me explain more...
Let us say that each user will upload his images to
website.com/Uploads/User/Images
Now creating the upload logic and displaying the images is not my issue here, my real problem is that say I have 200 GB hard Disk and if i have 20000 Customer where each client uploads 10 MB max, now as you see I will run out of space.
So how do I handle expansion in future without changing structure of web site, meaning that users will always upload to the same Path I have mentioned above, so obviously my front-end views will fetch images from same location too.
It may be stupid but I am lost on this. I mean, how guys like Facebook or other big sites do that ?
You can try using a cloud cdn(content delivery network), which will be dynamically expandable. amazon/rackspace, they are well known for this kind of service.
Ok after Tedious Searching, i have found the answer, basically it boils down to two Methods,
One Called Push, where you have to store the Files on the CDN Server by FTP, Api, etc...
The other called Pull Origin, where you dont have to change anything, you just Configure Your CDN to Fetch the Resources from your Servers, of course you have to Store files on Original Server First.
there is a lot more to it, but if anyone had my same wondering , just make Search about CDN Push or Pull in Google
Related
I'm fairly new to .net core but feel I now have a decent grasp of it but I am struggling with a problem that must have been encountered and solved before yet I can't find any tutorials, answers or pages that deal with it.
If I have a website where users can upload their own photos and those photos need to be private to the user how to I deal with <img> elements on pages? I can deal with uploading and storing the files. My problem is once I have the files on the server how to I service <img> elements that request those files. There are two parts to my question/problem:
Ideally I want a simple url (a) that doesn't reflect the physical storage location and (b) where the url has an id for the image - ie /files/images?id=123 rather than the url reflecting the directory structure of the server. The first problem is how do I intercept elements and map the images url to physical location and return the actual image file? ie what does look like from the servers perspective (how do I get the request) and what is the code that will intercept it (ie is it middleware)? Does .net core have any inbuilt mechanisms/features that can route the image requests (it seems like it is such a common requirement I would have thought that there were features to do it)?
When doing (1) how do I authenticate requests so that I only deliver images to authorised users particularly when it may not be a simple 1:1 relations (ie a user may give access to a particular image to multiple other users so I will need to verify access rights).
Thanks
I have a Magento website and I have been noticing an increase in warnings from Catchpoint that various images, CSS files, and javascript files are taking longer than usual to load. We use Edgecast for our CDN and have all images, CSS, and JS files hosted there. I have been in contact with them and they determined that the delays happen when the cache for the resource has expired and it must contact the origin for an updated file. The problem is that I can't figure out why it would take longer than a second to return a small image file. If I load the offending image off our server (not from the CDN) in my browser it always returns quickly. I assume that if you call up an image file directly using the full URL to the image file (say a product image, for example), that would bypass any Magento logic or database access and simply return the image to you. This should happen quickly, and it normally does, but sometimes it doesn't.
We have a number of things in play that may have an effect. There are API calls to the server for various integrations, though they are directed at a secondary server and not the web frontend. We may also have a large number of stale images since Magento doesn't delete any images even if you replace them or delete the product.
I realize this is a fairly open ended question, and I'm sorry if it breaks SO protocol, but I'm grasping at straws here. If anyone has any ideas on where to look or what could cause small resource files, like images, to take upwards of 8 seconds to load, I'm all ears. As an eCommerce site, it's getting close to peak season, and I can feel the hot breath of management on my neck. Any help would be greatly appreciated.
Thanks!
Turns out we had stumbled upon some problems with the CDN that they were somewhat aware of and not quick to admit. They made some changes to our account to work around the issues and things are much better now.
I'm depending on my users uploading image links, not image files. So my site actually requests this links from these external sites by including them on an element and getting the src from the database.
Could sites detect they are receiving lots of "requests" (no idea if it's the right term) and block your url from doing more requests?
If they can, how many times would go unnoticed? Also are there policies like having to add a link to the source or something?
Sites can detect and block your IP address, especially if you are sending a lot of requests that could slow down their site and cause performance issues on their end. Having users post links shouldn't be any issue though.
Short answer: yes, hotlinking to files such as images can be detected and blocked. How many times can you do it? That depends on the limitations set by the website hosting the files you are hotlinking. There isn't really a definitive answer for every server about the limitations.
I am working on a ecommerce site built on the top of Nop Commerce 2.3. We want to use CDN for loading all static contents including its images, but not sure how to do this with NopCommerce.
Nopcommerce is set to save binary of images in db at the time of inserting product, and then it generates thumb or re-sized / optimized images at the run time as and when required and stored them in the content folder of the same application for retrieving on page during load time.
Now, suppose on some page, lets say, Home Page, we have 70 product images. I want to distribute it across four host name, so each host name will serve 17/18 images.
This is definitely to save some time in image loading.
Now the Question is:
How to do it in best way in NopCommerce?
The challenges are:
Changing in nop commerce code to load images from CDN instead of its application\content folder. This is not an issue and is fine to manage.
To implement this correctly we might need some mechanism that checks for image on CDN if it doesn't exist, then we might need to transfer the image from content folder to appropriate folder at CDN maybe ? (suggest), and if it doesn't exist in content folder, then need to generate suitable image first and then transfer it.
I'm concerned about this 2nd challenge, and wants to understand the best approach to do this. Moreover, how to do this... specifically check if image exists in CDN or not?
Not very much sure, how to do this? And is it okay or do you suggest something else?
If you use the OVH CDN, you simply point your dns to the CDN dns, add the static file's extensions to the CDN configuration, and let it work for you. All other extensions will pass through. No code to change.
We host our Magento store internally with a not so amazing Internet connection. It's good enough to process the shop, but images take a while to upload to the visitor.
So what we have done is tweak the image.php file in app/code/core/Mage/Catalog/Model to point the images to a public space on RackSpace.
It's not an ideal method though with the admin console complaining that images don't exist and other random mess ups.
So I was wondering if anyone has implemented such a solution which I can use for our shop?
I imagine it will need to ignore the cache check as it cant upload to rackspace.
You can use the option of the backend of Magento
And redirect the trafic of images to another server(CDN).
There is one more solution, for example the module of one pica CDN with the option to select many CDN( amazon, custom cdn, etc)
http://www.magentocommerce.com/magento-connect/onepica-imagecdn.html