From Heroku's information page on the read-only file system
"There are two directories that are writeable: ./tmp and ./log (under your application root). If you wish to drop a file temporarily for the duration of the request, you can write to a filename like #{RAILS_ROOT}/tmp/myfile_#{Process.pid}. There is no guarantee that this file will be there on subsequent requests (although it might be), so this should not be used for any kind of permanent storage."
Does anyone know how often files are deleted from the /tmp folder in Heroku?
heroku never explicitely removes files from your tmp folder.
However it is not shared between instances of your application (your dynos).
This means you can assume the tmp folder to be emptied every time you deploy your application.
As you should always be able to deploy, you need, for your own sake, to architect your app with that in mind and not rely on the tmp folder to keep files longer than the user's HTTP request.
Related
I am searching for a backup strategy for my web application files.
I am hosting my (laravel) application at an ubuntu (18.04) server in the cloud and currently have around 80GB of storage that needs to be backed up (this grows fast). The biggest files are around ~30mb, the rest of it are small jpg/txt/pdf files.
I want to make at least 2 times a day a full backup of the storage directory and store it as a zip file on a local server. I have 2 reasons for this: independence from cloud providers, and for archiving.
My first backup strategy was to zip all the contents of the storage folder en rsync the zip, this goes well until a couple of gigabytes then the server is completely stuck on cpu usage.
My second approach is with rsync, but this i can't track when a file is deleted / added.
I am looking for a good backup strategy that preferable generate zips before or after backup and stores them so we can browse and examine back in time.
Strange enough i could not find anything that suits me, i hope anyone can help me out.
I agree with #RobertFridzema that the whole server becomes unresponsive when using ZIP functionality from spatie package.
Had the same situation with a customer project. My suggestion is to keep the source code files within version control. Just backup the dynamic/changing files with rsync (incremental works best and fast) and create a separate database backup strategy. For example with MySQL/Mariadb: mysqldump, encrypt the resulting file and move it to an external storage as well.
If ZIP creation still is a problem, I would maybe use a storage which is already set up with raid functionality or if that is not possible, I would definitly not use the ZIP functionality on the live server. rsync incremental to another server and do the backup strategy there.
Spatie has a package for Laravel backups that can be scheduled in the laravel job scheduler. It will create zips with the entire project including storage dirs
https://github.com/spatie/laravel-backup
I'm currently using H2O steam version 1.1.6 to deploy model endpoints which is working great!
However, steam uses the /tmp directory to store these deploys, which is actually only meant for temporary files. Because the /tmp has been cleared on my server, I've lost some deploys.
Is there a way to change where these files are stored?
Additionally it's also not possible to delete the deployments through the steam UI because the files are gone, is there a way to delete these as well?
I was wrong; the deployments can be found in steam/var/master/model
The files and directories that are located in /tmp are created by jetty.
I have a rails application that is running cron jobs and generating reports in the form of csv files. My question is what is best practice on where I should store these files before sending them to an sftp site? My thinking is as a tempfile or in the root and then delete the file after its sent?
There's no common place for this in a rails app file structure, so it's really up to you. As long as it's a known place on the file system then it should be fine. However, a couple of pointers:
Avoid storing them in the OS's temporary directory (or rails' tmp directory), as these are cleared in certain cases.
If you're going to use capistrano to deploy your application then it's probably best to keep the files in a directory that's outside of the rails app altogether, as a deployment will swap the app directory with a fresh copy. If this is a problem, and you're determined to keep the directory within the rails app, then you will have to put it in the shared directory that capistrano creates and create a symbolic link/shortcut.
suppose if i open my heroku webpage, it update a file, like a database.
now i want to retrieve it.
i tried git pull, when done, i checked, it is the old file what i pushed last time.
i tried heroku run bash and "cat"-ed the file, it gives old outputs. :/
but i can assure, the file is getting update, coz if i output the file content through server, like if i request for a particular path on my address, it will show the contents of that file on browser, then it shows updated data.
i have no idea why is this happening. any clue ?
i am using python3 with wsgiref module.
You shouldn't use the dyno filesystem for persistent file storage (like databases). The dyno filesystems are ephemeral and changes are not reflected in the git repository associated with you app. Use one of the data storage add-ons instead: https://addons.heroku.com
I'm using Python/Django on Heroku (Cedar Stack) and I've got a management command that I need to write that will pull a file out of an S3 bucket and process it. I'm not sure I understand how to use the ephemeral filesystem. Are there only certain directories that are writeable? I found an other article that implied that there were only certain folders that were writable (but, it doesn't seem to apply to the Cedar stack). I found this dev article but it doesn't go into much detail (note: I do understand that it's just temporary. I only need to unzip the file and process the file). Can I just create a folder anywhere under the application's root? And how would I get that? It seems like I could probably just use $HOME. I did a bit of testing by connecting to via
$ heroku run bash
and running:
$ echo #HOME
returns:
/app
and running:
$ mkdir $HOME/tmp
creates a folder in the app's root and gives with the same user and group as the other files and folders.
So... anything I'm missing here? A better way to do it? Is there an OS environment variable for this? I've run "env" and I don't see a better one.
To really understand the ephemeral filesystem, you need to understand what a dyno is. You can read more about how dynos work. In a nutshell, though, a process runs on Heroku in a virtual machine with its own filesystem. That virtual machine can stop for a number of reasons, taking the filesystem along with it.
The underlying filesystem will be destroyed when an app is restarted, reconfigured (e.g. heroku config ...), scaled, etc. For example, if you have two web dynos, write some files to the ephemeral filesystem, and scale to three dynos, those files will be destroyed because your app is running on new dynos.
In general, the ephemeral filesystem works just like any filesystem. Directories you have permission to write to, such as $HOME and /tmp, you can write files to. Any files that require permanence should be written to S3, or a similar durable store. S3 is preferred as Heroku runs on AWS and S3 offers some performance advantages. Any files that can be recreated at will can be stored on the dyno's ephemeral store.
You can create a file under the '/tmp' directory, and that file will be destroyed after the request is complete. I'm doing this on Cedar, and I haven't had any problems.