I have a rails application that is running cron jobs and generating reports in the form of csv files. My question is what is best practice on where I should store these files before sending them to an sftp site? My thinking is as a tempfile or in the root and then delete the file after its sent?
There's no common place for this in a rails app file structure, so it's really up to you. As long as it's a known place on the file system then it should be fine. However, a couple of pointers:
Avoid storing them in the OS's temporary directory (or rails' tmp directory), as these are cleared in certain cases.
If you're going to use capistrano to deploy your application then it's probably best to keep the files in a directory that's outside of the rails app altogether, as a deployment will swap the app directory with a fresh copy. If this is a problem, and you're determined to keep the directory within the rails app, then you will have to put it in the shared directory that capistrano creates and create a symbolic link/shortcut.
Related
I am searching for a backup strategy for my web application files.
I am hosting my (laravel) application at an ubuntu (18.04) server in the cloud and currently have around 80GB of storage that needs to be backed up (this grows fast). The biggest files are around ~30mb, the rest of it are small jpg/txt/pdf files.
I want to make at least 2 times a day a full backup of the storage directory and store it as a zip file on a local server. I have 2 reasons for this: independence from cloud providers, and for archiving.
My first backup strategy was to zip all the contents of the storage folder en rsync the zip, this goes well until a couple of gigabytes then the server is completely stuck on cpu usage.
My second approach is with rsync, but this i can't track when a file is deleted / added.
I am looking for a good backup strategy that preferable generate zips before or after backup and stores them so we can browse and examine back in time.
Strange enough i could not find anything that suits me, i hope anyone can help me out.
I agree with #RobertFridzema that the whole server becomes unresponsive when using ZIP functionality from spatie package.
Had the same situation with a customer project. My suggestion is to keep the source code files within version control. Just backup the dynamic/changing files with rsync (incremental works best and fast) and create a separate database backup strategy. For example with MySQL/Mariadb: mysqldump, encrypt the resulting file and move it to an external storage as well.
If ZIP creation still is a problem, I would maybe use a storage which is already set up with raid functionality or if that is not possible, I would definitly not use the ZIP functionality on the live server. rsync incremental to another server and do the backup strategy there.
Spatie has a package for Laravel backups that can be scheduled in the laravel job scheduler. It will create zips with the entire project including storage dirs
https://github.com/spatie/laravel-backup
I'm currently using H2O steam version 1.1.6 to deploy model endpoints which is working great!
However, steam uses the /tmp directory to store these deploys, which is actually only meant for temporary files. Because the /tmp has been cleared on my server, I've lost some deploys.
Is there a way to change where these files are stored?
Additionally it's also not possible to delete the deployments through the steam UI because the files are gone, is there a way to delete these as well?
I was wrong; the deployments can be found in steam/var/master/model
The files and directories that are located in /tmp are created by jetty.
I'm trying to upload images to my Phoenix app on Heroku. I have a simple app that follows the instructions for file uploading from Phoenix's docs.
I have a simple form and the controller uses File.cp_r() to copy the file from the temp directory.
def create(conn, %{"user" => user_params}) do
if upload = user_params["photo"] do
File.cp_r(upload.path, "priv/static/images/foo.jpg") #non-dynamic name, doens't matter
end
...
end
Works just file on my local. However when I upload this to Heroku, test the form and heroku run find on the directory, I don't see anything.
I have noticed that this directory on heroku has a seemingly forbidding privilege:
drwx------ 2 u25619 dyno 4096 Apr 23 05:14 images
I tried slipping in a nice little File.chmod("priv/static/images", 0o777), but to no avail; that directory seems locked away from me, so I think this is a heroku issue.
Any idea how to handle this?
EDIT: Resolved by using the phoenix dep ex_aws to upload to an Amazon S3 bucket.
ex_aws dependency
partial explanation (note: you need to add poison and hackney to make this work, they are not mentioned)
The file system on heroku is ephemeral, and you won't have access to any files that you save on it across deploys or when you start new instances.
Also, when you run heroku run, you're not connecting up to that same instance that's currently running your app, instead what it'll do is to launch a new instance so those uploaded files would not exist there.
A better approach is to save the uploaded files to S3 or similar where you can still access it across deploys.
From Heroku's information page on the read-only file system
"There are two directories that are writeable: ./tmp and ./log (under your application root). If you wish to drop a file temporarily for the duration of the request, you can write to a filename like #{RAILS_ROOT}/tmp/myfile_#{Process.pid}. There is no guarantee that this file will be there on subsequent requests (although it might be), so this should not be used for any kind of permanent storage."
Does anyone know how often files are deleted from the /tmp folder in Heroku?
heroku never explicitely removes files from your tmp folder.
However it is not shared between instances of your application (your dynos).
This means you can assume the tmp folder to be emptied every time you deploy your application.
As you should always be able to deploy, you need, for your own sake, to architect your app with that in mind and not rely on the tmp folder to keep files longer than the user's HTTP request.
I have a few resources (log files, database files, separate configuration files, etc.) that I would like to be able to access from my OSGi bundles. Up until now, I've been using a relative file path to access them. However, now my same bundles are running in different environments (plain old Felix and Glassfish).
Of course, the working directories are different and I would like to be able to use a method where the directory is known and deterministic. From what I can tell, the working directory for Glassfish shouldn't be assumed and isn't spec'ed (glassfish3/glassfish/domains/domain1/config currently).
I could try to embed these files in the bundle themselves, but then they would not be easily accessible. For instance, I want it to be easy to find the log files and not have to explode a cached bundle to access it. Also, I don't know that I can give my H2 JDBC driver a URL to something inside a bundle.
A good method is to store persistent files in a subdirectory of the current working directory (System.getProperty("user.dir") or of the users home directory (System.getProperty("user.home"))
Temporary and bundle specific files should be stored in the bundle's data area (BundleContext.getData()). Uninstalling the bundle will then automatically clean up. If different bundles need access to the same files, use a service to pass this information.
Last option is really long lived critically important files like major databases should be stored in /var or Window's equivalent. In those cases I would point out the location with Config Admin.
In general it is a good idea to deliver the files in a bundle and expand them to their proper place. This makes managing the system easier.
You have some options here. The first is to use the Configuration Admin service to specify a configuration directory, so you can access files if you have to.
For log files I recommend Ops4J Pax Logging. It allows you to simply use a logging API like slf4j and Pax Logging does the log management. It can be configured using a log4j config.
I think you should install the DB as a bundle too. For example I use Derby a lot in smaller projects. Derby can simply be started as a bundle and then manages the database files itself. I'm not sure about h2 but I guess it could work similarly.