I accessed the dyno via heroku run bash and made a file foo. However, when I check from my app, it still cannot find foo. So I dug deeper by trying to install nginx on a dyno, turning on autoindex, and I can confirm that the files are different from what's accessed via heroku run bash and from nginx. Why is that? How do I put files to the filesystem where my running process is showing.
When you issue heroku run bash a new dyno is created just for this one-off, and you are given access to it. Any file you create will "disappear" once your log-off, since the Heroku file-system is ephemeral.
That means the file-system is restored to its native state whenever a new dyno is created, or a dyno is rebooted. The "native" state is what's in your slug -- the "compiled" version of your app -- whatever is built by the build-pack after you "git push" to Heroku.
If you want a read-only file available to all your Dynos, either put it in your slug (for example: by including it in git, but also by using a different build-pack), or put it somewhere all your dynos can access (like a shared database, a Redis/Memcache instance, or most logically: S3).
Related
I have a discord bot that saves JSON files on the same directory he is in so it could work on more than one server without colliding (he saves variables that are not important to the question) .
I finished my code and I uploaded it to heroku for hosting. The thing is , when I ran the code from my pc I could see the files that were being created for each server for testing but now I don't know how to reach them.
Is there a way to check all the files I have in heroku?
(when I mean all, I mean also the JSON files that were created from the bot itself)
side not:
You can do heroku run bash -a APPNAME but it still doesn't let me see the files that were made in the dyno or directory.
On top of that, if someone has another good hosting site(preferred free) which doesn't use a ephemeral filesystem, that would be great if you can comment them down bellow.
(or if you have a way to save the files before the dyno deletes them)
What you are searching for in Heroku is called Heroku Exec (SSH Tunneling) which you can use to SSH into running dynos for debugging purposes.
I have a PWA made with NuxtJS correctly deployed and working on Heroku.
I would like to implement a file uploader and manager so that I can manage some files in a directory (~/static/files) from my front-end through some APIs.
On localhost, it works fine so I have my directory and when I add or delete the file, it deletes or creates it from the file system (as it should).
My question is: why can't I do the same on Heroku? I mean, I tried by uploading a file and deleting it and it works but the problem comes when I restart the app (through heroku ps:restart -a appname) because when I do so it deletes the file as if it was saved in RAM and not onto the file system.
If I try to see the files in the directory where they should be through heroku run bash -a appname and then down to the directory, no file is showed.
How can I fix this?
The Heroku filesystem is ephemeral - that means that any changes to the filesystem whilst the dyno is running only last until that dyno is shut down or restarted. Each dyno boots with a clean copy of the filesystem from the most recent deploy. This is similar to how many container based systems, such as Docker, operate.
In addition, under normal operations dynos will restart every day in a process known as "Cycling".
These two facts mean that the filesystem on Heroku is not suitable for persistent storage of data. In cases where you need to store data we recommend using a database addon such as Postgres (for data) or a dedicated file storage service such as AWS S3 (for static files).
I inherited an app, and while I intend to redeploy everything correctly shortly with a host of other changes, a client of mine wants new branding up quickly on a Heroku server.
I access the server this way:
heroku run bash --app APPNAME
and I pulled down the file via wget and put it into a static files directory.
However, after doing this, the change is not reflected live. Is there anyway to make a temporary change like this without a full redeploy?
File systems on Heroku are ephemeral - the files get deleted and there is no persistence.
The Heroku filesystem is ephemeral - that means that any changes to
the filesystem whilst the dyno is running only last until that dyno is
shut down or restarted. Each dyno boots with a clean copy of the
filesystem from the most recent deploy. This is similar to how many
container based systems, such as Docker, operate.
Why Are My Files Deleted - Heroku
Is it possible, and if so, how? I'd like to be able to reach it from my existing Heroku infrastructure.
Will I need a Procfile? From what I understand it's just a standalone binary written in Go! so it shouldn't be that hard to deploy it, I'm just curious how to deploy it because I don't think I understand the ins and outs of Heroku deployment.
Heroku Dynos should not be used to deploy a database application like InfluxDB.
Dynos are ephemeral servers. Data does not persist between dyno restarts and cannot be shared with other dynos. Practically speaking, any database application deployed on a dyno is essentially useless. This is why databases on Heroku (e.g. Postgres) are all Add-ons. InfluxDB should be set up on a different platform (like, AWS EC2 or a VPS) since a Heroku Add-on is not available.
That said, it is possible to deploy InfluxDB to a Heroku dyno.
To get started, it is important to understand the concept of a 'slug'. Slugs are containers (similar to a Docker images) which hold everything needed to run a program on Heroku's infrastructure. To deploy InfluxDB, an InfluxDB slug needs to be created.* There are two ways to create a slug for Go libraries:
Create a slug directly from a Go executable as described here.**
Build the slug from source using the Heroku Go buildpack (explained below).
To build the slug from source using a buildpack, first clone the InfluxDB Github repo. Then add a Procfile at the root of the repo, which tells Heroku the command to run when the dyno starts up.
echo 'web: ./influxd' > Procfile
The Go buildpack requires all dependencies be included in the directory. Use the godep dependency tool to vendor all dependencies into the directory.
go get github.com/tools/godep
godep save
Next, commit the changes made above to the git repo.
git add -A .
git commit -m dependencies
Finally, create a new app and tell it to compile with the Go buildpack.
heroku create -b https://github.com/kr/heroku-buildpack-go.git
git push heroku master
heroku open // Open the newly created InfluxDB instance in the browser.
Heroku will show an error page. An error will be displayed because Heroku's 'web' process type requires an app to listen for incoming requests on the port described by the $PORT environment variable, otherwise it will kill the dyno. InfluxDB's API and admin panel run on ports 8086 and 8083, respectively.
Unfortunately, InfluxDB does not allow those ports to be set from environment variables, only through the config file (/etc/config.toml). A small bash script executed before InfluxDB starts up could set the correct port in the config file before InfluxDB starts up.
Another problem, Heroku only exposes one port per dyno so the API and the admin panel cannot be exposed to the internet at the same time. A smart reverse proxy could work around that issue using Heroku's X-Forwarded-Port request header.
Bottom line, do not use Heroku dynos to run InfluxDB.
* This means the benefits of a standalone Go executable are lost when deploying to Heroku, since it needs to be recompiled for Heroku's stack.
** Creating a slug directly from the InfluxDB executable does not work because there is no built-in way to listen to the right port given by Heroku in the $PORT environment variable.
I like to think anything is possible on a Heroku node when using a custom buildpack, but there are some considerations when hosting with Heroku:
ops, e.g. backup, monitoring (does it entail installing extra services, opening extra ports, etc - Heroku might get in the way here)
performance, considering dyno size
and if you need a larger dyno, cost becomes an issue. You'll get more bang for your buck when you go the IaaS route.
other "features" of a dyno, e.g. disk ephemerality
I highly recommend hosted InfluxDB or spinning up your own on a VPS, all of which you can point your existing Heroku-based apps to. It will then help to get those instances as close together as possible (i.e. same region, or co-located if possible), presuming a need for low latency between DB and app stack.
I'm using Python/Django on Heroku (Cedar Stack) and I've got a management command that I need to write that will pull a file out of an S3 bucket and process it. I'm not sure I understand how to use the ephemeral filesystem. Are there only certain directories that are writeable? I found an other article that implied that there were only certain folders that were writable (but, it doesn't seem to apply to the Cedar stack). I found this dev article but it doesn't go into much detail (note: I do understand that it's just temporary. I only need to unzip the file and process the file). Can I just create a folder anywhere under the application's root? And how would I get that? It seems like I could probably just use $HOME. I did a bit of testing by connecting to via
$ heroku run bash
and running:
$ echo #HOME
returns:
/app
and running:
$ mkdir $HOME/tmp
creates a folder in the app's root and gives with the same user and group as the other files and folders.
So... anything I'm missing here? A better way to do it? Is there an OS environment variable for this? I've run "env" and I don't see a better one.
To really understand the ephemeral filesystem, you need to understand what a dyno is. You can read more about how dynos work. In a nutshell, though, a process runs on Heroku in a virtual machine with its own filesystem. That virtual machine can stop for a number of reasons, taking the filesystem along with it.
The underlying filesystem will be destroyed when an app is restarted, reconfigured (e.g. heroku config ...), scaled, etc. For example, if you have two web dynos, write some files to the ephemeral filesystem, and scale to three dynos, those files will be destroyed because your app is running on new dynos.
In general, the ephemeral filesystem works just like any filesystem. Directories you have permission to write to, such as $HOME and /tmp, you can write files to. Any files that require permanence should be written to S3, or a similar durable store. S3 is preferred as Heroku runs on AWS and S3 offers some performance advantages. Any files that can be recreated at will can be stored on the dyno's ephemeral store.
You can create a file under the '/tmp' directory, and that file will be destroyed after the request is complete. I'm doing this on Cedar, and I haven't had any problems.