My app has a Postgres DB, but the initial population is done from csv files that add up to 900MB total.
The way I do this now is by setting the DATABASE_URL environment variable on my local computer to the Heroku Postgres DB's URL, but this takes many hours since it has to send a lot of transactions over the network. For comparison, when I load the data to my SQLite DB on my computer, it only takes about a minute.
Is there a way I can upload my data files to Heroku so I can run the script local there (with heroku run), to make it faster? I'd like to upload them zipped and then unzip on the server, if possible.
I've looked at heroku push but as far as I understand, it deploys an entire directory and not specific files.
Related
I am doing a Discord.py bot and my bot needs a database to store guilds, members and stuff. I uploaded my files (a created database file too) with git to Heroku and ran my bot. When people join Discord server or something, it has to record that to the database.
When I downloaded all my files back from Heroku to my computer with heroku git:clone -a <name> I see that my database is still empty, even though a member has joined. When I run my bot on my computer directly, database works fine. Why? Does Heroku even update files when my code says so? Maybe I misunderstood something?
Maybe I misunderstood something?
Yes. You deploy code to heroku by pushing it to its git repo. That's when your blank/default database file is. When heroku starts a dyno, it essentially copies the files from the git repo into the filesystem on the new dyno. As your app works, it can read and write the files on the dyno. The important points are:
Writes to that dyno-local file are not propagated back to git.
The dyno filesystem is ephemeral. When your dyno shuts down, the files are gone and all the changes are gone with them.
If you want a persisted database that can survive dyno restarts, you should use an external database instead of a local file. For example, a Postgres add-on.
I have a discord bot that saves JSON files on the same directory he is in so it could work on more than one server without colliding (he saves variables that are not important to the question) .
I finished my code and I uploaded it to heroku for hosting. The thing is , when I ran the code from my pc I could see the files that were being created for each server for testing but now I don't know how to reach them.
Is there a way to check all the files I have in heroku?
(when I mean all, I mean also the JSON files that were created from the bot itself)
side not:
You can do heroku run bash -a APPNAME but it still doesn't let me see the files that were made in the dyno or directory.
On top of that, if someone has another good hosting site(preferred free) which doesn't use a ephemeral filesystem, that would be great if you can comment them down bellow.
(or if you have a way to save the files before the dyno deletes them)
What you are searching for in Heroku is called Heroku Exec (SSH Tunneling) which you can use to SSH into running dynos for debugging purposes.
I have a PWA made with NuxtJS correctly deployed and working on Heroku.
I would like to implement a file uploader and manager so that I can manage some files in a directory (~/static/files) from my front-end through some APIs.
On localhost, it works fine so I have my directory and when I add or delete the file, it deletes or creates it from the file system (as it should).
My question is: why can't I do the same on Heroku? I mean, I tried by uploading a file and deleting it and it works but the problem comes when I restart the app (through heroku ps:restart -a appname) because when I do so it deletes the file as if it was saved in RAM and not onto the file system.
If I try to see the files in the directory where they should be through heroku run bash -a appname and then down to the directory, no file is showed.
How can I fix this?
The Heroku filesystem is ephemeral - that means that any changes to the filesystem whilst the dyno is running only last until that dyno is shut down or restarted. Each dyno boots with a clean copy of the filesystem from the most recent deploy. This is similar to how many container based systems, such as Docker, operate.
In addition, under normal operations dynos will restart every day in a process known as "Cycling".
These two facts mean that the filesystem on Heroku is not suitable for persistent storage of data. In cases where you need to store data we recommend using a database addon such as Postgres (for data) or a dedicated file storage service such as AWS S3 (for static files).
We have a CMS on heroku, some files were generated by the CMS, how can I pull those changes down? Can I commit the changes remotely and pull them down? Is there an FTP option of some kind?
See: https://devcenter.heroku.com/articles/dynos#ephemeral-filesystem
It's not designed for persistent file generation and usage.
In practice, it works like this: User puts some code into a repository. That code is dynamically pulled into temporary Amazon EC instances and executed. The code can be pulled from virtual machine to virtual machine, node to node, without disruption, across data centers. There is no real "place" to get the products of your code from the environment, because anything generated by the checked-out code can (and will) be destroyed as your code deploy skips around between the temporary machines.
That being said, there are some workarounds:
If your app includes something like a file browser within your deployed code, you can grab the (entirely) temporary files using that file browser, and commit it back to your persistent code trunk.
Another option is using something like S3 for your persistent storage, with your application reading from, and writing to, a data storage service, knowing that while heroku will just re-write and destroy your local data on a frequent basis, the external service will maintain the files.
Similarly, you can change your application to use heroku's postgres for persistent data storage, or use Amazon's RDS, (etc.).
Alternately, you can edit your application in such a way as to ensure that any files generated by it will be regenerated every time the code is refreshed, redeployed, and moved around.
suppose if i open my heroku webpage, it update a file, like a database.
now i want to retrieve it.
i tried git pull, when done, i checked, it is the old file what i pushed last time.
i tried heroku run bash and "cat"-ed the file, it gives old outputs. :/
but i can assure, the file is getting update, coz if i output the file content through server, like if i request for a particular path on my address, it will show the contents of that file on browser, then it shows updated data.
i have no idea why is this happening. any clue ?
i am using python3 with wsgiref module.
You shouldn't use the dyno filesystem for persistent file storage (like databases). The dyno filesystems are ephemeral and changes are not reflected in the git repository associated with you app. Use one of the data storage add-ons instead: https://addons.heroku.com