Looking to setup a high performance environment running Mongo 3.4 on windows 2016 in azure. I come from a SQL\windows background and was wondering if there are any options with Mongo to spread out the IO workload of mongod. It seems odd that there is only a dbPath option and that you can not configure separate locations for the DB(s), opslogs and journal. Am i missing something ?
Thanks for any assistance
This is indeed possible, using a couple of different techniques:
The oplog is stored in the local database, so you can keep it in a separate folder by using the storage.directoryPerDB config option.
The journal is stored in a subfolder of the data directory; you can make MongoDB save its journal files in a separate directory by preparing a symbolic link called journal in the data directory, pointing to your other folder.
Related
I have almost an identical problem as this post:
How to migrate a Windows 10 installation of PostgreSQL 9.5.7 to a larger disk
I have a PostgreSQL database on my C drive which is running out of space. I want to move my database to my larger F drive. I'm running into the same issue as the user in the post I mentioned:
The path to executable under the service to start my server is
C:\PostgreSQL\pg10\pgservice.exe "//RS//PostgreSQL 10 Server"
There's no specific path to the data directory explicitly written. I'm not sure how to change where PostgreSQL looks to store data since there's not a -D variable defined there.
I think if I just copy my data over to the larger drive and pass the new data directory as a parameter argument on startup, my issue would be solved. Any ideas on how to do this given my current configuration?
I won’t call it migration rather just transferring files from one location to another.
It can be done by:
Stopping database server
Cut/paste data to your new drive location
reconfigure database server to use new location
Start server again or restart system if needed
I wrote a simple note taking program that's nothing more than a dictionary mapping a key to a value. IE
$ hlp -key age -value 25
$ hlp age
25
and it just stores information in a json file hardcoded to ~/.hlp.json. But I was wondering if there's likely some standard location I should be putting this file. Is there a standard location for databases like this?
A useful resource here is the hier(7) man page. (http://linux.die.net/man)
Data that is only going to be used by you belongs in $HOME, traditionally hosted under /home.
For something that is used to support the system itself, you'd be using /var. For applications that are just hosted on the system, you'd use /var/opt.
If the application is something big that could be replicated or moved to another system, you'd create a separate filesystem with a mount point outside any of those listed in hier(7). This could be a filesystem mounted from a SAN or NAS, which whould help mobility of the application.
Once you actually need to access the data from different machines, you'd have to move it to a network accessable key/value store or sql database.
After following this simple tutorial http://www.louisaslett.com/RStudio_AMI/ and video guide http://www.louisaslett.com/RStudio_AMI/video_guide.html I have setup an RStudio environment on EC2.
The only problem is, I can't upload large files (> 1GB).
I can upload small files just fine.
When I try to upload a file via RStudio, it gives me the following error:
Unexpected empty response from server
Does anyone know how I can upload these large files for use in RStudio? This is the whole reason I am using EC2 in the first place (to work with big data).
Ok so I had the same problem myself and it was incredibly frustrating, but eventually I realised what was going on here. The default home directory size for AWS is less than 8-10GB regardless of the size of your instance. As this as trying to upload to home then there was not enough room. An experienced linux user would not have fallen into this trap, but hopefully any other windows users new to this who come across this problem will see this. If you upload into a different drive on the instance then this can be solved. As the Louis Aslett Rstudio AMI is based in this 8-10GB space then you will have to set your working directory outside this, the home directory. Not intuitively apparent from Rstudio server interface. Whilst this is an advanced forum and this is a rookie error I am hoping no one deletes this question as I spent months on this and I think someone else will too. I hope this makes sense to you?
Don't you have shell access to your Amazon server? Don't rely on RStudio's upload (which may have a 2Gb limit, reasonably) and use proper unix dev tools:
rsync -avz myHugeFile.dat amazonusername#my.amazon.host.ip:
on your local PC command line (install cygwin or other unixy compatibility system) will transfer your huge file to your amazon server, and if interrupted will resume from that point, will compress the data for transfer too.
For a windows gui on something like this, WinSCP was what we used to do in the bad old days before Linux.
This could have something to do with your web server. Are you using nginx or apache as your web server. If so you can modify the upload feature in your nginx server. If you are running nginx on the front end of the web server I would recommend the following fix in your nginx.conf file.
http {
...
client_max_body_size 100M;
}
https://www.tecmint.com/limit-file-upload-size-in-nginx/
I had a similar problems with a 5GB file. What worked for me was to use SQLite to create a database with the csv file that I needed. Use SQLite code to bring create the database. Then I used a function in RStudio to communicate with the local database. In that way, I was able to bring in the csv file. I can track down the R code that I used if you like.
So, MSSQL is nice enough to have given us a nifty little sql code for creating a database backup from a command line:
BACKUP DATABASE [db_name] TO DISK = N'D:\backups\back.bak' WITH NOFORMAT, NOINIT, NAME = N'db_name', SKIP, NOREWIND, NOUNLOAD, STATS = 10
GO
However, I am looking to be able to run this command from a php or even shell script on a remote Mac server.
The Problem I am running into is when I try to change the DISK to say my admin home directory, it keeps complaining to me about:
Cannot open backup device 'D:\PATH\ON\SERVER\/Users/admin/back.bak'. Operating system error 3(The system cannot find the path specified.).
Anyone know what I am missing here? I would be very appreciative
SQL Server's BACKUP command does a backup to the database server's local disk. That means that setting the path to a directory on the client machine makes no sense.
If you want a database backup stored on your client machine, I can basically see 3 options;
Back up to a temporary location accessible from the database server, and copy it from there to your client.
Mount a disk shared from your client machine on your database server as for example X:\ and do the backup to that disk.
Find another backup solution that does backups in a different way (sorry, no, I have no recommendations)
You can use RasorSQL, it's a client for mac and windows.
https://razorsql.com/
A PHP scriptof mine wants to write into a log folder, the resulting error is:
Unable to open the log file "E:\approot\framework\log/dev.log" for writing.
When I set the writing permissions for the WebRole User RD001... manually it works fine.
Now I want to set the folder permissions automatically. Is there an easy way to get it done?
Please note that I'm very new to IIS and the stuff around, I would appreciate precise answers, thx.
Short/Technical Response:
You could probably set permissions on a particular folder using full-trust and a startup taks. However, you'd need to account for a stateless OS and changing drive letters (possible, not likely) in this script, which would make it difficult. Also, local storage is not persisted, so you'd have no way to ensure this data stayed in the case of a reboot.
Recommendation: Don't write local, read below ...
EDIT: Got to thinking about this, and while I still recommend against this, there is a 3rd option: You can allocate local storage in the service config, then access it from PHP using a dll reference, then you will have access to that folder. Please remember local storage is not persisted, so it's gone during a reboot.
Service Config for local:
http://blogs.mscommunity.net/blogs/dadamec/archive/2008/12/11/azure-reading-and-writing-with-localstorage.aspx
Accessing config from php:
http://phpazure.codeplex.com/discussions/64334?ProjectName=phpazure
Long / Detailed Response:
In Azure, you really are encouraged to approach things as a platform and not as "software on a server". What I mean there is that ideas such as "write something to a local log file" are somewhat incompatible with the cloud "idea". Depending on your usage, you could (and should) convert this script to output this data to some cloud-based or external storage, vs just placing it on the disk.
I would suggest modifying this script to leverage the PHP Azure SDK and write these log entries out to table or blob storage in Azure. If this sounds good, please provide the PHP and I can give an exact example.
The main reason for that (besides pushing the cloud idea) is that in Azure, you cannot assume the host machine ("role instance") will maintain an OS state, so while you can set some things such as folder permissions, you can't rely on them sticking that way. You have no real way to guarantee those permissions won't be reset when the fabric has to update your role and react to some lower level problem. For example, a hard-drive cage on the rack where your current instance lives could fail. If the failure were bad enough, the Fabric controller would need to rebuild your instance. When that happens, your code is moved to an entirely different server, so the need would arise to re-set those permissions. Also, depending on the changes, the E:\ could all of a sudden need to be the F:\ or X:\ drive and you wouldn't know.
Its much better to pretend (at some level) that your application is running "in Azure" and not "on a server in azure", so you make no assumptions about the hosting environment. So anything you need outside of your code (data, logs, audits, etc) should be stored somewhere you can control (Azure Storage, external call-out, etc)