I'm having a number of indices which are actually causing some space issues at the moment in my Ubuntu machine. The indices keep growing on a daily basis.
So I thought of moving it to another mount directory which has more space apparently. How can I do this safely?
And I have to make sure that the existing ES indices and the Kibana graphs would be safe enough after the doing the move.
What I did: Followed this SO and moved my data directory of Elasticsearch somehow to the directory (/data/es_data) I needed, but after I did that, I couldn't view my existing indices plus the Kibana graphs and dashboards which I created as well.
Am I doing something wrong? Any help could be appreciated.
FWIW If it were me, I would stop elasticsearch & kibana (& logstash if this is the only elasticsearch node in the cluster) then move the old data dir to a new location out of the way:
sudo mv /var/lib/elasticsearch /var/lib/elasticsearch-old
Then set up the new volume (which should be at least 15% larger than the size of the indexes you have on disk as elasticsearch won't create new indexes on a disk with less than 15% free space) with a file system and find out it's UUID and get ready to mount it:
sudo fdisk /dev/sdX # New volume, use all the space
sudo mkfs.ext4 /dev/sdX1
ls -la /dev/disk/by-uuid/ | grep /dev/sdX1 # Or forget the grep and manually look for it
Then add the following to your /etc/fstab, replacing with the UUID from previous command:
UUID=<RESPONSE> /var/lib/elasticsearch ext4 defaults 0 0
Make the new directory as the old one is gone, it probably wants chowning (I assume the owner should be elasticsearch but you can confirm by checking ownership of the old folder) and you want to copy the content from the old one:
sudo mkdir /var/lib/elasticsearch
sudo chown -R elasticsearch: /var/lib/elasticsearch
cp -rp /var/lib/elasticsearch-old/* /var/lib/elasticsearch
Once everything has finished copying across you should then be able to start elasticsearch back up, it should find the indexes as they haven't moved, config doesn't need updating.
Once you're happy that everything is working you can delete /var/lib/elasticsearch-old and reclaim your space. Failing that you can revert to the old data and it should continue to work.
Related
I'm trying to change /var/lib/clickhouse to something like /mnt/sdc/clickhouse so that i could have clickhouse in another hard disk. I've tried this steps:
1. Stop Clickhouse
2. Move directory /var/lib/clickhouse to /mnt/sdc/clickhouse
3. Replace all /var/lib/s to /mnt/sdc/ in file /etc/clickhouse-server/config.xml
4. Start Clickhouse
But the problem is /var/lib/clickhouse contains hard links so when i mv the directory, this hard links become corrupted.
Is this OK or not?
How should i change the clickhouse directory?
To copy files while preserving hard links, you can use rsync with --hard-links (or -H) option. For your setup, you should be able to run the following:
rsync -a -H /var/lib/clickhouse/ /mnt/sdc/clickhouse
Note the trailing slash after the first directory to copy the directory contents rather than the directory itself.
Then, as you mentioned, update the /var/lib/ paths to /mnt/sdc/ in /etc/clickhouse-server/config.xml, and restart ClickHouse with systemctl restart clickhouse-server.
I was able to follow these steps to migrate ClickHouse data to a new disk mount using rsync, and ClickHouse restarted successfully using the new disk (ClickHouse v22.3 on Ubuntu 18.04).
I found that moosefs trash take too much of my disk space. according to moosefs documentation, it will keep it for a while in case user want it back. But How to clean it up manually to save space?
In order to purge MooseFS' trash, you need to mount special directory called "MooseFS Meta".
Create mountdir for MooseFS Meta directory first:
mkdir /mnt/mfsmeta
and mount mfsmeta:
mfsmount -o mfsmeta /mnt/mfsmeta
If your Master Server Host Name differs from default mfsmaster and/or port differs from default 9421, use appropriate switch, e.g.:
mfsmount -H master.host.name -P PORT -o mfsmeta /mnt/mfsmeta
Then you can find your deleted files in /mnt/mfsmeta/trash/SUBTRASH directory. Subtrash is a directory inside /mnt/mfsmeta named 000..FFF. Subtrashes are helpful if you have many (e.g. millions) of files in trash, because you can easily operate on them using Unix tools like find, whereas if you had all the files in one directory, such tools may fail.
If you do not have many files in trash, mount Meta with mfsflattrash parameter:
mfsmount -o mfsmeta,mfsflattrash /mnt/mfsmeta
or if you use Master Host Name or Port other than default:
mfsmount -H master.host.name -P PORT -o mfsmeta,mfsflattrash /mnt/mfsmeta
In this case your deleted files will be available directly in /mnt/mfsmeta/trash (without subtrash).
In both cases you can remove files by simply using rm file or undelete them by moving them to undel directory available in trash or subtrash (mv file undel).
Remember, that if you do not want to have certain files moved to trash at all, set "trash time" (in seconds) for these files to 0 prior to deletion. If you set specific trash time for a directory, all the files created in this directory inherit trash time from parent, e.g.:
mfssettrashtime 0 /mnt/mfs/directory
You can also set a trash time to other value, e.g. 1 hour:
mfssettrashtime 3600 /mnt/mfs/directory
For more information on specific parameters passed to mfsmount or mfssettrashtime, see man mfsmount and man mfstrashtime.
Hope it helps!
Peter
I have been given the address to a very large folder on a shared Unix server. I've been given a path to some files on a unix server I'm working on through ssh. I don't want to waste space by creating a duplicate in my home area so I've linked the folder through ln -s. However I don't want to risk making any changes to the data within the folder.
How would I go about setting the files to read-only for myself? Do I have to ask the owner of the folder/file? Do I need sudo access? I am not the owner of the file and I do not have root access.
Read about chmod command to change the mask on the files the links point to.
The owner or root can restrict access to files.
Also you probably need to mount that shared folder as read-only. But I am not sure how your folder is connected
UPDATE
The desired behaviour can be achieved using mount tool. (man page for mount).
Note that the filesystem mount options will remain the same as those on the original mount point, and cannot be changed by passing the -o option along with --bind/--rbind. The mount options can be changed by a separate remount command, for example:
mount --bind olddir newdir
mount -o remount,ro newdir
Here is the similiar question to yours. Also solved via mount tool.
I was given the task to setup incremental backups for mongodb replicaset, as start point of course I googled about it and could not find anything on mongodb docs, I did find however this question which encouraged to develop my own solution as didn't find Tayra very active.
I read about oplog and realized it was very easy to develop something to replay the log, but it turns out that I didn't have to as mongorestore does that for me.
Now I have a working solution with bash scripts and it was quite easy, that's the reason I am asking here if there is any flaw in my logic, or maybe something that will bite me in the future.
Below how I implemented that:
Full backup procedure:
lock writes on a secondary member db.fsyncLock()
Take snapshot
Record last position from oplog
db.oplog.rs.find().sort({$natural:-1}).limit(1).next().ts
Unlock writes db.fsyncUnlock()
Incremental backup procedure:
lock writes on a secondary member
Dump oplog from the recorded oplog position on full (or latest incremental ) backup:
mongodump --host <secondary> -d local -c oplog.rs -o /mnt/mongo-test_backup/1 --query '{ "ts" : { $gt : Timestamp(1437725201, 50) } }'
Record latest oplog position (same way as for full backups)
Unlock writes
Full backup restore procedure:
stop all instances of mongod
copy snapshot to data dir of the box which will be the primary, but make sure to exclude all local* and mongod.lock this restore technique is called reconfigure by breaking mirror
Start primary
reconfigure replicaset
start secondaries without any data, let them perform initial sync. Or copy the data from the new primary with fresh local database
Restore incremental backup:
When we created incremental backup it stored it like this:
/mnt/mongo-test_backup/1/local/oplog.rs.bson
/mnt/mongo-test_backup/1/local/oplog.rs.metadata.json
We're instered on oplog.rs.bson but we will have to rename it, so here are the steps:
change directory to the backup: cd /mnt/mongo-test_backup/1/local
delete the json file rm *.json
rename the bson file mv oplog.rs.bson oplog.bson
restore it :
mongorestore -h <primary> --port <port> --oplogReplay /mnt/mongo-test_backup/1/local
I have it all scripted, I may commit it on github later.
Question is if there is any flaw in the logic. I am bit suspicious as the procedure is quite straight forward and still I couldn't find it documented anywhere.
I'm using Cloud9 (railstutorial.org) and noticed that the disk space used by my workspace is fastly growing toward the disk quota.
Is there a way to clean up the workspace and thereby reduce the disk space used?
The workspace is currently 817MB (see below using quota -s). I downloaded it to look at the size of the directories, and I don't understand it. The directory containing my project is only 170 MB in size and the .9 folder is only 3 MB. So together that doesn't come near the 817 MB... And the disk space used keeps growing even though I don't I'm making any major changes to the content of my project.
Size Used Avail Use%
1.1G 817M 222M 79%
Has it perhaps got to do with the .9 folder? For example, I've manually deleted several sub-projects but in the .9 folder these projects still exist, including their files. I also wonder if perhaps different versions of gems remain installed in the .9 folder... so that if you update a gem, it includes both versions of the gem.
I'm not sure how this folder or Cloud9 storage in general works, but my question is how to clean up disk space (without having to remove anything in my project)? Is there perhaps some clean-up function? I could of course create a new workspace and upload my project there, but perhaps there's an alternative while keeping the current workspace.
The du-c9 command lists all the files contributing to your quota. You can reclaim disk space by deleting files listed by this command.
For a user-friendly interface, you may want to install ncdu to see the size of all your folders. First, free some space for the install. A common way to do this is by removing your tmp folder:
rm -rf /tmp/*
Then install ncdu:
sudo apt-get install ncdu
Then run ncdu and navigate through your folders to see which ones are using up the most space:
ncdu ~
Reference: https://docs.c9.io/discuss/557ecf787eafa719001d1af8
For me the answers above unfortunately did not work (the first produced a list incomprehensibly long, so long that I run out of scroll space in the shell and the second one produced a strange list-- see at the end of this answer):
What did was the following:
1) From this support faq article: du -hx / -t 50000000
2) Identify the culprit from the easy to read, easy to understand list: in my case 1.1G /home/ubuntu/.local/share/heroku/tmp
3) From the examples of this article: rm -r /home/ubuntu/.local/share/heroku/tmp
Strange list:
1 ./.bundle
1 ./.git
1 ./README.md
1 ./Project_5
2 ./.c9
2 ./Project_1
3 ./Project_2
17 ./Project_3
28 ./Project_4
50 .
If you want to dig into more details of which file is affecting your workspace disk try this command: sudo du -h -t 50M / --exclude=/nix --exclude=/mnt --exclude=/proc
This will give you all the files on your Linux server and then you can remove any file by this command:
sudo rm -rf /fileThatNeedsToDelete/*
From AWS in Cloud9 this command df -hT /dev/xvda1 worked for me:
[ec2-user ~]$ df -hT /dev/xvda1
Filesystem Type Size Used Avail Use% Mounted on
/dev/xvda1 xfs 8.0G 1.2G 6.9G 15% /
more info here:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-describing-volumes.html