I have a problem regarding transferring files from a server to another server. I tried using PSCP putty. it worked in the first time from local to server. what I'm trying to do is zip all files then transfer to another server. what commands should I use to achieve this?
pscp -P 22 test.zip root#domain.com:/root
this line of code works when transferring local to remote server, however, I want to compress files from a server to another remote server, or at least remote to local, then local to remote, whatever method is possible. I cannot compress the files because it's almost 50 GB in total so I am searching for a much faster way to achieve this.
I am wondering if it is possible to RUN a remote file stored in an NFS share when building an image from a dockerfile.
Currently I am using the COPY command and then the RUN command to execute the files, however many of the files I need to create the image are extremely large.
Is it possible to execute files stored in an NFS share directly in the dockerfile without having to copy them all over?
You can only RUN files inside your container - so it needs to copied to your container.
What you can do is move the COPY commands to the beginning of your Dockerfile so that they are cached and don't need to be copied every time you change a command later in the Dockerfile.
You can RUN curl.... to grab the remote file ,then execute it sure.
But this will only run at image build time, not during lifecycle of the container
You could also mount the NFS volume to your host, then COPY the files.
Otherwise, remote execution is a pretty basic security flaw and shouldn't be possible under any circumstances
I have created jekyll site. Regarding the deployment I don't want to host on github pages. To host private domain I came know from documentation to copy the all files from _site folder. That's all wicked.
Question:
Each time I add new blog post, I am running cmd>jekyll build then I am copying newly created html to hosted domain. Is there any easy way to update without compiling each time ?
The reason, Why I am asking is because it will updated by non technical person
Thanks for the help!!
If you don't want to use GitHub Pages, AFAIK there's no other way than to compile your site each time you make a change.
But of course you can script/automate as much as possible.
That's what I do with my own blog as well. I'm hosting it on my own webspace instead of GitHub Pages, so I need to do these steps for each update:
Compile on local machine
Upload via FTP
I can do this with a single click (okay, a single double-click).
Note: I'm on Windows, so the following solution is for Windows.
But if you're using Linux/MacOS/whatever, of course you can use the tools given there to build something similar.
I'm using a batch file (the Windows equivalent to a shell script) to compile my site and then call WinSCP, a free command-line FTP client.
WinSCP allows me to store session configurations, so I saved the connection to my server there once.
Because of this, I didn't want to commit WinSCP to my (public) repository, so my script expects WinSCP in the parent folder.
The batch file looks like this:
call jekyll build
echo If the build succeeded, press RETURN to upload!
pause
set uploadpath=%~dp0\_site
%~dp0\..\winscp.com /script=build-upload.txt /xmllog=build-upload.log
pause
The first parameter in the WinSCP call (/script=build-upload.txt) specifies the script file which contains the actual WinSCP commands
This is in the script file:
option batch abort
option confirm off
open blog
synchronize remote -delete "%uploadpath%"
close
exit
Some explanations:
%~dp0 (in the batch file) is the folder where the current batch file is
The set uploadpath=... line (in the batch file) saves the complete path to the generated site into an environment variable
The open blog line (in the script file) opens a connection to the pre-saved session configuration (which I named blog)
The synchronize remote ... line (in the script file) uses the synchronize command to sync from the local folder (saved in %uploadpath%, the environment variable from step 2) to the server.
IMO this solution is suitable for non-technical persons as well.
If the technical person in your case doesn't know how to use source control, you could even script committing & pushing, too.
There are a number of options available which are mentioned in the documentation: http://jekyllrb.com/docs/deployment-methods/
If you are using Git, I would recommend the Git Post-Receive Hook approach. It simply builds the site after the new code is received:
GIT_REPO=$HOME/myrepo.git
TMP_GIT_CLONE=$HOME/tmp/myrepo
PUBLIC_WWW=/var/www/myrepo
git clone $GIT_REPO $TMP_GIT_CLONE
jekyll build -s $TMP_GIT_CLONE -d $PUBLIC_WWW
rm -Rf $TMP_GIT_CLONE
exit
Since you mentioned that it will be updated by a non-technical person, you might try something like rack-jekyll to automatically rebuild when new files are FTP'd.
I am attempting to mirror a directory on a remote server using rsync. However, I would like a copy of all newly created files to be stored in a separate directory on the local machine.
For example, if a new file is added on the remote server, I would like it to mirror regularly (for example, to ~/mirror), but save an additional copy of only the new file in another folder, (for example, ~/staging). To be clear, only the new files should appear in staging.
My first approach was to allow rsync to update the timestamps, and then use that to make a copy. However, I would now like to preserve timestamps.
Can anyone provide ideas on a simple approach? I am open to use of additional utilities other than rsync.
You might consider making hardlinks in the extra directory.
ln --force --target-directory=~/staging ~/mirror/*
Edit:
If this is a Linux system, incron will trigger on inotify events and would allow you to make copies of files as they are added to a directory you specify.
What methods are available for backing up repositories in a Windows environment?
You could use something like (Linux):
svnadmin dump repositorypath | gzip > backupname.svn.gz
Since Windows does not support GZip it is just:
svnadmin dump repositorypath > backupname.svn
We use svnadmin hotcopy, e.g.:
svnadmin hotcopy C:\svn\repo D:\backups\svn\repo
As per the book:
You can run this command at any time and make a safe copy of the repository, regardless of whether other processes are using the repository.
You can of course ZIP (preferably 7-Zip) the backup copy. IMHO It's the most straightforward of the backup options: in case of disaster there's little to do other than unzip it back into position.
There's a hotbackup.py script available on the Subversion web site that's quite handy for automating backups.
http://svn.apache.org/repos/asf/subversion/trunk/tools/backup/hot-backup.py.in
Here is a Perl script that will:
Backup the repo
Copy it to another server via SCP
Retrieve the backup
Create a test repository from the backup
Do a test checkout
Email you with any errors (via cron)
The script:
my $svn_repo = "/var/svn";
my $bkup_dir = "/home/backup_user/backups";
my $bkup_file = "my_backup-";
my $tmp_dir = "/home/backup_user/tmp";
my $bkup_svr = "my.backup.com";
my $bkup_svr_login = "backup";
$bkup_file = $bkup_file . `date +%Y%m%d-%H%M`;
chomp $bkup_file;
my $youngest = `svnlook youngest $svn_repo`;
chomp $youngest;
my $dump_command = "svnadmin -q dump $svn_repo > $bkup_dir/$bkup_file ";
print "\nDumping Subversion repo $svn_repo to $bkup_file...\n";
print `$dump_command`;
print "Backing up through revision $youngest... \n";
print "\nCompressing dump file...\n";
print `gzip -9 $bkup_dir/$bkup_file\n`;
chomp $bkup_file;
my $zipped_file = $bkup_dir . "/" . $bkup_file . ".gz";
print "\nCreated $zipped_file\n";
print `scp $zipped_file $bkup_svr_login\#$bkup_svr:/home/backup/`;
print "\n$bkup_file.gz transfered to $bkup_svr\n";
#Test Backup
print "\n---------------------------------------\n";
print "Testing Backup";
print "\n---------------------------------------\n";
print "Downloading $bkup_file.gz from $bkup_svr\n";
print `scp $bkup_svr_login\#$bkup_svr:/home/backup/$bkup_file.gz $tmp_dir/`;
print "Unzipping $bkup_file.gz\n";
print `gunzip $tmp_dir/$bkup_file.gz`;
print "Creating test repository\n";
print `svnadmin create $tmp_dir/test_repo`;
print "Loading repository\n";
print `svnadmin -q load $tmp_dir/test_repo < $tmp_dir/$bkup_file`;
print "Checking out repository\n";
print `svn -q co file://$tmp_dir/test_repo $tmp_dir/test_checkout`;
print "Cleaning up\n";
print `rm -f $tmp_dir/$bkup_file`;
print `rm -rf $tmp_dir/test_checkout`;
print `rm -rf $tmp_dir/test_repo`;
Script source and more details about the rational for this type of backup.
I use svnsync, which sets up a remote server as a mirror/slave. We had a server go down two weeks ago, and I was able to switch the slave into primary position quite easily (only had to reset the UUID on the slave repository to the original).
Another benefit is that the sync can be run by a middle-man, rather than as a task on either server. I've had a client to two VPNs sync a repository between them.
svnadmin hotcopy
svnadmin hotcopy REPOS_PATH NEW_REPOS_PATH
This subcommand makes a full “hot” backup of your repository, including all hooks, configuration files, and, of course, database files.
svnbackup over at Google Code, a .NET console application.
You can create a repository backup (dump) with svnadmin dump.
You can then import it in using svnadmin load.
Detailed reference in the SVNBook:
"Repository data migration using svnadmin"
Basically it's safe to copy the repository folder if the svn server is stopped. (source: https://groups.google.com/forum/?fromgroups#!topic/visualsvn/i_55khUBrys%5B1-25%5D )
So if you're allowed to stop the server, do it and just copy the repository, either with some script or a backup tool. Cobian Backup fits here nicely as it can stop and start services automatically, and it can do incremental backups so you're only backing up parts of repository that have changed recently (useful if the repository is large and you're backing up to remote location).
Example:
Install Cobian Backup
Add a backup task:
Set source to repository folder (e.g. C:\Repositories\),
Add pre-backup event "STOP_SERVICE" VisualSVN,
Add post-backup event, "START_SERVICE" VisualSVN,
Set other options as needed. We've set up incremental backups including removal of old ones, backup schedule, destination, compression incl. archive splitting etc.
Profit!
there are 2 main methods to backup a svn server, first is hotcopy that will create a copy of your repository files, the main problem with this approach is that it saves data about the underlying file system, so you may have some difficulties trying to repostore this kind of backup in another svn server kind or another machine.
there is another type of backup called dump, this backup wont save any information of the underlying file system and its potable to any kind of SVN server based in tigiris.org subversion.
about the backup tool you can use the svnadmin tool(it is able to do hotcopy and dump) from the command prompt, this console resides in the same directory where your svn server lives or you can google for svn backup tools.
my recommendation is that you do both kinds of backups and get them out of the office to your email acount, amazon s3 service, ftp, or azure services, that way you will have a securityy backup without having to host the svn server somewhere out of your office.
Here a GUI Windows tool for make a dump of local and remote subversion repositories:
https://falsinsoft-software.blogspot.com/p/svn-backup-tool.html
The tool description says:
This simply tool allow to make a dump backup of a local and remote subversion repository. The software work in the same way of the "svnadmin" but is not a GUI frontend over it. Instead use directly the subversion libraries for allow to create dump in standalone mode without any other additional tool.
Hope this help...
I like to just copy the entire repo directory to my backup location. That way, if something happens, you can just copy the directory back and be ready to go immediately.
Just make sure to preserve permissions, if needed. Usually, this is only a concern on Linux machines.
For hosted repositories you can since svn version 1.7 use svnrdump, which is analogous to svnadmin dump for local repositories. This article provides a nice walk-through, which essentially boils down to:
svnrdump dump /URL/to/remote/repository > myRepository.dump
After you have downloaded the dump file you can import it locally
svnadmin load /path/to/local/repository < myRepository.dump
or upload it to the host of your choice.
If you are using the FSFS repository format (the default), then you can copy the repository itself to make a backup. With the older BerkleyDB system, the repository is not platform independent and you would generally want to use svnadmin dump.
The svnbook documentation topic for backup recommends the svnadmin hotcopy command, as it will take care of issues like files in use and such.
#echo off
set hour=%time:~0,2%
if "%hour:~0,1%"==" " set hour=0%time:~1,1%
set folder=%date:~6,4%%date:~3,2%%date:~0,2%%hour%%time:~3,2%
echo Performing Backup
md "\\HOME\Development\Backups\SubVersion\%folder%"
svnadmin dump "C:\Users\Yakyb\Desktop\MainRepositary\Jake" | "C:\Program Files\7-Zip\7z.exe" a "\\HOME\Development\Backups\SubVersion\%folder%\Jake.7z" -sibackupname.svn
This is the Batch File i have running that performs my Backups
For the daily and full backup solution just use the SVN backup scripts here.
I have compiled the steps I followed for the purpose of taking a backup of the remote SVN
repository of my project.
install svk (http://svk.bestpractical.com/view/SVKWin32)
install svn (http://sourceforge.net/projects/win32svn/files/1.6.16/Setup-Subversion-1.6.16.msi/download)
svk mirror //local <remote repository URL>
svk sync //local
This takes time and says that it is fetching the logs from repository. It creates a set of files inside C:\Documents and Settings\nverma\.svk\local.
To update this local repository with the latest set of changes from the remote one, just run the previous command from time to time.
Now you can play with your local repository (/home/user/.svk/local in this example) as if it were a normal SVN repository!
The only problem with this approach is that the local repository is created with a revision increments by the actual revision in the remote repository. As someone wrote:
The svk miror command generates a commit in the just created repository. So all the commits created by the subsequent sync will have revision numbers incremented by one as compared to the remote public repository.
But, this was OK for me as I only wanted some backup of the remote repository time to time, nothing else.
Verification:
To verify, use the SVN client with the local repository like this:
svn checkout "file:///C:/Documents and Settings\nverma/.svk/local/" <local-dir-path-to-checkout-onto>
This command then goes to checkout the latest revision from the local repository. At the end it says Checked out revision N. This N was one more than the actual revision found in the remote repository (due to the problem mentioned above).
To verify that svk also brought all the history, the SVN checkout was run with various older revisions using -r with 2, 10, 50 etc. Then the files in <local-dir-path-to-checkout-onto> were confirmed to be from that revision.
At the end, zip the directory C:/Documents and Settings\nverma/.svk/local/ and store the zip somewhere. Keep doing this regularly.
as others have said, hot-backup.py from the Subversion team has some nice features over just plain svnadmin hotcopy
I run a scheduled task on a python script that spiders for all my repositories on the machine, and uses hotbackup to keep several days worth of hotcopies (paranoid of corruption) and an svnadmin svndump on a remote machine. Restoration is really easy from that - so far.
1.1 Create Dump from SVN (Subversion) repository
svnadmin dump /path/to/reponame > /path/to/reponame.dump
Real example
svnadmin dump /var/www/svn/testrepo > /backups/testrepo.dump
1.2 Gzip Created Dump
gzip -9 /path/to/reponame.dump
Real example
gzip -9 /backups/testrepo.dump
1.3 SVN Dump and Gzip Dump with One-liner
svnadmin dump /path/to/reponame | gzip -9 > /path/to/reponame.dump.gz
Real example
svnadmin dump /var/www/svn/testrepo |Â gzip -9 > /backups/testrepo.dump.gz
How to Backup (dump) and Restore (load) SVN (Subversion) repository on Linux.
Ref: svn subversion backup andrestore