List item
pipelines:
default:
- step:
name: Push changes to Commerce Cloud
script:
- dcu --putAll $OCCS_CODE_LOCATION --node $OCCS_ADMIN_URL --applicationKey $OCCS_APPLICATION_KEY
- step:
name: Publish changes Live Storefront
image: Python 3.5.1
script:
python publishDCUAuthoredChanges.py -u $OCCS_ADMIN_URL -k $OCCS_APPLICATION_KEY
environment variables:
$OCCS_CODE_LOCATION: Path to location of all OCCS code
$OCCS_ADMIN_URL: URL for the administration interface on the target Commerce Cloud instance
$OCCS_APPLICATION_KEY: application key to use to log into the target Commerce Cloud administration interface
So I want to use Azure Dev Repository to CI / CD.
in the above code block if you see I have specified - dcu & python code in two task.
dcu is nodejs third party oracle tool which needed to be used to migrate code to cloud system. I want to know how to use that tool in azure dev ops,
Second python (or) nodejs which I want to invoke to REST api to publish the changes.
So where to place those files and how do we invoke it.
*********** Update **************
I hosted the self pool agent and able to access the system.
Just start executing basic bash code, but end up in two issue -
1) the git extract files from the repository it is going to _work/1/s, not sure how that path is decided. How can I change that location s
2) I did 'pwd' to the correct path but it fails in 'dcu' command. I tried with npm and other few commands it fails. But things like mkdir , rmdir it create & remove folder correctly from the desired path. when I tried the 'dcu' cmd from the terminal manually from the system it works fine as expected.
You can follow below steps to use DCU tool and python in azure pipelines.
1, create a azure git repo to include dcu zip file and your .py files. You can follow the steps in this thread to create a azure git repo and push local files to azure repo.
2, create azure build pipeline. Please check here to create a yaml pipeline. Here is a good tutorial for you to get started.
To create a classic UI pipeline, please choose Use the classic editor in the pipeline setup wizard, and choose start with an Empty job to start with an empty pipeline and add your own steps.(I will use classic UI pipeline in below example.)
3, Click "+" and search for Extract files task to unzip the DCU zip file. Click the 3dots on the Destination folder field to select a destination folder for extracted dcu files. eg. $(agent.builddirectory). Please check my answer in this thread more information about predefined variables
4, click "+" to add a powershell task. Run below script in screenshot to install dcu and run dcu command. For environment variables (like $OCCS_CODE_LOCATION), please click the variables tab in below screenshot to define them
cd $(agent.builddirectory) #the folder where the unzipped dcu files reside. eg. $(agent.builddirectory)
npm install -g
.\dcu.cmd --putAll $(OCCS_CODE_LOCATION) --node $(OCCS_ADMIN_URL) --applicationKey $(OCCS_APPLICATION_KEY)
5, add Use python version task to define a python version to execute your .py file.
6, add Python script task to run your .py file. Click the 3dots on Script path field to locate your publishDCUAuthoredChanges.py file(this py file and the dcu zip file have been pushed to azure git repo in the above step 1).
You should be able to run the script of above question in the azure devops pipeline.
Update:
_work/1/s is the default working folder for the agent. You cannot change it. Though there are ways to change the location where the source code is cloned from git, the tasks' workingdirectory is still from the default folder.
However, You can change the workingdirectory inside the tasks. And there are predefined variables you can use to refer to the places in the agents. For below example:
$(Agent.BuildDirectory) is mapped to c:\agent_work\1
%(Build.ArtifactStagingDirectory) is mapped to c:\agent_work\1\a
$(Build.BinariesDirectory) is mapped to c:\agent_work\1\b
$(Build.SourcesDirectory) is mapped to c:\agent_work\1\s
The .sh scripts in the _temp folder are generated automatically by the agent which contains the scripts in the bash task.
For above dcu command not found error. You can try adding dcu command path to the system variables path for your local machine's environment variables. (path in user variables cannot be found by agent jobs, For the agent use a different user account to connect to local machine)
.
Or you can use the physically path to dcu command in the bash task. For example let's say the dcu.cmd in the c:\dcu\dcu.cmd on local machine. Then in the bash task use below script to run dcu command.
c:/dcu/dcu.cmd --putAll ...
Is there a way to backup & restore apache Nifi application data like nifi-flow-audit.h2.db & controller services configuration data?
The controller services configuration is serialized into $NIFI_HOME/conf/flow.xml.gz. You can copy this file to any NiFi instance with the same nifi.sensitive.props.key value in $NIFI_HOME/conf/nifi.properties and it will be used to define the entire flow (CS, processors, reporting tasks, etc.).
The nifi-flow-audit.h2.db is a local database containing the audit records of the flow changes. You can copy that file to remote disk and restore it from there as well.
In general, you want to back up everything in the $NIFI_HOME/conf directory to restore the application configuration, and the three repositories (content_repository, flowfile_repository, and provenance_repository) if you want the exact state of the application. More information on the repositories and setting up NiFi for easier upgrades here.
Check out the backup options in Nifi Toolkit's File Manager.
Sample command
# backup NiFi installation
# option -t may be provided to ensure backup of external boostrap.conf file
./file-manager.sh
-o backup
–b /tmp/nifi_bak
–c /usr/nifi_old
-v
Source: https://nifi.apache.org/docs/nifi-docs/components/nifi-docs/html/toolkit-guide.html#file-manager
This is really weird.
I am trying a clean Teamcity 9.1.1 install but the Data Directory is nowhere to be found.
if I access the Global Settings tab under Administration, it lists "C:\Windows\System32\config\systemprofile.BuildServer" - a folder that doesn't exist.
if I try to browse to that folder, it shows me a range of files; uploading a specific file there instead uploads it to C:\Windows\SysWOW64\config\systemprofile.BuildServer.
there is no teamcity-startup.properties file anywhere - I am unable to customize the location of the data directory.
when I restore a backup, the backup files are instead restored to C:\Users\[user name]\.BuildServer rather than in the correct data directory.
Does anyone has any suggestions on how to regain control of the situation? How can I tell TeamCity which data folder to use?
I resolved the situation by:
stopping TC services;
creating a teamcity-startup.properties in [install folder]\conf with the following content:
teamcity.data.path=D:\\[install folder]\\config
restarting TC services;
restoring my backup.
This restored the 9.1.1 install as well as stabilizing the location of the data directory. After this was done, the subsequent installation of 9.1.7 prompted me to uninstall 9.1.1 first (which it hadn’t done the first time around) and the upgrade succeeded.
I believe the system was already compromised at the beginning, unknown to me, due to the data folder being all over the place. Once that was resolved, everything else fell into place.
I've installed Collabnet Subversion Edge, and would like to make sure I have it backed up properly. I would like NOT to use the CloudBackup service offered.
I've went to the administration interface for collabnet (localhost:3343) and went to Repositories > Backup Schedule. There, one can choose between 3 different 'Type of Job':
Cloud Services Backup
Full Dump Backup
Hotcopy Backup
Neither lets you choose where to copy the backup. I've tried looking up how this works, but documentation seems to be lacking a lot.
What is the best way to backup such a repository? Shall I just keep a copy of the entire collabnet folder (c:\csvn)?
The Subversion Edge admin UI lets you specify the folder for backups. It defaults to a folder inside the normal data folder, but you can specify a different value. So, for example, if you have a D:\ drive that you want the backups to go on you can just specify that folder in the settings and the backups will go to that folder.
It does need to be a physically accessible hard drive though.
See the Backup Directory configuration item in this screenshot:
https://ctf.open.collab.net/sf/projects/svnedge/screenshots/screens/config/config.png
You can use Windows Server Backup to backup Subversion repositories. It allows you to shedule backups
to a network share, dedicated backup volume, writeable media. For example, wbadmin command-line tool allows you to safely backup your repositories. This simple command performs one-time copy backup of C:\foo\bar to X:\ volume:
wbadmin start backup –backupTarget:x: -include:c\foo\bar -vsscopy
(To install Windows Server Backup, run ocsetup WindowsServerBackup in elevated command-prompt).
You can setup backup in different ways:
wbadmin command-line tool,
PowerShell cmdlets, good for automation and customization of backup actions,
Windows Server Backup wizard (control panel, actually) MMC snap-in.
It's not required to stop server's service when you run the backup because FSFS repository backend is always in consistent state.
Here are general tips about recovering Subversion repository from a backup:
Recover repository backup to an empty directory to make sure that restored repository files won't mix with files of the broken one. After repository if recovered, you can delete broken repository and then replace it with the recovered one.
Stop-start cycle your Subversion server after recovering repository from a backup.
If your clients get errors after repository recover, run svnadmin recover against it. The command finishes instantly and makes repository accessible again.
If you have access to the repository directories then you should be able to use hotcopy directly and specify where the backups go.
It's enough to take a periodical backup of just csvn/data directory where all your repositories and configuration files are stored.
Visit this link for backup (and upgrade) options. The contents in the link is added below. Hope it helps.
Manual Upgrade/Reinstallation Steps
Subversion Edge includes an integrated mechanism for installing updates. This is the preferred way to do an upgrade as it handles whatever steps are needed to perform the upgrade and can be done remotely from your web browser. However, there are scenarios where you might want or need to do an upgrade manually, for example your Subversion Edge server might not be able to access the Internet to pull down the updates or maybe one or more critical installation files have become corrupted and you need to reinstall using the same version. Here are the steps for performing a manual upgrade or reinstallation:
Windows
If your existing Subversion Edge installation was installed using the installer from Subversion Edge 2.0.0 or later, then all you need to do to upgrade is download the latest installer and run it. This will uninstall the current version and install the new version (which is how the Windows Installer (.msi) process works for upgrades).
If you are not sure what version you installed with, you can always safely use this approach:
Stop the existing services and uninstall the current version from the
Windows Control Panel. This will leave behind your C:\csvn folder and
any files in it that have been modified since the original install.
Delete everything in the C:\csvn folder EXCEPT the data folder. So
you should be left with just the C:\csvn\data folder.
Install the new version. The installer will pick up the existing data folder and when the services start it will basically just be an upgrade to the new
version.
WARNING: Take note of this reported bug and backup the svn_access_file first:
artf7081 - Using Windows installer for updates can overwrite the svn_access_file
Linux/Solaris
To upgrade a Linux/Solaris installation, this is the safest way to do it:
Stop the servers $ bin/csvn stop $ bin/csvn-httpd stop
Rename the csvn folder $ mv csvn csvn-old
Untar the new release as a non-root user
Move the data folder back into the new release
$ mv csvn-old/data csvn
Important! Copy "dist" configuration files to data folder
$ cp -f csvn/dist/*.dist csvn/data/conf
Start the servers
$ bin/csvn start
$ bin/csvn-httpd start
What methods are available for backing up repositories in a Windows environment?
You could use something like (Linux):
svnadmin dump repositorypath | gzip > backupname.svn.gz
Since Windows does not support GZip it is just:
svnadmin dump repositorypath > backupname.svn
We use svnadmin hotcopy, e.g.:
svnadmin hotcopy C:\svn\repo D:\backups\svn\repo
As per the book:
You can run this command at any time and make a safe copy of the repository, regardless of whether other processes are using the repository.
You can of course ZIP (preferably 7-Zip) the backup copy. IMHO It's the most straightforward of the backup options: in case of disaster there's little to do other than unzip it back into position.
There's a hotbackup.py script available on the Subversion web site that's quite handy for automating backups.
http://svn.apache.org/repos/asf/subversion/trunk/tools/backup/hot-backup.py.in
Here is a Perl script that will:
Backup the repo
Copy it to another server via SCP
Retrieve the backup
Create a test repository from the backup
Do a test checkout
Email you with any errors (via cron)
The script:
my $svn_repo = "/var/svn";
my $bkup_dir = "/home/backup_user/backups";
my $bkup_file = "my_backup-";
my $tmp_dir = "/home/backup_user/tmp";
my $bkup_svr = "my.backup.com";
my $bkup_svr_login = "backup";
$bkup_file = $bkup_file . `date +%Y%m%d-%H%M`;
chomp $bkup_file;
my $youngest = `svnlook youngest $svn_repo`;
chomp $youngest;
my $dump_command = "svnadmin -q dump $svn_repo > $bkup_dir/$bkup_file ";
print "\nDumping Subversion repo $svn_repo to $bkup_file...\n";
print `$dump_command`;
print "Backing up through revision $youngest... \n";
print "\nCompressing dump file...\n";
print `gzip -9 $bkup_dir/$bkup_file\n`;
chomp $bkup_file;
my $zipped_file = $bkup_dir . "/" . $bkup_file . ".gz";
print "\nCreated $zipped_file\n";
print `scp $zipped_file $bkup_svr_login\#$bkup_svr:/home/backup/`;
print "\n$bkup_file.gz transfered to $bkup_svr\n";
#Test Backup
print "\n---------------------------------------\n";
print "Testing Backup";
print "\n---------------------------------------\n";
print "Downloading $bkup_file.gz from $bkup_svr\n";
print `scp $bkup_svr_login\#$bkup_svr:/home/backup/$bkup_file.gz $tmp_dir/`;
print "Unzipping $bkup_file.gz\n";
print `gunzip $tmp_dir/$bkup_file.gz`;
print "Creating test repository\n";
print `svnadmin create $tmp_dir/test_repo`;
print "Loading repository\n";
print `svnadmin -q load $tmp_dir/test_repo < $tmp_dir/$bkup_file`;
print "Checking out repository\n";
print `svn -q co file://$tmp_dir/test_repo $tmp_dir/test_checkout`;
print "Cleaning up\n";
print `rm -f $tmp_dir/$bkup_file`;
print `rm -rf $tmp_dir/test_checkout`;
print `rm -rf $tmp_dir/test_repo`;
Script source and more details about the rational for this type of backup.
I use svnsync, which sets up a remote server as a mirror/slave. We had a server go down two weeks ago, and I was able to switch the slave into primary position quite easily (only had to reset the UUID on the slave repository to the original).
Another benefit is that the sync can be run by a middle-man, rather than as a task on either server. I've had a client to two VPNs sync a repository between them.
svnadmin hotcopy
svnadmin hotcopy REPOS_PATH NEW_REPOS_PATH
This subcommand makes a full “hot” backup of your repository, including all hooks, configuration files, and, of course, database files.
svnbackup over at Google Code, a .NET console application.
You can create a repository backup (dump) with svnadmin dump.
You can then import it in using svnadmin load.
Detailed reference in the SVNBook:
"Repository data migration using svnadmin"
Basically it's safe to copy the repository folder if the svn server is stopped. (source: https://groups.google.com/forum/?fromgroups#!topic/visualsvn/i_55khUBrys%5B1-25%5D )
So if you're allowed to stop the server, do it and just copy the repository, either with some script or a backup tool. Cobian Backup fits here nicely as it can stop and start services automatically, and it can do incremental backups so you're only backing up parts of repository that have changed recently (useful if the repository is large and you're backing up to remote location).
Example:
Install Cobian Backup
Add a backup task:
Set source to repository folder (e.g. C:\Repositories\),
Add pre-backup event "STOP_SERVICE" VisualSVN,
Add post-backup event, "START_SERVICE" VisualSVN,
Set other options as needed. We've set up incremental backups including removal of old ones, backup schedule, destination, compression incl. archive splitting etc.
Profit!
there are 2 main methods to backup a svn server, first is hotcopy that will create a copy of your repository files, the main problem with this approach is that it saves data about the underlying file system, so you may have some difficulties trying to repostore this kind of backup in another svn server kind or another machine.
there is another type of backup called dump, this backup wont save any information of the underlying file system and its potable to any kind of SVN server based in tigiris.org subversion.
about the backup tool you can use the svnadmin tool(it is able to do hotcopy and dump) from the command prompt, this console resides in the same directory where your svn server lives or you can google for svn backup tools.
my recommendation is that you do both kinds of backups and get them out of the office to your email acount, amazon s3 service, ftp, or azure services, that way you will have a securityy backup without having to host the svn server somewhere out of your office.
Here a GUI Windows tool for make a dump of local and remote subversion repositories:
https://falsinsoft-software.blogspot.com/p/svn-backup-tool.html
The tool description says:
This simply tool allow to make a dump backup of a local and remote subversion repository. The software work in the same way of the "svnadmin" but is not a GUI frontend over it. Instead use directly the subversion libraries for allow to create dump in standalone mode without any other additional tool.
Hope this help...
I like to just copy the entire repo directory to my backup location. That way, if something happens, you can just copy the directory back and be ready to go immediately.
Just make sure to preserve permissions, if needed. Usually, this is only a concern on Linux machines.
For hosted repositories you can since svn version 1.7 use svnrdump, which is analogous to svnadmin dump for local repositories. This article provides a nice walk-through, which essentially boils down to:
svnrdump dump /URL/to/remote/repository > myRepository.dump
After you have downloaded the dump file you can import it locally
svnadmin load /path/to/local/repository < myRepository.dump
or upload it to the host of your choice.
If you are using the FSFS repository format (the default), then you can copy the repository itself to make a backup. With the older BerkleyDB system, the repository is not platform independent and you would generally want to use svnadmin dump.
The svnbook documentation topic for backup recommends the svnadmin hotcopy command, as it will take care of issues like files in use and such.
#echo off
set hour=%time:~0,2%
if "%hour:~0,1%"==" " set hour=0%time:~1,1%
set folder=%date:~6,4%%date:~3,2%%date:~0,2%%hour%%time:~3,2%
echo Performing Backup
md "\\HOME\Development\Backups\SubVersion\%folder%"
svnadmin dump "C:\Users\Yakyb\Desktop\MainRepositary\Jake" | "C:\Program Files\7-Zip\7z.exe" a "\\HOME\Development\Backups\SubVersion\%folder%\Jake.7z" -sibackupname.svn
This is the Batch File i have running that performs my Backups
For the daily and full backup solution just use the SVN backup scripts here.
I have compiled the steps I followed for the purpose of taking a backup of the remote SVN
repository of my project.
install svk (http://svk.bestpractical.com/view/SVKWin32)
install svn (http://sourceforge.net/projects/win32svn/files/1.6.16/Setup-Subversion-1.6.16.msi/download)
svk mirror //local <remote repository URL>
svk sync //local
This takes time and says that it is fetching the logs from repository. It creates a set of files inside C:\Documents and Settings\nverma\.svk\local.
To update this local repository with the latest set of changes from the remote one, just run the previous command from time to time.
Now you can play with your local repository (/home/user/.svk/local in this example) as if it were a normal SVN repository!
The only problem with this approach is that the local repository is created with a revision increments by the actual revision in the remote repository. As someone wrote:
The svk miror command generates a commit in the just created repository. So all the commits created by the subsequent sync will have revision numbers incremented by one as compared to the remote public repository.
But, this was OK for me as I only wanted some backup of the remote repository time to time, nothing else.
Verification:
To verify, use the SVN client with the local repository like this:
svn checkout "file:///C:/Documents and Settings\nverma/.svk/local/" <local-dir-path-to-checkout-onto>
This command then goes to checkout the latest revision from the local repository. At the end it says Checked out revision N. This N was one more than the actual revision found in the remote repository (due to the problem mentioned above).
To verify that svk also brought all the history, the SVN checkout was run with various older revisions using -r with 2, 10, 50 etc. Then the files in <local-dir-path-to-checkout-onto> were confirmed to be from that revision.
At the end, zip the directory C:/Documents and Settings\nverma/.svk/local/ and store the zip somewhere. Keep doing this regularly.
as others have said, hot-backup.py from the Subversion team has some nice features over just plain svnadmin hotcopy
I run a scheduled task on a python script that spiders for all my repositories on the machine, and uses hotbackup to keep several days worth of hotcopies (paranoid of corruption) and an svnadmin svndump on a remote machine. Restoration is really easy from that - so far.
1.1 Create Dump from SVN (Subversion) repository
svnadmin dump /path/to/reponame > /path/to/reponame.dump
Real example
svnadmin dump /var/www/svn/testrepo > /backups/testrepo.dump
1.2 Gzip Created Dump
gzip -9 /path/to/reponame.dump
Real example
gzip -9 /backups/testrepo.dump
1.3 SVN Dump and Gzip Dump with One-liner
svnadmin dump /path/to/reponame | gzip -9 > /path/to/reponame.dump.gz
Real example
svnadmin dump /var/www/svn/testrepo |Â gzip -9 > /backups/testrepo.dump.gz
How to Backup (dump) and Restore (load) SVN (Subversion) repository on Linux.
Ref: svn subversion backup andrestore