Is there a way to backup & restore apache Nifi application data like nifi-flow-audit.h2.db & controller services configuration data?
The controller services configuration is serialized into $NIFI_HOME/conf/flow.xml.gz. You can copy this file to any NiFi instance with the same nifi.sensitive.props.key value in $NIFI_HOME/conf/nifi.properties and it will be used to define the entire flow (CS, processors, reporting tasks, etc.).
The nifi-flow-audit.h2.db is a local database containing the audit records of the flow changes. You can copy that file to remote disk and restore it from there as well.
In general, you want to back up everything in the $NIFI_HOME/conf directory to restore the application configuration, and the three repositories (content_repository, flowfile_repository, and provenance_repository) if you want the exact state of the application. More information on the repositories and setting up NiFi for easier upgrades here.
Check out the backup options in Nifi Toolkit's File Manager.
Sample command
# backup NiFi installation
# option -t may be provided to ensure backup of external boostrap.conf file
./file-manager.sh
-o backup
–b /tmp/nifi_bak
–c /usr/nifi_old
-v
Source: https://nifi.apache.org/docs/nifi-docs/components/nifi-docs/html/toolkit-guide.html#file-manager
Related
I noticed someone creating a bunch of scripts to run on GemFire clusters, where they have multiple copies of the same script where the only difference between the scripts is the server name.
Here is a picture of the Github repo
What the script looks like:
#!/bin/bash
source /sys_data/gemfire/scripts/gf-common.env
#----------------------------------------------------------
# Start the servers
#----------------------------------------------------------
(ssh -n <SERVER_HOST_NAME_HERE> ". ${GF_INST_HOME}/scripts/gfsh-server.sh gf_cache1 start")
SERVER_HOST_NAME_HERE = the IP address or server name that the script was designed for, removed for the purposes of this questions.
I would like to create one script with a parameter for the server name. Problem is: I'm not exactly sure where the best location would be to store/retrieve the server ip/host name(s), and let the script reference it, any ideas? The number of cache servers will vary depending on environment, application, and cluster.
Our development pipeline should work like this ideally:
Users commit a file to GitHub repo
Triggers Jenkins job
Jenkins job copies file to each cache server, shuts down that server using the stop_cache.sh script, then runs the start_cache.sh script. The number of cache servers can vary from cluster to cluster.
GemFire cache servers are updated with new file.
Went with the method suggested by #nos
Right now you have them hardcoded in each file it seems. So extract them to a separate file(s), loop through entries in that file and run for host in $(cat cache_hostnames.txt) ; ./stop_cache.sh $host ; done and something similar for other kinds of services?
Placed the server names in a file, and looped through the file.
This project might be of interest:
https://github.com/Pivotal-Data-Engineering/gemfire-manager
I want to back my DynamoDB local server. I have install DynamoDB server in Linux machine. Some sites are refer to create a BASH file in Linux os and connect to S3 bucket, but in local machine we don't have S3 bucket.
So i am stuck with my work, Please help me Thanks
You need to find the database file created by DynamoDb local. From the docs:
-dbPath value — The directory where DynamoDB will write its database file. If you do not specify this option, the file will be written to
the current directory. Note that you cannot specify both -dbPath and
-inMemory at once.
The file name would be of the form youraccesskeyid_region.db. If you used the -sharedDb option, the file name would be shared-local-instance.db
By default, the file is created in the directory from which you ran dynamodb local. To restore you'll have to the copy the same file and while running dynamodb, specify the same dbPath.
I want to upgrade Teamcity 9.16 to 10. I want to proceed with manual back-up and then restore it . I am using external database - mysql . i want to upgrade database as well. How should i proceed with this?
TeamCity documentation states that
Backups created with TeamCity 6.0+ can be restored using the same or
more recent TeamCity versions
so you should be able to create a backup in TC9 and then restore it in TC10.
The simplest way to create a backup is to navigate to the Administration | Backup section in the server UI to specify some parameters and run the backup, as described here.
The other options are
backup via the maintainDB command-line tool — it is basically the same option, as backup via the UI
manual backup
which are described on the corresponding page of the TC documentation.
Restoring data from backup is performed using the maintainDB tool, basically the steps for your case are:
install new TeamCity (but do not start the server)
create a new empty Data Directory
create and configure an empty database
configure a temporary database.properties file
place the database drivers into the lib/jdbc in new data directory
use the maintainDB utility located in the <TeamCity Home>/bin to run restore command:
maintainDB.[cmd|sh] restore -A <absolute path to the Data Directory> -F <path to the TeamCity backup file> -T <absolute path to the database.properties file>
If the process completes successfully, copy over
/system/artifacts from the old directory
More details could be found on the corresponding page.
I am using spring cloud config server to host a centralized location for all the property files configurations to be used in the project.
I tried using the config files from a local file system using below and it works fine:
spring.profiles.active=native
spring.cloud.config.server.native.searchLocations=file://${HOME}/Documents/test-config/cloud-config-properties/
I also used the git repo using: spring.cloud.config.server.git.uri=ssh://xxxxxx.com:7999/test/cloud-config-properties.git
I would like to try using a combination of this in my project.
Example - for dev/test profile - i would like to use from local filesystem and for the production - I would like to use Git repository.
I enabled both the git uri and native profiles in my application.properties in config server application. But the properties are always picked up from the local file system. Is this possible?
Not supported out of the box, however there is a workaround for this. You can define the basedir for the configuration server, which is where it saves the files it fetches from the remote server, by setting the property (in the config server):
spring.cloud.config.server.git.basedir=<your_dir>
If you are working with docker, you can map this directory to the host filesystem.
Now whatever file you put in there will be picked up by configuration-server if it matches any of the application/profile in the request. For example you could put a file there called application-dynamic.properties, and have all your clients use dynamic as the last profile, for example
spring.profiles.active=systesting,dynamic
This way everything you will put in application-dynamic.properties will override whatever is defined in your config repo.
One thing to notice though is that you need to add the file only after configuartion server starts, because it deletes this folder during startup.
Needles to say, it's not a good practice doing this in production (for example a restart will cause the file to be deleted), but for test/dev this is the best option.
I am working on setting up a load balancing cluster on windows server 2012 and have a shared drive where I want the configuration files for Apache to exist at. This way each member of the LB can load the exact same config files. How do I change where the config file is located independently of where the ServerRoot is?
Start the Apache process with the -d parameter and give your alternative ServerRoot as an argument, though I'd imagine it would be a much better idea for you to use some mechanism to sync the files locally to each server.
Also read http://httpd.apache.org/docs/2.4/mod/core.html#mutex, as it's advised if you're running from a networked file system.
If you just want to specify the main config file, start the process with the -f parameter and the path to the config file as an argument.