I really am stumped - I've spent almost two hours searching for an answer to a ridiculously simple question: how can I continually keep two local files in sync on my mac? I investigated various tricks involving rsync, then settled on using lsyncd.
But for the life of me, I can't figure out how I can get lsyncd to sync two specific files. Is this even supported in the API? It was not clear in the documentation whether or not I could use rsync in this manner; I assume that lsyncd is passing CLI options which are preventing this. My configuration is as follows:
sync = {
default.rsync,
source = "/Users/username/Downloads/test1.txt",
target = "/Users/username/Downloads/test2.txt",
rsync = {
binary = "/usr/local/bin/rsync",
archive = "true"
}
}
It just says 'nothing to sync'. Help?
This had worked for me:
sync {
default.rsync,
source = "/Users/username/Downloads/",
target = "/Users/username/Downloads/",
rsync = {
binary = "/usr/bin/rsync",
archive = "true",
_extra = {
"--include=test1.txt",
"--exclude=*"
}
}
}
You have to use the include/exclude feature of lsyncd, which did not come out of the box. You have to use _extra field to set them.
In lsynd you can do like this
settings {
logfile = "/var/log/lsyncd.log",
statusFile = "/var/log/lsyncd-status.log",
statusInterval = 20,
nodaemon = true
}
sync {
default.rsync,
source="/srcdir/",
target="/dstdir/",
rsync = {
archive = true,
compress = true,
whole_file = false,
_extra = { "--include=asterisk", "--exclude=*" },
verbose = true
},
delay=5,
log=all,
}
After start lsyncd i have next
root#localhost:/srcdir# ls
12 aster asterisk
root#localhost:/dstdir# ls
asterisk
Related
Using Octopus Deploy to deploy a simple API.
The first step of our deployment process is to generate an HTML report with the delta of the scripts run vs the scripts required to run. I used this tutorial to create the step.
The relevant code in my console application is:
var reportLocationSection = appConfiguration.GetSection(previewReportCmdLineFlag);
if (reportLocationSection.Value is not null)
{
// Generate a preview file so Octopus Deploy can generate an artifact for approvals
try
{
var report = reportLocationSection.Value;
var fullReportPath = Path.Combine(report, deltaReportName);
Console.WriteLine($"Generating upgrade report at {fullReportPath}");
upgrader.GenerateUpgradeHtmlReport(fullReportPath);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
return operationError;
}
}
The Powershell which I am using in the script step is:
# Get the extracted path for the package
$packagePath = $OctopusParameters["Octopus.Action.Package[DatabaseUpdater].ExtractedPath"]
$connectionString = $OctopusParameters["Project.Database.ConnectionString"]
$reportPath = $OctopusParameters["Project.HtmlReport.Location"]
Write-Host "Report Path: $($reportPath)"
$exeToRun = "$($packagePath)\DatabaseUpdater.exe"
$generatedReport = "$($reportPath)\UpgradeReport.html"
Write-Host "Generated Report: $($generatedReport)"
if ((test-path $reportPath) -eq $false){
New-Item "Creating new directory..."
} else {
New-Item "Directory already exists."
}
# Run this .NET app, passing in the Connection String and a flag
# which tells the app to create a report, but not update the database
& $exeToRun --connectionString="$($connectionString)" --previewReportPath="$($reportPath)"
New-OctopusArtifact -Path "$($generatedReport)"
The error reported by Octopus is:
'Could not find file 'C:\DeltaReports\Some API\2.9.15-DbUp-Test-9\UpgradeReport.html'.'
I'm guessing that is being thrown when this powershell line is hit: New-OctopusArtifact ...
And that seems to indicate that the report was never created.
I've used a bit of logging to log out certain variables and the values look sound:
Report Path: C:\DeltaReports\Some API\2.9.15-DbUp-Test-9
Generated Report: C:\DeltaReports\Some API\2.9.15-DbUp-Test-9\UpgradeReport.html
Generating upgrade report at C:\DeltaReports\Some API\2.9.15-DbUp-Test-9\UpgradeReport.html
As you can see in the C#, the relevant code is wrapped in a try/catch block, but I'm not sure whether the error is being written out there or at a later point by Octopus (I'd need to do a pull request to add a marker in the code).
Can anyone see a way forward win resolving this? Has anyone else encountered this?
Cheers
I recently redid some of the work from that article for this video up on YouTube. I did run into some issues with the .SQL files not being included in the assembly. I think it was after I upgraded to .NET 6. But that might be a coincidence.
Anyway, because the files weren't being included in the assembly, when I ran the command line app via Octopus, it wouldn't properly generate the file for me. I ended up configuring the project to copy the .SQL files to a folder in the output directory instead of embedding them in the assembly. You can view a sample package here.
One thing that helped me is running the app in a debugger with the same parameters just to make sure it was actually generating the file. I'm sure you already thought of that, but I'd be remiss if I forgot to include it in my answer. :)
FWIW, this is my updated scripts.
First, the Octopus Script:
$packagePath = $OctopusParameters["Octopus.Action.Package[Trident.Database].ExtractedPath"]
$connectionString = $OctopusParameters["Project.Connection.String"]
$environmentName = $OctopusParameters["Octopus.Environment.Name"]
$reportPath = $OctopusParameters["Project.Database.Report.Path"]
cd $packagePath
$appToRun = ".\Octopus.Trident.Database.DbUp"
$generatedReport = "$reportPath\UpgradeReport.html"
& $appToRun --ConnectionString="$connectionString" --PreviewReportPath="$reportPath"
New-OctopusArtifact -Path "$generatedReport" -Name "$environmentName.UpgradeReport.html"
My C# code can be found here but for ease of use, you can see it all here (I'm not proud of how I parse the parameters).
static void Main(string[] args)
{
var connectionString = args.FirstOrDefault(x => x.StartsWith("--ConnectionString", StringComparison.OrdinalIgnoreCase));
connectionString = connectionString.Substring(connectionString.IndexOf("=") + 1).Replace(#"""", string.Empty);
var executingPath = Assembly.GetExecutingAssembly().Location.Replace("Octopus.Trident.Database.DbUp", "").Replace(".dll", "").Replace(".exe", "");
Console.WriteLine($"The execution location is {executingPath}");
var deploymentScriptPath = Path.Combine(executingPath, "DeploymentScripts");
Console.WriteLine($"The deployment script path is located at {deploymentScriptPath}");
var postDeploymentScriptsPath = Path.Combine(executingPath, "PostDeploymentScripts");
Console.WriteLine($"The deployment script path is located at {postDeploymentScriptsPath}");
var upgradeEngineBuilder = DeployChanges.To
.SqlDatabase(connectionString, null)
.WithScriptsFromFileSystem(deploymentScriptPath, new SqlScriptOptions { ScriptType = ScriptType.RunOnce, RunGroupOrder = 1 })
.WithScriptsFromFileSystem(postDeploymentScriptsPath, new SqlScriptOptions { ScriptType = ScriptType.RunAlways, RunGroupOrder = 2 })
.WithTransactionPerScript()
.LogToConsole();
var upgrader = upgradeEngineBuilder.Build();
Console.WriteLine("Is upgrade required: " + upgrader.IsUpgradeRequired());
if (args.Any(a => a.StartsWith("--PreviewReportPath", StringComparison.InvariantCultureIgnoreCase)))
{
// Generate a preview file so Octopus Deploy can generate an artifact for approvals
var report = args.FirstOrDefault(x => x.StartsWith("--PreviewReportPath", StringComparison.OrdinalIgnoreCase));
report = report.Substring(report.IndexOf("=") + 1).Replace(#"""", string.Empty);
if (Directory.Exists(report) == false)
{
Directory.CreateDirectory(report);
}
var fullReportPath = Path.Combine(report, "UpgradeReport.html");
if (File.Exists(fullReportPath) == true)
{
File.Delete(fullReportPath);
}
Console.WriteLine($"Generating the report at {fullReportPath}");
upgrader.GenerateUpgradeHtmlReport(fullReportPath);
}
else
{
var result = upgrader.PerformUpgrade();
// Display the result
if (result.Successful)
{
Console.ForegroundColor = ConsoleColor.Green;
Console.WriteLine("Success!");
}
else
{
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine(result.Error);
Console.WriteLine("Failed!");
}
}
}
I hope that helps!
After long and detailed investigation, we discovered the answer was quite obvious.
We assumed the existing deploy process configuration was sound. Because we never had a problem with it (until now). As it transpires, there was a problem which led to the Development deployments being deployed twice.
Hence, the errors like the one above and others which talked about file handles being held by another process.
It was actually obvious in hindsight, but we were blind to it as we thought the existing process was sound 😣
I'm new to packer. I've heard that you can add a vagrant post processor to get you an easy VM to test your new image in. Based on the examples and such I thought the code below would work. However, I get this error.
* Post-processor failed: ovf file couldn't be found
Here's my packer config/code.
source "digitalocean" "test" {
image = "ubuntu-20-10-x64"
region = "nyc1"
size = "s-1vcpu-1gb"
snapshot_name = "me-image-{{isotime \"2006-01-02T15:04\"}}"
snapshot_regions = [
"nyc1", "sgp1", "lon1", "nyc3", "ams3", "fra1", "tor1", "sfo2", "blr1",
"sfo3"
]
tags = ["delete"]
ssh_username = "root"
}
# a build block invokes sources and runs provisioning steps on them.
build {
sources = ["source.digitalocean.test"]
provisioner "file" {
source = "jump_host"
destination = "/tmp"
}
post-processor "vagrant" {
keep_input_artifact = true
provider_override = "virtualbox"
output = "out.box"
}
}
My packer version is 1.6.6
My vagrant version is 2.2.10
Had the Same(ish) Issue - Found the answer by bruteforce/chance
So I'm in the same boat as you, but I managed to find the hint for my solution here
Caveat: I'm working with an exported .vmdk, so this may not be a solution for you since you're looking for a way to get it straight from digital ocean?
The Hint
build {
sources = ["source.null.autogenerated_1"]
post-processor "shell-local" {
inline = ["echo Doing stuff..."]
}
post-processors {
post-processor "vagrant" {
--> include = ["image.iso"]
output = "proxycore_{{.Provider}}.box"
vagrantfile_template = "vagrantfile.tpl"
}
post-processor "vagrant-cloud" {
access_token = "${var.cloud_token}"
box_tag = "hashicorp/precise64"
version = "${local.version}"
}
}
}
This isn't listed on the Vagrant Post-Processor page, but it is on Vagrant Cloud Post-Processor. I just decided to try my luck and it worked.
Working Example
source "null" "example" {
communicator = "none"
}
build {
sources = ["source.null.example"]
post-processor "artifice" {
files = ["example-disk001.vmdk", "example.ovf"]
keep_input_artifact = true
}
post-processor "vagrant" {
include = ["example-disk001.vmdk", "example.ovf"]
keep_input_artifact = true
provider_override = "virtualbox"
}
}
Tl;dr it's not possible
What I wanted packer to do was build something for digitalocean then give me a copy so I could test it without paying for a vm from digitalocean and without needing internet. That isn't possible and after some reflection it makes sense why.
Digitalocean isn't just downloading the Ubuntu 20 ISO and throwing it on their servers. They configure and change the image so its optimized on their hardware. To expect their special images to run on some standard VM running on consumer hardware isn't realistic. Plus I'm not sure there's even a way to download a snapshot from DO.
But also in trying to do this I kind of missed the entire point of vagrant. If I'm testing a digitalocean image I will always need to connect for and pay for digitalocean. Vagrant is designed to make it easy for me to do that without having to click through the interface every single time. So I shouldn't even be trying to get this on my home computer.
PS: Thank you so much #RedGrin-Grumble for taking the time to add to this months old post.
I use SWUpdate to update different Hardware-Revisions of the same device with a double-copy strategy.
The bootloader environmnent of all those looks very similar. However, I have to set the mmc-partition to boot from depending on the active copy and the boot_file depending on the hardware-revision.
To keep the sw-description-file as comprehensive as possible and to make it easy to maintain I would like to set a "basic" boot-environment for all devices in a first step and in a second step overwrite some variables depending on hardware-revision and active copy:
software =
{
version = "1.1";
hardware-compatibility = ["0.1","1.0"];
device1=
{
copy-1:
{
images:
(
{
filename = "rootfs.ext3.gz";
device = "/dev/mmcblk0p3";
compressed = true;
},
{
filename = "u-boot-env-base"; #basic boot environment
type = "uboot";
}
);
bootenv: # device-specific boot variables
(
{
name = "boot_file"
value = "uImage1"
},
{
name = "mmcpart";
value = "3";
}
);
}
}
}
While parsing both bootloader environments are reported but only one is applied or both are, but in the wrong order, because when checking via fw_printenv the "u-boot-env-base" is unaltered.
I am using
SWUpdate v2018.11.0
U-Boot 2018.09.
I feel that I had this working in an older setup (SWUpdate 2016).
I have addressed the mailing list with this question. Stefano Babic, SWUpdate developer and maintainer, answered my question I am just trying to summarize it here.
What I have described is desired behaviour. It is not foreseen to set bootloader variables twice during an update. The u-boot variables defined in a file have priority over u-boot name-value-pairs in the bootenv section because the file is processed in the very end of the update. The solution in my case is to set the variables only in the bootenv section.
I am trying to get the system installation date by running a console app.
The only way I know how to do this is by parsing the /var/log/install.log file for the latest string containing OSInstaller & Install Complete items.
Is there a handy system API I am missing?
As a suggestion for exploration only:
You could try looking at the files /System/Library/Receipts.
You will probably see a com.apple.pkg.BaseSystemResources.plist file, its modification date may tell you when the OS was installed.
There are also com.apple.pkg.update.os.*.plist files for updates, again look at the modification dates and maybe parse the wildcard (*) bit if you can determined a naming convention.
HTH, Happy Hunting!
So in case, someone finds this useful.
Swift 3.0
SOLUTION 1
Initially I parsed /var/log/install.log file to get date string:
// To get OS installation date we'll need to check system log file and find entry which contains installation date
var systemDates : String? = nil
do {
let fullSystemLog = try NSString(contentsOf: URL(fileURLWithPath: "/var/log/install.log"), encoding: String.Encoding.utf8.rawValue)
let entries = fullSystemLog.components(separatedBy: "\n")
//Filter to get only entries about OS installation
let filtered = entries.filter{ entry in
return entry.contains("OSInstaller") && entry.contains("Install Complete") //Markers telling that OS was installed
}
var latestMention = ""
if filtered.count > 0 {
//If 1 or more entries found we'll pick last one
latestMention = filtered.last!
}
else if entries.count > 0 {
//If there are 0 mentions of OS installation - we'll use first entry in logs
latestMention = entries.first!
}
//parse picked entry for date
do {
let regex = try NSRegularExpression(pattern: ".+:[0-9]{2}", options: [])
let nsString = latestMention as NSString
let results = regex.matches(in: latestMention,
options: [], range: NSMakeRange(0, nsString.length))
let actualDateSubstrings = results.map { nsString.substring(with: $0.range)}
if let dateStringFromMention = actualDateSubstrings.first {
systemDates = dateStringFromMention
}
else {
systemDates = "<Error: no date results>"
}
} catch let error as NSError {
systemDates = "<Error: invalid regex: \(error.localizedDescription)>"
}
}
catch {
systemDates = "<Error: system log file not found>"
}
print("\tSYSTEM INSTALLED: \(systemDates)")
SOLUTION 2
The second solution appears much simpler. Look into the InstallDate field of /System/Library/Receipts/com.apple.pkg.BaseSystemResources.plist:
let systemDates = NSDictionary(contentsOfFile: "/System/Library/Receipts/com.apple.pkg.BaseSystemResources.plist")
print("\tSYSTEM INSTALLED: \(systemDates?["InstallDate"])")
I wan't to use lsyncd to create backups of the modified files using as a suffix a date/time string.
If I set the suffix option (in the lsyncd.conf file) using lua, the date/time is computed once, when I start the daemon, and is not updated at each sync iteration.
This leads to the creation of only one backup file per each modified file (with the same suffix) and I wish for the creation of a new backup file per modification per file.
The config file I use is the following:
-- global settings
settings {
delay = 5,
maxProcesses = 5,
statusFile = "<STATUS_FILE_PATH>",
logfile = "<LOG_FILE_PATH>",
insist = true
}
-- target nodes
nodes = {
{ source = "/home/<USER>/sync", target = "<TARGET_IP>:/home/<USER>/sync"},
}
-- execution
time = os.date("*t")
datetime = (time.year .. time.month .. time.day .. time.hour .. time.min .. time.sec)
for _, node in ipairs(nodes) do
sync {
default.rsync,
source = node.source,
target = node.target,
rsync = {
compress = true,
checksum = true,
perms = true,
rsh = "/usr/bin/ssh -i /home/<USER>/.ssh/id_dsa -o StrictHostKeyChecking=no",
times = true,
verbose = true,
_extra = { "--backup", "--suffix=" .. datetime },
}
}
end
If i try to pass the date function of bash in the suffix option, like this:
_extra = { "--backup", "--suffix=_$(date +\"%Y%m%d%H%M%S\")" },
it is converted to a string without computing the value, leading to a backup file with a name like this:
testfile.txt_$(date +"%Y%m%d%H%M%S")
I am limited to using the 2.1.4 version of lsyncd.
Is it possible to create dynamic backup file suffixes?
Not tested. But try this
--suffix=`date +"%F"`