shell script,how to print the newline in default position always? - bash

i am using tput sc/rc/ed and printf '\E[n<A|B|C|D>'|printf '\E[y;xH',here two ex:
tty_esc(){ printf "\e[%s" "$1"; }
tty_cursor_locate(){ tty_esc "${2};${1}H"; }
tty_cursor_right(){ tty_esc ${1}C; }
print_center()
{
local _width=$(tput cols)
local _str=$1
local _row=$2
local _cols=$((((${_width} - ${#_str})) / 2))
tty_cursor_locate ${_cols:-100} ${_row:-1}
printf "%s\n" " ${_str} "
}
show_net_adapter()
{
local _addr _iface _count
local _origin=$1
iface_line_count=
tty_cursor_locate ${_origin:-0} 4
printf "%s\n" "Current connected adapter(s):"
for _iface in $(get_net_adapter);do
if [[ "${_iface}" != "lo" ]];then
_addr=$(get_net_addr ${_iface})
test -z "${_addr}" && continue
let _count+=1
let iface_line_count+=1
if [[ ${_count} != 1 ]];then
unset _count
printf '%s' "${tty_rever}"
fi
tty_cursor_right ${_origin:-0}
print_fill 50 ${_iface} ${_addr:--}
printf "${tty_reset}"
fi
done
print_line -s "=" ${line_origin}
}
as above, I should locate the cursor before I print something.
BTW
I use trap "myfunc" WINCH, it only works once. when I try again to change my crt. size, it doesn't work.

Not really sure what you mean by in default position always, either intentations or overwrite the text but the following function can do both.
#!/bin/sh
preent () { # $1=indent start line, $2=spaces, $3=reset cursor, $4=update (1=itself, 2=delete previous lines, 0=newline), $5=save cursor, $6=text
if text="$6""$7""$8""$9""${10}" awk -v col="$COLUMNS" -v cp="$_CURSOR_POS" -v id="$1" -v spc="$2" -v rc="$3" -v u="$4" -v sc="$5" '
BEGIN { t=ENVIRON["text"]; lt=sprintf("%d",col/4); idx=1; len=length(t); y=0; sp=sprintf("% " spc "s",""); delete A;
for(ln=1;idx<len;ln++) { if(ln==id) { col=col-spc; y=1 } bd=col-lt; f=0
for(i=idx+col-1;i>bd;i--) { c=substr(t,i,1);
if(c==" ") {
if(y) { A[ln]=sprintf("%s%s",sp,substr(t,idx,i-idx)); idx=i+1; f=1; break
} else { A[ln]=substr(t,idx,i-idx); idx=i+1; f=1; break }
} else if(c=="") {
if(y) { A[ln]=sprintf("%s%s",sp,substr(t,idx,i-idx)); idx=i; f=1; break
} else { A[ln]=substr(t,idx,i-idx); idx=i; f=1; break; }
}
}
if(!f) {
if(y) { A[ln]=sprintf("%s%s",sp,substr(t,idx,col)); idx=idx+col
} else { A[ln]=substr(t,idx,col); idx=idx+col }
}
} if(rc=="true") cp=0;
if(u=="1") { for(i=1;i<ln;i++) printf("\x1B[1A\x1B[K"); cp=cp+1-i
} else if(u=="2") { for(i=0;i<cp;i++) printf("\x1B[1A\x1B[K"); cp=0 }
for(i=1;i<ln;i++) printf("%s\n",A[i]);
if(sc=="true") { exit ln-1+cp } else { exit 0 }
}'; then _CURSOR_POS="$?"; else _CURSOR_POS="$?"; fi
}
Example 1 (Long text, 5 spaces indent, indent starts from 2nd line)
preent 2 5 'false' 0 'false' 'You never say good-bye Handong-an monghani udukoni anja Dashi saenggakhatjiman ' \
'Momchul sun optkesso Ontong kudae saenggak hal subakke omnun ' \
"Nae jashini miwo Don't you let me go Baby don't you let me down"
Output
You never say good-bye Handong-an monghani
udukoni anja Dashi saenggakhatjiman Momchul
sun optkesso Ontong kudae saenggak hal
subakke omnun Nae jashini miwo Don't you
let me go Baby don't you let me down
Example 2 (Long text with no spaces)
preent 2 5 'false' 0 'false' 'Youneversaygood-byeHandong-anmonghaniudukonianjaDashisaenggakhatjiman' \
'MomchulsunoptkessoOntongkudaesaenggakhalsubakkeomnun' \
"NaejashinimiwoDon'tyouletmegoBabydon'tyouletmedown"
Output
Youneversaygood-byeHandong-anmonghaniudukonianjaD
ashisaenggakhatjimanMomchulsunoptkessoOntong
kudaesaenggakhalsubakkeomnunNaejashinimiwoDo
n'tyouletmegoBabydon'tyouletmedown

Related

I need to create a file with all hashicorp vault key value pairs data using shell script

I need to create a file with all hashicorp vault key value pairs data using shell script.
I want to dump all the data from vault to a flat file.
please advice best way to do it.
Thanks in advance
Prudhvi
Just for keys and values you can use my little Perl script 'vault-backup', that also freezes the data using the correct vault commands.
Please note that this does NOT create a full backup of your Vault! There are no methods being backed up, or any other (unlistable) stuff outside the secrets. It's only usable for simple keys and values. It also probably isn't usable for multiline or binary values. You can patch the script to support that, if you like. ;)
#!/usr/bin/perl
#
# Usage: vault-backup [<PATH> [stdout]]
use Data::Dumper;
use Storable qw(freeze thaw);
# Set vault environment variables
# Always end with a " && " for the actual command
my $setenv =
"VAULT_ADDR=https://myvault.somewhere.com:8200 && ".
"VAULT_CA_PATH=/etc/yourcertificates/ && ";
my $path = $ARGV[0] || "secret/";
if ($path!~/\/$/) {
$path="$path/";
}
push #list, getData($path);
if ($ARGV[1] eq "stdout") {
print Dumper(\#list);
} else {
my $fn="vault-backup-frozen-".time().".dat";
open W,">$fn";
print W freeze(\#list);
close W;
print STDERR "Wrote data to $fn\n";
}
sub getData {
my $path=shift;
print STDERR "Starting getData($path)\n";
my #ret=();
my $command="$setenv vault kv list -tls-skip-verify $path | tail -n+3 ";
print STDERR "starting command: $command\n";
my #lines = `$command`;
chomp #lines;
foreach my $line (#lines) {
if ($line=~/\/$/) {
my #result = getData($path.$line);
if (scalar(#result)>0) {
# Find deeper results
push #ret, #result;
} else {
# empty final dir, no values
push #ret, { path => $path.$line };
}
} else {
# Found a key!
my $command="$setenv vault kv get -tls-skip-verify $path$line";
print STDERR "starting command: $command\n";
my $values = `$command`;
push #ret, {path=>$path.$line, value=>$values};
}
}
return #ret;
}
To restore the data, you can use the script below. It handles data only, it does not act on metadata.
#!/usr/bin/perl
# Usage: vault-restore <backup-filename>
use Data::Dumper;
use Storable qw(thaw);
my %all_entries;
# Set vault environment variables
# Always end with a " && " for the actual command
my $setenv =
"VAULT_ADDR=https://myothervault.somewhere.com:8200 && ".
"VAULT_CA_PATH=/etc/mycertificates/ && ";
# Read the data
my $fn = $ARGV[0] || die("I need a filename with the frozen data");
open F,"<$fn";
my #list = #{ thaw(join("",<F>)) };
close F;
print STDERR "Read ".scalar(#list)." entries.\n";
# Process the data
foreach my $entry (#list) {
print STDERR "\n# adding entry -> $entry->{path}\n";
addEntry($entry);
}
foreach my $path (keys %all_entries) {
my $keyvalues="";
foreach my $key (keys %{$all_entries{$path}}) {
my $value=$all_entries{$path}{$key};
$keyvalues.="'$key=$value' ";
}
print STDERR "vault kv put $path $keyvalues\n";
# `$command`;
}
sub addEntry {
my $entry=shift;
my $path = $entry->{'path'};
if ($entry->{'value'}) {
my $values = $entry->{value};
my #list=split("\n", $values);
my $metadata_engage=0;
my $data_engage=0;
foreach my $keyvalue (#list) {
if ($keyvalue=~/==== Metadata ====/) {
$metadata_engage=1;
$data_engage=0;
} elsif ($keyvalue=~/==== Data ====/) {
$metadata_engage=0;
$data_engage=1;
} elsif ($data_engage) {
my ($key,$value)=($keyvalue=~/^([^ ]+) +(.*)$/);
if ($key ne "Key" && $key ne "---") {
# print STDERR "key=$key ; value=$value\n";
$all_entries{$path}{$key}=$value;
} else {
# print STDERR "-- separator\n";
}
}
}
} else {
print STDERR "Found a final but empty path: $path\n";
}
}

Local `commit-msg` doesn't work in conjunction with global `prepare-commit-msg`

When I have my global git hooks directory (which contains just a prepare-commit-msg hook) set up in config, my local commit-msg doesn't run (although the global hook does). However, when I disable the global prepare-commit-msg hook (by commenting out core.hookspath in gitconfig), the local commit-msg hook works just fine.
~/dotfiles/git-hooks/prepare-commit-msg
#!/usr/bin/env bash
pcregrep -Mv '(# Please.*|# with.*|^#$\n(?!#)|^#$(?=\n# On))' $1 > /tmp/msg && cat /tmp/msg > $1
./.git/hooks/commit-msg (Gerrit's hook to add change-id's if necessary, trimmed to remove license comments)
#!/bin/sh
...
unset GREP_OPTIONS
CHANGE_ID_AFTER="Bug|Depends-On|Issue|Test|Feature|Fixes|Fixed"
MSG="$1"
# Check for, and add if missing, a unique Change-Id
#
add_ChangeId() {
clean_message=`sed -e '
/^diff --git .*/{
s///
q
}
/^Signed-off-by:/d
/^#/d
' "$MSG" | git stripspace`
if test -z "$clean_message"
then
return
fi
# Do not add Change-Id to temp commits
if echo "$clean_message" | head -1 | grep -q '^\(fixup\|squash\)!'
then
return
fi
if test "false" = "`git config --bool --get gerrit.createChangeId`"
then
return
fi
# Does Change-Id: already exist? if so, exit (no change).
if grep -i '^Change-Id:' "$MSG" >/dev/null
then
return
fi
id=`_gen_ChangeId`
T="$MSG.tmp.$$"
AWK=awk
if [ -x /usr/xpg4/bin/awk ]; then
# Solaris AWK is just too broken
AWK=/usr/xpg4/bin/awk
fi
# Get core.commentChar from git config or use default symbol
commentChar=`git config --get core.commentChar`
commentChar=${commentChar:-#}
# How this works:
# - parse the commit message as (textLine+ blankLine*)*
# - assume textLine+ to be a footer until proven otherwise
# - exception: the first block is not footer (as it is the title)
# - read textLine+ into a variable
# - then count blankLines
# - once the next textLine appears, print textLine+ blankLine* as these
# aren't footer
# - in END, the last textLine+ block is available for footer parsing
$AWK '
BEGIN {
if (match(ENVIRON["OS"], "Windows")) {
RS="\r?\n" # Required on recent Cygwin
}
# while we start with the assumption that textLine+
# is a footer, the first block is not.
isFooter = 0
footerComment = 0
blankLines = 0
}
# Skip lines starting with commentChar without any spaces before it.
/^'"$commentChar"'/ { next }
# Skip the line starting with the diff command and everything after it,
# up to the end of the file, assuming it is only patch data.
# If more than one line before the diff was empty, strip all but one.
/^diff --git / {
blankLines = 0
while (getline) { }
next
}
# Count blank lines outside footer comments
/^$/ && (footerComment == 0) {
blankLines++
next
}
# Catch footer comment
/^\[[a-zA-Z0-9-]+:/ && (isFooter == 1) {
footerComment = 1
}
/]$/ && (footerComment == 1) {
footerComment = 2
}
# We have a non-blank line after blank lines. Handle this.
(blankLines > 0) {
print lines
for (i = 0; i < blankLines; i++) {
print ""
}
lines = ""
blankLines = 0
isFooter = 1
footerComment = 0
}
# Detect that the current block is not the footer
(footerComment == 0) && (!/^\[?[a-zA-Z0-9-]+:/ || /^[a-zA-Z0-9-]+:\/\//) {
isFooter = 0
}
{
# We need this information about the current last comment line
if (footerComment == 2) {
footerComment = 0
}
if (lines != "") {
lines = lines "\n";
}
lines = lines $0
}
# Footer handling:
# If the last block is considered a footer, splice in the Change-Id at the
# right place.
# Look for the right place to inject Change-Id by considering
# CHANGE_ID_AFTER. Keys listed in it (case insensitive) come first,
# then Change-Id, then everything else (eg. Signed-off-by:).
#
# Otherwise just print the last block, a new line and the Change-Id as a
# block of its own.
END {
unprinted = 1
if (isFooter == 0) {
print lines "\n"
lines = ""
}
changeIdAfter = "^(" tolower("'"$CHANGE_ID_AFTER"'") "):"
numlines = split(lines, footer, "\n")
for (line = 1; line <= numlines; line++) {
if (unprinted && match(tolower(footer[line]), changeIdAfter) != 1) {
unprinted = 0
print "Change-Id: I'"$id"'"
}
print footer[line]
}
if (unprinted) {
print "Change-Id: I'"$id"'"
}
}' "$MSG" > "$T" && mv "$T" "$MSG" || rm -f "$T"
}
_gen_ChangeIdInput() {
echo "tree `git write-tree`"
if parent=`git rev-parse "HEAD^0" 2>/dev/null`
then
echo "parent $parent"
fi
echo "author `git var GIT_AUTHOR_IDENT`"
echo "committer `git var GIT_COMMITTER_IDENT`"
echo
printf '%s' "$clean_message"
}
_gen_ChangeId() {
_gen_ChangeIdInput |
git hash-object -t commit --stdin
}
add_ChangeId
~/dotfiles/gitconfig (trimmed) when global hook is enabled.
...
[core]
editor = code -rwg $1:2
excludesfile = /Users/shreyasminocha/.gitignore
compression = 0
hookspath = /Users/shreyasminocha/dotfiles/git-hooks
...
Git version: 2.18.0
Edit: As #phd pointed out in the comments, "The problem is that global hookspath completely takes over local hooks. If the global hookspath is defined local hooks are never consulted. [I had] to create global /Users/shreyasminocha/dotfiles/git-hooks/commit-msg that [would] run local .git/hooks/commit-msg". Confirming duplicate.
Eventually solved this by adding some code to the global hook, just as #phd suggested. It looks for a local hook and runs it if it exists.
Global prepare-commit-msg hook:
#!/usr/bin/env bash
# if a local hook exists, run it
if [ -e ./.git/hooks/prepare-commit-msg ]; then
./.git/hooks/prepare-commit-msg "$#"
fi
if [ -e ./.git/hooks/commit-msg ]; then
./.git/hooks/commit-msg "$#"
fi
# global_hook-related things follow
pcregrep -Mv '(# Please.*|# with.*|^#$\n(?!#)|^#$(?=\n# On))' $1 > /tmp/msg && cat /tmp/msg > $1

Out of memory with running JavaScript by PhantomJS

My shell script is written in cygwin for windows:
// main.sh
#!/bin/bash
[ "$#" -lt 1 ] && echo "Usage: thisscript.sh <filename.txt>" && exit 0
filename=`basename -s .txt $1`
i=0
while [ $i == 0 ]
do
phantomjs --web-security=no myXHR.js $filename.txt
logLastLine=`tail -n 1 $filename.log`
if [[ "$logLastLine" =~ "Error" ]]; then
echo "Error occurs, now keep looping it..."
elseif [[ "$logLastLine" =~ "503" ]]; then
echo "Error occurs, now keep looping it..."
elseif [[ "$logLastLine" =~ "500" ]]; then
echo "Error occurs, now keep looping it..."
else
echo "Complete! Exiting the execution..."
i=1
fi
done
And here are the codes contained in the myXHR.js
// myXHR.js
phantom.onError = function(msg, trace) {
console.log("PhantomJS Error");
phantom.exit();
};
var fs = require('fs'), system = require('system');
if (system.args.length < 2) {
console.log("Usage: myXHR.js <FILE>");
}
var content = '',
f = null,
lines = null,
eol = "\n";
try {
f = fs.open(system.args[1], "r");
filename=system.args[1].replace(/\.txt/,"");
content = f.read();
} catch (e) {
console.log(e);
}
if (f) {
f.close();
}
var request = new XMLHttpRequest();
if (content) {
lines = content.split(eol);
for (i=0; i<(lines.length-1);i++) {
request.open('GET', "http://stackoverflow.com/", false);
request.send();
if (request.status === 200) {
try {
fs.write($filename.log, line[i] + "Succeed!", 'a');
} catch(e) {
console.log(e);
}
} else {
try {
fs.write($filename.log, line[i] + "Error!", 'a');
} catch(e) {
console.log(e);
}
}
}
phantom.exit();
To illustrate, the javascript, executed by PhantomJS, are reading 1st argument(a filename.txt file), passed into the shell script, line by line. For each line it sends a XMLHttpRequest to check the request status and writes it into filename.log file.
Error status number includes 503 and 500. Luckily these statuses are less likely to occur again if I resend the same XMLHttpRequest. So what I need to do is to set up a error handler which is for resend the same XMLHttpRequest when errors occur.
In this error handler, I use X=${tail -n 1 log} to see if there is a error status number(containing "503" or "500" string). For instance, if [[ "$X" =~ "503" ]]; then restart the execution of the javascript, by not giving i=1 and while loop never exits. Until it has finished reading the last line of the imported file without any error status numbers.
(I know it is awkward to handle error like this, but it was a quick solution that came to my mind.)
But this is theoretical. In practice, this script ended with an error "Memory exhausted". I reckon this error is triggered by the large amount of lines(>100k) in the $1 file, and it occurs in the JavaScript execution part. I used free -m command to get memory usage information, and I noticed that when Javascript is running, the used swap is increasing!
Could anybody teach me how to release the memory when the scripts is being executed.

bash replace multiline text in file, with pattern

Hi I need to edit some file but I din´t want to do it manually, I know that using sed command I can edit files using command line, but in this case, I don't know how to match the pattern to edit. for example I have this file:
(
AMI1
{
type patch; // <- relpace patch by cyclicAMI;
nFaces 1350;
startFace 2433406;
}
inlet
{
type patch;
nFaces 1125;
startFace 2434756;
}
outlet
{
type patch;
nFaces 1125;
startFace 2435881;
}
AMI2
{
type patch; // <- relpace patch by cyclicAMI;
nFaces 2850;
startFace 2440606;
}
)
And I want to edit ONLY the AMI keys to look like this:
(
AMI1
{
type cyclicAMI; // <-- Replaced
inGroups 1(cyclicAMI); // <-- Add
nFaces 1350;
startFace 2433406;
matchTolerance 0.0001; // <-- Add
transform noOrdering; // <-- Add
neighbourPatch AMI2; // <-- Add AMI2 in AMI1
}
inlet
{
type patch;
nFaces 1125;
startFace 2434756;
}
outlet
{
type patch;
nFaces 1125;
startFace 2435881;
}
AMI2
{
type cyclicAMI; // <-- Replaced
inGroups 1(cyclicAMI); // <-- Add
nFaces 2850;
startFace 2440606;
matchTolerance 0.0001; // <-- Add
transform noOrdering; // <-- Add
neighbourPatch AMI1; // <-- Add AMI1 in AMI2
}
)
Not overly elegant, but working.
state=0
while IFS= read -r line; do
case "${line// }" in
AMI[12]) state=${line##*AMI}
echo "$line";;
typepatch\;*) echo " type cyclicAMI;"
echo " inGroups 1(cyclicAMI);";;
else
echo "$line"
fi;;
\}) if [ "$state" != 0 ]; then
echo " matchTolerance 0.0001;"
echo " transform noOrdering;"
echo " neighbourPatch AMI$((3-state));"
echo " }"
state=0
else
echo " }"
fi;;
*) echo "$line"
esac
done < textfile.txt

how to improve Performance for this code?

this code is running on a file of 200M lines at least. and this takes a lot of time
I would like to know if I can improve the runtime of this loop.
my #bin_lsit; #list of 0's and 1's
while (my $line = $input_io->getline) {
if ($bin_list[$i]) {
$line =~ s/^.{3}/XXX/;
} else {
$line =~ s/^.{3}/YYY/;
}
$output_io->appendln($line);
$i++;
}
A regex solution may be overkill here. How about replacing the if/else blocks with:
substr($line, 0, 3, $bin_list[$i] ? 'XXX' : 'YYY';
Smallest change is probably to buffer between appendln's
my #bin_lsit; #list of 0's and 1's
my $i = 0;
while (my $line = $input_io->getline) {
if ($bin_list[$i]) {
$line =~ s/^.{3}/XXX/;
} else {
$line =~ s/^.{3}/YYY/;
}
$buffer .= $line;
if ( $i % 1000 == 0 ) {
$output_io->appendln($buffer);
$buffer = '';
}
$i++;
}
if ( $buffer ne '' ) {
$output_io->appendln($buffer);
}
Are you using IO::All?
I couldn't find anything else with appendln...
Replacing this:
my $input_io = io 'tmp.this';
my $output_io = io 'tmp.out';
while (my $line = $input_io->getline ) {
$output_io->appendln($line);
}
With this:
open(IFH, 'tmp.this');
open(OFH, '>>tmp.out');
while (my $line = <IFH> ) {
print OFH $line;
}
close IFH;
close OFH;
Is quite a bit faster (1 sec vs 23 in my test case).

Resources