I have this bash function that prints lines between Mode: org and # End of org. I would like for matched sections to be separated by a blank line.
capture ()
{
local efile="$1"
local charcl begorg endorg
charcl_ere='^[[:space:]]*([#;!]+|#c|\/\/)[[:space:]]*'
charcl_bre='^[[:space:]]*\([#;!]\+\|#c\|\/\/\)[[:space:]]*'
begorg="${charcl_bre}"'Mode: org$'
endorg="${charcl_bre}"'# End of org$'
mdr='^Mode: org$' ; edr='^# End of org$'
sed -n "/$begorg/,/$endorg/ s/$charcl_bre//p" "$efile" |
sed "/$mdr\|$edr/d"
}
This is the input
cat /home/flora/docs/recnotes.txt
## Mode: org
# Assigns shell positional parameters or changes the values of shell
# options. The -- option assigns the positional parameters to the
# arguments of {set}, even when some of them start with an option
# prefix `-'.
## # End of org
;; Mode: org
; Assigns shell positional parameters or changes the values of shell
; options. The -- option assigns the positional parameters to the
; arguments of {set}, even when some of them start with an option
; prefix `-'.
;; # End of org
#c Mode: org
#c Assigns shell positional parameters or changes the values of shell
#c options. The -- option assigns the positional parameters to the
#c arguments of {set}, even when some of them start with an option
#c prefix `-'.
#c # End of org
I assumed by lines between that you didn't mean including the lines with Mode: org and # End of org. You didn't specify that. Try:
perl -nlE'$p=/Mode: org/?1:/# End of org/?0*say$":$p&&$p+say' recnotes.txt
If it's important to avoid the empty separation line at the end, try:
perl -nlE'$p=/Mode: org/?1:/# End of org/?0*do{eof||say("")}:$p&&$p+say' r.txt
I have come up with a straightforward way using
sed -n "/$begorg/,/$endorg/ s/$charcl_bre//p" "$efile" |
sed "/$mdr/d" | sed "s/$edr//"
Related
Basically what foo(**bar) does in python, here I’d want something like
foo **bar.yaml
and that would become
foo --bar1=1 --bar2=2
Where bar.yaml would be
bar1: 1
bar2: 2
You could use a combination of sed and xargs:
sed -E 's/^(.+):[[:space:]]+(.+)$/--\1=\2/' bar.yaml | xargs -d '\n' foo
sed converts the format of bar.yaml lines (e.g. bar1: 1 -> --bar1=1) and xargs feeds the converted lines as arguments to foo.
You could of course modify/extend the sed part to support other formats or single-dash options like -v.
To test if this does what you want, you can run this Bash script instead of foo:
#!/usr/bin/env bash
echo "Arguments: $#"
for ((i=1; i <= $#; i++)); do
echo "Argument $i: '${!i}'"
done
Here's a version for zsh. Run this code or add it to ~/.zshrc:
function _yamlExpand {
setopt local_options extended_glob
# 'words' array contains the current command line
# yaml filename is the last value
yamlFile=${words[-1]}
# parse 'key : value' lines from file, create associative array
typeset -A parms=("${(#s.:.)${(f)"$(<${yamlFile})"}}")
# trim leading and trailing whitespace from keys and values
# requires extended_glob
parms=("${(kv#)${(kv#)parms##[[:space:]]##}%%[[:space:]]##}")
# add -- and = to create flags
typeset -a flags
for key val in "${(#kv)parms}"; do
flags+=("--${key}='${val}'")
done
# replace the value on the command line
compadd -QU -- "$flags"
}
# add the function as a completion and map it to ctrl-y
compdef -k _yamlExpand expand-or-complete '^Y'
At the zsh shell prompt, type in the command and the yaml file name:
% print -l -- ./bar.yaml▃
With the cursor immediately after the yaml file name, hit ctrl+y. The yaml filename will be replaced with the expanded parameters:
% print -l -- --bar1='1' --bar2='2' ▃
Now you're set; you can hit enter, or add parameters, just like any other command line.
Notes:
This only supports the yaml subset in your example.
You can add more yaml parsing to the function, possibly with yq.
In this version, the cursor must be next to the yaml filename - otherwise the last value in words will be empty. You can add code to detect that case and then alter the words array with compset -n.
compadd and compset are described in the zshcompwid man page.
zshcompsys has details on compdef; the section on autoloaded files describes another way to deploy something like this.
I have a variable equal to a string, which is a series of key/value pairs separated by newlines.
I want to then replace these newline characters with spaces, and set a new variable equal to the result
From various answers on the internet I've arrived at the following:
#test.txt has the content:
#test=example
#what=s0omething
vars="$(cat ./test.txt)"
formattedVars= $("$vars" | tr '\n' ' ')
echo "$taliskerEnvVars"
Problem is when I try to set formattedVars it tries to execute the second line:
script.sh: line 7: test=example
what=s0omething: command not found
I just want formattedVars to equal test=example what=s0omething
What trick am I missing?
Change your line to:
formattedVars=$(tr '\n' ' ' <<< "$secretsContent")
Notice the space of = in your code, which is not permitted in assignment statements.
I see that you are not setting secretsContent in your code, you are setting vars instead.
If possible, use an array to hold contents of the file:
readarray -t vars < ./test.txt # bash 4
or
# bash 3.x
declare -a vars
while IFS= read -r line; do
vars+=( "$line" )
done < ./test.txt
Then you can do what you need with the array. You can make your space-separated list with
formattedVars="${vars[*]}"
, but consider whether you need to. If the goal is to use them as a pre-command modifier, use, for instance,
"${vars[#]}" my_command arg1 arg2
I wrote a simple bash function which would read value from an ini file (defined by variable CONF_FILE) and output it
getConfValue() {
#getConfValue section variable
#return value of a specific variable from given section of a conf file
section=$1
var="$2"
val=$(sed -nr "/\[$section\]/,/\[/{/$var/p}" $CONF_FILE)
val=${val#$var=}
echo "$val"
}
The problem is that it does not ignore comments and runs into troubles if multiple variables within a section names share common substrings.
Example ini file:
[general]
# TEST=old
; TEST=comment
TEST=new
TESTING=this will be output too
PATH=/tmp/test
Running getConfValue general PATH would output /tmp/test as expected, but running getConfValue general TEST shows all the problems this approach has.
How to fix that?
I know there are dedicated tools like crudini or tools for python, perl and php out in the wild, but I do not want to add extra dependencies for simple config file parsing. A solution incorporating awk instead of sed would be just fine too.
Sticking with sed you could anchor your var search to the start of the record using ^ and end it with an equal sign:
"/\[$section\]/,/\[/{/^$var=/p}"
If you are concerned about whitespace in front of your record you could account for that:
"/\[$section\]/,/\[/{/^(\W|)$var=/p}"
That ^(\W|)$var= says "if there is whitespace at the beginning (^(\W) or nothing (|)) before your variable concatenated with an equal sign ($var=)."
If you wanted to switch over to awk you could use something like:
val=$(awk -F"=" -v section=$section -v var=$var '$1=="["section"]"{secFound=1}secFound==1 && $1==var{print $2; secFound=0}' $CONF_FILE)
That awk command splits the record by equal -F"=". Then if the first field in the record is your section ($1=="["section"]") then set variable secFound to 1. Then... if secFound is 1 and the first field is exactly equal to your var variable (secFound==1 && $1==var) then print out the second field ({print $2}) and sets secFound to 0 so we don't pick up any other Test keys.
I encountered this problem and came up with a solution similar to others here.
The main difference is it uses a single awk call to get a response suitable for creating an associative array of the property/value pairs for a section.
This will not ignore the commented properties. Though adding something to do that should not be to hard.
Here's a testing script demonstrating the awk and declare statements used;
#!/bin/bash
#
# Parse a INI style properties file and extract property values for a given section
#
# Author: Alan Carlyle
# License: CC0 (https://creativecommons.org/about/cclicenses/)
#
# Example Input: (example.properties)
# [SEC1]
# item1=value1
# item2="value 2"
#
# [Section 2]
# property 1="Value 1 of 'Section 2'"
# property 2='Single "quoted" value'
#
# Usage:
# $ read_props example.properties "Section 2" property\ 2
# $ Single "quoted" value
#
# Section names and properties with spaces do not need to be quoted.
# Values with spaces must be quoted. Values can use single or double quotes.
# The following characters [ = ] can not be used in names or values.
#
# If the property is not provided the the whole section is outputed.
#
propertiesFile=$1
section=$2
property=$3
# Extract the propetites for the section formated as for associtive array
sectionData="( "$(awk -F'=' -v s="$section" '/^\[/{ gsub(/[\[\]]/, "", $1); f = ($1 == s); next }
NF && f{ print "["$1"]="$2 }' $propertiesFile)" )"
# Create associtive array from extracted section data
declare +x -A "properties=$sectionData"
if [ -z "$property" ]
then
echo $sectionData
else
echo ${properties[$property]}
fi
I am creating a bash script to modify and summarize information with grep and sed. But it gets stuck.
#!/bin/bash
# This script extracts some basic information
# from text files and prints it to screen.
#
# Usage: ./myscript.sh </path/to/text-file>
#Extract lines starting with ">#HWI"
ONLY=`grep -v ^\>#HWI`
#replaces A and G with R in lines
ONLYR=`sed -e s/A/R/g -e s/G/R/g $ONLY`
grep R $ONLYR | wc -l
The correct way to write a shell script to do what you seem to be trying to do is:
awk '
!/^>#HWI/ {
gsub(/[AG]/,"R")
if (/R/) {
++cnt
}
END { print cnt+0 }
' "$#"
Just put that in the file myscript.sh and execute it as you do today.
To be clear - the bulk of the above code is an awk script, the shell script part is the first and last lines where the shell just calls awk and passes it the input file names.
If you WANT to have intermediate variables then you can create/print them with:
awk '
!/^>#HWI/ {
only = $0
onlyR = only
gsub(/[AG]/,"R",onlyR)
print "only:", only
print "onlyR:", onlyR
if (/R/) {
++cnt
}
END { print cnt+0 }
' "$#"
The above will work robustly, portably, and efficiently on all UNIX systems.
First of all, and as #fedorqui commented - you're not providing grep with a source of input, against which it will perform line matching.
Second, there are some problems in your script, which will result in unwanted behavior in the future, when you decide to manipulate some data:
Store matching lines in an array, or a file from which you'll later read values. The variable ONLY is not the right data structure for the task.
By convention, environment variables (PATH, EDITOR, SHELL, ...) and internal shell variables (BASH_VERSION, RANDOM, ...) are fully capitalized. All other variable names should be lowercase. Since
variable names are case-sensitive, this convention avoids accidentally overriding environmental and internal variables.
Here's a better version of your script, considering these points, but with an open question regarding what you were trying to do in the last line : grep R $ONLYR | wc -l :
#!/bin/bash
# This script extracts some basic information
# from text files and prints it to screen.
#
# Usage: ./myscript.sh </path/to/text-file>
input_file=$1
# Read lines not matching the provided regex, from $input_file
mapfile -t only < <(grep -v '^\>#HWI' "$input_file")
#replaces A and G with R in lines
for((i=0;i<${#only[#]};i++)); do
only[i]="${only[i]//[AG]/R}"
done
# DEBUG
printf '%s\n' "Here are the lines, after relpace:"
printf '%s\n' "${only[#]}"
# I'm not sure what you were trying to do here. Am I gueesing right that you wanted
# to count the number of R's in ALL lines ?
# grep R $ONLYR | wc -l
I am working on completion for a command that takes argument like "one:two:three".
In the simplest terms, I want ':' to be handled just like a space character is by default. Is there a simple way to do this that I am missing?
I've found the ':' is in COMP_WORDBREAKS, but that the character in COMP_WORDBREAKS are also treated as words as well.
So if the commandline is:
cmd one:tw[TAB]
COMP_CWORD will be 3 and COMP_WORDS[COMP_CWORD-1] will be ':'
For comparison, if the commandline is:
cmd one tw[TAB]
COMP_CWORD will be 2 and COMP_WORDS[COMP_CWORD-1] will be 'one'
Even worse is that if you hit the [TAB] right after the ':' delimiter it acts mostly like a space:
cmd one:[TAB]
Now COMP_CWORD will be 2 and COMP_WORDS[COMP_CWORD-1] will be 'one'.
I can parse the commandline myself from COMP_LINE easily enough, but nicer to find a way to just make ':' act like ' ' in my custom completion. Possible?
Unfortunately, not really. This is actually a 'feature' of bash.
While you could modify COMP_WORDBREAKS, modifying COMP_WORDBREAKS could cause other issues as it is a global variable and will affect the behavior of other completion scripts.
If you take a look at the source for bash-completion, two helper methods exist that can help with this:
_get_comp_words_by_ref with the -n option gets the word-to-complete without considering the characters in EXCLUDE as word breaks
# Available VARNAMES:
# cur Return cur via $cur
# prev Return prev via $prev
# words Return words via $words
# cword Return cword via $cword
#
# Available OPTIONS:
# -n EXCLUDE Characters out of $COMP_WORDBREAKS which should NOT be
# considered word breaks. This is useful for things like scp
# where we want to return host:path and not only path, so we
# would pass the colon (:) as -n option in this case.
# -c VARNAME Return cur via $VARNAME
# -p VARNAME Return prev via $VARNAME
# -w VARNAME Return words via $VARNAME
# -i VARNAME Return cword via $VARNAME
#
__ltrim_colon_completions removes colon containing prefix from COMPREPLY items
# word-to-complete.
# With a colon in COMP_WORDBREAKS, words containing
# colons are always completed as entire words if the word to complete contains
# a colon. This function fixes this, by removing the colon-containing-prefix
# from COMPREPLY items.
# The preferred solution is to remove the colon (:) from COMP_WORDBREAKS in
# your .bashrc:
#
# # Remove colon (:) from list of word completion separators
# COMP_WORDBREAKS=${COMP_WORDBREAKS//:}
#
# See also: Bash FAQ - E13) Why does filename completion misbehave if a colon
# appears in the filename? - http://tiswww.case.edu/php/chet/bash/FAQ
# #param $1 current word to complete (cur)
# #modifies global array $COMPREPLY
For example:
{
local cur
_get_comp_words_by_ref -n : cur
__ltrim_colon_completions "$cur"
}
complete -F _thing thing
First take on a custom parsed solution. Love to know if there is a better way:
parms=$(echo "$COMP_LINE" | cut -d ' ' -f 2)
vals="${parms}XYZZY"
IFS=$":"
words=( $vals )
unset IFS
count=${#words[#]}
cur="${words[$count-1]%%XYZZY}"