Problem
I have a variable called boiler, and I want the variable si1 to be expressed, and I am unsure of how to do this in a simple and minimal fashion.
boiler='#!/bin/bash
source ../../functions.sh
current="${si1}"
ready custom
title
breadcrumbs \""$current"\" \"Options\"
# END OF BOILER (DO NOT REMOVE ABOVE CODE OR MODIFY IT)
'
ISSUE
The issue is that i want everything to be ignored withing this string (aka printed raw) except for the ${si1} variable.
EXPECTED OUTPUT
How could I concatenate the first part the variable and then the rest of the string while keeping it minimal and saving it back into the boiler variable?
You can delimit the string around ${si1}.
boiler='#!/bin/bash
source ../../functions.sh
current='"${si1}"'
ready custom
title
breadcrumbs \""$current"\" \"Options\"
# END OF BOILER (DO NOT REMOVE ABOVE CODE OR MODIFY IT)
'
This is ordinary string concatenation. The strings delimited with ' will be literal, while the string delimited with " will have the variable expanded.
Difference between single and double quotes in Bash
I'm currently strugling to clean csv files generated automatically with fields containing the csv separator and the field delimiter using sed or awk or via a script.
The source software has no settings to play with to improve the situation.
Format of the csv:
"111111";"text";"";"text with ; and " sometimes "; or ;" multiple times";"user";
Fortunately, the csv is "well" formatted, the exporting software just doesn't escape or replace "forbidden" chars from the fields.
In the last few days I tried to improve my knowledge of regular expression and find expression to clean the files but I failed.
What I managed to do so far:
RegEx to find the fields (I wanted to find the fields and perform a replace inside but I didn't find a way to do it)
(?:";"|^")(.*?)(?=";"|";\n)
RegEx that find semicolon, does not work if the semicolon is the last char of the field only find one per field.
(?:^"|";")(?:.*?)(;)(?:[^"\n].*?)(?=";"|";\n)
RegEx to find the double quotes, seems to pick the first double quote of the line in online regex testers
(?:^"|";")(?:.*?)[^;](")(?:[^;].*?)(?=";"|";\n)
I thought of adding space between each chars in the fields then searching for lonely semi colon and double quotes and remove single space after that but I don't know if it's even possible and seems like a poor solution anyway.
Any standard library should be able to handle it if there is no explicit error in the CSV itself. This is why we have quote-characters and escape characters.
When you create a CSV by yourself - you may forgot handling such cases and let your final output file use this situation. AWK is not a CSV reader but simply a text processing utility.
This is what your row should rather look like.
"111111";"text";"";"text with \; and \" sometimes \"; or ;\" multiple times";"user";
So if you can still re-fetch the data, find a way to export the CSV either through the database's own functionality of csv library for the languages you work with.
In python, this would look like this:-
mywriter = csv.writer(csvfile, delimiter=';', quotechar='"', escapechar="\\")
But if you can't create csv again, the only hope is that you expect some pattern within the fields, as in this question:- parse a csv file that contains commans in the fields with awk
But this is rarely true in textual data - esp comments or posts on a webpage. Another idea in such situations would be to use '\t' as separator.
Looking at some documentation, I saw a multiline string defined using a percent literal:
command %Q{
do this;
do that;
}
In the past, I've always used heredocs when I needed multiline strings:
command <<-heredoc
echo "stuff" | do stuff;
heredoc
What are the semantic differences between them? Is there any reason why I would want to use %Q and not a heredoc?
I tend to evaluate how much text is being used when deciding which to use.
I use %Q when there's not a lot of text (for example, a single line), e.g. %Q|foobar|. The value that %Q provides, is it allows you to easily mix quotes, e.g.
%Q|"Get a Job" ~Mom's words|
I use "heredoc"s when there is a lot of text that spans multiple lines.
For example, suppose you're pasting a lot of text into a REPL (like the content of a YAML file). Unless you traverse the whole file, you can't be certain whether or not you will have a conflict with whatever %Q separator you have chosen. With a "heredoc" you just use some really obscure piece of text that you're fairly certain will not have a conflict, e.g.
<<-BatMobilePrettyObscure
... Lots of text ...
BatMobilePrettyObscure
As far as I know, semantically, there are just a few small differences:
%Q can only use one character to delimit strings
%Q can be multi-line or single-line
"heredoc"s must be Multi-line, with the closing "heredoc" standing alone
%Q delimiters can be "mashed" up against their strings, e.g. %Q|foobar|
There's a funky trick that you can use with heredocs: the first line can be used as if it was a complete string. For example, all of the following examples are valid Ruby code:
puts(<<-EOS)
Hello, world!
EOS
<<-EOS.upcase
Hello, world!
EOS
puts(<<-EOS.upcase)
Hello, world!
EOS
However, you will not find that very often in the wild. Other than that, they are the same as double quoted strings or %Q{} and %{} literals, except that you can choose multi-character delimiters. This comes in handy when all of the possible percent literal delimiters may occur in the string. This especially applies to long strings.
There isn't really a semantic difference, and it doesn't have to do with multiline strings either. All strings can be multiline in Ruby. These are all the same string:
'a
b
'
"a
b
"
%Q{a
b
}
<<-heredoc
a
b
heredoc
The question of which to use is decided by whether you need interpolation and the convenience of escaping characters. For example:
Do you need interpolation? If not then '' or %q()
Will there be lots of quote characters to escape? Then use %Q()
Do you want to write a lot of text without thinking about escaping characters? Use heredocs.
This is a common question, but just can't seem to find the answer without resorting to unreliable regular expressions.
Basically if there is a \302\240 or similar combination in a string I want to replace it with the real character.
I am using PLruby for this, hence the warn.
obj = {"a"=>"some string with special chars"}
warn obj.inspect
NOTICE: {"Outputs"=>["a\302\240b"]} <- chars are escaped
warn "\302\240"
NOTICE: <-- there is a non breaking space here, like I want
warn "#{json.inspect}"
NOTICE: {"Outputs"=>["a\302\240"b]} <- chars are escaped
So these can be decoded when I use a string literal, but with the "#{x}" format the \xxx placeholders are never decoded into characters.
How would I assign the same string as the middle command yields?
Ruby Version: 1.8.5
You mentioned that you're using PL/ruby. That suggests that your strings are actually bytea values (the PostgreSQL version of a BLOB) using the old "escape" format. The escape format encodes non-ASCII values in octal with a leading \ so a bit of gsub and Array#pack should sort you out:
bytes = s.gsub(/\\([0-8]{3})/) { [ $1.to_i(8) ].pack('C') }
That will expand the escape values in s to raw bytes and leave them in bytes. You're still dealing with binary data though so just trying to display it on a console won't necessarily do anything useful. If you know that you're dealing with comprehensible strings then you'll have to figure out what encoding they're in and use String methods to sort out the encoding.
Perhaps you just want to use .to_s instead?
For coding reasons which would horrify you (I'm too embarrassed to say), I need to store a number of text items in a single string.
I will delimit them using a character.
Which character is best to use for this, i.e. which character is the least likely to appear in the text? Must be printable and probably less than 128 in ASCII to avoid locale issues.
I would choose "Unit Separator" ASCII code "US": ASCII 31 (0x1F)
In the old, old days, most things were done serially, without random access. This meant that a few control codes were embedded into ASCII.
ASCII 28 (0x1C) File Separator - Used to indicate separation between files on a data input stream.
ASCII 29 (0x1D) Group Separator - Used to indicate separation between tables on a data input stream (called groups back then).
ASCII 30 (0x1E) Record Separator - Used to indicate separation between records within a table (within a group). These roughly map to a tuple in modern nomenclature.
ASCII 31 (0x1F) Unit Separator - Used to indicate separation between units within a record. The roughly map to fields in modern nomenclature.
Unit Separator is in ASCII, and there is Unicode support for displaying it (typically a "us" in the same glyph) but many fonts don't display it.
If you must display it, I would recommend displaying it in-application, after it was parsed into fields.
Assuming for some embarrassing reason you can't use CSV I'd say go with the data. Take some sample data, and do a simple character count for each value 0-127. Choose one of the ones which doesn't occur. If there is too much choice get a bigger data set. It won't take much time to write, and you'll get the answer best for you.
The answer will be different for different problem domains, so | (pipe) is common in shell scripts, ^ is common in math formulae, and the same is probably true for most other characters.
I personally think I'd go for | (pipe) if given a choice but going with real data is safest.
And whatever you do, make sure you've worked out an escaping scheme!
When using different languages, this symbol: ¬
proved to be the best. However I'm still testing.
Probably | or ^ or ~ you could also combine two characters
You said "printable", but that can include characters such as a tab (0x09) or form feed (0x0c). I almost always choose tabs rather than commas for delimited files, since commas can sometimes appear in text.
(Interestingly enough the ascii table has characters GS (0x1D), RS (0x1E), and US (0x1F) for group, record, and unit separators, whatever those are/were.)
If by "printable" you mean a character that a user could recognize and easily type in, I would go for the pipe | symbol first, with a few other weird characters (# or ~ or ^ or \, or backtick which I can't seem to enter here) as a possibility. These characters +=!$%&*()-'":;<>,.?/ seem like they would be more likely to occur in user input. As for underscore _ and hash # and the brackets {}[] I don't know.
How about you use a CSV style format? Characters can be escaped in a standard CSV format, and there's already a lot of parsers already written.
Can you use a pipe symbol? That's usually the next most common delimiter after comma or tab delimited strings. It's unlikely most text would contain a pipe, and ord('|') returns 124 for me, so that seems to fit your requirements.
For fast escaping I use stuff like this:
say you want to concatinate str1, str2 and str3
what I do is:
delimitedStr=str1.Replace("#","#a").Replace("|","#p")+"|"+str2.Replace("#","#a").Replace("|","#p")+"|"+str3.Replace("#","#a").Replace("|","#p");
then to retrieve original use:
splitStr=delimitedStr.Split("|".ToCharArray());
str1=splitStr[0].Replace("#p","|").Replace("#a","#");
str2=splitStr[1].Replace("#p","|").Replace("#a","#");
str3=splitStr[2].Replace("#p","|").Replace("#a","#");
note: the order of the replace is important
its unbreakable and easy to implement
Pipe for the win! |
We use ascii 0x7f which is pseudo-printable and hardly ever comes up in regular usage.
Well it's going to depend on the nature of your text to some extent but a vertical bar 0x7C doesn't crop up in text very often.
I don't think I've ever seen an ampersand followed by a comma in natural text, but you can check the file first to see if it contains the delimiter, and if so, use an alternative. If you want to always be able to know that the delimiter you use will not cause a conflict, then do a loop checking the file for the delimiter you want, and if it exists, then double the string until the file no longer has a match. It doesn't matter if there are similar strings because your program will only look for exact delimiter matches.
This can be good or bad (usually bad) depending on the situation and language, but keep mind mind that you can always Base64 encode the whole thing. You then don't have to worry about escaping and unescaping various patterns on each side, and you can simply seperate and split strings based on a character which isn't used in your Base64 charset.
I have had to resort to this solution when faced with putting XML documents into XML properties/nodes. Properties can't have CDATA blocks in them at all, and nodes escaped as CDATA obviously cannot have further CDATA blocks inside that without breaking the structure.
CSV is probably a better idea for most situations, though.
Both pipe and caret are the obvious choices. I would note that if users are expected to type the entire response, caret is easier to find on any keyboard than is pipe.
I've used double pipe and double caret before. The idea of a non printable char works if your not hand creating or modifying the file. For quick random access file storage and retrieval field width is used. You don't even have to read the file.. your literally pulling from the file by reference. This is how databases do some storage.. but they also manage the spaces between records and such. And it introduced the problem of max data element width. (Index attach a header which is used to define the width of each element and it's data type in the original old days.. later they introduced compression with remapping chars. This allows for a text file to get about 1/8 the size in transmission.. variable length char encoding for the win
make it dynamic : )
announce your control characters in the file header
for example
delimiter: ~
escape: \
wrapline: $
width: 19
hello world~this i$
s \\just\\ a sampl$
e text~$someVar$~h$
ere is some \~\~ma$
rkdown strikethrou$
gh\~\~ text
would give the strings
hello world
this is \just\ a sample text
$someVar$
here is some ~~markdown strikethrough~~ text
i have implemented something similar:
a plaintar text container format,
to escape and wrap utf16 text in ascii,
as an alternative to mime multipart messages.
see https://github.com/milahu/live-diff-html-editor