How to escape special characters in sphinxQL fulltext search? - full-text-search

in the sphinx changelog it says for 0.9.8:
"added query escaping support to query language, and EscapeString() API call"
can i assume, that there should be support for escaping special sphinx characters (#, !,
-, ...) for sphinxQL, too? if so, maybe someone could point me to an example on this. i'm
unable to find anything about it in the documentation or elsewhere on the net.
how do you do fulltext search (using spinxQL), if the search-phrase contains one of the special characters? i don't like the idea very much to "mask" them during indexing.
thanks!

The PHP version of the sphinxapi escape function did not work for me in tests. Also, it provides no protection against SQL-injection sorts of characters (e.g. single quote).
I needed this function:
function EscapeSphinxQL ( $string )
{
$from = array ( '\\', '(',')','|','-','!','#','~','"','&', '/', '^', '$', '=', "'", "\x00", "\n", "\r", "\x1a" );
$to = array ( '\\\\', '\\\(','\\\)','\\\|','\\\-','\\\!','\\\#','\\\~','\\\"', '\\\&', '\\\/', '\\\^', '\\\$', '\\\=', "\\'", "\\x00", "\\n", "\\r", "\\x1a" );
return str_replace ( $from, $to, $string );
}
Note the extra backslashes on the Sphinx-specific characters. I think what happens is that they put your whole query through an SQL parser, which removes escape backslashes 'extraneous' for SQL purposes (i.e. '\&' -> '&'). Then, it puts the MATCH clause through the fulltext parser, and suddenly '&' is a special character. So, you need the extra backslashes in the beginning.

There are corresponding functions EscapeString in each API ( php/python/java/ruby ) but to make escaping work with SphinxQL you have to write something similar in your application as SphinxQL hasn't such function.
The function itself is onliner
def EscapeString(self, string):
return re.sub(r"([=\(\)|\-!#~\"&/\\\^\$\=])", r"\\\1", string)
you could easy translate it to code of your application.

Related

How below REGEXP_REPLACE works?

I have query in my project and that is having REGEXP_REPLACE
i tried to find how it works by searching but i found it like
w+ Matches a word character (that is, an alphanumeric or underscore
(_) character).
but not able to find '"\w+\":' why these "" are used and what is mean by '{|}|"',''
UPDATE (SELECT data,data_value FROM TEMP) t
SET t.DATA_VALUE=REGEXP_REPLACE(REGEXP_REPLACE(t.data, '"\w+\":',''),'{|}|"','');
can you please tell me how it works?
This appear to be a regular expression for stripping keys and enclosing brackets from a JSON string - unfortunately, if this is the case then it does not work in all situations.
The regular expression
'"\w+\":'
will match:
A " double quotation mark;
\w+ one-or-more word (a-z or A-Z or 0-9 or _) characters;
\" another double quotation mark - note: the \ character is not necessary; then
A : colon.
So:
REGEXP_REPLACE(
'{"key":"value","key2":"value with \"quote"}',
'"\w+":', -- Pattern matched
'' -- Replacement string
)
Will output:
{"value","value with \"quote"}
The second pattern {|}|" will match either a {, or a } or a " character (and could have been equivalently written as [{}"]) so:
REGEXP_REPLACE(
'{"value","value with \"quote"}',
'{|}|"', -- Pattern matched
'' -- Replacement string
)
Will output:
value,value with \quote
Which is fine, until (like my example) you have an escaped double quote (or curly braces) in the value string; in which case those will also get stripped leaving the escape character.
(Note: you would not typically find this but it is possible to include escaped quotes in the key. So {"keywith\":quote":"value"} would get replaced to {quote":"value"} and then quote:value which is not the intended output.)
If parsing JSON is what you are trying to do (pre-Oracle 12) then you can use:
REGEXP_REPLACE(
'{"key":"value","key2":"value with \"quote","keywith\":quote":"value with \"{}"}',
'^{|"(\\"|[^"])+":(")?((\\"|[^"])+?)\2((,)|})',
'\3\6'
)
Which outputs:
value,value with \"quote,value with \"{}
Or in Oracle 12 you can do:
SELECT *
FROM JSON_TABLE(
'{"key":"value","key2":"value with \"quote","keywith\":quote":"value with \"{}"}',
'$.*' NULL ON ERROR
COLUMNS (
value VARCHAR2(4000) PATH '$'
)
)
Which outputs:
VALUE
-----------------
value
value with "quote
value with "{}
example:::REGEXP_REPLACE( string, pattern [, replacement_string [, start_position [, nth_appearance [, match_parameter ] ] ] ] )
| is or(CAN MEAN MORE THAN ONE ALTERNATIVE ) , is for at least as in {n,} at least n times
https://www.techonthenet.com/oracle/functions/regexp_replace.php
"where I got my info"
'"\w+\":' why these "" are used and what is mean by '{|}|"',''
Matches a word character(\w)One or more times(+) this has to be messed up it's missing the right quantity of close parentheses by putting \" w+ \"
they allow the " to be shown. This expression takes one expression changes it then uses that as the basis for the next change. Good luck figuring the rest out. Regular expressions aren't too bad, pretty intuitive once you get the basics down.

Replace special character with its index

I need to replace all special characters within a string with their index.
For example,
"I-need_to#change$all%special^characters^"
should become:
"I1need6to9change16all20special28characters39"
The index of all special character differs.
I have checked many links replacing all with single character, occurances of a character.
I found very similar link but it I do not want to adopt these replace its index number as I need to replace all of the special characters.
I have also tried to do something like this:
str.gsub!(/[^0-9A-Za-z]/, '')
Here str is my example string.
As this replaces all the characters but with space, and I want the index instead of space. Either all of the special character or these seven
\/*[]:?
I need to replace this seven mainly but it would be OK if we replace all of them.
I need a simpler way.
Thanks in advance.
You can use the global variable $` and the block form of gsub:
irb> str = "I-need_to#change$all%special^characters^"
=> "I-need_to#change$all%special^characters^"
irb> str.gsub(/[^0-9A-Za-z]/) { $`.length }
=> "I1need6to9change16all20special28characters39"

Treetop grammar line continuation

I'm trying to create a grammar for a language like the following
someVariable = This is a string, I know it doesn't have double quotes
anotherString = This string has a continuation _
this means I can write it on multiple line _
like this
anotherVariable = "This string is surrounded by quotes"
What are the correct Treetop grammar rules that parse the previous code correctly?
I should be able to extract the following values for the three variables
This is a string, I know it doesn't have double quotes
This string has a continuation this means I can write it on multiple line like this
This string is surrounded by quotes
Thank you
If you define the sequence "_\n" as if it was a single white-space character, and ensure that you test for that before you accept an end-of-line, your VB-style line continuation should just drop out. In VB, the newline "\n" is not white-space per se, but is a distinct statement termination character. You probably also need to deal with carriage returns, depending on your input character processing rules. I would write the white-space rule like this:
rule white
( [ \t] / "_\n" "\r"? )+
end
Then your statement rule looks like this:
rule variable_assignment
white* var:([[:alpha:]]+) white* "=" white* value:((white / !"\n" .)*) "\n"
end
and your top rule:
rule top
variable_assignment*
end
Your language doesn't seem to have any more apparent structure than that.

Ruby RegEx Commas Within Quotes [duplicate]

I would like to find a regex that will pick out all commas that fall outside quote sets.
For example:
'foo' => 'bar',
'foofoo' => 'bar,bar'
This would pick out the single comma on line 1, after 'bar',
I don't really care about single vs double quotes.
Has anyone got any thoughts? I feel like this should be possible with readaheads, but my regex fu is too weak.
This will match any string up to and including the first non-quoted ",". Is that what you are wanting?
/^([^"]|"[^"]*")*?(,)/
If you want all of them (and as a counter-example to the guy who said it wasn't possible) you could write:
/(,)(?=(?:[^"]|"[^"]*")*$)/
which will match all of them. Thus
'test, a "comma,", bob, ",sam,",here'.gsub(/(,)(?=(?:[^"]|"[^"]*")*$)/,';')
replaces all the commas not inside quotes with semicolons, and produces:
'test; a "comma,"; bob; ",sam,";here'
If you need it to work across line breaks just add the m (multiline) flag.
The below regexes would match all the comma's which are present outside the double quotes,
,(?=(?:[^"]*"[^"]*")*[^"]*$)
DEMO
OR(PCRE only)
"[^"]*"(*SKIP)(*F)|,
"[^"]*" matches all the double quoted block. That is, in this buz,"bar,foo" input, this regex would match "bar,foo" only. Now the following (*SKIP)(*F) makes the match to fail. Then it moves on to the pattern which was next to | symbol and tries to match characters from the remaining string. That is, in our output , next to pattern | will match only the comma which was just after to buz . Note that this won't match the comma which was present inside double quotes, because we already make the double quoted part to skip.
DEMO
The below regex would match all the comma's which are present inside the double quotes,
,(?!(?:[^"]*"[^"]*")*[^"]*$)
DEMO
While it's possible to hack it with a regex (and I enjoy abusing regexes as much as the next guy), you'll get in trouble sooner or later trying to handle substrings without a more advanced parser. Possible ways to get in trouble include mixed quotes, and escaped quotes.
This function will split a string on commas, but not those commas that are within a single- or double-quoted string. It can be easily extended with additional characters to use as quotes (though character pairs like « » would need a few more lines of code) and will even tell you if you forgot to close a quote in your data:
function splitNotStrings(str){
var parse=[], inString=false, escape=0, end=0
for(var i=0, c; c=str[i]; i++){ // looping over the characters in str
if(c==='\\'){ escape^=1; continue} // 1 when odd number of consecutive \
if(c===','){
if(!inString){
parse.push(str.slice(end, i))
end=i+1
}
}
else if(splitNotStrings.quotes.indexOf(c)>-1 && !escape){
if(c===inString) inString=false
else if(!inString) inString=c
}
escape=0
}
// now we finished parsing, strings should be closed
if(inString) throw SyntaxError('expected matching '+inString)
if(end<i) parse.push(str.slice(end, i))
return parse
}
splitNotStrings.quotes="'\"" // add other (symmetrical) quotes here
Try this regular expression:
(?:"(?:[^\\"]+|\\(?:\\\\)*[\\"])*"|'(?:[^\\']+|\\(?:\\\\)*[\\'])*')\s*=>\s*(?:"(?:[^\\"]+|\\(?:\\\\)*[\\"])*"|'(?:[^\\']+|\\(?:\\\\)*[\\'])*')\s*,
This does also allow strings like “'foo\'bar' => 'bar\\',”.
MarkusQ's answer worked great for me for about a year, until it didn't. I just got a stack overflow error on a line with about 120 commas and 3682 characters total. In Java, like this:
String[] cells = line.split("[\t,](?=(?:[^\"]|\"[^\"]*\")*$)", -1);
Here's my extremely inelegant replacement that doesn't stack overflow:
private String[] extractCellsFromLine(String line) {
List<String> cellList = new ArrayList<String>();
while (true) {
String[] firstCellAndRest;
if (line.startsWith("\"")) {
firstCellAndRest = line.split("([\t,])(?=(?:[^\"]|\"[^\"]*\")*$)", 2);
}
else {
firstCellAndRest = line.split("[\t,]", 2);
}
cellList.add(firstCellAndRest[0]);
if (firstCellAndRest.length == 1) {
break;
}
line = firstCellAndRest[1];
}
return cellList.toArray(new String[cellList.size()]);
}
#SocialCensus, The example you gave in the comment to MarkusQ, where you throw in ' alongside the ", doesn't work with the example MarkusQ gave right above that if we change sam to sam's: (test, a "comma,", bob, ",sam's,",here) has no match against (,)(?=(?:[^"']|["|'][^"']")$). In fact, the problem itself, "I don't really care about single vs double quotes", is ambiguous. You have to be clear what you mean by quoting either with " or with '. For example, is nesting allowed or not? If so, to how many levels? If only 1 nested level, what happens to a comma outside the inner nested quotation but inside the outer nesting quotation? You should also consider that single quotes happen by themselves as apostrophes (ie, like the counter-example I gave earlier with sam's). Finally, the regex you made doesn't really treat single quotes on par with double quotes since it assumes the last type of quotation mark is necessarily a double quote -- and replacing that last double quote with ['|"] also has a problem if the text doesn't come with correct quoting (or if apostrophes are used), though, I suppose we probably could assume all quotes are correctly delineated.
MarkusQ's regexp answers the question: find all commas that have an even number of double quotes after it (ie, are outside double quotes) and disregard all commas that have an odd number of double quotes after it (ie, are inside double quotes). This is generally the same solution as what you probably want, but let's look at a few anomalies. First, if someone leaves off a quotation mark at the end, then this regexp finds all the wrong commas rather than finding the desired ones or failing to match any. Of course, if a double quote is missing, all bets are off since it might not be clear if the missing one belongs at the end or instead belongs at the beginning; however, there is a case that is legitimate and where the regex could conceivably fail (this is the second "anomaly"). If you adjust the regexp to go across text lines, then you should be aware that quoting multiple consecutive paragraphs requires that you place a single double quote at the beginning of each paragraph and leave out the quote at the end of each paragraph except for at the end of the very last paragraph. This means that over the space of those paragraphs, the regex will fail in some places and succeed in others.
Examples and brief discussions of paragraph quoting and of nested quoting can be found here http://en.wikipedia.org/wiki/Quotation_mark .

How to correctly replace ereg with preg

I have a list of Mobile devices that I'm using to display content correctly. The depreciated function looks like this:
function detectPDA($query){
$browserAgent = $_SERVER['HTTP_USER_AGENT'];
$userAgents = $this->getBrowserAgentsToDetect(); // comma separated list of devices
foreach ( $userAgents as $userAgent ) {
if(eregi($userAgent,$browserAgent)){
if(eregi("iphone",$browserAgent) || eregi("ipod",$browserAgent) ){
$this->iphone = true;
}else{
$this->pda = true;
}
}
}
}
What is the correct way to replace the eregi functions?
If all the pattern strings ($userAgent and iphone) can be trusted not to contain special regex chars (()[]!|.^${}?*+), then you just surround the eregi regex with slashes (/) and add an i after the last slash (which means "case insensitive").
So:
eregi($userAgent,$browserAgent) --> preg_match("/$userAgent/i",$browserAgent)
eregi("iphone",$browserAgent) --> preg_match('/iphone/i',$browserAgent)
However, are you just trying to match $userAgent as-is within $browserAgent? For example, if a particular $userAgent was foo.bar, would you want the . to match a literal period, or would you want to interpret it in its regex sense ("match any character")?
If the former, I'd suggest you forgo regex entirely and use stripos($haystack,$needle), which searches for the string $needle in $haystack (case-insensitive). Then you don't need to worry about (say) an asterisk in $userAgent being interpreted in the regex sense instead of the literal sense.
If you do use stripos don't forget it can return a 0 which would evaluate to false, so you need to use === false or !== false (see the documentation I linked).

Resources