Freemarker ".vars" names can't contain dashes? - freemarker

We're using Freemarker version 2.3.16, and I've just tracked down a weird bug in one of our apps. It came down to there now being hyphens in some of our product code strings. The codes are used to pull hashes of localized text from the global scope using .vars.
Reducing the issue brought me to an example that anyone can try:
${.vars["foo-bar"]} in a template outputs 0
${.vars["foo+bar"]} outputs nullnull
${.vars["foobar"]} correctly triggers an InvalidReferenceException
All three should trigger exceptions. Instead, it appears the .vars parameter string is being evaluated! :-(
http://freemarker.sourceforge.net/docs/app_faq.html#faq_strange_variable_name implies this should work.
I saw mention of a similar issue a few weeks ago on the Freemarker mailing list, and it was suggested to prefix the parameter string with "#". That might work with other hashes, but it does NOT work with .vars. I just took a working example (.vars["resources_title"]) and changing it made it throw an InvalidReferenceException (.vars["#resources_title"]). I also tried it on the hyphenated reference, and it also threw the exception.
Upgrading to 2.3.18 did not seem to make a difference.

Sorry for the delay. After some good mailing-list help on places to put breakpoints, here's I wrote back to the list on June 10th:
Short story: It's not a Freemarker issue. Rather the Struts team chose to hard-wire Freemarker to treat .vars names as OGNL expressions, and there seems no way to tell OGNL to not parse them. So under Struts, "-" and "+" (and possibly other characters) cannot appear in .vars names.
Long story...
freemarker.core.BuiltinVariable (line 192) is where Freemarker starts to process .vars expressions
freemarker.core.Environment (line 1088) hands control over to the "rootDataModel" which the Struts team hard-wired to be an instance of org.apache.struts2.views.freemarker.ScopesHashModel
line 70 of that class (using version 2.1.8.1 of Struts) calls "stack.findValue"; "stack" has been wired to be an instance of com.opensymphony.xwork2.ognl.OgnlValueStack
at line 236 this class in turn asks an instance of OgnlUtil to find the object, and that's where the name is assumed to be an OGNL expression and is parsed, turning "foo-bar" into ( foo - bar )
At no point along the way does there seem to be a choice to NOT treat the .vars name as an expression (a comment in FreemarkerResult hints at the possibility, but the code doesn't follow through). In theory I could have my implementation of FreemarkerManager create a variant of ScopesHashModel, but that would take a lot of work to change all the associated classes with it.
(Nor does there seem to be a way to escape "-" characters in OGNL expressions. Seems there was discussion 5-6 years ago to do this, but.... .vars( "foo\\-bar" ) fails on finding "-" after "\", so presumably "-" isn't escapable?)
:-(
I'm not clear what the use-case is for treating .vars names as expressions... but I don't think Struts is going to change, now. Rather than override a half dozen Struts classes, I instead changed the code that loads our ResourceBundles into the value stack: it now changes the names to replace "-" and "_", and likewise my .vars names are changed the same way in the template and... tada. It works. Woo.

Works for me. And like already mentioned on the freemarker-user mailing list: maybe you use a strange data model, or even a fancy ObjectWrapper. But a discussion like this is probably better suited for the freemarker-user mailing list...

It works if it added with escape foo\-bar.
"Only single backslash"

Since freemarker version 2.3.22 is it possible to use dot (.), minus sign (-) or colon (:) in a variable name (details here).
In my case, it fails if I tried to use with freemarker 2.3.21 variables like :
api["x-link"]
If I change freemarker to version 2.3.22 it works.

Related

Object name becoming lowercase

I have some code which has worked in multiple installations for about a year. Today im doing a small change to a control and then another control seems to have developed an issue. When at runtime im getting a 91 error object variable or with block variable not set.
I therefore looked at the problem line which is: -
If Screen.ActiveForm.name = "frmFoutmelding" Then Exit Sub
so I noticed the name was lowercase. if i delete .name and rehit the "dot" then it shows me i can use .Name but as soon as i move from this line it drops back to .name
I've checked for instances of name and it appears everywhere in the code in different modules but i cant find if i have accidentally defined this lowercase name anywhere?
Googling doesn't seem to show much but i feel Im googling the wrong terms
chaps - thanks for your suggestions - this was the first instance of the lowercase name and searching as Jim suggested didn't reveal anything I'm afraid. What I did discover was that this was suddenly being run before any forms had actually been displayed and so the count was 0. I therefore, did an on error to check the form count and exit the sub if it =0 then if not to carry on with the line I thought I was having issues with.
It's likely that you did create a new variable or property called (lower case) name, or that some included reference did the same. It's possible to use reserved words as variable names in some cases, but it requires taking specific steps.
I would first search your code for instances of name As to see if you created a variable (this assumes you use Option Explicit, which is a must IMO). Then search for Property*name with * as a wildcard.
If those fail you could try unchecking references or components to see if any of them define name. If none of that finds anything, please post back here.
Jim Mack covers a lot of the potential issues. I think another is if you typed a lower case '.name' in association with Activeform at some point earlier in the same code module - the VB6 IDE checks in the current module and uses that to define what case to use. Look further up the same code module (sub or function).
Ultimately, check what changes you made by comparing the old source to the new in a file comparison tool like windiff - you do have backups, right?

Sitecore - rewrite " " to "-" in urls but still allow dashes as legal item names

I've read a half dozen guides on rewriting spaces to something more friendly in Sitecore, but all of them rely on Sitecore's <encodeNameReplacements/> element which also reverses the replacement requiring "-" to be an illegal character for names.
The problem with this is that the url of our application has a "-" in the hostname. Sitecore rewrites this resulting in a 404.
Does anyone have ideas on how to do this url rewriting in Sitecore without relying on <encodeNameReplacements and still allowing "-" as a legal item name character? Our current best idea is to use something slighty more complex than a plain "-" such as "--" or "_". This isn't a very good idea, so I'd appreciate any insight you guys have on the matter.
EDIT: We are running a multi-site setup with Sitecore 6.5
So, if I am assuming correctly, you want to replace spaces in item names with some other SEO-friendlier character. Whatever replacement you configure, you would need to apply the transformation on both sides of the equation (pun intended). So '--' or '_' will have to become illegal item name characters.
I generally think it's a bad idea to do this and would rather have my content editors determine the exact urls to their content.
But if you absolutely need to implement this as a rule, one of the solutions out there is to implement a custom handler to change your real item name while leaving the item display name in its original form.
Hope this helps.
Which version and build of Sitecore are you using?
I've just taken a look through Sitecore.Links.LinkProvider in Sitecore.Kernel and the BuildItemUrl method only replaces characters encodeNameReplacements on the path part of the URL so it should leave your hostname alone.
I would expect the same to happen to reverse, and looking at Sitecore.Pipelines.HttpRequest.SiteResolver the SiteContext is resolved by matching the requested Uri to the defined sites. Further down the process Sitecore.Pipelines.HttpRequest.ItemResolver decodes the Item Url but the Site has already been resolved at this stage.
I presume you have set the hostName attribute on the <site> elements? Having you tried setting the targetHostName attribute as well? Have you tried setting multiple hostNames, I don't expect it to work with spaces but worth a try if what you are saying is true:
<site name="website" hostName="my-site.com|my site.com" ... />
I tried replicating your issue on my local machine but it worked as expected for me... Unless you are working with very strange character sets then this is still the best way of encoding names in my opinion.

NSXMLParser with HTML Containing Javascript and "bad" characters

I am using NSXMLParser to parse HTML from web sites. Testing site is under my control but in operation sites will not be.
Problem is when parser encounters javascript which contains "bad" characters. For example, javascript containing if(screen.width<=521). The problem is the < in the code. I can see the problem but am unsure if there is any good way round it. (the NSXMLParser is reporting NSXMLParserErrorDomain error 68. and I can see why - it is treating the <= as the start of a new tag but = is not a valid tag name character...). But then what would I do with e.g. if(var<20) ?
I actually not interested in the specific content so could do things like global replace/removal of e.g. "<=" and ">=" (etc.) but in some regards that seems a bit of a mess as I was using NSXMLParser to avoid having to start messing around with the content. If substitution is the best way forward, I can envisage "<=" and ">=" but any other sequences I should include ?
I am new to Cocoa so may easily have missed something obvious - in which case many apologies. I did see that others have found similar problems but could not get a good way forward from the questions.
I am handling the error OK (in a tidy manner) but it is preventing my app from doing what it is meant to do - i.e. I need to avoid the error rather than handle it.
Background: that application is doing a "before" and "after" comparison on the html and looking for changes. I could swap "<=" for something really weird, then swap it back when necessary. I could even check the data for the replace content first to eliminate possible ambiguities (e.g. find a UID sequence not in the downloaded page, replace "<=" with UID sequence, parse page, if need be, replace UID with "<=", ditto for ">=".
(I have looked at e.g. libtidy of libxml2 but cannot find easy documentation and am wary about launching down such a route if it will not solve the issues.)
NSXMLParser, as its name implies, is not meant for parsing HTML. XML is much stricter than HTML, and the errors you've encountered are certainly not the only ones that are possible with real-world HTML. There are HTML documents that are also valid XML, but that is the exception, rather than the norm.
I would suggest using a proper HTML parser instead, such as this one, which is an Objective-C wrapper around libxml's HTML parsing functions.

How to remove special charecters in wordpress?

I am using Topsy, It returns me title of highest ranking article of my mebsite, It returns me one RSS file which contains post title with there link. For now i am only taking post name and using post title am trying to search in mysql database using following function like this:
get_post_by_title($postTitle,'post');
But the problem is topsy returns me post title but it also add some special characters in RSS file like " ' " replace with " ’ " this charecters.Because of this get_post_by_title() function does not return me post by title name.
EDIT : It returns me one post title like this :
iPad Applications In Bloom’s Taxonomy NEXT
Here single quote is special charecter.
Please help me. Thanks
First let's clear up a misconception: that character in your example is not a "special" character. It is Unicode code point U+2019, "RIGHT SINGLE QUOTATION MARK." Its HTML entity reference is ’. It's an ordinary character - it just happens to be an ordinary character that has no representation in ASCII. Before getting to an answer to your specific question, I need to tell you to read Joel Spolsky's article "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)" - it is just what it says on the tin, and unless you absorb at least a little more knowledge about Unicode, you will keep running into problems like this. Don't fret too much: everyone runs into problems like this until they learn how to deal with text. Unicode isn't "hard" so much as it is "prone to exposing unconscious assumptions we make about how text works." †
Now, to your question.
If I'm reading you right, what's happening to you is that you have posts with non-ASCII characters in their titles such as ’ which aren't showing up when you search for them with get_post_by_title() (it seems like you're using something similar to the accepted answer on this question - is that right?) There are two paths to a solution: store the titles in a format that's easier for you to search, or use a searching method that can find non-ASCII characters.
Storing the titles differently would require that you run them through PHP's built-in htmlentities() function or before storing them in your Wordpress DB - you would also want to make sure that you convert characters with no HTML entity equivalent to '\xNN' form, and to make sure that your DB's collation/charset is set to UTF-8 or another Unicode-aware encoding. This will be a nontrivial amount of effort. ‡
Using a different searching method doesn't require tinkering with your DB or digging into WordPress internals, but it does require very careful fiddling with search string. You'll need to either use the exact character you're looking for in a search, expressed as a '\xNN' character reference if necessary, or use wildcards carefully in the search.
Either way, good luck. It may be possible to offer more specific advice if more of your code is visible.
†: By the way, your life with regards to Unicode will also get much, much easier if you use better languages than PHP and better databases than MySQL. WordPress is inextricably tied to PHP and MySQL: PHP & MySQL are both woefully, horrendous, hilariously bad at handling Unicode issues correctly. Your life as a programmer will get better if you extirpate PHP & MySQL from it.
‡: Seriously, PHP is atrociously bad at this, and MySQL is in a shoelaces-tied-together state of fumbling. Avoid them.
remove from wp-config.php
//define('DB_CHARSET', 'utf8');
//define('DB_COLLATE','utf8_unicode_ci');
You can easily remove special characters using preg_replace, see this post -> http://code-tricks.com/filter-non-ascii-characters-using-php/

Do I really need to encode '&' as '&'?

I'm using an '&' symbol with HTML5 and UTF-8 in my site's <title>. Google shows the ampersand fine on its SERPs, as do all the browsers in their titles.
http://validator.w3.org is giving me this:
& did not start a character reference. (& probably should have been escaped as &.)
Do I really need to do &?
I'm not fussed about my pages validating for the sake of validating, but I'm curious to hear people's opinions on this and if it's important and why.
Yes. Just as the error said, in HTML, attributes are #PCDATA meaning they're parsed. This means you can use character entities in the attributes. Using & by itself is wrong and if not for lenient browsers and the fact that this is HTML not XHTML, would break the parsing. Just escape it as & and everything would be fine.
HTML5 allows you to leave it unescaped, but only when the data that follows does not look like a valid character reference. However, it's better just to escape all instances of this symbol than worry about which ones should be and which ones don't need to be.
Keep this point in mind; if you're not escaping & to &, it's bad enough for data that you create (where the code could very well be invalid), you might also not be escaping tag delimiters, which is a huge problem for user-submitted data, which could very well lead to HTML and script injection, cookie stealing and other exploits.
Please just escape your code. It will save you a lot of trouble in the future.
Validation aside, the fact remains that encoding certain characters is important to an HTML document so that it can render properly and safely as a web page.
Encoding & as & under all circumstances, for me, is an easier rule to live by, reducing the likelihood of errors and failures.
Compare the following: which is easier? Which is easier to bugger up?
Methodology 1
Write some content which includes ampersand characters.
Encode them all.
Methodology 2
(with a grain of salt, please ;) )
Write some content which includes ampersand characters.
On a case-by-case basis, look at each ampersand. Determine if:
It is isolated, and as such unambiguously an ampersand. eg. volt & amp > In that case don't bother encoding it.
It is not isolated, but you feel it is nonetheless unambiguous, as the resulting entity does not exist and will never exist since the entity list could never evolve. E.g., amp&volt >. In that case, don't bother encoding it.
It is not isolated, and ambiguous. E.g., volt&amp > Encode it.
??
HTML5 rules are different from HTML4. It's not required in HTML5 - unless the ampersand looks like it starts a parameter name. "&copy=2" is still a problem, for example, since © is the copyright symbol.
However it seems to me that it's harder work to decide to encode or not to encode depending on the following text. So the easiest path is probably to encode all the time.
I think this has turned into more of a question of "why follow the spec when browser's don't care." Here is my generalized answer:
Standards are not a "present" thing. They are a "future" thing. If we, as developers, follow web standards, then browser vendors are more likely to correctly implement those standards, and we move closer to a completely interoperable web, where CSS hacks, feature detection, and browser detection are not necessary. Where we don't have to figure out why our layouts break in a particular browser, or how to work around that.
Specifically, if HTML5 does not require using & in your specific situation, and you're using an HTML5 doctype (and also expecting your users to be using HTML5-compliant browsers), then there is no reason to do it.
Well, if it comes from user input then absolutely yes, for obvious reasons. Think if this very website didn't do it: the title of this question would show up as Do I really need to encode ‘&’ as ‘&’?
If it's just something like echo '<title>Dolce & Gabbana</title>'; then strictly speaking you don't have to. It would be better, but if you don't, no user will notice the difference.
Could you show us what your title actually is? When I submit
<!DOCTYPE html>
<html>
<title>Dolce & Gabbana</title>
<body>
<p>Am I allowed loose & mpersands?</p>
</body>
</html>
to http://validator.w3.org/ - explicitly asking it to use the experimental HTML 5 mode - it has no complaints about the &s...
In HTML, a & marks the begin of a reference, either of a character reference or of an entity reference. From that point on, the parser expects either a # denoting a character reference, or an entity name denoting an entity reference, both followed by a ;. That’s the normal behavior.
But if the reference name or just the reference opening & is followed by a white space or other delimiters like ", ', <, >, &, the ending ; and even a reference to represent a plain, & can be omitted:
<p title="&">foo & bar</p>
<p title="&amp">foo &amp bar</p>
<p title="&">foo & bar</p>
Only in these cases can the ending ; or even the reference itself be omitted (at least in HTML 4). I think HTML 5 requires the ending ;.
But the specification recommends to always use a reference like the character reference & or the entity reference & to avoid confusion:
Authors should use "&" (ASCII decimal 38) instead of "&" to avoid confusion with the beginning of a character reference (entity reference open delimiter). Authors should also use "&" in attribute values since character references are allowed within CDATA attribute values.
Update (March 2020): The W3C validator no longer complains about escaping URLs.
I was checking why image URLs need escaping and hence tried it in https://validator.w3.org. The explanation is pretty nice. It highlights that even URLs need to be escaped. [PS: I guess it will be unescaped when it's consumed since URLs need &. Can anyone clarify?]
<img alt="" src="foo?bar=qut&qux=fop" />
An entity reference was found in the document, but there is no
reference by that name defined. Often this is caused by misspelling
the reference name, unencoded ampersands, or by leaving off the
trailing semicolon (;). The most common cause of this error is
unencoded ampersands in URLs as described by the WDG in "Ampersands in
URLs". Entity references start with an ampersand (&) and end with a
semicolon (;). If you want to use a literal ampersand in your document
you must encode it as "&" (even inside URLs!). Be careful to end
entity references with a semicolon or your entity reference may get
interpreted in connection with the following text. Also keep in mind
that named entity references are case-sensitive; &Aelig; and æ
are different characters. If this error appears in some markup
generated by PHP's session handling code, this article has
explanations and solutions to your problem.
It depends on the likelihood of a semicolon ending up near your &, causing it to display something quite different.
For example, when dealing with input from users (say, if you include the user-provided subject of a forum post in your title tags), you never know where they might be putting random semicolons, and it might randomly display strange entities. So always escape in that situation.
For your own static HTML content, sure, you could skip it, but it's so trivial to include proper escaping, that there's no good reason to avoid it.
If the user passes it to you, or it will wind up in a URL, you need to escape it.
If it appears in static text on a page? All browsers will get this one right either way, and you don't worry much about it, since it will work.
Yes, you should try to serve valid code if possible.
Most browsers will silently correct this error, but there is a problem with relying on the error handling in the browsers. There is no standard for how to handle incorrect code, so it's up to each browser vendor to try to figure out what to do with each error, and the results may vary.
Some examples where browsers are likely to react differently is if you put elements inside a table but outside the table cells, or if you nest links inside each other.
For your specific example it's not likely to cause any problems, but error correction in the browser might for example cause the browser to change from standards compliant mode into quirks mode, which could make your layout break down completely.
So, you should correct errors like this in the code, if not for anything else so to keep the error list in the validator short, so that you can spot more serious problems.
A couple of years ago, we got a report that one of our web apps wasn't displaying correctly in Firefox. It turned out that the page contained a tag that looked like
<div style="..." ... style="...">
When faced with a repeated style attribute, Internet Explorer combines both of the styles, while Firefox only uses one of them, hence the different behavior. I changed the tag to
<div style="...; ..." ...>
and sure enough, it fixed the problem! The moral of the story is that browsers have more consistent handling of valid HTML than of invalid HTML. So, fix your damn markup already! (Or use HTML Tidy to fix it.)
If & is used in HTML then you should escape it.
If & is used in JavaScript strings, e.g., an alert('This & that'); or document.href, you don't need to use it.
If you're using document.write then you should use it, e.g. document.write(<p>this & that</p>).
If you're really talking about the static text
<title>Foo & Bar</title>
stored in some file on the hard disk and served directly by a server, then yes: it probably doesn't need to be escaped.
However, since there is very little HTML content nowadays that's completely static, I'll add the following disclaimer that assumes that the HTML content is generated from some other source (database content, user input, web service call result, legacy API result, ...):
If you don't escape a simple &, then chances are you also don't escape a & or a or <b> or <script src="http://attacker.com/evil.js"> or any other invalid text. That would mean that you are at best displaying your content wrongly and more likely are suspectible to XSS attacks.
In other words: when you're already checking and escaping the other more problematic cases, then there's almost no reason to leave the not-totally-broken-but-still-somewhat-fishy standalone-& unescaped.
The link has a fairly good example of when and why you may need to escape & to &
https://jsfiddle.net/vh2h7usk/1/
Interestingly, I had to escape the character in order to represent it properly in my answer here. If I were to use the built-in code sample option (from the answer panel), I can just type in & and it appears as it should. But if I were to manually use the <code></code> element, then I have to escape in order to represent it correctly :)

Resources