Ruby / Mechanize aborts when it finds an accentuated u letter - ruby

Question

I want to read this form of a php script using Ruby/Mechanize:
<form name="editevent" method="post" action="/index.php" enctype="multipart/form-data">
<input type="text" name="veranstaltung">
<select name='ortid'>
<option value='2'>Kaminwerk</option>
<option value='3'>Pitú</option>
<option value='4'>Apollo-Center</option>
</select>
<input type="text" name="neutermin" id="neutid" />
<textarea name="beschreibung" cols="40" rows="7"></textarea><br />
<input type="submit" name="button" value="Absenden">
</form>
In Ruby I have got:
form = page.forms.first
form.fields.each { |f| puts f.name }
However Ruby can find only the form elements with the name "veranstaltung" and "ortid"
I found out that the problem ist the "u" letter with the accent on it in the word "Pitú". Proofs: when I print the inner_html of the html code the part of the form looks like this:
<form name="editevent" method="post" action="/index.php" enctype="multipart/form-data">
<input type="text" name="veranstaltung">
<select name='ortid'>
<option value='2'>Kaminwerk</option>
<option value='3'>Pit</form>
The other part of the form has vanished! How can I use that form completely despite of the "ú"?
I would be very glad if anyone could help.

Answer 1

What version of Ruby? It smells like 1.8.7, which is not Unicode savvy. If you can, upgrade to 1.9.2.
It's also important to specify the code-set of the language when parsing the content. Often times that information is in the DOCTYPE statement, but if it isn't you have to give the language a hint of what to expect.
Because those characters are embedded in PHP, they could be UTF-8, or maybe a variant of WIN-1252 or ISO-8951 which implies they'd be a single byte character. Mechanize uses Nokogiri to parse, and it will want to know what the language is to give you the best decoding of the values. Nokogiri will put errors in the errors attribute when it can't parse something to its liking, so you might want to check there.
SO, if I were you, I'd look to see what the DOCTYPE is when the content is sent, and also check the HTTP headers, and see if something will define the codeset.
This is a problem I've encountered many times on the internet because HTML is so poorly written and so often fails to follow the specs.