Problems with character encoding in Resque - ruby

Question

I have a series of JSON messages that I need to process and I've run into some trouble with character encodings. I retrieve this message from a RabbitMQ queue like so:
#data = []
#queue.pop do |metadata, payload|
parsed_message = JSON.parse(payload)
#data << parsed_message
end
I then process them using Resque and enqueue them like so:
Resque.enqueue(Worker, #data)
When I run the worker it throws this error:
** [16:06:13 2012-09-14] 5375: Error reserving job: #<Encoding::InvalidByteSequenceError: "\xC3" on US-ASCII>
** [16:06:13 2012-09-14] 5375: /usr/local/lib/ruby/1.9.1/json/common.rb:148:in `encode'
/usr/local/lib/ruby/1.9.1/json/common.rb:148:in `initialize'
/usr/local/lib/ruby/1.9.1/json/common.rb:148:in `new'
/usr/local/lib/ruby/1.9.1/json/common.rb:148:in `parse'
/usr/local/lib/ruby/gems/1.9.1/gems/multi_json-1.3.6/lib/multi_json/adapters/json_common.rb:7:in `load'
/usr/local/lib/ruby/gems/1.9.1/gems/multi_json-1.3.6/lib/multi_json.rb:93:in `load'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/helpers.rb:38:in `decode'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque.rb:149:in `pop'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/job.rb:100:in `reserve'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque.rb:303:in `reserve'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/worker.rb:209:in `block in reserve'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/worker.rb:207:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/worker.rb:207:in `reserve'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/worker.rb:136:in `block in work'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/worker.rb:133:in `loop'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/worker.rb:133:in `work'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/tasks.rb:36:in `block (2 levels) in <top (required)>'
/usr/local/lib/ruby/1.9.1/rake/task.rb:205:in `call'
/usr/local/lib/ruby/1.9.1/rake/task.rb:205:in `block in execute'
/usr/local/lib/ruby/1.9.1/rake/task.rb:200:in `each'
/usr/local/lib/ruby/1.9.1/rake/task.rb:200:in `execute'
/usr/local/lib/ruby/1.9.1/rake/task.rb:158:in `block in invoke_with_call_chain'
/usr/local/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/usr/local/lib/ruby/1.9.1/rake/task.rb:151:in `invoke_with_call_chain'
/usr/local/lib/ruby/1.9.1/rake/task.rb:144:in `invoke'
/usr/local/lib/ruby/1.9.1/rake/application.rb:116:in `invoke_task'
/usr/local/lib/ruby/1.9.1/rake/application.rb:94:in `block (2 levels) in top_level'
/usr/local/lib/ruby/1.9.1/rake/application.rb:94:in `each'
/usr/local/lib/ruby/1.9.1/rake/application.rb:94:in `block in top_level'
/usr/local/lib/ruby/1.9.1/rake/application.rb:133:in `standard_exception_handling'
/usr/local/lib/ruby/1.9.1/rake/application.rb:88:in `top_level'
/usr/local/lib/ruby/1.9.1/rake/application.rb:66:in `block in run'
/usr/local/lib/ruby/1.9.1/rake/application.rb:133:in `standard_exception_handling'
/usr/local/lib/ruby/1.9.1/rake/application.rb:63:in `run'
/usr/local/bin/rake:32:in `<main>'
rake aborted!
I've wrapped all the code in the perform function from the worker in a begin/rescue block, to try to catch the exception but to no avail which leads me to think that it's from inside Resque somewhere.
I've tried enqueueing the Resque worker with the raw data and then parse it inside the worker into JSON and setting an UTF-8 encoding:
raw_data.each do |msg|
msg = msg.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "")
parsed_msg = JSON.parse(msg) rescue nil
next if parsed_msg.nil?
parsed_data << parsed_msg
end
and I still get the same error.
It's not always the same character, sometimes it's \xC2 or \xD8 if I remember correctly
Any ideas?

Answer 1

Is it possible the file encoding is incorrect?
Try inserting:
# encoding: UTF-8
at the top of the relevant ruby source files. This is one of the most obnoxious things in Ruby 1.9.