Can't upload images to Redmine anymore - ruby

Question

For some strange reason I can't upload images to tickets in Redmine anymore. I can upload a txt file or zip files. When I upload an image in the ticket it either says "Service Unavailable" or "Unprocessable". Weird thing is that it used to work. We updated to the latest Redmine (2.6.0.stable)
I looked at the production.log and this is the error (Can't verify CSRF token authenticity):
Started POST "/uploads.js?attachment_id=1&filename=test.png" for xx.xx.xxx.xxx at 2014-12-03 12:58:49 -0500
Processing by AttachmentsController#upload as JS
Parameters: {"attachment_id"=>"1", "filename"=>"test.png"}
WARNING: Can't verify CSRF token authenticity
Filter chain halted as :verify_authenticity_token rendered or redirected
Completed 422 Unprocessable Entity in 2.6ms (ActiveRecord: 0.3ms)
Here is my Redmine Information:
Default administrator account changed True
Attachments directory writable True
Plugin assets directory writable True
RMagick available (optional) Exclamation
ImageMagick convert available (optional) True
Environment:
Redmine version 2.6.0.stable
Ruby version 1.9.3-p547 (2014-05-14) [x86_64-linux]
Rails version 3.2.19
Environment production
Database adapter Mysql2
SCM:
Git 1.8.2.1
Filesystem
Redmine plugins:
redmine_agile 1.3.2
redmine_ckeditor 1.0.16
redmine_github_hook 2.1.0
redmine_my_page_queries 2.1.6
redmine_theme_changer 0.1.0

Answer 1

It turns out that this was a Varnish Issue. We got around this problem by adding this Varnish rule:
if (req.http.host ~ "my\.domain\.com$") {
return (pipe);
}
Here are some debugging things we did to try to figure out the problem.
Temporarily added config.action_controller.allow_forgery_protection = false to application.rb. When we tried to upload an image it I get a Popup: login required for Server on Redmine API. This gave me a clue that it must have been some kind of server issue.
Created additional_environment.rb and enabled config.log_level = :debug. This added more debug info to the log file.
Started POST "/uploads.js?attachment_id=1&filename=Screen%20Shot%202014-12-11%20at%2010.01.49%20AM.png" for xx.xx.xxx.xxx at 2014-12-11 11:07:41 -0500
Processing by AttachmentsController#upload as JS
Parameters: {"attachment_id"=>"1", "filename"=>"Screen Shot 2014-12-11 at 10.01.49 AM.png"}
^[[1m^[[35m (0.3ms)^[[0m SELECT MAX(`settings`.`updated_on`) AS max_id FROM `settings`
^[[1m^[[36mSetting Load (0.3ms)^[[0m ^[[1mSELECT `settings`.* FROM `settings` WHERE `settings`.`name` = 'rest_api_enabled' LIMIT 1^[[0m
^[[1m^[[35mAnonymousUser Load (0.3ms)^[[0m SELECT `users`.* FROM `users` WHERE `users`.`type` IN ('AnonymousUser') LIMIT 1
Current user: anonymous
^[[1m^[[36mSetting Load (0.3ms)^[[0m ^[[1mSELECT `settings`.* FROM `settings` WHERE `settings`.`name` = 'login_required' LIMIT 1^[[0m
^[[1m^[[35mSetting Load (0.2ms)^[[0m SELECT `settings`.* FROM `settings` WHERE `settings`.`name` = 'force_default_language_for_anonymous' LIMIT 1
^[[1m^[[36mSQL (1.2ms)^[[0m ^[[1mSELECT `members`.`id` AS t0_r0, `members`.`user_id` AS t0_r1, `members`.`project_id` AS t0_r2, `members`.`created_on` AS t0_r3, `members`.`mail_notification` AS t0_r4, `projects`.`id` AS t1_r0, `projects`.`name` AS t1_r1, `projects`.`description` AS t1_r2, `projects`.`homepage` AS t1_r3, `projects`.`is_public` AS t1_r4, `projects`.`parent_id` AS t1_r5, `projects`.`created_on` AS t1_r6, `projects`.`updated_on` AS t1_r7, `projects`.`identifier` AS t1_r8, `projects`.`status` AS t1_r9, `projects`.`lft` AS t1_r10, `projects`.`rgt` AS t1_r11, `projects`.`inherit_members` AS t1_r12, `roles`.`id` AS t2_r0, `roles`.`name` AS t2_r1, `roles`.`position` AS t2_r2, `roles`.`assignable` AS t2_r3, `roles`.`builtin` AS t2_r4, `roles`.`permissions` AS t2_r5, `roles`.`issues_visibility` AS t2_r6 FROM `members` LEFT OUTER JOIN `projects` ON `projects`.`id` = `members`.`project_id` LEFT OUTER JOIN `member_roles` ON `member_roles`.`member_id` = `members`.`id` LEFT OUTER JOIN `roles` ON `roles`.`id` = `member_roles`.`role_id` WHERE `members`.`user_id` = 2 AND (projects.status<>9) ORDER BY projects.name^[[0m
^[[1m^[[35mRole Load (0.2ms)^[[0m SELECT `roles`.* FROM `roles` WHERE `roles`.`builtin` = 2 LIMIT 1
Filter chain halted as :authorize_global rendered or redirected
Completed 401 Unauthorized in 54.3ms (ActiveRecord: 2.7ms)
Current user: anonymous in the log kind of helped lead to the fix.