Google Chrome not running when Chromedriver tries to access it - EC2 instance - ruby

Question

I've looked at so many suggested solutions to this issue, and yet none apply or have worked for me. I'm trying to test updating chromedriver and google-chrome on our staging server and having no luck.
We were using an older version of selenium-webdriver, so I updated as far as we could go (we are on Ruby 2.3.6), so our versions stand at:
selenium-webdriver: 3.142.7
capybara: 2.7.1
Operating System: Ubuntu 18.04.4 LTS
Kernel: Linux 5.4.0-1093-aws
Architecture: x86-64
$ which chromedriver
/usr/bin/chromedriver
$ which google-chrome
/usr/bin/google-chrome
$ chromedriver --version
ChromeDriver 109.0.5414.74 (e7c5703604daa9cc128ccf5a5d3e993513758913-refs/branch-heads/5414#{#1172})
$ google-chrome --version
Google Chrome 109.0.5414.74
I've been running it somewhat like this. I added the Service instance after moving to selenium webdriver 3.142.7
def set_defaults
options = set_options
capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(chromeOptions: options)
service = Selenium::WebDriver::Service.chrome(path: '/usr/bin/chromedriver')
selenium_options = { browser: :chrome, desired_capabilities: capabilities, service: service }
Capybara.register_driver :chrome do |app|
Capybara::Selenium::Driver.new(app, selenium_options)
end
Capybara.javascript_driver = :chrome
end
def set_options
user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:36.0) Gecko/20100101 Firefox/36.0'
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument('--no-sandbox')
options.add_argument('--headless')
options.add_argument("--user-agent='#{user_agent}'")
options.add_argument('--disable-popup-blocking')
options.add_argument('--disable-gpu')
options.add_argument('--window-size=1280,1024')
options.add_argument('--disable-dev-shm-usage') # I added this after reading other answers on SO, but it didn't make a difference
options
end
The vast majority of this was set up previously and working until we needed to update our chromedriver/google-chrome versions, which is what I'm testing.
Here is the error I'm seeing when trying to run the above code:
Selenium::WebDriver::Error::UnknownError: unknown error: Chrome failed to start: exited abnormally.
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
from #0 0x55b693029303 <unknown>
from #1 0x55b692dfdd37 <unknown>
from #2 0x55b692e26157 <unknown>
from #3 0x55b692e22330 <unknown>
from #4 0x55b692e634a6 <unknown>
from #5 0x55b692e5a753 <unknown>
from #6 0x55b692e2da14 <unknown>
from #7 0x55b692e2eb7e <unknown>
from #8 0x55b69307832e <unknown>
from #9 0x55b69307bc0e <unknown>
from #10 0x55b69305e610 <unknown>
from #11 0x55b69307cc23 <unknown>
from #12 0x55b693050545 <unknown>
from #13 0x55b69309d6a8 <unknown>
from #14 0x55b69309d836 <unknown>
from #15 0x55b6930b8d13 <unknown>
... 7 levels...
from /var/www/apps/einstein_staging/shared/bundle/ruby/2.3.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/bridge.rb:167:in `execute'
from /var/www/apps/einstein_staging/shared/bundle/ruby/2.3.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/bridge.rb:102:in `create_session'
from /var/www/apps/einstein_staging/shared/bundle/ruby/2.3.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/bridge.rb:56:in `handshake'
from /var/www/apps/einstein_staging/shared/bundle/ruby/2.3.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/chrome/driver.rb:43:in `initialize'
from /var/www/apps/einstein_staging/shared/bundle/ruby/2.3.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/driver.rb:46:in `new'
from /var/www/apps/einstein_staging/shared/bundle/ruby/2.3.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/driver.rb:46:in `for'
from /var/www/apps/einstein_staging/shared/bundle/ruby/2.3.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver.rb:88:in `for'
from /var/www/apps/einstein_staging/shared/bundle/ruby/2.3.0/gems/capybara-2.7.1/lib/capybara/selenium/driver.rb:14:in `browser'
from /var/www/apps/einstein_staging/shared/bundle/ruby/2.3.0/gems/capybara-2.7.1/lib/capybara/selenium/driver.rb:46:in `visit'
from /var/www/apps/einstein_staging/shared/bundle/ruby/2.3.0/gems/capybara-2.7.1/lib/capybara/session.rb:233:in `visit'
from (irb):21
from /var/www/apps/einstein_staging/shared/bundle/ruby/2.3.0/gems/railties-3.2.22.5/lib/rails/commands/console.rb:47:in `start'
from /var/www/apps/einstein_staging/shared/bundle/ruby/2.3.0/gems/railties-3.2.22.5/lib/rails/commands/console.rb:8:in `start'
from /var/www/apps/einstein_staging/shared/bundle/ruby/2.3.0/gems/railties-3.2.22.5/lib/rails/commands.rb:41:in `<top (required)>'
from script/rails:21:in `require'
from script/rails:21:in `<main>'