multi-stage docker build with bundler - ruby

Question

ARG RUBY_VERSION=3.1.0
ARG IMAGE_OS=slim-buster
FROM ruby:${RUBY_VERSION}-${IMAGE_OS} as prod-gem-base
ENV APP_ENVIRONMENT=production
ENV RACK_ENV=production
ARG CI_ACCESS_TOKEN
WORKDIR /contract-service
RUN apt-get update && apt-get install -qqy build-essential ca-certificates tzdata shared-mime-info git curl openssh-client libpq-dev && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
gem install bundler:2.3.13 && \
bundle config --global frozen 1 && \
bundle config set without 'development test' && \
mkdir -p -m 600 $HOME/.ssh && ssh-keyscan gitlab.xxx.de >> $HOME/.ssh/known_hosts && \
if [ -n $CI_ACCESS_TOKEN ]; then git config --global url."https://gitlab-ci-token:${CI_ACCESS_TOKEN}#gitlab.xxx.de /".insteadOf git#gitlab.xxx.de: ; fi
COPY Gemfile Gemfile.lock ./
RUN --mount=type=ssh bundle install --jobs 20 --retry 5 && \
bundle clean --force && \
rm -rf /usr/local/bundle/cache/*.gem && \
find /usr/local/bundle/gems/ -name "*.c" -delete && \
find /usr/local/bundle/gems/ -name "*.o" -delete
ARG RUBY_VERSION
ARG IMAGE_OS
FROM ruby:${RUBY_VERSION}-${IMAGE_OS} as prod-core-base
ENV APP_ENVIRONMENT=production
ENV RACK_ENV=production
WORKDIR /contract-service
RUN mkdir -p /tmp/pids && apt-get update && apt-get install -qqy --no-install-recommends shared-mime-info openssh-client libpq-dev && \
COPY --from=prod-gem-base /usr/local/bundle/ /usr/local/bundle/
COPY . .
EXPOSE 3005
CMD ["bundle", "exec", "puma", "--pidfile", "/tmp/pids/server.pid", "-b", "tcp://0.0.0.0", "-p", "3005"]
I am not sure how many ruby people are here. I am creating an with multi-stage build to cache the gems. I am building a production image but when the container is run the bundler outputs error as follows.
bundler: failed to load command: puma (/usr/local/bundle/bin/puma)
cntrct-contract.1.dfhae8xfhdip#stage-app01-ruh2 | /usr/local/bundle/gems/bundler-2.3.13/lib/bundler/definition.rb:486:in `materialize': Could not find rubocop-1.31.2, byebug-11.1.3, simplecov-0.21.2, simplecov-cobertura-2.1.0, webmock-3.13.0, cucumber-8.0.0, rspec-3.11.0, rspec-core-3.11.0, rspec_junit_formatter-0.5.1, rack-test-2.0.2, json-2.6.2, parallel-1.22.1, parser-3.1.2.0, rainbow-3.1.1, regexp_parser-2.5.0, rubocop-ast-1.19.1, ruby-progressbar-1.11.0, unicode-display_width-2.2.0, docile-1.4.0, simplecov-html-0.12.3, simplecov_json_formatter-0.1.4, crack-0.4.5, hashdiff-1.0.1, builder-3.2.4, cucumber-ci-environment-9.0.4, cucumber-core-11.0.0, cucumber-cucumber-expressions-15.2.0, cucumber-gherkin-23.0.1, cucumber-html-formatter-19.2.0, cucumber-messages-18.0.0, diff-lcs-1.5.0, multi_test-1.1.0, sys-uname-1.2.2, rspec-expectations-3.11.0, rspec-mocks-3.11.1, rspec-support-3.11.0, ast-2.4.2, cucumber-tag-expressions-4.1.0, ffi-1.15.5 in any of the sources (Bundler::GemNotFound)
cntrct-contract.1.dfhae8xfhdip#stage-app01-ruh2 | from /usr/local/bundle/gems/bundler-2.3.13/lib/bundler/definition.rb:191:in `specs'
cntrct-contract.1.8tfk50ss339q#stage-app01-ruh2 | from /usr/local/bundle/bin/bundle:25:in `<main>'
cntrct-contract.1.dfhae8xfhdip#stage-app01-ruh2 | from /usr/local/bundle/gems/bundler-2.3.13/lib/bundler/definition.rb:239:in `specs_for'
cntrct-contract.1.dfhae8xfhdip#stage-app01-ruh2 | from /usr/local/bundle/gems/bundler-2.3.13/lib/bundler/runtime.rb:18:in `setup'
cntrct-contract.1.dfhae8xfhdip#stage-app01-ruh2 | from /usr/local/bundle/gems/bundler-2.3.13/lib/bundler.rb:151:in `setup'
cntrct-contract.1.dfhae8xfhdip#stage-app01-ruh2 | from /usr/local/bundle/gems/bundler-2.3.13/lib/bundler/setup.rb:20:in `block in <top (required)>'
The issues is bundler is trying to load the development and test related gems which is not in the final image. Can someone tell me how to do it correctly with multi-stage docker build ?
Update: if I add RUN bundle config set without 'development test' && bundle after the COPY .. then everything works. I still didn't get the original issue!!! What am I missing ?