SonarQube upgrade to 6.7.1 LTS: Unrecoverable Indexation Failures - elasticsearch

Question

I have successfully upgraded SonarQube to ver. 6.5, including the database upgrade, and I am currently trying to upgrade SonarQube to ver. 6.7.1 LTS. The New SonarQube version is being installed on a Linux 64 bit system and is connected to a 2014 Microsoft SQL database. Every time I try to launch the 6.7.1 version of SonarQube it fails with the error "Background initialization failed". If I run the new SonarQube using an empty Microsoft SQL database, then it will start up fine with no issues. The "Background initialization failed" issue only occurs when I connect the new SonarQube to the upgraded database. I have tried adding memory to the heap for ElasticSearch and reducing the number of issues being processed. Any help to resolve this issue would be greatly appreciated.
Web log:
web[][o.s.p.ProcessEntryPoint] Starting web
web[][o.a.t.u.n.NioSelectorPool] Using a shared selector for servlet write/read
web[][o.e.p.PluginsService] no modules loaded
web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.join.ParentJoinPlugin]
web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
web[][i.n.c.MultithreadEventLoopGroup] -Dio.netty.eventLoopThreads: 64
web[][i.n.u.i.PlatformDependent0] -Dio.netty.noUnsafe: false
web[][i.n.u.i.PlatformDependent0] Java version: 8
web[][i.n.u.i.PlatformDependent0] sun.misc.Unsafe.theUnsafe: available
web[][i.n.u.i.PlatformDependent0] sun.misc.Unsafe.copyMemory: available
web[][i.n.u.i.PlatformDependent0] java.nio.Buffer.address: available
web[][i.n.u.i.PlatformDependent0] direct buffer constructor: available
web[][i.n.u.i.PlatformDependent0] java.nio.Bits.unaligned: available, true
web[][i.n.u.i.PlatformDependent0] jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9
web[][i.n.u.i.PlatformDependent0] java.nio.DirectByteBuffer.<init>(long, int): available
web[][i.n.u.i.PlatformDependent] sun.misc.Unsafe: available
web[][i.n.u.i.PlatformDependent] -Dio.netty.tmpdir: /../../sonarqube-6.7.1/temp (java.io.tmpdir)
web[][i.n.u.i.PlatformDependent] -Dio.netty.bitMode: 64 (sun.arch.data.model)
web[][i.n.u.i.PlatformDependent] -Dio.netty.noPreferDirect: false
web[][i.n.u.i.PlatformDependent] -Dio.netty.maxDirectMemory: 4772593664 bytes
web[][i.n.u.i.PlatformDependent] -Dio.netty.uninitializedArrayAllocationThreshold: -1
web[][i.n.u.i.CleanerJava6] java.nio.ByteBuffer.cleaner(): available
web[][i.n.c.n.NioEventLoop] -Dio.netty.noKeySetOptimization: false
web[][i.n.c.n.NioEventLoop] -Dio.netty.selectorAutoRebuildThreshold: 512
web[][i.n.u.i.PlatformDependent] org.jctools-core.MpscChunkedArrayQueue: available
web[][i.n.c.DefaultChannelId] -Dio.netty.processId: ***** (auto-detected)
web[][i.netty.util.NetUtil] -Djava.net.preferIPv4Stack: true
web[][i.netty.util.NetUtil] -Djava.net.preferIPv6Addresses: false
web[][i.netty.util.NetUtil] Loopback interface: lo (lo, 127.0.0.1)
web[][i.netty.util.NetUtil] /proc/sys/net/core/somaxconn: 128
web[][i.n.c.DefaultChannelId] -Dio.netty.machineId: ***** (auto-detected)
web[][i.n.u.ResourceLeakDetector] -Dio.netty.leakDetection.level: simple
web[][i.n.u.ResourceLeakDetector] -Dio.netty.leakDetection.maxRecords: 4
web[][i.n.b.PooledByteBufAllocator] -Dio.netty.allocator.numHeapArenas: 47
web[][i.n.b.PooledByteBufAllocator] -Dio.netty.allocator.numDirectArenas: 47
web[][i.n.b.PooledByteBufAllocator] -Dio.netty.allocator.pageSize: 8192
web[][i.n.b.PooledByteBufAllocator] -Dio.netty.allocator.maxOrder: 11
web[][i.n.b.PooledByteBufAllocator] -Dio.netty.allocator.chunkSize: 16777216
web[][i.n.b.PooledByteBufAllocator] -Dio.netty.allocator.tinyCacheSize: 512
web[][i.n.b.PooledByteBufAllocator] -Dio.netty.allocator.smallCacheSize: 256
web[][i.n.b.PooledByteBufAllocator] -Dio.netty.allocator.normalCacheSize: 64
web[][i.n.b.PooledByteBufAllocator] -Dio.netty.allocator.maxCachedBufferCapacity: 32768
web[][i.n.b.PooledByteBufAllocator] -Dio.netty.allocator.cacheTrimInterval: 8192
web[][i.n.b.PooledByteBufAllocator] -Dio.netty.allocator.useCacheForAllThreads: true
web[][i.n.b.ByteBufUtil] -Dio.netty.allocator.type: pooled
web[][i.n.b.ByteBufUtil] -Dio.netty.threadLocalDirectBufferSize: 65536
web[][i.n.b.ByteBufUtil] -Dio.netty.maxThreadLocalCharBufferSize: 16384
web[][i.n.b.AbstractByteBuf] -Dio.netty.buffer.bytebuf.checkAccessible: true
web[][i.n.u.ResourceLeakDetectorFactory] Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector#6c6be5c2
web[][i.n.util.Recycler] -Dio.netty.recycler.maxCapacityPerThread: 32768
web[][i.n.util.Recycler] -Dio.netty.recycler.maxSharedCapacityFactor: 2
web[][i.n.util.Recycler] -Dio.netty.recycler.linkCapacity: 16
web[][i.n.util.Recycler] -Dio.netty.recycler.ratio: 8
web[][o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [127.0.0.1:*****]
web[][o.s.s.p.LogServerVersion] SonarQube Server / 6.7.1.35068 / 426519346f51f7b980a76f9050f983110550509d
web[][o.sonar.db.Database] Create JDBC data source for jdbc:sqlserver:*****
web[][o.s.s.p.ServerFileSystemImpl] SonarQube home: /../../sonarqube-6.7.1
web[][o.s.s.u.SystemPasscodeImpl] System authentication by passcode is disabled
web[][o.s.c.i.DefaultI18n] Loaded 2094 properties from l10n bundles
web[][o.s.s.p.d.m.c.MssqlCharsetHandler] Verify that database collation is case-sensitive and accent-sensitive
web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceFilter#7e977d45 [pattern=UrlPattern{inclusions=[/api/system/migrate_db/*, ...], exclusions=[/api/properties*, ...]}]
web[][o.s.s.a.TomcatAccessLog] Tomcat is started
web[][o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port ****
web[][o.s.s.p.UpdateCenterClient] Update center:https://update.sonarsource.org/update-center.properties (no proxy)
web[][o.s.a.r.Languages] No language available
web[][o.s.s.e.RecoveryIndexer] Elasticsearch recovery - sonar.search.recovery.minAgeInMs=300000
web[][o.s.s.e.RecoveryIndexer] Elasticsearch recovery - sonar.search.recovery.loopLimit=10000
web[][o.s.s.s.LogServerId] Server ID: *****
web[][o.s.s.e.RecoveryIndexer] Elasticsearch recovery - sonar.search.recovery.delayInMs=300000
web[][o.s.s.e.RecoveryIndexer] Elasticsearch recovery - sonar.search.recovery.initialDelayInMs=26327
web[][o.s.s.t.TelemetryDaemon] Sharing of SonarQube statistics is enabled.
web[][o.s.s.n.NotificationDaemon] Notification service started (delay 60 sec.)
web[][o.s.s.s.GeneratePluginIndex] Generate scanner plugin index
web[][o.s.s.s.GeneratePluginIndex] Generate scanner plugin index (done) | time=1ms
web[][o.s.s.s.RegisterPlugins] Register plugins
web[][o.s.s.s.RegisterPlugins] Register plugins (done) | time=167ms
web[][o.s.s.s.RegisterMetrics] Register metrics
web[][o.s.s.s.RegisterMetrics] Register metrics (done) | time=2734ms
web[][o.s.s.r.RegisterRules] Register rules
web[][o.s.s.r.RegisterRules] Register rules (done) | time=685ms
web[][o.s.s.q.BuiltInQProfileRepositoryImpl] Load quality profiles
web[][o.s.s.q.BuiltInQProfileRepositoryImpl] Load quality profiles (done) | time=2ms
web[][o.s.s.s.RegisterPermissionTemplates] Register permission templates
web[][o.s.s.s.RegisterPermissionTemplates] Register permission templates (done) | time=153ms
web[][o.s.s.s.RenameDeprecatedPropertyKeys] Rename deprecated property keys
web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceFilter#3a6e54b [pattern=UrlPattern{inclusions=[/api/measures/component/*, ...], exclusions=[/api/properties*, ...]}]
web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.DeprecatedPropertiesWsFilter#3b2c45f3 [pattern=UrlPattern{inclusions=[/api/properties/*], exclusions=[]}]
web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceReroutingFilter#42ffe60e [pattern=UrlPattern{inclusions=[/api/components/bulk_update_key, ...], exclusions=[]}]
web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.InitFilter#3bc1cd0f [pattern=UrlPattern{inclusions=[/sessions/init/*], exclusions=[]}]
web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.OAuth2CallbackFilter#533fe992 [pattern=UrlPattern{inclusions=[/oauth2/callback/*], exclusions=[]}]
web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.LoginAction#54370dcd [pattern=UrlPattern{inclusions=[/api/authentication/login], exclusions=[]}]
web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.LogoutAction#7bc801b4 [pattern=UrlPattern{inclusions=[/api/authentication/logout], exclusions=[]}]
web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.ValidateAction#2e0576fc [pattern=UrlPattern{inclusions=[/api/authentication/validate], exclusions=[]}]
web[][o.s.s.e.IndexerStartupTask] Indexing of type [issues/issue] ...
web[][o.s.s.es.BulkIndexer] 1387134 requests processed (23118 items/sec)
web[][o.s.s.es.BulkIndexer] 2715226 requests processed (22134 items/sec)
web[][o.s.s.es.BulkIndexer] 3944404 requests processed (20486 items/sec)
web[][o.s.s.es.BulkIndexer] 5319447 requests processed (22917 items/sec)
web[][o.s.s.es.BulkIndexer] 6871423 requests processed (25866 items/sec)
web[][o.s.s.es.BulkIndexer] 7814247 requests processed (15713 items/sec)
web[][o.s.s.es.BulkIndexer] 7814247 requests processed (0 items/sec)
web[][o.s.s.es.BulkIndexer] 7814247 requests processed (0 items/sec)
web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube
java.lang.IllegalStateException: Unrecoverable indexation failures
at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:39)
at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:117)
at org.sonar.server.issue.index.IssueIndexer.doIndex(IssueIndexer.java:247)
at org.sonar.server.issue.index.IssueIndexer.indexOnStartup(IssueIndexer.java:95)
at org.sonar.server.es.IndexerStartupTask.indexUninitializedTypes(IndexerStartupTask.java:68)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at org.sonar.server.es.IndexerStartupTask.execute(IndexerStartupTask.java:55)
at java.util.Optional.ifPresent(Optional.java:159)
at org.sonar.server.platform.platformlevel.PlatformLevelStartup$1.doPrivileged(PlatformLevelStartup.java:84)
at org.sonar.server.user.DoPrivileged.execute(DoPrivileged.java:45)
at org.sonar.server.platform.platformlevel.PlatformLevelStartup.start(PlatformLevelStartup.java:80)
at org.sonar.server.platform.Platform.executeStartupTasks(Platform.java:196)
at org.sonar.server.platform.Platform.access$400(Platform.java:46)
at org.sonar.server.platform.Platform$1.lambda$doRun$1(Platform.java:121)
at org.sonar.server.platform.Platform$AutoStarterRunnable.runIfNotAborted(Platform.java:371)
at org.sonar.server.platform.Platform$1.doRun(Platform.java:121)
at org.sonar.server.platform.Platform$AutoStarterRunnable.run(Platform.java:355)
at java.lang.Thread.run(Thread.java:748)
web[][o.s.s.p.Platform] Background initialization of SonarQube done
web[][o.s.p.StopWatcher] Stopping process
===========================================================================
Edit: I have referenced the link provided prior to my initial post. The post referenced "free space" which I assumed to mean disk space, here is my disk space values where SonarQube 6.7.1 is installed:
1K-blocks Used Available Use%
251531268 16204576 235326692 7% /prod/appl
Also here is a portion of my elasticsearch log where the error in the web.log occurs. SonarQube 6.7.1 uses Elasticsearch-5.
Elasticsearch log:
es[][o.e.i.IndexingMemoryController] write indexing buffer to disk for shard [[issues][0]] to free up its [29.8mb] indexing buffer
es[][o.e.i.s.IndexShard] add [29.8mb] writing bytes for shard [[issues][0]]
es[][o.e.i.e.Engine] use refresh to write indexing buffer (heap size=[23.5mb]), to also clear version map (heap size=[6.3mb])
es[][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [_parent#authorization][49] took [462.3micros]
es[][o.e.i.s.IndexShard] remove [29.8mb] writing bytes for shard [[issues][0]]
es[][o.e.i.IndexingMemoryController] now write some indexing buffers: total indexing heap bytes used [104.3mb] vs indices.memory.index_buffer_size [98.9mb], currently writing bytes [0b], [5] shards with non-zero indexing buffer
es[][o.e.i.IndexingMemoryController] write indexing buffer to disk for shard [[issues][1]] to free up its [54.8mb] indexing buffer
es[][o.e.i.s.IndexShard] add [54.8mb] writing bytes for shard [[issues][1]]
es[][o.e.i.e.Engine] use IndexWriter.flush to write indexing buffer (heap size=[51.1mb]) since version map is small (heap size=[3.6mb])
es[][o.e.i.s.IndexShard] remove [54.8mb] writing bytes for shard [[issues][1]]
es[][o.e.i.IndexingMemoryController] now write some indexing buffers: total indexing heap bytes used [104.2mb] vs indices.memory.index_buffer_size [98.9mb], currently writing bytes [0b], [5] shards with non-zero indexing buffer
es[][o.e.i.IndexingMemoryController] write indexing buffer to disk for shard [[issues][1]] to free up its [50.7mb] indexing buffer
es[][o.e.i.s.IndexShard] add [50.7mb] writing bytes for shard [[issues][1]]
es[][o.e.i.e.Engine] use IndexWriter.flush to write indexing buffer (heap size=[43.9mb]) since version map is small (heap size=[6.7mb])
es[][o.e.i.s.IndexShard] remove [50.7mb] writing bytes for shard [[issues][1]]
es[][o.e.i.IndexingMemoryController] now write some indexing buffers: total indexing heap bytes used [100.1mb] vs indices.memory.index_buffer_size [98.9mb], currently writing bytes [0b], [5] shards with non-zero indexing buffer
es[][o.e.i.IndexingMemoryController] write indexing buffer to disk for shard [[issues][1]] to free up its [31.5mb] indexing buffer
es[][o.e.i.s.IndexShard] add [31.5mb] writing bytes for shard [[issues][1]]
es[][o.e.i.e.Engine] use refresh to write indexing buffer (heap size=[23.3mb]), to also clear version map (heap size=[8.2mb])
es[][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [_parent#authorization][46] took [988.8micros]
es[][o.e.i.s.IndexShard] remove [31.5mb] writing bytes for shard [[issues][1]]
es[][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [_parent#authorization][46] took [880.6micros]
es[][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [_parent#authorization][57] took [510.7micros]
es[][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [_parent#authorization][49] took [829.3micros]
es[][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [_parent#authorization][47] took [412.9micros]
es[][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [_parent#authorization][43] took [277.4micros]
es[][o.e.i.e.InternalEngine$EngineMergeScheduler] merge segment [_kh] done: took [30.9s], [343.7 MB], [3,159,200 docs], [0s stopped], [1.5s throttled], [169.4 MB written], [Infinity MB/sec throttle]
es[][o.e.i.e.InternalEngine$EngineMergeScheduler] merge segment [_oc] done: took [28.9s], [290.9 MB], [2,593,116 docs], [0s stopped], [0s throttled], [232.1 MB written], [Infinity MB/sec throttle]
es[][o.e.i.e.InternalEngine$EngineMergeScheduler] merge segment [_pz] done: took [30.6s], [341.3 MB], [2,573,716 docs], [0s stopped], [0s throttled], [266.1 MB written], [Infinity MB/sec throttle]
es[][o.e.i.e.InternalEngine$EngineMergeScheduler] merge segment [_th] done: took [35.2s], [346.3 MB], [3,102,397 docs], [0s stopped], [0s throttled], [262.0 MB written], [Infinity MB/sec throttle]
es[][o.e.c.s.ClusterService] processing [update-settings]: execute
es[][o.e.i.IndicesQueryCache] using [node] query cache with size [98.9mb] max filter count [10000]
es[][o.e.i.IndicesService] creating Index [[issues/WmTjz_-ITtyPeqpDlqPeFg]], shards [5]/[0] - reason [metadata verification]
es[][o.e.i.s.IndexStore] using index.store.throttle.type [NONE], with index.store.throttle.max_bytes_per_sec [null]
es[][o.e.i.m.MapperService] using dynamic[false]
es[][o.e.i.c.b.BitsetFilterCache] clearing all bitsets because [close]
es[][o.e.i.c.q.IndexQueryCache] full cache clear, reason [close]
es[][o.e.i.c.b.BitsetFilterCache] clearing all bitsets because [close]
es[][o.e.c.s.ClusterService] cluster state updated, version [17], source [update-settings]
es[][o.e.c.s.ClusterService] publishing cluster state version [17]
es[][o.e.c.s.ClusterService] applying cluster state version 17
es[][o.e.c.s.ClusterService] set local cluster state to version 17
es[][o.e.c.s.ClusterService] processing [update-settings]: took [19ms] done applying updated cluster_state (version: 17, uuid: dkhQacKBQGS5YsyMqp1kmQ)
es[][o.e.n.Node] stopping ...