VERIFIED SOLUTION i
X

Resolve java.io.IOException: Unable to connect to Elasticsearch instance in Spectrum 12

Issue

When Spectrum 12.0 is installed fresh / upgraded from previous release, the Spectrum server will not start.

The wrapper.log file indicates the following error stack:

INFO   | jvm 1    | 2017/09/05 11:21:07 | INFO  [ElasticsearchExternalBean] Starting Index from D:\Program Files\Pitney Bowes\Spectrum\index\elasticsearch
INFO   | jvm 1    | 2017/09/05 11:24:07 | INFO  [ElasticsearchExternalBean] Stopping Index instance
INFO   | jvm 1    | 2017/09/05 11:24:07 | INFO  [ElasticsearchExternalBean] Stopping Index instance
INFO   | jvm 1    | 2017/09/05 11:24:07 | INFO  [Neo4jExternalBean] Stopping Repository instance
INFO   | jvm 1    | 2017/09/05 11:24:18 | Sep 05, 2017 11:24:18 AM org.neo4j.driver.internal.logging.JULogger warn
INFO   | jvm 1    | 2017/09/05 11:24:18 | WARNING: WARNING: Failed to close tx due to error: org.neo4j.driver.v1.exceptions.ServiceUnavailableException: Connection terminated while receiving data. This can happen due to network instabilities, or due to restarts of the database.
INFO   | jvm 1    | 2017/09/05 11:24:19 | INFO  [ApplicationContextRegistry] Stopped
INFO   | jvm 1    | 2017/09/05 11:24:19 | INFO  [HazelcastBean] Stopped
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp: 
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error: Encountered an error running main:
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error: java.lang.reflect.InvocationTargetException
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at java.lang.reflect.Method.invoke(Method.java:498)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at com.g1.dcg.boot.Main.main(Main.java:82)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at java.lang.reflect.Method.invoke(Method.java:498)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at java.lang.Thread.run(Thread.java:745)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error: Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'enterpriseServer' defined in file [D:\Program Files\Pitney Bowes\Spectrum\server\app\conf\spring\enterprise-server.xml]: Invocation of init method failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'indexService' defined in URL [jar:file:/d:/Program%20Files/Pitney%20Bowes/Spectrum/server/app/lib/spectrum-server-index-elasticsearch-external-12.0.jar!/spring/spectrum-container-index-elasticsearch.xml]: Invocation of init method failed; nested exception is java.io.IOException: Unable to connect to Elasticsearch instance
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1578)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:772)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at com.pb.spectrum.platform.server.container.startup.Main.main(Main.java:37)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     ... 11 more
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error: Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'indexService' defined in URL [jar:file:/d:/Program%20Files/Pitney%20Bowes/Spectrum/server/app/lib/spectrum-server-index-elasticsearch-external-12.0.jar!/spring/spectrum-container-index-elasticsearch.xml]: Invocation of init method failed; nested exception is java.io.IOException: Unable to connect to Elasticsearch instance
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1578)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:772)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at com.pb.spectrum.platform.server.container.startup.Server.createApplicationContext(Server.java:191)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at com.pb.spectrum.platform.server.container.startup.Server.configureSpringAppContext(Server.java:176)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at com.pb.spectrum.platform.server.container.startup.Server.start(Server.java:125)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at java.lang.reflect.Method.invoke(Method.java:498)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1706)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1645)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     ... 21 more
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error: Caused by: java.io.IOException: Unable to connect to Elasticsearch instance
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at com.pb.spectrum.platform.server.index.elasticsearch.ElasticsearchExternalBean.waitUntilRunning(ElasticsearchExternalBean.java:213)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at com.pb.spectrum.platform.server.index.elasticsearch.ElasticsearchExternalBean.start(ElasticsearchExternalBean.java:129)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at java.lang.reflect.Method.invoke(Method.java:498)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1706)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1645)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574)
INFO   | jvm 1    | 2017/09/05 11:24:19 | WrapperSimpleApp Error:     ... 40 more
STATUS | wrapper  | 2017/09/05 11:24:23 | <-- Wrapper Stopped

index\elasticsearch\logs\spectrum-logs.log contains following error stack:

Exception in thread "main" BindTransportException[Failed to resolve publish address]; nested: IllegalArgumentException[publish address: {::} is wildcard, but multiple addresses specified: this makes no sense];
Likely root cause: java.lang.IllegalArgumentException: publish address: {::} is wildcard, but multiple addresses specified: this makes no sense

Cause

There are several possibilities that could cause this error:
  1. The "Cognos" application was also installed on the same server as Spectrum. Cognos uses port 9300 as it's main access port. Spectrum "elasticsearch" is trying to use the same port and it is unavailable.
  2. Wildcard (0.0.0.0) is returning multiple addresses for elasticsearch's publish host.
  3. The server Spectrum is installed on does not have enough memory(RAM) for Spectrum to start properly.

Resolution

UPDATED: December 18, 2017


If there is a port conflict:
  • In the "server\app\conf\spectrum-container.properties", change the "spectrum.index.tcp.port" to a different port other than 9300. To use a different port in a clustered environment:
    • Specify the port you want to use instead of 9300 in spectrum.index.tcp.portand after the colon in spectrum.index.client.addresses.
    • Specify the seed nodes for the search index in spectrum.index.server.cluster.seeds.
If wildcard (0.0.0.0) is the problem:
  • In the "index\elasticsearch.template" change default configuration for "network.publish_host" from "0.0.0.0" to machine's IP address.These errors can be identified by reviewing "index\elasticsearch\logs\spectrum-logs.log"
If memory is the problem:
  • More system memory(RAM) will need to allocated to the server in order to start. Memory problems can be identified by reviewing the "index\elasticsearch\logs\spectrum-logs.log"

Environment Details

Product Feature: Installation / Upgrade

 

Downloads

  • No Downloads