0

Im currently doing research about adding node in multiple machine to a cluster. The master node should be x.x.x.246 and the data node that i want to add is x.x.x.99. Running the latest elasticsearch 7.6. I've make sure that both elastic are the same version.

Here is my config file for master node.

cluster.name : elasticsearch
node.name : Node_master
node.master : true
node.data : false
network.host : x.x.x.246
http.port : 9200
transport.tcp.port: 9300
discovery.seed_hosts:
   - x.x.x.246:9300
   - x.x.x.99:9301
cluster.initial_master_nodes:
   - x.x.x.246:9300
   - x.x.x.99:9301

here is my config for data node

cluster.name : elasticsearch
node.name : Node-1
node.master : false
node.data : true
network.host : x.x.x.99
http.port : 9201
transport.tcp.port: 9301
discovery.seed_hosts:
   - x.x.x.246:9300
   - x.x.x.99:9301
cluster.initial_master_nodes:
   - x.x.x.246:9300
   - x.x.x.99:9301

here is the error returned when i restart elasticsearch on data node :

[root@dev elasticsearch]# service elasticsearch restart
Restarting elasticsearch (via systemctl):  Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
                                                           [FAILED]
[root@dev elasticsearch]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2020-03-13 10:29:21 WIB; 16s ago
     Docs: http://www.elastic.co
  Process: 10474 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 10474 (code=exited, status=1/FAILURE)

systemctl view on datanode

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2020-03-13 10:29:21 WIB; 15min ago
     Docs: http://www.elastic.co
  Process: 10474 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 10474 (code=exited, status=1/FAILURE)

Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: at org.elasticsearch.cli.Command.main(Command.java:90)
Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: Caused by: com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException: while scanning a simple key
Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: in 'reader', line 25, column 1:
Mar 13 10:29:21 dev.int.cbn.net.id systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Mar 13 10:29:21 dev.int.cbn.net.id systemd[1]: Failed to start Elasticsearch.
Mar 13 10:29:21 dev.int.cbn.net.id systemd[1]: Unit elasticsearch.service entered failed state.
Mar 13 10:29:21 dev.int.cbn.net.id systemd[1]: elasticsearch.service failed.

whats wrong with the code? i think i followed this instruction

edit master node config

discovery.seed_hosts:
   - x.x.x.246:9300
cluster.initial_master_nodes:
   - Node_master

error log of new master_node config:

[root@localhost elasticsearch]# tail -20 elasticsearch.log
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.6.1.jar:7.6.1]
[2020-03-13T12:21:22,387][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [Node_master] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Node is started with node.data=false, but has shard data: [/var/lib/elasticsearch/nodes/0/indices/e3cMeIVSQdmIwVVgh_Bd0w/0, /var/lib/elasticsearch/nodes/0/indices/Prq1jnKSSoyt3XWJO6iCUg/0, /var/lib/elasticsearch/nodes/0/indices/pnQ2AWwnSca8-hFVL_gtYA/0, /var/lib/elasticsearch/nodes/0/indices/bFsN-CkCSjufBACUL27u0w/0, /var/lib/elasticsearch/nodes/0/indices/mvFmVLaOSN-wAvMgXknzTg/0, /var/lib/elasticsearch/nodes/0/indices/XBSojsDSRs6IXYQNH9j-Jg/0, /var/lib/elasticsearch/nodes/0/indices/0X9qyFynTPac0YZvXn9Ufw/0, /var/lib/elasticsearch/nodes/0/indices/NQbHnD4CTweCU4LgfKR77A/0, /var/lib/elasticsearch/nodes/0/indices/vDe9aUIpQ8ipqbmPEXnTFA/0, /var/lib/elasticsearch/nodes/0/indices/97L23TQ1TJmKHDYzwaYfRg/0, /var/lib/elasticsearch/nodes/0/indices/udZhKWpVSVCAsJnOVcRVLQ/0, /var/lib/elasticsearch/nodes/0/indices/U1b1IEMWRYqzPlDu9TcG0Q/0, /var/lib/elasticsearch/nodes/0/indices/_CLujyNjQgmCxLFBjO79Zg/0, /var/lib/elasticsearch/nodes/0/indices/LNFkjsKnQMCD5yYBHKyyNw/0, /var/lib/elasticsearch/nodes/0/indices/HnFJxvS7TveyGoawLxDFtA/0, /var/lib/elasticsearch/nodes/0/indices/pqkhAZObTj63zylELg7ySQ/0, /var/lib/elasticsearch/nodes/0/indices/61ETZjspQzOk6q5fGn3Twg/0, /var/lib/elasticsearch/nodes/0/indices/5MC-yxWeS6WmubAOIHJt1w/0, /var/lib/elasticsearch/nodes/0/indices/abPk_EJyTluw36cmNf1ZrQ/0, /var/lib/elasticsearch/nodes/0/indices/s-lG4szwR8uMIQfLo_JA1w/0, /var/lib/elasticsearch/nodes/0/indices/Ii5lCjuKRfS5tW4cXEnJng/0, /var/lib/elasticsearch/nodes/0/indices/J6S12mDCR8KGnfWScyiqpw/0, /var/lib/elasticsearch/nodes/0/indices/Sr0g5j7VTzqfcLcd1yoHDQ/0, /var/lib/elasticsearch/nodes/0/indices/8Gk6QC7_TS2p0qRtx6XCmA/0, /var/lib/elasticsearch/nodes/0/indices/BjMiZoZ5SIyBNbwND7fZ5w/0, /var/lib/elasticsearch/nodes/0/indices/pCmnXkAIR6a7k_Kv2uiwig/0, /var/lib/elasticsearch/nodes/0/indices/fSpz7hUCRWuX_YqTOp7FUA/0, /var/lib/elasticsearch/nodes/0/indices/uXXph5eFSPGH7RRuEKDGHQ/0, /var/lib/elasticsearch/nodes/0/indices/nr3MS0J9TzuElqtUjqEwqQ/0, /var/lib/elasticsearch/nodes/0/indices/SNiHj03OT2-YC8wMb7LyLg/0, /var/lib/elasticsearch/nodes/0/indices/pzmAhBOES-KtmESbs0_EKQ/0, /var/lib/elasticsearch/nodes/0/indices/cXhosVdCSayJJrf3tvvG7w/0]. Use 'elasticsearch-node repurpose' tool to clean up
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.6.1.jar:7.6.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.6.1.jar:7.6.1]
Caused by: java.lang.IllegalStateException: Node is started with node.data=false, but has shard data: [/var/lib/elasticsearch/nodes/0/indices/e3cMeIVSQdmIwVVgh_Bd0w/0, /var/lib/elasticsearch/nodes/0/indices/Prq1jnKSSoyt3XWJO6iCUg/0, /var/lib/elasticsearch/nodes/0/indices/pnQ2AWwnSca8-hFVL_gtYA/0, /var/lib/elasticsearch/nodes/0/indices/bFsN-CkCSjufBACUL27u0w/0, /var/lib/elasticsearch/nodes/0/indices/mvFmVLaOSN-wAvMgXknzTg/0, /var/lib/elasticsearch/nodes/0/indices/XBSojsDSRs6IXYQNH9j-Jg/0, /var/lib/elasticsearch/nodes/0/indices/0X9qyFynTPac0YZvXn9Ufw/0, /var/lib/elasticsearch/nodes/0/indices/NQbHnD4CTweCU4LgfKR77A/0, /var/lib/elasticsearch/nodes/0/indices/vDe9aUIpQ8ipqbmPEXnTFA/0, /var/lib/elasticsearch/nodes/0/indices/97L23TQ1TJmKHDYzwaYfRg/0, /var/lib/elasticsearch/nodes/0/indices/udZhKWpVSVCAsJnOVcRVLQ/0, /var/lib/elasticsearch/nodes/0/indices/U1b1IEMWRYqzPlDu9TcG0Q/0, /var/lib/elasticsearch/nodes/0/indices/_CLujyNjQgmCxLFBjO79Zg/0, /var/lib/elasticsearch/nodes/0/indices/LNFkjsKnQMCD5yYBHKyyNw/0, /var/lib/elasticsearch/nodes/0/indices/HnFJxvS7TveyGoawLxDFtA/0, /var/lib/elasticsearch/nodes/0/indices/pqkhAZObTj63zylELg7ySQ/0, /var/lib/elasticsearch/nodes/0/indices/61ETZjspQzOk6q5fGn3Twg/0, /var/lib/elasticsearch/nodes/0/indices/5MC-yxWeS6WmubAOIHJt1w/0, /var/lib/elasticsearch/nodes/0/indices/abPk_EJyTluw36cmNf1ZrQ/0, /var/lib/elasticsearch/nodes/0/indices/s-lG4szwR8uMIQfLo_JA1w/0, /var/lib/elasticsearch/nodes/0/indices/Ii5lCjuKRfS5tW4cXEnJng/0, /var/lib/elasticsearch/nodes/0/indices/J6S12mDCR8KGnfWScyiqpw/0, /var/lib/elasticsearch/nodes/0/indices/Sr0g5j7VTzqfcLcd1yoHDQ/0, /var/lib/elasticsearch/nodes/0/indices/8Gk6QC7_TS2p0qRtx6XCmA/0, /var/lib/elasticsearch/nodes/0/indices/BjMiZoZ5SIyBNbwND7fZ5w/0, /var/lib/elasticsearch/nodes/0/indices/pCmnXkAIR6a7k_Kv2uiwig/0, /var/lib/elasticsearch/nodes/0/indices/fSpz7hUCRWuX_YqTOp7FUA/0, /var/lib/elasticsearch/nodes/0/indices/uXXph5eFSPGH7RRuEKDGHQ/0, /var/lib/elasticsearch/nodes/0/indices/nr3MS0J9TzuElqtUjqEwqQ/0, /var/lib/elasticsearch/nodes/0/indices/SNiHj03OT2-YC8wMb7LyLg/0, /var/lib/elasticsearch/nodes/0/indices/pzmAhBOES-KtmESbs0_EKQ/0, /var/lib/elasticsearch/nodes/0/indices/cXhosVdCSayJJrf3tvvG7w/0]. Use 'elasticsearch-node repurpose' tool to clean up
        at org.elasticsearch.env.NodeEnvironment.ensureNoShardData(NodeEnvironment.java:1083) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:326) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.1.jar:7.6.1]
        ... 6 more

any help or instructions will be appreciated so much. thank you.

Improve this question

1 Answer 1

Reset to default
1

discovery.seed_hosts : This should contain list of master eligible nodes in the form of host or host:port.

cluster.initial_master_nodes: This should be list of nodes which are master eligible and will be used to form cluster for the very first time. List contains node names.

In your case you have only one master eligible node therefore change the above mentioned properties as below.

For master node change to:

discovery.seed_hosts:
   - x.x.x.246:9300
cluster.initial_master_nodes:
   - Node_master

For data node cluster.initial_master_nodes and change seed hosts to:

discovery.seed_hosts:
       - x.x.x.246:9300
Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

hi there, thank you so much for the respond and edit. Ive tried the master node edit and when i run the elasticsearch it returning error as : [root@localhost elasticsearch]# service elasticsearch start Starting elasticsearch (via systemctl): Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details. [FAILED]
Also this answer and comments might help.
@yuliansen you can try what the error suggests i.e. use elasticsearch-node repurpose tool or try deleting data folder on both nodes. Note that after you delete data folder you will loose all the documents indexed so delete only if you are sure about that.
hi thank you for your response, is this really the procedure to be happened? because this doesnt meet my expectation if everytime i want to expanding my cluster i have to lose my data. judging by my conf, do you find any anomaly?
@yuliansen No, this is not for every time. What you seem to have tried is that when you first start up the cluster you started the master node as data node as well. Then you changed the config of the node to be only master node ( not a data node) and hence the error. You need to make sure and have understanding these configs. Adding/removing nodes (with correct configs) doesn't lead to such error and will not result into data loss
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.