Adding a new node to existing node cassandra cluster

Question

Starting from one host running Cassandra, I am trying to add a new node and form a cluster.

I update the seeds list on both hosts and after restarting both nodes, I do nodetool status and see both nodes forming a cluster. However, I am seeing some data loss issue. I am not seeing all the data that I added to a column family before I added the new node.

Steps to reproduce:

Start a node with following settings in cassandra.yaml
- initial_token:
- num_tokens:256
- seed_list: host1
Create a keyspace and a column family and enter some data
Start another node, exact same settings and host1 with the following settings changes on both - seeds: host1, host2
When I log in to cal from host2, I do not see all data.

No, the replication factor is still set to 1. What is a good practice, should I set the replication factor to the number of nodes in my cluster ? — Nitin Bhatt
– Nitin Bhatt, Commented Apr 3, 2014 at 20:32
Updating the replication factor does not help. Still seeing the issue. — Nitin Bhatt
– Nitin Bhatt, Commented Apr 4, 2014 at 1:09
The idea with the replication factor, is that you could lose a machine and still have all of your data. With 2 servers and a replication factor of 1, losing a machine means losing half of your data. So with 2 servers it makes sense to go with a replication factor of 2. But if you went to 3 nodes, staying with a RF of 2 would still allow you to get 100% of your data if you lost a node. — Aaron
– Aaron, Commented Apr 4, 2014 at 12:14
I am having this exact same problem. Did you determine a solution? — chrislovecnm
– chrislovecnm, Commented Jun 19, 2014 at 22:49

Alexis Wilke · Accepted Answer · 2016-07-02 18:42:11Z

1

Running:

nodetool cleanup
nodetool repair
nodetool rebuild

should solve the issue.

edited Jul 2, 2016 at 18:42

Alexis Wilke

21.2k11 gold badges111 silver badges183 bronze badges

answered Apr 3, 2014 at 7:06

vivek mishra

1,1628 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Nitin Bhatt Over a year ago

Tried this, still see the same issue. Anything else that I could try.

vivek mishra Over a year ago

With replication factor 1, it will be written only on 1 node.

Nitin Bhatt Over a year ago

Right, but doing a select * should return data from all the hosts right ? I am not sure why select * is not returning the entire data.

Alexis Wilke Over a year ago

@Nitin, you may need to use consistency of QUORUM or ALL if you are not already.

Alexis Wilke Over a year ago

I tried a repair, but somehow it blocks forever (or at least what looks like forever) without doing anything (at least seemingly, since the process does nothing and no data gets transferred...) Would there be a reason for the repair not to work in that situation?

Alexis Wilke · Accepted Answer · 2016-07-02 18:43:45Z

1

Will suggest you to run a nodetool cleanup in both the nodes so that keys get distributed.

edited Jul 2, 2016 at 18:43

Alexis Wilke

21.2k11 gold badges111 silver badges183 bronze badges

answered Apr 3, 2014 at 5:39

Ananth

9919 silver badges25 bronze badges

1 Comment

Nitin Bhatt Over a year ago

Does not work. Surprisingly when I decommission the newly added node, I can see all the data again from CQL in old host.

Collectives™ on Stack Overflow

Adding a new node to existing node cassandra cluster

2 Answers 2

5 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related