Server Errors While Writing With Python Cassandra Driver

Question

code=1000 [Unavailable exception] message="Cannot achieve consistency level ONE" info={'required_replicas': 1, 'alive_replicas': 0, 'consistency': 'ONE'}
code=1100 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}
code=1200 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}

I am inserting into Cassandra Cassandra 2.0.13(single node for testing) by python cassandra-driver version 2.6

The following are my keyspace and table definitions:

CREATE KEYSPACE test_keyspace WITH replication = {   'class': 'SimpleStrategy',   'replication_factor': '1' };


 CREATE TABLE test_table (
  key text PRIMARY KEY,
  column1 text,
  ...,
  column17 text
 ) WITH COMPACT STORAGE AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  read_repair_chance=0.100000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'SnappyCompressor'};

What I tried:

1) multiprocessing(protocol version set to 1) each process has its own cluster, session(default_timeout set to 30.0)

def get_cassandra_session():
    """creates cluster and gets the session base on key space"""
    # be aware that session cannot be shared between threads/processes
    # or it will raise OperationTimedOut Exception
    if CLUSTER_HOST2:
        cluster = cassandra.cluster.Cluster([CLUSTER_HOST1, CLUSTER_HOST2])
    else:
        # if only one address is available, we have to use older protocol version
        cluster = cassandra.cluster.Cluster([CLUSTER_HOST1], protocol_version=1)

    session = cluster.connect(KEY_SPACE)
    session.default_timeout = 30.0
    return session

2) batch insert (protocol version set to 2 because BatchStatement is enabled on Cassandra 2.X)

def batch_insert(session, batch_queue, batch):
    try:
        insert_user = session.prepare("INSERT INTO " + db.TABLE + " (" + db.COLUMN1 + "," + db.COLUMN2 + "," + db.COLUMN3 +
                                      "," + db.COLUMN4 + ") VALUES (?,?,?,?)")
        while batch_queue.qsize() > 0:
            '''batch queue size is 1000'''
            row_tuple = batch_queue.get()
            batch.add(insert_user, row_tuple)

        session.execute(batch)
    except Exception as e:
        logger.error("batch insert fail.... %s", e)

the above function is invoked by:

 batch = BatchStatement(consistency_level=ConsistencyLevel.ONE)
 batch_insert(session, batch_queue, batch)

tuples are stored in batch_queue.

3) synchronizing execution Several days ago I post another question Cassandra update fails , cassandra was complaining about TimeOut issue. I was using synchronize execution for updating.

Can anyone help, is this my code issue or python cassandra-driver issue or Cassandra itself ?

Thanks a million!

Adam Holmberg · Accepted Answer · 2015-07-13 21:55:45Z

3

If your question is about those errors at the top, those are server-side error responses.

The first says that the coordinator you contacted cannot satisfy the request at CL.ONE, with the nodes it believes are alive. This can happen if all replicas are down (more likely with a low replication factor).

The other two errors are timeouts, where the coordinator didn't get responses from 'live' nodes in a time configured in the cassandra.yaml.

All of these indicate that the cluster you're connected to is not healthy. This could be because it is overwhelmed (high GC pauses), or experiencing network issues. Check the server logs for clues.

answered Jul 13, 2015 at 21:55

Adam Holmberg

7,3653 gold badges35 silver badges54 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Haifeng Zhang Over a year ago

is it possible because of the cassandra.yaml configuration issue? I have no clue and cannot find any helpful message in the log file. +1 for your replay, thanks

Haifeng Zhang Over a year ago

Hi @Adam . Is the timeout a failure or is the row updated locally?

Adam Holmberg Over a year ago

It would have stored the hint locally, and handed it off assuming the replica came back. To be clear: I don't think there's a "cassandra.yaml configuration issue". That is where you might change those settings to extend timeouts, but the real issue is probably cluster stability. I'm surprised there was nothing helpful in the logs.

Aryan · Accepted Answer · 2020-11-21 11:10:08Z

0

I got the following error, which looks very similar:

cassandra.Unavailable: Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level LOCAL_ONE" info={'consistency': 'LOCAL_ONE', 'alive_replicas': 0, 'required_replicas': 1}

When I added a sleep(0.5) in the code, it worked fine. I was trying to write too much too fast...

edited Nov 21, 2020 at 11:10

Aryan

3,6465 gold badges23 silver badges49 bronze badges

answered Nov 18, 2019 at 20:35

Oeyvind

3577 silver badges22 bronze badges

Collectives™ on Stack Overflow

Server Errors While Writing With Python Cassandra Driver

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related