Inconsistent cache values using Zend Cache with AWS ElastiCache across multiple servers

Question

We are using Zend Cache with a memcached backend pointing to an AWS ElastiCache cluster with 2 cache nodes. Our cache setup looks like this:

$frontend = array(
    'lifetime' => (60*60*48),
    'automatic_serialization' => true,
    'cache_id_prefix' => $prefix
);
$backend = array(
    'servers' => array(
        array( 'host' => $node1 ),
        array( 'host' => $node2 )
    )
);
$cache = Zend_Cache::factory('Output', 'memecached', $frontend, $backend);

We have not noticed any problems with the cache in the past when using a single EC2 server to write and read from the cache.

However, we have recently introduced a second EC2 server and suddenly we're seeing issues when writing to the cache from one server and reading from another. Both servers are managed by the same AWS account, and neither server has issues writing to or reading from the cache individually. The same cache configuration is used for both.

Server A executes $cache->save('hello', 'message');

Subsequent calls to $cache->load('message'); from Server A return the expected result of hello.

However, when Server B executes $cache->load('message');, we get false.

As far as my understanding of ElastiCache goes, the server making the read request should have no bearing on the cache value returned. Can anyone shed some light on this?

I would assume this is a latency issue, have you tried to sleep(xxxx) and then perform the $cache->load from B? — Julius F
– Julius F, Commented Sep 17, 2012 at 22:33
Unfortunately, this is not the case. Even hours later a value set from A is not readable from B. — michaelxor
– michaelxor, Commented Sep 17, 2012 at 22:49
Which version of PHP are you using? I think the serialization is what is in play here. Try disabling the auto serialization and see what happens. The unfortunate side effect is that you have to serialize everything manually that is not a string. — Rich H.
– Rich H., Commented Oct 16, 2012 at 21:06
Additionally is the cache_id_prefix the same or a generated value? — Rich H.
– Rich H., Commented Oct 16, 2012 at 21:08
I would try writing some test code using PHP's Memcache, removing Zend_Cache from the equation. I never much liked the Memcache extension and have always favored Memcached (but don't get me started on why the hell they added a 'd' to the client). For one it supports compare-and-swap operations. — ficuscr
– ficuscr, Commented Oct 17, 2012 at 18:35

Jason · Accepted Answer · 2013-01-02 11:09:20Z

1

Can you tell what hash_strategy you are using for memcache? I've had problems in the past using the default standard but everything has been fine since changing to consistent:

http://php.net/manual/en/memcache.ini.php#ini.memcache.hash-strategy

answered Jan 2, 2013 at 11:09

Jason

5073 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2020-06-20 09:12:55Z

With a normal hashing algorithm, changing the number of servers can cause many keys to be remapped to different servers resulting in huge sets of cache misses.

Imagine you have 5 ElastiCache Nodes in your cache Cluster, adding an sixth server may cause 40%+ of your keys to suddenly point to different servers than normal. This activity is undesirable, may cause cache misses and eventually swamping your backend DB with requests. To minimize this remapping it is recommended to follow consistent Hashing model in your cache clients.

Consistent Hashing is a model that allows for more stable distribution of keys given addition or removal of servers. Consistent Hashing describes methods for mapping keys to a list of servers, where adding or removing servers causes a very minimal shift in where keys map to. Using this approach, adding an eleventh server should cause less than 10% of your keys to be reassigned. This % may vary in production but it is far more efficient in such elastic scenarios compared to normal hash algorithms. It is also advised to keep memcached server ordering and number of servers same in all the client configurations while using consistent Hashing. Java Applications can use “Ketama library” through spymemcached to integrate this algorithm into their systems. More information on consistent hashing can be found at http://www.last.fm/user/RJ/journal/2007/04/10/rz_libketama_-_a_consistent_hashing_algo_for_memcache_clients

Collectives™ on Stack Overflow

Inconsistent cache values using Zend Cache with AWS ElastiCache across multiple servers

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related