Why is my lambda not able to talk to elasticache?

Question

I have an Redis ElastiCache cluster that has a FQDN for the primary node in the format: master.clustername.x.euw1.cache.amazonaws.com. I also have a Route53 record with the CNAME pointing at that FQDN.

I have a .net core lambda in the same VPC as the cluster, with access to the cluster via security groups. The lambda talks to the cluster using the Redis library developed by Stack Overflow (Github repo here for reference).

If I give the lambda the hostname the FQDN for the Redis cluster (the one that starts with master) I can connect, save data and read it.

If I give the lambda the CNAME (and that CNAME gives the same IP address as the FQDN when I ping it from my local machine and also if I use Dns.GetHostEntry within the lambda) it doesn't connect and I get the following error message:

One or more errors occurred. (It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail. SocketFailure on PING): AggregateException
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at lambda_method(Closure , Stream , Stream , LambdaContextInternal )

at StackExchange.Redis.ConnectionMultiplexer.ConnectImpl(Func`1 multiplexerFactory, TextWriter log) in c:\code\StackExchange.Redis\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:line 890
at lambda.Redis.RedisClientBuilder.Build(String redisHost, String redisPort, Int32 redisDbId) in C:\BuildAgent\work\91d24911506461d0\src\Lambda\Redis\RedisClientBuilder.cs:line 9
at lambda.Ioc.ServiceBuilder.GetRedisClient() in C:\BuildAgent\work\91d24911506461d0\src\Lambda\IoC\ServiceBuilder.cs:line 18
at lambda.Ioc.ServiceBuilder.GetServices() in C:\BuildAgent\work\91d24911506461d0\src\Lambda\IoC\ServiceBuilder.cs:line 11
at Handlers.OrderHandler.Run(SNSEvent request, ILambdaContext context) in C:\BuildAgent\work\91d24911506461d0\src\Lambda\Handlers\OrderHandler.cs:line 26

Has anyone seen anything similar to this?

Can you add the inbound rules for all the security groups applied to both the lambda function and cache cluster — Josnidhin
– Josnidhin, Commented Aug 31, 2018 at 3:02
It's not a security group thing as if I remove the lambda sg from the cluster sg the lambda times out whereas if it's there I get the above error message if I use the route53 cname or I can connect using the fqdn — Piers Karsenbarg
– Piers Karsenbarg, Commented Aug 31, 2018 at 3:34
I misread that section in your question. Have you tired what is told in exception i.e setting abortConnect=false — Josnidhin
– Josnidhin, Commented Aug 31, 2018 at 4:10
I don't want a disconnected client. The only difference is the host I'm giving it (and they both resolve to the same IP) and I've tried the "resolveDns" property — Piers Karsenbarg
– Piers Karsenbarg, Commented Aug 31, 2018 at 17:21

Piers Karsenbarg · Accepted Answer · 2019-01-24 14:56:37Z

1

It turned out that because I was using an SSL certificate on the elasticache cluster and the SSL certificate was bound the the master. endpoint whereas I was trying to connect to the CNAME, the certificate validation was failing.

So I ended up querying the Route53 record within the code to get the master endpoint and it worked.

answered Jan 24, 2019 at 14:56

Piers Karsenbarg

3,2215 gold badges37 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Fabio Manzano · Accepted Answer · 2018-09-04 14:45:17Z

Possible workaround to isolate problems from your client lib -- follow AWS' tutorial and rewrite your Lambda to something like the code below (example in Python).

from __future__ import print_function
import time
import uuid
import sys
import socket
import elasticache_auto_discovery
from pymemcache.client.hash import HashClient

#elasticache settings
elasticache_config_endpoint = "your-elasticache-cluster-endpoint:port"
nodes = elasticache_auto_discovery.discover(elasticache_config_endpoint)
nodes = map(lambda x: (x[1], int(x[2])), nodes)
memcache_client = HashClient(nodes)

def handler(event, context):
    """
    This function puts into memcache and get from it.
    Memcache is hosted using elasticache
    """

    #Create a random UUID... this will the sample element we add to the cache.
    uuid_inserted = uuid.uuid4().hex
    #Put the UUID to the cache.
    memcache_client.set('uuid', uuid_inserted)
    #Get item (UUID) from the cache.
    uuid_obtained = memcache_client.get('uuid')
    if uuid_obtained.decode("utf-8") == uuid_inserted:
        # this print should go to the CloudWatch Logs and Lambda console.
        print ("Success: Fetched value %s from memcache" %(uuid_inserted))
    else:
        raise Exception("Value is not the same as we put :(. Expected %s got %s" %(uuid_inserted, uuid_obtained))

    return "Fetched value from memcache: " + uuid_obtained.decode("utf-8")

Reference: https://docs.aws.amazon.com/lambda/latest/dg/vpc-ec-deployment-pkg.html

Collectives™ on Stack Overflow

Why is my lambda not able to talk to elasticache?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related