Serverless Kafka Trigger Cannot Resolve Variables/!Ref

Question

Serverless is unable to resolve varaibles in kafka event source, looking for a solution to resolve variables instead of hardcoded arns, which are dynamic and not automated.

functions:
  compute:
    handler: handler.compute
    events:
      - kafka:
          accessConfigurations:
            clientCertificateTlsAuth: arn:aws:secretsmanager:us-east-1:01234567890:secret:ClientCertificateTLS
            serverRootCaCertificate: !Ref resource # <-- turn to [Object object] in cloudformation
          topic: MySelfManagedMTLSKafkaTopic
          consumerGroupId: MyConsumerGroupId
          bootstrapServers:
            - abc3.xyz.com:9092
            - abc2.xyz.com:9092

Reza · Accepted Answer · 2025-02-21 15:24:10Z

The solution to this is to make the event-source (trigger) separated from what serverless assmbles to have full control over raw cloudformation.

serverless.yml:

  myHandle:
    handler: src/handlers/handler.default
    # events: 'resources/event-source.yml' No handler specified here!

resources/event-source.yml:

  HandleMyHandlerEventSourceMappingKafkaTopicName:
    Type: AWS::Lambda::EventSourceMapping
    DependsOn:
      - IamRoleLambdaExecution
      - HandleMyHandleProvConcLambdaAlias
    Properties:
      FunctionName:
        Fn::Join:
          - ':'
          - - Fn::GetAtt:
                - HandleMyHandleLambdaFunction
                - Arn
            - provisioned
      StartingPosition: TRIM_HORIZON
      SelfManagedEventSource:
        Endpoints:
          KafkaBootstrapServers:
            - ${env:CONFLUENT_ENDPOINT}
      Topics:
        - ${env:KAFKA_TOPIC}
      SourceAccessConfigurations:
        - Type: SERVER_ROOT_CA_CERTIFICATE
          URI: ${self:custom.customConfigs.ROOT_CA_SECRET_ARN} # can be !Ref.arn
        - Type: BASIC_AUTH
          URI: ${self:custom.customConfigs.CREDENTIALS_SECRETS_ARN} # can be !Ref.arn
        - Type: VPC_SECURITY_GROUP
          URI: !Join
            - ':'
            - - 'security_group'
              - !Ref MySecurityGroup
        - Type: VPC_SUBNET
          URI: ${cf:${self:custom.MyStackName}.paymentsSubnet1}
        - Type: VPC_SUBNET
          URI: ${cf:${self:custom.MyStackName}.paymentsSubnet2}
        - Type: VPC_SUBNET
          URI: ${cf:${self:custom.MyStackName}.paymentsSubnet3}
      BatchSize: 1
      SelfManagedKafkaEventSourceConfig:
        ConsumerGroupId: ${env:MY_CONSUMER_GROUP}

The way to have this cloudformation peace is to dig into your generated .serverless and find it there, bringing it out and customize it in your own favour, this way you also know the right names for each config in cloudformation.

Collectives™ on Stack Overflow

Serverless Kafka Trigger Cannot Resolve Variables/!Ref

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related