AWS Lambda, Secrets Manager, VPC Timeout

Yesterday I needed to deploy a Lambda function to send out email reminders for my client. In setting this up I need to add it to a VPC so it could connect to the database which was straight forward. On our platform we use Secret Manager to store our connection strings for our connections to our databases. It all sounds great, but the function kept timing out and I wasn’t getting any information about the why.

Like any respected engineer (or desperate) I started to add console.logs every where. I figured that there was a connection issue to the database since it is not accessible to the public. After some annoying cycles of add log, redeploy, test rinse and repeat it turned out that it was hanging trying to connect to Secret Manager.

This didn’t make much sense at first, there were no security groups that were restricting outbound traffic. I checked the credentials that it was using and those were fine. Then I turned to the engineers best friend, Google. After some digging, I found that Lambda needs internet access to get to Secret Manager and I am not willing to give it access. So what were my options?

I remembered that I used an endpoint so I don’t have to route my database traffic over the internet. Endpoints allow you to access available services through the AWS internal networks and not over the internet. Not all services have endpoints that you can create, but it turns out that Secret Manager is available. It is incredibly simple to setup. Here are the steps.

You have to navigate to the VPC section of AWS, and select Endpoints item in the left nav menu.

Now you can click Create Endpoint and you can look through the list of services that are available. It should look something like this.

You want to select what is the category of services that you want to use, in my case it was the default AWS Services. Search for Secret Manager and it will pop into the list. Once you make the selection, your only task left is to select the VPC you want to attach it to.

That’s all it takes. After a short period (~45 seconds) your endpoint will be attached and all is well again.