Testing AWS service integrations using LocalStack

In this guide, you will learn how to

Create a Spring Boot application with Spring Cloud AWS integration
Use AWS S3 and SQS services
Test the application using Testcontainers and LocalStack

Prerequisites

Java 17+
Your favorite IDE (Intellij IDEA, Eclipse, NetBeans, VS Code)
A Docker environment supported by Testcontainers https://www.testcontainers.org/supported_docker_environment/

What we are going to achieve in this guide

We are going to create a Spring Boot application and use Spring Cloud AWS to work with AWS S3 and SQS services. We are going to send a message to an SQS queue and implement a listener to consume the messages and save the payload data in an S3 bucket. We are going to test this application using Testcontainers and LocalStack.

Getting Started

Spring Cloud AWS provides a higher-level abstraction on top of AWS Java SDK V2 and tight integration with Spring Boot following the Spring programming model.

LocalStack is a fully functional local cloud emulator to develop and test your AWS cloud and serverless applications.

Let’s create a Spring Boot application from Spring Initializr by selecting the Testcontainers dependency. Currently, Spring Cloud AWS starters are not available on Spring Initializr, so we need to manually add the required dependencies.

Let’s add the spring-cloud-aws-dependencies Bill Of Material (BOM) to our dependency management and add the S3, SQS starters as dependencies. Testcontainers provides a LocalStack module which we are going to use for testing AWS service integrations. Also, we are going to use Awaitility for testing asynchronous processing with SQS.

If you are using Gradle as your build tool, you should have the dependencies configured in build.gradle file as follows:

ext {
    set('testcontainers.version', "1.19.3")
    set('awspringVersion', "3.0.3")
}

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'io.awspring.cloud:spring-cloud-aws-starter-s3'
    implementation 'io.awspring.cloud:spring-cloud-aws-starter-sqs'

    testImplementation 'org.springframework.boot:spring-boot-devtools'
    testImplementation 'org.springframework.boot:spring-boot-starter-test'
    testImplementation 'org.springframework.boot:spring-boot-testcontainers'
    testImplementation 'org.testcontainers:junit-jupiter'
    testImplementation 'org.testcontainers:localstack'
    testImplementation 'org.awaitility:awaitility'
}

dependencyManagement {
    imports {
        mavenBom "io.awspring.cloud:spring-cloud-aws-dependencies:${awspringVersion}"
    }
}

For the complete code, please refer to the GitHub repository https://github.com/testcontainers/tc-guide-testing-aws-service-integrations-using-localstack

We are going to work with an SQS queue and S3 bucket. To make the queue and bucket names configurable, let’s create a configuration properties class as follows:

package com.testcontainers.demo;

import org.springframework.boot.context.properties.ConfigurationProperties;

@ConfigurationProperties(prefix = "app")
public record ApplicationProperties(String queue, String bucket) {}

Then add @ConfigurationPropertiesScan annotation on the main Application class to automatically scan for @ConfigurationProperties annotated classes and register them as beans.

Implement StorageService to work with S3 service

Spring Cloud AWS provides higher-level abstractions like S3Client and S3Template with convenience methods for performing the most common tasks such as uploading/downloading files, generating signed URLs, etc.

Let us create a StorageService class to upload and download files using an S3Template as follows:

package com.testcontainers.demo;

import io.awspring.cloud.s3.S3Template;
import java.io.IOException;
import java.io.InputStream;
import org.springframework.stereotype.Service;

@Service
public class StorageService {

  private final S3Template s3Template;

  public StorageService(S3Template s3Template) {
    this.s3Template = s3Template;
  }

  public void upload(String bucketName, String key, InputStream stream) {
    this.s3Template.upload(bucketName, key, stream);
  }

  public InputStream download(String bucketName, String key)
    throws IOException {
    return this.s3Template.download(bucketName, key).getInputStream();
  }

  public String downloadAsString(String bucketName, String key)
    throws IOException {
    try (InputStream is = this.download(bucketName, key)) {
      return new String(is.readAllBytes());
    }
  }
}

Implement SQS message sender and listener

Spring Cloud AWS provides the higher-level abstraction SqsTemplate on top of AWS Java SDK SqsAsyncClient and @SqsListener annotation driven listener support to work with SQS queues.

Let’s create a record Message that represents the message payload we are going to send to the SQS queue as follows:

package com.testcontainers.demo;

import java.util.UUID;

public record Message(UUID uuid, String content) {}

Create MessageSender class, which internally uses SqsTemplate, to publish messages to the SQS queue as follows:

package com.testcontainers.demo;

import io.awspring.cloud.sqs.operations.SqsTemplate;
import org.springframework.stereotype.Service;

@Service
public class MessageSender {

  private final SqsTemplate sqsTemplate;

  public MessageSender(SqsTemplate sqsTemplate) {
    this.sqsTemplate = sqsTemplate;
  }

  public void publish(String queueName, Message message) {
    sqsTemplate.send(to -> to.queue(queueName).payload(message));
  }
}

Create MessageListener class and implement a SQS queue message handler method annotated with @SqsListener specifying which queue(s) to listen to.

package com.testcontainers.demo;

import io.awspring.cloud.sqs.annotation.SqsListener;
import java.io.ByteArrayInputStream;
import java.nio.charset.StandardCharsets;
import org.springframework.stereotype.Service;

@Service
public class MessageListener {

  private final StorageService storageService;
  private final ApplicationProperties properties;

  public MessageListener(
    StorageService storageService,
    ApplicationProperties properties
  ) {
    this.storageService = storageService;
    this.properties = properties;
  }

  @SqsListener(queueNames = { "${app.queue}" })
  public void handle(Message message) {
    String bucketName = this.properties.bucket();
    String key = message.uuid().toString();
    ByteArrayInputStream is = new ByteArrayInputStream(
      message.content().getBytes(StandardCharsets.UTF_8)
    );
    this.storageService.upload(bucketName, key, is);
  }
}

Note that we have used the Spring’s property reference expression ${app.queue} to configure the queue name using the configured property value instead of hard-coding the queue name.

When a message is published to the SQS queue, Spring handles unmarshalling the payload data into a Message object, and we are uploading the message content into an S3 bucket with the message’s unique key.

Write integration test using LocalStack

We can spin up a container using the LocalStack Docker image and configure the following Spring Cloud AWS properties to talk to the LocalStack container instead of the actual AWS services.

spring.cloud.aws.s3.endpoint=http://localhost:4566
spring.cloud.aws.sqs.endpoint=http://localhost:4566
spring.cloud.aws.credentials.access-key=noop
spring.cloud.aws.credentials.secret-key=noop
spring.cloud.aws.region.static=us-east-1

However, for testing, it would be better to use an ephemeral container that starts on a random available port so that we can run multiple builds on CI in parallel without having port conflict issues. Testcontainers provides a LocalStack module that helps us to achieve this easily.

We can create a SpringBoot integration test that will start a LocalStack container and configure the Spring Cloud AWS properties as follows:

@SpringBootTest
@Testcontainers
class MessageListenerTest {

  @Container
  static LocalStackContainer localStack = new LocalStackContainer(
    DockerImageName.parse("localstack/localstack:3.0")
  );

  static final String BUCKET_NAME = UUID.randomUUID().toString();
  static final String QUEUE_NAME = UUID.randomUUID().toString();

  @DynamicPropertySource
  static void overrideProperties(DynamicPropertyRegistry registry) {
    registry.add("app.bucket", () -> BUCKET_NAME);
    registry.add("app.queue", () -> QUEUE_NAME);
    registry.add(
      "spring.cloud.aws.region.static",
      () -> localStack.getRegion()
    );
    registry.add(
      "spring.cloud.aws.credentials.access-key",
      () -> localStack.getAccessKey()
    );
    registry.add(
      "spring.cloud.aws.credentials.secret-key",
      () -> localStack.getSecretKey()
    );
    registry.add(
      "spring.cloud.aws.s3.endpoint",
      () -> localStack.getEndpointOverride(S3).toString()
    );
    registry.add(
      "spring.cloud.aws.sqs.endpoint",
      () -> localStack.getEndpointOverride(SQS).toString()
    );
  }
}

We have used Testcontainers JUnit 5 Extension annotations @Testcontainers and @Container to spin up a LocalStackContainer instance. Then we obtain the dynamic S3 and SQS endpoint URLs, region, access and secret keys from the container instance and configured Spring Cloud AWS configuration properties using DynamicPropertyRegistry mechanism.

Before writing our test, we need to create a SQS queue and S3 bucket. We can use @BeforeAll callback method to create the necessary resources (S3 buckets or SQS queues etc) by using localStack.execInContainer() API as follows:

@BeforeAll
static void beforeAll() throws IOException, InterruptedException {
  localStack.execInContainer("awslocal", "s3", "mb", "s3://" + BUCKET_NAME);
  localStack.execInContainer(
    "awslocal",
    "sqs",
    "create-queue",
    "--queue-name",
    QUEUE_NAME
  );
}

We have used localStack.execInContainer() API to run commands inside the container and used awslocal CLI tool, that comes pre-installed with LocalStack Docker image, to create the AWS resources.

Finally, we can write our test by publishing a message to the SQS queue which should be handled by the listener and persist the message content in the S3 bucket. We will use Awaitility to wait a maximum of 10 seconds to verify whether the expected message is present in the S3 bucket with the key or not.

The complete test would look like this:

package com.testcontainers.demo;

import static org.assertj.core.api.Assertions.assertThat;
import static org.awaitility.Awaitility.await;
import static org.testcontainers.containers.localstack.LocalStackContainer.Service.S3;
import static org.testcontainers.containers.localstack.LocalStackContainer.Service.SQS;

import java.io.IOException;
import java.time.Duration;
import java.util.UUID;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.DynamicPropertyRegistry;
import org.springframework.test.context.DynamicPropertySource;
import org.testcontainers.containers.localstack.LocalStackContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;

@SpringBootTest
@Testcontainers
class MessageListenerTest {

  @Container
  static LocalStackContainer localStack = new LocalStackContainer(
    DockerImageName.parse("localstack/localstack:3.0")
  );

  static final String BUCKET_NAME = UUID.randomUUID().toString();
  static final String QUEUE_NAME = UUID.randomUUID().toString();

  @DynamicPropertySource
  static void overrideProperties(DynamicPropertyRegistry registry) {
    registry.add("app.bucket", () -> BUCKET_NAME);
    registry.add("app.queue", () -> QUEUE_NAME);
    registry.add(
      "spring.cloud.aws.region.static",
      () -> localStack.getRegion()
    );
    registry.add(
      "spring.cloud.aws.credentials.access-key",
      () -> localStack.getAccessKey()
    );
    registry.add(
      "spring.cloud.aws.credentials.secret-key",
      () -> localStack.getSecretKey()
    );
    registry.add(
      "spring.cloud.aws.s3.endpoint",
      () -> localStack.getEndpointOverride(S3).toString()
    );
    registry.add(
      "spring.cloud.aws.sqs.endpoint",
      () -> localStack.getEndpointOverride(SQS).toString()
    );
  }

  @BeforeAll
  static void beforeAll() throws IOException, InterruptedException {
    localStack.execInContainer("awslocal", "s3", "mb", "s3://" + BUCKET_NAME);
    localStack.execInContainer(
      "awslocal",
      "sqs",
      "create-queue",
      "--queue-name",
      QUEUE_NAME
    );
  }

  @Autowired
  StorageService storageService;

  @Autowired
  MessageSender publisher;

  @Autowired
  ApplicationProperties properties;

  @Test
  void shouldHandleMessageSuccessfully() {
    Message message = new Message(UUID.randomUUID(), "Hello World");
    publisher.publish(properties.queue(), message);

    await()
      .pollInterval(Duration.ofSeconds(2))
      .atMost(Duration.ofSeconds(10))
      .ignoreExceptions()
      .untilAsserted(() -> {
        String msg = storageService.downloadAsString(
          properties.bucket(),
          message.uuid().toString()
        );
        assertThat(msg).isEqualTo("Hello World");
      });
  }
}

We have created a Message instance with a random UUID as a unique identifier and published the message to the SQS Queue. When the listener receives the message, the message content will be stored in S3 bucket with the random UUID as key. Then we are verifying whether there is an entry in the bucket with the expected key and the content is equal to what we sent or not.

Run tests

# If you are using Maven
./mvnw test

# If you are using Gradle
./gradlew test

You should see the LocalStack Docker container is started and the test should PASS. You can also notice that after the tests are executed, the containers are stopped and removed automatically.

Summary

LocalStack enables developing AWS services based applications locally and Testcontainers LocalStack module makes it simpler to write integration tests by using LocalStack ephemeral containers.

To learn more about Testcontainers visit http://testcontainers.com