Search for a command to run...
Docker containers for testing your Scala applications
Travel | Movies | History | Nature
I am a software developer. Started my career as a Java developer, but later switched to Scala. I wish to write Scala articles which are easier for newbies to follow. I love to travel, prefer to be alone or in very small group. Love to watch movies/series.
No comments yet. Be the first to comment.
Implementing Continuous Data Flows with FS2 Streaming
Building Resumable Data Pipelines with Redis-backed Offset Tracking and Batch Processing
Creating Flexible Data Integration Patterns for Cross-System Data Movement
How to leverage Scala 3 Mirrors to generically parse CSV files into case classes, making data handling more flexible and type-safe.
Introduction Recently, at work, there was a requirement to list out the direct dependencies of the our SBT project, something like Bill Of Materials. SBT itself provides the dependencies as list or tree or searchable HTML page using the built-in comm...
Integration Testing is a very important part of software development. It is generally better to have an integration testing environment as close to the production setup as to catch bugs early.
However, it is also cumbersome to set up all the required services and databases each time in all the required environments. In this short blog, let's look at TestContainers-Scala which makes integration testing very easy and realistic.
There are many different ways in which different companies and teams handle integration testing scenarios.
The most common and easy way is to have a shared environment for the integration testing in the CI server. This way, a shared database, and services are used while the tests run in the CI environment.
However, the problem with this approach is that since it is a shared setup, multiple CI jobs can interfere and cause flaky behavior for the tests.
To avoid problems with a shared environment, we can use embedded/in-memory modes of services such as databases, Kafka, etc. For example, there are in-memory instances for MongoDB, Kafka, PostgreSQL, and so on. However, some of the problems with this approach are:
There should be an available in-memory version of the databases/services we need
Not all services/databases could be made in-memory
There is a chance that some of these become unmaintained later
Not all features of a database/service might be available in the in-memory version
For the relational database, another option is to use the H2 database with the relevant compatibility mode. But, not all features of a database might be available in H2.
Another possibility is to use dockerized environment for all the services for a single run per CI job. This avoids all the previous issues. However, we still need to manually handle the creation and destruction of docker containers.
TestContainers is a java library that helps to create lightweight docker instances for integration testing. It also has good integration with testing libraries which makes writing integration tests much easier.
TestContainer-Scala is a scala wrapper over TestContainer-Java. It also has integration with ScalaTest and MUnit testing frameworks. It provides ways to create docker containers per test case and per test suite. It already has support for almost all popular services and databases. Apart from that, we can also create and use custom modules for the services which are not yet supported by TestContainer.
This solves all the previously mentioned issues with other modes of integration testing. In the next section, let's look at some of its features.
Let's first set up the required library dependencies in build.sbt:
libraryDependencies ++= Seq(
"com.dimafeng" %% "testcontainers-scala-scalatest" % "0.40.16" % "test",
"com.dimafeng" %% "testcontainers-scala-postgresql" % "0.40.16" % "test"
"org.postgresql" % "postgresql" % "42.6.0",
"org.scalatest" %% "scalatest-flatspec" % "3.2.16" % Test,
"org.tpolecat" %% "skunk-core" % "0.6.0"
)
This will add the test-container dependencies for Scalatest and PostgreSQL docker container. Additionally, we also have added the relevant libraries and drivers for the sample application to work.
Next, let's add the setting in build.sbt to use forked JVM for tests. This ensures a cleaner shutdown of the test setup:
Test / fork := true
Now, let's create a DAO file for writing database querying logic. In this case, I am using Skunk to connect to PostgreSQL database. I am not showing the DAO code here for the sake of brevity. You can find the code here on GitHub.
Now, let's start writing the integration test class. As mentioned before, I am using ScalaTest in this case. We can extend our test with the trait ForEachTestContainer provided by testcontainer-scala. This forces us to override a member container with the required docker instance:
class TestContainerPGSpec extends AnyFlatSpec with ForEachTestContainer {
override val container: PostgreSQLContainer = new PostgreSQLContainer()
}
The trait ForEachTestContainer instantiates a new docker container for every test case and destroys it after the test is completed. This is very useful in ensuring that the tests are not interfering with each other.
Make sure that the overriden member container is a val and NOT a def.
We can also use another trait ForAllTestContainer instead, which creates a docker instance per test suite.
Now, let's write a simple test to check if we are able to connect to this dockerized postgres container:
it should "get current date from postgresql database" in {
val dao = new PostgresDAO(getConfig)
val dateIO = dao.getCurrentDate
assert(dateIO.unsafeRunSync() == LocalDate.now)
}
If everything is fine, this test passes successfully.
In the above example, we created the postgres container with default options. Sometimes, we might need to create a more configurable container. We can do that by using the configure method:
override val container: PostgreSQLContainer =
PostgreSQLContainer().configure { c =>
c.withInitScript("init_scripts.sql")
c.withDatabaseName("test-database")
}
This creates a dockerized database with the databasse name as test-database and executes the initialisation scripts provided in the file init_scripts.sql that is placed in the resources directory.
With this configuration, let's write additional test. The init script inserts 2 records for movies to the table and we can verify if it exists:
it should "get rows from movie table" in {
val dao = new PostgresDAO(getConfig)
val rows = dao.getMovies.unsafeRunSync()
assert(rows.size == 2)
assert(rows.map(_.name) == List("Shawshank Redemptions", "The Prestige"))
assert(rows.map(_.id).forall(_ > 0))
}
TestContainer-Scala also provides before and after hooks. With this, we can execute some code before or after the container start and stop:
override def beforeStop(): Unit = {
println("Container is about to be stopped")
}
override def afterStart(): Unit = {
println("Container is started")
}
TestContainer-Scala also supports the usage of multiple containers within tests. This helps us to test integration with different services, all dockerized without any impact on other tests.
To use that, we need to override the container member with MultipleContainer instance with the required containers. In this case, let's use postgres and mongoDB containers. Apart from the existing libraries, firstly let's add the mongoDB driver and container in our build.sbt:
libraryDependencies ++= Seq(
"org.reactivemongo" %% "reactivemongo" % "1.0.10",
"com.dimafeng" %% "testcontainers-scala-mongodb" % "0.40.11" % "test"
)
Now, let's instantiate the required containers:
val mongoDBContainer: MongoDBContainer = new MongoDBContainer()
val postgresContainer = PostgreSQLContainer().configure { c =>
c.withInitScript("init_scripts.sql")
}
In the next step, we can create MultipleContainers and inject both the containers into it:
override val container = MultipleContainers(postgresContainer, mongoDBContainer)
Now, we can write the test where movies are fetched from postgreSQL database and inserted into mongoDB:
it should "get movies from postgres and insert into mongo" in {
val config =
MongoConfig(mongoDBContainer.container.getConnectionString(), "movies")
val mongoDao = new MongoDao(config)
val postgreDao = new PostgresDAO(getPostgresConfig)
import cats.effect.unsafe.implicits.global
for {
pgMovies <- postgreDao.getMovies.unsafeToFuture()
_ <- mongoDao.saveMovies(pgMovies)
all <- mongoDao.getMovies()
} yield {
assert(all.size == 2)
assert(all.map(_.name) == Seq("Shawshank Redemptions", "The Prestige"))
}
}
That's it, this uses both postgres and mongo containers to run the test. The full test class code is available here on GitHub.
If we need to use a container that is not supported by TestContainer-Scala, we can use GenericContainer. It allows to create custom containers with any configurations required for the application. So far, I didn't have to follow this approach, but more details are available here.
We can also use DockerCompose container to build a custom container for our test.
I hope that this blog helps you to get started with TestContainers-Scala and use it in your application to have better testing. Even though this brings an additional delay to the integration tests (to create and drop new containers), it is very useful in avoiding flaky test behaviours.
Please leave a reaction for this article, if you find it useful.
All the sample code and test cases are available here on GitHub.