Search for a command to run...
A Clear and Safe Configuration Library
Travel | Movies | History | Nature
I am a software developer. Started my career as a Java developer, but later switched to Scala. I wish to write Scala articles which are easier for newbies to follow. I love to travel, prefer to be alone or in very small group. Love to watch movies/series.
No comments yet. Be the first to comment.
Implementing Continuous Data Flows with FS2 Streaming
Building Resumable Data Pipelines with Redis-backed Offset Tracking and Batch Processing
Creating Flexible Data Integration Patterns for Cross-System Data Movement
How to leverage Scala 3 Mirrors to generically parse CSV files into case classes, making data handling more flexible and type-safe.
Introduction Recently, at work, there was a requirement to list out the direct dependencies of the our SBT project, something like Bill Of Materials. SBT itself provides the dependencies as list or tree or searchable HTML page using the built-in comm...
Configurations are essential parts of any software application. In the previous blog, we looked at PureConfig for handling configurations files in Scala. In this blog, we will look at another library which provides even more safety.
Even though PureConfig adds type-safety and loads the configurations without boiler-plate code, there still exists some risks. Let's look at different possible scenarios where PureConfig and other such libraries fail.
It is a general practice to have different configuration files for each environments like prod, test and local. After successfully testing in the test env and while deploying to prod env, we might find out that the app is not starting due to a config mistake in the prod environment config file. It is very difficult to verify each of these cases before every deployment.
Sometimes we may have unused configurations in the config files. It is a pain to search and find out the unused configs. Sometimes we might make some small spelling mistakes in one of the config file and might not even detect it until too late!
Sometime, we might need to override the config values. This is very useful in creating a hierarchy of config environments and use from the relevant levels based on the priority. For example, we can have a config for a path for a directory. It will be nice to dynamically change this value while testing. We can do this by passing java classpath arguments or by using environment variables in the target machine. However, it is a challenge to visualise and understand which config value is used for a particular scenario.
Clear-Config is a small and nice library which helps to solve all the above pain points.
Some of the advantages of using Clear-Config are:
Unfortunately, Clear-Config doesn't support HOCON style config format. So, each of the config needs to be provided with appropriate prefixes for clear separation. For instance, if we have separate config for postgres and mongo database, we can provide the config as:
postgres.dbName = "pgDB"
postgres.username = "pgUser"
mongo.dbName = "mongoDB"
mongo.username = "mongoUser"
To use Clear-Config, add the sbt dependency:
libraryDependencies += "com.github.japgolly.clearconfig" %% "core" % "3.0.0"
Note that, Clear-Config has dropped the support for Scala 2.12. As of now, it supports only Scala 2.13 and Scala 3.
In this section, let's go through different usages of Clear-Config.
Now, let's see how we can use Clear-Config. For the sample code, we can first create the necessary config file. Let's put the below content in application.conf file:
host = "postgresql://localhost:5432"
dbName = "configs"
username = "admin"
password = "pwd!#+"
maxConnection = 5
As a next step, let's create a case class corresponding to the config properties:
final case class DatabaseConfig(
host: String,
dbName: String,
username: String,
password: String,
maxConnection: Option[Int]
)
Now, we need to add the required imports for using this library. Clear-Config uses the library cats under the hood:
import japgolly.clearconfig.*
import cats.implicits.*
Next, we need to create a sort of mapping between the config properties and the case class fields. Clear-Config provides different methods like need, get and getOrUse, to handle mandatory, optional and default config properties respectively.
Let's create this mapping in the companion object of the case class. For simplicity, we will be using the effect type Id here. We need to add the necessary imports also for it to work:
import cats.Id
import cats.catsInstancesForId
object DatabaseConfig {
def config: ConfigDef[DatabaseConfig] = (
ConfigDef.need[String]("host"),
ConfigDef.need[String]("dbName"),
ConfigDef.need[String]("username"),
ConfigDef.need[String]("password"),
ConfigDef.get[Int]("maxConnection")
).mapN(apply)
}
In the next step, we need to define the config source. This is the place where we map the config file name to the config classes:
def configSources: ConfigSources[Id] = ConfigSource
.propFileOnClasspath[Id]("/application.conf", optional = false)
Now, we are ready to read the config file:
val dbCfg: DatabaseConfig = DatabaseConfig.config.run(configSources).getOrDie()
If there is some mistake with the config file, the application will fail to start and provide the reason.
In the previous section, we have only single source of config. Now, let's see how we can use multiple sources. Let's assume that we have we have config file for each environment, which adds the string ${env} based on each environment. For example, let's add another file application-prod.conf. We expect the application-prod.conf file has more priority in the production env and if any config is is provided in that file, it should be considered instead of from the the application.conf. For doing this, we need to define 2 sources and combine them together using > :
def configSources: ConfigSources[Id] =
ConfigSource.propFile[Id](sysFilePath, optional = true) >
ConfigSource
.propFileOnClasspath[Id]("application-prod.conf", optional = true) >
ConfigSource
.propFileOnClasspath[Id]("/application.conf", optional = false)
Now, if it finds any properties in application-prod file, it will be used. If some properties are not available, then it will use from application.conf. Also note that we have mentioned that the application-prod.conf file as optional.
Along with the files, we can also use environment variables to handle config. For instance, assume that some property is set in the environment of the target system, we can use them in the same way as the other sources:
def configSources: ConfigSources[Id] = ConfigSource.environment[Id] >
ConfigSource.propFileOnClasspath[Id]("application-prod.conf", optional = true) >
ConfigSource.propFileOnClasspath[Id]("/application.conf", optional = false)
Now, the first priority will be for the environment variable, then application-prod.conf and at last application.conf
We can also load config file from outside the classpath/resources directory. For this, instead of using propFileOnClassPath() method, we can use propFile() with absolute path to the file:
ConfigSource.propFile[Id](sysFilePath, optional = true)
Similar to environment variable, we can also read the property from javac flags we pass while the app is starting. For that, we can use the method ConfigSource.system[Id]. For example, we can pass the value for dbName as jvm arg as:
sbt -DdbName=jvmDB "project runMain <path_to_class>/ConfigLoader"
One of the most powerful feature of Clear-Config is the report it generates regarding the config properties. With this table structured report, we can identify which config is used from which file. We can generate the report by using the method .withReport(). This will return a tuple of config class and ConfigReport instance. We can invoke thee method full() on ConfigReport to generate a tabular report as shown below.
val (dbCfg, report): (DatabaseConfig, ConfigReport) =
DatabaseConfig.config
.withReport
.run(configSources)
.getOrDie()
It displays which config is used from which source, and the left one has the highest priority. It also provides a list of unused configurations.
If any of the configuration in the provided source is wrong, then the application will fail to start. That means, even if production config file has a mistake, running in test or local environment will cause the app to crash. This way we will not miss out the config mistakes from other sources.
There are many more features available in Clear-Config. Here are some of the ones which I am particularly interested in.
To read the configs without worrying about the case, we can use the method caseInsensitive while defining the sources.
ConfigSource.environment[Id].caseInsensitive
This will read the config properties ignoring the case.
As mentioned, one of the disadvantages of ClearConfig is that HOCON style config is not supported. So, to read the separated configs, we can use the withPrefix method. For accessing mongo properties:
DatabaseConfig.config.withPrefix("mongo.").run(configSources)
For using postgres:
DatabaseConfig.config.withPrefix("postgres.").run(configSources)
In this short article, we looked at Clear-Config and how to get started with it. There are still more powerful features which are available in Clear-Config. These can be explored from its GitHub repository. The code samples used in this article is available over on GitHub.