In our daily life, we have loads of secrets. Secrets are information that is not meant to be publicly known, such as the username and password combination you use to access your online bank account, or the code to your cell phone or your locker at the gym.
When writing software, we often need to use secrets, too. For example, we need some certificate to authenticate our application service with some external service we want to access, or we need a token to authenticate and authorize our service when accessing some other API. In the past, developers, for convenience, have just hardcoded those values or put them in clear text in some external configuration files. There, this very sensitive information has been accessible to a broad audience that in reality should never have had the opportunity to see those secrets.
Luckily, these days, orchestrators offer what's called secrets to deal with such sensitive information in a highly secure way. Secrets can be created by authorized or trusted personnel. The values of those secrets are then encrypted and stored in the highly available cluster state database. The secrets, since they are encrypted, are now secure at rest. Once a secret is requested by an authorized application service, the secret is only forwarded to the cluster nodes that actually run an instance of that particular service, and the secret value is never stored on the node but mounted into the container in a tmpfs RAM-based volume. Only inside the respective container is the secret value available in clear text.
We already mentioned that the secrets are secure at rest. Once they are requested by a service, the cluster manager or master decrypts the secret and sends it over the wire to the target nodes. So, what about the secrets being secure in transit? Well, we learned earlier that the cluster nodes use MTLS for their communication, thus the secret, although transmitted in clear text, is still secure since data packets will be encrypted by MTLS. Thus, secrets are secure at rest and in transit. Only services that are authorized to use secrets will ever have access to those secret values.