The first thing we need to think about is what our render array depends on. Are we rendering some entity data? Are we using some configuration values? Or anything that might be changed elsewhere impacting what we have to render? If the answer is yes, we need to use cache tags. If we don't use them, our render array gets cached as it is, and if the underlying data changes, we end up showing our users stale content or data.
To look at this another way, imagine a simple Article node. This content can be shown on its main detail page, in a listing of article teasers or even a listing of article titles (and many other places potentially). And since there is no way of knowing where it will be used, it is the responsibility of the render array that displays this content to mark this node entity as a dependency using cache tags. This way, when the node gets updated, all the render arrays that depend on it get invalidated as well.
Cache tags are simple strings and we can declare many cache tags for a single render array. They do have a special form in the following pattern: thing:identifier, or in some cases, just simply thing (if there is only one single element of that "thing"). For example, the cache tag for a given node would be in the format node:1, where the identifier is the actual node ID. Or for a configuration object it would be config:hello_world.custom_salutation.
I hinted before how, for example, some node content can be present in a list and therefore using the cache tags we can ensure that the render array for that node gets updated when the node does. Since render arrays are highly granular, this can present a small extra problem as the list itself can be a render array that may not even know which nodes it renders. Or even more so, it does not know when new nodes are created and should be included in it. To solve this issue, we have a special list cache tag we can use when rendering entities. For example, the node_list cache tag can be used for node entities, while the product_list cache tag can be used for product entities. These are automatically understood by the Drupal caching system, so all we have to do is use them appropriately.
To make life easier, however, all entities and configuration objects can be "interrogated" to provide their respective cache tags. For example:
$tags = $node->getCacheTags();
Where $tags will be an array containing one tag—node:[nid].
The same applies to configuration objects and this is handy because it prevents typos and errors. This is due to the generic CacheableDependencyInterface they implement which defines the methods for retrieving the cache metadata properties. In fact, any value that needs to be a cache dependency can and should implement this interface. As you'll find, there are a quite a few classes in Drupal core that do so.
We can also figure out the "list" cache tag specific to a given entity type. For example, instead of hardcoding the product_list tag, we can use the getListCacheTags() method on the EntityTypeInterface.
If your render array depends on something custom, you can use custom cache tags, but it will be your responsibility to also invalidate them when the underlying data is changed. We will see how this is done when we interact with the Cache API directly. It's always good to consistently use the CacheableDependencyInterface for any custom value objects.