NLP is the field of using computer science and AI to process and analyze natural language data and then make machines able to interpret it as humans do. During the 1980s, when this concept started to get hyped, language processing systems were designed by hand coding a set of rules. Later, following increases in calculation power, a different approach, mostly based on statistical models, replaced the original one. A later ML approach (supervised learning first, also semi-supervised or unsupervised at present time) brought advances in this field, such as voice recognition software and human language translation, and will probably lead to more complex scenarios, such as natural language understanding and generation.
Here is how NLP works. The first task, called the speech-to-text process, is to understand the natural language received. A built-in model performs speech recognition, which does the conversion from natural to programming language. This happens by breaking down speech into very small units and then comparing them to previous units coming speech that has been input previously. The output determines the words and sentences that most probably have been said. The next task, called part-of-speech (POS) tagging (or word-category disambiguation in some literature), identifies words as their grammatical forms (nouns, adjectives, verbs, and so on) using a set of lexicon rules. At the end of these two phases, a machine should understand the meaning of the input speech. A possible third task of an NLP process is text-to-speech conversion: at the end, the programming language is converted into a textual or audible format understandable by humans. That's the ultimate goal of NLP: to build software that analyzes, understands, and can generate human languages in a natural way, making computers communicate as if they were humans.
Given a piece of text, there are three things that need to be considered and understood when implementing NLP:
- Semantic information: The specific meaning of a single word. Consider, for example, the word pole, which could have different meanings (one end of a magnet, a long stick, and others). In a sentence like extreme right and extreme left are the two poles of the political system, in order to understand the correct meaning, it is important to know the relevant definition of pole. A reader would easily infer which one it is, but a machine can't without ML or DL.
- Syntax information: The phrase structure. Consider the sentence William joined the football team already with long international experience. Depending on how it is read, it has different meanings (it could be William or the football team that has long international experience).
- Context information: The context where a word or a phrase appears. Consider, for example, the adjective low. It is often positive when part of a context of convenience (for example, This mobile phone has a low price), but it is almost always negative when talking about supplies (for example, Supplies of drinkable water are running low).
The following subsections will explain the main concepts of NLP supervised learning.