Having the wrong memory configuration will result in different behaviors. It can be boiled down to two different situations: not having enough memory but having enough to run the system, and not having enough memory to the point that Elasticsearch can't even start.
For the first scenario, where there is a memory constraint, but there is just enough for Elasticsearch to start and run, the main problem would be that there is not enough memory for the page cache, which results in slow searches and reduced indexes per second. In this scenario, Elasticsearch is able to run, but with a reduced overall performance.
The other scenario can be split into two different situations: one where there's not enough memory to start Elasticsearch and the other where Elasticsearch is able to start, but as soon as some indexes are added, it runs out of memory. To avoid a system crash, Linux has a mechanism called out-of-memory killer (OOM killer).