What is "overlay" and what is its connection with memory?

This is used in primitive operating systems. The famous MS-DOS used this technique. Today it is only used on very limited devices, although some are becoming quite popular with the IoT.

In general the operating system does not have this for lack of space to create the Virtual Memory Engine and mainly because it is made to run on hardware that does not provide facilities to manage the memory in a simple, transparent and com performance.

Think of the DLL. It is a way to break an executable into parts. The overlay is the same thing, it's only done in a slightly different way. In fact one of the advantages is to break the executable into parts to be distributed separately, in the past more useful than today since the floppy disks had very little space and was the most used means of transport.

The engine is useful when the executable is too large to fit in memory. It splits the code in parts that can be loaded alternately as needed. There is usually code insertion to manage the need to load another part of the code that is not yet in memory.

In more complete operating systems in modern architectures there is the virtual memory system, so it does not matter if what you need is in physical memory or not, suffice it to say that it is in memory that exists beyond the limits of RAM, in general a part can be in disk. The application does not need to know any of this, for it there is a huge memory limited in 4GB in 32 bits or how much OS allow up to the limit of 16EB in 64 bits. The OS manages whether it needs to put a part in secondary storage or will keep in RAM.

A virtual memory is a management that determines where memory pages (commonly 4KB blocks) are in memory. These pages can be scattered throughout the RAM or they can be elsewhere. In thesis may even be on another machine. It controls this, it is not a problem of the user or the developer of the application. It's an abstraction to make a lot of things easier, including protecting areas of memory and meeting applications ' memory allocation requests consistently.

In general a modern executable is usually loaded with a technique of file mapped in memory , which makes it transparent whether it is on disk or RAM. Physically an excerpt from executable needs to be in RAM to run, so whenever you try to access and fail occurs a page fault (Wikipedia) and the OS will bring the page that should be on disk to RAM. So a code or data that is being used in two different processes does not need to be duplicated, physically it only exists once despite being referenced in two virtual memory spaces. So it is complicated to measure application memory consumption, there is the physical consumption of RAM, the total consumption and the virtual consumption.

The overlay is a poor virtual memory :) it is not so sophisticated, it only serves for the separation of part of the code, nothing else.