How to combine the power of several computers running a single Linux system?
There are a lot of unnecessary, working system blocks lying around in the office. So I decided to combine them into a single computer network, and in fact into a regular computer controlled by a single OS, but representing 3 + system blocks.
Bottom line: I need 5 + boxes to work as one, while I do not want to manage each server separately, my goal is a single machine consisting physically of several under the control of one system. What will she do? Yes, anything, even if will be a web server with an ip on the internet or just an Ubuntu or fedora workstation
4 answers
Usually solutions are built from the task, not from the platform
What exactly do you want to do?
From personal experience-analysis of logs on billing, and retarification: I did manual sharding at the application level (Python scripts), on a bundle of decommissioned desktops, an account on the office IBM eServer, and a couple of desktops under windows.
If you have interconnects on ancient 100mbit ethernet, and not at least 1G (not to mention a special device like Infiniband) - it will be cheaper to go to shop, buy the most filthy motherboard and fill it with memory to the brim, save on an expensive fast 1G switch and power consumption
Your configuration option can be effective only in one case: all parallel threads are completely independent, and each block of the task completely fits into the RAM and resources of one node, computing tasks with a complete lack of dependencies in the "one node - one calculation block" architecture are well solved
On the other hand, as a cluster workstation such utilization is quite interesting and has the right to life, and can be a winning option: no more than 2-3 heavy tasks are active on the workstation, the rest of the resources are spent on storing gigabyte tabs in the browser, text editors and rare runs of utilities on the file system. system
- https://en.wikipedia.org/wiki/Single_system_image
- https://en.wikipedia.org/wiki/Distributed_operating_system
- https://en.wikipedia.org/wiki/Multikernel
Linux implementations don't seem to exist:
- https://en.wikipedia.org/wiki/OpenMosix (R.I.P 2008, kernel 2.4)
So the solution of the problem in the original formulation is reduced in fact to the development of the system programming in full: write an analog of the Linux kernel that provides all the necessary DOS functionality, adapt the basic libraries, including those from the GNU copilators (libc, libatomic, gomp,...), and rebuild the entire system to a state where the entire software package that you use can work
Perhaps an easier solution would be to write a virtualization layer: rewrite only the layer of all system libraries that your application software uses, with the implementation distributed POSIX functionality on top of the usual distributions installed on each node, or hypervisors (I wish you a lot of fun with Xen and libgcc/libstdc++ 8 sources-)
From a practical point of view:
- look for tasks with the minimum amount of data exchange between threads, and
- write your own software: look in the direction of ready-made distributed platforms for web/microservices that require minimal work resources
- drag and drop business processes in your office on web technology, so that you can distribute weak hardware to users as terminals/browser launchers
Unfortunately, as far as I know, there are no free implementations of distributed Smalltalk, otherwise I would first of all advise it -- as an option, look for libraries for programming on distributed messaging between objects for mainstream languages
The answer is that you need to create a cluster of computers running some Linux-like OS.
Here are some ready-made solutions:
[useless link: has been removed].
For your reference:
It doesn't make much sense to describe the whole process in the answer. I think my answer is useful to you.
In this form - in which you ask: solutions will be ineffective. Systems from many computers (servers) are made separately, for each task their own, moreover, programs - for which such a system is created-are also written specifically for certain server systems.
They are called "high-load systems", and their creation/use is a very expensive process, which is called "scaling". The largest examples of such systems are social networks: for example work VK provides 10k machines, but comparable in power to a home computer(according to the owners). And StackOverflow itself certainly does not work on the same server.
If you're really interested in scaling: then you should master system administration, docker, and read the lyrics on the topic of hylod at a cool level https://ruhighload.com/scale
Most likely energy efficiency (and possibly reliability) such a system would be too low.
The task of cluster systems is to achieve such computing power and fault tolerance, which is difficult (or impossible) to achieve by means of a single modern server. And for the disposal of old iron, it is better to use other methods ;)