Wrapper for Remote Executor

Currently BAMPI uses remote executor to do various types of tasks. Apart from Ironic’s agent pattern, remote executor is based on SSH. For high availability design, we need to consider a lot of situations, one is “while during task execution, BAMPI-1 fails”. In this situation, remote executor is dead, but the task being run still remains on the target bare-metal. So how does BAMPI-2 take over this kind of situation?

By doing some experiments, we found that although the SSH connection is broken, the processes started by that SSH session still goes on. The real question is, how to prevent BAMPI-2 running new tasks on the same bare-metal which already has a running task? To differentiate whether a bare-metal has a previous executed task or not, try to use ps. For example, when Bob logins to a server and executes sleep 100 &, then he logouts, that “sleep” process is still running in the background. However, its parent process is no longer being Bob’s shell. These kind of processes are called “orphan process”: an orphan process is a computer process whose parent process has finished or terminated, though it remains running itself. All the orphan processes’ parent process is init which has PID 1.

So the answer is clear: find the orphan processes and wipe them out! But there might be some other processes’ parent process ID is 1, e.g. the processes spawned by init or other orphan processes not started by BAMPI at the beginning. And there are various types of tasks that BAMPI can run. So one suggestion is that, in addition to BAMPI’s remote executor, write another “wrapper” that runs on bare-metal. Therefore we can combine these two condition: parent process ID is 1 and process name equals to the name of wrapper. Ta-da!