Add node to cluster

Once a test chef-bach cluster is up and running users can manually add new nodes to the cluster by following these steps.

• On the hypervisor host run only the function create_cluster_VMs in vbox_create.sh script file with these changes.
• Update the for look in the function to include the new node names. For e.g. if bcpc-vm4 need to be added to the cluster as a new node change the for loop to

for vm in bcpc-vm4; do ... 

which is originally coded as

for vm in bcpc-vm1 bcpc-vm2 bcpc-vm3; do ...

• Comment the following lines at the end of vbox_create.sh script.

...
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
  ##download_VM_files
  ##create_bootstrap_VM
  create_cluster_VMs
  ##install_cluster
fi
...

• The assumption is that the user will be executing from the chef-bcpc directory from where the original cluster was created from and there has been no changes.
• Execute the modified vbox_create.sh script which will create the new virtual machine(s).

./vbox_create.sh

• We need to identify the id of NIC 1 of the newly created VM(s)
• Using bcpc-vm4 as example, run vboxmanage showvminfo bcpc-vm4 | grep 'NIC 1' on the hypervisor host which will generate the output similar to the following

NIC 1:           MAC: 0800276DB8F7, Attachment: Host-only Interface 'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82543GC, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth group: none

• Note the MAC: value for NIC 1 and logon to the bootstrap host for the cluster.
• Update the cluster.txt file in the bootstrap node with the details for the new VM(s). In the case of BCPC-VM4 which is going to be a single node ZooKeeper & Kafka server, the following will be the new entry in cluster.txt

bcpc-vm4 08:00:27:6D:B8:F7 10.0.100.14 - bcpc.example.com role[BCPC-Kafka-Head-Zookeeper],role[BCPC-Kafka-Head-Server] 

• On the bootstrap node, run the cluster-enroll-cobbler.sh to add new nodes to cobbler

./cluster-enroll-cobbler.sh add bcpc-vm4 

• Once the VM(s) are enrolled, power on the VM(s) which need to be done from the hypervisor host. For e.g.

vboxmanage startvm bcpc-vm4 --type=headless 

• When the VM(s) are up which take a bit of time since OS gets installed the first time, run cluster assign roles script from the bootstrap node to install the required components on the VMs through chef-client.

./cluster-assign-roles.sh Test-Laptop hadoop bcpc-vm4 

If kerberos authentication is enabled you will need to chef the bootstrap node first in order to generate keytabs for the newly created node.

Also if the new node is worker node run

yarn rmadmin -refreshServiceAcl

This will get a new VM node added to the cluster.