Updating and brainswapping your node
Last updated
Last updated
This tutorial is fully covered in the video shown above.
Brainswapping is the idea of having two nodes switch identities at the same time. We call it "brainswapping" because a node's identity dictates how it behaves.
It was implemented as a way to update the network without having to bring it down, as federated servers can "brainswap" with standby nodes that have already been updated with the new code.
The network will not perceive this as a node going offline, as the identity (and thus the associated federated server) is still online.
After transferring the Authority identity the node operator can now shutdown their original node, perform necessary updates, bring it back online and finally brainswap the identity back into the original server.
The procedure can also be used for migrating the Authority identity to a new physical server, by not performing the brainswap a second time to reverse the first swap .
First check if your node is running the latest version. This can be done by comparing your nodes version with the latest found here. If your follower is not running the latest version, you can proceed by stopping and removing the current factomd docker container.
Dont worry, this will not delete or remove the synchronized database.
This is also a good time to update and reboot the system, if you have any pending updates or reboots. To update simply run the following commands.
Now, run the following command exactly :
If you're updating or brain swapping a node on the test-net you need to add the following parameters to the command above, before -startdelay=600:
After this your node will be started. You can check for the existence of a Factom container using the command docker ps.
Please wait for your node to be fully synced by checking the control panel node sync statuses to be 100% before performing any next steps. Please also regard the initial wait period of 20 minutes before doing anything with your node. Furthermore, note that the factomd:v6.5.2-alpine version may have changed since the time of writing.
For experienced and seasoned brainswappers there's an automated brain swap script made by Stamp-IT found here. However, do not try this if this is the first time you're doing a brainswap, as it's important to understand the process in case something goes wrong.
To perform the brain swap you will need one standby node that is ready. The standby node should be running the most recent Factomd software version if you've completed the previous steps.
Federated Node is the node that holds your authority identity, and needs to be updated.
Standby Node is a follower node on the network that you control.
If your standby node is not in sync with the network, performing the brainswap will result in your authority server going offline and likely to be demoted, so it is crucial to first check the health of the standby node.
Check if the DBHeight matches that of the network.
This is done by comparing DBHeight in your control panel (localhost:8090) with the DBHeight of your federated server.
Check if the minutes are following that of the network.
This is done by comparing the control panel of the Stanby node to that of the Federated. On the summary tab of the more Detailed Node Information you will see something similar to this:
The 7/7 means you are on minute 7. You will want to make sure this number is the same on the Standby (_/7) and the Federated (7/7)
Check the process list for Nil(s), that indicates some network instability.
Process list is located in the control panel (localhost:8090) -> "more detailed node information". If any Nil lines show you should not move on with the brainswap.
Once you've confirmed that the standby node is ready, it's time to start the swap process.
If you haven't done a brain swap before, it is highly recommended that you read through this tutorial and watch the complete video before proceeding with the actual process.
The swap requires both config files (located on Federated Node and Standby Node) to be modified.
Open both config files in parallel in a text editor of your choice. The config files are located at /var/lib/docker/volumes/factom_keys/_data. Some examples with vim and nano:
vim: vi /var/lib/docker/volumes/factom_keys/_data/factomd.conf
nano: nano /var/lib/docker/volumes/factom_keys/_data/factomd.conf
Swap the following lines in the two config files:
If you prefer to do a brain-transfer instead of a swap, you can just comment out the lines in your Federated node by placing a ; in front of these lines.
Then add an additional line: ChangeAcksHeight = 0
This additional line is the brainswap logic. You will want to set the ChangeAcksHeight to some block height in the future (remember the block height on the control panel is the last saved height, not the current working height!).
The safe height is the one you see in the control panel + 4. (localhost:8090). If you know how to read the more detailed page, you can get away with a closer number.
Once you set the ChangeAcksHeight to DBHeight+4, save both files.
At the block height ChangeAcksHeight you should see both nodes change identities. If none of your nodes crash, and the identities change, the swap was successful.
