When deploying a Cosmos SDK based node such as Archway, we must keep in mind that p2p networks are exposed to the risk of denial of service attacks, and an effective way to mitigate this type of attacks is by using sentinel nodes.
Why is it important to avoid denial of service attacks?
When our node is suffering a denial of service attack that cannot be mitigated by the server provider, it is likely that the server will be inoperative for the duration of the attack because, being saturated, it will stop responding to the requests made by the rest of the nodes in the network.
If our node becomes inoperative, it means that we are no longer participating in the consensus rounds, our node will not be signing blocks or generating rewards for the delegators and therefore in a few minutes the node will be jailed and penalized.
In addition to being a risk for node operators due to the penalties involved in being jailed, it is also a risk for the network, since if an attack is carried out on several nodes at the same time, the stability and security of the network would be affected.
P2P networks
A p2p (peer-to-peer) network, such as Archway's network, is a network where a set of computers (2, 10 or 130, no matter the number) are interconnected with each other. Unlike what happens when we visit a web page where there is a server to which we make a request, in a p2p network there is no central server, but all the machines are equal to each other and are continuously communicating to perform the validation of the blocks.
The problem with p2p networks is that in order for the different machines (nodes) that are part of the network to communicate with each other, an entry point is required to establish connections; this entry point is the IP address and port. Knowing the machine's IP address and p2p port, the rest of the nodes in the network can make requests and, in short, make it part of the p2p network, but it is also the entry point for an attacker's denial-of-service attacks.
Sentinel nodes, hiding our validator's IP from the world
As we discussed earlier, when we expose the IP of our validator node, the rest of the nodes in the network can connect to us, but we are also allowing someone to saturate our machine.
Using sentinel nodes is a solution to prevent the rest of the network from knowing the IP of our validator node, but allowing our validator node to continue communicating with other validator nodes in the Archway network, since if the node were 100% isolated, it would not be able to participate in the network and therefore would not generate rewards for the delegators.
A sentinel node is a bridge between our validator node and the rest of the network, so that the rest of the network does not know the IP of the validator node, but the IP of the sentinel nodes.
As can be seen in the previous image, the "Validator node" is only connected to our sentinel nodes, but not to the rest of the network. Thus, only the sentinel nodes will know the IP of our validator node.
Can sentinel nodes suffer denial of service attacks?
Yes, but as they are nodes that are not validating transactions and only act as a bridge between the network and the validating node, we could quickly deploy new sentinel nodes or even change the IP of the sentinel node being attacked.
How many sentinel nodes can there be?
There is really no maximum number, the more sentinel nodes we have, the more resistance to denial of service attacks our validator node will have. However, it must be taken into account that the more sentinel nodes we have, the more complex it will be to maintain our nodes when carrying out maintenance or upgrades, in addition to the increase in server costs. You should have at least two sentinel nodes and if possible have one of them in a different datacenter from where the validator node is deployed.
Where should the sentinel nodes be deployed?
If two sentinel nodes are going to be mounted, one of them could be deployed in the same datacenter where our validator node is, this will reduce the latency between both servers, and therefore, the connection between both servers will be quite fast. The other sentinel node could be located in a different datacenter; this way, in case the network of the datacenter where our validator node is down for any reason, we would always have a node with the current block available to synchronize our validator node.
Step by step guide
To follow this guide, we will use a node created in Archway's torii-1 testnet network. In case you have not deployed the node yet, you can rely on the official Archway documentation and also on the torii-1 specific documentation.
Creating the sentinel nodes
Once we have contracted the two sentinel node servers and we have the access IPs, we must perform the same installation that we would perform as if it were a validator node on both servers.
🚀 Tip: If you use MobaXterm, you can use the MultiExec option to perform the same commands on both servers at the same time.
First, we will need to download and compile the archway binary.
git clone https://github.com/archway-network/archway
cd archway
git checkout main
Note: it is possible that in the future, the main branch will not be the most suitable and it will be necessary to download the version that is published in the official Archway documentation. In any case, the version must be the same that we have in our validator node.
Once downloaded, we will proceed to compile the binary:
make install
It will also be necessary to initialize the node so that the .archway directory is created, for this we can use the following commands, each one in the corresponding server:
- On sentinel node A:
archwayd init "Stakely.io - Sentry A" --chain-id torii-1
- On sentry node B:
archwayd init "Stakely.io - Sentry B" --chain-id torii-1
Once the previous step is done, the .archway folder will already exist, so we can download the genesis.json file:
curl -s https://raw.githubusercontent.com/archway-network/testnets/main/torii-1/genesis.json > ~/.archway/config/genesis.json
Finally, we will configure the archway service:
sudo nano /etc/systemd/system/system/archway.service
Remember to replace the user you are using where it says <user here>.
[Unit]
Description=Archway Daemon
After=network-online.target[Service]
User=<user here>
ExecStart=/home/<user here>/go/bin/archwayd start --p2p.laddr tcp://0.0.0.0.0:26656 --home /home/<user here>/.archway
Restart=on-failure
RestartSec=3
LimitNOFILE=4096[Install]
WantedBy=multi-user.target
Once we have finished editing, we will enable the service:
sudo systemctl enable archway
Base configuration of the sentinel nodes
For now we have not done anything different from what we would do when setting up a validator node, although we have stopped doing several things, such as creating the wallet or running the validator creation command "tx staking create-validator", since we do not want to create validator nodes, only nodes that synchronize with the rest of the nodes in the network and we can use to synchronize our validator node in a secure way without exposing our IP to the rest of the network.
The sentinel nodes (both), must have peers to be synchronized at all times, we will add the following peers to the config.toml file inside the config folder.
persistent_peers = "[email protected]:26656,[email protected]:46656,[email protected]:26656,[email protected]:26756,[email protected]:30273,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26686"
Note: the peers were obtained from Discord in the torii-1 channel.
It would also be advisable to add as persistent peer the sentinel nodes between them. That is to say, that the sentinel node B is persistent peer of the sentinel node A, in the same way that the sentinel node A is persistent peer of the sentinel node B; this will make us have more redundancy.
In the config.toml file of the config folder of the sentinel nodes (both), it is necessary to specify the id of the validator node in the private_peer_ids parameter. This is so that our sentinel nodes never share with the rest of the peers of the network the existence of our validator node.
private_peer_ids = "validator node".
The appearance should look something like this:
If you don't know how to get the id of your validator node, you can get it with the following command:
archwayd tendermint show-node-id
Note: to the received id, we must concatenate the ip and the p2p port as shown in the image above.
In the same configuration file, we will also find the unconditional-peer-ids parameter to which we must add our validator node as we did with the private_peer_ids parameter. This step is necessary because the nodes have a limited number of peers to which they can connect.
To avoid the risk of leaving our validator node incommunicado, by setting its id in the unconditional-peer-ids parameter, we will make the sentinel nodes always connected to the validator node even if the limit of peers to which they can be connected has been exceeded. Optionally we can also add in this same parameter (separated by commas) the homologous sentinel node to guarantee that the sentinel nodes will always be connected to each other.
unconditional-peer-ids = "validator node, homologous sentinel node".
Note: with homologous sentinel node we mean that if you are configuring the sentinel node A, the homologous would be the B, and in the case of editing the sentinel node B, the homologous would be the sentinel node A.
At this point, we could start our sentinel nodes and let them synchronize and be discovered by the rest of the network. For that we will use the following command:
sudo systemctl start archway
If we have done all the steps correctly, we should see how both sentinel nodes are synchronizing:
Validator node configuration
Once the sentinel nodes have been synchronized, we can configure the validator node with all certainty that our validator node is not going to stop being synchronized with the network. We will have to edit the config.toml file in the config folder, where we will find the following lines:
# Comma separated list of nodes to keep persistent connections to
# Do not add private peers to this list if you don't want them advertised
persistent_peers =[list of sentry nodes]
In persistent_peers we will add only the ids of the sentry nodes, that is to say that if the node was already mounted before, we will have to delete the content of this field before adding our sentry nodes.
The appearance should be similar to the following:
In the same configuration file we will find the pex parameter, which we must set to "false". This parameter does not discover other peers, it will only use the ones set in the persistent_peers parameter.
pex = false
If the validator has already been exposed to the network, we can delete the address book so that it only "knows" the sentinel nodes; if it is a validator that has never been started, this step is not necessary. Once inside the config folder and with the node stopped, perform the following command:
rm addrbook.json
At this point, we can start the node and it will synchronize using only the sentinel nodes.
To verify that we are really connected only to two peers and these are the sentinel nodes, we can write the following command in our validated node; the output is a JSON where it will show us the number and which peers we are connected to, where the number should be two and the peers should be our sentinel nodes.
curl -s localhost:26657/net_info
Here you can see the number of peers to which our validator node is connected, in this case it is two, which is correct.
And in the JSON object, if we continue observing, we will see that the two peers we have are "Sentry A" and "Sentry B".
Extra: Protecting the validator node through firewall
Right now the p2p port of our validator node is open and anyone can establish a connection. If we have just set up the validator node, with the configurations made, nobody would have to find our validator node, however, to add a plus of security it is advisable to close the p2p port and only allow traffic to the IP's of our sentinel nodes.
There are several ways to do this, it is possible that your server provider allows you to do it through a gui. Ufw is a simple to use alternative, you can read more about ufw here.
The commands to use with ufw are as follows:
sudo ufw allow from <sentry A IP> proto tcp to any port 26656
sudo ufw allow from <sentry B IP> proto tcp to any port 26656
Extra: Private networks
One possibility we have when our validator node is in the same datacenter as one of the sentry nodes, would be to use private addressing.
In case we want to use private addressing, we will have to edit the addr-book-strict parameter of the config.toml file and set it to false in both the validator node and the sentinel node that are communicating under private addressing. This parameter, when set to "true", will only add routable addresses to the address book, the private addresses defined in RFC-1918 are not routable, therefore, they would not be added to the address book, so we will have to change it to false to be able to use IP addresses from the private ranges.
addr-book-strict: boolean. By default, nodes with a routable address will be considered for connection. If this setting is disabled (false), non-routable IP addresses, such as addresses from a private network, can be added to the address book.
Source: https://docs.tendermint.com/master/nodes/validators.html
Conclusions
Having sentinel nodes will not only help our validator node not to fall victim to a denial of service attack, but will also make the Archway network more robust. The installation of the sentinel nodes does not differ much from that of a validator node, and the additional configurations to be performed are very simple and intuitive.
Now that Archway is doing the torii-1 testnet, it is a good time to try to set up sentinel nodes and once the mainnet is launched, to be able to replicate it with the experience of having been able to test it in torii-1. If you already have your node on torii-1, what are you waiting for to set up your sentinel nodes and protect yourself from unwanted attacks?