Not enough disk space:
If the issue is inadequate disk space, you see the alerts below before the error message is displayed:
Healthmon: V=1, AID=53, S=2, MSG='The partition is at least 96% full.'
Healthmon: V=1, AID=53, S=2, MSG='The partition is at least 100% full.'
Other symptoms include:
- Identifying a full partition in the df -h output.
- Entries as shown below in /var/log/RunESnode-*.log.
Caused by: java.nio.file.FileSystemException: /els_storage/name_20210101000000/els/<guid>/node-0/data/siem-es-cluster-0/nodes/0/_state/node-<id>.st.tmp: No space left on device
The storage has run out of space which corrupts the node and is unrecoverable. You must add more space to the storage device to avoid this issue in future.
To get the system working again, delete all current data using the following command:
rm -r /els_storage/name_20210101000000/els
Insufficient RAM:
If the issue is about less RAM capacity, you can see similar entries as shown below in
/var/log/RunESnode-*.log.
java.lang.OutOfMemoryError: Java heap space
To resolve this issue, you need to either reduce the allocated storage space to the ELS or adjust the RAM configuration. If the RAM configuration hasn’t been tuned before, follow the below steps to tune it:
- Identify the total RAM capacity of the appliance.
- Calculate 60% of the total RAM and divide the value equally by the number of nodes configured in /etc/els/cluster.json and round this value down to the nearest whole integer of GB of RAM.
- Edit /etc/els/cluster.json and enter the figure against each node as the "heap" value.
For example:
Based on the following cluster.json file:
{
"indexerInstances": 1,
"name": "siem-es-cluster-0",
"nodes": [
{
"heap": 1,
"httpPort": 9200,
"mode": "MD",
"name": "node-0",
"processors": 1,
"transportPort": 9300
},
{
"heap": 1,
"httpPort": 9201,
"mode": "MD",
"name": "node-1",
"processors": 1,
"transportPort": 9301
},
{
"heap": 1,
"httpPort": 9202,
"mode": "MD",
"name": "node-2",
"processors": 1,
"transportPort": 9302
}
]
}
And 16GB total RAM.
16GB * 0.6 = 9.2GB
9.2GB / 3 nodes = 3.066 GB/node
So the heap value is 3. The resulting configuration file is now:
{
"indexerInstances": 1,
"name": "siem-es-cluster-0",
"nodes": [
{
"heap": 3,
"httpPort": 9200,
"mode": "MD",
"name": "node-0",
"processors": 1,
"transportPort": 9300
},
{
"heap": 3,
"httpPort": 9201,
"mode": "MD",
"name": "node-1",
"processors": 1,
"transportPort": 9301
},
{
"heap": 3,
"httpPort": 9202,
"mode": "MD",
"name": "node-2",
"processors": 1,
"transportPort": 9302
}
]
}
After the configuration file has been updated, restart the ELS services with the following command:
NitroRestart --nod
