Ellas War is producing a lot of metrics and needs to be as available as possible to provide a good user experience. Few months ago, I was using Zabbix to monitor my servers. I have had until 9 machines to monitor and everything was working great. After an update, my main server that was also hosting the monitoring has started to become unstable. So I have taken the decision to remove the monitoring and only rely on Better Uptime and Grafana to detect problems.
Time has passed, and I was lacking of metrics, mostly for my main server. New Relic is a SaaS software that can be used to track logs and metrics. Its free tier is maybe the best one available on the market, one full platform user and 100GB of data per month.
Startup
The installation of the CLI is starting with a curl, I don’t like that so much but it’s easy to run and efficient. My server is using Ubuntu LTS.
curl -Ls https://download.newrelic.com/install/newrelic-cli/scripts/install.sh | bash && sudo NEW_RELIC_API_KEY=MY-KEY NEW_RELIC_ACCOUNT_ID=00000 /usr/local/bin/newrelic install
Main problem with this setup, to update the CLI an other curl needs to be run, so I can’t rely on unattended-upgrade.
We need to update your New Relic CLI version to continue.
Installed version: 0.60.3
Latest version: 0.61.0
To update your CLI and continue this installation, run this command:
curl -Ls https://download.newrelic.com/install/newrelic-cli/scripts/install.sh | bash
Services
Lets see which services are running or are used by Ellas War.
For the server:
- Linux
- Docker
- MariaDB
- Nginx
- NodeJS
- Redis
Ellas War has also a front-end, that you can find on GiHhub.
- Angular
- Android
- Cordova
I don’t want to track my users, so I will pass on the front-end part but the server part will be able to give me a lot of interesting metrics.
Configuration
Everything is running into Docker, so I will need to adjust a bit some configuration. From the New Relic web interface, click on Add data and start feeding it. NodeJS metrics will be scraped using the Docker plugin.
Linux & Docker
Nothing really special, the CLI provided by New Relic is working great
MariaDB
Install the package to get the compatibility and follow the documentation.
apt install -y nri-mysql
Since my MariaDB server is running into Docker, I have bound the port 3306 to localhost.
My configuration file:
integrations:
- name: nri-mysql
env:
HOSTNAME: localhost
PORT: 3306
USERNAME: newrelic
PASSWORD: mariadb-password
REMOTE_MONITORING: true
interval: 30s
labels:
env: production
role: write-replica
inventory_source: config/mysql
Redis
The configuration is very similar to MariaDB, but the port to bind is 6379.
apt install -y nri-redis
My configuration file:
integrations:
- name: nri-redis
env:
METRICS: true
HOSTNAME: localhost
PORT: 6379
KEYS: '{"0":["<KEY_1>"],"1":["<KEY_2>"]}'
USERNAME: newrelic
PASSWORD: my-redis-password
REMOTE_MONITORING: true
USE_UNIX_SOCKET: true
interval: 15s
labels:
environment: production
- name: nri-redis
env:
INVENTORY: true
HOSTNAME: localhost
PORT: 6379
USERNAME: newrelic
PASSWORD: my-redis-password
REMOTE_MONITORING: true
USE_UNIX_SOCKET: true
inventory_source: config/redis
interval: 60s
labels:
environment: production
Nginx
We also need to install the plugin and follow the documentation.
apt install -y nri-nginx
I have modified the default_server block of my Nginx configuration.
server {
listen 80 default_server;
root /var/www;
index index.html;
location = /status {
stub_status on;
allow 172.20.0.1;
deny all;
}
}
Now we can configure New Relic
integrations:
- name: nri-nginx
env:
METRICS: "true"
STATUS_URL: http://127.0.0.1/status
STATUS_MODULE: discover
REMOTE_MONITORING: true
interval: 30s
labels:
env: production
role: load_balancer
- name: nri-nginx
env:
INVENTORY: "true"
CONFIG_PATH: /etc/nginx/nginx.conf
REMOTE_MONITORING: true
STATUS_URL: http://127.0.0.1/status
interval: 60s
labels:
env: production
role: load_balancer
inventory_source: config/nginx
Conclusion
All my metrics are now stored in New Relic. Thanks to that I’m able to go back in the time and troubleshoot my services. With my few services, I’m already reaching 40% of the data limit, so I still have a margin.
I the future I will have to investigate to make custom dashboards and configure the alerting system. I also would like to find help to harden my system.