Implementation of New Relic on Ellas War

Ellas War is producing a lot of metrics and needs to be as available as possible to provide a good user experience. Few months ago, I was using Zabbix to monitor my servers. I have had until 9 machines to monitor and everything was working great. After an update, my main server that was also hosting the monitoring has started to become unstable. So I have taken the decision to remove the monitoring and only rely on Better Uptime and Grafana to detect problems.

Time has passed, and I was lacking of metrics, mostly for my main server. New Relic is a SaaS software that can be used to track logs and metrics. Its free tier is maybe the best one available on the market, one full platform user and 100GB of data per month.

Startup

The installation of the CLI is starting with a curl, I don’t like that so much but it’s easy to run and efficient. My server is using Ubuntu LTS.

curl -Ls https://download.newrelic.com/install/newrelic-cli/scripts/install.sh | bash && sudo  NEW_RELIC_API_KEY=MY-KEY NEW_RELIC_ACCOUNT_ID=00000 /usr/local/bin/newrelic install

Main problem with this setup, to update the CLI an other curl needs to be run, so I can’t rely on unattended-upgrade.

We need to update your New Relic CLI version to continue.

    Installed version: 0.60.3
    Latest version:    0.61.0

  To update your CLI and continue this installation, run this command:

    curl -Ls https://download.newrelic.com/install/newrelic-cli/scripts/install.sh | bash

Services

Lets see which services are running or are used by Ellas War.

For the server:

• Linux
• Docker
• MariaDB
• Nginx
• NodeJS
• Redis

Ellas War has also a front-end, that you can find on GiHhub.

• Angular
• Android
• Cordova

I don’t want to track my users, so I will pass on the front-end part but the server part will be able to give me a lot of interesting metrics.

Configuration

Everything is running into Docker, so I will need to adjust a bit some configuration. From the New Relic web interface, click on Add data and start feeding it. NodeJS metrics will be scraped using the Docker plugin.

Linux & Docker

Nothing really special, the CLI provided by New Relic is working great

MariaDB

Install the package to get the compatibility and follow the documentation.

apt install -y nri-mysql

Since my MariaDB server is running into Docker, I have bound the port 3306 to localhost.

My configuration file:

integrations:
- name: nri-mysql
  env:
    HOSTNAME: localhost
    PORT: 3306
    USERNAME: newrelic
    PASSWORD: mariadb-password
    REMOTE_MONITORING: true
  interval: 30s
  labels:
    env: production
    role: write-replica
  inventory_source: config/mysql

Redis

The configuration is very similar to MariaDB, but the port to bind is 6379.

apt install -y nri-redis

My configuration file:

integrations:
  - name: nri-redis
    env:
      METRICS: true
      HOSTNAME: localhost
      PORT: 6379
      KEYS: '{"0":["<KEY_1>"],"1":["<KEY_2>"]}'
      USERNAME: newrelic
      PASSWORD: my-redis-password
      REMOTE_MONITORING: true
      USE_UNIX_SOCKET: true
    interval: 15s
    labels:
      environment: production
  - name: nri-redis
    env:
      INVENTORY: true
      HOSTNAME: localhost
      PORT: 6379
      USERNAME: newrelic
      PASSWORD: my-redis-password
      REMOTE_MONITORING: true
      USE_UNIX_SOCKET: true
    inventory_source: config/redis
    interval: 60s
    labels:
      environment: production

Nginx

We also need to install the plugin and follow the documentation.

apt install -y nri-nginx

I have modified the default_server block of my Nginx configuration.

server {
  listen 80 default_server;
  
  root /var/www;
  index index.html;
  
  location = /status {
    stub_status on;
    allow 172.20.0.1;
    deny  all;
  }
}

Now we can configure New Relic

integrations:
- name: nri-nginx
  env:
    METRICS: "true"
    STATUS_URL: http://127.0.0.1/status
    STATUS_MODULE: discover
    REMOTE_MONITORING: true
  interval: 30s
  labels:
    env: production
    role: load_balancer

- name: nri-nginx
  env:
    INVENTORY: "true"
    CONFIG_PATH: /etc/nginx/nginx.conf
    REMOTE_MONITORING: true
    STATUS_URL: http://127.0.0.1/status
  interval: 60s
  labels:
    env: production
    role: load_balancer
  inventory_source: config/nginx

Conclusion

All my metrics are now stored in New Relic. Thanks to that I’m able to go back in the time and troubleshoot my services. With my few services, I’m already reaching 40% of the data limit, so I still have a margin.

I the future I will have to investigate to make custom dashboards and configure the alerting system. I also would like to find help to harden my system.