Skip to main content

9B2.Actuator/Admin Monitor

trydoforOriginalPracticeMonitorAbout 3 min

9B2.Actuator/Admin Monitor

In Wings, basic monitoring is enabled by default

  • Wings start/stop and log notification
  • Hazelcast Management Center
  • Spring Boot Admin
  • Prometheus Grafana
  • Sentry Log Monitor

9B2.1.Start/Stop and Log

By default, Wings self-monitoring has two parts: the script (wings-starter.sh) and the application itself (app), which posts notifications via Dingtalk.

  • wings-starter.sh - via cron: (1) whether pid exists, (2) whether log grows
  • wings app - start, stop, JVM metrics, and WARN in logs

1a.Script Monitoring

Using the warn argument to run the monitor, crontab is recommended if monitor periodically, the following properties should be config.

  • WARN_TXT - preceding text, e.g. keywords in the Dingtalk, etc.
  • WARN_AGO - seconds, if not empty and the log NOT grow in N seconds, it will report
  • WARN_RUN - webhook, if not empty and there is any warning, it will report

1b.Itself Monitoring

The disadvantage of the itself monitoring is "Who monitors the monitor", it needs the app itself to be able to execute healthily.

The default config is detailed in wings-monitor.properties, and the key points are,

  • spring.application.name - locate the app, required
  • wings.slardar.monitor.cron - scans every 17 minutes by default
  • .log.default.file - default log file logging.file.name
  • .view.domain - the url to view logs externally, recommended
  • .ding-notice - Dingtalk settings, monitor by default

9B2.2.hazelcast Monitor

Hazelcast keep the same version with spring boot, mainly used as caching, Session, Event and Lock, Messaging and so on.

Hazelcast IMDG The open source version is licensed under Apache2 (including standalone deployment and Runtime in SpringBoot). Compared to the Pro and Enterprise versions, it lacks the Persistence and Security suite, which has little impact in the intranet.

In the basic deployment of Wings, there is typically only a 3-node cluster (admin/devops/front) with log and ManagementCenter monitoring. Because the 3-node cluster is just robust enough to run in CP Subsystem Unsafe Mode, it only provides weak consistency assurance, which is not as strong as CP.

  • CP Subsystem Management APIs are not available.
  • split-brain protection is not supported.

Management Center can only monitor clusters with up to 3 members when no license is available. Be sure to set strong passwords when using it, and disable some insecure options on the public network, such as script

wget https://repository.hazelcast.com/download/management-center/hazelcast-management-center-5.0.4.zip
unzip hazelcast-management-center-5.0.4.zip
cd hazelcast-management-center-5.0.4
nohup bin/start.sh 8090 >hazelcast-management-center.log 2>&1 &

Just visit http://localhost:8090 and set a strong enough password. The wings-starter.sh sibling directory generates diagnostics.log to view Metric and other information. wings default period.seconds is 600, i.e. 10 minutes.

9B2.3.Spring Boot Admin

With devops as server and other apps as client, autostart and registration, see the wings-boot-admin.properties. Among them, we need to take care of the user permission settings, whether the password is strong, and whether the DingTalk token and keywords are correct.

Between server and client, there is a Wings adapted version of BASIC authentication, with timestamp and md5 digest, instead of plaintext transmission of password, but the security is still not high. Therefore, in addition to strong passwords, try to use intranet isolation, transport layer encryption, try to use https, etc.

As in the demo project structure, devops deps on spring-boot-admin-starter-server, And in other boot applications, such as admin, deps on spring-boot-admin-starter-client. Also, configure wings-boot-admin.properties separately. The demo project is configured in common uniformly.

9B2.4.Prometheus Grafana

Use prometheus to pull data. If you need push form, use pushgateway.

prometheus is only used as a database and is naked, without user authentication, and needs to be protected by itself.

# check prometheus
curl -G http://localhost:8093/actuator/prometheus -u "boot-admin-client:${DING_TALK_TOKEN}"
# prometheus config
tee /Users/trydofor/Docker/prometheus/conf/prometheus.yml << EOF
scrape_configs:
  - job_name: "wings-winx"
    metrics_path: "/actuator/prometheus"
    static_configs:
      - targets:
        - "host.docker.internal:8091"
        - "host.docker.internal:8092"
        - "host.docker.internal:8093"
    basic_auth:
      username: "boot-admin-client"
      password: "${DING_TALK_TOKEN}"
EOF

# start docker
docker run -d \
 --name prometheus \
 --restart=unless-stopped \
 -v /Users/trydofor/Docker/prometheus/conf:/etc/prometheus \
 -v /Users/trydofor/Docker/prometheus/data:/prometheus \
 -p 9090:9090 \
prom/prometheus

# prometheus config
tee /Users/trydofor/Docker/grafana/conf/grafana.ini << EOF
[security]
admin_user = admin
admin_password = ${DING_TALK_TOKEN}
EOF

# start docker
docker run -d \
 --name grafana-oss \
 --restart=unless-stopped \
 -v /Users/trydofor/Docker/grafana/conf:/etc/grafana \
 -v /Users/trydofor/Docker/grafana/data:/var/lib/grafana \
 -p 3000:3000 \
grafana/grafana-oss

Then import the predefined dashboards.

JVM Micrometer / 4701

9B2.5.Sentry Log Monitor

You can use official service or official docker

#https://develop.sentry.dev/self-hosted/
wget -O sendtry.tgz https://github.com/getsentry/self-hosted/archive/refs/tags/22.2.0.tar.gz
tar xzf sendtry.tgz
cd self-hosted-22.2.0/
./install.sh
# https://docs.docker.com/engine/install/ubuntu/
# https://docs.docker.com/engine/install/linux-postinstall/
# Modify the mail.* configuration
vi sentry/config.yml
#mail.backend: 'smtp'
#mail.host: 'smtp.exmail.qq.com'
#mail.port: 465
#mail.use-ssl: true
docker-compose up -d
# Modify others, you can stop and then up
docker-compose stop