services
Keeping services up and running requires practical knowledge of tooling.
Lifecycle management
starting a service
Using monit:
check host app_name with address 127.0.0.1
start "/sbin/start app_name"
stop "/sbin/stop app_name"
if failed port 80 protocol HTTP
request /ok
with timeout 5 seconds
then restart
Using upstart:
description "app_name"
start on startup
stop on shutdown
script
# Node needs HOME to be set
export HOME="path/to/node/app"
exec sudo -u nodejs \
/usr/local/bin/node \
path/to/node/app/server.js \
production \
2>> /var/log/app_name.error.log \
>>/var/log/app_name.log
end script
stopping a service
To stop a service, use the killproc(8) command passing it a PIDFILE.
$ killproc -p ${PIDFILE} -d ${STOP_TIMEOUT} -SIGTERM $PROG
Provisioning
Monitoring
Running systems must be monitored to check for irregularities and track performance.
Directory structures
┌──────────────┐
│ ~/ │ ┌──────────────────────┐
├──────────────┤ │ / │
│ ~/db ◀───┐ ├──────────────────────┤
│ ~/image │ └──┼─/var/db │
│ ~/log ◀┼──────┼─/var/log │
│ ~/script │ ┌───┼▶/var/www │
│ ~/service ─┼──┘ └──────────────────────┘
└──────────────┘
~/db/ # symlinks to service databases
~/image/ # downloaded images
~/log/ # symlinks to service logs
~/script/ # scripts to interact with services
~/service/ # symlinks to currently running services
/var/log/ # log files
/var/www/ # currently running services
/var/db/ # database data, can be symlinked to separate mount
Adding configuration to a service
This treads into the territory of opinions, and thus caution is adviced. In my opinion services must be self-contained; which means they should carry their own config. If config is dynamic, that too should be carried.
Testing a service should ideally be done in a containerized environment, to fully mimick the OS situation. Else there's a discrepancy between the way the service is interacted with during development, and during production. The closer the two are, the better.
To hold configuration, a service should have a config/ directory in its root
which would be overlaid 1:1 with the root directory (/). That way it's super
clear where all the files go. Symlinks are always preferred over hard copies in
this case; unless the application that's being instrumented is unable to follow
symlinks.
Naming of services
It's convention to prefix services with either www-, api- or service-.
Sometimes people will use www- to mark static websites, and api- or
service- to mark more backendy type of systems, but that kind of separation
can get messy fast. In the end, make a judgement call and monitor the services
you have, and eventually you'll find a naming scheme that creates a suitable
distinction.
Logs
Logs should go in /var/log/<service_name>.log. Logs should be rotated using
logrotate(1). Add to /etc/logrotate.d/<service_name>:
/var/log/myapp.log {
su root
daily
rotate 7
delaycompress
compress
notifempty
missingok
copytruncate
}