Keeping services up and running requires practical knowledge of tooling.

Lifecycle management

starting a service

Using monit:

check host app_name with address
  start "/sbin/start app_name"
  stop "/sbin/stop app_name"
  if failed port 80 protocol HTTP
    request /ok
    with timeout 5 seconds
    then restart

Using upstart:

description "app_name"

start on startup
stop on shutdown

    # Node needs HOME to be set
    export HOME="path/to/node/app"

    exec sudo -u nodejs \
      /usr/local/bin/node \
      path/to/node/app/server.js \
      production \
      2>> /var/log/app_name.error.log \
end script

stopping a service

To stop a service, use the killproc(8) command passing it a PIDFILE.

$ killproc -p ${PIDFILE} -d ${STOP_TIMEOUT} -SIGTERM $PROG



Running systems must be monitored to check for irregularities and track performance.

Directory structures

  │      ~/      │      ┌──────────────────────┐
  ├──────────────┤      │          /           │
  │ ~/db         ◀───┐  ├──────────────────────┤
  │ ~/image      │   └──┼─/var/db              │
  │ ~/log       ◀┼──────┼─/var/log             │
  │ ~/script     │  ┌───┼▶/var/www             │
  │ ~/service   ─┼──┘   └──────────────────────┘
~/db/        # symlinks to service databases
~/image/     # downloaded images
~/log/       # symlinks to service logs
~/script/    # scripts to interact with services
~/service/   # symlinks to currently running services
/var/log/    # log files
/var/www/    # currently running services
/var/db/     # database data, can be symlinked to separate mount

Adding configuration to a service

This treads into the territory of opinions, and thus caution is adviced. In my opinion services must be self-contained; which means they should carry their own config. If config is dynamic, that too should be carried.

Testing a service should ideally be done in a containerized environment, to fully mimick the OS situation. Else there's a discrepancy between the way the service is interacted with during development, and during production. The closer the two are, the better.

To hold configuration, a service should have a config/ directory in its root which would be overlaid 1:1 with the root directory (/). That way it's super clear where all the files go. Symlinks are always preferred over hard copies in this case; unless the application that's being instrumented is unable to follow symlinks.

Naming of services

It's convention to prefix services with either www-, api- or service-. Sometimes people will use www- to mark static websites, and api- or service- to mark more backendy type of systems, but that kind of separation can get messy fast. In the end, make a judgement call and monitor the services you have, and eventually you'll find a naming scheme that creates a suitable distinction.


Logs should go in /var/log/<service_name>.log. Logs should be rotated using logrotate(1). Add to /etc/logrotate.d/<service_name>:

/var/log/myapp.log {
  su root
  rotate 7

results matching ""

    No results matching ""