One good way to monitor your container status is to use Docker’s HEALTHCHECK feature.

As part of testing the followed actions upon an unhealthy container, I had to experiment how this works. I wrote a small Go app that replies to /health requests, and the status it responds is configurable.

The webserver listens on $PORT (defaults to 8080). To change its status, simply make an api call to one of the supported actions. You can either connect the container to execute it, or if you have mapped a port on the host, you can use it.

Supported actions are:

• /sabotage will make it respond with 500.
• /timeout will make it respond after 20s.
• /recover will return it back to healthy state, with 200 response code.

You can find the source code here.

From the Docker docs -

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

The HEALTHCHECK instruction has two forms:

    HEALTHCHECK [OPTIONS] CMD command (check container health by running a command inside the container)
    HEALTHCHECK NONE (disable any healthcheck inherited from the base image)

The HEALTHCHECK instruction tells Docker how to test a container to check that it is still working. 
This can detect cases such as a web server that is stuck in an infinite loop and unable 
to handle new connections, even though the server process is still running.

When a container has a healthcheck specified, it has a health status in addition to its normal status. 
This status is initially starting. Whenever a health check passes, it becomes healthy (whatever state 
it was previously in). After a certain number of consecutive failures, it becomes unhealthy.

The options that can appear before CMD are:

--interval=DURATION (default: 30s)
--timeout=DURATION (default: 30s)
--start-period=DURATION (default: 0s)
--retries=N (default: 3)

You can configure the HEALTHCHECK settings in the Dockerfile or the docker-compose.yml. In my example, I use it in the Dockerfile.

Demo

1. Clone the repo and build the image: docker build . -t go-healthchecker

2. Bring up the container, wait until it gets healthy.

3. To change the healthcheck response, connect to the container and update its status