Deploying a Private Routing Environment With Docker-Compose
Hylke Visser

TTN Core Team

Posted on 21-12-2016

After setting up The Things Network's routing services in a local or private environment as described in the previous article, we will now look at what changes are needed to deploy those routing services using Docker and Docker-Compose.

NOTE: This guide applies to the previous version (v2) of our network stack. To get started with the current version of our network stack (v3), go to ttn.fyi/v3/getting-started.

In this guide we will modify the configuration that we created in the previous guide, and use that to run the routing services as (component).local.thethings.network. If you have your own (sub)domains, that point to the IP address of your server, you can also use those.

Preparation

Additional to the prepartion steps from the previous guide, we have to

  • Install and start Docker.
  • Install Docker-Compose.
  • Pull the latest mosquitto Docker image: docker pull eclipse-mosquitto`
  • Pull the latest ttn Docker image: docker pull thethingsnetwork/ttn:latest.
  • Pull the latest gateway-connector-bridge Docker image: docker pull thethingsnetwork/gateway-connector-bridge:latest.
  • Stop redis and mosquitto if running.

We will again work from the same working directory as we did before (~/ttn). All commands are executed from this directory. In Docker, however, we will put the configuration in /etc/ttn/router/, /etc/ttn/broker/, etc.

Changing the Paths

Because we will use different directories for configuration of the routing services in Docker, we have to change a number of things in the configuration files from before.

The key-dir: "(component)" configuration option in all (component) folders, should be changed to /etc/ttn/(component), so for example, for the Router, the first part of the configuration should become:

id: mynetwork-router
tls: true
key-dir: /etc/ttn/router/
auth-servers:
  ttn-account-v2: "https://account.thethingsnetwork.org"

You should do the same for all other ttn.yml configuration files.

As we added a local: "file://discovery/server.pub" account server to the configuration of the Discovery server, we now have to change this to local: "file:///etc/ttn/discovery/server.pub" (note the extra /). Similarly, the Broker has a networkserver-cert: broker/networkserver.cert. This should become networkserver-cert: /etc/ttn/broker/networkserver.cert.

Changing the Hosts

We will no longer run the services on localhost, but we'll use hostnames in the form (component).local.thethings.network. We have to change this in the configuration to make it work.

  • In all configurations, change discovery-address: "localhost:1900" to discovery-address: "discovery.local.thethings.network:1900".
  • In the router/ttn.yml configuration, change the server-address-announce to router.local.thethings.network.
  • In the broker/ttn.yml configuration, change the server-address-announce to broker.local.thethings.network.
  • In the handler/ttn.yml configuration, change the server-address-announce to handler.local.thethings.network.
  • In the broker/ttn.yml configuration, change the networkserver-address to networkserver.local.thethings.network.

Setting Configuration for Redis and Mosquitto

Because Redis and Mosquitto will now be running in Docker, we can no longer use localhost, but have to use the hostname of the container instead. We will run Mosquitto as mosquitto and Redis as redis, so we have to point our routing services to those hostnames.

For Redis, we have to add a configuration option to the Discovery server, NetworkServer and Handler configurations.

  • In discovery/ttn.yml add redis-address: redis:6379 (including two leading spaces) below the discovery: line.
  • In handler/ttn.yml add redis-address: redis:6379 (including two leading spaces) below the handler: line.
  • In networkserver/ttn.yml add the following lines:
networkserver:
  redis-address: redis:6379

For MQTT and AMQP, change the mqtt-address and amqp-address to use mosquitto instead of localhost in the handler/ttn.yml configuration.

Generating new certificates

Because we will now reach the routing services at different hostnames, we have to generate new certificates that allow these new hostnames. You don't need to generate new keypairs and you don't need to generate new access tokens. The ones generated for your localhost setup will still work.

  • ttn discovery gen-cert localhost discovery discovery.local.thethings.network --config ./discovery/ttn.yml --key-dir ./discovery
    • This will be valid for localhost, discovery and discovery.local.thethings.network.
  • ttn router gen-cert localhost router --config ./router/ttn.yml --key-dir ./router
    • This will be valid for localhost, router and router.local.thethings.network (the last one comes from the configuration).
  • ttn broker gen-cert localhost broker --config ./broker/ttn.yml --key-dir ./broker
    • This will be valid for localhost, broker and broker.local.thethings.network.
  • ttn networkserver gen-cert localhost networkserver networkserver.local.thethings.network --config ./networkserver/ttn.yml --key-dir ./networkserver
    • This will be valid for localhost, networkserver and networkserver.local.thethings.network.
  • ttn handler gen-cert localhost handler --config ./handler/ttn.yml --key-dir ./handler
    • This will be valid for localhost, handler and handler.local.thethings.network.

The discovery server's new certificate is needed by the Router, Broker, Handler, Bridge and by ttnctl, so we add this to the trusted certificates:

cat discovery/server.cert > router/ca.cert
cat discovery/server.cert > broker/ca.cert
cat discovery/server.cert > handler/ca.cert
cat discovery/server.cert > bridge/ca.cert
cat networkserver/server.cert > broker/networkserver.cert
cat discovery/server.cert > ~/.ttnctl/ca.cert

If ttnctl uses $XDG_DATA_HOME/ttnctl or $XDG_CACHE_HOME/ttnctl, you should use those.

Docker-Compose

We start with a clean docker-compose.yml file that contains two services: redis and mosquitto

version: '2'

services:
  redis:
    image: redis
    command: redis-server --appendonly yes
    hostname: redis
    ports:
      - "6379:6379" # Note: you should not expose this port in production environments
    volumes:
      - /data
  mosquitto:
    image: eclipse-mosquitto
    restart: always
    mem_limit: 64m
    ports:
      - "1883:1883"  # Note: your MQTT broker should use authentication in production environments
      - "9001:9001"  # Note: you should not expose this port in production environments
    environment:
      MQTTNAUTH_CACHE_HOST: redis

We can now start these services:

docker-compose up -d

Now we can add the different ttn services. These should be at the same level as redis and mosquitto.

  discovery:
    image: thethingsnetwork/ttn:latest
    command: discovery --config /etc/ttn/discovery/ttn.yml
    depends_on:
      - redis
    networks:
      default:
        aliases:
          - discovery.local.thethings.network
    ports:
      - "1900:1900"
      - "8080:8080"
    volumes:
      - "./discovery:/etc/ttn/discovery"
  router:
    image: thethingsnetwork/ttn:latest
    command: router --config /etc/ttn/router/ttn.yml
    depends_on:
      - discovery
    networks:
      default:
        aliases:
          - router.local.thethings.network
    ports:
      - "1901:1901"
    volumes:
      - "./router:/etc/ttn/router"
  broker:
    image: thethingsnetwork/ttn:latest
    command: broker --config /etc/ttn/broker/ttn.yml
    depends_on:
      - discovery
      - networkserver
    networks:
      default:
        aliases:
          - broker.local.thethings.network
    ports:
      - "1902:1902"
    volumes:
      - "./broker:/etc/ttn/broker"
  networkserver:
    image: thethingsnetwork/ttn:latest
    command: networkserver --config /etc/ttn/networkserver/ttn.yml
    depends_on:
      - redis
    networks:
      default:
        aliases:
          - networkserver.local.thethings.network
    ports:
      - "1903:1903" # Note: you should not expose this port in production environments
    volumes:
      - "./networkserver:/etc/ttn/networkserver"
  handler:
    image: thethingsnetwork/ttn:latest
    command: handler --config /etc/ttn/handler/ttn.yml
    depends_on:
      - discovery
      - redis
      - rabbitmq
    networks:
      default:
        aliases:
          - handler.local.thethings.network
    ports:
      - "1904:1904"
      - "8084:8084"
    volumes:
      - "./handler:/etc/ttn/handler"

Now we start the discovery server:

docker-compose up -d discovery

Just as in the previous guide we have to register the device address prefix:

docker-compose run broker broker register-prefix 26000000/20 --config /etc/ttn/broker/ttn.yml

Because the bridge does not use configuration files, we will configure its settings with environment variables in our docker-compose.yml:

  bridge:
    image: thethingsnetwork/gateway-connector-bridge:latest
    depends_on:
      - router
    environment:
      BRIDGE_TTN_DISCOVERY_SERVER: discovery.local.thethings.network:1900
      BRIDGE_ROOT_CA_FILE: /etc/ttn/bridge/ca.cert
      BRIDGE_TTN_ROUTER: mynetwork-router
      BRIDGE_REDIS_ADDRESS: redis:6379
      BRIDGE_MQTT: mosquitto:1883
      BRIDGE_AQMP: disable
    ports:
      - "1700:1700/udp"
    volumes:
      - "./bridge:/etc/ttn/bridge"

Starting Everything Together

Now run docker-compose up -d to start all services.

ttnctl

In order to use ttnctl with the services that are now running in Docker, you have to add the following to your /etc/hosts file:

127.0.0.1 discovery.local.thethings.network router.local.thethings.network handler.local.thethings.network

You can also update discovery-address and mqtt-address in your ttnctl.yml configuration (although this is not strictly necessary).

Questions or Issues

If you have questions or remarks after following this guide, feel free use the forum or the #private-backend channels on Slack. This is a community-supported guide, so please help each other out.

What's Next

You should now have a fully operational private backend deployed in Docker. Some next steps:

  • Add restart: always to your docker-compose.yml to automatically restart the services on reboots or system crashes.
  • Periodically update all Docker images to the latest version
  • Configure backups
  • Secure the MQTT/AMQP broker with usernames and passwords
  • Help with development of TTN's open source routing services