Advice on Potential Architecture

[evanlalo]

Overview

I have been tasked with rebuilding a key piece of software for my company. The software is deployed on individual PCs across the country and each location only needs to be concerned with its own data. The app itself is pretty straightforward and consists of 3 containers running in a swarm.

• A react app
• Django backend
• MariaDB

I have added other containers for monitoring like prometheus, cAdvisor, promtail etc...

One main requirement of the app is that it can run offline. Meaning, even if a location looses external internet, the customer can still do their work.

My solution to this was to run the application on locahost, so as long as an intranet exists at the customer site, the app will work. I have verified and tested this. What this means though, is that each location is running its own instance of the UI, backend and DB.

Just as a sidenote, I can see why a native app might seem better but we have some requirements that make a PWA the best solution.

Problems / Question

My main problem is how do I manage this? What's the best way to take care of deployments?

My first solution was to make each client location a master node. I would then have a cron job running on each PC to handle deployments. This actually works but I can see some major issues, especially once we scale up and it seems like a missed opportunity to take full advantage of everything docker swarm has to offer.

My second solution that I'm working on is have a master node(s) running on an EC2 instance and each location is a worker. This obviously makes deployments and management much simpler but I'm running into the following issues but only on the worker nodes.

1. I can't get UI to show on the worker nodes. I should be able to navigate to http://localhost:8080/ but I keep getting "refused to connect". If I type in the public IP of my EC2 I can connect just fine.
2. My backend intermittently drops connection to the database.

FWIW here's the yaml file I'm using for the app stack.

version: '3.7'

secrets:
  loki_url:
    external: true

volumes:
  mysql_data:
  prometheus_data:


networks:
  app:
    name: app
    driver: overlay
    attachable: true

services:
  db:
    image: mariadb:10.7.3
    networks:
      - app
    secrets:
      - loki_url
    logging:
      driver: loki
      options:
        loki-url: /run/secret/loki_url
        loki-retries: 5
        loki-batch-size: 400
    volumes:
      - mysql_data:/var/lib/mysql
    environment:
      - MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=yes
    env_file:
      - ../server/.env
    deploy:
      mode: global
      endpoint_mode: dnsrr

  django:
    depends_on:
      - db
    image: ${DJANGO_IMAGE}
    networks:
      - app
    secrets:
      - loki_url
    logging:
      driver: loki
      options:
        loki-url: /run/secret/loki_url
        loki-retries: 5
        loki-batch-size: 400
    environment:
      - CONTAINER=True
    env_file:
      - ../server/.env  
    deploy:
      mode: global

  frontend:
    depends_on:
      - django
    image: ${FRONTEND_IMAGE}
    networks:
      - app
    secrets:
      - loki_url
    logging:
      driver: loki
      options:
        loki-url: /run/secret/loki_url
        loki-retries: 5
        loki-batch-size: 400
    ports:
      - target: 80
        published: 8080
        protocol: tcp
        mode: host      
    deploy:
      mode: global

Conclusion

Any pointers, links or general advice would be a tremendous help. I have watched a lot of your talks and tutorials and I really appreciate your willingness to share your knowledge.

Regards,

[BretFisher]

Hey there, sorry for the delay, as I was on vacation.

Some thoughts, in no particular order, about the general design of these containers

• Regardless of Swarm or Kubernetes, they are not designed for a single cluster to operate across the internet (as in one node onsite and one node in cloud). All nodes in a cluster should be well connected on reliable networks with a goal of <10ms latency between the nodes. This means your solution will all need to be on each site (no hybrid).
• Swarm is very simple. I don't know of a solution that will help automate it remotely other than Portainer. If it can't do what you need, then I don't think there are other options for Swarm. Kubernetes is seeing much more innovation in this space of "many clusters, managed centrally."
• You said PC's as the install method on-prem. You'll want full Linux servers. When you say PC, I assume Windows 10/11, and Docker Desktop (or Rancher Desktop) aren't meant to be production and are only for a local dev/devops user to work in, and not designed for remote access from other users on the same network.
• Unless you can afford 3 physical servers on each site to provide cluster redundancy, then assume you'll have one node that is both cluster control plane and worker node in one, and just realize you don't have hardware fault tolerance.
• Is this a small app for only a few users? Do you not have Linux servers on each site yet? If it's a small app for under 20 people and needs less than 8GB memory, and you don't have existing infrastructure (or can't afford real server hardware on each site) then a good design would be to use a IoT-like design, using Raspberry Pi's or other solid-state mini machines, Kubernetes with K3s, and a set of tools designed to manage many K8s clusters across firewalls and many private networks. Here's just a quick sample of videos from the last year in the Kubernetes community to tackle these problems:
  https://www.youtube.com/watch?v=_NkuspFW9IQ
  https://www.youtube.com/watch?v=CMthyqMhuq4
  https://www.youtube.com/watch?v=9wCQCV0m5Kk

These ideas probably present a hundred other questions, so here's how I would move forward, having the limited info I have:

1. Try a single-node Swarm on a single Linux VM, with Portainer running centrally and the Portainer agent in the Swarm. Try two Swarms this way, with a single central Portainer and see if the setup works. Portainer even talks about IoT/Edge as a use case for it, so reach out to them if you have issues getting it to work in that design.
2. If that won't meet your needs, you'll need to start looking for ways to do a single-node k3s cluster per-site and pick the tools to manage this IoT/Edge centrally. I wouldn't worry too much about the increased complexity of K8s, as your app is simple enough that it won't be that much more complex than Swarm... it just might be new for you.

[evanlalo]

Thanks for getting back to me, I understand you do this pro-bono so no apology needed!

That's a bummer to hear that the EC2 route isn't a viable solution.

At any rate, you've given me a lot to think about.

Thanks again for the reply, much appreciated 👍