Support (and set) launchd user domain target

When running `brew services` over `ssh` or `sudo` as a non-`root` user
the `gui` domain target for `launchd` does not work as expected. This
produces confusing failures for users.

It was tried before to detect if this was needed from the `.plist` or
changing globally but both approaches were unsuccessful/error-prone.

Instead, use the `user` domain automatically on the cases we know it's
needed: when running over `ssh` or through `sudo`. To make clear to
users what's happening in these cases: output a warning (which can
be hidden with an output environment variable).

For this to work, `launchctl list` is no longer sufficient. The output
here, even when run as `root`, does not properly list `user` domain
services. Instead, we need to use `launchctl print` to correctly
query the status of these services. This also seems to correctly
handle some `launchd` edge-cases where launched services cannot be
detected and lets `brew services` now stop them.

While we're here, fix some related issues I came upon while working on
this:
- improve the `brew services` command documentation to note it now
  runs on Linux/`systemd` too
- use `named_args` and remove the `custom_plist` deprecation: it's
  been long enough and this cleans up the code nicely.
- if a service is status `:other` (an edge-case that shouldn't be
  possible in normal operation): actually output this rather than
  failing with a `nil` error
- avoid running `launchctl list` multiple times when unnecessary
- `brew services --debug` now outputs the raw output from `launchctl`
  or `systemctl` to aid debugging/development
- cleanup some code style and avoid use of `plist?`
- add `System.systemctl_args` to avoid repeating `System.systemctl`
  and `System.systemctl_scope` in every call site

cmd/services.rb

@@ -10,13 +10,15 @@ def services_args
       usage_banner <<~EOS
         `services` [<subcommand>]
 
-        Manage background services with macOS' `launchctl`(1) daemon manager.
+        Manage background services with macOS' `launchctl`(1) daemon manager or
+        Linux's `systemctl`(1) service manager.
 
-        If `sudo` is passed, operate on `/Library/LaunchDaemons` (started at boot).
-        Otherwise, operate on `~/Library/LaunchAgents` (started at login).
+        If `sudo` is passed, operate on `/Library/LaunchDaemons`/`/usr/lib/systemd/system`  (started at boot).
+        Otherwise, operate on `~/Library/LaunchAgents`/`~/.config/systemd/user` (started at login).
 
-        [`sudo`] `brew services` [`list`] (`--json`):
+        [`sudo`] `brew services` [`list`] (`--json`) (`--debug`):
         List information about all managed services for the current user (or root).
+        Provides more output from Homebrew and `launchctl`(1) or `systemctl`(1) if run with `--debug`.
 
         [`sudo`] `brew services info` (<formula>|`--all`|`--json`):
         List all managed services for the current user (or root).
@@ -42,6 +44,7 @@ def services_args
       flag "--file=", description: "Use the service file from this location to `start` the service."
       switch "--all", description: "Run <subcommand> on all services."
       switch "--json", description: "Output as JSON."
+      named_args min: 1, max: 2
     end
   end
 
@@ -64,13 +67,7 @@ def services
     end
 
     # Parse arguments.
-    subcommand, formula, custom_plist, = args.named
-
-    if custom_plist.present?
-      odeprecated "with file as last argument", "`--file=` to specify a plist file"
-    else
-      custom_plist = args.file
-    end
+    subcommand, formula, = args.named
 
     if [*::Service::Commands::List::TRIGGERS, *::Service::Commands::Cleanup::TRIGGERS].include?(subcommand)
       raise UsageError, "The `#{subcommand}` subcommand does not accept a formula argument!" if formula
@@ -105,11 +102,11 @@ def services
     when *::Service::Commands::Info::TRIGGERS
       ::Service::Commands::Info.run(targets, verbose: args.verbose?, json: args.json?)
     when *::Service::Commands::Restart::TRIGGERS
-      ::Service::Commands::Restart.run(targets, custom_plist, verbose: args.verbose?)
+      ::Service::Commands::Restart.run(targets, args.file, verbose: args.verbose?)
     when *::Service::Commands::Run::TRIGGERS
       ::Service::Commands::Run.run(targets, verbose: args.verbose?)
     when *::Service::Commands::Start::TRIGGERS
-      ::Service::Commands::Start.run(targets, custom_plist, verbose: args.verbose?)
+      ::Service::Commands::Start.run(targets, args.file, verbose: args.verbose?)
     when *::Service::Commands::Stop::TRIGGERS
       ::Service::Commands::Stop.run(targets, verbose: args.verbose?)
     when *::Service::Commands::Kill::TRIGGERS

lib/service/commands/list.rb

@@ -73,6 +73,7 @@ def get_status_string(status)
         when :stopped, :none then "#{Tty.default}#{status}#{Tty.reset}"
         when :error   then "#{Tty.red}error  #{Tty.reset}"
         when :unknown then "#{Tty.yellow}unknown#{Tty.reset}"
+        when :other then "#{Tty.yellow}other#{Tty.reset}"
         end
       end
     end

lib/service/formula_wrapper.rb

@@ -98,13 +98,15 @@ def plist?
     end
 
     # Returns `true` if the service is loaded, else false.
-    def loaded?
-      if System.launchctl?
-        # TODO: find replacement for deprecated "list"
-        quiet_system System.launchctl, "list", service_name
-      elsif System.systemctl?
-        quiet_system System.systemctl, System.systemctl_scope, "status", service_file.basename
+    def loaded?(cached: false)
+      @status_output_success_type = nil unless cached
+      service_name = if System.launchctl?
+        self.service_name
+      else
+        service_file.basename
       end
+      _, status_success, = status_output_success_type(service_name)
+      status_success
     end
 
     # Returns `true` if service is present (e.g. .plist is present in boot or user service path), else `false`
@@ -137,30 +139,32 @@ def error?
     end
 
     def unknown_status?
-      status.blank? && !pid?
+      status_output.blank? && !pid?
     end
 
     # Get current PID of daemon process from status output.
     def pid
-      return Regexp.last_match(1).to_i if status =~ pid_regex
+      status_output, _, status_type = status_output_success_type
+      return Regexp.last_match(1).to_i if status_output =~ pid_regex(status_type)
     end
 
     # Get current exit code of daemon process from status output.
     def exit_code
-      return Regexp.last_match(1).to_i if status =~ exit_code_regex
+      status_output, _, status_type = status_output_success_type
+      return Regexp.last_match(1).to_i if status_output =~ exit_code_regex(status_type)
     end
 
     def to_hash
       hash = {
         name:         name,
         service_name: service_name,
         running:      pid?,
-        loaded:       loaded?,
+        loaded:       loaded?(cached: true),
         schedulable:  timed?,
         pid:          pid,
         exit_code:    exit_code,
         user:         owner,
-        status:       operational_status,
+        status:       status_symbol,
         file:         service_file_present? ? dest : service_file,
       }
 
@@ -192,10 +196,39 @@ def load_service
       formula.service
     end
 
-    def operational_status
+    def status_output_success_type(service_name = self.service_name)
+      @status_output_success_type ||= if System.launchctl?
+        cmd = [System.launchctl.to_s, "list", service_name]
+        output = Utils.popen_read(*cmd).chomp
+        if $CHILD_STATUS.present? && $CHILD_STATUS.success? && output.present?
+          success = true
+          odebug cmd.join(" "), output
+          [output, success, :launchctl_list]
+        else
+          cmd = [System.launchctl.to_s, "print", "#{System.domain_target}/#{service_name}"]
+          output = Utils.popen_read(*cmd).chomp
+          success = $CHILD_STATUS.present? && $CHILD_STATUS.success? && output.present?
+          odebug cmd.join(" "), output
+          [output, success, :launchctl_print]
+        end
+      elsif System.systemctl?
+        cmd = [*System.systemctl_args, "status", service_name]
+        output = Utils.popen_read(*cmd).chomp
+        success = $CHILD_STATUS.present? && $CHILD_STATUS.success? && output.present?
+        odebug cmd.join(" "), output
+        [output, success, :systemctl]
+      end
+    end
+
+    def status_output
+      status_output, = status_output_success_type
+      status_output
+    end
+
+    def status_symbol
       if pid?
         :started
-      elsif !loaded?
+      elsif !loaded?(cached: true)
         :none
       elsif exit_code.present? && exit_code.zero?
         if timed?
@@ -212,29 +245,22 @@ def operational_status
       end
     end
 
-    def status
-      @status ||= if System.launchctl?
-        Utils.popen_read(System.launchctl, "list", service_name).chomp
-      elsif System.systemctl?
-        Utils.popen_read(System.systemctl.to_s, System.systemctl_scope.to_s, "status",
-                         service_name.to_s).chomp
-      end
-    end
-
-    def exit_code_regex
-      if System.launchctl?
-        /"LastExitStatus"\ =\ ([0-9]*);/
-      elsif System.systemctl?
-        /\(code=exited, status=([0-9]*)\)|\(dead\)/
-      end
+    def exit_code_regex(status_type)
+      @exit_code_regex ||= {
+        launchctl_list:  /"LastExitStatus"\ =\ ([0-9]*);/,
+        launchctl_print: /last exit code = ([0-9]+)/,
+        systemctl:       /\(code=exited, status=([0-9]*)\)|\(dead\)/,
+      }
+      @exit_code_regex.fetch(status_type)
     end
 
-    def pid_regex
-      if System.launchctl?
-        /"PID"\ =\ ([0-9]*);/
-      elsif System.systemctl?
-        /Main PID: ([0-9]*) \((?!code=)/
-      end
+    def pid_regex(status_type)
+      @pid_regex ||= {
+        launchctl_list:  /"PID"\ =\ ([0-9]*);/,
+        launchctl_print: /pid = ([0-9]+)/,
+        systemctl:       /Main PID: ([0-9]*) \((?!code=)/,
+      }
+      @pid_regex.fetch(status_type)
     end
 
     def boot_path_service_file_present?

lib/service/formulae.rb

@@ -9,7 +9,10 @@ module Formulae
     def available_services
       require "formula"
 
-      Formula.installed.map { |formula| FormulaWrapper.new(formula) }.select(&:plist?).sort_by(&:name)
+      Formula.installed
+             .map { |formula| FormulaWrapper.new(formula) }
+             .select(&:service?)
+             .sort_by(&:name)
     end
 
     # List all available services with status, user, and path to the file.

lib/service/services_cli.rb

@@ -14,10 +14,9 @@ def bin
     # Find all currently running services via launchctl list or systemctl list-units.
     def running
       if System.launchctl?
-        # TODO: find replacement for deprecated "list"
-        Utils.popen_read("#{System.launchctl} list | grep homebrew")
+        Utils.popen_read(System.launchctl, "list")
       else
-        Utils.popen_read(System.systemctl, System.systemctl_scope, "list-units",
+        Utils.popen_read(*System.systemctl_args, "list-units",
                          "--type=service",
                          "--state=running",
                          "--no-pager",
@@ -126,6 +125,9 @@ def stop(targets, verbose: false)
               Service `#{service.name}` is started as `#{service.owner}`. Try:
                 #{"sudo " unless System.root?}#{bin} stop #{service.name}
             EOS
+          elsif System.launchctl? &&
+                quiet_system(System.launchctl, "bootout", "#{System.domain_target}/#{service.service_name}")
+            ohai "Successfully stopped `#{service.name}` (label: #{service.service_name})"
           else
             opoo "Service `#{service.name}` is not started."
           end
@@ -134,7 +136,7 @@ def stop(targets, verbose: false)
 
         puts "Stopping `#{service.name}`... (might take a while)"
         if System.systemctl?
-          quiet_system System.systemctl, System.systemctl_scope, "disable", "--now", service.service_name
+          quiet_system(*System.systemctl_args, "disable", "--now", service.service_name)
         elsif System.launchctl?
           quiet_system System.launchctl, "bootout", "#{System.domain_target}/#{service.service_name}"
           while $CHILD_STATUS.to_i == 9216 || service.loaded?
@@ -146,7 +148,7 @@ def stop(targets, verbose: false)
 
         rm service.dest if service.dest.exist?
         # Run daemon-reload on systemctl to finish unloading stopped and deleted service.
-        safe_system System.systemctl, System.systemctl_scope, "daemon-reload" if System.systemctl?
+        safe_system(*System.systemctl_args, "daemon-reload") if System.systemctl?
 
         if service.pid? || service.loaded?
           opoo "Unable to stop `#{service.name}` (label: #{service.service_name})"
@@ -166,7 +168,7 @@ def kill(targets, verbose: false)
         else
           puts "Killing `#{service.name}`... (might take a while)"
           if System.systemctl?
-            quiet_system System.systemctl, System.systemctl_scope, "stop", service.service_name
+            quiet_system(*System.systemctl_args, "stop", service.service_name)
           elsif System.launchctl?
             quiet_system System.launchctl, "stop", "#{System.domain_target}/#{service.service_name}"
           end
@@ -250,8 +252,8 @@ def launchctl_load(service, file:, enable:)
     end
 
     def systemd_load(service, enable:)
-      safe_system System.systemctl, System.systemctl_scope, "start", service.service_name
-      safe_system System.systemctl, System.systemctl_scope, "enable", service.service_name if enable
+      safe_system(*System.systemctl_args, "start", service.service_name)
+      safe_system(*System.systemctl_args, "enable", service.service_name) if enable
     end
 
     def service_load(service, enable:)
@@ -299,7 +301,7 @@ def install_service_file(service, file)
 
       chmod 0644, service.dest
 
-      safe_system System.systemctl, System.systemctl_scope, "daemon-reload" if System.systemctl?
+      safe_system(*System.systemctl_args, "daemon-reload") if System.systemctl?
     end
   end
 end

lib/service/system.rb

@@ -31,6 +31,11 @@ def systemctl_scope
       root? ? "--system" : "--user"
     end
 
+    # Arguments to run systemctl.
+    def systemctl_args
+      @systemctl_args ||= [systemctl, systemctl_scope]
+    end
+
     # Woohoo, we are root dude!
     def root?
       Process.uid.zero?
@@ -75,6 +80,20 @@ def path
     def domain_target
       if root?
         "system"
+      elsif (ssh_tty = ENV.fetch("HOMEBREW_SSH_TTY", nil).present?) || ENV.fetch("HOMEBREW_SUDO_USER", nil).present?
+        if @output_warning.blank? && ENV.fetch("HOMEBREW_SERVICES_NO_DOMAIN_WARNING", nil).blank?
+          if ssh_tty
+            opoo "running over SSH, using user/* instead of gui/* domain!"
+          else
+            opoo "running through sudo, using user/* instead of gui/* domain!"
+          end
+          unless Homebrew::EnvConfig.no_env_hints?
+            puts "Hide this warning by setting HOMEBREW_SERVICES_NO_DOMAIN_WARNING."
+            puts "Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`)."
+          end
+          @output_warning = true
+        end
+        "user/#{Process.uid}"
       else
         "gui/#{Process.uid}"
       end

spec/homebrew/commands/list_spec.rb

@@ -129,7 +129,7 @@
     end
 
     it "returns other" do
-      expect(described_class.get_status_string(:other)).to be_nil
+      expect(described_class.get_status_string(:other)).to eq("<YELLOW>other<RESET>")
     end
   end
 end

spec/homebrew/formula_wrapper_spec.rb

@@ -119,14 +119,14 @@
     it "macOS - outputs if the service is loaded" do
       allow(Service::System).to receive(:launchctl?).and_return(true)
       allow(Service::System).to receive(:systemctl?).and_return(false)
-      allow(service).to receive(:quiet_system).and_return(false)
+      allow(Utils).to receive(:safe_popen_read)
       expect(service.loaded?).to be(false)
     end
 
     it "systemD - outputs if the service is loaded" do
       allow(Service::System).to receive(:launchctl?).and_return(false)
       allow(Service::System).to receive(:systemctl?).and_return(true)
-      allow(service).to receive(:quiet_system).and_return(false)
+      allow(Utils).to receive(:safe_popen_read)
       expect(service.loaded?).to be(false)
     end