Making PHP Monitor actually testable

As a GUI that helps working with various existing software possible, the majority of PHP Monitor’s functionality is responsible for interacting with various external systems, like:

• Directly invoking commands via the command line (via /bin/sh)
• Directly launching (or terminating) processes (or via Homebrew)
• Indirectly affecting running processes by modifying configuration files on the local filesystem

These external systems are often either Valet or Homebrew, but these can also be configuration files related to PHP, nginx or dnsmasq. A lot of these interactions are relatively straightforward, in theory.

However, sometimes I get bug reports from users who tell me that the app behaves in a particular way, and reproducing the exact conditions can prove to be quite troublesome.

Unit Tests

The current test suite in PHP Monitor 5 is simply a bunch of basic unit tests. These tests help me verify that various core functionality works as expected.

This is usually functionality that is hard or time-consuming to manually check again and again. The other key features of the app I’d manually test before releasing an update.¹

Here’s one test that already exists today: the self-contained AppVersionTest class, which contains a test that verifies if the app’s version can be read from a Cask versioned string:

import XCTest

class AppVersionTest: XCTestCase {
    func testCanParseNormalVersionString() {
        let version = AppVersion.from("1.0.0")

        XCTAssertNotNil(version)
        XCTAssertEqual("1.0.0", version?.version)
        XCTAssertEqual(nil, version?.build)
        XCTAssertEqual(nil, version?.suffix)
    }

    func testCanParseCaskVersionString() {
        let version = AppVersion.from("1.0.0_600")

        XCTAssertNotNil(version)
        XCTAssertEqual("1.0.0", version?.version)
        XCTAssertEqual("600", version?.build)
        XCTAssertEqual(nil, version?.suffix)
    }
}

This test guarantees that the regex parsing of the version number specified in the Caskfile is handled correctly.

This way, the updater can determine if the publicly available version number is newer than what you’re currently running or not. (This must work for the app update notification to work correctly!)

This is what a Caskfile might look like:

cask 'phpmon' do
  depends_on formula: 'gnu-sed'

  version '5.6.3_980'
  sha256 '8fdfc79802bbf50c69a67afdc6a096991876940665bfb8db87781568c8d01f7f'

  url 'https://github.com/nicoverbruggen/phpmon/releases/download/v5.6.3/phpmon.zip'
  appcast 'https://github.com/nicoverbruggen/phpmon/releases.atom'
  name 'PHP Monitor'
  homepage 'https://phpmon.app'

  app 'PHP Monitor.app'
end

Dependency Hell

Now, these tests are great. They are the number one priority for testing: you can test complicated functionality in small chunks.

Unfortunately, sometimes we create our own (dependency) hell…

Here’s an example you’ve ended up in dependency hell: if testing a single class means you’ll need to instantiate many other classes just to be able to get the existing functionality to work.

Unfortunately, because PHP Monitor started as a hacky project and I did not do the proper refactoring in a timely fashion, the project suffers from this exact issue.²

For example, various key singletons are instantiated in the AppDelegate class (during initialization of the app), and some of these rely on an actual working Valet setup. Here’s what the initializer looks like:

@NSApplicationMain
class AppDelegate: NSObject, NSApplicationDelegate, UNUserNotificationCenterDelegate {
    override init() {
        Log.separator(as: .info)
        Log.info("PHP MONITOR by Nico Verbruggen")
        Log.info("Version \(App.version)")
        Log.separator(as: .info)

        self.state = App.shared
        self.menu = MainMenu.shared
        self.paths = Paths.shared
        self.valet = Valet.shared
        super.init()
    }
}

For example, one of the most used methods in PHP Monitor is Shell.pipe(). Sadly, Shell in PHP Monitor 5 is quite fragile and problematic. It invokes the user’s actual terminal, which may already be a non-starter.

Oh, and invoking Shell.pipe() can happen pretty much anywhere in the app. For example, during the boot sequence, where there are many checks such as this one, which checks if there’s a valid PHP installation on the system:

EnvironmentCheck(
    command: {
        // This is the actual check that occurs when the app is booting.
        // At this point, the `Paths` singleton need to be instantiated.
        return !Shell.pipe("ls \(Paths.optPath) | grep php").contains("php") 
    },
    name: "`ls \(Paths.optPath) | grep php` returned php result",
    titleText: "startup.errors.php_opt.title".localized,
    subtitleText: "startup.errors.php_opt.subtitle".localized(Paths.optPath),
    descriptionText: "startup.errors.php_opt.desc".localized
),

As you can see in the example above: in order to be able to run various commands, the Shell class may also make use of (depend on) the Paths class, which attempts to find out where one’s Homebrew installation is located, and what the user’s username is, for example.

Having a functional instance of this Paths singleton is necessary in order to be able to construct all the absolute paths in the app.

Initializing the Paths singleton may be unnecessary when running particular (simpler) unit tests. Some tests may wish to make use of the Paths singleton. In that case, for tests to work, the tests would have to get output from the actual system the tests are running on. That is obviously an issue for unit tests!

Currently, this means that you need to have your environment set up in a particular way (have Homebrew, Valet, PHP, etc.) installed in order to have various tests pass. Here’s a real test from PHP Monitor 5’s codebase:

// - MARK: LIVE TESTS

/// This test requires that you have a valid Homebrew installation set up,
/// and requires the Valet services to be installed: php, nginx and dnsmasq.
/// If this test fails, there is an issue with your Homebrew installation
/// or the JSON API of the Homebrew output may have changed.
func testCanParseServicesJsonFromCliOutput() throws {
    let services = try! JSONDecoder().decode(
        [HomebrewService].self,
        from: Shell.pipe(
            "sudo \(Paths.brew) services info --all --json",
            requiresPath: true
        ).data(using: .utf8)!
    ).filter({ service in
        return ["php", "nginx", "dnsmasq"].contains(service.name)
    })

    XCTAssertTrue(services.contains(where: {$0.name == "php"}))
    XCTAssertTrue(services.contains(where: {$0.name == "nginx"}))
    XCTAssertTrue(services.contains(where: {$0.name == "dnsmasq"}))
    XCTAssertEqual(services.count, 3)
}

Not only is this not ideal (interacting with the user’s real terminal may be slow!) but it also means that I can only test a few things without breaking the user’s own system when they run the unit tests.

Such a thing would be unacceptable, so in PHP Monitor 5’s codebase, unit tests that deal with the user’s actual shell don’t do anything that may alter the system configuration. (These are effectively read-only tests.)

In the future, I may wish to add a test that unlinks the active PHP version via Homebrew (effectively running brew unlink php via PHP Monitor). That test should not affect the user’s real environment!

The same issue applies to the filesystem: the app can read and/or modify various config files! The same caveat applies: I would not want a unit test to affect the local filesystem.

It sure sounds as if I want to be able to run various commands and do various filesystem operations, check whether an operation has occurred, but not affect the system configuration. If you’ve ever written a bunch of unit tests before, you probably know what’s coming next…

Faking It

The solution is simple. All of the key systems that the app interacts with, must also be allowed to be replaced with test versions. This means:

• A shell that always responds to a given command with a given response
• A filesystem that contains fake files and directories, and doesn’t write to the system’s actual storage
• Interaction with various binaries should also be possible via pre-fabbed responses (i.e. if I run /opt/homebrew/bin/php --version I will get some output as a result without actually running the binary)

So, in essense, I’d need:

• A shell that can either invoke a real shell command
  OR doesn’t actually run, but returns a predetermined response
• A system that can run a given binary and returns the output
  OR doesn’t actually run, but returns a predetermined response
• A filesystem that supports common filesystem operations that run locally
  OR on a set of in-memory “files” and “directories”

At runtime, I can then swap out the actual classes that interact with the system shell, filesystem and commands and swap them for fake instances which always produce the same output or represent some sort of virtualized in-memory system (like the fake filesystem).

Here’s what one such interface looks like:

protocol FileSystemProtocol {
    func createDirectory(_ path: String, withIntermediateDirectories: Bool) throws
    func writeAtomicallyToFile(_ path: String, content: String) throws
    func getStringFromFile(_ path: String) throws -> String
    func getShallowContentsOfDirectory(_ path: String) throws -> [String]
    func getDestinationOfSymlink(_ path: String) throws -> String

    func move(from path: String, to newPath: String) throws
    func remove(_ path: String) throws

    func makeExecutable(_ path: String) throws

    func isExecutableFile(_ path: String) -> Bool
    func isWriteableFile(_ path: String) -> Bool
    func anyExists(_ path: String) -> Bool
    func fileExists(_ path: String) -> Bool
    func directoryExists(_ path: String) -> Bool
    func isSymlink(_ path: String) -> Bool
    func isDirectory(_ path: String) -> Bool
}

This interface is implemented in two actual classes, RealFileSystem and TestableFileSystem. These classes can be swapped at will via the ActiveFileSystem class:

var FileSystem: FileSystemProtocol {
    return ActiveFileSystem.shared
}

class ActiveFileSystem {
    static var shared: FileSystemProtocol = RealFileSystem()

    public static func useTestable(_ files: [String: FakeFile]) {
        Self.shared = TestableFileSystem(files: files)
    }

    public static func useSystem() {
        Self.shared = RealFileSystem()
    }
}

Because of this helper class, swapping them and using them is super simple. It looks something like this:

// Use the system's real filesystem.
ActiveFileSystem.useSystem()

// Use a fake filesystem from now on.
ActiveFileSystem.useTestable([
    "/usr/local/bin/": .fake(.directory, readOnly: true),
    "/usr/local/bin/composer": .fake(.binary),
    "/opt/homebrew/bin/brew": .fake(.binary),
    "/opt/homebrew/bin/php": .fake(.binary),
    "/opt/homebrew/bin/valet": .fake(.binary),
])

// We can do tests now that check for the existence of a given binary that may or will not exist on the actual system.
XCTAssertTrue(FileSystem.isExecutableFile("/opt/homebrew/bin/php"))

In the example here, our test will always pass because the fake filesystem is used. If we used the user’s own system shell, they might not have the binary installed, or they might be on Intel (which has the Homebrew directory in a different place).

By simply replacing these three systems, we can now create a configuration file that contains all the expected output for a working installation of PHP Monitor, and test all the functionality of the app.

Configurations

To accomplish this, I’ve got a class that allows me to set up all of these systems: TestableConfiguration. I can set the outcome of various shell commands, and define what binaries are available on the target system (among other things).

Here’s what one such configuration instance might look like. Entries for each argument have been truncated because the actual configuration instance is quite big due to sheer number of files and commands affected:

return TestableConfiguration(
    architecture: "arm64",
    filesystem: [
        "/usr/local/bin/" : .fake(.directory, readOnly: true),
        "/usr/local/bin/composer" : .fake(.binary),
    ],
    shellOutput: [
        "sysctl -n sysctl.proc_translated" : .instant("0"),
        "id -un" : .instant("user"),
        "which node" : .instant("/opt/homebrew/bin/node"),
        "ls /opt/homebrew/opt | grep php" : .instant("php"),
    ],
    commandOutput: [
        "/opt/homebrew/bin/php-config --version": "8.1.10",
        "/opt/homebrew/bin/php -r echo ini_get('memory_limit');": "512M",
        "/opt/homebrew/bin/php -r echo ini_get('upload_max_filesize');": "512M",
        "/opt/homebrew/bin/php -r echo ini_get('post_max_size');": "512M",
    ]
)

When using this TestableConfiguration, running various commands will return the values that have been predetermined. This allows for easy test reproducibility.

Due to its sheer size, the file where the actual full-fledged TestableConfiguration instance is created (with all of its many large array arguments) is not part of the main target. This is done to 1) avoid a direct dependency and 2) to ensure that the main production binary does not get too big.

To actually make use of the preset, I can run a test that copies all of the configuration values to a JSON file:

class TestableConfigurationTest: XCTestCase {
    func test_configuration_can_be_saved_as_json() async {
        // Start off with a working configuration preset.
        var configuration = TestableConfigurations.working
  
        // Write the working configuration to the user's home directory.
        try! configuration.toJson()
             .write(toFile: NSHomeDirectory() + "/.phpmon_fconf_working.json", atomically: true, encoding: .utf8)

        // For the broken configuration, remove the PHP binary!
        configuration.filesystem["/opt/homebrew/bin/php"] = nil

        try! configuration.toJson()
             .write(toFile: NSHomeDirectory() + "/.phpmon_fconf_broken.json", atomically: true, encoding: .utf8)
    }
}

At that point, I can simply add the JSON file as an argument when launching the app, at which point the app will load the TestableConfiguration from the JSON file and use it as the basis for all of its functionality, allowing me to test various scenarios without touching my actual Valet and PHP setup:

/path/to/phpmon --configuration:~/.phpmon_fconf_working.json

Loading these configuration files only works for debug builds, but that is good enough for debugging purposes. I ensure that configuration files cannot be loaded on production builds by using a #if DEBUG flag:

#if DEBUG
if let profile = CommandLine.arguments.first(where: { $0.matches(pattern: "--configuration:*") }) {
    Self.initializeTestingProfile(profile.replacingOccurrences(of: "--configuration:", with: ""))
}
#endif

With all of these systems in place, I can:

• Set up predefined configuration files that represent a system configuration state
• Leverage said system configuration state when launching the app
• Not touch the user’s actual production setup when running tests
• Speed up (or slow down) the output of various fake (shell/FS) operations
• Run various unit, feature and UI tests efficiently (with no-delay terminal output for maximum speed)

So far, so good.

The source code of PHP Monitor 6 will be made available when PHP Monitor 6 is released. I figured I’d share some of my work-in-progress code here today.