When something is wrong

Start here:

dqx doctor

It reports what is broken in this project and what to do about it - not a health score, a list of findings with severities and remedies:

dqx doctor

router         ✓ running, 6 domain(s) certified
dns            ✓ *.test: resolves to 127.0.0.1
certificate    ✓ covers harbor-books.dqx and 5 more, 89 days left
database       ! the config declares mysql 8.0 and the stack runs 8.4
queue          ! QUEUE_CONNECTION=redis and nothing is consuming it - declare a daemon, or set it to sync
docroot        ✗ www/api.harbor-books.test/public does not exist
images         ✓ every data-owning service is pinned

✓ is information, ! will break something, ✗ is broken now. Rendered by the same function the command uses, so the shape of this report cannot drift from the real one.

What it checks

• The machine: is the backend running, does something else hold 80 and 443, is mkcert's CA installed
• The project: does every declared domain have a route and a certificate, does the docroot exist, is a container restarting in a loop
• The application: can each site reach the schema its .env names, is QUEUE_CONNECTION set to something nothing is running, does the config declare a PHP image the stack does not run
• The repository: does the config name a repository the checkout does not point at

Each finding says which of two possible things is wrong, and neither is guessed at. When an .env asks for a database the config does not declare, dqx does not pick a winner: it names both and offers both ways to make them agree.

The failures that are hardest to see

These are the ones dqx was written for, because none of them announce themselves:

what you see	what is actually happening
The site loads, but nothing saves	The schema exists but the user has no grant on it
Everything is 200, jobs never run	QUEUE_CONNECTION is set and nothing is consuming the queue
One page is 500, the rest is fine	A worker died and its container is restarting in a loop
It worked yesterday	Somebody changed the compose file and not the config, or the reverse
A colleague's build differs	The config names one repository and the checkout is at another
up never finishes, docker calls a healthy service unhealthy	The probe cannot run at all
Redis or the database refuses to start on data that was fine	A floating tag is not the same image twice

Two that are worth their own command

A healthcheck that can never pass. Compose hands a test: written as a string to a shell whole, so the container runs [CMD-SHELL, and answers exit 127, for ever. The service is usually fine: one project's Elasticsearch was serving 62 indices while docker called it unhealthy. What you see is up refusing to call the project started.

dqx healthchecks                 # what cannot run here
dqx healthchecks --write         # fix it: the check becomes a YAML list
dqx healthchecks --all ~/dockers # every project under a directory

First run on one fleet: 35 such checks in 23 of 32 projects.

An image tag with no version in it. redis:alpine is whatever your engine pulled last, so the same compose file is a different image on a colleague's laptop and on CI. For a service that keeps data on disk, that is a format mismatch waiting:

# Can't handle RDB format version 15
# Fatal error loading the DB, check server logs. Exiting.

The data was fine. The reader was older, and the message says neither.

dqx images                       # data-owning services on a floating tag
dqx images --write               # pin them to the minor series
dqx images --outdated            # which pins have fallen behind
dqx images --catalogue           # the house PHP images, with their PHP and node versions

Only services that own data are reported: the mail catcher keeps nothing a version can be wrong about.

A project that will not start after dqx kill

Two errors, one cause:

failed to set up container networking: network 94fce367… not found
Conflict. The container name "/nwd_db" is already in use by container "14147ecf…"

dqx kill used to clear up networks with docker network prune, which reads "unused" as nothing running is attached - and a stopped container still holds the network it was created with. It now removes only networks no container refers to at all, running or stopped. On an older dqx:

docker rm <the container the error names>   # it is stopped; compose recreates it
dqx up

When the name is held by another project

A stack older than the router fixes its container names, so the name is a machine-wide resource. Where a stopped container of this directory holds it, dqx clears it and says so. Where one from somewhere else holds it, dqx will not remove what it did not create:

dqx up

  nwd_db is a stopped container another project created, holding a name this stack needs.
  dqx will not remove what it did not create. To clear them:
    docker rm nwd_db

If it is running, nothing is broken: some other project is using that name. dqx down where it runs, then start this one.

The whole machine, in one command

dqx repair --dry-run     # says what is wrong, changes nothing
dqx doctor --all         # the same ground, as a report

Both cover the two things that produce a project which starts perfectly and still does not work:

The router and its certificates	Whether the router runs, what holds 80/443 when it does not, and whether every domain on this machine is covered by some certificate. A domain nothing covers is a browser saying not secure about a project that is otherwise faultless.
DNS, really resolved	One real lookup per suffix, not the presence of a resolver file. A /etc/resolver/test with no dnsmasq behind it, a VPN answering for the suffix, or rebinding protection dropping a loopback record all leave that file exactly where it should be.

dqx doctor --all

router         running, 80 domain(s) certified
dns            *.dqx: resolves to 127.0.0.1
dns            *.local.dolphiq.eu: answered by public DNS as 127.0.0.1

A missing name is named, and dqx repair reissues it itself - no password, because the root is already trusted and the certificates live in dqx's own directory:

dqx repair --dry-run

!  no certificate covers register.example.local.dolphiq.eu
→  reissuing every project's certificate that needs it

Where the machine's domain list comes from

Checking the machine means having a list for the machine, even though each certificate is one project's. dqx keeps it in ~/.local/share/dqx/domains.json, keyed by each project's own path, and refreshes a project's entry on every dqx up and dqx sync. An entry is dropped only when its directory is gone from disk.

dqx repair, dqx doctor --all, dqx router:up and dqx router:status read the project trees themselves and rewrite that file, which is what makes them the commands to believe. dqx up reads the file: 1 ms against 1.3 seconds for a list it can only add to.

--all also says whether dqx review can reach Bitbucket and GitHub:

bitbucket      ✓ Bitbucket token for you@example.com, 23 days left
github         gh not installed or not logged in - gh auth login (only needed for dqx review on GitHub)

A token that expires within five days is a warning with dqx auth --set bitbucket beside it. None of it fails the doctor: only dqx review needs either.

The router does nothing on a fresh machine

In this order, because each step depends on the one above:

dqx repair --dry-run	says what is missing - PATH, engine, backend binaries, mkcert root, resolvers, SSH agent - and changes nothing. dqx repair then does it.
dqx router:status	running or not running, and who holds 80 and 443.
docker running	The router is a compose stack. dqx up starts Docker Desktop itself; router:up does not.
mkcert -install	Needs your password once. Without a trusted root every domain answers NET::ERR_CERT_AUTHORITY_INVALID while the router runs perfectly.
dqx resolver status	.dqx needs /etc/resolver/dqx and sudo; local.dolphiq.eu resolves publicly and usually needs nothing. Each suffix sent to dqx is asked over the wire, so a resolver that is not running says so.
docker network ls | grep dqx_router	router:up creates it. Missing while the router reports running means something else is on that port.
dqx doctor in the project	Per project: not routed while the router runs, a certificate that does not cover its domains, ports already claimed.

Two more that no message mentions:

• Docker's automatic subnets. A stack that pins 172.20.0.0/16 clashes with whatever docker handed out first. See older projects for the default-address-pools setting.
• Gatekeeper. A binary downloaded through a browser is quarantined and refuses to run; the curl installer sets no such flag. See installing.

Domains stopped answering

Every domain at once, often after a reboot or after starting lerd, Herd or DDEV: something else took 80 and 443, and the router is up but no longer the one answering. Three places say so:

• The dashboard's banner names the holder and the command that frees the ports, and Recheck asks again after you ran it.
• dqx verify in a project says what each domain answers, and who answered instead of the router.
• dqx doctor reports the router not bound to 80 and 443, and the holder.

Stop the holder with the command they name, then dqx up in a project brings the router back on the ports. See one backend at a time.

A page that never finishes loading

The router runs, the containers are healthy, and the browser spins on every dqx domain without ever showing an error. That is a DNS lookup waiting for a timeout: /etc/resolver/<suffix> sends the suffix to dqx's resolver on 127.0.0.1:5354, and nothing answers there. launchd can list the agent with no process behind it:

$ launchctl list | grep dqx.resolver
-    0    nl.dolphiq.dqx.resolver

dqx doctor --all names it, and dqx repair or dqx resolver install restarts it:

dqx doctor --all

dns            ✗ *.local.dolphiq.eu: does not resolve here - /etc/resolver/local.dolphiq.eu sends it to dqx's resolver on 127.0.0.1:5354, and nothing answers there - `dqx resolver install`

dig gives the wrong answer here

dig skips /etc/resolver and asks the network's DNS server, so dig app.example.local.dolphiq.eu can answer 127.0.0.1 while the browser gets nothing. Ask the agent itself:

dig @127.0.0.1 -p 5354 +short app.example.local.dolphiq.eu   # a timeout means it is not running

By hand, the same restart is launchctl kickstart -k gui/$(id -u)/nl.dolphiq.dqx.resolver. Its errors go to /tmp/dqx-resolver.log.

Keeping the house images fresh

dqx update-images              # what this machine already has, brought up to date
dqx update-images --dry-run    # what would be pulled, pulling nothing
dqx update-images --major      # also cross a major version

Only what is already here. A machine on PHP 8.3 does not acquire the 8.5 image because it exists. Within a line it moves forward: :3 is refreshed in place, an exact 3.1.0 goes to the newest 3.x, and crossing to 4.x is reported rather than done.

dqx update is the project-scoped one: the images this project declares, whether you have them or not.

The rest

dqx verify        # every declared domain, with the status it actually returns
dqx logs          # everything
dqx logs app      # one service

dqx down && dqx up      # the ordinary reset
dqx clean               # also removes the data directories a bind-mounted database leaves behind
dqx kill --mine         # every dqx project, router included, and free 80 and 443
dqx kill                # every container on the machine, whoever started it
dqx kill --all          # that, and the runtimes too

A 200 from dqx verify is not proof the application works, but a 404 or a certificate warning tells you where in the chain to look: see domains, DNS and certificates.

dqx clean is the one that removes data. It asks first.

When dqx itself is the problem

dqx doctor reports what it can see. If it is silent and something is still wrong, the honest next step is the logs and then the issue tracker, with dqx info --json and dqx doctor --json attached.