Troubleshooting Philosophy for MediaWiki & Multi‑Layer Systems

Maintaining this MediaWiki installation involves several interacting layers:

IIS ARR (reverse proxy)
Apache
PHP-FPM
MediaWiki
ImageMagick
SELinux
Filesystem permissions
Hashed upload directories
Rewrite rules

When something breaks, it’s tempting to blame the most external or mysterious component — often the reverse proxy. This can lead to “ghost hunting”: chasing symptoms that look like proxy failures but originate elsewhere.

This page outlines a disciplined approach that avoids that trap.

1. Start with direct, observable facts

Before forming a hypothesis, gather concrete evidence:

Can the file be read on disk?
Does curl -I return a status code?
Does Apache log the request?
Does MediaWiki show a specific error?
Does the file exist in the hashed directory?

This prevents assumptions from steering the investigation.

2. Test each layer independently

A multi-layer system fails in multi-layer ways. Always test each layer in isolation:

Filesystem: ls -lZ, verify ownership and SELinux
Apache: direct URL tests, static file serving
Rewrite rules: confirm exceptions for static files
MediaWiki: file page metadata, thumbnail generation
ImageMagick: command-line conversion
Reverse proxy: external vs internal URL behavior

This reveals where the failure actually occurs.

3. Avoid the “reverse proxy reflex”

Reverse proxies are often blamed first because they sit at the edge and feel like the “gatekeeper.” But many failures originate deeper:

Apache rewrite rules
MediaWiki switching from GD → ImageMagick
SELinux blocking read access
PHP-FPM socket issues
Incorrect $wgUploadPath
Missing static-file exceptions

Only blame the proxy when evidence points there.

4. Prefer reproducible tests over assumptions

A single curl test can eliminate entire categories of hypotheses:

curl -I https://mwiki.example.com/images/x/xy/file.png

Interpretation:

200 OK → routing works
404 → proxy or rewrite issue
403 → permissions or SELinux
500 → PHP/ImageMagick
blank page → rewrite rule catching static files

This is faster and more reliable than guesswork.

5. Remember that MediaWiki changes behavior when ImageMagick is installed

This is a subtle but important point:

With GD, MediaWiki reads images directly from disk.
With ImageMagick, MediaWiki validates images via URL and MIME type.

This means:

A misconfigured rewrite rule may be invisible under GD.
The same misconfiguration becomes catastrophic under ImageMagick.

This explains why image failures sometimes appear “out of nowhere.”

6. Document the fix, not just the cause

Future admins benefit from:

The symptom
The root cause
The exact fix
The reasoning behind it

This prevents rediscovering the same issue years later.

7. Stay calm and methodical

The most important principle:

Don’t let any single hypothesis dominate before the evidence supports it.

This avoids tunnel vision and keeps the investigation grounded.