ICT:Debugging

Troubleshooting Philosophy for MediaWiki & Multi‑Layer Systems

Maintaining this MediaWiki installation involves several interacting layers:

• IIS ARR (reverse proxy)
• Apache
• PHP-FPM
• MediaWiki
• ImageMagick
• SELinux
• Filesystem permissions
• Hashed upload directories
• Rewrite rules

When something breaks, it’s tempting to blame the most external or mysterious component — often the reverse proxy. This can lead to “ghost hunting”: chasing symptoms that look like proxy failures but originate elsewhere.

This page outlines a disciplined approach that avoids that trap.

1. Start with direct, observable facts

Before forming a hypothesis, gather concrete evidence:

• Can the file be read on disk?
• Does curl -I return a status code?
• Does Apache log the request?
• Does MediaWiki show a specific error?
• Does the file exist in the hashed directory?

This prevents assumptions from steering the investigation.

2. Test each layer independently

A multi-layer system fails in multi-layer ways. Always test each layer in isolation:

• Filesystem: ls -lZ, verify ownership and SELinux
• Apache: direct URL tests, static file serving
• Rewrite rules: confirm exceptions for static files
• MediaWiki: file page metadata, thumbnail generation
• ImageMagick: command-line conversion
• Reverse proxy: external vs internal URL behavior

This reveals where the failure actually occurs.

3. Avoid the “reverse proxy reflex”

Reverse proxies are often blamed first because they sit at the edge and feel like the “gatekeeper.” But many failures originate deeper:

• Apache rewrite rules
• MediaWiki switching from GD → ImageMagick
• SELinux blocking read access
• PHP-FPM socket issues
• Incorrect $wgUploadPath
• Missing static-file exceptions

Only blame the proxy when evidence points there.

4. Prefer reproducible tests over assumptions

A single curl test can eliminate entire categories of hypotheses:

curl -I https://mwiki.example.com/images/x/xy/file.png

Interpretation:

• 200 OK → routing works
• 404 → proxy or rewrite issue
• 403 → permissions or SELinux
• 500 → PHP/ImageMagick
• blank page → rewrite rule catching static files

This is faster and more reliable than guesswork.

5. Remember that MediaWiki changes behavior when ImageMagick is installed

This is a subtle but important point:

• With GD, MediaWiki reads images directly from disk.
• With ImageMagick, MediaWiki validates images via URL and MIME type.

This means:

• A misconfigured rewrite rule may be invisible under GD.
• The same misconfiguration becomes catastrophic under ImageMagick.

This explains why image failures sometimes appear “out of nowhere.”

6. Document the fix, not just the cause

Future admins benefit from:

• The symptom
• The root cause
• The exact fix
• The reasoning behind it

This prevents rediscovering the same issue years later.

7. Stay calm and methodical

The most important principle:

Don’t let any single hypothesis dominate before the evidence supports it.

This avoids tunnel vision and keeps the investigation grounded.