View source for ICT:Reverse-Proxy on AlmaLinux with nginx to apache backend

= Reverse‑Proxy Architecture (nginx → Apache) =
This page documents the complete reverse‑proxy setup for the AlmaLinux frontend server.
It replaces the former IIS reverse‑proxy and Windows‑based certificate renewal system.
The configuration is now fully Linux‑native, stable, and successor‑friendly.

== Overview ==
* Frontend: AlmaLinux + nginx
* Backend: Apache (Drupal + MediaWiki)
* Certificates: Let’s Encrypt (certbot)
* Automatic renewal: systemd timer
* Windows/IIS: fully retired (no bindings, no renewals, no proxies)

The nginx server terminates HTTPS and forwards traffic to the internal Apache instance.

== Directory Layout ==
* '''/etc/nginx/nginx.conf''' – global nginx configuration  
* '''/etc/nginx/conf.d/''' – per‑site reverse‑proxy configs  
* '''/var/www/letsencrypt/''' – ACME challenge directory  
* '''/etc/letsencrypt/live/<domain>/''' – certificate files  

== ACME Challenge Directory ==
Certbot uses a shared directory for HTTP‑01 validation:

<syntaxhighlight lang="bash">
/var/www/letsencrypt/
</syntaxhighlight>

nginx exposes this path for both domains.

== Site: research.costasano.club ==
=== HTTP (port 80) ===
Redirects all traffic to HTTPS, except ACME challenges.

<syntaxhighlight lang="nginx">
server {
    listen 80;
    listen [::]:80;
    server_name research.costasano.club;

    location /.well-known/acme-challenge/ {
        root /var/www/letsencrypt;
    }

    return 301 https://$host$request_uri;
}
</syntaxhighlight>

=== HTTPS (port 443) ===
Terminates TLS and proxies to Apache.

<syntaxhighlight lang="nginx">
server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;

    server_name research.costasano.club;

    ssl_certificate     /etc/letsencrypt/live/research.costasano.club/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/research.costasano.club/privkey.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;

    location /.well-known/acme-challenge/ {
        root /var/www/letsencrypt;
    }

    location / {
        proxy_pass http://192.168.33.231;

        proxy_set_header Host research.costasano.club;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;

        proxy_buffering on;
        proxy_buffers 16 16k;
        proxy_buffer_size 16k;

        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }
}
</syntaxhighlight>

== Site: mwiki.costasano.club ==
=== HTTP (port 80) ===
<syntaxhighlight lang="nginx">
server {
    listen 80;
    listen [::]:80;
    server_name mwiki.costasano.club;

    location /.well-known/acme-challenge/ {
        root /var/www/letsencrypt;
    }

    return 301 https://$host$request_uri;
}
</syntaxhighlight>

=== HTTPS (port 443) ===
<syntaxhighlight lang="nginx">
server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;

    server_name mwiki.costasano.club;

    ssl_certificate     /etc/letsencrypt/live/mwiki.costasano.club/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mwiki.costasano.club/privkey.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;

    location /.well-known/acme-challenge/ {
        root /var/www/letsencrypt;
    }

    location / {
        proxy_pass http://192.168.33.231;

        proxy_set_header Host mwiki.costasano.club;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;

        proxy_buffering on;
        proxy_buffers 16 16k;
        proxy_buffer_size 16k;

        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }
}
</syntaxhighlight>

== Certificate Renewal ==
Certbot is installed natively on AlmaLinux.

=== Renewal Timer ===
<syntaxhighlight lang="bash">
systemctl status certbot-renew.timer
</syntaxhighlight>

The timer runs daily and triggers renewal when certificates approach expiry.

=== Manual Dry‑Run Test ===
<syntaxhighlight lang="bash">
sudo certbot renew --dry-run
</syntaxhighlight>

== Windows/IIS Retirement ==
The former IIS reverse‑proxy and win‑acme renewal system were removed:
* IIS sites deleted  
* win‑acme renewal entries cancelled  
* scheduled tasks removed  
* old certificates removed from certlm.msc  
* no bindings remain on ports 80/443  

== Verification Checklist ==
* nginx configs validated: <code>nginx -t</code>  
* nginx reloaded: <code>systemctl reload nginx</code>  
* certbot timer active  
* ACME challenge reachable  
* backend Apache reachable  
* Windows silent on ports 80/443  

== Architecture Diagram (ASCII) ==

The following diagram shows the full request flow from the internet to the backend applications.

<pre>
                 +----------------------+
                 |      Internet        |
                 +----------+-----------+
                            |
                            v
                 +----------------------+
                 |   AlmaLinux Server   |
                 |      (frontend)      |
                 +----------+-----------+
                            |
                HTTPS :443 / HTTP :80
                            |
                            v
                    +---------------+
                    |    nginx      |
                    |  reverse proxy|
                    +-------+-------+
                            |
        +-------------------+-------------------+
        |                                       |
        v                                       v
+---------------------+               +---------------------+
| research.costasano  |               | mwiki.costasano     |
| .club (vhost)       |               | .club (vhost)       |
+----------+----------+               +----------+----------+
           |                                     |
           | proxy_pass http://192.168.33.231    |
           +-------------------------+-----------+
                                     |
                                     v
                         +------------------------+
                         |   Apache (backend)     |
                         |  192.168.33.231        |
                         +-----------+------------+
                                     |
                 +-------------------+-------------------+
                 |                                       |
                 v                                       v
        +-------------------+                  +-------------------+
        |   Drupal site     |                  |  MediaWiki site   |
        | research.*        |                  | mwiki.*           |
        +-------------------+                  +-------------------+


ACME / Let’s Encrypt HTTP-01 validation:

    Let’s Encrypt  --->  nginx  --->  /var/www/letsencrypt/
                         (both vhosts expose
                          /.well-known/acme-challenge/)

Certificates:

    /etc/letsencrypt/live/research.costasano.club/
    /etc/letsencrypt/live/mwiki.costasano.club/

Renewal:

    systemd timer: certbot-renew.timer
    service:       certbot-renew.service
</pre>

== Rationale for Replacing IIS with nginx ==

This section explains why the former Windows/IIS reverse‑proxy was retired in favour of a native Linux‑based nginx setup. The decision was driven by stability, clarity, maintainability, and long‑term operational safety.

=== 1. Architectural Simplicity ===
The previous setup required:
* Windows Server
* IIS with URL Rewrite + ARR
* win-acme for certificate renewal
* Manual bindings for each hostname
* Coordination between Windows and the Linux backend

This created a multi‑platform dependency chain.  
The new setup collapses the entire frontend into a single, predictable layer:
* AlmaLinux
* nginx
* certbot (systemd‑managed)

Fewer moving parts → fewer failure points.

=== 2. Elimination of Cross‑Platform Coupling ===
The old architecture required Windows to proxy traffic to Linux.  
This meant:
* Two operating systems had to stay in sync  
* Two certificate systems had to be maintained  
* Two sets of logs had to be consulted during troubleshooting  

By moving the reverse‑proxy to the same machine that hosts the backend, the system becomes self‑contained and easier to reason about.

=== 3. Cleaner TLS and Certificate Management ===
The Windows setup used win‑acme, scheduled tasks, and IIS bindings.  
This caused:
* Duplicate certificate stores  
* Renewal tasks that were easy to forget  
* Certificates living on a machine that no longer served the content  

The new setup uses:
* certbot  
* systemd timers  
* a shared ACME challenge directory  
* certificates stored exactly where they are used  

This is simpler, safer, and fully automated.

=== 4. nginx Is Purpose‑Built for Reverse‑Proxying ===
nginx excels at:
* high‑performance proxying  
* buffering  
* header management  
* HTTP/2  
* TLS termination  

IIS can do these things, but only through multiple modules (ARR, URL Rewrite) and with more complexity. nginx provides them natively and predictably.

=== 5. Reduced Operational Overhead ===
The Windows/IIS layer required:
* OS patching  
* IIS updates  
* win‑acme maintenance  
* Monitoring of scheduled tasks  
* Manual cleanup of old bindings and certificates  

The nginx layer requires:
* a single systemd timer  
* a single config directory  
* one place to check logs  

This reduces cognitive load for both you and any future maintainer.

=== 6. Successor‑Friendly Documentation and Maintenance ===
The new architecture is:
* linear  
* transparent  
* fully documented  
* reproducible  
* platform‑consistent  

A successor only needs to understand:
* nginx vhost files  
* certbot  
* Apache backend routing  

No Windows knowledge is required.

=== 7. Removal of Legacy Dependencies ===
The IIS reverse‑proxy was originally introduced as a workaround during earlier infrastructure phases.  
Now that the backend is stable on AlmaLinux, the Windows layer no longer serves a purpose and only adds risk.

Removing it:
* reduces attack surface  
* removes unused services  
* avoids silent port conflicts  
* simplifies backups and disaster recovery  

=== Summary ===
The migration from IIS to nginx was not just a technical change but an architectural improvement.  
It aligns the system with:
* clarity  
* maintainability  
* long‑term stability  
* reduced operational burden  
* successor‑friendly design  

The result is a cleaner, more predictable, and more robust infrastructure.

== Before vs After: Reverse‑Proxy Architecture ==

The table below summarizes the differences between the former IIS‑based setup and the current nginx‑based architecture.

{| class="wikitable"
! Aspect
! '''Before: Windows + IIS'''
! '''After: AlmaLinux + nginx'''

|-
| Reverse‑proxy layer
| IIS with ARR + URL Rewrite  
(added modules, multi‑step configuration)
| nginx (native reverse‑proxy, single config file per site)

|-
| Operating system
| Windows Server  
(separate from backend)
| AlmaLinux  
(same platform as backend)

|-
| Certificate management
| win‑acme (scheduled task, IIS bindings)  
Certificates stored in Windows certificate store
| certbot (systemd timer)  
Certificates stored directly under /etc/letsencrypt/

|-
| ACME challenge handling
| IIS rewrite rules + filesystem mapping  
More fragile and harder to debug
| Direct nginx location block  
Shared ACME directory for all vhosts

|-
| Number of moving parts
| High: Windows + IIS + ARR + Rewrite + win‑acme  
| Low: nginx + certbot

|-
| Logging
| Split across Windows Event Viewer, IIS logs, win‑acme logs
| Unified: nginx logs + certbot logs

|-
| Maintenance burden
| Requires Windows patching, IIS updates, scheduled task monitoring
| Minimal: systemd‑managed services, simple config reloads

|-
| Failure modes
| Certificate renewal failures often silent  
Bindings could break after updates  
Proxy rules spread across GUI modules
| Transparent and predictable  
Config validated with nginx -t  
Renewal handled by systemd

|-
| Security surface
| Larger (Windows + IIS + ARR + Rewrite)
| Smaller (nginx only)

|-
| Successor‑friendliness
| Requires Windows + IIS knowledge  
Proxy logic hidden behind GUI layers
| Fully text‑based, documented, reproducible  
Easy for any Linux‑familiar maintainer

|-
| Architectural clarity
| Cross‑platform coupling (Windows → Linux)
| Single‑platform, self‑contained frontend

|}

== Status ==
The reverse‑proxy setup is stable, maintainable, and fully Linux‑native.


== Troubleshooting Appendix ==

This appendix lists common issues that may occur in the nginx → Apache reverse‑proxy setup, along with clear diagnostic steps and expected outcomes. It is designed to help future maintainers quickly identify where a failure originates.

=== 1. Site Not Loading (General) ===
If a site does not load at all:

# Check nginx status:
<syntaxhighlight lang="bash">
systemctl status nginx
</syntaxhighlight>

# Validate configuration:
<syntaxhighlight lang="bash">
nginx -t
</syntaxhighlight>

# Reload nginx:
<syntaxhighlight lang="bash">
systemctl reload nginx
</syntaxhighlight>

If nginx fails to start or reload, the error message will point to the exact file and line.

=== 2. HTTPS Works but HTTP Does Not Redirect ===
Symptoms:
* Visiting http://domain shows an error  
* Visiting https://domain works normally  

Check the port 80 server block for the domain:
* It must include the ACME location block  
* It must end with: <code>return 301 https://$host$request_uri;</code>

Restart nginx after correcting:
<syntaxhighlight lang="bash">
systemctl reload nginx
</syntaxhighlight>

=== 3. Certificate Renewal Fails ===
Symptoms:
* Certbot emails warnings  
* Browser shows certificate expiring soon  
* Dry‑run renewal fails  

Diagnostics:

# Test ACME challenge manually:
<syntaxhighlight lang="bash">
curl http://domain/.well-known/acme-challenge/test
</syntaxhighlight>

Expected:
* 404 Not Found (normal)
* NOT a 403 or 500

# Check permissions:
<syntaxhighlight lang="bash">
ls -ld /var/www/letsencrypt
</syntaxhighlight>

# Run a dry‑run:
<syntaxhighlight lang="bash">
sudo certbot renew --dry-run
</syntaxhighlight>

If renewal fails, the error message will indicate whether nginx routing or filesystem access is the cause.

=== 4. Backend (Apache) Not Responding ===
Symptoms:
* nginx returns 502 Bad Gateway  
* nginx returns 504 Gateway Timeout  

Diagnostics:

# Check Apache:
<syntaxhighlight lang="bash">
systemctl status httpd
</syntaxhighlight>

# Test backend directly:
<syntaxhighlight lang="bash">
curl http://192.168.33.231
</syntaxhighlight>

If Apache is down, restart it:
<syntaxhighlight lang="bash">
systemctl restart httpd
</syntaxhighlight>

If Apache is up but unreachable, check firewall rules.

=== 5. Wrong Site Appears (Host Header Issue) ===
Symptoms:
* Visiting research.* shows the MediaWiki site  
* Visiting mwiki.* shows the Drupal site  

Cause:
* Missing or incorrect <code>proxy_set_header Host ...</code> in nginx

Fix:
Ensure each vhost sets the correct Host header:
<syntaxhighlight lang="nginx">
proxy_set_header Host research.costasano.club;
</syntaxhighlight>

Reload nginx afterwards.

=== 6. ACME Challenge Returns 404 ===
Symptoms:
* Certbot renewal fails  
* Browser shows 404 for ACME path  

Check that both vhosts contain:
<syntaxhighlight lang="nginx">
location /.well-known/acme-challenge/ {
    root /var/www/letsencrypt;
}
</syntaxhighlight>

Ensure the directory exists:
<syntaxhighlight lang="bash">
ls /var/www/letsencrypt
</syntaxhighlight>

=== 7. Windows Still Interfering (Legacy Setup) ===
Symptoms:
* Port conflicts  
* Unexpected certificates  
* Old bindings still active  

Diagnostics on Windows:

# Check ports:
<syntaxhighlight lang="powershell">
netstat -ano | findstr :80
netstat -ano | findstr :443
</syntaxhighlight>

# Check IIS:
Ensure no sites remain in IIS Manager.

# Check scheduled tasks:
Remove any leftover win‑acme tasks.

If Windows is still listening on 80/443, disable the Web Server (IIS) role entirely.

=== 8. Browser Shows Old Certificate ===
Symptoms:
* Browser still shows the old expiry date  
* Renewal succeeded on the server  

Possible causes:
* Browser caching  
* Intermediate certificate caching  
* CDN or proxy caching (if applicable)

Fix:
* Hard refresh (Ctrl+F5)  
* Test with curl:
<syntaxhighlight lang="bash">
curl -v https://domain
</syntaxhighlight>

The certificate shown by curl is authoritative.

=== 9. nginx Reload Fails After Editing Config ===
Symptoms:
* <code>systemctl reload nginx</code> fails  
* nginx refuses to start  

Run:
<syntaxhighlight lang="bash">
nginx -t
</syntaxhighlight>

This will show:
* the file  
* the line  
* the exact syntax error  

Fix the issue and reload nginx.

=== 10. Slow Responses or Timeouts ===
Symptoms:
* Pages load slowly  
* MediaWiki or Drupal feels sluggish  

Check proxy timeouts:
<syntaxhighlight lang="nginx">
proxy_read_timeout 60s;
proxy_send_timeout 60s;
</syntaxhighlight>

Check backend load:
<syntaxhighlight lang="bash">
top
systemctl status httpd
</syntaxhighlight>

If Apache is overloaded, investigate modules, PHP-FPM, or database performance.

== Summary ==
This appendix provides a structured approach to diagnosing issues in the nginx → Apache reverse‑proxy setup. Most problems fall into one of three categories:
* nginx configuration  
* certificate renewal  
* backend availability  

Following the steps above should isolate the cause quickly and safely.


== Notes for AlmaLinux 10.1 Security Changes ==

AlmaLinux 10.1 introduces several security‑related changes that affect nginx, certbot, and
backend communication. The following points must be checked whenever the system is updated
or rebuilt.

=== 1. SELinux: Stricter Defaults ===
AlmaLinux 10.1 enforces tighter SELinux rules, especially for:
* nginx reading files outside /usr/share/nginx/html
* ACME challenge directories
* proxy connections to backend servers

Ensure:
<syntaxhighlight lang="bash">
semanage fcontext -a -t httpd_sys_content_t "/var/www/letsencrypt(/.*)?"
restorecon -Rv /var/www/letsencrypt
</syntaxhighlight>

If nginx cannot reach Apache:
<syntaxhighlight lang="bash">
setsebool -P httpd_can_network_connect 1
</syntaxhighlight>

=== 2. FirewallD: More Restrictive Defaults ===
After updates, FirewallD may re‑apply stricter profiles.

Verify:
<syntaxhighlight lang="bash">
firewall-cmd --list-services
</syntaxhighlight>

Ensure:
* http
* https

are enabled permanently.

=== 3. OpenSSL 3.x Enforcement ===
AlmaLinux 10.1 uses OpenSSL 3.x with stricter cipher policies.

Ensure nginx uses:
* TLSv1.2
* TLSv1.3

and avoid deprecated ciphers.

=== 4. Systemd Hardening of Services ===
Some systemd units (nginx, certbot) may receive tightened sandboxing.

If certbot renewal fails after an update:
<syntaxhighlight lang="bash">
journalctl -u certbot-renew.service
</syntaxhighlight>

Check for:
* permission denials
* sandbox restrictions
* missing ACME directory access

=== 5. nginx: HTTP/2 and TLS Requirements ===
AlmaLinux 10.1 enforces modern TLS for HTTP/2.

Ensure:
<syntaxhighlight lang="nginx">
http2 on;
ssl_protocols TLSv1.2 TLSv1.3;
</syntaxhighlight>

HTTP/2 will silently disable itself if insecure ciphers are used.

=== 6. Apache: Private IP Binding Verification ===
After updates, Apache may rebind or change Listen directives.

Verify:
<syntaxhighlight lang="bash">
grep -R "Listen" /etc/httpd/conf*
</syntaxhighlight>

Expected:
* Apache listens on 192.168.33.231:80
* NOT on 0.0.0.0:80 (to avoid conflicts)

=== 7. Certbot: Renewal Timer Integrity ===
System updates may reset or disable timers.

Verify:
<syntaxhighlight lang="bash">
systemctl status certbot-renew.timer
systemctl list-timers | grep certbot
</syntaxhighlight>

Expected:
* Timer active
* Next run scheduled within 24 hours

=== 8. Kernel-Level Networking Changes ===
AlmaLinux 10.1 includes updated nftables and TCP stack behaviour.

If proxying becomes slow or unstable:
<syntaxhighlight lang="bash">
sysctl net.ipv4.ip_local_port_range
sysctl net.core.somaxconn
</syntaxhighlight>

Ensure defaults are not overly restrictive.

== Summary ==
AlmaLinux 10.1 is more secure by default, but this can affect nginx, certbot, and Apache
interactions. After any major update, verify:
* SELinux permissions  
* FirewallD rules  
* nginx TLS configuration  
* Apache binding  
* certbot timers  

This ensures the reverse‑proxy remains stable and predictable.