#StackBounty: #amazon-web-services #apache-2.4 #virtualhost #amazon-elb apache 2.4 config to allow ELB health check

Bounty: 200

I have an EC2 instance behind an AWS Application Load Balancer, running apache 2.4

The health check is configured to do a GET on /health/

I have virtual hosts configured, and two vhost entries – one with the servername, and one to handle incoming requests directly to the IP address. aaa_first should be loaded first, and therefore be the default.

However, when I go directly to the public IP of the instance, I get the default apache welcome page, and the health check gets a 403:

"GET /health/ HTTP/1.1" 403 199 "-" "ELB-HealthChecker/2.0"

aaa_first.conf contains:

<VirtualHost *:80>
  ServerName aaa

  <Location /var/www/html>
     Require all denied
  </Location>

  <Location /var/www/html/health>
     Require ip 10.151.0.0/20
     Require all denied
  </Location>
  CustomLog logs/0000_access.log combined-elb-host
</VirtualHost>

default.conf contains:

<VirtualHost *:80>
  ServerName host.example.com

  DocumentRoot /var/www/html
  DirectoryIndex index.html

  ErrorLog logs/error.log
  CustomLog logs/access.log combined-elb-host

  <Directory "/var/www/html">
    AllowOverride All
    Require all granted
  </Directory>
</VirtualHost>

What do I need to ensure that requests to the IP are blocked, except for the health checks coming from the ELB?


Get this bounty!!!

#StackBounty: #linux #networking #apache-2.4 #tcp Apparent random slowniness on apache 2.4 on HTTPS and HTTP

Bounty: 50

Hello currently I experience random slowniness in apache 2.4.18 on HTTPS on Ubuntu server 16.04. Here follow observations, symptoms and info:

  • it doesn’t happen with HTTP
  • it doesn’t happen with about 2k used slots in apache’s /server-status. It happens from 2.7k upwards (I’ve seen the status reach even 4.2k used slots)
  • time curl -vI <url> is stuck in Trying <ip address> and responds in 1 to 30 seconds (the average processing time, when not slow, is 0.2s). While stuck, tcpdump -n | grep '<ip address of request>' shows no incoming packets in the server: they appear once curl goes on
  • apache is using mpm_prefork with the following settings
StartServers                     10
MinSpareServers           50
MaxSpareServers          100
ServerLimit     20000
MaxRequestWorkers         20000
MaxConnectionsPerChild   0
  • apache’s /server-status page shows many (hundreds) free slots
  • server resources average usage: cpu 30%, load average 20, ram used 35%, disk io r/w very low
  • some tuning attempts on the TCP queues (https://blog.cloudflare.com/syn-packet-handling-in-the-wild/)
    • ss -n state syn-recv sport = :443 | wc -l shows a maximum number of 130 when launched repeatedly. It seems that curl hangs when this number is close to 130
    • net.core.somaxconn was 128, I did set it to 4096 with sysctl -w net.core.somaxconn=4096, then even reloaded apache
    • I’ve launched sysctl -w net.ipv4.tcp_max_syn_backlog=4096
    • ss -plnt sport = :443 shows something like
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port
LISTEN     0      128         :::443                     :::*                   users:(("apache2",pid=184902,fd=8),<4000 more apache processes>

Honestly I don’t know if the queues tuning has been effective.

I’d like to first find out how to troubleshoot the problem to find out the part responsible of the slowniness, so any hint on troubleshooting is appreciated. Thank you

Edit 1

OK there’s 1 thing that maybe I didn’t try (for avoiding a downtime in production): the restart of apache. I thought that the TCP queue of the port 443 was handled by the system alone, i.e. without asking apache which was its limit: wrong. After the restart the queue was finally free to go over 130: in fact, in the first seconds after the apache restart the queue hit the apache limit of 511.

Now that the queue seems ok curl isn’t stuck anymore on Trying to connect; now the problem seems on the apache side, which seems slow even on HTTP.

More information:

  • the request url I’m using is being handled by an apache VirtualHost, which just issues a Redirect
  • here follows the information on the SSL cache
SSL/TLS Session Cache Status:
cache type: SHMCB, shared memory: 512000 bytes, current entries: 2176
subcaches: 32, indexes per subcache: 88
time left on oldest entries' objects: avg: 229 seconds, (range: 209...247)
index usage: 77%, cache usage: 99%
total entries stored since starting: 43916
total entries replaced since starting: 0
total entries expired since starting: 0
total (pre-expiry) entries scrolled out of the cache: 41740
total retrieves since starting: 586 hit, 83648 miss
total removes since starting: 0 hit, 0 miss

So please share any thoughts on how to troubleshoot the slowniness. Thank you


Get this bounty!!!

#StackBounty: #ubuntu #apache-2.4 #database #mongodb #ubuntu-18.04 Ubuntu MongoDB server not responding?

Bounty: 50

I have amazon aws ubuntu 18.04 server 8GB RAM and 50GB Storage. I have installed MongoDB 4.2 and Node JS 12 LTS.

In current server has 30 GB MongoDB storage. I have 4 micro service on same server. I have installed pm2 to start all 4 services.

Don’t know why my all services responds delay.

For example:

I have checked pm2 start 0 it takes 4-5 mins to show logs.

I have already remove db caching and flush pm2 logs. But still mongodb service takes 50% CPU usage.

I have checked all code and found there is no 3rd party APIs call and optimize all queries and adding indexing on server. But still server taking too much times.

Most of time I got server timeout error. Please assist me. How to make my server fast.


Get this bounty!!!

#StackBounty: #apache-2.4 #centos7 #monit Restart apache with monit

Bounty: 50

I’m trying to figure out some issues on my server where my CPU is reaching 100% utilization every other day. This causes all my websites and http services to fail.

How can I configure monit to restart the httpd service when my cpu hits 100% for say 10 cycles?

I’m still new to linux and centos. Learning my way around it and I’ve tried to research this for a while, but I could not find a proper answer to this.

Thanks for looking

Update

Thanks for your response. When the CPUs are running at 100%, I’ve run the top command and I can see that user nobody is consuming all of it on httpd service. I have a few apps that use php scripts, but their error logs all seem fine.

I’m suspecting a particular wordpress theme is causing this error due to a loop while updating the website. This particular website also shows me a loopback test failed error while all other wordpress sites on the same server are running fine.

So, until I can find out the culprit, I simply want to restart the service httpd when CPU utilization reaches say 95% for 10 cycles.

I hope that is enough clarity for someone to kindly give me a solution.


Get this bounty!!!

#StackBounty: #apache-2.4 #php-fpm PHP status page not working in Apache

Bounty: 50

Using PHP7.3 FPM along with Apache 2.4.25 I have a problem when enabling PHP FPM status page.

What is working:

# /etc/php/7.3/fpm/pool.d/www.conf
listen = /run/php/php7.3-fpm.sock
pm.status_path = /status-php

# /etc/apache2/sites-enabled/status.conf
<VirtualHost [::1]:80 127.0.0.1:80>
        ServerName localhost
        DocumentRoot /var/www
        <Location /status-php>
                SetHandler "proxy:unix:/run/php/php7.3-fpm.sock|fcgi://localhost"
        </Location>
</VirtualHost>

But what does not work is replacing /status-php with /status/php in both files (PHP and Apache config). This results in 404 and an Apache error saying:

AH01071: Got error 'Primary script unknown'

What is the reason for that? Why is the subfolder-variant not working?


Get this bounty!!!

#StackBounty: #ubuntu #php #apache-2.4 #apache2 Diagnosing apache cpu / memory / process spike

Bounty: 50

I have a server running Ubuntu 18.04 which has been experiencing huge CPU spike that nearly brings Apache to a halt once or twice a day. The server runs a couple of websites – all php & mysql driven applications. Here’s a bit more detail on things I’ve looked into:

MySQL: Slow query log is enabled and set to log queries taking longer than 1 second. Reviewing this log after a spike reveals nothing in particular. No long running queries to speak of.

CRON: I’ve reviewed all user cron jobs running on the server and there’s nothing that happens during the times when these spikes occur. There are only a couple of CPU intensive jobs and they run around 3am and take approx 5 minutes to complete.

PHP: Both max_input_time and max_execution_time are set to 60 seconds and memory_limit is 64M (this is a 16GB server which typically doesn’t come close to maxing out memory usage).

APACHE: Our host (Linode) has a tool called Longview which shows various diagnostics related to Apache. Despite the huge spike in resources consumed, requests seem to be happening at a normal rate. Manually inspecting the access logs confirms this. Here’s a screenshot of the Apache tab in Longview showing a spike in Workers, CPU and RAM this morning – as well as a relatively normal rate of Requests:

enter image description here

I’ve also added flags in the Apache access logs to show time and I/O data about each request. The end of the LogFormat is time:%T input:%I output:%O. None of the request or response sizes are unusually large (1MB might be the largest response I saw and that was for an image). The only thing standing out is the "time taken to serve the request" which is the %T flag. At a certain point in the morning many seemingly normal requests take 5 – 10 minutes to complete for no apparent reason.

I’m completely stumped at this point. Where can i go from here to diagnose the event that’s triggering this?


Get this bounty!!!

#StackBounty: #php #apache-2.4 #virtualhost #shared-hosting How did shared hosting providers ensure user isolation before containerizat…

Bounty: 50

Around 2000-2010, shared hosting was extremely popular as a cheap solution (sometimes a few $ / month, or sometimes even free for just a few MB) for people starting blogs, small websites, e.g. using WordPress.

There was usually:

  • just Apache + PHP + MySQL
  • no SSH, only (s)FTP access
  • something like 100 MB
  • as far as I remember, they probably didn’t create a new virtual machine for each account

Question: before containerization / Docker went popular, how did major shared hosting providers ensure user isolation?

Did they just used ChrootDirectory in sshd_config + different users like in How to create an isolated/jailed SFTP user?
+ <VirtualHost> config with open_basedir to prevent PHP code to access other accounts’ files?

More generally, what were the main isolation techniques, preventing user1234 to access user5678‘s files on the same server with some malicious PHP code?


Get this bounty!!!

#StackBounty: #apache-2.4 #gzip Apply Apache GZIP on proxied resources

Bounty: 50

I’m running an asp.net application on a Kestrel server. The front facing server is Apache. I’m have trouble applying gzip compression on the proxied resources, although any other resources served directly from Apache and not Kestrel are compresses normally using gzip. Below is myhttp.conf, not sure what im missing, help please.

# Admin email, Server Name (domain name) and any aliases
ServerAdmin yehiasalam@cube.com
ServerName  app.cube.com
ServerAlias  www.app.cube.com
DocumentRoot /var/www/cube/app

# Exclude the /widget folder from the proxy
<Location /widget>   
    ProxyPass !    
    Header set Access-Control-Allow-Origin "*"
</Location>

ProxyPreserveHost On
ProxyPass / http://127.0.0.1:5000/
ProxyPassReverse / http://127.0.0.1:5000/

RewriteEngine on 
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC] 
RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC] 
RewriteCond %{REQUEST_URI} !^/widget/
RewriteRule /(.*) ws://127.0.0.1:5000/$1 [P]
                                                     
# Gzip everything
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript

# Set the logs
ErrorLog /var/www/cube/app/error.log
CustomLog /var/www/cube/app/access.log common



Get this bounty!!!

#StackBounty: #apache-2.2 #apache-2.4 #reverse-proxy #mod-rewrite #proxypass apache rewriterule and proxypass 404 issue

Bounty: 50

I’m trying to set up group access to certain urls on my reverse proxy gateway. The previous questions that combine proxypass and mod_rewrite do so not for the reasons that I am doing so. I am combining them because I am trying to force an internal redirect (so I can see the HTTP headers.. This is because mod_rewrite cannot see the headers that I want unless I do an internal redirect). (This is a must, unfortunately)

If I remove the rewrite lines, the proxy works as expected (it’s serving the files correctly). However, the group access is not being enforced.

<VirtualHost *:*>
    ServerName mysubdomain.mydomain.com
    SSLProxyEngine on
    
    #I'm not an apache sys-admin professional so I'm not sure if any of these are necessary
    ProxyPreserveHost On
    ProxyRequests Off
    AllowEncodedSlashes On

    <Location /mypath>
        #this AuthType is what gets the HTTP header that I want (GROUPS)
        AuthType MyApacheAgt
        Order Deny,Allow
        Deny from all
        Allow from all

        RewriteEngine On
        RewriteCond %{ENV:REDIRECT_PASS} !1
        RewriteRule ^(.*)$ /$1 [L,E=PASS:1,PT]
        RewriteCond %{HTTP:GROUPS} !^.*some-group-to-match-to.*$
        RewriteRule ^(.*)$ /$1 [L,R=403,PT]

        ProxyPass        http://my-proxied-webserver.mydomain:8080/mypath disablereuse=On retry=0 nocanon
        ProxyPassReverse http://my-proxied-webserver.mydomain:8080/mypath
    </Location>

</VirtualHost>
        

From my apache logs, I see that I’m getting:

[pid 9:tid 140131688048384] [client XX] AH00128: File does not exist: /var/www/html/proxy:http:/my-proxied-webserver.mydomain:8020/mypath

One thing that alarms me is that there is only one / in the proxied url in the error log. Is this normal?

I am using apache 2.4 (and would prefer 2.4, but most 2.2 can be converted over)

Would it be simpler to merge the rewriterules and the proxy lines (if this is possible?)


Get this bounty!!!

#StackBounty: #ubuntu #apache-2.4 #logging #wildcard #wildcard-subdomain Dynamic Error and Custom logs in apache 2 using wildcard subdo…

Bounty: 50

I’m trying to make a wildcard VirtualHost conf file for apache2, and I’m not sure how to handle the ErrorLog and CustomLog settings to put the logs where I want them to go. As you can see in the second VirtualHost, I have the logs in a logs folder in the domains DocumentRoot. This works fine for static VirtualHosts, but how would I go about it for a wildcard VirtualHost. eg, the first VirtualHost.

NameVirtualHost *:80

# Wild card all subdomains
<VirtualHost *:80>
        ServerAlias *.example.com
        VirtualDocumentRoot /var/www/%0/public
        ErrorLog ?????
        CustomLog ????? combined
</VirtualHost>


# Main domain
<VirtualHost *:80>
        ServerName example.com
        ServerAlias www.example.com
        DocumentRoot /var/www/example.com/public
        ErrorLog /var/www/example.com/logs/error.log
        CustomLog /var/www/example.com/logs/access.log combined
</VirtualHost>

I have tried going ErrorLog /var/www/*.example.com/logs/error.log and ErrorLog /var/www/%0/logs/error.log and the same for CustomLog, but when I try to restart apache it throws an error.

What syntax should I use to get a working version of my ErrorLog example above?

I have seen Wildcards in Virtual Hosts with dynamic logs? , but it is not really what I am after as it still ends up putting all the logs into one big file rather than having them split into their own subdomain specific folders.


Get this bounty!!!