Hi folks,
We have a serious issue with the current Apache httpd package, which is an indispensable service for a server these days.
The issue is basically the default configuration as provided by the package: - if you create just one server in den .conf directory, it results in a non-functional https configuration - the configuration uses items, that are deprecated since version 2.4 - the default layout of the default configuration is currently not suitable
I wrote a bug report about it: https://bugzilla.redhat.com/show_bug.cgi?id=2258121
Unfortunately we are experiencing a common Fedora problem, nobody responds to it and the bug reports are left unprocessed and pile up. And because none of the maintainers respond at all, we have now also missed all the deadlines for improving something for f40.
Technically an improvement is more than simple. It’s just a change in the configuration files, as far as I see.
Any idea how to proceed with this?
- Would someone take the bug report and check my analyses? - Is there any way to bring this forward? - Should we create a new https-server package which takes the binaries and provides an alternative configuration? Probably similar to default-editor-vim / default-editor-nano?
As it currently stands, we are in no way competitive with, for example, Debian or ArchLinux.
-- Peter Boy https://fedoraproject.org/wiki/User:Pboy PBoy@fedoraproject.org
Timezone: CET (UTC+1) / CEST (UTC+2)
Fedora Server Edition Working Group member Fedora Docs team contributor and board member Java developer and enthusiast
* Peter Boy [13/02/2024 10:36] :
Unfortunately we are experiencing a common Fedora problem, nobody responds to it and the bug reports are left unprocessed and pile up. And because none of the maintainers respond at all, we have now also missed all the deadlines for improving something for f40.
Sad but not surprising...
- Would someone take the bug report and check my analyses?
Looking at httpd's git repo, it doesn't look like we change the configuration as shipped upstream to a significant degree. And, looking at httpd's subversion repo, _default_ is used both at the tip of the 2.4 branch and trunk.
I'll also argue that using virtual hosts for the domain matching your server's name is a mistake but I'm not sure how popular the sentiment is...
- Should we create a new https-server package which takes the binaries and provides an alternative configuration? Probably similar to default-editor-vim / default-editor-nano?
I really don't like this idea. I would much rather we work with upstream to improve the default configuration.
Emmanuel
Am 13.02.2024 um 16:10 schrieb Emmanuel Seyman emmanuel@seyman.fr:
- Peter Boy [13/02/2024 10:36] :
Unfortunately we are experiencing a common Fedora problem, nobody responds to it and the bug reports are left unprocessed and pile up. And because none of the maintainers respond at all, we have now also missed all the deadlines for improving something for f40.
Sad but not surprising...
- Would someone take the bug report and check my analyses?
Looking at httpd's git repo, it doesn't look like we change the configuration as shipped upstream to a significant degree. And, looking at httpd's subversion repo, _default_ is used both at the tip of the 2.4 branch and trunk.
Question is, what is upstream? If I remember correctly, we use historically httpd, which is a specially licensed version of Apache, so that Red Hat can distribute an Apache server under its own name. I don't remember the exact details, the origin is about 2 decades ago. In any case, the original, "real" upstream is Apache.
And "real" upstream has changed the meaning of the string _default_ with version 2.4. Up to 2.3, this specifically referred to the default server, which is used if none of the configured ServerNames or ServerAlias correspond to a request. This was abandoned with 2.4 and the string _default_ has been a synonym for "*" since the first release of version 2.4, so it applies to all Servers!
And since then, the default server is the configuration which was find first in the alphabetical sorting. So when the configuration for the default server is determined, the configuration from httpd.conf is taken first, then the one for * (now the one with the server name _default_) and then the vhost configuration file, which is the first in alphabetical order. And since the vhosts inherit the general settings, they also inherit, for example, the DocumentRoot on the _default_ server for port 443 in ssl.conf and - even worse - the self-signed certificate created there. The user therefore creates a certificate in his configuration file, but this is not used because it has already been configured in _default_.
If you look at a Fedora version with Apache 2.3 - which must be something around Fedora 20 or maybe even earlier - you will find the same ssl.conf as now. This lets me assume that the change to the string _default_ was overlooked back then. Presumably the admins have since developed various local customizations for themselves, or they have immediately used Debian or similar for web servers. In any case, I have been deleting the ssl.conf for years and configuring everything in the host files. But that really isn't a long-term solution.
I'll also argue that using virtual hosts for the domain matching your server's name is a mistake but I'm not sure how popular the sentiment is...
- Should we create a new https-server package which takes the binaries and provides an alternative configuration? Probably similar to default-editor-vim / default-editor-nano?
I really don't like this idea. I would much rather we work with upstream to improve the default configuration.
I really don’t like that either. But what can we do if the maintainer(s) won't budge? Should we contact them directly? I don’t even know, who is maintainer and co-maintainer.
A configuration that was good 10 or 20 years ago may no longer be suitable today.
-- Peter Boy https://fedoraproject.org/wiki/User:Pboy PBoy@fedoraproject.org
Timezone: CET (UTC+1) / CEST (UTC+2)
Fedora Server Edition Working Group member Fedora Docs team contributor and board member Java developer and enthusiast
* Peter Boy [14/02/2024 11:08] :
Question is, what is upstream? If I remember correctly, we use historically httpd, which is a specially licensed version of Apache,
I believe (and I hope I'm corrected if I'm wrong) that, 20 years ago, the ASF asked that the server be refferred to as 'httpd' or 'Apache httpd' and that 'Apache' serve primarily to refer to the foundation.
So the package name was changed from 'apache' to 'httpd'.
If you look at a Fedora version with Apache 2.3 - which must be something around Fedora 20 or maybe even earlier
FTR, we've never shipped 2.3 which was a developement branch. we shipped 2.2 until Fedora 17 and have shipped 2.4 since Fedora 18 included.
Emmanuel
On Wed, Feb 14, 2024 at 06:46:30PM +0100, Emmanuel Seyman wrote:
- Peter Boy [14/02/2024 11:08] :
Question is, what is upstream? If I remember correctly, we use historically httpd, which is a specially licensed version of Apache,
I believe (and I hope I'm corrected if I'm wrong) that, 20 years ago, the ASF asked that the server be refferred to as 'httpd' or 'Apache httpd' and that 'Apache' serve primarily to refer to the foundation.
So the package name was changed from 'apache' to 'httpd'.
Yup, exactly - we renamed the package when upgrading from 1.3 to 2.0 in Red Hat Linux 9 (IIRC) since that was a breaking change anyway.
"httpd" is the proper name of the project under the umbrella of the Apache Software Foundation - "Apache httpd" aka "Apache HTTP Server". The tarballs started being called "httpd-X" from upstream starting in 2.0.x as well. So it made sense to reflect that in the packaging.
Regards, Joe
Am 15.02.2024 um 11:39 schrieb Joe Orton jorton@redhat.com:
On Wed, Feb 14, 2024 at 06:46:30PM +0100, Emmanuel Seyman wrote:
- Peter Boy [14/02/2024 11:08] :
Question is, what is upstream? If I remember correctly, we use historically httpd, which is a specially licensed version of Apache,
I believe (and I hope I'm corrected if I'm wrong) that, 20 years ago, the ASF asked that the server be refferred to as 'httpd' or 'Apache httpd' and that 'Apache' serve primarily to refer to the foundation.
So the package name was changed from 'apache' to 'httpd'.
Yup, exactly - we renamed the package when upgrading from 1.3 to 2.0 in Red Hat Linux 9 (IIRC) since that was a breaking change anyway.
"httpd" is the proper name of the project under the umbrella of the Apache Software Foundation - "Apache httpd" aka "Apache HTTP Server". The tarballs started being called "httpd-X" from upstream starting in 2.0.x as well. So it made sense to reflect that in the packaging.
Regards, Joe
Thanks for the info!
My memory is obviously a bit cloudy :-)
-- Peter Boy https://fedoraproject.org/wiki/User:Pboy PBoy@fedoraproject.org
Timezone: CET (UTC+1) / CEST (UTC+2)
Fedora Server Edition Working Group member Fedora Docs team contributor and board member Java developer and enthusiast
On Tue, Feb 13, 2024 at 10:36:44AM +0100, Peter Boy wrote:
Hi folks,
We have a serious issue with the current Apache httpd package, which is an indispensable service for a server these days.
The issue is basically the default configuration as provided by the package:
- if you create just one server in den .conf directory, it results in a non-functional https configuration
- the configuration uses items, that are deprecated since version 2.4
- the default layout of the default configuration is currently not suitable
I wrote a bug report about it: https://bugzilla.redhat.com/show_bug.cgi?id=2258121
Hi Peter,
This is definitely an interesting & complex topic. Sorry nobody has replied on your bug, I did read it but... it's a big area and haven't had time to think it through properly.
Do you have a concrete proposal on exactly what you'd like to do differently? I'm guessing you do but it's not really laid out in the bug. Going through some of my thoughts:
1. The default ssl.conf & vhost definition is difficult/awkward, I definitely agree here. I suppose there are several parts to this -
a) All the stuff outside the vhost in ssl.conf should be a hard-coded default or unnecessary - some coding probably required, e.g. I already made SSLRandom a no-op upstream since it doesn't make sense with modern OpenSSL.
b) Can & should we retain a default :443 vhost with generated certs? I *think* yes, but am quite open to persuasion on this.
2. Including mod_ssl as part of httpd I definitely don't agree with and is technically not necessary; it may be better to argue that we should "Recommends: mod_ssl". We will never make https:// "work" out of the box without active user intervention, at least since ACME CAs require a subscriber agreement contract.
3. In thinking about how we do SSL vhost config, I think it's important to consider all use cases - mod_md-managed, certbot-driven configs, or "traditional CA" users. (Particularly for ACME with DNS validation it's easier to get going with certbot than mod_md.)
4. Backwards compatibility is important and hopefully everybody agrees that we shouldn't break the upgrade path (beyond maybe removing or moving ssl.conf itself).
5. Not sure about anybody else, but I'm still not a fan of the Debian style split -enabled/-available config directories which has always seemed over-engineered to me. I'm open to being convinced we should have a new config directory in /etc/httpd which is included by default if necessary.
So, thoughts?
Regards, Joe
Hi Joe,
Many thanks for your feedback. I am very excited that we can now make a positive impact together and pushing things forward.
Am 14.02.2024 um 17:01 schrieb Joe Orton jorton@redhat.com:
On Tue, Feb 13, 2024 at 10:36:44AM +0100, Peter Boy wrote:
Hi folks,
We have a serious issue with the current Apache httpd package, which is an indispensable service for a server these days.
The issue is basically the default configuration as provided by the package:
- if you create just one server in den .conf directory, it results in a non-functional https configuration
- the configuration uses items, that are deprecated since version 2.4
- the default layout of the default configuration is currently not suitable
I wrote a bug report about it: https://bugzilla.redhat.com/show_bug.cgi?id=2258121
Hi Peter,
This is definitely an interesting & complex topic. Sorry nobody has replied on your bug, I did read it but... it's a big area and haven't had time to think it through properly.
Yes, probably every one of us gets swamped at times. I was probably a bit too impatient and too keen to improve our server edition.
Do you have a concrete proposal on exactly what you'd like to do differently? I'm guessing you do but it's not really laid out in the bug. Going through some of my thoughts:
Yes, I do or better we, as Server WG do. And because httpd is such an important service I would we ready to create a PR for a new or adjusted configuration. And also assist with the configuration part (The last time I compiled a C program is back more then one decade).
- The default ssl.conf & vhost definition is difficult/awkward, I
definitely agree here. I suppose there are several parts to this -
a) All the stuff outside the vhost in ssl.conf should be a hard-coded default or unnecessary - some coding probably required, e.g. I already made SSLRandom a no-op upstream since it doesn't make sense with modern OpenSSL.
Yes, I agree here. We should remove the vhost part, but retain it and put it elsewere.
As far as I understand (but am really not sure I do) the openssl package provides a system wide default and secure configuration, e.g. the optimal cipher list and anything like that. If this is indeed the case, wouldn't it make sense to use this in ssl.conf or to refer to it? And to point out that you should not make any changes that you might find in any instructions on the Internet. What I keep finding in discussions about this is often adventurous and a bad improvement or outdated. Such a centrally maintained and tested security configuration would be a distinguishing feature for Fedora as a whole. (Sorry, I’m not a marketing guy, but have to promote Fedora and Fedora Server as part of my engagement in Server WG).
b) Can & should we retain a default :443 vhost with generated certs? I *think* yes, but am quite open to persuasion on this.
I agree. We should ensure, that httpd is ready to work out of the box after installation. And today it is (quite predominantly) https with automatic forwarding http.
- Including mod_ssl as part of httpd I definitely don't agree with and
is technically not necessary; it may be better to argue that we should "Recommends: mod_ssl". We will never make https:// "work" out of the box without active user intervention, at least since ACME CAs require a subscriber agreement contract.
Well, I think we can and should treat mod_ssl the same way as a lot of other modules, e.g proxy, mod_auth and anything else that we currently install as part of the httpd package, and activate and configure in ~/conf.modules.d/. I suppose, mod_ssl is used much more frequently today than many of the modules that we already install and configure. Virtually no web server can manage without https - except for backend servers in containers or internal VMs (Perhaps we should offer a separate configuration for this, which is much more slim and streamlined).
And we make it work out of the box as we currently do, using self signed certificates for a start, and provide info how to improve with md and „real“ certificates.
- In thinking about how we do SSL vhost config, I think it's important
to consider all use cases - mod_md-managed, certbot-driven configs, or "traditional CA" users. (Particularly for ACME with DNS validation it's easier to get going with certbot than mod_md.)
My idea / wish is to provide a minimal but complete default vhost configuration, that works out of the box with self-signed certificates to start with and contains templates how to engage md-managed and certbot-driven configs and various additional services.
You can find a first try of such a vhost config file in our Server documentation "Setting up a basic web server" (https://docs.fedoraproject.org/en-US/fedora-server/services/httpd-basic-setu...) or download the file (https://docs.fedoraproject.org/en-US/fedora-server/_attachments/services/htt...) to inspect it.
Im currently working on a Guide to configure an Apache proxy and html backend server and a proxy for a Java backends via ajp protocol. Currently, I’m planning this for server documentation, but we can also revive the Web part of the Fedora Administrator’s Guide and provide documentation and sample configuration files for various typical purposes there.
The key point is to include a sample configuration that is a good starting point for administrators to customize for their own purposes.
- Backwards compatibility is important and hopefully everybody agrees
that we shouldn't break the upgrade path (beyond maybe removing or moving ssl.conf itself).
This is also an essential requirement for Server WG.
Perhaps it is best to leave the existing directory structure as it is for existing installations, and to introduce a new configuration directory for vhosts that is included in http.conf after conf.d. On my servers I use ~/conf.vhosts.d/*.vhost files. And we can deprecate ~/conf.d for vhost and specific server configuration, but leaving it for general, cross-vhost basic configurations. Application programs such as wordpress can continue to store their basic configuration there.
- Not sure about anybody else, but I'm still not a fan of the Debian
style split -enabled/-available config directories which has always seemed over-engineered to me. I'm open to being convinced we should have a new config directory in /etc/httpd which is included by default if necessary.
So, thoughts?
I totally agree regarding the Debian configuration system. I find this not only over-engineered but also very confusing.
Even with Debian's system, you will eventually reach the point where you have to write a configuration file. It is then more effective to write a configuration file that combines everything necessary in one place and can be easily adapted and extended. To deactivate, simply add an .OFF to the file name. I also find this clearer, you have everything in one place.
I think it is still important to "officially" introduce a directory structure in /var/www for web server data that is vhost or domain-oriented. For example /var/www/<domain[1..x]/[htdocs|auth.d|cgi|...] "officially". Most installations will host more than one domain and even with a proxy server you need a separate data area for each proxy domain.
Maybe I'm thinking too much as a system administrator and trying to make it as easy as possible for them. But maybe that's not entirely wrong.
Best Peter
-- Peter Boy https://fedoraproject.org/wiki/User:Pboy PBoy@fedoraproject.org
Timezone: CET (UTC+1) / CEST (UTC+2)
Fedora Server Edition Working Group member Fedora Docs team contributor and board member Java developer and enthusiast
Hi Joe,
Amazing how time goes by. It’s about a year ago that we wrote about a possible improvement of the Fedora Apache default configuration. I was quite busy with various issues with Fedora Server Edition and some accidents, too, that handicapped me.
But now Server WG is dealing with httpd config again. And I wrote a reworked installation documentation for Fedora Server, which details the ideas we have for web service in Fedora. Some are very Server specific, e.g. the part about storage organizations and LVM volumes. Others are more httpd specific, e.g. SSL and vhosts.
See: https://docs.stg.fedoraproject.org/en-US/fedora-server/services/httpd-basic-...
The article includes some downloadable config file modifications (currently on my development server until we find a place in Fedora infrastructure).
It’s just the first part. Further parts deal with specific uses cases, especially with configuration as reverse proxy for containerized apps, classic web applications as Wildly, etc.
I would be happy if we could manage to discuss about it and find a way for Server WG to avoid having to tinker with the default configurations. I would very much appreciate it if we could work together to coordinate the configuration requirements. I have three potential applications scenarios in mind (full-fledged standalone servers, reverse proxies, and backend communicators for web apps like WordPress in a container). Different Apache modules could be put together for this and Apache could be made leaner. This would counter the criticism of sluggishness and a heavy footprint compared to alternatives such as nginx and lighttpd.
Thanks in advance
Peter
P.S. Maybe, we could write together a Fedora Magazine article? Or a series thereof?
P.P.S Are you attending FOSDEM in Brussels this year by any chance?
Am 14.02.2024 um 17:01 schrieb Joe Orton jorton@redhat.com:
On Tue, Feb 13, 2024 at 10:36:44AM +0100, Peter Boy wrote:
Hi folks,
We have a serious issue with the current Apache httpd package, which is an indispensable service for a server these days.
The issue is basically the default configuration as provided by the package:
- if you create just one server in den .conf directory, it results in a non-functional https configuration
- the configuration uses items, that are deprecated since version 2.4
- the default layout of the default configuration is currently not suitable
I wrote a bug report about it: https://bugzilla.redhat.com/show_bug.cgi?id=2258121
Hi Peter,
This is definitely an interesting & complex topic. Sorry nobody has replied on your bug, I did read it but... it's a big area and haven't had time to think it through properly.
Do you have a concrete proposal on exactly what you'd like to do differently? I'm guessing you do but it's not really laid out in the bug. Going through some of my thoughts:
- The default ssl.conf & vhost definition is difficult/awkward, I
definitely agree here. I suppose there are several parts to this -
a) All the stuff outside the vhost in ssl.conf should be a hard-coded default or unnecessary - some coding probably required, e.g. I already made SSLRandom a no-op upstream since it doesn't make sense with modern OpenSSL.
b) Can & should we retain a default :443 vhost with generated certs? I *think* yes, but am quite open to persuasion on this.
- Including mod_ssl as part of httpd I definitely don't agree with and
is technically not necessary; it may be better to argue that we should "Recommends: mod_ssl". We will never make https:// "work" out of the box without active user intervention, at least since ACME CAs require a subscriber agreement contract.
- In thinking about how we do SSL vhost config, I think it's important
to consider all use cases - mod_md-managed, certbot-driven configs, or "traditional CA" users. (Particularly for ACME with DNS validation it's easier to get going with certbot than mod_md.)
- Backwards compatibility is important and hopefully everybody agrees
that we shouldn't break the upgrade path (beyond maybe removing or moving ssl.conf itself).
- Not sure about anybody else, but I'm still not a fan of the Debian
style split -enabled/-available config directories which has always seemed over-engineered to me. I'm open to being convinced we should have a new config directory in /etc/httpd which is included by default if necessary.
So, thoughts?
Regards, Joe
server mailing list -- server@lists.fedoraproject.org To unsubscribe send an email to server-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
— Peter Boy https://fedoraproject.org/wiki/User:Pboy PBoy@fedoraproject.org
Timezone: CET (UTC+1) / CEST (UTC+2)
Fedora Server Edition Working Group member Fedora Docs team contributor and board member Java developer and enthusiast
server@lists.fedoraproject.org
-
Emmanuel Seyman -
Joe Orton -
Peter Boy -
Peter Boy Uni