AS7 Plugin - Secure Connections to Server

Stefan Negrea snegrea at redhat.com
Fri Aug 10 22:00:25 UTC 2012


Hello,

Sorry for the delayed response... I think we are converging unto something very interesting but at the same time it is good to keep in mind short vs. long term scope.


First, short term. The changes I made so far can be merged in master today and be available in the next RHQ release. It would allow users to use secure connections to AS7 immediately. For certificates signed with a root CA included by default in the JVM this code will work flawlessly without configuration. Also, no further action action is required for these certificates even if the long term solution from below is implemented. The code works well with self signed certificates too but the agents will need some extra manual configuration. For self-signed certificates we would need to document the two alternatives to configure the JVM itself to take such certificates (options 1 and 2 from my original email).

The million dollar question .... Do we add this small change today and enable secure connections (with some manual configuration required for self-signed certificates only)? Or wait for the more complex solution that will undoubtedly require a lot more development? The long term solution below just adds better support for self-signed certificates only ...


Now the long term solution. Here is what I can summarize from this long chain of emails and conversions I had about this feature:
1) There should be a central place where users configure certificate stores (admin section?). This central place should allow the users to enter detailed configuration for as many certificate stores as needed (eg. path, password for the certificate store, certificate identity to be used).
2) Then individual resource should have a single field "certificate configuration", allowing the user to pick a predefined certificate configuration. This should work exactly like the "dependent values" implemented for the AS7 plugin.
3) If 1 and 2 are not implemented, each resource that can use certificates will have same 100 extra fields needed to configure the certificate store. I am exaggerating a bit with 100 options but it can be overwhelming. Also, it would be very tedious for plugin writers since they would need to include a lot of extra fields to enable this functionality for a single resource type.
4) The server should send to each resource configured to use certificate stores all the nifty-gritty details about the selected certificate store.


Here are some open questions:
1) If the RHQ server-agent communication is not secured, should the users be allowed to use certificate stores? The server will push down sensitive data to the agent, such as the password for the certificate store. If the communication is not secured, then this data can be intercepted and exploited by an attacker. 
2) Should the certificate store be configurable at server level or service level? We could restrict the implementation to allow a single certificate store for a server resource; all sub-resources inherit the configuration from the parent. But we could also allow certificate stores at service level.
3) Should we allow users to upload certificate stores in RHQ, and then push the certificate stores down to the individual agents along with the rest of the configuration?


I will create a wiki design page for this feature to capture this long trail of emails.



But first lets see if we can decide about the short solution ....



Thank you,
Stefan Negrea


----- Original Message -----
> From: "Larry O'Leary" <loleary at redhat.com>
> To: rhq-devel at lists.fedorahosted.org
> Sent: Wednesday, August 1, 2012 3:22:41 PM
> Subject: Re: AS7 Plugin - Secure Connections to Server
> 
> On Wed, 2012-08-01 at 13:23 -0400, Stefan Negrea wrote:
> > 
> > ----- Original Message -----
> > > From: "Larry O'Leary" <loleary at redhat.com>
> > > To: rhq-devel at lists.fedorahosted.org
> > > Sent: Wednesday, August 1, 2012 10:53:16 AM
> > > Subject: Re: AS7 Plugin - Secure Connections to Server
> > > 
> > > On Wed, 2012-08-01 at 11:36 -0400, Stefan Negrea wrote:
> > > > 
> > > > 1) Rely on users to import self-signed certificates into the
> > > > default JVM keystore. Here is a good guide on how to do this
> > > > http://www.grim.se/guide/jre-cert . This approach does not
> > > > require
> > > > any changes to the code posted above, we will just need to
> > > > document this in the RHQ wiki pages.
> > > > 
> > > > *I expect security savvy users to create an internal
> > > > certificate
> > > > signer and then import this certificate in all the JVMs in the
> > > > infrastructure. All certificates signed with this root
> > > > certificate
> > > > will then be accepted by the JVM like any other certificate
> > > > signed
> > > > by one of the big certificate signers.
> > > 
> > > Bad idea and not secure as the JVM may be used for other
> > > applications.
> > > Additionally, this hadn't worked in the past when securing
> > > agent-to-server communications as the keystore defined there
> > > overrides
> > > the system's default.
> > 
> > 
> > The agent-to-server comm code explicitly overrides the default
> > keystore and only for the comm layer purposes (the settings are
> > not applied at JVM level). This is not the case for the AS7
> > plugin.
> > 
> > I will reiterate this, from my experience a common use case is for
> > users to create a local (for their entire environment) signing
> > authority and import only that in each JVM. In that case, no
> > specific certificate will ever be imported in the actual JVMs.
> > With this signer certificate imported, each application could use
> > any certificate signed by this "root" certificate as if it is a
> > certificate signed by one of the major certificate signers.
> 
> Perhaps in development, but not in a production system. Instead,
> trusted
> CAs are added to the JVM but application specific CAs are added to
> the
> application. This allows tighter control over certificate chain
> validation without the need to trust everyone for everything. For a
> simple client/server this approach
> 
> > > 
> > > > 2) Rely on users to provide the JVM with a certificate store
> > > > that
> > > > contains the certificate to be used. This is very similar to
> > > > Option 1 but the users will not modify the default store,
> > > > rather
> > > > they will configure the JVM to use a custom store. As previous
> > > > option, we will just need to document this in the RHQ wiki
> > > > pages.
> > > > 
> > > > Step 1) Export the certificate from the server
> > > > Step 2) Create the keystore with the given certificate:
> > > > http://docs.oracle.com/javase/tutorial/security/toolsign/rstep2.html
> > > > Step 3) Include the following line in the rhq-agent-env.sh:
> > > > RHQ_AGENT_ADDITIONAL_JAVA_OPTS="-Djavax.net.ssl.trustStore=/path/to/keystore
> > > > -Djavax.net.ssl.trustStorePassword=s3cr3t"
> > > 
> > > This is on the right track but still not ideal. Because other
> > > plug-ins
> > > are installed and require the use of trust stores, such an
> > > implementation is overriding the JVMs system-wide store.
> > 
> > In this case all the plugins will need to use one single keystore.
> 
> Which is what we need to prevent and is the exact reason that
> connection
> level properties should be used. As per the JVM API the default
> implementation is available but it is recommended that the connection
> handle access to a configured store. A store can only contain a
> single
> type of certificate while certificates range in type.
> 
> > > Additionally,
> > > this had not worked in the past when securing agent-to-server
> > > communications.
> > > 
> > > The standard implementation for this is to define properties
> > > specific
> > > to
> > > the plug-in and the user specify the location of the trust store
> > > (and
> > > key store if 2-way SSL is required) along with the required
> > > passwords
> > > and store types. You specify these values either via command-line
> > > arguments (non-optimal as they are for each resource) or via
> > > plug-in
> > > configuration for that connection. Then you would use a custom
> > > SSL
> > > factory which loaded the correct stores and configuration on a
> > > resource
> > > basis instead of globally in the JVM.
> > 
> > 
> > This leads to a bigger discussion. How to design a facility to make
> > certificates and certificate stores available to plugins. Since
> > this initial AS7 plugin patch would just provide support for
> > one-way encryption the code is much simpler and the requirements
> > are not as complex as agent-server comm layer. I have not
> > evaluated yet the need to use two-way encryption with AS7 but we
> > will not be able to implement two-way encryption without
> > implementing a generic certificate infrastructure for plugins.
> 
> Agreed. But the initial implementation doesn't have to be plug-in
> wide.
> Instead, it could be implemented at the plug-in level as a connection
> property. Basically, you need to throw out your idea of
> javax.net.ssl.*
> properties and properly implement the SSL socket factory. The
> javax.net.ssl.* were intended for simple use-cases in where a single
> JVM
> needed to communicate with a single server. In most cases, this was
> for
> RMI.
> 
> > Also, the individual certificate password problem will never occur
> > in this case because the agent is just like a simple browser. All
> > what users would need to import into a keystore is the public cert
> > (which is never password protected).
> 
> Although it is not common, public keys can be password protected at
> the
> store level. This prevent tampering with the store. Therefore, a
> password may still be required to read the fingerprint. Perhaps this
> is
> an edge case and many implementations do not support it but it is
> simple
> to implement.
> 
> > 
> > > 
> > > The fall-back is if the properties are not explicitly defined,
> > > the
> > > JVM
> > > global option is used allowing the user to still do what you
> > > propose
> > > as
> > > a fall-back method if they do not want to explicitly define it
> > > for
> > > each
> > > AS7 resource.
> > 
> > 
> > There is one more problem with properties listed in the
> > agent-configuration.xml. Every time something changes or a new
> > store needs to be added, the user will have to go through an agent
> > reconfiguration (rhq-agent.sh --cleanconfig). I do not think it is
> > a good idea to expose properties for every single plugin. We will
> > just make the configuration very tedious.
> 
> I am not sure how you got that from what I said. This is a connection
> property in the resource itself. Configuration is maintained on the
> server.
> 
> > Think about this use case. The agent is configured, running, and
> > has some resources inventoried. The user installs an AS7 server
> > and secures the http management interface with a self-signed
> > certificate. The user would then need to stop the agent, and then
> > manually go through the agent re-configuration process just to
> > pick up a keystore for one plugin. Far from ideal... That is why I
> > think JVM level options are much better alternative. The user
> > should put all these self-signed certificates in a single store
> > for our agent to use. It's a more natural approach and no full
> > agent reconfiguration every time a new certificate is needed.
> 
> No restart of the agent should be needed.
> 
> > Our target user for this feature should be somebody already savvy
> > with certificates and usage with Java. If somebody goes through
> > the trouble of securing AS7 communication channels with
> > certificates, then I expect them to have an advanced understanding
> > of SSL/TLS and certificates. Thus no reason to design the AS7
> > feature for somebody with no knowledge into this (eg. 4 layers of
> > fallback hoping that they will get it right somehow).
> 
> That is a scary statement. I have not met too many system
> administrators
> who are "already savvy with certificates and usage with Java." It is
> the
> exact opposite. The issues that I have dealt with in the RHQ arena
> require hand holding because our existing process is so complex.
> System
> administrators have more important things to do then understanding
> developer tools provided with a JDK.
> 
> > 
> > Do we have any indication that our users use self-signed
> > certificates?
> 
> Yes. A mix of self-signed certificates along with certificates issued
> by
> a CA which are not yet trusted by the JVM.
> 
> --
> Larry O'Leary
> https://plus.google.com/u/0/112645929986009801513
> 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> 


More information about the rhq-devel mailing list