These changes cause the hub to use generators for some potentially large queries internally. The point is to reduce memory usage.
In the case that a large query is returned via rpc, the marshaller has been extended to handle generators. The marshaller will still create a giant pile of xml, but we at least still save the memory for the original data.
I also have some work on an iterating marshaller that will not keep the entire xml return in memory, but that's a bit more complicated, so I'm going to leave that for another day.
There are other cases (e.g. repo_init) where we have very large queries internally. In these cases, this patch set will result in significant memory savings.
On 07/29/2014 05:43 PM, Mike McLean wrote:
These changes cause the hub to use generators for some potentially large queries internally. The point is to reduce memory usage.
In the case that a large query is returned via rpc, the marshaller has been extended to handle generators. The marshaller will still create a giant pile of xml, but we at least still save the memory for the original data.
I also have some work on an iterating marshaller that will not keep the entire xml return in memory, but that's a bit more complicated, so I'm going to leave that for another day.
There are other cases (e.g. repo_init) where we have very large queries internally. In these cases, this patch set will result in significant memory savings.
A few more related changes, additional logging and a bugfix.
Also, will probably drop the chunksize patch, doesn't seem to matter when unless the db connection has a lot of lag.
These changes are working well for me and are likely to land in git soon.
On 7/30/14, 2:32 PM, Mike McLean wrote:
On 07/29/2014 05:43 PM, Mike McLean wrote:
These changes cause the hub to use generators for some potentially large queries internally. The point is to reduce memory usage.
In the case that a large query is returned via rpc, the marshaller has been extended to handle generators. The marshaller will still create a giant pile of xml, but we at least still save the memory for the original data.
I also have some work on an iterating marshaller that will not keep the entire xml return in memory, but that's a bit more complicated, so I'm going to leave that for another day.
There are other cases (e.g. repo_init) where we have very large queries internally. In these cases, this patch set will result in significant memory savings.
A few more related changes, additional logging and a bugfix.
Also, will probably drop the chunksize patch, doesn't seem to matter when unless the db connection has a lot of lag.
These changes are working well for me and are likely to land in git soon.
Here's a small patch to fix chain-maven, which can override a G:A:V in the tag with a build from the chain. Tested locally and it fixes the regression in chain-maven builds for me.
buildsys@lists.fedoraproject.org
-
Mike Bonnet -
Mike McLean