On 07/29/2014 05:43 PM, Mike McLean wrote:
These changes cause the hub to use generators for some potentially
large
queries internally. The point is to reduce memory usage.
In the case that a large query is returned via rpc, the marshaller has
been extended to handle generators. The marshaller will still create a
giant pile of xml, but we at least still save the memory for the
original data.
I also have some work on an iterating marshaller that will not keep the
entire xml return in memory, but that's a bit more complicated, so I'm
going to leave that for another day.
There are other cases (e.g. repo_init) where we have very large queries
internally. In these cases, this patch set will result in significant
memory savings.
A few more related changes, additional logging and a bugfix.
Also, will probably drop the chunksize patch, doesn't seem to matter
when unless the db connection has a lot of lag.
These changes are working well for me and are likely to land in git soon.