Hi,
In the koji/__init__.py:fastUpload method there is a blocksize parameter that is used to determine how many bytes are uploaded to the koji hub per request. By default it is set to 1MB and as far as I can tell that is the value actually used. Is there a way to change this value via koji configuration? A larger blocksize value could significantly improve the upload throughput.
-- Dennis
On 03/01/2016 11:09 AM, Dennis Gregorovic wrote:
Hi,
In the koji/__init__.py:fastUpload method there is a blocksize parameter that is used to determine how many bytes are uploaded to the koji hub per request. By default it is set to 1MB and as far as I can tell that is the value actually used. Is there a way to change this value via koji configuration? A larger blocksize value could significantly improve the upload throughput.
There is no way to change it via configuration. It used to be smaller and we raised it for performance. At the time at least, 1M seemed to be roughly the point of diminishing returns. It really depends on the connection overhead.
Speaking of connection overhead, another factor to consider is whether keepalive is enabled on the httpd server that runs the hub.
Worth noting that there is also a hard coded block size on the other end.
Does anyone have time to experiment with different permutations?
-- Dennis _______________________________________________ koji-devel mailing list koji-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/koji-devel@lists.fedorahosted.org
On Tue, Mar 1, 2016 at 4:48 PM, Mike McLean mikem@redhat.com wrote:
On 03/01/2016 11:09 AM, Dennis Gregorovic wrote:
Hi,
In the koji/__init__.py:fastUpload method there is a blocksize parameter that is used to determine how many bytes are uploaded to the koji hub per request. By default it is set to 1MB and as far as I can tell that is the value actually used. Is there a way to change this value via koji configuration? A larger blocksize value could significantly improve the upload throughput.
For people on the local LAN sure, but the further away you are the more likely you'll have issues, I suspect people in India, Australia and other places much more will cause issues.
There is no way to change it via configuration. It used to be smaller and we raised it for performance. At the time at least, 1M seemed to be roughly the point of diminishing returns. It really depends on the connection overhead.
Is it hard to make it a config option?
Worth noting that there is also a hard coded block size on the other end.
Why?
Does anyone have time to experiment with different permutations?
Ah, time, I wish.
On 03/01/2016 12:05 PM, Peter Robinson wrote:
On Tue, Mar 1, 2016 at 4:48 PM, Mike McLean mikem@redhat.com wrote:
On 03/01/2016 11:09 AM, Dennis Gregorovic wrote:
Hi,
In the koji/__init__.py:fastUpload method there is a blocksize parameter that is used to determine how many bytes are uploaded to the koji hub per request. By default it is set to 1MB and as far as I can tell that is the value actually used. Is there a way to change this value via koji configuration? A larger blocksize value could significantly improve the upload throughput.
For people on the local LAN sure, but the further away you are the more likely you'll have issues, I suspect people in India, Australia and other places much more will cause issues.
There is no way to change it via configuration. It used to be smaller and we raised it for performance. At the time at least, 1M seemed to be roughly the point of diminishing returns. It really depends on the connection overhead.
Is it hard to make it a config option?
Worth noting that there is also a hard coded block size on the other end.
Why?
Does anyone have time to experiment with different permutations?
Ah, time, I wish.
I haven't tested with brew, but we did some brief testing with uploads to Pulp, which uses a similar upload model. Over a wan, we saw an 8x increase in upload throughput when using a 10MB block size vs 1MB.
I don't know if upload speed is generally an issue or not for koji deployments, but I wanted to throw this out there. I've created an RFE for the backlog.
https://pagure.io/koji/issue/30
Cheers -- Dennis
On Tuesday, March 01, 2016 01:44:12 PM Dennis Gregorovic wrote:
On 03/01/2016 12:05 PM, Peter Robinson wrote:
On Tue, Mar 1, 2016 at 4:48 PM, Mike McLean mikem@redhat.com wrote:
On 03/01/2016 11:09 AM, Dennis Gregorovic wrote:
Hi,
In the koji/__init__.py:fastUpload method there is a blocksize parameter that is used to determine how many bytes are uploaded to the koji hub per request. By default it is set to 1MB and as far as I can tell that is the value actually used. Is there a way to change this value via koji configuration? A larger blocksize value could significantly improve the upload throughput.
For people on the local LAN sure, but the further away you are the more likely you'll have issues, I suspect people in India, Australia and other places much more will cause issues.
There is no way to change it via configuration. It used to be smaller and we raised it for performance. At the time at least, 1M seemed to be roughly the point of diminishing returns. It really depends on the connection overhead.
Is it hard to make it a config option?
Worth noting that there is also a hard coded block size on the other end.
Why?
Does anyone have time to experiment with different permutations?
Ah, time, I wish.
I haven't tested with brew, but we did some brief testing with uploads to Pulp, which uses a similar upload model. Over a wan, we saw an 8x increase in upload throughput when using a 10MB block size vs 1MB.
I don't know if upload speed is generally an issue or not for koji deployments, but I wanted to throw this out there. I've created an RFE for the backlog.
https://pagure.io/koji/issue/30
Cheers -- Dennis
Thanks Dennis,
Uploads of big items like live iso's, cloud and arm images are extremely slow in fedora. It takes somewhere around 10 to 20 Minutes to upload 1Gb
Dennis
On 03/02/2016 08:57 AM, Dennis Gilmore wrote:
On Tuesday, March 01, 2016 01:44:12 PM Dennis Gregorovic wrote:
On 03/01/2016 12:05 PM, Peter Robinson wrote:
On Tue, Mar 1, 2016 at 4:48 PM, Mike McLean mikem@redhat.com wrote:
On 03/01/2016 11:09 AM, Dennis Gregorovic wrote:
Hi,
In the koji/__init__.py:fastUpload method there is a blocksize parameter that is used to determine how many bytes are uploaded to the koji hub per request. By default it is set to 1MB and as far as I can tell that is the value actually used. Is there a way to change this value via koji configuration? A larger blocksize value could significantly improve the upload throughput.
For people on the local LAN sure, but the further away you are the more likely you'll have issues, I suspect people in India, Australia and other places much more will cause issues.
There is no way to change it via configuration. It used to be smaller and we raised it for performance. At the time at least, 1M seemed to be roughly the point of diminishing returns. It really depends on the connection overhead.
Is it hard to make it a config option?
Worth noting that there is also a hard coded block size on the other end.
Why?
Does anyone have time to experiment with different permutations?
Ah, time, I wish.
I haven't tested with brew, but we did some brief testing with uploads to Pulp, which uses a similar upload model. Over a wan, we saw an 8x increase in upload throughput when using a 10MB block size vs 1MB.
I don't know if upload speed is generally an issue or not for koji deployments, but I wanted to throw this out there. I've created an RFE for the backlog.
https://pagure.io/koji/issue/30
Cheers -- Dennis
Thanks Dennis,
Uploads of big items like live iso's, cloud and arm images are extremely slow in fedora. It takes somewhere around 10 to 20 Minutes to upload 1Gb
Dennis
That very well could be related to the chunk size. I'll follow up on the Fedora rel-eng list to see about testing that.
koji-devel@lists.fedorahosted.org
-
Dennis Gilmore -
Dennis Gregorovic -
Mike McLean -
Peter Robinson