Are we still planning to completely replace termcap with ncurses in Fedora?
While I welcome reducing the number of dupe packages, I'm worried by the impact of this change on the size and performance of many core programs.
libncurses is much bigger than libtermcap, and also has oddly large .bss and .data sections:
bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2 text data bss dec hex filename 319006 56608 3592 379206 5c946 /lib64/libncurses.so.5 10483 788 112 11383 2c77 /lib64/libtermcap.so.2
this is going to impact very negatively on the RSS of several critical programs such as bash and python.
I'm also worried that the overall time required spent for a fork may increase considerably.
But maybe ncurses can easily be fixed to avoid global data and buffers?
Ralf Ertzinger wrote:
On Fri, 01 Dec 2006 10:13:55 +0100, Bernardo Innocenti wrote:
I'm also worried that the overall time required spent for a fork may increase considerably.
Why would that be? The library is shared, and Linux has been doing COW for ages.
COW only saves the memcpy(), you still have to do some per-page book keeping at fork time, I suppose (*).
And if those pages are actually being written by the forked process, it's going to cost a page fault and a memcpy(). That's why I wrote "overall time spent for a fork".
(*) on second thought, it may actually be a per-vma thing, which would make the nr. of pages irrelevant. I shall study the Linux VM one day...
On Fri, Dec 01, 2006 at 10:13:55AM +0100, Bernardo Innocenti wrote:
Are we still planning to completely replace termcap with ncurses in Fedora?
While I welcome reducing the number of dupe packages, I'm worried by the impact of this change on the size and performance of many core programs.
libncurses is much bigger than libtermcap, and also has oddly large .bss and .data sections:
bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2 text data bss dec hex filename 319006 56608 3592 379206 5c946 /lib64/libncurses.so.5 10483 788 112 11383 2c77 /lib64/libtermcap.so.2
this is going to impact very negatively on the RSS of several critical programs such as bash and python.
The really big writable section (at least in FC6 ncurses; I know many libraries are much bigger, but bash and similar programs are really performance critical, slowing down all configure scripts, libtool etc. by say 50% is not acceptable I guess) is .data.rel.ro:
[15] .ctors PROGBITS 000000346904e088 04e088 000010 00 WA 0 0 8 [16] .dtors PROGBITS 000000346904e098 04e098 000010 00 WA 0 0 8 [17] .jcr PROGBITS 000000346904e0a8 04e0a8 000008 00 WA 0 0 8 [18] .data.rel.ro PROGBITS 000000346904e0c0 04e0c0 00c4e8 00 WA 0 0 32 [19] .dynamic DYNAMIC 000000346905a5a8 05a5a8 0001a0 10 WA 3 0 8 [20] .got PROGBITS 000000346905a748 05a748 0001f0 08 WA 0 0 8 [21] .got.plt PROGBITS 000000346905a938 05a938 000940 08 WA 0 0 8 [22] .data PROGBITS 000000346905b280 05b280 000b40 00 WA 0 0 32 [23] .bss NOBITS 000000346905bdc0 05bdc0 000e08 00 WA 0 0 32
There is a bunch of quite big objects:
121: 0000003469050fc0 7960 OBJECT GLOBAL DEFAULT 18 _nc_cap_hash_table 129: 0000003468e244b0 18 FUNC GLOBAL DEFAULT 10 bkgd 176: 000000346904e0e0 360 OBJECT GLOBAL DEFAULT 18 boolcodes 189: 0000003468e2aa50 18 FUNC GLOBAL DEFAULT 10 scrollok 196: 0000003468e24490 18 FUNC GLOBAL DEFAULT 10 bkgdset 244: 0000003468e21c50 18 FUNC GLOBAL DEFAULT 10 clearok 249: 0000003469058160 3320 OBJECT GLOBAL DEFAULT 18 strnames 261: 0000003469057d60 360 OBJECT GLOBAL DEFAULT 18 boolfnames 262: 000000346904e260 320 OBJECT GLOBAL DEFAULT 18 numcodes 294: 0000003469057220 2480 OBJECT GLOBAL DEFAULT 18 _nc_key_names 311: 000000346904f0a0 7960 OBJECT GLOBAL DEFAULT 18 _nc_info_hash_table 341: 0000003469052ee0 1080 OBJECT GLOBAL DEFAULT 18 _nc_capalias_table 344: 0000003468e243a0 18 FUNC GLOBAL DEFAULT 10 echochar 345: 0000003469058020 320 OBJECT GLOBAL DEFAULT 18 numfnames 378: 0000003468e24370 18 FUNC GLOBAL DEFAULT 10 addch 403: 0000003469058e60 3320 OBJECT GLOBAL DEFAULT 18 strfnames 422: 0000003469057be0 360 OBJECT GLOBAL DEFAULT 18 boolnames 440: 0000003468e25af0 18 FUNC GLOBAL DEFAULT 10 leaveok 467: 0000003468e23f50 18 FUNC GLOBAL DEFAULT 10 insch 488: 000000346904e3a0 3320 OBJECT GLOBAL DEFAULT 18 strcodes 515: 0000003468e3ad40 18 FUNC GLOBAL DEFAULT 10 notimeout 525: 0000003469057ee0 320 OBJECT GLOBAL DEFAULT 18 numnames 547: 0000003469053320 168 OBJECT GLOBAL DEFAULT 18 _nc_infoalias_table 551: 0000003468e2d5b0 18 FUNC GLOBAL DEFAULT 10 syncok
and 3850 relative relocations.
Several of these are used only by code which parses the terminfo sources, does that really have to be in a library that is used by all programs? Is there anything but tic (or a couple of other ncurses utilities) which parses the terminfo sources?
I think if it makes sense to remove termcap and replace it only with ncurses (I'm not convinced), then the first step needs to be change ncurses so that it is not a kitchen sink library.
Jakub
On Fri, 2006-12-01 at 04:45 -0500, Jakub Jelinek wrote:
On Fri, Dec 01, 2006 at 10:13:55AM +0100, Bernardo Innocenti wrote:
Are we still planning to completely replace termcap with ncurses in Fedora?
While I welcome reducing the number of dupe packages, I'm worried by the impact of this change on the size and performance of many core programs.
libncurses is much bigger than libtermcap, and also has oddly large .bss and .data sections:
bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2 text data bss dec hex filename 319006 56608 3592 379206 5c946 /lib64/libncurses.so.5 10483 788 112 11383 2c77 /lib64/libtermcap.so.2
this is going to impact very negatively on the RSS of several critical programs such as bash and python.
The really big writable section
how much of that would be missing const ?
On Fri, Dec 01, 2006 at 11:00:54AM +0100, Arjan van de Ven wrote:
On Fri, 2006-12-01 at 04:45 -0500, Jakub Jelinek wrote:
On Fri, Dec 01, 2006 at 10:13:55AM +0100, Bernardo Innocenti wrote:
Are we still planning to completely replace termcap with ncurses in Fedora?
While I welcome reducing the number of dupe packages, I'm worried by the impact of this change on the size and performance of many core programs.
libncurses is much bigger than libtermcap, and also has oddly large .bss and .data sections:
bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2 text data bss dec hex filename 319006 56608 3592 379206 5c946 /lib64/libncurses.so.5 10483 788 112 11383 2c77 /lib64/libtermcap.so.2
this is going to impact very negatively on the RSS of several critical programs such as bash and python.
The really big writable section
how much of that would be missing const ?
The 2 biggest objects (8KB each) are const, pregenerated array of: const struct name_table_entry * const _nc_{info,cap}_hash_table[] = { 0, 0, 0, 0, 0, 0, _nc_info_table + 465, 0, 0, 0, 0, 0, _nc_info_table + 261, ... };
I think the easiest would be just nuke the terminfo source parsing routines from libncurses*.so, _nc_read_entry_source nor _nc_parse_entry aren't even prototyped in any installed ncurses headers. It can be IMHO moved to libtic.a which will be linked into ncurses utilities that need it.
If that is not possible, there are other options, e.g. switching to a more compact and relocation friendly representation. In those 2 big (sparse) tables all entries are either 0 or _nc_{info,cap}_table + small const offset >= 0 < 512. Representing it as array of short int where -1 would mean the current 0 and 0 .. 511 would mean _nc_info_table + that_value would be trivial, save 2x6KB of memory and save hundreds of relocations. Guess other large .data.rel.ro objects could be treated similarly, again with a < 10 line change in code using the tables and some changes in the generator.
Jakub
On Fri, Dec 01, 2006 at 05:32:12AM -0500, Jakub Jelinek wrote:
libncurses is much bigger than libtermcap, and also has oddly large .bss and .data sections:
bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2 text data bss dec hex filename 319006 56608 3592 379206 5c946 /lib64/libncurses.so.5 10483 788 112 11383 2c77 /lib64/libtermcap.so.2
[...]
I think the easiest would be just nuke the terminfo source parsing routines from libncurses*.so, _nc_read_entry_source nor _nc_parse_entry aren't even prototyped in any installed ncurses headers. It can be IMHO moved to libtic.a which will be linked into ncurses utilities that need it.
If that is not possible, there are other options, e.g. switching to a more compact and relocation friendly representation.
[...]
Thanks for the suggestions. ncurses-5.5-27.20061209.fc7 should be much better.
$ size /lib64/libncurses.so.5 text data bss dec hex filename 261273 20504 3144 284921 458f9 /lib64/libncurses.so.5
Linker statistics show that total startup time in dynamic loader for bash with libncurses is about 10% longer than with libtermcap.
Is it good enough to replace libtermcap?
On Fri, Dec 01, 2006 at 10:13:55AM +0100, Bernardo Innocenti wrote:
While I welcome reducing the number of dupe packages, I'm worried by the impact of this change on the size and performance of many core programs.
It shouldn't be that bad.
libncurses is much bigger than libtermcap, and also has oddly large .bss and .data sections:
bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2 text data bss dec hex filename 319006 56608 3592 379206 5c946 /lib64/libncurses.so.5 10483 788 112 11383 2c77 /lib64/libtermcap.so.2
this is going to impact very negatively on the RSS of several critical programs such as bash and python.
$ ps -o rss,comm RSS COMMAND 1428 bash_termcap 1480 bash_curses
Not sure it proves anything.
I'm also worried that the overall time required spent for a fork may increase considerably.
Note that termcap has everything in one big file. A simple program doing just tgetent() and tgetstr() is significantly faster with ncurses.
$ time ( a=0;while [ $a -lt 1000 ]; do a=$[$a + 1] && ./testtermcap < /dev/null &> /dev/null; done )
real 0m2.078s user 0m1.137s sys 0m0.936s
$ time ( a=0;while [ $a -lt 1000 ]; do a=$[$a + 1] && ./testncurses < /dev/null &> /dev/null; done )
real 0m1.464s user 0m0.496s sys 0m0.962s
Miroslav Lichvar wrote:
libncurses is much bigger than libtermcap, and also has oddly large .bss and .data sections:
bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2 text data bss dec hex filename 319006 56608 3592 379206 5c946 /lib64/libncurses.so.5 10483 788 112 11383 2c77 /lib64/libtermcap.so.2
this is going to impact very negatively on the RSS of several critical programs such as bash and python.
$ ps -o rss,comm RSS COMMAND 1428 bash_termcap 1480 bash_curses
Not sure it proves anything.
We see almost exactly the +50KB size difference reported by "size".
The fact that bash also carries an additional 1400KB of bloat does not make the ncurses problem more appealing : - )
Note that termcap has everything in one big file. A simple program doing just tgetent() and tgetstr() is significantly faster with ncurses.
I too dislike big binary files, but I would at least hope it's being loaded with mmap() and/or has at least an index to seek to the correct terminal as an O(1) operation.
$ time ( a=0;while [ $a -lt 1000 ]; do a=$[$a + 1] && ./testtermcap < /dev/null &> /dev/null; done )
real 0m2.078s user 0m1.137s sys 0m0.936s
$ time ( a=0;while [ $a -lt 1000 ]; do a=$[$a + 1] && ./testncurses < /dev/null &> /dev/null; done )
real 0m1.464s user 0m0.496s sys 0m0.962s
Hmmm! So termcap also has algoritmic disadvantages that outweight its smaller size relative to ncurses.
This makes me prefer ncurses over termcap, but fixing its abnormal data+bss size is still very desiderable.
Once upon a time, Bernardo Innocenti bernie@develer.com said:
Hmmm! So termcap also has algoritmic disadvantages that outweight its smaller size relative to ncurses.
Remember though that /bin/sh (aka bash) has many uses that are non-interactive and don't ever look up a termcap/terminfo entry (probably many more such uses than interactive). The lookup slowdown probably does not outweigh any dynamic linking slowdown or memory overhead.
Bernardo Innocenti wrote:
bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2 text data bss dec hex filename 319006 56608 3592 379206 5c946 /lib64/libncurses.so.5 10483 788 112 11383 2c77 /lib64/libtermcap.so.2
It's somewhat better now:
bender:/# size /lib64/libncurses.so.5 /lib64/libtinfo.so.5 /lib64/libtermcap.so.2 text data bss dec hex filename 122768 2296 520 125584 1ea90 /lib64/libncurses.so.5 110682 14048 2064 126794 1ef4a /lib64/libtinfo.so.5 10483 788 112 11383 2c77 /lib64/libtermcap.so.2
And now bash links against libtinfo only.
-
Arjan van de Ven -
Bernardo Innocenti -
Chris Adams -
Jakub Jelinek -
Miroslav Lichvar -
Ralf Ertzinger