Okay its come to my attention that the readahead configs have a difficult time being kept in sync as package updates roll out. For fc6 right now for example readahead is out of sync with firefox libraries.
Can we update readahead's implementation so we can get per package control of the default configs? Is a readahead.d/ structure appropriate here?
I want to point out that we lose significant value with read-ahead if we can't ensure that the listings will be in-sync as package updates are issued. And re-spinning readahead updates periodically to attempt to sync a centralized config file is horrible inefficient and may never be perfectly in-sync considering the churn of updates during a release cycle.
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=203329
-jef
On Sun, 2006-11-12 at 13:57 -0900, Jeff Spaleta wrote:
Can we update readahead's implementation so we can get per package control of the default configs? Is a readahead.d/ structure appropriate here?
I think this would be the Right Way(tm) to do it. Without native support for it, we could still at least cat all of those files into the proper readahead listing before running it.
On 11/12/06, Jeff Spaleta jspaleta@gmail.com wrote:
Okay its come to my attention that the readahead configs have a difficult time being kept in sync as package updates roll out. For fc6 right now for example readahead is out of sync with firefox libraries.
Can we update readahead's implementation so we can get per package control of the default configs? Is a readahead.d/ structure appropriate here?
Sorry... i didnt look hard enough... the readahead.d structure was added for fc6.... the problem is package maintainers aren't using it yet. I guess what I need to do is parse the existing config file... identify individual packages that could drop files in readahead.d/ and poke the appropriate maintainers in the eye about dropping a file in there as part of package payloading.
Anyone want to help me construct the list of packages that could make sure of readahead.d/ on a per package basis based on the default readahead configs we have right now?
-jef"First up... the firefox maintainer... where's my eye poking stick"spaleta
On Sun, Nov 12, 2006 at 02:15:57PM -0900, Jeff Spaleta wrote:
On 11/12/06, Jeff Spaleta jspaleta@gmail.com wrote:
Okay its come to my attention that the readahead configs have a difficult time being kept in sync as package updates roll out. For fc6 right now for example readahead is out of sync with firefox libraries.
Can we update readahead's implementation so we can get per package control of the default configs? Is a readahead.d/ structure appropriate here?
Sorry... i didnt look hard enough... the readahead.d structure was added for fc6.... the problem is package maintainers aren't using it yet. I guess what I need to do is parse the existing config file... identify individual packages that could drop files in readahead.d/ and poke the appropriate maintainers in the eye about dropping a file in there as part of package payloading.
Anyone want to help me construct the list of packages that could make sure of readahead.d/ on a per package basis based on the default readahead configs we have right now?
I don't have machine with full FC6 installation, but you can create the list very easily by shell command:
for f in `cat /etc/readahead.d/default.*`; do rpm -q --queryformat "%{NAME}\n" -f $f 2> /dev/null; done | sort -u
-jef"First up... the firefox maintainer... where's my eye poking stick"spaleta
The problem is not firefox only. It's pretty generic problem, because we have versions filenames for libraries:
$ grep '/usr/lib/lib.*[0-9].*' /etc/readahead.d/default.* | wc -l 136
It's means almost all things in /etc/readahead.d/default.* could be obsolete after an update (except stable things like /bin/*).
Karel
On 11/13/06, Karel Zak kzak@redhat.com wrote:
It's means almost all things in /etc/readahead.d/default.* could be obsolete after an update (except stable things like /bin/*).
I agree... which is why i think its important to have this discussion in the list.. instead of just in the bugzilla report. The rpm query is a neat trick. If you want to pump up a few throwaway readahead packages to test new ideas to solve this issue, I'll be happy to offer my system up for inadvertent destruction testing them.
-jef
On Sun, Nov 12, 2006 at 02:15:57PM -0900, Jeff Spaleta wrote:
On 11/12/06, Jeff Spaleta jspaleta@gmail.com wrote:
Okay its come to my attention that the readahead configs have a difficult time being kept in sync as package updates roll out. For fc6 right now for example readahead is out of sync with firefox libraries.
Can we update readahead's implementation so we can get per package control of the default configs? Is a readahead.d/ structure appropriate here?
Sorry... i didnt look hard enough... the readahead.d structure was added for fc6.... the problem is package maintainers aren't using it yet. I guess what I need to do is parse the existing config file... identify individual packages that could drop files in readahead.d/ and poke the appropriate maintainers in the eye about dropping a file in there as part of package payloading.
Anyone want to help me construct the list of packages that could make sure of readahead.d/ on a per package basis based on the default readahead configs we have right now?
Update: I've wrote a small and simple readahead-auditd that is able to collect all filenames from boot process. It means everyone will be able to generate unique list for his Fedora. And also I can maintain default lists more effective now. I'm going to release an experimental readahead package with this solution to FC7 next week.
Karel
At 10:41 AM +0100 2/14/07, Karel Zak wrote:
On Sun, Nov 12, 2006 at 02:15:57PM -0900, Jeff Spaleta wrote:
On 11/12/06, Jeff Spaleta jspaleta@gmail.com wrote:
Okay its come to my attention that the readahead configs have a difficult time being kept in sync as package updates roll out. For fc6 right now for example readahead is out of sync with firefox libraries.
Can we update readahead's implementation so we can get per package control of the default configs? Is a readahead.d/ structure appropriate here?
Sorry... i didnt look hard enough... the readahead.d structure was added for fc6.... the problem is package maintainers aren't using it yet. I guess what I need to do is parse the existing config file... identify individual packages that could drop files in readahead.d/ and poke the appropriate maintainers in the eye about dropping a file in there as part of package payloading.
Anyone want to help me construct the list of packages that could make sure of readahead.d/ on a per package basis based on the default readahead configs we have right now?
Update: I've wrote a small and simple readahead-auditd that is able to collect all filenames from boot process. It means everyone will be able to generate unique list for his Fedora. And also I can maintain default lists more effective now. I'm going to release an experimental readahead package with this solution to FC7 next week.
Would this work on FC6 as well? If the package can build cleanly on FC6 I'd try it out.
Does the daemon keep running after booting finishes? Will it have anything more to do then?
On Wed, Feb 14, 2007 at 09:09:22AM -0500, Tony Nelson wrote:
At 10:41 AM +0100 2/14/07, Karel Zak wrote:
On Sun, Nov 12, 2006 at 02:15:57PM -0900, Jeff Spaleta wrote:
On 11/12/06, Jeff Spaleta jspaleta@gmail.com wrote:
Okay its come to my attention that the readahead configs have a difficult time being kept in sync as package updates roll out. For fc6 right now for example readahead is out of sync with firefox libraries.
Can we update readahead's implementation so we can get per package control of the default configs? Is a readahead.d/ structure appropriate here?
Sorry... i didnt look hard enough... the readahead.d structure was added for fc6.... the problem is package maintainers aren't using it yet. I guess what I need to do is parse the existing config file... identify individual packages that could drop files in readahead.d/ and poke the appropriate maintainers in the eye about dropping a file in there as part of package payloading.
Anyone want to help me construct the list of packages that could make sure of readahead.d/ on a per package basis based on the default readahead configs we have right now?
Update: I've wrote a small and simple readahead-auditd that is able to collect all filenames from boot process. It means everyone will be able to generate unique list for his Fedora. And also I can maintain default lists more effective now. I'm going to release an experimental readahead package with this solution to FC7 next week.
Would this work on FC6 as well? If the package can build cleanly on FC6 I'd try it out.
No problem. I use FC6 too.
Does the daemon keep running after booting finishes? Will it have anything more to do then?
The default behaviour (depends on setting in a config file) will be automated finish few minutes after boot. It should be a good way how collect things that depend on user's habits (for example I start gnome-terminal, firefox and mutt after computer startup).
I assume two steps:
1) collect filenames from boot process 2) post-processing -- things like sort by first block position
All this things should be on demand. Later we can thing about more transparent and automated solution (or about completely different solution like fcache ;-).
Karel
At 6:11 PM +0100 2/14/07, Karel Zak wrote:
On Wed, Feb 14, 2007 at 09:09:22AM -0500, Tony Nelson wrote:
At 10:41 AM +0100 2/14/07, Karel Zak wrote:
...
Update: I've wrote a small and simple readahead-auditd that is able to collect all filenames from boot process. It means everyone will be able to generate unique list for his Fedora. And also I can maintain default lists more effective now. I'm going to release an experimental readahead package with this solution to FC7 next week.
Would this work on FC6 as well? If the package can build cleanly on FC6 I'd try it out.
No problem. I use FC6 too.
Good, thanks.
Does the daemon keep running after booting finishes? Will it have anything more to do then?
The default behaviour (depends on setting in a config file) will be automated finish few minutes after boot. It should be a good way how collect things that depend on user's habits (for example I start gnome-terminal, firefox and mutt after computer startup).
I assume two steps:
- collect filenames from boot process
- post-processing -- things like sort by first block position
All this things should be on demand. Later we can thing about more transparent and automated solution (or about completely different solution like fcache ;-).
I think you're saying that the daemon will terminate after collecting data during boot, but the readahead.whenever files will only be regenerated when a separate command is run. Thus, after installing, say, a new Firefox, the readahead.d/default.later file will have stale entries in it until the user notices and runs that command. I expect it will become clear when I examine the source.
On Wed, Feb 14, 2007 at 01:43:51PM -0500, Tony Nelson wrote:
I think you're saying that the daemon will terminate after collecting data during boot, but the readahead.whenever files will only be regenerated when
I'm not sure if we need to run the daemon for every boot. I don't know how expensive is our audit system... we will see.
a separate command is run. Thus, after installing, say, a new Firefox, the readahead.d/default.later file will have stale entries in it until the user
Prepare a new version of the readahead.d/default.later file is very simple (one qsort) and I think it will be possible to do it automatically. So, without a separate command.
More attractive is to generate final version of the lists where filenames are sorted by first data block -- see "fast mode" in https://hosted.fedoraproject.org/projects/readahead/browser/README.
Now in FC6 readahead supports the "full mode" only. The readahead in FC7 will support both modes, but regenerate lists for "fast mode" is more expensive -- it seems like a good job for cron or some manually started command.
notices and runs that command. I expect it will become clear when I examine the source.
Yeah, that's always more clear than my mails. Sorry ;-)
Karel
On 2/14/07, Karel Zak kzak@redhat.com wrote:
Update: I've wrote a small and simple readahead-auditd that is able to collect all filenames from boot process. It means everyone will be able to generate unique list for his Fedora. And also I can maintain default lists more effective now. I'm going to release an experimental readahead package with this solution to FC7 next week.
Are you interested in collecting those 'unique' lists for some unscientific comparison of what the common boot time items are out in the wild?
-jef
On Sun, 2006-11-12 at 13:57 -0900, Jeff Spaleta wrote:
Okay its come to my attention that the readahead configs have a difficult time being kept in sync as package updates roll out. For fc6 right now for example readahead is out of sync with firefox libraries.
Can we update readahead's implementation so we can get per package control of the default configs? Is a readahead.d/ structure appropriate here?
maybe. the downside is that you now add extra costs (seeks :) which.. well readahead was trying to avoid. So if this is going to get used massively it's less certain things will gain as much as before...
On 11/12/06, Arjan van de Ven arjan@fenrus.demon.nl wrote:
the downside is that you now add extra costs (seeks :) which.. well readahead was trying to avoid. So if this is going to get used massively it's less certain things will gain as much as before...
I'm not saying use it for everything.. im just saying lets make sure that the default items we have now in the readahead config actually stay in sync with the libraries on disk so users can derive consistent benefit from it over time. My current favorite example is firefox. The libraries are listed in the default readahead config..but now that firefox has seen an update, the listings are no longer in-sync with the files on disk.. because the library locations are in a versioned directory tree...so firefox won't get a readahead boost on an updated system.
I understand the seek time issue associated with doing readahead willy-nilly, so if you can think of a way to centralize a default config and prevent out-of-sync conditions like we have with firefox at the moment I'm all ears. Please go ahead and take a lot at the listing in default.later and see if anything else besides firefox is currently out of sync on an updated fc6 install.
-jef
Hi.
On Mon, 13 Nov 2006 09:06:20 +0100, Arjan van de Ven wrote:
the downside is that you now add extra costs (seeks :) which.. well readahead was trying to avoid. So if this is going to get used massively it's less certain things will gain as much as before...
Maybe run a script on system shutdown which reads all the files under /etc/readahead.d and creates a single config file for readahead to read on the next boot?
On Mon, 2006-11-13 at 10:48 +0100, Ralf Ertzinger wrote:
Hi.
On Mon, 13 Nov 2006 09:06:20 +0100, Arjan van de Ven wrote:
the downside is that you now add extra costs (seeks :) which.. well readahead was trying to avoid. So if this is going to get used massively it's less certain things will gain as much as before...
Maybe run a script on system shutdown which reads all the files under /etc/readahead.d and creates a single config file for readahead to read on the next boot?
I really like this idea; it's a simple "cat" and it can be done at a time where latency doesn't matter... (even in cron.daily)
Oh... this opens more options. This also allows the "sort by blocknumber" to be done at this point and taken out of the critical latency part......
great idea!
On Mon, Nov 13, 2006 at 11:07:41AM +0100, Arjan van de Ven wrote:
On Mon, 2006-11-13 at 10:48 +0100, Ralf Ertzinger wrote:
Hi.
On Mon, 13 Nov 2006 09:06:20 +0100, Arjan van de Ven wrote:
the downside is that you now add extra costs (seeks :) which.. well readahead was trying to avoid. So if this is going to get used massively it's less certain things will gain as much as before...
Maybe run a script on system shutdown which reads all the files under /etc/readahead.d and creates a single config file for readahead to read on the next boot?
I really like this idea; it's a simple "cat" and it can be done at a time where latency doesn't matter... (even in cron.daily)
Oh... this opens more options. This also allows the "sort by blocknumber" to be done at this point and taken out of the critical latency part......
Good idea. Added to "TODO" list:
http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=156442
Karel
Arjan van de Ven (arjan@fenrus.demon.nl) said:
I really like this idea; it's a simple "cat" and it can be done at a time where latency doesn't matter... (even in cron.daily)
Oh... this opens more options. This also allows the "sort by blocknumber" to be done at this point and taken out of the critical latency part......
great idea!
Of course, that then makes shutdown take twice as long.
What do you do if you have a box where readahead makes boot slower?
Bill
On Mon, Nov 13, 2006 at 10:23:24PM -0500, Bill Nottingham wrote:
What do you do if you have a box where readahead makes boot slower?
rpm -e readahead?
:)
Matthew Miller (mattdm@mattdm.org) said:
On Mon, Nov 13, 2006 at 10:23:24PM -0500, Bill Nottingham wrote:
What do you do if you have a box where readahead makes boot slower?
rpm -e readahead?
You can, but it does seem odd to install something tha makes the system less efficient. I should do more testing on laptops; I know at least on some it slows things down.
Bill
On Mon, Nov 13, 2006 at 10:33:14PM -0500, Bill Nottingham wrote:
What do you do if you have a box where readahead makes boot slower?
rpm -e readahead?
You can, but it does seem odd to install something tha makes the system less efficient. I should do more testing on laptops; I know at least on some it slows things down.
Okay, I admit I actually take the more drastic solution of removing it from comps.xml. I don't remember the numbers, but it was definitely making my FC4 laptop boot more slowly. I haven't given it another chance.... And on non-laptop systems that don't reboot often, it's basically pointless anyway.
Of course, that then makes shutdown take twice as long.
In most cases (no changes to the files since last shutdown), then a 'make' can shorten the latency to just a 'stat()' of each file. The whole computation can run in parallel with all other activities during shutdown, until just before the unmounts.
What do you do if you have a box where readahead makes boot slower?
Delete enough items from the list of filenames to be processed.
On Mon, 2006-11-13 at 22:23 -0500, Bill Nottingham wrote:
Arjan van de Ven (arjan@fenrus.demon.nl) said:
I really like this idea; it's a simple "cat" and it can be done at a time where latency doesn't matter... (even in cron.daily)
Oh... this opens more options. This also allows the "sort by blocknumber" to be done at this point and taken out of the critical latency part......
great idea!
Of course, that then makes shutdown take twice as long.
if you do it at shutdown, which is again a latency path :) cron.daily/weekly is less so ;)
On 11/14/2006 07:15 AM, Arjan van de Ven wrote:
On Mon, 2006-11-13 at 22:23 -0500, Bill Nottingham wrote:
Arjan van de Ven (arjan@fenrus.demon.nl) said:
I really like this idea; it's a simple "cat" and it can be done at a time where latency doesn't matter... (even in cron.daily)
Oh... this opens more options. This also allows the "sort by blocknumber" to be done at this point and taken out of the critical latency part......
great idea!
Of course, that then makes shutdown take twice as long.
if you do it at shutdown, which is again a latency path :) cron.daily/weekly is less so ;)
+1 for cron -1 for shutdown -> what about boxens, which are shutdown only on kernel updates? (like mine)
Regards, Dariusz
Send instant messages to your online friends http://uk.messenger.yahoo.com
On Tue, 2006-11-14 at 08:15 +0100, Arjan van de Ven wrote:
On Mon, 2006-11-13 at 22:23 -0500, Bill Nottingham wrote:
Arjan van de Ven (arjan@fenrus.demon.nl) said:
I really like this idea; it's a simple "cat" and it can be done at a time where latency doesn't matter... (even in cron.daily)
Oh... this opens more options. This also allows the "sort by blocknumber" to be done at this point and taken out of the critical latency part......
great idea!
Of course, that then makes shutdown take twice as long.
if you do it at shutdown, which is again a latency path :) cron.daily/weekly is less so ;)
Or post-{install,remove} of the packages that put files in readahead.d.
There's no need to run the script if nothing has changed, and if you upgrade affected packages and reboot before the next day, the libs and config will be out of sync.
Mark Rosenstand napsal(a):
On Tue, 2006-11-14 at 08:15 +0100, Arjan van de Ven wrote:
On Mon, 2006-11-13 at 22:23 -0500, Bill Nottingham wrote:
Arjan van de Ven (arjan@fenrus.demon.nl) said:
I really like this idea; it's a simple "cat" and it can be done at a time where latency doesn't matter... (even in cron.daily)
Oh... this opens more options. This also allows the "sort by blocknumber" to be done at this point and taken out of the critical latency part......
great idea!
Of course, that then makes shutdown take twice as long.
if you do it at shutdown, which is again a latency path :) cron.daily/weekly is less so ;)
Or post-{install,remove} of the packages that put files in readahead.d.
There's no need to run the script if nothing has changed,
Does prelink overwrite the modified files in place, or does it create new versions as separate files and rename()s them? Mirek
On Tue, Nov 14, 2006 at 08:15:26AM +0100, Arjan van de Ven wrote:
On Mon, 2006-11-13 at 22:23 -0500, Bill Nottingham wrote:
Arjan van de Ven (arjan@fenrus.demon.nl) said:
I really like this idea; it's a simple "cat" and it can be done at a time where latency doesn't matter... (even in cron.daily)
Oh... this opens more options. This also allows the "sort by blocknumber" to be done at this point and taken out of the critical latency part......
great idea!
Of course, that then makes shutdown take twice as long.
Definitely no. We don't talk about full readahead.
I've implemented the idea and it seems that we can use it during regular shutdown.
It's very fast, because almost all files from readahead lists are already in the cache (and don't forget we don't call readahead(2) and read full files. We need open(), stat() and read first block for each file only. This is 10x faster than full readahead which we know from system start up).
if you do it at shutdown, which is again a latency path :) cron.daily/weekly is less so ;)
Well, some numbers:
./readahead --timing --verbose \ --output /etc/readahead.d/early.sorted \ --sort /etc/readahead.d/default.early
List: /etc/readahead.d/default.early Time (read lists): 0 ms Create list with 850 files Time (read blocks): 103 ms <--------- Read blocks for 726/850 files (65778 KB)
List: /etc/readahead.d/default.later Time (read lists): 2 ms Create list with 3757 files Time (read blocks): 281 ms <---------- Read blocks for 3056/3757 files (110159 KB)
The worst numbers which I've ever seen was 2500ms (sum from both lists) and it was on an laptop with a lazy HDD.
Also, the idea that we can run it as parallel process is useful.
If you want to play with new readahead code, see:
http://people.redhat.com/kzak/readahead/readahead-11276006.tar.gz
Karel
I've created a new project page for readahead (with jkeating's help:-). It's fine if you edit the wiki page and contribute there.
http://hosted.fedoraproject.org/projects/readahead/
Note: GIT repo will be available next week.
Karel
On Sun, Nov 12, 2006 at 01:57:57PM -0900, Jeff Spaleta wrote:
Okay its come to my attention that the readahead configs have a difficult time being kept in sync as package updates roll out. For fc6 right now for example readahead is out of sync with firefox libraries.
Can we update readahead's implementation so we can get per package control of the default configs? Is a readahead.d/ structure appropriate here?
I want to point out that we lose significant value with read-ahead if we can't ensure that the listings will be in-sync as package updates are issued. And re-spinning readahead updates periodically to attempt to sync a centralized config file is horrible inefficient and may never be perfectly in-sync considering the churn of updates during a release cycle.
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=203329
-jef
-- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
-
Arjan van de Ven -
Bill Nottingham -
Dariusz J. Garbowski -
Jef Spaleta -
Jeffrey Ollie -
John Reiser -
Karel Zak -
Mark Rosenstand -
Matthew Miller -
Miloslav Trmač -
Peter Gordon -
Ralf Ertzinger -
Tony Nelson