Chris,
I see NO I/O messages.
I used less to look at history.sqlite. The file looks like a log file around the time of a system crash. LOTS of unprintable characters. Some text at the end that looks like they might be from dnf.
@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ 91652 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ 91652 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<81>^M<82><F8>5^F^@^B <82>^W^T<82>warning: file 91652 /usr/src/kernels/5.12.0-0.rc1.162.fc35.x86_64/include/config/video/tveeprom.h: remove failed: No such file or dire 91652 ctory<81>^L<82><F8>4^F^@^B <82>^U^T<82>warning: file /usr/src/kernels/5.12.0-0.rc1.162.fc35.x86_64/include/co 91652 nfig/video/tvp5150.h: remove failed: No such file or directory<81>^K<82><F8>3^F^@^B <82>^S^T<82>warning: file 91652 /usr/src/kernels/5.12.0-0.rc1.162.fc35.x86_64/include/config/video/tw2804.h: remove failed: No such file or direct 91652 ory<81>^K<82><F8>2^F^@^B <82>^S^T<82>warning: file /usr/src/kernels/5.12.0-0.rc1.162.fc35.x86_64/include/co 91652 nfig/video/tw686x.h: remove failed: No such file or directory<81>^K<82><F8>1^F^@^B <82>^S^T<82>warning: file 91652 /usr/src/kernels/5.12.0-0.rc1.162.fc35.x86_64/include/config/video/tw9903.h: remove failed: No such file or direct 91652 ory<81>^K<82><F8>0^F^@^B <82>^S^T<82>warning: file /usr/src/kernels/5.12.0-0.rc1.162.fc35.x86_64/include/co 91652 nfig/video/tw9906.h: remove failed: No such file or directory<81>^L<82><F8>/^F^@^B <82>^U^T<82>warning: file 91652 /usr/src/kernels/5.12.0-0.rc1.162.fc35.x86_64/include/config/video/uda1342.h: remove failed: No such file or direc 91652 tory<81>^N<82><F8>.^F^@^B <82>^Y^T<82>warning: file /usr/src/kernels/5.12.0-0.rc1.162.fc35.x86_64/include/co 91652 nfig/video/upd64031a.h: remove failed: No such file or directory<81>^M<82><F8>-^F^@^B <82>^W^T<82>warning: file 91652 /usr/src/kernels/5.12.0-0.rc1.162.fc35.x86_64/include/config/video/upd64083.h: remove failed: No such file or dire 91652 ctory<81> 91653 <82><F8>,^F^@^B <82>^Q^T<82>warning: file /usr/src/kernels/5.12.0-0.rc1.162.fc35.x86_64/include/config/video/usbtv 91653 .h: remove failed: No such file or directory<81>^G<82><F8>+^F^@^B <82>^K^T<82>warning: file /usr/src/kernels 91653 /5.12.0-0.rc1.162.fc35.x86_64/include/config/video/v4l2: remove failed: No such file or directory
Scrub is clean.
Odd messages from journalctl though:
-- Journal begins at Wed 2021-02-24 19:05:12 PST. -- Mar 13 17:32:57 fc35 kernel: BTRFS warning (device sda5): csum failed root 5 ino 2239235 off 1218498560 csum 0x694ef7e2 expected csum 0x204f8b8d mirror 1 Mar 13 17:32:57 fc35 kernel: BTRFS error (device sda5): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 82, gen 0 Mar 13 17:32:57 fc35 kernel: BTRFS warning (device sda5): csum failed root 5 ino 2239235 off 1218506752 csum 0x302a9a33 expected csum 0x204f8b8d mirror 1 Mar 13 17:32:57 fc35 kernel: BTRFS error (device sda5): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 83, gen 0 Mar 13 17:32:57 fc35 kernel: BTRFS warning (device sda5): csum failed root 5 ino 2239235 off 1218445312 csum 0x9ee6da83 expected csum 0x204f8b8d mirror 1 Mar 13 17:32:57 fc35 kernel: BTRFS error (device sda5): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 84, gen 0 Mar 13 17:32:57 fc35 kernel: BTRFS warning (device sda5): csum failed root 5 ino 2239235 off 1218445312 csum 0x9ee6da83 expected csum 0x204f8b8d mirror 1 Mar 13 17:32:57 fc35 kernel: BTRFS error (device sda5): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 85, gen 0 Mar 13 17:33:47 fc35 kernel: BTRFS info (device sda5): scrub: started on devid 1 Mar 13 17:34:39 fc35 kernel: BTRFS info (device sda5): scrub: finished on devid 1 with status: 0
----------------------------------------------------------------------
Date: Sat, 13 Mar 2021 11:20:19 -0700 From: Chris Murphy lists@colorremedies.com Subject: Re: Is there a procedure to rebuild /var/lib/dnf/history.sqlite? To: For testing and quality assurance of Fedora releases test@lists.fedoraproject.org Message-ID: CAJCQCtSUEbUAnPyxNXvkN1CTgJrbvhh4SjziZXQpj7c1+FOJ+Q@mail.gmail.com Content-Type: text/plain; charset="UTF-8"
On Sat, Mar 13, 2021 at 10:13 AM George R Goffe via test test@lists.fedoraproject.org wrote:
Chris,
I goofed and sent this with the wrong subject. Sorry about that.
Thanks for responding.
I converted /var to btrfs about 2 weeks ago so this isn't necessarily related to that (I hope).
Here's what "btrfs filesystem usage /var" says about /var: (it is NOT mounted ro)
fc35-bash 5.1 ~# btrfs filesystem usage /var Overall: Device size: 64.00GiB Device allocated: 63.94GiB Device unallocated: 65.68MiB Device missing: 0.00B Used: 50.30GiB Free (estimated): 13.01GiB (min: 13.01GiB) Free (statfs, df): 13.01GiB Data ratio: 1.00 Metadata ratio: 1.00 Global reserve: 61.03MiB (used: 0.00B) Multiple profiles: no
Data,single: Size:63.15GiB, Used:50.20GiB (79.49%) /dev/sda5 63.15GiB
Metadata,single: Size:776.00MiB, Used:105.33MiB (13.57%) /dev/sda5 776.00MiB
System,single: Size:32.00MiB, Used:16.00KiB (0.05%) /dev/sda5 32.00MiB
Unallocated: /dev/sda5 65.68MiB
dmesg buffer, recent boot:
yeah what I'm looking for are the kernel messages at the time of the I/O error you're getting from dnf. Do this:
journalctl -fk
And in another terminal shell, run the same dnf history command that triggers the error, switch back to the first shell and see what messages have appeared.
You will note what looks like a kernel dump related to duplicate lock requests. This and other kernel oops like events happen on a regular basis on "my" systems. I got tired of reporting them and getting NO response to the bug reports. Sigh.
It's an older 5.11.0-rc6 debug kernel. You don't need to report the lockdep warnings. They come and go in kernel rc development. Pretty sure that particular lock dep warning was fixed a while ago. The current Fedora kernel is 5.11.5 with 5.11.6 heading to stable soon, so you should update.
On Sat, Mar 13, 2021 at 8:03 PM George R Goffe via test test@lists.fedoraproject.org wrote:
Chris,
I see NO I/O messages.
You previously reported them.
History database is not writable: SQLite error on "/var/lib/dnf/history.sqlite": Executing an SQL statement failed: disk I/O error History database is not writable: SQLite error on "/var/lib/dnf/history.sqlite": Executing an SQL statement failed: disk I/O error
Odd messages from journalctl though:
-- Journal begins at Wed 2021-02-24 19:05:12 PST. -- Mar 13 17:32:57 fc35 kernel: BTRFS warning (device sda5): csum failed root 5 ino 2239235 off 1218498560 csum 0x694ef7e2 expected csum 0x204f8b8d mirror 1 Mar 13 17:32:57 fc35 kernel: BTRFS error (device sda5): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 82, gen 0 Mar 13 17:32:57 fc35 kernel: BTRFS warning (device sda5): csum failed root 5 ino 2239235 off 1218506752 csum 0x302a9a33 expected csum 0x204f8b8d mirror 1 Mar 13 17:32:57 fc35 kernel: BTRFS error (device sda5): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 83, gen 0 Mar 13 17:32:57 fc35 kernel: BTRFS warning (device sda5): csum failed root 5 ino 2239235 off 1218445312 csum 0x9ee6da83 expected csum 0x204f8b8d mirror 1 Mar 13 17:32:57 fc35 kernel: BTRFS error (device sda5): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 84, gen 0 Mar 13 17:32:57 fc35 kernel: BTRFS warning (device sda5): csum failed root 5 ino 2239235 off 1218445312 csum 0x9ee6da83 expected csum 0x204f8b8d mirror 1 Mar 13 17:32:57 fc35 kernel: BTRFS error (device sda5): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 85, gen 0 Mar 13 17:33:47 fc35 kernel: BTRFS info (device sda5): scrub: started on devid 1 Mar 13 17:34:39 fc35 kernel: BTRFS info (device sda5): scrub: finished on devid 1 with status: 0
That is strange. The scrub should find the same corruption. If you do:
ls -li /var/lib/dnf/history.sqlite
Is the reported inode number on the left 2239235?
What kind of drive is /dev/sda? HDD or SSD?
First, this shouldn't be necessary, therefore I'm kinda suspicious. But it's inspecific suspicion. Maybe it is conversion.
I also don't know if this will fix the SQLite I/O errors, or make things worse, in which case you'd lose dnf history (probably not a big loss).
sudo -i cd /var/lib/dnf/ sqlite3 history.sqlite 'VACUUM;'
That cleans the database. It will delete the other two files which will reappear later once dnf starts writing to this database again.
-- Chris Murphy