Chris,
The inode is NOT the same one. I tried a sqlite dump | restore operation but it didn't work...
With your journalctl -f and trying dnf history, dnf reported I/O errors but no kernel or /var/log/messages appeared to contain I/O error messages.
fc35-bash 5.1 /var/lib/dnf# sqlite3 history.sqlite 'VACUUM;' Error: file is encrypted or is not a database
The drive is a hybrid Seagate drive:
fc35-bash 5.1 /var/lib/dnf# smartctl -i /dev/sda smartctl 7.3 2021-03-02 r5210 [x86_64-linux-5.12.0-0.rc1.20210305git280d542f6ffa.164.fc35.x86_64] (local build) Copyright (C) 2002-21, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION === Model Family: Seagate FireCuda 2.5 Device Model: ST2000LX001-1RG174 Serial Number: ZDZ4TK3H LU WWN Device Id: 5 000c50 0b38c6722 Firmware Version: SDM1 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 2.5 inches Device is: In smartctl database 7.3/5170 ATA Version is: ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Mar 14 15:02:42 2021 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled
I guess it's time to rebuild from scratch?
I wonder why dnf reported I/O errors?
Regards,
George...
----------------------------------------------------------------------
Date: Sat, 13 Mar 2021 21:07:30 -0700 From: Chris Murphy lists@colorremedies.com Subject: Re: Is there a procedure to rebuild /var/lib/dnf/history.sqlite? (Chris Murphy) To: For testing and quality assurance of Fedora releases test@lists.fedoraproject.org Message-ID: CAJCQCtSqe01Ys6st9och6mTiZaLgEyq0mWTymSCCaiCECPnRgQ@mail.gmail.com Content-Type: text/plain; charset="UTF-8"
On Sat, Mar 13, 2021 at 8:03 PM George R Goffe via test test@lists.fedoraproject.org wrote:
Chris,
I see NO I/O messages.
You previously reported them.
History database is not writable: SQLite error on "/var/lib/dnf/history.sqlite": Executing an SQL statement failed: disk I/O error History database is not writable: SQLite error on "/var/lib/dnf/history.sqlite": Executing an SQL statement failed: disk I/O error
Odd messages from journalctl though:
-- Journal begins at Wed 2021-02-24 19:05:12 PST. -- Mar 13 17:32:57 fc35 kernel: BTRFS warning (device sda5): csum failed root 5 ino 2239235 off 1218498560 csum 0x694ef7e2 expected csum 0x204f8b8d mirror 1 Mar 13 17:32:57 fc35 kernel: BTRFS error (device sda5): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 82, gen 0 Mar 13 17:32:57 fc35 kernel: BTRFS warning (device sda5): csum failed root 5 ino 2239235 off 1218506752 csum 0x302a9a33 expected csum 0x204f8b8d mirror 1 Mar 13 17:32:57 fc35 kernel: BTRFS error (device sda5): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 83, gen 0 Mar 13 17:32:57 fc35 kernel: BTRFS warning (device sda5): csum failed root 5 ino 2239235 off 1218445312 csum 0x9ee6da83 expected csum 0x204f8b8d mirror 1 Mar 13 17:32:57 fc35 kernel: BTRFS error (device sda5): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 84, gen 0 Mar 13 17:32:57 fc35 kernel: BTRFS warning (device sda5): csum failed root 5 ino 2239235 off 1218445312 csum 0x9ee6da83 expected csum 0x204f8b8d mirror 1 Mar 13 17:32:57 fc35 kernel: BTRFS error (device sda5): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 85, gen 0 Mar 13 17:33:47 fc35 kernel: BTRFS info (device sda5): scrub: started on devid 1 Mar 13 17:34:39 fc35 kernel: BTRFS info (device sda5): scrub: finished on devid 1 with status: 0
That is strange. The scrub should find the same corruption. If you do:
ls -li /var/lib/dnf/history.sqlite
Is the reported inode number on the left 2239235?
What kind of drive is /dev/sda? HDD or SSD?
On Sun, Mar 14, 2021 at 4:05 PM George R Goffe via test test@lists.fedoraproject.org wrote:
Chris,
The inode is NOT the same one. I tried a sqlite dump | restore operation but it didn't work...
With your journalctl -f and trying dnf history, dnf reported I/O errors but no kernel or /var/log/messages appeared to contain I/O error messages.
fc35-bash 5.1 /var/lib/dnf# sqlite3 history.sqlite 'VACUUM;' Error: file is encrypted or is not a database
I don't have an explanation. I haven't ever seen a sqlite database go bad. And I know that file is a database and it's not encrypted, because the command works on the test VM's I've got. Pretty strange...
I guess it's time to rebuild from scratch?
I wonder why dnf reported I/O errors?
There's something about that history database file it doesn't like. I don't know what the instigator is, but often the problem happened earlier, but how much earlier is a bug hunt. Regression testing is tedious and time consuming. You can opt to just delete that history database, dnf should just start a new one. And then you'll have to see if it happens again.
Anyway, Btrfs reporting checksum errors is not normal. I mean, it's normal for it to complain anytime it detects even a single block that's corrupt. But routine corruptions are not normal, it suggests there's a hardware problem: memory, drive cable or connector or DRAM...
smartctl -x will report quite a lot more information about a drive, including internal read/write errors that don't always get reported to the kernel, as well as UDMA errors, which are common with connector+cable issues.
-- Chris Murphy