We did some more troubleshooting of AMD Radeon issues on ppc64
As with Nouveau, it looks like a change from 64k to 4k page size got it working again with RX 5700. I suspect it will be similar for RX 6800 if we can get some of them, they are a good complement for the compute power.
The issue is page size, not ppc64
Vikings[1] in Germany are about to launch their European Talos II workstations so there will be more users and many will want to try Fedora as this is the place to get newest versions of stuff.
How would people feel about making 4k the default page size for workstation users?
Given that the default file system, Btrfs, is sensitive to page size and there are not so many GPUs outside of those vendors mentioned above, I feel this is a compelling issue.
Regards,
Daniel
On Wed, Jan 06, 2021 at 02:11:37PM +0100, Daniel Pocock wrote:
We did some more troubleshooting of AMD Radeon issues on ppc64
Small aside: I know you mean ppc64le (because thats all we make anymore), but it confuses me since 'ppc64' is what we called the old big endian 64 bit ppc, which is retired. Can I ask you to please use 'ppc64le' when talking about the current thing? :)
As with Nouveau, it looks like a change from 64k to 4k page size got it working again with RX 5700. I suspect it will be similar for RX 6800 if we can get some of them, they are a good complement for the compute power.
The issue is page size, not ppc64
Vikings[1] in Germany are about to launch their European Talos II workstations so there will be more users and many will want to try Fedora as this is the place to get newest versions of stuff.
How would people feel about making 4k the default page size for workstation users?
Given that the default file system, Btrfs, is sensitive to page size and there are not so many GPUs outside of those vendors mentioned above, I feel this is a compelling issue.
I don't have any great opinion on changing it, but hopefully a few power folks will chime in with thoughts. Would this only need to be changed in the kernel?
kevin
On Wed, Jan 6, 2021 at 9:21 PM Kevin Fenzi kevin@scrye.com wrote:
On Wed, Jan 06, 2021 at 02:11:37PM +0100, Daniel Pocock wrote:
We did some more troubleshooting of AMD Radeon issues on ppc64
Small aside: I know you mean ppc64le (because thats all we make anymore), but it confuses me since 'ppc64' is what we called the old big endian 64 bit ppc, which is retired. Can I ask you to please use 'ppc64le' when talking about the current thing? :)
As with Nouveau, it looks like a change from 64k to 4k page size got it working again with RX 5700. I suspect it will be similar for RX 6800 if we can get some of them, they are a good complement for the compute power.
The issue is page size, not ppc64
Vikings[1] in Germany are about to launch their European Talos II workstations so there will be more users and many will want to try Fedora as this is the place to get newest versions of stuff.
How would people feel about making 4k the default page size for workstation users?
Given that the default file system, Btrfs, is sensitive to page size and there are not so many GPUs outside of those vendors mentioned above, I feel this is a compelling issue.
I don't have any great opinion on changing it, but hopefully a few power folks will chime in with thoughts. Would this only need to be changed in the kernel?
It is just kernel, there was a bunch of work done in the toolchain for aarch64 to ensure support here, one of the mass rebuilds around F-23 sorted the problem out.
We started aarch64 for 64K page sizes, which RHEL 7/8 still use, but due to devices like Raspberry Pi and the way the CMA allocations work on that arch meant we lost 512Mb of RAM, not sure if POWER (and I'm referring to both endians here) either uses CMA or has the same issue, we moved to 4K page sizes.
Ultimately it's software bugs and if the software is open it's a fixable problem. Are there bugs filed? Because I've always found the IBM team that are involved in Fedora are pretty active on addressing those sorts of problems.
Peter
On Wed, Jan 6, 2021 at 9:53 pm, Peter Robinson pbrobinson@gmail.com wrote:
Ultimately it's software bugs and if the software is open it's a fixable problem. Are there bugs filed? Because I've always found the IBM team that are involved in Fedora are pretty active on addressing those sorts of problems.
Where do you want architecture-specific bugs to be filed? Is there some component in Bugzilla for ppc64le issues?
WebKit disables its heap security features ppc64le due to the 64 KB page size. But reporting the bug to upstream would be pointless, because upstream does not care about ppc64le. Reporting the bug against webkit2gtk3 on Red Hat Bugzilla would mean reporting to myself, also not very useful.
On 06/01/2021 23:02, Michael Catanzaro wrote:
On Wed, Jan 6, 2021 at 9:53 pm, Peter Robinson pbrobinson@gmail.com wrote:
Ultimately it's software bugs and if the software is open it's a fixable problem. Are there bugs filed? Because I've always found the IBM team that are involved in Fedora are pretty active on addressing those sorts of problems.
Where do you want architecture-specific bugs to be filed? Is there some component in Bugzilla for ppc64le issues?
WebKit disables its heap security features ppc64le due to the 64 KB page size. But reporting the bug to upstream would be pointless, because upstream does not care about ppc64le. Reporting the bug against webkit2gtk3 on Red Hat Bugzilla would mean reporting to myself, also not very useful.
GPUs also require firmware blobs, we can not really see inside them. If there is something in the blob that only works on 4k then we have trouble.
These issues tend to crop up from time to time in different places, not just the kernel. I started a thread[1] here with some of them. For example, Firefox WebRTC with my Elgato Camlink was not working on 64k but it is fine on 4k.
Is there any way to report on how many packages call getconf PAGESIZE or getconf PAGE_SIZE during the build? There are not many of these but they have different risks if people are swapping between 4k and 64k kernels.
Most packages compiled on one type of kernel will run on the other type but there could be a very small number that have to be recompiled if the Fedora kernel changes down to 4k.
Regardless of the packages above, I still think the Btrfs issue is the biggest surprise for people and it is worse because it is the default filesystem now. People can reboot into a new kernel but they may not be able to easily reformat their disks after mkfs at 64k.
Regards,
Daniel
1. https://forums.raptorcs.com/index.php/topic,200.msg1463.html
On Wed, Jan 6, 2021 at 9:53 pm, Peter Robinson pbrobinson@gmail.com wrote:
Ultimately it's software bugs and if the software is open it's a fixable problem. Are there bugs filed? Because I've always found the IBM team that are involved in Fedora are pretty active on addressing those sorts of problems.
Where do you want architecture-specific bugs to be filed? Is there some component in Bugzilla for ppc64le issues?
WebKit disables its heap security features ppc64le due to the 64 KB page size. But reporting the bug to upstream would be pointless, because upstream does not care about ppc64le. Reporting the bug against webkit2gtk3 on Red Hat Bugzilla would mean reporting to myself, also not very useful.
GPUs also require firmware blobs, we can not really see inside them. If there is something in the blob that only works on 4k then we have trouble.
Yes, I'm well aware, I've dealt with the difference between Fedora on 4K and RHEL/CentOS on 64K, I've engaged with vendors and they've generally been receptive with bug reports even if they do sometimes take a little while to fix them. There's some pretty big ppc64le supercomputers running with GPUs with 64k page sizes for a number of years: https://www.olcf.ornl.gov/olcf-resources/compute-systems/summit/ https://computing.llnl.gov/computers/sierra
These issues tend to crop up from time to time in different places, not just the kernel. I started a thread[1] here with some of them. For example, Firefox WebRTC with my Elgato Camlink was not working on 64k but it is fine on 4k.
Having done 4 architecture boot straps on Fedora, including the ppc64le one, as well as being the non x85 release engineer for a number of years I'm well aware of the problems involved, I've fixed 100s of bugs on ppc64le, armv7hl, aarch64, risc-v and s390x.
Is there any way to report on how many packages call getconf PAGESIZE or getconf PAGE_SIZE during the build? There are not many of these but they have different risks if people are swapping between 4k and 64k kernels.
The way I dealt with it on aarch64 was to check out all the packages and sources, we've dealt with a lot of the key ones there on aarch64 because of the difference between Fedora on 4K and RHEL/CentOS on 64K.
Most packages compiled on one type of kernel will run on the other type but there could be a very small number that have to be recompiled if the Fedora kernel changes down to 4k.
Yes, and the small number of packages would be bugs, just like bugs between different architectures even if they're compiled with exactly the same version of the toolchain and libraries. Welcome to non x86 architectures, I've been involved in !x86 for 11 years in a few weeks, it's still a fun ride.
Regardless of the packages above, I still think the Btrfs issue is the biggest surprise for people and it is worse because it is the default filesystem now. People can reboot into a new kernel but they may not be able to easily reformat their disks after mkfs at 64k.
What's the situation on filesystems like xfs or ext4?
On Wed, Jan 6, 2021 at 6:03 PM Peter Robinson pbrobinson@gmail.com wrote:
Regardless of the packages above, I still think the Btrfs issue is the biggest surprise for people and it is worse because it is the default filesystem now. People can reboot into a new kernel but they may not be able to easily reformat their disks after mkfs at 64k.
What's the situation on filesystems like xfs or ext4?
This issue was mostly resolved for both filesystems years ago. There is a patchset in review in linux-btrfs@ mailing list that resolves this problem. I expect that it'll land in Linux 5.11 or 5.12, though Josef may know more exact details on timelines.
-- 真実はいつも一つ!/ Always, there's only one truth!
On Wed, Jan 6, 2021 at 11:25 PM Neal Gompa ngompa13@gmail.com wrote:
On Wed, Jan 6, 2021 at 6:03 PM Peter Robinson pbrobinson@gmail.com wrote:
Regardless of the packages above, I still think the Btrfs issue is the biggest surprise for people and it is worse because it is the default filesystem now. People can reboot into a new kernel but they may not be able to easily reformat their disks after mkfs at 64k.
What's the situation on filesystems like xfs or ext4?
This issue was mostly resolved for both filesystems years ago. There is a patchset in review in linux-btrfs@ mailing list that resolves this problem. I expect that it'll land in Linux 5.11 or 5.12, though Josef may know more exact details on timelines.
Unless it's a small fix it'll be 5.12+ now given the main merge window for 5.11 is closed.
On Wed, Jan 6, 2021 at 6:27 PM Peter Robinson pbrobinson@gmail.com wrote:
On Wed, Jan 6, 2021 at 11:25 PM Neal Gompa ngompa13@gmail.com wrote:
On Wed, Jan 6, 2021 at 6:03 PM Peter Robinson pbrobinson@gmail.com wrote:
Regardless of the packages above, I still think the Btrfs issue is the biggest surprise for people and it is worse because it is the default filesystem now. People can reboot into a new kernel but they may not be able to easily reformat their disks after mkfs at 64k.
What's the situation on filesystems like xfs or ext4?
This issue was mostly resolved for both filesystems years ago. There is a patchset in review in linux-btrfs@ mailing list that resolves this problem. I expect that it'll land in Linux 5.11 or 5.12, though Josef may know more exact details on timelines.
Unless it's a small fix it'll be 5.12+ now given the main merge window for 5.11 is closed.
Then yeah, we're probably looking at 5.12, since it's ~80 patches last I checked.
On 07/01/2021 02:02, Neal Gompa wrote:
On Wed, Jan 6, 2021 at 6:27 PM Peter Robinson pbrobinson@gmail.com wrote:
On Wed, Jan 6, 2021 at 11:25 PM Neal Gompa ngompa13@gmail.com wrote:
On Wed, Jan 6, 2021 at 6:03 PM Peter Robinson pbrobinson@gmail.com wrote:
Regardless of the packages above, I still think the Btrfs issue is the biggest surprise for people and it is worse because it is the default filesystem now. People can reboot into a new kernel but they may not be able to easily reformat their disks after mkfs at 64k.
What's the situation on filesystems like xfs or ext4?
This issue was mostly resolved for both filesystems years ago. There is a patchset in review in linux-btrfs@ mailing list that resolves this problem. I expect that it'll land in Linux 5.11 or 5.12, though Josef may know more exact details on timelines.
Unless it's a small fix it'll be 5.12+ now given the main merge window for 5.11 is closed.
Then yeah, we're probably looking at 5.12, since it's ~80 patches last I checked.
This will be good for the workstation installed with the latest Fedora if it can use the volumes formatted with 4k and create them that way by default.
If the user formats a disk such as a USB stick and tries to use it with any other GNU/Linux system with an older kernel, such as RHEL, they may be surprised.
With the changes mentioned, will it be possible to have 4k as default for mkfs.btrfs unless the user manually overrides it?
As far as I know, there are no problems using ext4 on ppc64el systems but as btrfs is now the default, some people may end up with the 64k block size if they do an install today using the current Fedora installer.
Another hack: maybe the installer can just default to ext4 on this architecture until 5.12 is available.
Regards,
Daniel
On Thu, Jan 7, 2021 at 7:54 AM Daniel Pocock daniel@pocock.pro wrote:
On 07/01/2021 02:02, Neal Gompa wrote:
On Wed, Jan 6, 2021 at 6:27 PM Peter Robinson pbrobinson@gmail.com wrote:
On Wed, Jan 6, 2021 at 11:25 PM Neal Gompa ngompa13@gmail.com wrote:
On Wed, Jan 6, 2021 at 6:03 PM Peter Robinson pbrobinson@gmail.com wrote:
Regardless of the packages above, I still think the Btrfs issue is the biggest surprise for people and it is worse because it is the default filesystem now. People can reboot into a new kernel but they may not be able to easily reformat their disks after mkfs at 64k.
What's the situation on filesystems like xfs or ext4?
This issue was mostly resolved for both filesystems years ago. There is a patchset in review in linux-btrfs@ mailing list that resolves this problem. I expect that it'll land in Linux 5.11 or 5.12, though Josef may know more exact details on timelines.
Unless it's a small fix it'll be 5.12+ now given the main merge window for 5.11 is closed.
Then yeah, we're probably looking at 5.12, since it's ~80 patches last I checked.
This will be good for the workstation installed with the latest Fedora if it can use the volumes formatted with 4k and create them that way by default.
I suspect that's for the btrfs tools and their maintainers to decide.
If the user formats a disk such as a USB stick and tries to use it with any other GNU/Linux system with an older kernel, such as RHEL, they may be surprised.
TBH that's no different for all sorts of filesystems and feature enhancements to them, always has been.
With the changes mentioned, will it be possible to have 4k as default for mkfs.btrfs unless the user manually overrides it?
I'm sure it's possible, as mentioned above that would be a decision for the upstream btrfs maintainers, they would be better positioned to know the impact of such a change.
As far as I know, there are no problems using ext4 on ppc64el systems but as btrfs is now the default, some people may end up with the 64k block size if they do an install today using the current Fedora installer.
I suspect the amount of people using btrfs on ppc64le and moving those disks to other architectures are quite limited especially given it was just introduced as default in F-33, like everything storage related on any filesystem backups are always recommended.
Another hack: maybe the installer can just default to ext4 on this architecture until 5.12 is available.
Well given 5.12 will likely be the shipping kernel for F-34 you've still got a few weeks to liaise with btrfs upstream to see if everything is going to be in place (kernel/tools etc) to submit a self contained change for F-34 to ensure this is resolved to your satisfaction for that release.
Peter
On Thu, Jan 7, 2021 at 4:51 AM Peter Robinson pbrobinson@gmail.com wrote:
On Thu, Jan 7, 2021 at 7:54 AM Daniel Pocock daniel@pocock.pro wrote:
On 07/01/2021 02:02, Neal Gompa wrote:
On Wed, Jan 6, 2021 at 6:27 PM Peter Robinson pbrobinson@gmail.com wrote:
On Wed, Jan 6, 2021 at 11:25 PM Neal Gompa ngompa13@gmail.com wrote:
On Wed, Jan 6, 2021 at 6:03 PM Peter Robinson pbrobinson@gmail.com wrote:
> Regardless of the packages above, I still think the Btrfs issue is the > biggest surprise for people and it is worse because it is the default > filesystem now. People can reboot into a new kernel but they may not be > able to easily reformat their disks after mkfs at 64k.
What's the situation on filesystems like xfs or ext4?
This issue was mostly resolved for both filesystems years ago. There is a patchset in review in linux-btrfs@ mailing list that resolves this problem. I expect that it'll land in Linux 5.11 or 5.12, though Josef may know more exact details on timelines.
Unless it's a small fix it'll be 5.12+ now given the main merge window for 5.11 is closed.
Then yeah, we're probably looking at 5.12, since it's ~80 patches last I checked.
This will be good for the workstation installed with the latest Fedora if it can use the volumes formatted with 4k and create them that way by default.
I suspect that's for the btrfs tools and their maintainers to decide.
If the user formats a disk such as a USB stick and tries to use it with any other GNU/Linux system with an older kernel, such as RHEL, they may be surprised.
TBH that's no different for all sorts of filesystems and feature enhancements to them, always has been.
With the changes mentioned, will it be possible to have 4k as default for mkfs.btrfs unless the user manually overrides it?
I'm sure it's possible, as mentioned above that would be a decision for the upstream btrfs maintainers, they would be better positioned to know the impact of such a change.
While I'm not entirely sure, I believe that btrfs-progs will format 4k by default on all arches once everything is done. There's no particularly good reason to keep doing 64k after this is done, since it's somewhat wasteful for storage devices.
As far as I know, there are no problems using ext4 on ppc64el systems but as btrfs is now the default, some people may end up with the 64k block size if they do an install today using the current Fedora installer.
I suspect the amount of people using btrfs on ppc64le and moving those disks to other architectures are quite limited especially given it was just introduced as default in F-33, like everything storage related on any filesystem backups are always recommended.
Another hack: maybe the installer can just default to ext4 on this architecture until 5.12 is available.
Well given 5.12 will likely be the shipping kernel for F-34 you've still got a few weeks to liaise with btrfs upstream to see if everything is going to be in place (kernel/tools etc) to submit a self contained change for F-34 to ensure this is resolved to your satisfaction for that release.
I'm reasonably confident that we're going to have this fixed for Fedora 34. I would not be surprised to see foreign page size read support merged in the coming days, and then the write support merged within the next few weeks. Upstream has made really good progress here and I'm confident in it being fixed in the Linux 5.12 timeframe.
-- 真実はいつも一つ!/ Always, there's only one truth!
On 06/01/2021 Daniel Pocock wrote:
Most packages compiled on one type of kernel will run on the other type but there could be a very small number that have to be recompiled if the Fedora kernel changes down to 4k.
I'm afraid changing the page size would be more complicated than that. The same way that some projects assume everyone has a 4K page size, there are projects that assume ppc64le has a 64K page size [1]. One could argue that both types of projects have to change their code and be independent of the page size and I'd agree with that. But there are projects whose storage format depend on the page size, e.g. PMDK.
Other projects may assume ppc64le has a 64K page size in order to improve performance, e.g. gperftools/tcmalloc. For this particular case, the current code works well on 4K page size, without any downgrade in performance. If modified to use 4K page size, it would have a negative impact on users with 64K page size.
[1] https://lists.ozlabs.org/pipermail/linuxppc-dev/2021-January/222993.html
Regardless of the packages above, I still think the Btrfs issue is the biggest surprise for people and it is worse because it is the default filesystem now. People can reboot into a new kernel but they may not be able to easily reformat their disks after mkfs at 64k.
Likewise for PMDK. I wouldn't be surprised if it works without any code changes, but I don't think anyone has tested it.
On Wed, Jan 6, 2021 at 10:01 PM Michael Catanzaro mcatanzaro@gnome.org wrote:
On Wed, Jan 6, 2021 at 9:53 pm, Peter Robinson pbrobinson@gmail.com wrote:
Ultimately it's software bugs and if the software is open it's a fixable problem. Are there bugs filed? Because I've always found the IBM team that are involved in Fedora are pretty active on addressing those sorts of problems.
Where do you want architecture-specific bugs to be filed? Is there some component in Bugzilla for ppc64le issues?
As per standard package process each architecture has a tracking bug where you file a bug against the package and link it to the bug for the architecture as a blocker. The POWER blocker alias is PPCTracker [1]
Peter
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1071880
WebKit disables its heap security features ppc64le due to the 64 KB page size. But reporting the bug to upstream would be pointless, because upstream does not care about ppc64le. Reporting the bug against webkit2gtk3 on Red Hat Bugzilla would mean reporting to myself, also not very useful.
But if you link it to the PPCTracker you'll find the associated team looks at it. Having done a number of architecture boot straps and dealt with various teams at IBM during that they are quite responsive and from my experience once they're aware of issues they engage with upstream and get things fixed.
Peter
On Wed, Jan 6, 2021 at 10:49 pm, Peter Robinson pbrobinson@gmail.com wrote:
As per standard package process each architecture has a tracking bug where you file a bug against the package and link it to the bug for the architecture as a blocker. The POWER blocker alias is PPCTracker [1]
Is there some page I can use to find these bugs for other architectures? I have an unrelated issue that only occurs on s390x....
On Wed, Jan 6, 2021 at 11:01 PM Michael Catanzaro mcatanzaro@gnome.org wrote:
On Wed, Jan 6, 2021 at 10:49 pm, Peter Robinson pbrobinson@gmail.com wrote:
As per standard package process each architecture has a tracking bug where you file a bug against the package and link it to the bug for the architecture as a blocker. The POWER blocker alias is PPCTracker [1]
Is there some page I can use to find these bugs for other architectures? I have an unrelated issue that only occurs on s390x....
Should be linked to from the architecture pages in the wiki, they also used to be linked in the packaging docs, but it was quicker for me to find it from the bug I checked for the power one, the Z-series alias is ZedoraTracker, bug is https://bugzilla.redhat.com/show_bug.cgi?id=467765
We did some more troubleshooting of AMD Radeon issues on ppc64
As with Nouveau, it looks like a change from 64k to 4k page size got it working again with RX 5700. I suspect it will be similar for RX 6800 if we can get some of them, they are a good complement for the compute power.
The issue is page size, not ppc64
This sounds like a kernel driver bug that needs to be bisected and reported upstream, do you have a link to the bug report(s)?
Hi Daniel,
On Wed, 6 Jan 2021 14:11:37 +0100 Daniel Pocock daniel@pocock.pro wrote:
We did some more troubleshooting of AMD Radeon issues on ppc64
As with Nouveau, it looks like a change from 64k to 4k page size got it working again with RX 5700. I suspect it will be similar for RX 6800 if we can get some of them, they are a good complement for the compute power.
The issue is page size, not ppc64
are you talking about Maurizio's issue? When I was talking with the guy who fixed the Navi 10 driver for Power, he wasn't aware of any page size issues. But it sounds plausible.
Dan
On 07/01/2021 12:35, Dan Horák wrote:
Hi Daniel,
On Wed, 6 Jan 2021 14:11:37 +0100 Daniel Pocock daniel@pocock.pro wrote:
We did some more troubleshooting of AMD Radeon issues on ppc64
As with Nouveau, it looks like a change from 64k to 4k page size got it working again with RX 5700. I suspect it will be similar for RX 6800 if we can get some of them, they are a good complement for the compute power.
The issue is page size, not ppc64
are you talking about Maurizio's issue? When I was talking with the guy who fixed the Navi 10 driver for Power, he wasn't aware of any page size issues. But it sounds plausible.
Yes, it is in the Raptor forum
To save reading, I gave him the following in an apt repository:
- 5.9.6 kernel with 4k page size - mesa 20.3 - deps for mesa (llvm-11, libdrm, ...)
He took the 2020-10-22 firmware snapshot from Debian unstable
He disabled Wayland, to use Xorg
It worked.
I'm hoping that exactly the same set of packages will work for me whenever I can get my hands on an RX 6800 XT card.
As mentioned elsewhere in this thread and in various parts of the Raptor forum, especially this[1] thread, there are a range of applications where page size appears to have some impact.
I'd like to emphasize that there are a range of issues where page size does not appear to be a factor, there are a backlog of these issues. Therefore, emphasizing this quite strongly, I feel it is better for people to run 4k page size on workstations from F-34 so we can clearly flush out the PPC issues on 4k before we even begin to address issues with 64k. As examples, I have issues with Blender[2], Firefox[3] and JPEGs in OBS[4] where page size has been eliminated but I also have another issue[5] with Firefox, or maybe v4l2-loopback, where page size is a factor.
Vikings and Raptor hope to sell a lot more of these systems when they go live in 2021. Vikings in particular has put great effort into a German-engineering water cooling solution, there is a leaked photo in the Raptor forum. Personally, I already identified the traps in this platform within hours of unboxing but I know a lot of the users, including Fedora users, will benefit from a slightly softer landing on POWER9. I can't personally spend time following every issue through to completion myself (unless somebody wants to offer me a bounty) but I try to document them as well as I can both here and in the bug trackers.
Regards,
Daniel
1. https://forums.raptorcs.com/index.php/topic,200.msg1463.html 2. https://developer.blender.org/T80912 3. https://bugzilla.mozilla.org/show_bug.cgi?id=1677216 4. https://developer.blender.org/T80912#1085758 5. https://forums.raptorcs.com/index.php/topic,200.msg1463.html#msg1463
On Thu, Jan 7, 2021 at 10:10 AM Daniel Pocock daniel@pocock.pro wrote:
On 07/01/2021 12:35, Dan Horák wrote:
Hi Daniel,
On Wed, 6 Jan 2021 14:11:37 +0100 Daniel Pocock daniel@pocock.pro wrote:
We did some more troubleshooting of AMD Radeon issues on ppc64
As with Nouveau, it looks like a change from 64k to 4k page size got it working again with RX 5700. I suspect it will be similar for RX 6800 if we can get some of them, they are a good complement for the compute power.
The issue is page size, not ppc64
are you talking about Maurizio's issue? When I was talking with the guy who fixed the Navi 10 driver for Power, he wasn't aware of any page size issues. But it sounds plausible.
Yes, it is in the Raptor forum
To save reading, I gave him the following in an apt repository:
- 5.9.6 kernel with 4k page size
- mesa 20.3
- deps for mesa (llvm-11, libdrm, ...)
He took the 2020-10-22 firmware snapshot from Debian unstable
He disabled Wayland, to use Xorg
It worked.
I'm hoping that exactly the same set of packages will work for me whenever I can get my hands on an RX 6800 XT card.
As mentioned elsewhere in this thread and in various parts of the Raptor forum, especially this[1] thread, there are a range of applications where page size appears to have some impact.
I'd like to emphasize that there are a range of issues where page size does not appear to be a factor, there are a backlog of these issues. Therefore, emphasizing this quite strongly, I feel it is better for people to run 4k page size on workstations from F-34 so we can clearly flush out the PPC issues on 4k before we even begin to address issues with 64k. As examples, I have issues with Blender[2], Firefox[3] and JPEGs in OBS[4] where page size has been eliminated but I also have another issue[5] with Firefox, or maybe v4l2-loopback, where page size is a factor.
Vikings and Raptor hope to sell a lot more of these systems when they go live in 2021. Vikings in particular has put great effort into a German-engineering water cooling solution, there is a leaked photo in the Raptor forum. Personally, I already identified the traps in this platform within hours of unboxing but I know a lot of the users, including Fedora users, will benefit from a slightly softer landing on POWER9. I can't personally spend time following every issue through to completion myself (unless somebody wants to offer me a bounty) but I try to document them as well as I can both here and in the bug trackers.
You could make a Change proposal to switch Fedora on POWER back to 4k page sizes. Or just talk to the Fedora kernel maintainers about switching it back. We did this four years ago for ARM (as Peter noted), and if it really continues to be a problem for desktop POWER systems, then switching back makes sense.
-- 真実はいつも一つ!/ Always, there's only one truth!
On Thu, Jan 7, 2021 at 3:24 PM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Jan 7, 2021 at 10:10 AM Daniel Pocock daniel@pocock.pro wrote:
On 07/01/2021 12:35, Dan Horák wrote:
Hi Daniel,
On Wed, 6 Jan 2021 14:11:37 +0100 Daniel Pocock daniel@pocock.pro wrote:
We did some more troubleshooting of AMD Radeon issues on ppc64
As with Nouveau, it looks like a change from 64k to 4k page size got it working again with RX 5700. I suspect it will be similar for RX 6800 if we can get some of them, they are a good complement for the compute power.
The issue is page size, not ppc64
are you talking about Maurizio's issue? When I was talking with the guy who fixed the Navi 10 driver for Power, he wasn't aware of any page size issues. But it sounds plausible.
Yes, it is in the Raptor forum
To save reading, I gave him the following in an apt repository:
- 5.9.6 kernel with 4k page size
- mesa 20.3
- deps for mesa (llvm-11, libdrm, ...)
He took the 2020-10-22 firmware snapshot from Debian unstable
He disabled Wayland, to use Xorg
It worked.
I'm hoping that exactly the same set of packages will work for me whenever I can get my hands on an RX 6800 XT card.
As mentioned elsewhere in this thread and in various parts of the Raptor forum, especially this[1] thread, there are a range of applications where page size appears to have some impact.
I'd like to emphasize that there are a range of issues where page size does not appear to be a factor, there are a backlog of these issues. Therefore, emphasizing this quite strongly, I feel it is better for people to run 4k page size on workstations from F-34 so we can clearly flush out the PPC issues on 4k before we even begin to address issues with 64k. As examples, I have issues with Blender[2], Firefox[3] and JPEGs in OBS[4] where page size has been eliminated but I also have another issue[5] with Firefox, or maybe v4l2-loopback, where page size is a factor.
Vikings and Raptor hope to sell a lot more of these systems when they go live in 2021. Vikings in particular has put great effort into a German-engineering water cooling solution, there is a leaked photo in the Raptor forum. Personally, I already identified the traps in this platform within hours of unboxing but I know a lot of the users, including Fedora users, will benefit from a slightly softer landing on POWER9. I can't personally spend time following every issue through to completion myself (unless somebody wants to offer me a bounty) but I try to document them as well as I can both here and in the bug trackers.
You could make a Change proposal to switch Fedora on POWER back to 4k page sizes. Or just talk to the Fedora kernel maintainers about switching it back. We did this four years ago for ARM (as Peter noted), and if it really continues to be a problem for desktop POWER systems, then switching back makes sense.
The arm team made that change by engaging with all the people that would be impacted after working with the toolchain teams to ensure the userspace had all the needed fixes to ensure it was purely a kernel config change, that was around 18 months work to get to that point (yes, and POWER64 likely benefits from that ~ 10 releases later) but the likes of IBM should be engaged because they obviously also have an interest and are active users/contributors of Fedora on POWER.
Peter
On Thu, 7 Jan 2021 18:00:42 +0000 Peter Robinson pbrobinson@gmail.com wrote:
On Thu, Jan 7, 2021 at 3:24 PM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Jan 7, 2021 at 10:10 AM Daniel Pocock daniel@pocock.pro wrote:
On 07/01/2021 12:35, Dan Horák wrote:
Hi Daniel,
On Wed, 6 Jan 2021 14:11:37 +0100 Daniel Pocock daniel@pocock.pro wrote:
We did some more troubleshooting of AMD Radeon issues on ppc64
As with Nouveau, it looks like a change from 64k to 4k page size got it working again with RX 5700. I suspect it will be similar for RX 6800 if we can get some of them, they are a good complement for the compute power.
The issue is page size, not ppc64
are you talking about Maurizio's issue? When I was talking with the guy who fixed the Navi 10 driver for Power, he wasn't aware of any page size issues. But it sounds plausible.
Yes, it is in the Raptor forum
To save reading, I gave him the following in an apt repository:
- 5.9.6 kernel with 4k page size
- mesa 20.3
- deps for mesa (llvm-11, libdrm, ...)
He took the 2020-10-22 firmware snapshot from Debian unstable
He disabled Wayland, to use Xorg
It worked.
I'm hoping that exactly the same set of packages will work for me whenever I can get my hands on an RX 6800 XT card.
As mentioned elsewhere in this thread and in various parts of the Raptor forum, especially this[1] thread, there are a range of applications where page size appears to have some impact.
I'd like to emphasize that there are a range of issues where page size does not appear to be a factor, there are a backlog of these issues. Therefore, emphasizing this quite strongly, I feel it is better for people to run 4k page size on workstations from F-34 so we can clearly flush out the PPC issues on 4k before we even begin to address issues with 64k. As examples, I have issues with Blender[2], Firefox[3] and JPEGs in OBS[4] where page size has been eliminated but I also have another issue[5] with Firefox, or maybe v4l2-loopback, where page size is a factor.
Vikings and Raptor hope to sell a lot more of these systems when they go live in 2021. Vikings in particular has put great effort into a German-engineering water cooling solution, there is a leaked photo in the Raptor forum. Personally, I already identified the traps in this platform within hours of unboxing but I know a lot of the users, including Fedora users, will benefit from a slightly softer landing on POWER9. I can't personally spend time following every issue through to completion myself (unless somebody wants to offer me a bounty) but I try to document them as well as I can both here and in the bug trackers.
You could make a Change proposal to switch Fedora on POWER back to 4k page sizes. Or just talk to the Fedora kernel maintainers about switching it back. We did this four years ago for ARM (as Peter noted), and if it really continues to be a problem for desktop POWER systems, then switching back makes sense.
The arm team made that change by engaging with all the people that would be impacted after working with the toolchain teams to ensure the userspace had all the needed fixes to ensure it was purely a kernel config change, that was around 18 months work to get to that point (yes, and POWER64 likely benefits from that ~ 10 releases later) but the likes of IBM should be engaged because they obviously also have an interest and are active users/contributors of Fedora on POWER.
I'm talking with them,
Dan
On 07/01/2021 19:15, Dan Horák wrote:
On Thu, 7 Jan 2021 18:00:42 +0000 Peter Robinson pbrobinson@gmail.com wrote:
On Thu, Jan 7, 2021 at 3:24 PM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Jan 7, 2021 at 10:10 AM Daniel Pocock daniel@pocock.pro wrote:
On 07/01/2021 12:35, Dan Horák wrote:
Hi Daniel,
On Wed, 6 Jan 2021 14:11:37 +0100 Daniel Pocock daniel@pocock.pro wrote:
We did some more troubleshooting of AMD Radeon issues on ppc64
As with Nouveau, it looks like a change from 64k to 4k page size got it working again with RX 5700. I suspect it will be similar for RX 6800 if we can get some of them, they are a good complement for the compute power.
The issue is page size, not ppc64
are you talking about Maurizio's issue? When I was talking with the guy who fixed the Navi 10 driver for Power, he wasn't aware of any page size issues. But it sounds plausible.
Yes, it is in the Raptor forum
To save reading, I gave him the following in an apt repository:
- 5.9.6 kernel with 4k page size
- mesa 20.3
- deps for mesa (llvm-11, libdrm, ...)
He took the 2020-10-22 firmware snapshot from Debian unstable
He disabled Wayland, to use Xorg
It worked.
I'm hoping that exactly the same set of packages will work for me whenever I can get my hands on an RX 6800 XT card.
As mentioned elsewhere in this thread and in various parts of the Raptor forum, especially this[1] thread, there are a range of applications where page size appears to have some impact.
I'd like to emphasize that there are a range of issues where page size does not appear to be a factor, there are a backlog of these issues. Therefore, emphasizing this quite strongly, I feel it is better for people to run 4k page size on workstations from F-34 so we can clearly flush out the PPC issues on 4k before we even begin to address issues with 64k. As examples, I have issues with Blender[2], Firefox[3] and JPEGs in OBS[4] where page size has been eliminated but I also have another issue[5] with Firefox, or maybe v4l2-loopback, where page size is a factor.
Vikings and Raptor hope to sell a lot more of these systems when they go live in 2021. Vikings in particular has put great effort into a German-engineering water cooling solution, there is a leaked photo in the Raptor forum. Personally, I already identified the traps in this platform within hours of unboxing but I know a lot of the users, including Fedora users, will benefit from a slightly softer landing on POWER9. I can't personally spend time following every issue through to completion myself (unless somebody wants to offer me a bounty) but I try to document them as well as I can both here and in the bug trackers.
You could make a Change proposal to switch Fedora on POWER back to 4k page sizes. Or just talk to the Fedora kernel maintainers about switching it back. We did this four years ago for ARM (as Peter noted), and if it really continues to be a problem for desktop POWER systems, then switching back makes sense.
The arm team made that change by engaging with all the people that would be impacted after working with the toolchain teams to ensure the userspace had all the needed fixes to ensure it was purely a kernel config change, that was around 18 months work to get to that point (yes, and POWER64 likely benefits from that ~ 10 releases later) but the likes of IBM should be engaged because they obviously also have an interest and are active users/contributors of Fedora on POWER.
I'm talking with them,
Can Fedora ship multiple flavours from a single source package of the kernel, as in Debian? This doesn't solve the issue of ensuring every other package in the archive is rebuilt on a host with the same page size, but it puts the 4k kernel within reach for many more people. In my case, it allowed me to "divide and conquer", probably not the best metaphor to use this week, but nonetheless, it allowed me to sideline the page size issues and focus on other issues.
If Fedora can ship both 64k and 4k kernel with an easy way for users to alternate between them, that could be a middle ground for F-34.
Going completely 4k or completely fixing 64k, either of these directions requires an effort now. I can see some hope though:
- can strace or some other mechanism be used during userspace package builds to see which packages are calling getconf(PAGESIZE / PAGE_SIZE) from their Makefiles, excluding the unit tests?
- can reproducible builds methods be used to compare userspace packages built on both 4k and 64k to try and work out which packages are built differently?
IBM already offered some bounties for work on POWER. There are a number of people who may be willing to contribute to parts of this effort if incentives can be aligned with their projects. If IBM can make the hardware more accessible, like the Lenovo offers for developers, that could also encourage people.
Regards,
Daniel
On Thu, Jan 7, 2021 at 1:31 PM Daniel Pocock daniel@pocock.pro wrote:
On 07/01/2021 19:15, Dan Horák wrote:
On Thu, 7 Jan 2021 18:00:42 +0000 Peter Robinson pbrobinson@gmail.com wrote:
On Thu, Jan 7, 2021 at 3:24 PM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Jan 7, 2021 at 10:10 AM Daniel Pocock daniel@pocock.pro wrote:
On 07/01/2021 12:35, Dan Horák wrote:
Hi Daniel,
On Wed, 6 Jan 2021 14:11:37 +0100 Daniel Pocock daniel@pocock.pro wrote:
> > We did some more troubleshooting of AMD Radeon issues on ppc64 > > As with Nouveau, it looks like a change from 64k to 4k page size got it > working again with RX 5700. I suspect it will be similar for RX 6800 if > we can get some of them, they are a good complement for the compute power. > > The issue is page size, not ppc64
are you talking about Maurizio's issue? When I was talking with the guy who fixed the Navi 10 driver for Power, he wasn't aware of any page size issues. But it sounds plausible.
Yes, it is in the Raptor forum
To save reading, I gave him the following in an apt repository:
- 5.9.6 kernel with 4k page size
- mesa 20.3
- deps for mesa (llvm-11, libdrm, ...)
He took the 2020-10-22 firmware snapshot from Debian unstable
He disabled Wayland, to use Xorg
It worked.
I'm hoping that exactly the same set of packages will work for me whenever I can get my hands on an RX 6800 XT card.
As mentioned elsewhere in this thread and in various parts of the Raptor forum, especially this[1] thread, there are a range of applications where page size appears to have some impact.
I'd like to emphasize that there are a range of issues where page size does not appear to be a factor, there are a backlog of these issues. Therefore, emphasizing this quite strongly, I feel it is better for people to run 4k page size on workstations from F-34 so we can clearly flush out the PPC issues on 4k before we even begin to address issues with 64k. As examples, I have issues with Blender[2], Firefox[3] and JPEGs in OBS[4] where page size has been eliminated but I also have another issue[5] with Firefox, or maybe v4l2-loopback, where page size is a factor.
Vikings and Raptor hope to sell a lot more of these systems when they go live in 2021. Vikings in particular has put great effort into a German-engineering water cooling solution, there is a leaked photo in the Raptor forum. Personally, I already identified the traps in this platform within hours of unboxing but I know a lot of the users, including Fedora users, will benefit from a slightly softer landing on POWER9. I can't personally spend time following every issue through to completion myself (unless somebody wants to offer me a bounty) but I try to document them as well as I can both here and in the bug trackers.
You could make a Change proposal to switch Fedora on POWER back to 4k page sizes. Or just talk to the Fedora kernel maintainers about switching it back. We did this four years ago for ARM (as Peter noted), and if it really continues to be a problem for desktop POWER systems, then switching back makes sense.
The arm team made that change by engaging with all the people that would be impacted after working with the toolchain teams to ensure the userspace had all the needed fixes to ensure it was purely a kernel config change, that was around 18 months work to get to that point (yes, and POWER64 likely benefits from that ~ 10 releases later) but the likes of IBM should be engaged because they obviously also have an interest and are active users/contributors of Fedora on POWER.
I'm talking with them,
Can Fedora ship multiple flavours from a single source package of the kernel, as in Debian? This doesn't solve the issue of ensuring every other package in the archive is rebuilt on a host with the same page size, but it puts the 4k kernel within reach for many more people. In my case, it allowed me to "divide and conquer", probably not the best metaphor to use this week, but nonetheless, it allowed me to sideline the page size issues and focus on other issues.
We have avoided doing so for much larger target markets than the power workstation market. It is not just a simple matter of "adding another config and flavor". Kernel builds are massive, we do multiple per week, and sometimes per day. the test matrix would have to be expanded, dnf and such have to learn how to deal with multiple variants on the same arch. This is not trivial, and certainly not a solution to take on as a short term band aid until things can be properly worked out for a long term solution.
If Fedora can ship both 64k and 4k kernel with an easy way for users to alternate between them, that could be a middle ground for F-34.
Going completely 4k or completely fixing 64k, either of these directions requires an effort now. I can see some hope though:
If effort is going to be spent, it should be spent here. Particularly as I see increased page sizes more likely among other architectures in the larger term, if I had to make a prediction. Memory isn't getting any smaller.
Justin
- can strace or some other mechanism be used during userspace package
builds to see which packages are calling getconf(PAGESIZE / PAGE_SIZE) from their Makefiles, excluding the unit tests?
- can reproducible builds methods be used to compare userspace packages
built on both 4k and 64k to try and work out which packages are built differently?
IBM already offered some bounties for work on POWER. There are a number of people who may be willing to contribute to parts of this effort if incentives can be aligned with their projects. If IBM can make the hardware more accessible, like the Lenovo offers for developers, that could also encourage people.
Regards,
Daniel _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org