Hi, here are my Gemini-assisted minutes for this week's meeting.
*Summary*
Meeting reviewed the Anthony voice control project with technical discussions on accessibility integration and security requirements.
*Voice Assistant Project Review*
Participants reviewed the Anthony voice assistant architecture, noting its use of large language models for desktop task automation. Discussions emphasized utilizing accessibility trees to improve integration.
*Technical and Security Strategy*
The team addressed hardware resource constraints and potential security risks regarding TCP port usage. Experts proposed utilizing D-Bus portals to enhance application security and sandboxing.
*Integration and Future Planning*
The group decided to align future efforts with existing ibus speech initiatives. Members also planned the summer meeting schedule to accommodate upcoming conference travel.
*Decisions*
Aligned
• *Anthony voice control project integration* The group decided to pursue exploring the integration of the Anthony voice control project into Fedora Workstation as a default feature.
• *IBus speech-to-text collaboration meeting* The group decided to organize a dedicated meeting to discuss potential collaboration between the Anthony project and general IBus speech-to-text efforts.
• *July meeting schedule pause* The group decided to hold one final meeting before pausing the regular meeting schedule for the month of July, with plans to reconvene in August.
*Next steps*
• [Matthias Clasen] File Design Issue: File or locate an existing design ticket for the voice control project to ensure proper integration.
• [Jiri Prajzner] Collaborate on Internationalization: Discuss ways to align and collaborate on the internationalization and speech to text efforts.
•[Matthias Clasen] Create Tracking Issue: Open a new ticket on Fedora Forge to track the progress and support for the Anthony project.
• [Allan Day] Share Design Tickets: Locate and distribute information about existing speech to text design tickets.
• [The group] Test Project: Experiment with the Anthony voice control project and report any feedback or issues to the author.
*Details*
• *Meeting Opening and Agenda*: Matthias Clasen opened the meeting with a quorum present and confirmed the absence of a pre-written agenda. Matthias proposed a discussion regarding a potential future feature for the desktop, specifically voice control, and asked other participants for any additional topics to discuss.
• *Anthony Voice Control Project Overview*: Jiri Prajzner introduced Anthony, a voice-driven desktop assistant designed to assist with tasks such as adjusting settings, managing windows, and engaging in conversational modes. The project includes vision capabilities that take desktop screenshots to assist users with low or no vision, and it utilizes an LLM for complex tasks while remaining capable of short-circuiting to tools for simpler requests. Jiri explained the technical architecture, which relies on Faster Whisper for speech-to-text, a sentence transformer embedding model, and Google’s Gemma 4B Mixture of Experts model to facilitate tool calling and conversational interaction.
• *Desktop Accessibility Strategy*: The group discussed the state of desktop accessibility, noting the fragmentation within the Linux ecosystem compared to the more unified systems on Apple and Microsoft platforms. Matthias Clasen emphasized that developing voice control features that benefit the general public—similar to mobile phone implementations—would improve the overall utility for those who rely on accessibility features. Jiri Prajzner noted that Canonical is exploring accessibility models and has also investigated the use of Gemma and Qwen, with Jiri favoring Gemma for its lightweight, mixture-of-experts architecture.
• *Hardware Constraints and Resources*: Neal Gompa and Jiri Prajzner discussed the resource-intensive nature of AI models, highlighting that many computers lack the dedicated accelerator hardware needed to run these models without significant CPU load. Jiri acknowledged these concerns and noted that they had performed performance tuning to enhance responsiveness. They also considered the use of smaller, specialized models that could run on older hardware, such as Jiri’s 10-year-old laptop, by avoiding the need for dedicated GPUs or neural processing units.
• *Accessibility Parsing and Design*: Neal Gompa suggested leveraging accessibility trees, which are already parsed as text, to simplify the process of reading desktop content. Allan Day emphasized the importance of a thoughtful user experience design for the project to ensure it is fully integrated rather than bolted on. Matthias Clasen committed to filing a design ticket to ensure that the project receives the necessary attention for proper integration into the desktop environment.
• *Security and Sandboxing*: Michael Catanzaro raised concerns regarding the project's reliance on a service running on a TCP port, which could grant unauthorized access to sandboxed applications. Jiri Prajzner agreed that security hardening is required, and the group discussed using D-Bus or portals as more secure alternatives to manage application access without needing to whitelist applications within Flatpak.
• *IBUS and Internationalization*: Jens-Ulrik Petersen recommended aligning the project's efforts with existing IBUS speech-to-text initiatives to better support locale selection and language switching. Jiri Prajzner agreed that this integration would simplify the technical requirements, and Matthias Clasen proposed setting up a dedicated meeting to coordinate these internationalization efforts across the relevant teams.
• *Cross-Platform Compatibility*: Neal Gompa noted that while the KDE ecosystem has explored the use of LLMs in applications, they face significant hardware constraints similar to those mentioned by the GNOME group. Jiri Prajzner confirmed that while Anthony is currently tied to GNOME, the project is not fundamentally limited to one desktop environment and could be made more general-purpose to support other platforms.
• *Fedora Packaging and Vulkan Support*: Matthias Clasen and Jiri Prajzner discussed challenges with packaging Llama CPP for Fedora, specifically regarding backend support for Intel GPUs. They noted that recent updates to the Fedora package now include Vulkan support, which resolves these issues and allows for broader hardware utilization without requiring users to manually build the software.
• *Feedback and Deployment Planning*: Jiri Prajzner invited participants to test the project and provide feedback to help refine the tool, as he is currently working on it alone. Regarding deployment, Matthias Clasen suggested that containerizing the project would simplify the installation process by providing a single package. Matthias also mentioned plans to discuss the project in relevant accessibility communication channels to raise awareness and gain further insight.
• *July Scheduling*: The group discussed the meeting schedule for July, noting that several members would be traveling for the GUADEC conference in Spain. They agreed to hold one meeting in early July and then suspend further meetings for the remainder of the month to accommodate summer schedules and travel plans, with the intention of resuming in August.
• *FOSSY Conference Announcement*: Neal Gompa announced that the schedule for FOSSY, the North American conference occurring in Vancouver, Canada, from August 6th through 9th, has been published and contains a variety of relevant topics.
• *Fontconfig 2.18 Status*: Jens-Ulrik Petersen provided a brief update on the change proposal for Fontconfig 2.18, noting that it includes new features for detecting font styles and provides a hashmap for default mappings to UI styles. Jens-Ulrik advised that the release might require a rebuild of all font packages due to changes in the cache format.
