mentos1386/gtk-transcription
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: use adw and add back whisper etc.

Browse source
This commit is contained in:
Tine Jozelj 2023-07-16 15:55:57 +02:00
parent 2965231a61
commit 65c3d07b3d
Signed by: mentos1386
SSH key fingerprint: SHA256:MNtTsLbihYaWF8j1fkOHfkKNlnN1JQfxEU/rBU8nCGw
12 changed files with 2048 additions and 112 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
.gitignore vendored
View file

@ -10,3 +10,4 @@
/.flatpak/
/vendor
/.vscode
.flatpak-builder/

1738
Cargo.lock generated Normal file
View file

File diff suppressed because it is too large Load diff

4
Cargo.toml
View file

@ -8,8 +8,12 @@ edition = "2021"
lto = true
[dependencies]
adw = { version = "0.4.4", package = "libadwaita", features = ["v1_2"] }
cpal = "0.15.2"
gettext-rs = { version = "0.7", features = ["gettext-system"] }
gtk = { version = "0.6", package = "gtk4", features = ["v4_8"] }
once_cell = "1.14"
ringbuf = "0.3.3"
tracing = "0.1.37"
tracing-subscriber = "0.3"
whisper-rs = { version = "0.8.0", features = [] }

69
README.md
View file

@ -1,77 +1,16 @@
# GTK + Rust + Meson + Flatpak = <3
A boilerplate template to get started with GTK, Rust, Meson, Flatpak made for GNOME. It can be adapted for other desktop environments like elementary.
<div align="center">
![Main window](data/resources/screenshots/screenshot1.png "Main window")
</div>
## What does it contains?
- A simple window with a headerbar
- Bunch of useful files that you SHOULD ship with your application on Linux:
- Metainfo: describe your application for the different application stores out there;
- Desktop: the application launcher;
- Icons: This repo contains three icons, a normal, a nightly & monochromatic icon (symbolic) per the GNOME HIG, exported using [App Icon Preview](https://flathub.org/apps/details/org.gnome.design.AppIconPreview).
- Flatpak Manifest for nightly builds
- Dual installation support
- Uses Meson for building the application
- Bundles the UI files & the CSS using gresources
- A pre-commit hook to run rustfmt on your code
- Tests to validate your Metainfo, Schemas & Desktop files
- Gsettings to store the window state, more settings could be added
- Gitlab CI to produce flatpak nightlies
- i18n support
## How to init a project ?
The template ships a simple python script to init a project easily. It asks you a few questions and replaces & renames all the necessary files.
The script requires having `git` installed on your system.
You can run it with,
```shell
python3 create-project.py
```
```shell
➜ python3 create-project.py
Welcome to GTK Rust Template
Name: Contrast
Project Name: contrast
Application ID (e.g. org.domain.MyAwesomeApp, see: https://developer.gnome.org/ChooseApplicationID/): org.gnome.design.Contrast
Author: Bilal Elmoussaoui
Email: bil.elmoussaoui@gmail.com
```
A new directory named `contrast` containing the generated project
# `gtk-Transcription`
## Building the project
Make sure you have `flatpak` and `flatpak-builder` installed. Then run the commands below. Replace `<application_id>` with the value you entered during project creation. Please note that these commands are just for demonstration purposes. Normally this would be handled by your IDE, such as GNOME Builder or VS Code with the Flatpak extension.
Make sure you have `flatpak` and `flatpak-builder` installed. Then run the commands below. Please note that these commands are just for demonstration purposes. Normally this would be handled by your IDE, such as GNOME Builder or VS Code with the Flatpak extension.
```
flatpak install --user org.gnome.Sdk//43 org.freedesktop.Sdk.Extension.rust-stable//22.08 org.gnome.Platform//43 org.freedesktop.Sdk.Extension.llvm14//22.08
flatpak-builder --user flatpak_app build-aux/<application_id>.Devel.json
flatpak-builder --user flatpak_app build-aux/dev.mnts.Transcription.Devel.json
```
## Running the project
Once the project is build, run the command below. Replace Replace `<application_id>` and `<project_name>` with the values you entered during project creation. Please note that these commands are just for demonstration purposes. Normally this would be handled by your IDE, such as GNOME Builder or VS Code with the Flatpak extension.
```
flatpak-builder --run flatpak_app build-aux/<application_id>.Devel.json <project_name>
flatpak-builder --run flatpak_app build-aux/dev.mnts.Transcription.Devel.json Transcription
```
## Community
Join the GNOME and gtk-rs community!
- [Matrix chat](https://matrix.to/#/#rust:gnome.org): chat with other developers using gtk-rs
- [Discourse forum](https://discourse.gnome.org/tag/rust): topics tagged with `rust` on the GNOME forum.
- [GNOME circle](https://circle.gnome.org/): take inspiration from applications and libraries already extending the GNOME ecosystem.
## Credits
- [Podcasts](https://gitlab.gnome.org/World/podcasts)
- [Shortwave](https://gitlab.gnome.org/World/Shortwave)

3
build-aux/dev.mnts.Transcription.Devel.json
View file

@ -1,7 +1,7 @@
{
"id": "dev.mnts.Transcription.Devel",
"runtime": "org.gnome.Platform",
"runtime-version": "43",
"runtime-version": "44",
"sdk": "org.gnome.Sdk",
"sdk-extensions": [
"org.freedesktop.Sdk.Extension.rust-stable",
@ -12,6 +12,7 @@
"--share=ipc",
"--socket=fallback-x11",
"--socket=wayland",
"--socket=pulseaudio",
"--device=dri",
"--env=RUST_LOG=gtk_transcription=debug",
"--env=G_MESSAGES_DEBUG=none",

9
data/dev.mnts.Transcription.metainfo.xml.in.in
View file

@ -6,9 +6,9 @@
<!-- Insert your license of choice here -->
<!-- <project_license>MIT</project_license> -->
<name>Transcription</name>
<summary>Write a GTK + Rust application</summary>
<summary>Automated Voice Transcription</summary>
<description>
<p>A boilerplate template for GTK + Rust. It uses Meson as a build system and has flatpak support by default.</p>
<p>Transcription of voice using Whisper by OpenAI.</p>
</description>
<screenshots>
<screenshot type="default">
@ -18,9 +18,9 @@
</screenshots>
<url type="homepage">https://gitlab.gnome.org/bilelmoussaoui/gtk-transcription</url>
<url type="bugtracker">https://gitlab.gnome.org/bilelmoussaoui/gtk-transcription/issues</url>
<content_rating type="oars-1.0" />
<content_rating type="oars-1.0"/>
<releases>
<release version="0.1.0" date="2019-07-11" />
<release version="0.1.0" date="2019-07-11"/>
</releases>
<kudos>
<!--
@ -35,3 +35,4 @@
<translation type="gettext">@gettext-package@</translation>
<launchable type="desktop-id">@app-id@.desktop</launchable>
</component>

58
data/resources/ui/window.ui
View file

@ -16,26 +16,48 @@
</item>
</section>
</menu>
<template class="ExampleApplicationWindow" parent="GtkApplicationWindow">
<child type="titlebar">
<object class="GtkHeaderBar" id="headerbar">
<child type="end">
<object class="GtkMenuButton" id="appmenu_button">
<property name="icon-name">open-menu-symbolic</property>
<property name="menu-model">primary_menu</property>
<property name="primary">True</property>
<property name="tooltip-text" translatable="yes">Main Menu</property>
<template class="ExampleApplicationWindow" parent="AdwApplicationWindow">
<property name="content">
<object class="GtkBox">
<property name="orientation">vertical</property>
<child>
<object class="AdwHeaderBar" id="headerbar">
<property name="title-widget">
<object class="AdwWindowTitle">
<property name="title" translatable="yes">Transcription</property>
</object>
</property>
<child type="end">
<object class="GtkMenuButton" id="appmenu_button">
<property name="icon-name">open-menu-symbolic</property>
<property name="menu-model">primary_menu</property>
</object>
</child>
</object>
</child>
<child>
<object class="GtkLabel" id="label">
<property name="label" translatable="yes">Hello world!</property>
<property name="hexpand">True</property>
<property name="vexpand">True</property>
<style>
<class name="title-header"/>
</style>
</object>
</child>
<child>
<object class="GtkButton" id="button">
<property name="label">Run it!</property>
<property name="action-name">app.run_it</property>
</object>
</child>
<child>
<object class="GtkTextView" id="text">
<property name="editable">False</property>
</object>
</child>
</object>
</child>
<child>
<object class="GtkLabel" id="label">
<property name="label" translatable="yes">Hello world!</property>
<style>
<class name="title-header"/>
</style>
</object>
</child>
</property>
</template>
</interface>

34
src/application.rs
View file

@ -1,11 +1,13 @@
use gettextrs::gettext;
use tracing::{debug, info};
use gtk::prelude::*;
use gtk::subclass::prelude::*;
use gtk::{gdk, gio, glib};
use adw::prelude::*;
use adw::subclass::prelude::*;
use gtk::{gio, glib};
use crate::config::{APP_ID, PKGDATADIR, PROFILE, VERSION};
use crate::whisper::run_whisper;
use crate::window::ExampleApplicationWindow;
mod imp {
@ -22,7 +24,7 @@ mod imp {
impl ObjectSubclass for ExampleApplication {
const NAME: &'static str = "ExampleApplication";
type Type = super::ExampleApplication;
type ParentType = gtk::Application;
type ParentType = adw::Application;
}
impl ObjectImpl for ExampleApplication {}
@ -55,18 +57,18 @@ mod imp {
// Set icons for shell
gtk::Window::set_default_icon_name(APP_ID);
app.setup_css();
app.setup_gactions();
app.setup_accels();
}
}
impl GtkApplicationImpl for ExampleApplication {}
impl AdwApplicationImpl for ExampleApplication {}
}
glib::wrapper! {
pub struct ExampleApplication(ObjectSubclass<imp::ExampleApplication>)
@extends gio::Application, gtk::Application,
@extends gio::Application, gtk::Application, adw::Application,
@implements gio::ActionMap, gio::ActionGroup;
}
@ -91,7 +93,13 @@ impl ExampleApplication {
app.show_about_dialog();
})
.build();
self.add_action_entries([action_quit, action_about]);
// Run it!
let action_run_it = gio::ActionEntry::builder("run_it")
.activate(|_, _, _| run_whisper().expect("Failed to run whisper"))
.build();
self.add_action_entries([action_quit, action_about, action_run_it]);
}
// Sets up keyboard shortcuts
@ -100,18 +108,6 @@ impl ExampleApplication {
self.set_accels_for_action("window.close", &["<Control>w"]);
}
fn setup_css(&self) {
let provider = gtk::CssProvider::new();
provider.load_from_resource("/dev/mnts/Transcription/style.css");
if let Some(display) = gdk::Display::default() {
gtk::StyleContext::add_provider_for_display(
&display,
&provider,
gtk::STYLE_PROVIDER_PRIORITY_APPLICATION,
);
}
}
fn show_about_dialog(&self) {
let dialog = gtk::AboutDialog::builder()
.logo_icon_name(APP_ID)

1
src/config.rs.in
View file

@ -5,3 +5,4 @@ pub const PKGDATADIR: &str = @PKGDATADIR@;
pub const PROFILE: &str = @PROFILE@;
pub const RESOURCES_FILE: &str = concat!(@PKGDATADIR@, "/resources.gresource");
pub const VERSION: &str = @VERSION@;

2
src/main.rs
View file

@ -1,6 +1,6 @@
mod application;
#[rustfmt::skip]
mod config;
mod whisper;
mod window;
use gettextrs::{gettext, LocaleCategory};

231
src/whisper.rs Normal file
View file

@ -0,0 +1,231 @@
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
use ringbuf::{Consumer, LocalRb, Rb, SharedRb};
use std::io::Write;
use std::mem::MaybeUninit;
use std::sync::Arc;
use std::time::{Duration, Instant};
use std::{cmp, thread};
use whisper_rs::{
print_system_info, FullParams, SamplingStrategy, WhisperContext, WhisperState, WhisperToken,
};
const LATENCY_MS: f32 = 4000.0;
const NUM_ITERS: usize = 2;
const NUM_ITERS_SAVED: usize = 2;
const MODEL_NAME: &str = "ggml-medium.en.bin";
pub fn run_whisper() -> Result<(), &'static str> {
// load a context and model
let ctx = WhisperContext::new(
format!("/home/tine/projects/whisper.cpp/models/{}", MODEL_NAME).as_str(),
)
.expect("failed to load model");
// make a state
let mut state = ctx.create_state().expect("failed to create state");
let host = cpal::default_host();
let device = host
.default_input_device()
.expect("failed to get default input device");
println!("Device: {}", device.name().unwrap());
let config = device
.supported_input_configs()
.unwrap()
.find(|c| c.channels() == 1 && c.sample_format() == cpal::SampleFormat::F32)
.expect("failed to find supported input config")
.with_sample_rate(cpal::SampleRate(16000))
.config();
println!("Config: {:?}", config);
println!("{}", print_system_info());
let latency_frames = (LATENCY_MS / 1_000.0) * config.sample_rate.0 as f32;
let latency_samples = latency_frames as usize * config.channels as usize;
let sampling_freq = config.sample_rate.0 as f32;
// The buffer to share samples
let ring = SharedRb::new(latency_samples * 2);
let (mut producer, mut consumer) = ring.split();
let stream = device
.build_input_stream(
&config,
move |data: &[f32], _: &cpal::InputCallbackInfo| {
let mut output_fell_behind = false;
for &sample in data {
if producer.push(sample).is_err() {
output_fell_behind = true;
}
}
if output_fell_behind {
eprintln!("output stream fell behind: try increasing latency");
}
},
move |err| {
eprintln!("an error occurred on stream: {}", err);
},
Some(Duration::from_secs(10)),
)
.expect("failed to build stream");
stream.play().expect("failed to play stream");
process_loop(
&mut consumer,
latency_samples,
sampling_freq,
&mut state,
&ctx,
)
.expect("failed to process loop");
Ok(())
}
pub fn process_loop(
consumer: &mut Consumer<f32, Arc<SharedRb<f32, Vec<MaybeUninit<f32>>>>>,
latency_samples: usize,
sampling_freq: f32,
state: &mut WhisperState,
ctx: &WhisperContext,
) -> Result<(), &'static str> {
let mut transcription = String::new();
// Variables used across loop iterations
let mut iter_samples = LocalRb::new(latency_samples * NUM_ITERS * 2);
let mut iter_num_samples = LocalRb::new(NUM_ITERS);
let mut iter_tokens = LocalRb::new(NUM_ITERS_SAVED);
for _ in 0..NUM_ITERS {
iter_num_samples
.push(0)
.expect("Error initailizing iter_num_samples");
}
consumer.pop_iter().count();
let mut start_time = Instant::now();
let mut num_chars_to_delete = 0;
let mut loop_num = 0;
let mut words = "".to_owned();
loop {
loop_num += 1;
// Only run every LATENCY_MS
let duration = start_time.elapsed();
let latency = Duration::from_millis(LATENCY_MS as u64);
println!(
"Duration: {} Latency: {}",
duration.as_millis(),
latency.as_millis()
);
if duration < latency {
let sleep_time = latency - duration;
thread::sleep(sleep_time);
} else {
println!("Classification got behind. It took to long. Try using a smaller model and/or more threads");
}
start_time = Instant::now();
// Collect the samples
let samples: Vec<_> = consumer.pop_iter().collect();
let num_samples_to_delete = iter_num_samples
.push_overwrite(samples.len())
.expect("Error num samples to delete is off");
for _ in 0..num_samples_to_delete {
iter_samples.pop();
}
iter_samples.push_iter(&mut samples.into_iter());
let (head, tail) = iter_samples.as_slices();
let current_samples = [head, tail].concat();
// Get tokens to be deleted
if loop_num > 1 {
let num_tokens = state.full_n_tokens(0).expect("Error getting num tokens");
let token_time_end = state
.full_get_segment_t1(0)
.expect("Error getting token time");
let token_time_per_ms =
token_time_end as f32 / (LATENCY_MS * cmp::min(loop_num, NUM_ITERS) as f32); // token times are not a value in ms, they're 150 per second
let ms_per_token_time = 1.0 / token_time_per_ms;
let mut tokens_saved = vec![];
// Skip beginning and end token
for i in 1..num_tokens - 1 {
let token = state
.full_get_token_data(0, i)
.expect("Error getting token data");
let token_t0_ms = token.t0 as f32 * ms_per_token_time;
let ms_to_delete = num_samples_to_delete as f32 / (sampling_freq / 1000.0);
// Save tokens for whisper context
if (loop_num > NUM_ITERS) && token_t0_ms < ms_to_delete {
tokens_saved.push(token.id);
}
}
num_chars_to_delete = words.chars().count();
if loop_num > NUM_ITERS {
num_chars_to_delete -= tokens_saved
.iter()
.map(|x| ctx.token_to_str(*x).expect("Error"))
.collect::<String>()
.chars()
.count();
}
iter_tokens.push_overwrite(tokens_saved);
}
// Make the model params
let (head, tail) = iter_tokens.as_slices();
let tokens = [head, tail]
.concat()
.into_iter()
.flatten()
.collect::<Vec<WhisperToken>>();
let mut params = whisper_params();
params.set_tokens(&tokens);
// Run the model
state
.full(params, &current_samples)
.expect("failed to convert samples");
// Update the words on screen
if num_chars_to_delete != 0 {
transcription = transcription
.split_at(transcription.len() - num_chars_to_delete)
.0
.to_string();
}
let num_tokens = state.full_n_tokens(0).expect("Error getting num tokens");
words = (1..num_tokens - 1)
.map(|i| {
state
.full_get_token_text(0, i)
.map_err(|_| "".to_string())
.expect("")
})
.collect::<String>();
transcription += &words;
print!("{}", words);
std::io::stdout().flush().unwrap();
}
}
pub fn whisper_params<'a>() -> FullParams<'a, 'a> {
let mut params = FullParams::new(SamplingStrategy::default());
params.set_print_progress(false);
params.set_print_special(false);
params.set_print_realtime(false);
params.set_print_timestamps(false);
params.set_suppress_blank(true);
params.set_language(Some("en"));
params.set_token_timestamps(true);
params.set_duration_ms(LATENCY_MS as i32);
params.set_no_context(true);
params.set_n_threads(10);
params
}

10
src/window.rs
View file

@ -1,5 +1,6 @@
use gtk::prelude::*;
use gtk::subclass::prelude::*;
use adw::prelude::*;
use adw::subclass::prelude::*;
use gtk::{gio, glib};
use crate::application::ExampleApplication;
@ -12,7 +13,7 @@ mod imp {
#[template(resource = "/dev/mnts/Transcription/ui/window.ui")]
pub struct ExampleApplicationWindow {
#[template_child]
pub headerbar: TemplateChild<gtk::HeaderBar>,
pub headerbar: TemplateChild<adw::HeaderBar>,
pub settings: gio::Settings,
}
@ -29,7 +30,7 @@ mod imp {
impl ObjectSubclass for ExampleApplicationWindow {
const NAME: &'static str = "ExampleApplicationWindow";
type Type = super::ExampleApplicationWindow;
type ParentType = gtk::ApplicationWindow;
type ParentType = adw::ApplicationWindow;
fn class_init(klass: &mut Self::Class) {
klass.bind_template();
@ -74,6 +75,7 @@ mod imp {
}
impl ApplicationWindowImpl for ExampleApplicationWindow {}
impl AdwApplicationWindowImpl for ExampleApplicationWindow {}
}
glib::wrapper! {