Vmode

key	body
[v]	start/stop
vmc	port:34539
osc	port:34540

VMC/OSC protocol support for character motion control
Works with external AI tools to create interactive characters for conversations and livestreaming

link

example

osc

character jump

# pacman -S rust
$ mkdir -p ./tmp/src
$ cd ./tmp

Cargo.toml

[package]
name = "example"
edition = "2021"

[dependencies]
rosc = "0.10"

src/main.rs

use rosc::{OscMessage, OscPacket, OscType};
use std::net::UdpSocket;

fn main() {
	let socket = UdpSocket::bind("0.0.0.0:0").unwrap();
	let target = "127.0.0.1:34540";
	// Jump
	send_osc(&socket, target, "/player/jump", vec![]);
}

fn send_osc(socket: &UdpSocket, target: &str, addr: &str, args: Vec<OscType>) {
	let msg = OscMessage { addr: addr.into(), args };
	let buf = rosc::encoder::encode(&OscPacket::Message(msg)).unwrap();
	socket.send_to(&buf, target).unwrap();
}

$ cargo build
$ ./target/debug/example

/player/move: [x, y, delay] ... 0, 1, 3
/player/look: [x, y, delay] ... -10, 10, 1
/player/fly: [start/stop, none, delay] ... 1, 0, 10
/player/fly: [start/stop, none, delay] ... 0, 0, 0
/player/teleport: [x, y, z] ... 2284050.0, 0, 642049020.0
/player/screenshot: ~/Library/Containers/ai.syui.rse/Data/Library/Application Support/Epic/Airse/Saved/Screenshots/Mac

vmc

character blink

src/main.rs

use rosc::{OscMessage, OscPacket, OscType};
use std::net::UdpSocket;
use std::thread::sleep;
use std::time::Duration;

fn main() {
    let socket = UdpSocket::bind("0.0.0.0:0").unwrap();
    let target = "127.0.0.1:39539";

    println!("Sending blink to {}", target);

    for i in 0..=5 {
        let value = if i <= 2 { i as f32 / 2.0 } else { (5 - i) as f32 / 2.0 };
        println!("  Blink: {}", value);
        blink(&socket, target, value);
        sleep(Duration::from_millis(30));
    }

    println!("Done");
}

fn blink(socket: &UdpSocket, target: &str, value: f32) {
    for name in &["Blink", "Blink_L", "Blink_R"] {
        let msg = OscMessage {
            addr: "/VMC/Ext/Blend/Val".to_string(),
            args: vec![OscType::String(name.to_string()), OscType::Float(value)],
        };
        let buf = rosc::encoder::encode(&OscPacket::Message(msg)).unwrap();
        socket.send_to(&buf, target).unwrap();
    }

    // Apply
    let apply = OscMessage {
        addr: "/VMC/Ext/Blend/Apply".to_string(),
        args: vec![],
    };
    let buf = rosc::encoder::encode(&OscPacket::Message(apply)).unwrap();
    socket.send_to(&buf, target).unwrap();
}

AI + character + game

Below is an example of VMC/OSC functionality.

chat with your character
they think and move on their own.

character voice

kawaii

elevenlabs.io: https://elevenlabs.io/docs/api-reference/authentication

{
		"elevenlabs": {
			"model": "eleven_turbo_v2.5",
			"voice_id": "lhTvHflPVOqgSWyuWQry"
		}
}

voice to text

Google Cloud Speech-to-Text API: https://docs.cloud.google.com/speech-to-text/docs/quickstarts/transcribe-api

Vmode

link

example

osc

vmc

AI + character + game

character voice

voice to text

AI

project

func

page

[ja]

project

asset

plugin

platform

ref