Deep dive into the technology behind automated funscript generation. Learn about optical flow, motion tracking, and ML pipelines that power AI-based haptic scripting.
Automated funscript generation uses computer vision and machine learning to convert video content into synchronized haptic device instructions. This article explains the underlying technology: how optical flow detects motion, how ML pipelines classify and quantify movement, and how the raw data gets transformed into a smooth, device-ready funscript.
Manual funscript creation requires watching a video frame by frame and placing keyframes that describe device movement at each moment. For high-quality scripts, scripters typically place keyframes at every significant change in motion direction or speed.
A one-minute video might require 120-240 keyframes for smooth playback. At a rate of roughly 2 keyframes per minute of work (including reviewing, adjusting, and testing), a single minute of scripted content takes 60-120 minutes of labor. This ratio makes manual scripting economically unviable at scale and limits the available library of scripted content.
The community has long needed a way to automate this process. Early attempts used audio waveform analysis, but haptic movement rarely correlates with audio. The breakthrough came from applying computer vision techniques that actually analyze what is happening visually in the video.
Optical flow is a computer vision technique that estimates the motion of pixels between consecutive frames. By analyzing how brightness patterns shift from one frame to the next, optical flow produces a vector field that describes the direction and magnitude of movement at every point in the image.
HaptiQ uses dense optical flow (specifically the Farneback algorithm via OpenCV) to compute motion vectors for every pixel in the frame. This produces a detailed motion map that captures all movement in the scene.
The Farneback method approximates each image neighborhood as a polynomial. By comparing polynomial coefficients between consecutive frames, the algorithm estimates displacement at each pixel. The result is two channels of data: horizontal (dx) and vertical (dy) displacement.
For funscript generation, the vertical component (dy) is typically the most relevant, as most haptic devices operate on a single linear axis. HaptiQ's pipeline extracts the dominant vertical motion vector within the region of interest and converts its magnitude to a position value.
Sparse methods like Lucas-Kanade track specific feature points rather than every pixel. While faster, sparse tracking can lose track of features during occlusions or rapid movement -- common in haptic content. Dense optical flow provides more robust motion estimation at the cost of computation time, which cloud-based processing makes acceptable.
Raw optical flow captures all motion in the frame: camera shake, background movement, multiple subjects, lighting changes. For funscript generation, only a specific subset of this motion is relevant.
HaptiQ's guided tracking system addresses this by letting users define a region of interest (ROI). The ROI tells the pipeline exactly where to focus motion analysis, dramatically improving accuracy.
In the guided tracking interface, you draw a rectangle over the area of the video that should drive haptic output. The pipeline then only analyzes optical flow vectors within this region, filtering out irrelevant motion from the rest of the frame.
For VR content with predictable framing, HaptiQ can automatically detect the optimal tracking region using YOLO-based object detection. The detection model identifies relevant subjects and automatically sets the ROI, requiring zero manual input.
Once the pipeline has isolated relevant motion vectors within the ROI, it needs to convert them into funscript position values (0-100). This conversion is not trivial and involves several steps:
Within the ROI, thousands of individual optical flow vectors are aggregated into a single representative value. HaptiQ uses a weighted average that gives more importance to high-magnitude vectors (significant motion) while filtering out noise (small random fluctuations).
Optical flow gives velocity, but funscripts need position. The pipeline integrates velocity over time to produce position values. A constant upward motion translates to a rising position value; a reversal in direction creates a peak in the funscript.
Raw integrated values can have arbitrary range. The pipeline normalizes these to the 0-100 range that funscript devices expect, preserving the relative differences between peaks and valleys while filling the full range of device motion.
Rather than creating a keyframe at every sampled frame (which would produce unnecessarily large files), the pipeline uses peak and valley detection to place keyframes at motion extremes. The RDP (Ramer-Douglas-Peucker) simplification algorithm further reduces keyframe count while preserving the essential shape of the motion curve.
The raw funscript from the generation stage is functional but rough. Post-processing transforms it into a polished, device-ready script. HaptiQ Pro includes 14 transformation plugins for this purpose:
Two smoothing approaches are available: moving average (fast, simple) and Savitzky-Golay (preserves peaks while reducing noise). The Savitzky-Golay filter fits polynomial curves to local neighborhoods of keyframes, producing smoother output that still feels responsive.
The anti-jerk plugin detects unrealistic acceleration in the script -- moments where the device would need to change direction instantly. It inserts transitional keyframes that smooth out these sharp reversals, making the output physically plausible for the device.
Different devices have different maximum speeds. The Handy, for example, has a lower maximum stroke speed than the Kiiroo Keon. The speed limiter plugin caps the rate of position change to a value that the target device can actually achieve, preventing the device from falling behind the script.
The amplify plugin scales position values around the midpoint. Increasing amplitude makes strokes longer and more intense; decreasing it creates gentler, subtler movements. This lets users tune the script's intensity to their preference without re-generating.
Plugins execute in sequence, with each plugin receiving the output of the previous one. HaptiQ's plugin chain editor lets you arrange, configure, and preview the effect of each plugin before committing changes. You can save plugin chains as presets for reuse across projects.
Standard funscripts control a single axis of motion (typically linear stroke). Multi-axis devices like the OSR2+ and SR6 support up to 8 axes of motion including rotation, twist, and vibration.
HaptiQ can generate multi-axis T-code output by analyzing different motion components from the optical flow data. Horizontal motion drives rotation axes while vertical motion drives the primary linear axis. Secondary axes can be derived algorithmically from the primary motion pattern.
After generation and post-processing, HaptiQ's quality analyzer scores the script on a 0-100 scale across several dimensions:
For users with large video libraries, HaptiQ's batch processing system processes multiple videos concurrently. Each video runs through the full 5-stage pipeline independently, with progress tracked via WebSocket for real-time monitoring.
Batch processing uses asyncio-based concurrency with configurable semaphore limits to balance throughput against server resources. Pro users get unlimited batch jobs; free tier users have 10 per month.
Accuracy depends heavily on the source material and ROI configuration. In our testing:
For most users, generating with AI and then spending 5-10 minutes on manual refinement produces results that are indistinguishable from fully manual scripts, at a fraction of the time cost.
Video content processed by HaptiQ is encrypted at rest, transmitted over TLS, and never shared with third parties. Videos are stored only for the duration needed to process them and can be deleted at any time from your project dashboard. See our privacy policy for complete details.
Upload a video and see HaptiQ's 5-stage pipeline in action. Free accounts include 5 video uploads.
HaptiQ uses a 5-stage ML pipeline built on OpenCV dense optical flow. Stage 1 extracts 5–10 frames per second from the source video. Stage 2 runs the Farneback optical-flow algorithm to compute pixel-level motion vectors between consecutive frames. Stage 3 scopes the analysis to a region of interest, either manually drawn in the guided tracker or auto-detected via YOLO on VR footage. Stage 4 aggregates vectors, integrates velocity over time, normalizes to the 0–100 funscript range, and places keyframes at motion extremes using RDP simplification. Stage 5 applies post-processing plugins — Savitzky-Golay smoothing, anti-jerk, speed-limiter — to produce a device-ready .funscript. The entire pipeline streams progress over WebSocket so you watch each stage live.
Optical flow estimates how pixels move between consecutive video frames by analyzing brightness-pattern shifts. It produces a vector field describing direction and magnitude at every point. HaptiQ uses dense optical flow (Farneback via OpenCV) instead of sparse feature tracking like Lucas-Kanade because sparse methods lose track during occlusions or rapid movement — common in haptic content. Dense flow computes motion at every pixel, giving a detailed map that holds up through difficult scenes. For funscript generation, the vertical component (dy) drives the primary linear axis since most devices operate on a single up/down stroke. Audio-waveform analysis was tried in earlier tools but abandoned because haptic motion rarely correlates with audio.
Accuracy depends heavily on source footage. Well-framed, stable camera content reaches 90–95 percent accuracy versus a skilled manual scripter. VR with auto-ROI detection scores 85–92 percent. Handheld or shaky footage drops to 70–80 percent and benefits from manual region-of-interest selection. Complex multi-subject scenes run 60–75 percent. In practice, generating with AI and spending 5–10 minutes on targeted manual refinement in the HaptiQ timeline editor produces results indistinguishable from fully hand-scripted content, at roughly one-tenth the time cost. The built-in quality analyzer scores each script 0–100 and flags specific weak sections — dead zones, sync gaps, unrealistic speeds — so you know exactly where to polish.
HaptiQ Pro includes 14 transformation plugins that run after generation. Two smoothing options are available: a fast moving average and Savitzky-Golay, which fits polynomial curves to local neighborhoods and preserves peaks while reducing noise. The anti-jerk plugin detects physically impossible direction reversals and inserts transitional keyframes. The speed-limiter caps rate of position change to device-specific maxima — 400 units/second for the Kiiroo Keon, lower for The Handy — preventing devices from falling behind the script. Amplify scales positions around the midpoint for longer or gentler strokes. RDP simplification reduces keyframe count while preserving motion shape. Plugins chain in any order, and you can save configurations as presets for reuse across projects.
Create a free HaptiQ account to generate a funscript from your own video.