diff --git a/.agents/skills/remotion-best-practices/SKILL.md b/.agents/skills/remotion-best-practices/SKILL.md new file mode 100644 index 0000000..1e47cf0 --- /dev/null +++ b/.agents/skills/remotion-best-practices/SKILL.md @@ -0,0 +1,59 @@ +--- +name: remotion-best-practices +description: Router for all Remotion skills +version: 4.0.520 +--- + +## Preserve user changes + +Users may make edits in the code outside of the conversation. + +If you detect a surprising change made in the meanwhile, don't overwrite it, assume it was intentional or ask for confirmation. + +## Creating a video + +If the user asks to make, create, or build a new video or composition, load [Create a new Remotion video](./remotion-create/REFERENCE.md), whether or not a Remotion project already exists. + +## New project setup + +If no Remotion project currently exists, load [Create a new Remotion project](./remotion-create/REFERENCE.md) + +## React Markup Best Practices + +If you are writing Remotion React Markup, load [Remotion Markup Best Practices](./remotion-markup/REFERENCE.md) + +## Maps + +For static maps, animated routes and markers, geographic explainers, Mapbox, MapLibre, MapTiler, GeoJSON, or 3D geographic flyovers, load [Remotion Maps](./remotion-maps/REFERENCE.md). + +## Multimedia + +For achieving multimedia tasks in the browser, such as trimming, cropping videos, or getting metadata from them, load [Remotion Multimedia](./remotion-multimedia/REFERENCE.md) + +## Improving Interactivity + +By structuring the Remotion markup well, we can allow users to interactively change things in the Studio and write back to code. If relevant: [Interactivity Best Practices](./remotion-interactivity/REFERENCE.md) + +## Rendering + +For advanced rendering beyond simple `npx remotion render`, see: [Rendering Best Practices](./remotion-render/REFERENCE.md) + +## Opening Remotion Studio + +To launch a project in Remotion Studio, open its exact local URL, or configure Studio CLI flags, load [Remotion Studio](./remotion-studio/REFERENCE.md). + +## Captions + +When working with Captions, load [Remotion Captions](./remotion-captions/REFERENCE.md). + +## Creating a SaaS, automation or application + +Use the [Remotion SaaS skill](./remotion-saas/REFERENCE.md) for knowledge about Remotion-powered SaaS apps, such as ``, rendering on Lambda, Vercel, Cloudflare, via Express.js, client-side rendering, or for finding the right SaaS template. + +## Looking up Remotion APIs and documentation + +To find and read current Remotion documentation, load [Remotion Docs](./remotion-docs/REFERENCE.md). + +## Upgrading + +To upgrade Remotion, related packages, compatible Mediabunny packages, and installed Remotion Agent Skills, load [Remotion Upgrade](./remotion-upgrade/REFERENCE.md). diff --git a/.agents/skills/remotion-best-practices/agents/openai.yaml b/.agents/skills/remotion-best-practices/agents/openai.yaml new file mode 100644 index 0000000..2dabb40 --- /dev/null +++ b/.agents/skills/remotion-best-practices/agents/openai.yaml @@ -0,0 +1,7 @@ +interface: + display_name: '/remotion-best-practices' + short_description: 'Route Remotion tasks to the right workflow' + icon_small: './assets/remotion-icon.png' + icon_large: './assets/remotion-icon.png' + brand_color: '#0B84F3' + default_prompt: '$remotion-best-practices: Make a promo video for my product' diff --git a/.agents/skills/remotion-best-practices/assets/remotion-icon.png b/.agents/skills/remotion-best-practices/assets/remotion-icon.png new file mode 100644 index 0000000..c17ab6e Binary files /dev/null and b/.agents/skills/remotion-best-practices/assets/remotion-icon.png differ diff --git a/.agents/skills/remotion-best-practices/remotion-captions/REFERENCE.md b/.agents/skills/remotion-best-practices/remotion-captions/REFERENCE.md new file mode 100644 index 0000000..2e08a35 --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-captions/REFERENCE.md @@ -0,0 +1,36 @@ +--- +name: remotion-captions +description: Transcribing, displaying and animating captions +version: 4.0.520 +--- + +All captions must be processed in JSON. The captions must use the [`Caption`](https://www.remotion.dev/docs/captions/caption.md) type which is the following: + +```ts +import type { Caption } from "@remotion/captions"; +``` + +This is the definition: + +```ts +type Caption = { + text: string; + startMs: number; + endMs: number; + timestampMs: number | null; + confidence: number | null; + pageBreakAfter?: boolean; +}; +``` + +## Generating captions + +To transcribe video and audio files to generate captions, load the [transcribe-captions.md](transcribe-captions.md) file for more instructions. + +## Displaying captions + +To display captions in your video, load the [display-captions.md](display-captions.md) file for more instructions. + +## Importing captions + +To import captions from a .srt file, load the [import-srt-captions.md](import-srt-captions.md) file for more instructions. diff --git a/.agents/skills/remotion-best-practices/remotion-captions/agents/openai.yaml b/.agents/skills/remotion-best-practices/remotion-captions/agents/openai.yaml new file mode 100644 index 0000000..1d482bb --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-captions/agents/openai.yaml @@ -0,0 +1,7 @@ +interface: + display_name: '/remotion-captions' + short_description: 'Transcribe, display, and animate captions' + icon_small: './assets/remotion-icon.png' + icon_large: './assets/remotion-icon.png' + brand_color: '#0B84F3' + default_prompt: '$remotion-captions: Transcribe my video and display captions' diff --git a/.agents/skills/remotion-best-practices/remotion-captions/assets/remotion-icon.png b/.agents/skills/remotion-best-practices/remotion-captions/assets/remotion-icon.png new file mode 100644 index 0000000..c17ab6e Binary files /dev/null and b/.agents/skills/remotion-best-practices/remotion-captions/assets/remotion-icon.png differ diff --git a/.agents/skills/remotion-best-practices/remotion-captions/display-captions.md b/.agents/skills/remotion-best-practices/remotion-captions/display-captions.md new file mode 100644 index 0000000..c720d0c --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-captions/display-captions.md @@ -0,0 +1,190 @@ +--- +name: display-captions +description: Displaying captions in Remotion with TikTok-style pages and word highlighting +metadata: + tags: captions, subtitles, display, tiktok, highlight +--- + +# Displaying captions in Remotion + +This guide explains how to display captions in Remotion, assuming you already have captions in the [`Caption`](https://www.remotion.dev/docs/captions/caption) format. + +## Prerequisites + +Read [Transcribing audio](transcribe-captions.md) for how to generate captions. + +First, the [`@remotion/captions`](https://www.remotion.dev/docs/captions) package needs to be installed. +If it is not installed, use the following command: + +```bash +npx remotion add @remotion/captions +``` + +## Fetching captions + +First, fetch your captions JSON file. Use [`useDelayRender()`](https://www.remotion.dev/docs/use-delay-render) to hold the render until the captions are loaded: + +```tsx +import { useState, useEffect, useCallback } from "react"; +import { AbsoluteFill, staticFile, useDelayRender } from "remotion"; +import type { Caption } from "@remotion/captions"; + +export const MyComponent: React.FC = () => { + const [captions, setCaptions] = useState(null); + const { delayRender, continueRender, cancelRender } = useDelayRender(); + const [handle] = useState(() => delayRender()); + + const fetchCaptions = useCallback(async () => { + try { + // Assuming captions.json is in the public/ folder. + const response = await fetch(staticFile("captions123.json")); + const data = await response.json(); + setCaptions(data); + continueRender(handle); + } catch (e) { + cancelRender(e); + } + }, [continueRender, cancelRender, handle]); + + useEffect(() => { + fetchCaptions(); + }, [fetchCaptions]); + + if (!captions) { + return null; + } + + return ( + + {/* Render captions here */} + + ); +}; +``` + +## Creating pages + +Use `createTikTokStyleCaptions()` to group captions into pages. The `combineTokensWithinMilliseconds` option controls how many words appear at once: + +```tsx +import { useMemo } from "react"; +import { createTikTokStyleCaptions } from "@remotion/captions"; +import type { Caption } from "@remotion/captions"; + +// How often captions should switch (in milliseconds) +// Higher values = more words per page +// Lower values = fewer words (more word-by-word) +const SWITCH_CAPTIONS_EVERY_MS = 1200; + +const { pages } = useMemo(() => { + return createTikTokStyleCaptions({ + captions, + combineTokensWithinMilliseconds: SWITCH_CAPTIONS_EVERY_MS, + }); +}, [captions]); +``` + +If a caption has `pageBreakAfter: true`, the current page ends after that caption and the next caption starts a new page. + +## Rendering with Sequences + +Map over the pages and render each one in a ``. Calculate the start frame and duration from the page timing: + +```tsx +import { Sequence, useVideoConfig, AbsoluteFill } from "remotion"; +import type { TikTokPage } from "@remotion/captions"; + +const CaptionedContent: React.FC = () => { + const { fps } = useVideoConfig(); + + return ( + + {pages.map((page, index) => { + const nextPage = pages[index + 1] ?? null; + const startFrame = (page.startMs / 1000) * fps; + const endFrame = Math.min( + nextPage ? (nextPage.startMs / 1000) * fps : Infinity, + startFrame + (SWITCH_CAPTIONS_EVERY_MS / 1000) * fps, + ); + const durationInFrames = endFrame - startFrame; + + if (durationInFrames <= 0) { + return null; + } + + return ( + + + + ); + })} + + ); +}; +``` + +## White-space preservation + +The captions are whitespace sensitive. You should include spaces in the `text` field before each word. Use `whiteSpace: "pre"` to preserve the whitespace in the captions. + +## Separate component for captions + +Put captioning logic in a separate component. +Make a new file for it. + +## Word highlighting + +A caption page contains `tokens` which you can use to highlight the currently spoken word: + +```tsx +import { AbsoluteFill, useCurrentFrame, useVideoConfig } from "remotion"; +import type { TikTokPage } from "@remotion/captions"; + +const HIGHLIGHT_COLOR = "#39E508"; + +const CaptionPage: React.FC<{ page: TikTokPage }> = ({ page }) => { + const frame = useCurrentFrame(); + const { fps } = useVideoConfig(); + + // Current time relative to the start of the sequence + const currentTimeMs = (frame / fps) * 1000; + // Convert to absolute time by adding the page start + const absoluteTimeMs = page.startMs + currentTimeMs; + + return ( + +
+ {page.tokens.map((token, tokenIndex) => { + const isActive = + token.fromMs <= absoluteTimeMs && token.toMs > absoluteTimeMs; + + return ( + + {token.text} + + ); + })} +
+
+ ); +}; +``` + +## Display captions alongside video content + +By default, put the captions alongside the video content, so the captions are in sync. +For each video, make a new captions JSON file. + +```tsx + + +``` diff --git a/.agents/skills/remotion-best-practices/remotion-captions/import-srt-captions.md b/.agents/skills/remotion-best-practices/remotion-captions/import-srt-captions.md new file mode 100644 index 0000000..5fdf83a --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-captions/import-srt-captions.md @@ -0,0 +1,73 @@ +--- +name: import-srt-captions +description: Importing .srt subtitle files into Remotion using @remotion/captions +metadata: + tags: captions, subtitles, srt, import, parse +--- + +# Importing .srt subtitles into Remotion + +If you have an existing `.srt` subtitle file, you can import it into Remotion using `parseSrt()` from `@remotion/captions`. + +If you don't have a .srt file, read [Transcribing audio](transcribe-captions.md) for how to generate captions instead. + +## Prerequisites + +First, the @remotion/captions package needs to be installed. +If it is not installed, use the following command: + +```bash +npx remotion add @remotion/captions # If project uses npm +bunx remotion add @remotion/captions # If project uses bun +yarn remotion add @remotion/captions # If project uses yarn +pnpm exec remotion add @remotion/captions # If project uses pnpm +``` + +## Reading an .srt file + +Use `staticFile()` to reference an `.srt` file in your `public` folder, then fetch and parse it: + +```tsx +import { useState, useEffect, useCallback } from "react"; +import { AbsoluteFill, staticFile, useDelayRender } from "remotion"; +import { parseSrt } from "@remotion/captions"; +import type { Caption } from "@remotion/captions"; + +export const MyComponent: React.FC = () => { + const [captions, setCaptions] = useState(null); + const { delayRender, continueRender, cancelRender } = useDelayRender(); + const [handle] = useState(() => delayRender()); + + const fetchCaptions = useCallback(async () => { + try { + const response = await fetch(staticFile("subtitles.srt")); + const text = await response.text(); + const { captions: parsed } = parseSrt({ input: text }); + setCaptions(parsed); + continueRender(handle); + } catch (e) { + cancelRender(e); + } + }, [continueRender, cancelRender, handle]); + + useEffect(() => { + fetchCaptions(); + }, [fetchCaptions]); + + if (!captions) { + return null; + } + + return ( + + {/* Use captions here */} + + ); +}; +``` + +Remote URLs are also supported - you can `fetch()` a remote file via URL instead of using `staticFile()`. + +## Using imported captions + +Once parsed, the captions are in the `Caption` format and can be used with all `@remotion/captions` utilities. diff --git a/.agents/skills/remotion-best-practices/remotion-captions/transcribe-captions.md b/.agents/skills/remotion-best-practices/remotion-captions/transcribe-captions.md new file mode 100644 index 0000000..ba649d5 --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-captions/transcribe-captions.md @@ -0,0 +1,70 @@ +--- +name: transcribe-captions +description: Transcribing audio to generate captions in Remotion +metadata: + tags: captions, transcribe, whisper, audio, speech-to-text +--- + +# Transcribing audio + +To transcribe audio to generate captions in Remotion, you can use the [`transcribe()`](https://www.remotion.dev/docs/install-whisper-cpp/transcribe) function from the [`@remotion/install-whisper-cpp`](https://www.remotion.dev/docs/install-whisper-cpp) package. + +## Prerequisites + +First, the @remotion/install-whisper-cpp package needs to be installed. +If it is not installed, use the following command: + +```bash +npx remotion add @remotion/install-whisper-cpp +``` + +## Transcribing + +Make a Node.js script to download Whisper.cpp and a model, and transcribe the audio. + +```ts +import path from "path"; +import { + downloadWhisperModel, + installWhisperCpp, + transcribe, + toCaptions, +} from "@remotion/install-whisper-cpp"; +import fs from "fs"; + +const to = path.join(process.cwd(), "whisper.cpp"); + +await installWhisperCpp({ + to, + version: "1.5.5", +}); + +await downloadWhisperModel({ + model: "medium.en", + folder: to, +}); + +// Convert the audio to a 16KHz wav file first if needed: +// import {execSync} from 'child_process'; +// execSync('ffmpeg -i /path/to/audio.mp4 -ar 16000 /path/to/audio.wav -y'); + +const whisperCppOutput = await transcribe({ + model: "medium.en", + whisperPath: to, + whisperCppVersion: "1.5.5", + inputPath: "/path/to/audio123.wav", + tokenLevelTimestamps: true, +}); + +// Optional: Apply our recommended postprocessing +const { captions } = toCaptions({ + whisperCppOutput, +}); + +// Write it to the public/ folder so it can be fetched from Remotion +fs.writeFileSync("captions123.json", JSON.stringify(captions, null, 2)); +``` + +Transcribe each clip individually and create multiple JSON files. + +See [Displaying captions](display-captions.md) for how to display the captions in Remotion. diff --git a/.agents/skills/remotion-best-practices/remotion-create/REFERENCE.md b/.agents/skills/remotion-best-practices/remotion-create/REFERENCE.md new file mode 100644 index 0000000..6d15c51 --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-create/REFERENCE.md @@ -0,0 +1,68 @@ +--- +name: remotion-create +description: Create a new Remotion video +version: 4.0.520 +--- + +These are instructions for making a new Remotion project and composition. +If this is not the next task, see [Remotion Best Practices](../SKILL.md) + +## Scaffold a project + +If a project already exists, skip this. +Ensure Node.js and Git is installed, and the current folder is appropriate for starting a new project. + +Scaffold one using: + +```bash +npx create-video@latest --yes --blank --no-tailwind my-video +cd my-video +npm i +``` + +Replace `my-video` with a suitable project name. + +## Designing a video + +Keep the scaffold and add React Markup. +Follow [Remotion React Markup Best Practices](../remotion-markup/REFERENCE.md) and [Video Layout Rules](video-layout.md) for video-first layout and text sizing guidance. + +## Is this a multi-scene video? + +If this is a video with multiple subsequence videos, follow guidance at [Multi-scene videos](../remotion-markup/multi-scene-video.md). + +## Interactivity Best Practices + +By structuring the React Markup following [Remotion Interactivity Best Practices](../remotion-interactivity/REFERENCE.md), you allow the user to make edits in the Studio which write back to code. + +## TailwindCSS + +If Tailwind is requested, see [tailwind.md](tailwind.md) for using TailwindCSS in Remotion. + +## Open the preview + +Start the preview server after building the composition: + +```bash +npx remotion studio --no-open +``` + +This will start a long-running process and print the server URL for the preview. +If the server is already started, it will print the URL. +If an in-harness browser is available, open it there. +You can visit a specific composition by navigating to `/[composition-id]`, for example `http://localhost:3000/MapAnimation`. + +## Render the video + +Only render if the user explicitly asks for it. + +``` +npx remotion render +``` + +For more options, see [Rendering](../remotion-render/REFERENCE.md). + +## Follow-up + +The video creation process has finished. +For follow-up prompts, use [Remotion Best Practices](../SKILL.md) diff --git a/.agents/skills/remotion-best-practices/remotion-create/agents/openai.yaml b/.agents/skills/remotion-best-practices/remotion-create/agents/openai.yaml new file mode 100644 index 0000000..33b709d --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-create/agents/openai.yaml @@ -0,0 +1,7 @@ +interface: + display_name: '/remotion-create' + short_description: 'Create a new Remotion video project' + icon_small: './assets/remotion-icon.png' + icon_large: './assets/remotion-icon.png' + brand_color: '#0B84F3' + default_prompt: '$remotion-create: Create a Remotion video and make a promo for my product' diff --git a/.agents/skills/remotion-best-practices/remotion-create/assets/remotion-icon.png b/.agents/skills/remotion-best-practices/remotion-create/assets/remotion-icon.png new file mode 100644 index 0000000..c17ab6e Binary files /dev/null and b/.agents/skills/remotion-best-practices/remotion-create/assets/remotion-icon.png differ diff --git a/.agents/skills/remotion-best-practices/remotion-create/tailwind.md b/.agents/skills/remotion-best-practices/remotion-create/tailwind.md new file mode 100644 index 0000000..b902131 --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-create/tailwind.md @@ -0,0 +1,11 @@ +--- +name: tailwind +description: Using TailwindCSS in Remotion. +metadata: +--- + +You can and should use TailwindCSS in Remotion, if TailwindCSS is installed in the project. + +Don't use `transition-*` or `animate-*` classes - always animate using the `useCurrentFrame()` hook. + +Tailwind must be installed and enabled first in a Remotion project - see https://www.remotion.dev/docs/tailwind. diff --git a/.agents/skills/remotion-best-practices/remotion-create/video-layout.md b/.agents/skills/remotion-best-practices/remotion-create/video-layout.md new file mode 100644 index 0000000..3e6bc43 --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-create/video-layout.md @@ -0,0 +1,9 @@ +You are designing a video, not a webpage. + +- Decide what the viewer should notice first in each scene. Build the frame around that one thing. +- Keep important content inside a generous safe area. For 1080px-wide videos, keep key text at least 80px from the sides and 100px from the top and bottom. +- Do not add redundant elements. +- For 1080px-wide compositions, use these rough minimums: + - Main headline: 84px + - Important supporting text: 44px +- Scale those values with the composition width. diff --git a/.agents/skills/remotion-best-practices/remotion-docs/REFERENCE.md b/.agents/skills/remotion-best-practices/remotion-docs/REFERENCE.md new file mode 100644 index 0000000..de53bfc --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-docs/REFERENCE.md @@ -0,0 +1,46 @@ +--- +name: remotion-docs +description: Search Remotion documentation +version: 4.0.520 +--- + +This skill teaches you how to discover and read current Remotion documentation. +If this is not relevant, load [Remotion Best Practices](../SKILL.md) instead. + +## Searching the docs + +Use the Algolia search API to find relevant documentation pages: + +``` +POST https://plsduol1ca-dsn.algolia.net/1/indexes/*/queries?x-algolia-api-key=3e42dbd4f895fe93ff5cf40d860c4a85&x-algolia-application-id=PLSDUOL1CA +Content-Type: application/x-www-form-urlencoded + +{ + "requests": [ + { + "query": "", + "indexName": "remotion", + "params": "attributesToRetrieve=[\"hierarchy.lvl0\",\"hierarchy.lvl1\",\"hierarchy.lvl2\",\"url\"]&hitsPerPage=10" + } + ] +} +``` + +Each hit contains a `url` field pointing to the documentation page. + +## Fetching a page as Markdown + +Append `.md` to any Remotion docs URL to retrieve its Markdown source (saves tokens): + +``` +https://www.remotion.dev/docs/use-video-config.md +https://www.remotion.dev/docs/sequence.md +https://www.remotion.dev/docs/lambda/rendermediaonlambda.md +``` + +## Workflow + +1. Search Algolia for the concept or API you need. +2. Pick the most relevant URL(s) from the results. +3. Fetch each URL with the `.md` suffix. +4. Implement using the current documentation rather than memorized API knowledge. diff --git a/.agents/skills/remotion-best-practices/remotion-docs/agents/openai.yaml b/.agents/skills/remotion-best-practices/remotion-docs/agents/openai.yaml new file mode 100644 index 0000000..12190bf --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-docs/agents/openai.yaml @@ -0,0 +1,7 @@ +interface: + display_name: '/remotion-docs' + short_description: 'Search current Remotion documentation' + icon_small: './assets/remotion-icon.png' + icon_large: './assets/remotion-icon.png' + brand_color: '#0B84F3' + default_prompt: '$remotion-docs: What server-side rendering options does Remotion have?' diff --git a/.agents/skills/remotion-best-practices/remotion-docs/assets/remotion-icon.png b/.agents/skills/remotion-best-practices/remotion-docs/assets/remotion-icon.png new file mode 100644 index 0000000..c17ab6e Binary files /dev/null and b/.agents/skills/remotion-best-practices/remotion-docs/assets/remotion-icon.png differ diff --git a/.agents/skills/remotion-best-practices/remotion-interactivity/REFERENCE.md b/.agents/skills/remotion-best-practices/remotion-interactivity/REFERENCE.md new file mode 100644 index 0000000..9b0c15e --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-interactivity/REFERENCE.md @@ -0,0 +1,247 @@ +--- +name: remotion-interactivity +description: Structure Remotion markup for interactivity +version: 4.0.520 +--- + +By writing Remotion markup in a specific way, the Remotion Studio is able to recognize the structure of the code and makes it interactive: + +- Allowing items to be selected by clicking on them +- Allowing drag+drop, resizing and rotation +- Editing the CSS styles +- Making keyframes and easing values editable + +If the markup is too complex for the Studio to make it interactive, then the values become grayed out. + +## Make an HTML element interactive using `Interactive` + +Every HTML and SVG element (except ``, it already is interactive) such as `
` can be turned interactive using `Interactive`: + +```tsx title="Interactive elements" + + Hello + +``` + +This allows styles and keyframes to be set in the Studio. Be sensible, if a component has many elements, the timeline might get messy. + +## Prefer inline text + +If text is fixed and only used once, write it directly inside the interactive element instead of extracting it into a constant. + +```tsx title="Inline text" +// πŸ‘ Fixed copy stays editable + + Remotion Best Practices + +``` + +Use a prop or variable only when the text is dynamic or reused. + +## Give interactive elements a descriptive name + +Add a `name` prop to elements to make them easily identifyable. +Avoid computed names, hardcode them. + +```tsx title="Interactive names" +<> + + Launch day + + +