nextav/docs/active/media-streaming/VIDEO_FORMAT_COMPATIBILITY_ANALYSIS.md

Video Format Compatibility Analysis: Jellyfin vs ArtPlayer

Executive Summary

This document provides a comprehensive analysis of video format support across Jellyfin's transcoding system, browser HTML5 capabilities, and ArtPlayer compatibility. The research focuses on determining when direct play is possible versus when transcoding is required, with specific attention to .avi, .mkv, and .ts formats.

Key Finding: Transcoding should be the last resort. Most modern video content can be delivered directly to browsers using MP4 or WebM containers with appropriate codecs.

1. Jellyfin's Direct Play Decision Logic

1.1 Core Architecture

Jellyfin uses a DeviceProfile system to determine playback method based on client capabilities:

// From StreamBuilder.cs - Container support check
if (!directPlayProfile.SupportsContainer(container))
{
    directPlayProfileReasons |= TranscodeReason.ContainerNotSupported;
}

// Video codec compatibility
if (!directPlayProfile.SupportsVideoCodec(videoCodec))
{
    directPlayProfileReasons |= TranscodeReason.VideoCodecNotSupported;
}

// Audio codec compatibility  
if (!directPlayProfile.SupportsAudioCodec(audioCodec))
{
    directPlayProfileReasons |= TranscodeReason.AudioCodecNotSupported;
}

1.2 Playback Method Priority

1. DirectPlay: Native file streaming (best performance)
2. DirectStream: Container remuxing only (good performance)
3. Transcode: Full video/audio re-encoding (resource intensive)

1.3 Format-Specific Analysis

MP4 Container

• Chrome Profile: ✅ DirectPlay
  • Video: H.264, VP8, VP9, AV1
  • Audio: AAC, MP3, Opus, FLAC, ALAC, Vorbis
• Result: Almost always direct play for web browsers

MKV Container

• Chrome Profile: ❌ Not in DirectPlayProfiles
• AndroidPixel Profile: ✅ DirectPlay
  • Video: H.264, HEVC, AV1, VP8, VP9
  • Audio: AAC, AC3, DTS, TrueHD, FLAC, etc.
• Result: Container-dependent, usually requires DirectStream to MP4

TS/MPEGTS Container

• Chrome Profile: ❌ Not supported → Transcoding required
• AndroidPixel Profile: ✅ DirectPlay
  • Video: H.264, HEVC, MPEG4
  • Audio: AAC, AC3, EAC3, MP3
• Result: Limited web browser support

AVI Container

• All Tested Profiles: ❌ Not found in any DirectPlayProfile
• Result: Always requires transcoding for web delivery

2. Browser HTML5 Video Format Support

2.1 Native Browser Compatibility Matrix

Container	Video Codec	Audio Codec	Chrome	Firefox	Safari	Edge	Support Level
MP4	H.264	AAC	✅	✅	✅	✅	Universal
MP4	H.265/HEVC	AAC	✅*	✅*	✅	✅*	Platform Dependent
MP4	AV1	AAC	✅	✅	❌	✅	Modern Browsers
WebM	VP8	Vorbis	✅	✅	❌	✅	Good
WebM	VP9	Opus	✅	✅	❌	✅	Good
WebM	AV1	Opus	✅	✅	❌	✅	Emerging
Ogg	Theora	Vorbis	✅	✅	❌	❌	Limited
MKV	Any	Any	❌	❌	❌	❌	None
AVI	Any	Any	❌	❌	❌	❌	None
TS	Any	Any	❌	❌	❌	❌	None

*Depends on hardware/OS support

2.2 Browser Codec Detection

// Runtime capability detection
function checkCodecSupport(container, videoCodec, audioCodec) {
    const video = document.createElement('video');
    const mimeType = `video/${container}; codecs="${videoCodec},${audioCodec}"`;
    
    const support = video.canPlayType(mimeType);
    return {
        probably: support === 'probably',
        maybe: support === 'maybe', 
        supported: support !== ''
    };
}

// Examples
checkCodecSupport('mp4', 'avc1.42E01E', 'mp4a.40.2');     // H.264 + AAC
checkCodecSupport('webm', 'vp9', 'opus');                  // VP9 + Opus
checkCodecSupport('mp4', 'hev1.1.6.L93.B0', 'mp4a.40.2'); // HEVC + AAC

3. ArtPlayer Capabilities Analysis

3.1 Native HTML5 Support

ArtPlayer leverages the browser's native <video> element, inheriting all HTML5 limitations:

✅ Direct Support:

• MP4 with H.264/H.265 + AAC
• WebM with VP8/VP9 + Vorbis/Opus
• Ogg with Theora + Vorbis

❌ Requires External Processing:

• MKV containers
• AVI containers
• TS streams
• Advanced codecs (depending on browser)

3.2 Plugin Extension Capabilities

ArtPlayer supports external libraries for enhanced format support:

// Available plugins from ArtPlayer ecosystem
const supportedPlugins = {
    'hls.js': {
        formats: ['m3u8', 'ts segments'],
        protocols: ['HLS'],
        note: 'HTTP Live Streaming support'
    },
    'dash.js': {
        formats: ['mpd', 'mp4 segments'],
        protocols: ['DASH'],
        note: 'Dynamic Adaptive Streaming'
    },
    'flv.js': {
        formats: ['flv'],
        protocols: ['HTTP-FLV'],
        note: 'Flash Video via MSE'
    },
    'mpegts.js': {
        formats: ['ts', 'mpegts'],
        protocols: ['HTTP'],
        note: 'MPEG-TS streams via MSE'
    }
};

3.3 Format Support Strategy

// Recommended ArtPlayer integration pattern
class SmartArtPlayer {
    private formatSupport = {
        native: ['mp4', 'webm', 'ogg'],
        plugin: ['ts', 'flv', 'm3u8'],
        transcode: ['mkv', 'avi', 'mov']
    };
    
    async initialize(mediaSource: MediaInfo) {
        const playbackStrategy = this.determineStrategy(mediaSource);
        
        switch (playbackStrategy.method) {
            case 'native':
                return this.createNativePlayer(mediaSource);
            case 'plugin':
                return this.createPluginPlayer(mediaSource, playbackStrategy.plugin);
            case 'transcode':
                return this.createTranscodedPlayer(mediaSource);
        }
    }
    
    private determineStrategy(media: MediaInfo): PlaybackStrategy {
        // Priority: Native > Plugin > Transcode
        if (this.canPlayNatively(media)) {
            return { method: 'native' };
        }
        
        if (this.canPlayWithPlugin(media)) {
            return { 
                method: 'plugin', 
                plugin: this.selectOptimalPlugin(media)
            };
        }
        
        return { method: 'transcode' };
    }
}

4. Format-Specific Recommendations

4.1 Container Priority for Web Delivery

Tier 1: Universal Support (No Transcoding)

1. MP4 + H.264 + AAC

   • ✅ Universal browser support
   • ✅ Hardware acceleration
   • ✅ Adaptive streaming compatible
   • ⚠️ Larger file sizes

2. WebM + VP9 + Opus

   • ✅ Excellent compression
   • ✅ Open source/royalty-free
   • ❌ No Safari support
   • ✅ Good for Chrome/Firefox

Tier 2: Limited Support (May Require DirectStream)

3. MP4 + HEVC + AAC

   • ✅ Excellent compression
   • ⚠️ Platform-dependent support
   • ✅ Future-proof for 4K content

4. MKV + H.264 + AAC

   • ❌ No native browser support
   • ✅ Excellent for archival
   • 🔄 Requires container remux to MP4

Tier 3: Transcoding Required

5. TS/MPEGTS

   • ❌ Limited browser support
   • ✅ Good for live streaming
   • 🔄 Usually requires transcoding

6. AVI

   • ❌ No modern browser support
   • ❌ Legacy format
   • 🔄 Always requires transcoding

4.2 Codec Compatibility Guidelines

Video Codecs

✅ H.264 (AVC)     - Universal support, hardware accelerated
✅ VP9             - Good compression, Chrome/Firefox
⚠️ H.265 (HEVC)   - Platform dependent, excellent compression  
⚠️ AV1            - Future codec, limited current support
❌ VP8             - Legacy, use VP9 instead
❌ Theora          - Legacy, limited support

Audio Codecs

✅ AAC             - Universal support, good quality
✅ MP3             - Universal support, larger files
✅ Opus            - Excellent compression, modern browsers
⚠️ Vorbis          - Open source, limited Safari support
❌ AC3/DTS         - Limited browser support
❌ FLAC            - Lossless but large, limited support

5. Implementation Strategy

5.1 Progressive Enhancement Approach

// Smart format selection algorithm
class FormatOptimizer {
    selectOptimalFormat(mediaItem: MediaFile, clientCapabilities: BrowserInfo): PlaybackPlan {
        const plans: PlaybackPlan[] = [];
        
        // Plan 1: Native direct play
        if (this.canDirectPlay(mediaItem, clientCapabilities)) {
            plans.push({
                method: 'direct',
                url: mediaItem.originalUrl,
                container: mediaItem.container,
                bandwidth: mediaItem.bitrate,
                quality: 'original'
            });
        }
        
        // Plan 2: Direct stream (remux only)
        if (this.canDirectStream(mediaItem, clientCapabilities)) {
            plans.push({
                method: 'directstream',
                url: this.buildDirectStreamUrl(mediaItem),
                container: this.getCompatibleContainer(clientCapabilities),
                bandwidth: mediaItem.bitrate,
                quality: 'original'
            });
        }
        
        // Plan 3: Transcoding (last resort)
        plans.push({
            method: 'transcode',
            url: this.buildTranscodeUrl(mediaItem, clientCapabilities),
            container: 'mp4', // Safe fallback
            bandwidth: this.calculateTranscodeBitrate(mediaItem),
            quality: 'transcoded'
        });
        
        // Return best available option
        return plans[0];
    }
    
    private canDirectPlay(media: MediaFile, browser: BrowserInfo): boolean {
        const supportMatrix = {
            'mp4': ['chrome', 'firefox', 'safari', 'edge'],
            'webm': ['chrome', 'firefox', 'edge'],
            'mkv': [], // No native browser support
            'avi': [], // No native browser support
            'ts': []   // No native browser support
        };
        
        return supportMatrix[media.container]?.includes(browser.name) || false;
    }
}

5.2 ArtPlayer Integration Pattern

// Complete ArtPlayer setup with format detection
class OptimalArtPlayer {
    private player: Artplayer | null = null;
    
    async initializePlayer(container: string, mediaSource: MediaSource) {
        const config = await this.buildOptimalConfig(mediaSource);
        
        this.player = new Artplayer({
            container,
            url: config.url,
            type: config.type,
            customType: config.customHandlers,
            plugins: config.plugins,
            // Quality selector for multiple formats
            quality: config.qualities,
            // Subtitle support
            subtitle: config.subtitle
        });
        
        return this.player;
    }
    
    private async buildOptimalConfig(media: MediaSource): Promise<PlayerConfig> {
        const browserSupport = await this.detectBrowserCapabilities();
        const optimalFormat = this.selectFormat(media, browserSupport);
        
        return {
            url: optimalFormat.url,
            type: optimalFormat.requiresPlugin ? optimalFormat.type : undefined,
            customHandlers: this.buildCustomHandlers(optimalFormat),
            plugins: this.getRequiredPlugins(optimalFormat),
            qualities: this.buildQualitySelector(media),
            subtitle: this.buildSubtitleConfig(media)
        };
    }
    
    private buildCustomHandlers(format: OptimalFormat): CustomTypeHandlers {
        const handlers: CustomTypeHandlers = {};
        
        if (format.container === 'ts' && format.requiresPlugin) {
            handlers.mpegts = (video: HTMLVideoElement, url: string) => {
                if (window.mpegts?.isSupported()) {
                    const player = mpegts.createPlayer({
                        type: 'mp2t',
                        url: url
                    });
                    player.attachMediaElement(video);
                    player.load();
                }
            };
        }
        
        if (format.container === 'm3u8') {
            handlers.hls = (video: HTMLVideoElement, url: string) => {
                if (window.Hls?.isSupported()) {
                    const hls = new Hls();
                    hls.loadSource(url);
                    hls.attachMedia(video);
                }
            };
        }
        
        return handlers;
    }
}

6. Performance Considerations

6.1 Resource Usage by Playback Method

Method	CPU Usage	Memory	Network	Latency	Quality
DirectPlay	Minimal	Low	Original	None	Original
DirectStream	Low	Low	Original	Minimal	Original
Transcode	High	High	Variable	High	Configurable

6.2 Bandwidth Optimization

// Adaptive quality selection based on connection
class AdaptiveFormatSelector {
    selectByConnection(mediaItem: MediaFile, connection: ConnectionInfo): FormatOption {
        const connectionSpeeds = {
            '4g': 25000000,      // 25 Mbps
            '3g': 5000000,       // 5 Mbps  
            'wifi': 100000000,   // 100 Mbps
            'ethernet': 1000000000 // 1 Gbps
        };
        
        const availableBandwidth = connectionSpeeds[connection.type] || 5000000;
        
        if (mediaItem.bitrate <= availableBandwidth * 0.8) {
            // Can handle original quality
            return this.getDirectPlayOption(mediaItem);
        } else {
            // Need transcoding for bandwidth
            return this.getTranscodedOption(mediaItem, availableBandwidth * 0.7);
        }
    }
}

7. Best Practices & Recommendations

7.1 Content Strategy

1. For New Content Creation:

   • Primary: MP4 + H.264 + AAC (universal compatibility)
   • Secondary: WebM + VP9 + Opus (size optimization)
   • Avoid: AVI, legacy codecs

2. For Existing Libraries:

   • Keep original files for archival
   • Create web-optimized versions on-demand
   • Use transcoding as fallback only

3. Quality Ladder:

   4K/2160p → H.265 + AAC (if supported) → H.264 + AAC
   1080p    → H.264 + AAC
   720p     → H.264 + AAC  
   480p     → H.264 + AAC (mobile fallback)
   

7.2 Implementation Checklist

• Format Detection: Implement runtime codec capability detection
• Progressive Fallback: Native → Plugin → Transcode
• Quality Selection: Provide multiple quality options
• Performance Monitoring: Track transcoding resource usage
• Caching Strategy: Cache transcoded segments
• Error Handling: Graceful fallback between methods

7.3 Common Pitfalls to Avoid

1. Over-transcoding: Don't transcode when DirectStream suffices
2. Safari Assumptions: Remember Safari doesn't support WebM
3. Plugin Dependencies: Load external libraries only when needed
4. Quality Loss: Prefer container remux over re-encoding
5. Resource Limits: Monitor server CPU/memory during transcoding

8. Testing Matrix

8.1 Browser Compatibility Testing

// Automated format compatibility testing
const testMatrix = {
    browsers: ['chrome', 'firefox', 'safari', 'edge'],
    formats: [
        { container: 'mp4', video: 'h264', audio: 'aac' },
        { container: 'webm', video: 'vp9', audio: 'opus' },
        { container: 'mkv', video: 'h264', audio: 'aac' },
        { container: 'avi', video: 'xvid', audio: 'mp3' }
    ],
    expected: {
        'mp4+h264+aac': { chrome: true, firefox: true, safari: true, edge: true },
        'webm+vp9+opus': { chrome: true, firefox: true, safari: false, edge: true },
        'mkv+h264+aac': { chrome: false, firefox: false, safari: false, edge: false },
        'avi+xvid+mp3': { chrome: false, firefox: false, safari: false, edge: false }
    }
};

9. Future Considerations

9.1 Emerging Technologies

• AV1 Codec: Better compression, growing browser support
• WebCodecs API: Native browser encoding/decoding
• WebAssembly: Client-side media processing
• HTTP/3: Improved streaming performance

9.2 Implementation Roadmap

1. Phase 1: Implement native format detection and fallback
2. Phase 2: Add plugin support for extended formats
3. Phase 3: Optimize transcoding pipeline
4. Phase 4: Implement adaptive streaming

Conclusion

The key to optimal video delivery is progressive enhancement: start with what browsers can handle natively (MP4+H.264+AAC), extend capabilities with plugins (HLS.js, mpegts.js), and use transcoding only as a last resort. ArtPlayer provides excellent flexibility for this approach through its plugin architecture and HTML5 foundation.

Bottom Line: With proper implementation, 80-90% of modern video content can be delivered without transcoding, significantly reducing server load and improving user experience.