Pixel Streaming

Pixel Streaming allows a high-fidelity interactive experience to be streamed directly to users' device, regardless of its graphical capabilities.

We provide a browser client for viewing pixel-streamed content from MSquared events:

• NPM package: @msquared/pixel-streaming-client

• Source code: github.com/msquared-io/pixel-streaming-client

Installation

When bundling with a modern build tool, you can install the package directly from npm:

npm install @msquared/pixel-streaming-client

This package provides ESM, CJS and UMD modules.

Alteratively, you can include the following script and use the client directly in the browser via window.MSquaredPixelStreamingClient:

<script src="https://cdn.jsdelivr.net/npm/@msquared/pixel-streaming-client@latest/dist/script/index.js"></script>

Usage

import { StreamingClient, StreamProvider, StreamTarget } from '@msquared/pixel-streaming-client';

// Initialize client
const client = new StreamingClient({
  auth: {
    // See "How to obtain an auth token"
    token: 'your-auth-token',
    organizationId: 'your-org-id'
  },
  logger: console
});

// Listen for state changes
client.addEventListener('streamStateUpdated', (event) => {
  console.log('Stream state:', event.detail.state);
});

// Fetch stream configuration
const config = await client.fetchStreamConfig({
  projectId: 'your-project-id',
  worldId: 'your-world-id',
  forceProvider: StreamProvider.GeforceNow,
});

// Start an embedded stream
await client.start({
    ...config,
  provider: StreamProvider.GeforceNow,
  target: StreamTarget.Embedded,
  container: '#stream-container',
});

Obtaining an Auth Token

To use the Pixel Streaming Client, an MSquared ID token is required.

If you need a temporary or throwaway account, you can generate one programmatically.

This must be done on your web server, as the API key is a secret and should never be exposed in your web client.

Below is a reference script for generating accounts using the MSquared API:

const apiKey = "..." // Speak to MSquared to obtain an API key
const organizationId = "..." // Your organization ID - shown on the MSquared dashboard
const userId = randomUUID() // Must be a unique per session
const displayName = "..." // The display name you want to display for the user's avatar
const yourDomain = "example.com" // Can be whatever value you want

const userResp = await fetch(
  `https://${organizationId}.m2worlds.io/api/accounts/auth/user/`,
  {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-api-key": apiKey,
    },
    body: JSON.stringify({
      email: `${userId}@${yourDomain}`,
      emailVerified: true,
      displayName: displayName,
    }),
  },
)
const user = await userResp.json()

const tokenResp = await fetch(
    `https://${organizationId}.m2worlds.io/api/accounts/token/generate`,
    {
        method: "POST",
        headers: {
            "Content-Type": "application/json",
            "x-api-key": apiKey,
        },
        body: JSON.stringify({
            userId: user.uid,
        }),
    },
)
const token = await tokenResp.json()

const signinResp = await fetch(
    `https://${organizationId}.m2worlds.io/api/accounts/token/signin`,
    {
        method: "POST",
        headers: {
            "Content-Type": "application/json",
            "x-api-key": apiKey,
        },
        body: JSON.stringify({
            token: token.token,
        }),
    },
)
const signin = await signinResp.json()

// Users must have a username set, or they will be unable to enter a world
await fetch(`https://${organizationId}.m2worlds.io/api/profile`, {
    method: "PUT",
    headers: {
        "Content-Type": "application/json",
        Authorization: `Bearer ${authToken}`,
    },
    body: JSON.stringify({
        username: displayName,
    }),
})
    
// The auth token needed to start a streaming session
const authToken = signin.idToken

Granting streaming permissions for your world

You will need to grant permission for users to stream your world.

If you are creating temporary/throwaway users (as described above) you can do the following:

1. Go to https://your-organisation.msquared.io/dashboard/admin/access

2. Create a new role with the following properties:

   1. Group name: Public Access

   2. Members:

      1. Type: Email Domain

      2. Enter the same domain you used to generate temporary accounts

   3. Access type:

      1. Type: World Access

      2. Select all projects you want these users to access

React

The usePixelStreaming hook provides an easy way to integrate pixel streaming in React:

"use client"

import { useRef, useState, useCallback, useEffect } from "react"
import { usePixelStreaming } from "@msquared/pixel-streaming-client"

type Params = {
    organizationId: string
    projectId: string
    worldId: string
    authToken: string
}

export function PixelStreamingClient({ organizationId, projectId, worldId, authToken }: Params) {
    const [started, setStarted] = useState(false)
    const streamContainerRef = useRef<HTMLDivElement | null>(null)
    const { startStreaming, stopStreaming } = usePixelStreaming({
        organizationId,
        projectId,
        worldId,
        authToken,
    })

    const start = useCallback(() => {
        if (started || !streamContainerRef.current) return
        setStarted(true)
        startStreaming({ ref: streamContainerRef.current })
    }, [started, startStreaming])

    useEffect(() => {
        return () => stopStreaming()
    }, [stopStreaming])

    return (
        <div className="h-screen w-full bg-black relative">
            <div className="absolute inset-0" ref={streamContainerRef} />
            {!started && (
                <button onClick={start} className="absolute bottom-8 left-1/2 transform -translate-x-1/2">
                    Start Streaming
                </button>
            )}
        </div>
    )
}

Aspect Ratio Support

Our primary pixel streaming provider, GeForce Now, only supports a limited set of common aspect ratios, and is principally designed to be used fullscreen on the user's device. If you are embedding the Pixel Streaming Client within your own UI, you may experience letterboxing, whereby black bars are present on either the top/bottom, or either side of the streaming video.

To minimise this, consider the aspect ratios that GeForce Now supports (see Supported Aspect Ratios, below) and try to ensure that the streaming element matches one of the given ratios. This may mean, for example, moving navigation menus from the top of the screen, to the side, to more closely match a supported aspect ratio.

Note that the aspect ratio is fixed from the moment the stream begins, and will remain fixed for the entire duration of the experience, regardless of whether the streaming element changes size or orientation (for example, as a result of the user resizing their browser window, or rotating their device.)

Supported Aspect Ratios

GeForce Now supports the following aspect ratios, and will select one that is as near as possible to that of the streaming element in your HTML:

• 16:9

• 16:10

• 4:3

• 21:9

• 19.5:9