Chat Tools Development

This guide provides instructions for adding Chat Tools to the Developer Mode of Lens Studio AI. Chat Tools are plugins that define tool extensions that allow AIs to interact with Lens Studio's functionality. AIs can chain multiple chat tools together to respond to the user, passing data between each tool to help themselves complete the task.

Search Chat Tool in the Asset Library to find examples of this type of plugin.

Architecture Overview

Chat Tools follow a consistent architecture:

• JSON Schema: Defines the tool's metadata, parameters, and validation rules
• Code Implementation: Contains the actual tool logic extending ChatTools.ChatTool class of the Editor API.

Lens Studio uses JavaScript for its code development. However, since TypeScript can compile into JavaScript, you can also use TypeScript. See an example .tsconfig below. You will usually use TypeScript when you’re making a more complex plugin to help you keep track of your code.

File Structure

One plugin can contain multiple Chat Tools.

Single Tool Structure

Each individual Chat Tool follows a standardized directory structure, where you have a folder for the tool, and a schema and implementation within it.

My-Chat-Tools/
├── Your-New-Tool/
│   ├── yourNewTool.json      # Schema definition
│   └── yourNewTool.js        # TypeScript implementation
│   └── module.json         # Describes the permission and entry point of the plugin

Also note that, like all plugins, you will have a module.json which describes the entry point of the plugin. In the example above, it will point to ./yourNewTool.js.

Plugin with Multiple Tools Structure

To have multiple tools in one plugin, you can make several of these folder structures.

Chat-Tools/
├── Your-Plugin-Name/
│   ├── index.js              # Plugin entry file (exports all tools)
│   ├── module.json         # Describes the permission and entry point of the plugin
│   ├── Tool-One/
│   │   ├── toolOne.json      # Schema definition
│   │   └── toolOne.ts        # TypeScript implementation
│   ├── Tool-Two/
│   │   ├── toolTwo.json      # Schema definition
│   │   └── toolTwo.ts        # TypeScript implementation
│   └── Shared/
│       ├── types.ts          # Shared types and interfaces
│       └── utils.ts          # Shared utility functions
└── ...

In addition, rather than having the entry point be a single tool, it can point to a JS file which just exports the rest of the tool.

export * from './Tool-One/toolOne.js';
export * from './Tool-Two/toolTwo.js';

Optionally, as in the example above, you might want to create a Shared folder, containing utilities and types that might be used across the tools.

Creating a New Tool

With your structure set up, we can start adding the relevant information we need to each file.

Create JSON Schema

Create yourNewTool.json with the following structure:

{
  "displayName": "Your Tool Display Name",
  "canBeReferencedInPrompt": true,
  "icon": "$(files)",
  "name": "YourToolCommand",
  "tags": ["tag1", "tag2", "relevant-keywords"],
  "modelDescription": "Description of what this tool does for the AI model to understand.",
  "inputSchema": {
    "type": "object",
    "required": ["requiredParam1", "requiredParam2"],
    "properties": {
      "requiredParam1": {
        "type": "string",
        "modelDescription": "Description of this parameter for the AI model."
      },
      "optionalParam": {
        "type": "string",
        "modelDescription": "Optional parameter description.",
        "default": "defaultValue"
      }
    }
  }
}

Core Schema Properties

• displayName: string, human-readable name shown in UI
• canBeReferencedInPrompt: boolean, required, whether AI can reference this tool
• icon: string, required, VS Code icon identifier (for example, "$(files)")
• name: string, required, unique command identifier
• tags: string[], required, keywords for tool discovery
• modelDescription: string, required, AI-friendly description of functionality
• inputSchema: object, required, JSON Schema for parameters

Input Schema Structure

{
  "inputSchema": {
    "type": "object",
    "required": ["param1", "param2"],
    "properties": {
      "parameterName": {
        "type": "string|number|boolean|integer",
        "modelDescription": "AI-friendly parameter description",
        "default": "optional-default-value",
        "enum": ["option1", "option2"],
        "minimum": 1,
        "maximum": 100
      }
    }
  }
}

Supported Parameter Types

• string: Text values
• number: Floating-point numbers
• integer: Whole numbers
• boolean: true/false values
• enum: Predefined string options

Advanced Schema Examples

You can further expand your schema to give more details to the AI. This is helpful to prevent AI hallucinations and incorrect assumptions.

Complex Parameter with Constraints

For example, even if we told the AI an input is a float, it needs guidance to know whether the float is between 0-1, 0-100, etc.

{
  "steps": {
    "type": "integer",
    "modelDescription": "Number of generation steps. Higher values may produce better quality but take longer. Range: 1-150.",
    "default": 50,
    "minimum": 1,
    "maximum": 150
  }
}

enum Parameter

In some cases, you need to constrain the AI to your specific arguments.

{
  "assetType": {
    "type": "string",
    "modelDescription": "The type of asset to create.",
    "enum": ["RenderTarget", "ObjectPrefab", "Material", "FileTexture"]
  }
}

Code Implementation

With your schema ready, we can now add the code to process inputs and generate outputs when the tool is called.

Basic Template

Create yourNewTool extending from the Editor’s ChatTools API.

import * as ChatTools from 'LensStudio:ChatTool';
import schemaJson from './yourNewTool.json';

export class YourNewTool extends ChatTools.ChatTool {
  static descriptor() {
    return {
      id: schemaJson.name,
      name: schemaJson.displayName,
      description: 'Brief description of what this tool does.',
      schema: schemaJson,
    };
  }

  async execute(parameters: any) {
    const result = new ChatTools.Result();

    try {
      // Extract parameters
      const param1: string = parameters?.data?.param1;
      const param2: string = parameters?.data?.param2;

      // Validate required parameters
      if (!param1 || typeof param1 !== 'string') {
        result.error = "Parameter 'param1' is required and must be a string.";
        return result;
      }

      // [Your tool logic here]
      const outputData = await this.performToolAction(param1, param2);

      // Set success result
      result.data = {
        message: 'Tool executed successfully',
        result: outputData,
      };

      return result;
    } catch (error: any) {
      result.error = `Tool execution failed: ${error?.message ?? error}`;
      return result;
    }
  }

  private async performToolAction(
    param1: string,
    param2?: string
  ): Promise<any> {
    // Implement your tool's core functionality
    return { success: true };
  }
}

In the example above, the core function of the tool is separated out for easier reuse by creating the performToolAction function from the checks and bookkeeping of the tool.

The above code uses TypeScript to help provide clarity. Take a look at the tsconfig below, or delete the types and optional markers (? character) to have it work in native JavaScript.

Installing and Distributing

With your plugins ready, you can load them to Lens Studio AI, by adding them like any other plugins.

Alternatively, you can use the Tools configuration menu, next to the Developer mode button, to + Add New Tool.

Toggling the checkbox in the Tools configuration menu does not restart the plugin, only enables and disables it. You will need to toggle the plugin in the Lens Studio preferences menu to reload the plugin.

If you’re happy with your ChatTool, you can share it to the Asset Library, and collaborate with the community.

Common Patterns

As you develop your tool, you’ll start to notice a pattern that you’ll frequently do. Below are some examples of code that you might find useful.

Common Imports

In addition to importing the tool and its JSON, you’ll likely want to import capabilities from Lens Studio.

// Core Chat Tools
import * as ChatTools from 'LensStudio:ChatTool';
import schemaJson from './yourTool.json';

// Lens Studio APIs
import * as FileSystem from 'LensStudio:FileSystem';
import * as Network from 'LensStudio:Network';
import * as App from 'LensStudio:App';

Parameter Validation

You’ll also want to provide validation before processing the input, since AI can hallucinate.

Make sure these validations are meaningful so that when the AI receives the output, it can try again with the correct input.

String Validation

const name: string = parameters?.data?.name;
if (!name || typeof name !== 'string' || name.trim() === '') {
  result.error = "Invalid or missing 'name' parameter.";
  return result;
}

enum Validation

const assetType: string = parameters?.data?.assetType;
const supportedTypes = ['Material', 'ScreenTexture', 'RenderTarget'];
if (!assetType || !supportedTypes.includes(assetType)) {
  result.error = `Asset type "${assetType}" is not supported. Supported types: ${supportedTypes.join(', ')}`;
  return result;
}

Number Validation

const steps: number = parameters?.data?.steps ?? 50;
if (typeof steps !== 'number' || steps < 1 || steps > 150) {
  result.error = 'Steps must be a number between 1 and 150.';
  return result;
}

Accessing Lens Studio project’s Scene, and Assets.

// Get the model interface
const model = (this as any).pluginSystem.findInterface(
  Editor.Model.IModel
) as Editor.Model.IModel;

// Access managers
const assetManager = model.project.assetManager;
const sceneManager = model.project.sceneManager;

// Work with assets
const asset = assetManager.findAssetByUUID(assetUUID);

Scene Object Operations

// Get scene manager
const scene = getScene();
const sceneObject = scene.findSceneObjectByUUID(objectUUID);

if (!sceneObject) {
  result.error = `Scene object with ID ${objectUUID} not found.`;
  return result;
}

Asset Operations

// Find asset by ID
const asset = getAssetById(assetUUID);
if (!asset) {
  result.error = `Asset with ID ${assetUUID} not found.`;
  return result;
}

// Convert to shared type for response
result.data = {
  asset: getSharedFromAsset(asset),
  message: `Asset "${asset.name}" found`,
};

Network Requests

import * as Network from 'LensStudio:Network';

const response = await Network.post(url, {
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify(requestData),
});

if (!response.ok) {
  result.error = `Network request failed: ${response.statusText}`;
  return result;
}

File Operations

import * as FileSystem from 'LensStudio:FileSystem';

// Create directory
FileSystem.createDir(targetPath, { recursive: true });

// Write file
const content = JSON.stringify(data, null, 2);
FileSystem.writeTextFile(filePath, content);

// Check if file exists
if (!FileSystem.exists(filePath)) {
  result.error = `File does not exist: ${filePath}`;
  return result;
}

Tsconfig for developing with TypeScript

Here's an example tsconfig that takes TypeScript code from the ./src folder and outputs JavaScript in the ./dist folder.

This means that you should have your module.json point to the entry point in the ./dist folder.

{
  "compilerOptions": {
    "target": "ES2019",
    "module": "ES2020",
    "rootDir": "./src",
    "outDir": "./dist",
    "strict": false,
    "lib": ["ES2020"],
    "baseUrl": "./",
    "moduleResolution": "bundler",
    "resolveJsonModule": true,
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "allowJs": true,
    "skipLibCheck": true
  },
  "include": ["../EditorLib.d.ts", "src/**/*"]
}

You will need to grab a TypeScript Definition file from the API documentation to let TypeScript know what APIs are available in the Lens Studio plugin environment, and make sure it's referenced correctly in the include array.

Best Practices

1. Always wrap tool logic in try-catch blocks and provide meaningful error messages:

try {
    // [Your tool logic here]
    result.data = { success: true };
} catch (error: any) {
    result.error = `Specific operation failed: ${error?.message ?? error}`;
}

2. Parameter Validation

Validate all parameters early and provide clear error messages:

if (!required_param || typeof required_param !== 'expected_type') {
    result.error = "Clear description of what's wrong and what's expected.";
    return result;
}

3. Name Sanitization

Sanitize user-provided names for Lens Studio compatibility:

const sanitizedName = (name as string)
  .replace(/[^a-zA-Z0-9 _-]/g, '')
  .substring(0, 255);
if (!sanitizedName) {
  result.error = 'Name contains only invalid characters.';
  return result;
}

4. Path Handling

Use Lens Studio's Path API for file system operations:

let targetFolder = new Editor.Path('');
if (folderPath) {
  const abs = assetManager.assetsDirectory.appended(
    new Editor.Path(folderPath)
  );
  FileSystem.createDir(abs, { recursive: true });
  targetFolder = abs;
}

5. Async Operations

Handle asynchronous operations properly:

async execute(parameters: any) {
    // Use await for async operations
    const networkResult = await Network.get(url);

    // Handle promises appropriately
    return result;
}

Troubleshooting

Common Issues

1. Tool Not Registered
   • Ensure export is added to chatTools-entry.ts
   • Check that class name matches export name
   • Verify JSON schema name matches tool ID
2. Schema Validation Errors
   • Ensure JSON schema is valid JSON
   • Check that required properties are properly defined
   • Verify parameter types match expected values
3. Runtime Errors
   • Check parameter extraction and validation
   • Ensure proper error handling in try-catch blocks
   • Verify Lens Studio API usage
4. Test Failures
   • Ensure test command ID matches schema name
   • Check that test parameters match schema requirements
   • Verify test assertions match expected behavior

Debug Tips

// Add logging for debugging
console.log('🔍 Tool parameters:', parameters);
console.log('📋 Extracted values:', { param1, param2 });

// Validate intermediate steps
console.log('✅ Validation passed for:', parameterName);

Conclusion

Following this guide ensures your Chat Tools integrate seamlessly with the Developer Mode of Lens Studio AI.

Remember to:

• Follow naming conventions consistently
• Provide comprehensive parameter validation
• Include proper error handling
• Document your tool's functionality

For questions or issues, refer to existing tools in the public plugins repository.