Streaming API

The Streaming API allows you to receive real-time responses from Oliver, delivering message chunks as they're generated rather than waiting for the complete response. This creates a more interactive and engaging user experience.

How Streaming Works

The Streaming API uses Server-Sent Events (SSE) to establish a unidirectional connection from the server to the client. This allows the server to push real-time updates to the client as soon as they're available.

Here's the general flow:

1. The client sends a message to the streaming endpoint
2. The server establishes an SSE connection and begins processing the message
3. As the AI generates content, the server sends chunks of the response to the client
4. The client receives and displays these chunks incrementally
5. When the response is complete, the server sends a completion event

Streaming Message Endpoint

Send a message and receive the response as a stream of events.

POST https://api.oliverchat.com/v1/chats/{chat_id}/messages/stream

{
  "content": "What investment strategies would you recommend for a moderate risk tolerance?",
  "attachments": []  // Optional, for file uploads
}

event: start
data: {"message_id":"msg_123456789","created_at":"2025-03-11T10:05:00Z"}

event: chunk
data: {"content":"For a moderate risk tolerance, "}

event: chunk
data: {"content":"I would recommend a balanced portfolio approach. "}

event: chunk
data: {"content":"This typically includes a mix of "}

event: chunk
data: {"content":"stocks, bonds, and possibly alternative investments."}

event: citation
data: {"citation_id":"citation_123456789","source_type":"rag_document","title":"Investment Strategy Guide 2025","url":"https://ragcontainer.oliverchat.com/docs/investment-strategy-2025.pdf"}

event: chunk
data: {"content":"\n\nA common allocation would be:"}

event: chunk
data: {"content":"\n\n1. 60% in equities (stocks)"}

event: chunk
data: {"content":"\n2. 30% in fixed income (bonds)"}

event: chunk
data: {"content":"\n3. 10% in alternatives or cash"}

event: end
data: {"message_id":"msg_123456789","complete":true}

Stream Event Types

The Streaming API uses different event types to communicate various kinds of information:

Start Event

Sent when the stream begins. Contains the message ID and creation timestamp.

event: start
data: {
  "message_id": "msg_123456789",
  "created_at": "2025-03-11T10:05:00Z"
}

Chunk Event

Contains a portion of the response content. Clients should concatenate these chunks to form the complete response.

event: chunk
data: {
  "content": "This is a portion of the response."
}

Citation Event

Sent when the AI cites a source. This allows clients to display citation information in real-time alongside the response.

event: citation
data: {
  "citation_id": "citation_123456789",
  "source_type": "web_page",
  "title": "Market Update: Q1 2025 Financial Outlook",
  "url": "https://www.example.com/financial-news/article"
}

Error Event

Sent if an error occurs during processing. The stream will typically end after an error event.

event: error
data: {
  "code": "processing_error",
  "message": "An error occurred while generating the response"
}

End Event

Sent when the stream is complete. Contains the message ID and completion status.

event: end
data: {
  "message_id": "msg_123456789",
  "complete": true
}

Client-Side Implementation

Here's a simple JavaScript implementation for consuming the Streaming API:

function streamChatMessage(chatId, message) {
  const outputElement = document.getElementById('response-container');
  let fullResponse = '';
  
  // Clear previous content
  outputElement.innerHTML = '';
  
  // Create a typing indicator
  const typingIndicator = document.createElement('div');
  typingIndicator.className = 'typing-indicator';
  typingIndicator.textContent = 'Oliver is thinking...';
  outputElement.appendChild(typingIndicator);
  
  // Prepare the request
  const requestOptions = {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': 'Bearer YOUR_API_KEY'
    },
    body: JSON.stringify({
      content: message,
      attachments: []
    })
  };
  
  // Create EventSource for SSE
  const fetchSSE = async () => {
    try {
      const response = await fetch(`https://api.oliverchat.com/v1/chats/${chatId}/messages/stream`, requestOptions);
      
      // Check for errors
      if (!response.ok) {
        const errorData = await response.json();
        throw new Error(errorData.error?.message || 'Unknown error');
      }
      
      // Process the stream
      const reader = response.body.getReader();
      const decoder = new TextDecoder();
      
      // Remove typing indicator
      outputElement.removeChild(typingIndicator);
      
      // Create response container
      const responseElement = document.createElement('div');
      responseElement.className = 'response-content';
      outputElement.appendChild(responseElement);
      
      // Create citations container
      const citationsElement = document.createElement('div');
      citationsElement.className = 'citations-container';
      citationsElement.style.display = 'none';
      outputElement.appendChild(citationsElement);
      
      // Process the stream
      while (true) {
        const { done, value } = await reader.read();
        
        if (done) {
          break;
        }
        
        // Convert Uint8Array to string
        const chunk = decoder.decode(value);
        
        // Process SSE format (data: {...}\n\n)
        const events = chunk.split('\n\n');
        
        for (const eventText of events) {
          if (!eventText.trim()) continue;
          
          // Parse event type and data
          const eventLines = eventText.split('\n');
          let eventType = '';
          let eventData = '';
          
          for (const line of eventLines) {
            if (line.startsWith('event:')) {
              eventType = line.substring(7).trim();
            } else if (line.startsWith('data:')) {
              eventData = line.substring(5).trim();
            }
          }
          
          if (eventType && eventData) {
            const data = JSON.parse(eventData);
            
            // Handle different event types
            switch (eventType) {
              case 'start':
                console.log('Stream started:', data);
                break;
                
              case 'chunk':
                fullResponse += data.content;
                responseElement.innerHTML = formatMarkdown(fullResponse);
                break;
                
              case 'citation':
                // Show citations container
                citationsElement.style.display = 'block';
                
                // Add citation
                const citationItem = document.createElement('div');
                citationItem.className = 'citation-item';
                citationItem.innerHTML = `
                  
                    ${data.title}
                  
                  ${data.source_type}
                `;
                citationsElement.appendChild(citationItem);
                break;
                
              case 'error':
                console.error('Stream error:', data);
                const errorElement = document.createElement('div');
                errorElement.className = 'error-message';
                errorElement.textContent = `Error: ${data.message}`;
                outputElement.appendChild(errorElement);
                break;
                
              case 'end':
                console.log('Stream ended:', data);
                break;
            }
          }
        }
      }
    } catch (error) {
      console.error('Error:', error);
      outputElement.innerHTML = `
Error: ${error.message}
`;
    }
  };
  
  fetchSSE();
}

// Helper function to format markdown text
function formatMarkdown(text) {
  // Simple markdown formatting (you may want to use a library like marked.js for more comprehensive formatting)
  return text
    .replace(/\*\*(.*?)\*\*/g, '$1')
    .replace(/\*(.*?)\*/g, '$1')
    .replace(/\n\n/g, '

')
    .replace(/\n/g, '
');
}

Browser Compatibility

Server-Sent Events are supported in all modern browsers, but there are some limitations:

• Internet Explorer does not support SSE natively (requires a polyfill)
• Some older browsers may have limited or no support for the Fetch API with streams
• For maximum compatibility, consider using a library like event-source-polyfill

Error Handling

In addition to the standard error responses, the Streaming API may emit error events during the stream:

event: error
data: {
  "code": "stream_timeout",
  "message": "The stream timed out due to inactivity"
}

Common error codes specific to streaming:

• stream_timeout - The stream timed out (typically after 60 seconds of inactivity)
• processing_error - An error occurred while generating the response
• connection_closed - The client closed the connection

Rate Limiting

Streaming requests count toward your API rate limits, but with specific considerations:

• Each streaming request counts as one API call toward your rate limit
• There is a separate concurrent streaming limit (default: 5 concurrent streams per API key)
• Streams have a maximum duration of 5 minutes before being automatically closed