Cymbal Insurance Provider - AI-Powered Outbound Calling System

A cloud-native outbound calling application that uses Twilio for telephony and Google Gemini Live API for real-time AI-powered conversations. The system enables automated policy renewal reminders and handles customer queries with natural voice interactions.

🏗️ Architecture Overview

The system consists of three main components:

┌─────────────┐         ┌─────────────┐         ┌─────────────┐
│   React     │  HTTP   │   FastAPI   │  WSS    │   Twilio    │
│  Frontend   │────────▶│   Backend   │◀────────│   Media     │
│             │         │             │         │   Stream    │
└─────────────┘         └──────┬──────┘         └─────────────┘
                               │
                               │ WebSocket
                               ▼
                        ┌─────────────┐
                        │   Gemini    │
                        │  Live API   │
                        │ (Vertex AI) │
                        └─────────────┘

How It Works

Call Initiation: Customer service representative enters a phone number in the React frontend
Twilio Connection: Backend initiates an outbound call via Twilio Voice API
Media Streaming: When customer answers, Twilio establishes a bidirectional WebSocket media stream
AI Processing: Audio from customer is sent to Gemini Live API for real-time processing
Natural Conversation: AI responds with voice, creating a natural conversation flow
Transcription: Full conversation is transcribed and displayed in real-time on the frontend

Audio Flow

Customer ─▶ Twilio (μ-law 8kHz) ─▶ Backend ─▶ Gemini (PCM 16kHz)
                                               │
Customer ◀─ Twilio (μ-law 8kHz) ◀─ Backend ◀──┘ (PCM 24kHz)

✨ Features

Real-time Voice AI: Powered by Google Gemini 2.5 Flash Live with native audio support
Natural Conversations: Supports interruptions, turn-taking, and multilingual interactions
Indian English Support: Optimized for Indian accents and regional languages (Hindi, Tamil, Telugu, etc.)
Live Transcription: Real-time conversation transcript displayed on frontend
WebSocket Streaming: Bidirectional audio streaming with minimal latency
Cloud-Native: Designed for deployment on Google Cloud Run
Scalable: Handles multiple concurrent calls with automatic scaling

📋 Prerequisites

Before you begin, ensure you have:

Python 3.11+ installed
Node.js 18+ and npm
Google Cloud Account with Vertex AI API enabled
Twilio Account with a phone number
uv package manager (pip install uv)

🚀 Quick Start

1. Clone the Repository

git clone git@github.com:kkrishnan90/Gemini-Live-Twilio-.git
cd Gemini-Live-Twilio-

2. Backend Setup

Install Dependencies

cd backend
uv venv
source .venv/bin/activate
uv pip install -r requirements.txt

Configure Environment Variables

Copy the example environment file:

cp .env.example .env

Edit .env with your credentials:

# Application Settings
DEBUG_MODE=false
LOG_LEVEL=INFO

# Server Configuration
BACKEND_HOST=0.0.0.0
BACKEND_PORT=8000
BACKEND_URL=http://localhost:8000

# Twilio Configuration
TWILIO_ACCOUNT_SID=your_twilio_account_sid_here
TWILIO_AUTH_TOKEN=your_twilio_auth_token_here
TWILIO_PHONE_NUMBER=your_twilio_phone_number_here

# Google Cloud / Vertex AI
GOOGLE_CLOUD_PROJECT=your_google_cloud_project_id
GOOGLE_CLOUD_LOCATION=us-central1
GOOGLE_APPLICATION_CREDENTIALS=/path/to/your/service-account-key.json

# Gemini Live API
GEMINI_MODEL=gemini-live-2.5-flash-preview-native-audio-09-2025
GEMINI_TEMPERATURE=0.7
GEMINI_TOP_P=0.95

# Frontend
FRONTEND_URL=http://localhost:3000

# Audio Settings (in Hz)
TWILIO_SAMPLE_RATE=8000
GEMINI_INPUT_SAMPLE_RATE=16000
GEMINI_OUTPUT_SAMPLE_RATE=24000

Run the Backend

source .venv/bin/activate
uv run python app.py

The backend will start on http://localhost:8000

3. Frontend Setup

Install Dependencies

cd frontend
npm install

Configure Environment Variables

Copy the example environment file:

cp .env.example .env

Edit .env:

REACT_APP_BACKEND_URL=http://localhost:8000

Run the Frontend

npm start

The frontend will open in your browser at http://localhost:3000

⚙️ Twilio Configuration

Step 1: Create a Twilio Account

Sign up at twilio.com
Get your Account SID and Auth Token from the dashboard
Purchase a phone number with voice capabilities

Step 2: Configure Webhook URLs

For local development with ngrok:

# Install ngrok
brew install ngrok  # macOS
# or download from https://ngrok.com

# Start ngrok tunnel
ngrok http 8000

# Copy the HTTPS URL (e.g., https://abc123.ngrok.io)

Configure your Twilio phone number:

Go to Twilio Console → Phone Numbers
Click on your phone number
Under Voice Configuration:
- A Call Comes In: https://your-ngrok-url.ngrok.io/voice/twiml (HTTP POST)
- Status Callback URL: https://your-ngrok-url.ngrok.io/voice/status (HTTP POST)
Save

For production (Cloud Run):

Replace ngrok URL with your Cloud Run URL
Example: https://your-service-name.run.app/voice/twiml

Step 3: Test Your Setup

Start the backend: uv run python app.py
Start ngrok: ngrok http 8000
Update Twilio webhook with ngrok URL
Make a test call through the frontend

🔧 Google Cloud Setup

Step 1: Enable Required APIs

gcloud services enable aiplatform.googleapis.com
gcloud services enable cloudresourcemanager.googleapis.com

Step 2: Create Service Account

# Create service account
gcloud iam service-accounts create gemini-calling-sa \
    --display-name="Gemini Calling Service Account"

# Grant Vertex AI User role
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
    --member="serviceAccount:gemini-calling-sa@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/aiplatform.user"

# Create and download key
gcloud iam service-accounts keys create ~/gemini-calling-key.json \
    --iam-account=gemini-calling-sa@YOUR_PROJECT_ID.iam.gserviceaccount.com

Step 3: Set Environment Variable

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/gemini-calling-key.json"

Or add it to your .env file:

GOOGLE_APPLICATION_CREDENTIALS=/path/to/gemini-calling-key.json

📁 Project Structure

Gemini-Live-Twilio-/
├── backend/
│   ├── app.py                      # FastAPI application
│   ├── config.py                   # Configuration management
│   ├── requirements.txt            # Python dependencies
│   ├── .env                        # Environment variables (not committed)
│   ├── .env.example               # Environment template
│   ├── services/
│   │   ├── twilio_service.py      # Twilio integration
│   │   ├── gemini_service.py      # Gemini Live API client
│   │   ├── audio_bridge.py        # Audio format conversion
│   │   └── call_orchestrator.py   # Call flow management
│   └── models/
│       ├── call_session.py        # Session state models
│       └── user_context.py        # User/policy models
├── frontend/
│   ├── src/
│   │   ├── App.js                 # Main React component
│   │   ├── components/
│   │   │   ├── CallInitiator.jsx  # Call initiation UI
│   │   │   └── CallStatus.jsx     # Real-time status display
│   │   └── services/
│   │       └── api.js             # Backend API client
│   ├── package.json
│   └── .env                       # Frontend environment (not committed)
├── deployment/
│   ├── Dockerfile                 # Docker configuration
│   ├── cloudbuild.yaml           # Cloud Build config
│   └── smoke_test.sh             # Health check script
├── .gitignore
├── README.md
└── TECHNICAL_ARCHITECTURE.md

🔑 Environment Variables Reference

Backend Variables

Variable	Description	Required	Default
`DEBUG_MODE`	Enable debug logging	No	`false`
`LOG_LEVEL`	Logging level (INFO, DEBUG, WARNING, ERROR)	No	`INFO`
`BACKEND_HOST`	Server host address	No	`0.0.0.0`
`BACKEND_PORT`	Server port	No	`8000`
`BACKEND_URL`	Public backend URL	Yes	-
`TWILIO_ACCOUNT_SID`	Twilio Account SID	Yes	-
`TWILIO_AUTH_TOKEN`	Twilio Auth Token	Yes	-
`TWILIO_PHONE_NUMBER`	Twilio phone number (E.164 format)	Yes	-
`GOOGLE_CLOUD_PROJECT`	GCP Project ID	Yes	-
`GOOGLE_CLOUD_LOCATION`	GCP Region	No	`us-central1`
`GOOGLE_APPLICATION_CREDENTIALS`	Path to service account key	Yes	-
`GEMINI_MODEL`	Gemini model name	No	`gemini-live-2.5-flash-preview-native-audio-09-2025`
`GEMINI_TEMPERATURE`	Model temperature (0.0-1.0)	No	`0.7`
`GEMINI_TOP_P`	Model top_p (0.0-1.0)	No	`0.95`
`FRONTEND_URL`	Frontend URL for CORS	No	`http://localhost:3000`

Frontend Variables

Variable	Description	Required	Default
`REACT_APP_BACKEND_URL`	Backend API URL	Yes	`http://localhost:8000`

🎯 API Endpoints

Health Check

GET /health

Response:

{
  "status": "healthy",
  "service": "Cymbal Insurance Provider Calling System"
}

Initiate Call

POST /api/calls/initiate
Content-Type: application/json

{
  "to_number": "+919876543210"
}

Response:

{
  "success": true,
  "call_sid": "CAxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "message": "Call initiated successfully"
}

Get Call Session

GET /api/calls/{call_sid}/session

Response:

{
  "call_sid": "CAxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "user_phone": "+919876543210",
  "state": "in_progress",
  "transcript": [
    {
      "speaker": "ai",
      "text": "Hello, this is Lucy from Cymbal Insurance Provider...",
      "timestamp": "2024-11-14T10:30:00Z"
    }
  ],
  "metrics": {
    "audio_packets_sent": 1500,
    "audio_packets_received": 1200,
    "ai_response_count": 5
  }
}

WebSocket Endpoint

WS /ws/media-stream/{call_sid}

Used by Twilio for bidirectional audio streaming.

🧪 Testing

Test Backend Health

curl http://localhost:8000/health

Test Call Initiation

curl -X POST http://localhost:8000/api/calls/initiate \
  -H "Content-Type: application/json" \
  -d '{"to_number": "+919876543210"}'

Run Smoke Tests

cd deployment
chmod +x smoke_test.sh
./smoke_test.sh http://localhost:8000

🚢 Deployment

Deploy to Google Cloud Run

Build and deploy:

gcloud builds submit \
  --config=deployment/cloudbuild.yaml \
  --project=YOUR_PROJECT_ID

Configure secrets in Secret Manager:

# Store Twilio credentials
echo -n "your_account_sid" | gcloud secrets create twilio-account-sid \
  --data-file=- --project=YOUR_PROJECT_ID

echo -n "your_auth_token" | gcloud secrets create twilio-auth-token \
  --data-file=- --project=YOUR_PROJECT_ID

echo -n "+1234567890" | gcloud secrets create twilio-phone-number \
  --data-file=- --project=YOUR_PROJECT_ID

Update Twilio webhook with your Cloud Run URL

See deployment/DEPLOYMENT_SUMMARY.md for detailed deployment instructions.

🔍 Troubleshooting

Backend won't start

Issue: ModuleNotFoundError or import errors

Solution:

cd backend
source .venv/bin/activate
uv pip install -r requirements.txt

Twilio webhook errors

Issue: 404 Not Found on Twilio webhook

Solution:

Verify backend is running
Check ngrok is active and URL is correct
Ensure webhook URL in Twilio console matches ngrok URL
Check backend logs: tail -f backend/logs/app.log

Gemini API authentication errors

Issue: 403 Forbidden or Authentication failed

Solution:

# Verify service account has correct permissions
gcloud projects get-iam-policy YOUR_PROJECT_ID \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:gemini-calling-sa@YOUR_PROJECT_ID.iam.gserviceaccount.com"

# Should show roles/aiplatform.user role

No audio in call

Issue: Call connects but no AI voice

Solution:

Check backend logs for WebSocket connection
Verify Gemini API quotas in GCP Console
Test audio conversion: Check TWILIO_SAMPLE_RATE and GEMINI_INPUT_SAMPLE_RATE settings
Ensure TwiML webhook is returning correct XML

Frontend can't connect to backend

Issue: CORS errors in browser console

Solution:

Verify REACT_APP_BACKEND_URL in frontend .env
Check backend is running on correct port
Ensure CORS is configured in backend/app.py

📊 Monitoring

View Logs (Local)

# Backend logs
tail -f backend/logs/app.log

# Frontend logs
# Check browser console (F12)

View Logs (Cloud Run)

gcloud logging read \
  "resource.type=cloud_run_revision AND resource.labels.service_name=cymbal-insurance-backend" \
  --limit=50 \
  --project=YOUR_PROJECT_ID

Metrics to Monitor

Call Success Rate: Percentage of successful calls
Average Call Duration: Mean duration of completed calls
Audio Latency: Time between user speech and AI response
API Error Rate: Rate of Gemini API errors
WebSocket Connection Drops: Number of unexpected disconnections

🛡️ Security Best Practices

Never commit .env files - They contain sensitive credentials
Use Secret Manager for production deployments
Rotate credentials regularly (Twilio tokens, service account keys)
Enable audit logging in Google Cloud Console
Use HTTPS for all production endpoints
Implement rate limiting to prevent abuse
Validate phone numbers before initiating calls
Sanitize user inputs to prevent injection attacks

📝 Customization

Modify AI System Instructions

Edit backend/services/gemini_service.py:

def _generate_system_instruction(self) -> str:
    instruction = f"""You are Lucy, a professional customer service representative...

    # Customize greeting, tone, and behavior here
    """
    return instruction

Change AI Voice

Edit backend/services/gemini_service.py:

voice_config=types.VoiceConfig(
    prebuilt_voice_config=types.PrebuiltVoiceConfig(
        voice_name="Aoede"  # Options: Aoede, Charon, Fenrir, Kore, Puck
    )
)

Customize Frontend UI

Edit frontend/src/App.js and component files in frontend/src/components/

🤝 Contributing

Contributions are welcome! Please follow these guidelines:

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Google Gemini Live API - Real-time AI conversation capabilities
Twilio - Voice and media streaming infrastructure
FastAPI - Modern Python web framework
React - Frontend user interface

📞 Support

For issues and questions:

Check the Troubleshooting section
Review TECHNICAL_ARCHITECTURE.md for architecture details
Open an issue on GitHub
Contact the development team

🗺️ Roadmap

Add support for multiple languages (Spanish, French, German)
Implement call recording and storage
Add analytics dashboard
Support for inbound calls
Integration with CRM systems
Automated testing suite
Multi-region deployment support

Built with ❤️ for modern customer service experiences

Name		Name	Last commit message	Last commit date
backend		backend
deployment		deployment
frontend		frontend
.gitignore		.gitignore
README.md		README.md
TECHNICAL_ARCHITECTURE.md		TECHNICAL_ARCHITECTURE.md

Latest commit

History

Folders and files

Repository files navigation

Cymbal Insurance Provider - AI-Powered Outbound Calling System

🏗️ Architecture Overview

How It Works

Audio Flow

✨ Features

📋 Prerequisites

🚀 Quick Start

1. Clone the Repository

2. Backend Setup

Install Dependencies

Configure Environment Variables

Run the Backend

3. Frontend Setup

Install Dependencies

Configure Environment Variables

Run the Frontend

⚙️ Twilio Configuration

Step 1: Create a Twilio Account

Step 2: Configure Webhook URLs

Step 3: Test Your Setup

🔧 Google Cloud Setup

Step 1: Enable Required APIs

Step 2: Create Service Account

Step 3: Set Environment Variable

📁 Project Structure

🔑 Environment Variables Reference

Backend Variables

Frontend Variables

🎯 API Endpoints

Health Check

Initiate Call

Get Call Session

WebSocket Endpoint

🧪 Testing

Test Backend Health

Test Call Initiation

Run Smoke Tests

🚢 Deployment

Deploy to Google Cloud Run

🔍 Troubleshooting

Backend won't start

Twilio webhook errors

Gemini API authentication errors

No audio in call

Frontend can't connect to backend

📊 Monitoring

View Logs (Local)

View Logs (Cloud Run)

Metrics to Monitor

🛡️ Security Best Practices

📝 Customization

Modify AI System Instructions

Change AI Voice

Customize Frontend UI

🤝 Contributing

📄 License

🙏 Acknowledgments

📞 Support

🗺️ Roadmap

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages