Troubleshooting
Solve common Brokle issues - connection errors, missing traces, SDK configuration problems, and performance optimization tips
Troubleshooting
Solutions for common issues when using Brokle.
SDK Issues
Traces not appearing in dashboard
Symptoms: SDK runs without errors, but traces don't appear.
Solutions:
-
Check API key configuration
import os print(os.getenv("BROKLE_API_KEY")) # Should show your key -
Ensure flush is called
# Always flush before exit brokle.flush() # Or use context manager with brokle: # Your code here pass # Auto-flushes on exit -
Check for errors
brokle = Brokle(debug=True) # Enable debug logging -
Verify network connectivity
curl https://api.brokle.dev/health
Import errors
Error: ModuleNotFoundError: No module named 'brokle'
Solutions:
# Ensure correct Python environment
which python
pip show brokle
# Reinstall
pip uninstall brokle
pip install brokle
# Check virtual environment
source venv/bin/activate # or your venv pathAsync context issues
Error: RuntimeError: no running event loop
Solution: Use AsyncBrokle for async code:
from brokle import AsyncBrokle
brokle = AsyncBrokle()
async def main():
async with brokle.start_as_current_span("operation"):
await some_async_function()Memory issues with large traces
Symptoms: High memory usage with large batches
Solutions:
# Reduce batch size
brokle = Brokle(
flush_at=20, # Smaller batch size
flush_interval=2.0 # More frequent flushes
)
# Limit trace size
with brokle.start_as_current_span("operation") as span:
# Truncate large outputs
span.update(output=large_output[:10000])Self-Hosting Issues
Server won't start
Check required environment variables:
# These are required for server mode
echo $JWT_SECRET # Must be set
echo $AI_KEY_ENCRYPTION_KEY # Must be set
echo $DATABASE_URL # Must be valid
echo $CLICKHOUSE_URL # Must be valid
echo $REDIS_URL # Must be validCheck logs:
# Docker
docker logs brokle-server
# Docker Compose
docker compose logs serverDatabase connection errors
PostgreSQL issues:
# Test connection
psql $DATABASE_URL -c "SELECT 1"
# Common fixes:
# - Check if PostgreSQL is running
# - Verify credentials
# - Check network/firewall
# - Ensure database existsClickHouse issues:
# Test connection
curl "http://localhost:8123/?query=SELECT%201"
# Check ClickHouse is running
docker ps | grep clickhouseRedis issues:
# Test connection
redis-cli -u $REDIS_URL ping
# Should return: PONGMigration errors
Error: migration failed: dirty database
# Check migration status
./brokle migrate status
# Force reset (CAUTION: for dev only)
./brokle migrate drop
./brokle migrate upError: relation already exists
# The migration may have partially run
# Check the migration version
./brokle migrate status
# Force the version if needed
./brokle migrate force <version>High memory usage
ClickHouse memory:
# Limit ClickHouse memory in docker-compose.yml
clickhouse:
deploy:
resources:
limits:
memory: 4GServer memory:
# Check what's using memory
docker stats
# Reduce worker concurrency
WORKER_CONCURRENCY=5
WORKER_BATCH_SIZE=50Slow queries
Check slow query logs:
-- PostgreSQL
SELECT * FROM pg_stat_statements
ORDER BY total_time DESC
LIMIT 10;
-- ClickHouse
SELECT * FROM system.query_log
WHERE type = 'QueryFinish'
ORDER BY query_duration_ms DESC
LIMIT 10;Common optimizations:
- Add appropriate indexes
- Reduce time range in queries
- Enable query caching
- Consider partitioning
API Issues
401 Unauthorized
Causes:
- Invalid or expired API key
- Wrong authentication header
Solutions:
# Check API key format
# Should be: bk_<40 characters>
# Correct header format
curl -H "Authorization: Bearer bk_..." \
https://api.brokle.dev/v1/traces
# Or use X-API-Key header
curl -H "X-API-Key: bk_..." \
https://api.brokle.dev/v1/traces429 Too Many Requests
Cause: Rate limit exceeded
Solutions:
# Implement exponential backoff
import time
def with_retry(func, max_retries=3):
for i in range(max_retries):
try:
return func()
except RateLimitError:
time.sleep(2 ** i)
raise Exception("Max retries exceeded")
# Or reduce request rate
brokle = Brokle(
flush_at=50, # Smaller batches
flush_interval=5.0 # Less frequent
)500 Internal Server Error
For self-hosted:
# Check server logs
docker logs brokle-server 2>&1 | tail -100
# Common causes:
# - Database connection issues
# - Out of memory
# - Disk space fullFor Brokle Cloud:
- Check status.brokle.dev
- Report issues to support@brokle.dev
Dashboard Issues
Can't login
Clear browser data:
- Clear cookies for the domain
- Clear local storage
- Try incognito mode
Check CORS (self-hosted):
# Ensure correct CORS settings
CORS_ALLOWED_ORIGINS=https://your-dashboard-url.comSlow dashboard
Common causes:
- Large date range selected
- Too many filters active
- Network latency
Solutions:
- Reduce date range
- Use pagination
- Check network speed
- Verify server resources
Charts not loading
Check browser console:
- Open Developer Tools (F12)
- Check Console tab for errors
- Check Network tab for failed requests
Common fixes:
- Clear browser cache
- Disable browser extensions
- Try different browser
Performance Issues
High latency in traces
Check SDK configuration:
# Use async for better performance
brokle = Brokle(
flush_at=100, # Batch more traces
flush_interval=5.0 # Less frequent flushes
)Network issues:
# Test latency to Brokle API
ping api.brokle.dev
# Use self-hosted for lower latency
# if you're in a region far from Brokle CloudWorker falling behind
Symptoms: Growing queue size, delayed traces
Solutions:
# Scale workers
docker compose up -d --scale worker=3
# Or increase concurrency
WORKER_CONCURRENCY=20
WORKER_BATCH_SIZE=200Getting Help
If you can't resolve your issue:
- Search existing issues: GitHub Issues
- Ask the community: Discord or Discussions
- Report a bug: Open a new issue
When reporting issues, include: Brokle version, SDK version, error messages, and steps to reproduce.