PSA/docs/architecture/file_storage_system.md

# File Storage System Implementation

This implementation provides a robust, multi-tenant file storage system with support for multiple storage providers and strong security controls.

## Core Features

### Multi-tenant File Storage
- Tenant isolation through RLS policies
- Tenant-specific storage configuration
- Separate storage paths per tenant

### Storage Provider Support
Community Edition:
- Base provider interface
- Extensible provider architecture
- Core file management features

Enterprise Edition:
- S3 provider implementation
- Advanced provider features
- Cloud storage integration

### File Metadata Tracking
- File metadata storage
- Original filename preservation
- MIME type tracking
- File size tracking
- Upload/modification timestamps

### Entity Relationship Management
- File references to business entities
- Many-to-many relationship support
- Reference tracking and cleanup

### Security Features
- Tenant isolation
- MIME type validation
- File size restrictions
- Storage quota management
- Access control through user relationships

### PDF Preview Caching
- Tenant-isolated preview caching
- Automatic cache expiration
- Performance metrics tracking
- Configurable cache settings
- File-based caching system

## Architecture Components

### Database Schema
- storage_providers: Storage provider configuration
- storage_buckets: Storage location configuration
- file_stores: File metadata storage
- file_references: Entity relationships
- provider_events: Provider operation monitoring
- document_content: Large text content storage for documents
- document_block_content: Block-based JSON content storage for documents

### Content Storage
The system provides three approaches for storing document content:
1. File Storage:
   - Binary files (PDFs, images, etc.)
   - Managed through storage providers
   - File metadata tracking
   
2. Text Content Storage:
   - Large text content stored in document_content table
   - 1-to-1 relationship with documents
   - Tenant isolation through RLS
   - Efficient querying and updates
   - Separation from core document metadata

3. Block-Based Content Storage:
   - Structured JSON content stored in document_block_content table
   - 1-to-1 relationship with documents
   - Tenant isolation through RLS
   - Integration with BlockNote editor:
     * Rich text editing capabilities
     * Block-based content structure
     * Standardized JSON format:
       ```typescript
       interface BlockContent {
         blocks: Block[];     // BlockNote's native block format
         version: number;     // Document format version (e.g., 1)
         time: string;       // ISO timestamp of last update
       }
       ```
   - Version-aware content storage:
     * Optional version_id reference to document_versions
     * Allows tracking block content changes across versions
     * Supports reverting to previous versions
     * Maintains version history without duplicating content
     * Enables gradual adoption of versioning
   - Enables future features like collaborative editing
   - UI Components:
     * TextEditor: Core editing component using BlockNote
     * Documents: Container component managing document state and UI
     * Supports manual save with validation
     * Real-time content updates
     * Error handling and validation
     * Drawer-based editing interface

4. Document Versioning:
   - Version tracking through document_versions table
   - Features:
     * Unique version numbers per document
     * Active version flagging
     * Creation metadata (timestamp, user)
     * Tenant isolation through RLS
     * Composite foreign key to documents
   - Integration with block content:
     * Optional version references
     * Flexible content organization
     * Efficient storage without content duplication
   - Benefits:
     * Track document evolution
     * Maintain version history
     * Support content rollback
     * Enable review workflows
     * Facilitate auditing

### Models
- StorageProviderModel: Provider management
- StorageBucketModel: Bucket management
- FileStoreModel: File metadata management
- FileReferenceModel: Entity reference management
- ProviderEventModel: Event tracking

### Storage Providers
Community Edition:
- Base provider interface
- Provider factory
- Error handling and retry logic

Enterprise Edition:
- S3 provider implementation with:
  - Automatic retries
  - Error handling
  - Streaming support
  - Metadata management

### Services
- StorageService: Core business logic
- File operation coordination
- Validation and error handling
- Event tracking

### Server Actions
- File upload handling
- File download processing
- File deletion management
- File listing and filtering
- Validation checks

### UI Components
- FileManager: Main file management component
- Drag & drop support
- Progress tracking
- Error handling
- File listing and management

## Usage

### Setting Up a Storage Provider (Enterprise Edition)
```typescript
const provider = await StorageProviderModel.create({
    provider_type: 's3',
    provider_name: 'Main Storage',
    config: {
        s3: {
            region: 'us-west-2',
            bucket: 'company-files',
            access_key: process.env.AWS_ACCESS_KEY,
            secret_key: process.env.AWS_SECRET_KEY
        }
    }
});
```

### Creating a Storage Bucket
```typescript
const bucket = await StorageBucketModel.create({
    provider_id: provider.provider_id,
    bucket_name: 'Customer Files',
    bucket_path: '/customer-files',
    allowed_mime_types: ['image/jpeg', 'image/png', 'application/pdf'],
    max_file_size: 10 * 1024 * 1024 // 10MB
});
```

### Using the FileManager Component
```typescript
<FileManager
    bucketId="bucket-id"
    entityType="ticket"
    entityId="ticket-id"
    userId="user-id"
    onUploadComplete={(file) => {
        console.log('File uploaded:', file);
    }}
    onDeleteComplete={(fileId) => {
        console.log('File deleted:', fileId);
    }}
/>
```

### Using the Preview Cache System
```typescript
import { CacheFactory } from '../lib/cache/CacheFactory';

// Get a cache instance for a tenant
const cache = CacheFactory.getPreviewCache('tenant-id');

// Check if preview exists
const exists = await cache.exists('file-id');

// Get cached preview path
const previewPath = await cache.get('file-id');

// Cache a new preview
await cache.set('file-id', previewBuffer);

// Get cache metrics
const metrics = cache.getMetrics();
```

## Security Considerations

1. Tenant Isolation
   - RLS policies on all tables
   - Tenant-specific storage paths
   - Access control validation

2. File Validation
   - MIME type checking
   - File size limits
   - Bucket-specific restrictions

3. Access Control
   - User-based permissions
   - Entity-based access control
   - Audit trail through events

4. Storage Security
   - Provider-level security
   - Secure credential management
   - Path sanitization

## Enterprise Edition Features

The S3 storage provider is available in the Enterprise Edition and includes:
1. AWS S3 Integration
   - Direct S3 bucket access
   - AWS credentials management
   - Region configuration

2. Advanced Features
   - Automatic retries
   - Error handling
   - Streaming support
   - Metadata management

3. Performance
   - Optimized uploads
   - Efficient downloads
   - Proper cleanup

## Maintenance

1. Monitoring
   - Provider event tracking
   - Error logging
   - Performance metrics

2. Cleanup
   - Soft deletion support
   - Reference cleanup
   - Storage quota management

3. Backup
   - Database backups
   - File backups through providers
   - Metadata consistency checks

## Future Considerations

### Cache System Enhancements
1. Distributed Caching
   - Redis-based caching implementation
   - Cluster support for high availability
   - Cache synchronization across nodes

2. Cache Management
   - Automated cache cleanup jobs
   - Size-based eviction policies
   - Priority-based caching strategies

3. Performance Optimizations
   - Memory-mapped file support
   - Compression algorithms
   - Streaming preview generation

4. Monitoring and Analytics
   - Real-time cache statistics
   - Performance metrics dashboard
   - Cache efficiency analysis

### Implementation Details

1. Cache Directory Structure
```
/tmp/preview-cache/
├── tenant1/
│   ├── file1.png
│   └── file2.png
├── tenant2/
│   └── file3.png
└── .gitignore
```

2. Environment Configuration
```
PREVIEW_CACHE_DIR=/tmp/preview-cache
PREVIEW_CACHE_MAX_AGE=604800000  # 7 days in milliseconds
PREVIEW_CACHE_MAX_SIZE=5368709120  # 5GB in bytes
```

3. Cache Metrics Collection
- Hit/miss ratios
- Error rates
- Cache size monitoring
- Access patterns analysis

4. Cleanup Strategies
- Time-based expiration
- LRU eviction policy
- Size-based cleanup
- Manual purge capabilities

5. Error Handling
- Disk space monitoring
- Corrupted cache detection
- Automatic recovery
- Fallback mechanisms