Interview preparation · Technical guide

System Design Interview Questions and Answers

System design concepts, trade-offs, and interview prompts. Numbers and named technologies are illustrative; capacity, consistency, and security decisions require explicit requirements.

Examples are independent teaching snippets and may require application types, imports, packages, schema, and configuration. Framework behavior is version-dependent. Corrections address identified issues; the complete source code collection has not been compiled or integration-tested.

1. How would you design a system to handle 1 million concurrent users?

A “million concurrent users” is not enough to design a system. Clarify active versus connected users, request rate, read/write mix, payload size, geographic distribution, latency SLO, data durability, and budget. Then estimate capacity, identify bottlenecks, choose stateless tiers and data partitioning, set overload behavior, and validate with load tests. Concurrency alone does not determine required servers.

2. How do you design a system that can scale from 100 to 10 million users?

Scalability Strategy: - Horizontal Scaling: Add more servers as load increases - Vertical Scaling: Upgrade server resources initially - Microservices: Break monolith into scalable services - Database Sharding: Distribute data across multiple databases - Event-Driven Architecture: Decouple services for better scaling

C# Implementation:

// Auto-scaling Configuration
public class AutoScalingService
{
    private readonly IMetricsCollector _metrics;
    private readonly IScalingManager _scalingManager;

    public async Task MonitorAndScaleAsync()
    {
        var cpuUsage = await _metrics.GetCpuUsageAsync();
        var memoryUsage = await _metrics.GetMemoryUsageAsync();
        var requestCount = await _metrics.GetRequestCountAsync();

        if (ShouldScaleOut(cpuUsage, memoryUsage, requestCount))
        {
            await _scalingManager.ScaleOutAsync();
        }
        else if (ShouldScaleIn(cpuUsage, memoryUsage, requestCount))
        {
            await _scalingManager.ScaleInAsync();
        }
    }

    private bool ShouldScaleOut(double cpu, double memory, int requests)
    {
        return cpu > 70 || memory > 80 || requests > 1000;
    }
}

// Database Sharding Implementation
public class ShardedDatabaseService
{
    private readonly Dictionary<string, IDbConnection> _shards;

    public async Task<T> ExecuteOnShardAsync<T>(string shardKey, Func<IDbConnection, Task<T>> operation)
    {
        var shard = GetShardForKey(shardKey);
        return await operation(shard);
    }

    private IDbConnection GetShardForKey(string key)
    {
        var hash = GetHash(key);
        var shardIndex = hash % _shards.Count;
        return _shards.Values.ElementAt(shardIndex);
    }
}

3. How would you implement horizontal scaling for a database?

Database Horizontal Scaling Strategies: - Read Replicas: Distribute read operations - Sharding: Partition data across multiple databases - Master-Slave Replication: Write to master, read from slaves - Multi-Master: Write to multiple masters

C# Implementation:

// Database Sharding with Consistent Hashing
public class ConsistentHashSharding
{
    private readonly SortedDictionary<int, string> _ring = new SortedDictionary<int, int>();
    private readonly List<string> _nodes = new List<string>();
    private const int VIRTUAL_NODES = 150;

    public void AddNode(string node)
    {
        _nodes.Add(node);
        for (int i = 0; i < VIRTUAL_NODES; i++)
        {
            var hash = GetHash($"{node}:{i}");
            _ring[hash] = _nodes.Count - 1;
        }
    }

    public string GetNode(string key)
    {
        if (_ring.Count == 0) return null;

        var hash = GetHash(key);
        var nodeIndex = _ring.FirstOrDefault(x => x.Key >= hash).Value;

        if (nodeIndex == 0 && _ring.First().Key < hash)
        {
            nodeIndex = _ring.Last().Value;
        }

        return _nodes[nodeIndex];
    }

    private int GetHash(string key)
    {
        using (var md5 = MD5.Create())
        {
            var hash = md5.ComputeHash(Encoding.UTF8.GetBytes(key));
            return BitConverter.ToInt32(hash, 0);
        }
    }
}

// Read Replica Implementation
public class ReadReplicaService
{
    private readonly IDbConnection _master;
    private readonly List<IDbConnection> _replicas;
    private int _currentReplica = 0;

    public async Task<T> ExecuteReadAsync<T>(Func<IDbConnection, Task<T>> operation)
    {
        var replica = GetNextReplica();
        return await operation(replica);
    }

    public async Task<T> ExecuteWriteAsync<T>(Func<IDbConnection, Task<T>> operation)
    {
        return await operation(_master);
    }

    private IDbConnection GetNextReplica()
    {
        var replica = _replicas[_currentReplica];
        _currentReplica = (_currentReplica + 1) % _replicas.Count;
        return replica;
    }
}

4. How do you design a system with 99.99% uptime?

99.99% uptime allows roughly 4 minutes 23 seconds of unavailability in a 30-day month, subject to the service-level definition. Design around measured failure modes: multi-instance deployment, redundancy across fault domains, automated health-based failover, tested rollbacks, dependency timeouts, and operational runbooks. A percentage target without a measurement boundary is incomplete.

5. How would you implement auto-scaling based on load?

Auto-scaling Strategy: - Metrics Collection: CPU, memory, request count, response time - Scaling Policies: Define thresholds for scale-out/in - Predictive Scaling: Use ML to predict load patterns - Scheduled Scaling: Scale based on known patterns

C# Implementation:

// Auto-scaling Manager
public class AutoScalingManager
{
    private readonly IMetricsCollector _metrics;
    private readonly IScalingService _scalingService;
    private readonly ILogger<AutoScalingManager> _logger;

    public async Task MonitorAndScaleAsync()
    {
        var metrics = await _metrics.GetCurrentMetricsAsync();

        if (ShouldScaleOut(metrics))
        {
            await ScaleOutAsync();
        }
        else if (ShouldScaleIn(metrics))
        {
            await ScaleInAsync();
        }
    }

    private bool ShouldScaleOut(SystemMetrics metrics)
    {
        return metrics.CpuUsage > 70 ||
               metrics.MemoryUsage > 80 ||
               metrics.RequestCount > 1000 ||
               metrics.ResponseTime > TimeSpan.FromSeconds(2);
    }

    private async Task ScaleOutAsync()
    {
        _logger.LogInformation("Scaling out due to high load");
        await _scalingService.AddInstanceAsync();
    }
}

// Predictive Scaling with ML
public class PredictiveScalingService
{
    private readonly IMLModel _mlModel;
    private readonly IHistoricalDataService _historicalData;

    public async Task<int> PredictRequiredInstancesAsync(DateTime targetTime)
    {
        var historicalData = await _historicalData.GetHistoricalLoadAsync(targetTime);
        var features = ExtractFeatures(historicalData);

        return await _mlModel.PredictAsync(features);
    }

    private double[] ExtractFeatures(HistoricalLoadData data)
    {
        return new double[]
        {
            data.HourOfDay,
            data.DayOfWeek,
            data.IsHoliday ? 1 : 0,
            data.AverageLoad,
            data.PeakLoad
        };
    }
}

6. How do you design a system to handle traffic spikes?

Traffic Spike Handling: - Rate Limiting: Prevent overwhelming the system - Queue Management: Buffer requests during spikes - Caching: Serve cached responses for repeated requests - Load Shedding: Drop non-critical requests during overload - CDN: Distribute load geographically

C# Implementation:

// Rate Limiting Implementation
public class RateLimiter
{
    private readonly ConcurrentDictionary<string, Queue<DateTime>> _requestHistory = new();
    private readonly int _maxRequests;
    private readonly TimeSpan _window;

    public RateLimiter(int maxRequests, TimeSpan window)
    {
        _maxRequests = maxRequests;
        _window = window;
    }

    public bool IsAllowed(string clientId)
    {
        var now = DateTime.UtcNow;
        var queue = _requestHistory.GetOrAdd(clientId, _ => new Queue<DateTime>());

        lock (queue)
        {
            // Remove old requests
            while (queue.Count > 0 && now - queue.Peek() > _window)
            {
                queue.Dequeue();
            }

            if (queue.Count < _maxRequests)
            {
                queue.Enqueue(now);
                return true;
            }

            return false;
        }
    }
}

// Request Queue with Priority
public class PriorityRequestQueue
{
    private readonly PriorityQueue<Request, int> _queue = new();
    private readonly SemaphoreSlim _semaphore = new SemaphoreSlim(100, 100);

    public async Task<Response> EnqueueAsync(Request request, int priority)
    {
        await _semaphore.WaitAsync();

        try
        {
            _queue.Enqueue(request, priority);
            return await ProcessRequestAsync(request);
        }
        finally
        {
            _semaphore.Release();
        }
    }

    private async Task<Response> ProcessRequestAsync(Request request)
    {
        // Process request based on priority
        return await request.ProcessAsync();
    }
}

// Load Shedding Implementation
public class LoadSheddingMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILoadMonitor _loadMonitor;

    public async Task InvokeAsync(HttpContext context)
    {
        var currentLoad = await _loadMonitor.GetCurrentLoadAsync();

        if (currentLoad > 90) // 90% capacity
        {
            // Drop non-critical requests
            if (!IsCriticalRequest(context.Request))
            {
                context.Response.StatusCode = 503;
                await context.Response.WriteAsync("Service temporarily unavailable");
                return;
            }
        }

        await _next(context);
    }

    private bool IsCriticalRequest(HttpRequest request)
    {
        return request.Path.StartsWithSegments("/api/critical") ||
               request.Method == "GET" && request.Path.StartsWithSegments("/api/health");
    }
}

7. How would you implement load balancing across multiple regions?

Multi-region load balancing needs clear traffic policy, health signals, data locality, failover criteria, DNS/cache behavior, and a data-consistency plan. Active-active improves locality but introduces cross-region write/conflict decisions; active-passive can simplify writes but requires tested promotion and capacity for failover.

8. How do you design a system with sub-second response times?

Performance Optimization: - Caching: Multi-level caching strategy - Database Optimization: Indexing, query optimization - Async Processing: Non-blocking operations - CDN: Serve static content globally - Connection Pooling: Efficient resource management

C# Implementation:

// Multi-Level Caching
public class MultiLevelCache
{
    private readonly IMemoryCache _l1Cache; // In-memory cache
    private readonly IRedisCache _l2Cache;  // Redis cache
    private readonly IDbConnection _database;

    public async Task<T> GetAsync<T>(string key, Func<Task<T>> factory)
    {
        // L1 Cache (Memory)
        if (_l1Cache.TryGetValue(key, out T l1Result))
        {
            return l1Result;
        }

        // L2 Cache (Redis)
        var l2Result = await _l2Cache.GetAsync<T>(key);
        if (l2Result != null)
        {
            _l1Cache.Set(key, l2Result, TimeSpan.FromMinutes(5));
            return l2Result;
        }

        // Database
        var dbResult = await factory();
        if (dbResult != null)
        {
            await _l2Cache.SetAsync(key, dbResult, TimeSpan.FromMinutes(30));
            _l1Cache.Set(key, dbResult, TimeSpan.FromMinutes(5));
        }

        return dbResult;
    }
}

// Optimized Database Queries
public class OptimizedDataService
{
    private readonly IDbConnection _database;
    private readonly IQueryOptimizer _optimizer;

    public async Task<List<User>> GetUsersAsync(UserFilter filter)
    {
        var query = _optimizer.OptimizeQuery(filter);

        using var connection = await _database.OpenConnectionAsync();
        using var command = connection.CreateCommand();
        command.CommandText = query.Sql;
        command.CommandType = CommandType.Text;

        // Add parameters
        foreach (var param in query.Parameters)
        {
            command.Parameters.AddWithValue(param.Key, param.Value);
        }

        var users = new List<User>();
        using var reader = await command.ExecuteReaderAsync();

        while (await reader.ReadAsync())
        {
            users.Add(MapToUser(reader));
        }

        return users;
    }
}

// Async Processing Pipeline
public class AsyncProcessingPipeline
{
    private readonly Channel<ProcessingRequest> _channel;
    private readonly IProcessor _processor;

    public async Task<ProcessingResult> ProcessAsync(ProcessingRequest request)
    {
        var tcs = new TaskCompletionSource<ProcessingResult>();
        request.CompletionSource = tcs;

        await _channel.Writer.WriteAsync(request);
        return await tcs.Task;
    }

    public async Task StartProcessingAsync()
    {
        await foreach (var request in _channel.Reader.ReadAllAsync())
        {
            try
            {
                var result = await _processor.ProcessAsync(request);
                request.CompletionSource.SetResult(result);
            }
            catch (Exception ex)
            {
                request.CompletionSource.SetException(ex);
            }
        }
    }
}

9. How would you implement caching at multiple levels?

Cache layers need an ownership model for freshness, keys, TTLs, invalidation, failure behavior, and stampede control. Cache-aside is common, but a cache miss must not overload the backing store. Never place authorization-sensitive content in a shared cache without partitioning its key by the relevant identity/context.

10. How do you design a system to handle 1TB of data per day?

Big Data Architecture: - Data Ingestion: Stream processing with Kafka/Azure Event Hubs - Data Storage: Distributed storage (HDFS, Azure Data Lake) - Data Processing: Batch and stream processing - Data Analytics: Real-time and batch analytics - Data Archival: Long-term storage with lifecycle management

C# Implementation:

// Data Ingestion Pipeline
public class DataIngestionService
{
    private readonly IEventHubClient _eventHub;
    private readonly IDataValidator _validator;
    private readonly IDataTransformer _transformer;

    public async Task IngestDataAsync(DataRecord record)
    {
        // Validate data
        if (!await _validator.ValidateAsync(record))
        {
            throw new InvalidDataException("Data validation failed");
        }

        // Transform data
        var transformedData = await _transformer.TransformAsync(record);

        // Send to event hub
        var eventData = new EventData(JsonSerializer.SerializeToUtf8Bytes(transformedData));
        await _eventHub.SendAsync(eventData);
    }

    public async Task ProcessDataStreamAsync()
    {
        var consumerGroup = _eventHub.CreateConsumerGroup("data-processors");

        await foreach (var partitionEvent in consumerGroup.ReadEventsAsync())
        {
            await ProcessEventAsync(partitionEvent);
        }
    }

    private async Task ProcessEventAsync(EventData eventData)
    {
        var data = JsonSerializer.Deserialize<DataRecord>(eventData.Body.Span);

        // Process data based on type
        switch (data.Type)
        {
            case DataType.UserActivity:
                await ProcessUserActivityAsync(data);
                break;
            case DataType.Transaction:
                await ProcessTransactionAsync(data);
                break;
            case DataType.Log:
                await ProcessLogAsync(data);
                break;
        }
    }
}

// Batch Processing Service
public class BatchProcessingService
{
    private readonly IDataLakeService _dataLake;
    private readonly IDataProcessor _processor;
    private readonly IAnalyticsService _analytics;

    public async Task ProcessBatchAsync(DateTime date)
    {
        var dataFiles = await _dataLake.GetFilesForDateAsync(date);

        var processingTasks = dataFiles.Select(file => ProcessFileAsync(file));
        await Task.WhenAll(processingTasks);

        // Generate daily analytics
        await GenerateDailyAnalyticsAsync(date);
    }

    private async Task ProcessFileAsync(string filePath)
    {
        using var stream = await _dataLake.OpenFileAsync(filePath);
        using var reader = new StreamReader(stream);

        var batch = new List<DataRecord>();
        string line;

        while ((line = await reader.ReadLineAsync()) != null)
        {
            var record = JsonSerializer.Deserialize<DataRecord>(line);
            batch.Add(record);

            if (batch.Count >= 1000)
            {
                await ProcessBatchAsync(batch);
                batch.Clear();
            }
        }

        // Process remaining records
        if (batch.Count > 0)
        {
            await ProcessBatchAsync(batch);
        }
    }

    private async Task ProcessBatchAsync(List<DataRecord> batch)
    {
        var processedData = await _processor.ProcessBatchAsync(batch);
        await _dataLake.WriteProcessedDataAsync(processedData);
    }
}

// Data Archival Service
public class DataArchivalService
{
    private readonly IDataLakeService _dataLake;
    private readonly IArchiveService _archive;
    private readonly IConfiguration _config;

    public async Task ArchiveOldDataAsync()
    {
        var retentionPolicy = _config.GetSection("DataRetention").Get<RetentionPolicy>();
        var cutoffDate = DateTime.UtcNow.AddDays(-retentionPolicy.DaysToKeep);

        var filesToArchive = await _dataLake.GetFilesOlderThanAsync(cutoffDate);

        foreach (var file in filesToArchive)
        {
            await ArchiveFileAsync(file);
        }
    }

    private async Task ArchiveFileAsync(string filePath)
    {
        // Compress file
        var compressedData = await CompressFileAsync(filePath);

        // Move to archive storage
        await _archive.StoreAsync(filePath, compressedData);

        // Delete from primary storage
        await _dataLake.DeleteFileAsync(filePath);
    }

    private async Task<byte[]> CompressFileAsync(string filePath)
    {
        using var inputStream = await _dataLake.OpenFileAsync(filePath);
        using var outputStream = new MemoryStream();
        using var gzipStream = new GZipStream(outputStream, CompressionMode.Compress);

        await inputStream.CopyToAsync(gzipStream);
        return outputStream.ToArray();
    }
}

// Real-time Analytics Service
public class RealTimeAnalyticsService
{
    private readonly IStreamProcessor _streamProcessor;
    private readonly IAnalyticsEngine _analytics;
    private readonly IAlertService _alertService;

    public async Task ProcessRealTimeDataAsync()
    {
        await foreach (var dataPoint in _streamProcessor.GetDataStreamAsync())
        {
            var result = await _analytics.ProcessDataPointAsync(dataPoint);

            // Check for anomalies
            if (result.IsAnomaly)
            {
                await _alertService.SendAlertAsync(new AnomalyAlert
                {
                    DataPoint = dataPoint,
                    Severity = result.Severity,
                    Timestamp = DateTime.UtcNow
                });
            }

            // Update real-time dashboards
            await UpdateDashboardAsync(result);
        }
    }

    private async Task UpdateDashboardAsync(AnalyticsResult result)
    {
        // Update various metrics and KPIs
        await UpdateMetricsAsync(result.Metrics);
        await UpdateKPIsAsync(result.KPIs);
    }
}

These implementations provide a comprehensive foundation for building scalable, high-performance systems. Each solution includes practical C# code examples that can be adapted and extended based on specific requirements and technology choices.

11. How would you design a database for a social media platform?

Design Approach: - Users & Authentication: Core user profiles with authentication - Content Management: Posts, comments, media storage - Social Graph: Followers, friends, relationships - Activity Feed: Real-time content aggregation - Notifications: Event-driven notification system

Database Schema Design:

// User Management
public class User
{
    public Guid Id { get; set; }
    public string Username { get; set; }
    public string Email { get; set; }
    public string PasswordHash { get; set; }
    public string ProfilePictureUrl { get; set; }
    public string Bio { get; set; }
    public DateTime CreatedAt { get; set; }
    public DateTime LastActiveAt { get; set; }
    public bool IsVerified { get; set; }
    public UserStatus Status { get; set; }
}

// Content Management
public class Post
{
    public Guid Id { get; set; }
    public Guid AuthorId { get; set; }
    public string Content { get; set; }
    public PostType Type { get; set; }
    public string MediaUrls { get; set; } // JSON array
    public int LikeCount { get; set; }
    public int CommentCount { get; set; }
    public int ShareCount { get; set; }
    public DateTime CreatedAt { get; set; }
    public DateTime? UpdatedAt { get; set; }
    public PostVisibility Visibility { get; set; }
    public bool IsDeleted { get; set; }
}

// Social Graph
public class UserRelationship
{
    public Guid Id { get; set; }
    public Guid FollowerId { get; set; }
    public Guid FollowingId { get; set; }
    public RelationshipType Type { get; set; }
    public DateTime CreatedAt { get; set; }
    public bool IsBlocked { get; set; }
}

// Activity Feed
public class ActivityFeed
{
    public Guid Id { get; set; }
    public Guid UserId { get; set; }
    public Guid PostId { get; set; }
    public ActivityType Type { get; set; }
    public DateTime CreatedAt { get; set; }
    public int Priority { get; set; }
}

Key Considerations: - Sharding: Shard by user ID for horizontal scaling - Caching: Redis for hot content and user sessions - Search: Elasticsearch for content discovery - Media Storage: CDN for images/videos - Real-time: WebSockets for live updates

12. How do you implement database sharding?

Sharding Strategies:

public interface IShardingStrategy
{
    int GetShardId(object key);
    string GetConnectionString(int shardId);
}

// Hash-based sharding
public class HashShardingStrategy : IShardingStrategy
{
    private readonly int _shardCount;
    private readonly Dictionary<int, string> _connectionStrings;

    public HashShardingStrategy(int shardCount, Dictionary<int, string> connectionStrings)
    {
        _shardCount = shardCount;
        _connectionStrings = connectionStrings;
    }

    public int GetShardId(object key)
    {
        int hash = key.GetHashCode();
        return Math.Abs(hash) % _shardCount;
    }

    public string GetConnectionString(int shardId)
    {
        return _connectionStrings[shardId];
    }
}

// Range-based sharding
public class RangeShardingStrategy : IShardingStrategy
{
    private readonly Dictionary<Range<long>, int> _ranges;
    private readonly Dictionary<int, string> _connectionStrings;

    public int GetShardId(object key)
    {
        long numericKey = Convert.ToInt64(key);
        foreach (var range in _ranges)
        {
            if (range.Key.Contains(numericKey))
                return range.Value;
        }
        throw new InvalidOperationException("No shard found for key");
    }
}

// Sharding Manager
public class ShardingManager
{
    private readonly IShardingStrategy _strategy;
    private readonly Dictionary<int, IDbConnection> _connections;

    public IDbConnection GetConnection(object key)
    {
        int shardId = _strategy.GetShardId(key);
        return _connections[shardId];
    }

    public async Task<T> ExecuteOnShardAsync<T>(object key, Func<IDbConnection, Task<T>> operation)
    {
        using var connection = GetConnection(key);
        return await operation(connection);
    }
}

Implementation Example:

// Usage
var shardingStrategy = new HashShardingStrategy(4, new Dictionary<int, string>
{
    { 0, "Server=shard1;Database=SocialMedia;..." },
    { 1, "Server=shard2;Database=SocialMedia;..." },
    { 2, "Server=shard3;Database=SocialMedia;..." },
    { 3, "Server=shard4;Database=SocialMedia;..." }
});

var shardingManager = new ShardingManager(shardingStrategy);

// Execute query on appropriate shard
var user = await shardingManager.ExecuteOnShardAsync(userId, async connection =>
{
    // Execute query on the specific shard
    return await GetUserByIdAsync(connection, userId);
});

13. How would you design a multi-tenant database architecture?

Multi-Tenant Patterns:

// Tenant Context
public class TenantContext
{
    public string TenantId { get; set; }
    public string ConnectionString { get; set; }
    public TenantConfig Config { get; set; }
}

// Tenant Configuration
public class TenantConfig
{
    public string DatabaseName { get; set; }
    public string Schema { get; set; }
    public Dictionary<string, string> CustomSettings { get; set; }
}

// Multi-tenant Repository
public class MultiTenantRepository<T> where T : class
{
    private readonly ITenantProvider _tenantProvider;
    private readonly IDbContextFactory _contextFactory;

    public async Task<T> GetByIdAsync(Guid id)
    {
        var tenant = _tenantProvider.GetCurrentTenant();
        using var context = _contextFactory.CreateContext(tenant);

        // Apply tenant filter
        return await context.Set<T>()
            .Where(e => EF.Property<string>(e, "TenantId") == tenant.TenantId)
            .FirstOrDefaultAsync(e => EF.Property<Guid>(e, "Id") == id);
    }
}

// Tenant-aware Entity Base
public abstract class TenantAwareEntity
{
    public string TenantId { get; set; }
    public DateTime CreatedAt { get; set; }
    public DateTime? UpdatedAt { get; set; }
}

// Tenant Provider
public interface ITenantProvider
{
    TenantContext GetCurrentTenant();
}

public class HttpHeaderTenantProvider : ITenantProvider
{
    private readonly IHttpContextAccessor _httpContextAccessor;
    private readonly ITenantService _tenantService;

    public TenantContext GetCurrentTenant()
    {
        var tenantId = _httpContextAccessor.HttpContext?.Request.Headers["X-Tenant-ID"].FirstOrDefault();
        return _tenantService.GetTenant(tenantId);
    }
}

Database Schema Approaches:

-- 1. Shared Database, Shared Schema (with tenant filter)
CREATE TABLE Users (
    Id UNIQUEIDENTIFIER PRIMARY KEY,
    TenantId NVARCHAR(50) NOT NULL,
    Username NVARCHAR(100) NOT NULL,
    Email NVARCHAR(255) NOT NULL,
    INDEX IX_Users_TenantId (TenantId)
);

-- 2. Shared Database, Separate Schemas
CREATE SCHEMA Tenant1;
CREATE SCHEMA Tenant2;

CREATE TABLE Tenant1.Users (
    Id UNIQUEIDENTIFIER PRIMARY KEY,
    Username NVARCHAR(100) NOT NULL,
    Email NVARCHAR(255) NOT NULL
);

-- 3. Separate Databases
-- Each tenant gets their own database

14. How do you implement read replicas for high availability?

Read Replica Implementation:

public interface IDatabaseConnection
{
    string ConnectionString { get; }
    bool IsReadOnly { get; }
    bool IsHealthy { get; }
}

public class DatabaseReplica
{
    public string Name { get; set; }
    public string ConnectionString { get; set; }
    public bool IsReadOnly { get; set; }
    public bool IsHealthy { get; set; }
    public int LagSeconds { get; set; }
    public DateTime LastHealthCheck { get; set; }
}

public class ReadReplicaManager
{
    private readonly List<DatabaseReplica> _replicas;
    private readonly DatabaseReplica _primary;
    private readonly IHealthChecker _healthChecker;

    public async Task<IDbConnection> GetReadConnectionAsync()
    {
        var healthyReplicas = _replicas.Where(r => r.IsHealthy && r.IsReadOnly).ToList();

        if (!healthyReplicas.Any())
            return await GetPrimaryConnectionAsync();

        // Round-robin or least lag selection
        var selectedReplica = healthyReplicas
            .OrderBy(r => r.LagSeconds)
            .First();

        return new SqlConnection(selectedReplica.ConnectionString);
    }

    public async Task<IDbConnection> GetWriteConnectionAsync()
    {
        return await GetPrimaryConnectionAsync();
    }

    private async Task<IDbConnection> GetPrimaryConnectionAsync()
    {
        return new SqlConnection(_primary.ConnectionString);
    }
}

// Repository with read/write separation
public class UserRepository
{
    private readonly ReadReplicaManager _replicaManager;

    public async Task<User> GetByIdAsync(Guid id)
    {
        using var connection = await _replicaManager.GetReadConnectionAsync();
        // Execute read query
        return await GetUserByIdAsync(connection, id);
    }

    public async Task<User> CreateAsync(User user)
    {
        using var connection = await _replicaManager.GetWriteConnectionAsync();
        // Execute write query
        return await CreateUserAsync(connection, user);
    }
}

Health Monitoring:

public class ReplicaHealthChecker : IHealthChecker
{
    public async Task<bool> IsHealthyAsync(DatabaseReplica replica)
    {
        try
        {
            using var connection = new SqlConnection(replica.ConnectionString);
            await connection.OpenAsync();

            // Check replication lag
            var lagQuery = "SELECT DATEDIFF(SECOND, last_commit_time, GETDATE()) FROM sys.dm_database_replica_states";
            var lag = await connection.ExecuteScalarAsync<int>(lagQuery);

            replica.LagSeconds = lag;
            replica.IsHealthy = lag < 30; // 30 seconds threshold
            replica.LastHealthCheck = DateTime.UtcNow;

            return replica.IsHealthy;
        }
        catch
        {
            replica.IsHealthy = false;
            return false;
        }
    }
}

15. How would you design a database migration strategy?

Migration Framework:

public interface IMigration
{
    string Version { get; }
    string Description { get; }
    Task UpAsync(IDbConnection connection);
    Task DownAsync(IDbConnection connection);
}

public class MigrationHistory
{
    public string Version { get; set; }
    public string Description { get; set; }
    public DateTime AppliedAt { get; set; }
    public string AppliedBy { get; set; }
    public TimeSpan Duration { get; set; }
}

public class MigrationManager
{
    private readonly IDbConnection _connection;
    private readonly IEnumerable<IMigration> _migrations;

    public async Task MigrateAsync()
    {
        await EnsureMigrationTableExistsAsync();

        var appliedMigrations = await GetAppliedMigrationsAsync();
        var pendingMigrations = _migrations
            .Where(m => !appliedMigrations.Contains(m.Version))
            .OrderBy(m => m.Version);

        foreach (var migration in pendingMigrations)
        {
            await ApplyMigrationAsync(migration);
        }
    }

    private async Task ApplyMigrationAsync(IMigration migration)
    {
        var stopwatch = Stopwatch.StartNew();

        try
        {
            await migration.UpAsync(_connection);

            await InsertMigrationHistoryAsync(new MigrationHistory
            {
                Version = migration.Version,
                Description = migration.Description,
                AppliedAt = DateTime.UtcNow,
                AppliedBy = Environment.UserName,
                Duration = stopwatch.Elapsed
            });
        }
        catch (Exception ex)
        {
            // Rollback logic
            throw new MigrationException($"Migration {migration.Version} failed", ex);
        }
    }
}

// Example Migration
public class CreateUsersTableMigration : IMigration
{
    public string Version => "2024.01.01.001";
    public string Description => "Create users table";

    public async Task UpAsync(IDbConnection connection)
    {
        var sql = @"
            CREATE TABLE Users (
                Id UNIQUEIDENTIFIER PRIMARY KEY DEFAULT NEWID(),
                Username NVARCHAR(100) NOT NULL UNIQUE,
                Email NVARCHAR(255) NOT NULL UNIQUE,
                CreatedAt DATETIME2 DEFAULT GETUTCDATE()
            );

            CREATE INDEX IX_Users_Email ON Users(Email);
        ";

        await connection.ExecuteAsync(sql);
    }

    public async Task DownAsync(IDbConnection connection)
    {
        await connection.ExecuteAsync("DROP TABLE Users");
    }
}

16. How do you implement database connection pooling?

Connection Pooling Implementation:

public class ConnectionPool
{
    private readonly SemaphoreSlim _semaphore;
    private readonly Queue<IDbConnection> _availableConnections;
    private readonly List<IDbConnection> _activeConnections;
    private readonly string _connectionString;
    private readonly int _maxPoolSize;
    private readonly int _minPoolSize;

    public ConnectionPool(string connectionString, int maxPoolSize = 100, int minPoolSize = 10)
    {
        _connectionString = connectionString;
        _maxPoolSize = maxPoolSize;
        _minPoolSize = minPoolSize;
        _semaphore = new SemaphoreSlim(maxPoolSize);
        _availableConnections = new Queue<IDbConnection>();
        _activeConnections = new List<IDbConnection>();

        InitializePool();
    }

    private void InitializePool()
    {
        for (int i = 0; i < _minPoolSize; i++)
        {
            var connection = CreateConnection();
            _availableConnections.Enqueue(connection);
        }
    }

    public async Task<IDbConnection> GetConnectionAsync()
    {
        await _semaphore.WaitAsync();

        IDbConnection connection = null;
        lock (_availableConnections)
        {
            if (_availableConnections.Count > 0)
            {
                connection = _availableConnections.Dequeue();
            }
        }

        if (connection == null)
        {
            connection = CreateConnection();
        }

        lock (_activeConnections)
        {
            _activeConnections.Add(connection);
        }

        return new PooledConnection(connection, this);
    }

    public void ReturnConnection(IDbConnection connection)
    {
        lock (_activeConnections)
        {
            _activeConnections.Remove(connection);
        }

        if (IsConnectionHealthy(connection))
        {
            lock (_availableConnections)
            {
                if (_availableConnections.Count < _maxPoolSize)
                {
                    _availableConnections.Enqueue(connection);
                }
                else
                {
                    connection.Dispose();
                }
            }
        }
        else
        {
            connection.Dispose();
        }

        _semaphore.Release();
    }

    private IDbConnection CreateConnection()
    {
        var connection = new SqlConnection(_connectionString);
        connection.Open();
        return connection;
    }

    private bool IsConnectionHealthy(IDbConnection connection)
    {
        try
        {
            using var command = connection.CreateCommand();
            command.CommandText = "SELECT 1";
            command.ExecuteScalar();
            return true;
        }
        catch
        {
            return false;
        }
    }
}

public class PooledConnection : IDbConnection
{
    private readonly IDbConnection _innerConnection;
    private readonly ConnectionPool _pool;
    private bool _disposed;

    public PooledConnection(IDbConnection connection, ConnectionPool pool)
    {
        _innerConnection = connection;
        _pool = pool;
    }

    public void Dispose()
    {
        if (!_disposed)
        {
            _pool.ReturnConnection(_innerConnection);
            _disposed = true;
        }
    }

    // Implement other IDbConnection members by delegating to _innerConnection
}

17. How would you design a data warehouse architecture?

Data Warehouse Architecture:

// ETL Pipeline
public interface IExtractService
{
    Task<IEnumerable<RawData>> ExtractAsync(DateTime fromDate, DateTime toDate);
}

public interface ITransformService
{
    Task<IEnumerable<TransformedData>> TransformAsync(IEnumerable<RawData> rawData);
}

public interface ILoadService
{
    Task LoadAsync(IEnumerable<TransformedData> data);
}

public class ETLPipeline
{
    private readonly IExtractService _extractService;
    private readonly ITransformService _transformService;
    private readonly ILoadService _loadService;
    private readonly ILogger<ETLPipeline> _logger;

    public async Task ExecuteAsync(DateTime fromDate, DateTime toDate)
    {
        try
        {
            // Extract
            var rawData = await _extractService.ExtractAsync(fromDate, toDate);

            // Transform
            var transformedData = await _transformService.TransformAsync(rawData);

            // Load
            await _loadService.LoadAsync(transformedData);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "ETL pipeline failed");
            throw;
        }
    }
}

// Data Models
public class SalesFact
{
    public int SalesId { get; set; }
    public int ProductId { get; set; }
    public int CustomerId { get; set; }
    public int StoreId { get; set; }
    public int DateId { get; set; }
    public decimal Amount { get; set; }
    public int Quantity { get; set; }
    public decimal UnitPrice { get; set; }
}

public class ProductDimension
{
    public int ProductId { get; set; }
    public string ProductName { get; set; }
    public string Category { get; set; }
    public string Brand { get; set; }
    public decimal Price { get; set; }
    public DateTime ValidFrom { get; set; }
    public DateTime? ValidTo { get; set; }
}

// Data Mart
public class SalesDataMart
{
    public async Task<IEnumerable<SalesReport>> GetSalesReportAsync(DateTime fromDate, DateTime toDate)
    {
        var sql = @"
            SELECT 
                p.ProductName,
                p.Category,
                SUM(s.Amount) as TotalSales,
                SUM(s.Quantity) as TotalQuantity,
                COUNT(*) as TransactionCount
            FROM SalesFact s
            JOIN ProductDimension p ON s.ProductId = p.ProductId
            JOIN DateDimension d ON s.DateId = d.DateId
            WHERE d.Date BETWEEN @FromDate AND @ToDate
            GROUP BY p.ProductName, p.Category
            ORDER BY TotalSales DESC
        ";

        using var connection = new SqlConnection(_connectionString);
        return await connection.QueryAsync<SalesReport>(sql, new { FromDate = fromDate, ToDate = toDate });
    }
}

18. How do you implement database backup and recovery?

Backup and Recovery System:

public interface IBackupService
{
    Task<BackupResult> CreateBackupAsync(BackupOptions options);
    Task<RestoreResult> RestoreBackupAsync(RestoreOptions options);
}

public class BackupOptions
{
    public string DatabaseName { get; set; }
    public string BackupPath { get; set; }
    public BackupType Type { get; set; }
    public bool Compress { get; set; }
    public bool Encrypt { get; set; }
    public string EncryptionKey { get; set; }
}

public class SqlServerBackupService : IBackupService
{
    private readonly string _connectionString;
    private readonly ILogger<SqlServerBackupService> _logger;

    public async Task<BackupResult> CreateBackupAsync(BackupOptions options)
    {
        var backupFileName = GenerateBackupFileName(options);
        var backupPath = Path.Combine(options.BackupPath, backupFileName);

        var sql = $@"
            BACKUP DATABASE [{options.DatabaseName}] 
            TO DISK = '{backupPath}'
            WITH FORMAT, 
                 COMPRESSION = {(options.Compress ? "ON" : "OFF")},
                 ENCRYPTION = {(options.Encrypt ? "ON" : "OFF")},
                 ALGORITHM = AES_256,
                 SERVER CERTIFICATE = BackupCertificate
        ";

        try
        {
            using var connection = new SqlConnection(_connectionString);
            await connection.ExecuteAsync(sql);

            var fileInfo = new FileInfo(backupPath);
            return new BackupResult
            {
                Success = true,
                BackupPath = backupPath,
                SizeInBytes = fileInfo.Length,
                CreatedAt = DateTime.UtcNow
            };
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Backup failed for database {DatabaseName}", options.DatabaseName);
            return new BackupResult { Success = false, ErrorMessage = ex.Message };
        }
    }

    public async Task<RestoreResult> RestoreBackupAsync(RestoreOptions options)
    {
        var sql = $@"
            RESTORE DATABASE [{options.DatabaseName}] 
            FROM DISK = '{options.BackupPath}'
            WITH REPLACE,
                 RECOVERY
        ";

        try
        {
            using var connection = new SqlConnection(_connectionString);
            await connection.ExecuteAsync(sql);

            return new RestoreResult
            {
                Success = true,
                RestoredAt = DateTime.UtcNow
            };
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Restore failed for database {DatabaseName}", options.DatabaseName);
            return new RestoreResult { Success = false, ErrorMessage = ex.Message };
        }
    }
}

// Automated Backup Scheduler
public class BackupScheduler : BackgroundService
{
    private readonly IBackupService _backupService;
    private readonly IConfiguration _configuration;
    private readonly ILogger<BackupScheduler> _logger;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                var options = new BackupOptions
                {
                    DatabaseName = _configuration["Database:Name"],
                    BackupPath = _configuration["Backup:Path"],
                    Type = BackupType.Full,
                    Compress = true,
                    Encrypt = true
                };

                var result = await _backupService.CreateBackupAsync(options);
                if (result.Success)
                {
                    _logger.LogInformation("Backup completed successfully: {BackupPath}", result.BackupPath);
                }
                else
                {
                    _logger.LogError("Backup failed: {Error}", result.ErrorMessage);
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Backup scheduler error");
            }

            // Wait for next backup (e.g., daily at 2 AM)
            var nextBackup = DateTime.Today.AddDays(1).AddHours(2);
            var delay = nextBackup - DateTime.Now;
            await Task.Delay(delay, stoppingToken);
        }
    }
}

19. How would you design a database for real-time analytics?

Real-time Analytics Architecture:

// Event Store for Real-time Data
public class EventStore
{
    private readonly IDbConnection _connection;
    private readonly IMessageBroker _messageBroker;

    public async Task StoreEventAsync(IAnalyticsEvent @event)
    {
        // Store in database
        var sql = @"
            INSERT INTO AnalyticsEvents (EventId, EventType, UserId, Data, Timestamp)
            VALUES (@EventId, @EventType, @UserId, @Data, @Timestamp)
        ";

        await _connection.ExecuteAsync(sql, @event);

        // Publish to message broker for real-time processing
        await _messageBroker.PublishAsync("analytics.events", @event);
    }
}

// Real-time Aggregator
public class RealTimeAggregator : IHostedService
{
    private readonly IMessageBroker _messageBroker;
    private readonly IAnalyticsService _analyticsService;
    private readonly Dictionary<string, object> _aggregates;

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        await _messageBroker.SubscribeAsync("analytics.events", HandleEventAsync);
    }

    private async Task HandleEventAsync(IAnalyticsEvent @event)
    {
        // Update in-memory aggregates
        UpdateAggregates(@event);

        // Periodically flush to database
        if (ShouldFlush())
        {
            await FlushAggregatesAsync();
        }
    }

    private void UpdateAggregates(IAnalyticsEvent @event)
    {
        var key = $"{@event.EventType}_{@event.UserId}";

        if (!_aggregates.ContainsKey(key))
        {
            _aggregates[key] = new EventAggregate();
        }

        var aggregate = (EventAggregate)_aggregates[key];
        aggregate.Count++;
        aggregate.LastEventTime = @event.Timestamp;
        aggregate.TotalValue += @event.Value;
    }
}

// Time-series Database Design
public class TimeSeriesData
{
    public DateTime Timestamp { get; set; }
    public string Metric { get; set; }
    public string Dimension { get; set; }
    public double Value { get; set; }
    public Dictionary<string, string> Tags { get; set; }
}

public class TimeSeriesRepository
{
    public async Task<IEnumerable<TimeSeriesData>> GetMetricsAsync(
        string metric, 
        DateTime from, 
        DateTime to, 
        string dimension = null)
    {
        var sql = @"
            SELECT Timestamp, Metric, Dimension, Value, Tags
            FROM TimeSeriesData
            WHERE Metric = @Metric 
              AND Timestamp BETWEEN @From AND @To
              AND (@Dimension IS NULL OR Dimension = @Dimension)
            ORDER BY Timestamp
        ";

        using var connection = new SqlConnection(_connectionString);
        return await connection.QueryAsync<TimeSeriesData>(sql, new { metric, from, to, dimension });
    }

    public async Task StoreMetricAsync(TimeSeriesData data)
    {
        var sql = @"
            INSERT INTO TimeSeriesData (Timestamp, Metric, Dimension, Value, Tags)
            VALUES (@Timestamp, @Metric, @Dimension, @Value, @Tags)
        ";

        using var connection = new SqlConnection(_connectionString);
        await connection.ExecuteAsync(sql, data);
    }
}

// Real-time Dashboard
public class RealTimeDashboard
{
    private readonly IAnalyticsService _analyticsService;
    private readonly IHubContext<AnalyticsHub> _hubContext;

    public async Task UpdateDashboardAsync()
    {
        var metrics = await _analyticsService.GetRealTimeMetricsAsync();
        await _hubContext.Clients.All.SendAsync("MetricsUpdated", metrics);
    }
}

20. How do you implement database versioning?

Database Versioning Strategies:

// Schema Versioning
public class SchemaVersion
{
    public int Version { get; set; }
    public string Description { get; set; }
    public DateTime AppliedAt { get; set; }
    public string AppliedBy { get; set; }
    public string Checksum { get; set; }
}

public class SchemaVersionManager
{
    private readonly IDbConnection _connection;

    public async Task<int> GetCurrentVersionAsync()
    {
        var sql = "SELECT MAX(Version) FROM SchemaVersions";
        return await _connection.ExecuteScalarAsync<int>(sql);
    }

    public async Task ApplyVersionAsync(SchemaVersion version)
    {
        using var transaction = _connection.BeginTransaction();
        try
        {
            // Apply schema changes
            await ApplySchemaChangesAsync(version.Version);

            // Record version
            await RecordVersionAsync(version);

            transaction.Commit();
        }
        catch
        {
            transaction.Rollback();
            throw;
        }
    }
}

// Data Versioning with Temporal Tables
public class VersionedEntity
{
    public Guid Id { get; set; }
    public string Name { get; set; }
    public string Description { get; set; }
    public DateTime ValidFrom { get; set; }
    public DateTime ValidTo { get; set; }
    public bool IsCurrent { get; set; }
}

public class TemporalTableRepository<T> where T : VersionedEntity
{
    public async Task<T> GetCurrentAsync(Guid id)
    {
        var sql = $@"
            SELECT * FROM {typeof(T).Name}
            WHERE Id = @Id AND IsCurrent = 1
        ";

        using var connection = new SqlConnection(_connectionString);
        return await connection.QueryFirstOrDefaultAsync<T>(sql, new { Id = id });
    }

    public async Task<IEnumerable<T>> GetHistoryAsync(Guid id)
    {
        var sql = $@"
            SELECT * FROM {typeof(T).Name}
            WHERE Id = @Id
            ORDER BY ValidFrom DESC
        ";

        using var connection = new SqlConnection(_connectionString);
        return await connection.QueryAsync<T>(sql, new { Id = id });
    }

    public async Task<T> GetAtPointInTimeAsync(Guid id, DateTime pointInTime)
    {
        var sql = $@"
            SELECT * FROM {typeof(T).Name}
            WHERE Id = @Id 
              AND ValidFrom <= @PointInTime 
              AND ValidTo > @PointInTime
        ";

        using var connection = new SqlConnection(_connectionString);
        return await connection.QueryFirstOrDefaultAsync<T>(sql, new { Id = id, PointInTime = pointInTime });
    }
}

// Soft Delete with Versioning
public class SoftDeleteRepository<T> where T : class
{
    private readonly IDbContext _context;

    public async Task DeleteAsync(T entity)
    {
        // Instead of hard delete, mark as deleted
        var deletedEntity = new DeletedEntity<T>
        {
            OriginalEntity = entity,
            DeletedAt = DateTime.UtcNow,
            DeletedBy = GetCurrentUser()
        };

        _context.Set<DeletedEntity<T>>().Add(deletedEntity);
        await _context.SaveChangesAsync();
    }

    public async Task<IEnumerable<T>> GetDeletedAsync()
    {
        return await _context.Set<DeletedEntity<T>>()
            .Where(e => e.DeletedAt != null)
            .Select(e => e.OriginalEntity)
            .ToListAsync();
    }
}

// Change Tracking
public class ChangeTracker
{
    private readonly IDbContext _context;

    public async Task TrackChangesAsync<T>(T entity, string operation) where T : class
    {
        var change = new EntityChange
        {
            EntityType = typeof(T).Name,
            EntityId = GetEntityId(entity),
            Operation = operation,
            Changes = SerializeChanges(entity),
            ChangedAt = DateTime.UtcNow,
            ChangedBy = GetCurrentUser()
        };

        _context.Set<EntityChange>().Add(change);
        await _context.SaveChangesAsync();
    }
}

These implementations provide comprehensive solutions for database architecture challenges, including scalability, high availability, data integrity, and real-time processing capabilities. Each solution includes practical C# code examples that can be adapted to specific requirements.

21. How would you break down a monolithic application into microservices?

Strategy: 1. Domain-Driven Design (DDD): Identify bounded contexts and aggregate roots 2. Database Decomposition: Split databases by domain 3. API Analysis: Identify cohesive API groups 4. Dependency Mapping: Analyze service dependencies

Example C# Domain Analysis:

// Before: Monolithic Order Service
public class OrderService
{
    public async Task<Order> CreateOrder(CreateOrderRequest request)
    {
        // Customer validation
        var customer = await _customerRepository.GetById(request.CustomerId);
        if (customer == null) throw new CustomerNotFoundException();

        // Inventory check
        var inventory = await _inventoryService.CheckAvailability(request.Items);
        if (!inventory.IsAvailable) throw new InsufficientInventoryException();

        // Payment processing
        var payment = await _paymentService.ProcessPayment(request.PaymentInfo);
        if (!payment.IsSuccessful) throw new PaymentFailedException();

        // Order creation
        var order = new Order(request);
        await _orderRepository.Create(order);

        // Notification
        await _notificationService.SendOrderConfirmation(order);

        return order;
    }
}

// After: Microservices Breakdown
// Order Service
public class OrderService
{
    private readonly IOrderRepository _orderRepository;
    private readonly ICustomerServiceClient _customerService;
    private readonly IInventoryServiceClient _inventoryService;
    private readonly IPaymentServiceClient _paymentService;
    private readonly INotificationServiceClient _notificationService;

    public async Task<Order> CreateOrder(CreateOrderRequest request)
    {
        // Orchestrate calls to other services
        await _customerService.ValidateCustomer(request.CustomerId);
        await _inventoryService.ReserveInventory(request.Items);
        await _paymentService.ProcessPayment(request.PaymentInfo);

        var order = new Order(request);
        await _orderRepository.Create(order);

        // Async notification
        _ = Task.Run(() => _notificationService.SendOrderConfirmation(order.Id));

        return order;
    }
}

// Customer Service
public class CustomerService
{
    public async Task<Customer> ValidateCustomer(Guid customerId)
    {
        var customer = await _customerRepository.GetById(customerId);
        if (customer == null) throw new CustomerNotFoundException();
        return customer;
    }
}

22. How do you implement service discovery in microservices?

Approaches: 1. Client-side discovery: Service clients query service registry 2. Server-side discovery: Load balancer queries service registry 3. Service mesh: Sidecar proxies handle discovery

C# Implementation with Consul:

// Service Registration
public class ServiceRegistration
{
    public string ServiceId { get; set; }
    public string ServiceName { get; set; }
    public string Address { get; set; }
    public int Port { get; set; }
    public string[] Tags { get; set; }
    public HealthCheck HealthCheck { get; set; }
}

// Consul Service Registry
public class ConsulServiceRegistry : IServiceRegistry
{
    private readonly IConsulClient _consulClient;

    public async Task RegisterServiceAsync(ServiceRegistration registration)
    {
        var serviceRegistration = new AgentServiceRegistration
        {
            ID = registration.ServiceId,
            Name = registration.ServiceName,
            Address = registration.Address,
            Port = registration.Port,
            Tags = registration.Tags,
            Check = new AgentServiceCheck
            {
                HTTP = $"http://{registration.Address}:{registration.Port}/health",
                Interval = TimeSpan.FromSeconds(10),
                Timeout = TimeSpan.FromSeconds(5)
            }
        };

        await _consulClient.Agent.ServiceRegister(serviceRegistration);
    }

    public async Task<List<ServiceInstance>> DiscoverServiceAsync(string serviceName)
    {
        var services = await _consulClient.Catalog.Service(serviceName);
        return services.Response.Select(s => new ServiceInstance
        {
            ServiceId = s.ServiceID,
            Address = s.ServiceAddress,
            Port = s.ServicePort
        }).ToList();
    }
}

// Service Discovery Client
public class ServiceDiscoveryClient : IServiceDiscoveryClient
{
    private readonly IServiceRegistry _serviceRegistry;
    private readonly ILoadBalancer _loadBalancer;

    public async Task<T> CallServiceAsync<T>(string serviceName, Func<string, Task<T>> operation)
    {
        var instances = await _serviceRegistry.DiscoverServiceAsync(serviceName);
        var selectedInstance = _loadBalancer.SelectInstance(instances);

        return await operation($"{selectedInstance.Address}:{selectedInstance.Port}");
    }
}

23. How would you design inter-service communication?

Patterns: 1. Synchronous: HTTP/REST, gRPC 2. Asynchronous: Message queues, event-driven 3. Hybrid: Command Query Responsibility Segregation (CQRS)

C# Implementation:

// Synchronous Communication with HttpClient
public class OrderServiceClient
{
    private readonly HttpClient _httpClient;
    private readonly IServiceDiscoveryClient _serviceDiscovery;

    public async Task<Customer> GetCustomerAsync(Guid customerId)
    {
        return await _serviceDiscovery.CallServiceAsync("customer-service", async (baseUrl) =>
        {
            var response = await _httpClient.GetAsync($"{baseUrl}/api/customers/{customerId}");
            response.EnsureSuccessStatusCode();
            return await response.Content.ReadFromJsonAsync<Customer>();
        });
    }
}

// Asynchronous Communication with RabbitMQ
public class OrderCreatedEvent
{
    public Guid OrderId { get; set; }
    public Guid CustomerId { get; set; }
    public decimal TotalAmount { get; set; }
    public DateTime CreatedAt { get; set; }
}

public class OrderEventPublisher
{
    private readonly IConnection _connection;
    private readonly IModel _channel;

    public async Task PublishOrderCreatedAsync(OrderCreatedEvent orderEvent)
    {
        var message = JsonSerializer.Serialize(orderEvent);
        var body = Encoding.UTF8.GetBytes(message);

        _channel.BasicPublish(
            exchange: "order.events",
            routingKey: "order.created",
            basicProperties: null,
            body: body);
    }
}

public class NotificationEventHandler
{
    public async Task HandleOrderCreatedAsync(OrderCreatedEvent orderEvent)
    {
        // Send notification to customer
        await _notificationService.SendOrderConfirmation(orderEvent.CustomerId, orderEvent.OrderId);
    }
}

// gRPC Communication
[ServiceContract]
public interface ICustomerService
{
    [OperationContract]
    Task<CustomerResponse> GetCustomerAsync(CustomerRequest request);
}

public class CustomerServiceClient
{
    private readonly CustomerService.CustomerServiceClient _grpcClient;

    public async Task<Customer> GetCustomerAsync(Guid customerId)
    {
        var request = new CustomerRequest { CustomerId = customerId.ToString() };
        var response = await _grpcClient.GetCustomerAsync(request);

        return new Customer
        {
            Id = Guid.Parse(response.Id),
            Name = response.Name,
            Email = response.Email
        };
    }
}

24. How do you implement distributed tracing?

Implementation with OpenTelemetry:

// Distributed Tracing Configuration
public class TracingConfiguration
{
    public static void ConfigureTracing(IServiceCollection services)
    {
        services.AddOpenTelemetry()
            .WithTracing(builder => builder
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddSqlClientInstrumentation()
                .AddJaegerExporter(options =>
                {
                    options.AgentHost = "localhost";
                    options.AgentPort = 6831;
                })
                .AddConsoleExporter());
    }
}

// Tracing in Services
public class OrderService
{
    private readonly ActivitySource _activitySource;

    public OrderService()
    {
        _activitySource = new ActivitySource("OrderService");
    }

    public async Task<Order> CreateOrder(CreateOrderRequest request)
    {
        using var activity = _activitySource.StartActivity("CreateOrder");
        activity?.SetTag("customer.id", request.CustomerId.ToString());
        activity?.SetTag("order.items.count", request.Items.Count);

        try
        {
            // Validate customer
            using var customerActivity = _activitySource.StartActivity("ValidateCustomer");
            await _customerService.ValidateCustomer(request.CustomerId);

            // Process payment
            using var paymentActivity = _activitySource.StartActivity("ProcessPayment");
            await _paymentService.ProcessPayment(request.PaymentInfo);

            // Create order
            var order = new Order(request);
            await _orderRepository.Create(order);

            activity?.SetTag("order.id", order.Id.ToString());
            activity?.SetStatus(ActivityStatusCode.Ok);

            return order;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            throw;
        }
    }
}

// Correlation ID Middleware
public class CorrelationIdMiddleware
{
    private readonly RequestDelegate _next;

    public async Task InvokeAsync(HttpContext context)
    {
        var correlationId = context.Request.Headers["X-Correlation-ID"].FirstOrDefault() 
                           ?? Activity.Current?.Id ?? Guid.NewGuid().ToString();

        context.Response.Headers["X-Correlation-ID"] = correlationId;

        using var activity = Activity.Current?.Source.StartActivity("HTTP Request");
        activity?.SetTag("correlation.id", correlationId);

        await _next(context);
    }
}

25. How would you design a microservices deployment strategy?

Strategies: 1. Blue-Green Deployment 2. Canary Deployment 3. Rolling Updates 4. Feature Flags

C# Implementation with Feature Flags:

// Feature Flag Service
public interface IFeatureFlagService
{
    Task<bool> IsEnabledAsync(string featureName, string userId = null);
}

public class FeatureFlagService : IFeatureFlagService
{
    private readonly IConfiguration _configuration;
    private readonly IRedisCache _cache;

    public async Task<bool> IsEnabledAsync(string featureName, string userId = null)
    {
        var cacheKey = $"feature:{featureName}:{userId}";
        var cached = await _cache.GetAsync<bool?>(cacheKey);

        if (cached.HasValue) return cached.Value;

        var isEnabled = await EvaluateFeatureFlag(featureName, userId);
        await _cache.SetAsync(cacheKey, isEnabled, TimeSpan.FromMinutes(5));

        return isEnabled;
    }

    private async Task<bool> EvaluateFeatureFlag(string featureName, string userId)
    {
        // Implement feature flag logic (percentage rollout, user targeting, etc.)
        return _configuration.GetValue<bool>($"FeatureFlags:{featureName}");
    }
}

// Canary Deployment with Load Balancing
public class CanaryLoadBalancer : ILoadBalancer
{
    private readonly IFeatureFlagService _featureFlagService;

    public ServiceInstance SelectInstance(List<ServiceInstance> instances, string userId = null)
    {
        var canaryEnabled = _featureFlagService.IsEnabledAsync("canary-deployment", userId).Result;

        if (canaryEnabled)
        {
            // Route 10% of traffic to canary
            var canaryInstances = instances.Where(i => i.Tags.Contains("canary")).ToList();
            if (canaryInstances.Any() && Random.Shared.NextDouble() < 0.1)
            {
                return canaryInstances[Random.Shared.Next(canaryInstances.Count)];
            }
        }

        // Route to stable instances
        var stableInstances = instances.Where(i => !i.Tags.Contains("canary")).ToList();
        return stableInstances[Random.Shared.Next(stableInstances.Count)];
    }
}

// Health Check for Deployment
public class HealthCheck : IHealthCheck
{
    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        try
        {
            // Check database connectivity
            await _dbContext.Database.CanConnectAsync(cancellationToken);

            // Check external service dependencies
            var customerServiceHealth = await _customerServiceClient.HealthCheckAsync();
            if (!customerServiceHealth.IsHealthy)
            {
                return HealthCheckResult.Degraded("Customer service is unhealthy");
            }

            return HealthCheckResult.Healthy();
        }
        catch (Exception ex)
        {
            return HealthCheckResult.Unhealthy("Health check failed", ex);
        }
    }
}

26. How do you implement circuit breakers in microservices?

Circuit Breaker Pattern Implementation:

// Circuit Breaker State
public enum CircuitBreakerState
{
    Closed,     // Normal operation
    Open,       // Circuit is open, calls fail fast
    HalfOpen    // Testing if service is back
}

// Circuit Breaker Implementation
public class CircuitBreaker
{
    private readonly object _lock = new object();
    private CircuitBreakerState _state = CircuitBreakerState.Closed;
    private int _failureCount = 0;
    private DateTime _lastFailureTime;

    private readonly int _failureThreshold;
    private readonly TimeSpan _timeout;
    private readonly int _successThreshold;

    public CircuitBreaker(int failureThreshold = 5, int timeoutSeconds = 60, int successThreshold = 2)
    {
        _failureThreshold = failureThreshold;
        _timeout = TimeSpan.FromSeconds(timeoutSeconds);
        _successThreshold = successThreshold;
    }

    public async Task<T> ExecuteAsync<T>(Func<Task<T>> operation)
    {
        if (ShouldAllowExecution())
        {
            try
            {
                var result = await operation();
                OnSuccess();
                return result;
            }
            catch (Exception ex)
            {
                OnFailure();
                throw;
            }
        }

        throw new CircuitBreakerOpenException("Circuit breaker is open");
    }

    private bool ShouldAllowExecution()
    {
        lock (_lock)
        {
            switch (_state)
            {
                case CircuitBreakerState.Closed:
                    return true;

                case CircuitBreakerState.Open:
                    if (DateTime.UtcNow - _lastFailureTime > _timeout)
                    {
                        _state = CircuitBreakerState.HalfOpen;
                        return true;
                    }
                    return false;

                case CircuitBreakerState.HalfOpen:
                    return true;

                default:
                    return false;
            }
        }
    }

    private void OnSuccess()
    {
        lock (_lock)
        {
            _failureCount = 0;
            if (_state == CircuitBreakerState.HalfOpen)
            {
                _state = CircuitBreakerState.Closed;
            }
        }
    }

    private void OnFailure()
    {
        lock (_lock)
        {
            _failureCount++;
            _lastFailureTime = DateTime.UtcNow;

            if (_failureCount >= _failureThreshold)
            {
                _state = CircuitBreakerState.Open;
            }
        }
    }
}

// Circuit Breaker Decorator
public class CircuitBreakerHttpClientHandler : DelegatingHandler
{
    private readonly CircuitBreaker _circuitBreaker;

    public CircuitBreakerHttpClientHandler()
    {
        _circuitBreaker = new CircuitBreaker();
    }

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        return await _circuitBreaker.ExecuteAsync(async () =>
        {
            var response = await base.SendAsync(request, cancellationToken);

            if (!response.IsSuccessStatusCode)
            {
                throw new HttpRequestException($"HTTP {response.StatusCode}");
            }

            return response;
        });
    }
}

// Usage in Service Client
public class CustomerServiceClient
{
    private readonly HttpClient _httpClient;

    public CustomerServiceClient(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }

    public async Task<Customer> GetCustomerAsync(Guid customerId)
    {
        var response = await _httpClient.GetAsync($"/api/customers/{customerId}");
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadFromJsonAsync<Customer>();
    }
}

// Service Registration
services.AddHttpClient<CustomerServiceClient>()
    .AddHttpMessageHandler<CircuitBreakerHttpClientHandler>();

27. How would you design a microservices data consistency strategy?

Distributed data consistency starts from invariants. Prefer one service/database as the owner of each write model; publish durable events through an outbox; make consumers idempotent; expose retries and compensation for cross-service workflows. “Exactly once” delivery is rarely an end-to-end guarantee; design for at-least-once delivery and deduplication.

28. How do you implement API versioning in microservices?

API Versioning Strategies:

// URL Versioning
[ApiController]
[Route("api/v{version:apiVersion}/[controller]")]
[ApiVersion("1.0")]
[ApiVersion("2.0")]
public class CustomersController : ControllerBase
{
    [HttpGet("{id}")]
    [MapToApiVersion("1.0")]
    public async Task<ActionResult<CustomerV1>> GetCustomerV1(Guid id)
    {
        var customer = await _customerService.GetCustomerAsync(id);
        return Ok(new CustomerV1
        {
            Id = customer.Id,
            Name = customer.Name,
            Email = customer.Email
        });
    }

    [HttpGet("{id}")]
    [MapToApiVersion("2.0")]
    public async Task<ActionResult<CustomerV2>> GetCustomerV2(Guid id)
    {
        var customer = await _customerService.GetCustomerAsync(id);
        return Ok(new CustomerV2
        {
            Id = customer.Id,
            Name = customer.Name,
            Email = customer.Email,
            PhoneNumber = customer.PhoneNumber,
            Address = customer.Address
        });
    }
}

// Header Versioning
[ApiController]
[Route("api/[controller]")]
public class CustomersController : ControllerBase
{
    [HttpGet("{id}")]
    public async Task<IActionResult> GetCustomer(Guid id, [FromHeader(Name = "Api-Version")] string version = "1.0")
    {
        switch (version)
        {
            case "1.0":
                var customerV1 = await _customerService.GetCustomerAsync(id);
                return Ok(new CustomerV1
                {
                    Id = customerV1.Id,
                    Name = customerV1.Name,
                    Email = customerV1.Email
                });

            case "2.0":
                var customerV2 = await _customerService.GetCustomerAsync(id);
                return Ok(new CustomerV2
                {
                    Id = customerV2.Id,
                    Name = customerV2.Name,
                    Email = customerV2.Email,
                    PhoneNumber = customerV2.PhoneNumber,
                    Address = customerV2.Address
                });

            default:
                return BadRequest("Unsupported API version");
        }
    }
}

// Content Negotiation Versioning
[ApiController]
[Route("api/[controller]")]
public class CustomersController : ControllerBase
{
    [HttpGet("{id}")]
    public async Task<IActionResult> GetCustomer(Guid id)
    {
        var customer = await _customerService.GetCustomerAsync(id);

        var acceptHeader = Request.Headers["Accept"].ToString();

        if (acceptHeader.Contains("application/vnd.company.customer-v2+json"))
        {
            return Ok(new CustomerV2
            {
                Id = customer.Id,
                Name = customer.Name,
                Email = customer.Email,
                PhoneNumber = customer.PhoneNumber,
                Address = customer.Address
            });
        }

        return Ok(new CustomerV1
        {
            Id = customer.Id,
            Name = customer.Name,
            Email = customer.Email
        });
    }
}

// API Version Configuration
public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddApiVersioning(options =>
        {
            options.DefaultApiVersion = new ApiVersion(1, 0);
            options.AssumeDefaultVersionWhenUnspecified = true;
            options.ReportApiVersions = true;
            options.ApiVersionReader = ApiVersionReader.Combine(
                new UrlSegmentApiVersionReader(),
                new HeaderApiVersionReader("Api-Version"),
                new MediaTypeApiVersionReader("version")
            );
        });

        services.AddVersionedApiExplorer(options =>
        {
            options.GroupNameFormat = "'v'VVV";
            options.SubstituteApiVersionInUrl = true;
        });
    }
}

29. How would you design a microservices monitoring system?

Comprehensive Monitoring Solution:

// Metrics Collection
public class MetricsCollector
{
    private readonly IMetricsRoot _metrics;
    private readonly Counter _requestCounter;
    private readonly Histogram _requestDuration;
    private readonly Gauge _activeConnections;

    public MetricsCollector(IMetricsRoot metrics)
    {
        _metrics = metrics;
        _requestCounter = _metrics.CreateCounter("http_requests_total", "Total HTTP requests", new CounterConfiguration
        {
            LabelNames = new[] { "method", "endpoint", "status_code" }
        });

        _requestDuration = _metrics.CreateHistogram("http_request_duration_seconds", "HTTP request duration", new HistogramConfiguration
        {
            LabelNames = new[] { "method", "endpoint" }
        });

        _activeConnections = _metrics.CreateGauge("active_connections", "Number of active connections");
    }

    public void RecordRequest(string method, string endpoint, int statusCode, TimeSpan duration)
    {
        _requestCounter.WithLabels(method, endpoint, statusCode.ToString()).Increment();
        _requestDuration.WithLabels(method, endpoint).Observe(duration.TotalSeconds);
    }

    public void SetActiveConnections(int count)
    {
        _activeConnections.Set(count);
    }
}

// Health Checks
public class DatabaseHealthCheck : IHealthCheck
{
    private readonly DbContext _dbContext;

    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        try
        {
            await _dbContext.Database.CanConnectAsync(cancellationToken);
            return HealthCheckResult.Healthy("Database is accessible");
        }
        catch (Exception ex)
        {
            return HealthCheckResult.Unhealthy("Database is not accessible", ex);
        }
    }
}

public class ExternalServiceHealthCheck : IHealthCheck
{
    private readonly HttpClient _httpClient;

    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        try
        {
            var response = await _httpClient.GetAsync("/health", cancellationToken);
            if (response.IsSuccessStatusCode)
            {
                return HealthCheckResult.Healthy("External service is healthy");
            }
            return HealthCheckResult.Degraded("External service returned non-success status");
        }
        catch (Exception ex)
        {
            return HealthCheckResult.Unhealthy("External service is not accessible", ex);
        }
    }
}

// Monitoring Middleware
public class MonitoringMiddleware
{
    private readonly RequestDelegate _next;
    private readonly MetricsCollector _metrics;
    private readonly ILogger<MonitoringMiddleware> _logger;

    public async Task InvokeAsync(HttpContext context)
    {
        var stopwatch = Stopwatch.StartNew();
        var originalBodyStream = context.Response.Body;

        using var memoryStream = new MemoryStream();
        context.Response.Body = memoryStream;

        try
        {
            await _next(context);

            stopwatch.Stop();
            _metrics.RecordRequest(
                context.Request.Method,
                context.Request.Path,
                context.Response.StatusCode,
                stopwatch.Elapsed
            );

            memoryStream.Position = 0;
            await memoryStream.CopyToAsync(originalBodyStream);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Request failed: {Method} {Path}", context.Request.Method, context.Request.Path);
            throw;
        }
        finally
        {
            context.Response.Body = originalBodyStream;
        }
    }
}

// Alerting Service
public class AlertingService
{
    private readonly ILogger<AlertingService> _logger;
    private readonly IEmailService _emailService;
    private readonly ISlackService _slackService;

    public async Task SendAlertAsync(Alert alert)
    {
        _logger.LogWarning("Alert triggered: {AlertType} - {Message}", alert.Type, alert.Message);

        switch (alert.Severity)
        {
            case AlertSeverity.Critical:
                await _emailService.SendAlertAsync(alert);
                await _slackService.SendAlertAsync(alert);
                break;

            case AlertSeverity.Warning:
                await _slackService.SendAlertAsync(alert);
                break;
        }
    }
}

// Dashboard Configuration
public class MonitoringConfiguration
{
    public static void ConfigureMonitoring(IServiceCollection services)
    {
        services.AddHealthChecks()
            .AddCheck<DatabaseHealthCheck>("database")
            .AddCheck<ExternalServiceHealthCheck>("external-service")
            .AddPrometheusGatewayPublisher();

        services.AddMetrics();
        services.AddMetricsEndpoints();

        services.AddSingleton<MetricsCollector>();
        services.AddSingleton<AlertingService>();
    }
}

30. How do you implement distributed logging?

Centralized Logging with ELK Stack:

// Structured Logging Configuration
public class LoggingConfiguration
{
    public static void ConfigureLogging(IServiceCollection services, IConfiguration configuration)
    {
        services.AddLogging(builder =>
        {
            builder.AddConsole();
            builder.AddSeq(configuration.GetSection("Seq"));
            builder.AddElasticsearch(configuration.GetSection("Elasticsearch"));
        });
    }
}

// Correlation ID Middleware
public class CorrelationIdMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<CorrelationIdMiddleware> _logger;

    public async Task InvokeAsync(HttpContext context)
    {
        var correlationId = context.Request.Headers["X-Correlation-ID"].FirstOrDefault() 
                           ?? Activity.Current?.Id ?? Guid.NewGuid().ToString();

        context.Response.Headers["X-Correlation-ID"] = correlationId;

        using var scope = _logger.BeginScope(new Dictionary<string, object>
        {
            ["CorrelationId"] = correlationId,
            ["RequestId"] = context.TraceIdentifier,
            ["UserAgent"] = context.Request.Headers["User-Agent"].ToString(),
            ["RemoteIpAddress"] = context.Connection.RemoteIpAddress?.ToString()
        });

        await _next(context);
    }
}

// Structured Logging Service
public class StructuredLogger<T>
{
    private readonly ILogger<T> _logger;

    public StructuredLogger(ILogger<T> logger)
    {
        _logger = logger;
    }

    public void LogOrderCreated(Guid orderId, Guid customerId, decimal totalAmount)
    {
        _logger.LogInformation("Order created successfully. OrderId: {OrderId}, CustomerId: {CustomerId}, TotalAmount: {TotalAmount}",
            orderId, customerId, totalAmount);
    }

    public void LogServiceCall(string serviceName, string operation, TimeSpan duration, bool success)
    {
        _logger.LogInformation("Service call completed. Service: {ServiceName}, Operation: {Operation}, Duration: {Duration}ms, Success: {Success}",
            serviceName, operation, duration.TotalMilliseconds, success);
    }

    public void LogError(Exception ex, string operation, object context = null)
    {
        _logger.LogError(ex, "Error occurred during {Operation}. Context: {@Context}", operation, context);
    }
}

// Log Aggregation Service
public class LogAggregationService
{
    private readonly ILogger<LogAggregationService> _logger;
    private readonly ElasticsearchClient _elasticsearchClient;

    public async Task IndexLogAsync(LogEntry logEntry)
    {
        try
        {
            var indexName = $"logs-{DateTime.UtcNow:yyyy.MM.dd}";
            await _elasticsearchClient.IndexAsync(logEntry, indexName);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to index log entry");
        }
    }

    public async Task<List<LogEntry>> SearchLogsAsync(LogSearchRequest request)
    {
        var searchRequest = new SearchRequest<LogEntry>
        {
            Query = new BoolQuery
            {
                Must = new List<QueryContainer>
                {
                    new MatchQuery { Field = "level", Query = request.Level },
                    new DateRangeQuery
                    {
                        Field = "timestamp",
                        GreaterThanOrEqualTo = request.From,
                        LessThanOrEqualTo = request.To
                    }
                }
            },
            Sort = new List<ISort>
            {
                new FieldSort { Field = "timestamp", Order = SortOrder.Descending }
            },
            Size = request.Limit
        };

        var response = await _elasticsearchClient.SearchAsync<LogEntry>(searchRequest);
        return response.Documents.ToList();
    }
}

// Log Entry Model
public class LogEntry
{
    public string Id { get; set; } = Guid.NewGuid().ToString();
    public string Level { get; set; }
    public string Message { get; set; }
    public string CorrelationId { get; set; }
    public string ServiceName { get; set; }
    public string Operation { get; set; }
    public DateTime Timestamp { get; set; } = DateTime.UtcNow;
    public Dictionary<string, object> Properties { get; set; } = new();
    public Exception Exception { get; set; }
}

// Usage in Services
public class OrderService
{
    private readonly StructuredLogger<OrderService> _logger;
    private readonly LogAggregationService _logAggregation;

    public async Task<Order> CreateOrder(CreateOrderRequest request)
    {
        using var scope = _logger.BeginScope(new Dictionary<string, object>
        {
            ["Operation"] = "CreateOrder",
            ["CustomerId"] = request.CustomerId,
            ["ItemsCount"] = request.Items.Count
        });

        try
        {
            _logger.LogInformation("Starting order creation process");

            var order = new Order(request);
            await _orderRepository.Create(order);

            _logger.LogOrderCreated(order.Id, request.CustomerId, order.TotalAmount);

            return order;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Order creation failed", new { Request = request });
            throw;
        }
    }
}

// Log Viewer API
[ApiController]
[Route("api/[controller]")]
public class LogsController : ControllerBase
{
    private readonly LogAggregationService _logAggregation;

    [HttpGet]
    public async Task<ActionResult<List<LogEntry>>> GetLogs([FromQuery] LogSearchRequest request)
    {
        var logs = await _logAggregation.SearchLogsAsync(request);
        return Ok(logs);
    }

    [HttpGet("correlation/{correlationId}")]
    public async Task<ActionResult<List<LogEntry>>> GetLogsByCorrelationId(string correlationId)
    {
        var request = new LogSearchRequest
        {
            CorrelationId = correlationId,
            Limit = 100
        };

        var logs = await _logAggregation.SearchLogsAsync(request);
        return Ok(logs);
    }
}

These implementations provide comprehensive solutions for microservices architecture challenges, covering all aspects from service decomposition to monitoring and logging. Each solution includes practical C# code examples that can be adapted to real-world scenarios.

31. How would you design a message queue system?

A message queue system should provide reliable, asynchronous communication between components. Here's a design approach:

Key Components: - Message Producer - Message Consumer
- Queue Storage - Message Broker - Dead Letter Queue - Monitoring/Health Checks

C# Implementation:

public interface IMessageQueue
{
    Task EnqueueAsync<T>(T message, string queueName);
    Task<T> DequeueAsync<T>(string queueName);
    Task<bool> IsEmptyAsync(string queueName);
}

public class Message
{
    public Guid Id { get; set; }
    public string Type { get; set; }
    public string Payload { get; set; }
    public DateTime CreatedAt { get; set; }
    public int RetryCount { get; set; }
    public MessageStatus Status { get; set; }
}

public enum MessageStatus
{
    Pending,
    Processing,
    Completed,
    Failed
}

public class InMemoryMessageQueue : IMessageQueue
{
    private readonly ConcurrentDictionary<string, ConcurrentQueue<Message>> _queues;
    private readonly ILogger<InMemoryMessageQueue> _logger;

    public InMemoryMessageQueue(ILogger<InMemoryMessageQueue> logger)
    {
        _queues = new ConcurrentDictionary<string, ConcurrentQueue<Message>>();
        _logger = logger;
    }

    public async Task EnqueueAsync<T>(T message, string queueName)
    {
        var messageObj = new Message
        {
            Id = Guid.NewGuid(),
            Type = typeof(T).Name,
            Payload = JsonSerializer.Serialize(message),
            CreatedAt = DateTime.UtcNow,
            Status = MessageStatus.Pending
        };

        var queue = _queues.GetOrAdd(queueName, _ => new ConcurrentQueue<Message>());
        queue.Enqueue(messageObj);

        _logger.LogInformation($"Message {messageObj.Id} enqueued to {queueName}");
    }

    public async Task<T> DequeueAsync<T>(string queueName)
    {
        if (!_queues.TryGetValue(queueName, out var queue))
            throw new InvalidOperationException($"Queue {queueName} not found");

        if (queue.TryDequeue(out var message))
        {
            message.Status = MessageStatus.Processing;
            return JsonSerializer.Deserialize<T>(message.Payload);
        }

        return default(T);
    }

    public async Task<bool> IsEmptyAsync(string queueName)
    {
        return !_queues.TryGetValue(queueName, out var queue) || queue.IsEmpty;
    }
}

32. How do you implement event sourcing?

Event sourcing stores all changes as a sequence of events rather than just the current state.

C# Implementation:

public abstract class Event
{
    public Guid Id { get; set; }
    public Guid AggregateId { get; set; }
    public long Version { get; set; }
    public DateTime Timestamp { get; set; }
    public string EventType { get; set; }
}

public abstract class AggregateRoot
{
    private readonly List<Event> _uncommittedEvents = new();
    public Guid Id { get; protected set; }
    public long Version { get; private set; }

    protected void Apply(Event @event)
    {
        @event.AggregateId = Id;
        @event.Version = Version + 1;
        @event.Timestamp = DateTime.UtcNow;
        @event.EventType = @event.GetType().Name;

        _uncommittedEvents.Add(@event);
        When(@event);
        Version++;
    }

    protected abstract void When(Event @event);

    public IEnumerable<Event> GetUncommittedEvents() => _uncommittedEvents;

    public void MarkEventsAsCommitted() => _uncommittedEvents.Clear();
}

// Example: Bank Account Aggregate
public class AccountCreated : Event
{
    public string AccountNumber { get; set; }
    public string OwnerName { get; set; }
    public decimal InitialBalance { get; set; }
}

public class MoneyDeposited : Event
{
    public decimal Amount { get; set; }
}

public class MoneyWithdrawn : Event
{
    public decimal Amount { get; set; }
}

public class BankAccount : AggregateRoot
{
    public string AccountNumber { get; private set; }
    public string OwnerName { get; private set; }
    public decimal Balance { get; private set; }

    public BankAccount(string accountNumber, string ownerName, decimal initialBalance)
    {
        Id = Guid.NewGuid();
        Apply(new AccountCreated
        {
            AccountNumber = accountNumber,
            OwnerName = ownerName,
            InitialBalance = initialBalance
        });
    }

    public void Deposit(decimal amount)
    {
        if (amount <= 0) throw new ArgumentException("Amount must be positive");
        Apply(new MoneyDeposited { Amount = amount });
    }

    public void Withdraw(decimal amount)
    {
        if (amount <= 0) throw new ArgumentException("Amount must be positive");
        if (Balance < amount) throw new InvalidOperationException("Insufficient funds");
        Apply(new MoneyWithdrawn { Amount = amount });
    }

    protected override void When(Event @event)
    {
        switch (@event)
        {
            case AccountCreated e:
                AccountNumber = e.AccountNumber;
                OwnerName = e.OwnerName;
                Balance = e.InitialBalance;
                break;
            case MoneyDeposited e:
                Balance += e.Amount;
                break;
            case MoneyWithdrawn e:
                Balance -= e.Amount;
                break;
        }
    }
}

public interface IEventStore
{
    Task SaveEventsAsync(Guid aggregateId, IEnumerable<Event> events, long expectedVersion);
    Task<IEnumerable<Event>> GetEventsAsync(Guid aggregateId);
}

public class InMemoryEventStore : IEventStore
{
    private readonly Dictionary<Guid, List<Event>> _events = new();

    public async Task SaveEventsAsync(Guid aggregateId, IEnumerable<Event> events, long expectedVersion)
    {
        if (!_events.ContainsKey(aggregateId))
        {
            _events[aggregateId] = new List<Event>();
        }

        var existingEvents = _events[aggregateId];
        if (existingEvents.Count != expectedVersion)
        {
            throw new ConcurrencyException();
        }

        existingEvents.AddRange(events);
    }

    public async Task<IEnumerable<Event>> GetEventsAsync(Guid aggregateId)
    {
        return _events.ContainsKey(aggregateId) ? _events[aggregateId] : Enumerable.Empty<Event>();
    }
}

33. How would you design a pub/sub system?

A publish/subscribe system allows decoupled communication between publishers and subscribers.

C# Implementation:

public interface IPublisher
{
    Task PublishAsync<T>(string topic, T message);
}

public interface ISubscriber
{
    Task SubscribeAsync<T>(string topic, Func<T, Task> handler);
    Task UnsubscribeAsync(string topic);
}

public class PubSubSystem : IPublisher, ISubscriber
{
    private readonly ConcurrentDictionary<string, List<object>> _subscribers;
    private readonly ILogger<PubSubSystem> _logger;

    public PubSubSystem(ILogger<PubSubSystem> logger)
    {
        _subscribers = new ConcurrentDictionary<string, List<object>>();
        _logger = logger;
    }

    public async Task PublishAsync<T>(string topic, T message)
    {
        if (_subscribers.TryGetValue(topic, out var handlers))
        {
            var tasks = handlers
                .OfType<Func<T, Task>>()
                .Select(handler => handler(message))
                .ToArray();

            await Task.WhenAll(tasks);
            _logger.LogInformation($"Published message to {handlers.Count} subscribers on topic {topic}");
        }
    }

    public async Task SubscribeAsync<T>(string topic, Func<T, Task> handler)
    {
        var handlers = _subscribers.GetOrAdd(topic, _ => new List<object>());
        lock (handlers)
        {
            handlers.Add(handler);
        }
        _logger.LogInformation($"Subscribed to topic {topic}");
    }

    public async Task UnsubscribeAsync(string topic)
    {
        _subscribers.TryRemove(topic, out _);
        _logger.LogInformation($"Unsubscribed from topic {topic}");
    }
}

// Usage Example
public class OrderService
{
    private readonly IPublisher _publisher;

    public OrderService(IPublisher publisher)
    {
        _publisher = publisher;
    }

    public async Task CreateOrderAsync(Order order)
    {
        // Process order creation
        await _publisher.PublishAsync("order.created", order);
    }
}

public class NotificationService
{
    public async Task HandleOrderCreated(Order order)
    {
        // Send notification
        Console.WriteLine($"Sending notification for order {order.Id}");
    }
}

34. How do you implement message ordering?

Ordering is usually scoped to a key, partition, or producer. Global ordering harms throughput and availability. Define the business ordering requirement, partition by the ordering key, store sequence/version metadata, and make consumers idempotent when retries or redelivery occur.

35. How would you design a dead letter queue?

A dead letter queue handles messages that cannot be processed successfully.

C# Implementation:

public class DeadLetterMessage
{
    public Guid Id { get; set; }
    public string OriginalQueue { get; set; }
    public string OriginalMessage { get; set; }
    public string ErrorMessage { get; set; }
    public int RetryCount { get; set; }
    public DateTime FailedAt { get; set; }
    public DateTime? ProcessedAt { get; set; }
}

public class DeadLetterQueue
{
    private readonly ConcurrentQueue<DeadLetterMessage> _dlq;
    private readonly ILogger<DeadLetterQueue> _logger;
    private readonly int _maxRetries;

    public DeadLetterQueue(ILogger<DeadLetterQueue> logger, int maxRetries = 3)
    {
        _dlq = new ConcurrentQueue<DeadLetterMessage>();
        _logger = logger;
        _maxRetries = maxRetries;
    }

    public async Task MoveToDeadLetterAsync(string originalQueue, string message, Exception error, int retryCount)
    {
        var dlqMessage = new DeadLetterMessage
        {
            Id = Guid.NewGuid(),
            OriginalQueue = originalQueue,
            OriginalMessage = message,
            ErrorMessage = error.Message,
            RetryCount = retryCount,
            FailedAt = DateTime.UtcNow
        };

        _dlq.Enqueue(dlqMessage);
        _logger.LogError($"Message moved to DLQ: {dlqMessage.Id}, Error: {error.Message}");
    }

    public async Task<DeadLetterMessage> DequeueAsync()
    {
        if (_dlq.TryDequeue(out var message))
        {
            return message;
        }
        return null;
    }

    public async Task<bool> ShouldMoveToDeadLetterAsync(int retryCount)
    {
        return retryCount >= _maxRetries;
    }

    public async Task<IEnumerable<DeadLetterMessage>> GetAllMessagesAsync()
    {
        return _dlq.ToArray();
    }

    public async Task<int> GetMessageCountAsync()
    {
        return _dlq.Count;
    }
}

public class MessageProcessor
{
    private readonly DeadLetterQueue _dlq;
    private readonly ILogger<MessageProcessor> _logger;

    public MessageProcessor(DeadLetterQueue dlq, ILogger<MessageProcessor> logger)
    {
        _dlq = dlq;
        _logger = logger;
    }

    public async Task ProcessMessageAsync(string message, int retryCount = 0)
    {
        try
        {
            // Process the message
            await ProcessMessageInternalAsync(message);
        }
        catch (Exception ex)
        {
            if (await _dlq.ShouldMoveToDeadLetterAsync(retryCount))
            {
                await _dlq.MoveToDeadLetterAsync("main-queue", message, ex, retryCount);
            }
            else
            {
                // Retry logic
                await Task.Delay(1000 * (retryCount + 1)); // Exponential backoff
                await ProcessMessageAsync(message, retryCount + 1);
            }
        }
    }

    private async Task ProcessMessageInternalAsync(string message)
    {
        // Actual message processing logic
        if (message.Contains("error"))
        {
            throw new InvalidOperationException("Simulated processing error");
        }

        _logger.LogInformation($"Successfully processed message: {message}");
    }
}

36. How do you implement message deduplication?

Deduplication needs a stable message identity, a retention window, and atomicity between recording that identity and applying the effect. A deduplication table/key with a unique constraint is common. It cannot provide unlimited historical deduplication without storage/cost trade-offs.

37. How would you design an event-driven architecture?

Event-driven architecture uses events to trigger and communicate between decoupled services.

C# Implementation:

public interface IEventBus
{
    Task PublishAsync<T>(T @event) where T : IEvent;
    Task SubscribeAsync<T>(IEventHandler<T> handler) where T : IEvent;
}

public interface IEvent
{
    Guid Id { get; }
    DateTime OccurredOn { get; }
}

public interface IEventHandler<in T> where T : IEvent
{
    Task HandleAsync(T @event);
}

public class EventBus : IEventBus
{
    private readonly Dictionary<Type, List<object>> _handlers;
    private readonly ILogger<EventBus> _logger;

    public EventBus(ILogger<EventBus> logger)
    {
        _handlers = new Dictionary<Type, List<object>>();
        _logger = logger;
    }

    public async Task PublishAsync<T>(T @event) where T : IEvent
    {
        var eventType = typeof(T);
        if (_handlers.TryGetValue(eventType, out var handlers))
        {
            var tasks = handlers
                .OfType<IEventHandler<T>>()
                .Select(handler => handler.HandleAsync(@event))
                .ToArray();

            await Task.WhenAll(tasks);
            _logger.LogInformation($"Published event {eventType.Name} to {handlers.Count} handlers");
        }
    }

    public async Task SubscribeAsync<T>(IEventHandler<T> handler) where T : IEvent
    {
        var eventType = typeof(T);
        if (!_handlers.ContainsKey(eventType))
        {
            _handlers[eventType] = new List<object>();
        }
        _handlers[eventType].Add(handler);
        _logger.LogInformation($"Subscribed handler for event {eventType.Name}");
    }
}

// Example Events
public class OrderCreatedEvent : IEvent
{
    public Guid Id { get; set; }
    public DateTime OccurredOn { get; set; }
    public Guid OrderId { get; set; }
    public string CustomerId { get; set; }
    public decimal TotalAmount { get; set; }
}

public class PaymentProcessedEvent : IEvent
{
    public Guid Id { get; set; }
    public DateTime OccurredOn { get; set; }
    public Guid OrderId { get; set; }
    public string PaymentId { get; set; }
    public decimal Amount { get; set; }
    public bool IsSuccessful { get; set; }
}

// Example Event Handlers
public class OrderCreatedEventHandler : IEventHandler<OrderCreatedEvent>
{
    private readonly ILogger<OrderCreatedEventHandler> _logger;

    public OrderCreatedEventHandler(ILogger<OrderCreatedEventHandler> logger)
    {
        _logger = logger;
    }

    public async Task HandleAsync(OrderCreatedEvent @event)
    {
        _logger.LogInformation($"Order created: {@event.OrderId} for customer {@event.CustomerId}");
        // Send confirmation email, update inventory, etc.
    }
}

public class PaymentProcessedEventHandler : IEventHandler<PaymentProcessedEvent>
{
    private readonly ILogger<PaymentProcessedEventHandler> _logger;

    public PaymentProcessedEventHandler(ILogger<PaymentProcessedEventHandler> logger)
    {
        _logger = logger;
    }

    public async Task HandleAsync(PaymentProcessedEvent @event)
    {
        if (@event.IsSuccessful)
        {
            _logger.LogInformation($"Payment successful for order {@event.OrderId}");
            // Update order status, send confirmation
        }
        else
        {
            _logger.LogWarning($"Payment failed for order {@event.OrderId}");
            // Handle failed payment
        }
    }
}

// Example Service
public class OrderService
{
    private readonly IEventBus _eventBus;
    private readonly ILogger<OrderService> _logger;

    public OrderService(IEventBus eventBus, ILogger<OrderService> logger)
    {
        _eventBus = eventBus;
        _logger = logger;
    }

    public async Task CreateOrderAsync(string customerId, decimal totalAmount)
    {
        var orderId = Guid.NewGuid();

        // Create order logic here

        var orderCreatedEvent = new OrderCreatedEvent
        {
            Id = Guid.NewGuid(),
            OccurredOn = DateTime.UtcNow,
            OrderId = orderId,
            CustomerId = customerId,
            TotalAmount = totalAmount
        };

        await _eventBus.PublishAsync(orderCreatedEvent);
        _logger.LogInformation($"Order created and event published: {orderId}");
    }
}

38. How do you implement event replay?

Event replay allows reprocessing events from a specific point in time.

C# Implementation:

public interface IEventStore
{
    Task SaveEventAsync(IEvent @event);
    Task<IEnumerable<IEvent>> GetEventsAsync(DateTime from, DateTime to);
    Task<IEnumerable<IEvent>> GetEventsAsync(Guid aggregateId);
    Task<long> GetEventCountAsync();
}

public class EventReplayer
{
    private readonly IEventStore _eventStore;
    private readonly IEventBus _eventBus;
    private readonly ILogger<EventReplayer> _logger;

    public EventReplayer(IEventStore eventStore, IEventBus eventBus, ILogger<EventReplayer> logger)
    {
        _eventStore = eventStore;
        _eventBus = eventBus;
        _logger = logger;
    }

    public async Task ReplayEventsAsync(DateTime from, DateTime to, bool republishToEventBus = true)
    {
        var events = await _eventStore.GetEventsAsync(from, to);
        var eventList = events.ToList();

        _logger.LogInformation($"Starting event replay for {eventList.Count} events from {from} to {to}");

        foreach (var @event in eventList.OrderBy(e => e.OccurredOn))
        {
            try
            {
                if (republishToEventBus)
                {
                    await _eventBus.PublishAsync(@event);
                }

                _logger.LogDebug($"Replayed event: {@event.GetType().Name} at {@event.OccurredOn}");
            }
            catch (Exception ex)
            {
                _logger.LogError($"Error replaying event {@event.Id}: {ex.Message}");
            }
        }

        _logger.LogInformation($"Completed event replay for {eventList.Count} events");
    }

    public async Task ReplayEventsForAggregateAsync(Guid aggregateId, bool republishToEventBus = true)
    {
        var events = await _eventStore.GetEventsAsync(aggregateId);
        var eventList = events.ToList();

        _logger.LogInformation($"Starting event replay for aggregate {aggregateId} with {eventList.Count} events");

        foreach (var @event in eventList.OrderBy(e => e.OccurredOn))
        {
            try
            {
                if (republishToEventBus)
                {
                    await _eventBus.PublishAsync(@event);
                }

                _logger.LogDebug($"Replayed event for aggregate {aggregateId}: {@event.GetType().Name}");
            }
            catch (Exception ex)
            {
                _logger.LogError($"Error replaying event {@event.Id} for aggregate {aggregateId}: {ex.Message}");
            }
        }

        _logger.LogInformation($"Completed event replay for aggregate {aggregateId}");
    }

    public async Task<EventReplayReport> GenerateReplayReportAsync(DateTime from, DateTime to)
    {
        var events = await _eventStore.GetEventsAsync(from, to);
        var eventList = events.ToList();

        var report = new EventReplayReport
        {
            From = from,
            To = to,
            TotalEvents = eventList.Count,
            EventTypes = eventList.GroupBy(e => e.GetType().Name)
                                 .ToDictionary(g => g.Key, g => g.Count()),
            ReplayDuration = TimeSpan.Zero
        };

        return report;
    }
}

public class EventReplayReport
{
    public DateTime From { get; set; }
    public DateTime To { get; set; }
    public int TotalEvents { get; set; }
    public Dictionary<string, int> EventTypes { get; set; }
    public TimeSpan ReplayDuration { get; set; }
    public List<string> Errors { get; set; } = new List<string>();
}

// Example usage
public class EventReplayService
{
    private readonly EventReplayer _replayer;
    private readonly ILogger<EventReplayService> _logger;

    public EventReplayService(EventReplayer replayer, ILogger<EventReplayService> logger)
    {
        _replayer = replayer;
        _logger = logger;
    }

    public async Task ReplayLast24HoursAsync()
    {
        var from = DateTime.UtcNow.AddDays(-1);
        var to = DateTime.UtcNow;

        await _replayer.ReplayEventsAsync(from, to);
    }

    public async Task ReplaySpecificAggregateAsync(Guid aggregateId)
    {
        await _replayer.ReplayEventsForAggregateAsync(aggregateId);
    }
}

39. How would you design a message routing system?

A message routing system directs messages to appropriate handlers based on routing rules.

C# Implementation:

public interface IMessageRouter
{
    Task RouteAsync<T>(T message);
    void AddRoute<T>(string route, IMessageHandler<T> handler);
    void RemoveRoute<T>(string route);
}

public interface IMessageHandler<in T>
{
    Task HandleAsync(T message);
}

public class MessageRouter : IMessageRouter
{
    private readonly Dictionary<string, Dictionary<Type, object>> _routes;
    private readonly ILogger<MessageRouter> _logger;

    public MessageRouter(ILogger<MessageRouter> logger)
    {
        _routes = new Dictionary<string, Dictionary<Type, object>>();
        _logger = logger;
    }

    public async Task RouteAsync<T>(T message)
    {
        var messageType = typeof(T);
        var route = DetermineRoute(message);

        if (_routes.TryGetValue(route, out var handlers) && 
            handlers.TryGetValue(messageType, out var handler))
        {
            var typedHandler = (IMessageHandler<T>)handler;
            await typedHandler.HandleAsync(message);
            _logger.LogInformation($"Routed message of type {messageType.Name} to route {route}");
        }
        else
        {
            _logger.LogWarning($"No handler found for message type {messageType.Name} on route {route}");
        }
    }

    public void AddRoute<T>(string route, IMessageHandler<T> handler)
    {
        if (!_routes.ContainsKey(route))
        {
            _routes[route] = new Dictionary<Type, object>();
        }
        _routes[route][typeof(T)] = handler;
        _logger.LogInformation($"Added route {route} for message type {typeof(T).Name}");
    }

    public void RemoveRoute<T>(string route)
    {
        if (_routes.ContainsKey(route))
        {
            _routes[route].Remove(typeof(T));
            if (_routes[route].Count == 0)
            {
                _routes.Remove(route);
            }
        }
    }

    private string DetermineRoute<T>(T message)
    {
        // Implement routing logic based on message properties
        if (message is OrderMessage orderMessage)
        {
            return orderMessage.Priority switch
            {
                "high" => "high-priority",
                "low" => "low-priority",
                _ => "normal-priority"
            };
        }

        return "default";
    }
}

// Example Messages
public class OrderMessage
{
    public Guid Id { get; set; }
    public string Priority { get; set; }
    public string CustomerId { get; set; }
    public decimal Amount { get; set; }
}

public class PaymentMessage
{
    public Guid Id { get; set; }
    public string PaymentMethod { get; set; }
    public decimal Amount { get; set; }
}

// Example Handlers
public class HighPriorityOrderHandler : IMessageHandler<OrderMessage>
{
    private readonly ILogger<HighPriorityOrderHandler> _logger;

    public HighPriorityOrderHandler(ILogger<HighPriorityOrderHandler> logger)
    {
        _logger = logger;
    }

    public async Task HandleAsync(OrderMessage message)
    {
        _logger.LogInformation($"Processing high priority order: {message.Id}");
        // Process high priority order
    }
}

public class NormalPriorityOrderHandler : IMessageHandler<OrderMessage>
{
    private readonly ILogger<NormalPriorityOrderHandler> _logger;

    public NormalPriorityOrderHandler(ILogger<NormalPriorityOrderHandler> logger)
    {
        _logger = logger;
    }

    public async Task HandleAsync(OrderMessage message)
    {
        _logger.LogInformation($"Processing normal priority order: {message.Id}");
        // Process normal priority order
    }
}

// Advanced Routing with Multiple Criteria
public class AdvancedMessageRouter : IMessageRouter
{
    private readonly List<RoutingRule> _rules;
    private readonly ILogger<AdvancedMessageRouter> _logger;

    public AdvancedMessageRouter(ILogger<AdvancedMessageRouter> logger)
    {
        _rules = new List<RoutingRule>();
        _logger = logger;
    }

    public async Task RouteAsync<T>(T message)
    {
        var applicableRules = _rules.Where(r => r.Matches(message)).ToList();

        foreach (var rule in applicableRules)
        {
            await rule.Handler.HandleAsync(message);
            _logger.LogInformation($"Applied routing rule {rule.Name} to message");
        }
    }

    public void AddRoute<T>(string route, IMessageHandler<T> handler)
    {
        var rule = new RoutingRule
        {
            Name = route,
            MessageType = typeof(T),
            Handler = handler,
            Condition = _ => true // Default condition
        };
        _rules.Add(rule);
    }

    public void AddConditionalRoute<T>(string route, IMessageHandler<T> handler, Func<T, bool> condition)
    {
        var rule = new RoutingRule
        {
            Name = route,
            MessageType = typeof(T),
            Handler = handler,
            Condition = obj => condition((T)obj)
        };
        _rules.Add(rule);
    }

    public void RemoveRoute<T>(string route)
    {
        _rules.RemoveAll(r => r.Name == route && r.MessageType == typeof(T));
    }
}

public class RoutingRule
{
    public string Name { get; set; }
    public Type MessageType { get; set; }
    public object Handler { get; set; }
    public Func<object, bool> Condition { get; set; }

    public bool Matches(object message)
    {
        return message.GetType() == MessageType && Condition(message);
    }
}

40. How do you implement message persistence?

Message persistence ensures messages are stored reliably and can be recovered.

C# Implementation:

public interface IMessageStore
{
    Task SaveMessageAsync<T>(T message, string queueName);
    Task<T> GetMessageAsync<T>(string queueName);
    Task<IEnumerable<T>> GetMessagesAsync<T>(string queueName, int count);
    Task DeleteMessageAsync(string messageId, string queueName);
    Task<int> GetMessageCountAsync(string queueName);
}

public class PersistentMessage
{
    public string Id { get; set; }
    public string QueueName { get; set; }
    public string MessageType { get; set; }
    public string SerializedMessage { get; set; }
    public DateTime CreatedAt { get; set; }
    public DateTime? ProcessedAt { get; set; }
    public bool IsProcessed { get; set; }
    public int RetryCount { get; set; }
}

public class SqlMessageStore : IMessageStore
{
    private readonly string _connectionString;
    private readonly ILogger<SqlMessageStore> _logger;

    public SqlMessageStore(string connectionString, ILogger<SqlMessageStore> logger)
    {
        _connectionString = connectionString;
        _logger = logger;
    }

    public async Task SaveMessageAsync<T>(T message, string queueName)
    {
        using var connection = new SqlConnection(_connectionString);
        await connection.OpenAsync();

        var persistentMessage = new PersistentMessage
        {
            Id = Guid.NewGuid().ToString(),
            QueueName = queueName,
            MessageType = typeof(T).Name,
            SerializedMessage = JsonSerializer.Serialize(message),
            CreatedAt = DateTime.UtcNow,
            IsProcessed = false,
            RetryCount = 0
        };

        var sql = @"
            INSERT INTO Messages (Id, QueueName, MessageType, SerializedMessage, CreatedAt, IsProcessed, RetryCount)
            VALUES (@Id, @QueueName, @MessageType, @SerializedMessage, @CreatedAt, @IsProcessed, @RetryCount)";

        await connection.ExecuteAsync(sql, persistentMessage);
        _logger.LogInformation($"Saved message {persistentMessage.Id} to queue {queueName}");
    }

    public async Task<T> GetMessageAsync<T>(string queueName)
    {
        using var connection = new SqlConnection(_connectionString);
        await connection.OpenAsync();

        var sql = @"
            SELECT TOP 1 Id, QueueName, MessageType, SerializedMessage, CreatedAt, IsProcessed, RetryCount
            FROM Messages 
            WHERE QueueName = @QueueName AND IsProcessed = 0
            ORDER BY CreatedAt";

        var persistentMessage = await connection.QueryFirstOrDefaultAsync<PersistentMessage>(sql, new { QueueName = queueName });

        if (persistentMessage != null)
        {
            return JsonSerializer.Deserialize<T>(persistentMessage.SerializedMessage);
        }

        return default(T);
    }

    public async Task<IEnumerable<T>> GetMessagesAsync<T>(string queueName, int count)
    {
        using var connection = new SqlConnection(_connectionString);
        await connection.OpenAsync();

        var sql = @"
            SELECT TOP (@Count) Id, QueueName, MessageType, SerializedMessage, CreatedAt, IsProcessed, RetryCount
            FROM Messages 
            WHERE QueueName = @QueueName AND IsProcessed = 0
            ORDER BY CreatedAt";

        var persistentMessages = await connection.QueryAsync<PersistentMessage>(sql, new { QueueName = queueName, Count = count });

        return persistentMessages.Select(pm => JsonSerializer.Deserialize<T>(pm.SerializedMessage));
    }

    public async Task DeleteMessageAsync(string messageId, string queueName)
    {
        using var connection = new SqlConnection(_connectionString);
        await connection.OpenAsync();

        var sql = "DELETE FROM Messages WHERE Id = @Id AND QueueName = @QueueName";
        await connection.ExecuteAsync(sql, new { Id = messageId, QueueName = queueName });

        _logger.LogInformation($"Deleted message {messageId} from queue {queueName}");
    }

    public async Task<int> GetMessageCountAsync(string queueName)
    {
        using var connection = new SqlConnection(_connectionString);
        await connection.OpenAsync();

        var sql = "SELECT COUNT(*) FROM Messages WHERE QueueName = @QueueName AND IsProcessed = 0";
        return await connection.ExecuteScalarAsync<int>(sql, new { QueueName = queueName });
    }

    public async Task MarkAsProcessedAsync(string messageId, string queueName)
    {
        using var connection = new SqlConnection(_connectionString);
        await connection.OpenAsync();

        var sql = @"
            UPDATE Messages 
            SET IsProcessed = 1, ProcessedAt = @ProcessedAt
            WHERE Id = @Id AND QueueName = @QueueName";

        await connection.ExecuteAsync(sql, new { Id = messageId, QueueName = queueName, ProcessedAt = DateTime.UtcNow });
    }

    public async Task IncrementRetryCountAsync(string messageId, string queueName)
    {
        using var connection = new SqlConnection(_connectionString);
        await connection.OpenAsync();

        var sql = "UPDATE Messages SET RetryCount = RetryCount + 1 WHERE Id = @Id AND QueueName = @QueueName";
        await connection.ExecuteAsync(sql, new { Id = messageId, QueueName = queueName });
    }
}

// In-Memory Implementation for Testing
public class InMemoryMessageStore : IMessageStore
{
    private readonly ConcurrentDictionary<string, ConcurrentQueue<PersistentMessage>> _queues;
    private readonly ILogger<InMemoryMessageStore> _logger;

    public InMemoryMessageStore(ILogger<InMemoryMessageStore> logger)
    {
        _queues = new ConcurrentDictionary<string, ConcurrentQueue<PersistentMessage>>();
        _logger = logger;
    }

    public async Task SaveMessageAsync<T>(T message, string queueName)
    {
        var persistentMessage = new PersistentMessage
        {
            Id = Guid.NewGuid().ToString(),
            QueueName = queueName,
            MessageType = typeof(T).Name,
            SerializedMessage = JsonSerializer.Serialize(message),
            CreatedAt = DateTime.UtcNow,
            IsProcessed = false,
            RetryCount = 0
        };

        var queue = _queues.GetOrAdd(queueName, _ => new ConcurrentQueue<PersistentMessage>());
        queue.Enqueue(persistentMessage);

        _logger.LogInformation($"Saved message {persistentMessage.Id} to queue {queueName}");
    }

    public async Task<T> GetMessageAsync<T>(string queueName)
    {
        if (_queues.TryGetValue(queueName, out var queue) && queue.TryDequeue(out var persistentMessage))
        {
            return JsonSerializer.Deserialize<T>(persistentMessage.SerializedMessage);
        }

        return default(T);
    }

    public async Task<IEnumerable<T>> GetMessagesAsync<T>(string queueName, int count)
    {
        var messages = new List<T>();

        if (_queues.TryGetValue(queueName, out var queue))
        {
            for (int i = 0; i < count && queue.TryDequeue(out var persistentMessage); i++)
            {
                messages.Add(JsonSerializer.Deserialize<T>(persistentMessage.SerializedMessage));
            }
        }

        return messages;
    }

    public async Task DeleteMessageAsync(string messageId, string queueName)
    {
        // In-memory implementation doesn't need explicit deletion
        _logger.LogInformation($"Deleted message {messageId} from queue {queueName}");
    }

    public async Task<int> GetMessageCountAsync(string queueName)
    {
        return _queues.TryGetValue(queueName, out var queue) ? queue.Count : 0;
    }
}

// Usage Example
public class PersistentMessageProcessor
{
    private readonly IMessageStore _messageStore;
    private readonly ILogger<PersistentMessageProcessor> _logger;

    public PersistentMessageProcessor(IMessageStore messageStore, ILogger<PersistentMessageProcessor> logger)
    {
        _messageStore = messageStore;
        _logger = logger;
    }

    public async Task ProcessMessagesAsync<T>(string queueName, Func<T, Task> processor)
    {
        while (true)
        {
            var message = await _messageStore.GetMessageAsync<T>(queueName);
            if (message == null)
            {
                await Task.Delay(1000); // Wait before checking again
                continue;
            }

            try
            {
                await processor(message);
                _logger.LogInformation($"Successfully processed message from queue {queueName}");
            }
            catch (Exception ex)
            {
                _logger.LogError($"Error processing message from queue {queueName}: {ex.Message}");
                // Could implement retry logic here
            }
        }
    }
}

These implementations provide comprehensive solutions for building robust, scalable messaging and event-driven systems. Each design pattern addresses specific requirements for reliability, performance, and maintainability in distributed systems.

41. Multi-Level Caching System A multi-level caching system uses multiple cache layers with different characteristics to optimize performance and cost.

• L1 (in-memory), L2 (distributed), L3 (persistent) cache layers
• Automatic population of higher levels from lower levels
• Double-check locking pattern for thread safety

using System;
using System.Collections.Concurrent;
using System.Threading.Tasks;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Caching.Distributed;
using System.Text.Json;

namespace CachingSystem
{
    public interface ICacheLevel
    {
        Task<T> GetAsync<T>(string key);
        Task SetAsync<T>(string key, T value, TimeSpan? expiration = null);
        Task RemoveAsync(string key);
        Task<bool> ExistsAsync(string key);
    }

    public class L1Cache : ICacheLevel
    {
        private readonly IMemoryCache _memoryCache;
        private readonly ConcurrentDictionary<string, SemaphoreSlim> _locks;

        public L1Cache(IMemoryCache memoryCache)
        {
            _memoryCache = memoryCache;
            _locks = new ConcurrentDictionary<string, SemaphoreSlim>();
        }

        public async Task<T> GetAsync<T>(string key)
        {
            if (_memoryCache.TryGetValue(key, out T value))
            {
                return value;
            }
            return default(T);
        }

        public async Task SetAsync<T>(string key, T value, TimeSpan? expiration = null)
        {
            var options = new MemoryCacheEntryOptions();
            if (expiration.HasValue)
            {
                options.AbsoluteExpirationRelativeToNow = expiration;
            }
            _memoryCache.Set(key, value, options);
        }

        public async Task RemoveAsync(string key)
        {
            _memoryCache.Remove(key);
        }

        public async Task<bool> ExistsAsync(string key)
        {
            return _memoryCache.TryGetValue(key, out _);
        }

        public SemaphoreSlim GetLock(string key)
        {
            return _locks.GetOrAdd(key, k => new SemaphoreSlim(1, 1));
        }
    }

    public class L2Cache : ICacheLevel
    {
        private readonly IDistributedCache _distributedCache;

        public L2Cache(IDistributedCache distributedCache)
        {
            _distributedCache = distributedCache;
        }

        public async Task<T> GetAsync<T>(string key)
        {
            var value = await _distributedCache.GetStringAsync(key);
            if (value != null)
            {
                return JsonSerializer.Deserialize<T>(value);
            }
            return default(T);
        }

        public async Task SetAsync<T>(string key, T value, TimeSpan? expiration = null)
        {
            var options = new DistributedCacheEntryOptions();
            if (expiration.HasValue)
            {
                options.AbsoluteExpirationRelativeToNow = expiration;
            }
            var serializedValue = JsonSerializer.Serialize(value);
            await _distributedCache.SetStringAsync(key, serializedValue, options);
        }

        public async Task RemoveAsync(string key)
        {
            await _distributedCache.RemoveAsync(key);
        }

        public async Task<bool> ExistsAsync(string key)
        {
            var value = await _distributedCache.GetStringAsync(key);
            return value != null;
        }
    }

    public class L3Cache : ICacheLevel
    {
        private readonly IDatabase _database; // Could be Redis, SQL Server, etc.

        public L3Cache(IDatabase database)
        {
            _database = database;
        }

        public async Task<T> GetAsync<T>(string key)
        {
            var value = await _database.GetAsync(key);
            if (value != null)
            {
                return JsonSerializer.Deserialize<T>(value);
            }
            return default(T);
        }

        public async Task SetAsync<T>(string key, T value, TimeSpan? expiration = null)
        {
            var serializedValue = JsonSerializer.Serialize(value);
            await _database.SetAsync(key, serializedValue, expiration);
        }

        public async Task RemoveAsync(string key)
        {
            await _database.RemoveAsync(key);
        }

        public async Task<bool> ExistsAsync(string key)
        {
            return await _database.ExistsAsync(key);
        }
    }

    public class MultiLevelCache
    {
        private readonly L1Cache _l1Cache;
        private readonly L2Cache _l2Cache;
        private readonly L3Cache _l3Cache;
        private readonly IDataProvider _dataProvider;

        public MultiLevelCache(L1Cache l1Cache, L2Cache l2Cache, L3Cache l3Cache, IDataProvider dataProvider)
        {
            _l1Cache = l1Cache;
            _l2Cache = l2Cache;
            _l3Cache = l3Cache;
            _dataProvider = dataProvider;
        }

        public async Task<T> GetAsync<T>(string key)
        {
            // L1 Cache Check (Fastest)
            var l1Value = await _l1Cache.GetAsync<T>(key);
            if (l1Value != null)
            {
                return l1Value;
            }

            // L2 Cache Check
            var l2Value = await _l2Cache.GetAsync<T>(key);
            if (l2Value != null)
            {
                // Populate L1 cache
                await _l1Cache.SetAsync(key, l2Value, TimeSpan.FromMinutes(5));
                return l2Value;
            }

            // L3 Cache Check
            var l3Value = await _l3Cache.GetAsync<T>(key);
            if (l3Value != null)
            {
                // Populate L2 and L1 caches
                await _l2Cache.SetAsync(key, l3Value, TimeSpan.FromHours(1));
                await _l1Cache.SetAsync(key, l3Value, TimeSpan.FromMinutes(5));
                return l3Value;
            }

            // Data Provider (Slowest)
            var lockObj = _l1Cache.GetLock(key);
            await lockObj.WaitAsync();
            try
            {
                // Double-check pattern
                var doubleCheckValue = await _l1Cache.GetAsync<T>(key);
                if (doubleCheckValue != null)
                {
                    return doubleCheckValue;
                }

                var data = await _dataProvider.GetDataAsync<T>(key);
                if (data != null)
                {
                    // Populate all cache levels
                    await _l3Cache.SetAsync(key, data, TimeSpan.FromDays(1));
                    await _l2Cache.SetAsync(key, data, TimeSpan.FromHours(1));
                    await _l1Cache.SetAsync(key, data, TimeSpan.FromMinutes(5));
                }
                return data;
            }
            finally
            {
                lockObj.Release();
            }
        }

        public async Task SetAsync<T>(string key, T value, TimeSpan? expiration = null)
        {
            // Update all cache levels
            await _l1Cache.SetAsync(key, value, expiration);
            await _l2Cache.SetAsync(key, value, expiration);
            await _l3Cache.SetAsync(key, value, expiration);
        }

        public async Task InvalidateAsync(string key)
        {
            // Remove from all cache levels
            await _l1Cache.RemoveAsync(key);
            await _l2Cache.RemoveAsync(key);
            await _l3Cache.RemoveAsync(key);
        }
    }

    // Interfaces for demonstration
    public interface IDatabase
    {
        Task<string> GetAsync(string key);
        Task SetAsync(string key, string value, TimeSpan? expiration = null);
        Task RemoveAsync(string key);
        Task<bool> ExistsAsync(string key);
    }

    public interface IDataProvider
    {
        Task<T> GetDataAsync<T>(string key);
    }
}

42. Cache Invalidation Strategies

• TTL (Time To Live) with automatic cleanup
• LRU (Least Recently Used) with capacity management
• Write-through and write-behind patterns
• Event-driven invalidation with callbacks

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using System.Timers;
using Microsoft.Extensions.Logging;

namespace CachingSystem
{
    public enum InvalidationStrategy
    {
        TTL,           // Time To Live
        LRU,           // Least Recently Used
        WriteThrough,  // Write-through
        WriteBehind,   // Write-behind
        EventDriven    // Event-driven invalidation
    }

    public interface ICacheInvalidationStrategy
    {
        Task InvalidateAsync(string key);
        Task InvalidatePatternAsync(string pattern);
        Task InvalidateAllAsync();
        void RegisterInvalidationCallback(Action<string> callback);
    }

    public class TTLInvalidationStrategy : ICacheInvalidationStrategy
    {
        private readonly ConcurrentDictionary<string, DateTime> _expirationTimes;
        private readonly Timer _cleanupTimer;
        private readonly ILogger<TTLInvalidationStrategy> _logger;

        public TTLInvalidationStrategy(ILogger<TTLInvalidationStrategy> logger)
        {
            _expirationTimes = new ConcurrentDictionary<string, DateTime>();
            _logger = logger;

            // Cleanup timer runs every minute
            _cleanupTimer = new Timer(60000);
            _cleanupTimer.Elapsed += CleanupExpiredItems;
            _cleanupTimer.Start();
        }

        public void SetExpiration(string key, TimeSpan ttl)
        {
            var expirationTime = DateTime.UtcNow.Add(ttl);
            _expirationTimes.AddOrUpdate(key, expirationTime, (k, v) => expirationTime);
        }

        public bool IsExpired(string key)
        {
            if (_expirationTimes.TryGetValue(key, out var expirationTime))
            {
                return DateTime.UtcNow > expirationTime;
            }
            return false;
        }

        private void CleanupExpiredItems(object sender, ElapsedEventArgs e)
        {
            var expiredKeys = _expirationTimes
                .Where(kvp => DateTime.UtcNow > kvp.Value)
                .Select(kvp => kvp.Key)
                .ToList();

            foreach (var key in expiredKeys)
            {
                _expirationTimes.TryRemove(key, out _);
                _logger.LogInformation($"TTL expired for key: {key}");
            }
        }

        public async Task InvalidateAsync(string key)
        {
            _expirationTimes.TryRemove(key, out _);
            await Task.CompletedTask;
        }

        public async Task InvalidatePatternAsync(string pattern)
        {
            var keysToRemove = _expirationTimes.Keys
                .Where(key => key.Contains(pattern))
                .ToList();

            foreach (var key in keysToRemove)
            {
                _expirationTimes.TryRemove(key, out _);
            }
            await Task.CompletedTask;
        }

        public async Task InvalidateAllAsync()
        {
            _expirationTimes.Clear();
            await Task.CompletedTask;
        }

        public void RegisterInvalidationCallback(Action<string> callback)
        {
            // TTL doesn't use callbacks, it's time-based
        }
    }

    public class LRUInvalidationStrategy : ICacheInvalidationStrategy
    {
        private readonly int _maxSize;
        private readonly LinkedList<string> _accessOrder;
        private readonly Dictionary<string, LinkedListNode<string>> _keyToNode;
        private readonly object _lock = new object();

        public LRUInvalidationStrategy(int maxSize = 1000)
        {
            _maxSize = maxSize;
            _accessOrder = new LinkedList<string>();
            _keyToNode = new Dictionary<string, LinkedListNode<string>>();
        }

        public void Access(string key)
        {
            lock (_lock)
            {
                if (_keyToNode.TryGetValue(key, out var node))
                {
                    _accessOrder.Remove(node);
                    _accessOrder.AddFirst(node);
                }
                else
                {
                    var newNode = new LinkedListNode<string>(key);
                    _keyToNode[key] = newNode;
                    _accessOrder.AddFirst(newNode);

                    // Evict least recently used if capacity exceeded
                    if (_accessOrder.Count > _maxSize)
                    {
                        var lruNode = _accessOrder.Last;
                        _accessOrder.RemoveLast();
                        _keyToNode.Remove(lruNode.Value);
                    }
                }
            }
        }

        public List<string> GetLRUKeys(int count)
        {
            lock (_lock)
            {
                return _accessOrder.TakeLast(count).ToList();
            }
        }

        public async Task InvalidateAsync(string key)
        {
            lock (_lock)
            {
                if (_keyToNode.TryGetValue(key, out var node))
                {
                    _accessOrder.Remove(node);
                    _keyToNode.Remove(key);
                }
            }
            await Task.CompletedTask;
        }

        public async Task InvalidatePatternAsync(string pattern)
        {
            lock (_lock)
            {
                var keysToRemove = _keyToNode.Keys
                    .Where(key => key.Contains(pattern))
                    .ToList();

                foreach (var key in keysToRemove)
                {
                    if (_keyToNode.TryGetValue(key, out var node))
                    {
                        _accessOrder.Remove(node);
                        _keyToNode.Remove(key);
                    }
                }
            }
            await Task.CompletedTask;
        }

        public async Task InvalidateAllAsync()
        {
            lock (_lock)
            {
                _accessOrder.Clear();
                _keyToNode.Clear();
            }
            await Task.CompletedTask;
        }

        public void RegisterInvalidationCallback(Action<string> callback)
        {
            // LRU doesn't use callbacks, it's capacity-based
        }
    }

    public class WriteThroughInvalidationStrategy : ICacheInvalidationStrategy
    {
        private readonly ICache _cache;
        private readonly IDataStore _dataStore;
        private readonly ILogger<WriteThroughInvalidationStrategy> _logger;

        public WriteThroughInvalidationStrategy(ICache cache, IDataStore dataStore, ILogger<WriteThroughInvalidationStrategy> logger)
        {
            _cache = cache;
            _dataStore = dataStore;
            _logger = logger;
        }

        public async Task WriteThroughAsync<T>(string key, T value)
        {
            // Write to both cache and data store simultaneously
            var cacheTask = _cache.SetAsync(key, value);
            var dataStoreTask = _dataStore.SetAsync(key, value);

            await Task.WhenAll(cacheTask, dataStoreTask);
            _logger.LogInformation($"Write-through completed for key: {key}");
        }

        public async Task InvalidateAsync(string key)
        {
            await _cache.RemoveAsync(key);
            await _dataStore.RemoveAsync(key);
        }

        public async Task InvalidatePatternAsync(string pattern)
        {
            var keys = await _cache.GetKeysByPatternAsync(pattern);
            foreach (var key in keys)
            {
                await InvalidateAsync(key);
            }
        }

        public async Task InvalidateAllAsync()
        {
            await _cache.ClearAsync();
            await _dataStore.ClearAsync();
        }

        public void RegisterInvalidationCallback(Action<string> callback)
        {
            // Write-through doesn't use callbacks, it's synchronous
        }
    }

    public class WriteBehindInvalidationStrategy : ICacheInvalidationStrategy
    {
        private readonly ICache _cache;
        private readonly IDataStore _dataStore;
        private readonly Queue<WriteOperation> _writeQueue;
        private readonly Timer _flushTimer;
        private readonly object _lock = new object();
        private readonly ILogger<WriteBehindInvalidationStrategy> _logger;

        public WriteBehindInvalidationStrategy(ICache cache, IDataStore dataStore, ILogger<WriteBehindInvalidationStrategy> logger)
        {
            _cache = cache;
            _dataStore = dataStore;
            _logger = logger;
            _writeQueue = new Queue<WriteOperation>();

            // Flush queue every 5 seconds
            _flushTimer = new Timer(5000);
            _flushTimer.Elapsed += FlushWriteQueue;
            _flushTimer.Start();
        }

        public async Task WriteBehindAsync<T>(string key, T value)
        {
            // Write to cache immediately
            await _cache.SetAsync(key, value);

            // Queue write to data store
            lock (_lock)
            {
                _writeQueue.Enqueue(new WriteOperation { Key = key, Value = value, Timestamp = DateTime.UtcNow });
            }
        }

        private async void FlushWriteQueue(object sender, ElapsedEventArgs e)
        {
            List<WriteOperation> operationsToProcess;
            lock (_lock)
            {
                operationsToProcess = _writeQueue.ToList();
                _writeQueue.Clear();
            }

            foreach (var operation in operationsToProcess)
            {
                try
                {
                    await _dataStore.SetAsync(operation.Key, operation.Value);
                    _logger.LogInformation($"Write-behind flushed for key: {operation.Key}");
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, $"Failed to flush write-behind for key: {operation.Key}");
                    // Re-queue failed operations
                    lock (_lock)
                    {
                        _writeQueue.Enqueue(operation);
                    }
                }
            }
        }

        public async Task InvalidateAsync(string key)
        {
            await _cache.RemoveAsync(key);
            // Remove from write queue
            lock (_lock)
            {
                var remainingOperations = _writeQueue.Where(op => op.Key != key).ToList();
                _writeQueue.Clear();
                foreach (var op in remainingOperations)
                {
                    _writeQueue.Enqueue(op);
                }
            }
        }

        public async Task InvalidatePatternAsync(string pattern)
        {
            var keys = await _cache.GetKeysByPatternAsync(pattern);
            foreach (var key in keys)
            {
                await InvalidateAsync(key);
            }
        }

        public async Task InvalidateAllAsync()
        {
            await _cache.ClearAsync();
            lock (_lock)
            {
                _writeQueue.Clear();
            }
        }

        public void RegisterInvalidationCallback(Action<string> callback)
        {
            // Write-behind doesn't use callbacks, it's queue-based
        }
    }

    public class EventDrivenInvalidationStrategy : ICacheInvalidationStrategy
    {
        private readonly ICache _cache;
        private readonly IEventBus _eventBus;
        private readonly Dictionary<string, List<Action<string>>> _callbacks;
        private readonly ILogger<EventDrivenInvalidationStrategy> _logger;

        public EventDrivenInvalidationStrategy(ICache cache, IEventBus eventBus, ILogger<EventDrivenInvalidationStrategy> logger)
        {
            _cache = cache;
            _eventBus = eventBus;
            _logger = logger;
            _callbacks = new Dictionary<string, List<Action<string>>>();

            // Subscribe to invalidation events
            _eventBus.Subscribe<CacheInvalidationEvent>(HandleInvalidationEvent);
        }

        public async Task PublishInvalidationEventAsync(string key, string reason = "manual")
        {
            var invalidationEvent = new CacheInvalidationEvent
            {
                Key = key,
                Reason = reason,
                Timestamp = DateTime.UtcNow
            };

            await _eventBus.PublishAsync(invalidationEvent);
        }

        private async Task HandleInvalidationEvent(CacheInvalidationEvent invalidationEvent)
        {
            _logger.LogInformation($"Handling invalidation event for key: {invalidationEvent.Key}, reason: {invalidationEvent.Reason}");

            // Remove from cache
            await _cache.RemoveAsync(invalidationEvent.Key);

            // Execute callbacks
            if (_callbacks.TryGetValue(invalidationEvent.Key, out var keyCallbacks))
            {
                foreach (var callback in keyCallbacks)
                {
                    try
                    {
                        callback(invalidationEvent.Key);
                    }
                    catch (Exception ex)
                    {
                        _logger.LogError(ex, $"Error executing callback for key: {invalidationEvent.Key}");
                    }
                }
            }

            // Execute pattern callbacks
            var patternCallbacks = _callbacks
                .Where(kvp => kvp.Key.Contains("*") && IsPatternMatch(invalidationEvent.Key, kvp.Key))
                .SelectMany(kvp => kvp.Value);

            foreach (var callback in patternCallbacks)
            {
                try
                {
                    callback(invalidationEvent.Key);
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, $"Error executing pattern callback for key: {invalidationEvent.Key}");
                }
            }
        }

        private bool IsPatternMatch(string key, string pattern)
        {
            // Simple wildcard pattern matching
            return pattern.Replace("*", "").Contains(key) || key.Contains(pattern.Replace("*", ""));
        }

        public async Task InvalidateAsync(string key)
        {
            await PublishInvalidationEventAsync(key, "manual");
        }

        public async Task InvalidatePatternAsync(string pattern)
        {
            var keys = await _cache.GetKeysByPatternAsync(pattern);
            foreach (var key in keys)
            {
                await InvalidateAsync(key);
            }
        }

        public async Task InvalidateAllAsync()
        {
            await PublishInvalidationEventAsync("*", "clear_all");
        }

        public void RegisterInvalidationCallback(Action<string> callback, string key = "*")
        {
            if (!_callbacks.ContainsKey(key))
            {
                _callbacks[key] = new List<Action<string>>();
            }
            _callbacks[key].Add(callback);
        }
    }

    // Supporting classes
    public class WriteOperation
    {
        public string Key { get; set; }
        public object Value { get; set; }
        public DateTime Timestamp { get; set; }
    }

    public class CacheInvalidationEvent
    {
        public string Key { get; set; }
        public string Reason { get; set; }
        public DateTime Timestamp { get; set; }
    }

    // Interfaces for demonstration
    public interface ICache
    {
        Task<T> GetAsync<T>(string key);
        Task SetAsync<T>(string key, T value);
        Task RemoveAsync(string key);
        Task ClearAsync();
        Task<List<string>> GetKeysByPatternAsync(string pattern);
    }

    public interface IDataStore
    {
        Task<T> GetAsync<T>(string key);
        Task SetAsync<T>(string key, T value);
        Task RemoveAsync(string key);
        Task ClearAsync();
    }

    public interface IEventBus
    {
        Task PublishAsync<T>(T eventData);
        void Subscribe<T>(Func<T, Task> handler);
    }
}

43. Distributed Cache

• Consistent hashing with virtual nodes
• Node health monitoring and failover
• Replication for fault tolerance
• HTTP-based node communication

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Threading.Tasks;
using System.Text.Json;
using Microsoft.Extensions.Logging;

namespace CachingSystem
{
    public class DistributedCacheNode
    {
        public string Id { get; set; }
        public string Host { get; set; }
        public int Port { get; set; }
        public bool IsActive { get; set; } = true;
        public DateTime LastHeartbeat { get; set; }
        public int VirtualNodes { get; set; } = 150; // Number of virtual nodes for consistent hashing

        public string Endpoint => $"{Host}:{Port}";

        public override string ToString()
        {
            return $"{Id} ({Endpoint}) - Active: {IsActive}";
        }
    }

    public class CacheEntry<T>
    {
        public string Key { get; set; }
        public T Value { get; set; }
        public DateTime CreatedAt { get; set; }
        public DateTime? ExpiresAt { get; set; }
        public int Version { get; set; } = 1;
        public List<string> ReplicaNodes { get; set; } = new List<string>();

        public bool IsExpired => ExpiresAt.HasValue && DateTime.UtcNow > ExpiresAt.Value;
    }

    public class ConsistentHashRing
    {
        private readonly SortedDictionary<uint, string> _hashRing;
        private readonly Dictionary<string, List<uint>> _nodeToHashes;
        private readonly object _lock = new object();

        public ConsistentHashRing()
        {
            _hashRing = new SortedDictionary<uint, string>();
            _nodeToHashes = new Dictionary<string, List<uint>>();
        }

        public void AddNode(DistributedCacheNode node)
        {
            lock (_lock)
            {
                var hashes = new List<uint>();
                for (int i = 0; i < node.VirtualNodes; i++)
                {
                    var virtualNodeKey = $"{node.Id}-{i}";
                    var hash = GetHash(virtualNodeKey);
                    _hashRing[hash] = node.Id;
                    hashes.Add(hash);
                }
                _nodeToHashes[node.Id] = hashes;
            }
        }

        public void RemoveNode(string nodeId)
        {
            lock (_lock)
            {
                if (_nodeToHashes.TryGetValue(nodeId, out var hashes))
                {
                    foreach (var hash in hashes)
                    {
                        _hashRing.Remove(hash);
                    }
                    _nodeToHashes.Remove(nodeId);
                }
            }
        }

        public string GetNodeForKey(string key)
        {
            lock (_lock)
            {
                if (_hashRing.Count == 0)
                    return null;

                var hash = GetHash(key);
                var node = _hashRing.FirstOrDefault(kvp => kvp.Key >= hash);

                if (node.Key == 0) // No node found with hash >= key hash
                {
                    // Wrap around to the first node
                    node = _hashRing.First();
                }

                return node.Value;
            }
        }

        public List<string> GetReplicaNodes(string key, int replicaCount)
        {
            lock (_lock)
            {
                var nodes = new List<string>();
                var hash = GetHash(key);
                var sortedNodes = _hashRing.OrderBy(kvp => kvp.Key).ToList();

                // Find the primary node
                var primaryNodeIndex = sortedNodes.FindIndex(kvp => kvp.Key >= hash);
                if (primaryNodeIndex == -1)
                    primaryNodeIndex = 0;

                // Add primary and replica nodes
                for (int i = 0; i < replicaCount && i < sortedNodes.Count; i++)
                {
                    var index = (primaryNodeIndex + i) % sortedNodes.Count;
                    var nodeId = sortedNodes[index].Value;
                    if (!nodes.Contains(nodeId))
                    {
                        nodes.Add(nodeId);
                    }
                }

                return nodes;
            }
        }

        private uint GetHash(string key)
        {
            // Simple hash function - in production, use a better one like MurmurHash
            uint hash = 0;
            foreach (char c in key)
            {
                hash = ((hash << 5) + hash) + c;
            }
            return hash;
        }
    }

    public class DistributedCache
    {
        private readonly ConsistentHashRing _hashRing;
        private readonly Dictionary<string, DistributedCacheNode> _nodes;
        private readonly ConcurrentDictionary<string, CacheEntry<object>> _localCache;
        private readonly ILogger<DistributedCache> _logger;
        private readonly int _replicaCount;
        private readonly Timer _heartbeatTimer;
        private readonly Timer _cleanupTimer;

        public DistributedCache(ILogger<DistributedCache> logger, int replicaCount = 2)
        {
            _hashRing = new ConsistentHashRing();
            _nodes = new Dictionary<string, DistributedCacheNode>();
            _localCache = new ConcurrentDictionary<string, CacheEntry<object>>();
            _logger = logger;
            _replicaCount = replicaCount;

            // Heartbeat timer to check node health
            _heartbeatTimer = new Timer(30000); // 30 seconds
            _heartbeatTimer.Elapsed += CheckNodeHealth;
            _heartbeatTimer.Start();

            // Cleanup timer for expired entries
            _cleanupTimer = new Timer(60000); // 1 minute
            _cleanupTimer.Elapsed += CleanupExpiredEntries;
            _cleanupTimer.Start();
        }

        public void AddNode(DistributedCacheNode node)
        {
            lock (_nodes)
            {
                _nodes[node.Id] = node;
                _hashRing.AddNode(node);
                _logger.LogInformation($"Added node: {node}");
            }
        }

        public void RemoveNode(string nodeId)
        {
            lock (_nodes)
            {
                if (_nodes.Remove(nodeId))
                {
                    _hashRing.RemoveNode(nodeId);
                    _logger.LogInformation($"Removed node: {nodeId}");
                }
            }
        }

        public async Task<T> GetAsync<T>(string key)
        {
            // Check local cache first
            if (_localCache.TryGetValue(key, out var localEntry) && !localEntry.IsExpired)
            {
                return (T)localEntry.Value;
            }

            // Get the primary node for this key
            var primaryNodeId = _hashRing.GetNodeForKey(key);
            if (string.IsNullOrEmpty(primaryNodeId))
            {
                throw new InvalidOperationException("No available nodes in the cluster");
            }

            // Try to get from primary node
            var value = await GetFromNodeAsync<T>(primaryNodeId, key);
            if (value != null)
            {
                // Cache locally
                var entry = new CacheEntry<object>
                {
                    Key = key,
                    Value = value,
                    CreatedAt = DateTime.UtcNow,
                    ExpiresAt = DateTime.UtcNow.AddMinutes(5) // Local cache TTL
                };
                _localCache.AddOrUpdate(key, entry, (k, v) => entry);
                return value;
            }

            // Try replica nodes if primary failed
            var replicaNodes = _hashRing.GetReplicaNodes(key, _replicaCount);
            foreach (var replicaNodeId in replicaNodes)
            {
                if (replicaNodeId != primaryNodeId)
                {
                    value = await GetFromNodeAsync<T>(replicaNodeId, key);
                    if (value != null)
                    {
                        return value;
                    }
                }
            }

            return default(T);
        }

        public async Task SetAsync<T>(string key, T value, TimeSpan? expiration = null)
        {
            var replicaNodes = _hashRing.GetReplicaNodes(key, _replicaCount);
            var tasks = new List<Task>();

            // Write to all replica nodes
            foreach (var nodeId in replicaNodes)
            {
                tasks.Add(SetToNodeAsync(nodeId, key, value, expiration));
            }

            // Wait for at least one write to succeed
            await Task.WhenAny(tasks);

            // Update local cache
            var entry = new CacheEntry<object>
            {
                Key = key,
                Value = value,
                CreatedAt = DateTime.UtcNow,
                ExpiresAt = expiration.HasValue ? DateTime.UtcNow.Add(expiration.Value) : null
            };
            _localCache.AddOrUpdate(key, entry, (k, v) => entry);
        }

        public async Task RemoveAsync(string key)
        {
            var replicaNodes = _hashRing.GetReplicaNodes(key, _replicaCount);
            var tasks = new List<Task>();

            // Remove from all replica nodes
            foreach (var nodeId in replicaNodes)
            {
                tasks.Add(RemoveFromNodeAsync(nodeId, key));
            }

            await Task.WhenAll(tasks);

            // Remove from local cache
            _localCache.TryRemove(key, out _);
        }

        private async Task<T> GetFromNodeAsync<T>(string nodeId, string key)
        {
            try
            {
                if (!_nodes.TryGetValue(nodeId, out var node) || !node.IsActive)
                {
                    return default(T);
                }

                // Simulate network call to the node
                await Task.Delay(10); // Simulate network latency

                // In a real implementation, this would be an HTTP call or TCP connection
                // For now, we'll simulate a successful response
                return default(T);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Failed to get key {key} from node {nodeId}");
                return default(T);
            }
        }

        private async Task SetToNodeAsync<T>(string nodeId, string key, T value, TimeSpan? expiration)
        {
            try
            {
                if (!_nodes.TryGetValue(nodeId, out var node) || !node.IsActive)
                {
                    return;
                }

                // Simulate network call to the node
                await Task.Delay(10); // Simulate network latency

                // In a real implementation, this would be an HTTP call or TCP connection
                _logger.LogDebug($"Set key {key} to node {nodeId}");
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Failed to set key {key} to node {nodeId}");
            }
        }

        private async Task RemoveFromNodeAsync(string nodeId, string key)
        {
            try
            {
                if (!_nodes.TryGetValue(nodeId, out var node) || !node.IsActive)
                {
                    return;
                }

                // Simulate network call to the node
                await Task.Delay(10); // Simulate network latency

                // In a real implementation, this would be an HTTP call or TCP connection
                _logger.LogDebug($"Removed key {key} from node {nodeId}");
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Failed to remove key {key} from node {nodeId}");
            }
        }

        private void CheckNodeHealth(object sender, ElapsedEventArgs e)
        {
            var now = DateTime.UtcNow;
            var inactiveNodes = new List<string>();

            lock (_nodes)
            {
                foreach (var node in _nodes.Values)
                {
                    // Consider node inactive if no heartbeat for 2 minutes
                    if (now - node.LastHeartbeat > TimeSpan.FromMinutes(2))
                    {
                        node.IsActive = false;
                        inactiveNodes.Add(node.Id);
                        _logger.LogWarning($"Node {node.Id} marked as inactive");
                    }
                }
            }

            // Remove inactive nodes from hash ring
            foreach (var nodeId in inactiveNodes)
            {
                _hashRing.RemoveNode(nodeId);
            }
        }

        private void CleanupExpiredEntries(object sender, ElapsedEventArgs e)
        {
            var expiredKeys = _localCache
                .Where(kvp => kvp.Value.IsExpired)
                .Select(kvp => kvp.Key)
                .ToList();

            foreach (var key in expiredKeys)
            {
                _localCache.TryRemove(key, out _);
            }

            if (expiredKeys.Count > 0)
            {
                _logger.LogDebug($"Cleaned up {expiredKeys.Count} expired entries");
            }
        }

        public void UpdateNodeHeartbeat(string nodeId)
        {
            lock (_nodes)
            {
                if (_nodes.TryGetValue(nodeId, out var node))
                {
                    node.LastHeartbeat = DateTime.UtcNow;
                    if (!node.IsActive)
                    {
                        node.IsActive = true;
                        _hashRing.AddNode(node);
                        _logger.LogInformation($"Node {nodeId} reactivated");
                    }
                }
            }
        }

        public List<DistributedCacheNode> GetActiveNodes()
        {
            lock (_nodes)
            {
                return _nodes.Values.Where(n => n.IsActive).ToList();
            }
        }

        public Dictionary<string, object> GetClusterStats()
        {
            lock (_nodes)
            {
                return new Dictionary<string, object>
                {
                    ["TotalNodes"] = _nodes.Count,
                    ["ActiveNodes"] = _nodes.Values.Count(n => n.IsActive),
                    ["LocalCacheSize"] = _localCache.Count,
                    ["ReplicaCount"] = _replicaCount
                };
            }
        }
    }

    // Node communication interface
    public interface INodeCommunication
    {
        Task<T> GetAsync<T>(string nodeId, string key);
        Task SetAsync<T>(string nodeId, string key, T value, TimeSpan? expiration = null);
        Task RemoveAsync(string nodeId, string key);
        Task<bool> PingAsync(string nodeId);
    }

    // HTTP-based node communication implementation
    public class HttpNodeCommunication : INodeCommunication
    {
        private readonly HttpClient _httpClient;
        private readonly Dictionary<string, string> _nodeEndpoints;
        private readonly ILogger<HttpNodeCommunication> _logger;

        public HttpNodeCommunication(ILogger<HttpNodeCommunication> logger)
        {
            _httpClient = new HttpClient();
            _nodeEndpoints = new Dictionary<string, string>();
            _logger = logger;
        }

        public void RegisterNode(string nodeId, string endpoint)
        {
            _nodeEndpoints[nodeId] = endpoint;
        }

        public async Task<T> GetAsync<T>(string nodeId, string key)
        {
            if (!_nodeEndpoints.TryGetValue(nodeId, out var endpoint))
            {
                throw new ArgumentException($"Node {nodeId} not registered");
            }

            try
            {
                var response = await _httpClient.GetAsync($"{endpoint}/cache/{key}");
                if (response.IsSuccessStatusCode)
                {
                    var content = await response.Content.ReadAsStringAsync();
                    return JsonSerializer.Deserialize<T>(content);
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Failed to get key {key} from node {nodeId}");
            }

            return default(T);
        }

        public async Task SetAsync<T>(string nodeId, string key, T value, TimeSpan? expiration = null)
        {
            if (!_nodeEndpoints.TryGetValue(nodeId, out var endpoint))
            {
                throw new ArgumentException($"Node {nodeId} not registered");
            }

            try
            {
                var request = new
                {
                    Key = key,
                    Value = value,
                    ExpirationSeconds = expiration?.TotalSeconds
                };

                var json = JsonSerializer.Serialize(request);
                var content = new StringContent(json, System.Text.Encoding.UTF8, "application/json");

                var response = await _httpClient.PostAsync($"{endpoint}/cache", content);
                if (!response.IsSuccessStatusCode)
                {
                    _logger.LogError($"Failed to set key {key} to node {nodeId}: {response.StatusCode}");
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Failed to set key {key} to node {nodeId}");
            }
        }

        public async Task RemoveAsync(string nodeId, string key)
        {
            if (!_nodeEndpoints.TryGetValue(nodeId, out var endpoint))
            {
                throw new ArgumentException($"Node {nodeId} not registered");
            }

            try
            {
                var response = await _httpClient.DeleteAsync($"{endpoint}/cache/{key}");
                if (!response.IsSuccessStatusCode)
                {
                    _logger.LogError($"Failed to remove key {key} from node {nodeId}: {response.StatusCode}");
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Failed to remove key {key} from node {nodeId}");
            }
        }

        public async Task<bool> PingAsync(string nodeId)
        {
            if (!_nodeEndpoints.TryGetValue(nodeId, out var endpoint))
            {
                return false;
            }

            try
            {
                var response = await _httpClient.GetAsync($"{endpoint}/health");
                return response.IsSuccessStatusCode;
            }
            catch
            {
                return false;
            }
        }
    }
}

44. Cache Warming

• Predictive warming based on access patterns
• Scheduled warming with time-based triggers
• Event-driven warming for specific scenarios
• Progressive warming with priority queues

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using System.Timers;
using Microsoft.Extensions.Logging;
using System.Collections.Concurrent;

namespace CachingSystem
{
    public enum WarmingStrategy
    {
        Predictive,    // Based on usage patterns and predictions
        Scheduled,     // Time-based warming
        EventDriven,   // Triggered by specific events
        Progressive    // Gradual warming based on priority
    }

    public interface ICacheWarmingStrategy
    {
        Task WarmCacheAsync();
        Task WarmSpecificKeysAsync(List<string> keys);
        void RegisterWarmingCallback(Func<string, Task> callback);
    }

    public class WarmingItem
    {
        public string Key { get; set; }
        public int Priority { get; set; } // 1 = highest priority
        public DateTime LastAccessed { get; set; }
        public int AccessCount { get; set; }
        public TimeSpan EstimatedExpiration { get; set; }
        public WarmingStrategy Strategy { get; set; }
    }

    public class PredictiveWarmingStrategy : ICacheWarmingStrategy
    {
        private readonly ICache _cache;
        private readonly IDataProvider _dataProvider;
        private readonly ILogger<PredictiveWarmingStrategy> _logger;
        private readonly ConcurrentDictionary<string, WarmingItem> _accessPatterns;
        private readonly Timer _warmingTimer;
        private readonly int _maxWarmingItems;
        private Func<string, Task> _warmingCallback;

        public PredictiveWarmingStrategy(
            ICache cache, 
            IDataProvider dataProvider, 
            ILogger<PredictiveWarmingStrategy> logger,
            int maxWarmingItems = 1000)
        {
            _cache = cache;
            _dataProvider = dataProvider;
            _logger = logger;
            _maxWarmingItems = maxWarmingItems;
            _accessPatterns = new ConcurrentDictionary<string, WarmingItem>();

            // Run predictive warming every 5 minutes
            _warmingTimer = new Timer(300000);
            _warmingTimer.Elapsed += async (sender, e) => await WarmCacheAsync();
            _warmingTimer.Start();
        }

        public void RecordAccess(string key, DateTime accessTime)
        {
            var item = _accessPatterns.GetOrAdd(key, k => new WarmingItem
            {
                Key = k,
                Priority = 1,
                LastAccessed = accessTime,
                AccessCount = 0,
                EstimatedExpiration = TimeSpan.FromHours(1),
                Strategy = WarmingStrategy.Predictive
            });

            item.AccessCount++;
            item.LastAccessed = accessTime;

            // Adjust priority based on access frequency and recency
            UpdatePriority(item);
        }

        private void UpdatePriority(WarmingItem item)
        {
            var timeSinceLastAccess = DateTime.UtcNow - item.LastAccessed;
            var frequencyScore = Math.Min(item.AccessCount / 10.0, 10.0); // Cap at 10
            var recencyScore = Math.Max(0, 10 - timeSinceLastAccess.TotalHours);

            item.Priority = (int)(frequencyScore + recencyScore);
        }

        public async Task WarmCacheAsync()
        {
            try
            {
                _logger.LogInformation("Starting predictive cache warming");

                // Get top priority items for warming
                var itemsToWarm = _accessPatterns.Values
                    .OrderBy(item => item.Priority)
                    .Take(_maxWarmingItems)
                    .ToList();

                var warmingTasks = new List<Task>();
                var semaphore = new SemaphoreSlim(10, 10); // Limit concurrent warming operations

                foreach (var item in itemsToWarm)
                {
                    await semaphore.WaitAsync();
                    warmingTasks.Add(Task.Run(async () =>
                    {
                        try
                        {
                            await WarmItemAsync(item);
                        }
                        finally
                        {
                            semaphore.Release();
                        }
                    }));
                }

                await Task.WhenAll(warmingTasks);
                _logger.LogInformation($"Completed predictive warming for {itemsToWarm.Count} items");
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error during predictive cache warming");
            }
        }

        private async Task WarmItemAsync(WarmingItem item)
        {
            try
            {
                // Check if item is already in cache
                var existingValue = await _cache.GetAsync<object>(item.Key);
                if (existingValue != null)
                {
                    return; // Already cached
                }

                // Fetch data from data provider
                var data = await _dataProvider.GetDataAsync<object>(item.Key);
                if (data != null)
                {
                    await _cache.SetAsync(item.Key, data, item.EstimatedExpiration);
                    _logger.LogDebug($"Warmed cache for key: {item.Key}");
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Failed to warm cache for key: {item.Key}");
            }
        }

        public async Task WarmSpecificKeysAsync(List<string> keys)
        {
            var warmingTasks = keys.Select(async key =>
            {
                try
                {
                    var data = await _dataProvider.GetDataAsync<object>(key);
                    if (data != null)
                    {
                        await _cache.SetAsync(key, data, TimeSpan.FromHours(1));
                        _logger.LogDebug($"Warmed specific key: {key}");
                    }
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, $"Failed to warm specific key: {key}");
                }
            });

            await Task.WhenAll(warmingTasks);
        }

        public void RegisterWarmingCallback(Func<string, Task> callback)
        {
            _warmingCallback = callback;
        }

        public List<WarmingItem> GetWarmingStats()
        {
            return _accessPatterns.Values.OrderByDescending(item => item.Priority).ToList();
        }
    }

    public class ScheduledWarmingStrategy : ICacheWarmingStrategy
    {
        private readonly ICache _cache;
        private readonly IDataProvider _dataProvider;
        private readonly ILogger<ScheduledWarmingStrategy> _logger;
        private readonly Dictionary<string, WarmingSchedule> _schedules;
        private readonly Timer _scheduleTimer;
        private Func<string, Task> _warmingCallback;

        public ScheduledWarmingStrategy(
            ICache cache,
            IDataProvider dataProvider,
            ILogger<ScheduledWarmingStrategy> logger)
        {
            _cache = cache;
            _dataProvider = dataProvider;
            _logger = logger;
            _schedules = new Dictionary<string, WarmingSchedule>();

            // Check schedules every minute
            _scheduleTimer = new Timer(60000);
            _scheduleTimer.Elapsed += async (sender, e) => await CheckSchedulesAsync();
            _scheduleTimer.Start();
        }

        public void AddSchedule(string key, WarmingSchedule schedule)
        {
            _schedules[key] = schedule;
            _logger.LogInformation($"Added warming schedule for key: {key}");
        }

        public void RemoveSchedule(string key)
        {
            _schedules.Remove(key);
        }

        private async Task CheckSchedulesAsync()
        {
            var now = DateTime.UtcNow;
            var itemsToWarm = new List<string>();

            foreach (var schedule in _schedules)
            {
                if (ShouldWarm(schedule.Value, now))
                {
                    itemsToWarm.Add(schedule.Key);
                    schedule.Value.LastWarmed = now;
                }
            }

            if (itemsToWarm.Any())
            {
                await WarmSpecificKeysAsync(itemsToWarm);
            }
        }

        private bool ShouldWarm(WarmingSchedule schedule, DateTime now)
        {
            switch (schedule.Type)
            {
                case ScheduleType.Daily:
                    return now.TimeOfDay >= schedule.TimeOfDay && 
                           (now - schedule.LastWarmed).TotalHours >= 24;

                case ScheduleType.Weekly:
                    return now.DayOfWeek == schedule.DayOfWeek && 
                           now.TimeOfDay >= schedule.TimeOfDay &&
                           (now - schedule.LastWarmed).TotalDays >= 7;

                case ScheduleType.Interval:
                    return (now - schedule.LastWarmed) >= schedule.Interval;

                default:
                    return false;
            }
        }

        public async Task WarmCacheAsync()
        {
            var keys = _schedules.Keys.ToList();
            await WarmSpecificKeysAsync(keys);
        }

        public async Task WarmSpecificKeysAsync(List<string> keys)
        {
            var warmingTasks = keys.Select(async key =>
            {
                try
                {
                    var data = await _dataProvider.GetDataAsync<object>(key);
                    if (data != null)
                    {
                        var schedule = _schedules[key];
                        await _cache.SetAsync(key, data, schedule.CacheExpiration);
                        _logger.LogInformation($"Scheduled warming completed for key: {key}");
                    }
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, $"Failed to warm scheduled key: {key}");
                }
            });

            await Task.WhenAll(warmingTasks);
        }

        public void RegisterWarmingCallback(Func<string, Task> callback)
        {
            _warmingCallback = callback;
        }
    }

    public class EventDrivenWarmingStrategy : ICacheWarmingStrategy
    {
        private readonly ICache _cache;
        private readonly IDataProvider _dataProvider;
        private readonly IEventBus _eventBus;
        private readonly ILogger<EventDrivenWarmingStrategy> _logger;
        private readonly Dictionary<string, List<string>> _eventToKeys;
        private Func<string, Task> _warmingCallback;

        public EventDrivenWarmingStrategy(
            ICache cache,
            IDataProvider dataProvider,
            IEventBus eventBus,
            ILogger<EventDrivenWarmingStrategy> logger)
        {
            _cache = cache;
            _dataProvider = dataProvider;
            _eventBus = eventBus;
            _logger = logger;
            _eventToKeys = new Dictionary<string, List<string>>();

            // Subscribe to warming events
            _eventBus.Subscribe<CacheWarmingEvent>(HandleWarmingEvent);
        }

        public void RegisterEventTrigger(string eventType, List<string> keys)
        {
            if (!_eventToKeys.ContainsKey(eventType))
            {
                _eventToKeys[eventType] = new List<string>();
            }
            _eventToKeys[eventType].AddRange(keys);
        }

        private async Task HandleWarmingEvent(CacheWarmingEvent warmingEvent)
        {
            _logger.LogInformation($"Handling warming event: {warmingEvent.EventType}");

            if (_eventToKeys.TryGetValue(warmingEvent.EventType, out var keys))
            {
                await WarmSpecificKeysAsync(keys);
            }
        }

        public async Task WarmCacheAsync()
        {
            var allKeys = _eventToKeys.Values.SelectMany(keys => keys).Distinct().ToList();
            await WarmSpecificKeysAsync(allKeys);
        }

        public async Task WarmSpecificKeysAsync(List<string> keys)
        {
            var warmingTasks = keys.Select(async key =>
            {
                try
                {
                    var data = await _dataProvider.GetDataAsync<object>(key);
                    if (data != null)
                    {
                        await _cache.SetAsync(key, data, TimeSpan.FromHours(1));
                        _logger.LogDebug($"Event-driven warming completed for key: {key}");
                    }
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, $"Failed to warm event-driven key: {key}");
                }
            });

            await Task.WhenAll(warmingTasks);
        }

        public void RegisterWarmingCallback(Func<string, Task> callback)
        {
            _warmingCallback = callback;
        }
    }

    public class ProgressiveWarmingStrategy : ICacheWarmingStrategy
    {
        private readonly ICache _cache;
        private readonly IDataProvider _dataProvider;
        private readonly ILogger<ProgressiveWarmingStrategy> _logger;
        private readonly Queue<WarmingItem> _warmingQueue;
        private readonly Timer _warmingTimer;
        private readonly int _batchSize;
        private Func<string, Task> _warmingCallback;

        public ProgressiveWarmingStrategy(
            ICache cache,
            IDataProvider dataProvider,
            ILogger<ProgressiveWarmingStrategy> logger,
            int batchSize = 10)
        {
            _cache = cache;
            _dataProvider = dataProvider;
            _logger = logger;
            _batchSize = batchSize;
            _warmingQueue = new Queue<WarmingItem>();

            // Process warming queue every 30 seconds
            _warmingTimer = new Timer(30000);
            _warmingTimer.Elapsed += async (sender, e) => await ProcessWarmingQueueAsync();
            _warmingTimer.Start();
        }

        public void AddToWarmingQueue(WarmingItem item)
        {
            lock (_warmingQueue)
            {
                _warmingQueue.Enqueue(item);
            }
        }

        public void AddToWarmingQueue(List<WarmingItem> items)
        {
            lock (_warmingQueue)
            {
                foreach (var item in items.OrderBy(i => i.Priority))
                {
                    _warmingQueue.Enqueue(item);
                }
            }
        }

        private async Task ProcessWarmingQueueAsync()
        {
            var itemsToProcess = new List<WarmingItem>();

            lock (_warmingQueue)
            {
                for (int i = 0; i < _batchSize && _warmingQueue.Count > 0; i++)
                {
                    itemsToProcess.Add(_warmingQueue.Dequeue());
                }
            }

            if (itemsToProcess.Any())
            {
                await WarmSpecificKeysAsync(itemsToProcess.Select(item => item.Key).ToList());
            }
        }

        public async Task WarmCacheAsync()
        {
            var allItems = new List<WarmingItem>();
            lock (_warmingQueue)
            {
                allItems = _warmingQueue.ToList();
                _warmingQueue.Clear();
            }

            var keys = allItems.Select(item => item.Key).ToList();
            await WarmSpecificKeysAsync(keys);
        }

        public async Task WarmSpecificKeysAsync(List<string> keys)
        {
            var warmingTasks = keys.Select(async key =>
            {
                try
                {
                    var data = await _dataProvider.GetDataAsync<object>(key);
                    if (data != null)
                    {
                        await _cache.SetAsync(key, data, TimeSpan.FromHours(1));
                        _logger.LogDebug($"Progressive warming completed for key: {key}");
                    }
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, $"Failed to warm progressive key: {key}");
                }
            });

            await Task.WhenAll(warmingTasks);
        }

        public void RegisterWarmingCallback(Func<string, Task> callback)
        {
            _warmingCallback = callback;
        }

        public int GetQueueSize()
        {
            lock (_warmingQueue)
            {
                return _warmingQueue.Count;
            }
        }
    }

    public class CacheWarmingManager
    {
        private readonly Dictionary<WarmingStrategy, ICacheWarmingStrategy> _strategies;
        private readonly ILogger<CacheWarmingManager> _logger;

        public CacheWarmingManager(ILogger<CacheWarmingManager> logger)
        {
            _logger = logger;
            _strategies = new Dictionary<WarmingStrategy, ICacheWarmingStrategy>();
        }

        public void RegisterStrategy(WarmingStrategy strategy, ICacheWarmingStrategy implementation)
        {
            _strategies[strategy] = implementation;
            _logger.LogInformation($"Registered warming strategy: {strategy}");
        }

        public async Task WarmCacheAsync(WarmingStrategy strategy = WarmingStrategy.Predictive)
        {
            if (_strategies.TryGetValue(strategy, out var warmingStrategy))
            {
                await warmingStrategy.WarmCacheAsync();
            }
            else
            {
                _logger.LogWarning($"Warming strategy {strategy} not found");
            }
        }

        public async Task WarmAllCachesAsync()
        {
            var warmingTasks = _strategies.Values.Select(strategy => strategy.WarmCacheAsync());
            await Task.WhenAll(warmingTasks);
        }

        public async Task WarmSpecificKeysAsync(List<string> keys, WarmingStrategy strategy = WarmingStrategy.Predictive)
        {
            if (_strategies.TryGetValue(strategy, out var warmingStrategy))
            {
                await warmingStrategy.WarmSpecificKeysAsync(keys);
            }
        }
    }

    // Supporting classes
    public class WarmingSchedule
    {
        public ScheduleType Type { get; set; }
        public TimeSpan TimeOfDay { get; set; }
        public DayOfWeek DayOfWeek { get; set; }
        public TimeSpan Interval { get; set; }
        public DateTime LastWarmed { get; set; }
        public TimeSpan CacheExpiration { get; set; } = TimeSpan.FromHours(1);
    }

    public enum ScheduleType
    {
        Daily,
        Weekly,
        Interval
    }

    public class CacheWarmingEvent
    {
        public string EventType { get; set; }
        public DateTime Timestamp { get; set; }
        public Dictionary<string, object> Parameters { get; set; }
    }

    // Interfaces for demonstration
    public interface ICache
    {
        Task<T> GetAsync<T>(string key);
        Task SetAsync<T>(string key, T value, TimeSpan? expiration = null);
        Task RemoveAsync(string key);
    }

    public interface IDataProvider
    {
        Task<T> GetDataAsync<T>(string key);
    }

    public interface IEventBus
    {
        Task PublishAsync<T>(T eventData);
        void Subscribe<T>(Func<T, Task> handler);
    }
}

45. CDN Cache

• Edge node management with geographic distribution
• Content compression and ETag support
• Stale-while-revalidate pattern
• Background content replication

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Threading.Tasks;
using System.Text.Json;
using Microsoft.Extensions.Logging;
using System.IO;
using System.Security.Cryptography;
using System.Text;

namespace CachingSystem
{
    public class CDNEdgeNode
    {
        public string Id { get; set; }
        public string Region { get; set; }
        public string Location { get; set; }
        public bool IsActive { get; set; } = true;
        public long StorageCapacity { get; set; } // in bytes
        public long UsedStorage { get; set; } // in bytes
        public int MaxConnections { get; set; }
        public int CurrentConnections { get; set; }
        public DateTime LastHeartbeat { get; set; }

        public double StorageUtilization => (double)UsedStorage / StorageCapacity;
        public double ConnectionUtilization => (double)CurrentConnections / MaxConnections;
    }

    public class CDNContent
    {
        public string Key { get; set; }
        public string Url { get; set; }
        public string ContentType { get; set; }
        public byte[] Data { get; set; }
        public long Size { get; set; }
        public string ETag { get; set; }
        public DateTime CreatedAt { get; set; }
        public DateTime LastModified { get; set; }
        public DateTime? ExpiresAt { get; set; }
        public Dictionary<string, string> Headers { get; set; } = new Dictionary<string, string>();
        public List<string> EdgeNodes { get; set; } = new List<string>();
        public int AccessCount { get; set; }
        public bool IsCompressed { get; set; }
        public string CompressionType { get; set; }

        public bool IsExpired => ExpiresAt.HasValue && DateTime.UtcNow > ExpiresAt.Value;
        public bool IsStale => DateTime.UtcNow > LastModified.AddDays(1); // Consider stale after 1 day
    }

    public class CDNCacheHeaders
    {
        public string CacheControl { get; set; } = "public, max-age=3600";
        public string ETag { get; set; }
        public DateTime? LastModified { get; set; }
        public DateTime? Expires { get; set; }
        public string Vary { get; set; }
        public bool NoCache { get; set; }
        public bool NoStore { get; set; }
        public bool MustRevalidate { get; set; }
        public TimeSpan MaxAge { get; set; } = TimeSpan.FromHours(1);
        public TimeSpan StaleWhileRevalidate { get; set; } = TimeSpan.FromMinutes(5);
        public TimeSpan StaleIfError { get; set; } = TimeSpan.FromDays(1);
    }

    public class CDNRequest
    {
        public string Url { get; set; }
        public string Method { get; set; } = "GET";
        public Dictionary<string, string> Headers { get; set; } = new Dictionary<string, string>();
        public string ClientIP { get; set; }
        public string UserAgent { get; set; }
        public string AcceptEncoding { get; set; }
        public string IfNoneMatch { get; set; }
        public DateTime? IfModifiedSince { get; set; }
        public string Region { get; set; }
    }

    public class CDNResponse
    {
        public int StatusCode { get; set; } = 200;
        public byte[] Content { get; set; }
        public string ContentType { get; set; }
        public Dictionary<string, string> Headers { get; set; } = new Dictionary<string, string>();
        public bool FromCache { get; set; }
        public string ServedFrom { get; set; } // "edge", "origin", "cache"
        public TimeSpan ResponseTime { get; set; }
    }

    public class CDNCache
    {
        private readonly ConcurrentDictionary<string, CDNContent> _contentCache;
        private readonly Dictionary<string, CDNEdgeNode> _edgeNodes;
        private readonly IOriginServer _originServer;
        private readonly ILogger<CDNCache> _logger;
        private readonly CDNCacheHeaders _defaultHeaders;
        private readonly Timer _cleanupTimer;
        private readonly Timer _replicationTimer;

        public CDNCache(IOriginServer originServer, ILogger<CDNCache> logger)
        {
            _contentCache = new ConcurrentDictionary<string, CDNContent>();
            _edgeNodes = new Dictionary<string, CDNEdgeNode>();
            _originServer = originServer;
            _logger = logger;
            _defaultHeaders = new CDNCacheHeaders();

            // Cleanup expired content every 5 minutes
            _cleanupTimer = new Timer(300000);
            _cleanupTimer.Elapsed += CleanupExpiredContent;
            _cleanupTimer.Start();

            // Replicate popular content every 10 minutes
            _replicationTimer = new Timer(600000);
            _replicationTimer.Elapsed += ReplicatePopularContent;
            _replicationTimer.Start();
        }

        public void AddEdgeNode(CDNEdgeNode node)
        {
            _edgeNodes[node.Id] = node;
            _logger.LogInformation($"Added edge node: {node.Id} in {node.Region}");
        }

        public async Task<CDNResponse> ServeContentAsync(CDNRequest request)
        {
            var startTime = DateTime.UtcNow;
            var cacheKey = GenerateCacheKey(request);

            try
            {
                // Check edge cache first
                if (_contentCache.TryGetValue(cacheKey, out var cachedContent))
                {
                    var response = await HandleCachedContentAsync(request, cachedContent);
                    response.ResponseTime = DateTime.UtcNow - startTime;
                    return response;
                }

                // Check if we should serve from origin
                var response = await ServeFromOriginAsync(request);
                response.ResponseTime = DateTime.UtcNow - startTime;
                return response;
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Error serving content for {request.Url}");
                return new CDNResponse
                {
                    StatusCode = 500,
                    Content = Encoding.UTF8.GetBytes("Internal Server Error"),
                    ContentType = "text/plain",
                    FromCache = false,
                    ServedFrom = "error"
                };
            }
        }

        private async Task<CDNResponse> HandleCachedContentAsync(CDNRequest request, CDNContent content)
        {
            // Check if content is expired
            if (content.IsExpired)
            {
                _logger.LogDebug($"Content expired for {request.Url}, removing from cache");
                _contentCache.TryRemove(content.Key, out _);
                return await ServeFromOriginAsync(request);
            }

            // Check ETag for conditional requests
            if (!string.IsNullOrEmpty(request.IfNoneMatch) && request.IfNoneMatch == content.ETag)
            {
                return new CDNResponse
                {
                    StatusCode = 304, // Not Modified
                    Headers = GenerateHeaders(content, request),
                    FromCache = true,
                    ServedFrom = "edge"
                };
            }

            // Check Last-Modified for conditional requests
            if (request.IfModifiedSince.HasValue && content.LastModified <= request.IfModifiedSince.Value)
            {
                return new CDNResponse
                {
                    StatusCode = 304, // Not Modified
                    Headers = GenerateHeaders(content, request),
                    FromCache = true,
                    ServedFrom = "edge"
                };
            }

            // Serve from cache
            content.AccessCount++;
            var response = new CDNResponse
            {
                StatusCode = 200,
                Content = content.Data,
                ContentType = content.ContentType,
                Headers = GenerateHeaders(content, request),
                FromCache = true,
                ServedFrom = "edge"
            };

            // Check if content is stale and should be revalidated in background
            if (content.IsStale)
            {
                _ = Task.Run(async () => await RevalidateContentAsync(content));
            }

            return response;
        }

        private async Task<CDNResponse> ServeFromOriginAsync(CDNRequest request)
        {
            try
            {
                var originResponse = await _originServer.GetContentAsync(request.Url);

                if (originResponse.StatusCode == 200)
                {
                    // Cache the content
                    var content = new CDNContent
                    {
                        Key = GenerateCacheKey(request),
                        Url = request.Url,
                        ContentType = originResponse.ContentType,
                        Data = originResponse.Content,
                        Size = originResponse.Content.Length,
                        ETag = GenerateETag(originResponse.Content),
                        CreatedAt = DateTime.UtcNow,
                        LastModified = DateTime.UtcNow,
                        Headers = originResponse.Headers,
                        AccessCount = 1
                    };

                    // Set expiration based on headers
                    SetExpirationFromHeaders(content, originResponse.Headers);

                    // Compress content if supported
                    if (ShouldCompress(content.ContentType))
                    {
                        content.Data = await CompressContentAsync(content.Data);
                        content.IsCompressed = true;
                        content.CompressionType = "gzip";
                    }

                    // Add to cache
                    _contentCache.TryAdd(content.Key, content);

                    // Replicate to edge nodes if popular
                    if (ShouldReplicate(content))
                    {
                        _ = Task.Run(async () => await ReplicateToEdgeNodesAsync(content));
                    }
                }

                return new CDNResponse
                {
                    StatusCode = originResponse.StatusCode,
                    Content = originResponse.Content,
                    ContentType = originResponse.ContentType,
                    Headers = originResponse.Headers,
                    FromCache = false,
                    ServedFrom = "origin"
                };
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Error fetching from origin: {request.Url}");
                throw;
            }
        }

        private async Task RevalidateContentAsync(CDNContent content)
        {
            try
            {
                var request = new CDNRequest { Url = content.Url };
                var originResponse = await _originServer.GetContentAsync(content.Url);

                if (originResponse.StatusCode == 200)
                {
                    var newETag = GenerateETag(originResponse.Content);

                    if (newETag != content.ETag)
                    {
                        // Content has changed, update cache
                        content.Data = originResponse.Content;
                        content.ETag = newETag;
                        content.LastModified = DateTime.UtcNow;
                        content.Headers = originResponse.Headers;
                        SetExpirationFromHeaders(content, originResponse.Headers);

                        _logger.LogInformation($"Revalidated and updated content: {content.Url}");
                    }
                    else
                    {
                        // Content hasn't changed, just update timestamp
                        content.LastModified = DateTime.UtcNow;
                    }
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Error revalidating content: {content.Url}");
            }
        }

        private async Task ReplicateToEdgeNodesAsync(CDNContent content)
        {
            var replicationTasks = new List<Task>();

            foreach (var edgeNode in _edgeNodes.Values.Where(n => n.IsActive))
            {
                if (edgeNode.StorageUtilization < 0.9) // Only replicate if node has space
                {
                    replicationTasks.Add(ReplicateToNodeAsync(edgeNode, content));
                }
            }

            await Task.WhenAll(replicationTasks);
        }

        private async Task ReplicateToNodeAsync(CDNEdgeNode edgeNode, CDNContent content)
        {
            try
            {
                // Simulate replication to edge node
                await Task.Delay(100); // Simulate network latency

                content.EdgeNodes.Add(edgeNode.Id);
                edgeNode.UsedStorage += content.Size;

                _logger.LogDebug($"Replicated content {content.Key} to edge node {edgeNode.Id}");
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Failed to replicate content to edge node {edgeNode.Id}");
            }
        }

        private void SetExpirationFromHeaders(CDNContent content, Dictionary<string, string> headers)
        {
            if (headers.TryGetValue("Cache-Control", out var cacheControl))
            {
                var maxAge = ParseMaxAge(cacheControl);
                if (maxAge.HasValue)
                {
                    content.ExpiresAt = DateTime.UtcNow.Add(maxAge.Value);
                }
            }
            else if (headers.TryGetValue("Expires", out var expires))
            {
                if (DateTime.TryParse(expires, out var expiresDate))
                {
                    content.ExpiresAt = expiresDate;
                }
            }
            else
            {
                // Default expiration
                content.ExpiresAt = DateTime.UtcNow.Add(_defaultHeaders.MaxAge);
            }
        }

        private TimeSpan? ParseMaxAge(string cacheControl)
        {
            var parts = cacheControl.Split(',');
            foreach (var part in parts)
            {
                var trimmed = part.Trim();
                if (trimmed.StartsWith("max-age="))
                {
                    var maxAgeStr = trimmed.Substring("max-age=".Length);
                    if (int.TryParse(maxAgeStr, out var maxAge))
                    {
                        return TimeSpan.FromSeconds(maxAge);
                    }
                }
            }
            return null;
        }

        private Dictionary<string, string> GenerateHeaders(CDNContent content, CDNRequest request)
        {
            var headers = new Dictionary<string, string>(content.Headers)
            {
                ["ETag"] = content.ETag,
                ["Last-Modified"] = content.LastModified.ToString("R"),
                ["Cache-Control"] = _defaultHeaders.CacheControl,
                ["X-Cache"] = "HIT",
                ["X-Served-By"] = "CDN-Edge"
            };

            if (content.IsCompressed)
            {
                headers["Content-Encoding"] = content.CompressionType;
            }

            return headers;
        }

        private string GenerateCacheKey(CDNRequest request)
        {
            var keyBuilder = new StringBuilder();
            keyBuilder.Append(request.Url);

            // Include region for geographic caching
            if (!string.IsNullOrEmpty(request.Region))
            {
                keyBuilder.Append($"|region:{request.Region}");
            }

            // Include Accept-Encoding for compression variants
            if (!string.IsNullOrEmpty(request.AcceptEncoding))
            {
                keyBuilder.Append($"|encoding:{request.AcceptEncoding}");
            }

            return keyBuilder.ToString();
        }

        private string GenerateETag(byte[] content)
        {
            using (var sha256 = SHA256.Create())
            {
                var hash = sha256.ComputeHash(content);
                return $"\"{Convert.ToBase64String(hash)}\"";
            }
        }

        private bool ShouldCompress(string contentType)
        {
            var compressibleTypes = new[]
            {
                "text/", "application/json", "application/xml", "application/javascript"
            };

            return compressibleTypes.Any(type => contentType.StartsWith(type));
        }

        private async Task<byte[]> CompressContentAsync(byte[] content)
        {
            using (var output = new MemoryStream())
            {
                using (var gzip = new System.IO.Compression.GZipStream(output, System.IO.Compression.CompressionMode.Compress))
                {
                    await gzip.WriteAsync(content, 0, content.Length);
                }
                return output.ToArray();
            }
        }

        private bool ShouldReplicate(CDNContent content)
        {
            // Replicate if content is small enough and accessed frequently
            return content.Size < 10 * 1024 * 1024 && content.AccessCount > 5; // 10MB limit, 5+ accesses
        }

        private void CleanupExpiredContent(object sender, ElapsedEventArgs e)
        {
            var expiredKeys = _contentCache
                .Where(kvp => kvp.Value.IsExpired)
                .Select(kvp => kvp.Key)
                .ToList();

            foreach (var key in expiredKeys)
            {
                _contentCache.TryRemove(key, out _);
            }

            if (expiredKeys.Count > 0)
            {
                _logger.LogInformation($"Cleaned up {expiredKeys.Count} expired content items");
            }
        }

        private async void ReplicatePopularContent(object sender, ElapsedEventArgs e)
        {
            var popularContent = _contentCache.Values
                .Where(c => c.AccessCount > 10 && c.EdgeNodes.Count < _edgeNodes.Count / 2)
                .OrderByDescending(c => c.AccessCount)
                .Take(10)
                .ToList();

            foreach (var content in popularContent)
            {
                await ReplicateToEdgeNodesAsync(content);
            }
        }

        public Dictionary<string, object> GetCacheStats()
        {
            return new Dictionary<string, object>
            {
                ["TotalContent"] = _contentCache.Count,
                ["TotalSize"] = _contentCache.Values.Sum(c => c.Size),
                ["EdgeNodes"] = _edgeNodes.Count,
                ["ActiveEdgeNodes"] = _edgeNodes.Values.Count(n => n.IsActive),
                ["AverageAccessCount"] = _contentCache.Values.Average(c => c.AccessCount),
                ["CompressedContent"] = _contentCache.Values.Count(c => c.IsCompressed)
            };
        }
    }

    // Interfaces for demonstration
    public interface IOriginServer
    {
        Task<CDNResponse> GetContentAsync(string url);
    }

    public class HttpOriginServer : IOriginServer
    {
        private readonly HttpClient _httpClient;
        private readonly ILogger<HttpOriginServer> _logger;

        public HttpOriginServer(ILogger<HttpOriginServer> logger)
        {
            _httpClient = new HttpClient();
            _logger = logger;
        }

        public async Task<CDNResponse> GetContentAsync(string url)
        {
            try
            {
                var response = await _httpClient.GetAsync(url);
                var content = await response.Content.ReadAsByteArrayAsync();

                var headers = new Dictionary<string, string>();
                foreach (var header in response.Headers)
                {
                    headers[header.Key] = string.Join(", ", header.Value);
                }

                return new CDNResponse
                {
                    StatusCode = (int)response.StatusCode,
                    Content = content,
                    ContentType = response.Content.Headers.ContentType?.ToString() ?? "application/octet-stream",
                    Headers = headers
                };
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Error fetching content from origin: {url}");
                throw;
            }
        }
    }
}

46. Cache Consistency

• Strong consistency with write-through
• Eventual consistency with reconciliation
• Read-your-writes consistency
• Version-based consistency tracking

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using System.Timers;
using Microsoft.Extensions.Logging;
using System.Text.Json;

namespace CachingSystem
{
    public enum ConsistencyLevel
    {
        Strong,         // Immediate consistency across all nodes
        Eventual,       // Eventually consistent
        ReadYourWrites, // Read-your-writes consistency
        MonotonicReads, // Monotonic reads
        MonotonicWrites // Monotonic writes
    }

    public interface ICacheConsistencyStrategy
    {
        Task<T> GetAsync<T>(string key);
        Task SetAsync<T>(string key, T value, TimeSpan? expiration = null);
        Task RemoveAsync(string key);
        Task<bool> IsConsistentAsync(string key);
    }

    public class CacheVersion
    {
        public string Key { get; set; }
        public long Version { get; set; }
        public DateTime Timestamp { get; set; }
        public string NodeId { get; set; }
        public bool IsCommitted { get; set; }
        public List<string> ReplicaNodes { get; set; } = new List<string>();
    }

    public class StrongConsistencyStrategy : ICacheConsistencyStrategy
    {
        private readonly ICache _cache;
        private readonly IDataStore _dataStore;
        private readonly IEventBus _eventBus;
        private readonly ILogger<StrongConsistencyStrategy> _logger;
        private readonly ConcurrentDictionary<string, SemaphoreSlim> _keyLocks;
        private readonly Dictionary<string, CacheVersion> _versions;

        public StrongConsistencyStrategy(
            ICache cache,
            IDataStore dataStore,
            IEventBus eventBus,
            ILogger<StrongConsistencyStrategy> logger)
        {
            _cache = cache;
            _dataStore = dataStore;
            _eventBus = eventBus;
            _logger = logger;
            _keyLocks = new ConcurrentDictionary<string, SemaphoreSlim>();
            _versions = new Dictionary<string, CacheVersion>();

            // Subscribe to consistency events
            _eventBus.Subscribe<ConsistencyEvent>(HandleConsistencyEvent);
        }

        public async Task<T> GetAsync<T>(string key)
        {
            var lockObj = _keyLocks.GetOrAdd(key, k => new SemaphoreSlim(1, 1));
            await lockObj.WaitAsync();

            try
            {
                // Check cache first
                var cachedValue = await _cache.GetAsync<T>(key);
                if (cachedValue != null)
                {
                    // Verify version consistency
                    if (await IsVersionConsistentAsync(key))
                    {
                        return cachedValue;
                    }
                }

                // Read from data store with strong consistency
                var dataStoreValue = await _dataStore.GetAsync<T>(key);
                if (dataStoreValue != null)
                {
                    // Update cache with latest version
                    await _cache.SetAsync(key, dataStoreValue);
                    await UpdateVersionAsync(key);
                }

                return dataStoreValue;
            }
            finally
            {
                lockObj.Release();
            }
        }

        public async Task SetAsync<T>(string key, T value, TimeSpan? expiration = null)
        {
            var lockObj = _keyLocks.GetOrAdd(key, k => new SemaphoreSlim(1, 1));
            await lockObj.WaitAsync();

            try
            {
                // Write to data store first (write-through)
                await _dataStore.SetAsync(key, value, expiration);

                // Update cache
                await _cache.SetAsync(key, value, expiration);

                // Update version and notify other nodes
                await UpdateVersionAsync(key);
                await NotifyConsistencyEventAsync(key, "write");

                _logger.LogInformation($"Strong consistency write completed for key: {key}");
            }
            finally
            {
                lockObj.Release();
            }
        }

        public async Task RemoveAsync(string key)
        {
            var lockObj = _keyLocks.GetOrAdd(key, k => new SemaphoreSlim(1, 1));
            await lockObj.WaitAsync();

            try
            {
                // Remove from data store first
                await _dataStore.RemoveAsync(key);

                // Remove from cache
                await _cache.RemoveAsync(key);

                // Update version and notify
                await UpdateVersionAsync(key);
                await NotifyConsistencyEventAsync(key, "delete");

                _logger.LogInformation($"Strong consistency delete completed for key: {key}");
            }
            finally
            {
                lockObj.Release();
            }
        }

        public async Task<bool> IsConsistentAsync(string key)
        {
            return await IsVersionConsistentAsync(key);
        }

        private async Task<bool> IsVersionConsistentAsync(string key)
        {
            if (!_versions.TryGetValue(key, out var version))
            {
                return true; // No version means consistent
            }

            // Check if all replica nodes have the same version
            var replicaVersions = await GetReplicaVersionsAsync(key);
            return replicaVersions.All(v => v.Version == version.Version);
        }

        private async Task UpdateVersionAsync(string key)
        {
            var version = _versions.GetOrAdd(key, k => new CacheVersion
            {
                Key = k,
                Version = 0,
                Timestamp = DateTime.UtcNow,
                NodeId = GetCurrentNodeId()
            });

            version.Version++;
            version.Timestamp = DateTime.UtcNow;
            version.IsCommitted = true;

            _logger.LogDebug($"Updated version for key {key}: {version.Version}");
        }

        private async Task<List<CacheVersion>> GetReplicaVersionsAsync(string key)
        {
            // In a real implementation, this would query replica nodes
            // For now, we'll simulate by returning the current version
            if (_versions.TryGetValue(key, out var version))
            {
                return new List<CacheVersion> { version };
            }
            return new List<CacheVersion>();
        }

        private async Task NotifyConsistencyEventAsync(string key, string operation)
        {
            var consistencyEvent = new ConsistencyEvent
            {
                Key = key,
                Operation = operation,
                Timestamp = DateTime.UtcNow,
                NodeId = GetCurrentNodeId()
            };

            await _eventBus.PublishAsync(consistencyEvent);
        }

        private async Task HandleConsistencyEvent(ConsistencyEvent consistencyEvent)
        {
            _logger.LogDebug($"Handling consistency event: {consistencyEvent.Operation} for key {consistencyEvent.Key}");

            // In a real implementation, this would handle consistency events
            // For now, we'll just log the event
        }

        private string GetCurrentNodeId()
        {
            return Environment.MachineName; // Simplified node ID
        }
    }

    public class EventualConsistencyStrategy : ICacheConsistencyStrategy
    {
        private readonly ICache _cache;
        private readonly IDataStore _dataStore;
        private readonly IEventBus _eventBus;
        private readonly ILogger<EventualConsistencyStrategy> _logger;
        private readonly ConcurrentDictionary<string, CacheVersion> _versions;
        private readonly Queue<ConsistencyEvent> _pendingEvents;
        private readonly Timer _reconciliationTimer;

        public EventualConsistencyStrategy(
            ICache cache,
            IDataStore dataStore,
            IEventBus eventBus,
            ILogger<EventualConsistencyStrategy> logger)
        {
            _cache = cache;
            _dataStore = dataStore;
            _eventBus = eventBus;
            _logger = logger;
            _versions = new ConcurrentDictionary<string, CacheVersion>();
            _pendingEvents = new Queue<ConsistencyEvent>();

            // Subscribe to consistency events
            _eventBus.Subscribe<ConsistencyEvent>(HandleConsistencyEvent);

            // Reconciliation timer - runs every 30 seconds
            _reconciliationTimer = new Timer(30000);
            _reconciliationTimer.Elapsed += async (sender, e) => await ReconcileAsync();
            _reconciliationTimer.Start();
        }

        public async Task<T> GetAsync<T>(string key)
        {
            // Read from cache first (eventual consistency)
            var cachedValue = await _cache.GetAsync<T>(key);
            if (cachedValue != null)
            {
                return cachedValue;
            }

            // Fallback to data store
            var dataStoreValue = await _dataStore.GetAsync<T>(key);
            if (dataStoreValue != null)
            {
                // Update cache asynchronously
                _ = Task.Run(async () => await _cache.SetAsync(key, dataStoreValue));
            }

            return dataStoreValue;
        }

        public async Task SetAsync<T>(string key, T value, TimeSpan? expiration = null)
        {
            // Write to cache immediately (write-behind)
            await _cache.SetAsync(key, value, expiration);

            // Update version
            await UpdateVersionAsync(key);

            // Queue write to data store
            var consistencyEvent = new ConsistencyEvent
            {
                Key = key,
                Operation = "write",
                Value = JsonSerializer.Serialize(value),
                Timestamp = DateTime.UtcNow,
                NodeId = GetCurrentNodeId()
            };

            lock (_pendingEvents)
            {
                _pendingEvents.Enqueue(consistencyEvent);
            }

            // Notify other nodes asynchronously
            _ = Task.Run(async () => await NotifyConsistencyEventAsync(consistencyEvent));

            _logger.LogInformation($"Eventual consistency write queued for key: {key}");
        }

        public async Task RemoveAsync(string key)
        {
            // Remove from cache immediately
            await _cache.RemoveAsync(key);

            // Update version
            await UpdateVersionAsync(key);

            // Queue delete from data store
            var consistencyEvent = new ConsistencyEvent
            {
                Key = key,
                Operation = "delete",
                Timestamp = DateTime.UtcNow,
                NodeId = GetCurrentNodeId()
            };

            lock (_pendingEvents)
            {
                _pendingEvents.Enqueue(consistencyEvent);
            }

            // Notify other nodes asynchronously
            _ = Task.Run(async () => await NotifyConsistencyEventAsync(consistencyEvent));

            _logger.LogInformation($"Eventual consistency delete queued for key: {key}");
        }

        public async Task<bool> IsConsistentAsync(string key)
        {
            // In eventual consistency, we check if the version is recent enough
            if (_versions.TryGetValue(key, out var version))
            {
                var age = DateTime.UtcNow - version.Timestamp;
                return age.TotalMinutes < 5; // Consider consistent if less than 5 minutes old
            }
            return true;
        }

        private async Task ReconcileAsync()
        {
            List<ConsistencyEvent> eventsToProcess;
            lock (_pendingEvents)
            {
                eventsToProcess = _pendingEvents.ToList();
                _pendingEvents.Clear();
            }

            foreach (var consistencyEvent in eventsToProcess)
            {
                try
                {
                    await ProcessConsistencyEventAsync(consistencyEvent);
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, $"Error processing consistency event for key: {consistencyEvent.Key}");

                    // Re-queue failed events
                    lock (_pendingEvents)
                    {
                        _pendingEvents.Enqueue(consistencyEvent);
                    }
                }
            }

            if (eventsToProcess.Any())
            {
                _logger.LogInformation($"Reconciled {eventsToProcess.Count} consistency events");
            }
        }

        private async Task ProcessConsistencyEventAsync(ConsistencyEvent consistencyEvent)
        {
            switch (consistencyEvent.Operation)
            {
                case "write":
                    if (!string.IsNullOrEmpty(consistencyEvent.Value))
                    {
                        var value = JsonSerializer.Deserialize<object>(consistencyEvent.Value);
                        await _dataStore.SetAsync(consistencyEvent.Key, value);
                    }
                    break;

                case "delete":
                    await _dataStore.RemoveAsync(consistencyEvent.Key);
                    break;
            }
        }

        private async Task UpdateVersionAsync(string key)
        {
            var version = _versions.GetOrAdd(key, k => new CacheVersion
            {
                Key = k,
                Version = 0,
                Timestamp = DateTime.UtcNow,
                NodeId = GetCurrentNodeId()
            });

            version.Version++;
            version.Timestamp = DateTime.UtcNow;
        }

        private async Task NotifyConsistencyEventAsync(ConsistencyEvent consistencyEvent)
        {
            await _eventBus.PublishAsync(consistencyEvent);
        }

        private async Task HandleConsistencyEvent(ConsistencyEvent consistencyEvent)
        {
            _logger.LogDebug($"Handling consistency event: {consistencyEvent.Operation} for key {consistencyEvent.Key}");

            // Update local cache based on event
            switch (consistencyEvent.Operation)
            {
                case "write":
                    if (!string.IsNullOrEmpty(consistencyEvent.Value))
                    {
                        var value = JsonSerializer.Deserialize<object>(consistencyEvent.Value);
                        await _cache.SetAsync(consistencyEvent.Key, value);
                    }
                    break;

                case "delete":
                    await _cache.RemoveAsync(consistencyEvent.Key);
                    break;
            }

            // Update version
            await UpdateVersionAsync(consistencyEvent.Key);
        }

        private string GetCurrentNodeId()
        {
            return Environment.MachineName;
        }
    }

    public class ReadYourWritesConsistencyStrategy : ICacheConsistencyStrategy
    {
        private readonly ICache _cache;
        private readonly IDataStore _dataStore;
        private readonly ILogger<ReadYourWritesConsistencyStrategy> _logger;
        private readonly Dictionary<string, HashSet<string>> _clientWrites;
        private readonly ConcurrentDictionary<string, CacheVersion> _versions;

        public ReadYourWritesConsistencyStrategy(
            ICache cache,
            IDataStore dataStore,
            ILogger<ReadYourWritesConsistencyStrategy> logger)
        {
            _cache = cache;
            _dataStore = dataStore;
            _logger = logger;
            _clientWrites = new Dictionary<string, HashSet<string>>();
            _versions = new ConcurrentDictionary<string, CacheVersion>();
        }

        public async Task<T> GetAsync<T>(string key, string clientId = null)
        {
            // Check if this client has written to this key recently
            if (!string.IsNullOrEmpty(clientId) && HasClientWritten(clientId, key))
            {
                // Read from data store to ensure read-your-writes consistency
                var dataStoreValue = await _dataStore.GetAsync<T>(key);
                if (dataStoreValue != null)
                {
                    await _cache.SetAsync(key, dataStoreValue);
                }
                return dataStoreValue;
            }

            // Otherwise, read from cache
            var cachedValue = await _cache.GetAsync<T>(key);
            if (cachedValue != null)
            {
                return cachedValue;
            }

            // Fallback to data store
            var fallbackValue = await _dataStore.GetAsync<T>(key);
            if (fallbackValue != null)
            {
                await _cache.SetAsync(key, fallbackValue);
            }

            return fallbackValue;
        }

        public async Task SetAsync<T>(string key, T value, TimeSpan? expiration = null, string clientId = null)
        {
            // Write to data store first
            await _dataStore.SetAsync(key, value, expiration);

            // Update cache
            await _cache.SetAsync(key, value, expiration);

            // Track client write
            if (!string.IsNullOrEmpty(clientId))
            {
                TrackClientWrite(clientId, key);
            }

            // Update version
            await UpdateVersionAsync(key);

            _logger.LogInformation($"Read-your-writes consistency write completed for key: {key}");
        }

        public async Task RemoveAsync(string key, string clientId = null)
        {
            // Remove from data store first
            await _dataStore.RemoveAsync(key);

            // Remove from cache
            await _cache.RemoveAsync(key);

            // Track client write
            if (!string.IsNullOrEmpty(clientId))
            {
                TrackClientWrite(clientId, key);
            }

            // Update version
            await UpdateVersionAsync(key);

            _logger.LogInformation($"Read-your-writes consistency delete completed for key: {key}");
        }

        public async Task<bool> IsConsistentAsync(string key)
        {
            // In read-your-writes consistency, we consider it consistent if the version is recent
            if (_versions.TryGetValue(key, out var version))
            {
                var age = DateTime.UtcNow - version.Timestamp;
                return age.TotalMinutes < 1; // Consider consistent if less than 1 minute old
            }
            return true;
        }

        private bool HasClientWritten(string clientId, string key)
        {
            lock (_clientWrites)
            {
                if (_clientWrites.TryGetValue(clientId, out var writtenKeys))
                {
                    return writtenKeys.Contains(key);
                }
            }
            return false;
        }

        private void TrackClientWrite(string clientId, string key)
        {
            lock (_clientWrites)
            {
                if (!_clientWrites.ContainsKey(clientId))
                {
                    _clientWrites[clientId] = new HashSet<string>();
                }
                _clientWrites[clientId].Add(key);
            }
        }

        private async Task UpdateVersionAsync(string key)
        {
            var version = _versions.GetOrAdd(key, k => new CacheVersion
            {
                Key = k,
                Version = 0,
                Timestamp = DateTime.UtcNow,
                NodeId = GetCurrentNodeId()
            });

            version.Version++;
            version.Timestamp = DateTime.UtcNow;
        }

        private string GetCurrentNodeId()
        {
            return Environment.MachineName;
        }
    }

    public class CacheConsistencyManager
    {
        private readonly Dictionary<ConsistencyLevel, ICacheConsistencyStrategy> _strategies;
        private readonly ILogger<CacheConsistencyManager> _logger;

        public CacheConsistencyManager(ILogger<CacheConsistencyManager> logger)
        {
            _logger = logger;
            _strategies = new Dictionary<ConsistencyLevel, ICacheConsistencyStrategy>();
        }

        public void RegisterStrategy(ConsistencyLevel level, ICacheConsistencyStrategy strategy)
        {
            _strategies[level] = strategy;
            _logger.LogInformation($"Registered consistency strategy: {level}");
        }

        public async Task<T> GetAsync<T>(ConsistencyLevel level, string key, string clientId = null)
        {
            if (_strategies.TryGetValue(level, out var strategy))
            {
                if (strategy is ReadYourWritesConsistencyStrategy rywStrategy)
                {
                    return await rywStrategy.GetAsync<T>(key, clientId);
                }
                return await strategy.GetAsync<T>(key);
            }

            throw new ArgumentException($"Consistency strategy {level} not found");
        }

        public async Task SetAsync<T>(ConsistencyLevel level, string key, T value, TimeSpan? expiration = null, string clientId = null)
        {
            if (_strategies.TryGetValue(level, out var strategy))
            {
                if (strategy is ReadYourWritesConsistencyStrategy rywStrategy)
                {
                    await rywStrategy.SetAsync(key, value, expiration, clientId);
                }
                else
                {
                    await strategy.SetAsync(key, value, expiration);
                }
            }
            else
            {
                throw new ArgumentException($"Consistency strategy {level} not found");
            }
        }

        public async Task RemoveAsync(ConsistencyLevel level, string key, string clientId = null)
        {
            if (_strategies.TryGetValue(level, out var strategy))
            {
                if (strategy is ReadYourWritesConsistencyStrategy rywStrategy)
                {
                    await rywStrategy.RemoveAsync(key, clientId);
                }
                else
                {
                    await strategy.RemoveAsync(key);
                }
            }
            else
            {
                throw new ArgumentException($"Consistency strategy {level} not found");
            }
        }

        public async Task<bool> IsConsistentAsync(ConsistencyLevel level, string key)
        {
            if (_strategies.TryGetValue(level, out var strategy))
            {
                return await strategy.IsConsistentAsync(key);
            }

            throw new ArgumentException($"Consistency strategy {level} not found");
        }
    }

    // Supporting classes
    public class ConsistencyEvent
    {
        public string Key { get; set; }
        public string Operation { get; set; } // "write", "delete"
        public string Value { get; set; } // Serialized value for writes
        public DateTime Timestamp { get; set; }
        public string NodeId { get; set; }
    }

    // Interfaces for demonstration
    public interface ICache
    {
        Task<T> GetAsync<T>(string key);
        Task SetAsync<T>(string key, T value, TimeSpan? expiration = null);
        Task RemoveAsync(string key);
    }

    public interface IDataStore
    {
        Task<T> GetAsync<T>(string key);
        Task SetAsync<T>(string key, T value, TimeSpan? expiration = null);
        Task RemoveAsync(string key);
    }

    public interface IEventBus
    {
        Task PublishAsync<T>(T eventData);
        void Subscribe<T>(Func<T, Task> handler);
    }
}

47. Session Cache

• Sticky sessions with load balancing
• Session replication and persistence
• Sliding expiration and cleanup
• User session management

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using System.Timers;
using Microsoft.Extensions.Logging;
using System.Text.Json;
using System.Security.Cryptography;
using System.Text;

namespace CachingSystem
{
    public class SessionData
    {
        public string SessionId { get; set; }
        public string UserId { get; set; }
        public Dictionary<string, object> Data { get; set; } = new Dictionary<string, object>();
        public DateTime CreatedAt { get; set; }
        public DateTime LastAccessed { get; set; }
        public DateTime ExpiresAt { get; set; }
        public bool IsActive { get; set; } = true;
        public string NodeId { get; set; }
        public int Version { get; set; } = 1;
        public List<string> ReplicaNodes { get; set; } = new List<string>();

        public bool IsExpired => DateTime.UtcNow > ExpiresAt;
        public TimeSpan TimeToLive => ExpiresAt - DateTime.UtcNow;
    }

    public class SessionConfiguration
    {
        public TimeSpan DefaultTimeout { get; set; } = TimeSpan.FromMinutes(30);
        public TimeSpan SlidingExpiration { get; set; } = TimeSpan.FromMinutes(20);
        public int MaxSessionsPerUser { get; set; } = 5;
        public bool EnableStickySessions { get; set; } = true;
        public bool EnableSessionReplication { get; set; } = true;
        public int ReplicaCount { get; set; } = 2;
        public bool EnableSessionPersistence { get; set; } = true;
        public string PersistencePath { get; set; } = "sessions";
    }

    public class SessionNode
    {
        public string Id { get; set; }
        public string Host { get; set; }
        public int Port { get; set; }
        public bool IsActive { get; set; } = true;
        public int MaxSessions { get; set; }
        public int CurrentSessions { get; set; }
        public DateTime LastHeartbeat { get; set; }

        public double SessionUtilization => (double)CurrentSessions / MaxSessions;
    }

    public class SessionCache
    {
        private readonly ConcurrentDictionary<string, SessionData> _sessions;
        private readonly Dictionary<string, SessionNode> _nodes;
        private readonly SessionConfiguration _config;
        private readonly ILogger<SessionCache> _logger;
        private readonly Timer _cleanupTimer;
        private readonly Timer _replicationTimer;
        private readonly ISessionPersistence _persistence;
        private readonly ILoadBalancer _loadBalancer;

        public SessionCache(
            SessionConfiguration config,
            ILogger<SessionCache> logger,
            ISessionPersistence persistence = null,
            ILoadBalancer loadBalancer = null)
        {
            _config = config;
            _logger = logger;
            _sessions = new ConcurrentDictionary<string, SessionData>();
            _nodes = new Dictionary<string, SessionNode>();
            _persistence = persistence;
            _loadBalancer = loadBalancer;

            // Cleanup expired sessions every minute
            _cleanupTimer = new Timer(60000);
            _cleanupTimer.Elapsed += CleanupExpiredSessions;
            _cleanupTimer.Start();

            // Replicate sessions every 30 seconds
            if (_config.EnableSessionReplication)
            {
                _replicationTimer = new Timer(30000);
                _replicationTimer.Elapsed += ReplicateSessions;
                _replicationTimer.Start();
            }

            // Load persisted sessions on startup
            if (_config.EnableSessionPersistence && _persistence != null)
            {
                _ = Task.Run(async () => await LoadPersistedSessionsAsync());
            }
        }

        public void AddNode(SessionNode node)
        {
            _nodes[node.Id] = node;
            _logger.LogInformation($"Added session node: {node.Id}");
        }

        public async Task<SessionData> GetSessionAsync(string sessionId)
        {
            if (_sessions.TryGetValue(sessionId, out var session))
            {
                if (session.IsExpired)
                {
                    await RemoveSessionAsync(sessionId);
                    return null;
                }

                // Update last accessed time and extend expiration if sliding
                session.LastAccessed = DateTime.UtcNow;
                if (_config.SlidingExpiration > TimeSpan.Zero)
                {
                    session.ExpiresAt = DateTime.UtcNow.Add(_config.SlidingExpiration);
                }

                return session;
            }

            // Try to load from persistence
            if (_config.EnableSessionPersistence && _persistence != null)
            {
                var persistedSession = await _persistence.LoadSessionAsync(sessionId);
                if (persistedSession != null && !persistedSession.IsExpired)
                {
                    _sessions.TryAdd(sessionId, persistedSession);
                    return persistedSession;
                }
            }

            return null;
        }

        public async Task<SessionData> CreateSessionAsync(string userId, Dictionary<string, object> initialData = null)
        {
            // Check if user has too many active sessions
            var userSessions = _sessions.Values
                .Where(s => s.UserId == userId && s.IsActive && !s.IsExpired)
                .ToList();

            if (userSessions.Count >= _config.MaxSessionsPerUser)
            {
                // Remove oldest session
                var oldestSession = userSessions.OrderBy(s => s.LastAccessed).First();
                await RemoveSessionAsync(oldestSession.SessionId);
            }

            var sessionId = GenerateSessionId();
            var session = new SessionData
            {
                SessionId = sessionId,
                UserId = userId,
                Data = initialData ?? new Dictionary<string, object>(),
                CreatedAt = DateTime.UtcNow,
                LastAccessed = DateTime.UtcNow,
                ExpiresAt = DateTime.UtcNow.Add(_config.DefaultTimeout),
                IsActive = true,
                NodeId = GetCurrentNodeId(),
                Version = 1
            };

            _sessions.TryAdd(sessionId, session);

            // Persist session if enabled
            if (_config.EnableSessionPersistence && _persistence != null)
            {
                await _persistence.SaveSessionAsync(session);
            }

            // Replicate session if enabled
            if (_config.EnableSessionReplication)
            {
                await ReplicateSessionAsync(session);
            }

            _logger.LogInformation($"Created session {sessionId} for user {userId}");
            return session;
        }

        public async Task<bool> UpdateSessionAsync(string sessionId, Dictionary<string, object> data)
        {
            if (_sessions.TryGetValue(sessionId, out var session))
            {
                if (session.IsExpired)
                {
                    await RemoveSessionAsync(sessionId);
                    return false;
                }

                // Update session data
                foreach (var kvp in data)
                {
                    session.Data[kvp.Key] = kvp.Value;
                }

                session.LastAccessed = DateTime.UtcNow;
                session.Version++;

                // Persist changes
                if (_config.EnableSessionPersistence && _persistence != null)
                {
                    await _persistence.SaveSessionAsync(session);
                }

                // Replicate changes
                if (_config.EnableSessionReplication)
                {
                    await ReplicateSessionAsync(session);
                }

                return true;
            }

            return false;
        }

        public async Task<bool> RemoveSessionAsync(string sessionId)
        {
            if (_sessions.TryRemove(sessionId, out var session))
            {
                // Remove from persistence
                if (_config.EnableSessionPersistence && _persistence != null)
                {
                    await _persistence.DeleteSessionAsync(sessionId);
                }

                // Notify replica nodes
                if (_config.EnableSessionReplication)
                {
                    await NotifySessionRemovalAsync(sessionId);
                }

                _logger.LogInformation($"Removed session {sessionId}");
                return true;
            }

            return false;
        }

        public async Task<bool> InvalidateUserSessionsAsync(string userId)
        {
            var userSessions = _sessions.Values
                .Where(s => s.UserId == userId && s.IsActive)
                .ToList();

            var removalTasks = userSessions.Select(s => RemoveSessionAsync(s.SessionId));
            await Task.WhenAll(removalTasks);

            _logger.LogInformation($"Invalidated {userSessions.Count} sessions for user {userId}");
            return userSessions.Count > 0;
        }

        public SessionNode GetOptimalNode(string sessionId = null)
        {
            if (_config.EnableStickySessions && !string.IsNullOrEmpty(sessionId))
            {
                // Try to find the node that already has this session
                if (_sessions.TryGetValue(sessionId, out var session))
                {
                    if (_nodes.TryGetValue(session.NodeId, out var node) && node.IsActive)
                    {
                        return node;
                    }
                }
            }

            // Use load balancer or find least loaded node
            if (_loadBalancer != null)
            {
                return _loadBalancer.GetOptimalNode(_nodes.Values.ToList());
            }

            return _nodes.Values
                .Where(n => n.IsActive && n.SessionUtilization < 0.9)
                .OrderBy(n => n.SessionUtilization)
                .FirstOrDefault();
        }

        private async Task ReplicateSessionAsync(SessionData session)
        {
            var replicaNodes = _nodes.Values
                .Where(n => n.IsActive && n.Id != session.NodeId)
                .OrderBy(n => n.SessionUtilization)
                .Take(_config.ReplicaCount)
                .ToList();

            var replicationTasks = replicaNodes.Select(node => ReplicateToNodeAsync(node, session));
            await Task.WhenAll(replicationTasks);

            session.ReplicaNodes = replicaNodes.Select(n => n.Id).ToList();
        }

        private async Task ReplicateToNodeAsync(SessionNode node, SessionData session)
        {
            try
            {
                // In a real implementation, this would send the session to the replica node
                await Task.Delay(50); // Simulate network latency
                _logger.LogDebug($"Replicated session {session.SessionId} to node {node.Id}");
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Failed to replicate session {session.SessionId} to node {node.Id}");
            }
        }

        private async Task NotifySessionRemovalAsync(string sessionId)
        {
            var removalTasks = _nodes.Values
                .Where(n => n.IsActive)
                .Select(node => NotifyNodeOfRemovalAsync(node, sessionId));

            await Task.WhenAll(removalTasks);
        }

        private async Task NotifyNodeOfRemovalAsync(SessionNode node, string sessionId)
        {
            try
            {
                // In a real implementation, this would notify the node to remove the session
                await Task.Delay(10); // Simulate network latency
                _logger.LogDebug($"Notified node {node.Id} to remove session {sessionId}");
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Failed to notify node {node.Id} of session removal {sessionId}");
            }
        }

        private async Task LoadPersistedSessionsAsync()
        {
            try
            {
                var persistedSessions = await _persistence.LoadAllSessionsAsync();
                foreach (var session in persistedSessions)
                {
                    if (!session.IsExpired)
                    {
                        _sessions.TryAdd(session.SessionId, session);
                    }
                }

                _logger.LogInformation($"Loaded {persistedSessions.Count} persisted sessions");
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error loading persisted sessions");
            }
        }

        private void CleanupExpiredSessions(object sender, ElapsedEventArgs e)
        {
            var expiredSessions = _sessions.Values
                .Where(s => s.IsExpired)
                .Select(s => s.SessionId)
                .ToList();

            foreach (var sessionId in expiredSessions)
            {
                _ = Task.Run(async () => await RemoveSessionAsync(sessionId));
            }

            if (expiredSessions.Count > 0)
            {
                _logger.LogInformation($"Cleaned up {expiredSessions.Count} expired sessions");
            }
        }

        private async void ReplicateSessions(object sender, ElapsedEventArgs e)
        {
            var sessionsToReplicate = _sessions.Values
                .Where(s => s.IsActive && !s.IsExpired && s.ReplicaNodes.Count < _config.ReplicaCount)
                .ToList();

            foreach (var session in sessionsToReplicate)
            {
                await ReplicateSessionAsync(session);
            }
        }

        private string GenerateSessionId()
        {
            using (var rng = new RNGCryptoServiceProvider())
            {
                var bytes = new byte[32];
                rng.GetBytes(bytes);
                return Convert.ToBase64String(bytes).Replace("/", "_").Replace("+", "-").Substring(0, 24);
            }
        }

        private string GetCurrentNodeId()
        {
            return Environment.MachineName;
        }

        public Dictionary<string, object> GetSessionStats()
        {
            return new Dictionary<string, object>
            {
                ["TotalSessions"] = _sessions.Count,
                ["ActiveSessions"] = _sessions.Values.Count(s => s.IsActive && !s.IsExpired),
                ["ExpiredSessions"] = _sessions.Values.Count(s => s.IsExpired),
                ["Nodes"] = _nodes.Count,
                ["ActiveNodes"] = _nodes.Values.Count(n => n.IsActive),
                ["AverageSessionAge"] = _sessions.Values.Any() ? 
                    _sessions.Values.Average(s => (DateTime.UtcNow - s.CreatedAt).TotalMinutes) : 0,
                ["SessionsWithReplicas"] = _sessions.Values.Count(s => s.ReplicaNodes.Any())
            };
        }
    }

    public class StickySessionLoadBalancer : ILoadBalancer
    {
        private readonly Dictionary<string, string> _sessionToNode;
        private readonly ILogger<StickySessionLoadBalancer> _logger;

        public StickySessionLoadBalancer(ILogger<StickySessionLoadBalancer> logger)
        {
            _sessionToNode = new Dictionary<string, string>();
            _logger = logger;
        }

        public SessionNode GetOptimalNode(List<SessionNode> nodes, string sessionId = null)
        {
            if (!string.IsNullOrEmpty(sessionId) && _sessionToNode.TryGetValue(sessionId, out var assignedNodeId))
            {
                var assignedNode = nodes.FirstOrDefault(n => n.Id == assignedNodeId && n.IsActive);
                if (assignedNode != null && assignedNode.SessionUtilization < 0.9)
                {
                    return assignedNode;
                }
            }

            // Find least loaded node
            var optimalNode = nodes
                .Where(n => n.IsActive && n.SessionUtilization < 0.9)
                .OrderBy(n => n.SessionUtilization)
                .FirstOrDefault();

            if (optimalNode != null && !string.IsNullOrEmpty(sessionId))
            {
                _sessionToNode[sessionId] = optimalNode.Id;
            }

            return optimalNode;
        }

        public void RemoveSessionMapping(string sessionId)
        {
            _sessionToNode.Remove(sessionId);
        }
    }

    public class FileSessionPersistence : ISessionPersistence
    {
        private readonly string _persistencePath;
        private readonly ILogger<FileSessionPersistence> _logger;

        public FileSessionPersistence(string persistencePath, ILogger<FileSessionPersistence> logger)
        {
            _persistencePath = persistencePath;
            _logger = logger;
            Directory.CreateDirectory(_persistencePath);
        }

        public async Task SaveSessionAsync(SessionData session)
        {
            try
            {
                var filePath = Path.Combine(_persistencePath, $"{session.SessionId}.json");
                var json = JsonSerializer.Serialize(session, new JsonSerializerOptions { WriteIndented = true });
                await File.WriteAllTextAsync(filePath, json);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Failed to save session {session.SessionId}");
            }
        }

        public async Task<SessionData> LoadSessionAsync(string sessionId)
        {
            try
            {
                var filePath = Path.Combine(_persistencePath, $"{sessionId}.json");
                if (File.Exists(filePath))
                {
                    var json = await File.ReadAllTextAsync(filePath);
                    return JsonSerializer.Deserialize<SessionData>(json);
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Failed to load session {sessionId}");
            }

            return null;
        }

        public async Task<List<SessionData>> LoadAllSessionsAsync()
        {
            var sessions = new List<SessionData>();

            try
            {
                var files = Directory.GetFiles(_persistencePath, "*.json");
                foreach (var file in files)
                {
                    var json = await File.ReadAllTextAsync(file);
                    var session = JsonSerializer.Deserialize<SessionData>(json);
                    if (session != null)
                    {
                        sessions.Add(session);
                    }
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Failed to load all sessions");
            }

            return sessions;
        }

        public async Task DeleteSessionAsync(string sessionId)
        {
            try
            {
                var filePath = Path.Combine(_persistencePath, $"{sessionId}.json");
                if (File.Exists(filePath))
                {
                    File.Delete(filePath);
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Failed to delete session {sessionId}");
            }
        }
    }

    // Interfaces for demonstration
    public interface ILoadBalancer
    {
        SessionNode GetOptimalNode(List<SessionNode> nodes, string sessionId = null);
    }

    public interface ISessionPersistence
    {
        Task SaveSessionAsync(SessionData session);
        Task<SessionData> LoadSessionAsync(string sessionId);
        Task<List<SessionData>> LoadAllSessionsAsync();
        Task DeleteSessionAsync(string sessionId);
    }
}

48. Cache Partitioning

• Hash-based partitioning with consistent hashing
• Range-based and directory-based partitioning
• Round-robin distribution
• Partition management and statistics

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using System.Text.Json;
using System.Security.Cryptography;
using System.Text;

namespace CachingSystem
{
    public enum PartitioningStrategy
    {
        HashBased,      // Consistent hashing
        RangeBased,     // Range-based partitioning
        DirectoryBased, // Directory-based partitioning
        RoundRobin      // Round-robin distribution
    }

    public interface ICachePartitioningStrategy
    {
        string GetPartition(string key);
        List<string> GetPartitions();
        void AddPartition(string partitionId);
        void RemovePartition(string partitionId);
        Dictionary<string, object> GetPartitionStats();
    }

    public class CachePartition
    {
        public string Id { get; set; }
        public string Name { get; set; }
        public bool IsActive { get; set; } = true;
        public long Capacity { get; set; } // in bytes
        public long UsedSpace { get; set; } // in bytes
        public int ItemCount { get; set; }
        public DateTime CreatedAt { get; set; }
        public DateTime LastAccessed { get; set; }

        public double Utilization => (double)UsedSpace / Capacity;
        public long AvailableSpace => Capacity - UsedSpace;
    }

    public class HashBasedPartitioningStrategy : ICachePartitioningStrategy
    {
        private readonly SortedDictionary<uint, string> _hashRing;
        private readonly Dictionary<string, List<uint>> _partitionToHashes;
        private readonly Dictionary<string, CachePartition> _partitions;
        private readonly object _lock = new object();
        private readonly int _virtualNodesPerPartition;

        public HashBasedPartitioningStrategy(int virtualNodesPerPartition = 150)
        {
            _hashRing = new SortedDictionary<uint, string>();
            _partitionToHashes = new Dictionary<string, List<uint>>();
            _partitions = new Dictionary<string, CachePartition>();
            _virtualNodesPerPartition = virtualNodesPerPartition;
        }

        public string GetPartition(string key)
        {
            lock (_lock)
            {
                if (_hashRing.Count == 0)
                    return null;

                var hash = GetHash(key);
                var partition = _hashRing.FirstOrDefault(kvp => kvp.Key >= hash);

                if (partition.Key == 0) // No partition found with hash >= key hash
                {
                    // Wrap around to the first partition
                    partition = _hashRing.First();
                }

                return partition.Value;
            }
        }

        public List<string> GetPartitions()
        {
            lock (_lock)
            {
                return _partitions.Keys.ToList();
            }
        }

        public void AddPartition(string partitionId)
        {
            lock (_lock)
            {
                if (_partitions.ContainsKey(partitionId))
                    return;

                var partition = new CachePartition
                {
                    Id = partitionId,
                    Name = $"partition-{partitionId}",
                    CreatedAt = DateTime.UtcNow,
                    LastAccessed = DateTime.UtcNow,
                    Capacity = 1024 * 1024 * 1024 // 1GB default
                };

                _partitions[partitionId] = partition;

                // Add virtual nodes to hash ring
                var hashes = new List<uint>();
                for (int i = 0; i < _virtualNodesPerPartition; i++)
                {
                    var virtualNodeKey = $"{partitionId}-{i}";
                    var hash = GetHash(virtualNodeKey);
                    _hashRing[hash] = partitionId;
                    hashes.Add(hash);
                }
                _partitionToHashes[partitionId] = hashes;
            }
        }

        public void RemovePartition(string partitionId)
        {
            lock (_lock)
            {
                if (_partitionToHashes.TryGetValue(partitionId, out var hashes))
                {
                    foreach (var hash in hashes)
                    {
                        _hashRing.Remove(hash);
                    }
                    _partitionToHashes.Remove(partitionId);
                }

                _partitions.Remove(partitionId);
            }
        }

        public Dictionary<string, object> GetPartitionStats()
        {
            lock (_lock)
            {
                return new Dictionary<string, object>
                {
                    ["TotalPartitions"] = _partitions.Count,
                    ["ActivePartitions"] = _partitions.Values.Count(p => p.IsActive),
                    ["HashRingSize"] = _hashRing.Count,
                    ["VirtualNodesPerPartition"] = _virtualNodesPerPartition,
                    ["PartitionDetails"] = _partitions.ToDictionary(
                        kvp => kvp.Key,
                        kvp => new
                        {
                            kvp.Value.Name,
                            kvp.Value.IsActive,
                            kvp.Value.Utilization,
                            kvp.Value.ItemCount,
                            kvp.Value.UsedSpace,
                            kvp.Value.Capacity
                        })
                };
            }
        }

        private uint GetHash(string key)
        {
            using (var md5 = MD5.Create())
            {
                var hash = md5.ComputeHash(Encoding.UTF8.GetBytes(key));
                return BitConverter.ToUInt32(hash, 0);
            }
        }
    }

    public class RangeBasedPartitioningStrategy : ICachePartitioningStrategy
    {
        private readonly SortedDictionary<string, string> _rangeMap;
        private readonly Dictionary<string, CachePartition> _partitions;
        private readonly object _lock = new object();

        public RangeBasedPartitioningStrategy()
        {
            _rangeMap = new SortedDictionary<string, string>();
            _partitions = new Dictionary<string, CachePartition>();
        }

        public string GetPartition(string key)
        {
            lock (_lock)
            {
                if (_rangeMap.Count == 0)
                    return null;

                // Find the partition that contains this key
                var partition = _rangeMap.FirstOrDefault(kvp => string.Compare(key, kvp.Key) >= 0);

                if (partition.Key == null) // No partition found
                {
                    // Use the first partition
                    partition = _rangeMap.First();
                }

                return partition.Value;
            }
        }

        public List<string> GetPartitions()
        {
            lock (_lock)
            {
                return _partitions.Keys.ToList();
            }
        }

        public void AddPartition(string partitionId, string startRange = null)
        {
            lock (_lock)
            {
                if (_partitions.ContainsKey(partitionId))
                    return;

                var partition = new CachePartition
                {
                    Id = partitionId,
                    Name = $"partition-{partitionId}",
                    CreatedAt = DateTime.UtcNow,
                    LastAccessed = DateTime.UtcNow,
                    Capacity = 1024 * 1024 * 1024 // 1GB default
                };

                _partitions[partitionId] = partition;

                // Add to range map
                var rangeKey = startRange ?? partitionId;
                _rangeMap[rangeKey] = partitionId;
            }
        }

        public void RemovePartition(string partitionId)
        {
            lock (_lock)
            {
                var rangeKey = _rangeMap.FirstOrDefault(kvp => kvp.Value == partitionId).Key;
                if (rangeKey != null)
                {
                    _rangeMap.Remove(rangeKey);
                }

                _partitions.Remove(partitionId);
            }
        }

        public Dictionary<string, object> GetPartitionStats()
        {
            lock (_lock)
            {
                return new Dictionary<string, object>
                {
                    ["TotalPartitions"] = _partitions.Count,
                    ["ActivePartitions"] = _partitions.Values.Count(p => p.IsActive),
                    ["RangeMapSize"] = _rangeMap.Count,
                    ["PartitionDetails"] = _partitions.ToDictionary(
                        kvp => kvp.Key,
                        kvp => new
                        {
                            kvp.Value.Name,
                            kvp.Value.IsActive,
                            kvp.Value.Utilization,
                            kvp.Value.ItemCount,
                            kvp.Value.UsedSpace,
                            kvp.Value.Capacity
                        })
                };
            }
        }
    }

    public class DirectoryBasedPartitioningStrategy : ICachePartitioningStrategy
    {
        private readonly Dictionary<string, string> _directoryMap;
        private readonly Dictionary<string, CachePartition> _partitions;
        private readonly object _lock = new object();

        public DirectoryBasedPartitioningStrategy()
        {
            _directoryMap = new Dictionary<string, string>();
            _partitions = new Dictionary<string, CachePartition>();
        }

        public string GetPartition(string key)
        {
            lock (_lock)
            {
                // Extract directory from key (assuming key format like "dir1/dir2/item")
                var directory = ExtractDirectory(key);

                if (_directoryMap.TryGetValue(directory, out var partitionId))
                {
                    return partitionId;
                }

                // Use default partition if no specific mapping
                return _partitions.Keys.FirstOrDefault();
            }
        }

        public List<string> GetPartitions()
        {
            lock (_lock)
            {
                return _partitions.Keys.ToList();
            }
        }

        public void AddPartition(string partitionId)
        {
            lock (_lock)
            {
                if (_partitions.ContainsKey(partitionId))
                    return;

                var partition = new CachePartition
                {
                    Id = partitionId,
                    Name = $"partition-{partitionId}",
                    CreatedAt = DateTime.UtcNow,
                    LastAccessed = DateTime.UtcNow,
                    Capacity = 1024 * 1024 * 1024 // 1GB default
                };

                _partitions[partitionId] = partition;
            }
        }

        public void MapDirectoryToPartition(string directory, string partitionId)
        {
            lock (_lock)
            {
                if (_partitions.ContainsKey(partitionId))
                {
                    _directoryMap[directory] = partitionId;
                }
            }
        }

        public void RemovePartition(string partitionId)
        {
            lock (_lock)
            {
                // Remove all directory mappings for this partition
                var directoriesToRemove = _directoryMap
                    .Where(kvp => kvp.Value == partitionId)
                    .Select(kvp => kvp.Key)
                    .ToList();

                foreach (var directory in directoriesToRemove)
                {
                    _directoryMap.Remove(directory);
                }

                _partitions.Remove(partitionId);
            }
        }

        public Dictionary<string, object> GetPartitionStats()
        {
            lock (_lock)
            {
                return new Dictionary<string, object>
                {
                    ["TotalPartitions"] = _partitions.Count,
                    ["ActivePartitions"] = _partitions.Values.Count(p => p.IsActive),
                    ["DirectoryMappings"] = _directoryMap.Count,
                    ["PartitionDetails"] = _partitions.ToDictionary(
                        kvp => kvp.Key,
                        kvp => new
                        {
                            kvp.Value.Name,
                            kvp.Value.IsActive,
                            kvp.Value.Utilization,
                            kvp.Value.ItemCount,
                            kvp.Value.UsedSpace,
                            kvp.Value.Capacity
                        }),
                    ["DirectoryMappings"] = _directoryMap
                };
            }
        }

        private string ExtractDirectory(string key)
        {
            var lastSlashIndex = key.LastIndexOf('/');
            if (lastSlashIndex > 0)
            {
                return key.Substring(0, lastSlashIndex);
            }
            return "root";
        }
    }

    public class RoundRobinPartitioningStrategy : ICachePartitioningStrategy
    {
        private readonly List<string> _partitions;
        private readonly Dictionary<string, CachePartition> _partitionDetails;
        private int _currentIndex;
        private readonly object _lock = new object();

        public RoundRobinPartitioningStrategy()
        {
            _partitions = new List<string>();
            _partitionDetails = new Dictionary<string, CachePartition>();
            _currentIndex = 0;
        }

        public string GetPartition(string key)
        {
            lock (_lock)
            {
                if (_partitions.Count == 0)
                    return null;

                // Use round-robin distribution
                var partitionId = _partitions[_currentIndex];
                _currentIndex = (_currentIndex + 1) % _partitions.Count;

                return partitionId;
            }
        }

        public List<string> GetPartitions()
        {
            lock (_lock)
            {
                return _partitions.ToList();
            }
        }

        public void AddPartition(string partitionId)
        {
            lock (_lock)
            {
                if (_partitions.Contains(partitionId))
                    return;

                var partition = new CachePartition
                {
                    Id = partitionId,
                    Name = $"partition-{partitionId}",
                    CreatedAt = DateTime.UtcNow,
                    LastAccessed = DateTime.UtcNow,
                    Capacity = 1024 * 1024 * 1024 // 1GB default
                };

                _partitions.Add(partitionId);
                _partitionDetails[partitionId] = partition;
            }
        }

        public void RemovePartition(string partitionId)
        {
            lock (_lock)
            {
                _partitions.Remove(partitionId);
                _partitionDetails.Remove(partitionId);

                // Adjust current index if necessary
                if (_currentIndex >= _partitions.Count)
                {
                    _currentIndex = 0;
                }
            }
        }

        public Dictionary<string, object> GetPartitionStats()
        {
            lock (_lock)
            {
                return new Dictionary<string, object>
                {
                    ["TotalPartitions"] = _partitions.Count,
                    ["ActivePartitions"] = _partitionDetails.Values.Count(p => p.IsActive),
                    ["CurrentIndex"] = _currentIndex,
                    ["PartitionDetails"] = _partitionDetails.ToDictionary(
                        kvp => kvp.Key,
                        kvp => new
                        {
                            kvp.Value.Name,
                            kvp.Value.IsActive,
                            kvp.Value.Utilization,
                            kvp.Value.ItemCount,
                            kvp.Value.UsedSpace,
                            kvp.Value.Capacity
                        })
                };
            }
        }
    }

    public class PartitionedCache
    {
        private readonly Dictionary<string, ConcurrentDictionary<string, object>> _partitionCaches;
        private readonly ICachePartitioningStrategy _partitioningStrategy;
        private readonly ILogger<PartitionedCache> _logger;

        public PartitionedCache(
            ICachePartitioningStrategy partitioningStrategy,
            ILogger<PartitionedCache> logger)
        {
            _partitioningStrategy = partitioningStrategy;
            _logger = logger;
            _partitionCaches = new Dictionary<string, ConcurrentDictionary<string, object>>();
        }

        public async Task<T> GetAsync<T>(string key)
        {
            var partitionId = _partitioningStrategy.GetPartition(key);
            if (string.IsNullOrEmpty(partitionId))
            {
                return default(T);
            }

            if (_partitionCaches.TryGetValue(partitionId, out var partitionCache))
            {
                if (partitionCache.TryGetValue(key, out var value))
                {
                    return (T)value;
                }
            }

            return default(T);
        }

        public async Task SetAsync<T>(string key, T value, TimeSpan? expiration = null)
        {
            var partitionId = _partitioningStrategy.GetPartition(key);
            if (string.IsNullOrEmpty(partitionId))
            {
                throw new InvalidOperationException("No available partition for key");
            }

            var partitionCache = _partitionCaches.GetOrAdd(partitionId, 
                _ => new ConcurrentDictionary<string, object>());

            partitionCache.AddOrUpdate(key, value, (k, v) => value);

            _logger.LogDebug($"Set key {key} in partition {partitionId}");
        }

        public async Task RemoveAsync(string key)
        {
            var partitionId = _partitioningStrategy.GetPartition(key);
            if (!string.IsNullOrEmpty(partitionId) && 
                _partitionCaches.TryGetValue(partitionId, out var partitionCache))
            {
                partitionCache.TryRemove(key, out _);
                _logger.LogDebug($"Removed key {key} from partition {partitionId}");
            }
        }

        public void AddPartition(string partitionId)
        {
            _partitioningStrategy.AddPartition(partitionId);
            _partitionCaches[partitionId] = new ConcurrentDictionary<string, object>();
            _logger.LogInformation($"Added partition: {partitionId}");
        }

        public void RemovePartition(string partitionId)
        {
            _partitioningStrategy.RemovePartition(partitionId);
            _partitionCaches.Remove(partitionId);
            _logger.LogInformation($"Removed partition: {partitionId}");
        }

        public Dictionary<string, object> GetPartitionStats()
        {
            var stats = _partitioningStrategy.GetPartitionStats();

            // Add cache-specific stats
            stats["TotalCachedItems"] = _partitionCaches.Values.Sum(cache => cache.Count);
            stats["PartitionItemCounts"] = _partitionCaches.ToDictionary(
                kvp => kvp.Key,
                kvp => kvp.Value.Count);

            return stats;
        }

        public List<string> GetPartitions()
        {
            return _partitioningStrategy.GetPartitions();
        }
    }

    public class PartitionedCacheManager
    {
        private readonly Dictionary<PartitioningStrategy, ICachePartitioningStrategy> _strategies;
        private readonly Dictionary<PartitioningStrategy, PartitionedCache> _caches;
        private readonly ILogger<PartitionedCacheManager> _logger;

        public PartitionedCacheManager(ILogger<PartitionedCacheManager> logger)
        {
            _logger = logger;
            _strategies = new Dictionary<PartitioningStrategy, ICachePartitioningStrategy>();
            _caches = new Dictionary<PartitioningStrategy, PartitionedCache>();

            // Initialize default strategies
            InitializeDefaultStrategies();
        }

        private void InitializeDefaultStrategies()
        {
            RegisterStrategy(PartitioningStrategy.HashBased, new HashBasedPartitioningStrategy());
            RegisterStrategy(PartitioningStrategy.RangeBased, new RangeBasedPartitioningStrategy());
            RegisterStrategy(PartitioningStrategy.DirectoryBased, new DirectoryBasedPartitioningStrategy());
            RegisterStrategy(PartitioningStrategy.RoundRobin, new RoundRobinPartitioningStrategy());
        }

        public void RegisterStrategy(PartitioningStrategy strategy, ICachePartitioningStrategy implementation)
        {
            _strategies[strategy] = implementation;
            _caches[strategy] = new PartitionedCache(implementation, _logger);
            _logger.LogInformation($"Registered partitioning strategy: {strategy}");
        }

        public async Task<T> GetAsync<T>(PartitioningStrategy strategy, string key)
        {
            if (_caches.TryGetValue(strategy, out var cache))
            {
                return await cache.GetAsync<T>(key);
            }

            throw new ArgumentException($"Partitioning strategy {strategy} not found");
        }

        public async Task SetAsync<T>(PartitioningStrategy strategy, string key, T value, TimeSpan? expiration = null)
        {
            if (_caches.TryGetValue(strategy, out var cache))
            {
                await cache.SetAsync(key, value, expiration);
            }
            else
            {
                throw new ArgumentException($"Partitioning strategy {strategy} not found");
            }
        }

        public async Task RemoveAsync(PartitioningStrategy strategy, string key)
        {
            if (_caches.TryGetValue(strategy, out var cache))
            {
                await cache.RemoveAsync(key);
            }
            else
            {
                throw new ArgumentException($"Partitioning strategy {strategy} not found");
            }
        }

        public void AddPartition(PartitioningStrategy strategy, string partitionId)
        {
            if (_caches.TryGetValue(strategy, out var cache))
            {
                cache.AddPartition(partitionId);
            }
            else
            {
                throw new ArgumentException($"Partitioning strategy {strategy} not found");
            }
        }

        public void RemovePartition(PartitioningStrategy strategy, string partitionId)
        {
            if (_caches.TryGetValue(strategy, out var cache))
            {
                cache.RemovePartition(partitionId);
            }
            else
            {
                throw new ArgumentException($"Partitioning strategy {strategy} not found");
            }
        }

        public Dictionary<string, object> GetStats(PartitioningStrategy strategy)
        {
            if (_caches.TryGetValue(strategy, out var cache))
            {
                return cache.GetPartitionStats();
            }

            throw new ArgumentException($"Partitioning strategy {strategy} not found");
        }

        public List<string> GetPartitions(PartitioningStrategy strategy)
        {
            if (_caches.TryGetValue(strategy, out var cache))
            {
                return cache.GetPartitions();
            }

            throw new ArgumentException($"Partitioning strategy {strategy} not found");
        }
    }
}

49. Search Result Cache

• Query normalization and fuzzy matching
• Result ranking and analytics
• Stale-while-revalidate for search results
• Query pattern invalidation

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using System.Timers;
using Microsoft.Extensions.Logging;
using System.Text.Json;
using System.Security.Cryptography;
using System.Text;
using System.Text.RegularExpressions;

namespace CachingSystem
{
    public class SearchQuery
    {
        public string OriginalQuery { get; set; }
        public string NormalizedQuery { get; set; }
        public Dictionary<string, object> Parameters { get; set; } = new Dictionary<string, object>();
        public string QueryHash { get; set; }
        public DateTime Timestamp { get; set; }
        public int ResultCount { get; set; }
        public TimeSpan ExecutionTime { get; set; }
        public int AccessCount { get; set; }
    }

    public class SearchResult
    {
        public string Id { get; set; }
        public string Title { get; set; }
        public string Content { get; set; }
        public double Score { get; set; }
        public Dictionary<string, object> Metadata { get; set; } = new Dictionary<string, object>();
        public DateTime IndexedAt { get; set; }
        public List<string> Tags { get; set; } = new List<string>();
    }

    public class CachedSearchResult
    {
        public string QueryHash { get; set; }
        public List<SearchResult> Results { get; set; }
        public DateTime CachedAt { get; set; }
        public DateTime ExpiresAt { get; set; }
        public int AccessCount { get; set; }
        public DateTime LastAccessed { get; set; }
        public TimeSpan OriginalExecutionTime { get; set; }
        public Dictionary<string, object> QueryParameters { get; set; }
        public bool IsStale { get; set; }

        public bool IsExpired => DateTime.UtcNow > ExpiresAt;
        public TimeSpan Age => DateTime.UtcNow - CachedAt;
    }

    public class SearchCacheConfiguration
    {
        public TimeSpan DefaultTTL { get; set; } = TimeSpan.FromHours(1);
        public TimeSpan StaleWhileRevalidate { get; set; } = TimeSpan.FromMinutes(5);
        public int MaxCachedQueries { get; set; } = 10000;
        public int MaxResultsPerQuery { get; set; } = 1000;
        public bool EnableQueryNormalization { get; set; } = true;
        public bool EnableFuzzyMatching { get; set; } = true;
        public double FuzzyThreshold { get; set; } = 0.8;
        public bool EnableResultRanking { get; set; } = true;
        public bool EnableQueryAnalytics { get; set; } = true;
    }

    public class SearchResultCache
    {
        private readonly ConcurrentDictionary<string, CachedSearchResult> _cache;
        private readonly Dictionary<string, SearchQuery> _queryAnalytics;
        private readonly SearchCacheConfiguration _config;
        private readonly ILogger<SearchResultCache> _logger;
        private readonly Timer _cleanupTimer;
        private readonly Timer _analyticsTimer;
        private readonly IQueryNormalizer _queryNormalizer;
        private readonly IResultRanker _resultRanker;

        public SearchResultCache(
            SearchCacheConfiguration config,
            ILogger<SearchResultCache> logger,
            IQueryNormalizer queryNormalizer = null,
            IResultRanker resultRanker = null)
        {
            _config = config;
            _logger = logger;
            _cache = new ConcurrentDictionary<string, CachedSearchResult>();
            _queryAnalytics = new Dictionary<string, SearchQuery>();
            _queryNormalizer = queryNormalizer ?? new DefaultQueryNormalizer();
            _resultRanker = resultRanker ?? new DefaultResultRanker();

            // Cleanup expired results every 5 minutes
            _cleanupTimer = new Timer(300000);
            _cleanupTimer.Elapsed += CleanupExpiredResults;
            _cleanupTimer.Start();

            // Update analytics every 10 minutes
            if (_config.EnableQueryAnalytics)
            {
                _analyticsTimer = new Timer(600000);
                _analyticsTimer.Elapsed += UpdateAnalytics;
                _analyticsTimer.Start();
            }
        }

        public async Task<List<SearchResult>> GetSearchResultsAsync(
            string query, 
            Dictionary<string, object> parameters = null,
            int maxResults = 100)
        {
            var startTime = DateTime.UtcNow;
            var normalizedQuery = await NormalizeQueryAsync(query);
            var queryHash = GenerateQueryHash(normalizedQuery, parameters);

            // Check cache first
            if (_cache.TryGetValue(queryHash, out var cachedResult))
            {
                if (!cachedResult.IsExpired)
                {
                    cachedResult.AccessCount++;
                    cachedResult.LastAccessed = DateTime.UtcNow;

                    // Update analytics
                    if (_config.EnableQueryAnalytics)
                    {
                        UpdateQueryAnalytics(query, cachedResult.OriginalExecutionTime);
                    }

                    _logger.LogDebug($"Cache hit for query: {query}");
                    return cachedResult.Results.Take(maxResults).ToList();
                }
                else if (cachedResult.Age < _config.StaleWhileRevalidate)
                {
                    // Return stale result while revalidating in background
                    _ = Task.Run(async () => await RevalidateResultsAsync(query, parameters, queryHash));
                    return cachedResult.Results.Take(maxResults).ToList();
                }
            }

            // Try fuzzy matching if enabled
            if (_config.EnableFuzzyMatching)
            {
                var fuzzyMatch = await FindFuzzyMatchAsync(normalizedQuery, parameters);
                if (fuzzyMatch != null)
                {
                    fuzzyMatch.AccessCount++;
                    fuzzyMatch.LastAccessed = DateTime.UtcNow;
                    _logger.LogDebug($"Fuzzy cache hit for query: {query}");
                    return fuzzyMatch.Results.Take(maxResults).ToList();
                }
            }

            // Execute search and cache results
            var results = await ExecuteSearchAsync(query, parameters, maxResults);
            var executionTime = DateTime.UtcNow - startTime;

            // Cache the results
            var newCachedResult = new CachedSearchResult
            {
                QueryHash = queryHash,
                Results = results,
                CachedAt = DateTime.UtcNow,
                ExpiresAt = DateTime.UtcNow.Add(_config.DefaultTTL),
                AccessCount = 1,
                LastAccessed = DateTime.UtcNow,
                OriginalExecutionTime = executionTime,
                QueryParameters = parameters ?? new Dictionary<string, object>()
            };

            _cache.TryAdd(queryHash, newCachedResult);

            // Update analytics
            if (_config.EnableQueryAnalytics)
            {
                UpdateQueryAnalytics(query, executionTime);
            }

            // Evict old entries if cache is full
            if (_cache.Count > _config.MaxCachedQueries)
            {
                await EvictOldEntriesAsync();
            }

            _logger.LogInformation($"Cached search results for query: {query} ({results.Count} results)");
            return results;
        }

        public async Task<List<SearchResult>> GetSimilarResultsAsync(string query, int maxResults = 10)
        {
            var normalizedQuery = await NormalizeQueryAsync(query);
            var similarQueries = new List<CachedSearchResult>();

            foreach (var cachedResult in _cache.Values)
            {
                var similarity = CalculateQuerySimilarity(normalizedQuery, cachedResult.QueryHash);
                if (similarity >= _config.FuzzyThreshold)
                {
                    similarQueries.Add(cachedResult);
                }
            }

            // Sort by similarity and recency
            var sortedResults = similarQueries
                .OrderByDescending(cr => cr.AccessCount)
                .ThenByDescending(cr => cr.LastAccessed)
                .Take(maxResults)
                .SelectMany(cr => cr.Results)
                .Distinct()
                .Take(maxResults)
                .ToList();

            return sortedResults;
        }

        public async Task InvalidateResultsAsync(string queryPattern)
        {
            var normalizedPattern = await NormalizeQueryAsync(queryPattern);
            var keysToRemove = new List<string>();

            foreach (var kvp in _cache)
            {
                if (IsQueryMatch(kvp.Value.QueryHash, normalizedPattern))
                {
                    keysToRemove.Add(kvp.Key);
                }
            }

            foreach (var key in keysToRemove)
            {
                _cache.TryRemove(key, out _);
            }

            _logger.LogInformation($"Invalidated {keysToRemove.Count} cached results for pattern: {queryPattern}");
        }

        public async Task WarmCacheAsync(List<string> popularQueries)
        {
            var warmingTasks = popularQueries.Select(async query =>
            {
                try
                {
                    await GetSearchResultsAsync(query);
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, $"Failed to warm cache for query: {query}");
                }
            });

            await Task.WhenAll(warmingTasks);
            _logger.LogInformation($"Warmed cache with {popularQueries.Count} queries");
        }

        private async Task<string> NormalizeQueryAsync(string query)
        {
            if (!_config.EnableQueryNormalization)
            {
                return query;
            }

            return await _queryNormalizer.NormalizeAsync(query);
        }

        private string GenerateQueryHash(string normalizedQuery, Dictionary<string, object> parameters)
        {
            var hashInput = normalizedQuery;

            if (parameters != null && parameters.Any())
            {
                var sortedParams = parameters
                    .OrderBy(kvp => kvp.Key)
                    .Select(kvp => $"{kvp.Key}:{kvp.Value}")
                    .ToList();
                hashInput += "|" + string.Join("|", sortedParams);
            }

            using (var sha256 = SHA256.Create())
            {
                var hash = sha256.ComputeHash(Encoding.UTF8.GetBytes(hashInput));
                return Convert.ToBase64String(hash).Replace("/", "_").Replace("+", "-").Substring(0, 16);
            }
        }

        private async Task<CachedSearchResult> FindFuzzyMatchAsync(string normalizedQuery, Dictionary<string, object> parameters)
        {
            var bestMatch = default(CachedSearchResult);
            var bestSimilarity = 0.0;

            foreach (var cachedResult in _cache.Values)
            {
                if (cachedResult.IsExpired)
                    continue;

                var similarity = CalculateQuerySimilarity(normalizedQuery, cachedResult.QueryHash);
                if (similarity > bestSimilarity && similarity >= _config.FuzzyThreshold)
                {
                    bestSimilarity = similarity;
                    bestMatch = cachedResult;
                }
            }

            return bestMatch;
        }

        private double CalculateQuerySimilarity(string query1, string query2)
        {
            // Simple Levenshtein distance-based similarity
            var distance = CalculateLevenshteinDistance(query1, query2);
            var maxLength = Math.Max(query1.Length, query2.Length);
            return maxLength == 0 ? 1.0 : 1.0 - ((double)distance / maxLength);
        }

        private int CalculateLevenshteinDistance(string s1, string s2)
        {
            var matrix = new int[s1.Length + 1, s2.Length + 1];

            for (int i = 0; i <= s1.Length; i++)
                matrix[i, 0] = i;

            for (int j = 0; j <= s2.Length; j++)
                matrix[0, j] = j;

            for (int i = 1; i <= s1.Length; i++)
            {
                for (int j = 1; j <= s2.Length; j++)
                {
                    var cost = s1[i - 1] == s2[j - 1] ? 0 : 1;
                    matrix[i, j] = Math.Min(
                        Math.Min(matrix[i - 1, j] + 1, matrix[i, j - 1] + 1),
                        matrix[i - 1, j - 1] + cost);
                }
            }

            return matrix[s1.Length, s2.Length];
        }

        private bool IsQueryMatch(string queryHash, string pattern)
        {
            // Simple pattern matching - in production, use regex
            return queryHash.Contains(pattern) || pattern.Contains(queryHash);
        }

        private async Task<List<SearchResult>> ExecuteSearchAsync(string query, Dictionary<string, object> parameters, int maxResults)
        {
            // Simulate search execution
            await Task.Delay(100); // Simulate search time

            var results = new List<SearchResult>();
            for (int i = 0; i < Math.Min(maxResults, 10); i++)
            {
                results.Add(new SearchResult
                {
                    Id = $"result-{i}",
                    Title = $"Search Result {i} for '{query}'",
                    Content = $"This is the content for result {i} matching query '{query}'",
                    Score = 1.0 - (i * 0.1),
                    IndexedAt = DateTime.UtcNow.AddDays(-i),
                    Tags = new List<string> { "tag1", "tag2" }
                });
            }

            // Apply ranking if enabled
            if (_config.EnableResultRanking)
            {
                results = await _resultRanker.RankResultsAsync(results, query, parameters);
            }

            return results;
        }

        private async Task RevalidateResultsAsync(string query, Dictionary<string, object> parameters, string queryHash)
        {
            try
            {
                var newResults = await ExecuteSearchAsync(query, parameters, _config.MaxResultsPerQuery);

                if (_cache.TryGetValue(queryHash, out var existingResult))
                {
                    existingResult.Results = newResults;
                    existingResult.CachedAt = DateTime.UtcNow;
                    existingResult.ExpiresAt = DateTime.UtcNow.Add(_config.DefaultTTL);
                    existingResult.IsStale = false;
                }

                _logger.LogDebug($"Revalidated results for query: {query}");
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Failed to revalidate results for query: {query}");
            }
        }

        private void UpdateQueryAnalytics(string query, TimeSpan executionTime)
        {
            var normalizedQuery = _queryNormalizer.NormalizeAsync(query).Result;

            if (_queryAnalytics.TryGetValue(normalizedQuery, out var existingQuery))
            {
                existingQuery.AccessCount++;
                existingQuery.ExecutionTime = TimeSpan.FromTicks(
                    (existingQuery.ExecutionTime.Ticks + executionTime.Ticks) / 2);
            }
            else
            {
                _queryAnalytics[normalizedQuery] = new SearchQuery
                {
                    OriginalQuery = query,
                    NormalizedQuery = normalizedQuery,
                    Timestamp = DateTime.UtcNow,
                    ExecutionTime = executionTime,
                    AccessCount = 1
                };
            }
        }

        private async Task EvictOldEntriesAsync()
        {
            var entriesToEvict = _cache.Values
                .OrderBy(cr => cr.AccessCount)
                .ThenBy(cr => cr.LastAccessed)
                .Take(_cache.Count - _config.MaxCachedQueries + 100) // Evict extra to make room
                .Select(cr => cr.QueryHash)
                .ToList();

            foreach (var queryHash in entriesToEvict)
            {
                _cache.TryRemove(queryHash, out _);
            }

            _logger.LogInformation($"Evicted {entriesToEvict.Count} old cache entries");
        }

        private void CleanupExpiredResults(object sender, ElapsedEventArgs e)
        {
            var expiredKeys = _cache
                .Where(kvp => kvp.Value.IsExpired)
                .Select(kvp => kvp.Key)
                .ToList();

            foreach (var key in expiredKeys)
            {
                _cache.TryRemove(key, out _);
            }

            if (expiredKeys.Count > 0)
            {
                _logger.LogDebug($"Cleaned up {expiredKeys.Count} expired search results");
            }
        }

        private void UpdateAnalytics(object sender, ElapsedEventArgs e)
        {
            // In a real implementation, this would send analytics to a monitoring system
            _logger.LogDebug($"Updated analytics for {_queryAnalytics.Count} queries");
        }

        public Dictionary<string, object> GetCacheStats()
        {
            return new Dictionary<string, object>
            {
                ["TotalCachedQueries"] = _cache.Count,
                ["ExpiredQueries"] = _cache.Values.Count(cr => cr.IsExpired),
                ["StaleQueries"] = _cache.Values.Count(cr => cr.IsStale),
                ["AverageAccessCount"] = _cache.Values.Any() ? 
                    _cache.Values.Average(cr => cr.AccessCount) : 0,
                ["AverageAge"] = _cache.Values.Any() ? 
                    _cache.Values.Average(cr => cr.Age.TotalMinutes) : 0,
                ["TotalAnalyticsQueries"] = _queryAnalytics.Count,
                ["MostPopularQueries"] = _queryAnalytics.Values
                    .OrderByDescending(q => q.AccessCount)
                    .Take(5)
                    .Select(q => new { q.OriginalQuery, q.AccessCount })
                    .ToList()
            };
        }
    }

    public class DefaultQueryNormalizer : IQueryNormalizer
    {
        public async Task<string> NormalizeAsync(string query)
        {
            if (string.IsNullOrWhiteSpace(query))
                return string.Empty;

            // Convert to lowercase
            var normalized = query.ToLowerInvariant();

            // Remove extra whitespace
            normalized = Regex.Replace(normalized, @"\s+", " ").Trim();

            // Remove common stop words (simplified)
            var stopWords = new[] { "the", "a", "an", "and", "or", "but", "in", "on", "at", "to", "for", "of", "with", "by" };
            var words = normalized.Split(' ');
            var filteredWords = words.Where(word => !stopWords.Contains(word)).ToArray();

            return string.Join(" ", filteredWords);
        }
    }

    public class DefaultResultRanker : IResultRanker
    {
        public async Task<List<SearchResult>> RankResultsAsync(
            List<SearchResult> results, 
            string query, 
            Dictionary<string, object> parameters)
        {
            // Simple ranking based on score and recency
            return results
                .OrderByDescending(r => r.Score)
                .ThenByDescending(r => r.IndexedAt)
                .ToList();
        }
    }

    // Interfaces for demonstration
    public interface IQueryNormalizer
    {
        Task<string> NormalizeAsync(string query);
    }

    public interface IResultRanker
    {
        Task<List<SearchResult>> RankResultsAsync(List<SearchResult> results, string query, Dictionary<string, object> parameters);
    }
}

50. Cache Compression

• Multiple compression algorithms (GZip, Deflate, Brotli)
• Adaptive compression with algorithm selection
• Content-type specific compression
• Compression analytics and optimization

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using System.Text.Json;
using System.IO;
using System.IO.Compression;
using System.Text;
using System.Security.Cryptography;

namespace CachingSystem
{
    public enum CompressionAlgorithm
    {
        None,
        GZip,
        Deflate,
        Brotli,
        LZ4,
        Zstd
    }

    public interface ICacheCompressionStrategy
    {
        Task<byte[]> CompressAsync(byte[] data);
        Task<byte[]> DecompressAsync(byte[] compressedData);
        CompressionAlgorithm Algorithm { get; }
        double GetCompressionRatio(byte[] original, byte[] compressed);
        bool ShouldCompress(byte[] data, string contentType = null);
    }

    public class CompressedCacheEntry
    {
        public byte[] Data { get; set; }
        public CompressionAlgorithm Algorithm { get; set; }
        public int OriginalSize { get; set; }
        public int CompressedSize { get; set; }
        public DateTime CompressedAt { get; set; }
        public string ContentType { get; set; }

        public double CompressionRatio => (double)CompressedSize / OriginalSize;
        public int SpaceSaved => OriginalSize - CompressedSize;
    }

    public class CompressionConfiguration
    {
        public int MinSizeToCompress { get; set; } = 1024; // 1KB
        public double MinCompressionRatio { get; set; } = 0.8; // 20% compression
        public bool EnableAdaptiveCompression { get; set; } = true;
        public Dictionary<string, CompressionAlgorithm> ContentTypeCompression { get; set; } = new Dictionary<string, CompressionAlgorithm>();
        public bool EnableCompressionAnalytics { get; set; } = true;
        public int CompressionLevel { get; set; } = 6; // Default compression level
    }

    public class GZipCompressionStrategy : ICacheCompressionStrategy
    {
        private readonly CompressionConfiguration _config;
        private readonly ILogger<GZipCompressionStrategy> _logger;

        public GZipCompressionStrategy(CompressionConfiguration config, ILogger<GZipCompressionStrategy> logger)
        {
            _config = config;
            _logger = logger;
        }

        public CompressionAlgorithm Algorithm => CompressionAlgorithm.GZip;

        public async Task<byte[]> CompressAsync(byte[] data)
        {
            try
            {
                using (var output = new MemoryStream())
                {
                    using (var gzip = new GZipStream(output, CompressionMode.Compress, true))
                    {
                        await gzip.WriteAsync(data, 0, data.Length);
                    }
                    return output.ToArray();
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error compressing data with GZip");
                return data; // Return original data if compression fails
            }
        }

        public async Task<byte[]> DecompressAsync(byte[] compressedData)
        {
            try
            {
                using (var input = new MemoryStream(compressedData))
                using (var output = new MemoryStream())
                {
                    using (var gzip = new GZipStream(input, CompressionMode.Decompress))
                    {
                        await gzip.CopyToAsync(output);
                    }
                    return output.ToArray();
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error decompressing data with GZip");
                throw;
            }
        }

        public double GetCompressionRatio(byte[] original, byte[] compressed)
        {
            if (original.Length == 0) return 1.0;
            return (double)compressed.Length / original.Length;
        }

        public bool ShouldCompress(byte[] data, string contentType = null)
        {
            if (data.Length < _config.MinSizeToCompress)
                return false;

            // Check content type specific settings
            if (!string.IsNullOrEmpty(contentType) && 
                _config.ContentTypeCompression.TryGetValue(contentType, out var algorithm))
            {
                return algorithm == CompressionAlgorithm.GZip;
            }

            return true;
        }
    }

    public class DeflateCompressionStrategy : ICacheCompressionStrategy
    {
        private readonly CompressionConfiguration _config;
        private readonly ILogger<DeflateCompressionStrategy> _logger;

        public DeflateCompressionStrategy(CompressionConfiguration config, ILogger<DeflateCompressionStrategy> logger)
        {
            _config = config;
            _logger = logger;
        }

        public CompressionAlgorithm Algorithm => CompressionAlgorithm.Deflate;

        public async Task<byte[]> CompressAsync(byte[] data)
        {
            try
            {
                using (var output = new MemoryStream())
                {
                    using (var deflate = new DeflateStream(output, CompressionMode.Compress, true))
                    {
                        await deflate.WriteAsync(data, 0, data.Length);
                    }
                    return output.ToArray();
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error compressing data with Deflate");
                return data;
            }
        }

        public async Task<byte[]> DecompressAsync(byte[] compressedData)
        {
            try
            {
                using (var input = new MemoryStream(compressedData))
                using (var output = new MemoryStream())
                {
                    using (var deflate = new DeflateStream(input, CompressionMode.Decompress))
                    {
                        await deflate.CopyToAsync(output);
                    }
                    return output.ToArray();
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error decompressing data with Deflate");
                throw;
            }
        }

        public double GetCompressionRatio(byte[] original, byte[] compressed)
        {
            if (original.Length == 0) return 1.0;
            return (double)compressed.Length / original.Length;
        }

        public bool ShouldCompress(byte[] data, string contentType = null)
        {
            if (data.Length < _config.MinSizeToCompress)
                return false;

            if (!string.IsNullOrEmpty(contentType) && 
                _config.ContentTypeCompression.TryGetValue(contentType, out var algorithm))
            {
                return algorithm == CompressionAlgorithm.Deflate;
            }

            return true;
        }
    }

    public class BrotliCompressionStrategy : ICacheCompressionStrategy
    {
        private readonly CompressionConfiguration _config;
        private readonly ILogger<BrotliCompressionStrategy> _logger;

        public BrotliCompressionStrategy(CompressionConfiguration config, ILogger<BrotliCompressionStrategy> logger)
        {
            _config = config;
            _logger = logger;
        }

        public CompressionAlgorithm Algorithm => CompressionAlgorithm.Brotli;

        public async Task<byte[]> CompressAsync(byte[] data)
        {
            try
            {
                using (var output = new MemoryStream())
                {
                    using (var brotli = new BrotliStream(output, CompressionMode.Compress, true))
                    {
                        await brotli.WriteAsync(data, 0, data.Length);
                    }
                    return output.ToArray();
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error compressing data with Brotli");
                return data;
            }
        }

        public async Task<byte[]> DecompressAsync(byte[] compressedData)
        {
            try
            {
                using (var input = new MemoryStream(compressedData))
                using (var output = new MemoryStream())
                {
                    using (var brotli = new BrotliStream(input, CompressionMode.Decompress))
                    {
                        await brotli.CopyToAsync(output);
                    }
                    return output.ToArray();
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error decompressing data with Brotli");
                throw;
            }
        }

        public double GetCompressionRatio(byte[] original, byte[] compressed)
        {
            if (original.Length == 0) return 1.0;
            return (double)compressed.Length / original.Length;
        }

        public bool ShouldCompress(byte[] data, string contentType = null)
        {
            if (data.Length < _config.MinSizeToCompress)
                return false;

            if (!string.IsNullOrEmpty(contentType) && 
                _config.ContentTypeCompression.TryGetValue(contentType, out var algorithm))
            {
                return algorithm == CompressionAlgorithm.Brotli;
            }

            return true;
        }
    }

    public class AdaptiveCompressionStrategy : ICacheCompressionStrategy
    {
        private readonly Dictionary<CompressionAlgorithm, ICacheCompressionStrategy> _strategies;
        private readonly CompressionConfiguration _config;
        private readonly ILogger<AdaptiveCompressionStrategy> _logger;
        private readonly Dictionary<string, CompressionAlgorithm> _contentTypeHistory;

        public AdaptiveCompressionStrategy(
            CompressionConfiguration config,
            ILogger<AdaptiveCompressionStrategy> logger)
        {
            _config = config;
            _logger = logger;
            _contentTypeHistory = new Dictionary<string, CompressionAlgorithm>();

            _strategies = new Dictionary<CompressionAlgorithm, ICacheCompressionStrategy>
            {
                [CompressionAlgorithm.GZip] = new GZipCompressionStrategy(config, logger),
                [CompressionAlgorithm.Deflate] = new DeflateCompressionStrategy(config, logger),
                [CompressionAlgorithm.Brotli] = new BrotliCompressionStrategy(config, logger)
            };
        }

        public CompressionAlgorithm Algorithm => CompressionAlgorithm.None; // Adaptive

        public async Task<byte[]> CompressAsync(byte[] data)
        {
            return await CompressAsync(data, null);
        }

        public async Task<byte[]> CompressAsync(byte[] data, string contentType)
        {
            if (!_config.EnableAdaptiveCompression || data.Length < _config.MinSizeToCompress)
            {
                return data;
            }

            var bestAlgorithm = await SelectBestAlgorithmAsync(data, contentType);
            var strategy = _strategies[bestAlgorithm];

            var compressedData = await strategy.CompressAsync(data);
            var compressionRatio = strategy.GetCompressionRatio(data, compressedData);

            // Only use compression if it meets the minimum ratio
            if (compressionRatio >= _config.MinCompressionRatio)
            {
                _logger.LogDebug($"Compressed {data.Length} bytes to {compressedData.Length} bytes using {bestAlgorithm} (ratio: {compressionRatio:F2})");
                return compressedData;
            }

            return data; // Return original if compression doesn't help
        }

        public async Task<byte[]> DecompressAsync(byte[] compressedData)
        {
            // This would need to be enhanced to detect the algorithm used
            // For now, we'll try each algorithm
            foreach (var strategy in _strategies.Values)
            {
                try
                {
                    return await strategy.DecompressAsync(compressedData);
                }
                catch
                {
                    // Try next algorithm
                }
            }

            throw new InvalidOperationException("Unable to decompress data with any known algorithm");
        }

        public double GetCompressionRatio(byte[] original, byte[] compressed)
        {
            if (original.Length == 0) return 1.0;
            return (double)compressed.Length / original.Length;
        }

        public bool ShouldCompress(byte[] data, string contentType = null)
        {
            if (data.Length < _config.MinSizeToCompress)
                return false;

            if (!string.IsNullOrEmpty(contentType) && 
                _config.ContentTypeCompression.TryGetValue(contentType, out var algorithm))
            {
                return algorithm != CompressionAlgorithm.None;
            }

            return true;
        }

        private async Task<CompressionAlgorithm> SelectBestAlgorithmAsync(byte[] data, string contentType)
        {
            // Check content type history first
            if (!string.IsNullOrEmpty(contentType) && _contentTypeHistory.TryGetValue(contentType, out var historicalAlgorithm))
            {
                return historicalAlgorithm;
            }

            // Test all algorithms and select the best one
            var results = new Dictionary<CompressionAlgorithm, double>();

            foreach (var strategy in _strategies)
            {
                try
                {
                    var compressed = await strategy.Value.CompressAsync(data);
                    var ratio = strategy.Value.GetCompressionRatio(data, compressed);
                    results[strategy.Key] = ratio;
                }
                catch (Exception ex)
                {
                    _logger.LogWarning(ex, $"Failed to test compression with {strategy.Key}");
                    results[strategy.Key] = 1.0; // No compression
                }
            }

            var bestAlgorithm = results.OrderBy(kvp => kvp.Value).First().Key;

            // Update content type history
            if (!string.IsNullOrEmpty(contentType))
            {
                _contentTypeHistory[contentType] = bestAlgorithm;
            }

            return bestAlgorithm;
        }
    }

    public class CompressedCache
    {
        private readonly ConcurrentDictionary<string, CompressedCacheEntry> _cache;
        private readonly ICacheCompressionStrategy _compressionStrategy;
        private readonly CompressionConfiguration _config;
        private readonly ILogger<CompressedCache> _logger;
        private readonly Dictionary<string, CompressionAnalytics> _analytics;

        public CompressedCache(
            ICacheCompressionStrategy compressionStrategy,
            CompressionConfiguration config,
            ILogger<CompressedCache> logger)
        {
            _compressionStrategy = compressionStrategy;
            _config = config;
            _logger = logger;
            _cache = new ConcurrentDictionary<string, CompressedCacheEntry>();
            _analytics = new Dictionary<string, CompressionAnalytics>();
        }

        public async Task<T> GetAsync<T>(string key)
        {
            if (_cache.TryGetValue(key, out var entry))
            {
                try
                {
                    var decompressedData = await _compressionStrategy.DecompressAsync(entry.Data);
                    var json = Encoding.UTF8.GetString(decompressedData);
                    var result = JsonSerializer.Deserialize<T>(json);

                    UpdateAnalytics(key, entry, false);
                    return result;
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, $"Error decompressing data for key: {key}");
                    _cache.TryRemove(key, out _);
                }
            }

            return default(T);
        }

        public async Task SetAsync<T>(string key, T value, string contentType = null)
        {
            try
            {
                var json = JsonSerializer.Serialize(value);
                var data = Encoding.UTF8.GetBytes(json);

                CompressedCacheEntry entry;

                if (_compressionStrategy.ShouldCompress(data, contentType))
                {
                    var compressedData = await _compressionStrategy.CompressAsync(data, contentType);
                    var compressionRatio = _compressionStrategy.GetCompressionRatio(data, compressedData);

                    entry = new CompressedCacheEntry
                    {
                        Data = compressedData,
                        Algorithm = _compressionStrategy.Algorithm,
                        OriginalSize = data.Length,
                        CompressedSize = compressedData.Length,
                        CompressedAt = DateTime.UtcNow,
                        ContentType = contentType
                    };

                    _logger.LogDebug($"Compressed {data.Length} bytes to {compressedData.Length} bytes for key {key} (ratio: {compressionRatio:F2})");
                }
                else
                {
                    entry = new CompressedCacheEntry
                    {
                        Data = data,
                        Algorithm = CompressionAlgorithm.None,
                        OriginalSize = data.Length,
                        CompressedSize = data.Length,
                        CompressedAt = DateTime.UtcNow,
                        ContentType = contentType
                    };
                }

                _cache.AddOrUpdate(key, entry, (k, v) => entry);
                UpdateAnalytics(key, entry, true);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Error setting compressed data for key: {key}");
            }
        }

        public async Task RemoveAsync(string key)
        {
            _cache.TryRemove(key, out _);
        }

        private void UpdateAnalytics(string key, CompressedCacheEntry entry, bool isWrite)
        {
            if (!_config.EnableCompressionAnalytics)
                return;

            var contentType = entry.ContentType ?? "unknown";

            if (!_analytics.TryGetValue(contentType, out var analytics))
            {
                analytics = new CompressionAnalytics();
                _analytics[contentType] = analytics;
            }

            if (isWrite)
            {
                analytics.TotalWrites++;
                analytics.TotalOriginalSize += entry.OriginalSize;
                analytics.TotalCompressedSize += entry.CompressedSize;
                analytics.TotalSpaceSaved += entry.SpaceSaved;

                if (entry.Algorithm != CompressionAlgorithm.None)
                {
                    analytics.CompressedWrites++;
                }
            }
            else
            {
                analytics.TotalReads++;
            }
        }

        public Dictionary<string, object> GetCompressionStats()
        {
            var stats = new Dictionary<string, object>
            {
                ["TotalEntries"] = _cache.Count,
                ["TotalOriginalSize"] = _cache.Values.Sum(e => e.OriginalSize),
                ["TotalCompressedSize"] = _cache.Values.Sum(e => e.CompressedSize),
                ["TotalSpaceSaved"] = _cache.Values.Sum(e => e.SpaceSaved),
                ["AverageCompressionRatio"] = _cache.Values.Any() ? 
                    _cache.Values.Average(e => e.CompressionRatio) : 1.0,
                ["CompressedEntries"] = _cache.Values.Count(e => e.Algorithm != CompressionAlgorithm.None),
                ["UncompressedEntries"] = _cache.Values.Count(e => e.Algorithm == CompressionAlgorithm.None)
            };

            if (_config.EnableCompressionAnalytics)
            {
                stats["ContentTypeAnalytics"] = _analytics.ToDictionary(
                    kvp => kvp.Key,
                    kvp => new
                    {
                        kvp.Value.TotalReads,
                        kvp.Value.TotalWrites,
                        kvp.Value.CompressedWrites,
                        kvp.Value.TotalOriginalSize,
                        kvp.Value.TotalCompressedSize,
                        kvp.Value.TotalSpaceSaved,
                        AverageCompressionRatio = kvp.Value.TotalOriginalSize > 0 ? 
                            (double)kvp.Value.TotalCompressedSize / kvp.Value.TotalOriginalSize : 1.0
                    });
            }

            return stats;
        }
    }

    public class CompressionAnalytics
    {
        public int TotalReads { get; set; }
        public int TotalWrites { get; set; }
        public int CompressedWrites { get; set; }
        public long TotalOriginalSize { get; set; }
        public long TotalCompressedSize { get; set; }
        public long TotalSpaceSaved { get; set; }
    }

    public class CompressedCacheManager
    {
        private readonly Dictionary<CompressionAlgorithm, ICacheCompressionStrategy> _strategies;
        private readonly Dictionary<CompressionAlgorithm, CompressedCache> _caches;
        private readonly CompressionConfiguration _config;
        private readonly ILogger<CompressedCacheManager> _logger;

        public CompressedCacheManager(
            CompressionConfiguration config,
            ILogger<CompressedCacheManager> logger)
        {
            _config = config;
            _logger = logger;
            _strategies = new Dictionary<CompressionAlgorithm, ICacheCompressionStrategy>();
            _caches = new Dictionary<CompressionAlgorithm, CompressedCache>();

            InitializeStrategies();
        }

        private void InitializeStrategies()
        {
            _strategies[CompressionAlgorithm.GZip] = new GZipCompressionStrategy(_config, _logger);
            _strategies[CompressionAlgorithm.Deflate] = new DeflateCompressionStrategy(_config, _logger);
            _strategies[CompressionAlgorithm.Brotli] = new BrotliCompressionStrategy(_config, _logger);
            _strategies[CompressionAlgorithm.None] = new AdaptiveCompressionStrategy(_config, _logger);

            foreach (var strategy in _strategies)
            {
                _caches[strategy.Key] = new CompressedCache(strategy.Value, _config, _logger);
            }
        }

        public async Task<T> GetAsync<T>(CompressionAlgorithm algorithm, string key)
        {
            if (_caches.TryGetValue(algorithm, out var cache))
            {
                return await cache.GetAsync<T>(key);
            }

            throw new ArgumentException($"Compression algorithm {algorithm} not supported");
        }

        public async Task SetAsync<T>(CompressionAlgorithm algorithm, string key, T value, string contentType = null)
        {
            if (_caches.TryGetValue(algorithm, out var cache))
            {
                await cache.SetAsync(key, value, contentType);
            }
            else
            {
                throw new ArgumentException($"Compression algorithm {algorithm} not supported");
            }
        }

        public async Task RemoveAsync(CompressionAlgorithm algorithm, string key)
        {
            if (_caches.TryGetValue(algorithm, out var cache))
            {
                await cache.RemoveAsync(key);
            }
            else
            {
                throw new ArgumentException($"Compression algorithm {algorithm} not supported");
            }
        }

        public Dictionary<string, object> GetStats(CompressionAlgorithm algorithm)
        {
            if (_caches.TryGetValue(algorithm, out var cache))
            {
                return cache.GetCompressionStats();
            }

            throw new ArgumentException($"Compression algorithm {algorithm} not supported");
        }

        public Dictionary<string, object> GetAllStats()
        {
            var allStats = new Dictionary<string, object>();

            foreach (var algorithm in _caches.Keys)
            {
                allStats[algorithm.ToString()] = GetStats(algorithm);
            }

            return allStats;
        }
    }
}

51. How would you design a secure authentication system?

Explanation: A secure authentication system should implement multiple layers of security including strong password policies, multi-factor authentication (MFA), secure session management, and protection against common attacks like brute force, session hijacking, and credential stuffing.

Key Components: - Password hashing with salt - JWT tokens for stateless authentication - Rate limiting - Account lockout mechanisms - Secure password reset flows - Session management

public class SecureAuthenticationService
{
    private readonly IUserRepository _userRepository;
    private readonly IPasswordHasher _passwordHasher;
    private readonly IJwtTokenService _jwtService;
    private readonly IRateLimiter _rateLimiter;
    private readonly ILogger<SecureAuthenticationService> _logger;

    public async Task<AuthenticationResult> AuthenticateAsync(string username, string password, string ipAddress)
    {
        // Rate limiting
        if (!await _rateLimiter.IsAllowedAsync(ipAddress, "login"))
        {
            throw new SecurityException("Too many login attempts");
        }

        // Get user
        var user = await _userRepository.GetByUsernameAsync(username);
        if (user == null)
        {
            await LogFailedAttemptAsync(username, ipAddress);
            return AuthenticationResult.Failed("Invalid credentials");
        }

        // Check if account is locked
        if (user.IsLocked)
        {
            return AuthenticationResult.Failed("Account is locked");
        }

        // Verify password
        if (!_passwordHasher.VerifyPassword(password, user.PasswordHash, user.PasswordSalt))
        {
            await LogFailedAttemptAsync(username, ipAddress);
            await CheckAndLockAccountAsync(user);
            return AuthenticationResult.Failed("Invalid credentials");
        }

        // Generate JWT token
        var token = await _jwtService.GenerateTokenAsync(user);

        // Log successful login
        await LogSuccessfulLoginAsync(user.Id, ipAddress);

        return AuthenticationResult.Success(token, user);
    }

    private async Task CheckAndLockAccountAsync(User user)
    {
        user.FailedLoginAttempts++;
        if (user.FailedLoginAttempts >= 5)
        {
            user.IsLocked = true;
            user.LockedUntil = DateTime.UtcNow.AddMinutes(30);
        }
        await _userRepository.UpdateAsync(user);
    }
}

public class PasswordHasher : IPasswordHasher
{
    public string HashPassword(string password, out string salt)
    {
        salt = GenerateSalt();
        using var pbkdf2 = new Rfc2898DeriveBytes(password, Convert.FromBase64String(salt), 10000);
        var hash = pbkdf2.GetBytes(32);
        return Convert.ToBase64String(hash);
    }

    public bool VerifyPassword(string password, string hash, string salt)
    {
        using var pbkdf2 = new Rfc2898DeriveBytes(password, Convert.FromBase64String(salt), 10000);
        var computedHash = pbkdf2.GetBytes(32);
        var storedHash = Convert.FromBase64String(hash);
        return computedHash.SequenceEqual(storedHash);
    }

    private string GenerateSalt()
    {
        var salt = new byte[32];
        using var rng = new RNGCryptoServiceProvider();
        rng.GetBytes(salt);
        return Convert.ToBase64String(salt);
    }
}

52. How do you implement OAuth 2.0?

Explanation: OAuth 2.0 is an authorization framework that allows third-party applications to access resources on behalf of users without sharing credentials. It supports multiple grant types for different use cases.

Grant Types: - Authorization Code (most secure) - Client Credentials - Resource Owner Password - Implicit (deprecated)

public class OAuth2Service
{
    private readonly IClientRepository _clientRepository;
    private readonly IUserRepository _userRepository;
    private readonly ITokenService _tokenService;
    private readonly IAuthorizationCodeRepository _authCodeRepository;

    public async Task<AuthorizationCodeResponse> CreateAuthorizationCodeAsync(
        string clientId, string redirectUri, string scope, string state)
    {
        var client = await _clientRepository.GetByIdAsync(clientId);
        if (client == null || !client.RedirectUris.Contains(redirectUri))
        {
            throw new SecurityException("Invalid client or redirect URI");
        }

        var authCode = new AuthorizationCode
        {
            Code = GenerateSecureCode(),
            ClientId = clientId,
            RedirectUri = redirectUri,
            Scope = scope,
            State = state,
            ExpiresAt = DateTime.UtcNow.AddMinutes(10)
        };

        await _authCodeRepository.SaveAsync(authCode);
        return new AuthorizationCodeResponse(authCode.Code, state);
    }

    public async Task<TokenResponse> ExchangeCodeForTokenAsync(string code, string clientId, string clientSecret)
    {
        var authCode = await _authCodeRepository.GetByCodeAsync(code);
        if (authCode == null || authCode.IsUsed || authCode.ExpiresAt < DateTime.UtcNow)
        {
            throw new SecurityException("Invalid or expired authorization code");
        }

        var client = await _clientRepository.GetByIdAsync(clientId);
        if (client == null || client.Secret != clientSecret)
        {
            throw new SecurityException("Invalid client credentials");
        }

        // Mark code as used
        authCode.IsUsed = true;
        await _authCodeRepository.UpdateAsync(authCode);

        // Generate access token
        var accessToken = await _tokenService.GenerateAccessTokenAsync(authCode.UserId, authCode.Scope);
        var refreshToken = await _tokenService.GenerateRefreshTokenAsync(authCode.UserId);

        return new TokenResponse
        {
            AccessToken = accessToken,
            RefreshToken = refreshToken,
            TokenType = "Bearer",
            ExpiresIn = 3600,
            Scope = authCode.Scope
        };
    }

    public async Task<TokenResponse> RefreshTokenAsync(string refreshToken, string clientId)
    {
        var token = await _tokenService.ValidateRefreshTokenAsync(refreshToken);
        if (token == null || token.ClientId != clientId)
        {
            throw new SecurityException("Invalid refresh token");
        }

        var newAccessToken = await _tokenService.GenerateAccessTokenAsync(token.UserId, token.Scope);
        var newRefreshToken = await _tokenService.GenerateRefreshTokenAsync(token.UserId);

        // Revoke old refresh token
        await _tokenService.RevokeRefreshTokenAsync(refreshToken);

        return new TokenResponse
        {
            AccessToken = newAccessToken,
            RefreshToken = newRefreshToken,
            TokenType = "Bearer",
            ExpiresIn = 3600,
            Scope = token.Scope
        };
    }
}

53. How would you design a role-based access control system?

Explanation: RBAC (Role-Based Access Control) assigns permissions to roles rather than directly to users. Users are assigned roles, and roles have permissions. This provides better manageability and security.

Key Concepts: - Users - Roles - Permissions - Resources - Hierarchical roles

public class RBACService
{
    private readonly IUserRepository _userRepository;
    private readonly IRoleRepository _roleRepository;
    private readonly IPermissionRepository _permissionRepository;

    public async Task<bool> HasPermissionAsync(int userId, string resource, string action)
    {
        var user = await _userRepository.GetByIdAsync(userId);
        if (user == null) return false;

        var userRoles = await _roleRepository.GetUserRolesAsync(userId);

        foreach (var role in userRoles)
        {
            if (await HasRolePermissionAsync(role.Id, resource, action))
            {
                return true;
            }
        }

        return false;
    }

    public async Task<bool> HasRolePermissionAsync(int roleId, string resource, string action)
    {
        var permissions = await _permissionRepository.GetRolePermissionsAsync(roleId);

        return permissions.Any(p => 
            p.Resource == resource && 
            p.Action == action && 
            p.IsActive);
    }

    public async Task<IEnumerable<Permission>> GetUserPermissionsAsync(int userId)
    {
        var userRoles = await _roleRepository.GetUserRolesAsync(userId);
        var permissions = new HashSet<Permission>();

        foreach (var role in userRoles)
        {
            var rolePermissions = await _permissionRepository.GetRolePermissionsAsync(role.Id);
            foreach (var permission in rolePermissions.Where(p => p.IsActive))
            {
                permissions.Add(permission);
            }
        }

        return permissions;
    }
}

[AttributeUsage(AttributeTargets.Method | AttributeTargets.Class)]
public class RequirePermissionAttribute : Attribute
{
    public string Resource { get; }
    public string Action { get; }

    public RequirePermissionAttribute(string resource, string action)
    {
        Resource = resource;
        Action = action;
    }
}

public class PermissionAuthorizationHandler : AuthorizationHandler<PermissionRequirement>
{
    private readonly RBACService _rbacService;

    protected override async Task HandleRequirementAsync(
        AuthorizationHandlerContext context, 
        PermissionRequirement requirement)
    {
        var user = context.User;
        var userId = int.Parse(user.FindFirst("sub")?.Value ?? "0");

        if (await _rbacService.HasPermissionAsync(userId, requirement.Resource, requirement.Action))
        {
            context.Succeed(requirement);
        }
    }
}

// Usage in controller
[ApiController]
[Route("api/[controller]")]
public class DocumentsController : ControllerBase
{
    [HttpGet]
    [RequirePermission("documents", "read")]
    public async Task<IActionResult> GetDocuments()
    {
        // Implementation
    }

    [HttpPost]
    [RequirePermission("documents", "create")]
    public async Task<IActionResult> CreateDocument()
    {
        // Implementation
    }
}

54. How do you implement API security?

Explanation: API security involves protecting APIs from various threats including unauthorized access, data breaches, and abuse. It includes authentication, authorization, input validation, rate limiting, and monitoring.

Security Measures: - API Keys - JWT Tokens - Rate Limiting - Input Validation - HTTPS/TLS - CORS Configuration - Request/Response Logging

public class ApiSecurityMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<ApiSecurityMiddleware> _logger;
    private readonly IRateLimiter _rateLimiter;
    private readonly IApiKeyValidator _apiKeyValidator;

    public async Task InvokeAsync(HttpContext context)
    {
        try
        {
            // Validate API key
            if (!await ValidateApiKeyAsync(context))
            {
                context.Response.StatusCode = 401;
                await context.Response.WriteAsync("Invalid API key");
                return;
            }

            // Rate limiting
            if (!await _rateLimiter.IsAllowedAsync(context))
            {
                context.Response.StatusCode = 429;
                await context.Response.WriteAsync("Rate limit exceeded");
                return;
            }

            // Add security headers
            AddSecurityHeaders(context);

            // Log request
            await LogRequestAsync(context);

            await _next(context);

            // Log response
            await LogResponseAsync(context);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Error in API security middleware");
            context.Response.StatusCode = 500;
            await context.Response.WriteAsync("Internal server error");
        }
    }

    private async Task<bool> ValidateApiKeyAsync(HttpContext context)
    {
        var apiKey = context.Request.Headers["X-API-Key"].FirstOrDefault();
        if (string.IsNullOrEmpty(apiKey))
        {
            return false;
        }

        return await _apiKeyValidator.ValidateAsync(apiKey);
    }

    private void AddSecurityHeaders(HttpContext context)
    {
        context.Response.Headers.Add("X-Content-Type-Options", "nosniff");
        context.Response.Headers.Add("X-Frame-Options", "DENY");
        context.Response.Headers.Add("X-XSS-Protection", "1; mode=block");
        context.Response.Headers.Add("Strict-Transport-Security", "max-age=31536000; includeSubDomains");
    }
}

public class ApiKeyValidator : IApiKeyValidator
{
    private readonly IApiKeyRepository _apiKeyRepository;

    public async Task<bool> ValidateAsync(string apiKey)
    {
        var key = await _apiKeyRepository.GetByKeyAsync(apiKey);
        return key != null && key.IsActive && key.ExpiresAt > DateTime.UtcNow;
    }
}

public class RateLimiter : IRateLimiter
{
    private readonly IDistributedCache _cache;

    public async Task<bool> IsAllowedAsync(HttpContext context)
    {
        var clientId = GetClientId(context);
        var key = $"rate_limit:{clientId}";

        var currentCount = await _cache.GetStringAsync(key);
        var count = currentCount == null ? 0 : int.Parse(currentCount);

        if (count >= 100) // 100 requests per minute
        {
            return false;
        }

        await _cache.SetStringAsync(key, (count + 1).ToString(), 
            new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(1) });

        return true;
    }

    private string GetClientId(HttpContext context)
    {
        return context.Request.Headers["X-API-Key"].FirstOrDefault() ?? 
               context.Connection.RemoteIpAddress?.ToString() ?? 
               "unknown";
    }
}

55. How would you design a secure file storage system?

Secure file storage needs authorization on every object operation, opaque object identifiers, scoped upload/download URLs where useful, validation of file size/type/content, malware scanning, encryption, retention/deletion policy, audit logs, and separation of metadata from object bytes. Do not rely on client-provided file names or MIME types.

56. How do you implement data encryption at rest and in transit?

Explanation: Data encryption protects sensitive information from unauthorized access. Encryption at rest protects stored data, while encryption in transit protects data during transmission.

Encryption Types: - Symmetric encryption (AES) - Asymmetric encryption (RSA) - Hashing (SHA-256) - Key management

public class DataEncryptionService
{
    private readonly IKeyVaultService _keyVault;
    private readonly IConfiguration _configuration;

    // Encryption at rest
    public async Task<string> EncryptDataAtRestAsync(string plainText, string keyName)
    {
        var key = await _keyVault.GetKeyAsync(keyName);

        using var aes = Aes.Create();
        aes.Key = key;
        aes.GenerateIV();

        using var encryptor = aes.CreateEncryptor();
        var plainBytes = Encoding.UTF8.GetBytes(plainText);
        var encryptedBytes = encryptor.TransformFinalBlock(plainBytes, 0, plainBytes.Length);

        var result = new byte[aes.IV.Length + encryptedBytes.Length];
        Buffer.BlockCopy(aes.IV, 0, result, 0, aes.IV.Length);
        Buffer.BlockCopy(encryptedBytes, 0, result, aes.IV.Length, encryptedBytes.Length);

        return Convert.ToBase64String(result);
    }

    public async Task<string> DecryptDataAtRestAsync(string encryptedText, string keyName)
    {
        var key = await _keyVault.GetKeyAsync(keyName);
        var encryptedBytes = Convert.FromBase64String(encryptedText);

        using var aes = Aes.Create();
        aes.Key = key;

        var iv = new byte[16];
        var cipherText = new byte[encryptedBytes.Length - 16];

        Buffer.BlockCopy(encryptedBytes, 0, iv, 0, 16);
        Buffer.BlockCopy(encryptedBytes, 16, cipherText, 0, cipherText.Length);

        aes.IV = iv;

        using var decryptor = aes.CreateDecryptor();
        var decryptedBytes = decryptor.TransformFinalBlock(cipherText, 0, cipherText.Length);

        return Encoding.UTF8.GetString(decryptedBytes);
    }

    // Encryption in transit (HTTPS/TLS)
    public async Task<HttpResponseMessage> SendSecureRequestAsync(string url, object data)
    {
        using var httpClient = new HttpClient();

        // Configure TLS settings
        httpClient.DefaultRequestHeaders.Add("User-Agent", "SecureApp/1.0");

        // Use HTTPS
        if (!url.StartsWith("https://"))
        {
            throw new SecurityException("HTTPS is required for secure communication");
        }

        var json = JsonSerializer.Serialize(data);
        var content = new StringContent(json, Encoding.UTF8, "application/json");

        return await httpClient.PostAsync(url, content);
    }
}

public class DatabaseEncryptionService
{
    private readonly IEncryptionService _encryptionService;
    private readonly string _encryptionKeyName;

    public async Task<T> SaveEncryptedEntityAsync<T>(T entity) where T : class
    {
        var entityType = typeof(T);
        var properties = entityType.GetProperties()
            .Where(p => p.GetCustomAttribute<EncryptedAttribute>() != null);

        foreach (var property in properties)
        {
            var value = property.GetValue(entity)?.ToString();
            if (!string.IsNullOrEmpty(value))
            {
                var encryptedValue = await _encryptionService.EncryptDataAtRestAsync(value, _encryptionKeyName);
                property.SetValue(entity, encryptedValue);
            }
        }

        // Save to database
        return entity;
    }

    public async Task<T> LoadEncryptedEntityAsync<T>(T entity) where T : class
    {
        var entityType = typeof(T);
        var properties = entityType.GetProperties()
            .Where(p => p.GetCustomAttribute<EncryptedAttribute>() != null);

        foreach (var property in properties)
        {
            var value = property.GetValue(entity)?.ToString();
            if (!string.IsNullOrEmpty(value))
            {
                var decryptedValue = await _encryptionService.DecryptDataAtRestAsync(value, _encryptionKeyName);
                property.SetValue(entity, decryptedValue);
            }
        }

        return entity;
    }
}

[AttributeUsage(AttributeTargets.Property)]
public class EncryptedAttribute : Attribute
{
}

public class User
{
    public int Id { get; set; }
    public string Username { get; set; }

    [Encrypted]
    public string SocialSecurityNumber { get; set; }

    [Encrypted]
    public string CreditCardNumber { get; set; }
}

57. How would you design a secure payment system?

A payment design requires a compliant payment provider boundary, idempotent payment requests, a durable order/payment state machine, webhook signature validation and replay handling, reconciliation, audit trails, least-privilege access, and observability. Do not store raw card data unless you are explicitly equipped to meet the required compliance obligations.

58. How do you implement audit logging?

Explanation: Audit logging records all security-relevant events for compliance, monitoring, and forensic analysis. It should be tamper-proof, comprehensive, and searchable.

Key Features: - Immutable logs - Structured logging - Event correlation - Retention policies - Real-time monitoring

public class AuditLogger : IAuditLogger
{
    private readonly ILogger<AuditLogger> _logger;
    private readonly IAuditRepository _auditRepository;
    private readonly ICurrentUserService _currentUserService;
    private readonly ICorrelationIdProvider _correlationIdProvider;

    public async Task LogAsync(AuditEvent auditEvent)
    {
        try
        {
            auditEvent.Timestamp = DateTime.UtcNow;
            auditEvent.UserId = _currentUserService.GetCurrentUserId();
            auditEvent.CorrelationId = _correlationIdProvider.GetCorrelationId();
            auditEvent.SessionId = _currentUserService.GetSessionId();
            auditEvent.IpAddress = _currentUserService.GetIpAddress();

            // Create immutable audit record
            var auditRecord = new AuditRecord
            {
                Id = Guid.NewGuid(),
                EventType = auditEvent.GetType().Name,
                EventData = JsonSerializer.Serialize(auditEvent),
                Timestamp = auditEvent.Timestamp,
                UserId = auditEvent.UserId,
                CorrelationId = auditEvent.CorrelationId,
                SessionId = auditEvent.SessionId,
                IpAddress = auditEvent.IpAddress,
                Hash = CalculateEventHash(auditEvent)
            };

            // Store in database
            await _auditRepository.SaveAsync(auditRecord);

            // Log to structured logging system
            _logger.LogInformation("Audit Event: {EventType} by User {UserId} at {Timestamp}",
                auditEvent.GetType().Name, auditEvent.UserId, auditEvent.Timestamp);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to log audit event");
            // Don't throw - audit logging should not break main functionality
        }
    }

    public async Task<IEnumerable<AuditRecord>> SearchAsync(AuditSearchCriteria criteria)
    {
        return await _auditRepository.SearchAsync(criteria);
    }

    public async Task<IEnumerable<AuditRecord>> GetUserActivityAsync(int userId, DateTime from, DateTime to)
    {
        return await _auditRepository.GetByUserAndDateRangeAsync(userId, from, to);
    }

    private string CalculateEventHash(AuditEvent auditEvent)
    {
        var eventData = JsonSerializer.Serialize(auditEvent);
        using var sha256 = SHA256.Create();
        var hashBytes = sha256.ComputeHash(Encoding.UTF8.GetBytes(eventData));
        return Convert.ToBase64String(hashBytes);
    }
}

public class AuditMiddleware
{
    private readonly RequestDelegate _next;
    private readonly IAuditLogger _auditLogger;

    public async Task InvokeAsync(HttpContext context)
    {
        var startTime = DateTime.UtcNow;
        var originalBodyStream = context.Response.Body;

        using var memoryStream = new MemoryStream();
        context.Response.Body = memoryStream;

        try
        {
            await _next(context);

            // Log successful request
            await LogRequestAsync(context, startTime, true, null);
        }
        catch (Exception ex)
        {
            // Log failed request
            await LogRequestAsync(context, startTime, false, ex);
            throw;
        }
        finally
        {
            memoryStream.Position = 0;
            await memoryStream.CopyToAsync(originalBodyStream);
        }
    }

    private async Task LogRequestAsync(HttpContext context, DateTime startTime, bool success, Exception exception)
    {
        var duration = DateTime.UtcNow - startTime;

        var auditEvent = new HttpRequestAuditEvent
        {
            Method = context.Request.Method,
            Path = context.Request.Path,
            StatusCode = context.Response.StatusCode,
            Duration = duration,
            Success = success,
            ErrorMessage = exception?.Message,
            UserAgent = context.Request.Headers["User-Agent"].ToString(),
            RequestSize = context.Request.ContentLength ?? 0,
            ResponseSize = context.Response.ContentLength ?? 0
        };

        await _auditLogger.LogAsync(auditEvent);
    }
}

public class DataAccessAuditInterceptor : IInterceptor
{
    private readonly IAuditLogger _auditLogger;

    public void Intercept(IInvocation invocation)
    {
        var startTime = DateTime.UtcNow;
        var methodName = invocation.Method.Name;
        var parameters = invocation.Arguments;

        try
        {
            invocation.Proceed();

            // Log successful data access
            var auditEvent = new DataAccessAuditEvent
            {
                Method = methodName,
                Parameters = JsonSerializer.Serialize(parameters),
                Duration = DateTime.UtcNow - startTime,
                Success = true
            };

            _auditLogger.LogAsync(auditEvent).Wait();
        }
        catch (Exception ex)
        {
            // Log failed data access
            var auditEvent = new DataAccessAuditEvent
            {
                Method = methodName,
                Parameters = JsonSerializer.Serialize(parameters),
                Duration = DateTime.UtcNow - startTime,
                Success = false,
                ErrorMessage = ex.Message
            };

            _auditLogger.LogAsync(auditEvent).Wait();
            throw;
        }
    }
}

59. How would you design a secure communication system?

Explanation: A secure communication system ensures confidentiality, integrity, and authenticity of messages between parties. It uses encryption, digital signatures, and secure protocols.

Security Features: - End-to-end encryption - Digital signatures - Certificate validation - Perfect forward secrecy - Message integrity

public class SecureCommunicationService
{
    private readonly ICertificateService _certificateService;
    private readonly IKeyExchangeService _keyExchangeService;
    private readonly IMessageEncryptionService _encryptionService;
    private readonly IDigitalSignatureService _signatureService;

    public async Task<SecureMessage> SendSecureMessageAsync(
        string recipientId, 
        string message, 
        MessagePriority priority = MessagePriority.Normal)
    {
        // Get recipient's public key
        var recipientCertificate = await _certificateService.GetCertificateAsync(recipientId);
        if (recipientCertificate == null)
        {
            throw new SecurityException("Recipient certificate not found");
        }

        // Generate session key for this message
        var sessionKey = _encryptionService.GenerateSessionKey();

        // Encrypt message with session key
        var encryptedMessage = await _encryptionService.EncryptAsync(message, sessionKey);

        // Encrypt session key with recipient's public key
        var encryptedSessionKey = await _encryptionService.EncryptAsymmetricAsync(
            sessionKey, recipientCertificate.PublicKey);

        // Create digital signature
        var messageHash = await _signatureService.ComputeHashAsync(message);
        var signature = await _signatureService.SignAsync(messageHash);

        // Create secure message
        var secureMessage = new SecureMessage
        {
            Id = Guid.NewGuid(),
            SenderId = GetCurrentUserId(),
            RecipientId = recipientId,
            EncryptedMessage = encryptedMessage,
            EncryptedSessionKey = encryptedSessionKey,
            Signature = signature,
            Timestamp = DateTime.UtcNow,
            Priority = priority,
            Nonce = GenerateNonce()
        };

        return secureMessage;
    }

    public async Task<string> ReceiveSecureMessageAsync(SecureMessage secureMessage)
    {
        // Verify sender's certificate
        var senderCertificate = await _certificateService.GetCertificateAsync(secureMessage.SenderId);
        if (senderCertificate == null)
        {
            throw new SecurityException("Sender certificate not found");
        }

        // Verify digital signature
        var messageHash = await _signatureService.ComputeHashAsync(secureMessage.EncryptedMessage);
        if (!await _signatureService.VerifyAsync(messageHash, secureMessage.Signature, senderCertificate.PublicKey))
        {
            throw new SecurityException("Message signature verification failed");
        }

        // Decrypt session key with our private key
        var sessionKey = await _encryptionService.DecryptAsymmetricAsync(
            secureMessage.EncryptedSessionKey, GetPrivateKey());

        // Decrypt message with session key
        var decryptedMessage = await _encryptionService.DecryptAsync(
            secureMessage.EncryptedMessage, sessionKey);

        // Verify message integrity
        var computedHash = await _signatureService.ComputeHashAsync(decryptedMessage);
        if (computedHash != messageHash)
        {
            throw new SecurityException("Message integrity check failed");
        }

        return decryptedMessage;
    }

    public async Task<SecureChannel> EstablishSecureChannelAsync(string remotePartyId)
    {
        // Perform key exchange (Diffie-Hellman)
        var keyExchange = await _keyExchangeService.InitiateKeyExchangeAsync(remotePartyId);

        // Verify remote party's identity
        var remoteCertificate = await _certificateService.GetCertificateAsync(remotePartyId);
        if (!await _certificateService.VerifyCertificateAsync(remoteCertificate))
        {
            throw new SecurityException("Remote party certificate verification failed");
        }

        // Generate shared secret
        var sharedSecret = await _keyExchangeService.ComputeSharedSecretAsync(keyExchange);

        // Derive encryption keys
        var encryptionKey = await _encryptionService.DeriveKeyAsync(sharedSecret, "encryption");
        var authenticationKey = await _encryptionService.DeriveKeyAsync(sharedSecret, "authentication");

        return new SecureChannel
        {
            RemotePartyId = remotePartyId,
            EncryptionKey = encryptionKey,
            AuthenticationKey = authenticationKey,
            EstablishedAt = DateTime.UtcNow,
            SessionId = Guid.NewGuid()
        };
    }
}

public class MessageEncryptionService : IMessageEncryptionService
{
    public async Task<byte[]> EncryptAsync(string message, byte[] key)
    {
        using var aes = Aes.Create();
        aes.Key = key;
        aes.GenerateIV();
        aes.Mode = CipherMode.CBC;
        aes.Padding = PaddingMode.PKCS7;

        using var encryptor = aes.CreateEncryptor();
        var messageBytes = Encoding.UTF8.GetBytes(message);
        var encryptedBytes = encryptor.TransformFinalBlock(messageBytes, 0, messageBytes.Length);

        // Combine IV and encrypted data
        var result = new byte[aes.IV.Length + encryptedBytes.Length];
        Buffer.BlockCopy(aes.IV, 0, result, 0, aes.IV.Length);
        Buffer.BlockCopy(encryptedBytes, 0, result, aes.IV.Length, encryptedBytes.Length);

        return result;
    }

    public async Task<string> DecryptAsync(byte[] encryptedData, byte[] key)
    {
        using var aes = Aes.Create();
        aes.Key = key;
        aes.Mode = CipherMode.CBC;
        aes.Padding = PaddingMode.PKCS7;

        // Extract IV
        var iv = new byte[16];
        var cipherText = new byte[encryptedData.Length - 16];
        Buffer.BlockCopy(encryptedData, 0, iv, 0, 16);
        Buffer.BlockCopy(encryptedData, 16, cipherText, 0, cipherText.Length);

        aes.IV = iv;

        using var decryptor = aes.CreateDecryptor();
        var decryptedBytes = decryptor.TransformFinalBlock(cipherText, 0, cipherText.Length);

        return Encoding.UTF8.GetString(decryptedBytes);
    }

    public async Task<byte[]> EncryptAsymmetricAsync(byte[] data, RSAParameters publicKey)
    {
        using var rsa = RSA.Create();
        rsa.ImportParameters(publicKey);

        return rsa.Encrypt(data, RSAEncryptionPadding.OaepSHA256);
    }

    public async Task<byte[]> DecryptAsymmetricAsync(byte[] encryptedData, RSAParameters privateKey)
    {
        using var rsa = RSA.Create();
        rsa.ImportParameters(privateKey);

        return rsa.Decrypt(encryptedData, RSAEncryptionPadding.OaepSHA256);
    }
}

60. How do you implement threat detection?

Explanation: Threat detection involves monitoring systems for malicious activities, anomalies, and security breaches. It typically includes log analysis, behavioral analytics, signature-based detection, and machine learning models.

Key Components: - Log aggregation and analysis - Real-time monitoring - Anomaly detection - Signature-based detection - Machine learning models - Alert system

public interface IThreatDetector
{
    Task<ThreatAnalysisResult> AnalyzeAsync(LogEntry logEntry);
    Task<bool> IsAnomalousAsync(UserBehavior behavior);
    Task<List<ThreatAlert>> GetActiveThreatsAsync();
}

public class ThreatDetectionService : IThreatDetector
{
    private readonly ILogger<ThreatDetectionService> _logger;
    private readonly IAnomalyDetector _anomalyDetector;
    private readonly ISignatureDetector _signatureDetector;
    private readonly IAlertService _alertService;
    private readonly IMemoryCache _cache;

    public ThreatDetectionService(
        ILogger<ThreatDetectionService> logger,
        IAnomalyDetector anomalyDetector,
        ISignatureDetector signatureDetector,
        IAlertService alertService,
        IMemoryCache cache)
    {
        _logger = logger;
        _anomalyDetector = anomalyDetector;
        _signatureDetector = signatureDetector;
        _alertService = alertService;
        _cache = cache;
    }

    public async Task<ThreatAnalysisResult> AnalyzeAsync(LogEntry logEntry)
    {
        var result = new ThreatAnalysisResult
        {
            LogEntry = logEntry,
            Timestamp = DateTime.UtcNow,
            Threats = new List<ThreatIndicator>()
        };

        // Signature-based detection
        var signatureThreats = await _signatureDetector.DetectAsync(logEntry);
        result.Threats.AddRange(signatureThreats);

        // Anomaly detection
        var userBehavior = await ExtractUserBehaviorAsync(logEntry);
        var isAnomalous = await _anomalyDetector.DetectAnomalyAsync(userBehavior);

        if (isAnomalous)
        {
            result.Threats.Add(new ThreatIndicator
            {
                Type = ThreatType.Anomaly,
                Severity = ThreatSeverity.Medium,
                Description = "Anomalous user behavior detected"
            });
        }

        // Rate limiting check
        var rateLimitExceeded = await CheckRateLimitAsync(logEntry.UserId);
        if (rateLimitExceeded)
        {
            result.Threats.Add(new ThreatIndicator
            {
                Type = ThreatType.RateLimit,
                Severity = ThreatSeverity.High,
                Description = "Rate limit exceeded"
            });
        }

        // Alert if threats detected
        if (result.Threats.Any())
        {
            await _alertService.SendAlertAsync(result);
        }

        return result;
    }

    private async Task<bool> CheckRateLimitAsync(string userId)
    {
        var cacheKey = $"rate_limit:{userId}";
        var requestCount = await _cache.GetOrSetAsync(cacheKey, 
            () => Task.FromResult(0), 
            TimeSpan.FromMinutes(1));

        if (requestCount > 100) // 100 requests per minute
            return true;

        await _cache.SetAsync(cacheKey, requestCount + 1, TimeSpan.FromMinutes(1));
        return false;
    }
}

public class AnomalyDetector : IAnomalyDetector
{
    public async Task<bool> DetectAnomalyAsync(UserBehavior behavior)
    {
        // Implement machine learning model for anomaly detection
        // This could use isolation forests, autoencoders, or statistical methods

        var features = ExtractFeatures(behavior);
        var anomalyScore = await CalculateAnomalyScoreAsync(features);

        return anomalyScore > 0.8; // Threshold for anomaly
    }
}

61. How would you design a real-time chat system?

Explanation: A real-time chat system requires low-latency message delivery, user presence management, message persistence, and scalability. Key components include WebSocket connections, message queuing, and database storage.

Architecture: - WebSocket connections for real-time communication - Message broker (Redis/RabbitMQ) for message routing - Database for message persistence - User presence tracking - Message delivery guarantees

public class ChatHub : Hub
{
    private readonly IChatService _chatService;
    private readonly IUserPresenceService _presenceService;
    private readonly IMessageBroker _messageBroker;

    public ChatHub(
        IChatService chatService,
        IUserPresenceService presenceService,
        IMessageBroker messageBroker)
    {
        _chatService = chatService;
        _presenceService = presenceService;
        _messageBroker = messageBroker;
    }

    public async Task SendMessage(ChatMessage message)
    {
        // Validate message
        if (string.IsNullOrEmpty(message.Content))
            throw new ArgumentException("Message content cannot be empty");

        // Store message in database
        var savedMessage = await _chatService.SaveMessageAsync(message);

        // Publish to message broker for distribution
        await _messageBroker.PublishAsync("chat.messages", savedMessage);

        // Send to all clients in the room
        await Clients.Group(message.RoomId).SendAsync("ReceiveMessage", savedMessage);
    }

    public async Task JoinRoom(string roomId)
    {
        await Groups.AddToGroupAsync(Context.ConnectionId, roomId);
        await _presenceService.UserJoinedRoomAsync(Context.UserIdentifier, roomId);

        // Notify others in the room
        await Clients.Group(roomId).SendAsync("UserJoined", Context.UserIdentifier);
    }

    public async Task LeaveRoom(string roomId)
    {
        await Groups.RemoveFromGroupAsync(Context.ConnectionId, roomId);
        await _presenceService.UserLeftRoomAsync(Context.UserIdentifier, roomId);

        await Clients.Group(roomId).SendAsync("UserLeft", Context.UserIdentifier);
    }

    public override async Task OnConnectedAsync()
    {
        await _presenceService.UserConnectedAsync(Context.UserIdentifier, Context.ConnectionId);
        await base.OnConnectedAsync();
    }

    public override async Task OnDisconnectedAsync(Exception exception)
    {
        await _presenceService.UserDisconnectedAsync(Context.UserIdentifier);
        await base.OnDisconnectedAsync(exception);
    }
}

public class ChatService : IChatService
{
    private readonly IChatRepository _repository;
    private readonly IMessageBroker _messageBroker;

    public async Task<ChatMessage> SaveMessageAsync(ChatMessage message)
    {
        message.Id = Guid.NewGuid();
        message.Timestamp = DateTime.UtcNow;

        await _repository.SaveMessageAsync(message);
        return message;
    }

    public async Task<List<ChatMessage>> GetRoomHistoryAsync(string roomId, int limit = 50)
    {
        return await _repository.GetMessagesByRoomAsync(roomId, limit);
    }
}

public class MessageBroker : IMessageBroker
{
    private readonly IConnectionMultiplexer _redis;
    private readonly ILogger<MessageBroker> _logger;

    public async Task PublishAsync(string channel, object message)
    {
        var subscriber = _redis.GetSubscriber();
        var serializedMessage = JsonSerializer.Serialize(message);

        await subscriber.PublishAsync(channel, serializedMessage);
    }

    public async Task SubscribeAsync(string channel, Action<string> handler)
    {
        var subscriber = _redis.GetSubscriber();
        await subscriber.SubscribeAsync(channel, (_, value) => handler(value));
    }
}

62. How do you implement WebSocket scaling?

Explanation: WebSocket scaling involves handling multiple server instances while maintaining connection state and message routing. Key strategies include sticky sessions, shared state management, and message broadcasting.

Scaling Strategies: - Sticky sessions with load balancers - Shared state (Redis) - Message broadcasting across instances - Connection pooling - Horizontal scaling

public class WebSocketManager : IWebSocketManager
{
    private readonly IConnectionMultiplexer _redis;
    private readonly ILogger<WebSocketManager> _logger;
    private readonly string _instanceId;
    private readonly ConcurrentDictionary<string, WebSocket> _connections;

    public WebSocketManager(IConnectionMultiplexer redis, ILogger<WebSocketManager> logger)
    {
        _redis = redis;
        _logger = logger;
        _instanceId = Guid.NewGuid().ToString();
        _connections = new ConcurrentDictionary<string, WebSocket>();
    }

    public async Task AddConnectionAsync(string userId, WebSocket webSocket)
    {
        _connections.TryAdd(userId, webSocket);

        // Register connection in Redis for cross-instance communication
        await _redis.GetDatabase().HashSetAsync(
            "websocket_connections", 
            userId, 
            _instanceId);
    }

    public async Task RemoveConnectionAsync(string userId)
    {
        _connections.TryRemove(userId, out _);

        await _redis.GetDatabase().HashDeleteAsync("websocket_connections", userId);
    }

    public async Task SendToUserAsync(string userId, string message)
    {
        // Check if user is connected to this instance
        var instanceId = await _redis.GetDatabase().HashGetAsync("websocket_connections", userId);

        if (instanceId == _instanceId && _connections.TryGetValue(userId, out var webSocket))
        {
            var buffer = Encoding.UTF8.GetBytes(message);
            await webSocket.SendAsync(new ArraySegment<byte>(buffer), WebSocketMessageType.Text, true, CancellationToken.None);
        }
        else
        {
            // User is connected to another instance, broadcast message
            await BroadcastToInstanceAsync(instanceId, userId, message);
        }
    }

    private async Task BroadcastToInstanceAsync(string instanceId, string userId, string message)
    {
        var subscriber = _redis.GetSubscriber();
        var broadcastMessage = new
        {
            InstanceId = instanceId,
            UserId = userId,
            Message = message
        };

        await subscriber.PublishAsync("websocket_broadcast", JsonSerializer.Serialize(broadcastMessage));
    }

    public async Task StartListeningAsync()
    {
        var subscriber = _redis.GetSubscriber();
        await subscriber.SubscribeAsync("websocket_broadcast", async (_, value) =>
        {
            var broadcast = JsonSerializer.Deserialize<BroadcastMessage>(value);

            if (broadcast.InstanceId == _instanceId && _connections.TryGetValue(broadcast.UserId, out var webSocket))
            {
                var buffer = Encoding.UTF8.GetBytes(broadcast.Message);
                await webSocket.SendAsync(new ArraySegment<byte>(buffer), WebSocketMessageType.Text, true, CancellationToken.None);
            }
        });
    }
}

public class LoadBalancerConfig
{
    public static void ConfigureStickySessions(IServiceCollection services)
    {
        services.Configure<IISServerOptions>(options =>
        {
            options.MaxRequestBodySize = int.MaxValue;
        });

        services.Configure<ForwardedHeadersOptions>(options =>
        {
            options.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
        });
    }
}

63. How would you design a real-time notification system?

Explanation: A real-time notification system delivers instant notifications to users across multiple channels (push, email, SMS, in-app). It requires message queuing, channel routing, and delivery tracking.

Components: - Notification service - Channel providers (push, email, SMS) - Message queuing - Delivery tracking - User preferences

public interface INotificationService
{
    Task<NotificationResult> SendAsync(NotificationRequest request);
    Task<List<Notification>> GetUserNotificationsAsync(string userId);
    Task MarkAsReadAsync(string userId, string notificationId);
}

public class NotificationService : INotificationService
{
    private readonly IMessageBroker _messageBroker;
    private readonly INotificationRepository _repository;
    private readonly IUserPreferencesService _preferencesService;
    private readonly Dictionary<NotificationChannel, INotificationProvider> _providers;

    public NotificationService(
        IMessageBroker messageBroker,
        INotificationRepository repository,
        IUserPreferencesService preferencesService,
        IEnumerable<INotificationProvider> providers)
    {
        _messageBroker = messageBroker;
        _repository = repository;
        _preferencesService = preferencesService;
        _providers = providers.ToDictionary(p => p.Channel);
    }

    public async Task<NotificationResult> SendAsync(NotificationRequest request)
    {
        var notification = new Notification
        {
            Id = Guid.NewGuid(),
            UserId = request.UserId,
            Title = request.Title,
            Message = request.Message,
            Type = request.Type,
            CreatedAt = DateTime.UtcNow,
            Status = NotificationStatus.Pending
        };

        // Save to database
        await _repository.SaveAsync(notification);

        // Get user preferences
        var preferences = await _preferencesService.GetUserPreferencesAsync(request.UserId);

        // Queue notifications for each enabled channel
        foreach (var channel in preferences.EnabledChannels)
        {
            if (_providers.TryGetValue(channel, out var provider))
            {
                var channelNotification = new ChannelNotification
                {
                    NotificationId = notification.Id,
                    Channel = channel,
                    UserId = request.UserId,
                    Content = request.Message,
                    Status = NotificationStatus.Pending
                };

                await _messageBroker.PublishAsync($"notifications.{channel}", channelNotification);
            }
        }

        return new NotificationResult
        {
            NotificationId = notification.Id,
            Status = NotificationStatus.Queued
        };
    }

    public async Task<List<Notification>> GetUserNotificationsAsync(string userId)
    {
        return await _repository.GetUserNotificationsAsync(userId, 50);
    }

    public async Task MarkAsReadAsync(string userId, string notificationId)
    {
        await _repository.MarkAsReadAsync(userId, notificationId);
    }
}

public class PushNotificationProvider : INotificationProvider
{
    public NotificationChannel Channel => NotificationChannel.Push;

    public async Task<DeliveryResult> SendAsync(ChannelNotification notification)
    {
        // Implement push notification logic (Firebase, Apple Push, etc.)
        try
        {
            // Send push notification
            var result = await SendPushNotificationAsync(notification);

            return new DeliveryResult
            {
                Success = result.Success,
                Message = result.Message,
                DeliveredAt = DateTime.UtcNow
            };
        }
        catch (Exception ex)
        {
            return new DeliveryResult
            {
                Success = false,
                Message = ex.Message,
                DeliveredAt = DateTime.UtcNow
            };
        }
    }
}

public class NotificationProcessor : BackgroundService
{
    private readonly IMessageBroker _messageBroker;
    private readonly Dictionary<NotificationChannel, INotificationProvider> _providers;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        foreach (var provider in _providers)
        {
            await _messageBroker.SubscribeAsync($"notifications.{provider.Key}", async (message) =>
            {
                var notification = JsonSerializer.Deserialize<ChannelNotification>(message);
                await provider.Value.SendAsync(notification);
            });
        }

        await Task.Delay(Timeout.Infinite, stoppingToken);
    }
}

64. How do you implement real-time analytics?

Explanation: Real-time analytics processes data streams to provide immediate insights. It involves data ingestion, stream processing, aggregation, and visualization.

Components: - Data ingestion pipeline - Stream processing engine - Real-time aggregations - Dashboard updates - Alert system

public interface IRealTimeAnalyticsService
{
    Task TrackEventAsync(AnalyticsEvent analyticsEvent);
    Task<AnalyticsResult> GetRealTimeMetricsAsync(string metricName);
    Task<List<AnalyticsEvent>> GetRecentEventsAsync(int count);
}

public class RealTimeAnalyticsService : IRealTimeAnalyticsService
{
    private readonly IMessageBroker _messageBroker;
    private readonly IAnalyticsRepository _repository;
    private readonly IMemoryCache _cache;
    private readonly ILogger<RealTimeAnalyticsService> _logger;

    public async Task TrackEventAsync(AnalyticsEvent analyticsEvent)
    {
        analyticsEvent.Timestamp = DateTime.UtcNow;
        analyticsEvent.Id = Guid.NewGuid();

        // Store in database
        await _repository.SaveEventAsync(analyticsEvent);

        // Publish to stream for real-time processing
        await _messageBroker.PublishAsync("analytics.events", analyticsEvent);

        // Update in-memory cache for immediate access
        await UpdateCacheMetricsAsync(analyticsEvent);
    }

    public async Task<AnalyticsResult> GetRealTimeMetricsAsync(string metricName)
    {
        var cacheKey = $"analytics:{metricName}";

        if (_cache.TryGetValue(cacheKey, out AnalyticsResult result))
        {
            return result;
        }

        // Calculate from database if not in cache
        result = await CalculateMetricAsync(metricName);
        _cache.Set(cacheKey, result, TimeSpan.FromMinutes(1));

        return result;
    }

    private async Task UpdateCacheMetricsAsync(AnalyticsEvent analyticsEvent)
    {
        // Update various metrics based on event type
        switch (analyticsEvent.EventType)
        {
            case "page_view":
                await UpdatePageViewMetricsAsync(analyticsEvent);
                break;
            case "user_action":
                await UpdateUserActionMetricsAsync(analyticsEvent);
                break;
            case "error":
                await UpdateErrorMetricsAsync(analyticsEvent);
                break;
        }
    }

    private async Task UpdatePageViewMetricsAsync(AnalyticsEvent analyticsEvent)
    {
        var cacheKey = "analytics:page_views";
        var currentCount = await _cache.GetOrSetAsync(cacheKey, () => Task.FromResult(0L));

        _cache.Set(cacheKey, currentCount + 1, TimeSpan.FromMinutes(5));

        // Update page-specific metrics
        var pageKey = $"analytics:page:{analyticsEvent.Properties["page"]}";
        var pageCount = await _cache.GetOrSetAsync(pageKey, () => Task.FromResult(0L));
        _cache.Set(pageKey, pageCount + 1, TimeSpan.FromMinutes(5));
    }
}

public class AnalyticsStreamProcessor : BackgroundService
{
    private readonly IMessageBroker _messageBroker;
    private readonly IAnalyticsRepository _repository;
    private readonly IMemoryCache _cache;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await _messageBroker.SubscribeAsync("analytics.events", async (message) =>
        {
            var analyticsEvent = JsonSerializer.Deserialize<AnalyticsEvent>(message);
            await ProcessEventAsync(analyticsEvent);
        });

        await Task.Delay(Timeout.Infinite, stoppingToken);
    }

    private async Task ProcessEventAsync(AnalyticsEvent analyticsEvent)
    {
        // Perform real-time aggregations
        await AggregateByTimeWindowAsync(analyticsEvent);
        await AggregateByUserAsync(analyticsEvent);
        await AggregateByEventTypeAsync(analyticsEvent);
    }

    private async Task AggregateByTimeWindowAsync(AnalyticsEvent analyticsEvent)
    {
        var timeWindow = GetTimeWindow(analyticsEvent.Timestamp);
        var cacheKey = $"aggregation:time:{timeWindow}";

        var aggregation = await _cache.GetOrSetAsync(cacheKey, () => Task.FromResult(new TimeWindowAggregation()));

        aggregation.EventCount++;
        aggregation.LastEventTime = analyticsEvent.Timestamp;

        _cache.Set(cacheKey, aggregation, TimeSpan.FromMinutes(10));
    }
}

public class AnalyticsDashboardHub : Hub
{
    private readonly IRealTimeAnalyticsService _analyticsService;

    public async Task SubscribeToMetrics(string metricName)
    {
        await Groups.AddToGroupAsync(Context.ConnectionId, $"metrics:{metricName}");
    }

    public async Task UnsubscribeFromMetrics(string metricName)
    {
        await Groups.RemoveFromGroupAsync(Context.ConnectionId, $"metrics:{metricName}");
    }
}

65. How would you design a live streaming platform?

Explanation: A live streaming platform requires video processing, content delivery networks (CDN), real-time transcoding, and viewer management. It involves handling high-bandwidth video streams and scaling to thousands of concurrent viewers.

Components: - Video ingestion - Real-time transcoding - CDN distribution - Viewer management - Chat system - Analytics

public interface ILiveStreamingService
{
    Task<StreamSession> StartStreamAsync(StreamRequest request);
    Task StopStreamAsync(string streamId);
    Task<StreamInfo> GetStreamInfoAsync(string streamId);
    Task<List<Viewer>> GetViewersAsync(string streamId);
}

public class LiveStreamingService : ILiveStreamingService
{
    private readonly IStreamRepository _repository;
    private readonly IVideoProcessor _videoProcessor;
    private readonly ICDNService _cdnService;
    private readonly IMessageBroker _messageBroker;
    private readonly ILogger<LiveStreamingService> _logger;

    public async Task<StreamSession> StartStreamAsync(StreamRequest request)
    {
        var streamSession = new StreamSession
        {
            Id = Guid.NewGuid().ToString(),
            StreamerId = request.StreamerId,
            Title = request.Title,
            Description = request.Description,
            Status = StreamStatus.Starting,
            StartedAt = DateTime.UtcNow,
            IngestUrl = await _videoProcessor.CreateIngestUrlAsync(request.StreamerId),
            PlaybackUrls = new Dictionary<string, string>()
        };

        // Create video processing pipeline
        var processingConfig = new VideoProcessingConfig
        {
            StreamId = streamSession.Id,
            QualityLevels = new[] { "1080p", "720p", "480p", "360p" },
            Codec = "H.264",
            FrameRate = 30
        };

        await _videoProcessor.StartProcessingAsync(processingConfig);

        // Generate CDN URLs for different qualities
        foreach (var quality in processingConfig.QualityLevels)
        {
            var playbackUrl = await _cdnService.CreatePlaybackUrlAsync(streamSession.Id, quality);
            streamSession.PlaybackUrls[quality] = playbackUrl;
        }

        // Save to database
        await _repository.SaveStreamSessionAsync(streamSession);

        // Notify viewers that stream is starting
        await _messageBroker.PublishAsync("streams.started", streamSession);

        return streamSession;
    }

    public async Task StopStreamAsync(string streamId)
    {
        var streamSession = await _repository.GetStreamSessionAsync(streamId);
        if (streamSession == null)
            throw new ArgumentException("Stream not found");

        streamSession.Status = StreamStatus.Ended;
        streamSession.EndedAt = DateTime.UtcNow;
        streamSession.Duration = streamSession.EndedAt - streamSession.StartedAt;

        // Stop video processing
        await _videoProcessor.StopProcessingAsync(streamId);

        // Save to database
        await _repository.UpdateStreamSessionAsync(streamSession);

        // Notify viewers that stream has ended
        await _messageBroker.PublishAsync("streams.ended", streamSession);
    }
}

public class VideoProcessor : IVideoProcessor
{
    private readonly ILogger<VideoProcessor> _logger;
    private readonly string _processingEndpoint;

    public async Task<string> CreateIngestUrlAsync(string streamerId)
    {
        // Create RTMP ingest URL for the streamer
        var ingestUrl = $"{_processingEndpoint}/live/{streamerId}";

        _logger.LogInformation("Created ingest URL: {IngestUrl} for streamer: {StreamerId}", 
            ingestUrl, streamerId);

        return ingestUrl;
    }

    public async Task StartProcessingAsync(VideoProcessingConfig config)
    {
        // Start real-time transcoding for multiple qualities
        foreach (var quality in config.QualityLevels)
        {
            await StartTranscodingAsync(config.StreamId, quality, config);
        }
    }

    private async Task StartTranscodingAsync(string streamId, string quality, VideoProcessingConfig config)
    {
        // Implement transcoding logic (could use FFmpeg, AWS MediaLive, etc.)
        var transcodingJob = new TranscodingJob
        {
            StreamId = streamId,
            Quality = quality,
            InputUrl = $"{_processingEndpoint}/live/{streamId}",
            OutputUrl = $"{_processingEndpoint}/output/{streamId}/{quality}",
            Codec = config.Codec,
            FrameRate = config.FrameRate
        };

        // Start transcoding process
        await StartTranscodingProcessAsync(transcodingJob);
    }
}

public class StreamViewerManager
{
    private readonly IMessageBroker _messageBroker;
    private readonly IStreamRepository _repository;
    private readonly ConcurrentDictionary<string, HashSet<string>> _viewers;

    public async Task AddViewerAsync(string streamId, string viewerId)
    {
        _viewers.AddOrUpdate(streamId, 
            new HashSet<string> { viewerId },
            (key, existing) => { existing.Add(viewerId); return existing; });

        // Update database
        await _repository.AddViewerAsync(streamId, viewerId);

        // Notify streamer
        await _messageBroker.PublishAsync("streams.viewer_joined", new
        {
            StreamId = streamId,
            ViewerId = viewerId,
            ViewerCount = _viewers[streamId].Count
        });
    }

    public async Task RemoveViewerAsync(string streamId, string viewerId)
    {
        if (_viewers.TryGetValue(streamId, out var streamViewers))
        {
            streamViewers.Remove(viewerId);
        }

        await _repository.RemoveViewerAsync(streamId, viewerId);

        await _messageBroker.PublishAsync("streams.viewer_left", new
        {
            StreamId = streamId,
            ViewerId = viewerId,
            ViewerCount = _viewers[streamId]?.Count ?? 0
        });
    }
}

66. How do you implement real-time collaboration?

Explanation: Real-time collaboration enables multiple users to work on the same document simultaneously. It requires operational transformation, conflict resolution, and state synchronization.

Components: - Operational transformation - Conflict resolution - State synchronization - User presence - Change tracking

public interface ICollaborationService
{
    Task<OperationResult> ApplyOperationAsync(string documentId, DocumentOperation operation);
    Task<List<DocumentOperation>> GetPendingOperationsAsync(string documentId);
    Task<DocumentState> GetDocumentStateAsync(string documentId);
    Task JoinDocumentAsync(string documentId, string userId);
    Task LeaveDocumentAsync(string documentId, string userId);
}

public class CollaborationService : ICollaborationService
{
    private readonly IOperationTransformer _transformer;
    private readonly IDocumentRepository _repository;
    private readonly IMessageBroker _messageBroker;
    private readonly IUserPresenceService _presenceService;
    private readonly ConcurrentDictionary<string, List<DocumentOperation>> _pendingOperations;

    public async Task<OperationResult> ApplyOperationAsync(string documentId, DocumentOperation operation)
    {
        // Get current document state
        var documentState = await GetDocumentStateAsync(documentId);

        // Transform operation against pending operations
        var transformedOperation = await TransformOperationAsync(documentId, operation);

        // Apply operation to document state
        var newState = await ApplyOperationToStateAsync(documentState, transformedOperation);

        // Save new state
        await _repository.SaveDocumentStateAsync(documentId, newState);

        // Add to pending operations for other clients
        AddPendingOperation(documentId, transformedOperation);

        // Broadcast to other users
        await _messageBroker.PublishAsync($"document.{documentId}.operations", transformedOperation);

        return new OperationResult
        {
            Success = true,
            OperationId = transformedOperation.Id,
            NewState = newState
        };
    }

    private async Task<DocumentOperation> TransformOperationAsync(string documentId, DocumentOperation operation)
    {
        var pendingOps = GetPendingOperations(documentId);
        var transformedOp = operation;

        foreach (var pendingOp in pendingOps)
        {
            transformedOp = await _transformer.TransformAsync(transformedOp, pendingOp);
        }

        return transformedOp;
    }

    private async Task<DocumentState> ApplyOperationToStateAsync(DocumentState state, DocumentOperation operation)
    {
        switch (operation.Type)
        {
            case OperationType.Insert:
                return await ApplyInsertOperationAsync(state, operation);
            case OperationType.Delete:
                return await ApplyDeleteOperationAsync(state, operation);
            case OperationType.Update:
                return await ApplyUpdateOperationAsync(state, operation);
            default:
                throw new ArgumentException($"Unknown operation type: {operation.Type}");
        }
    }

    private async Task<DocumentState> ApplyInsertOperationAsync(DocumentState state, DocumentOperation operation)
    {
        var newContent = state.Content.Insert(operation.Position, operation.Content);
        return new DocumentState
        {
            Content = newContent,
            Version = state.Version + 1,
            LastModified = DateTime.UtcNow
        };
    }
}

public class OperationTransformer : IOperationTransformer
{
    public async Task<DocumentOperation> TransformAsync(DocumentOperation op1, DocumentOperation op2)
    {
        // Implement operational transformation logic
        if (op1.Type == OperationType.Insert && op2.Type == OperationType.Insert)
        {
            return await TransformInsertInsertAsync(op1, op2);
        }
        else if (op1.Type == OperationType.Insert && op2.Type == OperationType.Delete)
        {
            return await TransformInsertDeleteAsync(op1, op2);
        }
        // Add more transformation rules...

        return op1;
    }

    private async Task<DocumentOperation> TransformInsertInsertAsync(DocumentOperation op1, DocumentOperation op2)
    {
        if (op1.Position <= op2.Position)
        {
            return op1; // No transformation needed
        }
        else
        {
            return new DocumentOperation
            {
                Id = op1.Id,
                Type = op1.Type,
                Position = op1.Position + op2.Content.Length,
                Content = op1.Content,
                UserId = op1.UserId,
                Timestamp = op1.Timestamp
            };
        }
    }
}

public class CollaborationHub : Hub
{
    private readonly ICollaborationService _collaborationService;
    private readonly IUserPresenceService _presenceService;

    public async Task JoinDocument(string documentId)
    {
        var userId = Context.UserIdentifier;

        await Groups.AddToGroupAsync(Context.ConnectionId, $"document:{documentId}");
        await _presenceService.UserJoinedDocumentAsync(userId, documentId);

        // Notify other users
        await Clients.Group($"document:{documentId}").SendAsync("UserJoined", userId);
    }

    public async Task LeaveDocument(string documentId)
    {
        var userId = Context.UserIdentifier;

        await Groups.RemoveFromGroupAsync(Context.ConnectionId, $"document:{documentId}");
        await _presenceService.UserLeftDocumentAsync(userId, documentId);

        await Clients.Group($"document:{documentId}").SendAsync("UserLeft", userId);
    }

    public async Task ApplyOperation(string documentId, DocumentOperation operation)
    {
        var result = await _collaborationService.ApplyOperationAsync(documentId, operation);

        // Send result back to the client
        await Clients.Caller.SendAsync("OperationApplied", result);
    }

    public override async Task OnConnectedAsync()
    {
        await _presenceService.UserConnectedAsync(Context.UserIdentifier, Context.ConnectionId);
        await base.OnConnectedAsync();
    }

    public override async Task OnDisconnectedAsync(Exception exception)
    {
        await _presenceService.UserDisconnectedAsync(Context.UserIdentifier);
        await base.OnDisconnectedAsync(exception);
    }
}

67. How would you design a real-time gaming system?

Explanation: A real-time gaming system requires low-latency communication, game state synchronization, player management, and scalable infrastructure to handle thousands of concurrent players.

Components: - Game server architecture - State synchronization - Player management - Matchmaking - Anti-cheat systems

public interface IGameServer
{
    Task<GameSession> CreateGameAsync(GameRequest request);
    Task<bool> JoinGameAsync(string gameId, string playerId);
    Task<GameState> GetGameStateAsync(string gameId);
    Task<bool> SubmitPlayerActionAsync(string gameId, string playerId, PlayerAction action);
}

public class GameServer : IGameServer
{
    private readonly IGameRepository _repository;
    private readonly IPlayerManager _playerManager;
    private readonly IGameStateManager _stateManager;
    private readonly IMessageBroker _messageBroker;
    private readonly ILogger<GameServer> _logger;
    private readonly ConcurrentDictionary<string, GameSession> _activeGames;

    public async Task<GameSession> CreateGameAsync(GameRequest request)
    {
        var gameSession = new GameSession
        {
            Id = Guid.NewGuid().ToString(),
            GameType = request.GameType,
            MaxPlayers = request.MaxPlayers,
            Status = GameStatus.Waiting,
            CreatedAt = DateTime.UtcNow,
            Players = new List<Player>(),
            GameState = new GameState()
        };

        // Add creator as first player
        var creator = new Player
        {
            Id = request.CreatorId,
            Name = request.CreatorName,
            JoinedAt = DateTime.UtcNow
        };

        gameSession.Players.Add(creator);
        gameSession.GameState.Players.Add(creator);

        // Save to database
        await _repository.SaveGameSessionAsync(gameSession);
        _activeGames.TryAdd(gameSession.Id, gameSession);

        // Start game loop
        _ = Task.Run(() => RunGameLoopAsync(gameSession.Id));

        return gameSession;
    }

    public async Task<bool> JoinGameAsync(string gameId, string playerId)
    {
        if (!_activeGames.TryGetValue(gameId, out var gameSession))
            return false;

        if (gameSession.Players.Count >= gameSession.MaxPlayers)
            return false;

        var player = new Player
        {
            Id = playerId,
            Name = await _playerManager.GetPlayerNameAsync(playerId),
            JoinedAt = DateTime.UtcNow
        };

        gameSession.Players.Add(player);
        gameSession.GameState.Players.Add(player);

        // Check if game should start
        if (gameSession.Players.Count >= gameSession.MaxPlayers)
        {
            gameSession.Status = GameStatus.Starting;
            await StartGameAsync(gameSession);
        }

        // Notify all players
        await _messageBroker.PublishAsync($"game.{gameId}.player_joined", player);

        return true;
    }

    public async Task<bool> SubmitPlayerActionAsync(string gameId, string playerId, PlayerAction action)
    {
        if (!_activeGames.TryGetValue(gameId, out var gameSession))
            return false;

        if (gameSession.Status != GameStatus.Active)
            return false;

        // Validate action
        if (!await ValidateActionAsync(gameSession, playerId, action))
            return false;

        // Add action to queue
        gameSession.ActionQueue.Enqueue(new QueuedAction
        {
            PlayerId = playerId,
            Action = action,
            Timestamp = DateTime.UtcNow
        });

        return true;
    }

    private async Task RunGameLoopAsync(string gameId)
    {
        const int tickRate = 60; // 60 FPS
        const int tickInterval = 1000 / tickRate;

        while (_activeGames.TryGetValue(gameId, out var gameSession))
        {
            var startTime = DateTime.UtcNow;

            // Process queued actions
            await ProcessActionsAsync(gameSession);

            // Update game state
            await UpdateGameStateAsync(gameSession);

            // Broadcast state to all players
            await BroadcastGameStateAsync(gameSession);

            // Check for game end conditions
            if (await CheckGameEndConditionsAsync(gameSession))
            {
                await EndGameAsync(gameSession);
                break;
            }

            // Maintain tick rate
            var elapsed = DateTime.UtcNow - startTime;
            var sleepTime = Math.Max(0, tickInterval - (int)elapsed.TotalMilliseconds);

            if (sleepTime > 0)
                await Task.Delay(sleepTime);
        }
    }

    private async Task ProcessActionsAsync(GameSession gameSession)
    {
        while (gameSession.ActionQueue.TryDequeue(out var queuedAction))
        {
            await ApplyActionToGameStateAsync(gameSession.GameState, queuedAction);
        }
    }

    private async Task UpdateGameStateAsync(GameSession gameSession)
    {
        // Update game logic (physics, AI, etc.)
        await _stateManager.UpdateGameStateAsync(gameSession.GameState);

        gameSession.GameState.Tick++;
        gameSession.GameState.LastUpdated = DateTime.UtcNow;
    }

    private async Task BroadcastGameStateAsync(GameSession gameSession)
    {
        var stateUpdate = new GameStateUpdate
        {
            GameId = gameSession.Id,
            State = gameSession.GameState,
            Tick = gameSession.GameState.Tick
        };

        await _messageBroker.PublishAsync($"game.{gameSession.Id}.state_update", stateUpdate);
    }
}

public class GameHub : Hub
{
    private readonly IGameServer _gameServer;
    private readonly IPlayerManager _playerManager;

    public async Task JoinGame(string gameId)
    {
        var playerId = Context.UserIdentifier;
        var success = await _gameServer.JoinGameAsync(gameId, playerId);

        if (success)
        {
            await Groups.AddToGroupAsync(Context.ConnectionId, $"game:{gameId}");
            await Clients.Caller.SendAsync("GameJoined", gameId);
        }
        else
        {
            await Clients.Caller.SendAsync("GameJoinFailed", "Unable to join game");
        }
    }

    public async Task SubmitAction(string gameId, PlayerAction action)
    {
        var playerId = Context.UserIdentifier;
        var success = await _gameServer.SubmitPlayerActionAsync(gameId, playerId, action);

        if (!success)
        {
            await Clients.Caller.SendAsync("ActionRejected", "Invalid action");
        }
    }

    public async Task GetGameState(string gameId)
    {
        var state = await _gameServer.GetGameStateAsync(gameId);
        await Clients.Caller.SendAsync("GameState", state);
    }
}

68. How do you implement real-time monitoring?

Explanation: Real-time monitoring tracks system health, performance metrics, and alerts administrators to issues immediately. It involves data collection, processing, visualization, and alerting.

Components: - Metrics collection - Data processing - Alert system - Dashboard - Notification system

public interface IMonitoringService
{
    Task TrackMetricAsync(MetricData metric);
    Task<List<Alert>> GetActiveAlertsAsync();
    Task<SystemHealth> GetSystemHealthAsync();
    Task<List<MetricData>> GetMetricsAsync(string metricName, TimeSpan duration);
}

public class MonitoringService : IMonitoringService
{
    private readonly IMetricCollector _metricCollector;
    private readonly IAlertService _alertService;
    private readonly IHealthChecker _healthChecker;
    private readonly ITimeSeriesDatabase _timeSeriesDb;
    private readonly ILogger<MonitoringService> _logger;

    public async Task TrackMetricAsync(MetricData metric)
    {
        metric.Timestamp = DateTime.UtcNow;

        // Store in time-series database
        await _timeSeriesDb.StoreMetricAsync(metric);

        // Check for threshold violations
        await CheckThresholdsAsync(metric);

        // Update real-time dashboards
        await UpdateDashboardsAsync(metric);
    }

    public async Task<SystemHealth> GetSystemHealthAsync()
    {
        var healthChecks = await _healthChecker.RunHealthChecksAsync();

        var overallHealth = healthChecks.All(h => h.Status == HealthStatus.Healthy) 
            ? HealthStatus.Healthy 
            : HealthStatus.Unhealthy;

        return new SystemHealth
        {
            Status = overallHealth,
            Checks = healthChecks,
            LastChecked = DateTime.UtcNow
        };
    }

    private async Task CheckThresholdsAsync(MetricData metric)
    {
        var thresholds = await GetThresholdsForMetricAsync(metric.Name);

        foreach (var threshold in thresholds)
        {
            if (IsThresholdViolated(metric, threshold))
            {
                await CreateAlertAsync(metric, threshold);
            }
        }
    }

    private bool IsThresholdViolated(MetricData metric, Threshold threshold)
    {
        switch (threshold.Operator)
        {
            case ThresholdOperator.GreaterThan:
                return metric.Value > threshold.Value;
            case ThresholdOperator.LessThan:
                return metric.Value < threshold.Value;
            case ThresholdOperator.Equals:
                return Math.Abs(metric.Value - threshold.Value) < 0.001;
            default:
                return false;
        }
    }

    private async Task CreateAlertAsync(MetricData metric, Threshold threshold)
    {
        var alert = new Alert
        {
            Id = Guid.NewGuid(),
            MetricName = metric.Name,
            Threshold = threshold,
            CurrentValue = metric.Value,
            Severity = threshold.Severity,
            CreatedAt = DateTime.UtcNow,
            Status = AlertStatus.Active
        };

        await _alertService.CreateAlertAsync(alert);
    }
}

public class MetricCollector : IMetricCollector
{
    private readonly ILogger<MetricCollector> _logger;
    private readonly IMonitoringService _monitoringService;

    public async Task CollectSystemMetricsAsync()
    {
        // Collect CPU usage
        var cpuUsage = await GetCpuUsageAsync();
        await _monitoringService.TrackMetricAsync(new MetricData
        {
            Name = "cpu_usage",
            Value = cpuUsage,
            Unit = "percentage",
            Tags = new Dictionary<string, string> { { "host", Environment.MachineName } }
        });

        // Collect memory usage
        var memoryUsage = await GetMemoryUsageAsync();
        await _monitoringService.TrackMetricAsync(new MetricData
        {
            Name = "memory_usage",
            Value = memoryUsage,
            Unit = "percentage",
            Tags = new Dictionary<string, string> { { "host", Environment.MachineName } }
        });

        // Collect disk usage
        var diskUsage = await GetDiskUsageAsync();
        await _monitoringService.TrackMetricAsync(new MetricData
        {
            Name = "disk_usage",
            Value = diskUsage,
            Unit = "percentage",
            Tags = new Dictionary<string, string> { { "host", Environment.MachineName } }
        });
    }

    private async Task<double> GetCpuUsageAsync()
    {
        // Implement CPU usage collection
        var cpuCounter = new PerformanceCounter("Processor", "% Processor Time", "_Total");
        return cpuCounter.NextValue();
    }

    private async Task<double> GetMemoryUsageAsync()
    {
        // Implement memory usage collection
        var memoryCounter = new PerformanceCounter("Memory", "% Committed Bytes In Use");
        return memoryCounter.NextValue();
    }
}

public class HealthChecker : IHealthChecker
{
    private readonly ILogger<HealthChecker> _logger;
    private readonly IConfiguration _configuration;

    public async Task<List<HealthCheck>> RunHealthChecksAsync()
    {
        var healthChecks = new List<HealthCheck>();

        // Database health check
        healthChecks.Add(await CheckDatabaseHealthAsync());

        // External service health check
        healthChecks.Add(await CheckExternalServiceHealthAsync());

        // Disk space health check
        healthChecks.Add(await CheckDiskSpaceHealthAsync());

        return healthChecks;
    }

    private async Task<HealthCheck> CheckDatabaseHealthAsync()
    {
        try
        {
            // Implement database connectivity check
            var isHealthy = await TestDatabaseConnectionAsync();

            return new HealthCheck
            {
                Name = "Database",
                Status = isHealthy ? HealthStatus.Healthy : HealthStatus.Unhealthy,
                Message = isHealthy ? "Database is accessible" : "Database connection failed",
                LastChecked = DateTime.UtcNow
            };
        }
        catch (Exception ex)
        {
            return new HealthCheck
            {
                Name = "Database",
                Status = HealthStatus.Unhealthy,
                Message = $"Database check failed: {ex.Message}",
                LastChecked = DateTime.UtcNow
            };
        }
    }
}

public class MonitoringHub : Hub
{
    private readonly IMonitoringService _monitoringService;

    public async Task SubscribeToMetrics(string metricName)
    {
        await Groups.AddToGroupAsync(Context.ConnectionId, $"metrics:{metricName}");
    }

    public async Task GetSystemHealth()
    {
        var health = await _monitoringService.GetSystemHealthAsync();
        await Clients.Caller.SendAsync("SystemHealth", health);
    }

    public async Task GetActiveAlerts()
    {
        var alerts = await _monitoringService.GetActiveAlertsAsync();
        await Clients.Caller.SendAsync("ActiveAlerts", alerts);
    }
}

69. How would you design a real-time recommendation system?

Explanation: A real-time recommendation system provides personalized suggestions based on user behavior, preferences, and contextual data. The architecture typically includes:

1. Data Collection Layer: Captures user interactions, preferences, and contextual data
2. Real-time Processing: Processes events using stream processing technologies
3. Recommendation Engine: Generates recommendations using ML models
4. Caching Layer: Stores pre-computed recommendations for fast retrieval
5. API Gateway: Serves recommendations to clients

Key Components: - Event streaming (Kafka, Event Hubs) - Real-time processing (Spark Streaming, Flink) - ML model serving (TensorFlow Serving, ONNX Runtime) - Caching (Redis, Azure Cache) - Database (Cosmos DB, PostgreSQL)

// Real-time Recommendation Service
public class RealTimeRecommendationService
{
    private readonly IEventProcessor _eventProcessor;
    private readonly IRecommendationEngine _recommendationEngine;
    private readonly ICacheService _cacheService;
    private readonly IUserProfileService _userProfileService;

    public RealTimeRecommendationService(
        IEventProcessor eventProcessor,
        IRecommendationEngine recommendationEngine,
        ICacheService cacheService,
        IUserProfileService userProfileService)
    {
        _eventProcessor = eventProcessor;
        _recommendationEngine = recommendationEngine;
        _cacheService = cacheService;
        _userProfileService = userProfileService;
    }

    public async Task<List<Recommendation>> GetRecommendationsAsync(string userId, string context)
    {
        // Check cache first
        var cacheKey = $"recommendations:{userId}:{context}";
        var cachedRecommendations = await _cacheService.GetAsync<List<Recommendation>>(cacheKey);

        if (cachedRecommendations != null)
            return cachedRecommendations;

        // Get user profile and recent interactions
        var userProfile = await _userProfileService.GetUserProfileAsync(userId);
        var recentInteractions = await _eventProcessor.GetRecentInteractionsAsync(userId);

        // Generate real-time recommendations
        var recommendations = await _recommendationEngine.GenerateRecommendationsAsync(
            userProfile, recentInteractions, context);

        // Cache recommendations for 5 minutes
        await _cacheService.SetAsync(cacheKey, recommendations, TimeSpan.FromMinutes(5));

        return recommendations;
    }

    public async Task ProcessUserInteractionAsync(UserInteraction interaction)
    {
        // Process interaction in real-time
        await _eventProcessor.ProcessEventAsync(interaction);

        // Update user profile
        await _userProfileService.UpdateUserProfileAsync(interaction);

        // Invalidate cached recommendations
        await _cacheService.RemoveAsync($"recommendations:{interaction.UserId}:*");
    }
}

// Event Processor for real-time data
public class EventProcessor : IEventProcessor
{
    private readonly IEventHubClient _eventHubClient;
    private readonly IStreamProcessor _streamProcessor;

    public async Task ProcessEventAsync(UserInteraction interaction)
    {
        // Send to event hub for real-time processing
        await _eventHubClient.SendAsync(interaction);

        // Process in stream
        await _streamProcessor.ProcessAsync(interaction);
    }

    public async Task<List<UserInteraction>> GetRecentInteractionsAsync(string userId)
    {
        // Get recent interactions from stream storage
        return await _streamProcessor.GetRecentInteractionsAsync(userId, TimeSpan.FromHours(1));
    }
}

70. How do you implement real-time data synchronization?

Explanation: Real-time data synchronization ensures data consistency across distributed systems. Common patterns include:

1. Event-Driven Architecture: Using message queues and event streaming
2. Change Data Capture (CDC): Capturing database changes in real-time
3. Conflict Resolution: Handling concurrent updates
4. Event Sourcing: Storing events instead of state
5. CQRS: Separating read and write operations

// Real-time Data Synchronization Service
public class RealTimeDataSyncService
{
    private readonly IEventBus _eventBus;
    private readonly IChangeDataCapture _cdc;
    private readonly IConflictResolver _conflictResolver;
    private readonly Dictionary<string, IDataStore> _dataStores;

    public async Task SyncDataAsync(string entityId, object data, string source)
    {
        // Create sync event
        var syncEvent = new DataSyncEvent
        {
            EntityId = entityId,
            Data = data,
            Source = source,
            Timestamp = DateTime.UtcNow,
            Version = await GetNextVersionAsync(entityId)
        };

        // Publish to event bus
        await _eventBus.PublishAsync(syncEvent);

        // Apply to local store
        await ApplyToLocalStoreAsync(syncEvent);
    }

    public async Task HandleSyncEventAsync(DataSyncEvent syncEvent)
    {
        // Check for conflicts
        var existingData = await GetExistingDataAsync(syncEvent.EntityId);

        if (existingData != null && existingData.Version != syncEvent.Version - 1)
        {
            // Resolve conflict
            var resolvedData = await _conflictResolver.ResolveConflictAsync(
                existingData, syncEvent.Data);

            syncEvent.Data = resolvedData;
        }

        // Apply to all stores
        foreach (var store in _dataStores.Values)
        {
            await store.UpdateAsync(syncEvent.EntityId, syncEvent.Data);
        }
    }
}

// Change Data Capture Implementation
public class ChangeDataCapture : IChangeDataCapture
{
    private readonly IEventBus _eventBus;
    private readonly ILogger<ChangeDataCapture> _logger;

    public async Task StartCaptureAsync()
    {
        // Subscribe to database change notifications
        await SubscribeToDatabaseChangesAsync();
    }

    private async Task OnDatabaseChangeAsync(DatabaseChange change)
    {
        var syncEvent = new DataSyncEvent
        {
            EntityId = change.EntityId,
            Data = change.NewData,
            Source = "database",
            Timestamp = DateTime.UtcNow,
            ChangeType = change.ChangeType
        };

        await _eventBus.PublishAsync(syncEvent);
    }
}

// Conflict Resolution Strategy
public class ConflictResolver : IConflictResolver
{
    public async Task<object> ResolveConflictAsync(object existingData, object newData)
    {
        // Implement conflict resolution logic
        // Example: Last Write Wins, Merge, or Custom Business Logic

        if (IsLastWriteWins(existingData, newData))
        {
            return newData;
        }

        return await MergeDataAsync(existingData, newData);
    }
}

71. How would you design a data pipeline for ETL?

Explanation: An ETL (Extract, Transform, Load) pipeline processes data from source systems to target data warehouses. Key components include:

1. Extract: Data extraction from various sources
2. Transform: Data cleaning, validation, and transformation
3. Load: Loading processed data into target systems
4. Orchestration: Coordinating the entire pipeline
5. Monitoring: Tracking pipeline health and performance

// ETL Pipeline Design
public class ETLPipeline
{
    private readonly IDataExtractor _extractor;
    private readonly IDataTransformer _transformer;
    private readonly IDataLoader _loader;
    private readonly IPipelineOrchestrator _orchestrator;
    private readonly IPipelineMonitor _monitor;

    public async Task ExecutePipelineAsync(PipelineConfig config)
    {
        try
        {
            _monitor.StartPipeline(config.PipelineId);

            // Extract phase
            var rawData = await _extractor.ExtractAsync(config.SourceConfig);
            _monitor.LogExtractMetrics(rawData.Count);

            // Transform phase
            var transformedData = await _transformer.TransformAsync(rawData, config.TransformRules);
            _monitor.LogTransformMetrics(transformedData.Count);

            // Load phase
            await _loader.LoadAsync(transformedData, config.TargetConfig);
            _monitor.LogLoadMetrics(transformedData.Count);

            _monitor.CompletePipeline(config.PipelineId);
        }
        catch (Exception ex)
        {
            _monitor.FailPipeline(config.PipelineId, ex);
            throw;
        }
    }
}

// Data Extractor
public class DataExtractor : IDataExtractor
{
    public async Task<List<RawData>> ExtractAsync(SourceConfig config)
    {
        switch (config.SourceType)
        {
            case SourceType.Database:
                return await ExtractFromDatabaseAsync(config);
            case SourceType.File:
                return await ExtractFromFileAsync(config);
            case SourceType.Api:
                return await ExtractFromApiAsync(config);
            default:
                throw new NotSupportedException($"Source type {config.SourceType} not supported");
        }
    }

    private async Task<List<RawData>> ExtractFromDatabaseAsync(SourceConfig config)
    {
        using var connection = new SqlConnection(config.ConnectionString);
        var query = config.Query;

        var data = await connection.QueryAsync<dynamic>(query);
        return data.Select(d => new RawData { Content = d, Source = config.SourceName }).ToList();
    }
}

// Data Transformer
public class DataTransformer : IDataTransformer
{
    public async Task<List<TransformedData>> TransformAsync(
        List<RawData> rawData, 
        List<TransformRule> rules)
    {
        var transformedData = new List<TransformedData>();

        foreach (var data in rawData)
        {
            var transformed = new TransformedData { OriginalData = data };

            foreach (var rule in rules)
            {
                transformed = await ApplyTransformRuleAsync(transformed, rule);
            }

            transformedData.Add(transformed);
        }

        return transformedData;
    }

    private async Task<TransformedData> ApplyTransformRuleAsync(
        TransformedData data, 
        TransformRule rule)
    {
        switch (rule.Type)
        {
            case TransformType.Clean:
                data.Content = await CleanDataAsync(data.Content);
                break;
            case TransformType.Validate:
                await ValidateDataAsync(data.Content, rule.ValidationRules);
                break;
            case TransformType.Enrich:
                data.Content = await EnrichDataAsync(data.Content, rule.EnrichmentConfig);
                break;
        }

        return data;
    }
}

// Data Loader
public class DataLoader : IDataLoader
{
    public async Task LoadAsync(List<TransformedData> data, TargetConfig config)
    {
        switch (config.TargetType)
        {
            case TargetType.DataWarehouse:
                await LoadToDataWarehouseAsync(data, config);
                break;
            case TargetType.DataLake:
                await LoadToDataLakeAsync(data, config);
                break;
            case TargetType.Database:
                await LoadToDatabaseAsync(data, config);
                break;
        }
    }

    private async Task LoadToDataWarehouseAsync(List<TransformedData> data, TargetConfig config)
    {
        // Implement data warehouse loading logic
        // Could use Azure Synapse, Snowflake, or other DW solutions
    }
}

72. How do you implement stream processing?

Explanation: Stream processing handles continuous data streams in real-time. Key concepts include:

1. Event Streams: Continuous flow of events
2. Windowing: Processing data within time or count windows
3. State Management: Maintaining state across events
4. Fault Tolerance: Handling failures and ensuring exactly-once processing
5. Scalability: Horizontal scaling of processing nodes

// Stream Processing Implementation
public class StreamProcessor
{
    private readonly IEventStream _eventStream;
    private readonly IStreamProcessorEngine _engine;
    private readonly IStateStore _stateStore;

    public async Task StartProcessingAsync()
    {
        await _eventStream.SubscribeAsync(ProcessEventAsync);
    }

    private async Task ProcessEventAsync(StreamEvent streamEvent)
    {
        try
        {
            // Get current state
            var state = await _stateStore.GetStateAsync(streamEvent.Key);

            // Process event
            var result = await _engine.ProcessEventAsync(streamEvent, state);

            // Update state
            await _stateStore.UpdateStateAsync(streamEvent.Key, result.NewState);

            // Emit result
            await EmitResultAsync(result);
        }
        catch (Exception ex)
        {
            await HandleProcessingErrorAsync(streamEvent, ex);
        }
    }
}

// Windowed Stream Processing
public class WindowedStreamProcessor
{
    private readonly Dictionary<string, List<StreamEvent>> _windows;
    private readonly TimeSpan _windowSize;
    private readonly TimeSpan _slideInterval;

    public async Task ProcessWindowedEventsAsync(StreamEvent streamEvent)
    {
        var windowKey = GetWindowKey(streamEvent.Timestamp);

        if (!_windows.ContainsKey(windowKey))
        {
            _windows[windowKey] = new List<StreamEvent>();
        }

        _windows[windowKey].Add(streamEvent);

        // Process window if it's complete
        if (IsWindowComplete(windowKey))
        {
            await ProcessWindowAsync(windowKey);
            CleanupWindow(windowKey);
        }
    }

    private async Task ProcessWindowAsync(string windowKey)
    {
        var events = _windows[windowKey];

        // Aggregate events in window
        var aggregatedResult = await AggregateEventsAsync(events);

        // Emit aggregated result
        await EmitAggregatedResultAsync(aggregatedResult);
    }

    private async Task<AggregatedResult> AggregateEventsAsync(List<StreamEvent> events)
    {
        // Implement aggregation logic (sum, average, count, etc.)
        var result = new AggregatedResult
        {
            Count = events.Count,
            Sum = events.Sum(e => e.Value),
            Average = events.Average(e => e.Value),
            Min = events.Min(e => e.Value),
            Max = events.Max(e => e.Value)
        };

        return result;
    }
}

// Stateful Stream Processing
public class StatefulStreamProcessor
{
    private readonly IStateStore _stateStore;

    public async Task<ProcessingResult> ProcessWithStateAsync(StreamEvent streamEvent)
    {
        // Get current state
        var currentState = await _stateStore.GetStateAsync(streamEvent.Key);

        // Apply event to state
        var newState = ApplyEventToState(currentState, streamEvent);

        // Update state
        await _stateStore.UpdateStateAsync(streamEvent.Key, newState);

        return new ProcessingResult
        {
            ProcessedEvent = streamEvent,
            NewState = newState,
            Output = GenerateOutput(newState)
        };
    }

    private State ApplyEventToState(State currentState, StreamEvent streamEvent)
    {
        // Implement state transition logic
        return new State
        {
            Count = currentState.Count + 1,
            LastEventTime = streamEvent.Timestamp,
            RunningTotal = currentState.RunningTotal + streamEvent.Value
        };
    }
}

73. How would you design a recommendation engine?

Explanation: A recommendation engine suggests items to users based on various algorithms:

1. Collaborative Filtering: Based on similar users' preferences
2. Content-Based Filtering: Based on item features
3. Hybrid Approaches: Combining multiple methods
4. Real-time Updates: Incorporating user feedback
5. A/B Testing: Testing different recommendation strategies

// Recommendation Engine Design
public class RecommendationEngine
{
    private readonly ICollaborativeFilter _collaborativeFilter;
    private readonly IContentBasedFilter _contentBasedFilter;
    private readonly IHybridRecommender _hybridRecommender;
    private readonly IUserProfileService _userProfileService;
    private readonly IItemCatalogService _itemCatalogService;

    public async Task<List<Recommendation>> GetRecommendationsAsync(
        string userId, 
        RecommendationRequest request)
    {
        var userProfile = await _userProfileService.GetUserProfileAsync(userId);
        var availableItems = await _itemCatalogService.GetAvailableItemsAsync();

        var recommendations = new List<Recommendation>();

        // Collaborative filtering
        if (request.UseCollaborativeFiltering)
        {
            var collaborativeRecs = await _collaborativeFilter.GetRecommendationsAsync(
                userId, availableItems, request.Limit);
            recommendations.AddRange(collaborativeRecs);
        }

        // Content-based filtering
        if (request.UseContentBasedFiltering)
        {
            var contentBasedRecs = await _contentBasedFilter.GetRecommendationsAsync(
                userProfile, availableItems, request.Limit);
            recommendations.AddRange(contentBasedRecs);
        }

        // Hybrid approach
        if (request.UseHybridApproach)
        {
            var hybridRecs = await _hybridRecommender.GetRecommendationsAsync(
                userId, userProfile, availableItems, request.Limit);
            recommendations.AddRange(hybridRecs);
        }

        // Rank and return top recommendations
        return await RankRecommendationsAsync(recommendations, request.Limit);
    }
}

// Collaborative Filtering Implementation
public class CollaborativeFilter : ICollaborativeFilter
{
    private readonly IUserSimilarityCalculator _similarityCalculator;
    private readonly IUserRatingService _ratingService;

    public async Task<List<Recommendation>> GetRecommendationsAsync(
        string userId, 
        List<Item> availableItems, 
        int limit)
    {
        // Find similar users
        var similarUsers = await _similarityCalculator.FindSimilarUsersAsync(userId);

        // Get ratings from similar users
        var similarUserRatings = new Dictionary<string, List<UserRating>>();
        foreach (var similarUser in similarUsers)
        {
            var ratings = await _ratingService.GetUserRatingsAsync(similarUser.UserId);
            similarUserRatings[similarUser.UserId] = ratings;
        }

        // Calculate predicted ratings
        var predictedRatings = new List<PredictedRating>();
        foreach (var item in availableItems)
        {
            var predictedRating = CalculatePredictedRating(userId, item, similarUserRatings);
            if (predictedRating.Score > 0)
            {
                predictedRatings.Add(predictedRating);
            }
        }

        // Convert to recommendations
        return predictedRatings
            .OrderByDescending(r => r.Score)
            .Take(limit)
            .Select(r => new Recommendation
            {
                ItemId = r.ItemId,
                Score = r.Score,
                Algorithm = "CollaborativeFiltering",
                Confidence = r.Confidence
            })
            .ToList();
    }

    private PredictedRating CalculatePredictedRating(
        string userId, 
        Item item, 
        Dictionary<string, List<UserRating>> similarUserRatings)
    {
        var weightedSum = 0.0;
        var totalWeight = 0.0;

        foreach (var kvp in similarUserRatings)
        {
            var similarUserId = kvp.Key;
            var ratings = kvp.Value;

            var itemRating = ratings.FirstOrDefault(r => r.ItemId == item.Id);
            if (itemRating != null)
            {
                var similarity = GetUserSimilarity(userId, similarUserId);
                weightedSum += itemRating.Rating * similarity;
                totalWeight += similarity;
            }
        }

        var predictedRating = totalWeight > 0 ? weightedSum / totalWeight : 0;
        var confidence = CalculateConfidence(similarUserRatings, item.Id);

        return new PredictedRating
        {
            ItemId = item.Id,
            Score = predictedRating,
            Confidence = confidence
        };
    }
}

// Content-Based Filtering Implementation
public class ContentBasedFilter : IContentBasedFilter
{
    private readonly IItemFeatureExtractor _featureExtractor;
    private readonly IUserPreferenceLearner _preferenceLearner;

    public async Task<List<Recommendation>> GetRecommendationsAsync(
        UserProfile userProfile, 
        List<Item> availableItems, 
        int limit)
    {
        // Extract user preferences
        var userPreferences = await _preferenceLearner.ExtractPreferencesAsync(userProfile);

        var recommendations = new List<Recommendation>();

        foreach (var item in availableItems)
        {
            // Extract item features
            var itemFeatures = await _featureExtractor.ExtractFeaturesAsync(item);

            // Calculate similarity between user preferences and item features
            var similarity = CalculateSimilarity(userPreferences, itemFeatures);

            if (similarity > 0.5) // Threshold for recommendation
            {
                recommendations.Add(new Recommendation
                {
                    ItemId = item.Id,
                    Score = similarity,
                    Algorithm = "ContentBasedFiltering",
                    Confidence = CalculateConfidence(userPreferences, itemFeatures)
                });
            }
        }

        return recommendations
            .OrderByDescending(r => r.Score)
            .Take(limit)
            .ToList();
    }

    private double CalculateSimilarity(UserPreferences preferences, ItemFeatures features)
    {
        // Implement similarity calculation (cosine similarity, Jaccard, etc.)
        var dotProduct = 0.0;
        var preferenceNorm = 0.0;
        var featureNorm = 0.0;

        foreach (var preference in preferences.Features)
        {
            if (features.Features.ContainsKey(preference.Key))
            {
                dotProduct += preference.Value * features.Features[preference.Key];
            }
            preferenceNorm += preference.Value * preference.Value;
        }

        foreach (var feature in features.Features.Values)
        {
            featureNorm += feature * feature;
        }

        if (preferenceNorm == 0 || featureNorm == 0)
            return 0;

        return dotProduct / (Math.Sqrt(preferenceNorm) * Math.Sqrt(featureNorm));
    }
}

74. How do you implement real-time analytics?

Explanation: Real-time analytics processes data as it arrives to provide immediate insights. Key components include:

1. Stream Processing: Processing data streams in real-time
2. Time-Series Databases: Storing time-stamped data efficiently
3. Aggregation Engines: Computing metrics in real-time
4. Alerting Systems: Triggering alerts based on thresholds
5. Dashboard Updates: Real-time visualization updates

// Real-time Analytics Implementation
public class RealTimeAnalyticsService
{
    private readonly IStreamProcessor _streamProcessor;
    private readonly ITimeSeriesDatabase _timeSeriesDb;
    private readonly IAggregationEngine _aggregationEngine;
    private readonly IAlertingService _alertingService;
    private readonly IDashboardService _dashboardService;

    public async Task ProcessAnalyticsEventAsync(AnalyticsEvent analyticsEvent)
    {
        // Store raw event
        await _timeSeriesDb.StoreEventAsync(analyticsEvent);

        // Process in stream
        await _streamProcessor.ProcessEventAsync(analyticsEvent);

        // Update aggregations
        await UpdateAggregationsAsync(analyticsEvent);

        // Check alerts
        await CheckAlertsAsync(analyticsEvent);

        // Update dashboards
        await UpdateDashboardsAsync(analyticsEvent);
    }

    private async Task UpdateAggregationsAsync(AnalyticsEvent analyticsEvent)
    {
        var aggregations = await _aggregationEngine.CalculateAggregationsAsync(analyticsEvent);

        foreach (var aggregation in aggregations)
        {
            await _timeSeriesDb.StoreAggregationAsync(aggregation);
        }
    }

    private async Task CheckAlertsAsync(AnalyticsEvent analyticsEvent)
    {
        var alerts = await _alertingService.CheckAlertsAsync(analyticsEvent);

        foreach (var alert in alerts)
        {
            await _alertingService.TriggerAlertAsync(alert);
        }
    }

    private async Task UpdateDashboardsAsync(AnalyticsEvent analyticsEvent)
    {
        await _dashboardService.UpdateMetricsAsync(analyticsEvent);
    }
}

// Time-Series Database Implementation
public class TimeSeriesDatabase : ITimeSeriesDatabase
{
    private readonly IDbConnection _connection;

    public async Task StoreEventAsync(AnalyticsEvent analyticsEvent)
    {
        var sql = @"
            INSERT INTO analytics_events (timestamp, event_type, user_id, data, metadata)
            VALUES (@Timestamp, @EventType, @UserId, @Data, @Metadata)";

        await _connection.ExecuteAsync(sql, analyticsEvent);
    }

    public async Task<List<AnalyticsEvent>> GetEventsAsync(
        DateTime startTime, 
        DateTime endTime, 
        string eventType = null)
    {
        var sql = @"
            SELECT * FROM analytics_events 
            WHERE timestamp BETWEEN @StartTime AND @EndTime";

        if (!string.IsNullOrEmpty(eventType))
        {
            sql += " AND event_type = @EventType";
        }

        sql += " ORDER BY timestamp DESC";

        return (await _connection.QueryAsync<AnalyticsEvent>(sql, 
            new { StartTime = startTime, EndTime = endTime, EventType = eventType })).ToList();
    }

    public async Task StoreAggregationAsync(Aggregation aggregation)
    {
        var sql = @"
            INSERT INTO aggregations (timestamp, metric_name, value, window_size)
            VALUES (@Timestamp, @MetricName, @Value, @WindowSize)
            ON CONFLICT (timestamp, metric_name, window_size) 
            DO UPDATE SET value = @Value";

        await _connection.ExecuteAsync(sql, aggregation);
    }
}

// Aggregation Engine
public class AggregationEngine : IAggregationEngine
{
    private readonly Dictionary<string, AggregationWindow> _windows;

    public async Task<List<Aggregation>> CalculateAggregationsAsync(AnalyticsEvent analyticsEvent)
    {
        var aggregations = new List<Aggregation>();

        // Update sliding windows
        await UpdateWindowsAsync(analyticsEvent);

        // Calculate aggregations for each window
        foreach (var window in _windows.Values)
        {
            var aggregation = await CalculateWindowAggregationAsync(window);
            aggregations.Add(aggregation);
        }

        return aggregations;
    }

    private async Task UpdateWindowsAsync(AnalyticsEvent analyticsEvent)
    {
        foreach (var window in _windows.Values)
        {
            window.AddEvent(analyticsEvent);
        }
    }

    private async Task<Aggregation> CalculateWindowAggregationAsync(AggregationWindow window)
    {
        return new Aggregation
        {
            Timestamp = DateTime.UtcNow,
            MetricName = window.MetricName,
            Value = window.CalculateAggregation(),
            WindowSize = window.WindowSize
        };
    }
}

// Alerting Service
public class AlertingService : IAlertingService
{
    private readonly List<AlertRule> _alertRules;

    public async Task<List<Alert>> CheckAlertsAsync(AnalyticsEvent analyticsEvent)
    {
        var triggeredAlerts = new List<Alert>();

        foreach (var rule in _alertRules)
        {
            if (await EvaluateAlertRuleAsync(rule, analyticsEvent))
            {
                var alert = new Alert
                {
                    RuleId = rule.Id,
                    Message = rule.Message,
                    Severity = rule.Severity,
                    Timestamp = DateTime.UtcNow,
                    EventData = analyticsEvent
                };

                triggeredAlerts.Add(alert);
            }
        }

        return triggeredAlerts;
    }

    private async Task<bool> EvaluateAlertRuleAsync(AlertRule rule, AnalyticsEvent analyticsEvent)
    {
        // Implement alert rule evaluation logic
        switch (rule.Condition)
        {
            case AlertCondition.ThresholdExceeded:
                return analyticsEvent.Value > rule.Threshold;
            case AlertCondition.ThresholdBelow:
                return analyticsEvent.Value < rule.Threshold;
            case AlertCondition.AnomalyDetected:
                return await DetectAnomalyAsync(analyticsEvent);
            default:
                return false;
        }
    }

    public async Task TriggerAlertAsync(Alert alert)
    {
        // Send alert through various channels (email, SMS, webhook, etc.)
        await SendEmailAlertAsync(alert);
        await SendWebhookAlertAsync(alert);
        await LogAlertAsync(alert);
    }
}

75. How would you design a search engine?

Explanation: A search engine indexes and searches through large volumes of data efficiently. Key components include:

1. Crawler: Discovers and fetches content
2. Indexer: Creates searchable indexes
3. Query Processor: Processes search queries
4. Ranking Algorithm: Ranks search results
5. Search API: Provides search functionality

// Search Engine Design
public class SearchEngine
{
    private readonly ICrawler _crawler;
    private readonly IIndexer _indexer;
    private readonly IQueryProcessor _queryProcessor;
    private readonly IRankingEngine _rankingEngine;
    private readonly ISearchIndex _searchIndex;

    public async Task IndexContentAsync(ContentSource source)
    {
        // Crawl content
        var documents = await _crawler.CrawlAsync(source);

        // Index documents
        foreach (var document in documents)
        {
            await _indexer.IndexDocumentAsync(document);
        }
    }

    public async Task<SearchResult> SearchAsync(SearchQuery query)
    {
        // Process query
        var processedQuery = await _queryProcessor.ProcessQueryAsync(query);

        // Search index
        var candidates = await _searchIndex.SearchAsync(processedQuery);

        // Rank results
        var rankedResults = await _rankingEngine.RankResultsAsync(candidates, query);

        return new SearchResult
        {
            Query = query,
            Results = rankedResults,
            TotalCount = candidates.Count,
            SearchTime = DateTime.UtcNow
        };
    }
}

// Document Indexer
public class DocumentIndexer : IIndexer
{
    private readonly ISearchIndex _searchIndex;
    private readonly ITextAnalyzer _textAnalyzer;

    public async Task IndexDocumentAsync(Document document)
    {
        // Analyze document text
        var tokens = await _textAnalyzer.AnalyzeAsync(document.Content);

        // Create inverted index entries
        var indexEntries = CreateIndexEntries(document, tokens);

        // Store in search index
        await _searchIndex.StoreIndexEntriesAsync(indexEntries);
    }

    private List<IndexEntry> CreateIndexEntries(Document document, List<Token> tokens)
    {
        var entries = new List<IndexEntry>();
        var tokenPositions = new Dictionary<string, List<int>>();

        // Build token positions
        for (int i = 0; i < tokens.Count; i++)
        {
            var token = tokens[i].Text.ToLower();
            if (!tokenPositions.ContainsKey(token))
            {
                tokenPositions[token] = new List<int>();
            }
            tokenPositions[token].Add(i);
        }

        // Create index entries
        foreach (var kvp in tokenPositions)
        {
            var entry = new IndexEntry
            {
                Term = kvp.Key,
                DocumentId = document.Id,
                Positions = kvp.Value,
                Frequency = kvp.Value.Count,
                DocumentMetadata = document.Metadata
            };

            entries.Add(entry);
        }

        return entries;
    }
}

// Query Processor
public class QueryProcessor : IQueryProcessor
{
    private readonly ITextAnalyzer _textAnalyzer;
    private readonly IQueryParser _queryParser;

    public async Task<ProcessedQuery> ProcessQueryAsync(SearchQuery query)
    {
        // Parse query syntax
        var parsedQuery = await _queryParser.ParseAsync(query.Text);

        // Analyze query terms
        var analyzedTerms = new List<AnalyzedTerm>();
        foreach (var term in parsedQuery.Terms)
        {
            var analyzed = await _textAnalyzer.AnalyzeAsync(term);
            analyzedTerms.AddRange(analyzed);
        }

        return new ProcessedQuery
        {
            OriginalQuery = query,
            Terms = analyzedTerms,
            Filters = parsedQuery.Filters,
            SortOptions = parsedQuery.SortOptions
        };
    }
}

// Ranking Engine
public class RankingEngine : IRankingEngine
{
    private readonly ITfIdfCalculator _tfIdfCalculator;
    private readonly IPageRankCalculator _pageRankCalculator;

    public async Task<List<RankedResult>> RankResultsAsync(
        List<SearchCandidate> candidates, 
        SearchQuery query)
    {
        var rankedResults = new List<RankedResult>();

        foreach (var candidate in candidates)
        {
            var score = await CalculateScoreAsync(candidate, query);

            rankedResults.Add(new RankedResult
            {
                Document = candidate.Document,
                Score = score,
                Highlights = await GenerateHighlightsAsync(candidate, query)
            });
        }

        return rankedResults
            .OrderByDescending(r => r.Score)
            .ToList();
    }

    private async Task<double> CalculateScoreAsync(SearchCandidate candidate, SearchQuery query)
    {
        // TF-IDF score
        var tfIdfScore = await _tfIdfCalculator.CalculateScoreAsync(candidate, query);

        // PageRank score
        var pageRankScore = await _pageRankCalculator.CalculateScoreAsync(candidate.Document);

        // Combine scores (weighted combination)
        var finalScore = 0.7 * tfIdfScore + 0.3 * pageRankScore;

        // Apply boost factors
        if (candidate.Document.Metadata.ContainsKey("boost"))
        {
            finalScore *= double.Parse(candidate.Document.Metadata["boost"]);
        }

        return finalScore;
    }

    private async Task<List<string>> GenerateHighlightsAsync(
        SearchCandidate candidate, 
        SearchQuery query)
    {
        var highlights = new List<string>();
        var content = candidate.Document.Content;

        foreach (var term in query.Terms)
        {
            var positions = candidate.Positions[term];
            foreach (var position in positions.Take(3)) // Limit highlights
            {
                var highlight = ExtractHighlight(content, position, 50);
                highlights.Add(highlight);
            }
        }

        return highlights.Distinct().ToList();
    }

    private string ExtractHighlight(string content, int position, int contextLength)
    {
        var start = Math.Max(0, position - contextLength / 2);
        var length = Math.Min(contextLength, content.Length - start);
        return content.Substring(start, length);
    }
}

// TF-IDF Calculator
public class TfIdfCalculator : ITfIdfCalculator
{
    public async Task<double> CalculateScoreAsync(SearchCandidate candidate, SearchQuery query)
    {
        var score = 0.0;

        foreach (var term in query.Terms)
        {
            var tf = CalculateTermFrequency(term, candidate);
            var idf = await CalculateInverseDocumentFrequencyAsync(term);
            score += tf * idf;
        }

        return score;
    }

    private double CalculateTermFrequency(string term, SearchCandidate candidate)
    {
        if (candidate.Positions.ContainsKey(term))
        {
            return (double)candidate.Positions[term].Count / candidate.Document.Content.Length;
        }
        return 0;
    }

    private async Task<double> CalculateInverseDocumentFrequencyAsync(string term)
    {
        // This would typically query the index to get document frequency
        // For simplicity, returning a placeholder value
        return 1.0;
    }
}

76. How do you implement data warehousing?

Explanation: Data warehousing involves collecting, storing, and managing data from multiple sources for analysis and reporting. Key components include:

1. ETL Processes: Extract, Transform, Load data
2. Data Models: Star schema, snowflake schema
3. Data Marts: Subject-specific data subsets
4. OLAP Cubes: Multi-dimensional analysis
5. Data Quality: Ensuring data accuracy and consistency

// Data Warehouse Implementation
public class DataWarehouse
{
    private readonly IETLService _etlService;
    private readonly IDataModelService _dataModelService;
    private readonly IDataQualityService _dataQualityService;
    private readonly IOLAPService _olapService;

    public async Task LoadDataAsync(DataSource source)
    {
        // Extract data
        var rawData = await _etlService.ExtractAsync(source);

        // Transform data
        var transformedData = await _etlService.TransformAsync(rawData);

        // Validate data quality
        var qualityReport = await _dataQualityService.ValidateAsync(transformedData);

        if (qualityReport.IsValid)
        {
            // Load to warehouse
            await _etlService.LoadAsync(transformedData);

            // Update OLAP cubes
            await _olapService.UpdateCubesAsync();
        }
        else
        {
            await HandleDataQualityIssuesAsync(qualityReport);
        }
    }

    public async Task<DataMart> CreateDataMartAsync(DataMartConfig config)
    {
        var dataMart = new DataMart
        {
            Name = config.Name,
            Subject = config.Subject,
            Schema = await _dataModelService.CreateSchemaAsync(config.Schema)
        };

        // Populate data mart
        await PopulateDataMartAsync(dataMart, config);

        return dataMart;
    }
}

// Star Schema Implementation
public class StarSchema : IDataModel
{
    public FactTable FactTable { get; set; }
    public List<DimensionTable> DimensionTables { get; set; }

    public async Task CreateSchemaAsync()
    {
        // Create dimension tables
        foreach (var dimension in DimensionTables)
        {
            await CreateDimensionTableAsync(dimension);
        }

        // Create fact table
        await CreateFactTableAsync(FactTable);
    }

    private async Task CreateDimensionTableAsync(DimensionTable dimension)
    {
        var sql = $@"
            CREATE TABLE {dimension.Name} (
                {dimension.KeyColumn} INT PRIMARY KEY IDENTITY(1,1),
                {string.Join(", ", dimension.Attributes.Select(a => $"{a.Name} {a.DataType}"))}
            )";

        await ExecuteSqlAsync(sql);
    }

    private async Task CreateFactTableAsync(FactTable factTable)
    {
        var foreignKeys = string.Join(", ", 
            factTable.DimensionKeys.Select(k => $"{k.Name} INT FOREIGN KEY REFERENCES {k.DimensionTable}({k.KeyColumn})"));

        var measures = string.Join(", ", 
            factTable.Measures.Select(m => $"{m.Name} {m.DataType}"));

        var sql = $@"
            CREATE TABLE {factTable.Name} (
                {factTable.KeyColumn} INT PRIMARY KEY IDENTITY(1,1),
                {foreignKeys},
                {measures}
            )";

        await ExecuteSqlAsync(sql);
    }
}

// ETL Service for Data Warehouse
public class DataWarehouseETLService : IETLService
{
    private readonly IDataExtractor _extractor;
    private readonly IDataTransformer _transformer;
    private readonly IDataLoader _loader;
    private readonly IDataQualityService _qualityService;

    public async Task<List<RawData>> ExtractAsync(DataSource source)
    {
        var rawData = new List<RawData>();

        switch (source.Type)
        {
            case SourceType.Database:
                rawData = await ExtractFromDatabaseAsync(source);
                break;
            case SourceType.File:
                rawData = await ExtractFromFileAsync(source);
                break;
            case SourceType.Api:
                rawData = await ExtractFromApiAsync(source);
                break;
        }

        return rawData;
    }

    public async Task<List<TransformedData>> TransformAsync(List<RawData> rawData)
    {
        var transformedData = new List<TransformedData>();

        foreach (var data in rawData)
        {
            var transformed = await TransformRecordAsync(data);
            transformedData.Add(transformed);
        }

        return transformedData;
    }

    public async Task LoadAsync(List<TransformedData> transformedData)
    {
        // Load dimensions first
        await LoadDimensionsAsync(transformedData);

        // Load facts
        await LoadFactsAsync(transformedData);
    }

    private async Task LoadDimensionsAsync(List<TransformedData> data)
    {
        foreach (var record in data)
        {
            // Load customer dimension
            var customerKey = await LoadCustomerDimensionAsync(record.CustomerData);

            // Load product dimension
            var productKey = await LoadProductDimensionAsync(record.ProductData);

            // Load time dimension
            var timeKey = await LoadTimeDimensionAsync(record.TimeData);

            // Store keys for fact table
            record.DimensionKeys = new Dictionary<string, int>
            {
                ["CustomerKey"] = customerKey,
                ["ProductKey"] = productKey,
                ["TimeKey"] = timeKey
            };
        }
    }

    private async Task LoadFactsAsync(List<TransformedData> data)
    {
        foreach (var record in data)
        {
            var sql = @"
                INSERT INTO SalesFact (CustomerKey, ProductKey, TimeKey, Quantity, Amount)
                VALUES (@CustomerKey, @ProductKey, @TimeKey, @Quantity, @Amount)";

            await ExecuteSqlAsync(sql, record);
        }
    }
}

// OLAP Cube Implementation
public class OLAPCube
{
    public string Name { get; set; }
    public List<Dimension> Dimensions { get; set; }
    public List<Measure> Measures { get; set; }
    public Dictionary<string, object> Data { get; set; }

    public async Task<CubeResult> QueryAsync(CubeQuery query)
    {
        var result = new CubeResult();

        // Apply filters
        var filteredData = ApplyFilters(Data, query.Filters);

        // Group by dimensions
        var groupedData = GroupByDimensions(filteredData, query.Dimensions);

        // Calculate measures
        result.Data = CalculateMeasures(groupedData, query.Measures);

        return result;
    }

    private Dictionary<string, object> ApplyFilters(
        Dictionary<string, object> data, 
        List<CubeFilter> filters)
    {
        var filteredData = data;

        foreach (var filter in filters)
        {
            filteredData = filteredData
                .Where(kvp => EvaluateFilter(kvp.Value, filter))
                .ToDictionary(kvp => kvp.Key, kvp => kvp.Value);
        }

        return filteredData;
    }

    private Dictionary<string, object> GroupByDimensions(
        Dictionary<string, object> data, 
        List<string> dimensions)
    {
        // Implement grouping logic
        return data.GroupBy(kvp => GetDimensionKey(kvp.Value, dimensions))
                  .ToDictionary(g => g.Key, g => g.ToList());
    }

    private Dictionary<string, double> CalculateMeasures(
        Dictionary<string, object> groupedData, 
        List<string> measures)
    {
        var results = new Dictionary<string, double>();

        foreach (var group in groupedData)
        {
            foreach (var measure in measures)
            {
                var value = CalculateMeasure(group.Value, measure);
                results[$"{group.Key}_{measure}"] = value;
            }
        }

        return results;
    }
}

// Data Quality Service
public class DataQualityService : IDataQualityService
{
    public async Task<QualityReport> ValidateAsync(List<TransformedData> data)
    {
        var report = new QualityReport();

        foreach (var record in data)
        {
            // Check for null values
            var nullCheck = CheckNullValues(record);
            report.NullValueIssues.AddRange(nullCheck);

            // Check for data type consistency
            var typeCheck = CheckDataTypeConsistency(record);
            report.DataTypeIssues.AddRange(typeCheck);

            // Check for business rules
            var businessCheck = CheckBusinessRules(record);
            report.BusinessRuleIssues.AddRange(businessCheck);

            // Check for duplicates
            var duplicateCheck = CheckDuplicates(record, data);
            report.DuplicateIssues.AddRange(duplicateCheck);
        }

        report.IsValid = !report.HasIssues();
        return report;
    }

    private List<QualityIssue> CheckNullValues(TransformedData record)
    {
        var issues = new List<QualityIssue>();

        foreach (var property in record.GetType().GetProperties())
        {
            var value = property.GetValue(record);
            if (value == null && IsRequired(property))
            {
                issues.Add(new QualityIssue
                {
                    Type = IssueType.NullValue,
                    Field = property.Name,
                    Message = $"Required field {property.Name} is null"
                });
            }
        }

        return issues;
    }

    private List<QualityIssue> CheckDataTypeConsistency(TransformedData record)
    {
        var issues = new List<QualityIssue>();

        // Implement data type validation logic
        // Check if values match expected data types

        return issues;
    }

    private List<QualityIssue> CheckBusinessRules(TransformedData record)
    {
        var issues = new List<QualityIssue>();

        // Implement business rule validation
        // Example: Amount must be positive, Date must be in valid range, etc.

        return issues;
    }

    private List<QualityIssue> CheckDuplicates(TransformedData record, List<TransformedData> allData)
    {
        var issues = new List<QualityIssue>();

        // Implement duplicate detection logic
        // Check for exact duplicates or near-duplicates

        return issues;
    }
}

Data & Analytics Architecture

77. How would you design a machine learning pipeline?

Explanation: A machine learning pipeline is a systematic approach to building, deploying, and maintaining ML models. It includes data ingestion, preprocessing, feature engineering, model training, evaluation, deployment, and monitoring.

Key Components: - Data Ingestion & Validation - Feature Engineering & Selection - Model Training & Validation - Model Deployment & Serving - Monitoring & Retraining

C# Implementation:

using Microsoft.ML;
using Microsoft.ML.Data;
using System;
using System.Collections.Generic;
using System.Threading.Tasks;

public class MLPipeline
{
    private readonly MLContext _mlContext;
    private readonly IDataIngestionService _dataIngestion;
    private readonly IFeatureEngineeringService _featureEngineering;
    private readonly IModelTrainingService _modelTraining;
    private readonly IModelDeploymentService _modelDeployment;
    private readonly IMonitoringService _monitoring;

    public MLPipeline(
        IDataIngestionService dataIngestion,
        IFeatureEngineeringService featureEngineering,
        IModelTrainingService modelTraining,
        IModelDeploymentService modelDeployment,
        IMonitoringService monitoring)
    {
        _mlContext = new MLContext(seed: 42);
        _dataIngestion = dataIngestion;
        _featureEngineering = featureEngineering;
        _modelTraining = modelTraining;
        _modelDeployment = modelDeployment;
        _monitoring = monitoring;
    }

    public async Task<PipelineResult> ExecutePipelineAsync(PipelineConfig config)
    {
        try
        {
            // 1. Data Ingestion
            var rawData = await _dataIngestion.IngestDataAsync(config.DataSource);

            // 2. Data Validation
            var validationResult = await ValidateDataAsync(rawData);
            if (!validationResult.IsValid)
                return PipelineResult.Failed(validationResult.Errors);

            // 3. Feature Engineering
            var features = await _featureEngineering.ProcessFeaturesAsync(rawData);

            // 4. Data Splitting
            var (trainData, testData) = SplitData(features, config.TestSize);

            // 5. Model Training
            var model = await _modelTraining.TrainModelAsync(trainData, config.ModelType);

            // 6. Model Evaluation
            var evaluation = await EvaluateModelAsync(model, testData);

            // 7. Model Deployment
            if (evaluation.Metrics.Accuracy > config.MinAccuracy)
            {
                var deploymentResult = await _modelDeployment.DeployModelAsync(model, config.Environment);

                // 8. Monitoring Setup
                await _monitoring.SetupMonitoringAsync(model.ModelId, config.MonitoringConfig);

                return PipelineResult.Success(model.ModelId, evaluation.Metrics);
            }

            return PipelineResult.Failed(new[] { "Model accuracy below threshold" });
        }
        catch (Exception ex)
        {
            return PipelineResult.Failed(new[] { ex.Message });
        }
    }

    private async Task<ValidationResult> ValidateDataAsync(IDataView data)
    {
        // Implement data validation logic
        return new ValidationResult { IsValid = true };
    }

    private (IDataView train, IDataView test) SplitData(IDataView data, double testSize)
    {
        return _mlContext.Data.TrainTestSplit(data, testFraction: testSize);
    }

    private async Task<ModelEvaluation> EvaluateModelAsync(ITransformer model, IDataView testData)
    {
        // Implement model evaluation logic
        return new ModelEvaluation { Metrics = new ModelMetrics { Accuracy = 0.95 } };
    }
}

public class PipelineConfig
{
    public string DataSource { get; set; }
    public string ModelType { get; set; }
    public double TestSize { get; set; } = 0.2;
    public double MinAccuracy { get; set; } = 0.8;
    public string Environment { get; set; }
    public MonitoringConfig MonitoringConfig { get; set; }
}

public class PipelineResult
{
    public bool IsSuccess { get; set; }
    public string ModelId { get; set; }
    public ModelMetrics Metrics { get; set; }
    public string[] Errors { get; set; }

    public static PipelineResult Success(string modelId, ModelMetrics metrics) =>
        new PipelineResult { IsSuccess = true, ModelId = modelId, Metrics = metrics };

    public static PipelineResult Failed(string[] errors) =>
        new PipelineResult { IsSuccess = false, Errors = errors };
}

78. How do you implement data lake architecture?

Explanation: A data lake is a centralized repository that stores all data in its raw format. It supports structured, semi-structured, and unstructured data, enabling data scientists and analysts to access data without predefined schemas.

Key Components: - Raw Data Zone (Bronze) - Processed Data Zone (Silver) - Curated Data Zone (Gold) - Data Governance & Security - Data Catalog & Metadata Management

C# Implementation:

using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
using System;
using System.Collections.Generic;
using System.Threading.Tasks;

public class DataLakeArchitecture
{
    private readonly BlobServiceClient _blobServiceClient;
    private readonly IDataCatalogService _dataCatalog;
    private readonly IDataGovernanceService _dataGovernance;
    private readonly IDataProcessingService _dataProcessing;

    public DataLakeArchitecture(
        string connectionString,
        IDataCatalogService dataCatalog,
        IDataGovernanceService dataGovernance,
        IDataProcessingService dataProcessing)
    {
        _blobServiceClient = new BlobServiceClient(connectionString);
        _dataCatalog = dataCatalog;
        _dataGovernance = dataGovernance;
        _dataProcessing = dataProcessing;
    }

    public async Task<DataIngestionResult> IngestDataAsync(DataIngestionRequest request)
    {
        try
        {
            // 1. Validate and classify data
            var dataClassification = await ClassifyDataAsync(request.Data);

            // 2. Apply data governance policies
            var governanceResult = await _dataGovernance.ApplyPoliciesAsync(request.Data, dataClassification);
            if (!governanceResult.IsCompliant)
                return DataIngestionResult.Failed(governanceResult.Violations);

            // 3. Store in Bronze zone (raw data)
            var bronzePath = await StoreInBronzeZoneAsync(request.Data, dataClassification);

            // 4. Process and store in Silver zone
            var silverPath = await ProcessAndStoreInSilverZoneAsync(bronzePath, dataClassification);

            // 5. Curate and store in Gold zone
            var goldPath = await CurateAndStoreInGoldZoneAsync(silverPath, dataClassification);

            // 6. Update data catalog
            await _dataCatalog.RegisterDataAsync(new DataCatalogEntry
            {
                BronzePath = bronzePath,
                SilverPath = silverPath,
                GoldPath = goldPath,
                Classification = dataClassification,
                Metadata = request.Metadata
            });

            return DataIngestionResult.Success(bronzePath, silverPath, goldPath);
        }
        catch (Exception ex)
        {
            return DataIngestionResult.Failed(new[] { ex.Message });
        }
    }

    private async Task<DataClassification> ClassifyDataAsync(byte[] data)
    {
        // Implement data classification logic
        return new DataClassification
        {
            DataType = DataType.Structured,
            Sensitivity = SensitivityLevel.Public,
            RetentionPeriod = TimeSpan.FromDays(365)
        };
    }

    private async Task<string> StoreInBronzeZoneAsync(byte[] data, DataClassification classification)
    {
        var containerName = "bronze-zone";
        var blobName = $"{classification.DataType}/{DateTime.UtcNow:yyyy/MM/dd}/{Guid.NewGuid()}.json";

        var containerClient = _blobServiceClient.GetBlobContainerClient(containerName);
        await containerClient.CreateIfNotExistsAsync();

        var blobClient = containerClient.GetBlobClient(blobName);
        await blobClient.UploadAsync(new BinaryData(data), overwrite: true);

        return $"{containerName}/{blobName}";
    }

    private async Task<string> ProcessAndStoreInSilverZoneAsync(string bronzePath, DataClassification classification)
    {
        // Implement data processing logic
        var processedData = await _dataProcessing.ProcessDataAsync(bronzePath);

        var containerName = "silver-zone";
        var blobName = $"{classification.DataType}/{DateTime.UtcNow:yyyy/MM/dd}/{Guid.NewGuid()}.parquet";

        var containerClient = _blobServiceClient.GetBlobContainerClient(containerName);
        await containerClient.CreateIfNotExistsAsync();

        var blobClient = containerClient.GetBlobClient(blobName);
        await blobClient.UploadAsync(new BinaryData(processedData), overwrite: true);

        return $"{containerName}/{blobName}";
    }

    private async Task<string> CurateAndStoreInGoldZoneAsync(string silverPath, DataClassification classification)
    {
        // Implement data curation logic
        var curatedData = await _dataProcessing.CurateDataAsync(silverPath);

        var containerName = "gold-zone";
        var blobName = $"{classification.DataType}/{DateTime.UtcNow:yyyy/MM/dd}/{Guid.NewGuid()}.parquet";

        var containerClient = _blobServiceClient.GetBlobContainerClient(containerName);
        await containerClient.CreateIfNotExistsAsync();

        var blobClient = containerClient.GetBlobClient(blobName);
        await blobClient.UploadAsync(new BinaryData(curatedData), overwrite: true);

        return $"{containerName}/{blobName}";
    }
}

public class DataIngestionRequest
{
    public byte[] Data { get; set; }
    public Dictionary<string, object> Metadata { get; set; }
    public string Source { get; set; }
}

public class DataIngestionResult
{
    public bool IsSuccess { get; set; }
    public string BronzePath { get; set; }
    public string SilverPath { get; set; }
    public string GoldPath { get; set; }
    public string[] Errors { get; set; }

    public static DataIngestionResult Success(string bronze, string silver, string gold) =>
        new DataIngestionResult { IsSuccess = true, BronzePath = bronze, SilverPath = silver, GoldPath = gold };

    public static DataIngestionResult Failed(string[] errors) =>
        new DataIngestionResult { IsSuccess = false, Errors = errors };
}

79. How would you design a reporting system?

Explanation: A reporting system provides insights through data visualization, dashboards, and scheduled reports. It includes data aggregation, report generation, distribution, and user access management.

Key Components: - Data Sources & ETL - Report Engine - Dashboard Framework - Scheduling & Distribution - User Management & Security

C# Implementation:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Linq;

public class ReportingSystem
{
    private readonly IDataSourceService _dataSource;
    private readonly IReportEngine _reportEngine;
    private readonly IDashboardService _dashboardService;
    private readonly ISchedulingService _schedulingService;
    private readonly IUserManagementService _userManagement;
    private readonly INotificationService _notificationService;

    public ReportingSystem(
        IDataSourceService dataSource,
        IReportEngine reportEngine,
        IDashboardService dashboardService,
        ISchedulingService schedulingService,
        IUserManagementService userManagement,
        INotificationService notificationService)
    {
        _dataSource = dataSource;
        _reportEngine = reportEngine;
        _dashboardService = dashboardService;
        _schedulingService = schedulingService;
        _userManagement = userManagement;
        _notificationService = notificationService;
    }

    public async Task<ReportResult> GenerateReportAsync(ReportRequest request)
    {
        try
        {
            // 1. Validate user permissions
            var userPermissions = await _userManagement.GetUserPermissionsAsync(request.UserId);
            if (!userPermissions.CanAccessReport(request.ReportType))
                return ReportResult.AccessDenied();

            // 2. Fetch data from sources
            var data = await _dataSource.GetDataAsync(request.DataSources, request.Filters);

            // 3. Apply business logic and aggregations
            var processedData = await ProcessDataAsync(data, request.Aggregations);

            // 4. Generate report
            var report = await _reportEngine.GenerateReportAsync(processedData, request.ReportType, request.Format);

            // 5. Store report metadata
            var reportMetadata = await StoreReportMetadataAsync(request, report);

            return ReportResult.Success(report, reportMetadata);
        }
        catch (Exception ex)
        {
            return ReportResult.Failed(ex.Message);
        }
    }

    public async Task<DashboardResult> CreateDashboardAsync(DashboardRequest request)
    {
        try
        {
            // 1. Validate user permissions
            var userPermissions = await _userManagement.GetUserPermissionsAsync(request.UserId);
            if (!userPermissions.CanCreateDashboard())
                return DashboardResult.AccessDenied();

            // 2. Create dashboard layout
            var dashboard = await _dashboardService.CreateDashboardAsync(request.Layout, request.Widgets);

            // 3. Configure data sources for widgets
            foreach (var widget in request.Widgets)
            {
                var data = await _dataSource.GetDataAsync(widget.DataSource, widget.Filters);
                await _dashboardService.ConfigureWidgetAsync(dashboard.Id, widget.Id, data);
            }

            // 4. Set up real-time updates if needed
            if (request.EnableRealTimeUpdates)
            {
                await _dashboardService.EnableRealTimeUpdatesAsync(dashboard.Id, request.UpdateInterval);
            }

            return DashboardResult.Success(dashboard);
        }
        catch (Exception ex)
        {
            return DashboardResult.Failed(ex.Message);
        }
    }

    public async Task<ScheduleResult> ScheduleReportAsync(ScheduleRequest request)
    {
        try
        {
            // 1. Validate schedule configuration
            if (!IsValidSchedule(request.Schedule))
                return ScheduleResult.InvalidSchedule();

            // 2. Create scheduled job
            var jobId = await _schedulingService.CreateScheduledJobAsync(new ScheduledJob
            {
                ReportRequest = request.ReportRequest,
                Schedule = request.Schedule,
                Recipients = request.Recipients,
                Format = request.Format
            });

            // 3. Set up notifications
            await _notificationService.SetupReportNotificationsAsync(jobId, request.Recipients);

            return ScheduleResult.Success(jobId);
        }
        catch (Exception ex)
        {
            return ScheduleResult.Failed(ex.Message);
        }
    }

    private async Task<ProcessedData> ProcessDataAsync(RawData data, List<Aggregation> aggregations)
    {
        var processedData = new ProcessedData();

        foreach (var aggregation in aggregations)
        {
            switch (aggregation.Type)
            {
                case AggregationType.Sum:
                    processedData.Results[aggregation.Field] = data.Records.Sum(r => r.GetValue(aggregation.Field));
                    break;
                case AggregationType.Average:
                    processedData.Results[aggregation.Field] = data.Records.Average(r => r.GetValue(aggregation.Field));
                    break;
                case AggregationType.Count:
                    processedData.Results[aggregation.Field] = data.Records.Count();
                    break;
                case AggregationType.GroupBy:
                    var grouped = data.Records.GroupBy(r => r.GetValue(aggregation.GroupByField))
                                             .ToDictionary(g => g.Key, g => g.Count());
                    processedData.Results[aggregation.Field] = grouped;
                    break;
            }
        }

        return processedData;
    }

    private async Task<ReportMetadata> StoreReportMetadataAsync(ReportRequest request, GeneratedReport report)
    {
        return new ReportMetadata
        {
            ReportId = Guid.NewGuid().ToString(),
            GeneratedAt = DateTime.UtcNow,
            ReportType = request.ReportType,
            DataSources = request.DataSources,
            Filters = request.Filters,
            GeneratedBy = request.UserId
        };
    }

    private bool IsValidSchedule(Schedule schedule)
    {
        // Implement schedule validation logic
        return schedule != null && schedule.Frequency != ScheduleFrequency.None;
    }
}

public class ReportRequest
{
    public string UserId { get; set; }
    public ReportType ReportType { get; set; }
    public List<string> DataSources { get; set; }
    public Dictionary<string, object> Filters { get; set; }
    public List<Aggregation> Aggregations { get; set; }
    public ReportFormat Format { get; set; }
}

public class DashboardRequest
{
    public string UserId { get; set; }
    public DashboardLayout Layout { get; set; }
    public List<Widget> Widgets { get; set; }
    public bool EnableRealTimeUpdates { get; set; }
    public TimeSpan UpdateInterval { get; set; }
}

public class ScheduleRequest
{
    public ReportRequest ReportRequest { get; set; }
    public Schedule Schedule { get; set; }
    public List<string> Recipients { get; set; }
    public ReportFormat Format { get; set; }
}

80. How do you implement data governance?

Explanation: Data governance ensures data quality, security, compliance, and proper usage across an organization. It includes policies, procedures, roles, and tools for managing data throughout its lifecycle.

Key Components: - Data Classification & Cataloging - Access Control & Security - Data Quality Management - Compliance & Audit - Data Lineage & Metadata

C# Implementation:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Linq;

public class DataGovernanceSystem
{
    private readonly IDataCatalogService _dataCatalog;
    private readonly IAccessControlService _accessControl;
    private readonly IDataQualityService _dataQuality;
    private readonly IComplianceService _compliance;
    private readonly IAuditService _auditService;
    private readonly IDataLineageService _dataLineage;

    public DataGovernanceSystem(
        IDataCatalogService dataCatalog,
        IAccessControlService accessControl,
        IDataQualityService dataQuality,
        IComplianceService compliance,
        IAuditService auditService,
        IDataLineageService dataLineage)
    {
        _dataCatalog = dataCatalog;
        _accessControl = accessControl;
        _dataQuality = dataQuality;
        _compliance = compliance;
        _auditService = auditService;
        _dataLineage = dataLineage;
    }

    public async Task<GovernanceResult> ApplyDataGovernanceAsync(DataGovernanceRequest request)
    {
        try
        {
            // 1. Classify and catalog data
            var classification = await ClassifyDataAsync(request.Data);
            var catalogEntry = await _dataCatalog.RegisterDataAsync(request.Data, classification);

            // 2. Apply access controls
            var accessResult = await _accessControl.ApplyAccessControlsAsync(catalogEntry, request.AccessPolicies);

            // 3. Validate data quality
            var qualityResult = await _dataQuality.ValidateDataQualityAsync(request.Data, request.QualityRules);

            // 4. Check compliance
            var complianceResult = await _compliance.CheckComplianceAsync(request.Data, request.ComplianceRules);

            // 5. Track data lineage
            await _dataLineage.TrackLineageAsync(catalogEntry, request.Source, request.Transformations);

            // 6. Audit the governance process
            await _auditService.LogGovernanceEventAsync(new GovernanceAuditEvent
            {
                DataId = catalogEntry.Id,
                UserId = request.UserId,
                Action = "DataGovernanceApplied",
                Timestamp = DateTime.UtcNow,
                Results = new { accessResult, qualityResult, complianceResult }
            });

            return GovernanceResult.Success(catalogEntry.Id, qualityResult, complianceResult);
        }
        catch (Exception ex)
        {
            return GovernanceResult.Failed(ex.Message);
        }
    }

    public async Task<AccessResult> RequestDataAccessAsync(DataAccessRequest request)
    {
        try
        {
            // 1. Validate user identity
            var userIdentity = await _accessControl.ValidateUserIdentityAsync(request.UserId);

            // 2. Check user permissions
            var permissions = await _accessControl.GetUserPermissionsAsync(request.UserId, request.DataId);

            // 3. Apply role-based access control
            var accessGranted = await _accessControl.ApplyRBACAsync(userIdentity, permissions, request.DataId);

            if (accessGranted)
            {
                // 4. Log access event
                await _auditService.LogAccessEventAsync(new AccessAuditEvent
                {
                    UserId = request.UserId,
                    DataId = request.DataId,
                    AccessType = request.AccessType,
                    Timestamp = DateTime.UtcNow,
                    Purpose = request.Purpose
                });

                return AccessResult.Granted(request.DataId);
            }

            return AccessResult.Denied("Insufficient permissions");
        }
        catch (Exception ex)
        {
            return AccessResult.Denied(ex.Message);
        }
    }

    public async Task<QualityResult> MonitorDataQualityAsync(string dataId)
    {
        try
        {
            // 1. Get data quality rules
            var qualityRules = await _dataQuality.GetQualityRulesAsync(dataId);

            // 2. Fetch current data
            var data = await _dataCatalog.GetDataAsync(dataId);

            // 3. Run quality checks
            var qualityChecks = new List<QualityCheck>();

            foreach (var rule in qualityRules)
            {
                var check = await _dataQuality.RunQualityCheckAsync(data, rule);
                qualityChecks.Add(check);
            }

            // 4. Generate quality report
            var qualityReport = new DataQualityReport
            {
                DataId = dataId,
                Timestamp = DateTime.UtcNow,
                Checks = qualityChecks,
                OverallScore = CalculateOverallScore(qualityChecks)
            };

            // 5. Store quality report
            await _dataQuality.StoreQualityReportAsync(qualityReport);

            return QualityResult.Success(qualityReport);
        }
        catch (Exception ex)
        {
            return QualityResult.Failed(ex.Message);
        }
    }

    public async Task<ComplianceResult> AuditComplianceAsync(ComplianceAuditRequest request)
    {
        try
        {
            // 1. Get compliance requirements
            var requirements = await _compliance.GetComplianceRequirementsAsync(request.Regulations);

            // 2. Audit data against requirements
            var auditResults = new List<ComplianceAuditResult>();

            foreach (var requirement in requirements)
            {
                var auditResult = await _compliance.AuditRequirementAsync(request.DataId, requirement);
                auditResults.Add(auditResult);
            }

            // 3. Generate compliance report
            var complianceReport = new ComplianceReport
            {
                DataId = request.DataId,
                Regulations = request.Regulations,
                AuditResults = auditResults,
                OverallCompliance = CalculateComplianceScore(auditResults),
                Timestamp = DateTime.UtcNow
            };

            // 4. Store compliance report
            await _compliance.StoreComplianceReportAsync(complianceReport);

            return ComplianceResult.Success(complianceReport);
        }
        catch (Exception ex)
        {
            return ComplianceResult.Failed(ex.Message);
        }
    }

    private async Task<DataClassification> ClassifyDataAsync(byte[] data)
    {
        // Implement data classification logic
        return new DataClassification
        {
            Sensitivity = SensitivityLevel.Public,
            DataType = DataType.Structured,
            RetentionPeriod = TimeSpan.FromDays(365),
            EncryptionRequired = false
        };
    }

    private double CalculateOverallScore(List<QualityCheck> checks)
    {
        if (!checks.Any()) return 0;
        return checks.Average(c => c.Score);
    }

    private double CalculateComplianceScore(List<ComplianceAuditResult> results)
    {
        if (!results.Any()) return 0;
        return results.Count(r => r.IsCompliant) / (double)results.Count * 100;
    }
}

public class DataGovernanceRequest
{
    public byte[] Data { get; set; }
    public string UserId { get; set; }
    public string Source { get; set; }
    public List<string> Transformations { get; set; }
    public List<AccessPolicy> AccessPolicies { get; set; }
    public List<QualityRule> QualityRules { get; set; }
    public List<ComplianceRule> ComplianceRules { get; set; }
}

public class DataAccessRequest
{
    public string UserId { get; set; }
    public string DataId { get; set; }
    public AccessType AccessType { get; set; }
    public string Purpose { get; set; }
}

public class ComplianceAuditRequest
{
    public string DataId { get; set; }
    public List<string> Regulations { get; set; }
}

Cloud & Infrastructure Architecture

81. How would you design a multi-cloud architecture?

Explanation: Multi-cloud architecture distributes applications across multiple cloud providers to avoid vendor lock-in, improve reliability, and optimize costs. It includes cloud abstraction layers, load balancing, and data synchronization.

Key Components: - Cloud Abstraction Layer - Load Balancing & Traffic Management - Data Synchronization - Monitoring & Observability - Disaster Recovery

C# Implementation:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Linq;

public class MultiCloudArchitecture
{
    private readonly ICloudProviderService _awsService;
    private readonly ICloudProviderService _azureService;
    private readonly ICloudProviderService _gcpService;
    private readonly ILoadBalancer _loadBalancer;
    private readonly IDataSynchronizationService _dataSync;
    private readonly IMonitoringService _monitoring;

    public MultiCloudArchitecture(
        ICloudProviderService awsService,
        ICloudProviderService azureService,
        ICloudProviderService gcpService,
        ILoadBalancer loadBalancer,
        IDataSynchronizationService dataSync,
        IMonitoringService monitoring)
    {
        _awsService = awsService;
        _azureService = azureService;
        _gcpService = gcpService;
        _loadBalancer = loadBalancer;
        _dataSync = dataSync;
        _monitoring = monitoring;
    }

    public async Task<DeploymentResult> DeployMultiCloudAsync(MultiCloudDeploymentRequest request)
    {
        try
        {
            var deploymentResults = new List<CloudDeploymentResult>();

            // 1. Deploy to AWS
            if (request.Clouds.Contains(CloudProvider.AWS))
            {
                var awsResult = await _awsService.DeployAsync(request.Application, request.AWSConfig);
                deploymentResults.Add(new CloudDeploymentResult { Provider = CloudProvider.AWS, Result = awsResult });
            }

            // 2. Deploy to Azure
            if (request.Clouds.Contains(CloudProvider.Azure))
            {
                var azureResult = await _azureService.DeployAsync(request.Application, request.AzureConfig);
                deploymentResults.Add(new CloudDeploymentResult { Provider = CloudProvider.Azure, Result = azureResult });
            }

            // 3. Deploy to GCP
            if (request.Clouds.Contains(CloudProvider.GCP))
            {
                var gcpResult = await _gcpService.DeployAsync(request.Application, request.GCPConfig);
                deploymentResults.Add(new CloudDeploymentResult { Provider = CloudProvider.GCP, Result = gcpResult });
            }

            // 4. Configure load balancer
            var loadBalancerConfig = new LoadBalancerConfig
            {
                Endpoints = deploymentResults.Select(r => r.Result.Endpoint).ToList(),
                Strategy = request.LoadBalancingStrategy,
                HealthCheckInterval = TimeSpan.FromSeconds(30)
            };

            await _loadBalancer.ConfigureAsync(loadBalancerConfig);

            // 5. Set up data synchronization
            if (request.EnableDataSync)
            {
                await _dataSync.SetupSynchronizationAsync(deploymentResults.Select(r => r.Result.DatabaseEndpoint).ToList());
            }

            // 6. Configure monitoring
            await _monitoring.SetupMultiCloudMonitoringAsync(deploymentResults);

            return DeploymentResult.Success(deploymentResults);
        }
        catch (Exception ex)
        {
            return DeploymentResult.Failed(ex.Message);
        }
    }

    public async Task<RoutingResult> RouteRequestAsync(ServiceRequest request)
    {
        try
        {
            // 1. Get available cloud providers
            var availableProviders = await GetAvailableProvidersAsync();

            // 2. Apply routing strategy
            var selectedProvider = await _loadBalancer.SelectProviderAsync(availableProviders, request);

            // 3. Route request to selected provider
            var response = await RouteToProviderAsync(selectedProvider, request);

            // 4. Monitor performance
            await _monitoring.LogRequestMetricsAsync(selectedProvider, request, response);

            return RoutingResult.Success(response, selectedProvider);
        }
        catch (Exception ex)
        {
            return RoutingResult.Failed(ex.Message);
        }
    }

    public async Task<FailoverResult> PerformFailoverAsync(CloudProvider failedProvider)
    {
        try
        {
            // 1. Detect failure
            var failureDetection = await _monitoring.DetectFailureAsync(failedProvider);

            // 2. Select failover target
            var failoverTarget = await SelectFailoverTargetAsync(failedProvider);

            // 3. Update load balancer configuration
            await _loadBalancer.UpdateConfigurationAsync(failedProvider, failoverTarget);

            // 4. Synchronize data if needed
            await _dataSync.SynchronizeDataAsync(failedProvider, failoverTarget);

            // 5. Update DNS/routing
            await UpdateRoutingAsync(failedProvider, failoverTarget);

            return FailoverResult.Success(failoverTarget);
        }
        catch (Exception ex)
        {
            return FailoverResult.Failed(ex.Message);
        }
    }

    private async Task<List<CloudProvider>> GetAvailableProvidersAsync()
    {
        var providers = new List<CloudProvider>();

        if (await _awsService.IsHealthyAsync()) providers.Add(CloudProvider.AWS);
        if (await _azureService.IsHealthyAsync()) providers.Add(CloudProvider.Azure);
        if (await _gcpService.IsHealthyAsync()) providers.Add(CloudProvider.GCP);

        return providers;
    }

    private async Task<CloudProvider> SelectFailoverTargetAsync(CloudProvider failedProvider)
    {
        var availableProviders = await GetAvailableProvidersAsync();
        return availableProviders.FirstOrDefault(p => p != failedProvider);
    }

    private async Task<ServiceResponse> RouteToProviderAsync(CloudProvider provider, ServiceRequest request)
    {
        switch (provider)
        {
            case CloudProvider.AWS:
                return await _awsService.ProcessRequestAsync(request);
            case CloudProvider.Azure:
                return await _azureService.ProcessRequestAsync(request);
            case CloudProvider.GCP:
                return await _gcpService.ProcessRequestAsync(request);
            default:
                throw new ArgumentException($"Unsupported cloud provider: {provider}");
        }
    }

    private async Task UpdateRoutingAsync(CloudProvider failedProvider, CloudProvider targetProvider)
    {
        // Implement DNS/routing update logic
        await Task.CompletedTask;
    }
}

public class MultiCloudDeploymentRequest
{
    public Application Application { get; set; }
    public List<CloudProvider> Clouds { get; set; }
    public AWSConfig AWSConfig { get; set; }
    public AzureConfig AzureConfig { get; set; }
    public GCPConfig GCPConfig { get; set; }
    public LoadBalancingStrategy LoadBalancingStrategy { get; set; }
    public bool EnableDataSync { get; set; }
}

public class ServiceRequest
{
    public string ServiceName { get; set; }
    public string Operation { get; set; }
    public Dictionary<string, object> Parameters { get; set; }
    public string UserId { get; set; }
}

82. How do you implement infrastructure as code?

Explanation: Infrastructure as Code (IaC) manages and provisions infrastructure through code rather than manual processes. It ensures consistency, version control, and automated deployment of infrastructure resources.

Key Components: - Resource Definitions - Configuration Management - Deployment Automation - State Management - Testing & Validation

C# Implementation:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Text.Json;

public class InfrastructureAsCode
{
    private readonly IResourceProvider _resourceProvider;
    private readonly IConfigurationManager _configManager;
    private readonly IDeploymentEngine _deploymentEngine;
    private readonly IStateManager _stateManager;
    private readonly IValidationService _validationService;

    public InfrastructureAsCode(
        IResourceProvider resourceProvider,
        IConfigurationManager configManager,
        IDeploymentEngine deploymentEngine,
        IStateManager stateManager,
        IValidationService validationService)
    {
        _resourceProvider = resourceProvider;
        _configManager = configManager;
        _deploymentEngine = deploymentEngine;
        _stateManager = stateManager;
        _validationService = validationService;
    }

    public async Task<DeploymentResult> DeployInfrastructureAsync(InfrastructureDefinition definition)
    {
        try
        {
            // 1. Validate infrastructure definition
            var validationResult = await _validationService.ValidateDefinitionAsync(definition);
            if (!validationResult.IsValid)
                return DeploymentResult.ValidationFailed(validationResult.Errors);

            // 2. Load current state
            var currentState = await _stateManager.GetCurrentStateAsync(definition.Environment);

            // 3. Calculate required changes
            var changes = await CalculateChangesAsync(definition, currentState);

            // 4. Execute deployment plan
            var deploymentPlan = await CreateDeploymentPlanAsync(changes);
            var deploymentResult = await _deploymentEngine.ExecutePlanAsync(deploymentPlan);

            // 5. Update state
            await _stateManager.UpdateStateAsync(definition.Environment, deploymentResult.NewState);

            return DeploymentResult.Success(deploymentResult);
        }
        catch (Exception ex)
        {
            return DeploymentResult.Failed(ex.Message);
        }
    }

    public async Task<InfrastructureDefinition> CreateInfrastructureDefinitionAsync(InfrastructureRequest request)
    {
        var definition = new InfrastructureDefinition
        {
            Environment = request.Environment,
            Resources = new List<IResource>()
        };

        // Add Virtual Network
        definition.Resources.Add(new VirtualNetwork
        {
            Name = $"{request.Environment}-vnet",
            AddressSpace = "10.0.0.0/16",
            Subnets = new List<Subnet>
            {
                new Subnet { Name = "default", AddressPrefix = "10.0.1.0/24" },
                new Subnet { Name = "database", AddressPrefix = "10.0.2.0/24" }
            }
        });

        // Add Application Gateway
        definition.Resources.Add(new ApplicationGateway
        {
            Name = $"{request.Environment}-appgw",
            Sku = "Standard_v2",
            Capacity = 2,
            VirtualNetwork = $"{request.Environment}-vnet",
            Subnet = "default"
        });

        // Add Virtual Machine Scale Set
        definition.Resources.Add(new VirtualMachineScaleSet
        {
            Name = $"{request.Environment}-vmss",
            Sku = "Standard_D2s_v3",
            Capacity = request.VmCount,
            VirtualNetwork = $"{request.Environment}-vnet",
            Subnet = "default",
            ImageReference = new ImageReference
            {
                Publisher = "MicrosoftWindowsServer",
                Offer = "WindowsServer",
                Sku = "2019-Datacenter",
                Version = "latest"
            }
        });

        // Add Database
        definition.Resources.Add(new Database
        {
            Name = $"{request.Environment}-db",
            Sku = "Standard",
            Capacity = request.DatabaseCapacity,
            VirtualNetwork = $"{request.Environment}-vnet",
            Subnet = "database"
        });

        // Add Monitoring
        definition.Resources.Add(new MonitoringWorkspace
        {
            Name = $"{request.Environment}-monitoring",
            RetentionDays = 30
        });

        return definition;
    }

    public async Task<ValidationResult> ValidateInfrastructureAsync(InfrastructureDefinition definition)
    {
        var errors = new List<string>();

        // Validate resource dependencies
        foreach (var resource in definition.Resources)
        {
            var dependencies = resource.GetDependencies();
            foreach (var dependency in dependencies)
            {
                if (!definition.Resources.Any(r => r.Name == dependency))
                {
                    errors.Add($"Resource {resource.Name} depends on {dependency} which is not defined");
                }
            }
        }

        // Validate naming conventions
        foreach (var resource in definition.Resources)
        {
            if (!IsValidName(resource.Name, definition.Environment))
            {
                errors.Add($"Resource name {resource.Name} does not follow naming conventions");
            }
        }

        // Validate resource limits
        var totalVMs = definition.Resources.OfType<VirtualMachineScaleSet>().Sum(vmss => vmss.Capacity);
        if (totalVMs > 100)
        {
            errors.Add("Total VM capacity exceeds limit of 100");
        }

        return new ValidationResult
        {
            IsValid = !errors.Any(),
            Errors = errors
        };
    }

    public async Task<StateComparison> CompareStatesAsync(string environment)
    {
        var currentState = await _stateManager.GetCurrentStateAsync(environment);
        var desiredState = await _configManager.GetDesiredStateAsync(environment);

        return new StateComparison
        {
            CurrentState = currentState,
            DesiredState = desiredState,
            Differences = await CalculateDifferencesAsync(currentState, desiredState)
        };
    }

    private async Task<List<InfrastructureChange>> CalculateChangesAsync(
        InfrastructureDefinition definition, 
        InfrastructureState currentState)
    {
        var changes = new List<InfrastructureChange>();

        foreach (var resource in definition.Resources)
        {
            var existingResource = currentState.Resources.FirstOrDefault(r => r.Name == resource.Name);

            if (existingResource == null)
            {
                changes.Add(new InfrastructureChange
                {
                    Type = ChangeType.Create,
                    Resource = resource
                });
            }
            else if (!resource.Equals(existingResource))
            {
                changes.Add(new InfrastructureChange
                {
                    Type = ChangeType.Update,
                    Resource = resource,
                    PreviousResource = existingResource
                });
            }
        }

        // Check for resources to delete
        foreach (var existingResource in currentState.Resources)
        {
            if (!definition.Resources.Any(r => r.Name == existingResource.Name))
            {
                changes.Add(new InfrastructureChange
                {
                    Type = ChangeType.Delete,
                    Resource = existingResource
                });
            }
        }

        return changes;
    }

    private async Task<DeploymentPlan> CreateDeploymentPlanAsync(List<InfrastructureChange> changes)
    {
        var plan = new DeploymentPlan();

        // Sort changes by dependencies
        var sortedChanges = await SortByDependenciesAsync(changes);

        foreach (var change in sortedChanges)
        {
            plan.Steps.Add(new DeploymentStep
            {
                Change = change,
                EstimatedDuration = EstimateDuration(change),
                RollbackPlan = CreateRollbackPlan(change)
            });
        }

        return plan;
    }

    private bool IsValidName(string name, string environment)
    {
        return name.StartsWith($"{environment}-") && name.Length <= 63;
    }

    private TimeSpan EstimateDuration(InfrastructureChange change)
    {
        return change.Type switch
        {
            ChangeType.Create => TimeSpan.FromMinutes(5),
            ChangeType.Update => TimeSpan.FromMinutes(3),
            ChangeType.Delete => TimeSpan.FromMinutes(2),
            _ => TimeSpan.FromMinutes(1)
        };
    }
}

public class InfrastructureDefinition
{
    public string Environment { get; set; }
    public List<IResource> Resources { get; set; }
    public Dictionary<string, object> Variables { get; set; }
    public List<string> Tags { get; set; }
}

public interface IResource
{
    string Name { get; set; }
    string Type { get; }
    List<string> GetDependencies();
}

public class VirtualNetwork : IResource
{
    public string Name { get; set; }
    public string Type => "Microsoft.Network/virtualNetworks";
    public string AddressSpace { get; set; }
    public List<Subnet> Subnets { get; set; }

    public List<string> GetDependencies() => new List<string>();
}

public class ApplicationGateway : IResource
{
    public string Name { get; set; }
    public string Type => "Microsoft.Network/applicationGateways";
    public string Sku { get; set; }
    public int Capacity { get; set; }
    public string VirtualNetwork { get; set; }
    public string Subnet { get; set; }

    public List<string> GetDependencies() => new List<string> { VirtualNetwork };
}

83. How would you design a container orchestration system?

Explanation: A container orchestration system manages the deployment, scaling, and operation of containerized applications across multiple hosts. Key components include:

• Scheduler: Determines where containers run based on resource availability and constraints
• Service Discovery: Enables containers to find and communicate with each other
• Load Balancing: Distributes traffic across multiple container instances
• Health Monitoring: Tracks container health and automatically restarts failed containers
• Resource Management: Allocates CPU, memory, and storage resources

C# Implementation Example:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;

namespace ContainerOrchestration
{
    public class Container
    {
        public string Id { get; set; }
        public string Image { get; set; }
        public Dictionary<string, string> EnvironmentVariables { get; set; }
        public ResourceRequirements Resources { get; set; }
        public ContainerStatus Status { get; set; }
    }

    public class ResourceRequirements
    {
        public double CpuRequest { get; set; }
        public double CpuLimit { get; set; }
        public long MemoryRequest { get; set; }
        public long MemoryLimit { get; set; }
    }

    public enum ContainerStatus
    {
        Pending,
        Running,
        Failed,
        Stopped
    }

    public class Node
    {
        public string Id { get; set; }
        public string Hostname { get; set; }
        public ResourceCapacity Capacity { get; set; }
        public List<Container> RunningContainers { get; set; } = new List<Container>();
    }

    public class ResourceCapacity
    {
        public double Cpu { get; set; }
        public long Memory { get; set; }
        public long Storage { get; set; }
    }

    public class ContainerScheduler
    {
        private readonly List<Node> _nodes;
        private readonly ILoadBalancer _loadBalancer;
        private readonly IHealthMonitor _healthMonitor;

        public ContainerScheduler(List<Node> nodes, ILoadBalancer loadBalancer, IHealthMonitor healthMonitor)
        {
            _nodes = nodes;
            _loadBalancer = loadBalancer;
            _healthMonitor = healthMonitor;
        }

        public async Task<Node> ScheduleContainerAsync(Container container)
        {
            // Find the best node based on resource availability and constraints
            var bestNode = FindBestNode(container);

            if (bestNode == null)
            {
                throw new InvalidOperationException("No suitable node found for container");
            }

            // Deploy container to the selected node
            await DeployContainerAsync(container, bestNode);

            // Register with load balancer
            await _loadBalancer.RegisterContainerAsync(container);

            // Start health monitoring
            await _healthMonitor.StartMonitoringAsync(container);

            return bestNode;
        }

        private Node FindBestNode(Container container)
        {
            Node bestNode = null;
            double bestScore = double.MinValue;

            foreach (var node in _nodes)
            {
                if (CanNodeHostContainer(node, container))
                {
                    var score = CalculateNodeScore(node, container);
                    if (score > bestScore)
                    {
                        bestScore = score;
                        bestNode = node;
                    }
                }
            }

            return bestNode;
        }

        private bool CanNodeHostContainer(Node node, Container container)
        {
            var availableCpu = node.Capacity.Cpu - node.RunningContainers.Sum(c => c.Resources.CpuRequest);
            var availableMemory = node.Capacity.Memory - node.RunningContainers.Sum(c => c.Resources.MemoryRequest);

            return availableCpu >= container.Resources.CpuRequest && 
                   availableMemory >= container.Resources.MemoryRequest;
        }

        private double CalculateNodeScore(Node node, Container container)
        {
            // Simple scoring based on resource utilization
            var cpuUtilization = node.RunningContainers.Sum(c => c.Resources.CpuRequest) / node.Capacity.Cpu;
            var memoryUtilization = node.RunningContainers.Sum(c => c.Resources.MemoryRequest) / node.Capacity.Memory;

            // Prefer nodes with lower utilization
            return 1.0 - (cpuUtilization + memoryUtilization) / 2.0;
        }

        private async Task DeployContainerAsync(Container container, Node node)
        {
            // Simulate container deployment
            container.Status = ContainerStatus.Running;
            node.RunningContainers.Add(container);

            await Task.Delay(1000); // Simulate deployment time
        }
    }

    public interface ILoadBalancer
    {
        Task RegisterContainerAsync(Container container);
        Task<Container> GetNextContainerAsync(string serviceName);
    }

    public interface IHealthMonitor
    {
        Task StartMonitoringAsync(Container container);
        Task<bool> IsHealthyAsync(Container container);
    }
}

84. How do you implement blue-green deployment?

Explanation: Blue-green deployment involves maintaining two identical production environments (blue and green). One environment serves live traffic while the other is used for deployment and testing. After successful deployment and testing, traffic is switched from the old environment to the new one.

C# Implementation Example:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;

namespace BlueGreenDeployment
{
    public class Environment
    {
        public string Name { get; set; }
        public string Url { get; set; }
        public EnvironmentStatus Status { get; set; }
        public List<Container> Containers { get; set; } = new List<Container>();
        public DateTime LastDeployment { get; set; }
    }

    public enum EnvironmentStatus
    {
        Active,
        Inactive,
        Deploying,
        Testing
    }

    public class BlueGreenDeploymentManager
    {
        private readonly ILoadBalancer _loadBalancer;
        private readonly IHealthChecker _healthChecker;
        private readonly IDeploymentService _deploymentService;
        private Environment _blueEnvironment;
        private Environment _greenEnvironment;

        public BlueGreenDeploymentManager(
            ILoadBalancer loadBalancer,
            IHealthChecker healthChecker,
            IDeploymentService deploymentService)
        {
            _loadBalancer = loadBalancer;
            _healthChecker = healthChecker;
            _deploymentService = deploymentService;

            InitializeEnvironments();
        }

        private void InitializeEnvironments()
        {
            _blueEnvironment = new Environment
            {
                Name = "Blue",
                Url = "https://blue.example.com",
                Status = EnvironmentStatus.Active
            };

            _greenEnvironment = new Environment
            {
                Name = "Green",
                Url = "https://green.example.com",
                Status = EnvironmentStatus.Inactive
            };
        }

        public async Task DeployAsync(DeploymentPackage package)
        {
            try
            {
                // Determine which environment is currently inactive
                var targetEnvironment = _blueEnvironment.Status == EnvironmentStatus.Active 
                    ? _greenEnvironment 
                    : _blueEnvironment;

                Console.WriteLine($"Starting deployment to {targetEnvironment.Name} environment");

                // Deploy to inactive environment
                targetEnvironment.Status = EnvironmentStatus.Deploying;
                await _deploymentService.DeployAsync(targetEnvironment, package);
                targetEnvironment.LastDeployment = DateTime.UtcNow;

                // Run health checks
                targetEnvironment.Status = EnvironmentStatus.Testing;
                var isHealthy = await _healthChecker.RunHealthChecksAsync(targetEnvironment);

                if (!isHealthy)
                {
                    throw new DeploymentException($"Health checks failed for {targetEnvironment.Name} environment");
                }

                // Switch traffic
                await SwitchTrafficAsync(targetEnvironment);

                Console.WriteLine($"Successfully switched traffic to {targetEnvironment.Name} environment");

            }
            catch (Exception ex)
            {
                Console.WriteLine($"Deployment failed: {ex.Message}");
                await RollbackAsync();
                throw;
            }
        }

        private async Task SwitchTrafficAsync(Environment newActiveEnvironment)
        {
            // Update load balancer configuration
            await _loadBalancer.UpdateRoutingAsync(newActiveEnvironment.Url);

            // Update environment statuses
            if (newActiveEnvironment.Name == "Blue")
            {
                _blueEnvironment.Status = EnvironmentStatus.Active;
                _greenEnvironment.Status = EnvironmentStatus.Inactive;
            }
            else
            {
                _greenEnvironment.Status = EnvironmentStatus.Active;
                _blueEnvironment.Status = EnvironmentStatus.Inactive;
            }

            // Wait for traffic to stabilize
            await Task.Delay(TimeSpan.FromSeconds(30));
        }

        private async Task RollbackAsync()
        {
            Console.WriteLine("Rolling back deployment...");

            // Keep current active environment as active
            // Clean up failed deployment
            var inactiveEnvironment = _blueEnvironment.Status == EnvironmentStatus.Active 
                ? _greenEnvironment 
                : _blueEnvironment;

            inactiveEnvironment.Status = EnvironmentStatus.Inactive;
            await _deploymentService.CleanupAsync(inactiveEnvironment);
        }

        public async Task<DeploymentStatus> GetDeploymentStatusAsync()
        {
            return new DeploymentStatus
            {
                ActiveEnvironment = _blueEnvironment.Status == EnvironmentStatus.Active ? "Blue" : "Green",
                BlueEnvironmentStatus = _blueEnvironment.Status,
                GreenEnvironmentStatus = _greenEnvironment.Status,
                LastDeployment = _blueEnvironment.LastDeployment > _greenEnvironment.LastDeployment 
                    ? _blueEnvironment.LastDeployment 
                    : _greenEnvironment.LastDeployment
            };
        }
    }

    public class DeploymentPackage
    {
        public string Version { get; set; }
        public string ImageTag { get; set; }
        public Dictionary<string, string> Configuration { get; set; }
    }

    public class DeploymentStatus
    {
        public string ActiveEnvironment { get; set; }
        public EnvironmentStatus BlueEnvironmentStatus { get; set; }
        public EnvironmentStatus GreenEnvironmentStatus { get; set; }
        public DateTime LastDeployment { get; set; }
    }

    public interface ILoadBalancer
    {
        Task UpdateRoutingAsync(string targetUrl);
    }

    public interface IHealthChecker
    {
        Task<bool> RunHealthChecksAsync(Environment environment);
    }

    public interface IDeploymentService
    {
        Task DeployAsync(Environment environment, DeploymentPackage package);
        Task CleanupAsync(Environment environment);
    }

    public class DeploymentException : Exception
    {
        public DeploymentException(string message) : base(message) { }
    }
}

85. How would you design a serverless architecture?

Explanation: Serverless architecture eliminates the need to manage servers by abstracting infrastructure management. Key components include:

• Function-as-a-Service (FaaS): Event-driven functions that scale automatically
• Backend-as-a-Service (BaaS): Managed services for databases, authentication, etc.
• Event Sources: Triggers that invoke functions (HTTP requests, database changes, etc.)
• API Gateway: Manages HTTP endpoints and routing
• Message Queues: Handles asynchronous processing

C# Implementation Example:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Text.Json;

namespace ServerlessArchitecture
{
    public class ServerlessFunction
    {
        public string Id { get; set; }
        public string Name { get; set; }
        public string Runtime { get; set; }
        public string Handler { get; set; }
        public Dictionary<string, string> EnvironmentVariables { get; set; }
        public int Timeout { get; set; }
        public int MemorySize { get; set; }
        public List<string> EventSources { get; set; } = new List<string>();
    }

    public class EventSource
    {
        public string Id { get; set; }
        public string Type { get; set; } // HTTP, SQS, DynamoDB, etc.
        public string Configuration { get; set; }
        public string FunctionId { get; set; }
    }

    public class FunctionInvocation
    {
        public string FunctionId { get; set; }
        public object Payload { get; set; }
        public Dictionary<string, string> Headers { get; set; }
        public DateTime InvocationTime { get; set; }
    }

    public class ServerlessPlatform
    {
        private readonly Dictionary<string, ServerlessFunction> _functions;
        private readonly Dictionary<string, EventSource> _eventSources;
        private readonly IExecutionEngine _executionEngine;
        private readonly IScalingManager _scalingManager;
        private readonly IMonitoringService _monitoringService;

        public ServerlessPlatform(
            IExecutionEngine executionEngine,
            IScalingManager scalingManager,
            IMonitoringService monitoringService)
        {
            _functions = new Dictionary<string, ServerlessFunction>();
            _eventSources = new Dictionary<string, EventSource>();
            _executionEngine = executionEngine;
            _scalingManager = scalingManager;
            _monitoringService = monitoringService;
        }

        public async Task<string> DeployFunctionAsync(ServerlessFunction function)
        {
            function.Id = Guid.NewGuid().ToString();
            _functions[function.Id] = function;

            // Register function with execution engine
            await _executionEngine.RegisterFunctionAsync(function);

            // Set up auto-scaling
            await _scalingManager.ConfigureScalingAsync(function);

            return function.Id;
        }

        public async Task<string> CreateEventSourceAsync(EventSource eventSource)
        {
            eventSource.Id = Guid.NewGuid().ToString();
            _eventSources[eventSource.Id] = eventSource;

            // Register event source
            await RegisterEventSourceAsync(eventSource);

            return eventSource.Id;
        }

        public async Task<object> InvokeFunctionAsync(string functionId, object payload)
        {
            if (!_functions.ContainsKey(functionId))
            {
                throw new ArgumentException($"Function {functionId} not found");
            }

            var function = _functions[functionId];
            var invocation = new FunctionInvocation
            {
                FunctionId = functionId,
                Payload = payload,
                InvocationTime = DateTime.UtcNow
            };

            // Check if we need to scale up
            await _scalingManager.CheckAndScaleAsync(function);

            // Execute function
            var result = await _executionEngine.ExecuteFunctionAsync(invocation);

            // Record metrics
            await _monitoringService.RecordInvocationAsync(invocation, result);

            return result;
        }

        private async Task RegisterEventSourceAsync(EventSource eventSource)
        {
            switch (eventSource.Type.ToLower())
            {
                case "http":
                    await RegisterHttpEventSourceAsync(eventSource);
                    break;
                case "sqs":
                    await RegisterSQSEventSourceAsync(eventSource);
                    break;
                case "dynamodb":
                    await RegisterDynamoDBEventSourceAsync(eventSource);
                    break;
                default:
                    throw new NotSupportedException($"Event source type {eventSource.Type} not supported");
            }
        }

        private async Task RegisterHttpEventSourceAsync(EventSource eventSource)
        {
            // Register HTTP endpoint
            await Task.CompletedTask;
        }

        private async Task RegisterSQSEventSourceAsync(EventSource eventSource)
        {
            // Set up SQS polling
            await Task.CompletedTask;
        }

        private async Task RegisterDynamoDBEventSourceAsync(EventSource eventSource)
        {
            // Set up DynamoDB streams
            await Task.CompletedTask;
        }
    }

    public class APIGateway
    {
        private readonly ServerlessPlatform _platform;
        private readonly Dictionary<string, string> _routes; // path -> functionId

        public APIGateway(ServerlessPlatform platform)
        {
            _platform = platform;
            _routes = new Dictionary<string, string>();
        }

        public void AddRoute(string path, string functionId)
        {
            _routes[path] = functionId;
        }

        public async Task<object> HandleRequestAsync(string path, object payload, Dictionary<string, string> headers)
        {
            if (!_routes.ContainsKey(path))
            {
                throw new ArgumentException($"No route found for path: {path}");
            }

            var functionId = _routes[path];
            return await _platform.InvokeFunctionAsync(functionId, payload);
        }
    }

    public interface IExecutionEngine
    {
        Task RegisterFunctionAsync(ServerlessFunction function);
        Task<object> ExecuteFunctionAsync(FunctionInvocation invocation);
    }

    public interface IScalingManager
    {
        Task ConfigureScalingAsync(ServerlessFunction function);
        Task CheckAndScaleAsync(ServerlessFunction function);
    }

    public interface IMonitoringService
    {
        Task RecordInvocationAsync(FunctionInvocation invocation, object result);
    }

    // Example usage
    public class ServerlessExample
    {
        public async Task RunExampleAsync()
        {
            var platform = new ServerlessPlatform(
                new MockExecutionEngine(),
                new MockScalingManager(),
                new MockMonitoringService()
            );

            // Deploy a function
            var function = new ServerlessFunction
            {
                Name = "UserProcessor",
                Runtime = "dotnet6",
                Handler = "UserProcessor::Process",
                Timeout = 30,
                MemorySize = 512
            };

            var functionId = await platform.DeployFunctionAsync(function);

            // Create HTTP event source
            var eventSource = new EventSource
            {
                Type = "HTTP",
                Configuration = "/api/users",
                FunctionId = functionId
            };

            await platform.CreateEventSourceAsync(eventSource);

            // Set up API Gateway
            var apiGateway = new APIGateway(platform);
            apiGateway.AddRoute("/api/users", functionId);

            // Handle request
            var result = await apiGateway.HandleRequestAsync("/api/users", new { userId = "123" }, new Dictionary<string, string>());
        }
    }

    // Mock implementations
    public class MockExecutionEngine : IExecutionEngine
    {
        public Task RegisterFunctionAsync(ServerlessFunction function) => Task.CompletedTask;
        public Task<object> ExecuteFunctionAsync(FunctionInvocation invocation) => Task.FromResult<object>("Success");
    }

    public class MockScalingManager : IScalingManager
    {
        public Task ConfigureScalingAsync(ServerlessFunction function) => Task.CompletedTask;
        public Task CheckAndScaleAsync(ServerlessFunction function) => Task.CompletedTask;
    }

    public class MockMonitoringService : IMonitoringService
    {
        public Task RecordInvocationAsync(FunctionInvocation invocation, object result) => Task.CompletedTask;
    }
}

86. How do you implement disaster recovery?

Explanation: Disaster recovery ensures business continuity by implementing strategies to recover from catastrophic failures. Key components include:

• Backup Strategy: Regular data backups with different retention policies
• Recovery Point Objective (RPO): Maximum acceptable data loss
• Recovery Time Objective (RTO): Maximum acceptable downtime
• Failover Mechanisms: Automatic or manual switching to backup systems
• Data Replication: Real-time or near-real-time data synchronization

C# Implementation Example:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Text.Json;

namespace DisasterRecovery
{
    public class DisasterRecoveryManager
    {
        private readonly IBackupService _backupService;
        private readonly IReplicationService _replicationService;
        private readonly IFailoverService _failoverService;
        private readonly IMonitoringService _monitoringService;
        private readonly DisasterRecoveryConfig _config;

        public DisasterRecoveryManager(
            IBackupService backupService,
            IReplicationService replicationService,
            IFailoverService failoverService,
            IMonitoringService monitoringService,
            DisasterRecoveryConfig config)
        {
            _backupService = backupService;
            _replicationService = replicationService;
            _failoverService = failoverService;
            _monitoringService = monitoringService;
            _config = config;
        }

        public async Task InitializeDisasterRecoveryAsync()
        {
            // Set up backup schedules
            await SetupBackupSchedulesAsync();

            // Initialize data replication
            await _replicationService.InitializeReplicationAsync(_config.PrimaryDatabase, _config.SecondaryDatabase);

            // Set up monitoring
            await _monitoringService.StartMonitoringAsync();

            // Register failover event handlers
            _monitoringService.SystemFailureDetected += OnSystemFailureDetected;
        }

        private async Task SetupBackupSchedulesAsync()
        {
            foreach (var backupPolicy in _config.BackupPolicies)
            {
                await _backupService.ScheduleBackupAsync(backupPolicy);
            }
        }

        private async void OnSystemFailureDetected(object sender, SystemFailureEventArgs e)
        {
            Console.WriteLine($"System failure detected: {e.FailureType}");

            if (e.FailureType == FailureType.Critical)
            {
                await InitiateFailoverAsync();
            }
            else
            {
                await HandleNonCriticalFailureAsync(e);
            }
        }

        public async Task InitiateFailoverAsync()
        {
            try
            {
                Console.WriteLine("Initiating disaster recovery failover...");

                // Stop replication to prevent data corruption
                await _replicationService.StopReplicationAsync();

                // Verify secondary system health
                var isSecondaryHealthy = await _failoverService.VerifySecondarySystemAsync();
                if (!isSecondaryHealthy)
                {
                    throw new DisasterRecoveryException("Secondary system is not healthy");
                }

                // Perform failover
                await _failoverService.PerformFailoverAsync();

                // Update DNS/routing
                await UpdateRoutingAsync();

                // Start monitoring new primary
                await _monitoringService.SwitchToNewPrimaryAsync();

                Console.WriteLine("Failover completed successfully");
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Failover failed: {ex.Message}");
                await RollbackFailoverAsync();
                throw;
            }
        }

        public async Task PerformRecoveryAsync()
        {
            Console.WriteLine("Starting disaster recovery process...");

            // Restore from latest backup
            var backup = await _backupService.GetLatestBackupAsync();
            await _backupService.RestoreFromBackupAsync(backup);

            // Re-sync data
            await _replicationService.ResyncDataAsync();

            // Verify system integrity
            var isHealthy = await _monitoringService.VerifySystemHealthAsync();
            if (!isHealthy)
            {
                throw new DisasterRecoveryException("System health verification failed");
            }

            Console.WriteLine("Disaster recovery completed successfully");
        }

        private async Task HandleNonCriticalFailureAsync(SystemFailureEventArgs e)
        {
            // Log the failure
            await _monitoringService.LogFailureAsync(e);

            // Attempt automatic recovery
            var recoveryResult = await _monitoringService.AttemptAutomaticRecoveryAsync(e);

            if (!recoveryResult.Success)
            {
                // Notify administrators
                await NotifyAdministratorsAsync(e, recoveryResult);
            }
        }

        private async Task UpdateRoutingAsync()
        {
            // Update load balancer configuration
            await Task.Delay(1000); // Simulate routing update
        }

        private async Task RollbackFailoverAsync()
        {
            Console.WriteLine("Rolling back failover...");
            await _failoverService.RollbackFailoverAsync();
        }

        private async Task NotifyAdministratorsAsync(SystemFailureEventArgs e, RecoveryResult result)
        {
            // Send notification to administrators
            await Task.CompletedTask;
        }
    }

    public class DisasterRecoveryConfig
    {
        public string PrimaryDatabase { get; set; }
        public string SecondaryDatabase { get; set; }
        public TimeSpan RPO { get; set; } = TimeSpan.FromMinutes(15);
        public TimeSpan RTO { get; set; } = TimeSpan.FromHours(4);
        public List<BackupPolicy> BackupPolicies { get; set; } = new List<BackupPolicy>();
        public bool AutomaticFailover { get; set; } = true;
    }

    public class BackupPolicy
    {
        public string Name { get; set; }
        public BackupType Type { get; set; }
        public TimeSpan Frequency { get; set; }
        public TimeSpan Retention { get; set; }
        public string StorageLocation { get; set; }
    }

    public enum BackupType
    {
        Full,
        Incremental,
        Differential
    }

    public enum FailureType
    {
        Minor,
        Major,
        Critical
    }

    public class SystemFailureEventArgs : EventArgs
    {
        public FailureType FailureType { get; set; }
        public string Component { get; set; }
        public string ErrorMessage { get; set; }
        public DateTime Timestamp { get; set; }
    }

    public class RecoveryResult
    {
        public bool Success { get; set; }
        public string Message { get; set; }
        public TimeSpan RecoveryTime { get; set; }
    }

    public interface IBackupService
    {
        Task ScheduleBackupAsync(BackupPolicy policy);
        Task<Backup> GetLatestBackupAsync();
        Task RestoreFromBackupAsync(Backup backup);
    }

    public interface IReplicationService
    {
        Task InitializeReplicationAsync(string primary, string secondary);
        Task StopReplicationAsync();
        Task ResyncDataAsync();
    }

    public interface IFailoverService
    {
        Task<bool> VerifySecondarySystemAsync();
        Task PerformFailoverAsync();
        Task RollbackFailoverAsync();
    }

    public interface IMonitoringService
    {
        event EventHandler<SystemFailureEventArgs> SystemFailureDetected;
        Task StartMonitoringAsync();
        Task SwitchToNewPrimaryAsync();
        Task<bool> VerifySystemHealthAsync();
        Task LogFailureAsync(SystemFailureEventArgs e);
        Task<RecoveryResult> AttemptAutomaticRecoveryAsync(SystemFailureEventArgs e);
    }

    public class Backup
    {
        public string Id { get; set; }
        public DateTime CreatedAt { get; set; }
        public BackupType Type { get; set; }
        public string Location { get; set; }
        public long Size { get; set; }
    }

    public class DisasterRecoveryException : Exception
    {
        public DisasterRecoveryException(string message) : base(message) { }
    }
}

87. How would you design a hybrid cloud architecture?

Explanation: Hybrid cloud architecture combines on-premises infrastructure with public cloud services. Key components include:

• Cloud Connector: Secure connection between on-premises and cloud
• Data Synchronization: Keeps data consistent across environments
• Load Balancing: Distributes traffic between on-premises and cloud
• Identity Federation: Unified authentication across environments
• Resource Orchestration: Manages resources across both environments

C# Implementation Example:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Text.Json;

namespace HybridCloudArchitecture
{
    public class HybridCloudManager
    {
        private readonly ICloudConnector _cloudConnector;
        private readonly IDataSynchronizer _dataSynchronizer;
        private readonly ILoadBalancer _loadBalancer;
        private readonly IIdentityProvider _identityProvider;
        private readonly IResourceOrchestrator _resourceOrchestrator;
        private readonly HybridCloudConfig _config;

        public HybridCloudManager(
            ICloudConnector cloudConnector,
            IDataSynchronizer dataSynchronizer,
            ILoadBalancer loadBalancer,
            IIdentityProvider identityProvider,
            IResourceOrchestrator resourceOrchestrator,
            HybridCloudConfig config)
        {
            _cloudConnector = cloudConnector;
            _dataSynchronizer = dataSynchronizer;
            _loadBalancer = loadBalancer;
            _identityProvider = identityProvider;
            _resourceOrchestrator = resourceOrchestrator;
            _config = config;
        }

        public async Task InitializeHybridCloudAsync()
        {
            // Establish secure connection to cloud
            await _cloudConnector.EstablishConnectionAsync(_config.CloudEndpoint, _config.Credentials);

            // Set up identity federation
            await _identityProvider.ConfigureFederationAsync(_config.IdentityConfig);

            // Initialize data synchronization
            await _dataSynchronizer.InitializeAsync(_config.SyncConfig);

            // Configure load balancing
            await _loadBalancer.ConfigureHybridLoadBalancingAsync(_config.LoadBalancingConfig);

            // Set up resource orchestration
            await _resourceOrchestrator.InitializeAsync(_config.OrchestrationConfig);
        }

        public async Task<DeploymentResult> DeployToHybridCloudAsync(DeploymentRequest request)
        {
            var result = new DeploymentResult();

            try
            {
                // Determine optimal deployment strategy
                var strategy = DetermineDeploymentStrategy(request);

                // Deploy to on-premises if required
                if (strategy.DeployOnPremises)
                {
                    result.OnPremisesDeployment = await DeployOnPremisesAsync(request);
                }

                // Deploy to cloud if required
                if (strategy.DeployToCloud)
                {
                    result.CloudDeployment = await DeployToCloudAsync(request);
                }

                // Configure load balancing
                await _loadBalancer.UpdateRoutingAsync(strategy);

                // Set up data synchronization if needed
                if (strategy.RequiresDataSync)
                {
                    await _dataSynchronizer.ConfigureSyncAsync(request.ApplicationId);
                }

                result.Success = true;
            }
            catch (Exception ex)
            {
                result.Success = false;
                result.ErrorMessage = ex.Message;
            }

            return result;
        }

        public async Task<ResourceMetrics> GetResourceMetricsAsync()
        {
            var onPremisesMetrics = await _resourceOrchestrator.GetOnPremisesMetricsAsync();
            var cloudMetrics = await _resourceOrchestrator.GetCloudMetricsAsync();

            return new ResourceMetrics
            {
                OnPremises = onPremisesMetrics,
                Cloud = cloudMetrics,
                Timestamp = DateTime.UtcNow
            };
        }

        public async Task<bool> MigrateWorkloadAsync(MigrationRequest request)
        {
            try
            {
                // Validate migration request
                await ValidateMigrationRequestAsync(request);

                // Prepare target environment
                await PrepareTargetEnvironmentAsync(request);

                // Perform data migration
                await MigrateDataAsync(request);

                // Deploy application to target
                await DeployToTargetAsync(request);

                // Update routing
                await UpdateRoutingAfterMigrationAsync(request);

                // Verify migration
                await VerifyMigrationAsync(request);

                return true;
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Migration failed: {ex.Message}");
                await RollbackMigrationAsync(request);
                return false;
            }
        }

        private DeploymentStrategy DetermineDeploymentStrategy(DeploymentRequest request)
        {
            var strategy = new DeploymentStrategy();

            // Check data residency requirements
            if (request.DataResidency == DataResidency.OnPremises)
            {
                strategy.DeployOnPremises = true;
            }
            else if (request.DataResidency == DataResidency.Cloud)
            {
                strategy.DeployToCloud = true;
            }
            else
            {
                // Hybrid deployment based on load
                var onPremisesLoad = _resourceOrchestrator.GetOnPremisesLoadAsync().Result;
                var cloudLoad = _resourceOrchestrator.GetCloudLoadAsync().Result;

                if (onPremisesLoad < cloudLoad)
                {
                    strategy.DeployOnPremises = true;
                }
                else
                {
                    strategy.DeployToCloud = true;
                }
            }

            // Check if data synchronization is needed
            strategy.RequiresDataSync = request.RequiresDataSync;

            return strategy;
        }

        private async Task<DeploymentInfo> DeployOnPremisesAsync(DeploymentRequest request)
        {
            // Deploy to on-premises infrastructure
            return await _resourceOrchestrator.DeployOnPremisesAsync(request);
        }

        private async Task<DeploymentInfo> DeployToCloudAsync(DeploymentRequest request)
        {
            // Deploy to cloud infrastructure
            return await _resourceOrchestrator.DeployToCloudAsync(request);
        }

        private async Task ValidateMigrationRequestAsync(MigrationRequest request)
        {
            // Validate source and target environments
            await Task.CompletedTask;
        }

        private async Task PrepareTargetEnvironmentAsync(MigrationRequest request)
        {
            // Prepare target environment for migration
            await Task.CompletedTask;
        }

        private async Task MigrateDataAsync(MigrationRequest request)
        {
            // Migrate data from source to target
            await _dataSynchronizer.MigrateDataAsync(request.SourceEnvironment, request.TargetEnvironment);
        }

        private async Task DeployToTargetAsync(MigrationRequest request)
        {
            // Deploy application to target environment
            await Task.CompletedTask;
        }

        private async Task UpdateRoutingAfterMigrationAsync(MigrationRequest request)
        {
            // Update routing to point to new environment
            await _loadBalancer.UpdateRoutingAsync(new DeploymentStrategy());
        }

        private async Task VerifyMigrationAsync(MigrationRequest request)
        {
            // Verify migration was successful
            await Task.CompletedTask;
        }

        private async Task RollbackMigrationAsync(MigrationRequest request)
        {
            // Rollback migration if it failed
            await Task.CompletedTask;
        }
    }

    public class HybridCloudConfig
    {
        public string CloudEndpoint { get; set; }
        public CloudCredentials Credentials { get; set; }
        public IdentityConfig IdentityConfig { get; set; }
        public SyncConfig SyncConfig { get; set; }
        public LoadBalancingConfig LoadBalancingConfig { get; set; }
        public OrchestrationConfig OrchestrationConfig { get; set; }
    }

    public class DeploymentRequest
    {
        public string ApplicationId { get; set; }
        public string ApplicationName { get; set; }
        public DataResidency DataResidency { get; set; }
        public bool RequiresDataSync { get; set; }
        public Dictionary<string, string> Configuration { get; set; }
    }

    public class DeploymentResult
    {
        public bool Success { get; set; }
        public string ErrorMessage { get; set; }
        public DeploymentInfo OnPremisesDeployment { get; set; }
        public DeploymentInfo CloudDeployment { get; set; }
    }

    public class DeploymentInfo
    {
        public string Environment { get; set; }
        public string Url { get; set; }
        public DateTime DeployedAt { get; set; }
        public string Status { get; set; }
    }

    public class DeploymentStrategy
    {
        public bool DeployOnPremises { get; set; }
        public bool DeployToCloud { get; set; }
        public bool RequiresDataSync { get; set; }
    }

    public class MigrationRequest
    {
        public string ApplicationId { get; set; }
        public string SourceEnvironment { get; set; }
        public string TargetEnvironment { get; set; }
        public MigrationType MigrationType { get; set; }
    }

    public class ResourceMetrics
    {
        public EnvironmentMetrics OnPremises { get; set; }
        public EnvironmentMetrics Cloud { get; set; }
        public DateTime Timestamp { get; set; }
    }

    public class EnvironmentMetrics
    {
        public double CpuUtilization { get; set; }
        public double MemoryUtilization { get; set; }
        public int ActiveConnections { get; set; }
        public double ResponseTime { get; set; }
    }

    public enum DataResidency
    {
        OnPremises,
        Cloud,
        Hybrid
    }

    public enum MigrationType
    {
        OnPremisesToCloud,
        CloudToOnPremises
    }

    public class CloudCredentials
    {
        public string AccessKey { get; set; }
        public string SecretKey { get; set; }
        public string Region { get; set; }
    }

    public class IdentityConfig
    {
        public string FederationEndpoint { get; set; }
        public string CertificateThumbprint { get; set; }
    }

    public class SyncConfig
    {
        public TimeSpan SyncInterval { get; set; }
        public bool RealTimeSync { get; set; }
        public List<string> SyncDatabases { get; set; }
    }

    public class LoadBalancingConfig
    {
        public string PrimaryEndpoint { get; set; }
        public string SecondaryEndpoint { get; set; }
        public LoadBalancingStrategy Strategy { get; set; }
    }

    public class OrchestrationConfig
    {
        public string OnPremisesEndpoint { get; set; }
        public string CloudEndpoint { get; set; }
        public Dictionary<string, string> ResourceMappings { get; set; }
    }

    public enum LoadBalancingStrategy
    {
        RoundRobin,
        LeastConnections,
        Geographic
    }

    public interface ICloudConnector
    {
        Task EstablishConnectionAsync(string endpoint, CloudCredentials credentials);
    }

    public interface IDataSynchronizer
    {
        Task InitializeAsync(SyncConfig config);
        Task ConfigureSyncAsync(string applicationId);
        Task MigrateDataAsync(string source, string target);
    }

    public interface ILoadBalancer
    {
        Task ConfigureHybridLoadBalancingAsync(LoadBalancingConfig config);
        Task UpdateRoutingAsync(DeploymentStrategy strategy);
    }

    public interface IIdentityProvider
    {
        Task ConfigureFederationAsync(IdentityConfig config);
    }

    public interface IResourceOrchestrator
    {
        Task InitializeAsync(OrchestrationConfig config);
        Task<DeploymentInfo> DeployOnPremisesAsync(DeploymentRequest request);
        Task<DeploymentInfo> DeployToCloudAsync(DeploymentRequest request);
        Task<EnvironmentMetrics> GetOnPremisesMetricsAsync();
        Task<EnvironmentMetrics> GetCloudMetricsAsync();
        Task<double> GetOnPremisesLoadAsync();
        Task<double> GetCloudLoadAsync();
    }
}

I'll act as a technical architect and provide comprehensive answers to these three interview questions, including explanations and C# code examples where relevant.

88. How do you implement auto-scaling?

Explanation

Auto-scaling is a cloud computing technique that automatically adjusts the number of computing resources (servers, containers, etc.) based on demand. It ensures optimal performance during peak loads while minimizing costs during low usage periods.

Key Components: - Metrics Collection: CPU, memory, request count, response time - Scaling Policies: Rules that determine when to scale up/down - Scaling Triggers: Thresholds that activate scaling actions - Resource Management: Provisioning/deprovisioning of resources

Types of Auto-scaling: 1. Horizontal Scaling: Adding/removing instances 2. Vertical Scaling: Increasing/decreasing instance capacity 3. Predictive Scaling: Based on historical patterns

C# Implementation Example

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;

namespace AutoScalingExample
{
    // Configuration for auto-scaling
    public class AutoScalingConfig
    {
        public int MinInstances { get; set; } = 2;
        public int MaxInstances { get; set; } = 10;
        public double CpuThreshold { get; set; } = 70.0;
        public double MemoryThreshold { get; set; } = 80.0;
        public int RequestThreshold { get; set; } = 1000;
        public TimeSpan ScalingCooldown { get; set; } = TimeSpan.FromMinutes(5);
        public TimeSpan MetricsCollectionInterval { get; set; } = TimeSpan.FromSeconds(30);
    }

    // Metrics data structure
    public class InstanceMetrics
    {
        public string InstanceId { get; set; }
        public double CpuUsage { get; set; }
        public double MemoryUsage { get; set; }
        public int RequestCount { get; set; }
        public double ResponseTime { get; set; }
        public DateTime Timestamp { get; set; }
    }

    // Scaling decision
    public enum ScalingAction
    {
        None,
        ScaleUp,
        ScaleDown
    }

    // Auto-scaling service
    public class AutoScalingService
    {
        private readonly ILogger<AutoScalingService> _logger;
        private readonly AutoScalingConfig _config;
        private readonly IResourceManager _resourceManager;
        private readonly IMetricsCollector _metricsCollector;
        private readonly ConcurrentDictionary<string, InstanceMetrics> _instanceMetrics;
        private readonly Timer _scalingTimer;
        private DateTime _lastScalingAction = DateTime.MinValue;

        public AutoScalingService(
            ILogger<AutoScalingService> logger,
            IOptions<AutoScalingConfig> config,
            IResourceManager resourceManager,
            IMetricsCollector metricsCollector)
        {
            _logger = logger;
            _config = config.Value;
            _resourceManager = resourceManager;
            _metricsCollector = metricsCollector;
            _instanceMetrics = new ConcurrentDictionary<string, InstanceMetrics>();

            // Start periodic scaling evaluation
            _scalingTimer = new Timer(EvaluateScaling, null, 
                _config.MetricsCollectionInterval, _config.MetricsCollectionInterval);
        }

        public async Task StartAsync()
        {
            _logger.LogInformation("Auto-scaling service started");
            await _resourceManager.EnsureMinimumInstances(_config.MinInstances);
        }

        public async Task StopAsync()
        {
            _scalingTimer?.Dispose();
            _logger.LogInformation("Auto-scaling service stopped");
        }

        private async void EvaluateScaling(object state)
        {
            try
            {
                // Check if we're in cooldown period
                if (DateTime.UtcNow - _lastScalingAction < _config.ScalingCooldown)
                {
                    return;
                }

                // Collect current metrics
                await CollectMetrics();

                // Determine scaling action
                var scalingAction = DetermineScalingAction();

                // Execute scaling if needed
                if (scalingAction != ScalingAction.None)
                {
                    await ExecuteScaling(scalingAction);
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error during scaling evaluation");
            }
        }

        private async Task CollectMetrics()
        {
            var instances = await _resourceManager.GetActiveInstances();

            foreach (var instance in instances)
            {
                var metrics = await _metricsCollector.GetInstanceMetrics(instance.Id);
                _instanceMetrics.AddOrUpdate(instance.Id, metrics, (key, oldValue) => metrics);
            }
        }

        private ScalingAction DetermineScalingAction()
        {
            var currentInstanceCount = _instanceMetrics.Count;
            var averageCpu = _instanceMetrics.Values.Average(m => m.CpuUsage);
            var averageMemory = _instanceMetrics.Values.Average(m => m.MemoryUsage);
            var totalRequests = _instanceMetrics.Values.Sum(m => m.RequestCount);
            var averageResponseTime = _instanceMetrics.Values.Average(m => m.ResponseTime);

            _logger.LogInformation(
                "Current metrics - Instances: {InstanceCount}, CPU: {Cpu}%, Memory: {Memory}%, " +
                "Requests: {Requests}, ResponseTime: {ResponseTime}ms",
                currentInstanceCount, averageCpu, averageMemory, totalRequests, averageResponseTime);

            // Scale up conditions
            if (currentInstanceCount < _config.MaxInstances &&
                (averageCpu > _config.CpuThreshold ||
                 averageMemory > _config.MemoryThreshold ||
                 totalRequests > _config.RequestThreshold ||
                 averageResponseTime > 2000)) // 2 seconds threshold
            {
                return ScalingAction.ScaleUp;
            }

            // Scale down conditions
            if (currentInstanceCount > _config.MinInstances &&
                averageCpu < _config.CpuThreshold * 0.5 &&
                averageMemory < _config.MemoryThreshold * 0.5 &&
                totalRequests < _config.RequestThreshold * 0.5 &&
                averageResponseTime < 500) // 500ms threshold
            {
                return ScalingAction.ScaleDown;
            }

            return ScalingAction.None;
        }

        private async Task ExecuteScaling(ScalingAction action)
        {
            try
            {
                switch (action)
                {
                    case ScalingAction.ScaleUp:
                        await _resourceManager.ScaleUp();
                        _logger.LogInformation("Scaling up - added new instance");
                        break;

                    case ScalingAction.ScaleDown:
                        await _resourceManager.ScaleDown();
                        _logger.LogInformation("Scaling down - removed instance");
                        break;
                }

                _lastScalingAction = DateTime.UtcNow;
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error executing scaling action: {Action}", action);
            }
        }
    }

    // Resource management interface
    public interface IResourceManager
    {
        Task<List<InstanceInfo>> GetActiveInstances();
        Task EnsureMinimumInstances(int minInstances);
        Task ScaleUp();
        Task ScaleDown();
    }

    // Metrics collection interface
    public interface IMetricsCollector
    {
        Task<InstanceMetrics> GetInstanceMetrics(string instanceId);
    }

    // Instance information
    public class InstanceInfo
    {
        public string Id { get; set; }
        public string Status { get; set; }
        public DateTime CreatedAt { get; set; }
    }

    // Azure-specific resource manager implementation
    public class AzureResourceManager : IResourceManager
    {
        private readonly ILogger<AzureResourceManager> _logger;
        private readonly List<InstanceInfo> _instances = new();

        public AzureResourceManager(ILogger<AzureResourceManager> logger)
        {
            _logger = logger;
        }

        public async Task<List<InstanceInfo>> GetActiveInstances()
        {
            // In real implementation, this would query Azure App Service, VMSS, or AKS
            await Task.Delay(100); // Simulate API call
            return _instances.Where(i => i.Status == "Running").ToList();
        }

        public async Task EnsureMinimumInstances(int minInstances)
        {
            var currentCount = _instances.Count;
            var needed = minInstances - currentCount;

            for (int i = 0; i < needed; i++)
            {
                await ScaleUp();
            }
        }

        public async Task ScaleUp()
        {
            // In real implementation, this would create new Azure resources
            var newInstance = new InstanceInfo
            {
                Id = Guid.NewGuid().ToString(),
                Status = "Running",
                CreatedAt = DateTime.UtcNow
            };

            _instances.Add(newInstance);
            _logger.LogInformation("Created new instance: {InstanceId}", newInstance.Id);
            await Task.Delay(1000); // Simulate provisioning time
        }

        public async Task ScaleDown()
        {
            // In real implementation, this would remove Azure resources
            var instanceToRemove = _instances.FirstOrDefault(i => i.Status == "Running");
            if (instanceToRemove != null)
            {
                instanceToRemove.Status = "Terminated";
                _logger.LogInformation("Terminated instance: {InstanceId}", instanceToRemove.Id);
            }
            await Task.Delay(500); // Simulate deprovisioning time
        }
    }

    // Metrics collector implementation
    public class MetricsCollector : IMetricsCollector
    {
        private readonly Random _random = new();

        public async Task<InstanceMetrics> GetInstanceMetrics(string instanceId)
        {
            // In real implementation, this would query monitoring systems like Azure Monitor
            await Task.Delay(50); // Simulate API call

            return new InstanceMetrics
            {
                InstanceId = instanceId,
                CpuUsage = _random.NextDouble() * 100,
                MemoryUsage = _random.NextDouble() * 100,
                RequestCount = _random.Next(0, 2000),
                ResponseTime = _random.NextDouble() * 3000,
                Timestamp = DateTime.UtcNow
            };
        }
    }

    // Program example
    public class Program
    {
        public static async Task Main(string[] args)
        {
            // Configure services
            var config = new AutoScalingConfig
            {
                MinInstances = 2,
                MaxInstances = 10,
                CpuThreshold = 70.0,
                MemoryThreshold = 80.0,
                RequestThreshold = 1000,
                ScalingCooldown = TimeSpan.FromMinutes(5),
                MetricsCollectionInterval = TimeSpan.FromSeconds(30)
            };

            var loggerFactory = LoggerFactory.Create(builder => builder.AddConsole());
            var logger = loggerFactory.CreateLogger<AutoScalingService>();

            var resourceManager = new AzureResourceManager(loggerFactory.CreateLogger<AzureResourceManager>());
            var metricsCollector = new MetricsCollector();

            var autoScalingService = new AutoScalingService(
                logger,
                Options.Create(config),
                resourceManager,
                metricsCollector);

            // Start auto-scaling
            await autoScalingService.StartAsync();

            // Keep running for demonstration
            Console.WriteLine("Auto-scaling service is running. Press any key to exit.");
            Console.ReadKey();

            await autoScalingService.StopAsync();
        }
    }
}

89. How would you design a CDN architecture?

Explanation

A Content Delivery Network (CDN) is a distributed network of servers that deliver web content to users based on their geographic location. It improves performance, reduces latency, and provides high availability.

Key Components: - Origin Server: The original source of content - Edge Servers: Distributed servers located close to users - Load Balancer: Distributes requests across edge servers - Cache Management: Determines what content to cache and for how long - Geographic Routing: Routes users to the nearest edge server

CDN Architecture Layers: 1. DNS Layer: Geographic routing and load balancing 2. Edge Layer: Content caching and delivery 3. Origin Layer: Source content management 4. Analytics Layer: Performance monitoring and reporting

C# Implementation Example

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging;
using System.Text.Json;
using System.Text;

namespace CDNArchitectureExample
{
    // CDN Configuration
    public class CDNConfig
    {
        public string OriginServerUrl { get; set; } = "https://origin.example.com";
        public List<EdgeServer> EdgeServers { get; set; } = new();
        public TimeSpan DefaultCacheDuration { get; set; } = TimeSpan.FromHours(1);
        public int MaxCacheSize { get; set; } = 1000; // MB
        public bool EnableCompression { get; set; } = true;
        public bool EnableSSL { get; set; } = true;
    }

    // Edge Server Information
    public class EdgeServer
    {
        public string Id { get; set; }
        public string Region { get; set; }
        public string Location { get; set; }
        public string Url { get; set; }
        public double Latitude { get; set; }
        public double Longitude { get; set; }
        public bool IsHealthy { get; set; } = true;
        public int CurrentLoad { get; set; }
        public DateTime LastHealthCheck { get; set; }
    }

    // Content Cache Item
    public class CacheItem
    {
        public string Key { get; set; }
        public byte[] Content { get; set; }
        public string ContentType { get; set; }
        public Dictionary<string, string> Headers { get; set; }
        public DateTime ExpiresAt { get; set; }
        public DateTime LastAccessed { get; set; }
        public int AccessCount { get; set; }
        public long Size { get; set; }
    }

    // Geographic Location
    public class GeoLocation
    {
        public string Country { get; set; }
        public string Region { get; set; }
        public string City { get; set; }
        public double Latitude { get; set; }
        public double Longitude { get; set; }
    }

    // CDN Service
    public class CDNService
    {
        private readonly ILogger<CDNService> _logger;
        private readonly CDNConfig _config;
        private readonly IMemoryCache _cache;
        private readonly HttpClient _httpClient;
        private readonly IGeoLocationService _geoService;
        private readonly ILoadBalancer _loadBalancer;
        private readonly ConcurrentDictionary<string, EdgeServer> _edgeServers;

        public CDNService(
            ILogger<CDNService> logger,
            CDNConfig config,
            IMemoryCache cache,
            HttpClient httpClient,
            IGeoLocationService geoService,
            ILoadBalancer loadBalancer)
        {
            _logger = logger;
            _config = config;
            _cache = cache;
            _httpClient = httpClient;
            _geoService = geoService;
            _loadBalancer = loadBalancer;
            _edgeServers = new ConcurrentDictionary<string, EdgeServer>();

            // Initialize edge servers
            foreach (var server in config.EdgeServers)
            {
                _edgeServers.TryAdd(server.Id, server);
            }
        }

        public async Task<CDNResponse> GetContentAsync(string path, string clientIp)
        {
            try
            {
                // Generate cache key
                var cacheKey = GenerateCacheKey(path);

                // Try to get from cache first
                var cachedContent = await GetFromCacheAsync(cacheKey);
                if (cachedContent != null)
                {
                    _logger.LogInformation("Cache hit for path: {Path}", path);
                    return cachedContent;
                }

                // Get client location
                var clientLocation = await _geoService.GetLocationAsync(clientIp);

                // Select best edge server
                var edgeServer = await SelectEdgeServerAsync(clientLocation);

                // Get content from origin or edge server
                var content = await FetchContentAsync(path, edgeServer);

                // Cache the content
                await CacheContentAsync(cacheKey, content);

                return content;
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error serving content for path: {Path}", path);
                return new CDNResponse
                {
                    Content = Encoding.UTF8.GetBytes("Service temporarily unavailable"),
                    ContentType = "text/plain",
                    StatusCode = HttpStatusCode.ServiceUnavailable
                };
            }
        }

        private string GenerateCacheKey(string path)
        {
            return $"cdn:{path.ToLowerInvariant()}";
        }

        private async Task<CDNResponse> GetFromCacheAsync(string cacheKey)
        {
            if (_cache.TryGetValue(cacheKey, out CacheItem cachedItem))
            {
                // Update access statistics
                cachedItem.LastAccessed = DateTime.UtcNow;
                cachedItem.AccessCount++;

                return new CDNResponse
                {
                    Content = cachedItem.Content,
                    ContentType = cachedItem.ContentType,
                    Headers = cachedItem.Headers,
                    StatusCode = HttpStatusCode.OK,
                    ServedFromCache = true
                };
            }

            return null;
        }

        private async Task<EdgeServer> SelectEdgeServerAsync(GeoLocation clientLocation)
        {
            // Get healthy edge servers
            var healthyServers = _edgeServers.Values
                .Where(s => s.IsHealthy)
                .ToList();

            if (!healthyServers.Any())
            {
                throw new InvalidOperationException("No healthy edge servers available");
            }

            // Use load balancer to select server
            return await _loadBalancer.SelectServerAsync(healthyServers, clientLocation);
        }

        private async Task<CDNResponse> FetchContentAsync(string path, EdgeServer edgeServer)
        {
            // Try edge server first
            try
            {
                var edgeUrl = $"{edgeServer.Url}{path}";
                var response = await _httpClient.GetAsync(edgeUrl);

                if (response.IsSuccessStatusCode)
                {
                    var content = await response.Content.ReadAsByteArrayAsync();
                    return new CDNResponse
                    {
                        Content = content,
                        ContentType = response.Content.Headers.ContentType?.ToString(),
                        Headers = response.Headers.ToDictionary(h => h.Key, h => string.Join(", ", h.Value)),
                        StatusCode = response.StatusCode,
                        ServedFromEdge = true
                    };
                }
            }
            catch (Exception ex)
            {
                _logger.LogWarning(ex, "Failed to fetch from edge server: {EdgeServer}", edgeServer.Id);
            }

            // Fallback to origin server
            var originUrl = $"{_config.OriginServerUrl}{path}";
            var originResponse = await _httpClient.GetAsync(originUrl);

            var originContent = await originResponse.Content.ReadAsByteArrayAsync();
            return new CDNResponse
            {
                Content = originContent,
                ContentType = originResponse.Content.Headers.ContentType?.ToString(),
                Headers = originResponse.Headers.ToDictionary(h => h.Key, h => string.Join(", ", h.Value)),
                StatusCode = originResponse.StatusCode,
                ServedFromOrigin = true
            };
        }

        private async Task CacheContentAsync(string cacheKey, CDNResponse content)
        {
            var cacheItem = new CacheItem
            {
                Key = cacheKey,
                Content = content.Content,
                ContentType = content.ContentType,
                Headers = content.Headers,
                ExpiresAt = DateTime.UtcNow.Add(_config.DefaultCacheDuration),
                LastAccessed = DateTime.UtcNow,
                AccessCount = 1,
                Size = content.Content.Length
            };

            var cacheOptions = new MemoryCacheEntryOptions
            {
                AbsoluteExpirationRelativeToNow = _config.DefaultCacheDuration,
                Size = cacheItem.Size
            };

            _cache.Set(cacheKey, cacheItem, cacheOptions);
            _logger.LogInformation("Cached content: {CacheKey}, Size: {Size} bytes", cacheKey, cacheItem.Size);
        }

        public async Task UpdateEdgeServerHealthAsync()
        {
            foreach (var server in _edgeServers.Values)
            {
                try
                {
                    var healthUrl = $"{server.Url}/health";
                    var response = await _httpClient.GetAsync(healthUrl, TimeSpan.FromSeconds(5));

                    server.IsHealthy = response.IsSuccessStatusCode;
                    server.LastHealthCheck = DateTime.UtcNow;

                    if (response.IsSuccessStatusCode)
                    {
                        var healthData = await response.Content.ReadAsStringAsync();
                        var health = JsonSerializer.Deserialize<HealthResponse>(healthData);
                        server.CurrentLoad = health?.CurrentLoad ?? 0;
                    }
                }
                catch (Exception ex)
                {
                    _logger.LogWarning(ex, "Health check failed for edge server: {ServerId}", server.Id);
                    server.IsHealthy = false;
                }
            }
        }
    }

    // CDN Response
    public class CDNResponse
    {
        public byte[] Content { get; set; }
        public string ContentType { get; set; }
        public Dictionary<string, string> Headers { get; set; }
        public HttpStatusCode StatusCode { get; set; }
        public bool ServedFromCache { get; set; }
        public bool ServedFromEdge { get; set; }
        public bool ServedFromOrigin { get; set; }
    }

    // Health Response
    public class HealthResponse
    {
        public string Status { get; set; }
        public int CurrentLoad { get; set; }
        public DateTime Timestamp { get; set; }
    }

    // Geographic Location Service Interface
    public interface IGeoLocationService
    {
        Task<GeoLocation> GetLocationAsync(string ipAddress);
    }

    // Load Balancer Interface
    public interface ILoadBalancer
    {
        Task<EdgeServer> SelectServerAsync(List<EdgeServer> servers, GeoLocation clientLocation);
    }

    // Geographic-based Load Balancer
    public class GeoLoadBalancer : ILoadBalancer
    {
        private readonly ILogger<GeoLoadBalancer> _logger;

        public GeoLoadBalancer(ILogger<GeoLoadBalancer> logger)
        {
            _logger = logger;
        }

        public async Task<EdgeServer> SelectServerAsync(List<EdgeServer> servers, GeoLocation clientLocation)
        {
            // Find the closest server based on geographic distance
            var closestServer = servers
                .OrderBy(s => CalculateDistance(
                    clientLocation.Latitude, clientLocation.Longitude,
                    s.Latitude, s.Longitude))
                .ThenBy(s => s.CurrentLoad)
                .First();

            _logger.LogInformation(
                "Selected edge server {ServerId} for client in {City}, {Country}",
                closestServer.Id, clientLocation.City, clientLocation.Country);

            return closestServer;
        }

        private double CalculateDistance(double lat1, double lon1, double lat2, double lon2)
        {
            const double R = 6371; // Earth's radius in kilometers
            var dLat = ToRadians(lat2 - lat1);
            var dLon = ToRadians(lon2 - lon1);
            var a = Math.Sin(dLat / 2) * Math.Sin(dLat / 2) +
                    Math.Cos(ToRadians(lat1)) * Math.Cos(ToRadians(lat2)) *
                    Math.Sin(dLon / 2) * Math.Sin(dLon / 2);
            var c = 2 * Math.Atan2(Math.Sqrt(a), Math.Sqrt(1 - a));
            return R * c;
        }

        private double ToRadians(double degrees)
        {
            return degrees * Math.PI / 180;
        }
    }

    // IP-based Geographic Location Service
    public class IPGeoLocationService : IGeoLocationService
    {
        private readonly ILogger<IPGeoLocationService> _logger;
        private readonly HttpClient _httpClient;

        public IPGeoLocationService(ILogger<IPGeoLocationService> logger, HttpClient httpClient)
        {
            _logger = logger;
            _httpClient = httpClient;
        }

        public async Task<GeoLocation> GetLocationAsync(string ipAddress)
        {
            try
            {
                // In real implementation, this would call a geolocation API
                // For demo purposes, we'll return mock data based on IP ranges
                return GetMockLocation(ipAddress);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Failed to get location for IP: {IP}", ipAddress);
                return new GeoLocation
                {
                    Country = "Unknown",
                    Region = "Unknown",
                    City = "Unknown",
                    Latitude = 0,
                    Longitude = 0
                };
            }
        }

        private GeoLocation GetMockLocation(string ipAddress)
        {
            // Simple mock implementation based on IP ranges
            var ipParts = ipAddress.Split('.');
            var firstOctet = int.Parse(ipParts[0]);

            return firstOctet switch
            {
                < 64 => new GeoLocation { Country = "US", Region = "West", City = "San Francisco", Latitude = 37.7749, Longitude = -122.4194 },
                < 128 => new GeoLocation { Country = "US", Region = "East", City = "New York", Latitude = 40.7128, Longitude = -74.0060 },
                < 192 => new GeoLocation { Country = "UK", Region = "England", City = "London", Latitude = 51.5074, Longitude = -0.1278 },
                _ => new GeoLocation { Country = "DE", Region = "Berlin", City = "Berlin", Latitude = 52.5200, Longitude = 13.4050 }
            };
        }
    }

    // CDN Controller
    public class CDNController
    {
        private readonly CDNService _cdnService;
        private readonly ILogger<CDNController> _logger;

        public CDNController(CDNService cdnService, ILogger<CDNController> logger)
        {
            _cdnService = cdnService;
            _logger = logger;
        }

        public async Task<CDNResponse> HandleRequestAsync(string path, string clientIp)
        {
            _logger.LogInformation("CDN request: {Path} from {ClientIP}", path, clientIp);

            var response = await _cdnService.GetContentAsync(path, clientIp);

            _logger.LogInformation(
                "CDN response: {Path} - Status: {StatusCode}, From: {Source}",
                path, response.StatusCode,
                response.ServedFromCache ? "Cache" : 
                response.ServedFromEdge ? "Edge" : "Origin");

            return response;
        }
    }

    // Program Example
    public class Program
    {
        public static async Task Main(string[] args)
        {
            // Configure CDN
            var config = new CDNConfig
            {
                OriginServerUrl = "https://origin.example.com",
                EdgeServers = new List<EdgeServer>
                {
                    new() { Id = "us-west", Region = "US-West", Location = "San Francisco", Url = "https://cdn-west.example.com", Latitude = 37.7749, Longitude = -122.4194 },
                    new() { Id = "us-east", Region = "US-East", Location = "New York", Url = "https://cdn-east.example.com", Latitude = 40.7128, Longitude = -74.0060 },
                    new() { Id = "eu-west", Region = "EU-West", Location = "London", Url = "https://cdn-eu.example.com", Latitude = 51.5074, Longitude = -0.1278 },
                    new() { Id = "asia-east", Region = "Asia-East", Location = "Tokyo", Url = "https://cdn-asia.example.com", Latitude = 35.6762, Longitude = 139.6503 }
                },
                DefaultCacheDuration = TimeSpan.FromHours(1),
                MaxCacheSize = 1000,
                EnableCompression = true,
                EnableSSL = true
            };

            var loggerFactory = LoggerFactory.Create(builder => builder.AddConsole());
            var cache = new MemoryCache(new MemoryCacheOptions { SizeLimit = config.MaxCacheSize });
            var httpClient = new HttpClient();
            var geoService = new IPGeoLocationService(loggerFactory.CreateLogger<IPGeoLocationService>(), httpClient);
            var loadBalancer = new GeoLoadBalancer(loggerFactory.CreateLogger<GeoLoadBalancer>());

            var cdnService = new CDNService(
                loggerFactory.CreateLogger<CDNService>(),
                config,
                cache,
                httpClient,
                geoService,
                loadBalancer);

            var controller = new CDNController(cdnService, loggerFactory.CreateLogger<CDNController>());

            // Simulate CDN requests
            var testRequests = new[]
            {
                ("/images/logo.png", "192.168.1.100"),
                ("/css/style.css", "10.0.0.50"),
                ("/js/app.js", "172.16.0.25")
            };

            foreach (var (path, clientIp) in testRequests)
            {
                var response = await controller.HandleRequestAsync(path, clientIp);
                Console.WriteLine($"Request: {path} -> Status: {response.StatusCode}, Served from: {(response.ServedFromCache ? "Cache" : response.ServedFromEdge ? "Edge" : "Origin")}");
            }

            // Update edge server health
            await cdnService.UpdateEdgeServerHealthAsync();
        }
    }
}

90. How do you implement edge computing?

Explanation

Edge computing is a distributed computing paradigm that brings computation and data storage closer to the location where it's needed, reducing latency and bandwidth usage. It processes data at the edge of the network, near the data source.

Key Components: - Edge Devices: IoT devices, sensors, mobile devices - Edge Nodes: Local processing units (gateways, routers, servers) - Edge Orchestrator: Manages edge resources and workloads - Cloud Backend: Centralized processing and storage - Data Pipeline: Real-time data processing and analytics

Edge Computing Benefits: 1. Low Latency: Processing close to data source 2. Bandwidth Optimization: Reduced data transmission 3. Privacy: Local data processing 4. Reliability: Offline operation capability 5. Scalability: Distributed processing

C# Implementation Example

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using System.Text.Json;
using System.Text;
using System.Net.Http;
using System.Net;

namespace EdgeComputingExample
{
    // Edge Computing Configuration
    public class EdgeConfig
    {
        public string EdgeNodeId { get; set; }
        public string CloudEndpoint { get; set; } = "https://cloud.example.com";
        public TimeSpan SyncInterval { get; set; } = TimeSpan.FromMinutes(5);
        public int MaxLocalStorageSize { get; set; } = 1000; // MB
        public bool EnableOfflineMode { get; set; } = true;
        public TimeSpan DataRetentionPeriod { get; set; } = TimeSpan.FromDays(7);
        public int MaxConcurrentWorkloads { get; set; } = 10;
    }

    // Edge Device Information
    public class EdgeDevice
    {
        public string DeviceId { get; set; }
        public string DeviceType { get; set; }
        public string Location { get; set; }
        public Dictionary<string, object> Capabilities { get; set; }
        public bool IsOnline { get; set; } = true;
        public DateTime LastSeen { get; set; }
    }

    // Data Processing Workload
    public class Workload
    {
        public string WorkloadId { get; set; }
        public string Type { get; set; } // "data-processing", "analytics", "ml-inference"
        public Dictionary<string, object> Parameters { get; set; }
        public byte[] InputData { get; set; }
        public WorkloadPriority Priority { get; set; }
        public DateTime CreatedAt { get; set; }
        public DateTime? CompletedAt { get; set; }
        public WorkloadStatus Status { get; set; }
        public object Result { get; set; }
        public string ErrorMessage { get; set; }
    }

    public enum WorkloadPriority
    {
        Low,
        Normal,
        High,
        Critical
    }

    public enum WorkloadStatus
    {
        Pending,
        Running,
        Completed,
        Failed,
        Cancelled
    }

    // Processed Data
    public class ProcessedData
    {
        public string DataId { get; set; }
        public string SourceDeviceId { get; set; }
        public string DataType { get; set; }
        public object ProcessedValue { get; set; }
        public Dictionary<string, object> Metadata { get; set; }
        public DateTime ProcessedAt { get; set; }
        public DateTime CreatedAt { get; set; }
        public bool SyncedToCloud { get; set; }
    }

    // Edge Node Service
    public class EdgeNodeService
    {
        private readonly ILogger<EdgeNodeService> _logger;
        private readonly EdgeConfig _config;
        private readonly IWorkloadExecutor _workloadExecutor;
        private readonly IDataProcessor _dataProcessor;
        private readonly ICloudSyncService _cloudSync;
        private readonly ILocalStorage _localStorage;
        private readonly ConcurrentDictionary<string, EdgeDevice> _devices;
        private readonly ConcurrentQueue<Workload> _workloadQueue;
        private readonly ConcurrentDictionary<string, ProcessedData> _processedData;
        private readonly SemaphoreSlim _workloadSemaphore;
        private readonly Timer _syncTimer;
        private readonly Timer _cleanupTimer;

        public EdgeNodeService(
            ILogger<EdgeNodeService> logger,
            EdgeConfig config,
            IWorkloadExecutor workloadExecutor,
            IDataProcessor dataProcessor,
            ICloudSyncService cloudSync,
            ILocalStorage localStorage)
        {
            _logger = logger;
            _config = config;
            _workloadExecutor = workloadExecutor;
            _dataProcessor = dataProcessor;
            _cloudSync = cloudSync;
            _localStorage = localStorage;
            _devices = new ConcurrentDictionary<string, EdgeDevice>();
            _workloadQueue = new ConcurrentQueue<Workload>();
            _processedData = new ConcurrentDictionary<string, ProcessedData>();
            _workloadSemaphore = new SemaphoreSlim(_config.MaxConcurrentWorkloads);

            // Start periodic tasks
            _syncTimer = new Timer(SyncToCloud, null, _config.SyncInterval, _config.SyncInterval);
            _cleanupTimer = new Timer(CleanupOldData, null, TimeSpan.FromHours(1), TimeSpan.FromHours(1));
        }

        public async Task StartAsync()
        {
            _logger.LogInformation("Edge node {NodeId} started", _config.EdgeNodeId);

            // Start workload processing
            _ = Task.Run(ProcessWorkloadsAsync);

            // Initialize local storage
            await _localStorage.InitializeAsync();
        }

        public async Task StopAsync()
        {
            _syncTimer?.Dispose();
            _cleanupTimer?.Dispose();
            _workloadSemaphore?.Dispose();

            // Final sync before stopping
            await SyncToCloud();

            _logger.LogInformation("Edge node {NodeId} stopped", _config.EdgeNodeId);
        }

        public async Task RegisterDeviceAsync(EdgeDevice device)
        {
            _devices.TryAdd(device.DeviceId, device);
            _logger.LogInformation("Device {DeviceId} registered", device.DeviceId);

            // Notify cloud about new device
            await _cloudSync.NotifyDeviceRegistrationAsync(device);
        }

        public async Task<Workload> SubmitWorkloadAsync(Workload workload)
        {
            workload.WorkloadId = Guid.NewGuid().ToString();
            workload.CreatedAt = DateTime.UtcNow;
            workload.Status = WorkloadStatus.Pending;

            _workloadQueue.Enqueue(workload);
            _logger.LogInformation("Workload {WorkloadId} submitted", workload.WorkloadId);

            return workload;
        }

        public async Task<ProcessedData> ProcessDeviceDataAsync(string deviceId, object rawData, string dataType)
        {
            try
            {
                // Process data locally
                var processedValue = await _dataProcessor.ProcessAsync(rawData, dataType);

                var processedData = new ProcessedData
                {
                    DataId = Guid.NewGuid().ToString(),
                    SourceDeviceId = deviceId,
                    DataType = dataType,
                    ProcessedValue = processedValue,
                    Metadata = new Dictionary<string, object>
                    {
                        ["processed_at_edge"] = true,
                        ["edge_node_id"] = _config.EdgeNodeId
                    },
                    ProcessedAt = DateTime.UtcNow,
                    CreatedAt = DateTime.UtcNow,
                    SyncedToCloud = false
                };

                _processedData.TryAdd(processedData.DataId, processedData);

                // Store locally
                await _localStorage.StoreDataAsync(processedData);

                _logger.LogInformation("Data processed for device {DeviceId}: {DataType}", deviceId, dataType);

                return processedData;
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error processing data for device {DeviceId}", deviceId);
                throw;
            }
        }

        private async void ProcessWorkloadsAsync()
        {
            while (true)
            {
                try
                {
                    if (_workloadQueue.TryDequeue(out var workload))
                    {
                        await _workloadSemaphore.WaitAsync();

                        _ = Task.Run(async () =>
                        {
                            try
                            {
                                await ExecuteWorkloadAsync(workload);
                            }
                            finally
                            {
                                _workloadSemaphore.Release();
                            }
                        });
                    }
                    else
                    {
                        await Task.Delay(100); // Wait for new workloads
                    }
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, "Error in workload processing loop");
                    await Task.Delay(1000);
                }
            }
        }

        private async Task ExecuteWorkloadAsync(Workload workload)
        {
            try
            {
                workload.Status = WorkloadStatus.Running;
                _logger.LogInformation("Executing workload {WorkloadId}: {Type}", workload.WorkloadId, workload.Type);

                var result = await _workloadExecutor.ExecuteAsync(workload);

                workload.Result = result;
                workload.Status = WorkloadStatus.Completed;
                workload.CompletedAt = DateTime.UtcNow;

                _logger.LogInformation("Workload {WorkloadId} completed successfully", workload.WorkloadId);
            }
            catch (Exception ex)
            {
                workload.Status = WorkloadStatus.Failed;
                workload.ErrorMessage = ex.Message;
                workload.CompletedAt = DateTime.UtcNow;

                _logger.LogError(ex, "Workload {WorkloadId} failed", workload.WorkloadId);
            }
        }

        private async void SyncToCloud(object state = null)
        {
            try
            {
                if (!_config.EnableOfflineMode || await _cloudSync.IsConnectedAsync())
                {
                    // Sync processed data
                    var unsyncedData = _processedData.Values
                        .Where(d => !d.SyncedToCloud)
                        .ToList();

                    foreach (var data in unsyncedData)
                    {
                        await _cloudSync.SyncDataAsync(data);
                        data.SyncedToCloud = true;
                    }

                    // Sync workload results
                    var completedWorkloads = _workloadQueue
                        .Where(w => w.Status == WorkloadStatus.Completed || w.Status == WorkloadStatus.Failed)
                        .ToList();

                    foreach (var workload in completedWorkloads)
                    {
                        await _cloudSync.SyncWorkloadResultAsync(workload);
                    }

                    _logger.LogInformation("Synced {DataCount} data items and {WorkloadCount} workloads to cloud", 
                        unsyncedData.Count, completedWorkloads.Count);
                }
                else
                {
                    _logger.LogWarning("Cloud sync skipped - offline mode enabled and no connection");
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error during cloud sync");
            }
        }

        private async void CleanupOldData(object state)
        {
            try
            {
                var cutoffDate = DateTime.UtcNow.Subtract(_config.DataRetentionPeriod);

                // Remove old processed data
                var oldDataKeys = _processedData.Values
                    .Where(d => d.CreatedAt < cutoffDate && d.SyncedToCloud)
                    .Select(d => d.DataId)
                    .ToList();

                foreach (var key in oldDataKeys)
                {
                    _processedData.TryRemove(key, out _);
                }

                // Clean up local storage
                await _localStorage.CleanupOldDataAsync(cutoffDate);

                _logger.LogInformation("Cleaned up {Count} old data items", oldDataKeys.Count);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error during data cleanup");
            }
        }

        public async Task<EdgeNodeStatus> GetStatusAsync()
        {
            return new EdgeNodeStatus
            {
                NodeId = _config.EdgeNodeId,
                IsOnline = true,
                DeviceCount = _devices.Count,
                PendingWorkloads = _workloadQueue.Count,
                ProcessedDataCount = _processedData.Count,
                AvailableStorage = await _localStorage.GetAvailableSpaceAsync(),
                LastSyncTime = DateTime.UtcNow // In real implementation, track this
            };
        }
    }

    // Edge Node Status
    public class EdgeNodeStatus
    {
        public string NodeId { get; set; }
        public bool IsOnline { get; set; }
        public int DeviceCount { get; set; }
        public int PendingWorkloads { get; set; }
        public int ProcessedDataCount { get; set; }
        public long AvailableStorage { get; set; }
        public DateTime LastSyncTime { get; set; }
    }

    // Workload Executor Interface
    public interface IWorkloadExecutor
    {
        Task<object> ExecuteAsync(Workload workload);
    }

    // Data Processor Interface
    public interface IDataProcessor
    {
        Task<object> ProcessAsync(object rawData, string dataType);
    }

    // Cloud Sync Service Interface
    public interface ICloudSyncService
    {
        Task<bool> IsConnectedAsync();
        Task SyncDataAsync(ProcessedData data);
        Task SyncWorkloadResultAsync(Workload workload);
        Task NotifyDeviceRegistrationAsync(EdgeDevice device);
    }

    // Local Storage Interface
    public interface ILocalStorage
    {
        Task InitializeAsync();
        Task StoreDataAsync(ProcessedData data);
        Task<ProcessedData> RetrieveDataAsync(string dataId);
        Task CleanupOldDataAsync(DateTime cutoffDate);
        Task<long> GetAvailableSpaceAsync();
    }

    // Sample Workload Executor Implementation
    public class SampleWorkloadExecutor : IWorkloadExecutor
    {
        private readonly ILogger<SampleWorkloadExecutor> _logger;

        public SampleWorkloadExecutor(ILogger<SampleWorkloadExecutor> logger)
        {
            _logger = logger;
        }

        public async Task<object> ExecuteAsync(Workload workload)
        {
            _logger.LogInformation("Executing workload {WorkloadId} of type {Type}", workload.WorkloadId, workload.Type);

            // Simulate processing time
            await Task.Delay(TimeSpan.FromSeconds(1));

            return workload.Type switch
            {
                "data-processing" => ProcessData(workload),
                "analytics" => PerformAnalytics(workload),
                "ml-inference" => PerformMLInference(workload),
                _ => throw new NotSupportedException($"Workload type {workload.Type} not supported")
            };
        }

        private object ProcessData(Workload workload)
        {
            // Simulate data processing
            var data = Encoding.UTF8.GetString(workload.InputData);
            return new { processed_data = data.ToUpper(), timestamp = DateTime.UtcNow };
        }

        private object PerformAnalytics(Workload workload)
        {
            // Simulate analytics processing
            var random = new Random();
            return new { 
                average = random.NextDouble() * 100,
                min = random.NextDouble() * 50,
                max = random.NextDouble() * 150,
                count = random.Next(100, 1000)
            };
        }

        private object PerformMLInference(Workload workload)
        {
            // Simulate ML inference
            var random = new Random();
            return new { 
                prediction = random.NextDouble(),
                confidence = random.NextDouble(),
                model_version = "1.0.0"
            };
        }
    }

    // Sample Data Processor Implementation
    public class SampleDataProcessor : IDataProcessor
    {
        private readonly ILogger<SampleDataProcessor> _logger;

        public SampleDataProcessor(ILogger<SampleDataProcessor> logger)
        {
            _logger = logger;
        }

        public async Task<object> ProcessAsync(object rawData, string dataType)
        {
            _logger.LogInformation("Processing {DataType} data", dataType);

            return dataType switch
            {
                "temperature" => ProcessTemperature(rawData),
                "humidity" => ProcessHumidity(rawData),
                "motion" => ProcessMotion(rawData),
                _ => rawData // Return as-is for unknown types
            };
        }

        private object ProcessTemperature(object rawData)
        {
            if (rawData is double temp)
            {
                return new { 
                    temperature_celsius = temp,
                    temperature_fahrenheit = (temp * 9/5) + 32,
                    status = temp > 30 ? "hot" : temp < 10 ? "cold" : "normal"
                };
            }
            return rawData;
        }

        private object ProcessHumidity(object rawData)
        {
            if (rawData is double humidity)
            {
                return new { 
                    humidity_percentage = humidity,
                    status = humidity > 70 ? "high" : humidity < 30 ? "low" : "normal"
                };
            }
            return rawData;
        }

        private object ProcessMotion(object rawData)
        {
            if (rawData is bool motion)
            {
                return new { 
                    motion_detected = motion,
                    timestamp = DateTime.UtcNow
                };
            }
            return rawData;
        }
    }

    // Sample Cloud Sync Service Implementation
    public class SampleCloudSyncService : ICloudSyncService
    {
        private readonly ILogger<SampleCloudSyncService> _logger;
        private readonly HttpClient _httpClient;
        private readonly string _cloudEndpoint;

        public SampleCloudSyncService(ILogger<SampleCloudSyncService> logger, HttpClient httpClient, string cloudEndpoint)
        {
            _logger = logger;
            _httpClient = httpClient;
            _cloudEndpoint = cloudEndpoint;
        }

        public async Task<bool> IsConnectedAsync()
        {
            try
            {
                var response = await _httpClient.GetAsync($"{_cloudEndpoint}/health");
                return response.IsSuccessStatusCode;
            }
            catch
            {
                return false;
            }
        }

        public async Task SyncDataAsync(ProcessedData data)
        {
            try
            {
                var json = JsonSerializer.Serialize(data);
                var content = new StringContent(json, Encoding.UTF8, "application/json");
                var response = await _httpClient.PostAsync($"{_cloudEndpoint}/data", content);

                if (response.IsSuccessStatusCode)
                {
                    _logger.LogInformation("Data {DataId} synced to cloud", data.DataId);
                }
                else
                {
                    _logger.LogWarning("Failed to sync data {DataId} to cloud", data.DataId);
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error syncing data {DataId} to cloud", data.DataId);
            }
        }

        public async Task SyncWorkloadResultAsync(Workload workload)
        {
            try
            {
                var json = JsonSerializer.Serialize(workload);
                var content = new StringContent(json, Encoding.UTF8, "application/json");
                var response = await _httpClient.PostAsync($"{_cloudEndpoint}/workloads", content);

                if (response.IsSuccessStatusCode)
                {
                    _logger.LogInformation("Workload {WorkloadId} result synced to cloud", workload.WorkloadId);
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error syncing workload {WorkloadId} result to cloud", workload.WorkloadId);
            }
        }

        public async Task NotifyDeviceRegistrationAsync(EdgeDevice device)
        {
            try
            {
                var json = JsonSerializer.Serialize(device);
                var content = new StringContent(json, Encoding.UTF8, "application/json");
                await _httpClient.PostAsync($"{_cloudEndpoint}/devices", content);

                _logger.LogInformation("Device {DeviceId} registration notified to cloud", device.DeviceId);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error notifying device {DeviceId} registration to cloud", device.DeviceId);
            }
        }
    }

    // Sample Local Storage Implementation
    public class SampleLocalStorage : ILocalStorage
    {
        private readonly ILogger<SampleLocalStorage> _logger;
        private readonly ConcurrentDictionary<string, ProcessedData> _storage;
        private readonly long _maxStorageSize;
        private long _currentStorageSize;

        public SampleLocalStorage(ILogger<SampleLocalStorage> logger, long maxStorageSize = 1000 * 1024 * 1024) // 1GB default
        {
            _logger = logger;
            _storage = new ConcurrentDictionary<string, ProcessedData>();
            _maxStorageSize = maxStorageSize;
            _currentStorageSize = 0;
        }

        public async Task InitializeAsync()
        {
            _logger.LogInformation("Local storage initialized with {MaxSize} bytes capacity", _maxStorageSize);
            await Task.CompletedTask;
        }

        public async Task StoreDataAsync(ProcessedData data)
        {
            var dataSize = EstimateDataSize(data);

            if (_currentStorageSize + dataSize > _maxStorageSize)
            {
                _logger.LogWarning("Storage full, cannot store data {DataId}", data.DataId);
                return;
            }

            _storage.TryAdd(data.DataId, data);
            _currentStorageSize += dataSize;

            _logger.LogInformation("Data {DataId} stored locally, size: {Size} bytes", data.DataId, dataSize);
            await Task.CompletedTask;
        }

        public async Task<ProcessedData> RetrieveDataAsync(string dataId)
        {
            _storage.TryGetValue(dataId, out var data);
            await Task.CompletedTask;
            return data;
        }

        public async Task CleanupOldDataAsync(DateTime cutoffDate)
        {
            var oldDataKeys = _storage.Values
                .Where(d => d.CreatedAt < cutoffDate)
                .Select(d => d.DataId)
                .ToList();

            foreach (var key in oldDataKeys)
            {
                if (_storage.TryRemove(key, out var data))
                {
                    _currentStorageSize -= EstimateDataSize(data);
                }
            }

            _logger.LogInformation("Cleaned up {Count} old data items from local storage", oldDataKeys.Count);
            await Task.CompletedTask;
        }

        public async Task<long> GetAvailableSpaceAsync()
        {
            await Task.CompletedTask;
            return _maxStorageSize - _currentStorageSize;
        }

        private long EstimateDataSize(ProcessedData data)
        {
            // Simple size estimation
            return 1024; // Assume 1KB per data item
        }
    }

    // Program Example
    public class Program
    {
        public static async Task Main(string[] args)
        {
            // Configure edge computing
            var config = new EdgeConfig
            {
                EdgeNodeId = "edge-node-001",
                CloudEndpoint = "https://cloud.example.com",
                SyncInterval = TimeSpan.FromMinutes(5),
                MaxLocalStorageSize = 1000,
                EnableOfflineMode = true,
                DataRetentionPeriod = TimeSpan.FromDays(7),
                MaxConcurrentWorkloads = 10
            };

            var loggerFactory = LoggerFactory.Create(builder => builder.AddConsole());
            var httpClient = new HttpClient();

            var workloadExecutor = new SampleWorkloadExecutor(loggerFactory.CreateLogger<SampleWorkloadExecutor>());
            var dataProcessor = new SampleDataProcessor(loggerFactory.CreateLogger<SampleDataProcessor>());
            var cloudSync = new SampleCloudSyncService(loggerFactory.CreateLogger<SampleCloudSyncService>(), httpClient, config.CloudEndpoint);
            var localStorage = new SampleLocalStorage(loggerFactory.CreateLogger<SampleLocalStorage>(), config.MaxLocalStorageSize * 1024 * 1024);

            var edgeNode = new EdgeNodeService(
                loggerFactory.CreateLogger<EdgeNodeService>(),
                config,
                workloadExecutor,
                dataProcessor,
                cloudSync,
                localStorage);

            // Start edge node
            await edgeNode.StartAsync();

            // Register sample devices
            var devices = new[]
            {
                new EdgeDevice { DeviceId = "temp-sensor-001", DeviceType = "temperature-sensor", Location = "room-1" },
                new EdgeDevice { DeviceId = "motion-sensor-001", DeviceType = "motion-sensor", Location = "entrance" }
            };

            foreach (var device in devices)
            {
                await edgeNode.RegisterDeviceAsync(device);
            }

            // Process sample data
            var random = new Random();
            for (int i = 0; i < 5; i++)
            {
                var temperature = random.NextDouble() * 40; // 0-40°C
                await edgeNode.ProcessDeviceDataAsync("temp-sensor-001", temperature, "temperature");

                var motion = random.Next(0, 2) == 1;
                await edgeNode.ProcessDeviceDataAsync("motion-sensor-001", motion, "motion");

                await Task.Delay(1000);
            }

            // Submit sample workloads
            var workloads = new[]
            {
                new Workload { Type = "data-processing", Priority = WorkloadPriority.Normal, InputData = Encoding.UTF8.GetBytes("sample data") },
                new Workload { Type = "analytics", Priority = WorkloadPriority.High, InputData = Encoding.UTF8.GetBytes("analytics data") },
                new Workload { Type = "ml-inference", Priority = WorkloadPriority.Critical, InputData = Encoding.UTF8.GetBytes("ml data") }
            };

            foreach (var workload in workloads)
            {
                await edgeNode.SubmitWorkloadAsync(workload);
            }

            // Get status
            var status = await edgeNode.GetStatusAsync();
            Console.WriteLine($"Edge Node Status: {JsonSerializer.Serialize(status, new JsonSerializerOptions { WriteIndented = true })}");

            // Keep running for demonstration
            Console.WriteLine("Edge computing node is running. Press any key to exit.");
            Console.ReadKey();

            await edgeNode.StopAsync();
        }
    }
}

Summary

88. Auto-Scaling Implementation

Key Components: - Metrics Collection: CPU, memory, request count, response time monitoring - Scaling Policies: Configurable thresholds and rules - Resource Management: Azure-specific implementation for provisioning/deprovisioning - Cooldown Periods: Prevents rapid scaling oscillations - Health Monitoring: Continuous evaluation of instance health

C# Features Demonstrated: - Concurrent collections for thread-safe operations - Timer-based periodic evaluation - Dependency injection with interfaces - Async/await patterns for scalability - Comprehensive logging and error handling

89. CDN Architecture Design

Key Components: - Edge Servers: Distributed geographically - Geographic Load Balancing: Distance-based server selection - Caching Strategy: Multi-level caching with TTL - Health Monitoring: Continuous edge server health checks - Fallback Mechanisms: Origin server fallback

C# Features Demonstrated: - Geographic distance calculations - Memory caching with size limits - HTTP client management - JSON serialization for data transfer - Concurrent operations for high performance

90. Edge Computing Implementation

Key Components: - Edge Nodes: Local processing units - Device Management: IoT device registration and monitoring - Workload Distribution: Priority-based task execution - Local Storage: Offline-capable data storage - Cloud Synchronization: Periodic data sync with central cloud

C# Features Demonstrated: - Semaphore for concurrency control - Queue-based workload processing - Timer-based periodic tasks - Offline mode capabilities - Comprehensive status monitoring

Each implementation follows enterprise-level patterns with: - Separation of Concerns: Clear interface boundaries - Dependency Injection: Testable and maintainable code - Error Handling: Comprehensive exception management - Logging: Structured logging for monitoring - Configuration: Externalized configuration management - Async Patterns: Non-blocking operations for scalability

These examples demonstrate real-world architectural patterns that would be expected in a technical architect role, showing both theoretical understanding and practical implementation skills.

Technical Architect Interview: System Design Questions & Answers

91. How would you design Twitter/X?

System Overview

Twitter/X is a real-time social media platform that allows users to post short messages (tweets), follow other users, and engage with content through likes, retweets, and replies.

Key Requirements

• Functional: Post tweets, follow users, like/retweet, timeline generation
• Non-Functional: High availability, low latency, scalability, consistency
• Scale: 330M+ users, 500M+ tweets/day, 1M+ requests/second

Architecture Components

1. Core Services

• User Service: Authentication, profiles, following relationships
• Tweet Service: Tweet creation, storage, retrieval
• Timeline Service: Generate home timelines for users
• Notification Service: Real-time notifications
• Media Service: Image/video upload and storage

2. Data Storage

• User Data: PostgreSQL (profiles, relationships)
• Tweets: Distributed database (Cassandra/DynamoDB)
• Timeline: Redis cache + database
• Media: Object storage (S3/Blob Storage)

3. Caching Strategy

• Redis: User sessions, trending topics, timeline cache
• CDN: Static content, media files
• Application Cache: Frequently accessed user data

C# Implementation Example

// Tweet Service
public class TweetService
{
    private readonly ITweetRepository _tweetRepository;
    private readonly ICacheService _cacheService;
    private readonly INotificationService _notificationService;

    public async Task<Tweet> CreateTweetAsync(CreateTweetRequest request)
    {
        var tweet = new Tweet
        {
            Id = Guid.NewGuid(),
            UserId = request.UserId,
            Content = request.Content,
            CreatedAt = DateTime.UtcNow,
            MediaUrls = request.MediaUrls
        };

        await _tweetRepository.CreateAsync(tweet);

        // Invalidate timeline cache for followers
        await InvalidateTimelineCacheForFollowers(request.UserId);

        // Send notifications to followers
        await _notificationService.NotifyFollowersAsync(tweet);

        return tweet;
    }

    public async Task<List<Tweet>> GetUserTimelineAsync(Guid userId, int page = 1, int pageSize = 20)
    {
        var cacheKey = $"timeline:{userId}:{page}";
        var cachedTimeline = await _cacheService.GetAsync<List<Tweet>>(cacheKey);

        if (cachedTimeline != null)
            return cachedTimeline;

        var timeline = await _tweetRepository.GetUserTimelineAsync(userId, page, pageSize);
        await _cacheService.SetAsync(cacheKey, timeline, TimeSpan.FromMinutes(5));

        return timeline;
    }
}

// Timeline Service
public class TimelineService
{
    private readonly IUserRepository _userRepository;
    private readonly ITweetRepository _tweetRepository;
    private readonly ICacheService _cacheService;

    public async Task<List<Tweet>> GenerateHomeTimelineAsync(Guid userId, int page = 1, int pageSize = 20)
    {
        var following = await _userRepository.GetFollowingAsync(userId);
        var tweetIds = following.SelectMany(u => u.TweetIds).OrderByDescending(id => id);

        var tweets = await _tweetRepository.GetTweetsByIdsAsync(tweetIds.Skip((page - 1) * pageSize).Take(pageSize));
        return tweets.ToList();
    }
}

---

92. How do you design a URL shortener?

System Overview

A URL shortener converts long URLs into short, manageable links while maintaining the ability to redirect users to the original URL.

Key Requirements

• Functional: Shorten URLs, redirect to original URLs, track analytics
• Non-Functional: High availability, low latency, scalability
• Scale: 100M+ URLs, 1B+ redirects/day

Architecture Components

1. Core Services

• URL Shortening Service: Generate short URLs
• Redirect Service: Handle redirects to original URLs
• Analytics Service: Track clicks and metrics
• User Service: User management and custom URLs

2. Data Storage

• URL Mappings: Key-value store (Redis) + database (PostgreSQL)
• Analytics: Time-series database (InfluxDB)
• User Data: PostgreSQL

3. URL Generation Strategy

• Hash-based: MD5/SHA + base62 encoding
• Sequential: Auto-incrementing IDs
• Custom: User-defined short codes

C# Implementation Example

public class UrlShortenerService
{
    private readonly IUrlRepository _urlRepository;
    private readonly ICacheService _cacheService;
    private readonly IAnalyticsService _analyticsService;
    private const string ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";

    public async Task<string> ShortenUrlAsync(string originalUrl, string customCode = null)
    {
        if (!string.IsNullOrEmpty(customCode))
        {
            if (await _urlRepository.ExistsAsync(customCode))
                throw new InvalidOperationException("Custom code already exists");

            await _urlRepository.CreateAsync(new UrlMapping
            {
                ShortCode = customCode,
                OriginalUrl = originalUrl,
                CreatedAt = DateTime.UtcNow
            });

            return customCode;
        }

        var shortCode = GenerateShortCode();
        while (await _urlRepository.ExistsAsync(shortCode))
        {
            shortCode = GenerateShortCode();
        }

        await _urlRepository.CreateAsync(new UrlMapping
        {
            ShortCode = shortCode,
            OriginalUrl = originalUrl,
            CreatedAt = DateTime.UtcNow
        });

        return shortCode;
    }

    public async Task<string> GetOriginalUrlAsync(string shortCode)
    {
        // Check cache first
        var cachedUrl = await _cacheService.GetAsync<string>($"url:{shortCode}");
        if (!string.IsNullOrEmpty(cachedUrl))
        {
            await _analyticsService.TrackClickAsync(shortCode);
            return cachedUrl;
        }

        var urlMapping = await _urlRepository.GetByShortCodeAsync(shortCode);
        if (urlMapping == null)
            throw new NotFoundException("URL not found");

        // Cache for 24 hours
        await _cacheService.SetAsync($"url:{shortCode}", urlMapping.OriginalUrl, TimeSpan.FromHours(24));

        await _analyticsService.TrackClickAsync(shortCode);
        return urlMapping.OriginalUrl;
    }

    private string GenerateShortCode()
    {
        var random = new Random();
        var shortCode = new char[6];

        for (int i = 0; i < 6; i++)
        {
            shortCode[i] = ALPHABET[random.Next(ALPHABET.Length)];
        }

        return new string(shortCode);
    }
}

public class AnalyticsService
{
    private readonly IAnalyticsRepository _analyticsRepository;

    public async Task TrackClickAsync(string shortCode)
    {
        var click = new UrlClick
        {
            ShortCode = shortCode,
            Timestamp = DateTime.UtcNow,
            IpAddress = GetClientIpAddress(),
            UserAgent = GetUserAgent()
        };

        await _analyticsRepository.TrackClickAsync(click);
    }

    public async Task<UrlAnalytics> GetAnalyticsAsync(string shortCode, DateTime from, DateTime to)
    {
        return await _analyticsRepository.GetAnalyticsAsync(shortCode, from, to);
    }
}

---

93. How would you design a ride-sharing app like Uber?

System Overview

Uber is a ride-sharing platform that connects riders with drivers, handles payments, and provides real-time tracking and matching.

Key Requirements

• Functional: User registration, ride booking, driver matching, payment processing, real-time tracking
• Non-Functional: High availability, low latency, real-time updates, scalability
• Scale: 100M+ users, 15M+ rides/day, real-time location updates

Architecture Components

1. Core Services

• User Service: Rider and driver management
• Matching Service: Driver-rider matching algorithm
• Location Service: Real-time GPS tracking
• Payment Service: Payment processing and billing
• Notification Service: Push notifications
• Rating Service: Driver and rider ratings

2. Data Storage

• User Data: PostgreSQL
• Ride Data: Distributed database (Cassandra)
• Location Data: Time-series database (InfluxDB)
• Payment Data: PostgreSQL with encryption
• Real-time Data: Redis

3. Real-time Communication

• WebSocket: Real-time location updates
• Message Queue: Event-driven architecture
• Push Notifications: Mobile notifications

C# Implementation Example

public class RideMatchingService
{
    private readonly ILocationService _locationService;
    private readonly IDriverService _driverService;
    private readonly INotificationService _notificationService;
    private readonly ICacheService _cacheService;

    public async Task<Driver> FindNearestDriverAsync(Location pickupLocation, VehicleType vehicleType)
    {
        var nearbyDrivers = await _locationService.GetNearbyDriversAsync(pickupLocation, 5.0); // 5km radius
        var availableDrivers = nearbyDrivers.Where(d => d.IsAvailable && d.VehicleType == vehicleType);

        if (!availableDrivers.Any())
            return null;

        // Find the closest driver
        var nearestDriver = availableDrivers
            .OrderBy(d => CalculateDistance(pickupLocation, d.CurrentLocation))
            .First();

        return nearestDriver;
    }

    public async Task<Ride> CreateRideAsync(CreateRideRequest request)
    {
        var driver = await FindNearestDriverAsync(request.PickupLocation, request.VehicleType);
        if (driver == null)
            throw new NoDriverAvailableException();

        var ride = new Ride
        {
            Id = Guid.NewGuid(),
            RiderId = request.RiderId,
            DriverId = driver.Id,
            PickupLocation = request.PickupLocation,
            DropoffLocation = request.DropoffLocation,
            Status = RideStatus.Requested,
            CreatedAt = DateTime.UtcNow
        };

        await _rideRepository.CreateAsync(ride);

        // Notify driver
        await _notificationService.NotifyDriverAsync(driver.Id, ride);

        // Cache ride status
        await _cacheService.SetAsync($"ride:{ride.Id}", ride, TimeSpan.FromHours(1));

        return ride;
    }

    public async Task UpdateRideStatusAsync(Guid rideId, RideStatus status)
    {
        var ride = await _rideRepository.GetByIdAsync(rideId);
        ride.Status = status;
        ride.UpdatedAt = DateTime.UtcNow;

        await _rideRepository.UpdateAsync(ride);

        // Notify relevant parties
        await _notificationService.NotifyRideStatusChangeAsync(ride);

        // Update cache
        await _cacheService.SetAsync($"ride:{rideId}", ride, TimeSpan.FromHours(1));
    }
}

public class LocationService
{
    private readonly ILocationRepository _locationRepository;
    private readonly ICacheService _cacheService;

    public async Task UpdateDriverLocationAsync(Guid driverId, Location location)
    {
        var driverLocation = new DriverLocation
        {
            DriverId = driverId,
            Latitude = location.Latitude,
            Longitude = location.Longitude,
            Timestamp = DateTime.UtcNow
        };

        await _locationRepository.UpdateDriverLocationAsync(driverLocation);

        // Cache current location
        await _cacheService.SetAsync($"driver:location:{driverId}", location, TimeSpan.FromMinutes(1));
    }

    public async Task<List<Driver>> GetNearbyDriversAsync(Location location, double radiusKm)
    {
        var cacheKey = $"nearby:{location.Latitude}:{location.Longitude}:{radiusKm}";
        var cachedDrivers = await _cacheService.GetAsync<List<Driver>>(cacheKey);

        if (cachedDrivers != null)
            return cachedDrivers;

        var nearbyDrivers = await _locationRepository.GetDriversWithinRadiusAsync(location, radiusKm);

        // Cache for 30 seconds
        await _cacheService.SetAsync(cacheKey, nearbyDrivers, TimeSpan.FromSeconds(30));

        return nearbyDrivers;
    }
}

---

94. How do you design a video streaming platform like Netflix?

System Overview

Netflix is a video streaming platform that delivers high-quality video content to millions of users worldwide with adaptive bitrate streaming.

Key Requirements

• Functional: Video upload, encoding, streaming, user management, recommendations
• Non-Functional: High bandwidth, low latency, global CDN, scalability
• Scale: 200M+ subscribers, 1B+ hours watched/day, global distribution

Architecture Components

1. Core Services

• Content Management Service: Video metadata and catalog
• Encoding Service: Video transcoding and compression
• Streaming Service: Video delivery and adaptive bitrate
• Recommendation Service: Content recommendations
• User Service: User profiles and preferences
• Analytics Service: Viewing analytics

2. Data Storage

• Video Files: Object storage (S3) with CDN
• Metadata: PostgreSQL
• User Data: Distributed database (Cassandra)
• Analytics: Data warehouse (Redshift/Snowflake)
• Cache: Redis for session and metadata

3. CDN Strategy

• Global CDN: Multiple edge locations
• Adaptive Bitrate: HLS/DASH streaming
• Geographic Distribution: Regional content delivery

C# Implementation Example

public class VideoStreamingService
{
    private readonly IVideoRepository _videoRepository;
    private readonly IEncodingService _encodingService;
    private readonly ICDNService _cdnService;
    private readonly IRecommendationService _recommendationService;

    public async Task<VideoStream> GetVideoStreamAsync(Guid videoId, Guid userId, string quality)
    {
        var video = await _videoRepository.GetByIdAsync(videoId);
        if (video == null)
            throw new NotFoundException("Video not found");

        // Get user's preferred quality or auto-detect based on bandwidth
        var streamQuality = await DetermineOptimalQualityAsync(userId, quality);

        var streamUrl = await _cdnService.GetStreamUrlAsync(videoId, streamQuality);

        // Track viewing analytics
        await TrackViewingAsync(videoId, userId, streamQuality);

        return new VideoStream
        {
            VideoId = videoId,
            StreamUrl = streamUrl,
            Quality = streamQuality,
            Duration = video.Duration,
            Subtitles = await GetSubtitlesAsync(videoId)
        };
    }

    public async Task<List<Video>> GetRecommendationsAsync(Guid userId, int limit = 20)
    {
        var userProfile = await _userRepository.GetProfileAsync(userId);
        var viewingHistory = await _analyticsRepository.GetViewingHistoryAsync(userId);

        var recommendations = await _recommendationService.GetRecommendationsAsync(
            userProfile, viewingHistory, limit);

        return recommendations;
    }

    private async Task<string> DetermineOptimalQualityAsync(Guid userId, string requestedQuality)
    {
        // Check user's bandwidth and device capabilities
        var userBandwidth = await GetUserBandwidthAsync(userId);
        var deviceCapabilities = await GetDeviceCapabilitiesAsync(userId);

        var availableQualities = new[] { "1080p", "720p", "480p", "360p" };

        foreach (var quality in availableQualities)
        {
            if (CanSupportQuality(userBandwidth, deviceCapabilities, quality))
                return quality;
        }

        return "360p"; // Fallback to lowest quality
    }
}

public class EncodingService
{
    private readonly IQueueService _queueService;
    private readonly IStorageService _storageService;

    public async Task<EncodingJob> EncodeVideoAsync(Guid videoId, string sourceUrl)
    {
        var encodingJob = new EncodingJob
        {
            Id = Guid.NewGuid(),
            VideoId = videoId,
            Status = EncodingStatus.Queued,
            CreatedAt = DateTime.UtcNow
        };

        await _encodingRepository.CreateAsync(encodingJob);

        // Queue encoding job
        await _queueService.EnqueueAsync("video-encoding", new EncodingRequest
        {
            JobId = encodingJob.Id,
            VideoId = videoId,
            SourceUrl = sourceUrl,
            OutputFormats = new[] { "hls", "dash" },
            Qualities = new[] { "1080p", "720p", "480p", "360p" }
        });

        return encodingJob;
    }

    public async Task ProcessEncodingJobAsync(EncodingRequest request)
    {
        try
        {
            // Download source video
            var sourcePath = await DownloadVideoAsync(request.SourceUrl);

            // Encode to different formats and qualities
            foreach (var format in request.OutputFormats)
            {
                foreach (var quality in request.Qualities)
                {
                    var outputPath = await EncodeVideoAsync(sourcePath, format, quality);
                    var cdnUrl = await _storageService.UploadToCDNAsync(outputPath, request.VideoId, format, quality);

                    await _videoRepository.AddStreamUrlAsync(request.VideoId, format, quality, cdnUrl);
                }
            }

            await _encodingRepository.UpdateStatusAsync(request.JobId, EncodingStatus.Completed);
        }
        catch (Exception ex)
        {
            await _encodingRepository.UpdateStatusAsync(request.JobId, EncodingStatus.Failed);
            throw;
        }
    }
}

---

95. How would you design an e-commerce platform like Amazon?

System Overview

Amazon is a comprehensive e-commerce platform that handles product catalog, inventory, orders, payments, and recommendations.

Key Requirements

• Functional: Product catalog, inventory management, order processing, payment, shipping, reviews
• Non-Functional: High availability, consistency, scalability, security
• Scale: 300M+ users, 1M+ products, 100K+ orders/day

Architecture Components

1. Core Services

• Catalog Service: Product information and search
• Inventory Service: Stock management
• Order Service: Order processing and fulfillment
• Payment Service: Payment processing
• Shipping Service: Delivery tracking
• Recommendation Service: Product recommendations
• Review Service: Product reviews and ratings

2. Data Storage

• Product Catalog: Search engine (Elasticsearch) + database
• Inventory: Distributed database (Cassandra)
• Orders: PostgreSQL with read replicas
• User Data: PostgreSQL
• Analytics: Data warehouse

3. Search and Recommendations

• Search Engine: Elasticsearch with faceted search
• Recommendations: Machine learning models
• Cache: Redis for product data and sessions

C# Implementation Example

public class ProductCatalogService
{
    private readonly IProductRepository _productRepository;
    private readonly ISearchService _searchService;
    private readonly ICacheService _cacheService;

    public async Task<List<Product>> SearchProductsAsync(SearchRequest request)
    {
        var cacheKey = $"search:{request.Query}:{request.Category}:{request.Page}";
        var cachedResults = await _cacheService.GetAsync<List<Product>>(cacheKey);

        if (cachedResults != null)
            return cachedResults;

        var searchResults = await _searchService.SearchAsync(new SearchQuery
        {
            Query = request.Query,
            Category = request.Category,
            PriceRange = request.PriceRange,
            SortBy = request.SortBy,
            Page = request.Page,
            PageSize = request.PageSize
        });

        await _cacheService.SetAsync(cacheKey, searchResults, TimeSpan.FromMinutes(5));

        return searchResults;
    }

    public async Task<Product> GetProductAsync(Guid productId)
    {
        var cacheKey = $"product:{productId}";
        var cachedProduct = await _cacheService.GetAsync<Product>(cacheKey);

        if (cachedProduct != null)
            return cachedProduct;

        var product = await _productRepository.GetByIdAsync(productId);
        if (product == null)
            throw new NotFoundException("Product not found");

        await _cacheService.SetAsync(cacheKey, product, TimeSpan.FromHours(1));

        return product;
    }
}

public class OrderService
{
    private readonly IOrderRepository _orderRepository;
    private readonly IInventoryService _inventoryService;
    private readonly IPaymentService _paymentService;
    private readonly INotificationService _notificationService;

    public async Task<Order> CreateOrderAsync(CreateOrderRequest request)
    {
        // Validate inventory
        foreach (var item in request.Items)
        {
            var availableQuantity = await _inventoryService.GetAvailableQuantityAsync(item.ProductId);
            if (availableQuantity < item.Quantity)
                throw new InsufficientInventoryException(item.ProductId);
        }

        // Reserve inventory
        await _inventoryService.ReserveInventoryAsync(request.Items);

        // Process payment
        var paymentResult = await _paymentService.ProcessPaymentAsync(new PaymentRequest
        {
            Amount = CalculateTotal(request.Items),
            PaymentMethod = request.PaymentMethod,
            Currency = "USD"
        });

        if (!paymentResult.IsSuccessful)
        {
            await _inventoryService.ReleaseInventoryAsync(request.Items);
            throw new PaymentFailedException(paymentResult.ErrorMessage);
        }

        // Create order
        var order = new Order
        {
            Id = Guid.NewGuid(),
            UserId = request.UserId,
            Items = request.Items,
            TotalAmount = CalculateTotal(request.Items),
            Status = OrderStatus.Confirmed,
            PaymentId = paymentResult.PaymentId,
            CreatedAt = DateTime.UtcNow
        };

        await _orderRepository.CreateAsync(order);

        // Send confirmation
        await _notificationService.SendOrderConfirmationAsync(order);

        return order;
    }

    public async Task<Order> GetOrderAsync(Guid orderId, Guid userId)
    {
        var order = await _orderRepository.GetByIdAsync(orderId);
        if (order == null || order.UserId != userId)
            throw new NotFoundException("Order not found");

        return order;
    }
}

public class InventoryService
{
    private readonly IInventoryRepository _inventoryRepository;
    private readonly ICacheService _cacheService;

    public async Task<int> GetAvailableQuantityAsync(Guid productId)
    {
        var cacheKey = $"inventory:{productId}";
        var cachedQuantity = await _cacheService.GetAsync<int?>(cacheKey);

        if (cachedQuantity.HasValue)
            return cachedQuantity.Value;

        var inventory = await _inventoryRepository.GetByProductIdAsync(productId);
        var availableQuantity = inventory?.AvailableQuantity ?? 0;

        await _cacheService.SetAsync(cacheKey, availableQuantity, TimeSpan.FromMinutes(5));

        return availableQuantity;
    }

    public async Task ReserveInventoryAsync(List<OrderItem> items)
    {
        foreach (var item in items)
        {
            await _inventoryRepository.ReserveQuantityAsync(item.ProductId, item.Quantity);
            await _cacheService.DeleteAsync($"inventory:{item.ProductId}");
        }
    }
}

---

96. How do you design a social media platform like Facebook?

System Overview

Facebook is a social networking platform that connects people, shares content, and provides real-time communication.

Key Requirements

• Functional: User profiles, friend connections, posts, comments, messaging, news feed
• Non-Functional: High availability, real-time updates, scalability, privacy
• Scale: 2.9B+ users, 500M+ posts/day, real-time interactions

Architecture Components

1. Core Services

• User Service: User profiles and authentication
• Friend Service: Friend relationships and connections
• Post Service: Content creation and sharing
• News Feed Service: Personalized content feed
• Messaging Service: Real-time messaging
• Notification Service: Real-time notifications
• Privacy Service: Privacy controls and content visibility

2. Data Storage

• User Data: Distributed database (Cassandra)
• Posts: Distributed database with CDN for media
• Friendships: Graph database (Neo4j) + cache
• Messages: Real-time database (Firebase)
• News Feed: Cache + database
• Media: Object storage with CDN

3. Real-time Features

• WebSocket: Real-time messaging and notifications
• Push Notifications: Mobile notifications
• Live Streaming: Real-time video streaming

C# Implementation Example

public class NewsFeedService
{
    private readonly IPostRepository _postRepository;
    private readonly IFriendService _friendService;
    private readonly ICacheService _cacheService;
    private readonly IRecommendationService _recommendationService;

    public async Task<List<Post>> GetNewsFeedAsync(Guid userId, int page = 1, int pageSize = 20)
    {
        var cacheKey = $"newsfeed:{userId}:{page}";
        var cachedFeed = await _cacheService.GetAsync<List<Post>>(cacheKey);

        if (cachedFeed != null)
            return cachedFeed;

        // Get user's friends
        var friends = await _friendService.GetFriendsAsync(userId);
        var friendIds = friends.Select(f => f.Id).ToList();
        friendIds.Add(userId); // Include user's own posts

        // Get posts from friends
        var posts = await _postRepository.GetPostsByUserIdsAsync(friendIds, page, pageSize);

        // Apply privacy filters
        posts = await ApplyPrivacyFiltersAsync(posts, userId);

        // Sort by relevance and recency
        posts = await _recommendationService.SortByRelevanceAsync(posts, userId);

        await _cacheService.SetAsync(cacheKey, posts, TimeSpan.FromMinutes(5));

        return posts;
    }

    private async Task<List<Post>> ApplyPrivacyFiltersAsync(List<Post> posts, Guid userId)
    {
        var filteredPosts = new List<Post>();

        foreach (var post in posts)
        {
            if (await CanUserViewPostAsync(post, userId))
                filteredPosts.Add(post);
        }

        return filteredPosts;
    }

    private async Task<bool> CanUserViewPostAsync(Post post, Guid userId)
    {
        if (post.UserId == userId)
            return true;

        switch (post.Privacy)
        {
            case PrivacyLevel.Public:
                return true;
            case PrivacyLevel.Friends:
                return await _friendService.AreFriendsAsync(post.UserId, userId);
            case PrivacyLevel.FriendsOfFriends:
                return await _friendService.AreFriendsOfFriendsAsync(post.UserId, userId);
            default:
                return false;
        }
    }
}

public class MessagingService
{
    private readonly IMessageRepository _messageRepository;
    private readonly IWebSocketManager _webSocketManager;
    private readonly INotificationService _notificationService;

    public async Task<Message> SendMessageAsync(SendMessageRequest request)
    {
        var message = new Message
        {
            Id = Guid.NewGuid(),
            SenderId = request.SenderId,
            ReceiverId = request.ReceiverId,
            Content = request.Content,
            MessageType = request.MessageType,
            CreatedAt = DateTime.UtcNow
        };

        await _messageRepository.CreateAsync(message);

        // Send real-time notification
        await _webSocketManager.SendToUserAsync(request.ReceiverId, "new_message", message);

        // Send push notification if user is offline
        if (!await _webSocketManager.IsUserOnlineAsync(request.ReceiverId))
        {
            await _notificationService.SendPushNotificationAsync(request.ReceiverId, "New message", message.Content);
        }

        return message;
    }

    public async Task<List<Message>> GetConversationAsync(Guid user1Id, Guid user2Id, int page = 1, int pageSize = 50)
    {
        return await _messageRepository.GetConversationAsync(user1Id, user2Id, page, pageSize);
    }
}

public class PostService
{
    private readonly IPostRepository _postRepository;
    private readonly INotificationService _notificationService;
    private readonly ICacheService _cacheService;

    public async Task<Post> CreatePostAsync(CreatePostRequest request)
    {
        var post = new Post
        {
            Id = Guid.NewGuid(),
            UserId = request.UserId,
            Content = request.Content,
            MediaUrls = request.MediaUrls,
            Privacy = request.Privacy,
            CreatedAt = DateTime.UtcNow
        };

        await _postRepository.CreateAsync(post);

        // Invalidate news feed cache for friends
        await InvalidateNewsFeedCacheForFriends(request.UserId);

        // Notify friends about new post
        await NotifyFriendsAboutPostAsync(post);

        return post;
    }

    public async Task<Comment> AddCommentAsync(AddCommentRequest request)
    {
        var comment = new Comment
        {
            Id = Guid.NewGuid(),
            PostId = request.PostId,
            UserId = request.UserId,
            Content = request.Content,
            CreatedAt = DateTime.UtcNow
        };

        await _commentRepository.CreateAsync(comment);

        // Notify post owner about new comment
        var post = await _postRepository.GetByIdAsync(request.PostId);
        if (post.UserId != request.UserId)
        {
            await _notificationService.NotifyUserAsync(post.UserId, "new_comment", comment);
        }

        return comment;
    }
}

---

97. How would you design a search engine like Google?

System Overview

Google is a web search engine that indexes billions of web pages and provides relevant search results based on user queries.

Key Requirements

• Functional: Web crawling, indexing, search, ranking, suggestions
• Non-Functional: High availability, low latency, massive scalability, relevance
• Scale: 100B+ indexed pages, 5B+ searches/day, sub-second response times

Architecture Components

1. Core Services

• Crawler Service: Web page crawling and discovery
• Indexing Service: Document processing and indexing
• Search Service: Query processing and result retrieval
• Ranking Service: Result ranking and relevance scoring
• Suggestion Service: Query suggestions and autocomplete
• Analytics Service: Search analytics and user behavior

2. Data Storage

• Search Index: Distributed search engine (Elasticsearch/Solr)
• Web Pages: Distributed file system (GFS/HDFS)
• Metadata: Distributed database (Bigtable)
• Analytics: Data warehouse (BigQuery)
• Cache: Multi-level caching (Redis, CDN)

3. Search Algorithm

• PageRank: Link-based ranking
• Content Analysis: Text processing and relevance
• Machine Learning: Personalized ranking
• Real-time Updates: Fresh content indexing

C# Implementation Example

public class SearchService
{
    private readonly ISearchIndex _searchIndex;
    private readonly IRankingService _rankingService;
    private readonly ICacheService _cacheService;
    private readonly IAnalyticsService _analyticsService;

    public async Task<SearchResults> SearchAsync(SearchQuery query)
    {
        var cacheKey = $"search:{query.Query}:{query.Page}";
        var cachedResults = await _cacheService.GetAsync<SearchResults>(cacheKey);

        if (cachedResults != null)
        {
            await _analyticsService.TrackSearchAsync(query, cachedResults);
            return cachedResults;
        }

        // Process query
        var processedQuery = await ProcessQueryAsync(query);

        // Search index
        var rawResults = await _searchIndex.SearchAsync(processedQuery);

        // Apply ranking
        var rankedResults = await _rankingService.RankResultsAsync(rawResults, query);

        // Apply filters
        var filteredResults = ApplyFilters(rankedResults, query.Filters);

        var searchResults = new SearchResults
        {
            Query = query.Query,
            Results = filteredResults,
            TotalCount = rawResults.TotalCount,
            Page = query.Page,
            PageSize = query.PageSize
        };

        await _cacheService.SetAsync(cacheKey, searchResults, TimeSpan.FromMinutes(5));
        await _analyticsService.TrackSearchAsync(query, searchResults);

        return searchResults;
    }

    private async Task<ProcessedQuery> ProcessQueryAsync(SearchQuery query)
    {
        // Tokenize query
        var tokens = Tokenize(query.Query);

        // Remove stop words
        tokens = RemoveStopWords(tokens);

        // Apply stemming
        tokens = ApplyStemming(tokens);

        // Expand synonyms
        tokens = await ExpandSynonymsAsync(tokens);

        return new ProcessedQuery
        {
            OriginalQuery = query.Query,
            Tokens = tokens,
            QueryType = DetermineQueryType(query.Query)
        };
    }
}

public class WebCrawlerService
{
    private readonly IUrlQueue _urlQueue;
    private readonly IPageRepository _pageRepository;
    private readonly ILinkExtractor _linkExtractor;
    private readonly IRobotsTxtParser _robotsTxtParser;

    public async Task CrawlWebsiteAsync(string url)
    {
        // Check robots.txt
        if (!await IsAllowedToCrawlAsync(url))
            return;

        // Fetch page
        var page = await FetchPageAsync(url);
        if (page == null)
            return;

        // Extract content
        var extractedContent = await ExtractContentAsync(page);

        // Store page
        await _pageRepository.StoreAsync(new WebPage
        {
            Url = url,
            Title = extractedContent.Title,
            Content = extractedContent.Content,
            Links = extractedContent.Links,
            LastCrawled = DateTime.UtcNow
        });

        // Add new links to queue
        foreach (var link in extractedContent.Links)
        {
            await _urlQueue.EnqueueAsync(link);
        }
    }

    private async Task<PageContent> ExtractContentAsync(WebPage page)
    {
        // Parse HTML
        var htmlDocument = ParseHtml(page.RawContent);

        // Extract title
        var title = ExtractTitle(htmlDocument);

        // Extract main content
        var content = ExtractMainContent(htmlDocument);

        // Extract links
        var links = ExtractLinks(htmlDocument);

        return new PageContent
        {
            Title = title,
            Content = content,
            Links = links
        };
    }
}

public class RankingService
{
    private readonly IPageRankService _pageRankService;
    private readonly IContentAnalyzer _contentAnalyzer;
    private readonly IUserBehaviorAnalyzer _userBehaviorAnalyzer;

    public async Task<List<SearchResult>> RankResultsAsync(List<SearchResult> results, SearchQuery query)
    {
        var rankedResults = new List<RankedResult>();

        foreach (var result in results)
        {
            var score = await CalculateRelevanceScoreAsync(result, query);
            rankedResults.Add(new RankedResult
            {
                Result = result,
                Score = score
            });
        }

        // Sort by score
        rankedResults = rankedResults.OrderByDescending(r => r.Score).ToList();

        return rankedResults.Select(r => r.Result).ToList();
    }

    private async Task<double> CalculateRelevanceScoreAsync(SearchResult result, SearchQuery query)
    {
        var score = 0.0;

        // PageRank score
        var pageRankScore = await _pageRankService.GetPageRankAsync(result.Url);
        score += pageRankScore * 0.3;

        // Content relevance score
        var contentScore = await _contentAnalyzer.CalculateRelevanceAsync(result.Content, query.Query);
        score += contentScore * 0.4;

        // Freshness score
        var freshnessScore = CalculateFreshnessScore(result.LastUpdated);
        score += freshnessScore * 0.1;

        // User behavior score
        var behaviorScore = await _userBehaviorAnalyzer.GetClickThroughRateAsync(result.Url);
        score += behaviorScore * 0.2;

        return score;
    }
}

---

I'll act as a technical architect and provide comprehensive answers to these system design questions, including explanations and C# code examples where relevant.

98. How do you design a payment system like PayPal?

System Architecture Overview

A payment system like PayPal requires high availability, security, scalability, and compliance with financial regulations. Here's a comprehensive design:

Core Components

1. User Management System
2. Payment Processing Engine
3. Transaction Management
4. Security & Fraud Detection
5. Compliance & Reporting
6. Notification System

High-Level Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Web/Mobile    │    │   API Gateway   │    │   Load Balancer │
│     Clients     │───▶│   (Rate Limiting│───▶│   (Round Robin) │
└─────────────────┘    │   & Auth)       │    └─────────────────┘
                       └─────────────────┘              │
                                                        ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Notification  │    │   Transaction   │    │   Payment       │
│     Service     │◀───│   Service       │◀───│   Processor     │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                        │
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Fraud         │    │   User          │    │   Compliance    │
│   Detection     │    │   Management    │    │   Service       │
└─────────────────┘    └─────────────────┘    └─────────────────┘

C# Implementation Examples

1. Payment Transaction Model

public class PaymentTransaction
{
    public Guid Id { get; set; }
    public string SenderId { get; set; }
    public string RecipientId { get; set; }
    public decimal Amount { get; set; }
    public string Currency { get; set; }
    public PaymentStatus Status { get; set; }
    public PaymentMethod Method { get; set; }
    public DateTime CreatedAt { get; set; }
    public DateTime? CompletedAt { get; set; }
    public string Description { get; set; }
    public string TransactionReference { get; set; }
    public List<TransactionEvent> Events { get; set; } = new();
}

public enum PaymentStatus
{
    Pending,
    Processing,
    Completed,
    Failed,
    Cancelled,
    Refunded
}

public enum PaymentMethod
{
    CreditCard,
    DebitCard,
    BankTransfer,
    PayPalBalance,
    Crypto
}

2. Payment Processing Service

public interface IPaymentProcessor
{
    Task<PaymentResult> ProcessPaymentAsync(PaymentRequest request);
    Task<PaymentResult> RefundPaymentAsync(string transactionId, decimal amount);
}

public class PaymentProcessor : IPaymentProcessor
{
    private readonly ITransactionService _transactionService;
    private readonly IFraudDetectionService _fraudDetection;
    private readonly IComplianceService _complianceService;
    private readonly INotificationService _notificationService;
    private readonly ILogger<PaymentProcessor> _logger;

    public PaymentProcessor(
        ITransactionService transactionService,
        IFraudDetectionService fraudDetection,
        IComplianceService complianceService,
        INotificationService notificationService,
        ILogger<PaymentProcessor> logger)
    {
        _transactionService = transactionService;
        _fraudDetection = fraudDetection;
        _complianceService = complianceService;
        _notificationService = notificationService;
        _logger = logger;
    }

    public async Task<PaymentResult> ProcessPaymentAsync(PaymentRequest request)
    {
        try
        {
            // 1. Validate request
            var validationResult = await ValidatePaymentRequest(request);
            if (!validationResult.IsValid)
                return PaymentResult.Failed(validationResult.Errors);

            // 2. Fraud detection
            var fraudCheck = await _fraudDetection.CheckTransactionAsync(request);
            if (fraudCheck.IsSuspicious)
                return PaymentResult.Failed(new[] { "Transaction flagged for review" });

            // 3. Compliance check
            var complianceCheck = await _complianceService.ValidateTransactionAsync(request);
            if (!complianceCheck.IsCompliant)
                return PaymentResult.Failed(complianceCheck.Violations);

            // 4. Create transaction record
            var transaction = new PaymentTransaction
            {
                Id = Guid.NewGuid(),
                SenderId = request.SenderId,
                RecipientId = request.RecipientId,
                Amount = request.Amount,
                Currency = request.Currency,
                Status = PaymentStatus.Processing,
                Method = request.Method,
                CreatedAt = DateTime.UtcNow,
                Description = request.Description
            };

            await _transactionService.CreateTransactionAsync(transaction);

            // 5. Process payment based on method
            var paymentResult = await ProcessPaymentByMethodAsync(request, transaction);

            // 6. Update transaction status
            transaction.Status = paymentResult.Success ? PaymentStatus.Completed : PaymentStatus.Failed;
            transaction.CompletedAt = DateTime.UtcNow;
            await _transactionService.UpdateTransactionAsync(transaction);

            // 7. Send notifications
            await _notificationService.SendPaymentNotificationAsync(transaction);

            return paymentResult;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Error processing payment for request {RequestId}", request.Id);
            return PaymentResult.Failed(new[] { "Internal server error" });
        }
    }

    private async Task<PaymentResult> ProcessPaymentByMethodAsync(PaymentRequest request, PaymentTransaction transaction)
    {
        return request.Method switch
        {
            PaymentMethod.CreditCard => await ProcessCreditCardPaymentAsync(request, transaction),
            PaymentMethod.BankTransfer => await ProcessBankTransferAsync(request, transaction),
            PaymentMethod.PayPalBalance => await ProcessBalanceTransferAsync(request, transaction),
            _ => PaymentResult.Failed(new[] { "Unsupported payment method" })
        };
    }
}

3. Fraud Detection Service

public class FraudDetectionService : IFraudDetectionService
{
    private readonly IRedisCache _cache;
    private readonly IMachineLearningService _mlService;

    public async Task<FraudCheckResult> CheckTransactionAsync(PaymentRequest request)
    {
        var riskScore = 0.0;

        // Check for velocity (frequency) of transactions
        var recentTransactions = await GetRecentTransactionsAsync(request.SenderId);
        if (recentTransactions.Count > 10) // More than 10 transactions in last hour
            riskScore += 0.3;

        // Check for unusual amounts
        if (request.Amount > 10000) // Large amount
            riskScore += 0.2;

        // Check for geographic anomalies
        var senderLocation = await GetUserLocationAsync(request.SenderId);
        var recipientLocation = await GetUserLocationAsync(request.RecipientId);
        if (senderLocation.Country != recipientLocation.Country)
            riskScore += 0.1;

        // ML-based anomaly detection
        var mlScore = await _mlService.PredictFraudProbabilityAsync(request);
        riskScore += mlScore;

        return new FraudCheckResult
        {
            IsSuspicious = riskScore > 0.7,
            RiskScore = riskScore,
            Reasons = GetRiskReasons(riskScore)
        };
    }
}

Key Design Considerations

1. Security: End-to-end encryption, PCI DSS compliance, tokenization
2. Scalability: Microservices architecture, horizontal scaling
3. Availability: Multi-region deployment, failover mechanisms
4. Compliance: KYC/AML, regulatory reporting, audit trails
5. Performance: Caching, database optimization, async processing

---

99. How would you design a messaging app like WhatsApp?

System Architecture Overview

A messaging app requires real-time communication, message persistence, user management, and media handling. Here's the design:

Core Components

1. Real-time Communication Service
2. Message Storage & Retrieval
3. User Management & Authentication
4. Media Storage & Delivery
5. Push Notification Service
6. Group Chat Management

High-Level Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Mobile Apps   │    │   API Gateway   │    │   Load Balancer │
│   (iOS/Android) │───▶│   (WebSocket    │───▶│   (Round Robin) │
└─────────────────┘    │   & REST)       │    └─────────────────┘
                       └─────────────────┘              │
                                                        ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Push          │    │   Message       │    │   WebSocket     │
│   Notification  │◀───│   Service       │◀───│   Service       │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                        │
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Media         │    │   User          │    │   Group         │
│   Service       │    │   Management    │    │   Management    │
└─────────────────┘    └─────────────────┘    └─────────────────┘

C# Implementation Examples

1. Message Model

public class Message
{
    public Guid Id { get; set; }
    public string SenderId { get; set; }
    public string ChatId { get; set; }
    public MessageType Type { get; set; }
    public string Content { get; set; }
    public MessageStatus Status { get; set; }
    public DateTime CreatedAt { get; set; }
    public DateTime? DeliveredAt { get; set; }
    public DateTime? ReadAt { get; set; }
    public string? ReplyToMessageId { get; set; }
    public Dictionary<string, object> Metadata { get; set; } = new();
    public List<MessageAttachment> Attachments { get; set; } = new();
}

public enum MessageType
{
    Text,
    Image,
    Video,
    Audio,
    Document,
    Location,
    Contact
}

public enum MessageStatus
{
    Sent,
    Delivered,
    Read,
    Failed
}

public class MessageAttachment
{
    public Guid Id { get; set; }
    public string FileName { get; set; }
    public string FileUrl { get; set; }
    public long FileSize { get; set; }
    public string MimeType { get; set; }
    public string ThumbnailUrl { get; set; }
}

2. WebSocket Service for Real-time Communication

public class WebSocketService : IWebSocketService
{
    private readonly IConnectionManager _connectionManager;
    private readonly IMessageService _messageService;
    private readonly IUserService _userService;
    private readonly ILogger<WebSocketService> _logger;

    public async Task HandleWebSocketConnectionAsync(WebSocket webSocket, string userId)
    {
        var connection = new UserConnection
        {
            UserId = userId,
            WebSocket = webSocket,
            ConnectedAt = DateTime.UtcNow
        };

        _connectionManager.AddConnection(userId, connection);

        try
        {
            var buffer = new byte[1024 * 4];
            while (webSocket.State == WebSocketState.Open)
            {
                var result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);

                if (result.MessageType == WebSocketMessageType.Text)
                {
                    var message = Encoding.UTF8.GetString(buffer, 0, result.Count);
                    await ProcessIncomingMessageAsync(userId, message);
                }
                else if (result.MessageType == WebSocketMessageType.Close)
                {
                    break;
                }
            }
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "WebSocket error for user {UserId}", userId);
        }
        finally
        {
            _connectionManager.RemoveConnection(userId);
        }
    }

    public async Task SendMessageToUserAsync(string userId, Message message)
    {
        var connection = _connectionManager.GetConnection(userId);
        if (connection?.WebSocket.State == WebSocketState.Open)
        {
            var messageJson = JsonSerializer.Serialize(new
            {
                Type = "message",
                Data = message
            });

            var bytes = Encoding.UTF8.GetBytes(messageJson);
            await connection.WebSocket.SendAsync(new ArraySegment<byte>(bytes), WebSocketMessageType.Text, true, CancellationToken.None);
        }
        else
        {
            // User is offline, send push notification
            await SendPushNotificationAsync(userId, message);
        }
    }

    private async Task ProcessIncomingMessageAsync(string userId, string messageJson)
    {
        try
        {
            var messageData = JsonSerializer.Deserialize<IncomingMessageData>(messageJson);

            switch (messageData.Type)
            {
                case "send_message":
                    await ProcessSendMessageAsync(userId, messageData);
                    break;
                case "typing":
                    await BroadcastTypingStatusAsync(userId, messageData);
                    break;
                case "read_receipt":
                    await ProcessReadReceiptAsync(userId, messageData);
                    break;
            }
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Error processing incoming message from user {UserId}", userId);
        }
    }
}

3. Message Service

public class MessageService : IMessageService
{
    private readonly IMessageRepository _messageRepository;
    private readonly IWebSocketService _webSocketService;
    private readonly IMediaService _mediaService;
    private readonly IChatService _chatService;
    private readonly ILogger<MessageService> _logger;

    public async Task<Message> SendMessageAsync(SendMessageRequest request)
    {
        // Validate chat exists and user is participant
        var chat = await _chatService.GetChatAsync(request.ChatId);
        if (!chat.Participants.Contains(request.SenderId))
            throw new UnauthorizedAccessException("User not part of this chat");

        // Create message
        var message = new Message
        {
            Id = Guid.NewGuid(),
            SenderId = request.SenderId,
            ChatId = request.ChatId,
            Type = request.Type,
            Content = request.Content,
            Status = MessageStatus.Sent,
            CreatedAt = DateTime.UtcNow,
            ReplyToMessageId = request.ReplyToMessageId
        };

        // Handle attachments
        if (request.Attachments?.Any() == true)
        {
            foreach (var attachment in request.Attachments)
            {
                var uploadedFile = await _mediaService.UploadFileAsync(attachment);
                message.Attachments.Add(new MessageAttachment
                {
                    Id = Guid.NewGuid(),
                    FileName = uploadedFile.FileName,
                    FileUrl = uploadedFile.Url,
                    FileSize = uploadedFile.Size,
                    MimeType = uploadedFile.MimeType,
                    ThumbnailUrl = uploadedFile.ThumbnailUrl
                });
            }
        }

        // Save message
        await _messageRepository.SaveMessageAsync(message);

        // Send to all participants in real-time
        var participants = chat.Participants.Where(p => p != request.SenderId);
        foreach (var participantId in participants)
        {
            await _webSocketService.SendMessageToUserAsync(participantId, message);
        }

        return message;
    }

    public async Task<List<Message>> GetChatMessagesAsync(string chatId, string userId, int limit = 50, DateTime? before = null)
    {
        // Validate user has access to chat
        var chat = await _chatService.GetChatAsync(chatId);
        if (!chat.Participants.Contains(userId))
            throw new UnauthorizedAccessException("User not part of this chat");

        return await _messageRepository.GetMessagesAsync(chatId, limit, before);
    }

    public async Task MarkMessageAsReadAsync(string messageId, string userId)
    {
        var message = await _messageRepository.GetMessageAsync(messageId);
        if (message == null) return;

        // Validate user is recipient
        var chat = await _chatService.GetChatAsync(message.ChatId);
        if (!chat.Participants.Contains(userId) || message.SenderId == userId)
            return;

        message.Status = MessageStatus.Read;
        message.ReadAt = DateTime.UtcNow;
        await _messageRepository.UpdateMessageAsync(message);

        // Send read receipt to sender
        await _webSocketService.SendMessageToUserAsync(message.SenderId, new
        {
            Type = "read_receipt",
            MessageId = messageId,
            ReadBy = userId,
            ReadAt = message.ReadAt
        });
    }
}

4. Chat Management Service

public class ChatService : IChatService
{
    private readonly IChatRepository _chatRepository;
    private readonly IUserService _userService;

    public async Task<Chat> CreateDirectChatAsync(string user1Id, string user2Id)
    {
        // Check if direct chat already exists
        var existingChat = await _chatRepository.GetDirectChatAsync(user1Id, user2Id);
        if (existingChat != null)
            return existingChat;

        var chat = new Chat
        {
            Id = Guid.NewGuid().ToString(),
            Type = ChatType.Direct,
            Participants = new List<string> { user1Id, user2Id },
            CreatedAt = DateTime.UtcNow,
            LastMessageAt = DateTime.UtcNow
        };

        await _chatRepository.SaveChatAsync(chat);
        return chat;
    }

    public async Task<Chat> CreateGroupChatAsync(string name, string creatorId, List<string> participantIds)
    {
        var allParticipants = new List<string> { creatorId };
        allParticipants.AddRange(participantIds);

        var chat = new Chat
        {
            Id = Guid.NewGuid().ToString(),
            Type = ChatType.Group,
            Name = name,
            CreatorId = creatorId,
            Participants = allParticipants,
            CreatedAt = DateTime.UtcNow,
            LastMessageAt = DateTime.UtcNow
        };

        await _chatRepository.SaveChatAsync(chat);
        return chat;
    }
}

Key Design Considerations

1. Real-time Communication: WebSocket connections, connection pooling
2. Message Persistence: Database storage, message ordering
3. Scalability: Horizontal scaling, message queuing
4. Offline Support: Message queuing, push notifications
5. Media Handling: File upload, compression, CDN delivery
6. Security: End-to-end encryption, message authentication

---

100. How do you design a file storage system like Dropbox?

System Architecture Overview

A file storage system requires efficient storage, synchronization, versioning, and sharing capabilities. Here's the design:

Core Components

1. File Storage Service
2. Synchronization Engine
3. Version Control System
4. Sharing & Collaboration
5. Search & Indexing
6. Backup & Recovery

High-Level Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Desktop/Mobile│    │   API Gateway   │    │   Load Balancer │
│     Clients     │───▶│   (File Upload  │───▶│   (Round Robin) │
└─────────────────┘    │   & Download)   │    └─────────────────┘
                       └─────────────────┘              │
                                                        ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Search        │    │   File          │    │   Sync          │
│   Service       │◀───│   Management    │◀───│   Engine        │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                        │
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Version       │    │   Sharing       │    │   Storage       │
│   Control       │    │   Service       │    │   Service       │
└─────────────────┘    └─────────────────┘    └─────────────────┘

C# Implementation Examples

1. File System Models

public class FileNode
{
    public Guid Id { get; set; }
    public string Name { get; set; }
    public string Path { get; set; }
    public FileNodeType Type { get; set; }
    public long Size { get; set; }
    public string OwnerId { get; set; }
    public string ParentId { get; set; }
    public DateTime CreatedAt { get; set; }
    public DateTime ModifiedAt { get; set; }
    public string Hash { get; set; }
    public FileStatus Status { get; set; }
    public List<FileVersion> Versions { get; set; } = new();
    public List<FileShare> Shares { get; set; } = new();
    public Dictionary<string, object> Metadata { get; set; } = new();
}

public enum FileNodeType
{
    File,
    Folder
}

public enum FileStatus
{
    Syncing,
    Synced,
    Conflict,
    Deleted
}

public class FileVersion
{
    public Guid Id { get; set; }
    public int VersionNumber { get; set; }
    public string Hash { get; set; }
    public long Size { get; set; }
    public DateTime CreatedAt { get; set; }
    public string StorageLocation { get; set; }
    public string CreatedBy { get; set; }
    public string Comment { get; set; }
}

public class FileShare
{
    public Guid Id { get; set; }
    public string FileId { get; set; }
    public string SharedWithUserId { get; set; }
    public SharePermission Permission { get; set; }
    public DateTime SharedAt { get; set; }
    public DateTime? ExpiresAt { get; set; }
    public string SharedBy { get; set; }
}

public enum SharePermission
{
    Read,
    Write,
    Admin
}

2. File Storage Service

public class FileStorageService : IFileStorageService
{
    private readonly IFileRepository _fileRepository;
    private readonly IStorageProvider _storageProvider;
    private readonly ISyncEngine _syncEngine;
    private readonly IHashService _hashService;
    private readonly ILogger<FileStorageService> _logger;

    public async Task<FileNode> UploadFileAsync(UploadFileRequest request)
    {
        try
        {
            // Calculate file hash
            var fileHash = await _hashService.CalculateHashAsync(request.FileStream);

            // Check if file already exists (deduplication)
            var existingFile = await _fileRepository.GetFileByHashAsync(fileHash);
            if (existingFile != null)
            {
                // Create reference to existing file
                return await CreateFileReferenceAsync(request, existingFile);
            }

            // Upload to storage provider
            var storageLocation = await _storageProvider.UploadFileAsync(request.FileStream, fileHash);

            // Create file node
            var fileNode = new FileNode
            {
                Id = Guid.NewGuid(),
                Name = request.FileName,
                Path = request.Path,
                Type = FileNodeType.File,
                Size = request.FileSize,
                OwnerId = request.UserId,
                ParentId = request.ParentId,
                CreatedAt = DateTime.UtcNow,
                ModifiedAt = DateTime.UtcNow,
                Hash = fileHash,
                Status = FileStatus.Synced
            };

            // Create initial version
            fileNode.Versions.Add(new FileVersion
            {
                Id = Guid.NewGuid(),
                VersionNumber = 1,
                Hash = fileHash,
                Size = request.FileSize,
                CreatedAt = DateTime.UtcNow,
                StorageLocation = storageLocation,
                CreatedBy = request.UserId
            });

            // Save to database
            await _fileRepository.SaveFileAsync(fileNode);

            // Trigger sync for other devices
            await _syncEngine.NotifyFileChangedAsync(fileNode);

            return fileNode;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Error uploading file {FileName}", request.FileName);
            throw;
        }
    }

    public async Task<Stream> DownloadFileAsync(Guid fileId, string userId)
    {
        var fileNode = await _fileRepository.GetFileAsync(fileId);
        if (fileNode == null)
            throw new FileNotFoundException("File not found");

        // Check permissions
        if (!await HasAccessAsync(fileNode, userId))
            throw new UnauthorizedAccessException("Access denied");

        var latestVersion = fileNode.Versions.OrderByDescending(v => v.VersionNumber).First();
        return await _storageProvider.DownloadFileAsync(latestVersion.StorageLocation);
    }

    public async Task<FileNode> CreateFolderAsync(CreateFolderRequest request)
    {
        var folder = new FileNode
        {
            Id = Guid.NewGuid(),
            Name = request.FolderName,
            Path = request.Path,
            Type = FileNodeType.Folder,
            Size = 0,
            OwnerId = request.UserId,
            ParentId = request.ParentId,
            CreatedAt = DateTime.UtcNow,
            ModifiedAt = DateTime.UtcNow,
            Status = FileStatus.Synced
        };

        await _fileRepository.SaveFileAsync(folder);
        await _syncEngine.NotifyFileChangedAsync(folder);

        return folder;
    }

    public async Task DeleteFileAsync(Guid fileId, string userId)
    {
        var fileNode = await _fileRepository.GetFileAsync(fileId);
        if (fileNode == null) return;

        if (fileNode.OwnerId != userId)
            throw new UnauthorizedAccessException("Only owner can delete file");

        // Soft delete
        fileNode.Status = FileStatus.Deleted;
        await _fileRepository.UpdateFileAsync(fileNode);

        // Notify sync engine
        await _syncEngine.NotifyFileChangedAsync(fileNode);
    }
}

3. Synchronization Engine

public class SyncEngine : ISyncEngine
{
    private readonly IFileRepository _fileRepository;
    private readonly IUserSessionService _sessionService;
    private readonly IWebSocketService _webSocketService;
    private readonly ILogger<SyncEngine> _logger;

    public async Task<SyncResult> SyncFilesAsync(string userId, List<FileChange> localChanges)
    {
        var syncResult = new SyncResult
        {
            UserId = userId,
            AppliedChanges = new List<FileChange>(),
            Conflicts = new List<FileConflict>()
        };

        // Get server state
        var serverFiles = await _fileRepository.GetUserFilesAsync(userId);

        foreach (var localChange in localChanges)
        {
            var serverFile = serverFiles.FirstOrDefault(f => f.Id == localChange.FileId);

            switch (localChange.ChangeType)
            {
                case ChangeType.Created:
                    await HandleFileCreatedAsync(localChange, syncResult);
                    break;
                case ChangeType.Modified:
                    await HandleFileModifiedAsync(localChange, serverFile, syncResult);
                    break;
                case ChangeType.Deleted:
                    await HandleFileDeletedAsync(localChange, serverFile, syncResult);
                    break;
            }
        }

        // Send server changes to client
        var serverChanges = await GetServerChangesAsync(userId, localChanges);
        syncResult.ServerChanges = serverChanges;

        return syncResult;
    }

    private async Task HandleFileModifiedAsync(FileChange localChange, FileNode serverFile, SyncResult syncResult)
    {
        if (serverFile == null)
        {
            // File doesn't exist on server, create it
            await CreateFileFromChangeAsync(localChange, syncResult);
            return;
        }

        // Check for conflicts
        if (serverFile.ModifiedAt > localChange.Timestamp)
        {
            // Server version is newer, create conflict
            syncResult.Conflicts.Add(new FileConflict
            {
                FileId = localChange.FileId,
                LocalVersion = localChange,
                ServerVersion = serverFile,
                Resolution = ConflictResolution.Pending
            });
        }
        else
        {
            // Apply local change
            await ApplyFileChangeAsync(localChange, syncResult);
        }
    }

    public async Task NotifyFileChangedAsync(FileNode fileNode)
    {
        // Notify all online users who have access to this file
        var usersToNotify = await GetUsersWithAccessAsync(fileNode);

        foreach (var userId in usersToNotify)
        {
            if (_sessionService.IsUserOnline(userId))
            {
                await _webSocketService.SendToUserAsync(userId, new
                {
                    Type = "file_changed",
                    FileId = fileNode.Id,
                    ChangeType = "modified",
                    Timestamp = DateTime.UtcNow
                });
            }
        }
    }
}

4. Version Control Service

public class VersionControlService : IVersionControlService
{
    private readonly IFileRepository _fileRepository;
    private readonly IStorageProvider _storageProvider;

    public async Task<FileVersion> CreateVersionAsync(Guid fileId, Stream fileStream, string userId, string comment = null)
    {
        var fileNode = await _fileRepository.GetFileAsync(fileId);
        if (fileNode == null)
            throw new FileNotFoundException("File not found");

        var fileHash = await _hashService.CalculateHashAsync(fileStream);

        // Check if this version is different from the latest
        var latestVersion = fileNode.Versions.OrderByDescending(v => v.VersionNumber).FirstOrDefault();
        if (latestVersion?.Hash == fileHash)
        {
            // No changes, return existing version
            return latestVersion;
        }

        // Upload new version
        var storageLocation = await _storageProvider.UploadFileAsync(fileStream, fileHash);

        var newVersion = new FileVersion
        {
            Id = Guid.NewGuid(),
            VersionNumber = (latestVersion?.VersionNumber ?? 0) + 1,
            Hash = fileHash,
            Size = fileStream.Length,
            CreatedAt = DateTime.UtcNow,
            StorageLocation = storageLocation,
            CreatedBy = userId,
            Comment = comment
        };

        fileNode.Versions.Add(newVersion);
        fileNode.ModifiedAt = DateTime.UtcNow;
        fileNode.Size = newVersion.Size;

        await _fileRepository.UpdateFileAsync(fileNode);

        return newVersion;
    }

    public async Task<List<FileVersion>> GetFileVersionsAsync(Guid fileId)
    {
        var fileNode = await _fileRepository.GetFileAsync(fileId);
        return fileNode?.Versions.OrderByDescending(v => v.VersionNumber).ToList() ?? new List<FileVersion>();
    }

    public async Task<Stream> GetVersionContentAsync(Guid fileId, int versionNumber)
    {
        var fileNode = await _fileRepository.GetFileAsync(fileId);
        var version = fileNode?.Versions.FirstOrDefault(v => v.VersionNumber == versionNumber);

        if (version == null)
            throw new FileNotFoundException("Version not found");

        return await _storageProvider.DownloadFileAsync(version.StorageLocation);
    }
}

5. Sharing Service

public class SharingService : ISharingService
{
    private readonly IFileRepository _fileRepository;
    private readonly IUserService _userService;
    private readonly INotificationService _notificationService;

    public async Task<FileShare> ShareFileAsync(ShareFileRequest request)
    {
        var fileNode = await _fileRepository.GetFileAsync(request.FileId);
        if (fileNode == null)
            throw new FileNotFoundException("File not found");

        if (fileNode.OwnerId != request.SharedBy)
            throw new UnauthorizedAccessException("Only owner can share file");

        // Check if already shared
        var existingShare = fileNode.Shares.FirstOrDefault(s => s.SharedWithUserId == request.SharedWithUserId);
        if (existingShare != null)
        {
            existingShare.Permission = request.Permission;
            existingShare.ExpiresAt = request.ExpiresAt;
            await _fileRepository.UpdateFileAsync(fileNode);
            return existingShare;
        }

        var share = new FileShare
        {
            Id = Guid.NewGuid(),
            FileId = request.FileId,
            SharedWithUserId = request.SharedWithUserId,
            Permission = request.Permission,
            SharedAt = DateTime.UtcNow,
            ExpiresAt = request.ExpiresAt,
            SharedBy = request.SharedBy
        };

        fileNode.Shares.Add(share);
        await _fileRepository.UpdateFileAsync(fileNode);

        // Send notification
        await _notificationService.SendFileSharedNotificationAsync(share);

        return share;
    }

    public async Task<List<FileNode>> GetSharedFilesAsync(string userId)
    {
        return await _fileRepository.GetSharedFilesAsync(userId);
    }

    public async Task RevokeAccessAsync(Guid fileId, string sharedWithUserId, string revokedBy)
    {
        var fileNode = await _fileRepository.GetFileAsync(fileId);
        if (fileNode.OwnerId != revokedBy)
            throw new UnauthorizedAccessException("Only owner can revoke access");

        var share = fileNode.Shares.FirstOrDefault(s => s.SharedWithUserId == sharedWithUserId);
        if (share != null)
        {
            fileNode.Shares.Remove(share);
            await _fileRepository.UpdateFileAsync(fileNode);
        }
    }
}

Key Design Considerations

1. Storage Efficiency: Deduplication, compression, tiered storage
2. Synchronization: Conflict resolution, change tracking, real-time sync
3. Versioning: File history, rollback capabilities, storage optimization
4. Sharing: Access control, link sharing, collaboration features
5. Performance: CDN, caching, parallel uploads/downloads
6. Security: Encryption at rest and in transit, access controls
7. Scalability: Distributed storage, horizontal scaling, load balancing

These designs provide a solid foundation for building robust, scalable systems that can handle the complexities of modern applications while maintaining performance, security, and user experience.

Official references

• Google SRE workbook
• AWS Well-Architected Reliability Pillar
• CAP theorem overview