feat: migrate system metadata from 'index' to '_system' directory with backward compatibility

BREAKING CHANGE: System metadata location changed from 'index/' to '_system/' directory

- Rename INDEX_DIR to SYSTEM_DIR following database conventions
- Implement dual-read/write strategy for zero-downtime migration
- Add automatic migration from old to new location on first access
- Support mixed service versions sharing S3/cloud storage
- Add 30-day grace period for gradual rollout (configurable)
- Store distributed config alongside statistics in _system folder
- Add comprehensive migration guide and documentation

Migration features:
- Read from both locations (new first, fallback to old)
- Write to both during migration period
- Automatic data migration when found only in old location
- Services can update independently without coordination
- Full backward compatibility for production deployments

The change improves clarity ('_system' better represents system metadata than 'index')
and follows standard database conventions (MongoDB's _system, PostgreSQL's pg_*).

src/storage/adapters/memoryStorage.ts

@@ -621,7 +621,11 @@ export class MemoryStorage extends BaseStorage {
       verbCount: {...statistics.verbCount},
       metadataCount: {...statistics.metadataCount},
       hnswIndexSize: statistics.hnswIndexSize,
-      lastUpdated: statistics.lastUpdated
+      lastUpdated: statistics.lastUpdated,
+      // Include distributedConfig if present
+      ...(statistics.distributedConfig && { 
+        distributedConfig: JSON.parse(JSON.stringify(statistics.distributedConfig)) 
+      })
     }
     
     // Since this is in-memory, there's no need for time-based partitioning
@@ -643,7 +647,11 @@ export class MemoryStorage extends BaseStorage {
       verbCount: {...this.statistics.verbCount},
       metadataCount: {...this.statistics.metadataCount},
       hnswIndexSize: this.statistics.hnswIndexSize,
-      lastUpdated: this.statistics.lastUpdated
+      lastUpdated: this.statistics.lastUpdated,
+      // Include distributedConfig if present
+      ...(this.statistics.distributedConfig && { 
+        distributedConfig: JSON.parse(JSON.stringify(this.statistics.distributedConfig)) 
+      })
     }
     
     // Since this is in-memory, there's no need for fallback mechanisms