feat: migrate system metadata from 'index' to '_system' directory with backward compatibility

BREAKING CHANGE: System metadata location changed from 'index/' to '_system/' directory

- Rename INDEX_DIR to SYSTEM_DIR following database conventions
- Implement dual-read/write strategy for zero-downtime migration
- Add automatic migration from old to new location on first access
- Support mixed service versions sharing S3/cloud storage
- Add 30-day grace period for gradual rollout (configurable)
- Store distributed config alongside statistics in _system folder
- Add comprehensive migration guide and documentation

Migration features:
- Read from both locations (new first, fallback to old)
- Write to both during migration period
- Automatic data migration when found only in old location
- Services can update independently without coordination
- Full backward compatibility for production deployments

The change improves clarity ('_system' better represents system metadata than 'index')
and follows standard database conventions (MongoDB's _system, PostgreSQL's pg_*).

src/storage/baseStorage.ts

@@ -12,9 +12,13 @@ export const VERBS_DIR = 'verbs'
 export const METADATA_DIR = 'metadata'
 export const NOUN_METADATA_DIR = 'noun-metadata'
 export const VERB_METADATA_DIR = 'verb-metadata'
-export const INDEX_DIR = 'index'
+export const INDEX_DIR = 'index'  // Legacy - kept for backward compatibility
+export const SYSTEM_DIR = '_system'  // New location for system data
 export const STATISTICS_KEY = 'statistics'
 
+// Migration version to track compatibility
+export const STORAGE_SCHEMA_VERSION = 2  // Increment when making breaking changes
+
 /**
  * Base storage adapter that implements common functionality
  * This is an abstract class that should be extended by specific storage adapters
@@ -893,6 +897,22 @@ export abstract class BaseStorage extends BaseStorageAdapter {
     return obj
   }
 
+  /**
+   * Save statistics data to storage (public interface)
+   * @param statistics The statistics data to save
+   */
+  public async saveStatistics(statistics: StatisticsData): Promise<void> {
+    return this.saveStatisticsData(statistics)
+  }
+
+  /**
+   * Get statistics data from storage (public interface)
+   * @returns Promise that resolves to the statistics data or null if not found
+   */
+  public async getStatistics(): Promise<StatisticsData | null> {
+    return this.getStatisticsData()
+  }
+
   /**
    * Save statistics data to storage
    * This method should be implemented by each specific adapter