fix(storage): resolve pagination warnings and improve S3 adapter performance

- Fix deprecated getAllNodes() warnings by using getNodesWithPagination() in S3 adapter
- Add getNounsWithPagination() method to S3CompatibleStorage for proper pagination support
- Create optimizedS3Search module for efficient pagination and filtering
- Update baseStorage to properly detect pagination support in adapters
- Add comprehensive documentation for performance and logging fixes
- Ensure backward compatibility with existing code

This resolves the following warnings in dependent projects:
- "getAllNodes() is deprecated and will be removed in a future version"
- "Storage adapter does not support pagination, falling back to loading all nouns"
- "Only returning the first 1000 nodes. There are more nodes available"

docs/PERFORMANCE_AND_LOGGING_FIXES.md

@@ -0,0 +1,101 @@
+# Performance and Logging Fixes for Brainy
+
+## Overview
+
+This document outlines the fixes implemented to address pagination warnings and improve performance in the Brainy framework.
+
+## Issues Addressed
+
+### 1. Pagination Warnings
+
+The following warnings were appearing when using Brainy in dependent projects:
+
+```
+WARNING: Only returning the first 1000 nodes. There are more nodes available. Use getNodesWithPagination() for proper pagination.
+Storage adapter does not support pagination, falling back to loading all nouns. This may cause performance issues with large datasets.
+WARNING: getAllNodes() is deprecated and will be removed in a future version. Use getNodesWithPagination() instead.
+```
+
+### 2. Root Causes
+
+1. **Missing Pagination Support**: The S3 storage adapter didn't expose a `getNounsWithPagination()` method, causing the base storage to fall back to `getAllNouns_internal()`.
+2. **Deprecation Warning Logic**: The `getAllNouns_internal()` method was calling the deprecated `getAllNodes()` method, which triggered deprecation warnings.
+3. **Inefficient Data Loading**: The fallback approach could load all data into memory, causing performance issues with large datasets.
+
+## Solutions Implemented
+
+### 1. Updated S3 Storage Adapter
+
+**File**: `src/storage/adapters/s3CompatibleStorage.ts`
+
+- Modified `getAllNouns_internal()` to use the paginated `getNodesWithPagination()` method instead of the deprecated `getAllNodes()`.
+- Added `getNounsWithPagination()` method to properly support pagination with filtering.
+
+```typescript
+public async getNounsWithPagination(options: {
+  limit?: number
+  cursor?: string
+  filter?: {
+    nounType?: string | string[]
+    service?: string | string[]
+    metadata?: Record<string, any>
+  }
+} = {}): Promise<{
+  items: HNSWNoun[]
+  totalCount?: number
+  hasMore: boolean
+  nextCursor?: string
+}>
+```
+
+### 2. Created Optimized S3 Search Module
+
+**File**: `src/storage/adapters/optimizedS3Search.ts`
+
+A new module that provides:
+- Efficient pagination using S3's ListObjectsV2 command
+- Parallel batch loading for better performance
+- Support for complex filtering without loading all data
+- Proper cursor-based pagination for large datasets
+
+### 3. Improved Base Storage Logic
+
+**File**: `src/storage/baseStorage.ts`
+
+- The base storage now properly detects when adapters support pagination
+- Falls back gracefully when pagination isn't supported
+- Limits the amount of data loaded to prevent memory issues
+
+## Benefits
+
+1. **Eliminated Warnings**: No more deprecation warnings in dependent projects.
+2. **Better Performance**: Pagination prevents loading entire datasets into memory.
+3. **Improved Scalability**: Can handle large datasets efficiently.
+4. **Backward Compatibility**: Existing code continues to work without changes.
+
+## Usage in Dependent Projects
+
+Your code in github-package is already using the correct approach:
+
+```typescript
+const response = await brainy.getNouns({
+  pagination: { limit: 10, offset: 0 },
+  filter: { nounType: 'Template' }
+})
+```
+
+With these fixes, this will now:
+- Use proper pagination without warnings
+- Load only the requested data
+- Support efficient filtering
+
+## Future Improvements
+
+1. **Native S3 Filtering**: Implement server-side filtering using S3 object tags or metadata.
+2. **Index-based Search**: Create indexes for common query patterns.
+3. **Caching Layer**: Add intelligent caching to reduce S3 API calls.
+4. **Streaming Support**: For very large datasets, implement streaming APIs.
+
+## Migration Notes
+
+No changes are required in dependent projects. The fixes are backward compatible and will automatically improve performance and eliminate warnings.