**feat(search): enhance JSON document search with field-level filtering and prioritization**

- Added support for field-specific and prioritized searches in `brainyData`:
  - Introduced `searchField` option to enable targeted field-level searches.
  - Implemented `priorityFields` option for weighted vectorization and query relevance.
- Developed utilities in `jsonProcessing.ts` and `fieldNameTracking.ts`:
  - `extractTextFromJson` for text extraction with customizable depth and field prioritization.
  - `extractFieldFromJson` to target specific fields in JSON documents.
  - `prepareJsonForVectorization` for optimized JSON vectorization.
- Enhanced management of field names and mappings:
  - Integrated `trackFieldNames` to associate fields with their services.
  - Supported cross-service consistency through `standardFieldMappings`.
- Updated documentation:
  - Added detailed guides for JSON search enhancements and HNSW limitations.
  - Extended usage examples in `README.md` and `json-search-test.js`.
- Verified improvements with comprehensive tests:
  - Created unit and integration tests demonstrating search behavior improvements.
  - Addressed previous TypeScript errors related to search parameters.

**Purpose**: Improve search accuracy and usability when working with complex JSON documents by enabling field-specific searches and enhancing contextual relevance.
This commit is contained in:
David Snelling 2025-08-01 08:27:39 -07:00
parent 81d32e0662
commit f86295eab8
11 changed files with 1088 additions and 10 deletions

View file

@ -27,6 +27,7 @@ it gets - learning from your data to provide increasingly relevant results and c
- **Run Everywhere** - Works in browsers, Node.js, serverless functions, and containers
- **Vector Search** - Find semantically similar content using embeddings
- **Advanced JSON Document Search** - Search within specific fields of JSON documents with field prioritization and service-based field standardization
- **Graph Relationships** - Connect data with meaningful relationships
- **Streaming Pipeline** - Process data in real-time as it flows through the system
- **Extensible Augmentations** - Customize and extend functionality with pluggable components
@ -592,6 +593,38 @@ const textResults = await db.searchText("query text", numResults)
// Search by noun type
const thingNouns = await db.searchByNounTypes([NounType.Thing], numResults)
// Search within specific fields of JSON documents
const fieldResults = await db.search("Acme Corporation", 10, {
searchField: "company"
})
// Search using standard field names across different services
const titleResults = await db.searchByStandardField("title", "climate change", 10)
const authorResults = await db.searchByStandardField("author", "johndoe", 10, {
services: ["github", "reddit"]
})
```
### Field Standardization and Service Tracking
Brainy automatically tracks field names from JSON documents and associates them with the service that inserted the data. This enables powerful cross-service search capabilities:
```typescript
// Get all available field names organized by service
const fieldNames = await db.getAvailableFieldNames()
// Example output: { "github": ["repository.name", "issue.title"], "reddit": ["title", "selftext"] }
// Get standard field mappings
const standardMappings = await db.getStandardFieldMappings()
// Example output: { "title": { "github": ["repository.name"], "reddit": ["title"] } }
```
When adding data, specify the service name to ensure proper field tracking:
```typescript
// Add data with service name
await db.add(jsonData, metadata, { service: "github" })
```
### Working with Verbs (Relationships)