🔨 Fixes to searchmatchscorer

Comicvine improvements

Dockerfile

@@ -1,20 +1,37 @@
-FROM node:12-alpine
+# Use Node 21 as the base image for the builder stage
+FROM node:21-alpine AS builder
 LABEL maintainer="Rishi Ghan <rishi.ghan@gmail.com>"
 
-# Working directory
+# Set the working directory
 WORKDIR /metadata-service
-# Install dependencies
+
+# Copy and install dependencies
 COPY package.json package-lock.json ./
 RUN npm ci --silent
 
-# Copy source
+# Copy source code and build the application
 COPY . .
+RUN npm run build
 
-# Build and cleanup
+# Clean up development dependencies
+RUN npm prune --production
+
+# Final image using Node 21
+FROM node:21-alpine
+
+LABEL maintainer="Rishi Ghan <rishi.ghan@gmail.com>"
+
+# Set the working directory
+WORKDIR /metadata-service
+
+# Copy the necessary files from the builder image
+COPY --from=builder /metadata-service /metadata-service
+
+# Set environment variables
 ENV NODE_ENV=production
-RUN npm run build \
- && npm prune
 
+# Expose the application's port
 EXPOSE 3080
-# Start server
-CMD ["npm", "start"]
+
+# Start the application
+CMD ["npm", "start"]

README_SCHEMA_STITCHING.md

@@ -0,0 +1,337 @@
+# GraphQL Schema Stitching - Unified Gateway
+
+This service now implements **GraphQL Schema Stitching** to combine multiple GraphQL schemas into a single unified endpoint.
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    Client Application                        │
+└────────────────────────┬────────────────────────────────────┘
+                         │
+                         │ Single GraphQL Endpoint
+                         ▼
+┌─────────────────────────────────────────────────────────────┐
+│          API Gateway (port 3080)                             │
+│          /graphql endpoint                                   │
+└────────────────────────┬────────────────────────────────────┘
+                         │
+                         │ Moleculer RPC
+                         ▼
+┌─────────────────────────────────────────────────────────────┐
+│          Gateway Service (Schema Stitching)                  │
+│          - Combines local + remote schemas                   │
+│          - Routes queries to appropriate service             │
+└────────┬────────────────────────────────────┬───────────────┘
+         │                                    │
+         │ Local Schema                       │ Remote Schema
+         ▼                                    ▼
+┌────────────────────┐              ┌────────────────────────┐
+│  Local Services    │              │  Remote GraphQL Server │
+│  - ComicVine       │              │  (port 3000)           │
+│  - Metron          │              │  - Core Service        │
+│  - Metadata        │              │  - Other queries       │
+└────────────────────┘              └────────────────────────┘
+```
+
+## What is Schema Stitching?
+
+Schema stitching combines multiple GraphQL schemas into a single unified schema. This allows you to:
+
+1. **Query multiple services** through a single endpoint
+2. **Combine data** from different sources in one request
+3. **Maintain service independence** while providing a unified API
+4. **Gradually migrate** services without breaking existing clients
+
+## Configuration
+
+### Environment Variables
+
+Set the remote GraphQL server URL:
+
+```bash
+export REMOTE_GRAPHQL_URL="http://localhost:3000/graphql"
+```
+
+If not set, it defaults to `http://localhost:3000/graphql`.
+
+### Service Files
+
+- **[`services/gateway.service.ts`](services/gateway.service.ts)** - Gateway service with schema stitching logic
+- **[`services/api.service.ts`](services/api.service.ts)** - API gateway routing to the gateway service
+- **[`services/graphql.service.ts`](services/graphql.service.ts)** - Original local GraphQL service (still available)
+
+## How It Works
+
+### 1. Schema Introspection
+
+On startup, the gateway service:
+1. Introspects the remote GraphQL server at port 3000
+2. Builds a client schema from the introspection result
+3. Creates a local executable schema from the metadata service
+4. Stitches both schemas together
+
+### 2. Query Routing
+
+When a query is received:
+1. The gateway analyzes the query
+2. Routes local queries (ComicVine, Metron) to local resolvers
+3. Routes remote queries to the remote GraphQL server
+4. Combines results if the query spans both schemas
+
+### 3. Fallback Behavior
+
+If the remote server is unavailable:
+- The gateway starts with **local schema only**
+- Logs a warning about remote unavailability
+- Continues to serve local queries normally
+- Remote queries will fail gracefully
+
+## Usage
+
+### Starting the Service
+
+```bash
+npm run dev
+```
+
+The unified GraphQL endpoint will be available at: `http://localhost:3080/graphql`
+
+### Example: Local Query (Metadata Service)
+
+```graphql
+query SearchComicVine {
+  searchComicVine(input: { 
+    query: "Batman", 
+    resources: "volume", 
+    limit: 5 
+  }) {
+    number_of_total_results
+    results {
+      id
+      name
+      start_year
+    }
+  }
+}
+```
+
+### Example: Remote Query (Core Service on port 3000)
+
+Assuming your remote server has queries like `getUser`, `getComics`, etc.:
+
+```graphql
+query GetUser {
+  getUser(id: "123") {
+    id
+    username
+    email
+  }
+}
+```
+
+### Example: Combined Query (Both Services)
+
+```graphql
+query CombinedQuery {
+  # Local metadata service
+  searchComicVine(input: { 
+    query: "Batman", 
+    resources: "volume", 
+    limit: 3 
+  }) {
+    results {
+      id
+      name
+    }
+  }
+  
+  # Remote core service
+  getUser(id: "123") {
+    id
+    username
+  }
+}
+```
+
+## Benefits
+
+### 1. Single Endpoint
+- Clients only need to know about one GraphQL endpoint
+- Simplifies frontend configuration
+- Easier to manage authentication/authorization
+
+### 2. Flexible Queries
+- Query data from multiple services in one request
+- Reduce network round trips
+- Better performance for complex data requirements
+
+### 3. Service Independence
+- Each service maintains its own schema
+- Services can be developed and deployed independently
+- No tight coupling between services
+
+### 4. Gradual Migration
+- Add new services without breaking existing clients
+- Migrate queries between services transparently
+- Maintain backward compatibility
+
+## Monitoring & Debugging
+
+### Logs
+
+The gateway service logs important events:
+
+```
+[GATEWAY] Initializing Apollo Gateway with Schema Stitching...
+[GATEWAY] Attempting to introspect remote schema at http://localhost:3000/graphql
+[GATEWAY] Successfully introspected remote schema
+[GATEWAY] Stitching local and remote schemas together...
+[GATEWAY] Schema stitching completed successfully
+[GATEWAY] Apollo Gateway Server started successfully
+```
+
+### Introspection
+
+Query the stitched schema:
+
+```graphql
+query IntrospectionQuery {
+  __schema {
+    queryType {
+      name
+      fields {
+        name
+        description
+      }
+    }
+  }
+}
+```
+
+### Health Check
+
+Check gateway status:
+
+```graphql
+query GetGatewayInfo {
+  __typename
+}
+```
+
+## Troubleshooting
+
+### Remote Server Unavailable
+
+**Symptom**: Warning logs about remote schema introspection failure
+
+**Solution**: 
+1. Ensure the remote server is running on port 3000
+2. Check the `REMOTE_GRAPHQL_URL` environment variable
+3. Verify network connectivity
+4. The gateway will continue with local schema only
+
+### Query Routing Issues
+
+**Symptom**: Queries to remote service fail or return null
+
+**Solution**:
+1. Check that the remote server is responding
+2. Verify the query syntax matches the remote schema
+3. Use introspection to see available fields
+4. Check gateway logs for routing errors
+
+### Type Conflicts
+
+**Symptom**: Errors about duplicate types or conflicting definitions
+
+**Solution**:
+1. Ensure type names are unique across schemas
+2. Use schema transformation if needed
+3. Consider renaming conflicting types in one schema
+4. Check the `mergeTypes` configuration in [`gateway.service.ts`](services/gateway.service.ts)
+
+## Advanced Configuration
+
+### Custom Executors
+
+Modify the executor in [`gateway.service.ts`](services/gateway.service.ts:95) to add:
+- Authentication headers
+- Request logging
+- Error handling
+- Caching
+
+### Schema Transformations
+
+Use `@graphql-tools/wrap` to transform schemas:
+- Rename types
+- Filter fields
+- Add custom directives
+- Modify field arguments
+
+### Performance Optimization
+
+Consider implementing:
+- **DataLoader** for batching requests
+- **Response caching** at the gateway level
+- **Query complexity analysis** to prevent expensive queries
+- **Rate limiting** per client or query type
+
+## Migration from Separate Endpoints
+
+### Before (Separate Endpoints)
+
+```typescript
+// Frontend code
+const metadataClient = new ApolloClient({
+  uri: 'http://localhost:3080/graphql'
+});
+
+const coreClient = new ApolloClient({
+  uri: 'http://localhost:3000/graphql'
+});
+```
+
+### After (Unified Gateway)
+
+```typescript
+// Frontend code
+const client = new ApolloClient({
+  uri: 'http://localhost:3080/graphql'  // Single endpoint!
+});
+```
+
+## Comparison with Apollo Federation
+
+| Feature | Schema Stitching | Apollo Federation |
+|---------|------------------|-------------------|
+| Setup Complexity | Moderate | Higher |
+| Service Independence | Good | Excellent |
+| Type Merging | Manual | Automatic |
+| Best For | Existing services | New microservices |
+| Learning Curve | Lower | Higher |
+
+## Related Documentation
+
+- [GraphQL API Documentation](README_GRAPHQL.md)
+- [Architecture Overview](ARCHITECTURE.md)
+- [Main README](README.md)
+
+## Support
+
+For issues or questions:
+1. Check the gateway service logs
+2. Verify both servers are running
+3. Test each service independently
+4. Review the schema stitching configuration
+
+## Future Enhancements
+
+- [ ] Add authentication/authorization at gateway level
+- [ ] Implement DataLoader for batching
+- [ ] Add response caching
+- [ ] Implement query complexity analysis
+- [ ] Add rate limiting
+- [ ] Support for GraphQL subscriptions
+- [ ] Schema transformation utilities
+- [ ] Automated schema versioning

models/graphql/resolvers.ts

@@ -0,0 +1,196 @@
+/**
+ * GraphQL Resolvers for ThreeTwo Metadata Service
+ * Maps GraphQL queries to Moleculer service actions
+ */
+
+export const resolvers = {
+	Query: {
+		/**
+		 * Search ComicVine for volumes, issues, characters, etc.
+		 */
+		searchComicVine: async (_: any, { input }: any, context: any) => {
+			const { broker } = context;
+			
+			if (!broker) {
+				throw new Error("Broker not available in context");
+			}
+
+			return broker.call("comicvine.search", {
+				query: input.query,
+				resources: input.resources,
+				format: input.format || "json",
+				sort: input.sort,
+				field_list: input.field_list,
+				limit: input.limit?.toString(),
+				offset: input.offset?.toString(),
+			});
+		},
+
+		/**
+		 * Advanced volume-based search with scoring and filtering
+		 */
+		volumeBasedSearch: async (_: any, { input }: any, context: any) => {
+			const { broker } = context;
+			
+			if (!broker) {
+				throw new Error("Broker not available in context");
+			}
+
+			const result = await broker.call("comicvine.volumeBasedSearch", {
+				query: input.query,
+				resources: input.resources,
+				format: input.format || "json",
+				limit: input.limit,
+				offset: input.offset,
+				fieldList: input.fieldList,
+				scorerConfiguration: input.scorerConfiguration,
+				rawFileDetails: input.rawFileDetails,
+			});
+
+			// Transform the result to match GraphQL schema
+			return {
+				results: result.results || result,
+				totalResults: result.totalResults || result.length || 0,
+			};
+		},
+
+		/**
+		 * Get volume details by URI
+		 */
+		getVolume: async (_: any, { input }: any, context: any) => {
+			const { broker } = context;
+			
+			if (!broker) {
+				throw new Error("Broker not available in context");
+			}
+
+			return broker.call("comicvine.getVolumes", {
+				volumeURI: input.volumeURI,
+				fieldList: input.fieldList,
+			});
+		},
+
+		/**
+		 * Get all issues for a series by comic object ID
+		 */
+		getIssuesForSeries: async (_: any, { comicObjectId }: any, context: any) => {
+			const { broker } = context;
+			
+			if (!broker) {
+				throw new Error("Broker not available in context");
+			}
+
+			return broker.call("comicvine.getIssuesForSeries", {
+				comicObjectId,
+			});
+		},
+
+		/**
+		 * Get generic ComicVine resource (issues, volumes, etc.)
+		 */
+		getComicVineResource: async (_: any, { input }: any, context: any) => {
+			const { broker } = context;
+			
+			if (!broker) {
+				throw new Error("Broker not available in context");
+			}
+
+			return broker.call("comicvine.getResource", {
+				resources: input.resources,
+				filter: input.filter,
+				fieldList: input.fieldList,
+			});
+		},
+
+		/**
+		 * Get story arcs for a volume
+		 */
+		getStoryArcs: async (_: any, { volumeId }: any, context: any) => {
+			const { broker } = context;
+			
+			if (!broker) {
+				throw new Error("Broker not available in context");
+			}
+
+			return broker.call("comicvine.getStoryArcs", {
+				volumeId,
+			});
+		},
+
+		/**
+		 * Get weekly pull list from League of Comic Geeks
+		 */
+		getWeeklyPullList: async (_: any, { input }: any, context: any) => {
+			const { broker } = context;
+			
+			if (!broker) {
+				throw new Error("Broker not available in context");
+			}
+
+			const locgResponse = await broker.call("comicvine.getWeeklyPullList", {
+				startDate: input.startDate,
+				currentPage: input.currentPage.toString(),
+				pageSize: input.pageSize.toString(),
+			});
+
+			// Transform LOCG response to match GraphQL schema
+			return {
+				result: locgResponse.result.map((item: any) => ({
+					name: item.issueName,
+					publisher: item.publisher,
+					url: item.issueUrl,
+					cover: item.coverImageUrl,
+					description: item.description || null,
+					price: item.price || null,
+					rating: item.rating || null,
+					pulls: item.pulls || null,
+					potw: item.potw || null,
+					publicationDate: item.publicationDate || null,
+				})),
+				meta: locgResponse.meta,
+			};
+		},
+
+		/**
+		 * Fetch resource from Metron API
+		 */
+		fetchMetronResource: async (_: any, { input }: any, context: any) => {
+			const { broker } = context;
+			
+			if (!broker) {
+				throw new Error("Broker not available in context");
+			}
+
+			const result = await broker.call("metron.fetchResource", {
+				resource: input.resource,
+				method: input.method,
+				query: input.query,
+			});
+
+			return {
+				data: result,
+				status: 200,
+			};
+		},
+	},
+
+	Mutation: {
+		/**
+		 * Placeholder for future mutations
+		 */
+		_empty: (): null => null,
+	},
+
+	// Custom scalar resolver for JSON
+	JSON: {
+		__parseValue(value: any): any {
+			return value;
+		},
+		__serialize(value: any): any {
+			return value;
+		},
+		__parseLiteral(ast: any): any {
+			return ast.value;
+		},
+	},
+};

models/graphql/typedef.ts

@@ -0,0 +1,357 @@
+import { gql } from "graphql-tag";
+
+/**
+ * GraphQL Type Definitions for ThreeTwo Metadata Service
+ * Covers ComicVine and Metron API endpoints
+ */
+export const typeDefs = gql`
+	# ============================================
+	# ComicVine Types
+	# ============================================
+
+	# Image URLs for various sizes
+	type ImageUrls {
+		icon_url: String
+		medium_url: String
+		screen_url: String
+		screen_large_url: String
+		small_url: String
+		super_url: String
+		thumb_url: String
+		tiny_url: String
+		original_url: String
+		image_tags: String
+	}
+
+	# Publisher information
+	type Publisher {
+		id: Int
+		name: String
+		api_detail_url: String
+	}
+
+	# Volume information
+	type Volume {
+		id: Int!
+		name: String!
+		api_detail_url: String
+		site_detail_url: String
+		start_year: String
+		publisher: Publisher
+		count_of_issues: Int
+		image: ImageUrls
+		description: String
+		deck: String
+	}
+
+	# Issue information
+	type Issue {
+		id: Int!
+		name: String
+		issue_number: String
+		api_detail_url: String
+		site_detail_url: String
+		cover_date: String
+		store_date: String
+		volume: Volume
+		image: ImageUrls
+		description: String
+		person_credits: [PersonCredit!]
+		character_credits: [CharacterCredit!]
+		team_credits: [TeamCredit!]
+		location_credits: [LocationCredit!]
+		story_arc_credits: [StoryArcCredit!]
+	}
+
+	# Person credit (writer, artist, etc.)
+	type PersonCredit {
+		id: Int
+		name: String
+		api_detail_url: String
+		site_detail_url: String
+		role: String
+	}
+
+	# Character credit
+	type CharacterCredit {
+		id: Int
+		name: String
+		api_detail_url: String
+		site_detail_url: String
+	}
+
+	# Team credit
+	type TeamCredit {
+		id: Int
+		name: String
+		api_detail_url: String
+		site_detail_url: String
+	}
+
+	# Location credit
+	type LocationCredit {
+		id: Int
+		name: String
+		api_detail_url: String
+		site_detail_url: String
+	}
+
+	# Story arc credit
+	type StoryArcCredit {
+		id: Int
+		name: String
+		api_detail_url: String
+		site_detail_url: String
+		deck: String
+		description: String
+		image: ImageUrls
+	}
+
+	# ComicVine search result
+	type ComicVineSearchResult {
+		error: String!
+		limit: Int!
+		offset: Int!
+		number_of_page_results: Int!
+		number_of_total_results: Int!
+		status_code: Int!
+		results: [SearchResultItem!]!
+	}
+
+	# Generic search result item (can be volume, issue, etc.)
+	type SearchResultItem {
+		id: Int
+		name: String
+		api_detail_url: String
+		site_detail_url: String
+		image: ImageUrls
+		description: String
+		deck: String
+		# Volume-specific fields
+		start_year: String
+		publisher: Publisher
+		count_of_issues: Int
+		# Issue-specific fields
+		issue_number: String
+		volume: Volume
+		cover_date: String
+	}
+
+	# Volume-based search result with scoring
+	type VolumeSearchResult {
+		volume: Volume!
+		score: Float
+		matchedIssues: [Issue!]
+	}
+
+	# Volume-based search response
+	type VolumeBasedSearchResponse {
+		results: [VolumeSearchResult!]!
+		totalResults: Int!
+	}
+
+	# Weekly pull list item (from League of Comic Geeks)
+	type MetadataPullListItem {
+		name: String
+		publisher: String
+		url: String
+		cover: String
+		description: String
+		price: String
+		rating: Float
+		pulls: Int
+		potw: Int
+		publicationDate: String
+	}
+
+	# Paginated pull list response
+	type MetadataPullListResponse {
+		result: [MetadataPullListItem!]!
+		meta: MetadataPaginationMeta!
+	}
+
+	# Pagination metadata
+	type MetadataPaginationMeta {
+		currentPage: Int!
+		totalPages: Int!
+		pageSize: Int!
+		totalCount: Int!
+		hasNextPage: Boolean!
+		hasPreviousPage: Boolean!
+	}
+
+	# Story arc with enriched data
+	type StoryArc {
+		id: Int!
+		name: String!
+		deck: String
+		description: String
+		image: ImageUrls
+		issues: [Issue!]
+	}
+
+	# Generic ComicVine resource response
+	type ComicVineResourceResponse {
+		error: String!
+		limit: Int!
+		offset: Int!
+		number_of_page_results: Int!
+		number_of_total_results: Int!
+		status_code: Int!
+		results: [SearchResultItem!]!
+	}
+
+	# Volume detail response
+	type VolumeDetailResponse {
+		error: String!
+		status_code: Int!
+		results: Volume!
+	}
+
+	# Issues for series response
+	type IssuesForSeriesResponse {
+		error: String!
+		limit: Int!
+		offset: Int!
+		number_of_page_results: Int!
+		number_of_total_results: Int!
+		status_code: Int!
+		results: [Issue!]!
+	}
+
+	# ============================================
+	# Metron Types
+	# ============================================
+
+	# Generic Metron resource (flexible JSON response)
+	scalar JSON
+
+	type MetronResponse {
+		data: JSON
+		status: Int!
+	}
+
+	# ============================================
+	# Input Types
+	# ============================================
+
+	# Search parameters
+	input SearchInput {
+		query: String!
+		resources: String!
+		format: String
+		sort: String
+		field_list: String
+		limit: Int
+		offset: Int
+	}
+
+	# Volume-based search configuration
+	input VolumeSearchInput {
+		query: String!
+		resources: String!
+		format: String
+		limit: Int
+		offset: Int
+		fieldList: String
+		scorerConfiguration: ScorerConfigurationInput
+		rawFileDetails: JSON
+	}
+
+	# Scorer configuration for matching
+	input ScorerConfigurationInput {
+		searchParams: SearchParamsInput
+	}
+
+	# Search parameters for scoring
+	input SearchParamsInput {
+		name: String
+		number: String
+		year: String
+		volume: String
+	}
+
+	# Get volumes input
+	input GetVolumesInput {
+		volumeURI: String!
+		fieldList: String
+	}
+
+	# Get resource input
+	input GetResourceInput {
+		resources: String!
+		filter: String
+		fieldList: String
+	}
+
+	# Weekly pull list input
+	input WeeklyPullListInput {
+		startDate: String!
+		currentPage: Int!
+		pageSize: Int!
+	}
+
+	# Metron fetch resource input
+	input MetronFetchInput {
+		resource: String!
+		method: String!
+		query: String
+	}
+
+	# ============================================
+	# Queries
+	# ============================================
+
+	type Query {
+		"""
+		Search ComicVine for volumes, issues, characters, etc.
+		"""
+		searchComicVine(input: SearchInput!): ComicVineSearchResult!
+
+		"""
+		Advanced volume-based search with scoring and filtering
+		"""
+		volumeBasedSearch(input: VolumeSearchInput!): VolumeBasedSearchResponse!
+
+		"""
+		Get volume details by URI
+		"""
+		getVolume(input: GetVolumesInput!): VolumeDetailResponse!
+
+		"""
+		Get all issues for a series by comic object ID
+		"""
+		getIssuesForSeries(comicObjectId: ID!): IssuesForSeriesResponse!
+
+		"""
+		Get generic ComicVine resource (issues, volumes, etc.)
+		"""
+		getComicVineResource(input: GetResourceInput!): ComicVineResourceResponse!
+
+		"""
+		Get story arcs for a volume
+		"""
+		getStoryArcs(volumeId: Int!): [StoryArc!]!
+
+		"""
+		Get weekly pull list from League of Comic Geeks
+		"""
+		getWeeklyPullList(input: WeeklyPullListInput!): MetadataPullListResponse!
+
+		"""
+		Fetch resource from Metron API
+		"""
+		fetchMetronResource(input: MetronFetchInput!): MetronResponse!
+	}
+
+	# ============================================
+	# Mutations
+	# ============================================
+
+	type Mutation {
+		"""
+		Placeholder for future mutations
+		"""
+		_empty: String
+	}
+`;

package.json

@@ -20,6 +20,7 @@
   ],
   "author": "",
   "devDependencies": {
+    "@faker-js/faker": "^9.7.0",
     "@types/jsdom": "^16.2.14",
     "@types/lodash": "^4.14.171",
     "@types/string-similarity": "^4.0.0",
@@ -31,11 +32,21 @@
     "jest": "^25.1.0",
     "jest-cli": "^25.1.0",
     "moleculer-repl": "^0.6.2",
+    "puppeteer": "^24.7.1",
+    "puppeteer-extra": "^3.3.6",
+    "puppeteer-extra-plugin-stealth": "^2.11.2",
+    "telnet-client": "^2.2.5",
     "threetwo-ui-typings": "^1.0.14",
     "ts-jest": "^25.3.0",
-    "ts-node": "^8.8.1"
+    "ts-node": "^10.9.2",
+    "typescript": "^5.9.3"
   },
   "dependencies": {
+    "@graphql-tools/delegate": "^12.0.8",
+    "@graphql-tools/schema": "^10.0.31",
+    "@graphql-tools/stitch": "^10.1.12",
+    "@graphql-tools/utils": "^11.0.0",
+    "@graphql-tools/wrap": "^11.1.8",
     "@types/axios": "^0.14.0",
     "@types/jest": "^25.1.4",
     "@types/mkdirp": "^1.0.0",
@@ -46,18 +57,20 @@
     "delay": "^5.0.0",
     "dotenv": "^10.0.0",
     "got": "^12.0.1",
+    "graphql": "^16.13.1",
+    "graphql-tag": "^2.12.6",
     "imghash": "^0.0.9",
     "ioredis": "^4.28.1",
     "jsdom": "^19.0.0",
     "leven": "^3.1.0",
     "lodash": "^4.17.21",
     "moleculer": "^0.14.28",
+    "moleculer-apollo-server": "^0.4.0",
     "moleculer-web": "^0.10.5",
     "nats": "^1.3.2",
     "paginate-info": "^1.0.4",
     "query-string": "^7.0.1",
-    "string-similarity": "^4.0.4",
-    "typescript": "^3.8.3"
+    "string-similarity": "^4.0.4"
   },
   "engines": {
     "node": ">= 10.x.x"

services/api.service.ts

@@ -56,6 +56,163 @@ export default class ApiService extends Service {
 						// Enable/disable logging
 						logging: true,
 					},
+					// GraphQL Gateway endpoint with schema stitching
+					{
+						path: "/graphql",
+						whitelist: ["gateway.query"],
+						cors: {
+							origin: "*",
+							methods: ["GET", "POST", "OPTIONS"],
+							allowedHeaders: ["*"],
+							exposedHeaders: [],
+							credentials: false,
+							maxAge: 3600,
+						},
+						aliases: {
+							"POST /": async (req: any, res: any) => {
+								try {
+									const { query, variables, operationName } = req.body;
+									
+									const result = await req.$ctx.broker.call("gateway.query", {
+										query,
+										variables,
+										operationName,
+									});
+
+									res.setHeader("Content-Type", "application/json");
+									res.end(JSON.stringify(result));
+								} catch (error: any) {
+									res.statusCode = 500;
+									res.setHeader("Content-Type", "application/json");
+									res.end(JSON.stringify({
+										errors: [{
+											message: error.message,
+											extensions: {
+												code: error.code || "INTERNAL_SERVER_ERROR",
+											},
+										}],
+									}));
+								}
+							},
+							"GET /": async (req: any, res: any) => {
+								// Support GraphQL Playground/introspection via GET
+								const query = req.$params.query;
+								const variables = req.$params.variables
+									? JSON.parse(req.$params.variables)
+									: undefined;
+								const operationName = req.$params.operationName;
+
+								try {
+									const result = await req.$ctx.broker.call("gateway.query", {
+										query,
+										variables,
+										operationName,
+									});
+
+									res.setHeader("Content-Type", "application/json");
+									res.end(JSON.stringify(result));
+								} catch (error: any) {
+									res.statusCode = 500;
+									res.setHeader("Content-Type", "application/json");
+									res.end(JSON.stringify({
+										errors: [{
+											message: error.message,
+											extensions: {
+												code: error.code || "INTERNAL_SERVER_ERROR",
+											},
+										}],
+									}));
+								}
+							},
+						},
+						bodyParsers: {
+							json: {
+								strict: false,
+								limit: "1MB",
+							},
+						},
+						mappingPolicy: "restrict",
+						logging: true,
+					},
+					// Standalone metadata GraphQL endpoint (no stitching)
+					// This endpoint exposes only the local metadata schema for external services to stitch
+					{
+						path: "/metadata-graphql",
+						whitelist: ["gateway.queryLocal"],
+						cors: {
+							origin: "*",
+							methods: ["GET", "POST", "OPTIONS"],
+							allowedHeaders: ["*"],
+							exposedHeaders: [],
+							credentials: false,
+							maxAge: 3600,
+						},
+						aliases: {
+							"POST /": async (req: any, res: any) => {
+								try {
+									const { query, variables, operationName } = req.body;
+									
+									const result = await req.$ctx.broker.call("gateway.queryLocal", {
+										query,
+										variables,
+										operationName,
+									});
+
+									res.setHeader("Content-Type", "application/json");
+									res.end(JSON.stringify(result));
+								} catch (error: any) {
+									res.statusCode = 500;
+									res.setHeader("Content-Type", "application/json");
+									res.end(JSON.stringify({
+										errors: [{
+											message: error.message,
+											extensions: {
+												code: error.code || "INTERNAL_SERVER_ERROR",
+											},
+										}],
+									}));
+								}
+							},
+							"GET /": async (req: any, res: any) => {
+								// Support GraphQL Playground/introspection via GET
+								const query = req.$params.query;
+								const variables = req.$params.variables
+									? JSON.parse(req.$params.variables)
+									: undefined;
+								const operationName = req.$params.operationName;
+
+								try {
+									const result = await req.$ctx.broker.call("gateway.queryLocal", {
+										query,
+										variables,
+										operationName,
+									});
+
+									res.setHeader("Content-Type", "application/json");
+									res.end(JSON.stringify(result));
+								} catch (error: any) {
+									res.statusCode = 500;
+									res.setHeader("Content-Type", "application/json");
+									res.end(JSON.stringify({
+										errors: [{
+											message: error.message,
+											extensions: {
+												code: error.code || "INTERNAL_SERVER_ERROR",
+											},
+										}],
+									}));
+								}
+							},
+						},
+						bodyParsers: {
+							json: {
+								strict: false,
+								limit: "1MB",
+							},
+						},
+						mappingPolicy: "restrict",
+						logging: true,
+					},
 				],
 				// Do not log client side errors (does not log an error response when the error.code is 400<=X<500)
 				log4XXResponses: false,

services/comicvine.service.ts

@@ -5,11 +5,7 @@ import axios from "axios";
 import { isNil, isUndefined } from "lodash";
 import { fetchReleases, FilterTypes, SortTypes } from "comicgeeks";
 import { matchScorer, rankVolumes } from "../utils/searchmatchscorer.utils";
-import {
-	scrapeIssuesFromSeriesPage,
-	scrapeIssuePage,
-	getWeeklyPullList,
-} from "../utils/scraping.utils";
+import { scrapeIssuePage, getWeeklyPullList } from "../utils/scraping.utils";
 const { calculateLimitAndOffset, paginate } = require("paginate-info");
 const { MoleculerError } = require("moleculer").Errors;
 
@@ -108,22 +104,10 @@ export default class ComicVineService extends Service {
 						return issues.data;
 					},
 				},
-				scrapeLOCGForSeries: {
-					rest: "POST /scrapeLOCGForSeries",
-					params: {},
-					handler: async (ctx: Context<{}>) => {
-						const seriesURIFragment = await scrapeIssuePage(
-							"https://leagueofcomicgeeks.com/comic/5878833/hulk-4"
-						);
-						return await scrapeIssuesFromSeriesPage(
-							`https://leagueofcomicgeeks.com/${seriesURIFragment}`
-						);
-					},
-				},
 				getWeeklyPullList: {
-					rest: "GET /getWeeklyPullList",
+					rest: "POST /scrapeLOCGForSeries",
+					timeout: 30000,
 					params: {},
-					timeout: 10000000,
 					handler: async (
 						ctx: Context<{
 							startDate: string;
@@ -131,26 +115,32 @@ export default class ComicVineService extends Service {
 							pageSize: string;
 						}>
 					) => {
-						const { currentPage, pageSize } = ctx.params;
+						const { currentPage, pageSize, startDate } = ctx.params;
+						console.log(`date for the pull list: ${startDate}`);
 						const { limit, offset } = calculateLimitAndOffset(
-							currentPage,
-							pageSize
+							parseInt(currentPage, 10),
+							parseInt(pageSize, 10)
 						);
 
-						const response = await getWeeklyPullList();
-						console.log(JSON.stringify(response, null, 4));
+						const url = `https://leagueofcomicgeeks.com/comics/new-comics/${startDate}`;
+						const issues = await getWeeklyPullList(url);
 
-						const count = response.length;
-						const paginatedData = response.slice(
+						const count = issues.length;
+						const paginatedData = issues.slice(
 							offset,
 							offset + limit
 						);
+
 						const paginationInfo = paginate(
-							currentPage,
+							parseInt(currentPage, 10),
 							count,
 							paginatedData
 						);
-						return { result: paginatedData, meta: paginationInfo };
+
+						return {
+							result: paginatedData,
+							meta: paginationInfo,
+						};
 					},
 				},
 				getResource: {
@@ -186,7 +176,7 @@ export default class ComicVineService extends Service {
 								field_list: `${fieldList}`,
 							},
 							headers: {
-								Accept: "application/json",
+								"Accept": "application/json",
 								"User-Agent": "ThreeTwo",
 							},
 						});
@@ -230,29 +220,70 @@ export default class ComicVineService extends Service {
 								"passed to fetchVolumesFromCV",
 								ctx.params
 							);
+							
+							// Send initial status to client
+							await this.broker.call("socket.broadcast", {
+								namespace: "/",
+								event: "CV_SCRAPING_STATUS",
+								args: [
+									{
+										message: `Starting volume search for: ${ctx.params.scorerConfiguration.searchParams.name}`,
+										stage: "fetching_volumes"
+									},
+								],
+							});
+							
 							const volumes = await this.fetchVolumesFromCV(
 								ctx.params,
 								results
 							);
+							
+							// Notify client that volume fetching is complete
+							await this.broker.call("socket.broadcast", {
+								namespace: "/",
+								event: "CV_SCRAPING_STATUS",
+								args: [
+									{
+										message: `Fetched ${volumes.length} volumes, now ranking matches...`,
+										stage: "ranking_volumes"
+									},
+								],
+							});
+							
 							// 1. Run the current batch of volumes through the matcher
 							const potentialVolumeMatches = rankVolumes(
 								volumes,
 								ctx.params.scorerConfiguration
 							);
-
+							
+							// Sort by totalScore in descending order to prioritize best matches
+							potentialVolumeMatches.sort((a: any, b: any) => b.totalScore - a.totalScore);
+							
+							// Notify client about ranked matches
+							await this.broker.call("socket.broadcast", {
+								namespace: "/",
+								event: "CV_SCRAPING_STATUS",
+								args: [
+									{
+										message: `Found ${potentialVolumeMatches.length} potential volume matches, searching for issues...`,
+										stage: "searching_issues"
+									},
+								],
+							});
+							
 							// 2. Construct the filter string
 							// 2a. volume: 1111|2222|3333
 							let volumeIdString = "volume:";
 							potentialVolumeMatches.map(
-								(volumeId: string, idx: number) => {
+								(volumeMatch: any, idx: number) => {
 									if (
 										idx >=
 										potentialVolumeMatches.length - 1
 									) {
-										volumeIdString += `${volumeId}`;
+										volumeIdString += `${volumeMatch.id}`;
 										return volumeIdString;
 									}
-									volumeIdString += `${volumeId}|`;
+									volumeIdString += `${volumeMatch.id}|`;
 								}
 							);
 
@@ -289,13 +320,46 @@ export default class ComicVineService extends Service {
 									filter: filterString,
 								},
 								headers: {
-									Accept: "application/json",
+									"Accept": "application/json",
 									"User-Agent": "ThreeTwo",
 								},
 							});
 							console.log(
 								`Total issues matching the criteria: ${issueMatches.data.results.length}`
 							);
+							
+							// Handle case when no issues are found
+							if (issueMatches.data.results.length === 0) {
+								await this.broker.call("socket.broadcast", {
+									namespace: "/",
+									event: "CV_SCRAPING_STATUS",
+									args: [
+										{
+											message: `No matching issues found. Try adjusting your search criteria.`,
+											stage: "complete"
+										},
+									],
+								});
+								
+								return {
+									finalMatches: [],
+									rawFileDetails,
+									scorerConfiguration,
+								};
+							}
+							
+							// Notify client about issue matches found
+							await this.broker.call("socket.broadcast", {
+								namespace: "/",
+								event: "CV_SCRAPING_STATUS",
+								args: [
+									{
+										message: `Found ${issueMatches.data.results.length} issue matches, fetching volume details...`,
+										stage: "fetching_volume_details"
+									},
+								],
+							});
+							
 							// 3. get volume information for the issue matches
 							if (issueMatches.data.results.length === 1) {
 								const volumeInformation =
@@ -309,9 +373,44 @@ export default class ComicVineService extends Service {
 									);
 								issueMatches.data.results[0].volumeInformation =
 									volumeInformation;
-								return issueMatches.data;
+								
+								// Notify scoring for single match
+								await this.broker.call("socket.broadcast", {
+									namespace: "/",
+									event: "CV_SCRAPING_STATUS",
+									args: [
+										{
+											message: `Scoring 1 match...`,
+											stage: "scoring_matches"
+										},
+									],
+								});
+								
+								// Score the single match
+								const scoredMatch = await this.broker.call(
+									"comicvine.getComicVineMatchScores",
+									{
+										finalMatches: issueMatches.data.results,
+										rawFileDetails,
+										scorerConfiguration,
+									}
+								);
+								
+								// Notify completion
+								await this.broker.call("socket.broadcast", {
+									namespace: "/",
+									event: "CV_SCRAPING_STATUS",
+									args: [
+										{
+											message: `Search complete! Found 1 match.`,
+											stage: "complete"
+										},
+									],
+								});
+								
+								return scoredMatch;
 							}
-							const finalMatches = issueMatches.data.results.map(
+							const finalMatchesPromises = issueMatches.data.results.map(
 								async (issue: any) => {
 									const volumeDetails =
 										await this.broker.call(
@@ -325,9 +424,24 @@ export default class ComicVineService extends Service {
 									return issue;
 								}
 							);
-
+	
+							// Wait for all volume details to be fetched
+							const finalMatches = await Promise.all(finalMatchesPromises);
+	
+							// Notify client about scoring
+							await this.broker.call("socket.broadcast", {
+								namespace: "/",
+								event: "CV_SCRAPING_STATUS",
+								args: [
+									{
+										message: `Scoring ${finalMatches.length} matches...`,
+										stage: "scoring_matches"
+									},
+								],
+							});
+	
 							// Score the final matches
-							const foo = await this.broker.call(
+							const scoredMatches = await this.broker.call(
 								"comicvine.getComicVineMatchScores",
 								{
 									finalMatches,
@@ -335,17 +449,52 @@ export default class ComicVineService extends Service {
 									scorerConfiguration,
 								}
 							);
-							return Promise.all(finalMatches);
+							
+							// Notify completion
+							await this.broker.call("socket.broadcast", {
+								namespace: "/",
+								event: "CV_SCRAPING_STATUS",
+								args: [
+									{
+										message: `Search complete! Returning scored matches.`,
+										stage: "complete"
+									},
+								],
+							});
+							
+							return scoredMatches;
 						} catch (error) {
-							console.log(error);
+							console.error("Error in volumeBasedSearch:", error);
+							
+							// Surface error to UI
+							await this.broker.call("socket.broadcast", {
+								namespace: "/",
+								event: "CV_SCRAPING_STATUS",
+								args: [
+									{
+										message: `Error during search: ${error.message || 'Unknown error'}`,
+										stage: "error",
+										error: {
+											message: error.message,
+											code: error.code,
+											type: error.type,
+											retryable: error.retryable
+										}
+									},
+								],
+							});
+							
+							// Re-throw or return error response
+							throw error;
 						}
 					},
 				},
 				getComicVineMatchScores: {
 					rest: "POST /getComicVineMatchScores",
+					timeout: 120000, // 2 minutes - allows time for image downloads and hash calculations
 					handler: async (
 						ctx: Context<{
-							finalMatches: Array<any>;
+							finalMatches: any[];
 							rawFileDetails: any;
 							scorerConfiguration: any;
 						}>
@@ -382,7 +531,7 @@ export default class ComicVineService extends Service {
 									resources: "volumes",
 								},
 								headers: {
-									Accept: "application/json",
+									"Accept": "application/json",
 									"User-Agent": "ThreeTwo",
 								},
 							});
@@ -401,7 +550,7 @@ export default class ComicVineService extends Service {
 															format: "json",
 														},
 														headers: {
-															Accept: "application/json",
+															"Accept": "application/json",
 															"User-Agent":
 																"ThreeTwo",
 														},
@@ -493,7 +642,7 @@ export default class ComicVineService extends Service {
 									limit: 100,
 								},
 								headers: {
-									Accept: "application/json",
+									"Accept": "application/json",
 									"User-Agent": "ThreeTwo",
 								},
 							});
@@ -508,7 +657,7 @@ export default class ComicVineService extends Service {
 										: null; // Extract the year from cover_date
 									return {
 										...issue,
-										year: year,
+										year,
 										description: issue.description || "",
 										image: issue.image || {},
 									};
@@ -548,7 +697,7 @@ export default class ComicVineService extends Service {
 							resources,
 						},
 						headers: {
-							Accept: "application/json",
+							"Accept": "application/json",
 							"User-Agent": "ThreeTwo",
 						},
 					});

services/gateway.service.ts

@@ -0,0 +1,193 @@
+import { Service, ServiceBroker } from "moleculer";
+import { ApolloServer } from "@apollo/server";
+import { stitchSchemas } from "@graphql-tools/stitch";
+import { print, getIntrospectionQuery, buildClientSchema } from "graphql";
+import { AsyncExecutor } from "@graphql-tools/utils";
+import axios from "axios";
+import { typeDefs } from "../models/graphql/typedef";
+import { resolvers } from "../models/graphql/resolvers";
+
+/**
+ * GraphQL Gateway Service with Schema Stitching
+ * Combines the local metadata schema with the remote GraphQL server
+ */
+export default class GatewayService extends Service {
+	private apolloServer?: ApolloServer;
+	private localApolloServer?: ApolloServer;
+
+	public constructor(broker: ServiceBroker) {
+		super(broker);
+
+		this.parseServiceSchema({
+			name: "gateway",
+
+			settings: {
+				remoteGraphQLUrl: process.env.REMOTE_GRAPHQL_URL || "http://localhost:3000/graphql",
+			},
+
+			actions: {
+				/**
+				 * Execute a GraphQL query through the stitched schema
+				 */
+				query: {
+					params: {
+						query: "string",
+						variables: { type: "object", optional: true },
+						operationName: { type: "string", optional: true },
+					},
+					async handler(ctx: any) {
+						if (!this.apolloServer) {
+							throw new Error("Apollo Gateway Server not initialized");
+						}
+
+						const { query, variables, operationName } = ctx.params;
+
+						const response = await this.apolloServer.executeOperation(
+							{ query, variables, operationName },
+							{ contextValue: { broker: this.broker, ctx } }
+						);
+
+						return response.body.kind === "single" ? response.body.singleResult : response;
+					},
+				},
+
+				/**
+				 * Execute a GraphQL query against local metadata schema only
+				 */
+				queryLocal: {
+					params: {
+						query: "string",
+						variables: { type: "object", optional: true },
+						operationName: { type: "string", optional: true },
+					},
+					async handler(ctx: any) {
+						if (!this.localApolloServer) {
+							throw new Error("Local Apollo Server not initialized");
+						}
+
+						const { query, variables, operationName } = ctx.params;
+
+						const response = await this.localApolloServer.executeOperation(
+							{ query, variables, operationName },
+							{ contextValue: { broker: this.broker, ctx } }
+						);
+
+						return response.body.kind === "single" ? response.body.singleResult : response;
+					},
+				},
+			},
+
+			methods: {
+				/**
+				 * Create an executor for the remote GraphQL server
+				 */
+				createRemoteExecutor(): AsyncExecutor {
+					const remoteUrl = this.settings.remoteGraphQLUrl;
+
+					return async ({ document, variables }) => {
+						try {
+							const response = await axios.post(
+								remoteUrl,
+								{ query: print(document), variables },
+								{ headers: { "Content-Type": "application/json" }, timeout: 30000 }
+							);
+							return response.data;
+						} catch (error: any) {
+							return {
+								errors: [{
+									message: `Remote server error: ${error.message}`,
+									extensions: { code: "REMOTE_GRAPHQL_ERROR" },
+								}],
+							};
+						}
+					};
+				},
+
+				/**
+				 * Initialize Apollo Server with stitched schema
+				 */
+				async initApolloGateway() {
+					this.logger.info("Initializing Apollo Gateway...");
+
+					const { makeExecutableSchema } = await import("@graphql-tools/schema");
+					const { execute } = await import("graphql");
+
+					// Create local schema
+					const localSchema = makeExecutableSchema({ typeDefs, resolvers });
+
+					// Create standalone local Apollo Server for /metadata-graphql endpoint
+					this.localApolloServer = new ApolloServer({ schema: localSchema, introspection: true });
+					await this.localApolloServer.start();
+					this.logger.info("Local metadata Apollo Server started");
+
+					// Create local executor
+					const localExecutor: AsyncExecutor = async ({ document, variables, context }) => {
+						return execute({
+							schema: localSchema,
+							document,
+							variableValues: variables,
+							contextValue: { broker: context?.broker || this.broker, ctx: context?.ctx },
+						}) as any;
+					};
+
+					// Try to introspect remote schema
+					let remoteSchema = null;
+					try {
+						const response = await axios.post(
+							this.settings.remoteGraphQLUrl,
+							{ query: getIntrospectionQuery() },
+							{ headers: { "Content-Type": "application/json" }, timeout: 30000 }
+						);
+
+						if (!response.data.errors) {
+							remoteSchema = buildClientSchema(response.data.data);
+							this.logger.info("Remote schema introspected successfully");
+						}
+					} catch (error: any) {
+						this.logger.warn(`Remote schema unavailable: ${error.message}`);
+					}
+
+					// Stitch schemas or use local only
+					const schema = remoteSchema
+						? stitchSchemas({
+								subschemas: [
+									{ schema: localSchema, executor: localExecutor },
+									{ schema: remoteSchema, executor: this.createRemoteExecutor() },
+								],
+								mergeTypes: false,
+						  })
+						: localSchema;
+
+					this.apolloServer = new ApolloServer({ schema, introspection: true });
+					await this.apolloServer.start();
+					this.logger.info("Apollo Gateway started");
+				},
+
+				/**
+				 * Stop Apollo Gateway Server
+				 */
+				async stopApolloGateway() {
+					if (this.localApolloServer) {
+						await this.localApolloServer.stop();
+						this.localApolloServer = undefined;
+					}
+					if (this.apolloServer) {
+						await this.apolloServer.stop();
+						this.apolloServer = undefined;
+					}
+				},
+			},
+
+			/**
+			 * Service lifecycle hooks
+			 */
+			started: async function (this: any) {
+				await this.initApolloGateway();
+			},
+
+			stopped: async function (this: any) {
+				await this.stopApolloGateway();
+			},
+		});
+	}
+}

services/metron.service.ts

@@ -27,19 +27,15 @@ export default class MetronService extends Service {
 						console.log(ctx.params);
 						const results = await axios({
 							method: "GET",
-							url: `https://metron.cloud/api/${ctx.params.resource}`,
+							url: `https://metron.cloud/api/${ctx.params.resource}/`,
 							params: {
 								name: ctx.params.query.name,
 								page: ctx.params.query.page,
 							},
-							headers: {
-								"Authorization": "Basic ZnJpc2hpOlRpdHVAMTU4OA=="
-							},
 							auth: {
-								"username": "frishi",
-								"password": "Titu@1588"
+								username: "frishi",
+								password: "Titu@1588"
 							}
-
 						});
 						return results.data;
 					},

tsconfig.json

@@ -8,7 +8,8 @@
 		"sourceMap": true,
 		"pretty": true,
 		"target": "es6",
-		"outDir": "dist"
+		"outDir": "dist",
+		"skipLibCheck": true,
 	},
 	"include": ["./**/*"],
 	"exclude": [

utils/scraping.utils.ts

@@ -1,64 +1,103 @@
-import jsdom from "jsdom";
+import puppeteer from "puppeteer-extra";
+import StealthPlugin from "puppeteer-extra-plugin-stealth";
+import { faker } from "@faker-js/faker";
 import axios from "axios";
-const { JSDOM } = jsdom;
+import { JSDOM } from "jsdom";
 
-export const scrapeIssuesFromSeriesPage = async (url: string) => {
-	const response = await axios(url);
-	const dom = new JSDOM(response.data, {
-		url,
-		referrer: url,
-		contentType: "text/html",
-		includeNodeLocations: true,
-		storageQuota: 10000000,
+// Optional Tor
+const useTor = process.env.USE_TOR === "true";
+const torProxy = process.env.TOR_SOCKS_PROXY || "socks5://192.168.1.119:9050";
+
+// Apply stealth plugin
+puppeteer.use(StealthPlugin());
+
+export const getWeeklyPullList = async (url: string) => {
+	const browser = await puppeteer.launch({
+		headless: true,
+		slowMo: 50,
+		args: useTor
+			? [`--proxy-server=${torProxy}`]
+			: ["--no-sandbox", "--disable-setuid-sandbox"],
 	});
-	const seriesId = dom.window.document
-		.querySelector("#comic-list-block")
-		.getAttribute("data-series-id");
-	const issueNodes = dom.window.document.querySelectorAll(
-		"ul.comic-list-thumbs > li"
-	);
 
-	const issues: any = [];
-	issueNodes.forEach(node => {
-		const comicHref = node.querySelector("a").getAttribute("href");
-		const issueCoverImage = node.querySelector("img").getAttribute("src");
-		const issueDetails = node.querySelector("img").getAttribute("alt");
-        const issueDate = node.querySelector("span.date").getAttribute("data-date");
-        const formattedIssueDate = node.querySelector("span.date").textContent.trim();
-        const publisher = node.querySelector("div.publisher").textContent.trim();
+	const page = await browser.newPage();
 
-		issues.push({
-			comicHref,
-			issueCoverImage,
-			issueDetails,
-            issueDate,
-            formattedIssueDate,
-            publisher,
+	await page.setExtraHTTPHeaders({
+		"Accept-Language": "en-US,en;q=0.9",
+		"Referer": "https://leagueofcomicgeeks.com/",
+	});
+
+	await page.setUserAgent(faker.internet.userAgent());
+
+	await page.setViewport({
+		width: faker.number.int({ min: 1024, max: 1920 }),
+		height: faker.number.int({ min: 768, max: 1080 }),
+	});
+
+	try {
+		await page.goto(url, {
+			waitUntil: "domcontentloaded", // Faster and more reliable for JS-rendered content
+			timeout: 30000, // Give it time on Tor or slow networks
 		});
-	});
-	return {
-		seriesId,
-		issues,
-	};
+
+		await page.waitForSelector(".issue", { timeout: 30000 });
+		console.log("✅ Found .issue blocks");
+
+		return await page.evaluate(() => {
+			const issues = Array.from(document.querySelectorAll(".issue"));
+
+			return issues.map(issue => {
+				const issueUrlElement = issue.querySelector(".cover a");
+				const coverImageElement =
+					issue.querySelector(".cover img.lazy");
+				const publisherText =
+					issue.querySelector("div.publisher")?.textContent?.trim() ||
+					null;
+				const issueName =
+					issue
+						.querySelector("div.title")
+						?.getAttribute("data-sorting") || null;
+
+				// Convert Unix timestamp (in seconds) to YYYY-MM-DD
+				const publicationDateRaw = issue
+					.querySelector(".date")
+					?.getAttribute("data-date");
+				const publicationDate = publicationDateRaw
+					? new Date(parseInt(publicationDateRaw, 10) * 1000)
+							.toISOString()
+							.split("T")[0]
+					: null;
+
+				const imageUrl =
+					coverImageElement?.getAttribute("data-src") ||
+					coverImageElement?.getAttribute("src") ||
+					null;
+
+				const coverImageUrl = imageUrl
+					? imageUrl.replace(/\/medium-(\d+\.jpg)/, "/large-$1")
+					: null;
+
+				const issueUrl = issueUrlElement?.getAttribute("href") || null;
+
+				return {
+					issueName,
+					coverImageUrl,
+					issueUrl,
+					publisher: publisherText,
+					publicationDate,
+				};
+			});
+		});
+	} catch (err) {
+		console.error("❌ Scraper error:", err);
+		throw err;
+	} finally {
+		await browser.close();
+	}
 };
 
+
 export const scrapeIssuePage = async (url: string) => {
-    const response = await axios(url);
-    const dom = new JSDOM(response.data, {
-		url,
-		referrer: url,
-		contentType: "text/html",
-		includeNodeLocations: true,
-		storageQuota: 10000000,
-	});
-    const seriesDOMElement = dom.window.document
-		.querySelector("div.series-pagination > a.series").getAttribute("href");
-    return seriesDOMElement;
-};
-
-
-export const getWeeklyPullList = async () => {
-	const url = "https://www.tfaw.com/comics/new-releases.html";
 	const response = await axios(url);
 	const dom = new JSDOM(response.data, {
 		url,
@@ -67,22 +106,8 @@ export const getWeeklyPullList = async () => {
 		includeNodeLocations: true,
 		storageQuota: 10000000,
 	});
-
-	const pullList: any[] = [];
-	// Node for the comics container
-	const issueNodes = dom.window.document.querySelectorAll("ol.products > li");
-
-	issueNodes.forEach(node => {
-		const coverImageUrl = node.querySelector("img.photo").getAttribute("data-src");
-		const name = node.querySelector("div.product > a.product").textContent.trim();
-		const publicationDate = node.querySelector("div.product-item-date").textContent.trim();
-		pullList.push({
-			coverImageUrl,
-			name,
-			publicationDate,
-		});
-	});
-
-	return pullList;
-
+	const seriesDOMElement = dom.window.document
+		.querySelector("div.series-pagination > a.series")
+		.getAttribute("href");
+	return seriesDOMElement;
 };

utils/searchmatchscorer.utils.ts

@@ -42,16 +42,15 @@ import { isAfter, isSameYear, parseISO } from "date-fns";
 const imghash = require("imghash");
 
 export const matchScorer = async (
-	searchMatches: Promise<any>[],
+	searchMatches: any[],
 	searchQuery: any,
 	rawFileDetails: any
 ): Promise<any> => {
 	const scoredMatches: any = [];
 
 	try {
-		const matches = await Promise.all(searchMatches);
-
-		for (const match of matches) {
+		// searchMatches is already an array of match objects, not promises
+		for (const match of searchMatches) {
 			match.score = 0;
 
 			// Check for the issue name match
@@ -93,7 +92,7 @@ export const rankVolumes = (volumes: any, scorerConfiguration: any) => {
 	// 2. If there is a strong string comparison between the volume name and the issue  name ??
 	const issueNumber = parseInt(scorerConfiguration.searchParams.number, 10);
 	const issueYear = parseISO(scorerConfiguration.searchParams.year);
-	const foo = volumes.map((volume: any, idx: number) => {
+	const rankedVolumes = volumes.map((volume: any, idx: number) => {
 		let volumeMatchScore = 0;
 		const volumeStartYear = !isNil(volume.start_year)
 			? parseISO(volume.start_year)
@@ -132,67 +131,77 @@ export const rankVolumes = (volumes: any, scorerConfiguration: any) => {
 		// 3. If issue number falls in the range of candidate volume's first issue # and last issue #, +3 to volumeMatchScore
 		if (!isNil(firstIssueNumber) && !isNil(lastIssueNumber)) {
 			if (
-				firstIssueNumber <= issueNumber ||
+				firstIssueNumber <= issueNumber &&
 				issueNumber <= lastIssueNumber
 			) {
 				volumeMatchScore += 3;
 			}
 		}
 		if (issueNameMatchScore > 0.5 && volumeMatchScore > 2) {
-			console.log(`Found a match for criteria, volume ID: ${volume.id}`);
-			return volume.id;
+			console.log(`Found a match for criteria, volume ID: ${volume.id}, score: ${volumeMatchScore}, name match: ${issueNameMatchScore.toFixed(2)}`);
+			return {
+				id: volume.id,
+				volumeMatchScore,
+				issueNameMatchScore,
+				totalScore: volumeMatchScore + issueNameMatchScore
+			};
 		}
+		return null;
 	});
-	return foo.filter((item: any) => !isNil(item));
+	return rankedVolumes.filter((item: any) => !isNil(item));
 };
 
 const calculateLevenshteinDistance = async (match: any, rawFileDetails: any) =>
-	new Promise((resolve, reject) => {
+	new Promise((resolve) => {
 		https.get(match.image.small_url, (response: any) => {
 			console.log(rawFileDetails.cover.filePath);
 			const fileName = match.id + "_" + rawFileDetails.name + ".jpg";
 			// Ensure the `temporary` directory exists
-			if (!existsSync("temporary")) {
-				mkdirSync("temporary", { recursive: true });
+			const tempDir = path.join(`${process.env.USERDATA_DIRECTORY}`, "temporary");
+			if (!existsSync(tempDir)) {
+				mkdirSync(tempDir, { recursive: true });
 			}
 			const file = createWriteStream(
 				`${process.env.USERDATA_DIRECTORY}/temporary/${fileName}`
 			);
 			const fileStream = response.pipe(file);
 			fileStream.on("finish", async () => {
-				// 1. hash of the cover image we have on hand
-				const coverFileName = rawFileDetails.cover.filePath
-					.split("/")
-					.at(-1);
-				const coverDirectory = rawFileDetails.containedIn
-					.split("/")
-					.at(-1);
-				const hash1 = await imghash.hash(
-					path.resolve(
-						`${process.env.USERDATA_DIRECTORY}/covers/${coverDirectory}/${coverFileName}`
-					)
-				);
-				// 2. hash of the cover of the potential match
-				const hash2 = await imghash.hash(
-					path.resolve(
-						`${process.env.USERDATA_DIRECTORY}/temporary/${fileName}`
-					)
-				);
-				if (!isUndefined(hash1) && !isUndefined(hash2)) {
-					const levenshteinDistance = leven(hash1, hash2);
-					if (levenshteinDistance === 0) {
-						match.score += 2;
-					} else if (
-						levenshteinDistance > 0 &&
-						levenshteinDistance <= 2
-					) {
-						match.score += 1;
-					} else {
-						match.score -= 2;
+				try {
+					// 1. hash of the cover image we have on hand
+					const coverFileName = rawFileDetails.cover.filePath
+						.split("/")
+						.at(-1);
+					const coverDirectory = rawFileDetails.containedIn
+						.split("/")
+						.at(-1);
+					const hash1 = await imghash.hash(
+						path.resolve(
+							`${process.env.USERDATA_DIRECTORY}/covers/${coverDirectory}/${coverFileName}`
+						)
+					);
+					// 2. hash of the cover of the potential match
+					const hash2 = await imghash.hash(
+						path.resolve(
+							`${process.env.USERDATA_DIRECTORY}/temporary/${fileName}`
+						)
+					);
+					if (!isUndefined(hash1) && !isUndefined(hash2)) {
+						const levenshteinDistance = leven(hash1, hash2);
+						if (levenshteinDistance === 0) {
+							match.score += 2;
+						} else if (
+							levenshteinDistance > 0 &&
+							levenshteinDistance <= 2
+						) {
+							match.score += 1;
+						} else {
+							match.score -= 2;
+						}
 					}
 					resolve(match);
-				} else {
-					reject({ error: "Couldn't calculate hashes." });
+				} catch (err) {
+					console.warn(`Image hashing failed for ${fileName}, skipping score adjustment:`, err.message);
+					resolve(match);
 				}
 			});
 		});