Select dates to stream, decompress in-memory, and ingest raw news events directly into the PostgreSQL database (raw_gdelt_news) with deduplication.
ON CONFLICT (id) DO NOTHING.| Export File Name | Date | Size | Total Ingested | Remaining in Stage 1 | Status | Actions | |
|---|---|---|---|---|---|---|---|
| Loading Stage 1 imported files... | |||||||
raw_gdelt_news)| Published Date | Actor 1 | Actor 2 | Country | Source URL | Status | Action | |
|---|---|---|---|---|---|---|---|
| Loading Stage 1 raw records... | |||||||
Filters raw items from Stage 1 using NLP keyword analysis to determine PESTLE/Porter alignment, discards clickbait/noise, deduplicates headlines, and moves relevant records to stage2_filtered_news.
raw_gdelt_news) upon moving. Unwanted noise is filtered out and deduplicated across the pipeline.Select any imported archive from Stage 1 below. Click "Convert / Filter File" to run NLP categorization, assign strategic dimensions, move items into Stage 2, and purge them from Stage 1.
| Stage 1 File / Archive | Date | Remaining Items | Conversion Batch Size | Actions | |
|---|---|---|---|---|---|
| Loading Stage 1 files for conversion... | |||||
stage2_filtered_news)| Date | Extracted Headline | PESTLE/Porter Classification | Location | Source URL | Status | Action | |
|---|---|---|---|---|---|---|---|
| Loading Stage 2 filtered records... | |||||||
Takes items from stage2_filtered_news, generates 768-D embeddings, converts them using the linear projection model matrix to 11 strategic dimensions, and ingests them into the Master News Database (news_articles).
stage2_filtered_news). The master table enforces deduplication on existing headlines.
Select any filtered batch from Stage 2 below. Click "Convert & Project 11-D" to generate embeddings, project to 11 dimensions using the trained matrix, ingest into news_articles, and purge from Stage 2.
| Stage 2 Batch / File | Date | Filtered Items Waiting | Categories in Batch | Projection Batch Size | Actions | |
|---|---|---|---|---|---|---|
| Loading Stage 2 filtered batches... | ||||||
news_articles)| Date | Headline | Primary Driver | Impact Score | 11-D Strategic Vector Preview | Source | Action | |
|---|---|---|---|---|---|---|---|
| Loading Master News records... | |||||||
Execute end-to-end multi-stage pipeline runs. The orchestrator transitions data seamlessly from Stage 1 to Stage 2 and into Stage 3, enforcing automatic purging and deduplication at every step.
raw_gdelt_news with ID/URL deduplication.