Full-Text Extraction (NewsSync PRO)

🎯 What it does

Full-Text Extraction automatically fetches the complete article content from the source website instead of using the truncated RSS excerpt.

Benefits:

  • ✅ Complete articles — Full content instead of “read more…” teasers
  • ✅ Better user experience — Readers stay on your site (reduced bounce rate)
  • ✅ Richer content — Images, formatting, embedded media preserved
  • ✅ Automatic attribution — Source link added at bottom (proper credit)
  • ✅ Works with most sites — Uses Mozilla’s Readability algorithm (same as Firefox Reader Mode)

⚠️ Important: This feature extracts public content for aggregation purposes. Always respect copyright and terms of service.

🤔 When to use Full-Text Extraction

✅ Good Use Cases

1. News aggregators / Content hubs

  • Aggregate multiple feeds into one site
  • Users want all content in one place
  • Example: Tech news hub combining TechCrunch, Wired, The Verge

2. Internal knowledge bases

  • Company-wide feed reader
  • Private site for team/organization
  • Archive industry news for reference

3. Research / Academic

  • Collect articles for analysis
  • Preserve content that might disappear
  • Compare coverage across sources

4. RSS feeds with truncated content

  • Many feeds only provide first paragraph
  • Full article requires clicking through
  • Example: Medium, WordPress.com (default settings)

❌ When NOT to use

1. SEO-focused sites (duplicate content risk)

  • If your site is public and indexed by Google
  • Publishers may issue DMCA takedown notices
  • Can harm your search rankings

2. Ad-supported content sites

  • Publishers lose ad revenue when content is copied
  • Ethical concern (taking traffic from original)
  • Risk of legal action

3. Paywalled / Private content

  • Extraction often fails (login required)
  • Violates terms of service
  • May be illegal in some jurisdictions

⚙️ How to Enable

Quick Setup (2 steps)

Step 1: Go to WordPress Admin → NewsSync PRO → Settings → Display

Step 2: Check “Enable full-text extraction” and click Save Changes

✅ Done! New imports will now fetch full articles automatically.

🧪 Testing (verify it works)

Step-by-step test

  1. Find a test article: Choose a public article with good content (e.g., TechCrunch, BBC News)
  2. Import the feed: Use the article’s RSS feed
  3. Check the post: Open the created post in WordPress
  4. Verify:
    • ✅ Content is longer than RSS excerpt
    • ✅ Images are included
    • ✅ Source link appears at bottom: “Originally published at [source]”

Tip: Compare the WordPress post content with the original article URL to confirm extraction worked.

🌐 Site Compatibility

Full-Text Extraction works with most modern websites, but some sites block it:

✅ Usually Works (High Success Rate)

Site TypeExemplosSuccess Rate
News sitesBBC, Reuters, AP News, The Guardian~95%
Tech blogsTechCrunch, Ars Technica, The Verge~90%
WordPress sitesMost WP blogs (if not heavily customized)~85%
MediumPublic Medium articles~80%
Academic journalsOpen-access papers (PLOS, arXiv)~85%

⚠️ Sometimes Works (Medium Success Rate)

Site TypeExemplosIssues
Social platformsLinkedIn articles, Facebook notesLogin walls, anti-scraping
News aggregatorsGoogle News, Apple NewsOften just links (no content)
Heavily styled sitesMagazine layouts (Vogue, GQ)Content mixed with ads/layout

❌ Usually Fails (Low Success Rate)

Site TypeWhy it fails
Paywalled sitesNY Times, WSJ, Financial Times — Require login
JavaScript-heavySingle-page apps (React/Vue) — No HTML on first load
Anti-scraping sitesCloudFlare challenge, Captchas, IP blocking
PDF/Document sitesNot HTML (plugin can’t parse)

🔒 SEO & Legal Considerations

Duplicate Content (SEO Risk)

Problem: Google penalizes sites that copy content from other sites.

Solutions:

Option 1: Use noindex meta tag (Recommended)

// Add to theme's functions.php
add_action('wp_head', function() {
    if (is_singular() && has_tag('imported')) {
        echo '<meta name="robots" content="noindex, follow">';
    }
});

This tells Google to NOT index imported posts.

Option 2: Use Canonical tags (Advanced)

// Add canonical pointing to original article
add_action('wp_head', function() {
    if (is_singular()) {
        $original_url = get_post_meta(get_the_ID(), '_newssync_original_url', true);
        if ($original_url) {
            echo '<link rel="canonical" href="' . esc_url($original_url) . '">';
        }
    }
});

This tells Google the original source is the “real” version.

Option 3: Make site private

  • Use WordPress Privacy Settings → “Discourage search engines”
  • Or use password protection / membership plugin
  • Best for internal/private sites

Legal disclaimer: We are not lawyers. Consult legal counsel if unsure.

General guidance:

  • ✅ OK: Personal use, research, internal company use
  • ✅ OK (usually): News aggregation with attribution + source link
  • ⚠️ Risky: Public site with full copied articles (even with attribution)
  • ❌ NOT OK: Removing attribution, claiming content as yours, monetizing copied content

Best practice:

  1. Add clear attribution: “Source: Publisher Name“
  2. Only aggregate from sites that allow it (check Terms of Service)
  3. If publisher requests removal, comply immediately
  4. Consider excerpts + “Read more” links instead of full text

🔧 Troubleshooting

Extraction returned empty content

Causes:

  • Website uses JavaScript to load content (not in HTML)
  • Site blocked the request (anti-scraping protection)
  • Article is behind paywall/login

Fixes:

  • ✅ Test with different article from same site
  • ✅ Check if article loads in browser (not 404)
  • ✅ Try site that doesn’t use heavy JavaScript (test with BBC or TechCrunch)

Extraction includes ads / navigation / junk

Causes:

  • Site has unusual HTML structure
  • Readability algorithm confused by layout

Fixes:

  • Enable “Strip Scripts & Iframes” in PRO → Advanced
  • Enable “Strip Inline Styles” to clean formatting
  • Some junk is normal (algorithm isn’t perfect)

Extraction is slow (timeouts)

Causes:

  • Slow source website
  • Large articles (10,000+ words, many images)
  • Server has low memory/CPU

Fixes:

  • ✅ Increase PHP max_execution_time (ask hosting)
  • ✅ Disable full-text for high-volume feeds (enable selectively)
  • ✅ Use caching (default: 7 days — results cached per article)

Check:

  • Attribution is automatic (added by plugin)
  • Appears at bottom of post content
  • Look for text: “Originally published at [source]”

If missing:

  • Check if theme overrides the_content() filter
  • Try different theme temporarily to test

Extracted content looks broken

Causes:

  • Source site uses complex CSS/styling
  • Your theme conflicts with imported HTML

Fixes:

  • Enable “Strip Inline Styles” (PRO → Advanced)
  • Add custom CSS to your theme to style imported content
  • Accept some formatting loss (algorithm focuses on text, not design)

⚙️ Advanced Settings

Cache Duration

Default: 7 days

Extractions are cached to avoid repeated requests (faster + reduces server load).

When to adjust:

  • Longer (30 days): For archive/static content (saves resources)
  • Shorter (1 day): If articles update frequently (live news, stocks)

Clear cache:

// WordPress admin → Transients (or use WP-CLI)
wp transient delete --all

Content Cleaning (PRO → Advanced)

Strip Scripts & Iframes: ✅ ON (recommended)

  • Removes <script>, <iframe>, <object>, <embed>
  • Security: Prevents XSS attacks from untrusted sites

Strip Inline Styles: ⚠️ Optional

  • Removes style="..." attributes
  • Use if imported content breaks your layout

Excerpt Source:

  • Default: Use RSS description field
  • Alternative: Extract first paragraph from full text

💡 Pro Tips

1. Combine with AI Summaries

  • Enable both Full-Text Extraction + AI Summaries
  • Result: Full article + AI-generated excerpt for listings

2. Use separate post type for imports

  • Create “Imported Articles” CPT (see Managed Feeds docs)
  • Apply noindex only to imported CPT (keeps original content indexed)

3. Add “Read Original” button

// Add to theme's single.php or functions.php
$original_url = get_post_meta(get_the_ID(), '_newssync_original_url', true);
if ($original_url) {
    echo '<a href="/pt/' . esc_url($original_url) . '/" class="read-original-btn" target="_blank">📄 Read Original Article</a>';
}

4. Monitor extraction success rate

  • Check PRO Log regularly (Settings → PRO → Tools)
  • If many failures, disable for problematic feeds

5. Respect robots.txt

  • Check if site allows scraping: https://example.com/robots.txt
  • Look for User-agent: * and Disallow rules

🆘 Need Help?

Quick debugging:

  1. Check WordPress Debug Log (wp-content/debug.log)
  2. Look for errors: NewsSync PRO: Full-text extraction failed
  3. Test extraction manually: View source of original article (should be HTML, not JavaScript)

Support channels:

⚖️ Legal Reminder: Full-Text Extraction is a tool. You are responsible for how you use it. Respect copyright, terms of service, and ethical publishing practices. When in doubt, use excerpts + source links instead of full content.

Back to Documentation