Tools Documentation & User Guide

Overview

This is your complete toolkit for managing Chia news content, built to transition from manual HTML editing to a modern JSON-based system while maintaining your current workflow.

Project Structure

/mnt/artoo/e/DEV/twic-parser/
|
+-- Data Files
|   +-- chia_news_cleaned.json         # Your main dataset (validated & cleaned)
|   +-- chia_news.json                 # Original parsed data
|   +-- test_data.json                 # Safe testing environment
|   +-- chia_news_validation_report.json
|
+-- Core Tools
|   +-- tweet-converter.js             # Convert single tweets to JSON
|   +-- batch-tweets.js                # Batch processing & interactive mode + HTML gen
|   +-- json-manager.js                # Manage & clean JSON data
|   +-- validate.js                    # Data quality validation
|   +-- cleanup.js                     # Fix data quality issues
|
+-- Development
|   +-- parser.js                      # Original HTML parser
|   +-- index.html                     # Current production site
|
+-- Node Dependencies
    +-- node_modules/                  # (not needed for basic tools)

Tool Reference Guide

# Convert single tweet
node tweet-converter.js https://x.com/username/status/1234567890

# With custom JSON file
node tweet-converter.js https://x.com/username/status/1234567890 my_data.json

node batch-tweets.js interactive

node batch-tweets.js test

# List recent posts
node json-manager.js list 20

# Search posts
node json-manager.js search "chia wallet"

# Remove specific post
node json-manager.js remove 2025-06-17-123

# Remove multiple posts
node json-manager.js bulk 2025-06-17-123,2025-06-16-456

# Show statistics
node json-manager.js stats

# Validate your main dataset
node validate.js chia_news_cleaned.json

# Validate with detailed reporting
node validate.js chia_news.json

# Analyze warning patterns
node cleanup.js analyze chia_news.json

# Clean the data automatically  
node cleanup.js clean chia_news.json

Daily Workflow Recommendations

📱 For Regular News Posting (UPDATED)

cd /mnt/artoo/e/DEV/twic-parser
node batch-tweets.js interactive

node batch-tweets.js test

# Collect URLs in a file during the day
# Process all at once:
node batch-tweets.js file todays_tweets.txt

🔧 For Data Maintenance

📅 Weekly Validation

# Check data health
node validate.js chia_news_cleaned.json

# Clean if needed
node cleanup.js clean chia_news_cleaned.json

🗓️ Monthly Cleanup

# Interactive cleanup session
node json-manager.js interactive
# Use: list, search, remove as needed

HTML Generation Features NEW

✨ What Gets Generated

• 🎯 Category Icons: 🌎🚀 with proper tooltips
• 📝 Post Types: Special styling for X Spaces, Releases, etc.
• 👤 Author Attribution: "Hosted by" for spaces, regular for others
• 🔗 Quote Tweets: Plain URLs → "Quote Tweet" links
• 🎨 Smart Links: Watch/Listen/Read article based on destination
• 🛡️ Proper Escaping: Prevents HTML entity issues (like —)

📋 HTML Output Examples

🚀 X Space

<li class='post'><span title="Community">🌎</span> <span title="Space">🚀</span> <span style='color:#8C52FE'>X Space</span> - Hosted by <a href='https://x.com/DracattusDev' target='_blank'>@DracattusDev</a> "Weekly Chia discussion". • <a href='https://example.com' target='_blank'>Listen</a> • <a href='https://x.com/source' target='_blank'>Source</a></li>

📢 Announcement with Quote Tweet

<li class='post'><span title="Chia">🌱</span> <span title="News">📢</span> <a href='https://x.com/chia_project' target='_blank'>@chia_project</a> announces new partnership! <a href='https://x.com/partner/status/123' target='_blank'>Quote Tweet</a> for details. • <a href='https://x.com/chia_project/status/456' target='_blank'>Source</a></li>

▶️ Video Release

<li class='post'><span title="Community">🌎</span> <span title="Video">▶️</span> <a href='https://x.com/creator' target='_blank'>@creator</a> releases new tutorial video! • <a href='https://youtube.com/watch?v=123' target='_blank'>Watch</a> • <a href='https://x.com/creator/status/789' target='_blank'>Source</a></li>

Data Quality Metrics

📊 Current Status

🎯 Quality Indicators

• 🟢 Green (90-100): Production ready
• 🟡 Yellow (70-89): Minor issues, still usable
• 🔴 Red (0-69): Requires cleanup before use

Troubleshooting Guide

🔧 Common Issues & Solutions

🌐 Network Issues & Error Recovery

# Check error log files (auto-generated)
ls -la failed_tweets_*.txt

# Reprocess failed URLs
node batch-tweets.js file failed_tweets_2025-06-20T15-30-45-123Z.txt

# Test connectivity
curl -I https://publish.twitter.com
curl -I https://x.com

Advanced Usage

🧪 Working with Test Mode

# Start clean testing session
node batch-tweets.js test

# Test with custom file
node batch-tweets.js test my_experiments.json

# Commands available in test mode:
# clear - Reset test data
# stats - View test statistics
# quit - Exit test mode

🔄 Batch Error Recovery

# After batch processing, check for error logs
ls -la failed_tweets_*.txt

# Reprocess failed URLs (file format is ready-to-use)
node batch-tweets.js file failed_tweets_2025-06-20T15-30-45-123Z.txt

# Or manually retry individual URLs
node batch-tweets.js interactive
# Then paste the failed URLs one by one

🌐 Integration with Current Site

📊 Data Flow (UPDATED)

Twitter URL → tweet-converter.js → JSON Entry → batch-tweets.js → HTML + JSON
                                                      ↓
                                               Current Website ← Copy/Paste HTML
                                                      ↓
                                            chia_news_cleaned.json ← Future Site Data

⚡ Performance Metrics

🗺️ Development Roadmap

✅ COMPLETED

• HTML to JSON parser
• Tweet converter with URL resolution
• Batch processing tools
• Data validation & cleanup
• JSON management utilities
• HTML generation for current website
• Test mode for safe experimentation
• Retry logic and error handling
• Quote tweet link conversion

🚧 IN PROGRESS

• Documentation updates (this file!)

📋 PLANNED

• Twitter List webapp (keyboard-driven workflow)
• Manual entry tool (web form)
• Production server deployment
• Weekly/monthly automated reports

🎯 Success Metrics

You'll know the tools are working well when:

• Daily posting takes < 30 seconds per tweet
• HTML generated instantly for current website
• Data quality stays > 90% health score
• No manual JSON editing required
• No manual HTML coding required
• Backups created automatically
• Search finds content quickly
• Failed URLs automatically retried and logged
• Test mode experiments safely

🌟 What's New in This Version