Austin Breuker | Technical Operations

Overview

Transitioned the Network Latency Monitor from a basic diagnostic script into a resilient, decoupled observability platform. This release introduces continuous data logging, advanced network metrics (Jitter and Packet Loss), and an architecture engineered to bypass restrictive public cloud firewalls.

Key Improvements & Engineering Steps

1. Automated Continuity & Flat-File Database Architecture

• Action: Integrated an append-only long-term storage engine within the Python core.
• Impact: The system now automatically logs every data point to a continuous latency_history.csv file alongside the live latency_data.json payload. This establishes a historical record without the overhead or cost of a traditional relational database.

2. Linux Environment Optimization (Case-Sensitivity Fix)

• Action: Resolved a file execution block by aligning local development naming conventions with Linux production environments.
• Impact: Corrected the script call to strictly account for case sensitivity (Dashboard_test.py) within the Ubuntu-based GitHub Actions environment, ensuring flawless automation execution.

3. Firewall Traversal via TCP Handshakes

• Action: Migrated the core polling engine away from native OS ICMP ping commands to low-level Python socket connections (socket.AF_INET, socket.SOCK_STREAM).
• Impact: Overcame restrictive cloud provider security policies (Azure/GitHub Actions dropping outbound ICMP traffic). By targeting legitimate service ports (Port 443 for web hosts, Port 53 for DNS), the monitor accurately measures latency, packet loss, and jitter without being dropped by firewalls.

4. Full-Stack Data Visualization (NOC Trend Graph)

• Action: Deployed an interactive frontend rendering engine powered by Chart.js.
• Impact: The client-side dashboard safely fetches the raw historical CSV file from GitHub’s global content delivery network (CDN). It parses the time-series telemetry and dynamically draws a rolling 24-hour network trend line chart directly on the page, keeping server resource consumption at zero.

5. Decentralized Event Logging & Alerting Panel

• Action: Built an automated incident detection pipeline that outputs to an active alerts.json feed.
• Impact: The backend engine continuously evaluates connection statuses during each run. If a target goes “Offline” or becomes “Degraded,” a structured CRITICAL or WARNING event is generated. The WordPress frontend asynchronously hooks into this feed to display a live, color-coded System Alerts panel.

Technical Stack & Infrastructure Blueprint

• Compute: Serverless Cron via GitHub Actions (On-Demand Ubuntu Runners)
• Backend Engine: Python 3.14 (Socket, JSON, and Time Subsystems)
• Data Layer: Flat-File Store (.json for live state, .csv for time-series logs)
• Frontend Delivery: Vanilla JS + Chart.js embedded in custom web container

The Objective: Transform the Network Latency Monitor from a reactive dashboard into an intelligent, AIOps-driven observability platform by integrating automated, daily natural-language network summaries.

The Crucible (What Happened): The transition from local code to a fully automated cloud pipeline exposed the strict security and operational limits of free-tier cloud environments. The build was actively challenged by:

• YAML Syntax Strictness: GitHub Actions immediately rejected minor indentation discrepancies.
• Ephemeral Environments: The virtual runners lacked the external Python libraries (like requests) needed for API communication.
• Aggressive Cloud Firewalls: GitHub’s free-tier runners actively blocked DNS resolution for the HuggingFace AI Inference endpoints to prevent abuse.
• WAF Blocking: When a proxy was built, HuggingFace’s Web Application Firewall (WAF) blocked the incoming IP ranges from Google’s servers.
• Live Model Deprecation: The initial fallback model (gemini-1.5-flash) returned a 404 error because the endpoint had been deprecated in real-time by the vendor.

The Engineering (What I Had To Do): To force the pipeline through those roadblocks, we executed a series of advanced systems engineering pivots:

• Infrastructure as Code Mastery: Re-architected the update_latency.yml file with perfect strict-YAML alignment and injected automated dependency installations (pip install requests) directly into the runner environment.
• Network Diagnostics: Implemented custom Python socket scripts (socket.gethostbyname) to force network visibility checks and isolate the exact layer of the DNS failure.
• Serverless API Gateway Construction: Engineered and deployed a custom JavaScript proxy via Google Apps Script to act as a webhook relay, attempting to bypass GitHub’s outgoing firewall.
• Rapid Vendor Pivot: When HuggingFace rejected the proxy, we executed a clean, mid-flight migration to Google’s Gemini REST API, completely rewriting the payload structure and authentication headers.
• Real-Time Version Control: Identified the 404 deprecation error and instantly bumped the API endpoint to the production-ready gemini-2.5-flash model.

The Result: A fully autonomous, self-documenting CI/CD observability stack that successfully pings infrastructure, compiles CSV telemetry, queries a live LLM for an executive summary, and deploys the JSON data to a headless frontend—with zero human intervention.

1. The Secure Architecture (GitHub Secrets)

We didn’t hardcode the webhook URL into your Python script. Instead, we generated a secure Webhook URL directly from your Discord #noc-alerts channel and stored it as a repository secret (DISCORD_WEBHOOK).

We then updated your update_latency.yml file to inject that secret into the runner environment right before execution. This ensures enterprise-grade security by keeping credentials strictly out of version control.

2. The Trigger Logic (Python Engine)

We completely upgraded how your ping script (Dashboard_test.py) handles the data it collects by introducing a new process_alerts() function to act as the analytical brain.

• The Evaluation: After pinging the targets, the script evaluates the array of results.
• The Threshold: It specifically looks for any instance where packet loss is greater than 0% or the status drops from Online.
• The Failsafe: We engineered a schema failsafe (if 'target' not in alert...) to ensure that legacy JSON data from older versions didn’t crash the script during the rolling update.

3. The Payload Construction

When a failure condition is met, the script doesn’t just send raw text. It constructs a rich JSON payload formatted specifically for Discord’s API.

We utilized Discord “Embeds” to make the alert look like a professional NOC system, injecting a critical red hex color code (16711680), an urgent title, and the exact telemetry data (latency and packet loss) directly into the message block. The script then fires this payload over HTTP via the requests library.

4. The “Single Source of Truth” Paradigm

This was the most advanced part of the build. Instead of fetching data from Discord to show on the website (which would require running a custom 24/7 bot), we used a Native Architecture approach.

When an alert triggers, the Python script acts as the absolute single source of truth and branches the data:

• Branch A: Fires the webhook to Discord (alerting the engineers). https://discord.gg/JQRBjaZsxz
• Branch B: Saves the alert state into a new, self-rotating file called alerts.json.