Silent JSON Parser & Log Discovery Conflicts

Getting your logs into a new system can sometimes feel like trying to assemble IKEA furniture without the instructions – you know it should work, but something’s just not clicking. If you're setting up LogLynx for the first time and find yourself staring at an empty dashboard, with API calls returning null data, you’re likely hitting one of two major roadblocks. These aren't minor glitches; they're critical bugs in the ingestion service that can stop your data flow dead in its tracks. We've dug into these issues, and they boil down to a startup conflict between log discovery settings and a silently failing JSON parser. Let's break down what's happening and how to get your logs flowing.

Issue 1: The UNIQUE constraint failed Log Discovery Conflict

One of the first hurdles you might encounter is a database error that seems to come out of nowhere: UNIQUE constraint failed: log_sources.name. This error message, while cryptic, points to a fundamental problem in how LogLynx handles log file detection at startup. It occurs when you configure your .env file in a way that seems perfectly logical: you want automatic log discovery (LOG_AUTO_DISCOVER=true, which is the default), but you also explicitly tell LogLynx where to find a specific log file, like Traefik’s access logs (TRAEFIK_LOG_PATH=traefik/logs/access.log). This combination triggers a race condition. The system ends up detecting the same log file twice. Once through the automatic discovery process and again because you explicitly pointed to it. When it tries to register this log file in its database (the log_sources table), it hits a snag because database tables typically require unique entries for each source. Trying to insert the same source twice leads to the UNIQUE constraint failed error, and crucially, this error doesn't just stop the duplicate entry; it halts all log ingestion entirely. The entire data pipeline effectively crashes before it even begins to process any log entries. It’s a critical bug because it makes initial setup incredibly frustrating, especially when your .env configuration is intentionally set up for ease of use and explicit path definition.

Steps to Reproduce the Conflict

To see this issue in action, you can follow these simple steps. First, ensure you're starting with a clean slate by deleting any existing loglynx.db file. Then, modify your .env file. Make sure LOG_AUTO_DISCOVER is set to true (or simply leave it as the default). Simultaneously, define the specific path for Traefik logs by setting TRAEFIK_LOG_PATH=traefik/logs/access.log. It’s also important to create the actual log file at the specified path, traefik/logs/access.log; it can even be empty to start. Once your .env is configured and the file exists, run the server using go run ./cmd/server/main.go. The expected behavior here is that LogLynx should be intelligent enough to recognize that the file discovered automatically is the same one you’ve explicitly defined. It should then register this log source only once and proceed with its operations. However, the current behavior is that the server detects the file through both mechanisms, attempts to add it to the log_sources table twice, and triggers the fatal database error. You’ll see output in your console logs indicating the file being detected, registered, and then detected again by the discovery engine, culminating in the UNIQUE constraint failed error. This error message, ERROR Detection failed, error: UNIQUE constraint failed: log_sources.name, effectively stops the server from processing any logs, leaving you with an empty dashboard and no clear indication of why, other than this specific database error.

The Underlying Problem: Race Condition in Discovery

The core of this problem lies in a race condition within the log discovery logic. When LOG_AUTO_DISCOVER is enabled, the system actively scans defined directories for log files. If TRAEFIK_LOG_PATH is also set, it provides a direct, explicit path to a log file. The bug occurs because these two detection mechanisms operate independently and can potentially identify the same file concurrently. The automatic discovery might find traefik/logs/access.log during its scan, and then, shortly after or in parallel, the explicit path handler also picks it up. Both try to create an entry in the log_sources table. Since the name column in this table is expected to be unique (likely derived from the file path or name to identify the source), the second attempt to insert fails. The database constraint is violated, and instead of gracefully handling the duplicate or simply ignoring it, the server crashes with a fatal error. This effectively breaks the entire ingestion process before any log data can even be read or parsed. The ideal scenario would involve a de-duplication step, where the system checks if a discovered log source already exists before attempting to add it. This would ensure that even with overlapping detection methods, the source is registered only once, allowing the server to continue processing logs without interruption. This is a critical bug that directly impacts the initial user experience and the system’s reliability when faced with common and logical configuration choices.

Issue 2: The Silent JSON Parser Failure

Perhaps even more insidious than the startup conflict is the silent failure of the JSON parser. While LogLynx seems to handle logs in the Common Log Format (CLF) without a hitch, attempting to ingest logs formatted as JSON results in a complete data blackout. The server will indicate that it has processed files, but no actual data makes its way into the system, and crucially, no errors are reported. This makes debugging a nightmare, as you’re left guessing why your meticulously formatted JSON logs are being ignored. The TRAEFIK_LOG_FORMAT=auto setting becomes a gamble; it might work if your logs are CLF, but it’s a coin toss if they’re JSON, and you won’t know it failed until you check your empty dashboard.

Analysis: Why JSON Fails Silently

Our deep dive into this issue revealed a stark contrast in behavior between different log formats. When TRAEFIK_LOG_FORMAT is set to auto or explicitly to common, LogLynx successfully parses and ingests logs formatted in the Common Log Format (CLF). A typical CLF log line, like `192.168.1.100 - user1 [16/Nov/2025:10:15:23 +0000]