Debugging a Production Nightmare

I was assigned a seemingly simple task that spiralled into a production issue affecting unrelated parts of the system. What followed was a valuable (and painful) lesson in system design, JavaScript quirks, and the risks of shared mutable state in a monolithic architecture.

The setup

The system was a large monolith composed of multiple logical components all running in the same process. One of these components used a hardcoded list of “questions” as part of a dynamic form flow.

My task was to add a new question to this list. This was the first "new" question after the feature was first implemented almost a year ago.

No database migration, no configuration system, just append an object to a static array declared in code.

I wasn’t thrilled about the hardcoded setup, but it was meant to be a low-risk change. I added the question, tested the feature, passed QA, and deployed to production.

The bug

The next day, a different component of the codebase started showing a pre-filled answer to this new question. Except no one had filled them in. There was no data entry. These answers were ghosts.

This should have been impossible.

I checked the database: nothing. I read through the code paths: nothing. I added logs directly in production. That’s when I saw it.

The cause

The array of questions was being shared across modules, passed around by reference. Somewhere downstream, another part of the application was mutating that array, modifying it in place by appending answers.

So my new question, added to the shared array, started receiving mutations intended for other workflows. That corrupted shared state leaked across service boundaries and showed up in user-facing API responses.

const questions = 
[
    { id: "name", type: "text" },
	{ id: "age", type: "number" },
];
 
const  getAllQuestions = () => questions;
 
module.exports = getAllQuestions;

getAllQuestions was imported in various modules leading to adding an answer somewhere along the way...

The quick fix

Deep-copy the array before passing it between modules.

const getAllQuestions = () => JSON.parse(JSON.stringify(questions));

After that, everything worked as expected. The newly added question remained untouched, and the API behaviour stabilised.

Long-term fix

Deep-copying solved the immediate issue, but the real problem was deeper: the system had no clear boundaries or ownership of data, and critical config was being shared and mutated across modules.

A better long-term solution would involve moving the questions into a database where they are not entangled with application logic. Write tests that assert immutability and fail if shared structured are unexpectedly mutated.

Eventually, parts of the monolith should probably be refactored into loosely coupled services, or at least well-encapsulated modules that communicate through defined interfaces, not shared memory.

Final thoughts

const in JS/TS gives you a false sense of safety, it only protects the reference, not the contents.

Always assume your data can be mutated unless you’ve made it truly read-only.

If you’re building anything more complex than a to-do list, shared mutable state is a trap. Wrap your arrays. Clone your objects. Trust no one. Especially not past developers. Especially not yourself.

And maybe, just maybe, don’t hardcode production-critical data into a single shared array.