Why Timing Is Everything in a Live Integration

Share
Why Timing Is Everything in a Live Integration

Nobody has ever called support to report an API call is misfiring or not sending at the right time.

They open tickets to report that messages aren't going through/the integration "stopped working".

Integrations connect two systems so that something in software A triggers an action in software B. Software A sends an API call; software B receives it, acknowledges it, and does what it’s supposed to do.

The same thing happens inside a single product. When you click something in the browser, one part of the software calls another. Live chat is exactly that kind of integration — between your screen and the other person’s screen — and timing is everything. The same is true for asking Alexa to turn on the lights, expecting an alert, or having your HVAC turn on based on an event.

When a customer says live chat is “broken,” they can mean anything from the moment they expect to send a message to the moment they expect to see it was read. If any part of that doesn’t happen in real time, the integration looks broken to them.

Every API call can still succeed. Every log can be green. Every response can be a clean 200 OK. But if the message arrives at 2:40 — ten minutes after the other person closed their laptop at 2:30 — the verdict from their side is the same. Integrations are built for a trigger followed by a result at the right time. If the timing is off, it technically worked but didn’t actually work, because it didn’t happen when it was expected to. A message that arrives minutes later is still a problem, because that’s not how sending and receiving a message is supposed to work.

Important Note:

Integrations between softwares and features within a software that use live API calls are built with all sorts of different tech stacks. Meaning different companies use different programming for the user interface (UI which is known as the front end), different programming for the source code (the company's main codebase which is the service layer), and the database (the back end, where all the data for settings, files, user info etc is stored).

  • In the example below, this article uses DuckBuddy's tech stack to help explain why timing in live API integrations is so important.
  • This example is adapted from a real life troubleshooting scenario that I experience at work, but the details have changed for company privacy reasons/respect.
  • DuckBuddy is a social network for remote workers — people whose whole social work circle can live inside one app. This is how it's live chat, along with live chat on other platforms, work: after you type something and you press Enter, the chat service writes your words to Cassandra (one of DuckBuddy's 5 databases that stores and runs the data in the platform), the delivery layer finds the other person's live WebSocket session (live connection) in Redis (another database), and the message lands on their screen across the planet in under a second. Just like with other live chats, there is a green dot that says they're there (online), a typing indicator that says they're answering, and a seen receipt that says it arrived/was read by the other person.

The Part That Isn't Timing

Now there are other things about sending and receiving an API call (example message on a live chat) that have to work on a platform as well.

  • The Trigger: A user action or system event kicks off the request inside the client application.
  • The Address & Verb: The request targets the exact URL endpoint using the correct HTTP method (like GET or POST).
  • The Structure: The payload data is perfectly formatted (usually as a JSON string) and query parameters are correctly written.
  • The Authorization: The client attaches a valid, unexpired token or API key to the request headers.
  • The Network Routing: DNS and API gateways successfully direct the traffic from the public internet into the internal company servers.
  • The Web Framework Routing: The backend matches the URL path to the precise code handler (Controller) built to receive it.
  • The Authentication Gate: Middleware decrypts the token to verify the user’s identity and checks that they have the right permissions to run the request.
  • The Input Validation: The Controller confirms all required fields are present and typed correctly before passing them forward.
  • The Business Logic: The Service Layer processes the request rules, calculations, and internal workflows.
  • The Database Mutation: The Data Access Layer safely queries or updates the database without running into errors or timeouts.
  • The Response Return: The server sends back the correct HTTP status code (like 200 OK) and the requested data payload to the client.

Everything beyond timing must also work.

Other failures are loud. A bad payload returns an error. An expired push token triggers a webhook. A rejected write shows up in logs, dashboards, and alerts. When something breaks in the path from one user to another, the system usually tells the team what went wrong, where, and when.

Timing fails silently. The call succeeds, but the message arrives late. In live chat, that makes it look like the other person never replied — even though they did. Users don’t see the delay, and nothing alerts the team. The pipeline did everything it was asked to do, just not when it mattered. Technically it worked, but for a conversation the expected window is only about ninety seconds wide. Outside that window, it didn’t work.

That's why timing is the whole ballgame. Everything else is the entry fee that allows the spectators in.


Four Ways to Deliver a Message - Deeper Live

As software grows and the customer base grows, companies have to program their software architecture to be more and more complex. For years, companies have commonly used microservices—specialized pieces of code that handle one task separately from the rest—to keep platforms from getting overloaded. But they rarely start there. Most begin with everything living in one big codebase (often called a monolith) and only break it into smaller pieces as usage grows.

More people chatting means more messages and more strain on the servers. If one part of the platform suddenly gets too busy and the rest of the code can’t keep up, everything can slow down for all users.

In chat, the interesting challenge isn’t saving the message to a database—that’s usually quick and straightforward. The crucial part is delivery: getting the message onto the other person’s screen while they’re still online and able to see it.

This is why teams split the code apart. No matter how the connection is built—whether with an Inline Call (most simple option), a Background Job, a Queue, or a Microservice (most complex option)—the goal is the same: making sure the delivery happens at the right moment.

1. The Inline Call — Timing Is “Now”

This is the simplest approach: delivery happens right inside the same request that sends the message. The user hits Enter, the platform writes the message, finds the recipient’s live session, pushes the message through, and finishes. Everything happens in one continuous flow.

Why use it: It’s straightforward and the result is available immediately. If you need to show the message back to the sender the instant they hit Enter, or check whether a room is open before joining, an Inline Call is often the right choice.

What it does to timing: Delivery happens right now. That’s its strength—and its risk. If anything in the delivery path is slow or down, the send itself slows down or fails, even if the rest of the chat system is fine. It also has no way to wait for the future. You can’t keep a request alive from “message sent at 2:00 PM” until “recipient comes back online at 6:00 PM.”

When to move on to the next option: When delivery delays start showing up in response times, or when you need to deliver messages later (for example, when someone comes back online).

2. The Background Job — Timing Is “When the Sweep Gets to It”

Here, the user’s action just records that a message needs to be delivered. A separate background process (something that wakes up on a schedule) later looks for waiting messages and sends them out.

Why use it: The user’s experience is no longer tied to how healthy the delivery system is. The message appears instantly for the sender. A background job can also handle future delivery—it can wake up later and push messages when people come back online.

What it does to timing: Delivery happens whenever the job runs, not instantly. If the job checks every few minutes, messages can arrive anywhere inside that window. That’s often fine for asynchronous chat. But when both people are online and chatting live, a delay of several minutes breaks the feeling of a real-time conversation. One slow item can also hold up everything else in the batch.

When to move on to the next option: When volume gets too high for one sweep to handle comfortably, when one problem starts affecting other messages, or when you need better visibility into what succeeded, what failed, and what’s still waiting.

3. The Queue — Timing Is “In Order, Eventually”

Deliveries become individual items on a durable queue. Workers pull them off one by one and handle them. Each message can have its own retry logic and failure path.

Why use it: Reliability improves because one failing delivery no longer blocks everything else. You also get better visibility—you can see what’s pending, what’s in progress, and what’s failed. A crash doesn’t lose the message; it simply goes back on the queue.

What it does to timing: Speed now depends on how deep the queue is. A short queue means fast delivery; a deep one means delays. Even with good retry logic, a message can sit behind a backlog and arrive after the conversation has already moved on. You now have two clocks to watch: when the message was sent, and when it actually left the queue.

When to move on to the next option: When the delivery layer grows its own rules and responsibilities—refreshing tokens, checking live sessions, managing retries, and needing its own monitoring. At that point it’s no longer just a consumer of a queue; it has become its own system.

4. The Microservice — Timing Is “On Schedule, With Receipts”

The delivery layer becomes its own dedicated service (a standalone piece of software that owns one specific job and runs separately from the rest of the platform). It owns presence tracking, the queue of undelivered messages, the process that pushes messages when recipients are available, token refreshes, retries, receipts, logging, and health monitoring.

Why use it: Delivery timing is now a first-class responsibility with a clear owner. Everything related to “when and whether this message actually reached someone” lives in one place. The service can answer: What’s due right now? What already went out and when? What’s still waiting and why? What should happen if something fails repeatedly?

It also keeps clear records. Every delivery gets timestamps, retry counts, and responses. In chat, that can include delivered and seen receipts coming back from the other side.

This is the architecture behind the chat system at DuckBuddy—a dedicated delivery service with its own presence tracking, queue, and delivery loop. And that’s what leads to the rest of the story: a case where the logs finally showed which clock was actually in control.

The Progression

  • Inline: timing is now — exact but fragile, and it can't wait for anyone.
  • Background job: timing is the sweep — independent of the user, but loose.
  • Queue: timing is throughput — reliable per item, but the backlog has its own clock.
  • Microservice: timing is the product — an owned schedule, and the receipts to prove it happened.

Every transition from in-line to microservice is really the same decision: the previous stage's answer to "what happens to the timing when things go wrong" stopped being good enough.


Real Life Example: Pass on 9/8, Fail on 9/22

*This example is adapted from a situation at work to explain my process, what I did, while keeping company info private. *

One of the QA test cases for a customer-reported bug fix was TC-LIVECHAT-001 — “chat room opens and messages deliver.” Under that simple description, it was really a timing test: when a message is sent to someone who is online, it had to appear on their screen within 2 seconds. Two seconds was the bar/the limit/the goal — not “eventually,” and not “when the system gets around to it on another pass.” Live chat is a conversation happening "right now" (just like external third party integrations), and the green "this message was delivered" dot is a promise about timing.

The fix for this issue went on for awhile. On 9/8, TC-LIVECHAT-001 passed. The message arrived 1.4 seconds after Send — comfortably on time. However, on 9/22, the same test, same accounts, same room, and same code path failed. Deliveries were running 3–4 minutes late, and up to about 16 minutes late in the worst cases.

Nothing actually broke in between. Every message still went out. Every call still succeeded. Every bubble still appeared. There were no errors and no warnings. The message content was right, the chat room was right, and the recipient was the correct one. The only thing that changed between the passing and failing runs was when the message arrived. From the system’s side, everything still looked fine. From the customer's perspective, it looks like the message was never sent and there was a bug.

So the question was simple: the message didn’t change, the endpoint didn’t change, and the room didn’t change. What did?

Troubleshooting Step 1: The Fingerprint in the Timestamps

The agent pulled the delivery timestamps from the message log for both periods — the same scenario, two different weeks — and compared them.

From 9/8 through 9/18, deliveries landed at any millisecond of any minute: 1:03:13.4 AM, 1:33:24.9 AM, 10:47:51.2 AM, and so on (all UTC). There was no fixed cycle. That’s the signature of event-driven delivery. The system isn’t following a schedule of its own; it reacts when people type. A message sent at 3:48 gets pushed almost immediately because delivery is triggered, not scheduled. The timestamps in the log simply follow the rhythm of the conversation.

On 9/22, after about 1:30 PM Pacific (20:30 UTC), every single delivery landed 30–125 seconds after a quarter hour — :00, :15, :30, or :45 — and nothing went out in between. That’s the signature of a 15-minute sync sweep. People typed at 2:07, 2:29, and 3:13, but the log only shows activity around the quarter hours.

Nobody changed the message. Someone changed the clock. And the agent spotted the change purely from the timing pattern in the logs — no deploy log, no config screen, no feature flag. Just timestamps. When every delivery clusters after quarter hours with nothing in between, a 15-minute sweep is in control. The timing pattern itself is the architecture’s signature. Once you can read it, you can tell which layer is delivering the messages without opening a single settings page.

Troubleshooting Step 2: The Minute It Changed — and What Else Changed With It

The agent narrowed the switch to a specific moment: around 1:30 PM Pacific (20:30 UTC) on 9/22. Two other things also changed at that exact minute:

  • The log wording. Older entries said “Delivered via Fan-out Service” and “Presence via Fan-out Service.” Newer ones said “Message synced to recipient” and “Presence refreshed.”
  • The log fields. Before, the message_id was usually present. After, it was null on every new delivery row.

Wording, fields, and timing all shifting at the same moment pointed to one conclusion: a deploy or configuration change around 1:30 PM — not a bug in the existing code. Three independent clues, one timestamp, one answer.

Before (~9/8–9/18) After (9/22, post-1:30 PM)
When deliveries fire Any millisecond of any minute Only 30–125 seconds after a quarter hour, nothing in between
Implied cadence Event-driven (no cadence) About 15 minutes
Delivery latency vs. Send 1.4 seconds (9/8); up to about 12.5 minutes late (9/9–9/10) +1m52s, +3m02s, +4m05s, worst case about 16 minutes
Delivery log label "Delivered via Fan-out Service" "Message synced to recipient"
message_id on delivery rows Usually present Always null
What a "pass" meant A good sample Luck, standing right next to a quarter-hour boundary

Troubleshooting Step 3: The Pass That Was Luck

The agent went back and looked closely at the 9/8 pass. Message 797326 was sent at 2:48 PM. The log recorded it at 21:48:00 UTC and delivered it at 21:48:01.4 UTC — 1.4 seconds later. A clean pass.

That turned out to be a single good sample. Retests on 9/9 and 9/10 for TC-LIVECHAT-004 were already 11.5 and 12.5 minutes late. Even under the old system, the test was never reliably passing. The 9/8 run simply happened when the queue was quiet. Nearby runs missed by double-digit minutes whenever the queue backed up.

The 9/22 “passes” were luck the other way. The two green runs that day both had sends within two seconds of a quarter-hour boundary. Under a 15-minute clock, that’s the luckiest place to be. A message sent at 2:29:58 only waits until the 2:30 sweep. A message sent at 2:07 waits until 2:30 and arrives twenty-three minutes into a conversation that ended at 2:10.

So the full picture was: it passed by luck on 9/8, failed by design on 9/22, and “passed” by luck again on 9/22. That story doesn’t exist in any single log row. It only appears when you compare across dates and test cases.

Troubleshooting Step 4: The Clocks Didn’t Agree

The timezone issue almost turned the story into fiction. The 9/8 test note used Pacific time. The delivery log stores UTC. The platform runs on UTC. Users are in thirty-eight different time zones. The same delivery is 2:48 PM on the tester’s screen, 21:48 in the log, and 23:48 on a recipient’s screen in Berlin. All are “correct,” but hours apart, and none of the screens label the zone.

Do that correlation by hand and one unlabeled timezone can quietly turn “delivered in 1.4 seconds” into “delivered about seven hours late.” You end up debugging a lag that doesn’t exist while the real problem continues. The agent caught it by comparing the written note against the stored log. Going forward, test notes will label the zone. In a timing investigation, the first thing you align is the clocks.

The irony is hard to miss: matching people who share the same clock is part of the product. The platform already knows every user’s time zone. The QA process just hadn’t been printing it. This case closed that gap.

Troubleshooting Step 5: The New Clock’s Collateral

The new 15-minute cadence didn’t just fail one test. It changed behavior across the pipeline:

  • The math stopped working. A message sent at 3:13 waits for the 3:15 sweep and arrives at 3:16:02. A message sent at 3:42 waits for 3:45 and arrives at 3:46:05. Under a 15-minute cadence, the 2-second acceptance criterion can only be met if the send falls within the two seconds before a quarter hour — a tiny window of luck. If the new cadence is intentional, the criterion and the cadence are in conflict, and one of them has to change.
  • Presence lingered — and so did messages. After a user closed the app, their green dot stayed green for 10+ minutes, until the next quarter-hour sweep. People kept starting conversations with someone who was already gone. After a user deleted their account, the dot stayed green and messages kept arriving for another ten minutes. Under the old system, presence flipped almost instantly. Under the new clock, deletion promised the right to be forgotten, while the delivery layer still treated the person as online for up to fifteen more minutes.
  • Traceability broke. With message_id null on the new delivery rows, the log could no longer say which send landed on which screen. That’s why the TC-LIVECHAT-004 automation kept looking for a delivery it could never find.

The Honest Limits — and the Questions Only a Developer Could Answer

The agent was clear about what it couldn’t see. It only had access to the QA delivery-log page — up to 200 rows per conversation. No job monitor, no feature flags, no production data, and no service records. So the deploy and the flag state weren’t confirmed. But based on what the agent found, it knew the right questions to ask to help QA and the dev team continue in their investigation. Those along with the test results were posted on Jira issue DB-241 through DB-244:

  1. What was deployed or reconfigured on QA around 1:30 PM Pacific (20:30 UTC) on 9/22?
  2. Which dispatcher is live right now — the event-driven fan-out, or the sync sweep? Is the real-time delivery flag still off?
  3. Is the 15-minute cadence intended? If so, the 2-second criterion on the timing tests can’t be met, and one of them needs to change.
  4. Is message_id null on the new delivery rows intended? It breaks message-to-delivery tracing.

Plus one instruction that captures the whole case: rerun TC-LIVECHAT-001 with the send mid-quarter — at :07 or :37 — so timing luck can’t produce a pass.

That last line is the investigation’s best output. It didn’t just explain the failure. It redesigned the test so luck can never fake a verdict again. That’s what comparing detailed logs across scenes and dates actually buys you: not just what happened, but how to never be fooled by what happens next.


Why Comparing the Same Event Across Scenes Is the Whole Game

Since timing is the result of a live pipeline, the troubleshooting question is never “did it error?” On 9/22, nothing errored. The real question is “did the right thing happen at the right moment — and can you prove it?”

No single view can answer that on its own. Each one only holds part of the timeline, on its own clock, in its own format:

  • The test note recorded the send time in Pacific.
  • The delivery log recorded the landings in UTC.
  • The recipient’s screen showed the moment in Berlin.
  • The acceptance criterion defined the contract (two seconds).
  • The two date windows showed the actual behavior (any millisecond versus quarter hours).

The real answer only appeared once all of those pieces were aligned.

This is where using an agent to test and troubleshoot became extremely valuable! Here’s the method it used:

  1. Pull every scene for the same events — test notes, delivery rows, seen receipts, acceptance criteria, and both date windows — as structured records, not screenshots.
  2. Align the clocks first. Convert every timestamp before doing any math, so one unlabeled timezone can’t turn the whole story fictional.
  3. Align on identity. Stitch the same message through every scene using message IDs, conversation IDs, and test case IDs so a delivery can’t lose the send that caused it.
  4. Diff across dates, not just rows. Same test, 14 days apart: any millisecond versus quarter hours, 1.4 seconds versus sixteen minutes, luck versus design.

What a human does with multiple platforms, notes, and mental timezone math, the agent did in a single pass. It handed back the exact timestamps, the row counts (223 of 252, 312 of 324, 0 of 4), the minute the clock changed (about 1:30 PM Pacific / 20:30 UTC on 9/22), and the four questions only a developer could answer. “Is it working?” became a clear timeline — either the story holds across every scene on one clock, or you can see exactly which second, message, or scene stopped agreeing.


Bottom Line

There are four ways of delivering a message: the inline call, which sends now or never, the background job, which sends when the sweep gets to it, the queue, which sends in order (eventually, with retries), and the microservice, which sends one on schedule (on purpose, with receipts).

Each one depends on timing. If the message is sent at the wrong time, it will still appear as though the message didn't send (a software bug), even if it did later and sent the right call and got the right response. A message sent late in a social networking app makes it look at though the message sending feature isn't working.

Troubleshooting timing issue in integrations can be the trickiest part sometimes. This is where an agent can step in and become the most helpful tool in the troubleshooting process. It tracks the detailed logging, it pulls the same event out of every scene and every date, it aligns the clocks before the IDs, and it tells you whether the every piece worked and if it worked "on time" or not. Just like when someone doesn't receive a package in the mail when they expect and conclude that something must be wrong, an integration or feature firing an API call at the wrong moment makes it looks like there is a bug in the software.


Next time a live integration is reported as "broken," don't just ask what errored the customer might have seen, ask your agent to review the integration document to learn how it's supposed to work, then ask it to review the event that triggered the integration and the logs, maybe even perform it's own test in your demo account. Then review it's assessment to gain a better understanding of what the issue might be in a much shorter time frame, letting your agent do the tedious and heavy lifting.