Between the Log and the Answer
Nine ways health tracking apps get the math wrong, and how to check yours.
Apps are giving people confident, specific, wrong health advice, and the statistics are rarely to blame.
The problem. Apps that find health patterns in self-tracked data are producing findings that aren't real. Not occasionally, but systematically, and in a consistent direction.
Why now. Millions of people track symptoms, food, sleep and medication daily, and a growing number of apps promise to tell them what's driving how they feel. Analyzing one person's data every night now costs pennies a year. The barrier to building this has vanished, so a lot of it is being built, largely without anyone checking whether the answers hold up.
What's going wrong. The errors sit in the unglamorous steps that turn log entries into a table a model can read: filling in blanks, choosing time zones, grouping into days, deciding what counts as a possible cause. Each step has an obvious default. Most of the obvious defaults are wrong, and each pushes results in a predictable direction, so you don't get random noise, you get consistent, plausible, backwards answers.
Key findings. Nine decisions account for most of the damage. Filling missing entries with zero makes every under-logged habit look protective, because people log less when they feel worse. Cutting a day short at the symptom, but not on symptom-free days, invents protective effects for anything done in the evening. And averaging an effect across all days, including ones the behavior never happened, shrinks real findings to nothing and convinces teams their data has no signal in it.
What to do. Fix the nine decisions: most are a few lines of code, and several change whether a result points the right way. Then build six automated tests that measure your own error rate, so “we think it works” becomes a number you track.
Who it's for. Engineers, data scientists and product managers building personal health analytics. Also useful to clinicians and investors judging whether a product's claims hold up.
Every one of these looks like plumbing. Every one changes what a user is told.
One example, start to finish
A health tracking app tells a user: “Eating breakfast reduces your migraines by 40%.”
The user believes it. They start forcing down breakfast on days they feel terrible. Nothing improves.
The finding was never real. This person doesn't log breakfast when they have a migraine, because they're lying in a dark room. The app filled the blank with a zero, saw thirty days of “no breakfast, migraine,” and drew the obvious conclusion.
The statistics were fine. The regression was correctly implemented. The bug was three steps earlier, in a line of code that filled empty cells with zero, a line nobody would flag in review, because it looks like data cleaning.
Why nobody catches this
Two groups of people know things about this problem, and neither sees the whole of it.
Researchers who study one person over time work with data collected on a protocol. Fixed schedule. Defined windows. Missed entries recorded as missed. Almost none of these problems come up, so almost nothing is written about them.
Engineers building the app see a data pipeline. Group the rows, fill the blanks, aggregate, hand it to the model. Every step looks like ordinary plumbing. Nothing signals that filling a blank is a decision that will change what the app tells someone about their health.
So the mistakes get made by competent people doing reasonable work, and nobody catches them, because catching them means standing in both places at once.
The nine problems, in detail
Empty boxes are not zeros
Someone didn't log lunch on Tuesday. Most teams put a zero there. It's what fillna(0) does. But an empty cell can mean “didn't do it” or “did it and didn't write it down,” and those two aren't interchangeable.
People log less when they feel worse.
So bad days have fewer entries. Fill those blanks with zero and bad days become “low everything” days. The model sees bad day, low exposure and concludes this protects you.
Full detail
An empty cell means one of three different things: they genuinely didn't do it; they did it and didn't record it; or the question doesn't apply to them. Treating all three as zero would be harmless if the second happened at random. It doesn't.
That people log less when they feel worse isn't a theory. It's the consistent finding across decades of research on self-reporting, and you can see it in your own data in an afternoon: count each person's daily entries, compare against how they rated that day, and watch the line slope down.
Every single thing a person stops logging on bad days will show up as protective. All of them, in the same direction, invisibly. And it's worst for the things that take effort to record, which in most apps means food, the exact thing users most want answers about.
What to do instead
Cheapest fix, do this today. Add one more column to every model: how many entries the person made that day. Now the model has somewhere to put the “how much did they log” effect instead of smearing it across everything else.
This is also an instant test. Any “protective” finding that disappears when you add this column was never about the behavior. It was about when the person felt like typing.
Better. Change the schema so you can tell “didn't happen” from “didn't say.” A single end-of-day tap (“nothing else to log today”) converts most ambiguous blanks into real zeros. Two seconds of user effort, worth more than any extra tracking field you could add.
Best. If you can predict who logs and when, you can correct for it statistically. Real work, and only worth doing once the two cheap fixes are in.
Regress daily entry count against same-day symptom severity, per person. A downward slope confirms you have this. Then rerun every model with and without the entry-count column, and count how many findings change sign or vanish. That count is your exposure.
A day starts where the person lives
UTC is the right answer to almost every other timestamp question in engineering. Here it's wrong. Bodies run on local time, and a UTC day boundary cuts through the middle of the night for most of the world, splitting one night's sleep across two rows, and filing last night's dinner under a different day than this morning's headache.
Most teams work that out. The trap is the repair: looking up the person's offset and adding it to every timestamp. That breaks in two ways that are easy to ship and hard to spot.
Full detail
The two ways the repair breaks
- Using today's offset on old data. Look it up once, apply it to their whole history, and every daylight-saving change shifts their entire past by an hour. Anything you conclude about timing across that boundary is quietly wrong.
- Writing a shifted time and labeling it UTC. A local wall-clock time with a
Zon the end isn't a real timestamp. It works fine until a second system reads it, and then it doesn't.
What to do instead
Store timestamps in UTC. Store the person's time zone name (like America/Denver), not a fixed offset. Convert to local time at the moment you group, one timestamp at a time, using a library that knows what the offset was on that specific date. If you need to move a local time between systems, send the local time and the zone name as separate fields. Never a shifted instant pretending to be UTC.
Take someone whose data spans a daylight-saving change. Plot what hours of the day they log, four weeks before and four weeks after. People don't change their habits by exactly one hour overnight. If the chart jumps, you have the bug.
A calendar day is the wrong box
Someone gets a headache at 9am Tuesday. The most likely cause is Monday evening, but calendar grouping puts those in two different rows and cuts the connection. Meanwhile everything they eat Tuesday afternoon, which cannot possibly have caused a 9am headache, lands in the same row and waters it down.
The default manages to exclude the real causes and include the impossible ones. And it forces one box size onto every outcome, when different things in the body move at wildly different speeds.
Full detail
What to do instead
Build the box around the symptom, not the calendar. Every time the symptom happens, look backwards a fixed amount of time and gather up what happened in that stretch. Days with no symptom get regular fixed-length boxes, to give the model something to compare against.
Make the look-back length depend on what you're tracking. Hours for blood sugar. A day for headaches and gut symptoms. Several days for flares. This isn't a setting to tune. It's a fact about the body, and it should be written down with the reason next to it.
Run everything again with the window half as long, then twice as long. A real effect gets weaker as you move away from the right window, but it's still there. Something that shows up at exactly one window length and vanishes on both sides was never real.
Causes come before effects
This one produces the most embarrassing findings in the category, because people take medicine after they get sick. Most pipelines never ask whether something logged after the symptom should count as a possible cause.
Users spot this instantly, and it's the fastest way to destroy their trust, because it reveals that the app doesn't understand the order things happened in.
Once you drop everything logged after the symptom, you've cut symptom days short, but not normal days. Now an evening habit is uncounted on symptom days and fully counted on normal days. The app reports it as protective, and the effect is entirely manufactured by your own cutoff rule.
Full detail
Applying the filter
For each symptom, throw out anything logged after it. Two details matter in practice.
Let whole-day numbers back in. Total sleep, morning check-ins, and anything else describing the whole day but carrying one timestamp will get thrown out for any morning symptom. Give them a start-of-day timestamp and let them through on purpose.
Decide what to do about repeats. If the symptom happened three times, cutting at the last one is the safe choice.
The asymmetry, in full
Say someone drinks wine in the evening. On headache days, cut off at 10am, the wine isn't counted. On good days the whole day is counted, so it is. The model sees less wine on headache days than on good days and reports that wine protects against headaches. There is nothing in the data.
The worse the mismatch (an evening habit and a morning symptom), the bigger the fake effect, easily large enough to be the main thing the app tells the user.
Cut the normal days short too. Either chop everyone at the person's typical symptom hour, or pick a random cutoff hour for each normal day drawn from the hours their symptoms actually happen. Now both kinds of day cover the same slice of time.
For each thing you track, work out what time of day people usually log it, and how much of it falls after the cutoff on symptom days. Put that number on a dashboard. Anything with a high number and a protective-looking result is a suspect. This takes an afternoon and will probably explain several results that have puzzled you for months.
“Today” and “this week” shouldn't overlap
Some things act immediately, some build up. So teams add a rolling total alongside today's amount. If that rolling window includes today, the two columns carry the same information twice.
The model can't tell them apart. Coefficients go unstable, flipping sign from one night to the next on data that barely changed. And because it looks like ordinary noise, everyone assumes they need more data. They don't. They need to fix two lines.
Full detail
The natural way to write it is:
today = amount today
last 3 days = rolling sum over the last 3 daysMove the window back one day so the two don't overlap:
today = amount today
prior 3 days = sum over the three days before todayNow they're independent and each means something on its own. You also get something genuinely useful: comparing the two tells you whether something is an immediate trigger or a build-up. Those need completely different advice. You avoid a trigger in the moment; you manage a build-up by cutting down over time.
One warning. Every extra time window multiplies how many things you're testing. Three windows across two hundred tracked items is six hundred tests per symptom per night. Windows aren't free. See #9.
Correlate each today/rolling pair. Above about 0.7 and they're overlapping. Then check whether those coefficients flip sign between nightly runs: that's the tell.
Five spellings of the same pill
“advil”, “Advil”, “advil 200”, “ibuprofen”, “ibuprofen 200mg”. One pill, five columns. Say they took it twelve times across those spellings, and you require three appearances to analyze something. None of the five clear it. The pill gets dropped entirely.
The signal was in the data. Your naming layer destroyed it. The model never sees it, the user never hears about it, and nothing in your output says anything went missing.
This looks like a design choice. It's a math choice, and getting it wrong throws away data nothing downstream can recover.
Full detail
Use a fixed list with alias matching, organized into categories. Yes, it adds friction. It's worth it.
And here's the part nobody talks about. Those categories aren't just tidiness. They're a dial, and the right setting is different for every single user.
Take dairy → cheese → aged cheese → parmesan. Each step down is more useful to know and happens less often.
- Someone who logs everything for two years:
parmesanmight appear 40 times. Analyzable, and far more useful than “dairy.” - Someone who logs occasionally:
parmesanappears twice and gets dropped. Butdairyappears 60 times and works fine.
Set the level per person: use the most specific category their own data can support.
Light users get answers about food groups. Heavy users get answers about specific foods. Everyone gets something, and nobody gets a claim more precise than their data can back.
Two bonuses. You can show this as progress (“keep logging and we can go from 'dairy' to specific cheeses”), a rare case where what's good for the math is also good for engagement. And when “dairy” shows up as a result, you know exactly what to ask them to log more precisely next.
Count distinct typed strings per user, then see how many would collapse together with basic normalizing. A high collapse rate means you're fragmenting right now. Then work out what share of each person's logged items sits in columns too sparse to analyze. In free-text apps that's often most of them.
Stop averaging over days it didn't happen
Someone drinks wine on 20 days out of 400. Average the effect across all 400 days and you've included 380 days where the comparison is “no wine versus no wine.” A real effect present on those 20 days gets divided by twenty and reported as basically nil.
Everything comes out tiny, nothing looks actionable, and the reasonable conclusion is that health data is just noisy. It isn't. You diluted it.
Full detail
A raw coefficient is a change in log-odds. Nobody outside a statistics department can read that, and showing it is giving up. The averaging fix is right in spirit and usually wrong in practice.
There's a second error stacked on top. “Typical amount” usually gets computed across all days, which answers “how much wine does this person drink per day on average?” That's a consumption rate, not a serving. Comparing against it produces a scenario nobody cares about.
On the days I actually do this, how much does it change things?
So work it out that way:
- Keep only the days they actually did it.
- Work out the typical amount from those days only: the serving when they have one, not the daily average.
- Predict twice on those days: once with it set to zero, once at the typical amount.
- Report the difference.
Now you have a percentage-point change, on days the thing happens, at the amount they actually use. Readable, honest, and it doesn't dilute.
Cap the extremes. Small-sample models occasionally spit out absurd numbers. Squash or cap what you display so one glitch can't dominate the list. Be clear with yourself that this is display hygiene, not a fix.
Work out every finding both ways: across all days, and across active days only. If you're using the all-days version, the gap between them is how much real signal you've been hiding. It's usually a factor of five to fifty.
Sometimes the honest answer is “not yet”
Below a certain amount of data, these models don't give you a weak answer. They give you an arbitrary one. With few data points the math systematically overstates effects, and in one specific situation it breaks down entirely and returns an enormous number that a naive display renders as your strongest finding ever.
Products get built to fill space, so empty screens feel like failure. They shouldn't. Saying nothing is a real, designed outcome.
Full detail
Enough time, and enough days. Require both a minimum stretch of calendar time and a minimum number of days they actually logged, and require them together. Time alone lets through the person who logged twice a month for a year. Day count alone lets through the person who logged intensively for two weeks and stopped. Neither can support a reliable answer. Only requiring both keeps them both out.
Enough of the symptom. You need enough occurrences to model, and the floor should rise as you add more things to test.
Enough of the thing you're testing. Something that happened once can't be told apart from something that never varies.
If you build rolling-total columns (#5), one single occurrence spreads forward into several days of the rolling column. So something that genuinely happened once produces a rolling column that appears to have happened three times, and sails straight past your threshold while carrying one day of actual information.
Always check the count on the original column, never the rolling one. Two lines of code. Nearly impossible to spot from the output, because the resulting finding looks completely normal.
On what you output. When you skip something, say so and say why, rather than just leaving it out. Anything reading your results needs to tell apart “we looked and found nothing,” “not enough data yet,” and “we didn't look.” If all it gets is a list of findings, it will show users “no problems found” when the truth is “we haven't got enough to say.”
Report how many of your users clear each threshold, and how many of their symptoms clear it. If nearly everyone gets through, your thresholds are too loose. If almost nobody does, the thing to fix is how you collect data, not the math.
Running the same test nightly is not one test
Two problems, either of which alone breaks the number you're showing.
First: you picked which variables to test using the same data you're testing them on. A p-value assumes you decided beforehand. So a “0.03” here does not mean a 3% chance of being a fluke. It's a ranking score, not a false-positive rate.
Second: textbook corrections assume you run the analysis once. You run it every night, for as long as someone uses the app. Two years, thirty symptoms, three findings each: roughly 65,000 reported associations for one person.
A finding that showed up once and never again looks identical on screen to one that has shown up every single night for eight months. The second is vastly more trustworthy. Your system works out that difference every night, and then throws it away, by overwriting yesterday's results with today's.
Full detail
Why the nightly runs matter
Those runs aren't independent: last night and tonight share almost all their data, so it isn't really 65,000 separate tests. But it isn't one test either. The real number sits somewhere in between, it grows the longer someone uses your app, and no standard correction addresses it, because no normal research setting produces it.
What to do instead
1. Change the label. Stop calling it a p-value in your schemas, docs, and UI. Call it a strength score. Costs nothing, removes a false claim.
2. Correct within each run. Apply a false discovery rate correction across everything you tested that night. The goal is a shortlist where most entries are real, not a guarantee that no single one is wrong.
3. Rank by how often it comes back. If you take one thing from this guide, take this.
Keep every night's results. Rank findings first by how consistently they survive night after night as data accumulates, and only then by how strong they look tonight.
The reasoning is simple. A fluke happens when the noise lines up by chance, and it doesn't line up again tomorrow. A real effect holds, or gets stronger, as evidence piles up. So how often something comes back is a direct measurement of how much to trust it.
And unlike a p-value, it doesn't care what your screening step did, because it doesn't assume anything about the math at all. It just counts.
It's also nearly free. Your system already produces the raw material every night and deletes it. Keeping a history table and running a count is a schema change and a query.
Keep results for sixty days. Then group findings by how many nights in a row they've survived, and ask what share of each group survives the next night. That's the chart above, drawn from your own data.
The new user problem
Results get more reliable the more data someone has. Your thresholds let people in the moment they scrape past the minimum. Put those two facts together:
The very first results a person ever sees are the least reliable results your app will ever show them.
They're produced from the smallest amount of data you accept, using math that overstates effects when data is thin. And they land at the exact moment the person is deciding whether to trust your app at all.
Four things that help
Use math built for small samples. Swap standard logistic regression for a penalized version. Firth's method is the standard one. It corrects the overstatement, and returns sensible numbers in the exact situation that breaks the normal approach. In most statistics libraries it's close to a one-line change, and it improves precisely the users your thresholds just let in. Probably the best return per line of code in this entire guide.
Label how sure you are, visibly. Don't show everything with the same confidence.
| Label | What it means | What to say |
|---|---|---|
| Early signal | Showed up once, new user | “Keep logging to confirm this” |
| Consistent | Has held up across many nights | “This keeps showing up in your data” |
| Confirmed | Tested properly (see below) | “You tested this and it held” |
This turns your biggest liability into visible progress. People watch findings earn their status.
Ask for more before the first one. Consider requiring more evidence for someone's first result than their fiftieth. The first sets how they read everything after it.
Use the empty state. Someone who hasn't hit the threshold should be told what you're collecting and why, not shown a blank screen. It's the one place where being honest costs nothing and buys patience.
Six tests you can build this month
Everything above takes judgment. This part doesn't. These are six tests you can build, run automatically, and turn into a number you track over time.
Your pipeline is a measuring instrument, and instruments need calibrating. Nobody would trust a scale that had never been checked against a known weight. Most apps in this space have never been checked against a known nothing.
Shuffle the timestamps
Randomly shuffle the timestamps on a real user's data. This destroys any real relationship while keeping everything else identical. Run the whole pipeline on it, a few hundred times. Count what it finds. There should be almost nothing.
This one test is worth more than any amount of theorizing, because it measures what your actual system actually does when there is definitely nothing there. It also hands you something directly useful: now you know what score a fluke typically produces, so you can report each real finding as “stronger than 97% of pure noise”, a number you can honestly put in front of a user.
If you build one thing from this guide, build this.
Plant fake variables
Add made-up tracked items that look exactly like real ones (same frequency, same clustering) but generated with no connection to anything. They should never show up. How often they do is a false-positive rate you can measure continuously, on live data.
Matching the pattern is what makes this work. A smooth random number is a weak test. One that mimics “logged in bursts, a few times a month, more at weekends” is a real one.
Add the logging-count column
Rerun everything with the entry-count column added (#1). Count what disappears and what flips sign. That's your exposure to the biggest bias in the category, in one number. Track it permanently, not once.
Toggle the cutoff rule
Rerun with the “causes come first” filter off, on, and on for normal days too (#4). Anything whose direction depends on which version you ran is an artifact.
Stretch and squeeze the window
Rerun at half and double the window (#3). Real effects fade gradually. Artifacts appear at one setting and vanish either side.
Track how often findings come back
Keep the history and plot survival rate against how many nights something has held (#9). Both a test and your ranking signal.
A team that can say “our shuffle test finds false results 2% of the time, our planted fakes show up 1.8% of the time, and findings that have held ten nights hold an eleventh 94% of the time” is in a completely different position from a team that can't, with users, with clinicians, with regulators, and internally when deciding what to build next.
These six don't prove anything causes anything, and they don't fix any of the nine. What they do is make your error rate measurable. You can't improve what you can't measure, and in this field being able to measure it is rare enough to be a real advantage.
What this can and cannot tell you
Some things can't be fixed by better math. You can only account for what you track. If someone drinks wine mainly at restaurants, and restaurant meals differ in a dozen ways they never log, then “wine” quietly absorbs all of it. No model recovers what was never recorded.
But comparing a person to themselves buys you something real. Everything about them that doesn't change (genetics, baseline health, where they live, their other conditions) is perfectly accounted for. Not approximately. What it leaves open is everything that does change over time.
A pattern, in this one person, that holds up when you respect the order events happened in, account for the other things they track, and keep re-checking it as more data arrives.
That's genuinely valuable. It is not proof of cause, and your wording should never suggest it is. “Worth testing” is accurate. “Causes” is not. The gap between those two words is where trust gets lost.
How to get past the ceiling
If you can narrow things down to a few strong candidates, you can then help someone test one properly. Pick something the person controls. Have them deliberately do it for a block of days, then deliberately not do it for a block, then repeat, with your app deciding the schedule, not them.
When a coin flip decides whether they drink wine this week, the wine is no longer tied to their social calendar. The restaurant problem disappears, along with every other hidden factor you never tracked.
This is a real experiment, and it supports a claim about cause that no amount of pattern-finding can.
The pattern-finding isn't the product. It's how you work out what's worth testing.
Observational data is bad at proving cause and excellent at cheaply narrowing a huge list down to a few candidates. Nobody can run structured experiments on two hundred things. They can run three. Everything in this guide exists to pick the right three.
It also gives your confidence labels a real top rung: early signal → consistent → you tested it yourself.
Practical limits, all manageable. Only things the person controls can be tested. They'll know which block they're in, so expectation plays a part. Say so. Anything with lasting effects needs a gap between blocks. And you have to track whether they actually stuck to the schedule.
The checklist
A half-day review. Each row points back to the problem it comes from.
Open the 27-point checklist
| # | Check | What it looks like when it's wrong | § |
|---|---|---|---|
| 1 | Do you tell “didn't happen” apart from “didn't log”? | Lots of unexplained “protective” findings | 01 |
| 2 | Is a daily entry-count column in every model? | Findings track logging habits, not behavior | 01 |
| 3 | Are timestamps converted per-event using the zone name? | Logging hours jump by an hour at daylight saving | 02 |
| 4 | Are windows built around the symptom, not the calendar? | Evening→morning links never show up | 03 |
| 5 | Does window length depend on what's being tracked? | Fast-moving things show nothing | 03 |
| 6 | Do you drop anything logged after the symptom? | Medicines “cause” the thing they treat | 04 |
| 7 | Do you cut normal days short too? | Evening habits look protective | 04 |
| 8 | Do “today” and “rolling” windows overlap? | Coefficients flip sign between nights | 05 |
| 9 | Is there a fixed list with alias matching? | Users have hundreds of distinct typed strings | 06 |
| 10 | Does category depth adapt to how much someone logs? | Light users get no findings at all | 06 |
| # | Check | What it looks like when it's wrong | § |
|---|---|---|---|
| 11 | Are effects averaged over active days only? | Everything's tiny; “there's no signal” | 07 |
| 12 | Is “typical amount” from active days only? | You're comparing against a daily rate | 07 |
| 13 | Are time and active-day thresholds both required? | Occasional loggers get findings | 08 |
| 14 | Is the occurrence count checked on the original column? | Findings built on a single logged event | 08 |
| 15 | Do you say when you skipped something, and why? | “Nothing found” and “didn't look” are identical | 08 |
| 16 | Is the score labeled honestly, not as a p-value? | Users read a ranking score as a certainty | 09 |
| 17 | Do you correct for multiple tests within each run? | More symptoms tracked = more findings, always | 09 |
| 18 | Are findings ranked by how often they come back? | One-off flukes look like solid patterns | 09 |
| 19 | Are you using small-sample math (Firth)? | Wild numbers for brand-new users | New user |
| 20 | Are confidence levels shown in the UI? | Everything looks equally certain | New user |
| # | Check | What it looks like when it's wrong | § |
|---|---|---|---|
| 21 | Shuffle test running and tracked? | You don't know your false-positive rate | Test 1 |
| 22 | Planted fake variables in production? | No live measurement of false positives | Test 2 |
| 23 | Logging-count comparison tracked over time? | Adherence bias unmeasured | Test 3 |
| 24 | Cutoff and window comparisons run? | Can't tell artifacts from real findings | Tests 4–5 |
| 25 | Survival curve maintained and reviewed? | No evidence behind your confidence claims | Test 6 |
| 26 | Does your wording avoid claiming cause? | “Causes” or “triggers” in the UI | Limits |
| 27 | Can a user turn a finding into a real test? | You've accepted the ceiling | Limits |
Where to start
If this applies to something you're building, here's the order to do it in.
Frequently asked questions
Why does filling blank entries with zero create false findings in health tracking apps?
An empty cell can mean the person didn’t do something, or that they did it and didn’t write it down, and those two aren’t interchangeable. People log less when they feel worse, so bad days have fewer entries. If those blanks become zeros, bad days turn into “low everything” days and every under-logged habit looks protective. The cheapest fix is to add one column to every model: how many entries the person made that day.
Should a health app group days by UTC or by the person’s time zone?
By the person’s local time. Bodies run on local time, and a UTC day boundary cuts through the middle of the night for most of the world, splitting one night’s sleep across two rows. Store timestamps in UTC along with the person’s time zone name (like America/Denver), not a fixed offset, and convert each timestamp to local time at the moment you group. Applying today’s offset to old data shifts someone’s entire past by an hour at every daylight-saving change.
Why is a calendar day the wrong window for symptom data?
Calendar grouping can cut a cause off from its effect. A headache at 9am Tuesday most likely has its cause on Monday evening, which lands in a different row, while Tuesday afternoon, which cannot have caused it, lands in the same row. Build the window around the symptom instead, looking back a fixed amount of time, and let the length depend on what you’re tracking: hours for blood sugar, a day for headaches and gut symptoms, several days for flares.
Why do painkillers appear to cause headaches in app data?
Because people take medicine after they get sick, and most pipelines never ask whether something logged after the symptom should count as a possible cause. Drop anything logged after the symptom, then cut normal days short too so both kinds of day cover the same slice of time. Otherwise an evening habit goes uncounted on symptom days and fully counted on normal days, and the app reports it as protective.
Can I trust a p-value from a health app that re-runs its analysis every night?
Not as a false-positive rate. The variables were picked using the same data they’re tested on, so a p-value of 0.03 isn’t a 3% chance of being a fluke; it’s a ranking score. And because the analysis runs every night, two years with thirty symptoms and three findings each adds up to roughly 65,000 reported associations for one person. A better trust signal is how often a finding comes back: a fluke doesn’t line up again tomorrow, while a real effect holds as evidence accumulates.
When should a health tracking app say “not yet” instead of showing a finding?
When the person hasn’t cleared minimum thresholds. Below a certain amount of data, these models don’t give a weak answer, they give an arbitrary one. The guide recommends three thresholds: enough calendar time and enough logged days (both required together), enough occurrences of the symptom, and enough occurrences of the thing being tested. Saying nothing is a designed outcome, and the empty state should tell the person what you’re collecting and why.
How do I test whether my health analytics pipeline is producing real findings?
Build six automated tests: shuffle the timestamps on real data and count what the pipeline still finds (there should be almost nothing); plant fake variables that look like real ones and measure how often they show up; rerun with the entry-count column added; toggle the “causes come first” cutoff rule; stretch and squeeze the time window; and track how often findings come back night after night. Together they turn “we think it works” into error rates you can track.
Can pattern-finding in self-tracked data prove that something causes a symptom?
No. It gives you a pattern in one person that holds up over time, which is worth testing but is not proof of cause, so the wording in your app should say “worth testing,” not “causes.” To get past that ceiling, help the person run a real experiment on a strong candidate: do it for a block of days, then deliberately don’t for a block, then repeat, with the app deciding the schedule rather than the person.
What is an N-of-1 study in health tracking?
It means studying one person over time and comparing them to themselves, rather than comparing different people. Everything about that person that doesn’t change, such as genetics, baseline health, where they live and their other conditions, is accounted for perfectly. What it leaves open is everything that does change over time and that the person never tracked.
The plumbing matters more than the model
The methods for studying one person properly have been around for forty years. The computing to do it for everyone, every night, for pennies, now exists too. What sits between them isn't a modeling problem. It's a construction problem, spread across a dozen decisions that each look like plumbing and each push results in a particular direction.
The plumbing matters more than the model. A week auditing how you build your table will do more than a quarter improving your math. Several of the nine don't make results slightly worse. They flip them backwards.
How often something comes back beats how strong it looks. Your system already works this out nightly and deletes it. It needs no assumptions, can't be fooled by your screening step, and costs a table and a query.
Pattern-finding is how you choose what to test, not the answer itself. No amount of care turns observational data into proof. But it's excellent at narrowing two hundred possibilities to three, and a person who can't test two hundred things can easily test three.
The uncomfortable summary: most apps in this category are showing people results whose error rate has never been measured. That's fixable in a month, and the six tests above are how. Everything else here gets easier once that number exists, because for the first time you can see whether a change helped.
Glossary
Open the glossary
| Term | What it means |
|---|---|
| Coefficient | The number a model produces for each thing it tested, showing how strongly it's linked to the outcome. Not readable by users as-is. |
| Regression | The family of math used to work out which of many things is linked to an outcome, while accounting for the others. |
| Logistic regression | The version used when the outcome is yes/no: did the headache happen or not. |
| p-value | Meant to express how likely a result is to be a fluke. Only means that if you decided what to test before looking, which these systems don't. See #9. |
| False discovery rate | A correction applied when you test many things at once, so most of what you report is real. |
| Firth's method | A version of logistic regression built for small amounts of data. Corrects the tendency to overstate effects. |
| Confounding | When something you didn't track is the real explanation. Wine looks harmful, but it's the restaurant meals that come with it. |
| Marginal effect | The plain-language answer: how much the probability changes if they do this versus don't. |
| N-of-1 | Studying one person over time and comparing them to themselves, rather than comparing different people. |
- Guyatt, G. et al. Determining optimal therapy: randomized trials in individual patients. New England Journal of Medicine, 1986.
- Lillie, E.O. et al. The n-of-1 clinical trial: the ultimate strategy for individualizing medicine? Personalized Medicine, 2011.
- Kravitz, R.L., Duan, N. (eds.). Design and Implementation of N-of-1 Trials: A User's Guide. Agency for Healthcare Research and Quality, 2014.
- Shiffman, S., Stone, A.A., Hufford, M.R. Ecological momentary assessment. Annual Review of Clinical Psychology, 2008.
- Benjamini, Y., Hochberg, Y. Controlling the false discovery rate. Journal of the Royal Statistical Society Series B, 1995.
- Firth, D. Bias reduction of maximum likelihood estimates. Biometrika, 1993.
- Heinze, G., Schemper, M. A solution to the problem of separation in logistic regression. Statistics in Medicine, 2002.
- Meinshausen, N., Bühlmann, P. Stability selection. Journal of the Royal Statistical Society Series B, 2010.
- Little, R.J.A., Rubin, D.B. Statistical Analysis with Missing Data, 3rd ed. Wiley, 2019.
- Hernán, M.A., Robins, J.M. Causal Inference: What If. Chapman & Hall/CRC, 2020.
This guide describes problems and fixes that apply to any app finding patterns in one person's self-tracked health data over time. It names no product or company.