<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[🗄️ Where IoT data actually goes]]></title><description><![CDATA[<p dir="auto">One device sending telemetry every 5 seconds produces about 17,000 rows a day. A hundred devices produce 1.7 million rows a day — all timestamped, all needing fast answers for dashboards and alerts: average temperature per hour, last-seen per device, events inside a time window. This is where most IoT projects quietly succeed or fail, long after the firmware works.</p>
<h3>Start with Postgres, think in time</h3>
<p dir="auto">Postgres handles millions to low billions of rows on ordinary hardware when the schema is narrow, indexed on <code>(device_id, ts)</code>, and old data is dropped by time range. The discipline that makes it scale — and yes, this is a data-modelling problem first, which is why it helps to have the fundamentals straight:<br />
→ <a href="https://embiq.grapesiot.com/application-development/31-database-fundamentals/" rel="nofollow ugc">Database fundamentals: modelling and normalisation</a></p>
<ul>
<li><strong>One time column to rule queries.</strong> Every analytical query bounds <code>ts</code>. Queries that filter on payload alone (<code>WHERE temperature_c &gt; 30</code>, no time bound) are full-table scans at any scale — always add the time bound first.</li>
<li><strong>Narrow rows, typed columns.</strong> Keep hot fields relational. Unmodelled JSON accumulates five spellings of temperature and breaks every downstream query — and columnar compression works best on typed columns, not opaque JSON.</li>
<li><strong>Retention is a partition operation.</strong> Dropping whole time partitions by age runs in milliseconds; deleting a year row-by-row can stall a cluster for days. Design deletes by time range from day one.<br />
→ <a href="https://embiq.grapesiot.com/application-development/32-sql-postgresql/chapter-05" rel="nofollow ugc">Native partitioning, compression thinking and retention</a></li>
<li><strong>Precompute the dashboards.</strong> An hourly rollup table turns a million-row dashboard scan into a 24-row lookup. Same <code>GROUP BY date_trunc</code> shape, precomputed — keep the bucket expression identical in raw and rollup form.<br />
→ <a href="https://embiq.grapesiot.com/application-development/32-sql-postgresql/chapter-02" rel="nofollow ugc">Joins, aggregation and subqueries</a></li>
</ul>
<h3>The Timescale upgrade path (concepts, no new queries)</h3>
<p dir="auto">When the fleet grows, TimescaleDB automates exactly these patterns with zero query rewrites — because your queries already bound <code>ts</code> and group by <code>(device_id, date_trunc)</code>:</p>
<ul>
<li><strong>Hypertables</strong> auto-create time chunks behind a normal-looking table (<code>create_hypertable</code> on <code>ts</code>), with per-chunk indexing and pruning that behaves like your manual partitions.</li>
<li><strong>Compression</strong> on chunks older than N days (roughly 10x on narrow numeric telemetry) for cheap long retention.</li>
<li><strong>Continuous aggregates</strong> replace hand-maintained hourly tables with background materialisation.</li>
<li><strong>Retention policies</strong> (<code>add_retention_policy</code>, e.g. raw 90 days, rollups 2 years) replace the detach-and-drop routine.</li>
</ul>
<p dir="auto">The rule from the field: earn the extension with growth numbers. A Timescale dependency for a million rows loses to a clean native design with pruning, two composite indexes, and a documented upgrade path.</p>
<hr />
<p dir="auto"><em>Finish with <a href="https://embiq.grapesiot.com/application-development/32-sql-postgresql/chapter-06" rel="nofollow ugc">the telemetry schema capstone</a> — a 500-sensor fleet designed and measured end to end. Post your schema, the slow query with its EXPLAIN, and the time range involved. Tag with <code>database</code>.</em></p>
]]></description><link>https://forum.grapesiot.com/topic/74/where-iot-data-actually-goes</link><generator>RSS for Node</generator><lastBuildDate>Thu, 19 Nov 2026 13:02:48 GMT</lastBuildDate><atom:link href="https://forum.grapesiot.com/topic/74.rss" rel="self" type="application/rss+xml"/><pubDate>Sat, 03 Oct 2026 11:29:39 GMT</pubDate><ttl>60</ttl></channel></rss>