aaverbitskiy/mapdash
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

bookings/map: read pf_board and board_info with FINAL to dedup ReplacingMergeTree

Browse Source 
Both tables are ReplacingMergeTree; dedup happens only during background part
merges, not at read time. The ongoing etl_rw feed keeps inserting into pf_board,
so right after an insert that updates an existing row there is a short window
(before the forced merge, min_age_to_force_merge_seconds=15) where the same
task_id / board_key exists in two parts. A plain SELECT reads both parts and
returns the row twice.

The /api/boards query already collapses duplicates via argMax + GROUP BY, but
/api/bookings and the two /api/map queries read raw rows, so they could
intermittently double a booking (timeline/balloon) or a surface marker.

Add FINAL to those three reads (pf_board in bookings + map current, board_info
in map surfaces). At 463/515 rows the cost is negligible and it removes the
class of transient duplicates regardless of merge timing. Output verified
unchanged against the single-part baseline.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
aaverbitskiy 2026-08-05 19:25:40 +00:00
parent 6816ba90f4
commit 1f5b393675
1 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
main.py
View File

@ -221,7 +221,7 @@ async def bookings(city: str = "", dimension: str = "", brand: str = "", manager
toString(end_date) AS end_date, toString(end_date) AS end_date,
brand, brand,
company_name company_name
FROM default.pf_board FROM default.pf_board FINAL
WHERE {where} WHERE {where}
ORDER BY board_id, start_date ORDER BY board_id, start_date
""" """
@ -255,7 +255,7 @@ async def map_data(city: str = "", dimension: str = "", brand: str = "", manager
side, side,
latitude AS lat, latitude AS lat,
longitude AS lon longitude AS lon
FROM default.board_info FROM default.board_info FINAL
WHERE {where} WHERE {where}
ORDER BY city, address ORDER BY city, address
""" """
@ -271,7 +271,7 @@ async def map_data(city: str = "", dimension: str = "", brand: str = "", manager
toString(end_date) AS end_date, toString(end_date) AS end_date,
brand, brand,
company_name company_name
FROM default.pf_board FROM default.pf_board FINAL
WHERE board_key != 0 WHERE board_key != 0
AND pf_board.start_date <= today() AND pf_board.start_date <= today()
AND pf_board.end_date >= today() AND pf_board.end_date >= today()