API: Reduce JSON data sizes to cut traffic costs and improve performance

[RalphBike]

Disclaimer 1: I know this is not the right place for this ticket, but the API repository is closed and read-only. As discussed with @mltbnz , I am opening this ticket here, please feel free to move it.
Disclaimer 2: I am not sure if all my assumptions or findings are correct. Please excuse if I got something wrong or do not have a full picture.

TL;DR: The current JSON format is inefficient and generates lots of traffic. Therefore I suggest several actions to reduce data size and costs. All actions are sorted from least intrusive to more complex.

I had a closer look at the data provided by the back end and would like to suggest some improvements to reduce traffic, cut hosting costs and improve performance and stability for all users.

Current status

• The app gets the user positions from the server in JSON format. The current JSON is valid but not very efficient: it contains non-functional unnecessary white space and long identifiers like "device", "latitude", "longitude", "timestamp" which are used repeatedly.
• The values for lat/long, ID, and timestamp are longer than needed.
• There are very little or none optimizations on the server in place: server side compression (transfer encoding) is not supported.
• I do not think there is any location filter in place which would provide each user only positions in his area. Instead every user downloads all the device positions on the planet all the time.
• Users of the mobile clients (at least on Android) are never asked if they still want to send there position or if they would like to either become an observer or shut down the app. This creates useless traffic, spoils data quality and invades user privacy.

Suggested changes

(sorted by effort to implement, least expensive first)

1. Requiring no changes on the clients
1.1. Reduce all unnecessary white space/blanks from the JSON. Or minify it.
1.2. Enable server side compression (transfer encoding).
1.3. Reduce device ID length from 128 characters(android)/40 characters (Iphone) to 40/40 or 32/32. Even further reductions should be doable without collisions.

2. Little changes on the clients
2.1. Implement a region of interest (ROI) filter on the back end to make sure only devices are downloaded that are relevant to the user (visible in the current view port plus some border around). The clients should only requests data for their current viewport (plus some more) and the server should only provide the needed positions. In case of changes to the map (zoom/move/etc.), the clients needs to request new data. URL could be something like ...?latMin=53.528&latMax=53.579&lonMin=9.818&lonMax=10.181.
2.2. Implement a timeout for position sending and a position check on all clients to prevent people from sending their positions for many hours or sending their unchanged position for a long time. E.g.: ask the users after 4 hours of sending, if they really want do continue. And: ask the user if they still want to send after 30 minutes without a change of the position.

3. Larger changes to the clients
3.1. Use a more efficient JSON format, without the repeated use of long identifiers, see below.
3.2. Reduce GPS precision from 6 decimals (Precision 11.11 cm) to 5 digits (Precision 1.11 meters).
3.3. Use delta timestamps within the JSON and only one absolute timestamp per JSON.

Sample JSON code for one device

Old JSON, 245 characters per device:

{"device": "3a65e13300839ab0727a355179b4eb7319490e363a65e13300839ab0727a355179b4eb7319490e363a65e13300839ab0727a355179b4eb7319490e363a65e13300839ab0727a355179b4eb7319490e36", "latitude": 53183718, "longitude": 12974962, "timestamp": 1779483918},

New JSON, 56 characters per device:

["4a4ae4a6a0c559a40cb94e04226b398e",5249335,1337942,27],

Combined with the compression I would expect to bring traffic down by factor 5-10, with the ROI filter even more.

I have working proof-of-concept code for all of the changes and am happy to assist.

[RalphBike]

@stephanlindauer FYI

[stephanlindauer]

Hi @RalphBike ,

Thanks for putting time and effort into this!

Combined with the compression I would expect to bring traffic down by factor 5-10, with the ROI filter even more.

My assumption would be that run-length encoding as part of the normal compression makes that difference insignificant. With this assumption there was greater benefit in having a more readable structure than a short one.

1.2. Enable server side compression (transfer encoding).

Is turned on, if it's requested. What request headers is your client sending?

➜  ~ curl -i -H "Accept-Encoding: gzip" https://api.criticalmaps.net
HTTP/2 200
content-type: application/json
date: Wed, 15 Jul 2026 10:09:55 GMT
content-encoding: gzip
...

1.3. Reduce device ID length from 128 characters(android)/40 characters (Iphone) to 40/40 or 32/32. Even further reductions should be doable without collisions.

Yes, point taken.

3.2. Reduce GPS precision from 6 decimals (Precision 11.11 cm) to 5 digits (Precision 1.11 meters).

That would also make sense IMHO.

3.3. Use delta timestamps within the JSON and only one absolute timestamp per JSON.

We use timestamps in the requests because we can't guarantee order of request and just use that in the backend to let the last timestamp win. The temporarily persistent data will only contain a timestamp, which the server has set.

Users of the mobile clients (at least on Android) are never asked if they still want to send there position or if they would like to either become an observer or shut down the app. This creates useless traffic, spoils data quality and invades user privacy.

Afaik that is not true because we explicitly display a persistent notification that makes users aware of their location being shared. On the very first start of the app the users is also shown the data privacy information and asked to confirm this. If this is not the case on your device, we should definitely have a deeper look there.

I do not think there is any location filter in place which would provide each user only positions in his area. Instead every user downloads all the device positions on the planet all the time.

This optimization was already discussed (either to limit by distance to user or by viewport of device) and dropped because of the following assumptions:

• Egress is cheap.
• Compute is expensive.
• There are potentially different solutions to that problem that don't tend to go into O(n^2) complexity immediately and therefore into a greater risk of spending too much money. One of those would be bucketing/sharding into different areas (like cities, countries, etc.), but the code and infra complexity was something we wanted to avoid.
• Only events in roughly the same timezone would overlap (which creates a bit of a separation between different continents at least.
• It is nice to see bike protests on a global scale.

In principle simplicity typically won over optimizations. There were even certain features removed from the API over time to make it even less complex. The problem has always been that it was always just me working on this, with barely any time these days and for potential contributors the idea was to keep the entry barrier as low as possible.

I think there is some good stuff in your suggestion, and some assumptions on both sides that have not been proven yet. If you want to implement them, we need to talk about who will own the backend and those changes long term.

[RalphBike]

Hey @stephanlindauer,

thank you for the feedback and taking the time for it. I am going to answer only a few points, because I don't think it makes much sense to discuss everything in great depth. If you are interested in something specific, please let me know.

"We use timestamps in the requests because we can't guarantee order of request and just use that in the backend to let the last timestamp win. The temporarily persistent data will only contain a timestamp, which the server has set."

Well, currently very similar epoch timestamps are sent over and over again, this makes little sense from data size perspective. Instead you could send one single absolute/full timestamp (10 digits) and N relative ones with 1-3 digits, regardless of any order or age.

Readability of the JSON to humans:
Is this really a relevant topic here? My reduced JSON is also readable, with the constant ID lengths maybe even better. If readability is important, I'd rather remove the whitespace and add a linebreak after each item.

"Checks if users still want to send their position"
I do not think there are enough working checks in place. Just have a look at the number of active people after big events. E.g. how many people are sending their (mostly static) positions after Sternfahrt Berlin after several hours or even days. It takes quite some time for the number of users to go down to a regular level. Or check the number of people sitting at home during the mass sending their couch positions for hours.

ROI filter:
My idea is not to stop people from seeing all demonstrations worldwide. I want to provide only the positions that are interesting to each user.
I would not overcomplicate things with time zones, etc. My assumption is that the computation is not that expensive (only a modified DB query) and would reduce CPU load due to smaller data size (less devices, less JSON, less compression time, shorter download times, less parallel downloads, etc). If you want I could try to simulate the traffic reduction with an ROI filter, I have some old data available. This would of course be based on assumptions and not cover CPU, RAM, etc..

retrieve_locations.sql (new point):
As we are talking about CPU load during retrieval of the positions: I think the SQL query could be more efficient. It makes limited sense to do DELETE and GET in the same query. This way you create another O(n2) action and double the amount of DB write actions per device. It also makes little sense to check the age of the positions again after the old ones have just been deleted. I'd rather do the delete in a separate job, running at a constant low frequency and not on every request. Like every 5 min to several days which would mean O(n) instead of O(n2) for deleting. With the saved CPU load, you could easily add a ROI check to that query.

Implementation:
Thanks for the offer, however I am currently not able to do the implementation and take the responsibility for all the connected plattforms. This should rather be a team effort than a one man show. I just had the feeling that you guys would like lower traffic bills, therefore I felt I had to give you feedback.

What are your thoughts now? Do you intend to implement some of the ideas?