Is Meshtastic becoming a victim of its own Success? Default Channel Congestion and New User Experience

[korbinianbauer]

Hey everyone,

I'd like to share our experience and would like to know what you think about it. Long post, no tl;dr.

---

A friend of mine and I have been looking for quite a while for an infrastructure-independent way of communication for my friends, which are scattered across ~50km in the swiss and austrian alps.
Point-to-point stuff like CB radio doesn't work here and relays are inhibitively expensive, big and energy-hungry.

We went with Meshtastic, since it appeared to be the most mature and widespread project of all the LoRa mesh things.
We bought some nodes and tested direct line-of-sight communication, which worked well. I was pleased to see a lot of other nodes whenever I took my T-Deck up some mountain. So we set up a nice solar node on a summit that a my friend and I both had LOS to and started testing.

Unfortunately messaging via that 1 hop was extremely unreliable at best, maybe 1 out of 10 DMs arrived.
We were hesitant to switch to a different radio preset, since we knew that would mean not being able to communicate with all the other nodes we were seeing. I also thought ~35% ChUtil at the mountaintop node isn't too bad, so switching probably wouldn't help anyways.

We spent a couple weeks testing, reading code and logs, kept watching videos and reading blogs comparing Meshtastic to other projects.
The gist always seemed to be about how that other thing that calls itself Mesh but somehow isn't(?) is so much more reliable at getting messages delivered, but lacks many features.

But since getting a fair comparison to MC would mean we'd have to hike to the mountaintop node again and put another solar node there, we decided to give a different radio preset a shot anyways.

We switched our personal nodes and the mountainttop node over from LongFast to MediumFast and suddenly the experience was completely different.
Still not as reliable as what you're used to from internet-based messengers, but now ~8-9 out of 10 DMs arrived

Turns out our mountaintop node was just getting bombarded on LongFast and most packets got lost with ChUtil sitting around 35%.

Now that I knew LoRa is in principle able to get data across in our scenario, I wondered if MC could make it 10 out of 10.

I assumed MC's protocol is somehow fundamentally different, that repeaters would inherently store and forward messages, so each of them makes sure the message gets delivered eventually, even if e.g. sender and receiver aren't reachable at the same time. Something like that.

But testing it, I found it actually isn't any more reliable then Meshtastic on a channel that's also almost empty. At all. Digging into the Code I realized MC is actually not doing any of that for DMs and collisions swallow messages just as much.

---

So it seems Meshtastic is kind of suffering from success: The default LongFast on SF11 is getting congested in many places, while MC's default SF8 channel still works well, because hardly anyone uses it yet, and the lack of features also means less traffic inherently.

New users who experience the congested LongFast channel won't think, "Ah, RF congestion on this specific frequency and modem setting." They'll likely think, "Meshtastic is unreliable.", then try a competitor on an empty channel and think, "Wow, this other project is so much better!"

I feel like getting the congestion issue under control is a strategically important matter for Meshtastic as a whole, because it probably pushes many users away to other projects.
Not because the other projects actually have a solution for it, but because they're not suffering from it YET.

I am working on a throttling approach based on Token Bucket, with the goal of getting ChUtil back to e.g. 25% under all circumstances.
I don't know yet if that's going to work, and of course it comes at a cost: Scaling down broadcast frequencies and also potentially hop_start automatically and more aggressively than it is done now. But this discussion isn't about my proposal, but about the motivation behind it.

Before I sink more time into that project, I'd like to know if this is even as relevant as I think it is, and if my goals are aligned with the project.

• Am I on the wrong path thinking this likely is a universal experience for Meshtastic users?
• Is trying to fix this without having to isolate to a different preset worth the effort?

Maybe the Leads of the Meshtastic project have an entirely different view than I do, e.g. that local-ish meshes with their own presets is the way to go.

For us, Meshtastic does what we need it to do now. But it required a lot of research and persistence, setting up all of the infrastructure nodes we need to cover all our friend group's homes and isolating our mesh from almost everyone else around us.

[thebentern]

I largely agree with your assessment, and in fact, I detailed some of the history and details of the default LongFast choice in the blog post:
https://meshtastic.org/blog/why-your-mesh-should-switch-from-longfast/
Meshtastic blew up from an era when the average mesh size was under 50 nodes and the focus was more ad-hoc private meshes for outdoor activities where we were exploring what kinds of data we could send over the mesh. We've done a lot to curb and throttle traffic since then, as the idea of a larger scale "public mesh" formed out of community, as well as encouraged folks to try out higher bandwidth presets, custom channels, etc. Unforunately, a number of the issues of the past continue to plague existing meshes as well, since there are large quantity of abandoned nodes on older firmware versions in some dense meshes with less conservative airtime utilization. I'm glad to see that you made an apples-to-apples comparison between Meshtastic and other projects in the space, because many times the comparisons are based on the default LongFast preset, which in my mind is has not been a fair comparison, due to bandwidth constraints and existing polution of problematic nodes, etc. And we are unlikely to change default presets until we introduce breaking changes in the 3.0 protocol.

That said, I look forward to seeing your proposal(s) as well as @erayd's packet replay work to help increase reliability of meshes in the existing situation we find ourselves in.

[NomDeTom]

I've already pointed you towards one of the ideas (#8115) on The Big List which concerns managing chutil in a neighbour-friendly way.

Another one that might help in the EU space (which you and I share) is this proposal which is erroneously marked for 3.0. The idea of chopping the 250khz band into smaller slots, and using one edge, would achieve similar clear-water separation between current nodes and those older ones with lower levels of control.

In local trials, the "EdgeFast" preset was successful in bypassing much of the noise on the centre frequency, although subsequent discussion has revealed some further considerations. If you wanted to discuss how we can do that for the current iteration, I'd be very grateful for the help.

Finally, one of the ideas in The List is for time-limited 'router' roles, so that they can gracefully retire to civilian client roles if they're not needed or maintained.

[shalberd]

In local trials, the "EdgeFast" preset was successful in bypassing much of the noise on the centre frequency, although subsequent discussion has revealed some further considerations.

in Europe, or at least in Switzerland, on EU_868, the upper edge of the band has MeshCore using it:
Frequency= 869.618 MHz
Bandwidth = 62.5 kHz
SF = 8
CR = 8

Source: https://www.meshcore.ch/settings

[shalberd]

is for time-limited 'router' roles .. so that they can gracefully retire to civilian client roles if they're not needed or maintained.

even good for ROUTER_LATE and CLIENT_BASE. It is safe to assume that, in a healthy mesh with involved node owners, even up on high mountains, getting to a Router once every year, or at least once every 1.5 years, is possible and can be expected.
How do you define the "not maintained" status? Long time between reboots?

[NomDeTom]

getting to a Router once every year, or at least once every 1.5 years, is possible and can be expected. How do you define the "not maintained" status? Long time between reboots?

Still up for debate, tbh. The main idea is a persistent uptime count, so even intermittent nodes will eventually fall back. It would be a remote reset of the count, so physical access is not required, just evidence that you still have admin access.

[shalberd]

The main idea is a persistent uptime count, so even intermittent nodes will eventually fall back.

of course, for the two non-ideal nodes we have here in Switzerland of type ultra-aggressive Router, cause they won't upgrade their firmware most likely, this won't help. But for all future nodes in the firmware to come, this would help, yeah. In any case, around here with Medium_Fast, high node density, nodes on mountains seeing each other 360 degrees, we have found Router_Late to be much better than Router. But even then, we have to closely watch channel util, so it does not go above 25%.
We have 500+ nodes, not all of them active at the same time (around 150 - 250 online), here in Switzerland.
Convincing people to use Client_Mute more, only using Client in strategic locations .... trying Client first in mountain locations, then switching over to Router_Late on an as-needed basis ...

[dem1tris]

Hands down, this is true! I want to emphasise that the main reason for congestion is rather the default "apps" flood and suboptimal default settings, not the LongFast limitations. Here's a screenshot from one of the network monitoring services showing the payload types ratios.

That's overwhelming. The network could be barely alive without any text messages. Thanks for the forced Telemetry shutdown in newer firmware versions, but still. Smart Position is also a major issue. Enabled by default with 15-30 seconds delay. Even for equipment intended for home use, even in powered state! With slightly worse GPS quality, the home node becomes a flood source for the mesh.

To cut to the chase, here's a quick list of essential tasks to prioritize:

• [Feature Request]: Patch and release the stable firmware version with the telemetry disabled by default (network congestion) #8764
  Because the flasher suggests the latest stable version by default;
• [Feature Request]: Rework Smart Position for home-based nodes (network congestion) #8765
  Fully rework Smart Position and the entire POSITION_APP defaults. Differentiate defaults by hardware type, power state, and so on. Home nodes mustn't flood with position on default settings;
• [Feature Request]: Power-fault Flooding Protection (network congestion) #8763
  Every reboot triggers NODEINFO_APP (and position, and telemetry) if I'm right, and no one can even notice this. So the firmware must protect the node from being a flood source even if it faces regular power faults.

[dem1tris]

Ok, let's even start with the calculations.

• 98 packets per hour
• On long moderate ~1 sec per Telemetry and Position, ~1.3 per NodeInfo. Let it be 1.1
• I'll hear them. 1
• I'll retranslate most of them. +0.88 (7 chances from 8 possible hop options 0-7)
• I'll hear the retranslation of the next hop of mine. +0.86 (6/7)
• Maaaaybe I'll even hear one else retranslation. Let it be + 0.1?
  98 * 1.1 sec * (1 + 0.88 + 0.86 + 0.1) = 98 * 1.1 sec * 2,84 = 306 seconds per hour. I's already 8,5% ChUtil.

And this is for the single send attempt. The impact and probability of 3-5 collision-caused sending repetitions is uncountable.

Welcome to add what I missed here :)

[korbinianbauer]

despite 90% of packets can be literally garbage

How do you know? Not everyone is using Meshtastic for the same thing.

IMO, air utilization just needs to be A) limited to a healthy amount and B) shared fairly.

If everyone else uses their share for what you consider literal garbage, so be it.

[dem1tris]

How do you know?

The histogram shows this.

Not everyone is using Meshtastic for the same thing.

That’s true, and that’s why the default settings must be chat-friendly and flood-protecting. If you use Meshtastic for some specific position-related or telemetry-related things, you can just set it up as it needs.

healthy amount and fair share

I don’t argue with that. But how we define healthy amount and fair channel share for Position packets of home-based node? For reboot-caused NodeInfos? That’s what I call garbage.

uses their share

In the mesh it is never “their share”, it’s always “our share”, am I right? I don’t want to restrict any intentional and careful usage of the mesh. I only against out-of-the-box flood-prone default settings, that’s all

[korbinianbauer]

The histogram shows this.

What about it?

But how we define healthy amount and fair channel share for...

Wrong question. Healthy is whatever keeps collisions at an acceptable level, fair means equal here.

In the mesh it is never “their share”, it’s always “our share”, am I right?

No. It's "their" share of "our" shared resource, which is time on air.

[dem1tris]

Sorry, I’m not quite follow what are you arguing with. I see that you protecting the rights of mesh members to use it in different ways. Don’t get me wrong, nothing of my proposals or thoughts is about usage patterns restrictions.

I’d happy to be the member of the mesh which is healthy and useful, not only for text messaging. And I’m eager for maintaining such health. That’s why I left comments here and share “ChUtil and flooding” awareness within the local community.