7 Vive Pro/16 base station setup issues

[VizionVR]

Hey there, I seem to be having issues with my setup.
When plugging 4 base stations into power, my Vive Pro tracks and detects the base stations flawlessly. However, I have 16 base station 2.0's in my area, and when all of them are on, SteamVR does not detect a single base station on any computers

Worth noting:
-Every station is labeled with their channel number and there are no conflicting channels (all 16 used)
-Happens on both SteamVR and SteamVR Beta
-User is not admin, but also happens on an admin user
-Every base station was individually tested and updated to latest firmware
-Every headset and controller was tested and updated to latest firmware
-SteamVR/headset/Computer were restarted in-between tests and changes
-At any point in time, 6-7 base stations may be visible to the vive, this changes as I move around (odd shaped play space)
-All 16 stations show up in 'base station channel management' in steamVR settings.
-There are no reflective surfaces
-There are 6 other Vive pros in the area
-There is no outside light
-All Vives are wired
-This same issue happens on all 7 computers in the area

Thanks for any help or direction you can point me in, I was led to believe that multi-vive setup with 2.0 was solid enough for normal use, but it's completely unusable as far as I can tell. Is there anything I can do to fix this?
Attached is what SteamVR looks like after a couple minutes of the headset sitting in the center of the area.

[HackPerception]

@VizionVR -

A couple of notes:

• Bluetooth power management must be disabled on each and every single SteamVR instance. You can ensure it's disabled via SteamVR -> Settings -> Bluetooth as well as SteamVR -> Settings -> Developer -> Disable Power Management. If you do not disable this on each and every station, an idle command can be sent out from one SteamVR instance that can affect multiple stations used for tracking by another HMD/session. Based on your description, this is my primary suspect.
• Currently, a SteamVR instance can only accept signals from a maximum of 4 stations during a tracked session. You cannot track an HMD using more than 4 stations currently.

SteamVR doesn't have the most robust UX/UI for multistation setups beyond the basic channel configuration tool and they haven't released any documentation on the topic. Here are some tips that can be used:

• When setting up advanced spaces like this, we'd generally recommend that you run roomsetup for each station with only the stations that will be used for that tracking session plugged in. For instance, if you plan to have an HMD tracked by 4 specific stations, ensure that those are the only ones that are plugged in when you run roomsetup and ensure all others in the space are unplugged.
• The basestation channel manager has an icon that denotes a station is "observed in a tracking session". This icon generally means that a roomsetup has been run with those stations used as the primary tracking stations. Use this to validate that roomsetup is being trained to the correct stations.
• "Reflective surfaces" in practice are extraordinarily hard to track down without specialized equipment because materials interact differently with IR light than optical light. A material may not be reflective in the visible spectrum but may be highly reflective in the IR spectrum; the same is true with opacity, some visibly opaque materials will be complete transparent in IR and vice versa. To detect reflections, generate a system report or open up the SteamVR web console and search for the term "back-facing". I'll post an example of what a reflection looks like in the logs below.
  • Sun Jun 26 2016 23:02:09.676 - lighthouse: LHR-4E8EF209 H: Dropped 32312 back-facing hits, 2069 non-clustered hits during the previous tracking session

I'll shoot you an email with some more information but I'd also generally recommend contacting support.enterprise@htc.com as this is an advanced use case.

[VizionVR]

Thank you for your help, I need all I can get.
I can confirm that bluetooth power management is disabled for every SteamVR instance. My initial screenshot is a bit misleading, as in reality it will occasionally 'connect' to 1-5 base stations. They will light up on steamVR, but the headset will never come off of 'searching' while urging me to make sure it's visible to the base stations.
I am more than happy to only use 4 stations per headset, I had initially  planned for it that way when mounting the stations since I was unsure of 4+ base station support at the time. Attached is a layout of my space, as well as the location and channel of my lighthouses.
You mention that I should only turn on each base station that will be used per tracked area when setting up room scale. If I were to do this, would steamVR then ignore or block all other stations once they are turned on again? Is there a way to permanently ignore or block all other base stations based on channel, or whitelist specific channels in some sort of config? for instance C:\Program Files (x86)\Steam\config\lighthouse ?
I'll be sure to message HTC's enterprise email, though I was unsure if they were more of a sales team and less of a technical support.
I have also attached the log files of the headset sitting in the middle of the playspace for a while, if that clears anything up.

SteamVR.txt

[HackPerception]

@VizionVR - SteamVR tracking is a proprietary Valve technology (alongside the SteamVR compositor and the SteamVR/OpenVR SDK) which means that Valve controls the overall hardware and software tech-stack and it's implementation. Valve hasn't really released any confirmed information about how 2.0 tracking works beyond the basics for in-home setup. At a high level, it would appear the current behavior is that it pulls data from the first 4 stations it sees but overall, the system is dynamic and those dynamic elements have not been documented publicly. We recommend training room setup with only the desired stations powered on to generate a clean lighthousedb file and essentially try to get the system to know which stations are essentially primary stations which are the desired ones for that specific tracking universe. Past a certain point, Valve is the only entity who can truly speak to the function and behavior of their algorithms.

One of our engineers has created an open source tool for arcades that attempts to "lock" configurations. It specifically has a feature to prevent SteamVR from making changes to the lighthousedb file which may be useful in your case. You can access it here. Doing roomsetup with the desired stations and then applying this tool may help.

I can see the behavior your describing in the logs - it looks like a bunch of this:

Tue Oct 22 2019 15:09:58.061 - lighthouse: ... SOB: add S-8 also seeing S-9 S-11 S-13
Tue Oct 22 2019 15:09:58.208 - lighthouse: ... SOB: add S-12 also seeing S-3 S-4 S-10
Tue Oct 22 2019 15:09:58.477 - lighthouse: ... SOB: add S-3 also seeing S-8 S-9 S-11 S-13
Tue Oct 22 2019 15:09:59.047 - lighthouse: ... SOB: drop S-11 seeing S-4 S-8
Tue Oct 22 2019 15:09:59.277 - lighthouse: ... SOB: drop S-11 seeing S-3 S-8 S-9 S-13
Tue Oct 22 2019 15:09:59.326 - lighthouse: ... SOB: add S-13 also seeing S-4 S-8
Tue Oct 22 2019 15:09:59.885 - lighthouse: ... SOB: drop S-4 seeing S-3 S-10 S-12

Outside of the software/compositor side stuff, I would note that putting 2.0 stations in corners cannibalizes their horizontal FOV. They have a 150x110 degree FOV with the long axis being the horizontal axis (hence the curved front). When you mount in a right angle corner, you effectively reduce that 150 degree horizontal FOV to 90 degrees and can cause bounceback issues (although I'm not seeing bounceback in your file). Mounting them in locations that maximize the 150 degree horizontal FOV and are within 5.5m of each other will help you reduce your overall station count and maximize each stations coverage potential.

[VizionVR]

@VibrantNebula

Thanks much for your help again, as much as I'd like to report that the third party fix worked, unfortunately it doesn't seem to have any effect. First thing I noticed is that it had a 'whitespace' error on startup which only seemed to prevent the audio controls and 'launch at startup', but it's possible it could be affecting the base station blocking in the background.
I deleted the lighthousedb and all associated Lighthouse folders, ran room setup with only 4 lighthouses present, then ran this program to lock the new lighthousedb file. I made sure I used the correct universe, then only checked the 4 base stations that were used (no other showed up). After committing changes, I turned on the rest of the lighthouses and the SteamVR window started to become populated with the other lighthouses, and my Vive Pro began to do the usual massive drifting/white flashes/bad tracking issues it had before. Log files showed it switching from lighthouse to lighthouse the same way as before the arcade fix.
If you have any other solutions or ideas, I'd be more than grateful. I've emailed both SteamVR and Vive enterprise with no response so far. If I could get in contact with the engineer who worked on this arcade software, maybe I could speak with them as well. We're a small family run business trying to open this VR arcade and currently this is one of the final issues we're facing that's preventing us from opening. Our setup as is does not allow for partitions or walls between stations, so I was banking on the idea that the 16 channels in Lighthouse 2.0s would be able to set up in a way that does not cause any interference. It was only until recently did I realize that SteamVR has had the idea of 'channels' on the backburner with no major support as far as I can tell.

[VARtex_0]

Good day VisionVR
It has now been 2-1/2 months since your last post.  Have you managed to resolve this issue?
I ask because I am also opening a VR Arcade with no walls and have been trying to setup 10 stations.  I am experiencing the same issues and would appreciate any assistance based on your experience.

@VibrantNebula

[dagillespie]

Hi - we are also looking at tracking across large spaces with Vive and are looking at how to achieve this. Is it possible to create a setup with more than 4 lighthouses successfully yet? We would like to understand the maximum number we could use as would be looking to track as large a space as possible (think warehouse scale so ideally 16).

@VibrantNebula

[HackPerception]

@VARtex_0 @dagillespie

Valve has not released any updates that would allow you to use more than 4 primary basestations per SteamVR instance. Valve manages and owns SteamVR tracking and the SteamVR compositor overall so these kinds of updates and features would come from them rather than Vive. The official maximum supported tracked volume remains 10x10m per Valve.

Currently, you can have 4 "primary stations". Technically, SteamVR can accept additional data from additional "secondary stations" and use it for the pose estimation but all of this is highly proprietary and Valve has created no front end GUI or API access to actually manage or control the behavior.