I just need to log it and show it to them so that they will finally take this seriously. The network works great while connected to a "backup" 200Mb/200Mb connection that I got access to even without reconfiguring any traffic shaping, so I'm pretty sure it's something up on the ISP's side. Trust me, I'll be running and testing and logging this thoroughly. Unfortunately, everyone headed out for the long weekend there, so I'll have to pick up Monday. The network is currently running on a backup connection and that one is working fine, so the ONLY thing this connection will be handling is the load test. I think that stress-testing the connection into and out of the ISP's network will probably be sufficient to bring out the bad behavior. I'm also going to bump my internet service to 500/500 - I've actually been meaning to do that anyway even before all this, so I'll probably throw it on a VM, put that on an isolated VLAN, poke a hole in my firewall to that vlan/subnet, and go to town. I've used it a few times in the past, but it's been a few years and I never delved into all of the switches. I'm taking a crash course in iPerf right now. We don't have any direct traffic into our corporate network and our corporate network engineers don't get involved in or support any of our "field" networks anyway, so there is no single point to test from. All of our systems are hosted or cloud-based, and it's almost all going over port 443 (https). I've also had a user connect a laptop directly to the ISPs router while this was happening, and I'd see dropped pings coinciding with our network issues, so I'm pretty sure the issues are not just within our network. I'd see drops on Skype traffic, PoS/Credit card traffic, etc. I've had the fiber extensions from the MPOE to the router tested thoroughly, and I've replaced the SFP modules and the NVG router itself. Generally it can work fine for most of the day or crap out several times that day. It happens in spurts for anywhere from a few seconds to a few minutes, then spontaneously goes away for anywhere from a few minutes to a few hours, then repeats. The issue affects all traffic and all hosts from all subnets at the same time. PS - I'm not at the location, and any PC I can log into remotely to work on this is running Windows 10, so Linux is, unfortunately, out of scope). I need to be able to decisively show to the ISP that the issue is on their end, and I need to show them the packet drops etc while they are logged into their equipment. I need to be able to generate a limited amount of traffic - meaning not "flat-out", but targeting specific bandwidth (about 200-300 Mb) so that contention isn't a problem. I need to be able to generate a measured continuous load on the ISP, emulating a mix of HTTP, some streaming (we use Skype conferencing), and client-server-type transaction traffic which has tiny packets but is very latency sensitive (PoS credit card and payroll deduct database lookups). When I don't have a load on the service, ping times and latencies are perfect (2-9 ms). We're not generating more than about 150Mb-250Mb of traffic, so it shouldn't be contention. When we're having issues with TCP timeouts and packet loss, we see the same thing when we're plugged directly into the ISP's router as well, so I'm pretty sure it's nothing on our end, but the ISP is denying that there are issues. We have our own router (Fortinet 100E) behind that with various PoS systems and various office networks. The service is 500Mb/500Mb fiber, dynamic IP, and we pick up a private IP (.x) off of the ISP's router (Arris NVG 595). Have a remote site experiencing packet loss and latencies. I have a very urgent need for this and very little time to research this.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |