Forum is extremely slow at the moment.

[berntd]

It takes about 1 minute every time I click on anything on the forum.
Eventually the screen will come up with the heading, background gears and mechanism but no contents. That will take another xxs before it comes up.


Regards
Bernt

 

[berntd]

The slowness is exactly the same to today.

Is this is know issue?

 

[Dave Coatsworth]

Yes, it is a know problem. The folks at HQ are trying to track down the problem.

 

[zedric]

Looks like it is working much better now...

 

[John Matthews]

Can someone explain what caused the problem and how it was resolved?

John

 

[Dave Coatsworth]

The problem was with Comcast. Apparantly an area problem affecting upload speed.

 

[John Matthews]

Dave - thank you for your response.

However, if that is the case can someone explain why the loading of www.nawcc.org hosted elements was much quicker than www.mb.nawcc.org - the forum was slower than the main site and the loading of text was much faster than images. If these elements are stored on different servers, those servers would have to be linked by Comcast infrastructure elements for that to be the explanation. What I experienced didn't appear to be due to a network area problem.

I would love to see a diagram showing how all the NAWCC IT infrastructure is configured - does such a diagram exist?

John

 

[Dave Coatsworth]

John,
Upload speeds were equally bad from all servers. I suspect you were looking at content that was cached on www.nawcc.org since it does not change that often. Thus, it only appeared to be faster.

 

[John Matthews]

Upload speeds were equally bad from all servers. I suspect you were looking at content that was cached on www.nawcc.org since it does not change that often. Thus, it only appeared to be faster.

Dave Coatsworth , Tom McIntyre Leroy Baker

Just to clarify to those of you who believe this is due to cached content - it is not.

The reason for the difference in performance is because the routing is different between the nodes nawcc.org (ezsolutions) and mb/doc/net.nawcc.org

End of the routing to nawcc.org



No loss of packets addressed to both nodes nor loss of packets on the link between

Compare this with mb.nawcc.org



This clearly shows that the route is different and there are packets being lost on some of the links. Dropping 15% of the packets between 162.151.69.158 & 174.54.64.96 - is a severe network problem, which will seriously disrupt/fail the transfer of images and pdf files. There is also an issue between 174.54.64.96 and 50.244.233.125. None of the nodes are dropping packets directly addressed to them.

The routes diverge earlier in the comcast network. For nawcc.org



For the mb nodes etc




Given I have done the traces in the last 30 minutes after downloading a pdf file failed, I suspect when America awakes the situation will not improve.

I assume someone is already aware that these are the links that are causing the problems. I respectfully suggest we deserve a statement explaining in detail the nature of the problem, the actions being taken and how it will be fixed.

John

 

[John Matthews]

Update at 12:22 CET - just got worse as expected.



To demonstrate the impact ...



or to make it more obvious, compare



with



John

 

[novicetimekeeper]

I have tried to contribute on a few threads but the inability to get the larger pictures means I can't really answer questions.

 

[Walt Wallgren]

I am curious why nothing is being done. I log in almost every morning and for more than a week, the download speed is terrible. Somebody said the upload speed was causing a problem with the download speed. Huh Maybe John Matthews needs to talk to our ISP to get some answers.

The ideal time (according to internet marketers) for a website to load is 3 seconds. Our page-load speeds are a joke. Because this is a MB and we are wanting to see what is new, we will wait longer. BUT somebody who doesn't know about us and just finds us on google or elsewhere, if they have to wait more than 10 seconds for a page to load, they are all too often gone. We have lost a potential member but even more importantly, we have failed in our mission to provide education and information to the community.

This needs to be fixed or I will be gone for a while until it is.

 

[Dave Coatsworth]

Hello everyone,
I have been on the phone with Jim Price today. He has been in constant contact with the IT group in Columbia who, in turn, have been in contact with Comcast, as that is where we believe the problem lies. They will be sending a technician out tomorrow between 10am and 12pm to investigate the problem. I will post an update as soon as we know anything further.

In the meantime, we are also working on a proposal to move us to a 24/7 hosting environment. That's not going to happen overnight, but we are treating it with the highest priority.

 

[John Matthews]

Dave

The problem resides in the link with the errors and/or the devices at either end of the link. If all of these are Comcast's responsibility, then it is their problem. If not then it may not be.

I would expect a commercial network provider to have resilience in their network, ie multiple routes. It would be worth asking the question - is alternative routing possible? Unfortunately if the defective link & devices are local, there may be no alternatives.

In that situation you need to swap out the end devices and check the integrity of the link.

John

 

[Dave Coatsworth]

Thank you, John. I have passed your info along to the IT folks.

 

[John Matthews]

So we appear to be back to something like normal. Someone has fixed the link (just 1% loss) and/or devices that were losing packets. It would nice to know who manages this part of the system (I assume Comcast), the nature of the fault and what was done to fix it.



John

 

[Walt Wallgren]

Much better, Thank you!

 

[Dave T]

Yep, looks like we're back on track today! Thanks for all who have worked to correct this problem.

 

[Bill Stuntz]

It's nice to be able to use the MB again. Like John, I'd like to know what the problem was, and how it was fixed. If it was Comcast's problem, is there any compensation for the fact the system was essentially unusable?

 

[Dave Coatsworth]

Comcast was at HQ this morning. The tech said the problem was indeed theirs and was resolved last night. (Consistent with our experience.) Most of Columbia was affected. Unfortunately, since it was their issue, there's not much data available on what was fixed.

This is not the end of it. We will continue to pursue alternatives that will give us more reliable 24/7 service.

 

[John Matthews]

With all due respect there is. The problem was the link or the end nodes. They will know and they should be willing to share information to explain why it took so long to diagnose and repair.

John

 

[Bill Stuntz]

Does the fact that Camcast's service made our web sites essentially unusable entitle the NAWCC to compensation? Discounted service, rebates, service credit, etc?

 

[Tom McIntyre]

I agree with Dave. There is very little to be gained by pursuing this with ComCast. If we move the forums off site, it will solve the problem. We cannot move our telephone system off site and probably not the iMIS System or the MSSQL Server, or the Exchange Server all of which are critical to local operations.

We may be entitled to some compensation if we have a Service Level Agreement as we should.

 

[John Matthews]

There is very little to be gained by pursuing this with ComCast. If we move the forums off site, it will solve the problem.

I disagree. There is very good reason to understanding the nature of the failure and the time taken to resolve. Moving the forums offsite will NOT necessarily solve the problem.

Just to be clear moving the servers to another site does not necessarily remove the reliance on the comcast network. nawcc.org (67.225.191.105) is located at ezsolutions.com and if you trace the routing this is what you find ...



The reason for the slow performance was due to a failure in one specific part of the comcast network. If the failure had been elsewhere in their network then nawcc.org could also have been impacted and may well have been so in the past. This is the reason why nawc needs to understand the circumstances relating to the recent failure and the time delay in resolving the problem. Questions need to be asked and answers given. This is the way that lessons will be learnt and problems avoided in the future.

If relocation is being considered, I strongly advise that someone understands precisely the nature of the network, its resilience and the service level the providers are willing to guarantee before reaching a decision.

John

 

[Bill Stuntz]

The MB is very slow again today. More Comcast problems?

 

[Kevin W.]

Yes, here we go again, i gave up on a thread to read it, as its taking a long time for the pictures to load.

 

[Dave Coatsworth]

Thanks, and I do apologize for this. We are aware of the problem. We are working to address it (both short and long term).

 

[Kevin W.]

Thanks Dave, we all appreciate your help.

 

[Bill Stuntz]

And, of course, it happens on a weekend, when nobody is around to actually deal with it.

 

[John Matthews]

The forum slow response is a the cumulative impact of degradation across the Comcast network in the Ashburn and Lancaster areas in addition to errors on the local links ...



Fewer problems on the route to nawcc.org, but also suffering from some regional problems ...



John

 

[Dick C]

Is Comcast throttling traffic on purpose because of network inadequacies?

 

[Tom McIntyre]

Just my opinion, but there is little of value for us in working on ComCast problems. The obvious answer is to get rid of them as soon as we can and move to a professionally managed site.

 

[bruce linde]

Just my opinion, but there is little of value for us in working on ComCast problems. The obvious answer is to get rid of them as soon as we can and move to a professionally managed site.

if 'move to a professionally hosted site' means 'move the servers/sites to somewhere where comcast is not between our content/functionality and our users', sure.

 

[Jerry Treiman]

It is hardly worth visiting the site these days since the images take so long to load (if ever).
--- extremely frustrating
I fear we are going to lose a lot of our audience.

 

[John Matthews]

Anyone interested in understanding more about the impact of network performance might like to read this, particularly the explanation of packet loss.

Network latency and packet loss effects on performance | Noction

The triumvirate of network performance metrics are network latency, packet loss and jitter...

www.noction.com

John

 

[Dick C]

Isn't it time to remove or modify the banner at the top of the page where it says that all is well?

 

[Dave Coatsworth]

Yes, it is. We were just working on that.

 

[Tom McIntyre]

The banner is dismissable, but it should probably also be deleted/hidden.

 

[zedric]

Not sure how long it will last, but the Forums are now back to a useable speed, for me at least

 

[Jim Haney]

It would be nice & common courtesy to post the results of the latest web site problems and what was done to correct it, and what is being done to eliminate it.

 

[Tom McIntyre]

Jim, you can try writing to ComCast and asking them how they fixed whatever they may have fixed. They do not tell us.

The real problem is not fixed. Our server is still located in Columbia.

 

[Jim Haney]

Jim, you can try writing to ComCast and asking them how they fixed whatever they may have fixed. They do not tell us.

The real problem is not fixed. Our server is still located in Columbia.

Tom,

What would you propose to fix the MB problem? You want to move the server and that would accomplish what?

It seems to me that Comcast is holding us at bay by not providing a representative who we can talk to directly or withhold payment until good service to restored.

I realize that the BOD is trying to run things because we have no ED in place, but I believe that the ED can runs things by phone or email from England, like all large corporations do.

In other words someone has to be in charge and they should be able to make the situation workable or plan a setup that will give good service.

Is Comcast the only game in town? What would moving the server do? These things need addressed and solved.

All good managers (ED's) thrive on identifying and solving problems.That is what the job is.

 

[Tom McIntyre]

In Columbia ComCast is the only game.
The forum has no need to be served from Columbia. The main NAWCC site www.nawcc.org is not in Columbia.

 

[Dave Coatsworth]

Hi Jim,
We are working diligently to resolve the forum performance issues. We have an indepenent analyst coming to HQ this week to examine everything about our computing environment. He will have the ability to dial in over the weekend if we experience another slowdown. We will then determine how to best proceed based on that analysis. This is a top priority. It's just not the type of problem that is going to get resolved overnight. I ask for everyone's patience for the next couple of weeks while we work this issue out. We will get there - and sooner rather than later.

I will post updates as they are available.

 

[Jim Haney]

Dave,
Thanks for the explanation. That is all anyone wants, is to know that the problems are being addressed and kept up to date on progress & failures..

 

[zedric]

Dave - looks like the analyst won’t have to wait for the weekend to see slow speeds - the gremlins are back already…

 

[gmorse]

Hi zedric,

Dave - looks like the analyst won’t have to wait for the weekend to see slow speeds - the gremlins are back already…

I don't know if anyone else has noticed it, but every evening recently at around 8pm GMT, the whole board has stopped responding for 5 to 10 minutes before going back to normal. Images and all content load normally all the time apart from this brief but annoying hiatus.

Regards,

Graham

 

[John Matthews]

Hi Graham

Packet loss on the Comcast network was impacting both nawcc.org (corporate site) and mb.nawcc.org (forums) between 20:00 & 21:00 GMT this evening. More errors at 21:00 GMT today.

Forums @ 21:00 GMT



Corporate @ 21:00 GMT (fibre link to EZSolutions where the server is located)



Latest traces show problems have been cleared ...

Forums @ 23:00 GMT



I can confirm that there were a similar problems impacting both servers at the same times on January 21, 20, 19, 18, 17, 16, 15, 14 .... gave up checking the logs at this point. There may be other times I only checked 14:00 & 15:00 EST.

As can be seen a local fibre link to the host server, does not protect you from area network degradation.

John

 

[Dave Coatsworth]

As can be seen a local fibre link to the host server, does not protect you from area network degradation.

Note that our dedicated fiber service is not yet installed. Installation is currently scheduled for March 7.

 

[John Matthews]

Dave I was referring to the link to EZSolutions that is fibre. When the degradation is outwith the local fibre link, it is no protection against slow response.

John