• Member Voting Now through June 6. Check Your Email for a Link to the Online Ballot. The Ballot Contains Links to Each Proposed Amendment to Bylaws and Articles of Incorporation.

Forum is extremely slow at the moment.

berntd

Registered User
Jun 21, 2009
1,111
14
38
Sydney
Country
It takes about 1 minute every time I click on anything on the forum.
Eventually the screen will come up with the heading, background gears and mechanism but no contents. That will take another xxs before it comes up.


Regards
Bernt
 

zedric

NAWCC Member
Aug 8, 2012
1,955
404
83
Country
Region
Looks like it is working much better now...
 
  • Like
Reactions: berntd

John Matthews

NAWCC Member
Sep 22, 2015
3,700
1,823
113
France
Country
Region
Can someone explain what caused the problem and how it was resolved?

John
 

John Matthews

NAWCC Member
Sep 22, 2015
3,700
1,823
113
France
Country
Region
Dave - thank you for your response.

However, if that is the case can someone explain why the loading of www.nawcc.org hosted elements was much quicker than www.mb.nawcc.org - the forum was slower than the main site and the loading of text was much faster than images. If these elements are stored on different servers, those servers would have to be linked by Comcast infrastructure elements for that to be the explanation. What I experienced didn't appear to be due to a network area problem.

I would love to see a diagram showing how all the NAWCC IT infrastructure is configured - does such a diagram exist?

John
 

John Matthews

NAWCC Member
Sep 22, 2015
3,700
1,823
113
France
Country
Region
Upload speeds were equally bad from all servers. I suspect you were looking at content that was cached on www.nawcc.org since it does not change that often. Thus, it only appeared to be faster.
Dave Coatsworth , Tom McIntyre Leroy Baker

Just to clarify to those of you who believe this is due to cached content - it is not.

The reason for the difference in performance is because the routing is different between the nodes nawcc.org (ezsolutions) and mb/doc/net.nawcc.org

End of the routing to nawcc.org

1637658032544.png

No loss of packets addressed to both nodes nor loss of packets on the link between

Compare this with mb.nawcc.org

1637658210905.png

This clearly shows that the route is different and there are packets being lost on some of the links. Dropping 15% of the packets between 162.151.69.158 & 174.54.64.96 - is a severe network problem, which will seriously disrupt/fail the transfer of images and pdf files. There is also an issue between 174.54.64.96 and 50.244.233.125. None of the nodes are dropping packets directly addressed to them.

The routes diverge earlier in the comcast network. For nawcc.org

1637659137392.png

For the mb nodes etc

1637659222193.png


Given I have done the traces in the last 30 minutes after downloading a pdf file failed, I suspect when America awakes the situation will not improve.

I assume someone is already aware that these are the links that are causing the problems. I respectfully suggest we deserve a statement explaining in detail the nature of the problem, the actions being taken and how it will be fixed.

John
 

John Matthews

NAWCC Member
Sep 22, 2015
3,700
1,823
113
France
Country
Region
Update at 12:22 CET - just got worse as expected.

1637666560425.png

To demonstrate the impact ...

1637666624075.png

or to make it more obvious, compare

1637666976985.png

with

1637667005537.png

John
 
Last edited:

novicetimekeeper

Registered User
Jul 26, 2015
11,791
1,174
113
Dorset
Country
Region
I have tried to contribute on a few threads but the inability to get the larger pictures means I can't really answer questions.
 

Walt Wallgren

NAWCC Member
Aug 16, 2012
1,223
34
48
Bay Area, California
Country
Region
I am curious why nothing is being done. I log in almost every morning and for more than a week, the download speed is terrible. Somebody said the upload speed was causing a problem with the download speed. Huh:???: Maybe John Matthews needs to talk to our ISP to get some answers.

The ideal time (according to internet marketers) for a website to load is 3 seconds. Our page-load speeds are a joke. Because this is a MB and we are wanting to see what is new, we will wait longer. BUT somebody who doesn't know about us and just finds us on google or elsewhere, if they have to wait more than 10 seconds for a page to load, they are all too often gone. We have lost a potential member but even more importantly, we have failed in our mission to provide education and information to the community.

This needs to be fixed or I will be gone for a while until it is.
 

Dave Coatsworth

Senior Administrator
Staff member
NAWCC Business
NAWCC Fellow
Sponsor
Feb 11, 2005
8,705
3,431
113
62
Camarillo, CA
www.daveswatchparts.com
Country
Region
Hello everyone,
I have been on the phone with Jim Price today. He has been in constant contact with the IT group in Columbia who, in turn, have been in contact with Comcast, as that is where we believe the problem lies. They will be sending a technician out tomorrow between 10am and 12pm to investigate the problem. I will post an update as soon as we know anything further.

In the meantime, we are also working on a proposal to move us to a 24/7 hosting environment. That's not going to happen overnight, but we are treating it with the highest priority.
 
  • Like
Reactions: Jim Haney

John Matthews

NAWCC Member
Sep 22, 2015
3,700
1,823
113
France
Country
Region
Dave

The problem resides in the link with the errors and/or the devices at either end of the link. If all of these are Comcast's responsibility, then it is their problem. If not then it may not be.

I would expect a commercial network provider to have resilience in their network, ie multiple routes. It would be worth asking the question - is alternative routing possible? Unfortunately if the defective link & devices are local, there may be no alternatives.

In that situation you need to swap out the end devices and check the integrity of the link.

John
 

John Matthews

NAWCC Member
Sep 22, 2015
3,700
1,823
113
France
Country
Region
So we appear to be back to something like normal. Someone has fixed the link (just 1% loss) and/or devices that were losing packets. It would nice to know who manages this part of the system (I assume Comcast), the nature of the fault and what was done to fix it.

1637709396110.png

John
 

Dave T

NAWCC Member
Dec 8, 2011
4,031
326
83
NC
Country
Region
Yep, looks like we're back on track today! Thanks for all who have worked to correct this problem.
 

Bill Stuntz

Registered User
Apr 6, 2012
4,958
45
48
73
Columbus. OH
Country
Region
It's nice to be able to use the MB again. Like John, I'd like to know what the problem was, and how it was fixed. If it was Comcast's problem, is there any compensation for the fact the system was essentially unusable?
 

Dave Coatsworth

Senior Administrator
Staff member
NAWCC Business
NAWCC Fellow
Sponsor
Feb 11, 2005
8,705
3,431
113
62
Camarillo, CA
www.daveswatchparts.com
Country
Region
Comcast was at HQ this morning. The tech said the problem was indeed theirs and was resolved last night. (Consistent with our experience.) Most of Columbia was affected. Unfortunately, since it was their issue, there's not much data available on what was fixed.

This is not the end of it. We will continue to pursue alternatives that will give us more reliable 24/7 service.
 
  • Like
Reactions: Jim Haney

John Matthews

NAWCC Member
Sep 22, 2015
3,700
1,823
113
France
Country
Region
With all due respect there is. The problem was the link or the end nodes. They will know and they should be willing to share information to explain why it took so long to diagnose and repair.

John
 

Bill Stuntz

Registered User
Apr 6, 2012
4,958
45
48
73
Columbus. OH
Country
Region
Does the fact that Camcast's service made our web sites essentially unusable entitle the NAWCC to compensation? Discounted service, rebates, service credit, etc?
 

Tom McIntyre

Technical Admin
Staff member
NAWCC Star Fellow
NAWCC Ruby Member
Sponsor
Golden Circle
Aug 24, 2000
84,726
2,358
113
85
Boston
awco.org
Country
Region
I agree with Dave. There is very little to be gained by pursuing this with ComCast. If we move the forums off site, it will solve the problem. We cannot move our telephone system off site and probably not the iMIS System or the MSSQL Server, or the Exchange Server all of which are critical to local operations.

We may be entitled to some compensation if we have a Service Level Agreement as we should.
 

John Matthews

NAWCC Member
Sep 22, 2015
3,700
1,823
113
France
Country
Region
There is very little to be gained by pursuing this with ComCast. If we move the forums off site, it will solve the problem.
I disagree. There is very good reason to understanding the nature of the failure and the time taken to resolve. Moving the forums offsite will NOT necessarily solve the problem.

Just to be clear moving the servers to another site does not necessarily remove the reliance on the comcast network. nawcc.org (67.225.191.105) is located at ezsolutions.com and if you trace the routing this is what you find ...

1637793119111.png

The reason for the slow performance was due to a failure in one specific part of the comcast network. If the failure had been elsewhere in their network then nawcc.org could also have been impacted and may well have been so in the past. This is the reason why nawc needs to understand the circumstances relating to the recent failure and the time delay in resolving the problem. Questions need to be asked and answers given. This is the way that lessons will be learnt and problems avoided in the future.

If relocation is being considered, I strongly advise that someone understands precisely the nature of the network, its resilience and the service level the providers are willing to guarantee before reaching a decision.

John
 
Last edited:

Bill Stuntz

Registered User
Apr 6, 2012
4,958
45
48
73
Columbus. OH
Country
Region
The MB is very slow again today. More Comcast problems?
 

Kevin W.

NAWCC Member
Apr 11, 2002
23,443
644
113
64
Nepean, Ontario, Canada
Country
Region
Yes, here we go again, i gave up on a thread to read it, as its taking a long time for the pictures to load.
 

Kevin W.

NAWCC Member
Apr 11, 2002
23,443
644
113
64
Nepean, Ontario, Canada
Country
Region
Thanks Dave, we all appreciate your help.
 

Bill Stuntz

Registered User
Apr 6, 2012
4,958
45
48
73
Columbus. OH
Country
Region
And, of course, it happens on a weekend, when nobody is around to actually deal with it.
 

John Matthews

NAWCC Member
Sep 22, 2015
3,700
1,823
113
France
Country
Region
The forum slow response is a the cumulative impact of degradation across the Comcast network in the Ashburn and Lancaster areas in addition to errors on the local links ...

1638659026217.png

Fewer problems on the route to nawcc.org, but also suffering from some regional problems ...

1638659422834.png

John
 
  • Like
Reactions: musicguy

Dick C

Registered User
Oct 14, 2009
2,037
132
63
Country
Is Comcast throttling traffic on purpose because of network inadequacies?
 

Tom McIntyre

Technical Admin
Staff member
NAWCC Star Fellow
NAWCC Ruby Member
Sponsor
Golden Circle
Aug 24, 2000
84,726
2,358
113
85
Boston
awco.org
Country
Region
Just my opinion, but there is little of value for us in working on ComCast problems. The obvious answer is to get rid of them as soon as we can and move to a professionally managed site.
 

bruce linde

NAWCC Member
Donor
Nov 13, 2011
10,077
1,866
113
oakland, ca.
clockhappy.com
Country
Region
Just my opinion, but there is little of value for us in working on ComCast problems. The obvious answer is to get rid of them as soon as we can and move to a professionally managed site.

if 'move to a professionally hosted site' means 'move the servers/sites to somewhere where comcast is not between our content/functionality and our users', sure.
 

Jerry Treiman

NAWCC Member
Golden Circle
Aug 25, 2000
7,098
4,082
113
Los Angeles, CA
Country
Region
It is hardly worth visiting the site these days since the images take so long to load (if ever).
--- extremely frustrating
I fear we are going to lose a lot of our audience.
 
  • Like
Reactions: Kevin W.

John Matthews

NAWCC Member
Sep 22, 2015
3,700
1,823
113
France
Country
Region
Anyone interested in understanding more about the impact of network performance might like to read this, particularly the explanation of packet loss.


John
 

Dick C

Registered User
Oct 14, 2009
2,037
132
63
Country
Isn't it time to remove or modify the banner at the top of the page where it says that all is well?
 

Tom McIntyre

Technical Admin
Staff member
NAWCC Star Fellow
NAWCC Ruby Member
Sponsor
Golden Circle
Aug 24, 2000
84,726
2,358
113
85
Boston
awco.org
Country
Region
The banner is dismissable, but it should probably also be deleted/hidden.
 

zedric

NAWCC Member
Aug 8, 2012
1,955
404
83
Country
Region
Not sure how long it will last, but the Forums are now back to a useable speed, for me at least
 

Jim Haney

NAWCC Member
Sponsor
Donor
Sep 21, 2002
7,277
2,317
113
72
Decatur, TN.
Country
Region
It would be nice & common courtesy to post the results of the latest web site problems and what was done to correct it, and what is being done to eliminate it.
 

Tom McIntyre

Technical Admin
Staff member
NAWCC Star Fellow
NAWCC Ruby Member
Sponsor
Golden Circle
Aug 24, 2000
84,726
2,358
113
85
Boston
awco.org
Country
Region
Jim, you can try writing to ComCast and asking them how they fixed whatever they may have fixed. They do not tell us.

The real problem is not fixed. Our server is still located in Columbia.
 

Jim Haney

NAWCC Member
Sponsor
Donor
Sep 21, 2002
7,277
2,317
113
72
Decatur, TN.
Country
Region
Jim, you can try writing to ComCast and asking them how they fixed whatever they may have fixed. They do not tell us.

The real problem is not fixed. Our server is still located in Columbia.

Tom,

What would you propose to fix the MB problem? You want to move the server and that would accomplish what?

It seems to me that Comcast is holding us at bay by not providing a representative who we can talk to directly or withhold payment until good service to restored.

I realize that the BOD is trying to run things because we have no ED in place, but I believe that the ED can runs things by phone or email from England, like all large corporations do.

In other words someone has to be in charge and they should be able to make the situation workable or plan a setup that will give good service.

Is Comcast the only game in town? What would moving the server do? These things need addressed and solved.

All good managers (ED's) thrive on identifying and solving problems.That is what the job is.
 

Tom McIntyre

Technical Admin
Staff member
NAWCC Star Fellow
NAWCC Ruby Member
Sponsor
Golden Circle
Aug 24, 2000
84,726
2,358
113
85
Boston
awco.org
Country
Region
In Columbia ComCast is the only game.
The forum has no need to be served from Columbia. The main NAWCC site www.nawcc.org is not in Columbia.
 

Dave Coatsworth

Senior Administrator
Staff member
NAWCC Business
NAWCC Fellow
Sponsor
Feb 11, 2005
8,705
3,431
113
62
Camarillo, CA
www.daveswatchparts.com
Country
Region
Hi Jim,
We are working diligently to resolve the forum performance issues. We have an indepenent analyst coming to HQ this week to examine everything about our computing environment. He will have the ability to dial in over the weekend if we experience another slowdown. We will then determine how to best proceed based on that analysis. This is a top priority. It's just not the type of problem that is going to get resolved overnight. I ask for everyone's patience for the next couple of weeks while we work this issue out. We will get there - and sooner rather than later.

I will post updates as they are available.
 
  • Like
Reactions: Jim Haney

Jim Haney

NAWCC Member
Sponsor
Donor
Sep 21, 2002
7,277
2,317
113
72
Decatur, TN.
Country
Region
Dave,
Thanks for the explanation. That is all anyone wants, is to know that the problems are being addressed and kept up to date on progress & failures..;)
 

zedric

NAWCC Member
Aug 8, 2012
1,955
404
83
Country
Region
Dave - looks like the analyst won’t have to wait for the weekend to see slow speeds - the gremlins are back already…
 
  • Like
Reactions: Jerry Treiman

gmorse

NAWCC Member
Jan 7, 2011
13,705
2,875
113
Breamore, Hampshire, UK
Country
Region
Hi zedric,
Dave - looks like the analyst won’t have to wait for the weekend to see slow speeds - the gremlins are back already…
I don't know if anyone else has noticed it, but every evening recently at around 8pm GMT, the whole board has stopped responding for 5 to 10 minutes before going back to normal. Images and all content load normally all the time apart from this brief but annoying hiatus.

Regards,

Graham
 

John Matthews

NAWCC Member
Sep 22, 2015
3,700
1,823
113
France
Country
Region
Hi Graham

Packet loss on the Comcast network was impacting both nawcc.org (corporate site) and mb.nawcc.org (forums) between 20:00 & 21:00 GMT this evening. More errors at 21:00 GMT today.

Forums @ 21:00 GMT

1642891920518.png

Corporate @ 21:00 GMT (fibre link to EZSolutions where the server is located)

1642892083937.png

Latest traces show problems have been cleared ...

Forums @ 23:00 GMT

1642892293977.png

I can confirm that there were a similar problems impacting both servers at the same times on January 21, 20, 19, 18, 17, 16, 15, 14 .... gave up checking the logs at this point. There may be other times I only checked 14:00 & 15:00 EST.

As can be seen a local fibre link to the host server, does not protect you from area network degradation.

John
 

Dave Coatsworth

Senior Administrator
Staff member
NAWCC Business
NAWCC Fellow
Sponsor
Feb 11, 2005
8,705
3,431
113
62
Camarillo, CA
www.daveswatchparts.com
Country
Region
As can be seen a local fibre link to the host server, does not protect you from area network degradation.
Note that our dedicated fiber service is not yet installed. Installation is currently scheduled for March 7.
 

John Matthews

NAWCC Member
Sep 22, 2015
3,700
1,823
113
France
Country
Region
Dave I was referring to the link to EZSolutions that is fibre. When the degradation is outwith the local fibre link, it is no protection against slow response.

John
 

Forum statistics

Threads
173,631
Messages
1,516,175
Members
51,848
Latest member
Dutch Calibre
Encyclopedia Pages
1,062
Total wiki contributions
2,969
Last update
-