Help with server speeds please

philiporchard · May 9, 2020

Hi,

We've hit a snag, and really hoping someone here can help. Hopefully there will be some consensus about what the issue is!

We've been working on a new web application.

We've been running the UAT on a server with Rackspace (shared space with the team doing the dev)
Yesterday, we pushed the web app to the live server for a soft launch. It's a dedicated server with Fasthosts.

There is a noticeable discrepancy between the performance of the UAT and Live sites.

We updated the admin site with the main code disabled (our code doing almost nothing) and the Live server is 3x slower on average than UAT.

The stats below show the time taken to complete the same task on UAT and Live (see images with green bars and time in seconds highlighted). But our code is doing almost nothing in this test, so it implies that something in-between the browser and the web server seems to be slow.

We're not sure, But possibly this would point to something in the network or server infrastructure.

Comparingthe Rackspace server (on UAT) with the Fasthosts server (on Live) with the actual software the same in both cases we pulled the CPU and Memory details. On the face of it the live server should be as fast or faster than the UAT.

We're no expert on these stats but it's interesting that the 'Speed' is showing 1.17GHz on the Live server, even though the CPU is supposed to be 2.20GHz. Whereas on the UAT server, the 'Speed' is showing 2.88GHz, even though the CPU is supposed to be 2.40GHz.

I've pasted the screenshots of the CPU and Memory for UAT and Live servers below, along with the time to save a question on UAT and Live (3x slower on Live). It is interesting that the 'Speed' showing on Live (1.17GHz) is almost 3x slower than the 'Speed' showing on UAT (2.88GHz) so that tallies with what we are seeing.

Any help would be appreciated. Hopefully I will be buying one of you a drink soon to say thanks!

cyberpunk · May 9, 2020

What's the app doing during that time? Does it read from a database? Are those times consistent for every request? What language is the back-end coded in?

Hay · May 9, 2020

are you running a java app? Can you screenshot the processes tab to show whats using the resources?

philiporchard · May 9, 2020

Thanks for questions.

The above info was provided by the dev team, but they are not available now until next week.
(it was only a soft launch, so it's not time critical)

We know there is room to improve the software, I guess the main question is why the discrepency between the UAT server and live server running the same software.

And as a bit of background info, the task is just to do with saving a question that we've created for our new site (maths dotcouk) back to a database.

Cheers!

philiporchard · May 9, 2020

Does anything at all jump out as being an obvious issue here between the two servers?

Hopefully there are some Acorners who are hardware gurus!

seemly · May 9, 2020

Anything regarding server configuration will never truly be obvious and jump out.

I think the 2 questions asked above will need to be answered prior to any valid potential feedback. Well, failing you giving individuals direct access to snoop and poke around.

super-whois · May 9, 2020

If your http headers are correct:
Server Microsoft-IIS/8.5
X-AspNetMvc-Version 4.0
X-AspNet-Version 4.0.30319
X-Powered-By ASP.NET

You should upgrade urgently, mainstream support for Windows 2012 R2 Mainstream support ended on October 9, 2018, you're using software that isn't supported and has no security upgrades.

philiporchard · May 9, 2020

Thanks for the info.

I think we are using this:
https://www.lizengo.co.uk/microsoft/microsoft-sql-server-2016-standard

Although we might be talking about two different things!

DaveP · May 10, 2020

Too many possibilities to help diagnose such a problem in this manner. Even 8 seconds to write a record to a database is astonishingly slow. Is the web app doing anything else other than writing to the database during this request?

If it's database related it could be an issue with indexes. Though my guess is that it looks like there is something fundamentally flawed at the web application level.

Ryan Ewen · May 10, 2020

are both servers running windows 2012 R2? - or are Rackspace running 2016?

And also IIS? same version on both?

Ben Thomas · May 11, 2020

Move to AWS and do yourself a favour whilst you're at the beginning of this little venture. Fasthosts are terrible.

Fred Steven · May 11, 2020

8 Seconds on the UAT is way too high, is the database server on the same machine or on a different machine.?

philiporchard · May 11, 2020

Hi Ben,

Thanks for comments. Would be interested in hearing your experience with both Fasthosts and AWS.

Fred Steven · May 11, 2020

philiporchard said:
Hi Ben,

Thanks for comments. Would be interested in hearing your experience with both Fasthosts and AWS.

I use both AWS and Fasthosts. I have used fasthosts for years, even back when they were hit with the floods in 2007

I have never had an issue with fasthosts, a couple of times we have lost a disk in a raid array and its was fixed very quickly and the other benefit is the support is quite good and you can speak with real people without any hassle.

I don't think your issue is with fasthosts itself. Something within your app or server is causing the long delay before the request is returned to the browser, it should be in the milliseconds.

You need to find out why the server is also only running at 1.17ghz. I found this which could be a possible:

http://ilantz.com/2015/02/16/dont-forget-to-modify-your-windows-server-power-options/

philiporchard · May 11, 2020

Thanks Fred. That's really useful

Fred Steven · May 11, 2020

You can also run this query in MSSMS which will at least give you insight if there are any crazy long queries executing.

SELECT top 100
SUBSTRING(t.text, ( s.statement_start_offset / 2 ) + 1,
( ( CASE statement_end_offset
WHEN -1 THEN DATALENGTH(t.text)
ELSE s.statement_end_offset
END - s.statement_start_offset ) / 2 ) + 1)
AS statement_text,
text,
objtype,
cacheobjtype,
usecounts,
last_execution_time,
total_worker_time,
total_worker_time / execution_count AS [Avg CPU Time],
execution_count ,
qp.query_plan
FROM sys.dm_exec_query_stats AS s
inner join sys.dm_exec_cached_plans cp
on s.plan_handle = cp.plan_handle
CROSS APPLY sys.dm_exec_sql_text(s.sql_handle) AS t
CROSS APPLY sys.dm_exec_query_plan(s.plan_handle) qp
--where text like ''
order by total_worker_time desc, last_execution_time desc
--order by last_execution_time desc

Ben Thomas · May 11, 2020

philiporchard said:
Hi Ben,

Thanks for comments. Would be interested in hearing your experience with both Fasthosts and AWS.

I can't speak for anybody else, but my experience and other people that I know who have used dedi's from Fasthosts have had terrible experiences. I moved to AWS a long time ago and would not turn back. You have complete control over everything, easily scale up on demand and I've never encountered any performance issues. It's also relatively cheap if you are careful about what you do. Either way, there's nothing like it, it's like owning the metal yourself minus the ballache of maintaining the racks.

philiporchard · May 11, 2020

Fred Steven said:
I use both AWS and Fasthosts. I have used fasthosts for years, even back when they were hit with the floods in 2007

I have never had an issue with fasthosts, a couple of times we have lost a disk in a raid array and its was fixed very quickly and the other benefit is the support is quite good and you can speak with real people without any hassle.

I don't think your issue is with fasthosts itself. Something within your app or server is causing the long delay before the request is returned to the browser, it should be in the milliseconds.

You need to find out why the server is also only running at 1.17ghz. I found this which could be a possible:

http://ilantz.com/2015/02/16/dont-forget-to-modify-your-windows-server-power-options/

Thanks for this. Excellent. It's definitely showing a faster CPU now. Might be some other improvements still to make, but that was a step in the right direction.

philiporchard · May 11, 2020

Thanks for the help with this Fred.

We're also testing a new server with Fasthosts.

One of their better spec machines.

I think this is the processor: Intel Xeon Gold 6126 2.6ghz
198gb ram and 12 cores.

Fred Steven · May 12, 2020

philiporchard said:
Thanks for the help with this Fred.

We're also testing a new server with Fasthosts.

One of their better spec machines.

I think this is the processor: Intel Xeon Gold 6126 2.6ghz
198gb ram and 12 cores.

Hi, you need to fix what the issue is, throwing more CPU at a problem is not the best method normally

Just as an example, the image attached is a 4 core fasthost server which runs approx 120 CMS websites each with its own database and it never max's out the cpu.