Enjoy unlimited access to all forum features for FREE! Optional upgrade available for extra perks.

Help with server speeds please

Discussion in 'Services Wanted' started by philiporchard, May 9, 2020.

  1. philiporchard United Kingdom

    philiporchard Well-Known Member

    Joined:
    May 2007
    Posts:
    1,994
    Likes Received:
    156
    Hi,

    We've hit a snag, and really hoping someone here can help. Hopefully there will be some consensus about what the issue is!

    We've been working on a new web application.

    We've been running the UAT on a server with Rackspace (shared space with the team doing the dev)
    Yesterday, we pushed the web app to the live server for a soft launch. It's a dedicated server with Fasthosts.

    There is a noticeable discrepancy between the performance of the UAT and Live sites.

    We updated the admin site with the main code disabled (our code doing almost nothing) and the Live server is 3x slower on average than UAT.

    The stats below show the time taken to complete the same task on UAT and Live (see images with green bars and time in seconds highlighted). But our code is doing almost nothing in this test, so it implies that something in-between the browser and the web server seems to be slow.

    We're not sure, But possibly this would point to something in the network or server infrastructure.

    Comparingthe Rackspace server (on UAT) with the Fasthosts server (on Live) with the actual software the same in both cases we pulled the CPU and Memory details. On the face of it the live server should be as fast or faster than the UAT.

    We're no expert on these stats but it's interesting that the 'Speed' is showing 1.17GHz on the Live server, even though the CPU is supposed to be 2.20GHz. Whereas on the UAT server, the 'Speed' is showing 2.88GHz, even though the CPU is supposed to be 2.40GHz.

    I've pasted the screenshots of the CPU and Memory for UAT and Live servers below, along with the time to save a question on UAT and Live (3x slower on Live). It is interesting that the 'Speed' showing on Live (1.17GHz) is almost 3x slower than the 'Speed' showing on UAT (2.88GHz) so that tallies with what we are seeing.

    Any help would be appreciated. Hopefully I will be buying one of you a drink soon to say thanks!
     

    Attached Files:

  2. Domain Forum

    Acorn Domains Elite Member

    Joined:
    1999
    Messages:
    Many
    Likes Received:
    Lots
    IWA Meetup
     
  3. cyberpunk United Kingdom

    cyberpunk Active Member

    Joined:
    Jul 2019
    Posts:
    151
    Likes Received:
    28
    What's the app doing during that time? Does it read from a database? Are those times consistent for every request? What language is the back-end coded in?
     
  4. Hay

    Hay Active Member

    Joined:
    Jul 2019
    Posts:
    384
    Likes Received:
    97
    are you running a java app? Can you screenshot the processes tab to show whats using the resources?
     
  5. philiporchard United Kingdom

    philiporchard Well-Known Member

    Joined:
    May 2007
    Posts:
    1,994
    Likes Received:
    156
    Thanks for questions.

    The above info was provided by the dev team, but they are not available now until next week.
    (it was only a soft launch, so it's not time critical)

    We know there is room to improve the software, I guess the main question is why the discrepency between the UAT server and live server running the same software.

    And as a bit of background info, the task is just to do with saving a question that we've created for our new site (maths dotcouk) back to a database.

    Cheers!
     
  6. philiporchard United Kingdom

    philiporchard Well-Known Member

    Joined:
    May 2007
    Posts:
    1,994
    Likes Received:
    156
    Does anything at all jump out as being an obvious issue here between the two servers?

    Hopefully there are some Acorners who are hardware gurus!
     
  7. seemly

    seemly Well-Known Member

    Joined:
    Feb 2011
    Posts:
    1,607
    Likes Received:
    493
    Anything regarding server configuration will never truly be obvious and jump out.

    I think the 2 questions asked above will need to be answered prior to any valid potential feedback. Well, failing you giving individuals direct access to snoop and poke around.
     
  8. super-whois

    super-whois Active Member

    Joined:
    Oct 2008
    Posts:
    347
    Likes Received:
    86
    If your http headers are correct:
    Server Microsoft-IIS/8.5
    X-AspNetMvc-Version 4.0
    X-AspNet-Version 4.0.30319
    X-Powered-By ASP.NET

    You should upgrade urgently, mainstream support for Windows 2012 R2 Mainstream support ended on October 9, 2018, you're using software that isn't supported and has no security upgrades.
     
  9. philiporchard United Kingdom

    philiporchard Well-Known Member

    Joined:
    May 2007
    Posts:
    1,994
    Likes Received:
    156
  10. DaveP United Kingdom

    DaveP Well-Known Member

    Joined:
    Mar 2010
    Posts:
    2,941
    Likes Received:
    56
    Too many possibilities to help diagnose such a problem in this manner. Even 8 seconds to write a record to a database is astonishingly slow. Is the web app doing anything else other than writing to the database during this request?

    If it's database related it could be an issue with indexes. Though my guess is that it looks like there is something fundamentally flawed at the web application level.
     
  11. Ryan Ewen

    Ryan Ewen Active Member

    Joined:
    Oct 2016
    Posts:
    868
    Likes Received:
    256
    are both servers running windows 2012 R2? - or are Rackspace running 2016?

    And also IIS? same version on both?
     
  12. Ben Thomas

    Ben Thomas Well-Known Member

    Joined:
    Mar 2018
    Posts:
    2,621
    Likes Received:
    363
    Move to AWS and do yourself a favour whilst you're at the beginning of this little venture. Fasthosts are terrible.
     
    • Like Like x 1
  13. Fred Steven Cyprus

    Fred Steven Member

    Joined:
    Jan 2020
    Posts:
    34
    Likes Received:
    8
    8 Seconds on the UAT is way too high, is the database server on the same machine or on a different machine.?
     
  14. philiporchard United Kingdom

    philiporchard Well-Known Member

    Joined:
    May 2007
    Posts:
    1,994
    Likes Received:
    156
    Hi Ben,

    Thanks for comments. Would be interested in hearing your experience with both Fasthosts and AWS.
     
  15. Fred Steven Cyprus

    Fred Steven Member

    Joined:
    Jan 2020
    Posts:
    34
    Likes Received:
    8
    I use both AWS and Fasthosts. I have used fasthosts for years, even back when they were hit with the floods in 2007 :)

    I have never had an issue with fasthosts, a couple of times we have lost a disk in a raid array and its was fixed very quickly and the other benefit is the support is quite good and you can speak with real people without any hassle.

    I don't think your issue is with fasthosts itself. Something within your app or server is causing the long delay before the request is returned to the browser, it should be in the milliseconds.

    You need to find out why the server is also only running at 1.17ghz. I found this which could be a possible:

    http://ilantz.com/2015/02/16/dont-forget-to-modify-your-windows-server-power-options/
     
  16. philiporchard United Kingdom

    philiporchard Well-Known Member

    Joined:
    May 2007
    Posts:
    1,994
    Likes Received:
    156
    Thanks Fred. That's really useful
     
  17. Fred Steven Cyprus

    Fred Steven Member

    Joined:
    Jan 2020
    Posts:
    34
    Likes Received:
    8
    You can also run this query in MSSMS which will at least give you insight if there are any crazy long queries executing.

    SELECT top 100
    SUBSTRING(t.text, ( s.statement_start_offset / 2 ) + 1,
    ( ( CASE statement_end_offset
    WHEN -1 THEN DATALENGTH(t.text)
    ELSE s.statement_end_offset
    END - s.statement_start_offset ) / 2 ) + 1)
    AS statement_text,
    text,
    objtype,
    cacheobjtype,
    usecounts,
    last_execution_time,
    total_worker_time,
    total_worker_time / execution_count AS [Avg CPU Time],
    execution_count ,
    qp.query_plan
    FROM sys.dm_exec_query_stats AS s
    inner join sys.dm_exec_cached_plans cp
    on s.plan_handle = cp.plan_handle
    CROSS APPLY sys.dm_exec_sql_text(s.sql_handle) AS t
    CROSS APPLY sys.dm_exec_query_plan(s.plan_handle) qp
    --where text like ''
    order by total_worker_time desc, last_execution_time desc
    --order by last_execution_time desc
     
  18. Ben Thomas

    Ben Thomas Well-Known Member

    Joined:
    Mar 2018
    Posts:
    2,621
    Likes Received:
    363
    I can't speak for anybody else, but my experience and other people that I know who have used dedi's from Fasthosts have had terrible experiences. I moved to AWS a long time ago and would not turn back. You have complete control over everything, easily scale up on demand and I've never encountered any performance issues. It's also relatively cheap if you are careful about what you do. Either way, there's nothing like it, it's like owning the metal yourself minus the ballache of maintaining the racks.
     
  19. philiporchard United Kingdom

    philiporchard Well-Known Member

    Joined:
    May 2007
    Posts:
    1,994
    Likes Received:
    156
    Thanks for this. Excellent. It's definitely showing a faster CPU now. Might be some other improvements still to make, but that was a step in the right direction.
     
  20. philiporchard United Kingdom

    philiporchard Well-Known Member

    Joined:
    May 2007
    Posts:
    1,994
    Likes Received:
    156
    Thanks for the help with this Fred.

    We're also testing a new server with Fasthosts.

    One of their better spec machines.

    I think this is the processor: Intel Xeon Gold 6126 2.6ghz
    198gb ram and 12 cores.
     
  21. Fred Steven Cyprus

    Fred Steven Member

    Joined:
    Jan 2020
    Posts:
    34
    Likes Received:
    8
    Hi, you need to fix what the issue is, throwing more CPU at a problem is not the best method normally:)

    Just as an example, the image attached is a 4 core fasthost server which runs approx 120 CMS websites each with its own database and it never max's out the cpu. fasthost.jpg