High Memory Usage - Build 8496

Problem reported by Scarab - 4/12/2023 at 11:56 AM

Resolved

After upgrading to Build 8496 we experienced two immediate issues:

"Your system clock is more than 15 seconds off". Although w32tm has the correct time on the DC (with a -6s drift between polling), and members of the AD have the correct time according to w32tm SmarterMail was giving this warning on all members in the same AD. We finally figured out this is due to High Accuracy not being enabled by default in Windows Server as per https://learn.microsoft.com/en-us/windows-server/networking/windows-time-service/configuring-systems-for-high-accuracy. Enabling High Accuracy on the DC and all members in the AD and restarting w32tm service resolved the issue. Note: that this means the DC and all servers running SmarterMail in the AD must have the minimum requirement of Windows Server 2016 or higher. High Accuracy is not available for DCs or AD members running Windows Server 2012R2 or those running in a Virtual environment on a host other than Hyper-V (such as VMWare).
High Memory Usage. The most important issue is dramatically increased Memory usage. We are seeing an average of 24GB memory usage for SmarterMail across the board. This is not an issue for our primary SmarterMail server which has 128GB RAM. However, on our Incoming Gateways that only do inbound SMTP & Anti-Spam checks SmarterMail is using 100% Memory usage (even the secondary and tertiary Gateways which only see a whopping 30 messages per hour!). SmarterMail v8451 was never using more than 2-4GB on the Incoming Gateways (that's definitely a significant increase in memory usage between 8451 and 8496!). The minimum memory requirements for SmarterMail is definitely no longer 4GB. Any suggestions on how to decrease memory usage on our Incoming Gateways? (I'm going to try to decrease the size of the SECURITY > BLACKLISTS as we noticed @ build 7929 there was a memory leak in SmarterMail at precisely 7000 entries which we remedied by consolidating our Blacklists to be < 7000 entries. It may be that 6000 is now too high in build 8496. I will update if this resolves the issue on our Incoming Gateways.)

10 Replies

Reply to Thread

Zach Sylvester Replied

4/12/2023 at 12:03 PM

Employee Post

Hello,

Thank you for reporting this issue. I'm going to open a ticket for the High memory usage but I want to ask you a couple of questions before opening one for the system clock issue.

If you go to https://time.is what does it say?
If you do this command in the CMD does it fix the issue?
```
w32tm /resync
```
If you go to Settings->Diagnostic what time does it show? And what timezone are you in?

Looking forward to hearing from you.

Thanks,

Zach Sylvester

Software Developer
SmarterTools Inc.

www.smartertools.com

Scarab Replied

4/12/2023 at 12:18 PM

Zach,

We are PDT (UTC -8 Pacific Time)

SmarterMail Self-Diagnostic shows the following:

After a resync of the w32tm service we show the following on the DC and the SmarterMail AD member:

Domain Controller:
w32tm /query /status
Leap Indicator: 0(no warning)
Stratum: 3 (secondary reference - syncd by (S)NTP)
Precision: -6 (15.625ms per tick)
Root Delay: 0.0621185s
Root Dispersion: 7.7767087s
ReferenceId: 0x9BF8C41C (source IP: 155.248.196.28)
Last Successful Sync Time: 4/12/2023 10:44:46 AM
Source: 2.us.pool.ntp.org
Poll Interval: 6 (64s)

SmarterMail Server:
w32tm /query /status
Leap Indicator: 0(no warning)
Stratum: 4 (secondary reference - syncd by (S)NTP)
Precision: -23 (119.209ns per tick)
Root Delay: 0.0636210s
Root Dispersion: 7.8205975s
ReferenceId: 0xC0A80103 (source IP: 192.168.1.3)
Last Successful Sync Time: 4/12/2023 10:55:33 AM
Source: dc.scarabmedia.com
Poll Interval: 6 (64s)

We were able to resolve the problem by enabling High Accuracy on the DC & all AD members running SmarterMail, although in the case of one of our Incoming Gateways we had to migrate the Windows 2016 Server from a VMWare Host to a Hyper-V Host that had High Accuracy enabled in the Windows registry.

The site https://time.is shows the following (but this is after enabling High Accuracy and moving from VMWare Host to Hyper-V with High Accuracy enabled):

Domain Controller
Your time is exact! The difference from Time.is was -0.002 seconds (±0.040 seconds)

SmarterMail Server
Your time is exact! The difference from Time.is was +0.014 seconds (±0.026 seconds)

Scarab Replied

4/12/2023 at 12:49 PM

Zach,

We just noticed that the Incoming Gateway we migrated from VMWare to Hyper-V to resolve the Time issue has SmarterMail showing a max of 1.2GB Memory usage (it routinely recycles upon trash collection and drops down to @ 700MB and works it way back up to @ 1.2, rinse and repeat). Since this looks good we restarted the other Incoming Gateways. Although ClamAV - clamd is using a larger amount of RAM on one of them (@2GB) the SmarterMail service itself is showing consistently at @ 1.2GB across all three Incoming Gateways after a system reboot. So, it may have been a sub-process running post-installation/conversion that was causing a memory leak.

After decades with SmarterMail I should know by now that if there is a problem post-upgrade I should reboot the SmarterMail server first and foremost to troubleshoot.

Zach Sylvester Replied

4/12/2023 at 1:14 PM

Employee Post

Hello,

Thanks for the updates.

So, the memory issue was resolved by restarting the system. The time issue has been fixed by enabling high accuracy. I'm happy to hear that worked for you!

I suggest that you keep an eye on ClamAV and check if it continues to use a lot of memory. In some cases, ClamAV has been known to consume up to 2.5GB of memory, but it should go down after some time.

Next week, I'll follow up with you on the ticket to check if everything is going well and if you're experiencing any more problems. If everything is working fine, we can close the ticket.

Thank you,

Zach Sylvester

Software Developer
SmarterTools Inc.

www.smartertools.com

Brian Replied

4/12/2023 at 1:29 PM

Ours doubled from 4GB to 8GB and slowly rising.... nothing critical though.

Webio Replied

4/13/2023 at 2:05 AM

I'm using SM build 8454 (but I've noticed this earlier) as only Incoming Gateways and all of them (3) use: 11GB, 10GB and 20GB or RAM.

Spool levels are at level about 1400, 1400 and 400 messages in Waiting to deliver sections so this are not a big numbers I think. I use incoming gateways only for SPAM filtering (rspamd and other built in functions) and I don't use ClamAV on them. Also I can't update to later builds since SPAM headers being added by latest build are encoded which is being decoded on destination server but I don't want to update main server yet.

RAM usage which I provided at the beggining is from task manager and here you have stats from SM dashboards of Incoming Gateways

EDIT: I don't use AD and systems are virtualised on VMWare and higher memory usage I've noticed when I started using preview versions of latest SM release since I wanted to integrated rspamd solution for spam checking.

EDIT2: Memory usage is something which returns after some time of service uptime. It returns after system restart or service restart after some period of time so this is not something which occured one time.

Seph Parshall Replied

5/16/2023 at 7:08 AM

Just had to restart server because RAM was maxxed at 99%. I upgraded weeks ago so I know nothing is being parsed still because of an upgrade. I've never had memory issues with SmarterMail until recently. I'm currently on v8524 because there's too many problems with the latest version.

I also am getting an occasional warning about time not being synced.

Brian Replied

5/16/2023 at 9:08 AM

Hi Seph. Can you check if the clock is synchronizing with the host via vmtools??
If it does, turn it off.

Seph Parshall Replied

5/16/2023 at 9:25 PM

My server is not a VM. It's a single, dedicated, hardware host. My RAM is already at 89% and I restarted service this morning.

Gabriele Maoret - SERSIS Replied

5/22/2023 at 4:52 AM

Using SM custom build 8536 for several days adn it seems that work better.
Today I have upgraded to official 8538 and I had no RAM nor CPU spikes till now

Gabriele Maoret - Head of SysAdmins and CISO at SERSIS

Currently manages 7 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 6 on-premise for customers who prefer to have their mail server in-house)

Back to Community Threads

Reply to Thread

Enter the verification text