RAID1 Mirror Corruption on 2008 R2 Server with Intel RSTe Controller with Intel SSD Drives

If the problem goes away with 3.6 driver and 3.8 option ROM, then it is likely that you have found the same bug as us.

Under light load, this problem does not happen for a few days or weeks. But you really don’t want to leave this problem unaddressed, unless you like rebuilding RAID arrays - I have found that the novelty quickly wore off for our IT guy :slight_smile:

Please report this problem to [email protected].

There doesn’t appear to be any way to report this problem to Intel - they slopey-shoulder the problem onto SuperMicro.

They will probably ask you to do all sorts of tests. But make sure you don’t waste your time doing tests that have already been done and are documented here - we already know that no recent driver fixes the problem. Please point them to this thread and one day there might be enough documented cases of the problem for it to get prioritised to be fixed.

I reported it this morning.

Found a Way to install 3.6 drivers last night, I’m currently running stress test until next week, I’ll report back

So far so good, up to about 70 million transactions and no errors but the real test will be Friday evening when I run a parity verification.

Okay, 3.6 with hdds, 3.8 orom works fine, 13 trillion transactions in stress test and parity verification for raid 1 volume 2 hdds came back no parity or media errors. I will switch to 2 ssds tomorrow and try stress test again.

Okay,
I just wanted to point out some findings. I have swapped in a raid 1 volume made up of 2 ssds (intel 530 240gb) already initialized and verified from another machine, running on the backup server with 2008r2 and the 3.6 drivers from intel’s website, trim does not appear to be working. I have run “fsutil behavior query disabledeletenotify” and the response is “DisableDeleteNotify = 0” but when I check with the tool trimcheck, it shows that trim is not enabled. I have begun a stress test to see what if anything will happen after a few trillion test transactions. I will report back when the results are in.

I found another version of the driver that Intel still keeps up but ONLY for their server boards, 3.8.0.1111 available here
https://downloadcenter.intel.com/downloa…ver-for-Windows-
This version indeed supports trim on raid 1 and trimcheck tool confirms that indeed it works. I installed this version and have begun a stress test, this version also closely matches my sata orom. Will report back.

3.8.0.1111 IRSTe fails the parity test on verification after 4 days stress test (1280 parity errors). Put back hdds for now because I will be away. Will return and try a few more tests with different versions. Will report back in 2 weeks.

Yes, 3.6 does not support TRIM.
But 3.6 was the newest driver that worked.
My guess is that their implementation of TRIM is what has broken the driver.

Btw, we’ve only ever replicated the problem using SSDs. Perhaps that is because only SSDs use TRIM.

Best regards

Steve

I was starting to believe the very same thing, that trim being functional is what causes the problems. I have at least 15 systems setup using raid 1 or 10 (windows 8.1) and none have trim enabled and they work perfectly. I read somewhere that trim is not enabled on server 2012. Is that true?

If this theory should be valid, it has to be checked, whether the issue is caused by the TRIM command itself, which is sent by the OS, or by the individual system components, which are realizing the execution of the TRIM command (Intel SATA RAID Controller, Intel RSTe RAID BIOS modules and Intel RSTe RAID driver).
As you certainly know, it is very easy to disable the TRIM support of any OS from Win7 up completely by running the following command:
fsutil behavior set disabledeletenotify 1

Fernando, I have a question regarding your last comment.
Why is it that on Windows 8.1 systems that have raid 1 ssd volumes and I check
fsutil behavior query disabledeletenotify

I receive this message
DisableDeleteNotify = 0 = TRIM support enabled

but I know trim is actually disabled since I run trimcheck tool and see that deleted files are not erased, besides trim command in admin tools is disabled.

Something more to all this that I am not aware of?

@ billydv:

The OS message "DisableDeleteNotify = 0" certifies, that it actually sends TRIM commands into the direction of the connected SSDs.
With the TrimCheck tool you can verify, whether the TRIM commands really have passed through the on-board Intel SATA Controller and arrived within the SSDs resp.the RAID array.

TRIM acticity within a single or RAIDed SSD requires
a) an OS, which is able to send TRIM commands (all Windows OSes from Win7 up),
b) an OS with activated continuous execution of TRIM commands (message: "DisableDeleteNotify = 0"),
c) a SATA Controller, which lets the TRIM commands pass through (depends on the SATA Controller, the related BIOS modules and the SATA driver) and
d) an SSD Controller, which supports TRIM.

Got it,
So Windows may indeed say that Yes I have an SSD installed on my hardware so lets enable the trim command, whether or not it actually passes through the sata controller and ultimately to the raided ssds depends on the other factors.

Okay,
I have finally tested with a driver that works with raid 1 ssd volume and does not fall apart. I used the same stress test that messed up every other raid 1 mirror. As I see it the problem lies in the enabling of trim on Windows 7/Server 2008r2 based systems. The 3.6 driver available from the intel website is the only driver that will not destroy the raid 1 mirror upon being stressed. I will rerun the test later this week but I am quite confident that the issue is the enabling of trim for raid 1. I have countless machines setup running win 8.1 with 2 ssd raid 1 volumes (or 4 ssd raid 10 volumes), I have never seen a single parity error on any of them even when power is suddenly shut off.
I will report back after rerunning this test.
Thanks

Oh BTW Stephen,
Yes my findings are exactly the same as yours, only the 3.6 driver works if you are running server 2008r2.
I believe this matter is irrelevant in server 2012r2 because if I am not mistaken, trim is disabled for raid 1 (server 2012r2 or win 8)

I’m afraid I’m not able to comment usefully on that, as all our systems are 2008r2.

Hi,
I have the same problem with 3 servers running with Windows 2008R2.
Mainboard: Supermicro X9SCM-F (C204 PCH Chipset).

Using 2 RAID1 arrays (one for the system consisting of Samsung 830 and Intel 400i, one other for hosting the hyperv vm files consisting of Samsung 850 evo and Samsung 850 pro).
Speed is really good and I did not have any problems until now.
But one week before I just had the idea "Can be no bad idea to start a verify run" … and saw some repaired verification errors!
For this phenomenom, google seems not to be our friend because it took a long time to find this thread!

What about the following idea: all SSDs have some garbage collector processes, maybe the verification errors are a result of these?

Here is my report:
Betriebssystemname: Microsoft Windows Server 2008 R2 Standard
Betriebssystemversion: 6.1.7601 Service Pack 1 7601
Systemname: SOL-5
Systemhersteller: Supermicro
Systemmodell: X9SCL/X9SCM
Prozessor: GenuineIntel Intel64 Family 6 Model 42 Stepping 7 3,301 GHz
BIOS: American Megatrends Inc., 2.10

Intel® Rapid Storage-Technologie für Unternehmen Information
Installiertes Kit: 13.6.0.1002
Benutzeroberflächenversion: 13.6.0.1002
Sprache: Deutsch (Deutschland)
Version des RAID-Erweiterungs-ROMs: 11.6.0.1702
Treiberversion: 13.6.0.1002
ISDI-Version: 13.6.0.1002

Speichersysteminformationen
RAID-Konfiguration

Array-Name: SATA_Array_0000
Größe: 1.907.739 MB
Verfügbarer Speicherplatz: 186.700 MB
Anzahl der Volumes: 1
Volume-Mitglied: vm_V0
Anzahl der Array-Festplatten: 2
Array-Festplatte: S21DNWAFC15630J
Array-Festplatte: S2BBNEAG113442N
Festplatten-Daten-Cache: Aktiviert

Array-Name: SATA_Array_0001
Größe: 457.872 MB
Verfügbarer Speicherplatz: 27.792 MB
Anzahl der Volumes: 1
Volume-Mitglied: sys_V1
Anzahl der Array-Festplatten: 2
Array-Festplatte: S0VZNYAC203032
Array-Festplatte: CVCV205302JG240CGN
Festplatten-Daten-Cache: Aktiviert

Volume-Name: vm_V0
Status: Normal
Typ: RAID 1
Größe: 860.519 MB
System-Volume: Nein
Datenstreifengröße: 64 KB
Write-Back-Cache: Write-Back
Initialisiert: Ja
Paritätsfehler: 1
Blöcke mit Medienfehlern: 0
Größe des physischen Sektors: 512 Bytes
Größe des logischen Sektors: 512 Bytes

Volume-Name: sys_V1
Status: Normal
Typ: RAID 1
Größe: 215.040 MB
System-Volume: Ja
Datenstreifengröße: 64 KB
Write-Back-Cache: Write-Back
Initialisiert: Ja
Paritätsfehler: 0
Blöcke mit Medienfehlern: 0
Größe des physischen Sektors: 512 Bytes
Größe des logischen Sektors: 512 Bytes

Hardwareinformationen

Controller-Name: Intel(R) Desktop/Workstation/Server Express Chipset SATA RAID Controller
Typ: SATA
Modus: RAID
Anzahl der SATA-Anschlüsse: 6
Anzahl der Volumes: 2
Volume: vm_V0
Volume: sys_V1
Anzahl der Ersatzlaufwerke: 0
Anzahl verfügbarer Festplatten: 1
Neuaufbau nach Hot-Plug: Deaktiviert
Hersteller: 32902
Modellnummer: 10274
Produktüberarbeitung: 5
Direkt angeschlossene Festplatte: S21DNWAFC15630J
Direkt angeschlossene Festplatte: S2BBNEAG113442N
Direkt angeschlossene Festplatte: S0VZNYAC203032
Direkt angeschlossene Festplatte: CVCV205302JG240CGN
Direkt angeschlossene Festplatte: WD-WCASY7878803

Festplatte an Controller 0, Anschluss 0
Status: Normal
Typ: SATA SSD
Positionstyp: Intern
Verwendung: Array-Festplatte
Größe: 932 GB
Systemfestplatte: Nein
Festplatten-Daten-Cache: Aktiviert
Command Queuing: NCQ
Übertragungsrate: 6 Gb/s
Modell: Samsung SSD 850 EVO 1TB
Seriennummer: S21DNWAFC15630J
SCSI Geräte-ID: 0
Firmware: EMT01B6Q
Größe des physischen Sektors: 512 Bytes
Größe des logischen Sektors: 512 Bytes

Festplatte an Controller 0, Anschluss 1
Status: Normal
Typ: SATA SSD
Positionstyp: Intern
Verwendung: Array-Festplatte
Größe: 954 GB
Systemfestplatte: Nein
Festplatten-Daten-Cache: Aktiviert
Command Queuing: NCQ
Übertragungsrate: 6 Gb/s
Modell: Samsung SSD 850 PRO 1TB
Seriennummer: S2BBNEAG113442N
SCSI Geräte-ID: 1
Firmware: EXM02B6Q
Größe des physischen Sektors: 512 Bytes
Größe des logischen Sektors: 512 Bytes

Festplatte an Controller 0, Anschluss 2
Status: Normal
Typ: SATA SSD
Positionstyp: Intern
Verwendung: Array-Festplatte
Größe: 238 GB
Systemfestplatte: Nein
Festplatten-Daten-Cache: Aktiviert
Command Queuing: NCQ
Übertragungsrate: 3 Gb/s
Modell: SAMSUNG SSD 830 Series
Seriennummer: S0VZNYAC203032
SCSI Geräte-ID: 2
Firmware: CXM03B1Q
Größe des physischen Sektors: 512 Bytes
Größe des logischen Sektors: 512 Bytes

Festplatte an Controller 0, Anschluss 3
Status: Normal
Typ: SATA SSD
Positionstyp: Intern
Verwendung: Array-Festplatte
Größe: 224 GB
Systemfestplatte: Nein
Festplatten-Daten-Cache: Aktiviert
Command Queuing: NCQ
Übertragungsrate: 3 Gb/s
Modell: INTEL SSDSC2CW240A
Seriennummer: CVCV205302JG240CGN
SCSI Geräte-ID: 3
Firmware: 400i
Größe des physischen Sektors: 512 Bytes
Größe des logischen Sektors: 512 Bytes

Festplatte an Controller 0, Anschluss 4
Status: Normal
Typ: SATA Festplatte
Positionstyp: Intern
Verwendung: Verfügbar
Größe: 466 GB
Systemfestplatte: Nein
Festplatten-Daten-Cache: Aktiviert
Command Queuing: NCQ
Übertragungsrate: 3 Gb/s
Modell: WDC WD5002ABYS-02B1B0
Seriennummer: WD-WCASY7878803
SCSI Geräte-ID: 4
Firmware: 02.03B03
Größe des physischen Sektors: 512 Bytes
Größe des logischen Sektors: 512 Bytes

Freier Anschluss
Anschluss: 5
Anschlussposition: Intern

-------------------------------------------------
System Report

System Information
OS name: Microsoft Windows Server 2008 R2 Standard
OS version: 6.1.7601 Service Pack 1 7601
System name: SOL-5
System manufacturer: Supermicro
System model: X9SCL/X9SCM
Processor: GenuineIntel Intel64 Family 6 Model 42 Stepping 7 3.301 GHz
BIOS: American Megatrends Inc., 2.10

Intel® Rapid Storage Technology enterprise Information
Kit installed: 13.6.0.1002
User interface version: 13.6.0.1002
Language: English (United States)
RAID option ROM version: 11.6.0.1702
Driver version: 13.6.0.1002
ISDI version: 13.6.0.1002

Storage System Information
RAID Configuration

Array Name: SATA_Array_0000
Size: 1,907,739 MB
Available space: 186,700 MB
Number of volumes: 1
Volume member: vm_V0
Number of array disks: 2
Array disk: S21DNWAFC15630J
Array disk: S2BBNEAG113442N
Disk data cache: Enabled

Array Name: SATA_Array_0001
Size: 457,872 MB
Available space: 27,792 MB
Number of volumes: 1
Volume member: sys_V1
Number of array disks: 2
Array disk: S0VZNYAC203032
Array disk: CVCV205302JG240CGN
Disk data cache: Enabled

Volume name: vm_V0
Status: Normal
Type: RAID 1
Size: 860,519 MB
System volume: No
Data stripe size: 64 KB
Write-back cache: Write back
Initialized: Yes
Parity errors: 1
Blocks with media errors: 0
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Volume name: sys_V1
Status: Normal
Type: RAID 1
Size: 215,040 MB
System volume: Yes
Data stripe size: 64 KB
Write-back cache: Write back
Initialized: Yes
Parity errors: 0
Blocks with media errors: 0
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Hardware Information

Controller name: Intel(R) Desktop/Workstation/Server Express Chipset SATA RAID Controller
Type: SATA
Mode: RAID
Number of SATA ports: 6
Number of volumes: 2
Volume: vm_V0
Volume: sys_V1
Number of spares: 0
Number of available disks: 1
Rebuild on Hot Plug: Disabled
Manufacturer: 32902
Model number: 10274
Product revision: 5
Direct attached disk: S21DNWAFC15630J
Direct attached disk: S2BBNEAG113442N
Direct attached disk: S0VZNYAC203032
Direct attached disk: CVCV205302JG240CGN
Direct attached disk: WD-WCASY7878803

Disk on Controller 0, Port 0
Status: Normal
Type: SATA SSD
Location type: Internal
Usage: Array disk
Size: 932 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Transfer rate: 6 Gb/s
Model: Samsung SSD 850 EVO 1TB
Serial number: S21DNWAFC15630J
SCSI device ID: 0
Firmware: EMT01B6Q
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 0, Port 1
Status: Normal
Type: SATA SSD
Location type: Internal
Usage: Array disk
Size: 954 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Transfer rate: 6 Gb/s
Model: Samsung SSD 850 PRO 1TB
Serial number: S2BBNEAG113442N
SCSI device ID: 1
Firmware: EXM02B6Q
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 0, Port 2
Status: Normal
Type: SATA SSD
Location type: Internal
Usage: Array disk
Size: 238 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Transfer rate: 3 Gb/s
Model: SAMSUNG SSD 830 Series
Serial number: S0VZNYAC203032
SCSI device ID: 2
Firmware: CXM03B1Q
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 0, Port 3
Status: Normal
Type: SATA SSD
Location type: Internal
Usage: Array disk
Size: 224 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Transfer rate: 3 Gb/s
Model: INTEL SSDSC2CW240A
Serial number: CVCV205302JG240CGN
SCSI device ID: 3
Firmware: 400i
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 0, Port 4
Status: Normal
Type: SATA disk
Location type: Internal
Usage: Available
Size: 466 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Transfer rate: 3 Gb/s
Model: WDC WD5002ABYS-02B1B0
Serial number: WD-WCASY7878803
SCSI device ID: 4
Firmware: 02.03B03
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Empty port
Port: 5
Port location: Internal

--------------------------------

@ theRealBibo:
Welcome at Win-RTAID Forum!

Although I do not know, whether this has something to do with your problem, but the in-use Intel RST ROM/driver combination seems to be not the best (the RAID ROM module is too old, the RAID driver too new for your mainboard’s chipset). If you want to get the best results (performance and stability), both RAID components should belong to the same development branch (11.6/11.7 or 13.5/13.6).
My suggestion wouled be to update the Intel RAID ROM to v12.9.0.2006 and to "downgrade" the Intel RST driver to v12.9.4.1000 WHQL.

Regards
Dieter (alias Fernando)