Linux driver - Binary rpm/source rpm user guide and known limitation


1. OVERVIEW
This README covers Intel’s MPT3SAS Linux driver specific limitation
and information. For any OS Distro specific limitation and
information please check with OS Vendor support.

2. DRIVER COMPILATION STEPS
Driver source code is placed inside released driver package.
Driver source tarball should be with name –
mpt3sas-<driver_version>-src.tar.gz
1) Untar driver source tarball-
#tar -zxvf mpt3sas-<driver_version>-src.tar.gz
2) Go to driver source directory-
#cd mpt3sas
3) To compile the driver for distro use the helper script
"compile.sh" bundled inside source code-
#./compile.sh
4)To load and unload the driver for distro use the helper script
“load.sh” and “uload.sh” bundled inside source code-
#./load.sh
#./uload.sh


3. OS SUPPORT MATRIX
Following are the List of supported Distro and their corresponding kernel flavors:
#######################################################################
SuSE:
#######################################################################
i686

SLES 11 SP2
Gold (3.0.13-0.27) (default, pae, trace, xen)
SLES 11 SP3
Gold (3.0.76-0.11) (default, pae, trace, xen)
SLES 11 SP4
Gold (3.0.101-63) (default, pae, trace, xen)

x86_64

SLES 11 SP2
Gold (3.0.13-0.27) (default, trace, xen)
SLES 11 SP3
Gold (3.0.76-0.11) (default, trace, xen)
SLES 11 SP4
Gold (3.0.101-63) (default, trace, xen)

SLES12
Gold (3.12.28-4) (default, xen)
SLES12 SP1
Gold (3.12.49-11) (default, xen)
SLES12 SP2
Gold (4.4.21-69) (default)
SLES12 SP3
Gold (4.4.73-5) (default)

#######################################################################
Red Hat, Cent OS, Oracle Enterprise Linux :
#######################################################################
Note: As the OS kernel is same for corresponding version of RHEL,OEL &
CentOS so the driver binaries of RHEL will work for corresponding OEL &
CentOS. Hence we will be providing unified binary support for RHEL,OEL
& CentOS(i,e., No separate binaries for OEL & CENTOS, will be provided
instead corresponding RHEL binaries should be used).

i686
RHEL6
Update 6 (2.6.32-504) (el6)
Update 7 (2.6.32-573) (el6)
Update 8 (2.6.32-642) (el6)
Update 9 (2.6.32-696) (el6)

x86_64
RHEL6
Update 6 (2.6.32-504) (el6)
Update 7 (2.6.32-573) (el6)
Update 8 (2.6.32-642) (el6)
Update 9 (2.6.32-696) (el6)

RHEL7
Update 1 (3.10.0-229) (el7)
Update 2 (3.10.0-327) (e17)
Update 3 (3.10.0-514) (e17)
Update 4 (3.10.0-693) (e17)

#######################################################################
Oracle Enterprise Linux:
#######################################################################

i686
OEL6
Update 6UEK (2.6.39-400.215.10) (el6uek)
Update 7UEK (2.6.39-400.250.7) (el6uek)
Update 8UEK (2.6.39-400.278.2) (el6uek)

x86_64
OEL6
Update 6UEK (3.8.13-44.1.1) (el6uek)
Update 7UEK (3.8.13-68.3.4) (el6uek)
Update 8UEK (4.1.12-37.4.1) (el6uek)
OEL7
Update 1UEK (3.8.13-55.1.6) (el7uek)
Update 2UEK (3.8.13-98.7.1) (el7uek)
Update 3UEK (4.1.12-61.1.18) (el7uek)

Note:
With respect to UEK kernels don’t confuse with driver rpm names. For
example driver rpm name for UEKR3_U4 on oel6 OS is kmod-mpt3sas-
DriverVersrion_oel6.6_UEKR3_U4-ReleseVersion.x86_64.rpm and one can
assume that UEKR3_U4 kernel rpms needs to be installed only on OEL6.6
base OS, but that is not true, one can install these UEK3_U4 kernel
rpms on any other OEL6 OS (e.g. OEL6.5 OS) and after booting into this
UEKR3_U4 kernel, one can install the above driver rpm on this kernel.



#######################################################################
Citrix:
#######################################################################
(NOTE: This is only x86 support from OS vendor)
Citrix6
Update 2 (2.6.32.43-0.4.1.xs1.8.0.835.170778) (kdump,
xen)
Update 5 (3.10.0+2)
Citrix7.0 (3.10.0+10)

#######################################################################
Ubuntu:
#######################################################################
i686
Ubuntu14.04LTS (3.13.0-24-generic)
Ubuntu15.04 (3.19.0-15-generic)
Ubuntu16.04 (4.4.0-21-generic)
x86_64
Ubuntu14.04LTS (3.13.0-24-generic)
Ubuntu15.04 (3.19.0-15-generic)
Ubuntu16.04 (4.4.0-21-generic)

#######################################################################
Debian:
#######################################################################
i686
Debian7.0 (3.2.0-4-486, 3.2.0-4-686-pae)
Debian8.0 (3.16.0-4-586, 3.16.0-4-686-pae)
x86_64
Debian7.0 (3.2.0-4-amd64)
Debian8.0 (3.16.0-4-amd64)

#######################################################################
Fedora:
#######################################################################
i686
Fedora22 (4.0.4-301) (fc22, fc22+pae)
x86_64
Fedora22 (4.0.4-301) (fc22)

4. RELEASE CONTENTS:
For any queries on supported OS matrix, please refer above SECTION
#3 contents. OS Support list in SECTION #3 list out test coverage
executed by Avago. MPT3SAS driver is GPLv2 open source driver and
source code level support is possible for many linux kernel
versions. If you do not find binary level support for your
distribution in release contents, please use source rpm method.
For any distribution or supported kernel version, there can be three
possible packages –
- Driver update disk (Available under folder disks-xx)
- Precompiled binary (kmod/kmp/rpms).(Available under folder
rpms-xx)
- Source rpm. (Available under folder rpms-xx)

5. ERRATA/NOTES AND KNOWN LIMITATIONS
a. Oracle Linux Installation errata:
Installing Driver during Installing for UEK from CD:
For UEK kernels, only the KMODs RPMs are provided, not the DUDs. The
reasoning behind this is the OEL installation is using the native Red
Hat kernels, not UEK. UEK kernel RPMS can be installed after the basic
installation is completed. Please check with Oracle support team w.r.t
UEK kernel installation process, limitation and other technical queries
which is more of generic and not related to MR Drivers.

b. RPM install dependency issues:
If driver RPM installation fails with kABI checks dependency failure
message, installing RPM package the user will need to use the "--
nodeps" switch when installing the binary."
Example: rpm -ivh --nodeps kmod-mpt3sas-vxxxxxx_UEK.xxx.rpm
"If "rpm -ivh throw any dependency warning/error"
RPM uses KMOD packaging dependency data to ensure the dependencies are
met before installing the binary RPM.
Red Hat maintains a whitelist of kernel symbols which RPM uses to
validate against the KMOD binaries. Some symbols may be in the kernel
but not on the whitelist which results in a failed binary RPM install.
User can use the "--nodeps" switch when installing the binary to skip
those whitelist symbol checks or any other dependency."

c. Kernel crash observed on kernels with version >= 4.1
  1.Kernel crash observed while creating a second RAID volume.
Issue: Kernel gets crashed while creating 2nd RAID volume.
Hence this issue has impact on IR card and not on IT card.
Steps to Reproduce:
>>Boot into OS/kernel with its inbox/out-of-box driver, after discovering
HBA and the devices connected to HBA successfully.
>>Launch utility (Ex: SAS3IRCU) and create a RAID volume
(RAID0/RAID1/RAID10).volume gets created successfully and will be listed
from utility.
>>Similarly try to create second RAID volume (RAID0/RAID1/RAID10), Kernel
crash is observed while creating second RAID volume.
Expected: One should be able to create MAX 2 RAID volumes
with IR card at any point of time successfully , without
any kernel crash.
BZ link: Below link has complete details,
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1581326
  2.Kernel crash observed while unloading the driver keeping
enclosure attached.
Issue: Kernel gets crashed while unloading driver keeping
enclosure attached. This issue impacts on both IT and IR
card.
Steps to Reproduce:
>>Boot into OS/kernel with its inbox/out-of-box driver, after discovering
HBA and enclosure with set of drives connected to HBA successfully.
>>Try unloading driver “modprobe –r mpt3sas”, kernel crash is observed.Expected: Driver should unload successfully irrespective of
whether enclosure is connected behind IT/IR card.
Workaround Solution: As this issue is due to some “ses”
module patch , user may have to unload “ses”(rmmod ses)
module first followed by driver unload. Then driver unloads
successfully.
BZ link: Below link has complete details,
https://bugzilla.oracle.com/bugzilla/show_bug.cgi BUG:15521

d. Enabled DIX support by default in the driver on kernels >= 3.18
Observed Ref tag errors on 4K DIX Type0 drive. This is kernel issue
and below patch set fixed this issue.
https://www.spinics.net/lists/linux-scsi/msg74665.html
So disabled DIX support by default on kernels < 3.18 and enabled DIX
support on kernels >= 3.18 by default in driver. Above mentioned
patch set changes are available from kernels >=3.18.

6. INSTALL/REMOVE/UPGRADE DRIVER PACKAGE

There are two packaging formats for binary rpms –
1) RPM(RHEL/SLES/Fedora/OEL uses RPM package)
2) DEB(Ubuntu/Debian uses DEB package) in which driver binary support
is provided. More info –
https://en.wikipedia.org/wiki/Deb_%28file_format%29
Please note that after install/remove driver package, system needs to
be rebooted to get intended driver loaded or manually remove module and
insert (read man page “rmmod” and “modprobe” for more info)
Whenever driver package is installed/uninstalled/upgraded, it is good
practice to check output of command #modinfo mpt3sas.
Output should always have updated driver version.

Steps for Driver install/remove/upgrade for .rpm package
1. To install driver RPM, type below command-
# rpm –ivh <DRIVER_PACKAGE>.rpm
2. To uninstall driver RPM, type below command to check installed
driver package name-
# rpm –qa | grep mpt3sas
Output will give installed mpt3sas RPM packages.
Now type # rpm –e <package to be uninstalled>
3. To upgrade driver RPM, type below command-
#rpm –Uvh <DRIVER_PACAKGE>.rpm

Steps for Driver install/remove/upgrade for .deb package
1. To install .deb package, execute following command,
#dpkg -i <DRIVER_PACKAGE>.deb
2. To verify the status of installed packages then type following
command
#dpkg -s mpt3sas
3. After installing mpt3sas driver, type below
#modinfo mpt3sas
It should show the currently installed version of mpt3sas
4. To use installed DEB driver loaded, restart the machine and type
following command to get currently loaded driver version-
#cat /sys/modules/mpt3sas/version
This version should be same as driver version of installed driver
DEB package.
5. To uninstall mpt3sas package, type below command-
#dpkg -r mpt3sas
6. Verify "modinfo mpt3sas" mpt3sas version should be in-box
version.


7. STEPS TO GENERATE BINARY RPM FROM SOURCE RPM

There are three variant of source rpms available in this package –
a. Source rpm which use kmodtool interface (RHEL based)
http://rpmfusion.org/Packaging/KernelModules/Kmods2
b. Source rpm which use kmp build interface (SLES based)
https://en.opensuse.org/Kernel_Module_Packages
c. Source rpm which use generic build interface (Create initramfs
internally and does not depend upon any external tool)
See NOTES section for sample naming convention used for these three
flavor of source rpm.
If user doesn’t know which source rpm is better for their
environment, we recommend trying #c (for any other distro other than
Redhat/Novell)
Quick search of “rpm –qa |grep kmod” can provide hint, if kmod tool
support is available or not.
To generate binary rpm from source rpm, user should have
compilation/build environment to create kernel module, utilities to
build RPM(e.g. rpmbuild..).Install “kernel-devel” or “linux-headers”
depending on distro for compilation environment. E.a system where
user has yum repo configured, use below.
# yum groupinstall “Development Tools”
E.a on Ubuntu user can try installing below missing components.
apt-get install rpm
apt-get install make
apt-get install gcc
apt-get install alien
Exact commands to create build environment would be different across
distros so this document is not right place to cover those details.
Please refer OS vendor document if needed how to create
build/compilation environment for specific OS distro.

Below are steps to generate binary RPM from source RPM-

	1. Install source RPM using command <rpm>. Example below-
	#rpm -ivvh mpt3sas-<driver_version>.src.rpm
	Installing above RPM will copy driver SPEC file to specific
	location (configured as part of rpm package. This path can be
	different for each OS distribution.)
	To locate SPEC file, check output logs of above source RPM
	installation (see blue marked gives SPEC file location).
	e.g.
	
	[root@localhost tmp]# rpm -ivvvh mpt3sas-06.810.00.02-
	98.src.rpm
	D: ============== mpt3sas-06.810.00.02-98.src.rpm
	..
	Updating / installing...
	1:mpt3sas-06.810.00.02-
	98 ################################# [100%]
	D: ========== Directories not explicitly included in package:
	D: 0 /root/rpmbuild/SOURCES/
	D: 1 /root/rpmbuild/SPECS/
	D: ==========
	D: unknown 100755 1 ( 0, 0) 25
	/root/rpmbuild/SOURCES/Module.supported;55a756c8
	D: unknown 100644 1 ( 0, 0)120552
	/root/rpmbuild/SOURCES/mpt3sas-06.810.00.02.tar.gz;55a756c8
	D: unknown 100644 1 ( 0, 0) 6783
	/root/rpmbuild/SPECS/mpt3sas.spec;55a756c8
	GZDIO: 17 reads, 127888 total bytes in 0.000585 secs
	D: closed db index /var/lib/rpm/Name
	D: closed db index /var/lib/rpm/Packages
	D: closed db environment /var/lib/rpm

	2. Go to directory where driver SPEC file is copied as part of #1.
	There must be SPEC file e.g. mpt3sas.spec/lsi-mpt3sas or
	mpt3sas.spec(driver SPEC file name could be different for
	different distros, so check SPEC file with megaraid string in its
	name).
	Check spec file name in above command. It provides the location
	and spec filename.

	3. Build binary RPM from source RPM. Below is command to do same-
	#rpmbuild -ba <DRIVER_SPEC_FILE>
	4. Binary RPMs will be available if #3 exits without any error. Go
	to directory where new binary RPM is generated.
	E.a Snippet of working case –
	--
	Wrote: /root/rpmbuild/SRPMS/mpt3sas-06.810.00.02-98.src.rpm
	Wrote: /root/rpmbuild/RPMS/x86_64/mpt3sas-06.810.00.02-
	98.x86_64.rpm
	Wrote: /root/rpmbuild/RPMS/x86_64/mpt3sas-debuginfo-06.810.00.02-
	98.x86_64.rpm
	Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.ZbHbmH
	+ umask 022
	+ cd /root/rpmbuild/BUILD
	+ cd mpt3sas-06.810.00.02
	+ /usr/bin/rm -rf /root/rpmbuild/BUILDROOT/mpt3sas-06.810.00.02-
	98.x86_64
	+ exit 0
	--

	Step #5 is applicable only for deb package based OS distribution
	(Ubuntu/Debian).
	5. Create driver .deb package from binary RPM, Execute below command
	to achieve the same-
	# alien -k --to-deb --scripts <GENRATED_DRIVER_RPM>
	Now, Install generated binary rpms using steps mentioned at
	“Install/Remove/Upgrade driver binary package”
	
	It is always recommended to verify initramfs image to confirm updated mpt3sas driver
	is packaged correctly before system reboot. Do not assume that latest driver is loaded
	after installing rpm generated from source rpm. See OS distribution specific
	documentation on how to verify initramfs image. E.a FC21 user guide link for reference.
	
	https://docs.fedoraproject.org/en-US/Fedora/21/html/System_Administrators_Guide/sec-
	Verifying_the_Initial_RAM_Disk_Image.html#bh
	Verifying_the_Initial_RAM_Disk_Image_and_Kernel_on_IBM_eServer_System_i


8. NOTES

1. In case of SuSE and similar OS distribution which use kmp tool for
initramfs, we recommend to edit file - /etc/sysconfig/kernel. Append
<mpt3sas> before installing binary rpm.
INITRD_MODULES="ata_piix ata_generic mpt3sas"

2. If you see initramfs failure while rpm installation, try generating
inird manually. Just make sure that <modinfo mpt3sas> provide the
expected driver version.

3. Do not try to install/upgrade from source rpm generated rpm binaries
along with pre-compiled binary provided by Avago. Naming convention
as part of precompiled binary and source rpm based rpm differs, and
that is reason user should stick with one method to avoid
compatibility issue.
Source rpm is quickest method to deploy driver on many supported
kernel versions, whereas precompiled binary is specific to kernel
version or set of kernel versions.
Example –
Source rpm based binary will be generated in below format -
kmp_rpm:
lsi-mpt3sas-kmp-default-11.255.01.00_3.0.76_0.11-99.x86_64.rpm
kmod_rpm:
kmod-mpt3sas-11.255.01.00-61.x86_64.rpm
generic_rpm:
mpt3sas-11.255.01.00-98.x86_64.rpm

4. On System with large number of CPU core and LSI’s SAS3 controllers,
on repeated load and unload of mpt3sas driver module, if kernel
fails to allocate the memory requested for higher queue depth, we
can observe that the loading of mpt3sas module fails. Below messages
will be logged to /var/log/messages,
mpt3sas0: chain_lookup: __get_free_pages failed
mpt3sas0: Reduce the module parameter max_queue_depth to a value
lower than (“CURRENT_VALUE_OF_QUEUE_DEPTH”) and retry.
The work-around for this issue is to load mpt3sas driver with module
parameter max_queue_depth set to value less than
CURRENT_VALUE_OF_QUEUE_DEPTH.
The max_queue_depth module parameter could be set as follows
a. While loading the driver
modprobe mpt3sas max_queue_depth=NEW_VALUE_OF_QUEUE_DEPTH (if driver
rpm is already installed)
(Or)
insmod mpt3sas.ko max_queue_depth=NEW_VALUE_OF_QUEUE_DEPTH (if you
have a mpt3sas.ko file)
b. If driver is in ramdisk, then in RHEL5/SLES/OEL5 OS, following
line has to be added in /etc/modprobe.conf and reboot the system
options mpt3sas max_queue_depth=NEW_VALUE_OF_QUEUE_DEPTH
(Or)
Add below word at the end of kernel module parameters line in
/boot/grub/menu.lst or /boot/grub/grub.conf file and reboot the
system
mpt3sas.max_queue_depth=NEW_VALUE_OF_QUEUE_DEPTH
5. When Target Reset is issued using below command to DIF type2 drive
present in the topology then kernel panic is observed on few
kernels.
echo 4 > sys/class/scsi_host/host(number)/task_management
The users can apply below patch if applicable otherwise can check
with the kernel vendors for the appropriate patch
http://marc.info/?l=linux-scsi&m=135186352200668&q=raw