This paper is part of the Esprimo P910 w/ Tesla K80 installation manual serie.
This article is related to the setup of the connections with/in the Fujitsu Esprimo P910 E85+, early Ubuntu 24.04.1 LTS configuration and to do some tests in preparation of installing the Nvidia Tesla K80 accelerator board. This article has been created starting from the notes taken at the end of the article about dual PSU installation, in the final version of the manual which is going to collect all the contribution, this article will be incorporated at the following chapter after the P910's BIOS upgrade and hardware diagnostic tests.
Front USB 3.0 ports
The Tesla K80 is so long that it goes over the USB 3.0 connector but it is too high. In order to keep the two USB 3.0 front ports, a 90° adapter can be used. There are two similar adapters quite similar and I took the one with 15 mm height which it seems to fit in place but the other which is 1 mm lower, I think it fits even better.
right click menu to enlarge (x2) the image
Because the front USB 3.0 ports are working at 40 MB/s - while those in the back at 450 MB/s - can be fine living only with 2x USB 2.0 at 30 MB/s on the front panel. Therefore, this adapter should be considered an optional. It is better to buy an USB 3.0 extension cable and leave it connected to one of the ports on the back of the case.
Ethernet configuration
First of all we need to set up the Ethernet network link between the workstation and the GPU server.
Hostname
IPv4
IPv6
IP address
Netmask
Gateway
DNS
Route
Device
X390
manual
disable
10.10.10.1
255.255.255.0
off
automatic
usbeth0
P100
manual
disable
10.10.10.2
255.255.255.0
off
automatic
pcieth0
These configuration settings are provided to be used with the Ubuntu Network Manager. Please note that installing Ubuntu 24.04.1 LTS is not mandatory but for lowering the entry barrier for those users who are not used to operating as system administrators.
ⓘ
For those prefer to use Microsoft Windows, please notice that I strongly doubt that a 2014 system will be support by Windows 11 and hence you will be forced to be stick with Windows 10 at the little extra cost $30/year support plus the a license than you might provide yourself buying a refurbished OEM sata disk/system. However, if you decide to go with Windows 10 and pay for its support, then refer to those you are paying to give you support!
Esprimo P910 has an Intel PRO/1000 Gigabit ethernet which linked with a cheap USB FastEthernet 100 Mbit/s network adapter will provide such performace below:
k80user@p910:~$ nc -l 1111 > /dev/null
roberto@x390:~$ dd if=/dev/zero bs=1500 count=16K | nc -N 10.10.10.2 1111
16384+0 records in
16384+0 records out
24576000 bytes (25 MB, 23 MiB) copied, 2.01924 s, 12.2 MB/s
Which is good-enough for a reactive GUI remote control but relatively slow to transfer large amount of data. So, a Gigabit USB adapter is worth its 2x price.
Remote control
WARNING!
Please, note that this is NOT the proper way to go with a system in "production" but a setup shortcut.
In order to avoid to waste our time switching between our laptop/PC and the GPU server:
login into Gnome GUI session with the k80usr's password (e.g. no autologin session)
Settings → System → Remote Desktop → Desktop Sharing → Remote Control → ON
Settings → System → Users → Unlock → Automatic Login → OFF
Settings → System → Secure Shell → ON
sudo apt install openssh-server -y ← in whatever rather than Ubuntu 24.04
These settings will allow us to access the GPU server with Remmina and any SSH client.
Root SSH password-less login
We are lazy and we are proud of it, especially when repetitive tasks are at stake. This includes password digitization for frequent accesses. One in particular is the root access to a remote host using SSH without entering the password. Moreover, by the password is a way of login that usually is disabled by default for the root user, at least.
# On the client side
ssh-keygen -t rsa
ssh-copy-id k80usr@10.10.10.2
ssh k80usr@10.10.10.2 # On the server side
sudo -s
mkdir -p /root/.ssh
cat /home/k80usr/.ssh/authorized_keys >>/root/.ssh/authorized_keys
chmod go-wrx -R /root/.ssh
echo "tput cvvis" >> ~/.bashrc
echo "set nocompatible" >> ~/.vimrc
touch ~/.Xauthority
exit
echo "tput cvvis" >> ~/.bashrc
echo "set nocompatible" >> ~/.vimrc
touch ~/.Xauthority
exit # On the client side
printf "\nrl() { ssh -XY root@10.10.10.2; }\n" >> ~/.bashrc
printf "sl() { ssh -XY k80usr@10.10.10.2; }\n" >> ~/.bashrc
bash; rl
Appending two functions at the .bashrc file, in every new bash shell we can just digit sl or rl for user or root login on our GPU server.
DVI to VGA adapter
The Esprimo P910 comes with a DVI port, and in case you plan to couple with an old VGA monitor like I did, then expect that a cheap DVI-VGA adapter will limit the monitor resolution to 1024x768 despite being advertised differently. Which is enough for doing the preliminary stuff and remoting the desktop.
Unfortunately, it also affects the resolution of the shared desktop by Gnome RDP. In order to mitigate this limitation, it is worth installing the Gnome tweaks and injecting a new resolution from a Gnome terminal:
With Gnome tweaks we can change the fonts and icons sizes, plus setting the zoom at 75%. Which combined with the maximum resolution supported by the monitor (e.g. 1280x1024, 5:4) provides a display equivalent area (e.g. 1700x1366) enlarged by 78% (and 3x than the 1024x768 given by the adapter) but at lower 96 → 72 DPI resolution. Not bad at all, for an old piece of trashware!
Wi-Fi/LAN networking
The Wi-Fi dongle is NOT an alternative to the cabled network for transferring data to the GPU server. Because configuring the Wi-Fi in a way that clients over that network can see each other is NOT a good idea, in terms of security nor data privacy.
On the other hand, connecting the GPU server in a cabled network requires a proxy/firewall for reaching the Internet which can be useful for maintenance, updates and downloads. In a home network the Wi-Fi usually provides Internet access.
Providing the GPU server with an independent connection to the Internet, which can physically disable removing the USB dongle, allows us to downsize the RJ45 cabled network to a single link point-to-point with our workstation without configuring it as Internet proxy as well.
Please note that within the RTL8188’s family there are adapters which support 2.4GHz at 150 Mbit/s, only. Just in case your Wi-Fi network is working only at 5GHz, instead. Moreover, it will result relatively slow to leverage a full-fledged optic fiber Internet connection.
Thermal control
Let start from the basics, here below some line commands for Ubuntu just for starting with the P910 before even installing the Tesla K80 within, with sudo -s root priviledges:
Moreover, the application gkrellm - even if it is not particularly well integrated in Ubuntu - can help us keep the system under careful supervision. Here github.com/robang74/gkrellm2-config a suggested way to configure and theming it
Package id 0: +44.0°C (high = +85.0°C, crit = +105.0°C)
Which is the highest temperature seen up to now, with the case open and laying on its closed side. Which makes the CPU radiator operate in a sub-optimal way, as per its factory design. Without the support of an active device like a fan, it relies on the "hot-air is lighter" physics principle that lets the air flow bottom-up through its fins for natural convection. But in that position, laying down 90° tilted, the air tends to remain trapped among its fins instead of flowing and the main fan - with the case open - is too far to bring a sensitive benefit before its flow spreads around unguided.
Noise control
Time to close the tower case and raise it on its feet again.
dd if=/dev/zero bs=1M | pigz -11 -p4 - >/dev/null &
for i in $(seq 1 40); do sensors | grep Package;
sleep 1; done & sleep 30; killall pigz; echo;
Running this code for two times in a row, lead to warm up the CPU core:
Package id 0: +77.0°C (high = +85.0°C, crit = +105.0°C)
At 75°C the main 12 cm fan starts to be loud, working at almost the full throttle. This is because the default BIOS configuration is tuned to be silent as much as possible. However, these settings could be changed and the difference is very sensitive.
F2 → BIOS → Advanced → System Monitoring → Fan Control → auto, enhanced, disabled
While acoustic management has an impact on "auto" and "enhanced" fan control modes, it has not on "disabled" for which all the fans are running at their full throttle.
Cooling noise tests
In order to have a quantitative idea about the noise figure, I have installed this Android application on my smartphone and put it on the tower case.
Sound meter : SPL & dB meter by KTW Apps, 4.8* on Google Play
Between the case and the smartphone I put a mouse pad, just to absorb those low-frequencies which are below the human ability to hear but a capacitor microphone may catch. Usually, this kind of measures should be done after a calibration and reading the value at a standard distance (usually 1 mt) from the geometric center of the noise source.
Instead, I took the value on an arbitrary and generic mouse pad with an uncalibrated app because I am interested in the relative metric, not in the absolute. In fact, a quiet room is about 40 dB while the keyboard beep scales at 62 dB. My studio in the night is as quiet as 25 db while the modded P910 with its 12 cm fan at full throttle showed a steady 64 db figure, the double of the most silent BIOS configuration.
Using this script saved in data, this table can be completed with some temperature ranges, just to provide a quick and basic reference.
Acoustic Management
Fan Control
Noise R(dB)
Noise R(A²)
Start
Highest
Ending
T-Gap
enabled
auto
32
1.00 x
49-61°C
67-74°C
51-57°C
83%
enabled
enhanced
40
6.31 x
42-48°C
60-63°C
42-45°C
66%
irrelevant
disabled
64
1585 x
32-34°C
46-48°C
28-38°C
43%
fan w/ splitter
disabled
70
6310 x
31-37°C
49-52°C
31-35°C
49%
Notice that decibel are a logarithmic way of measuring the ratio between two sound intensities, alike human hearing are working. Hence considering a referencing intensity (32 dB) then the difference with another value gives us the absolute ratio between the two related intensities despite the scale was not standardly calibrated.
ⓘ
In the final version of the manual, this part will end up after the BIOS upgrade chapter, so before every hardware change. Instead, the temperatures above reported, have been taken after having modified the Esprimo P910 moving its main 12cm fan and installing the handcraft baffle to increase the case air-flow. At the time of the measures, the two middle PCI slot covers were removed. Hence, it could be possible to find different values with the original hardware configuration.
For sake of completeness, these temperatures have been measured in a 20°C room temperature. Sometimes, the ending temperature is lower than the starting temperature but this should not surprise us because in the meantime the fan started to cool down the CPU and the ending temperature is taken after 15 seconds the 30s job is completed.
When the Fan Control and the Acoustic Management are both disabled, and an air-flow splitter is added to cooling down the Tesla K80, then because of the hand-crafted splitter the noise increases by a six decibels compared when the Tesla K80 nor the splitter were installed.
However, the reference noise level is 40 dB, when the GPU heating additive contribute reasonably requires an enhanced cooling policy but instead we force the fan at its full throtling with the air-flow splitter as per iper-prudence initial testing policy. Also, the splitter does not reduce sensitively the CPU cooling efficiency.
Just to have an idea, using the inverse square law and ISO 9613-1:1996 calculator (440Hz, 50%, 1atm, 20°C) having 40 dB at 20 cm is equivalent to 70 dB at 5 mt. Considering 40 dB noise level as a "quiet room in daylight hours", then others house's rooms are not sensitively impacted by the GPU server in terms of noise when the noisy room's door is closed.
Spinning the whirlybird
Unsurprisingly, noisier it gets, cooler it ends up. How much? The absolute temperature scale is graded in Kelvin but its zero is too far away. However, neither Celsius nor Fahrenheit are good in providing a relative reference. The main point is the same: setting the zero, but in this case the zero is the environment temperature which is 20°C, our reference zero.
While 85°C is our "boiling water" point which is a reference temperature for the CPU that has been indicated by the CPU foundry process. For comparison, the GPU incorporated into the Tesla K80 card, has an 88°C alarming temperature trigger. Under this PoV, 43% is 43°R, while the room temperature is 0°R and 100°R is the warning temperature.
The table presented in the previous section shows that the original system can potentially be quieter than a quiet room in the night but also annoying like the keyboard beep when a key is kept pressed down, but at lower frequency. Once completed the system software configuration, the GPU card will be installed and tested.
Prudently, we will test the system in its initial stages of configuration by unleashing its "wanna-be an helicopter" character...
Kernel boot config
These options are the suggested for the Linux kernel command line:
mtrr_gran_size=8M mtrr_chunk_size=64M
to solve the issue about MTRR losing 230 MB of RAM mitigated to 6 MB
pcie_aspm=off mem_encrypt=off, nouveau.modeset=0 and blacklist=chromeos_pstore
for improving the performances of the Tesla K80, but it consumes more energy
to prevent the generic open-source nvidia cards driver to be loaded on boot
avoid kernel load the Chrome OS related module because P910 hardware specs
pci=realloc and intel_iommu=on iommu=pt intremap=on pci=assign-busses,routeirq
due to an explicit suggestion for adding that argument by the dmesg output
more parameters about the PCI/e subsystem for the sake of Nvidia K80 board
The above parameters should go into the GRUB_CMDLINE_LINUX_DEFAULT variable which is recorded into the /etc/default/grub file. Then sudo update-grub to write the change in the grub's boot record. Moreover, blacklisting these two kernel modules and rebuild all the initramfs files, is a good idea:
( echo; echo "blacklist chromeos_pstore"; echo "blacklist nvidiafb";
echo "blacklist nouveau" )| sudo tee -a /etc/modprobe.d/blacklist.conf
sudo update-initramfs -u -k all
Then reboot to let the system restart with the new settings. Finally, after the reboot cat /proc/cmdline and lsmod to check.
BIOS settings
First of all, reset the BIOS settings to its optimal default and then on the top of that settings we can make some adjustments:
Trust Computing: disabled
PCIe ASPM support: Disabled
Intel TXT support: Disabled
Secure Boot: Disabled
It is better to disable executing code isolation and secure boot options until the system is correctly configured. Moreover, in order to avoid possible conflicts by legacy support from BIOS, also:
Legacy USB support: Disabled
Serial and parallel ports: Disabled
CSM launch: Disabled
Because the 2nd PSU installed, we might want use a multi-plug extender with 0/1 switch, in that case these make our life easier:
Power-On source: ACPI
Power failure recovery: Disabled
Enabling the "Secure Boot" requires the CSM disabled while the USB legacy support is requested by non-UEFI bootable devices.
However, none of these changes has a sensitive impact on the dmesg output which means that the 6.x Linux kernel is not affected by. Despite this, it is better to start with a BIOS configuration as much friendly as possible and then when we know that the system is working correctly, make changes that matter for our specific user case.