Installation of Eiger computer at ESRF

BIOS

Disable CPU power management

‘lid10eiger1’ - Detector PC Type B

High Performance Detector PC Type B

In the ‘Advanced -> CPU Configuration’ menu

Hyper-Threading

Enable

In the ‘Advanced -> CPU Configuration -> CPU Power Management Configuration’ menu

Power Technology

Disable

Energy Performance

Performance

Disable boot on Intel 10 Gigabit Converged Network adapter

In the ‘Boot -> Set Boot Priority’ menu select:

1st Boot Device

Hard Disk:P0: SATADOM-SL

In the ‘Advanced -> Intel(R) Ethernet Controller 10 Gigabit X540-AT2’ (first) menu note the field:

PCI Address

07:00:00

It is referred to in other menu as Slot 0700. Then in the ‘Boot -> Network Device BBS Priorities’ menu disable all the Intel X520 ports, and leave only the Intel X540-AT2 ports:

1st Device

IBA XE Slot 0700 v2304

2nd Device

Disabled

3rd Device

Disabled

4th Device

Disabled

5th Device

Disabled

Debian 7

Linux kernel

Update the Linux kernel from Debian repository in order to improve system reliability:

# as root
lid10eiger1:~ # linux_ver=$(dpkg --list linux-image-3.2\* | grep '^ii' | awk '{print $3}')
lid10eiger1:~ # echo ${linux_ver}
3.2.82-1

lid10eiger1:~ # linux_pkgs=$(dpkg --list linux\* | grep ${linux_ver} | awk '{print $2}')
lid10eiger1:~ # echo "${linux_pkgs}"
linux-doc-3.2
linux-headers-3.2.0-4-all-amd64
linux-headers-3.2.0-4-amd64
linux-headers-3.2.0-4-common
linux-headers-3.2.0-4-common-rt
linux-headers-3.2.0-4-rt-amd64
linux-image-3.2.0-4-amd64
linux-libc-dev:amd64
linux-manual-3.2

lid10eiger1:~ # apt-get install ${linux_pkgs}
...
Get:1 http://lin-repo-master.esrf.fr/debian/stable/debian-security/ wheezy/updates/main linux-image-3.2.0-4-amd64 amd64 3.2.96-2 [23.5 MB]
Get:2 http://lin-repo-master.esrf.fr/debian/stable/debian-security/ wheezy/updates/main linux-doc-3.2 all 3.2.102-1 [6,501 kB]
Get:3 http://lin-repo-master.esrf.fr/debian/stable/debian-security/ wheezy/updates/main linux-headers-3.2.0-4-all-amd64 amd64 3.2.96-2 [270 kB]
Get:4 http://lin-repo-master.esrf.fr/debian/stable/debian-security/ wheezy/updates/main linux-headers-3.2.0-4-amd64 amd64 3.2.96-2 [671 kB]
Get:5 http://lin-repo-master.esrf.fr/debian/stable/debian-security/ wheezy/updates/main linux-headers-3.2.0-4-common amd64 3.2.96-2 [3,641 kB]
Get:6 http://lin-repo-master.esrf.fr/debian/stable/debian-security/ wheezy/updates/main linux-headers-3.2.0-4-rt-amd64 amd64 3.2.96-2 [671 kB]
Get:7 http://lin-repo-master.esrf.fr/debian/stable/debian-security/ wheezy/updates/main linux-headers-3.2.0-4-common-rt amd64 3.2.96-2 [3,646 kB]
Get:8 http://lin-repo-master.esrf.fr/debian/stable/debian-security/ wheezy/updates/main linux-libc-dev amd64 3.2.102-1 [890 kB]
Get:9 http://lin-repo-master.esrf.fr/debian/stable/debian-security/ wheezy/updates/main linux-manual-3.2 all 3.2.102-1 [3,051 kB]
...

libc

The Debian 7 libc is 2.13-38+deb7u11, built from eglibc-2.13. This version has a bug in malloc, corrupting memory in heavyly multi-threaded applications. A recompiled version of libc, modified with selected upstream patches, must be installed so the SlsDetector software can run properly:

# as root
lid10eiger1:~ # mkdir -p ~/libc && cd ~/libc
lid10eiger1:~/libc # scp lid01eiger1:/nobackup/lid01eiger12/devel/libc/apt-get-source/eglibc-2.13/modified-02/*.deb .
...
lid10eiger1:~/libc # dpkg --install libc6_2.13-38+deb7u10_amd64.deb \
  libc6-dbg_2.13-38+deb7u10_amd64.deb libc6-dev_2.13-38+deb7u10_amd64.deb \
  libc6-dev-i386_2.13-38+deb7u10_amd64.deb \
  libc6-i386_2.13-38+deb7u10_amd64.deb \
  libc-bin_2.13-38+deb7u10_amd64.deb  libc-dev-bin_2.13-38+deb7u10_amd64.deb \
  locales_2.13-38+deb7u10_all.deb locales-all_2.13-38+deb7u10_amd64.deb \
  multiarch-support_2.13-38+deb7u10_amd64.deb libc6_2.13-38+deb7u10_i386.deb \
  libc-bin_2.13-38+deb7u10_i386.deb libc6-i686_2.13-38+deb7u10_i386.deb
...
lid10eiger1:~/libc # dpkg --list libc\* | grep 2.13-38
ii  libc-bin         2.13-38+deb7u10  i386   Embedded GNU C Library: Binaries
ii  libc-dev-bin     2.13-38+deb7u10  amd64  Embedded GNU C Library: Development binaries
ii  libc6:amd64      2.13-38+deb7u10  amd64  Embedded GNU C Library: Shared libraries
ii  libc6:i386       2.13-38+deb7u10  i386   Embedded GNU C Library: Shared libraries
ii  libc6-dbg:amd64  2.13-38+deb7u10  amd64  Embedded GNU C Library: detached debugging symbols
ii  libc6-dev:amd64  2.13-38+deb7u10  amd64  Embedded GNU C Library: Development Libraries and Header Files
ii  libc6-dev-i386   2.13-38+deb7u10  amd64  Embedded GNU C Library: 32-bit development libraries for AMD64
ii  libc6-i386       2.13-38+deb7u10  amd64  Embedded GNU C Library: 32-bit shared libraries for AMD64
ii  libc6-i686:i386  2.13-38+deb7u10  i386   Embedded GNU C Library: Shared libraries [i686 optimized]

Copy the good getconf binary (64-bit):

lid10eiger1:~ # cd /usr/bin
lid10eiger1:/usr/bin # mv getconf getconf.32
lid10eiger1:~ # scp lid01eiger1:/usr/bin/getconf .
...

cpufrequtils

The previous settings disable the ACPI CPU power management interface, so the loadcpufreq INIT service will not be able to load acpi-cpufreq kernel module. In case this is enabled in BIOS in the future, force the ‘performance’ governor in cpufrequtils INIT service:

# as root
lid10eiger1:~ # cat > /etc/default/cpufrequtils <<'EOF'
# valid values: userspace conservative powersave ondemand performance
# get them from cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
GOVERNOR="performance"
EOF

irqbalance

The irqbalance must be installed in order to distribute hardware IRQs to different CPU cores:

# as root
lid10eiger1:~ # apt-get install irqbalance
...

Disk configuration

Find the 4 TByte RAID-0 array in Debian (/dev/sda):

# as root
lid10eiger1:~ # for d in /sys/block/sd?; do \
    echo "$(basename ${d}): model=$(cat ${d}/device/model)" \
         "size=$(python -c "print \"%.1f MB\" % ($(cat ${d}/size) / (2*1024.0**2))")"; \
done
sda: model=LSI2208          size=3811.0 MB
sdb: model=SATADOM-SL 3ME   size=59.6 MB

Install parted and use it to create a GPT and a RAID partition on the entire disk:

lid10eiger1:~ # apt-get install parted
...
lid10eiger1:~ # parted /dev/sda
GNU Parted 2.3
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.

(parted) mklabel gpt

(parted) unit s

(parted) print free
Model: SMC LSI2208 (scsi)
Model: SMC LSI2208 (scsi)
Disk /dev/sda: 7992180736s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End          Size         File system  Name  Flags
        34s    7992180702s  7992180669s  Free Space

(parted) mkpart logical 2048s 7992178687s

(parted) set 1 raid on

(parted) print free
Model: SMC LSI2208 (scsi)
Disk /dev/sda: 7992180736s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start        End          Size         File system  Name     Flags
        34s          2047s        2014s        Free Space
 1      2048s        7992178687s  7992176640s               logical  raid
        7992178688s  7992180702s  2015s        Free Space

(parted) quit
Information: You may need to update /etc/fstab.

Note

the partition is aligned to 2048 sectors (1 MByte). The end sector is obtained by:

7992180702 - 7992180702 % 2048 - 1 = 7992178687

Create the filesystem and mount it:

lid10eiger1:~ # mkfs.ext4 /dev/sda1
mke2fs 1.42.5 (29-Jul-2012)
...

lid10eiger1:~ # blkid /dev/sda1
/dev/sda1: UUID="aff827d8-a744-470d-a753-998919f36d77" TYPE="ext4"

lid10eiger1:~ # mkdir -p /nobackup/lid10eiger12

lid10eiger1:~ # cat /etc/fstab
...
UUID="aff827d8-a744-470d-a753-998919f36d77" /nobackup/lid10eiger12        ext4    relatime,nodev,nosuid 0       2

lid10eiger1:~ # mount /nobackup/lid10eiger12

lid10eiger1:~ # df -h /nobackup/lid10eiger12
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       3.7T  196M  3.5T   1% /nobackup/lid10eiger12

lid10eiger1:~ # mkdir /nobackup/lid10eiger12/data
lid10eiger1:~ # chmod a+w /nobackup/lid10eiger12/data

Test the effective write speed:

lid10eiger1:~ # mkdir /nobackup/lid10eiger12/data/eiger
lid10eiger1:~ # chmod a+w /nobackup/lid10eiger12/data/eiger
lid10eiger1:~ # dd if=/dev/zero bs=8M count=4096 of=/nobackup/lid10eiger12/data/eiger/test.raw
4096+0 records in
4096+0 records out
34359738368 bytes (34 GB) copied, 32.3067 s, 1.1 GB/s

Conda volume

Reduce the data volume in order to create a conda volume and mount it on /opt/bliss/conda:

# as root

lid10eiger1:~ # (
    # create conda volume
    vg="vg"
    data_name="data"
    conda_name="conda"
    conda_size="20"
    conda_size_unit="GiB"
    conda_size_suffix=${conda_size_unit:0:1}
    part="/dev/mapper/${vg}-${data_name}"
    umount ${part}
    e2fsck -f ${part}
    curr_size_full=$(lvdisplay ${part} | grep 'LV Size' | awk '{print $3 " " $4}')
    curr_size=$(echo ${curr_size_full} | cut -d" " -f1)
    curr_size_unit=$(echo ${curr_size_full} | cut -d" " -f2)
    [ ${curr_size_unit} == ${conda_size_unit} ] || exit ]
    new_size=$(expr $(printf "%.0f" ${curr_size}) - ${conda_size})
    resize2fs ${part} ${new_size}${conda_size_unit:0:1}
    lvresize -L${new_size}${conda_size_suffix} ${part}
    e2fsck -f ${part}
    mount ${part}

    free_size_full=$(vgdisplay ${vg} |
                         grep 'Free \+PE / Size' | awk '{print $7 " " $8}')
    free_size=$(echo ${free_size_full} | cut -d" " -f1)
    free_size_unit=$(echo ${free_size_full} | cut -d" " -f2)
    [ ${free_size_unit} == ${conda_size_unit} ] || exit ]
    size_opt="-L ${conda_size}${conda_size_suffix}"
    [ $(printf "%.0f" ${free_size}) -eq ${conda_size} ] && \
      size_opt="-l 100%FREE"
    lvcreate -n ${conda_name} ${size_opt} ${vg}
    part="/dev/mapper/${vg}-${conda_name}"
    mkfs.ext4 ${part}
    part_dir="/opt/bliss/conda"
    mkdir -p ${part_dir}
    echo "${part} ${part_dir} ext4 relatime,nodev,nosuid 0 2" >> /etc/fstab
    mount ${part_dir}
    chown blissadm:bliss ${part_dir}
)

Network performance

Add opid00 user:

# as root
lid10eiger1:~ # mkuser opid00
...

Create netperf group and add affected users to it:

lid10eiger1:~ # groupadd netperf

lid10eiger1:~ # for u in ahoms opid00 opid10; do \
    usermod -a -G netperf ${u}; \
done

Allow netperf users to set real-time (SCHED_RR) scheduling policy with the highest priority:

lid10eiger1:~ # cat > /etc/security/limits.d/net-performance.conf <<'EOF'
@netperf         -       rtprio 99
EOF

Compile the netdev_set_queue_cpu_affinity util, used by the SlsDetector plugin to change the network packet dispatching tasks’ CPU affinity, and install it in /usr/local/bin:

lid10eiger1:~ # (
    cd /tmp
    prog_name="netdev_set_queue_cpu_affinity"
    cat > ${prog_name}.c <<'EOF'
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

int main(int argc, char *argv[])
{
        char *dev, *irq_queue, *p, fname[256], buffer[128];
        int irq, rps, fd, len, ret;
        long aff;

        if (argc != 5)
                exit(1);
        irq = (strcmp(argv[1], "-i") == 0);
        rps = (strcmp(argv[1], "-r") == 0);
        if (!irq && !rps)
                exit(2);
        if (!strlen(argv[2]) || !strlen(argv[3]) || !strlen(argv[4]))
                exit(2);

        dev = argv[2];
        irq_queue = argv[3];

        errno = 0;
        aff = strtol(argv[4], &p, 0);
        if (errno || *p)
                exit(3);

        len = sizeof(fname);
        if (irq)
                ret = snprintf(fname, len, "/proc/irq/%s/smp_affinity",
                               irq_queue);
        else
                ret = snprintf(fname, len, "/sys/class/net/%s/queues/%s/rps_cpus",
                               dev, irq_queue);
        if ((ret < 0) || (ret == len))
                exit(4);

        len = sizeof(buffer);
        ret = snprintf(buffer, len, "%016lx", aff);
        if ((ret < 0) || (ret == len))
                exit(5);

        fd = open(fname, O_WRONLY);
        if (fd < 0)
                exit(6);

        for (p = buffer; *p; p += ret)
                if ((ret = write(fd, p, strlen(p))) < 0)
                        exit(7);

        if (close(fd) < 0)
                exit(8);
        return 0;
}
EOF

    gcc -Wall -o ${prog_name} ${prog_name}.c
    rm -f ${prog_name}.c
    mv ${prog_name} /usr/local/bin
)

Allow netperf users to execute sudo in order to change other tasks’ CPU affinity (taskset and netdev_set_queue_cpu_affinity), to configure the network devices (ethtool and ifconfig) and start/stop system services like irqbalance (service):

lid10eiger1:~ # cat > /etc/sudoers.d/netperf <<'EOF'
%netperf        ALL=(root) NOPASSWD: /usr/bin/taskset, /sbin/ethtool, \
                                     /sbin/ifconfig, \
                                     /usr/local/bin/netdev_set_queue_cpu_affinity, \
                                     /usr/sbin/service
EOF

Tune the OS network buffer sizes:

lid10eiger1:~ # cat > /etc/sysctl.d/net-performance.conf <<'EOF'
# Tune network buffers for UDP RX performance

# Original values: sysctl -a | grep net
#...
#net.core.wmem_max = 131071
#net.core.rmem_max = 131071
#net.core.wmem_default = 229376
#net.core.rmem_default = 229376
#...
#net.core.netdev_max_backlog = 1000
#...
#net.ipv4.tcp_mem = 1549845 2066462 3099690
#net.ipv4.tcp_wmem = 4096   16384   4194304
#net.ipv4.tcp_rmem = 4096   87380   6291456
#...
#net.ipv4.udp_mem = 1549845 2066462 3099690
#net.ipv4.udp_rmem_min = 4096
#net.ipv4.udp_wmem_min = 4096

# Max OS socket receive buffer size (in bytes) for all types
net.core.rmem_max = 134217728

# Size of per-device buffer (in packets) before Linux kernel dispatching
net.core.netdev_max_backlog = 262144
EOF

cmake & libnuma-dev

Conda includes cmake, needed to compile Lima. De-install the Debian 7 package, if present, or the manually installed cmake-3.8.0:

# as root
lid10eiger1:~ # p=$(dpkg --list cmake\* | grep '^ii' | awk '{print $2}'); \
    [ -n "${p}" ] && dpkg --purge ${p}
...

lid10eiger1:~ # \
    curr_cmake=$(which cmake)
    if [ -n "${curr_cmake}" ] && [ $(dirname ${curr_cmake}) == "/usr/local/bin" ]; then
        cmake_src=$(find ~ ~opid00 -type d -name cmake-3.8.0)
        [ -n "${cmake_src}" ] && cd ${cmake_src} && su -c "make uninstall"
    fi
...

The same applies to libnuma-dev:

# as root
lid10eiger1:~ # p=$(dpkg --list libnuma-dev | grep '^ii' | awk '{print $2}'); \
    [ -n "${p}" ] && dpkg --purge ${p}
...

Xsession

Xsession executes ssh-agent, which has the setgid bit set. This forces Linux to clear its LD_LIBRARY_PATH, and hence that of its descendant processes - the full X11 session. The following patch propagates the LD_LIBRARY_PATH configured at login (.bash_profile) to the X11 session:

# as root
lid10eiger1:~ # (
    Xsession_patch="/etc/X11/Xsession.d/80ld_library_path"
    [ -f ${Xsession_patch} ] || cat > ${Xsession_patch} <<'EOF'
# This file is sourced by Xsession(5), not executed.

# ensure LD_LIBRARY_PATH is propagated after ssh-agent is executed
STARTUP="${LD_LIBRARY_PATH:+env LD_LIBRARY_PATH=$LD_LIBRARY_PATH} $STARTUP"

# vim:set ai et sts=2 sw=2 tw=80:
EOF
)

Network configuration

Intel 10 Gigabit Converged Adapter

Interface association

Force eth2 and eth3 to be in PCI-Express slot #2 ports and eth4 and eth5 to be in slot #1.

First locate the Intel 10 Gigabit X520 Ethernet adapters (reported as Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection):

lid10eiger1:~ # lspci | grep Ethernet
02:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
07:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
07:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

Then check the PCI tree:

lid10eiger1:~ # lspci -t
-+-[0000:ff]-+-08.0
...
 +-[0000:80]-+-01.0-[81]--
...
 +-[0000:7f]-+-08.0
...
 \-[0000:00]-+-00.0
...
             +-02.0-[02-03]--+-00.0
             |               \-00.1
             +-02.2-[04]--
             +-03.0-[05-06]--+-00.0
             |               \-00.1
             +-03.2-[07-08]--+-00.0
             |               \-00.1
...

From the tree we identify the parent root device of each Intel X520 Ethernet adapter:

Node

Parent

0000:02:00.x

0000:00:02.0

0000:05:00.x

0000:00:03.0

Find the PCI-e slot from the parent root port in the CPU:

lid10eiger1:~ # lspci -s 0:02.0 -vvv | grep Slot
    Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
        LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
            Slot #2, PowerLimit 25.000W; Interlock- NoCompl-
lid10eiger1:~ # lspci -s 0:03.0 -vvv | grep Slot
    Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
        LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
            Slot #1, PowerLimit 25.000W; Interlock- NoCompl-

This means that:

Adapter

Slot

0000:02:00.x

2

0000:05:00.x

1

So we must force the following association:

PCI-e Device

Network Device

0000:02:00.0

eth2

0000:02:00.1

eth3

0000:05:00.0

eth4

0000:05:00.1

eth5

This is obtained by the following udev network rules configuration:

lid10eiger1:~ # cat /etc/udev/rules.d/70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x8086:/sys/devices/pci0000:00/0000:00:03.2/0000:07:00.1 (ixgbe)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="0c:c4:7a:bc:d0:35", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:/sys/devices/pci0000:00/0000:00:03.2/0000:07:00.0 (ixgbe)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="0c:c4:7a:bc:d0:34", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.1 (ixgbe)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="90:e2:ba:86:28:65", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"

# PCI device 0x8086:/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0 (ixgbe)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="90:e2:ba:86:28:64", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

# PCI device 0x8086:/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.1 (ixgbe)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="90:e2:ba:86:2e:15", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth5"

# PCI device 0x8086:/sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0 (ixgbe)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="90:e2:ba:86:2e:14", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"

Reboot and verify that the association is OK:

lid10eiger1:~ # ls -l /sys/class/net/eth?/device
lrwxrwxrwx 1 root root 0 Sep  7 21:05 /sys/class/net/eth0/device -> ../../../0000:07:00.0
lrwxrwxrwx 1 root root 0 Sep  7 21:05 /sys/class/net/eth1/device -> ../../../0000:07:00.1
lrwxrwxrwx 1 root root 0 Sep  7 21:05 /sys/class/net/eth2/device -> ../../../0000:02:00.0
lrwxrwxrwx 1 root root 0 Sep  7 21:05 /sys/class/net/eth3/device -> ../../../0000:02:00.1
lrwxrwxrwx 1 root root 0 Sep  7 21:05 /sys/class/net/eth4/device -> ../../../0000:05:00.0
lrwxrwxrwx 1 root root 0 Sep  7 21:05 /sys/class/net/eth5/device -> ../../../0000:05:00.1

IP assignments

Assign the following addresses to the Eiger interfaces:

Network Device

Function

IP Address

eth2

Top-Half Control

192.168.11.1

eth3

Top-Half Data

192.168.12.1

eth4

Bottom-Half Control

192.168.13.1

eth5

Bottom-Half Data

192.168.12.1

For the 10 Gigabit data interfaces, we force:

  • MTU 9000

  • Rx adaptive interrupt moderation cycle of 100 usecs: /sbin/ethtool -C ethX rx-usecs 100

  • Rx ring buffer size of 4096 entries: /sbin/ethtool -G ethX rx 4096

The resulting /etc/network/interfaces file is:

lid10eiger1:~ # cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
allow-hotplug eth0
iface eth0 inet dhcp
#   up sleep 5; /sbin/ethtool -s eth0 autoneg off speed 100 duplex full
    up sleep 5; /sbin/ethtool -s eth0 autoneg on speed 1000 duplex full

# The secondary network interface
auto eth1
allow-hotplug eth1
iface eth1 inet static
        address 192.168.1.1
        netmask 255.255.255.0

# The 10 Gbps FO network interfaces - Top half
auto eth2
allow-hotplug eth2
iface eth2 inet static
        address 192.168.11.1
        netmask 255.255.255.0
auto eth3
allow-hotplug eth3
iface eth3 inet static
        address 192.168.12.1
        netmask 255.255.255.0
        mtu 9000
        up while /sbin/ethtool eth3 | grep 'Link detected' | grep -q no; do sleep 1; done; /sbin/ethtool -C eth3 rx-usecs 100; /sbin/ethtool -G eth3 rx 4096

# The 10 Gbps FO network interfaces - Bottom half
auto eth4
allow-hotplug eth4
iface eth4 inet static
        address 192.168.13.1
        netmask 255.255.255.0
auto eth5
allow-hotplug eth5
iface eth5 inet static
        address 192.168.14.1
        netmask 255.255.255.0
        mtu 9000
        up while /sbin/ethtool eth5 | grep 'Link detected' | grep -q no; do sleep 1; done; /sbin/ethtool -C eth5 rx-usecs 100; /sbin/ethtool -G eth5 rx 4096

PSI/Eiger modules

Define the PSI/Eiger module IPs (data interfaces are not actually needed):

lid10eiger1:~ # cat /etc/hosts
#============= OS ====================
127.0.0.1   localhost
127.0.1.1   lid10eiger1.esrf.fr lid10eiger1

#============= Eiger ====================
# Direct Connection - Top half
192.168.11.10   beb021.esrf.fr  beb021
#192.168.12.20  beb02110ge1.esrf.fr     beb02110ge1

# Direct Connection - Bottom half
192.168.13.11   beb020.esrf.fr  beb020
#192.168.14.21  beb02010ge1.esrf.fr     beb02010ge1

#============= OS ====================
# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Modify nsswitch.conf to first look at /etc/hosts when resolving names:

lid10eiger1:~ # cat /etc/nsswitch.conf
# /etc/nsswitch.conf
...
hosts:          files dns [NOTFOUND=return] mdns4_minimal mdns4
...

DHCP configuration

Install DHCP server software:

# as root
lid10eiger1:~ # apt-get install isc-dhcp-server
...
[FAIL] Starting ISC DHCP server: dhcpd[....] check syslog for diagnostics. ... failed!
 failed!
invoke-rc.d: initscript isc-dhcp-server, action "start" failed.
ldegjfrau1:~ # apt-get install isc-dhcp-server
...

Define the dynamic addresses and Eiger MAC/name relations:

lid10eiger1:~ # cat /etc/dhcp/dhcpd.conf
...
option domain-name "esrf.fr";
option domain-name-servers dns1.esrf.fr, dns2.esrf.fr;
...
# This is a very basic subnet declaration.

subnet 192.168.1.0 netmask 255.255.255.0 {
  range 192.168.1.128 192.168.1.191;
}

subnet 192.168.11.0 netmask 255.255.255.0 {
  range 192.168.11.128 192.168.11.191;
}

subnet 192.168.12.0 netmask 255.255.255.0 {
  range 192.168.12.128 192.168.12.191;
}

subnet 192.168.13.0 netmask 255.255.255.0 {
  range 192.168.13.128 192.168.13.191;
}

subnet 192.168.14.0 netmask 255.255.255.0 {
  range 192.168.14.128 192.168.14.191;
}

# PSI Eiger 500k detectors

host beb021 {
  hardware ethernet 00:50:c2:46:d9:2a;
  fixed-address beb021.esrf.fr;
}

host beb020 {
  hardware ethernet 00:50:c2:46:d9:28;
  fixed-address beb020.esrf.fr;
}
...

Specify the interfaces DHCP server will listen on:

lid10eiger1:~ # cat /etc/default/isc-dhcp-server
...
INTERFACES="eth2 eth4"

Reboot the computer for the changes to be applied. Verify that the dhcp server is running on the given interfaces:

lid10eiger1:~ # ps -ef | grep dhcpd | grep -v grep
root      3923     1  0 21:48 ?        00:00:00 /usr/sbin/dhcpd -q -cf /etc/dhcp/dhcpd.conf -pf /var/run/dhcpd.pid eth2 eth4

Restart the detector, wait for 20 sec and check that the links are OK:

lid10eiger1:~ # for i in $(seq 2 5); do n="eth${i}"; ifconfig ${n} | grep UP; done
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1

lid10eiger1:~ # ping -c 1 beb021; ping -c 1 beb020
PING beb021.esrf.fr (192.168.11.10) 56(84) bytes of data.
64 bytes from beb021.esrf.fr (192.168.11.10): icmp_req=1 ttl=64 time=0.357 ms

--- beb021.esrf.fr ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.357/0.357/0.357/0.000 ms
PING beb020.esrf.fr (192.168.13.11) 56(84) bytes of data.
64 bytes from beb020.esrf.fr (192.168.13.11): icmp_req=1 ttl=64 time=0.399 ms

--- beb020.esrf.fr ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.399/0.399/0.399/0.000 ms

SlsDetectors Software

ROOT installation

Install development packages necessary as root:

lid10eiger1:~ # apt-get install libxpm-dev libldap2-dev libmysqlclient-dev \
                               libavahi-client-dev libavahi-compat-libdnssd-dev \
                               libfftw3-dev graphviz-dev libxml2-dev libcfitsio3-dev
...

Unpack the ROOT sources on a user’s directory (~opid00), build it (12 parallel jobs: one per core), and install on /opt/root:

lid10eiger1:~/Downloads % mkdir -p ~/Downloads
lid10eiger1:~/Downloads % cd ~/Downloads
lid10eiger1:~/Downloads % scp lisgeiger1:Downloads/root_v5.34.34.source.tar.gz .
root_v5.34.34.source.tar.gz                   100%   72MB  71.7MB/s   00:01
lid10eiger1:~/Downloads % tar -xzf root_v5.34.34.source.tar.gz
lid10eiger1:~/Downloads % mkdir rootbuild
lid10eiger1:~/Downloads % cd rootbuild
lid10eiger1:~/Downloads/rootbuild % cmake ~/Downloads/root
...
lid10eiger1:~/Downloads/rootbuild % cmake --build . -- -j12
...
lid10eiger1:~/Downloads/rootbuild % su
Password:
lid10eiger1:/users/opid00/Downloads/rootbuild # cmake -DCMAKE_INSTALL_PREFIX=/opt/root -P cmake_install.cmake
...

Include ROOT initialisation script in the global /etc/profile.d chain:

lid10eiger1:~ # echo ". /opt/root/bin/thisroot.sh" > /etc/profile.d/root.sh

Qt4 environment

Do the same for Qt4:

lid10eiger1:~ # echo "QTDIR=/usr/share/qt4
export QTDIR" > /etc/profile.d/qt4.sh

Qwt development package installation

Qwt development package is needed by some applications in the SlsDetectorsSoftware:

lid10eiger1:~ # apt-get install libqwt-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  libqwt5-qt4-dev
The following NEW packages will be installed:
  libqwt-dev
0 upgraded, 1 newly installed, 1 to remove and 210 not upgraded.
Need to get 111 kB of archives.
After this operation, 35.8 kB of additional disk space will be used.
Do you want to continue [Y/n]? y
...

lid10eiger1:~ # dpkg --list libqwt\* | grep '^ii'
ii  libqwt-dev          6.0.0-1.2       amd64  Qt widgets library for technical applications (development)
ii  libqwt5-doc         5.2.2-3         all    Qt widgets library for technical applications (documentation)
ii  libqwt5-qt4         5.2.2-3         amd64  Qt4 widgets library for technical applications (runtime)
ii  libqwt6             6.0.0-1.2       amd64  Qt widgets library for technical applications (runtime)
ii  libqwtplot3d-qt4-0  0.2.7+svn191-7  amd64  3D plotting library based on Qt4/OpenGL (runtime)

Conda software installation

Before installing Conda, first stop the BLISS daemons:

# as root
lid10eiger1:~ # /users/blissadm/admin/etc/S70daemons stop

and delete the previous installation of the BLISS pew environment:

# as blissadm
lid10eiger1:~ % (
  rm -rf ${HOME}/.local/bin ${HOME}/.local/lib ${HOME}/lib
  sed -i '/\(PEW\|virtualenv\)/D' ${HOME}/local/BLISS_ENV_VAR
  find ${HOME}/bin -type l |
      while read l; do
          t=$(readlink ${l})
          if echo ${t} | grep -q "${HOME}/.local/bin"; then
              echo "Deleting ${l} -> ${t}"
              rm -f ${l}
          fi
      done
  )
Deleting /users/blissadm/bin/virtualenv-clone -> /users/blissadm/.local/bin/virtualenv-clone
Deleting /users/blissadm/bin/virtualenv -> /users/blissadm/.local/bin/virtualenv
Deleting /users/blissadm/bin/pythonz -> /users/blissadm/.local/bin/pythonz
Deleting /users/blissadm/bin/pythonz_install -> /users/blissadm/.local/bin/pythonz_install
Deleting /users/blissadm/bin/pew -> /users/blissadm/.local/bin/pew

Download and install conda:

# as blissadm
lid10eiger1:~ % (
    ln -s /opt/bliss/conda ${HOME}
    export http_proxy=http://proxy.esrf.fr:3128
    export https_proxy=${http_proxy}
    export no_proxy=.esrf.fr
    cat >> ${HOME}/.bash_profile <<EOF

# Proxy
export http_proxy=${http_proxy}
export https_proxy=${https_proxy}
export no_proxy=${no_proxy}
EOF
    cd ${HOME}/Downloads
    wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
    chmod a+x Miniconda2-latest-Linux-x86_64.sh
    ./Miniconda2-latest-Linux-x86_64.sh
    # installation directory: /users/blissadm/conda/miniconda
) && . ${HOME}/.bash_profile

Configure the conda BCU ESRF channel, update the software and create the slsdetector environment:

# as blissadm
lid10eiger1:~ % (
  cat > ${HOME}/conda/miniconda/.condarc <<'EOF'
channels:
  - http://bcu-ci.esrf.fr/stable
  - defaults
  - conda-forge
EOF
  ${HOME}/conda/miniconda/bin/conda update -n base conda
  ${HOME}/conda/miniconda/bin/conda create -n slsdetector-py37 python=3.7
)

Patch blissrc to use conda environments:

# as blissadm
lid10eiger1:~ % (
  cat > /tmp/blissrc.patch <<'EOF'
diff -Naur a/blissrc b/blissrc
--- a/blissrc   2018-08-09 20:42:27.798785071 +0200
+++ b/blissrc   2018-08-09 20:39:24.325593908 +0200
@@ -359,17 +359,65 @@
     export LD_LIBRARY_PATH
   fi

-  # Check if pew is required and load the good environment
+  # Check if pew/conda is required and load the good environment
   if [ -n "${BLISS_PEW_DEFAULT_ENV}" ] && [ -z "${BLISS_PEW_ENV}" ]; then
     BLISS_PEW_ENV=${BLISS_PEW_DEFAULT_ENV}
     export BLISS_PEW_ENV
   fi
+  if [ -n "${BLISS_CONDA_DEFAULT_ENV}" ] && [ -z "${BLISS_CONDA_ENV}" ]; then
+    BLISS_CONDA_ENV=${BLISS_CONDA_DEFAULT_ENV}
+    export BLISS_CONDA_ENV
+  fi
+  if [ -n "${BLISS_PEW_ENV}" -a -n "${BLISS_CONDA_ENV}" ]; then
+    print "Warning: both PEW and CONDA are configured. Ignoring PEW"
+    unset BLISS_PEW_ENV
+  fi
+  redo_cd=0
   if [ -n "${BLISS_PEW_ENV}" ] && [ "${BLISS_PEW_ENV}" != "NONE" ] && \
      [ -x "$(which pew)" ]; then
-    ve_dir=$(HOME=$BLISSADM pew in ${BLISS_PEW_ENV} bash -c "echo \$VIRTUAL_ENV") && \
-      . ${ve_dir}/bin/activate
+    ve_dir=$(HOME=$BLISSADM \
+             pew in ${BLISS_PEW_ENV} bash -c "echo \$VIRTUAL_ENV") && \
+             . ${ve_dir}/bin/activate
     unset ve_dir
-
+    redo_cd=1
+  fi
+  if [ -n "${BLISS_CONDA_ENV}" ] && [ "${BLISS_CONDA_ENV}" != "NONE" ]; then
+    this_prog=$(basename $(cd /proc/self && readlink exe))
+    if ([ -z "${CONDA_DEFAULT_ENV}" ] ||
+        [ "${CONDA_DEFAULT_ENV}" != "${BLISS_CONDA_ENV}" ]) && \
+       [ -f "${BLISS_CONDA_BASE}/bin/activate" ] && \
+       [ ${this_prog} == "bash" ]; then
+      if [ -n "${BLISS_CONDA_BASE}" ]; then
+        PATH=${BLISS_CONDA_BASE}/bin:${PATH}
+        export PATH
+      fi
+      . activate ${BLISS_CONDA_ENV} > /dev/null
+      CONDA_SYSROOT=$(echo ${CONDA_PREFIX}/*/sysroot)
+      export CONDA_SYSROOT
+      redo_cd=1
+      BLISS_CONDA_PREV_PYTHONPATH=${PYTHONPATH}
+    fi
+    unset this_prog
+    PYTHONPATH=$(echo ${PYTHONPATH} |
+                 sed -e "s,${BLISSADM}/python/bliss_modules\(/[^:]\+\)\?:,,g")
+    export BLISS_CONDA_PREV_PYTHONPATH
+  fi
+  if [ -n "${BLISS_CONDA_ENV}" ] && [ "${BLISS_CONDA_ENV}" == "NONE" ] && \
+     [ -n "${CONDA_DEFAULT_ENV}" ] && \
+     [ -f "${BLISS_CONDA_BASE}/bin/deactivate" ]; then
+    if [ $(basename $0) == "bash" ]; then
+      . deactivate > /dev/null
+      redo_cd=1
+    else
+      PATH=$(echo ${PATH} | sed "s,${CONDA_PREFIX}/bin:,,")
+      export PATH
+    fi
+    PATH=$(echo ${PATH} | sed "s,${BLISS_CONDA_BASE}/bin:,,")
+    PYTHONPATH=${BLISS_CONDA_PREV_PYTHONPATH}
+    export PATH PYTHONPATH
+    unset CONDA_SYSROOT BLISS_CONDA_PREV_PYTHONPATH
+  fi
+  if [ ${redo_cd} -ne 0 ]; then
     # redo the _esrf_cd if it is already defined (ESRF standard)
     if type _esrf_cd > /dev/null 2>&1; then
       _esrf_cd () {
@@ -377,6 +425,7 @@
       }
     fi
   fi
+  unset redo_cd

   # force BLISSADM [local/]bin to lead the PATH
   one_dir_re="${BLISSADM}/\(local/\)\?bin"
EOF
  (cd ${HOME}/bin && patch -b -p1 < /tmp/blissrc.patch)
)

Configure blissadm software to use conda with the default slsdetector-py37 environment:

# as blissadm
lid10eiger1:~ % \
  sed -i -e '/^BLISS_LIB_PATH/i\
BLISS_CONDA_BASE=${BLISSADM}/conda/miniconda export BLISS_CONDA_BASE' -e '/^BLISS_LIB_PATH/i\
BLISS_CONDA_DEFAULT_ENV=slsdetector-py37 export BLISS_CONDA_DEFAULT_ENV' ~/local/BLISS_ENV_VAR

Patch bliss_drivers and bliss_daemons so they do not use the conda environment:

# as blissadm
lid10eiger1:~ % (
  cat > /tmp/bliss_drivers.patch <<'EOF'
diff -Naur a/bliss_drivers b/bliss_drivers
--- a/bliss_drivers     2018-08-10 15:49:35.306633892 +0200
+++ b/bliss_drivers     2018-08-10 15:50:01.846229119 +0200
@@ -7,6 +7,9 @@
 # Author(s):    A. Homs (ahoms@esrf.fr)
 ##########################################################################

+BLISS_CONDA_ENV=NONE
+export BLISS_CONDA_ENV
+
 . blissrc

 COLUMNS=
EOF
  (cd ${HOME}/bin && patch -b -p1 < /tmp/bliss_drivers.patch)

  cat > /tmp/bliss_daemons.patch <<'EOF'
diff -Naur a/blisswatch b/blisswatch
--- a/blisswatch        2018-08-09 20:03:15.274806479 +0200
+++ b/blisswatch        2018-08-09 20:00:00.109181262 +0200
@@ -2,6 +2,9 @@
 # Copyright (c) 2004 Bliss group, ESRF, France
 # System startup script for HardwareRepositoryServer

+BLISS_CONDA_ENV=NONE
+export BLISS_CONDA_ENV
+
 . blissrc

 BLISSWATCHHOME=${BLISSADM}/admin/bliss_tools/blisswatch/
diff -Naur a/dserver_daemon b/dserver_daemon
--- a/dserver_daemon    2018-08-09 20:06:40.700207938 +0200
+++ b/dserver_daemon    2018-08-09 20:07:01.347931182 +0200
@@ -1,6 +1,9 @@
 #! /bin/ksh
 # Copyright (c) 2004 Bliss group, ESRF, France

+BLISS_CONDA_ENV=NONE
+export BLISS_CONDA_ENV
+
 . blissrc
 DSERVERHOME=${BLISSADM}/admin/bliss_tools/dserver
 exec ${DSERVERHOME}/dserver_daemon.py 2>&1 >/dev/null
EOF
  (cd ${HOME}/admin/daemon/src && patch -b -p1 < /tmp/bliss_daemons.patch)
)

Install basic conda packages:

# as blissadm
lid10eiger1:~ % (
  . blissrc
  conda install gxx_linux-64 gxx-dbg_linux-64 cmake \
                sip="4.19*" numpy gsl lz4-c=1.8.2 hdf5="1.10*" \
                libpng gevent
  conda install -c valkyriesystemscorporation libnuma
)

Restart the BLISS daemons:

# as root
lid10eiger1:~ # /users/blissadm/admin/etc/S70daemons start

BLISS software installation

Install PyTango, needed by Lima:

# as blissadm
lid10eiger1:~ % (. blissrc && conda install pytango)

Install the Python modules needed for building the HTML documentation with Doxygen, Sphinx and Read-the-Docs:

# as blissadm
lid10eiger1:~ (. blissrc && conda install sphinx_rtd_theme breathe)
...

Include the Python Scapy interface por network packet capture and decoding:

# as blissadm
lid10eiger1:~ % (. blissrc && pip install scapy)
...

Eiger calibration development: Seaborn and Spyder

The seaborn Python module and the spyder IDE for are used by Eiger calibration development (Erik Frojdh). First, de-install any OS installation:

# as root
lid10eiger1:~ # p=$(dpkg --list spyder\* | grep '^ii' | awk '{print $2}'); \
    [ -n "${p}" ] && dpkg --purge ${p}
# as blissadm
lid10eiger1:~ % (. blissrc && conda install seaborn spyder)
...

Detector software and development account: opid00

Define the Eiger software home

Add the eiger.sh entry in the system-wide Bash login setup scripts:

# as root
lid10eiger1:~ # cat > /etc/profile.d/eiger.sh <<'EOF'
EIGER_HOME=~opid00
export EIGER_HOME
EOF

Eiger environment setup

Create eiger_setup.sh, oriented to prepare the Eiger environment. In the beginning it just contains the BLISS environment:

# as opid00
lid10eiger1:~ % cat > eiger_setup.sh <<'EOF'
# Setup the Eiger data acquisition environment

# include the BLISS environment
. blissrc
EOF

and include it in the .bash_profile so it is executed at every login shell:

lid10eiger1:~ % cat >> .bash_profile <<'EOF'

# include the PSI/Eiger environment
. ${EIGER_HOME}/eiger_setup.sh
EOF

git-sig Bash helper

Add the git-sig Bash helper for authoring future commits:

lid10eiger1:~ % cat >> .bashrc <<'EOF'

# Signature: from dev-gitlab dot_bashrc

read_esrf_3612()
{
    echo $1 | nc 160.103.180.14 3612
}

git-sig()
{
    read_esrf=$(read_esrf_3612 $1)
    if [ "${read_esrf}" != "NO USER" ]; then
        GIT_AUTHOR_NAME=$(echo "${read_esrf}" | cut -d' ' -f4- | sed 's/"//g')
        GIT_AUTHOR_EMAIL="$1@esrf.fr"
    else
        GIT_AUTHOR_NAME="$1@esrf.fr"
        GIT_AUTHOR_EMAIL="$1@esrf.fr"
    fi

    export GIT_AUTHOR_NAME GIT_AUTHOR_EMAIL
    echo "Now git will use \"$GIT_AUTHOR_NAME\" to commits until SHELL ends"
}
EOF

Logout from opid00 and re-login so changes are taken into account for next steps.

Eiger-500k configuration

Copy the Eiger-500k detector configuration file and adapt to the new computer directories:

(slsdetector-py37) lid10eiger1:~ % \
    EIGER_DIR=${EIGER_HOME}/eiger/eiger_v3.1.1
    EIGER_CONFIG=${EIGER_DIR}/config/beb-021-020-direct-FO-10g.config
    mkdir -p $(dirname ${EIGER_CONFIG})
    scp lisgeiger1:${EIGER_CONFIG} $(dirname ${EIGER_CONFIG})
    sed -i 's:lisgeiger1:lid10eiger1:g' ${EIGER_CONFIG}
...

The resulting configuration file:

(slsdetector-py37) lid10eiger1:~ % cat ${EIGER_CONFIG}
detsizechan 1024 512

#type Eiger+
#top+bottom+
hostname beb021+beb020+

rx_hostname lid10eiger1

#port 1952
#stopport 1953

#top
0:rx_tcpport 1961
0:rx_udpport 50010
0:rx_udpport2 50011
0:rx_udpip 192.168.12.1
0:detectorip 192.168.12.22
0:detectormac 00:50:c2:46:d9:2b
0:flippeddatax 0

#bottom
1:rx_tcpport 1962
1:rx_udpport 50012
1:rx_udpport2 50013
1:rx_udpip 192.168.14.1
1:detectorip 192.168.14.23
1:detectormac 00:50:c2:46:d9:29
1:flippeddatax 1

settingsdir /users/opid00/eiger/eiger_v3.1.1/settingsdir/eiger
lock 0
#caldir /users/opid00/eiger/eiger_v3.1.1/settingsdir/eiger
outdir /nobackup/lid10eiger12/data/eiger

tengiga 1
threaded 1
flags parallel
iodelay 651

trimen 7 3000 3700 4500 5400 6400 8000 9900

index 250

Copy the detector calibration data:

(slsdetector-py37) lid10eiger1:~ % \
    SLS_DETECTOR_SETTINGS=$(grep ^settings ${EIGER_CONFIG} | awk '{print $2}')/standard
    mkdir -p $(dirname ${SLS_DETECTOR_SETTINGS})
    scp -r lisgeiger1:${SLS_DETECTOR_SETTINGS} $(dirname ${SLS_DETECTOR_SETTINGS})
...

Add the configuration file to eiger_setup.sh and decode the EIGER_MODULES, together with the calibration directory:

(slsdetector-py37) lid10eiger1:~ % cat >> eiger_setup.sh <<'EOF'

EIGER_DIR=${EIGER_HOME}/eiger/eiger_v3.1.1
EIGER_CONFIG=${EIGER_DIR}/config/beb-021-020-direct-FO-10g.config
EIGER_MODULES=$(grep "^hostname" ${EIGER_CONFIG} | cut -d" " -f2 | tr '+' ' ')
export EIGER_DIR EIGER_CONFIG EIGER_MODULES

SLS_DETECTOR_SETTINGS=$(grep ^settings ${EIGER_CONFIG} | awk '{print $2}')/standard
export SLS_DETECTOR_SETTINGS
EOF

Logout from opid00 and login again in order to apply the previous changes.

ESRF package for SlsDetectors

Install the [GitLab Hardware/sls_detectors project|https://gitlab.esrf.fr/Hardware/sls_detectors]:

(slsdetector-py37) lid10eiger1:~ % mkdir -p ~/esrf && cd ~/esrf
(slsdetector-py37) lid10eiger1:~/esrf % git clone -o gitlab git://gitlab.esrf.fr/Hardware/sls_detectors.git
Cloning into 'sls_detectors'...
...

Add the ESRF scripts to eiger_setup.sh:

(slsdetector-py37) lid10eiger1:~ % cat >> eiger_setup.sh <<'EOF'

SLS_DETECTORS=${EIGER_HOME}/esrf/sls_detectors
export SLS_DETECTORS
PATH=${SLS_DETECTORS}/eiger/scripts:${PATH}
export PATH
EOF

Logout and re-login as opid00 to have the previous environment set.

Lima installation in detector software account

First install flex, which might needed to compile some Lima subsystems:

# as root
lid10eiger1:~ # apt-get install flex
...

De-install libgsl and libnuma-dev:

# as root
lid10eiger1:~ # \
    dpkg --purge inkscape bogofilter bogofilter-bdb libgsl0ldbl libgsl0-dev
    dpkg --purge libnuma-dev
...

Lima is referenced as a submodule by the sls_detectors project installed before:

# as opid00
(slsdetector-py37) lid10eiger1:~ % \
    cd ${SLS_DETECTORS}
    git submodule init Lima
    git submodule update
    LIMA_DIR=${SLS_DETECTORS}/Lima
    cd ${LIMA_DIR}
    submod="third-party/Processlib
        third-party/bitshuffle
        camera/slsdetector
        applications/spec
        applications/tango/python"
    ext_submod_names="bitshuffle"
    ext_submod=$(for s in ${submod}; do
            for m in ${ext_submod_names}; do
                echo ${s} | grep ${m}
            done
        done)
    re_pat="(${ext_submod_names// /|})"
    gitlab_submod=$(echo "${submod}" | grep -Ev ${re_pat})
    git submodule init ${submod}
    git submodule update
    for s in ${gitlab_submod}; do
            (cd ${s} &&
                 git remote rename origin gitlab &&
                 git remote add github.bliss \
                     $(git config remote.gitlab.url |
                         sed "s%git://gitlab.esrf.fr/limagroup%git://github.com/esrf-bliss%"))
        done
    git remote rename origin gitlab
    git remote add github.bliss git://github.com/esrf-bliss/Lima.git
    git submodule foreach git fetch --all
    git fetch --all
...

Add Lima to environment variables in eiger_setup.sh:

(slsdetector-py37) lid10eiger1:~ % cat >> eiger_setup.sh <<'EOF'

LIMA_DIR=${SLS_DETECTORS}/Lima
export LIMA_DIR
EOF

Eiger software: slsDetectorPackage

The slsDetectorPackage is in turn a submodule of the Lima/camera/slsdetector plugin:

# as opid00
(slsdetector-py37) lid10eiger1:~ % \
    cd ${LIMA_DIR}/camera/slsdetector
    git submodule init
    git submodule update
    cd slsDetectorPackage
    git remote rename origin github.bliss
    git remote add github.slsdetectorgroup \
        git://github.com/slsdetectorgroup/slsDetectorPackage.git
    git fetch --all
...

Lima compilation

Compile Lima as blissadm, including slsDetectorPackage using the CMake commands in Conda:

# as opid00
(slsdetector-py37) lid10eiger1:~ % (
    cd ${LIMA_DIR}
    for d in third-party/Processlib . camera/slsdetector camera/slsdetector/tango; do
      (cd ${d} && mkdir -p build && chmod a+w build);
    done)
# as blissadm
lid10eiger1:~ % (
    . ${EIGER_HOME}/eiger_setup.sh
    cd ${LIMA_DIR}
    (PREFIX=${CONDA_PREFIX} \
     SP_DIR=$(python -c "import sysconfig; print(sysconfig.get_paths().get('platlib'))") \
     bash -c '(cd third-party/Processlib && bash -e conda/debug/build.sh) && \
              (cd . && bash -e conda/debug/build.sh) && \
              (cd applications/tango/python && \
                 bash -e conda/build.sh && cp scripts/Lima* ${PREFIX}/bin) && \
              (cd camera/slsdetector && \
                 bash -e conda/camera/build.sh && bash -e conda/tango/build.sh)'))
...

Build the documentation:

(slsdetector-py37) lid10eiger1:~/esrf/sls_detectors/Lima % make -C docs html
...

eigerDetectorServer and detector firmwares

If necessary, the eigerDetectorServer corresponding to the installed slsDetectorPackage version must be copied into the modules embedded Linux. Please refer to Installation of PSI/Eiger embedded server and firmwares

Test the slsdetector Lima plugin

Logout and re-login as opid00, so the previous changes can be tested. Test the Lima plugin without and with CtControl instantiation:

(slsdetector-py37) lid10eiger1:~ % \
    cd ${LIMA_DIR}
    (rm -f /tmp/eiger.edf &&
         build/camera/slsdetector/test/test_slsdetector -c ${EIGER_CONFIG})
...
(slsdetector-py37) lid10eiger1:~/esrf/sls_detectors/Lima % \
    mkdir -p /nobackup/lid10eiger12/data/eiger/lima
    ln -s /nobackup/lid10eiger12/data/eiger/lima data
    (rm -f data/img*.edf &&
         python camera/slsdetector/test/test_slsdetector_control.py -c ${EIGER_CONFIG})
...

Clean the shared memory segments used by the SlsDetector library, so thay can be re-created by opid10:

# as opid00
(slsdetector-py37) lid10eiger1:~ % \
    for m in $(ipcs -m | grep '^0x000016' | awk '{print $2}'); do
        ipcrm -m ${m}
    done

Setup opid10 account

Include the Eiger environment at login:

# as opid10
lid10eiger1:~ % cat >> .bash_profile <<'EOF'

# include the PSI/Eiger environment
. ${EIGER_HOME}/eiger_setup.sh
EOF

Lima Python Tango server configuration in blissadm

Use jive server wizard to add the LimaCCDs/eiger500k server to the Tango database:

(slsdetector-py37) lid10eiger1:~ % jive > /dev/null 2>&1 &

and start it:

# as opid10
(slsdetector-py37) lid10eiger1:~ % LimaCCDs eiger500k
...

Add LimaCCDs and SlsDetector class devices.

LimaCCDs/eiger500k/DEVICE/LimaCCDs

id10/limaccds/eiger500k

id10/limaccds/eiger500k->LimaCameraType

SlsDetector

id10/limaccds/eiger500k->NbProcessingThread

12

id10/limaccds/eiger500k->BufferMaxMemory

25

LimaCCDs/eiger500k/DEVICE/SlsDetector

id10/slsdetector/eiger500k

id10/slsdetector/eiger500k->config_fname

/users/opid00/eiger/eiger_v3.1.1/config/ beb-021-020-direct-FO-10g.config

id10/slsdetector/eiger500k->apply_corrections

0

id10/slsdetector/eiger500k->pixel_depth_cpu_affinity_map

{ 4: ((((CPU( 6), CPU( 7)), (CPU(18), CPU(19))),
((CPU( 9), CPU(10)), (CPU(21), CPU(22)))),
CPU(*chain(range(0, 6), range(12, 18))),
CPU(0),
((‘eth0,eth1,eth2,eth4,eth6,eth7,eth8,eth9’, {-1: (CPU(0), CPU(0))}),
(‘eth3’, {-1: (CPU( 8), CPU(20))}),
(‘eth5’, {-1: (CPU(11), CPU(23))}))),
8: ‘@4’,
16: ‘@4’,
32: ‘@4’}

Note

in order to perform high frame rate acquisitions, the CPU affinity must be fixed for the following tasks:

  • Receivers’ packet stream threads (passive mode equivalent to listeners): 2

  • Eiger processing threads: 2 per receiver, configurable

  • Lima processing threads

  • Other OS processes

  • Net-dev group packet dispatching for Rx queues: irq & processing

The previous example is based on a dual 6-core CPUs backend with Hyper-Threading Technology (12 cores, 24 threads). The configuration is optimised to work in pipe-line mode: cores/threads in Socket #1 (6-11, 18-23) will perform network data acquisition and basic image reconstruction (4-to-8 bit conversion, gap pixel insertion), and cores/threads in Socket #0 (0-5, 12-17) will perform Lima processing tasks. That’s why the NbProcessingThread is set to 12. Please note that there are three network groups and four pixel_depth->cpu_affinity settings (4-, 8-, 16- and 32-bit). The special global_affinity ‘@X’ is a reference to pixel_depth X.

Note

The Intel 10 Gigabit Ethernet Server Adapter has multiple hardware FIFOs per port, called queues in the OS terminology. The hardware uses a hash algorithm to dispatch packets into the active queues, which includes the source IP address. Each FIFO has an associated IRQ, so the irqbalance service activates the Receive-Side Scaling (RSS) mechanism by distributing the queues IRQ Service Routine (ISR) CPU affinity on different cores.

The destination FIFO in the Intel adapter depends on the Eiger 10 Gigabit Ethernet data interface IP, and thus the CPU where the corresponding ISR will run. ISRs have higher priority than packet dispatch tasks. If the queue IRQs are serviced by the same CPU that does the packet dispatching, the latter is affected when the frame rate is important. Care must be taken to avoid this kind of CPU conflict by having a deterministic CPU task distribution. With this aim, the Lima plugin sets not only the (network stack dispatching) Receive Packet Steering (RPS) CPU Affinity for both data interfaces eth3/eth5 but also their effective hardware FIFO ISR CPU Affinity (netdev_group CPU affinity).

Lima uses the ethtool -S command, which shows the statistics of a network device, including the bytes/packets received per FIFO, in order to determine the active FIFOs. Then it uses the files:

  • /proc/interrupts

  • /sys/class/net/<netdev>/device/msi_irqs/<irq>

  • /proc/irq/<irq>/smp_affinity_list

to find out the active device IRQs and to set their corresponding CPU affinities. If running, the irqbalance service is stopped before setting the IRQ affinity and restored during application cleanup.

SPEC

Install SPEC and CCD/Lima macros

Install the following packages with Blissinstaller:

  • Control/Spec/Core/SPEC

  • Control/Spec/Macros/ccd

  • Control/Spec/Macros/ccdbpm

  • Control/Spec/Macros/lima.mac

  • Applications/Analysis/Oxidis

Add a symbolic link to the Lima SlsDetector macros in the development version compiled on opid00:

# as blissadm
lid10eiger1:~ % (
    . ${EIGER_HOME}/eiger_setup.sh
    cd ~/spec/macros/lima
    ln -s ${LIMA_DIR}/applications/spec/limaslsdetector.mac
)

SPEC configuration

Include the ccd & Lima macros by default in SPEC:

# as blissadm
lid10eiger1:~ % cat ~/local/spec/macros/ID10setup.mac
...
need ccd
need lima/limacore
need lima/limaacq
need lima/limaroi
need lima/limatools
need lima/limasimulator
need lima/limafrelon
need lima/limaslsdetector
...

Create the eiger SPEC session:

# as blissadm
lid10eiger1:~/spec/macros/lima % spec_version add eiger
...

SPEC config file

Configure the LimaCCDs/eiger500k Taco interface server.

(slsdetector-py37) lid10eiger1:~ % cat ~blissadm/local/spec/spec.d/eiger/config
# ID @(#)getinfo.c  6.5  03/14/15 CSS
# Device nodes
PSE_MAC_MOT  = slsdetmot 32 eiger500k
SW_SFTWARE   = 1 POLL
VM_CCD_PC    = tango:id10/limatacoccds/eiger500k 2 TCP  @img_0
# CAMAC Slot Assignments
#  CA_name_unit = slot [crate_number]
# Motor    cntrl steps sign slew base backl accel nada  flags   mne  name
MOT000 = MAC_MOT:0/0   2000  1  2000  200   50  125    0 0x003    en_th  en_th
MOTPAR:read_mode = 7
MOTPAR:name = threshold_energy
# Counter   ctrl unit chan scale flags    mne  name
CNT000 =  SFTWARE  0  0 1000000 0x001      sec  Seconds
CNT001 =     NONE  0  0      1 0x000   imgall  imgall

SPEC setup file

Configure the LimaCCDs/eiger500k server control and specific interfaces.

(slsdetector-py37) lid10eiger1:~ % cat ~blissadm/local/spec/spec.d/eiger/setup
#
# Add or modify setup lines.
# Comment out the lines you want to cancel temporarily.
#

def lima_ccd_resetup_all '{
    local ccd_u ccd_dev

    _ccd_globals
    limasetup

    for (ccd_u = 0; ccd_u < CCDS; ccd_u++) {
        ccd_dev = image_par(ccd_u, "device_id")
        if (!image_par(ccd_u,"responsive") || (ccd_dev == "?"))
            continue

        ccdresetup ccd_u

        ##########################
        #     eiger500k
        #########################
        if (index(ccd_dev, "eiger500k") > 0) {
            limaccdsetup eiger500k ccd_u id10/limaccds/eiger500k
            taco_io(ccd_dev, "timeout", 30)
            tango_io("id10/slsdetector/eiger500k", "timeout", 30)
        }
    }
}'

lima_ccd_resetup_all

Note

the 30 seconds timeout is necessary for large memory allocations (long sequences)