Installation of PSI/Eiger embedded server and firmwares ======================================================= Requirements ------------ The following elements are required in order to perform an upgrade of the PSI/Eiger embedded server and firmwares (FWs): * The detector names are expected to be in registered in the *hosts* name service (*/etc/hosts*, DNS), and reachable through the IPv4 layer. * The *GitLab Hardware/sls_detectors* project is expected to be installed in *${SLS_DETECTORS_DIR}* (defaults to *~blissadm/local/sls_detectors*). The different Git submodules are recursiverly cloned, so *slsDetectorFirmware* and *Lima/camera/slsdetector/slsDetectorPackage* are available. * The *${SLS_DETECTORS_DIR}/common/scripts/det_env_setup.sh* is available and sourced, defining *LIMA_DIR*, *SLS_DETECTOR_NAME*, *SLS_DETECTOR_CONFIG*, *SLS_DETECTOR_MODULES*, *SLS_DETECTOR_LOG* and *SLS_DETECTOR_BACKEND_SETUP* Please refer to :doc:`installation` in order to fulfill these requirements. SSH keys -------- Backup the current PSI/Eiger-500k modules Linux root account SSH *authorized_keys* and check that the *opid00@lisgeiger1* SSH public key is included: :: lisgeiger1:~ % \ [ -n "${SLS_DETECTOR_MODULES}" ] || . ${SLS_DETECTORS_DIR}/common/scripts/det_env_setup.sh EIGER_MODULE_TOP=$(echo ${SLS_DETECTOR_MODULES} | cut -f1 -d" ") cat ${SLS_DETECTORS_DIR}/eiger/ssh/id_dsa.pub echo base_dir="${SLS_DETECTOR_LOG}/${SLS_DETECTOR_NAME}" this_dir="${base_dir}/$(date +%Y-%m-%d-%H%M)" mkdir -p ${this_dir} && cd ${this_dir} for m in ${SLS_DETECTOR_MODULES}; do ssh -x root@${m} cat .ssh/authorized_keys > ssh_authorized_keys_${m} done cat ssh_authorized_keys_${EIGER_MODULE_TOP} ssh-dss AAAAB3NzaC1kc3MAAACBALGVR0qC2i/HgaJl4fuiwmOVrq46Bz3bs+o3/jdw/dqMaPjx35Ha shyC4zS+2wHyZVSjwTMIbVT8LPsNMGxL40ZxqWaAUyzn0XnjJMe3XT7h+yyx+iLUXvyCK489PAwT0srE iWbGNeQTgEYiwX+jqezQTiwss2sgypOrrwIrGrZBAAAAFQDjOUdgHjbCc1UMW37Zu+7b/AV1cQAAAIAu tumVMJlCqWv30RRscEVMxGuv7UYanyMSnp+fI6pIfAKfcme/tGGKYiiF6biHVmDKcL+tnYloZvuDvfM+ T7k1vMIf9UcX9ZglJu+6B1N7ZRV8wO9ZKCHcTeVkNqbTu7clhsjjEjwO/EWQrPv7aZPSu8hcJr/dFfP3 AskvvG04rgAAAIEAjS61ZiP1iGoP0UhJCf4uHDnGra41mvMyqGBFwji93XON49UmVYXVFfYcWeoDzpaH JQguFttMCNduiXmZfDYITld+86c9aWCR6g7re977ElFTbutWe+isu/ZFINXOvDEHmBUKd7++4lGDCjsM NOQQmG/Ftsi4jE2iWBaI91oyTQI= opid00@lisgeiger1 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAxHKsWjHzIsIt0Pt1lkZA9Px5x0v6e2ZzCh+AEiQz1nk0 l7kU6K0IzKvYbAj6HnQB3/epYHz6SBRrpsMQcFPHgtQIKlTzeKHNqS6pHBKxKR77jGGq70i7SWZsSBXP 1/8QxUhV9CNbFJKAkEmvXa5ZO1fKgtiXYXR26X72foOKwZzOtPzOqz2IQ6icxqTMELK1H4R+s0Rvuurl lDibdzo5g3xuV9bu/HLYt19zUgIR0KoTzpZEyPSiNwNizUy5oqXiNL4ZdcCjsSwQ1lYDwbzrg5brQA8W onqu5dtK4/LSh97jJ7u2V7LCUPr1KZj7OUy1FOHg2MUUjJbJApAMAffXgQ== root@pc6698 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2cWXxdmZHHfBq0n7iE4aNB6iyHWIUSA1EbLbuCfGhSog cn+uTeRXdAQfBvHXAp2JbR+mX30ftMzq28TfWlMKPWvKgcEwh05WbhL9swRIj7+FTryxnZG+LDDpIAUg kaGEmrX/9gtQ0wBwsbWPPPuHP6rlvSfoX2QPl6DSEt4AJqKf2QJNfm1SdqpQZOK5bG77VcEt3GRob9Mu d4j9Y+Tb8CWnruDQl6+Z66ccWwjCI5mlhKkjzgkaJytURcvEP4y+htqpeW6maoXJmvtS900+SCv1dqTe lbiSd2hRMx2ElZ7WCdaspHwkxwYyDtaPyHrGYuunDUhLFk+nDBjCUcuFgQ== theidel@pc6698 ssh-dss AAAAB3NzaC1kc3MAAACBALeM8Rw5A/3q3oWqP8OuVU1H12nL+UxQiHlpQj/rQ1C02+O1x3SZ M32gJZo99s8uFb2YTlbzxYDroLdxw5Z1fM22Dsf9vUDcNcAWfXQK0JV2LIsNLJKC/s3/O51GwvtgxtC4 wMgG3MMe5kmyI34tJSuVMggcnBepssvH6oWGVNbJAAAAFQCbtQeAikwu20rUfn7eEiCOP/0p9wAAAIBE hXhaxl5Ht29VldHfYMMAjY3V1+d2XbleLl7ixOdQLcy2igssBR4w0QRz4V6p0ePCiIhzqiWf5Hvgc1u/ TBK0Rj9STI1qU1s8A6x/qdNDTbgebSeO6CNGqDTvhIdMgX7dUhgtwx4lzgbKtOtYSZ1XkMr1omwMAK9n K7xFCNu4IAAAAIBS0Is0ryjPYiOuEnWptKy9FiBYv2edS3vJ3Ln74HjzJ8FW3jT76xwEiBcaBtvXZ5kM VLyDtR3LR63i11Nnrg7nNJa43S4sng9dbBnO3f5/7Ylv2ku/vhATtMouVRn8c+xP1CMEYpGhSIw+OzrG xs33FGKr0xkL2yofwrDtHNRA/Q== theidel@pc6698 ssh-dss AAAAB3NzaC1kc3MAAACBALGVR0qC2i/HgaJl4fuiwmOVrq46Bz3bs+o3/jdw/dqMaPjx35Ha shyC4zS+2wHyZVSjwTMIbVT8LPsNMGxL40ZxqWaAUyzn0XnjJMe3XT7h+yyx+iLUXvyCK489PAwT0srE iWbGNeQTgEYiwX+jqezQTiwss2sgypOrrwIrGrZBAAAAFQDjOUdgHjbCc1UMW37Zu+7b/AV1cQAAAIAu tumVMJlCqWv30RRscEVMxGuv7UYanyMSnp+fI6pIfAKfcme/tGGKYiiF6biHVmDKcL+tnYloZvuDvfM+ T7k1vMIf9UcX9ZglJu+6B1N7ZRV8wO9ZKCHcTeVkNqbTu7clhsjjEjwO/EWQrPv7aZPSu8hcJr/dFfP3 AskvvG04rgAAAIEAjS61ZiP1iGoP0UhJCf4uHDnGra41mvMyqGBFwji93XON49UmVYXVFfYcWeoDzpaH JQguFttMCNduiXmZfDYITld+86c9aWCR6g7re977ElFTbutWe+isu/ZFINXOvDEHmBUKd7++4lGDCjsM NOQQmG/Ftsi4jE2iWBaI91oyTQI= opid00@lisgeiger1 Check that all the keys are identical: :: lisgeiger1:~/eiger/psi_eiger_500k_024_025/2018-04-01-1828 % \ md5sum ssh_authorized_keys_beb* 1c183bdaa3a2f27029fca84b9cb3b857 ssh_authorized_keys_beb024 1c183bdaa3a2f27029fca84b9cb3b857 ssh_authorized_keys_beb025 In case the *opid00@lisgeiger1* public key is not included (not the case before), add them in order to open SSH sessions automatically on the detector modules: :: lisgeiger1:~/eiger/psi_eiger_500k_024_025/2018-04-01-1828 % \ if grep -q 'opid00@lisgeiger1' ssh_authorized_keys_beb*; then echo "SSH keys already installed in detector ${SLS_DETECTOR_NAME}" else echo "Installing SSH keys in detector ${SLS_DETECTOR_NAME}" for m in ${SLS_DETECTOR_MODULES}; do ssh -x root@${m} sh -c '"cat >> .ssh/authorized_keys"' \ < ${SLS_DETECTORS_DIR}/eiger/ssh/id_dsa.pub done fi Also check that the SSH public host keys are identical (same Linux image): :: lisgeiger1:~/eiger/psi_eiger_500k_024_025/2018-04-01-1828 % \ for m in ${SLS_DETECTOR_MODULES}; do ssh-keygen -f ~/.ssh/known_hosts -F ${m} -l done # Host beb024 found: line 82 type RSA 1040 21:78:5d:39:d5:cc:92:7a:42:f8:4d:69:38:3b:40:40 |1|m9PYbOqjp0h4qI8tq9u9H8x7pKQ=|wXZou5Y2oMKiULF5ZOuBjV0U7oo= (RSA) # Host beb025 found: line 84 type RSA 1040 21:78:5d:39:d5:cc:92:7a:42:f8:4d:69:38:3b:40:40 |1|Hp2vpbDpXlxTjMSfxo+n+r3B+ZQ=|LaI693N3AKzUER5lgBOOwReHOpI= (RSA) *eigerDetectorServer* --------------------- Before transferring the new version of the *eigerDetectorServer*, keep track of the current versions stored on the modules: :: lisgeiger1:~/eiger/psi_eiger_500k_024_025/2018-04-01-1828 % \ server_dir="executables" server_name="eigerDetectorServer" server="${server_dir}/${server_name}" server_str=$(echo ${server} | sed 's:/:_:g') full_server="/home/root/${server}" for m in ${SLS_DETECTOR_MODULES}; do ssh -x root@${m} 'ls -l '${server}'*' \ > ls_${server_str}_${m}.out done cat ls_${server_str}_${EIGER_MODULE_TOP}.out echo for m in ${SLS_DETECTOR_MODULES}; do ssh -x root@${m} 'md5sum '${server}'*' \ > md5sum_${server_str}_${m}.out done cat md5sum_${server_str}_${EIGER_MODULE_TOP}.out echo md5sum md5sum_${server_str}_beb* -rwxr-xr-x 1 root root 280601 Jan 1 01:15 executables/eigerDetectorServer -rwxr-xr-x 1 root root 277442 Aug 26 2016 executables/eigerDetectorServer_bkp -rwxr-xr-x 1 root root 277442 Aug 26 2016 executables/eigerDetectorServerv2.0.5.14.3 -rwxr-xr-x 1 root root 280601 Jan 1 01:14 executables/eigerDetectorServerv2.3.0.16.2 4fca193db64ed991da785043e7769082 executables/eigerDetectorServer e8a39956bbcb4aac62f109188e8ddbb2 executables/eigerDetectorServer_bkp e8a39956bbcb4aac62f109188e8ddbb2 executables/eigerDetectorServerv2.0.5.14.3 4fca193db64ed991da785043e7769082 executables/eigerDetectorServerv2.3.0.16.2 754a871d0608c28aa7544230ca728f86 md5sum_executables_eigerDetectorServer_beb024.out 754a871d0608c28aa7544230ca728f86 md5sum_executables_eigerDetectorServer_beb025.out Kill the running servers and disable the automatic startup: :: lisgeiger1:~/eiger/psi_eiger_500k_024_025/2018-04-01-1828 % \ for m in ${SLS_DETECTOR_MODULES}; do ssh -x root@${m} killall ${server_name} done for m in ${SLS_DETECTOR_MODULES}; do ssh -x root@${m} sed -i '"s:^#\?\('${full_server}'\).*$:#\1 \&:"' \ /etc/init.d/board_com.sh done Force a filesystem *sync* on each host to make the changes persistent, just before power-cycling: :: lisgeiger1:~/eiger/psi_eiger_500k_024_025/2018-04-01-1828 % \ for m in ${SLS_DETECTOR_MODULES}; do ssh -x root@${m} sync done Power-cycle the detector and check that no *eigerDetectorServer* is running: :: lisgeiger1:~/eiger/psi_eiger_500k_024_025/2018-04-01-1828 % \ for m in ${SLS_DETECTOR_MODULES}; do \ ssh -x root@${m} 'ps -ef | grep '${server}' | grep -v grep'; \ done Backup the current version, and transfer the new version: :: lisgeiger1:~/eiger/psi_eiger_500k_024_025/2018-04-01-1828 % \ for m in ${SLS_DETECTOR_MODULES}; do ssh -x root@${m} 'mv '${server}' '${server}'_bkp' done SLS_DETECTOR_PACKAGE=${LIMA_DIR}/camera/slsdetector/slsDetectorPackage new_servers=$(cd ${SLS_DETECTOR_PACKAGE} && find -name ${server_name}v\*) (cd ${SLS_DETECTOR_PACKAGE} && md5sum ${new_servers}) echo new_server=${SLS_DETECTOR_PACKAGE}/$(echo "${new_servers}" | head -n 1) for m in ${SLS_DETECTOR_MODULES}; do scp ${new_server} root@${m}:${server_dir} done for m in ${SLS_DETECTOR_MODULES}; do ssh -x root@${m} "cp ${server_dir}/$(basename ${new_server}) ${server}" done 50ef053f1ddd0b49314479a558c9c330 ./slsDetectorSoftware/eigerDetectorServer/bin/eigerDetectorServerv3.1.1.16.0 50ef053f1ddd0b49314479a558c9c330 ./serverBin/eigerDetectorServerv3.1.1.16.0 eigerDetectorServerv3.1.1.16.0 100% 286KB 286.2KB/s 00:00 eigerDetectorServerv3.1.1.16.0 100% 286KB 286.2KB/s 00:00 Check that all is as expected: :: lisgeiger1:~/eiger/psi_eiger_500k_024_025/2018-04-01-1828 % \ cd this_dir="${base_dir}/$(date +%Y-%m-%d-%H%M)" mkdir -p ${this_dir} && cd ${this_dir} for m in ${SLS_DETECTOR_MODULES}; do ssh -x root@${m} 'ls -l '${server}'*' \ > ls_${server_str}_${m}.out done cat ls_${server_str}_${EIGER_MODULE_TOP}.out echo for m in ${SLS_DETECTOR_MODULES}; do ssh -x root@${m} 'md5sum '${server}'*' \ > md5sum_${server_str}_${m}.out done cat md5sum_${server_str}_${EIGER_MODULE_TOP}.out echo md5sum md5sum_${server_str}_beb* -rwxr-xr-x 1 root root 293085 Jan 10 02:35 executables/eigerDetectorServer -rwxr-xr-x 1 root root 280601 Jan 1 01:15 executables/eigerDetectorServer_bkp -rwxr-xr-x 1 root root 277442 Aug 26 2016 executables/eigerDetectorServerv2.0.5.14.3 -rwxr-xr-x 1 root root 280601 Jan 1 01:14 executables/eigerDetectorServerv2.3.0.16.2 -rwxr-xr-x 1 root root 293085 Jan 10 02:34 executables/eigerDetectorServerv3.1.1.16.0 50ef053f1ddd0b49314479a558c9c330 executables/eigerDetectorServer 4fca193db64ed991da785043e7769082 executables/eigerDetectorServer_bkp e8a39956bbcb4aac62f109188e8ddbb2 executables/eigerDetectorServerv2.0.5.14.3 4fca193db64ed991da785043e7769082 executables/eigerDetectorServerv2.3.0.16.2 50ef053f1ddd0b49314479a558c9c330 executables/eigerDetectorServerv3.1.1.16.0 4168a104e53ee71f763ed5f0e0b43859 md5sum_executables_eigerDetectorServer_beb024.out 4168a104e53ee71f763ed5f0e0b43859 md5sum_executables_eigerDetectorServer_beb025.out Force a another filesystem *sync*: :: lisgeiger1:~/eiger/psi_eiger_500k_024_025/2018-04-01-1927 % \ cd for m in ${SLS_DETECTOR_MODULES}; do ssh -x root@${m} sync done And finally perform a *paranoid* check after power-cycling the detector: :: lisgeiger1:~ % \ prev_dir=${this_dir} this_dir="${base_dir}/$(date +%Y-%m-%d-%H%M)" mkdir -p ${this_dir} && cd ${this_dir} for m in ${SLS_DETECTOR_MODULES}; do ssh -x root@${m} 'md5sum '${server}'*' \ > md5sum_${server_str}_${m}.out done cd .. for m in ${SLS_DETECTOR_MODULES}; do (diff ${prev_dir}/md5sum_${server_str}_${m}.out ${this_dir} && echo "${m} OK" || echo "${m} changed") done beb024 OK beb025 OK Firmware flash -------------- .. note:: older modules *beb021/020* (Eiger-500k #1) and *beb074/071/064/102/072/073/087/088* (Eiger-2M) use bigger Xilinx Virtex5 FX70T FPGAs in Front-End-Board (FEB). New modules like *beb024/025* (Eiger-500k #2) use in their FEBs Xilinx Virtex5 FX30T FPGAs. **Updated**: The module *beb024/025* FEBs were changed during its front-end update with Xilinx Virtex5 FX70T FPGAs. No automatic determination of the FPGA type can be performed by software. **Solved**: the *flash.config* file includes the *Feb* section with the *FpgaType*, which is used by the code below. The current FWs (v18 and later) allow entering into flash mode from the Linux environment, without the need of pressing the button in the rear panel. The latestversion of the *eiger_flash* utility exploits this and enters into flash mode automatically. .. note:: two BEB FWs variants allow using fiber optic or twisted-pair (copper) transceivers: *beb_fiber.bit* and *beb_copper.bit*. The good file must be specified in the command below. Run the *eiger_flash* utility to update the FEB left/right and BEB FWs, as well as the kernel image: :: lisgeiger1:~ % ( [ -n "${SLS_DETECTOR_MODULES}" ] || . ${SLS_DETECTORS_DIR}/common/scripts/det_env_setup.sh; base_dir="${SLS_DETECTOR_LOG}/${SLS_DETECTOR_NAME}"; detector_dir="${SLS_DETECTOR_CONFIG_DIR}/setup/${SLS_DETECTOR_BACKEND_SETUP}/detector"; fw_ver=$(cat ${detector_dir}/fw); fw_dir="${SLS_DETECTORS_DIR}/slsDetectorFirmware/binaries/eiger/${fw_ver}"; flash_config="${detector_dir}/flash.config"; fpga_type=$(python < /dev/null 2>&1 &' done Once verified that the new server runs fine with the new firmware, restore automatic startup: :: lisgeiger1:~ % \ for m in ${SLS_DETECTOR_MODULES}; do ssh -x root@${m} sed -i '"s:^#\?\('${full_server}'\).*$:\1 \&:"' \ /etc/init.d/board_com.sh done for m in ${SLS_DETECTOR_MODULES}; do ssh -x root@${m} sync done Power-cycle the detector and verify that the servers start automatically: :: lisgeiger1:~ % \ for m in ${SLS_DETECTOR_MODULES}; do \ ssh -x root@${m} 'ps -ef | grep '${server}' | grep -v grep'; \ done 961 root 0:00 /home/root/executables/eigerDetectorServer 965 root 0:00 /home/root/executables/eigerDetectorServer -stopserver 961 root 0:00 /home/root/executables/eigerDetectorServer 965 root 0:00 /home/root/executables/eigerDetectorServer -stopserver