elasticsearch, reindex using logstash

Well, I'm not an elasticsearch expert, not at all. And, as usual, my english sucks.

Let's say that you have indexed a bunch of stuff, and now you need to change an index type.
For instance, in my case, I've seen that a filed named "hostname" was splitted if it was containing a dash, like "tc-pi.pacs.mydomain" was splitted in two parts when creating graphs using kibana.
The solution, in order to avoid this hostname splitting, is to define "index" : "not_analyzed" in logstash mapping.

Well, reading around it is not possible to change mappings once the document were indexed.

So a solution, a workaround, thanks to this post is the following.

"Download" the old index

curl -XGET 'http://127.0.0.1:9200/dcmaudit/_mappings/'

Copy the result in a text editor for your convenience, then change the mapping, like

...
"hostname":{"type":"string", "index" : "not_analyzed"}
...

Create a new index:

curl -XPOST http://localhost:9200/dcmaudit2 -d '{"mappings":{"logs":{"properties":{"@timestamp":{"type":"date","format":"dateOptionalTime"},"@version":{"type":"string"},"ParticipantObjectIdentification2.ParticipantObjectTypeCode.displayName":{"type":"string", "index" : "not_analyzed"},"hostname":{"type":"string", "index" : "not_analyzed"},"message":{"type":"string"},"tags":{"type":"string"},"timestamp":{"type":"date","format":"dateOptionalTime"}}}}}'

Now let's create a logstash configuration file like this:

input {
  # We read from the "old" index
  elasticsearch {
    hosts => [ "localhost" ]
    port => "9200"
    index => "dcmaudit"
    size => 500
    scroll => "5m"
    docinfo => true
  }
}

filter {
  mutate {
    remove_field => [ "@timestamp", "@version" ]
  }
}

output {

elasticsearch {
    host => "localhost"
    port => "9200"
    protocol => "http"
    index => "dcmaudit2"
    index_type => "%{[@metadata][_type]}"
    document_id => "%{[@metadata][_id]}"
  }

      stdout {
        codec => rubydebug
      }
}

Launch logstash

./bin/logstash -f conf.json

Now all the stuff from one index (dcmaudit) will be copied to the new one (dcmaudit2).

At this point you can delete the old index.

curl -XDELETE localhost:9200/dcmaudit

If you want, and if you need it, you can run this task again, recreating the old index name (dcmaudit) but whit the new mapping, and then repeat the logstash task changing the input and the output index accordingly.

Oracle Java, SmartOS Centos 6 LX branded zone: error while loading shared libraries libjli.so

SmartOS 20150820T062742Z

centos-6 lx-dataset image_uuid b1e082f6-460d-11e5-991a-0bc7df451392 (version 20150819)

I don't know if this happens in other versions or linux distributions.

I've installed Java (using the tar.gz archive downloaded from Oracle). Java 6 and Java 7: same issue.

/usr/java/latest/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory

Weird:

 # ldd /usr/java/latest/bin/java

lx_vdso.so.1 =>  (0x00007fffff0b0000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffffee00000)

libjli.so => /usr/java/latest/bin/../lib/amd64/jli/libjli.so (0x00007ffffecf5000)

libdl.so.2 => /lib64/libdl.so.2 (0x00007ffffea00000)

libc.so.6 => /lib64/libc.so.6 (0x00007ffffe600000)

/lib64/ld-linux-x86-64.so.2 (0x00007fffff200000)

Solution:

Create the file /etc/ld.so.conf.d/java.conf and put the following line in it

/usr/java/latest/lib/amd64/jli/

Then run the /sbin/ldconfig command.

# /usr/java/latest/bin/java -version

java version "1.6.0_65"

Java(TM) SE Runtime Environment (build 1.6.0_65-b32)

Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04, mixed mode)

# ldd /usr/java/latest/bin/java         

lx_vdso.so.1 =>  (0x00007fffff0b0000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffffee00000)

libjli.so => /usr/java/latest/bin/../lib/amd64/jli/libjli.so (0x00007ffffecf5000)

libdl.so.2 => /lib64/libdl.so.2 (0x00007ffffea00000)

libc.so.6 => /lib64/libc.so.6 (0x00007ffffe600000)

/lib64/ld-linux-x86-64.so.2 (0x00007fffff200000)

How to expand GLPI entity tree sons by default

vi ajax/entitytreesons.php

...

 83                   $path['state'] = 'open';
...

119                   $path['state'] = 'open';

...

Nagios plugin: check oracle query

Here is a Nagios plugin useful to connect to an Oracle database, perform a query and check the result by a regular expression.

Please note: it works for me, but it is far from perfect.

https://github.com/alcir/nagios/tree/master/check_oracle_query

Examples

./check_oracle_query.sh -H 192.168.1.10 -u user -w pass -p 1521 -s SID -n ID -q "select * from table" -e "^[1-9].* rows selected"

./check_oracle_query.sh -H 10.0.0.47 -u user -w pass -p 1521 -s SID01 -n ID1 -q "select * from dual" -e "^X"

How-To: Ubuntu Kiosk

How-To: Ubuntu Kiosk

This article first appeared in Full Circle Magazine, issue #93.

If you work in a place open to the public, for example a library or a hospital, sometimes it could be useful to provide a computer to the visitors. Such a computer, even if equipped with a mouse and a keyboard, must be configured in a way that people should not be able to use all the system functions, but only a limited set of them. For instance, people could use the computer to access the Internet, or only an internal website providing information related to your company or your activity. And you must configure the system in order to prevent shell access, and inhibit the launch of all the other programs you don't wish to offer to the visitors.

Such a computer is a kiosk. (http://en.wikipedia.org/wiki/Interactive_kiosk).

You can eventually configure a PC attached to a large TV, without keyboard and mouse, in order to show information slides or videos. To achieve this last goal, you can take a look to Xibo instead, which is a digital signage content management system (http://xibo.org.uk/).

Installing

Let's start with installing Ubuntu server 14.04 LTS. Why Ubuntu server? Well, because, in your kiosk, you don't need a lot of software. Also note the use of the –no-install-recommends parameter used with apt-get.

Please keep in mind that after the configuration of the kiosk environment, it is supposed that you will access the computer from SSH, and not from the terminal.

So, let's install Ubuntu, enabling automatic security updates and installing Openssh server.

After the first reboot, log in as the (administrator) user you have created during the installation and then upgrade the system:

sudo apt update
sudo apt upgrade 

And install these packages:

sudo apt install --no-install-recommends xorg openbox firefox xscreensaver plymouth-theme-sabily cups 

Install CUPS if you plan to allow printing to your visitors.

Create a user, the kiosk user.

sudo useradd -m kiosk  
sudo mkdir /opt/kiosk

Set up the environment

In this configuration, you will not launch a desktop manager, but you will invoke X11 and start only the Firefox executable.

sudo vi /opt/kiosk/kiosk.sh

#!/bin/bash 
home_tar="/opt/kiosk/kiosk_home.tar"  
rewritehome() {         
  echo $home_tar         
  if [ -f $home_tar ]         
  then                 
    rm -rf $HOME/*                 
    tar xvf $home_tar -C /         
  fi
}  
xset -dpms 
xset s 
off openbox-session &
rewritehome
xscreensaver -nosplash &
while true; 
do         
  rewritehome         
  /usr/bin/firefox 
done

The script shown above will be invoked by upstart when the system boots up. It essentially replaces the kiosk user home directory before starting Firefox. And the Firefox executable is placed inside a loop, in order to relaunch it whenever it is closed.

sudo chmod +x /opt/kiosk/kiosk.sh

Now you need to create an upstart script:

sudo vi /etc/init/kiosk.conf 

start on (filesystem and stopped udevtrigger)  
stop on runlevel [06]  
console output
emits starting-x  
respawn  
exec sudo -u kiosk startx /etc/X11/Xsession /opt/kiosk/kiosk.sh

Since X needs root privileges to start, you need to issue this command and select “Anybody”.

sudo dpkg-reconfigure x11-common 

At this point we are nearly finished. You need to reboot the machine.

sudo reboot 

And Firefox should start automatically.

Make the desired changes now, like adding CA certificates, security devices, adding some kind of extensions, etc.

Finally you need to install the srkiosk extension https://addons.mozilla.org/en-US/firefox/addon/srkiosk/

Please carefully read the documents (http://blogoless.blogspot.it/2012/07/firefox-srkiosk-add-on-full-screen.html), in order to understand this handcrafted item and better suit it to your needs, like changing the home page, or showing the address bar.

Close Firefox. At this point Firefox will restart in a locked-down kiosk mode.

To continue the configuration, you have to remotely log in via SSH.

Last steps.

The use of the screensaver can be useful – not only to have a nice effect on the screen after an amount of time. It is useful to “reset” the session after a period of inactivity. For instance, it is nice to go to the home page, instead of leaving the kiosk in the last visited page by the last user. So you need to configure the screensaver only if you need to reset your kiosk periodically.

Place the xscreensaver configuration file in the kiosk user home directory. The timeout option is the one you need to change in order to suit your needs (in this example the kiosk will be reloaded every quarter of an hour).

sudo vi /home/kiosk/.xscreensaver

timeout:        0:15:00 
lock:           False 
verbose:        False 
splash:         False 
captureStderr:  True 
dpmsEnabled:    False 
dpmsQuickOff:   False 
mode:           one 
selected:       0 
programs: resetkiosk.sh -root 
\n\

Next, you need the script that xscreensaver will invoke after the period of inactivity (Listing 4). As said, xscreensaver, instead of launching a real screen saver, will actually run this script.

sudo vi /usr/lib/xscreensaver/resetkiosk.sh
#!/bin/bash
PID=`ps -u $UID -f |grep firefox |grep -v grep | awk '{ print $3 }'`
kill $PID

Snapshot Time

Finally, you need to take a snapshot of the kiosk user home directory. Such snapshot will overwrite the home directory every time the machine reboots, or when the kiosk session is reset, or when Firefox is closed.

So, remember that every time you make changes to the kiosk user home directory, you will need to make a tar of that directory, and place it in the appropriate place: take a look at the kiosk.sh script (previous page, top right).

rm -rf /home/kiosk/.cache/  
tar cpvf /opt/kiosk/kiosk_home.tar /home/kiosk/  

Configuring GRUB and disabling ttys  

Another useful task to accomplish is related to some hardening of the system. For instance, you can prevent naughty users from booting the system into single-user mode, or from switching between virtual consoles (the terminals you can use by pressing functions keys like CTRL+ALT+F1).  Let's start with GRUB. First of all, it needs to be as quiet (less verbose) as possible, just for aesthetic purposes (Listing 5).

sudo vi /etc/default/grub

... 
GRUB_TIMEOUT=0 
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash vga=current loglevel=0"
...

Then, you can set a password in order to disallow users to boot Linux into single-user mode. Use the command grub-mkpasswd-pbkdf2 to create an encrypted password, then edit some files.  Put the following lines at the end of /etc/grub.d/00_header

cat << EOF  
set superusers="root"
password_pbkdf2 root  <the_sting_obtained_from_the_grub-mkpasswd-pbkdf2_command>
EOF

At this point, a user and a password will be requested for each GRUB menu entry, even for booting Linux in the normal way; that is what we don't want. So add the –unrestricted parameter (shown below in black) in each menuentry line of the configuration files. For instance in /etc/grub.d/10_linux

...
   echo "menuentry '$(echo "$title" | grub_quote)'  --unrestricted ${CLASS} \$menuentry_id_option 'gnulinux-$version-$type-$boot_device_id' {" | sed "s    /^/$submenu_indentation/"
else
   echo "menuentry '$(echo "$os" | grub_quote)'  --unrestricted ${CLASS} \$menuentry_id_option 'gnulinux-simple-$boot_device_id' {" | sed "s/^/$submenu    _indentation/"
...

In this way, a user and a password will be asked only if the user tries to edit the GRUB menu entries, using the 'e' key.

Now you must issue the sudo update-grub command to make the changes effective.

At the end, you can disable virtual terminal consoles by deleting the tty files under the /etc/init directory:

sudo rm /etc/init/tty*

Further works and conclusion 

In order to harden the machine, you should set up a password in the BIOS, and prevent boot from CD/DVD or USB ports. And about USB, you should prevent the use of such ports at least by making the PC inaccessible from the users.

Other things to do? Configure a custom theme for the boot screen. Look at ways to disable possible keyboard shortcuts.

Instead of running Firefox, you might wish to run a video player or a pictures slide-show: you can give it a try by putting some other command in the while loop.

This paper is based on a post found on https://thepcspy.com/read/building-a-kiosk-computer-ubuntu-1404-chrome/

Maybe a kiosk solution could be better achieved, but as far as I know, I have not found any better alternatives. This solution works for my needs: maybe it could be useful to you or it could be a starting point for a better implementation.

SmartOS zlogin Resource temporarily unavailable

Unable to perform zlogin to a SmartOS zone

[root@gz ~]# zlogin 461026ec-a298-4172-ba14-ca392b13e471
[Connected to zone '461026ec-a298-4172-ba14-ca392b13e471' pts/6]
malloc: Resource temporarily unavailable
ld.so.1: login: fatal: /lib/libc.so.1: Resource temporarily unavailable

[Connection to zone '461026ec-a298-4172-ba14-ca392b13e471' pts/6 closed]

Dmesg says

[root@gz ~]# dmesg

2015-03-13T08:56:29.246047+00:00 gz tmpfs: [ID 518458 kern.warning] WARNING: /zones/461026ec-a298-4172-ba14-ca392b13e471/root/etc/svc/volatile: File system full, swap space limit exceeded
2015-03-13T08:56:51.747916+00:00 gz genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 14479 (zlogin)

Solution

[root@hcn4 ~]# vmadm update 461026ec-a298-4172-ba14-ca392b13e471 max_swap=4096

Autenticazione Apache con la carta sanitaria Regione Toscana

E' stata una tribolazione: andare a tentoni richiede del tempo.

Utilizzare questo script https://gist.github.com/3v1n0/e371f58162795e0635f2 che fa quanto scritto nei commenti:

Get Italian government Certification Authority certificates from used by

by various National Service SmartCards (Carta Nazionale dei Servizi- CNS) 

mkdir /var/tmp/certs
./parse-gov-certs.py --output-folder /var/tmp/certs

Qui ci saranno vari certificati di Certification Authority di vari enti italiani usati per firmare i certificati di varie smart card, fra cui quelli utili al nostro scopo.

cd /var/tmp/certs
cat Regione_Toscana_-_CA_Cittadini_Servizi_di_Certificazione_* >> /etc/pki/tls/certs/ca-CNS-bundle.crt

In Apache, aprire il file ssl.conf (per esempio) e nel virtualhost opportuno mettere:

<VirtualHost _default_:443>
...
SSLCACertificateFile /etc/pki/tls/certs/ca-CNS-bundle.crt
...
</VirtualHost>

Non sono riuscito a far funzionare la direttiva SSLCACertificatePath, ovvero indicare una directory dove mettere i certificati scaricati con lo script Python.

E per richiedere la verifica del certificato del client (quindi il browser chiede il PIN della carta e passa il cerificato al server), mettere sempre all'interno della definizione del virtual host, o in qualche direttiva Location o Directory:

SSLVerifyClient require
SSLOptions +ExportCertData +StdEnvVars
SSLVerifyDepth 4

Questa è certamente una configurazione di base.
Il client viene autorizzato semplicemente se il certificato che presenta è stato firmato da una di queste CA. 
E' possibile autorizzare le carte in base a specifici attributi definiti con le direttive SSLRequire, e probabilmente configurare una CRL (Certificate Revocation List).

Installing check_mk agent on a SmartOS global zone

The check_mk agent uses inetd (or xinetd).
On Solaris, and on SmartOS, inetd is a legacy facility.
In addition, on the SmartOS GZ, the stuff configured in the /etc directory doesn't survive to reboots, as you may know.

So, since I already use a method to set the hostname at boot time, as described in this blog https://nickebo.net/creating-a-custom-motd-in-the-smartos-global-zone/, I added a pair of lines in the postboot script.


echo "check_mk        6556/tcp" >> /etc/services
echo "check_mk stream tcp nowait root /usr/sbin/tcpd /opt/custom/bin/check_mk_agent" >> /etc/inet/inetd.conf
inetconv
inetconv -e


Then download the check_mk solaris agent from github and put it in the /opt/custom/bin/ directory.


Obviously, if you don't want to reboot the GZ, but enable the service right now, issue the same commands as above.

Remember to change the MK_LIBDIR and the MK_CONFDIR variables in the /opt/custom/bin/check_mk_agent script in order to use additional plugins or configurations.

Nagios NConf, Apache, PHP and systemd

I struggled a bit with NConf on CentOS 7.

Following the NConf example configuration files, in order to perform deployment, the default temporary location is /tmp

So, it was supposed that temporary files goes to /tmp, but no file was here.

The right location is /tmp/systemd-private-<randomchars>/tmp/

So, if you want things to work, select another temp dir: neither /var/tmp is a good place (there is a systemd-private directory)

How-To: dcm4chee on Ubuntu

This article first appeared in Full Circle Magazine, issue #89.

Disclaimer

Maybe this article is not intended for a wide range of people reading this magazine, as I will describe the installation of a niche software related to the medical and health-care enterprise world.  It is not a home software; indeed nobody has a positron emission tomography (PET) facility at home, have they?

Intro

Maybe I’m wrong, but reading magazines like this, I have the feeling that Ubuntu, Linux and FOSS in general are mainly perceived, by common people, as something related to or useful for:

• a hobby for geeks, computer enthusiasts and programmers
• a free and legal way to achieve, using a PC, day by day personal interests (an alternative to Windows and paid software) and tasks such as:
• multimedia and amusement: games, image editing and video playback 
• personal and home income management
• office software 
• and for professional use, it seems something strictly related to information technology, like something made by programmers for other programmers or IT staff:
• network tools
• web server and stuff like that 
• programming languages and development tools 
• anything strictly related to IT, computer science, network and so on
• and at last something related to business  

But there are other aspects of Linux and FOSS: the real world, the serious things. Like health care applications, as stated above.

So, in this article I will explain how to install dcm4chee on Ubuntu. Maybe, as I said before, no reader of this magazine will have the need or the knowledge to install and understand this software, but if you are a student, or if you know a dentist or a radiological study, maybe you could be inspired by knowing the existence of this software.

DICOM and PACS

DICOM (Digital Imaging and Communications in Medicine) “is a standard for handling, storing, printing, and transmitting information in medical imaging”.

PACS (Picture archiving and communication system) is a medical imaging technology which provides storage of and access to images from multiple modalities.

A modality is a source machine where patient’s images are acquired from, for example, an X-Ray machine or a Magnetic Resonance scanner. Some other actors of a PACS are: servers, storage (NAS, SAN, tapes), printers, viewers, CD/DVD robots, etc.

dcm4chee (http://dcm4che.org/) is a DICOM archive and image manager, that is the server side of a PACS system, the server where medical images (an x-ray, an ultrasound, a magnetic resonance) will be stored, and from which a radiologist will retrieve your examination in order to carry out a medical report.

“The application contains the DICOM, HL7 services and interfaces that are required to provide storage, retrieval, and workflow to a health-care environment”. It is written in Java and is prepackaged and deployed within the JBoss application server. It is open source and licensed under an MPL/GPL/LGPL triple license. There are also a lot of DICOM related tools provided by the same developers and community (called dcm4che toolkit), and some DICOM viewers, like Mayam, Oviyam and Weasis.

If you need help and assistance there is a Google group where a vibrant and nice community will be willing to give you some advice. On the other hand, the wiki lacks a little bit in documentation, that is some pages are dated or are incomplete.

Let's install it

Even if some willing person has packaged dcm4che in a “.deb” package, and even if I’m a lover of package management systems, in this case I feel better if I install this software by hand.

So, let’s see how to install dcm4chee on Ubuntu server 14.04 LTS.

First of all we need to install Java. I’m not sure if dcm4chee works with OpenJDK. And I’m not sure if it works with Java 1.7, so let’s install Oracle Java 1.6, using a PPA repository.

sudo apt-get install software-properties-common  
sudo add-apt-repository ppa:webupd8team/java 
sudo apt-get update  
sudo apt-get install oracle-java6-installer

We need to accept the license, and so on.  dcm4chee needs a database: Postgres, DB2, Oracle, MySQL, etc. If you want to try it without a RDBMS, you can opt for the HSQL package, but let’s install MySQL.

sudo apt-get install mysql-server 

Now we need to create a schema and grant all rights to a user.

$ mysql -uroot -p***** 
mysql> create schema pacsdb;  
mysql> grant all on pacsdb.* to pacs@localhost identified by 'pacs';  
mysql> flush privileges;  
mysql> \q

Now we need to download JBoss and dcm4chee zip files. Download JBoss 4.2.3.GA (don’t use a newer version!) from
http://sourceforge.net/projects/jboss/files/JBoss/JBoss-4.2.3.GA/jboss-4.2.3.GA-jdk6.zip/download
Download the latest release of dcm4chee-mysql, 2.18.0 at time of writing, from

http://sourceforge.net/projects/dcm4che/files/dcm4chee/

Put the zip files in a temporary folder, like /var/tmp/dcminstall and unzip them

sudo apt-get install unzip  
unzip dcm4chee-2.18.0-mysql.zip 
unzip jboss-4.2.3.GA-jdk6.zip 

Since we are on a 64-bit operating system, and the dcm4chee zip file contains a 32-bit version of compression library, we have to download the proper Java Advanced Imaging Image I/O Tools for linux 64-bit from
http://download.java.net/media/jai-imageio/builds/release/1.1/
and replace the Linux version of libclib_jiio.so

wget http://download.java.net/media/jai-imageio/builds/release/1.1/jai_imageio-1_1-lib-linux-amd64.tar.gz
tar xzvf jai_imageio-1_1-lib-linux-amd64.tar.gz
cp /var/tmp/install/jai_imageio-1_1/lib/libclib_jiio.so /var/tmp/install/dcm4chee-2.18.0-mysql/bin/native/libclib_jiio.so

Now we have to copy files from JBoss to Dcm4chee:

cd /var/tmp/install/dcm4chee-2.18.0-mysql/bin/  
./install_jboss.sh /var/tmp/install/jboss-4.2.3.GA

Now we have to import the provided SQL file in order to create the database tables and indexes:

cd /var/tmp/install/dcm4chee-2.18.0-mysql/sql/ 
mysql -upacs -ppacs pacsdb < create.mysql

And now we have to move the dcm4chee directory to a definitive place, like /opt, and then it is useful to create a symbolic link (so, in case of an upgrade, we don’t need to modify any script or anything else pointing to the dcm4chee directory):

cd /  
sudo mv /var/tmp/install/dcm4chee-2.18.0-mysql/ /opt/ 
sudo ln -s /opt/dcm4chee-2.18.0-mysql /opt/dcm4chee

It is a good idea to create a dcm4chee user and change the owner of the installation dir.

useradd dcm4chee 
chown -R dcm4chee /opt/dcm4chee-2.18.0-mysql

Now we can try to run dcm4chee in foreground:

su - dcm4chee -c /opt/dcm4chee/bin/run.sh

Using a web browser, connect to http://yourubuntuserver:8080/dcm4chee-web3
The default user is “admin” and the default password is “admin”.

In order to make changes to the default configuration, we need to use the JBoss jmx console http://yourubuntuserver:8080/jmx-console/, and this task could be painful since there are no manuals covering a step-by-step configuration: you have to dig inside the wiki and on the old forum or, as I said before, you can call for help in the mailing list.

More info: 

• wiki - http://www.dcm4che.org/confluence/display/proj/The+Project 
• mailing list - https://groups.google.com/forum/m/?hl=en#!forum/dcm4che 
• discontinued forum - http://forums.dcm4che.org