How to print ZFS filesystems ordered by space used

How to print ZFS filesystems ordered by space used

zfs get -o value,name -Hp used|sort -n

Percona Monitoring Plugins for Nagios on SmartOS

Using Percona monitoring plugins inside a SmartOS branded zone, I get an error related to the local command.

percona-monitoring-plugins-1.1.7/nagios/bin/pmp-check-mysql-innodb[58]: local: not found [No such file or directory]

To solve this problem, perform the following command against every script (i.e. inside the nagios/bin/ directory).

sed -i 's/^\( \{2,4\}\)local /\1typeset /' *

SmartOS: how to clone a KVM machine

Get the disk0 filesystem of the VM to clone.

vmadm get uuid-originvm | json disks

[
  {
    "path": "/dev/zvol/rdsk/zones/uuid-originvm-disk0",
    "boot": true,
    "model": "virtio",
    "media": "disk",
    "image_size": 10240,
    "image_uuid": "d8e65ea2-1f3e-11e5-8557-6b43e0a88b38",
    "zfs_filesystem": "zones/uuid-originvm-disk0",
    "zpool": "zones",
    "size": 10240,
    "compression": "off",
    "refreservation": 10240,
    "block_size": 8192
  }
]

Take a snapshot

zfs snapshot zones/uuid-originvm-disk0@toclone

Create a json file in order to create the new VM.
vi targetvm.json
Please note: there are no disks.

{

  "alias": "targetvm",

  "hostname": "targetvm",

  "dns_domain": "mydomain.it",

  "brand": "kvm",

  "resolvers": [

    "192.168.0.2"

  ],

  "ram": "2048",

  "vcpus": "2",

  "nics": [

    {

      "nic_tag": "admin",

      "ip": "192.168.0.201",

      "netmask": "255.255.255.0",

      "gateway": "192.168.0.1",

      "model": "virtio",

      "primary": true

    }

  ]

}

vmadm create -f targetvm.json

Let's clone the filesystem of the origin VM.

zfs clone zones/uuid-originvm-disk0@toclone zones/uuid-targetvm-disk0

Create a json in order to add a disk to the new VM.
vi adddisk.json

{

   "add_disks": [

      {

          "media": "disk",

          "model": "virtio",

          "boot": true,

          "nocreate": true,

          "size": 10240,

          "path": "/dev/zvol/rdsk/zones/435ca788-4935-ead3-f4e4-94e6eb7dd4b6-disk0"

      }

   ]

}

Let's add the disk to the newly created VM.

vmadm stop uuid-targetvm -F

vmadm update uuid-targetvm -f adddisk.json

This is important.

zfs promote  zones/uuid-targetvm-disk0

From man zfs:

zfs promote clone-filesystem

    Promotes a clone file system to no longer be dependent on its "ori-

    gin" snapshot. This makes it possible to destroy  the  file  system

    that  the clone was created from. The clone parent-child dependency

    relationship is reversed, so that the origin file system becomes  a

    clone of the specified file system.

    The  snapshot  that  was cloned, and any snapshots previous to this

    snapshot, are now owned by the promoted clone. The space  they  use

    moves  from the origin file system to the promoted clone, so enough

    space must be available to  accommodate  these  snapshots.  No  new

    space  is  consumed  by this operation, but the space accounting is

    adjusted. The promoted clone must not have any conflicting snapshot

    names  of  its own. The rename subcommand can be used to rename any

    conflicting snapshots.

mod_jk apache 2.4 smartos zone

If, compiling mod_jk, you get

/usr/bin/bash: ../build/libtool: No such file or directory

Then

cd native/
./configure --with-apxs=/opt/local/bin/apxs
mkdir build
cp libtool build/
make
cp ./apache-2.0/mod_jk.so /opt/local/lib/httpd/

tomcat sloooow to start, smartos, kvm: urandom?

A fresh installation of CentOS 7 in a SmartOS KVM virtual machine using a dataset (     image_uuid: d8e65ea2-1f3e-11e5-8557-6b43e0a88b38).
CentOS updated just after the installation.

• apache-tomcat-7.0.64
• Java(TM) SE Runtime Environment (build 1.7.0_80-b15)

I noticed that Tomcat, at startup was very very very, help me, very slow to start.

I used strace to look at what was happening (even if I don't understand anything).

I got a lot of

[pid  1279] futex(0x7f5a880a7154, FUTEX_WAIT_BITSET_PRIVATE, 1, {348, 455527976}, ffffffff) = -1 ETIMEDOUT (Connection timed out)

[pid  1279] futex(0x7f5a880a7128, FUTEX_WAKE_PRIVATE, 1) = 0

Just googled around. I found something similar.

So the solution is: to add the following option to java in the tomcat startup script

-Djava.security.egd=file:/dev/urandom

 (like CATALINA_OPTS="$CATALINA_OPTS -Djava.security.egd=file:/dev/./urandom").

Who knows?

Oracle Java, SmartOS Centos 6 LX branded zone: error while loading shared libraries libjli.so

SmartOS 20150820T062742Z

centos-6 lx-dataset image_uuid b1e082f6-460d-11e5-991a-0bc7df451392 (version 20150819)

I don't know if this happens in other versions or linux distributions.

I've installed Java (using the tar.gz archive downloaded from Oracle). Java 6 and Java 7: same issue.

/usr/java/latest/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory

Weird:

 # ldd /usr/java/latest/bin/java

lx_vdso.so.1 =>  (0x00007fffff0b0000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffffee00000)

libjli.so => /usr/java/latest/bin/../lib/amd64/jli/libjli.so (0x00007ffffecf5000)

libdl.so.2 => /lib64/libdl.so.2 (0x00007ffffea00000)

libc.so.6 => /lib64/libc.so.6 (0x00007ffffe600000)

/lib64/ld-linux-x86-64.so.2 (0x00007fffff200000)

Solution:

Create the file /etc/ld.so.conf.d/java.conf and put the following line in it

/usr/java/latest/lib/amd64/jli/

Then run the /sbin/ldconfig command.

# /usr/java/latest/bin/java -version

java version "1.6.0_65"

Java(TM) SE Runtime Environment (build 1.6.0_65-b32)

Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04, mixed mode)

# ldd /usr/java/latest/bin/java         

lx_vdso.so.1 =>  (0x00007fffff0b0000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffffee00000)

libjli.so => /usr/java/latest/bin/../lib/amd64/jli/libjli.so (0x00007ffffecf5000)

libdl.so.2 => /lib64/libdl.so.2 (0x00007ffffea00000)

libc.so.6 => /lib64/libc.so.6 (0x00007ffffe600000)

/lib64/ld-linux-x86-64.so.2 (0x00007fffff200000)

SmartOS zlogin Resource temporarily unavailable

Unable to perform zlogin to a SmartOS zone

[root@gz ~]# zlogin 461026ec-a298-4172-ba14-ca392b13e471
[Connected to zone '461026ec-a298-4172-ba14-ca392b13e471' pts/6]
malloc: Resource temporarily unavailable
ld.so.1: login: fatal: /lib/libc.so.1: Resource temporarily unavailable

[Connection to zone '461026ec-a298-4172-ba14-ca392b13e471' pts/6 closed]

Dmesg says

[root@gz ~]# dmesg

2015-03-13T08:56:29.246047+00:00 gz tmpfs: [ID 518458 kern.warning] WARNING: /zones/461026ec-a298-4172-ba14-ca392b13e471/root/etc/svc/volatile: File system full, swap space limit exceeded
2015-03-13T08:56:51.747916+00:00 gz genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 14479 (zlogin)

Solution

[root@hcn4 ~]# vmadm update 461026ec-a298-4172-ba14-ca392b13e471 max_swap=4096

Installing check_mk agent on a SmartOS global zone

The check_mk agent uses inetd (or xinetd).
On Solaris, and on SmartOS, inetd is a legacy facility.
In addition, on the SmartOS GZ, the stuff configured in the /etc directory doesn't survive to reboots, as you may know.

So, since I already use a method to set the hostname at boot time, as described in this blog https://nickebo.net/creating-a-custom-motd-in-the-smartos-global-zone/, I added a pair of lines in the postboot script.


echo "check_mk        6556/tcp" >> /etc/services
echo "check_mk stream tcp nowait root /usr/sbin/tcpd /opt/custom/bin/check_mk_agent" >> /etc/inet/inetd.conf
inetconv
inetconv -e


Then download the check_mk solaris agent from github and put it in the /opt/custom/bin/ directory.


Obviously, if you don't want to reboot the GZ, but enable the service right now, issue the same commands as above.

Remember to change the MK_LIBDIR and the MK_CONFDIR variables in the /opt/custom/bin/check_mk_agent script in order to use additional plugins or configurations.

SmartOS: move (migrate) a VM to another Global Zone

To move a running virtual machine, either a zone or a KVM, from an hypervisor (Global Zone) to another, I usually follow these steps in an handicraft way.

In order to minimize downtime, there are 2 steps.
The first snapshot and transfer of the ZFS filesystem will be done without halting the VM.
In the second step we will shutdown the VM, take another snapshot, then we will send it using an incremental transfer, that is very fast.

First of all you need the UUID of the VM you want to move from a global zone to another GZ.

[root@gz1 /]#vmadm list
UUID                                  TYPE  RAM      STATE             ALIAS
561b686e-3119-4ab0-932e-20fc944fb001  KVM   512      running           vm1
e44f3c76-4acb-11e3-a536-a7cfa8b66838  OS    512      running           vm2
b803b5b9-bc86-4d0f-b450-16862e7bd7ed  OS    2048     running           vm3


Now we need the list of all the ZFS filesystem related to that VM.

[root@gz1 ~]# zfs list -o name | grep 561b686e-3119-4ab0-932e-20fc944fb001

zones/561b686e-3119-4ab0-932e-20fc944fb001
zones/561b686e-3119-4ab0-932e-20fc944fb001-disk0
zones/561b686e-3119-4ab0-932e-20fc944fb001-disk1
zones/cores/561b686e-3119-4ab0-932e-20fc944fb001

Now we must create a snapshot of every ZFS.
Note: at this time the VM doesn't need to be stopped, you can leave it up and running, just to minimize downtime.

[root@gz1 ~]# zfs snapshot zones/561b686e-3119-4ab0-932e-20fc944fb001@tosend
[root@gz1 ~]# zfs snapshot zones/561b686e-3119-4ab0-932e-20fc944fb001-disk0@tosend
[root@gz1 ~]# zfs snapshot zones/561b686e-3119-4ab0-932e-20fc944fb001-disk1@tosend
[root@gz1 ~]# zfs snapshot zones/cores/561b686e-3119-4ab0-932e-20fc944fb001@tosend

Now it is time to send these snapshots to the destination. It may require a lot of time.

[root@gz1 ~]# zfs send zones/561b686e-3119-4ab0-932e-20fc944fb001@tosend | ssh gz2.domain zfs receive -v zones/561b686e-3119-4ab0-932e-20fc944fb001[root@gz1 ~]# zfs send zones/561b686e-3119-4ab0-932e-20fc944fb001-disk0@tosend | ssh gz2.domain zfs receive -v zones/561b686e-3119-4ab0-932e-20fc944fb001-disk0[root@gz1 ~]# zfs send zones/561b686e-3119-4ab0-932e-20fc944fb001-disk1@tosend | ssh gz2.domain zfs receive -v zones/561b686e-3119-4ab0-932e-20fc944fb001-disk1
[root@gz1 ~]# zfs send zones/cores/561b686e-3119-4ab0-932e-20fc944fb001@tosend | ssh gz2.domain zfs receive -v zones/cores/561b686e-3119-4ab0-932e-20fc944fb001

After that, we have to stop the virtual machine.

[root@gz1 ~]# vmadm stop 561b686e-3119-4ab0-932e-20fc944fb001

And get additional snapshots of the ZFS filesystems, like before, giving them a different name. Now, these snapshots are potentially in a consistent state, since the operating system inside the virtual machine is not running.

[root@gz1 ~]# zfs snapshot zones/cores/561b686e-3119-4ab0-932e-20fc944fb001@tosend-last 
[root@gz1 ~]# zfs snapshot zones/561b686e-3119-4ab0-932e-20fc944fb001-disk1@tosend-last 
[root@gz1 ~]# zfs snapshot zones/561b686e-3119-4ab0-932e-20fc944fb001-disk0@tosend-last 
[root@gz1 ~]# zfs snapshot zones/561b686e-3119-4ab0-932e-20fc944fb001@tosend-last


Finally, we have to send these ZFS snapshots using incremental send. It will take a little bit of time.

[root@gz1 ~]# zfs send -i zones/561b686e-3119-4ab0-932e-20fc944fb001@tosend zones/561b686e-3119-4ab0-932e-20fc944fb001@tosend-last | ssh gz2.domain zfs receive -Fv zones/561b686e-3119-4ab0-932e-20fc944fb001
[root@gz1 ~]# zfs send -i zones/561b686e-3119-4ab0-932e-20fc944fb001-disk0@tosend zones/561b686e-3119-4ab0-932e-20fc944fb001-disk0@tosend-last | ssh gz2.domain zfs receive -Fv zones/561b686e-3119-4ab0-932e-20fc944fb001-disk0
[root@gz1 ~]# zfs send -i zones/561b686e-3119-4ab0-932e-20fc944fb001-disk1@tosend zones/561b686e-3119-4ab0-932e-20fc944fb001-disk1@tosend-last | ssh gz2.domain zfs receive -Fv zones/561b686e-3119-4ab0-932e-20fc944fb001-disk1

[root@gz1 ~]# zfs send -i zones/cores/561b686e-3119-4ab0-932e-20fc944fb001@tosend zones/cores/561b686e-3119-4ab0-932e-20fc944fb001@tosend-last | ssh gz2.domain zfs receive -v zones/cores/561b686e-3119-4ab0-932e-20fc944fb001

Some last operations.
Grab the line related to the VM from inside the /etc/zones/index in the source global zone, and paste it at the end of same file in the destination global zone.

[root@gz1 ~]# cat /etc/zones/index | grep 561b686e-3119-4ab0-932e-20fc944fb001


[root@gz2 ~]# echo "561b686e-3119-4ab0-932e-20fc944fb001:installed:/zones/561b686e-3119-4ab0-932e-20fc944fb001:561b686e-3119-4ab0-932e-20fc944fb001" >> /etc/zones/index

Finally, copy the xml configuration file from the source global zone, to the destination one.


[root@gz1 ~]#scp /etc/zones/561b686e-3119-4ab0-932e-20fc944fb001.xml gz2.domain:/etc/zones/561b686e-3119-4ab0-932e-20fc944fb001.xml 

At this point, you can boot the VM on the destination global zone, check if all is working as expected, then delete the old VM from the source global zone.

smartos pxe: operation not permitted

I have downloaded the latest SmartOS image (20140904T175324Z) suitable for PXE booting.
Unpacking such file, platform-20140904T175324Z.tgz, and following the steps like the ones described here, http://wiki.smartos.org/display/DOC/PXE+Booting+SmartOS, I stumbled in an error that prevent the server to boot. This problem had never happened with previous releases.

Operation not permitted (http://ipxe.org/410c613c)
Could not boot image: Operation not permitted (http://ipxe.org/410c613c)

Visiting the proposed link, and looking at Apache log file (I've configured iPXE to download the images via HTTP) I found where the problem was.

 [Wed Sep 10 xx:xx:xx 2014] [error] [client 192.168.56.123] (13)Permission denied: file permissions deny server access: /srv/tftp/images/smartos/20140904T175324Z/platform/i86pc/amd64/boot_archive


Untarring the image file, the permission on the boot_archive file was 600 and not 644 as expected, and as it has been up to now.

install gparted inside smartos Centos VM

Well. I'm not so virtuous to use parted inside a Centos VM to expand the filesystem.
By default the Centos Linux image dataset from SmartOS is large 10GB.
You can add another disk (it will be mounted in /data).
Or you can grow the ZFS volume related to disk0.

So, from the global zone grow the volume related to the Linux VM:

zfs set volsize=20G zones/<UUID>-disk0

I don't know, maybe you must stop the VM and start it again.

Now, from inside the VM, yes, you must install various rpms.
Firts of all the EPEL repository.

rpm --import https://fedoraproject.org/static/0608B895.txt
rpm -ivh https://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

yum install gparted
yum install xauth
yum install dejavu-lgc-sans-fonts.noarch

Now you can connect to the VM via ssh with the -X option (or -Y) and run gparted

Fifo: vm reported as "provisioning", but it is false

Fifo: https://project-fifo.net

Why a VM status is reported as "Provisioning (Running)" even if it is already provisioned, and it is (only) running?

<killfill> that's because fifo knows if a vm is provisioning if the file 'provisioning' exists.
<killfill> Probably when creating the VM, the tools didn't wipe that file
<killfill> or maybe because you created it manually.. :P
<killfill> it's in /zones/:uuid/root/svc/ something like that.
<killfill> /zones/<uuid>/root/var/svc/provisioning

So, rm /zones/<uuid>/root/var/svc/provisioning
Et voilà.

SmartOS: add (modify) the gateway to a zone

I often forget to add the gateway to a new created zone.

So, get the nic MAC address.

vmadm get zone-uuid | json -a nics.0.mac

52:d9:d7:ab:c7:56

Create a file, like /var/tmp/updategw.json

{
   "update_nics": [
      {
         "mac": "52:d9:d7:ab:c7:56",
         "gateway": "192.168.0.1"
      }
   ]
}

Stop the zone, then issue an update.

vmadm update zone-uuid < /var/tmp/updategw.json

Verify.

vmadm get <zone-uuid> | json -a nics.0

Start the zone.

watchman under SmartOS

"A file watching service.
Purpose
Watchman exists to watch files and record when they actually change. It can also trigger actions (such as rebuilding assets) when matching files change."
https://github.com/facebook/watchman

You can find pkgsrc definitions, useful to build the package under SmartOS (pkgsrc on SmartOS - zone creation and basic builds) and a binary package to be installed using pkg_add

pkg_add watchman-2.9.1nb2.tgz

This is the link: https://github.com/alcir/watchman-pkgsrc

xtrabackup_55: Error writing file 'UNOPENED' (Errcode: 32)

SmartOS and percona-cluster-5.5.31nb3

Error in /var/log/mysql/error.log

WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.0.5 4444 (20131210 11:16:26.461)
WSREP_SST: [ERROR] innobackupex finished with error: 3.  Check /var/mysql//innobackup.backup.log (20131210 11:16:27.891)
WSREP_SST: [ERROR] Cleanup after exit with status:22 (20131210 11:16:27.912)
131210 11:16:27 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup --role 'donor' --address '192.168.0.5:4444/xtrabackup_sst' --auth 'sst:secret' --socket '/tmp/mysql.sock' --datadir '/var/mysql/' --defaults-file '/opt/local/etc/my.cnf' --gtid '6ceed930-6185-11e3-ae0d-8e7504b63f54:2'
131210 11:16:27 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'donor' --address '192.168.0.5:4444/xtrabackup_sst' --auth 'sst:secret' --socket '/tmp/mysql.sock' --datadir '/var/mysql/' --defaults-file '/opt/local/etc/my.cnf' --gtid '6ceed930-6185-11e3-ae0d-8e7504b63f54:2': 22 (Invalid argument)
131210 11:16:27 [Warning] WSREP: 0 (node1): State transfer to 1 (node2) failed: -1 (Not owner)
 

And in /var/mysql//innobackup.backup.log you see

xtrabackup_55: Error writing file 'UNOPENED' (Errcode: 32) 

I've removed

 [sst]
streamfmt = xbstream

from /opt/local/etc/my.cnf

Install Baikal server on a SmartOS zone

Baïkal is a Lightweight CalDAV+CardDAV server. The home page is http://baikal-server.com/

Baïkal needs some PHP modules. I use Apache as web server.

pkgin in apache-2.4.6 ap24-php53-5.3.27
pkgin in php53-pdo-5.3.27
pkgin in php53-xmlrpc-5.3.27
pkgin in php53-pdo_sqlite-5.3.27 php53-pdo_mysql-5.3.27
pkgin in php53-dom-5.3.27

Now you must configure php.ini, adding the modules.


vi /opt/local/etc/php.ini

extension=xmlrpc.so
extension=pdo.so
extension=pdo_mysql.so
extension=pdo_sqlite.so
extension=mbstring.so
extension=dom.so

The rest is pretty simple, following the install guide.

SmartOS: purge pkgin

rm -rf /opt/local
rm -rf /var/db/pkgin

download old versions of SmartOS

Here https://download.joyent.com/pub/iso/ you can find older SmartOS versions.

MegaCLI on SmartOS

I'm working on a Dell R710 with PERC H700 controller.

# prtconf -v|grep -i raid
                        value='RAID controller'
                        value='MegaRAID SAS 2108 [Liberator]'

# prtconf -v|grep -i perc
                        value='PERC H700 Integrated'
...

Firts of all go to LSI website http://www.lsi.com/support/Pages/Download-Results.aspx and search by keyword "MegaCLI. Then download Latest MegaCLI (there is Solaris OS).

Unzip it and copy MegaCLI/MegaCli_Solaris/x86/MegaCli to your SmartOS global zone (under /opt/custom/bin for example).

Now edit it with vi (yes, with vi), search for mr_sas and change this word (the driver name) to dr_sas.

Without doing this the utility doesn't detect any controller.
i.e.

# ./MegaCli -adpCount                              

Controller Count: 0.

Exit Code: 0x00

Thanks to jacques, Nils and elijah on the smartos-discuss mailing list.

using send_nsca from SmartOS global zone

If you don't want to install pkgin in the global zone, but you want to use Nagios send_nsca command, this is how I did it.

Inside a zone install the nagios-nsca package.

pkgin in nagios-nsca-2.9.1nb2


From the global zone, cd to the zone zfs and copy these files in a place of your own.

cd /zones/<uuid>

cp opt/local/gcc47/x86_64-sun-solaris2.11/lib/amd64/libgcc_s.so.1 /opt/custom/nsca/

cp opt/local/gcc47/x86_64-sun-solaris2.11/lib/amd64/libgcc_s.so.1 /opt/custom/nsca/

cp opt/local/lib/libmcrypt.so.4 /opt/custom/nsca/


Create a configuration file

vi /opt/custom/nsca/send_nsca.cfg


Now set the LD_LIBRARY_PATH variable and run the command

export LD_LIBRARY_PATH=/opt/custom/nsca; echo "host;passive_check;0;All seems to be OK" | /opt/custom/nsca/send_nsca -H nagios.ip -p 5667 -d ";" -c /opt/custom/nsca/send_nsca.cfg