Skip to main content

Author: sysadmin

CEPH: SATA and SSD pools on the same server without editing crushmap

sysadmin 3 Comments

I had some free slots in two on my ceph nodes and I used them to set a new SSD only pool. Because the ssd OSDs are sharing the server with an existing SATA pool we have to do some additional steps. There are some good posts out there that explain how to set up two pools sharing the same server, but you have to edit manually the ceph crushmap. Although it’s not very difficult to do it in that way I achieve the same goal setting the crush location parameter for those OSDs. I’ve tested it in Hammer release.

First create a new root bucket for the ssd pool. This bucket will be used to set the ssd pool location using a crush rule.

ceph osd crush add-bucket ssds root

We already have some servers with SATA OSDs in production, but we have to add two new host buckets for the faked hostnames that we are going to use to set the ssd OSDs.

ceph osd crush add-bucket ceph-node1-ssd  host
ceph osd crush add-bucket ceph-node2-ssd  host

Move the host buckets to the ssds root:

ceph osd crush move ceph-node1-ssd root=ssds
ceph osd crush move ceph-node2-ssd root=ssds

In the ceph configuration file (ceph.conf) set the crush location for the SSD OSDs. This is necessary because the default location is always the hostname obtained with the command hostname -s 

[osd.35]
host =  ceph-node1
osd_journal = /dev/disk/by-id/ata-INTEL_SSDSC2BB016T6_BTWA543204R11P6KGN-part1
crush_location =  root=ssds host=ceph-node1-ssd

You can check the location of the osd running this command:

$ ceph-crush-location --id 35 --type osd
 root=ssds host=ceph-node1-ssd

For each new ssd OSD move the osd to ssds root:

ceph osd crush add 35 1.5 root=ssds
ceph osd crush set osd.35 1.5 root=ssds host=ceph-node1-ssd

Create a new SSD pool:

ceph osd pool create ssdpool 128 128

Crate a crush rule in the ssds root:

ceph osd crush rule create-simple ssdpool ssds host

Finally assign the new rule to the ssdpool:

$ ceph osd pool set ssdpool crush_ruleset 4
 set pool 5 crush_ruleset to 4

Got! Now, we have a new only SSD pool:

$ ceph osd tree
ID  WEIGHT   TYPE NAME                       UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-25  3.00000 root ssds                                                    
-26  1.50000     host ceph-node1-ssd                                          
 35  1.50000         osd.35                       up  1.00000          1.00000 
-27  1.50000     host ceph-node2-ssd                                          
 36  1.50000         osd.36                       up  1.00000          1.00000 
-21 48.22978 root sata                                                    
-24  6.50995     host ceph-node1                                              
  1  0.92999         osd.1                        up  1.00000          1.00000 
  5  0.92999         osd.5                        up  1.00000          1.00000 
 16  0.92999         osd.16                       up  1.00000          1.00000 
 17  0.92999         osd.17                       up  1.00000          1.00000 
 18  0.92999         osd.18                       up  1.00000          1.00000 
 19  0.92999         osd.19                       up  1.00000          1.00000 
 20  0.92999         osd.20                       up  1.00000          1.00000 
-22  6.50995     host ceph-node2                                              
 21  0.92999         osd.21                       up  1.00000          1.00000 
 22  0.92999         osd.22                       up  1.00000          1.00000 
 23  0.92999         osd.23                       up  1.00000          1.00000 
 24  0.92999         osd.24                       up  1.00000          1.00000 
 25  0.92999         osd.25                       up  1.00000          1.00000 
 26  0.92999         osd.26                       up  1.00000          1.00000 
 27  0.92999         osd.27                       up  1.00000          1.00000 
 -7 13.29996     host ceph-node3                                              
  0  1.89999         osd.0                        up  1.00000          1.00000 
  6  1.89999         osd.6                        up  1.00000          1.00000 
  9  1.89999         osd.9                        up  1.00000          1.00000 
 11  1.89999         osd.11                       up  1.00000          1.00000 
 14  1.89999         osd.14                       up  1.00000          1.00000 
 15  1.89999         osd.15                       up  1.00000          1.00000 
  2  1.89999         osd.2                        up  1.00000          1.00000 
-20  6.50995     host ceph-node4                                              
 28  0.92999         osd.28                       up  1.00000          1.00000 
 29  0.92999         osd.29                       up  1.00000          1.00000 
 30  0.92999         osd.30                       up  1.00000          1.00000 
 31  0.92999         osd.31                       up  1.00000          1.00000 
 32  0.92999         osd.32                       up  1.00000          1.00000 
 33  0.92999         osd.33                       up  1.00000          1.00000 
 34  0.92999         osd.34                       up  1.00000          1.00000 
-14 15.39998     host ceph-node5                                              
  3  2.20000         osd.3                        up  1.00000          1.00000 
  4  2.20000         osd.4                        up  1.00000          1.00000 
  7  2.20000         osd.7                        up  1.00000          1.00000 
  8  2.20000         osd.8                        up  1.00000          1.00000 
 10  2.20000         osd.10                       up  1.00000          1.00000 
 12  2.20000         osd.12                       up  1.00000          1.00000 
 13  2.20000         osd.13                       up  1.00000          1.00000 
[...]

Changing osd configuration without restarting

sysadmin Leave a comment

If you want to change some settings in your OSDs but you don’t want to reboot them this is the way to do it. There are two ways to change the settings on the osd:

If you are in a server where a specific OSD is running on like the OSD.3 in the example, you can use ceph daemon command to get and set settings on the OSD:

# ceph daemon osd.3 config get osd_deep_scrub_interval
{
    "osd_deep_scrub_interval": "604800"
}

 ceph daemon osd.3 config set osd_deep_scrub_interval 4838400
{
"success": "osd_deep_scrub_interval = '4.8384e+06' "
}

If you want to change settings in remote OSDs or you want to change all OSDs at once you can use injectargs:

ceph tell osd.4 injectargs '--osd-deep-scrub-interval 4838400'
osd_deep_scrub_interval = '4.8384e+06'

To change all of them:

ceph tell osd.* injectargs '--osd-deep-scrub-interval 4838400'

Configurando DKIM con Postfix

sysadmin Leave a comment

DKIM es un standard de internet que permite relacionar un mensaje de correo con un nombre de dominio, utiliza un cifrado de clave asimétrica para poder validar un mensaje de correo con su emisor. El MTA del emisor firma los mensajes de correo con la clave privada y el receptor puede validar a partir de la clave pública obtenida del dominio del emisor que las cabeceras del mensaje no han sido alteradas.

Instalar OpenDKIM y sus dependencias:

sudo apt-get install opendkim opendkim-tools

Editar el fichero /etc/opendkim.conf

AutoRestart             Yes
AutoRestartRate         10/1h
UMask                   002
Syslog                  yes
SyslogSuccess           Yes
LogWhy                  Yes

Canonicalization        relaxed/simple

ExternalIgnoreList      refile:/etc/opendkim/TrustedHosts
InternalHosts           refile:/etc/opendkim/TrustedHosts
KeyTable                refile:/etc/opendkim/KeyTable
SigningTable            refile:/etc/opendkim/SigningTable

Mode                    sv
PidFile                 /var/run/opendkim/opendkim.pid
SignatureAlgorithm      rsa-sha256

UserID                  opendkim:opendkim

Socket                  inet:12301@localhost

Esto es un ejemplo de configuración que permite firmar mensajes para varios dominios, para ver con mayor detalle la configuración puedes ir aquí

Conectar el filtro milter con Postfix

editar el fichero /etc/default/opendkim y añadir la siguiente línea:

SOCKET="inet:12301@localhost"

Editar el fichero /etc/postfix/main.cf de postfix y asegurarse de que estas opciones existen:

milter_protocol = 2
milter_default_action = accept

Si no tienes más filtros milter configurados añadir las siguientes líneas:

smtpd_milters = inet:localhost:12301
non_smtpd_milters = inet:localhost:12301

Crear la estructura de directorios para albergar las claves y los ficheros de configuración de OpenDKIM:

sudo mkdir -p /etc/opendkim/keys

Especificar el fichero de servidores de confianza:

vim /etc/opendkim/TrustedHosts

127.0.0.1
localhost
192.168.0.1/24

*.example.com

#*.example.net
#*.example.org

Crear el fichero KeyTable con la tabla de claves, la tabla contiene el par selector/dominio y la ruta a la clave privada a utilizar para firmar los mensajes:

vim /etc/opendkim/KeyTable

mail._domainkey.example.com example.com:mail:/etc/opendkim/keys/example.com/mail.private
mail._domainkey.example.es example.es:mail:/etc/opendkim/keys/example.es/mail.private

Crear el fichero SigningTable. El fichero se utiliza para especificar los dominios y sus selectores:

vim /etc/opendkim/SigningTable

*@example.com mail._domainkey.example.com
*@example.es mail._domainkey.example.es

Generar el par de claves para cada dominio:

cd /etc/opendkim/keys
mkdir example.com
cd example.com
opendkim-genkey -s mail -d example.com
chown opendkim:opendkim mail.private

Añade la clave pública al registro DNS
Por cada dominio abre el fichero mail.txt 

mail._domainkey IN TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDN12F+VM4TCEMm8/5vGjhT42Zo/UHbf+N6CZx5Aj3p20u1dR8mqeWLM3TqE+9EpvKsx4GKtrl/8QBL1g7ZmdluVSlz6AIMarDXnjqmKqN4dlpCj15bnOjiHxH6r/bpll36dJrlmRuOL61xnilxTydpWQ4uJtTel2eUG5zeUG1CnQIDAQAB" ; ----- DKIM key mail for example.com

Puedes probar que tu DNS responde correctamente a través del comando dig:

dig txt mail._domainkey.example.com

[...]
;; ANSWER SECTION:
mail._domainkey.example.com. 14400 IN	TXT	"v=DKIM1\; k=rsa\; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDN12F+VM4TCEMm8/5vGjhT42Zo/UHbf+N6CZx5Aj3p20u1dR8mqeWLM3TqE+9EpvKsx4GKtrl/8QBL1g7ZmdluVSlz6AIMarDXnjqmKqN4dlpCj15bnOjiHxH6r/bpll36dJrlmRuOL61xnilxTydpWQ4uJtTel2eUG5zeUG1CnQIDAQAB"

Una vez que está todo correcto reinicia el servicio de OpenDKIM y Postfix

# service opendkim restart
# service postfix restart

y para probar que la configuración es correcta envía un mail a la dirección check-auth@verifier.port25.com, el servicio nos responderá con un mensaje en el que se 

==========================================================
Summary of Results
==========================================================
SPF check:          pass
DomainKeys check:   neutral
DKIM check:         pass
Sender-ID check:    pass
SpamAssassin check: ham

Debian 8 Jessie The following signatures couldn’t be verified because the public key is not available

sysadmin 2 Comments

Trying the upgrade from Debian 7 to Debian 8 I got key warnings when I changed the sources.list

W: Error de GPG: http://security.debian.org jessie/updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9D6D8F6BC857C906 NO_PUBKEY 8B48AD6246925553
W: Error de GPG: http://ftp.fr.debian.org jessie Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B48AD6246925553 NO_PUBKEY 7638D0442B90D010 NO_PUBKEY CBF8D6FD518E17E1
W: Error de GPG: http://ftp.fr.debian.org jessie-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B48AD6246925553 NO_PUBKEY 7638D0442B90D0

I’ve fixed it running the following commands:

# wget -O - https://ftp-master.debian.org/keys/archive-key-8.asc | apt-key add -
# wget -O - https://ftp-master.debian.org/keys/archive-key-8-security.asc | apt-key add -

and get the public key:

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 8B48AD6246925553

or simply run this command:

aptitude install debian-keyring debian-archive-keyring

High Availability RabbitMQ cluster in OpenStack

sysadmin Leave a comment

The RabbitMQ service is the heart of the processes communication in OpenStack and in a PROD deployment you want to configure a rabbitmq cluster in order to achieve hight availability for the messages queues.

There are two types of RabbitMQ nodes, disk nodes and ram nodes. Ram nodes require less IOPS because the resource management is not written in disk, but in a cluster, at least, a disk node is required.

In this post I’m going to configure a three node cluster: one disk node and two ram nodes.

Installing the RabbitMQ server in all your nodes is as simple as running these commands:

$ sudo echo "deb http://www.rabbitmq.com/debian/ testing main" > /etc/apt/sources.list.d/rabbitmq.list
$ sudo wget -O - https://www.rabbitmq.com/rabbitmq-signing-key-public.asc | sudo apt-key add - 
$ sudo apt-get update
$ sudo apt-get install rabbitmq-server

To start from scratch stop the rabbitmq service in your nodes and reset the queues and configuration:

$ sudo rabbitmqctl stop_app
$ sudo rabbitmqctl reset
$ sudorabbitmqctl start_app

Note: After the cluster creation the ram nodes can be reset without problems, but the disk node cannot be reset because it is the only disk node in the cluster, to reset it you can do it removing the data from the disk:

rabbitmqctl stop_app
/etc/init.d/rabbitmq-server stop
rm -rf /var/lib/rabbitmq/mnesia/rabbit@node1*
/etc/init.d/rabbitmq-server start
rabbitmqctl start_app

Now, if you run the cluster status command you can see the cluster running with your disk node only:

rabbitmqctl cluster_status
Cluster status of node 'rabbit@node1' ...
[{nodes,[{disc,['rabbit@node1']},
         {ram,[]}]},
 {running_nodes,['rabbit@node1']},
 {cluster_name,<<"rabbit@node1">>},
 {partitions,[]}]

In the disk node create the user for the openstack services, set its permissions and set the cluster name:

$ sudo rabbitmqctl add_user openstack openstack_pass
$ sudo rabbitmqctl set_permissions -p / openstack ".*" ".*" "."
$ sudo rabbitmqctl set_cluster_name openstack

Set the queues ha policy to ensure that all queues except those with auto-generated names are mirrored across all running nodes:

rabbitmqctl set_policy ha-all '^(?!amq\.).*' '{"ha-mode": "all"}'

To make a RabbitMQ cluster all the members have to have the same Erlang cookie, find the cookie in the first node and copy it to the other nodes. The cookie is located at:

$ cat /var/lib/rabbitmq/.erlang.cookie
SRITXWMZBCBIRFZMQOAQ

Join the other two nodes to the cluster as a ram nodes, to do that run the following command in node2 and in node3:

$ sudo rabbitmqctl stop_app
$ sudo rabbitmqctl join_cluster --ram rabbit@node1
$ sudo rabbitmqctl start_app

The cluster is completed now:

$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit@node1' ...
[{nodes,[{disc,['rabbit@node1']},
         {ram,['rabbit@node2','rabbit@node3']}]},
 {running_nodes,['rabbit@node1','rabbit@node2',
                 'rabbit@node3']},
 {cluster_name,<<"openstack">>},
 {partitions,[]}]

As an additional step you can enable the rabbitmq management plugin in one or all of your nodes:

$ sudo rabbitmq-plugins enable rabbitmq_management

Create a new user for the management interface:

$ sudo rabbitmqctl add_user admin admin_pass
$ sudo rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"
$ sudo rabbitmqctl set_user_tags  admin administrator

And finally go to your browser an and type:

http://server-name:15672
user: admin
pass: admin_pass

Once you have configured your RabbitMQ cluster you can configure the OpenStack services to use the cluster and mirrored queues. Just in case you should configure check OpenStack documentation for each service:

[oslo_messaging_rabbit]
rabbit_hosts=node1:5672,node2:5672,node3:5672
rabbit_retry_interval=1
rabbit_retry_backoff=2
rabbit_max_retries=0
rabbit_ha_queues=true
rabbit_userid = openstack
rabbit_password = openstack_pass
amqp_auto_delete = true
amqp_durable_queues=True