I decided to upgrade my Elasticsearch installation over 3 nodes - following the documentation in Rolling Upgrades

After the initial steps:

curl -XPUT 'localhost:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d'
  "transient": {
    "cluster.routing.allocation.enable": "none"
curl -XPOST 'localhost:9200/_flush/synced?pretty'

I upgraded elkserver1 with:

root@elkserver1:~# service elasticsearch stop
root@elkserver1:~# apt-get install elasticsearch
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  linux-headers-4.4.0-53 linux-headers-4.4.0-53-generic linux-headers-4.4.0-57 linux-headers-4.4.0-57-generic
  linux-image-4.4.0-53-generic linux-image-4.4.0-57-generic linux-image-extra-4.4.0-53-generic
Use 'sudo apt autoremove' to remove them.
The following packages will be upgraded:
1 upgraded, 0 newly installed, 0 to remove and 63 not upgraded.
Need to get 33.4 MB of archives.
After this operation, 236 kB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 elasticsearch all 5.2.0 [33.4 MB]
Fetched 33.4 MB in 3s (9,591 kB/s)
(Reading database ... 218942 files and directories currently installed.)
Preparing to unpack .../elasticsearch_5.2.0_all.deb ...
Unpacking elasticsearch (5.2.0) over (5.1.1) ...
Processing triggers for systemd (229-4ubuntu13) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up elasticsearch (5.2.0) ...

Configuration file '/etc/elasticsearch/elasticsearch.yml'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** elasticsearch.yml (Y/I/N/O/D/Z) [default=N] ?
Installing new version of config file /etc/elasticsearch/jvm.options ...
Installing new version of config file /usr/lib/systemd/system/elasticsearch.service ...
Installing new version of config file /etc/init.d/elasticsearch ...
Processing triggers for systemd (229-4ubuntu13) ...
Processing triggers for ureadahead (0.100.0-19) ...
root@elkserver1:~# service elasticsearch start

But it did not start - looking at the log I found:

[2017-02-07T19:56:57,523][ERROR][o.e.b.Bootstrap          ] Exception
java.lang.IllegalArgumentException: Plugin [x-pack] is incompatible with Elasticsearch [5.2.0]. Was designed for version [5.1.1]
	at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:108) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:292) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:131) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.node.Node.<init>(Node.java:297) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.node.Node.<init>(Node.java:232) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:241) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.bootstrap.Boots

DOH - Read the fucking manual - Step 4: Upgrade any plugins


So I removed and added the X-Pack again:

root@elkserver1:/usr/share/elasticsearch# bin/elasticsearch-plugin remove x-pack
root@elkserver1:/usr/share/elasticsearch# bin/elasticsearch-plugin install x-pack
-> Downloading x-pack from elastic
[=================================================] 100%
@     WARNING: plugin requires additional permissions     @
* java.lang.RuntimePermission accessClassInPackage.com.sun.activation.registries
* java.lang.RuntimePermission getClassLoader
* java.lang.RuntimePermission setContextClassLoader
* java.lang.RuntimePermission setFactory
* java.security.SecurityPermission createPolicy.JavaPolicy
* java.security.SecurityPermission getPolicy
* java.security.SecurityPermission putProviderProperty.BC
* java.security.SecurityPermission setPolicy
* java.util.PropertyPermission * read,write
* java.util.PropertyPermission sun.nio.ch.bugLevel write
* javax.net.ssl.SSLPermission setHostnameVerifier
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]

And restarted Elasticsearch, and It cam up alright.

After a short while, elkserver3 and elkserver2 got the same (x-pack excluded) - between node upgrades went aproximatly 10 minutes.

And then I enabled Shard allocation again:

curl -XPUT 'localhost:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d'
  "transient": {
    "cluster.routing.allocation.enable": "all"

During the entire upgrade, Cluster health was either red or yellow - and I was somewhat concerned about the state of everything in the cluster.

But it came back to green and the number of unassigned shards went towards a nice 0 (zero)

This actually costed me some data loss... first of all, after rebooting my elkserver1, the filebeat service did not start and I did not realized this for 2 days.

Before that realization, searching is the "syslog-*" index gave me "Courier Fetch: X of 5 shards failed" and looking at the Shards from the Upgrade time and onwards for the syslog-* index, size was closed to 0.

I never found the reason, but ended up deleting the "closed to 0 ones" in Kibana with "DELETE /syslog-dd.mm.yyyy" and then everything worked again. The X-Pack could be a possibility, but all other index'es works and have worked fine the entire time.

Also, the "Courier Fetch: X of 5 shards failed" is a common problem it seems, when googling it.

But I should have closed all Logstash instances before the upgrade....