Elasticsearch provides a distributed full-text, JSON-based search engine with a RESTful web interface.
With Elasticsearch you can index and search petabytes of data every day and get actionable insights in real-time from almost any type of structured or unstructured data source.
- High-performance bare metal servers that provide consistent, reliable performance over time.
- Direct memory access on bare metal instances.
- 40 Gbps per instance for high throughput across the cluster.
- Bare metal environment with no hypervisor to eat away system performance.
- All-SSD storage provides maximum IOPS and scalability.
It only takes a few clicks in the Control Center or an API call to setup an Elasticsearch cluster:
- We finely tune the software configuration for the underlying hardware of the selected instance types.
- The OS is optimized to provide the maximum possible performance.
- We configure Elasticsearch to automatically scale with the size of the underlying instance arrays. When the instance arrays are expanded, Elasticsearch is notified and shards are moved to the new instances. When the instance arrays are scaled down, shards are transferred before the instances are decommissioned.
We organize the Elasticsearch cluster in one InstanceArray:
The InstanceArray contains services that do both operational work and indexing and resource intensive queries. The instances in the instance array should be sized according to the dataset.
The architecture allows you to scale both vertically and horizontally extremely fast.
- Instance type: FMCI 16.64
- Solid Store Drive size: 150GB
- OS: Centos 7.x
- Instance count: 1
Scaling Elasticsearch on Metal Cloud
The bare metal cloud enables horizontal as well as vertical scaling of Elasticsearch at any time, as follows:
New instances can be added to the cluster within a minute; the hosts are registered in Elasticsearch and shards are relocated to the new nodes.
Instance configuration can be changed at any time with a simple restart; no additional delays are required for the data to be transferred out of each individual host.
Sizing an Elasticsearch Cluster
A few best practices to be taken into account when sizing an Elasticsearch cluster:
- Keep the active dataset in memory as much as possible.
- The number of master shards should match the number of data instances in the cluster.
- Elasticsearch’s memory should be 32GB.
The rest of the RAM will be used by Lucene as a filesystem cache and will yield additional performance improvements.
- FMCI 16.64 is a good balanced choice but you might go above as well with FMCI 32.128 to get extra CPU power.
- Elasticsearch recommends having at least 3 nodes while in critical production environments.
- Account owners have root access to all the machines.
- LAN networks can be used to improve the isolation of the Elasticsearch cluster.
- We pre-install the Jetty plugin but it is disabled by default.