{"_id":"5af218a6d7f67b0003d8dfef","project":"5633ebff7e9e880d00af1a53","version":{"_id":"5a8fae0268264c001f20cc00","project":"5633ebff7e9e880d00af1a53","__v":4,"createdAt":"2018-02-23T06:00:34.961Z","releaseDate":"2018-02-23T06:00:34.961Z","categories":["5a8fae0268264c001f20cc01","5a8fae0268264c001f20cc02","5a8fae0368264c001f20cc03","5a8fae0368264c001f20cc04","5a8fae0368264c001f20cc05","5a8fae0368264c001f20cc06","5a8fae0368264c001f20cc07","5a8fae0368264c001f20cc08","5a8fae0368264c001f20cc09","5abaa7eb72d6dc0028a07bf3","5b8ee7842790f8000333f9ba","5b8ee8f244a21a00034b5cd9"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"","version_clean":"2.0.0","version":"2.0"},"category":{"_id":"5a8fae0368264c001f20cc07","version":"5a8fae0268264c001f20cc00","project":"5633ebff7e9e880d00af1a53","__v":0,"sync":{"url":"","isSync":false},"reference":false,"createdAt":"2015-11-12T21:22:16.300Z","from_sync":false,"order":8,"slug":"troubleshooting-common-errors","title":"Troubleshooting"},"user":"5637d336aa96490d00a64f81","githubsync":"","__v":0,"parentDoc":null,"updates":[],"next":{"pages":[],"description":""},"createdAt":"2018-05-08T21:37:42.711Z","link_external":false,"link_url":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":1,"body":"Troubleshooting and resolving connection issues can be time consuming and frustrating. This article aims to reduce the friction of resolving connection problems by offering suggestions for quickly identifying a root cause.\n[block:api-header]\n{\n  \"title\": \"First Steps\"\n}\n[/block]\nIf you're seeing connection errors while attempting to reach your Bonsai cluster, the first step is **Don't Panic.** Virtually all connection issues can be fixed in under 5 minutes once the root cause is identified.\n\nThe next step is to review the documentation on [Connecting to Bonsai](doc:connecting-to-elasticsearch), which contains plenty of information about creating connections and testing the availability of your Bonsai cluster.\n\nIf the basic documentation doesn't help resolve the problem, the next step is to read the error message very carefully. Often the error message contains enough information to explain the problem. For example, something like this may show up in your logs:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"Faraday::ConnectionFailed (Connection refused - connect(2) for \\\"localhost\\\" port 9200):\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\nThis message tells us that the client tried to reach Elasticsearch over [localhost](https://en.wikipedia.org/wiki/Localhost), which is a red flag (discussed [below](doc:connection-issues#connection-refused--connection-failed)). It means the client is not pointed at your Bonsai cluster.\n\nNext, look for an HTTP error code. Bonsai provides an entire article dedicated to explaining what different HTTP codes mean here: [HTTP Error Codes](doc:error-codes). Like the error message, the HTTP error code often provides enough diagnostic information to identify the underlying problem.\n\nLast, don't make any changes until you understand the error message and HTTP code (if one is present), and have read the relevant documentation. Most of these errors occur during the initial set up and configuration step, so also read the relevant Quickstart guide if one exists for your language / framework. \n[block:api-header]\n{\n  \"title\": \"Common Connection Issues\"\n}\n[/block]\nThere are issues which do not return HTTP error codes because the request can not be made at all, or because the request is not recognized by Elasticsearch. These issues are commonly one of the following:\n\n* [Connection Refused](doc:connection-issues#connection-refused) \n* [No handler found for URI](doc:connection-issues#no-handler-found-for-uri) \n* [DNS Resolution Problem](doc:connection-issues#name-or-service-not-known)\n* [Missing Authentication Headers](doc:connection-issues#missing-authentication-headers)\n* [Missing SSL Root Certificate](doc:connection-issues#missing-ssl-root-certificate)\n[block:api-header]\n{\n  \"title\": \"Connection Refused\"\n}\n[/block]\nThis error indicates that a request was made to a server that is not accepting connections.\n\nThis error most commonly occurs when your Elasticsearch client has not been properly configured. By default, many clients will try to connect to something like `localhost:9200`. This is a problem because Bonsai will never be running on your localhost or network, regardless of your platform. \n[block:callout]\n{\n  \"type\": \"info\",\n  \"title\": \"What is Localhost?\",\n  \"body\": \"Simply, [localhost](https://en.wikipedia.org/wiki/Localhost) can be interpreted as \\\"this computer.\\\" _Which_ computer is \\\"this\\\" computer is determined by whatever machine is running the code. If you're running the code on your laptop, then _localhost_ is the machine in front of you; if you're running the code on AWS or in Heroku, then the _localhost_ is the node running your application.\\n\\nBonsai clusters run in a private network that does not include any of your infrastructure, which is why trying to reach it via _localhost_ will always fail.\"\n}\n[/block]\n\n[block:callout]\n{\n  \"type\": \"info\",\n  \"title\": \"Why Port 9200?\",\n  \"body\": \"By default, Elasticsearch runs on port 9200. While you _can_ access your Bonsai cluster over port 9200, this is not recommended due to lack of encryption. (See [Protocols and Ports](doc:connecting-to-elasticsearch#protocols-and-ports)).\"\n}\n[/block]\nDirect users, Heroku users and Manifold users will all need to ensure their Elasticsearch client is pointed at the correct URL. Please see the appropriate documentation for more information:\n\n* [Direct Users](doc:getting-started-with-bonsaiio) \n* [Heroku Users](doc:getting-started-with-heroku) \n* [Manifold Users](doc:getting-started-with-manifold) \n\nThe solution to this problem is to ensure that the client is properly configured. \n\n### Ruby / Rails\n\nIf you’re using the [elastisearch-rails client](https://github.com/elasticsearch/elasticsearch-rails), simply add the following gem to your Gemfile:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"gem 'bonsai-elasticsearch-rails'\",\n      \"language\": \"ruby\"\n    }\n  ]\n}\n[/block]\nThe bonsai-elasticsearch-rails gem is a shim. All it does is configure your Elasticsearch client to load the cluster URL from an environment variable called `BONSAI_URL`. You can read more about it on the [project repository](https://github.com/omc/bonsai-elasticsearch-rails).\n\nIf you'd prefer to keep your Gemfile sparse, you can initialize the client yourself like so:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"# config/initializers/elasticsearch.rb\\nElasticsearch::Model.client = Elasticsearch::Client.new url: ENV['BONSAI_URL']\",\n      \"language\": \"ruby\"\n    }\n  ]\n}\n[/block]\nIf you opt for this method, make sure to add the `BONSAI_URL` to your environment. It will be automatically created for Heroku and Manifold users. Users managing their own application environment will need to run something like:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"$ export BONSAI_URL=\\\"https://randomuser:randompass:::at:::something-12345.us-east-1.bonaisearch.net\\\"\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\n### Other Languages / Frameworks\n\nThe Elasticsearch client is probably using the default `localhost:9200` or `127.0.0.1:9200` (127.0.0.1 is the IPv4 equivalent of \"localhost\"). You'll need to make sure that the client is configured to use the correct URL for your cluster, and that this configuration is not being overwritten somewhere.\n[block:api-header]\n{\n  \"title\": \"No handler found for URI\"\n}\n[/block]\nThe Elasticsearch API has a variety of endpoints defined, like `/_cat/indices`. Each of these endpoints can be called with at a specific [HTTP method](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods). This error simply indicates a request was made to an endpoint using the wrong method.\n\nHere are some examples:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"# Request to /_cat/indices with GET method:\\n$ curl -XGET https://user:pass@something-12345.us-east-1.bonsai.io/_cat/indices\\ngreen open test1           1 1 0 0  318b  159b \\ngreen open test2           1 1 0 0  318b  159b \\n\\n# Request to /_cat/indices with PUT method:\\n$ curl -XPUT https://user:pass@something-12345.us-east-1.bonsai.io/_cat/indices\\nNo handler found for uri [/_cat/indices] and method [PUT]\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\nThe solution for this issue is simple: use the correct HTTP method. The [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) will offer guidance on which methods pertain to the endpoint you're trying to use.\n[block:api-header]\n{\n  \"title\": \"Name or service not known\"\n}\n[/block]\nThe [Domain Name System (DNS)](https://en.wikipedia.org/wiki/Domain_Name_System) is a worldwide, decentralized and distributed directory service which translates human-readable domains like www.example.com in to network addresses like 93.184.216.119. When a client makes a request to a URL like \"google.com,\" the application's networking layer will use DNS to translate the domain to an IP address so that it can pass the request along to the right server.\n\nThe \"Name or service not known\" error indicates that there has been a failure in determining an IP address for the URL's domain. This typically implicates one of several root causes:\n\n1. There is a typo in the URL.\n2. There is a [TLD](https://en.wikipedia.org/wiki/Top-level_domain) outage.\n3. The Internet Service Provider (ISP) handling your application's traffic may have lost a DNS server, or is having another outage.\n\nThe first troubleshooting step is to carefully read the error and double-check that the URL (particularly the domain name) is spelled correctly. If this is your first time accessing the cluster, then a typo is almost certainly the problem.\n\nIf this error arises during regular production use, then there is probably a DNS outage. **DNS outages are outside Bonsai's control**. There are a couple types of outages, but the ones that have affected our users before are:\n\n## TLD outage\nA TLD is something like \".com,\" \".net,\" \".org,\" etc. A TLD outage affects all domains under the TLD. So an outage of the \".net\" TLD would affect all domains with the \".net\" suffix.\n\nFortunately, all Bonsai clusters can be accessed via either of two domains:\n\n* bonsai.io\n* bonsaisearch.net\n\nIf there is a TLD outage, you should be able to restore service by switching to the other domain. In other words, if your application is sending traffic to https://user:pass@something-12345.us-east-1.bonsai.io, and the \".io\" TLD goes down, you can switch over to https://user:pass@something-12345.us-east-1.bonsaisearch.net, and it will fix the error.\n\n## ISP Outage\nMany ISPs operate their own DNS servers. This way, requests made from a node in their network can get low-latency responses for IP addresses. Most ISPs also have a fleet of DNS servers, at minimum a primary and a secondary. However, this is not a requirement, and there have been instances where an ISP's entire DNS service is taken offline.\n\nThere are also multitenant services which have [Software Defined Networks (SDN) running Internal DNS (iDNS)](https://docs.microsoft.com/en-us/windows-server/networking/sdn/technologies/idns-for-sdn). Regressions in this software can also lead to application-level DNS name resolution problems.\n\nIf you've already confirmed the domain is correct and swapped to another TLD for your cluster, and you're _still_ having issues, then you are probably dealing with an ISP/DNS or SDN/iDNS outage. One way to confirm this is to try making requests to other common domains like google.com. A name resolution error on google.com or example.com points to a local DNS problem.\n\nIf this happens, there is basically nothing you can do about it as a user, aside from complaining to your application sysadmin.\n[block:api-header]\n{\n  \"title\": \"Missing Authentication Headers\"\n}\n[/block]\nUsers who are seeing persistent [HTTP 401 Error Codes](doc:error-codes#http-401-authorization-required) may be using a client that is not handling authentication properly. As explained in the error code documentation, as well as in [Connecting to Bonsai](doc:connecting-to-elasticsearch), all Bonsai clusters have a randomly-generated username and password which must be present in the request in order for it to be accepted.\n\nWhat's less clear from this documentation is that including the complete URL in your Elasticsearch client may not be enough to create a secure connection.\n\nThis is due to how [HTTP Basic Access Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) works. In short, there needs to be a request header present. This header has an \"Authorization\" field and a value of `Basic <base64 string>`. The base64 string is a base64 representation of the username and password, concatenated with a colon (\":\").\n\nHere is a basic example in Ruby using `Net::HTTP`, demonstrating how a URL with auth can still receive an HTTP 401 response:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"require 'base64'\\nrequire 'net/http'\\n\\n# URL with credentials:\\nuri = URI(\\\"https://randomuser:randompass@something-12345.us-east-1.bonsaisearch.net\\\")\\n\\n# Net::HTTP does not automatically detect the presence of \\n# authentication credentials and insert the proper Authorization header.\\nreq = Net::HTTP::Get.new(uri)\\n\\n# This request will fail with an HTTP 401, even though the credentials\\n# are in the URI:\\nres = Net::HTTP.start(uri.hostname, uri.port, :use_ssl => true) {|http|\\n  http.request(req)\\n}\\n\\n# The proper header must be added manually:\\ncredentials = \\\"randomuser:randompass\\\"\\nreq['Authorization'] = \\\"Basic \\\" + Base64::encode64(credentials).chomp\\n\\n# The request now succeeds\\nres = Net::HTTP.start(uri.hostname, uri.port, :use_ssl => true) {|http|\\n  http.request(req)\\n}\",\n      \"language\": \"ruby\",\n      \"name\": null\n    }\n  ]\n}\n[/block]\nFrom the Ruby example, it's clear that there are some cases where the credentials are simply ignored instead of being automatically put in to a header. This causes the Basic authentication to fail, and receive the HTTP 401.\n\nMost clients handle the headers for you automatically and in the background. But not all do, especially if the client is part of a bleeding edge language or framework, or if it's something homebrewed/forked/patched.\n\nSimply, if you're seeing HTTP 401 responses even while including the credentials in the URL, _and_ you've confirmed that the credentials are entered correctly and have not been expired, then the problem is probably a missing header. You can detect this with tools like `socat` or `wireshark` if you're familiar with network traffic inspection. Or, you can try adding the headers manually.\n\nHere are some examples of calculating the base64 string and adding the request header in several different languages:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"public static Map<String, String> getHeaders() {\\n  Map<String, String> headers = new HashMap<>();\\n  String credentials = \\\"randomuser:randompass\\\";\\n  String auth = \\\"Basic \\\" + Base64.encodeToString(credentials.getBytes(), Base64.NO_WRAP);\\n  headers.put(\\\"Authorization\\\", auth);\\n  headers.put(\\\"Content-type\\\", \\\"application/json\\\");\\n  return headers;\\n}\",\n      \"language\": \"java\",\n      \"name\": \"Android\"\n    },\n    {\n      \"code\": \"public static Map<String, String> getHeaders() {\\n  Map<String, String> headers = new HashMap<>();\\n  String credentials = \\\"randomuser:randompass\\\";\\n  String auth = \\\"Basic \\\" + Base64.getEncoder().encodeToString(credentials.getBytes());\\n  headers.put(\\\"Authorization\\\", auth);\\n  headers.put(\\\"Content-type\\\", \\\"application/json\\\");\\n  return headers;\\n}\",\n      \"language\": \"java\"\n    },\n    {\n      \"code\": \"require 'base64'\\nrequire 'net/http'\\n\\nuri = URI(\\\"https://something-12345.us-east-1.bonsaisearch.net\\\")\\nreq = Net::HTTP::Get.new(uri)\\ncredentials = \\\"randomuser:randompass\\\"\\nreq['Authorization'] = \\\"Basic \\\" + Base64::encode64(credentials).chomp\\nres = Net::HTTP.start(uri.hostname, uri.port, :use_ssl => true) {|http|\\n  http.request(req)\\n}\",\n      \"language\": \"ruby\"\n    }\n  ]\n}\n[/block]\n\n[block:api-header]\n{\n  \"title\": \"Missing SSL Root Certificate\"\n}\n[/block]\nAs mentioned in the [Security](doc:security) documentation, all Bonsai clusters support SSL/TLS. This enables your traffic to be encrypted over the wire. In some rare cases, users may see something like this when trying to access their cluster over HTTPS:\n\n```\nSSL: CERTIFICATE_VERIFY_FAILED\n```\n\nThis is almost certainly due to a server level misconfiguration.\n\nA complete examination of how SSL works is well outside the scope of this article, but in short: it utilizes [cryptographic signatures](https://en.wikipedia.org/wiki/Digital_signature) to facilitate a [chain of trust](https://en.wikipedia.org/wiki/Chain_of_trust) from a [root certificate](https://en.wikipedia.org/wiki/Root_certificate) issued by a [certificate authority (CA)](https://en.wikipedia.org/wiki/Certificate_authority) to a certificate deployed to a server. The latter is used to prove the server's ownership before initiating [public key exchange](https://en.wikipedia.org/wiki/Public-key_cryptography).\n\nThink of a certificate authority as a mediator of sorts. A company like Bonsai goes to the CA and provides proof of identity and ownership of a domain. The CA issues Bonsai a unique certificate cryptographically signed by the CA's root certificate. Then Bonsai configures its servers to respond to SSL/TLS requests using the certificates signed by the CA's root.\n\nWhen your application reaches out to Bonsai over HTTPS, the Bonsai server will respond with this certificate. The application can then inspect the certificate it receives from Bonsai and determine whether the CA's cryptographic signature is valid. If it's valid, then the application knows it really is talking to Bonsai and not an impersonator. It then performs a key exchange to open an encrypted connection and then sends/receives data.\n\nOne weakness of this system is that the CA has to be recognized by both parties; if your application doesn't have a copy of the CA's root certificate, it can't validate that the server it's talking to is genuine. You may start seeing errors like this:\n\n```\nCould not verify the SSL certificate for...\n\nSSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed\n\npeer certificate won't be verified in this SSL session\n\nSSL certificate problem, verify that the CA cert is OK\n\nSSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed\n```\n\nThese are just a sample of the errors you might see, but it all points to the application's inability to verify Bonsai's certificate.\n\nIt's _possible_ that this is caused by an active [man in the middle (MITM) attack](https://en.wikipedia.org/wiki/Man-in-the-middle_attack), but the greater probability is that the application does not have access to the right certificates. There are a few things to check:\n\n## 1. Make sure the CA root certificate for Bonsai is available\nBonsai's nodes are using [Amazon Trust Services](https://www.amazontrust.com/repository/) as a certificate authority. These CA's need to be added to your trust store. They should be automatically installed in most browsers / operating systems. If you're seeing SSL issues, make sure that you have the correct root certificates in your trust store.\n\nThis is sometimes easier said than done. Typically a sysadmin will need to handle this. If you're using a platform like Manifold or Heroku, you'll need to open a request to their support team to verify that the proper root certificate is deployed to their systems. There are some hackish workarounds that involve bundling the certificate in your deploy and configuring the application to read it from somewhere other than the OS' trust store.\n\n## 2. Make sure the certificate(s) for Bonsai are up to date\nCertificate Authorities typically do not issue permanent certificates. Most certificates expire every 1-2 years and need to be renewed. Some browsers and applications will cache SSL certificates to improve latency. It's possible to be using a cached certificate that has passed its expiration date, thus rendering it invalid. Flush your caches and make sure that the Bonsai certificate is up to date.","excerpt":"","slug":"connection-issues","type":"basic","title":"Connection Issues"}
Troubleshooting and resolving connection issues can be time consuming and frustrating. This article aims to reduce the friction of resolving connection problems by offering suggestions for quickly identifying a root cause. [block:api-header] { "title": "First Steps" } [/block] If you're seeing connection errors while attempting to reach your Bonsai cluster, the first step is **Don't Panic.** Virtually all connection issues can be fixed in under 5 minutes once the root cause is identified. The next step is to review the documentation on [Connecting to Bonsai](doc:connecting-to-elasticsearch), which contains plenty of information about creating connections and testing the availability of your Bonsai cluster. If the basic documentation doesn't help resolve the problem, the next step is to read the error message very carefully. Often the error message contains enough information to explain the problem. For example, something like this may show up in your logs: [block:code] { "codes": [ { "code": "Faraday::ConnectionFailed (Connection refused - connect(2) for \"localhost\" port 9200):", "language": "shell" } ] } [/block] This message tells us that the client tried to reach Elasticsearch over [localhost](https://en.wikipedia.org/wiki/Localhost), which is a red flag (discussed [below](doc:connection-issues#connection-refused--connection-failed)). It means the client is not pointed at your Bonsai cluster. Next, look for an HTTP error code. Bonsai provides an entire article dedicated to explaining what different HTTP codes mean here: [HTTP Error Codes](doc:error-codes). Like the error message, the HTTP error code often provides enough diagnostic information to identify the underlying problem. Last, don't make any changes until you understand the error message and HTTP code (if one is present), and have read the relevant documentation. Most of these errors occur during the initial set up and configuration step, so also read the relevant Quickstart guide if one exists for your language / framework. [block:api-header] { "title": "Common Connection Issues" } [/block] There are issues which do not return HTTP error codes because the request can not be made at all, or because the request is not recognized by Elasticsearch. These issues are commonly one of the following: * [Connection Refused](doc:connection-issues#connection-refused) * [No handler found for URI](doc:connection-issues#no-handler-found-for-uri) * [DNS Resolution Problem](doc:connection-issues#name-or-service-not-known) * [Missing Authentication Headers](doc:connection-issues#missing-authentication-headers) * [Missing SSL Root Certificate](doc:connection-issues#missing-ssl-root-certificate) [block:api-header] { "title": "Connection Refused" } [/block] This error indicates that a request was made to a server that is not accepting connections. This error most commonly occurs when your Elasticsearch client has not been properly configured. By default, many clients will try to connect to something like `localhost:9200`. This is a problem because Bonsai will never be running on your localhost or network, regardless of your platform. [block:callout] { "type": "info", "title": "What is Localhost?", "body": "Simply, [localhost](https://en.wikipedia.org/wiki/Localhost) can be interpreted as \"this computer.\" _Which_ computer is \"this\" computer is determined by whatever machine is running the code. If you're running the code on your laptop, then _localhost_ is the machine in front of you; if you're running the code on AWS or in Heroku, then the _localhost_ is the node running your application.\n\nBonsai clusters run in a private network that does not include any of your infrastructure, which is why trying to reach it via _localhost_ will always fail." } [/block] [block:callout] { "type": "info", "title": "Why Port 9200?", "body": "By default, Elasticsearch runs on port 9200. While you _can_ access your Bonsai cluster over port 9200, this is not recommended due to lack of encryption. (See [Protocols and Ports](doc:connecting-to-elasticsearch#protocols-and-ports))." } [/block] Direct users, Heroku users and Manifold users will all need to ensure their Elasticsearch client is pointed at the correct URL. Please see the appropriate documentation for more information: * [Direct Users](doc:getting-started-with-bonsaiio) * [Heroku Users](doc:getting-started-with-heroku) * [Manifold Users](doc:getting-started-with-manifold) The solution to this problem is to ensure that the client is properly configured. ### Ruby / Rails If you’re using the [elastisearch-rails client](https://github.com/elasticsearch/elasticsearch-rails), simply add the following gem to your Gemfile: [block:code] { "codes": [ { "code": "gem 'bonsai-elasticsearch-rails'", "language": "ruby" } ] } [/block] The bonsai-elasticsearch-rails gem is a shim. All it does is configure your Elasticsearch client to load the cluster URL from an environment variable called `BONSAI_URL`. You can read more about it on the [project repository](https://github.com/omc/bonsai-elasticsearch-rails). If you'd prefer to keep your Gemfile sparse, you can initialize the client yourself like so: [block:code] { "codes": [ { "code": "# config/initializers/elasticsearch.rb\nElasticsearch::Model.client = Elasticsearch::Client.new url: ENV['BONSAI_URL']", "language": "ruby" } ] } [/block] If you opt for this method, make sure to add the `BONSAI_URL` to your environment. It will be automatically created for Heroku and Manifold users. Users managing their own application environment will need to run something like: [block:code] { "codes": [ { "code": "$ export BONSAI_URL=\"https://randomuser:randompass@something-12345.us-east-1.bonaisearch.net\"", "language": "shell" } ] } [/block] ### Other Languages / Frameworks The Elasticsearch client is probably using the default `localhost:9200` or `127.0.0.1:9200` (127.0.0.1 is the IPv4 equivalent of "localhost"). You'll need to make sure that the client is configured to use the correct URL for your cluster, and that this configuration is not being overwritten somewhere. [block:api-header] { "title": "No handler found for URI" } [/block] The Elasticsearch API has a variety of endpoints defined, like `/_cat/indices`. Each of these endpoints can be called with at a specific [HTTP method](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods). This error simply indicates a request was made to an endpoint using the wrong method. Here are some examples: [block:code] { "codes": [ { "code": "# Request to /_cat/indices with GET method:\n$ curl -XGET https://user:pass@something-12345.us-east-1.bonsai.io/_cat/indices\ngreen open test1 1 1 0 0 318b 159b \ngreen open test2 1 1 0 0 318b 159b \n\n# Request to /_cat/indices with PUT method:\n$ curl -XPUT https://user:pass@something-12345.us-east-1.bonsai.io/_cat/indices\nNo handler found for uri [/_cat/indices] and method [PUT]", "language": "shell" } ] } [/block] The solution for this issue is simple: use the correct HTTP method. The [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) will offer guidance on which methods pertain to the endpoint you're trying to use. [block:api-header] { "title": "Name or service not known" } [/block] The [Domain Name System (DNS)](https://en.wikipedia.org/wiki/Domain_Name_System) is a worldwide, decentralized and distributed directory service which translates human-readable domains like www.example.com in to network addresses like 93.184.216.119. When a client makes a request to a URL like "google.com," the application's networking layer will use DNS to translate the domain to an IP address so that it can pass the request along to the right server. The "Name or service not known" error indicates that there has been a failure in determining an IP address for the URL's domain. This typically implicates one of several root causes: 1. There is a typo in the URL. 2. There is a [TLD](https://en.wikipedia.org/wiki/Top-level_domain) outage. 3. The Internet Service Provider (ISP) handling your application's traffic may have lost a DNS server, or is having another outage. The first troubleshooting step is to carefully read the error and double-check that the URL (particularly the domain name) is spelled correctly. If this is your first time accessing the cluster, then a typo is almost certainly the problem. If this error arises during regular production use, then there is probably a DNS outage. **DNS outages are outside Bonsai's control**. There are a couple types of outages, but the ones that have affected our users before are: ## TLD outage A TLD is something like ".com," ".net," ".org," etc. A TLD outage affects all domains under the TLD. So an outage of the ".net" TLD would affect all domains with the ".net" suffix. Fortunately, all Bonsai clusters can be accessed via either of two domains: * bonsai.io * bonsaisearch.net If there is a TLD outage, you should be able to restore service by switching to the other domain. In other words, if your application is sending traffic to https://user:pass@something-12345.us-east-1.bonsai.io, and the ".io" TLD goes down, you can switch over to https://user:pass@something-12345.us-east-1.bonsaisearch.net, and it will fix the error. ## ISP Outage Many ISPs operate their own DNS servers. This way, requests made from a node in their network can get low-latency responses for IP addresses. Most ISPs also have a fleet of DNS servers, at minimum a primary and a secondary. However, this is not a requirement, and there have been instances where an ISP's entire DNS service is taken offline. There are also multitenant services which have [Software Defined Networks (SDN) running Internal DNS (iDNS)](https://docs.microsoft.com/en-us/windows-server/networking/sdn/technologies/idns-for-sdn). Regressions in this software can also lead to application-level DNS name resolution problems. If you've already confirmed the domain is correct and swapped to another TLD for your cluster, and you're _still_ having issues, then you are probably dealing with an ISP/DNS or SDN/iDNS outage. One way to confirm this is to try making requests to other common domains like google.com. A name resolution error on google.com or example.com points to a local DNS problem. If this happens, there is basically nothing you can do about it as a user, aside from complaining to your application sysadmin. [block:api-header] { "title": "Missing Authentication Headers" } [/block] Users who are seeing persistent [HTTP 401 Error Codes](doc:error-codes#http-401-authorization-required) may be using a client that is not handling authentication properly. As explained in the error code documentation, as well as in [Connecting to Bonsai](doc:connecting-to-elasticsearch), all Bonsai clusters have a randomly-generated username and password which must be present in the request in order for it to be accepted. What's less clear from this documentation is that including the complete URL in your Elasticsearch client may not be enough to create a secure connection. This is due to how [HTTP Basic Access Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) works. In short, there needs to be a request header present. This header has an "Authorization" field and a value of `Basic <base64 string>`. The base64 string is a base64 representation of the username and password, concatenated with a colon (":"). Here is a basic example in Ruby using `Net::HTTP`, demonstrating how a URL with auth can still receive an HTTP 401 response: [block:code] { "codes": [ { "code": "require 'base64'\nrequire 'net/http'\n\n# URL with credentials:\nuri = URI(\"https://randomuser:randompass@something-12345.us-east-1.bonsaisearch.net\")\n\n# Net::HTTP does not automatically detect the presence of \n# authentication credentials and insert the proper Authorization header.\nreq = Net::HTTP::Get.new(uri)\n\n# This request will fail with an HTTP 401, even though the credentials\n# are in the URI:\nres = Net::HTTP.start(uri.hostname, uri.port, :use_ssl => true) {|http|\n http.request(req)\n}\n\n# The proper header must be added manually:\ncredentials = \"randomuser:randompass\"\nreq['Authorization'] = \"Basic \" + Base64::encode64(credentials).chomp\n\n# The request now succeeds\nres = Net::HTTP.start(uri.hostname, uri.port, :use_ssl => true) {|http|\n http.request(req)\n}", "language": "ruby", "name": null } ] } [/block] From the Ruby example, it's clear that there are some cases where the credentials are simply ignored instead of being automatically put in to a header. This causes the Basic authentication to fail, and receive the HTTP 401. Most clients handle the headers for you automatically and in the background. But not all do, especially if the client is part of a bleeding edge language or framework, or if it's something homebrewed/forked/patched. Simply, if you're seeing HTTP 401 responses even while including the credentials in the URL, _and_ you've confirmed that the credentials are entered correctly and have not been expired, then the problem is probably a missing header. You can detect this with tools like `socat` or `wireshark` if you're familiar with network traffic inspection. Or, you can try adding the headers manually. Here are some examples of calculating the base64 string and adding the request header in several different languages: [block:code] { "codes": [ { "code": "public static Map<String, String> getHeaders() {\n Map<String, String> headers = new HashMap<>();\n String credentials = \"randomuser:randompass\";\n String auth = \"Basic \" + Base64.encodeToString(credentials.getBytes(), Base64.NO_WRAP);\n headers.put(\"Authorization\", auth);\n headers.put(\"Content-type\", \"application/json\");\n return headers;\n}", "language": "java", "name": "Android" }, { "code": "public static Map<String, String> getHeaders() {\n Map<String, String> headers = new HashMap<>();\n String credentials = \"randomuser:randompass\";\n String auth = \"Basic \" + Base64.getEncoder().encodeToString(credentials.getBytes());\n headers.put(\"Authorization\", auth);\n headers.put(\"Content-type\", \"application/json\");\n return headers;\n}", "language": "java" }, { "code": "require 'base64'\nrequire 'net/http'\n\nuri = URI(\"https://something-12345.us-east-1.bonsaisearch.net\")\nreq = Net::HTTP::Get.new(uri)\ncredentials = \"randomuser:randompass\"\nreq['Authorization'] = \"Basic \" + Base64::encode64(credentials).chomp\nres = Net::HTTP.start(uri.hostname, uri.port, :use_ssl => true) {|http|\n http.request(req)\n}", "language": "ruby" } ] } [/block] [block:api-header] { "title": "Missing SSL Root Certificate" } [/block] As mentioned in the [Security](doc:security) documentation, all Bonsai clusters support SSL/TLS. This enables your traffic to be encrypted over the wire. In some rare cases, users may see something like this when trying to access their cluster over HTTPS: ``` SSL: CERTIFICATE_VERIFY_FAILED ``` This is almost certainly due to a server level misconfiguration. A complete examination of how SSL works is well outside the scope of this article, but in short: it utilizes [cryptographic signatures](https://en.wikipedia.org/wiki/Digital_signature) to facilitate a [chain of trust](https://en.wikipedia.org/wiki/Chain_of_trust) from a [root certificate](https://en.wikipedia.org/wiki/Root_certificate) issued by a [certificate authority (CA)](https://en.wikipedia.org/wiki/Certificate_authority) to a certificate deployed to a server. The latter is used to prove the server's ownership before initiating [public key exchange](https://en.wikipedia.org/wiki/Public-key_cryptography). Think of a certificate authority as a mediator of sorts. A company like Bonsai goes to the CA and provides proof of identity and ownership of a domain. The CA issues Bonsai a unique certificate cryptographically signed by the CA's root certificate. Then Bonsai configures its servers to respond to SSL/TLS requests using the certificates signed by the CA's root. When your application reaches out to Bonsai over HTTPS, the Bonsai server will respond with this certificate. The application can then inspect the certificate it receives from Bonsai and determine whether the CA's cryptographic signature is valid. If it's valid, then the application knows it really is talking to Bonsai and not an impersonator. It then performs a key exchange to open an encrypted connection and then sends/receives data. One weakness of this system is that the CA has to be recognized by both parties; if your application doesn't have a copy of the CA's root certificate, it can't validate that the server it's talking to is genuine. You may start seeing errors like this: ``` Could not verify the SSL certificate for... SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed peer certificate won't be verified in this SSL session SSL certificate problem, verify that the CA cert is OK SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed ``` These are just a sample of the errors you might see, but it all points to the application's inability to verify Bonsai's certificate. It's _possible_ that this is caused by an active [man in the middle (MITM) attack](https://en.wikipedia.org/wiki/Man-in-the-middle_attack), but the greater probability is that the application does not have access to the right certificates. There are a few things to check: ## 1. Make sure the CA root certificate for Bonsai is available Bonsai's nodes are using [Amazon Trust Services](https://www.amazontrust.com/repository/) as a certificate authority. These CA's need to be added to your trust store. They should be automatically installed in most browsers / operating systems. If you're seeing SSL issues, make sure that you have the correct root certificates in your trust store. This is sometimes easier said than done. Typically a sysadmin will need to handle this. If you're using a platform like Manifold or Heroku, you'll need to open a request to their support team to verify that the proper root certificate is deployed to their systems. There are some hackish workarounds that involve bundling the certificate in your deploy and configuring the application to read it from somewhere other than the OS' trust store. ## 2. Make sure the certificate(s) for Bonsai are up to date Certificate Authorities typically do not issue permanent certificates. Most certificates expire every 1-2 years and need to be renewed. Some browsers and applications will cache SSL certificates to improve latency. It's possible to be using a cached certificate that has passed its expiration date, thus rendering it invalid. Flush your caches and make sure that the Bonsai certificate is up to date.