Using the horizontal pod autoscaler with custom metrics requires some configuration when launching your cluster. First, you need to enable the API aggregation layer. Then you need to register your resource metrics API and your custom metrics API. Heapster provides an implementation of the resource metrics API you can use. Just start Heapster with the --api-server flag set to true. You need to run a separate server that exposes the custom metrics API. A good starting point is this: https://github.com/kubernetes-incubator/custom-metrics-apiserver.

The next step is to start the kube-controller-manager with the following flags:

--horizontal-pod-autoscaler-use-rest-clients=true
--kubeconfig <path-to-kubeconfig> OR --master <ip-address-of-apiserver>  

The --master flag will override --kubeconfig if both are specified. These flags specify the location of the API aggregation layer, allowing the controller manager to communicate to the API server.

In Kubernetes 1.7, the standard aggregation layer that Kubernetes provides runs in-process with the kube-apiserver, so the target IP address can be found with this:

> kubectl get pods --selector k8s-app=kube-apiserver --namespace kube-system -o jsonpath='{.items[0].status.podIP}'