I was thinking about creating a Kubernetes cluster on-premise, using a set of 3 used mini PCs. The main problem that we need to resolve is pricing. We don’t want to spend too much money on a development environment. The cheapest solution is always to be Minikube, which is the Kubernetes cluster running on our own machine. The thing is, my current machine does not have enough space for the Minikube and the VMs. It’s good enough to develop with a few containers running at the same time along with IDEs (IntelliJ, PyCharm, and VScode) and browsers. Therefore, for a development environment for DevOps, I believe it’s necessary to have a large enough set of nodes, the minimum in my thinking is 3 nodes with 4 vCPU, 8GB memory, and 200 GB of disk space for each node. This should be sufficient to host at least two to three different databases (eg: OpenSearch, Postgres) and some kind of CICD pipelines (eg: Jenkins). To have this capacity set up, we can think of public clouds like GCP, and AWS or we can choose to purchase a set of used PCs and a router, a switch, and CAT6 cables.
If I choose to use GCP and AWS, the costs are different. Both offer spot instances. For GCP, we can have 3 nodes with each node having exactly 4 vCPU, 8 GB memory, and 200 disk space, this already included the GKE standard edition cost. The cost for the whole managed Kubernetes cluster with the above 3 nodes using spot instances is going to be 187.68 USD each month, the reference link for the cost calculation is here. But we also need a load balancer for the cluster right? the CLB pricing is roughly 0.025 per hour and we have 730 hours a month therefore the cost for load balancing is 18.25 USD. The final number is 205.83 USD. CLB pricing reference can be found here.
For AWS, the closest equivalent for the N1 series with the same spec is the c6g.xlarge instance family, the total cost for 3 spot instances of this kind of machine is 175.92 USD per month, note that this already includes 200 GB disk space for each machine (I’m choosing HDD for cost saving). For the managed EKS control plane of the cluster (without LB) we also need to have to pay an extra 73 USD per month. Therefore the cost of having a cluster is 248.92 USD. If we add an Application Load Balancer (ALB) or Elastic Load Balancing (ELB) to provide load-balancing capability for the cluster, it will cost roughly 29.57 USD per month, assuming that we only have 2 GB of data forwarding in and out of the load balancer for both TLS and TCP/UDP. The total cost should be 278.49 for the Kubernetes cluster with a load balancer in AWS. The reference link for cost calculation is here.
For On-premise, the almost-as-good machine that provides the same capacity as N1 series or c6g.xlarge is the Lenovo ThinkCentre M93P, this machine is equipped with Intel® Core i5-4570T Processor, which can provide 4 vCPU and 8 GB of DDR3 of memory and 240 of Micron SSD. The refurnished in Amazon is 200 USD, and the marketplace can be much lower (I see some folks sell it for only 100 USD), but for the shake of cost consistency, we can use the one from Amazon, the reference link is here. For the router, we can go with the TP-Link AC750 Wireless Portable Nano Travel Router, reference to here, this will cost 50 CAD or 36.63 USD at the time of writing. For the switch, we can use TL-SG105E, reference to here, the cost is 17.58 USD. We also need patch cables, for this, we can use Cat 6 Ethernet Cable 0.5ft 5 Pack, reference to here, the cost is 7.32 USD. We will also need a power bar, to supply power for those 5 devices, the cost is around 16 USD. The total is 200 * 3 + 36.63 + 17.58 + 7.32 = 661.53 USD. The big disadvantage of the on-premise cluster is the global network, and it will not have a public address to be visible from outside. The reliability is not good as well since the cluster can be shut down anytime and it can only be used as a local development environment. Besides that, it requires Kubernetes setup and it takes time to do that. Therefore, despite that with 661 USD you can start getting profit after 3 months (since it’s twice the cost of AWS and triple the cost of GCP). I assume that it won’t be able to be useful in the long term if you decide to have your services running in the cloud.
Final words: on-premise cluster is profitable if we use it for development and learning purposes. If we want to deploy the real thing, the cloud is the way to go. The above pricing quote is quite a minimal (or just enough) setup for a production workload since we have up to 3 nodes for high availability. Overall, a Kubernetes cluster is expensive. I mean, anything that comes with ‘highly available’ is expensive.