https://www.thecodebinge.com/blog/Recent content in Blog on CodeBingeHugo -- gohugo.ioen-USThu, 13 Sep 2018 22:31:21 +0800https://www.thecodebinge.com/blog/2022-02/big-ideas-behind-mads-one-stop-no-code-platform-for-iiot/Wed, 09 Feb 2022 13:59:48 +0800https://www.thecodebinge.com/blog/2022-02/big-ideas-behind-mads-one-stop-no-code-platform-for-iiot/Fast, flexible, and user-friendly, MADS is the leading platform for building IIOT solutions. What is MADS? MADS is a 100% self-service no-code intuitive IoT platform for rapid implementation of enterprise IoT projects. It enables quick development, management, and scaling of IoT use-cases. Any enterprise IoT project has some desired outcome to realize the the ROI of the project. The desired outcome is what the stakeholders are interested in, however to realize that outcome there are certain foundational steps: Create a design of the IoT project Connect and collect all the IoT data Define data pipelines and apply the business logic Analyze and visualize the generated information from raw data. Take action to complete the feedback loop. The above steps are fundamental to deployment of any IoT project. Conventionally these steps were developed as applications and tools by software developers in collaboration with domain experts who understood the usecase. However, since these steps are common an abstraction layer can be created to stop reinventing the wheel and create a horizontal IoT platform which comes with such features baked in. This kind of support allows the domain experts to use the platform to build the usecase on their own and rapidly customize and deploy it for their users.https://www.thecodebinge.com/blog/2020-03/elasticsearch-introduction/Tue, 10 Mar 2020 08:23:48 +0800https://www.thecodebinge.com/blog/2020-03/elasticsearch-introduction/What is elastic Search Add data Built with Java, based on Apache lucene. Easy to use, and highly scalable. Use Cases Add data Overview of ElasticStack ElasticStack consists of a few different products: X-Pack: Adds features to ElasticSearch and Kibana, Security: Authentication and authroization. Monitoring: Gain insights into how ELK stack is working. Alerts: For CPU usage, user behaviour, etc. Reporting: Machine Learning: Enables Kibana to use machine learning. Graph: Relationships Kibana: Analytics and visualization engine. Anomaly detection and machine learning. Beats: Logstash: Data processing pipeline. ElasticSearch Storing Data in Elastic Search Add data Document A document in elastic search corresponds to a row in database. { “name”: “iron man”, “work_place: “stark industries”, “team”: “avengers” } Each field in the document corresponds to a column in a table.https://www.thecodebinge.com/blog/2020-03/exploring-the-elasticsearch-directory/Tue, 10 Mar 2020 08:23:48 +0800https://www.thecodebinge.com/blog/2020-03/exploring-the-elasticsearch-directory/https://www.thecodebinge.com/blog/2020-03/elasticsearch-different-node-roles/Tue, 10 Mar 2020 08:22:48 +0800https://www.thecodebinge.com/blog/2020-03/elasticsearch-different-node-roles/https://www.thecodebinge.com/blog/2020-03/elasticsearch-installation/Tue, 10 Mar 2020 08:22:48 +0800https://www.thecodebinge.com/blog/2020-03/elasticsearch-installation/Installation There are multiple ways to install and use ElasticSearch and Kibana. ElasticCloud Elastic Cloud is a software as a service provided by elastic. The above allows you to deploy a cluster in the cloud and run your application. It’s the easiest option, however it costs you to run the application in cloud. There is free trial of 14 days available though. Running on Linux, MacOS, Windows The elasticsearch consists of a set of binaries to run on respective operating system, it can be downloaded from on elasticsearch downloads page. MacOS and Linux Easy steps to download and run Elasticsearch on linux. Please change the package version if you want a different version. $ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.1-darwin-x86_64.tar.gz //download the binaries $ tar -xvzf elasticsearch-7.6.1-darwin-x86_64.tar.gz // extract the files $ cd elasticsearch-7.6.1 $ bin/elasticsearch // runs the elasticsearch server in forgeground But be warned elastic search needs java to be properly set on the machine and it might lead to some trouble if setup is not proper. To avoid any java requirement related issues, we can use the dockerized deployment which is much easier to setup. provide a resolution for the java problem. Add steps for windows Running with Dockerhttps://www.thecodebinge.com/blog/2020-03/elasticsearch-replication/Tue, 10 Mar 2020 08:22:48 +0800https://www.thecodebinge.com/blog/2020-03/elasticsearch-replication/https://www.thecodebinge.com/blog/2020-03/elasticsearch-sharding-and-scalability/Tue, 10 Mar 2020 08:22:48 +0800https://www.thecodebinge.com/blog/2020-03/elasticsearch-sharding-and-scalability/https://www.thecodebinge.com/blog/2019-12/local-setup-for-learning-kubernetes/Mon, 09 Dec 2019 13:59:48 +0800https://www.thecodebinge.com/blog/2019-12/local-setup-for-learning-kubernetes/Introduction There are multiple ways of setting up kubernetes. To play around or learn kubernetes you can set it up via multiple ways such as: Minikube MicroK8s Kubeadm The kubeadm tool can also be used for setting up a local viable cluster with the help of a config management tools such as Ansible. A detailed step by step tutorial for installing kuberetes with ansible on a local cluster can be found here. However for running a prodcuction grade cluster in cloud it’s better to use a managed service from providers such as GCP, AWS, Azure or DigitalOcean. For a local development or learning setup minikube is the easiest to start with and we will go through steps to setup a local minikube cluster. A typical kubernetes deployments consists of master and worker node with components shown in the image below. Minikube bundles all these components together in a single node kubernetes cluser. This gives us a simple setup to work and interact with. Installing Kubectl To interact with a kubernetes cluster we need a utility kubectl. The kubectl utility can interact with both local as well as a remote setup with some configuration. Inorder to install kubectl utility on linuxhttps://www.thecodebinge.com/blog/2019-11/inspecting-kubernetes-objects/Sun, 10 Nov 2019 13:59:48 +0800https://www.thecodebinge.com/blog/2019-11/inspecting-kubernetes-objects/During a deployment the orchestrator performs some operations on the containers. To see the operations, we can make use of some commands which paints a clear picture on series of steps taken to reach the goal. The two main commands are logs and describe. If you have followed the previous part till end then you can run the following commands else follow steps to create replica sets. Log If we use the log command to inspect the deployment we created my-apache, kubectl logs deploy/my-apache it spits out the following on the terminal. Found 2 pods, using pod/my-apache-5d589d69c7-mb7wx AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.1.10.3. Set the 'ServerName' directive globally to suppress this message AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.1.10.3. Set the 'ServerName' directive globally to suppress this message [Thu Nov 21 14:23:27.128189 2019] [mpm_event:notice] [pid 1:tid 139734356759680] AH00489: Apache/2.4.41 (Unix) configured -- resuming normal operations [Thu Nov 21 14:23:27.150818 2019] [core:notice] [pid 1:tid 139734356759680] AH00094: Command line: 'httpd -D FOREGROUND' Inorder to inspect the pods based on some label, we can use the command kubectl logs -l run=my-apache here -l is for label and when we created the replica sets for my-apache, by default a label is associated with the pods with the name of the cluster.https://www.thecodebinge.com/blog/2019-11/kubernetes-deployment-strategies/Sat, 09 Nov 2019 13:59:48 +0800https://www.thecodebinge.com/blog/2019-11/kubernetes-deployment-strategies/Deployment strategies are methods use to deploy applications on Kubernetes. Recreate Strategy Rolling Update Blue Green Strategy New deployment is made across the old deployment. Then the traffic is moved from old to new in one go. deployment and service with label v1 create deployment with label v2 on kubernetes change label in service to v2 Canary In canary deployment a small percentage of requests are routed to the new deployment. Comparison of different Strategieshttps://www.thecodebinge.com/blog/2019-11/kubernetes-helm-charts/Sat, 09 Nov 2019 13:59:48 +0800https://www.thecodebinge.com/blog/2019-11/kubernetes-helm-charts/https://www.thecodebinge.com/blog/2019-11/kubernetes-introduction/Sat, 09 Nov 2019 13:59:48 +0800https://www.thecodebinge.com/blog/2019-11/kubernetes-introduction/Basic Terms Kubernetes: The whole orchestration system for containers. Also called K8s. Kubectl: CLI to configure kubernetes and manage apps. Pronounces Kube-cuttle or kube-c-t-l. Node: A single server in kubernetes cluster. Kubelet: Kubernetes agent running on nodes. Control Plane(Managers): Set of containers managing the cluster. Runs APIs, DNS, Database, scheduler, etcd, controller manager etc. Control Plane(Master) etcd -> A distributed key value storage system which uses Raft consensus algorithm to manage a highly-available replicated log. API -> A set of calls to talk to the kubernetes engine running on master nodes. Scheduler -> Controls how or where containers are placed on nodes. Controller Manager -> Looks at state of whole cluster using APIs. Receives orders from user and makes the system work accordingly. Core DNS -> Manage networking. Nodes(Workers) kubelet -> kube-proxy -> Installing locally To use kubernetes locally we can use MicroK8s. There are other ways to install, but I find this one, the most convenient. Kubernetes Container Abstractions Pod: One or more nodes running together on one Node. - Basic unit of deployment. Containers are always in pod. Controller: For creating/updating pods and other objects. Many types of controllers Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, CronJob etc.https://www.thecodebinge.com/blog/2019-11/kubernetes-operators/Sat, 09 Nov 2019 13:59:48 +0800https://www.thecodebinge.com/blog/2019-11/kubernetes-operators/https://www.thecodebinge.com/blog/2019-11/kubernetes-pods/Sat, 09 Nov 2019 13:59:48 +0800https://www.thecodebinge.com/blog/2019-11/kubernetes-pods/PODS Kubernetes’s ultimate aim is to deploy an application in the form of containers on machines. Kubernetes doesn’t deploy containers directly, it encapsulates it into an object known as a pod. As the traffic increases to a cluster these pods are scaled to meet the demand. A pod can contain multiple containers of different types. Sometimes helper containers are deployed on the same pod however such instances are very rare. It’s a good practice to have only one container per pod as it keeps the architecture simple. Containers and their use A genuine question comes to mind whenever we start with Kubernetes is why we need pods if there are containers. Containers are used for deploying a single application(process) in an isolated way. The main idea behind this philosohpy is that it becomes easier to manage logs, and if there is an exception then the orchestrator can restart the process. We will have hard time figuring out the issues if there are multiple processes running in a single container. Also, there might be a need that some of the services should keep running in case another fails, when a container goes down it brings all the processes to halt.https://www.thecodebinge.com/blog/2019-10/elixir-genservers-101-part-1/Wed, 09 Oct 2019 12:47:47 +0800https://www.thecodebinge.com/blog/2019-10/elixir-genservers-101-part-1/What are Server Processes A server process can be defined as one with the following behaviour : Long running process. It can maintain state. It can process messages and react to them. It may modify the internal state. It may send a reply back to the client. Server processes are important in building highly concurrent systems. Erlang provides an important abstraction to handle server processes in it’s portion of OTP framework, known as a Genserver. It’s the essential backbone over which much of the parallel processing in Elixir is built. Implementing the Server Process Inorder to have a better understanding of the Genserver behaviour. Let’s build a minimal version of this abstraction ourselves using the very basic construct of OTP, processes. To do this, we have to implement a process which has all the characteristics mentioned above, of a long running server process. What we want to achieve at the end: 1. Define a behaviour(for a long running process with state) which can be extended using callback modules. 2. The behaviour should create a process and initialize it by calling the respective callback module. 3. The behaviour should have a mechanism to keep the process runnning until a signal to stop or kill the process is given.https://www.thecodebinge.com/blog/2019-10/elixir-processes-good-bad-and-ugly/Mon, 07 Oct 2019 13:59:48 +0800https://www.thecodebinge.com/blog/2019-10/elixir-processes-good-bad-and-ugly/What are Elixir Processes Spawn Communication Mechanism Receive State Management through infinite loop Process State Process Dictionary Process Supervision Links and Monitors Erlang Schedulerhttps://www.thecodebinge.com/blog/2019-10/elixir-tasks/Mon, 07 Oct 2019 13:59:48 +0800https://www.thecodebinge.com/blog/2019-10/elixir-tasks/Overview Tasks are an abstraction build over BEAM processes which allow easy management of the spawned process instead of directly handling the nitty gritties. Elixir/Erlang provides spwan and recieve functions for spawning and handling communication with process however they are cumbersome to deal with in a complex setting and may require us to write code for error handling, receiving requests and managing the entire lifecycle of the process. The Task module provides a simple API to interact with the processes. In this blog we will try to understand how to use the Task module functions. How to start tasks, retrieve results, process large amount of data using the Task module. I have been working in the IoT domain and will be showing examples from the same. IoT is quite data intensive condierring use caess where data packets can be range from generation every millisecond to a single data point per day. A detailed list of functions exposed by the Task and Task.Supervisor module can be found here However, the set of functins we will take a look at in this blog are the following: Creating the Project Let’s give a cool name to our project we can call it SenseEx.https://www.thecodebinge.com/blog/2019-10/if-case-cond-and-with-in-elixir/Mon, 07 Oct 2019 13:59:48 +0800https://www.thecodebinge.com/blog/2019-10/if-case-cond-and-with-in-elixir/https://www.thecodebinge.com/blog/2019-10/pattern-matching/Mon, 07 Oct 2019 13:59:48 +0800https://www.thecodebinge.com/blog/2019-10/pattern-matching/https://www.thecodebinge.com/blog/1-01/kubernetes-deployments/Mon, 01 Jan 0001 00:00:00 +0000https://www.thecodebinge.com/blog/1-01/kubernetes-deployments/