Evolution of Linux Containers and Future

Linux containers are an operating system level virtualization technology for providing multiple isolated Linux environments on a single Linux host. Unlike virtual machines (VMs), containers do not run dedicated guest operating systems. Rather, they share the host operating system kernel and make use of the guest operating system system libraries for providing the required OS capabilities. Since there is no dedicated operating system, containers start much faster than VMs.Image credit: Docker Inc.Containers make use of Linux kernel features such as Namespaces, Apparmor, SELinux profiles, chroot, and CGroups for providing an isolated environment similar to VMs. Linux security modules guarantee that access to the host machine and the kernel from the containers is properly managed to avoid any intrusion activities. In addition containers can run different Linux distributions from its host operating system if both operating systems can run on the same CPU architecture.In general, containers provide a means of creating container images based on various Linux distributions, an API for managing the lifecycle of the containers, client tools for interacting with the API, features to take snapshots, migrating container instances from one container host to another, etc.Container HistoryBelow is a short summary of container history extracted from Wikipedia and other sources:1979 — chrootThe concept of containers was started way back in 1979 with UNIX chroot. It’s an UNIX operating-system system call for changing the root directory of a process and it's children to a new location in the filesystem which is only visible to a given process. The idea of this feature is to provide an isolated disk space for each process. Later in 1982 this was added to BSD.2000 — FreeBSD JailsFreeBSD Jails is one of the early container technologies introduced by Derrick T. Woolworth at R&D Associates for FreeBSD in year 2000. It is an operating-system system call similar to chroot, but included additional process sandboxing features for isolating the filesystem, users, networking, etc. As a result it could provide means of assigning an IP address for each jail, custom software installations and configurations, etc.2001 — Linux VServerLinux VServer is a another jail mechanism that can be used to securely partition resources on a computer system (file system, CPU time, network addresses and memory). Each partition is called a security context, and the virtualized system within it is called a virtual private server.2004 — Solaris ContainersSolaris Containers were introduced for x86 and SPARC systems, first released publicly in February 2004 in build 51 beta of Solaris 10, and subsequently in the first full release of Solaris 10, 2005. A Solaris Container is a combination of system resource controls and the boundary separation provided by zones. Zones act as completely isolated virtual servers within a single operating system instance.2005 — OpenVZOpenVZ is similar to Solaris Containers and makes use of a patched Linux kernel for providing virtualization, isolation, resource management, and checkpointing. Each OpenVZ container would have an isolated file system, users and user groups, a process tree, network, devices, and IPC objects.2006 — Process ContainersProcess Containers was implemented at Google in year 2006 for limiting, accounting, and isolating resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes. Later on it was renamed to Control Groups to avoid the confusion multiple meanings of the term “container” in the Linux kernel context and merged to the Linux kernel 2.6.24. This shows how early Google was involved in container technology and how they have contributed back.2007 — Control GroupsAs explained above, Control Groups AKA cgroups was implemented by Google and added to the Linux Kernel in 2007.2008 — LXCLXC stands for LinuX Containers and it is the first, most complete implementation of Linux container manager. It was implemented using cgroups and Linux namespaces. LXC was delivered in liblxc library and provided language bindings for the API in Python3, Python2, Lua, Go, Ruby, and Haskell. Contrast to other container technologies LXC works on vanila Linux kernel without requiring any patches. Today LXC project is sponsored by Canonical Ltd. and hosted here.2011 — WardenWarden was implemented by CloudFoundry in year 2011 by using LXC at the initial stage and later on replaced with their own implementation. Unlike LXC, Warden is not tightly coupled to Linux. Rather, it can work on any operating system that can provide ways of isolating environments. It runs as a daemon and provides an API for managing the containers. Refer to Warden documentation and this blog post for more detailed information on Warden.2013 — LMCTFYlmctfy stands for “Let Me Contain That For You”. It is the open source version of Google’s container stack, which provides Linux application containers. Google started this project with the intention of providing guaranteed performance, high resource utilization, shared resources, over-commitment, and near zero overhead with containers (Ref: lmctfy presentation). The cAdvisor tool used by Kubernetes today was started as a result of lmctfy project. The initial release of lmctfy was made in Oct 2013 and in year 2015 Google has decided to contribute core lmctfy concepts and abstractions to libcontainer. As a result now no active development is done in LMCTFY.The libcontainer project was initially started by Docker and now it has been moved to Open Container Foundation.2013 — DockerDocker is the most popular and widely used container management system as of January 2016. It was developed as an internal project at a platform-as-a-service company called dotCloud and later renamed to Docker. Similar to Warden, Docker also used LXC at the initial stages and later replaced LXC with it’s own library called libcontainer. Unlike any other container platform, Docker introduced an entire ecosystem for managing containers. This includes a highly efficient, layered container image model, a global and local container registries, a clean REST API, a CLI, etc. At a later stage, Docker also took an initiative to implement a container cluster management solution called Docker Swarm.2014 — RocketRocket is a much similar initiative to Docker started by CoreOS for fixing some of the drawbacks they found in Docker. CoreOS has mentioned that their aim is to provide more rigorous security and production requirements than Docker. More importantly, it is implemented on App Container specifications to be a more open standard. In addition to Rocket, CoreOS also develops several other container related products used by Docker and Kubernetes: CoreOS Operating System, etcd, and flannel.2016 — Windows ContainersMicrosoft also took an initiative to add container support to the Microsoft Windows Server operating system in 2015 for Windows based applications, called Windows Containers. This is to be released with Microsoft Windows Server 2016. With this implementation Docker would be able to run Docker containers on Windows natively without having to run a virtual machine to run Docker (earlier Docker ran on Windows using a Linux VM).The Future of ContainersAs of today (Jan 2016) there is a significant trend in the industry to move towards containers from VMs for deploying software applications. The main reasons for this are the flexibility and low cost that containers provide compared to VMs. Google has used container technology for many years with Borg and Omega container cluster management platforms for running Google applications at scale. More importantly, Google has contributed to container space by implementing cgroups and participating in libcontainer projects. Google may have gained a huge gain in performance, resource utilization, and overall efficiency using containers during past years. Very recently Microsoft, who did not had an operating system level virtualization on the Windows platform took immediate action to implement native support for containers on Windows Server.Docker, Rocket, and other container platforms cannot run on a single host in a production environment, the reason is that they are exposed to single point of failure. While a collection of containers are run on a single host, if the host fails, all the containers that run on that host will also fail. To avoid this, a container host cluster needs to be used. Google took a step to implement an open source container cluster management system called Kubernetes with the experience they got from Borg. Docker also started a solution called Docker Swarm. Today these solutions are at their very early stages and it may take several months and may be another year to complete their full feature set, become stable and widely used in the industry in production environments.Microservices are another groundbreaking technology rather a software architecture which uses containers for their deployment. A microservice is nothing new but a lightweight implementation of a web service which can start extremely fast compared to a standard web service. This is done by packaging a unit of functionality (may be a single service/API method) in one service and embedding it into a lightweight web server binary.By considering the above facts we can predict that in next few years, containers may take over virtual machines, and sometimes might replace them completely. Last year I worked with a handful of enterprises on implementing container-based solutions on a POC level. There were few who wanted to take the challenge and put them in production. This may change very quickly as the container cluster management systems get more mature. Read more

Gatling Tool Review for Performance Tests (Written in Scala)

Have you heard of Gatling for performance tests? It seems to be a relatively new tool (created in 2012, so pretty new), that has recently been gaining certain popularity (250,000 downloads in four years, 60,000 of those in the last three months, meaning it has been gaining attention from the community). So that you don’t have to dedicate too much time out of your day to learn more about this tool, I wrote this review to sum up some of the tests I ran with it. Hopefully, within just a few minutes, this gatling tool review will give you a good idea of what you can do with it. As there are hardly an articles about the topic in Spanish, this a translation of my original post (written in espanõl!).Key features of Gatling:●  Tool for performance testing●   Free and opensource (developed in Java / Scala)●   The scripting language is Scala, with its own DSL●   It works with whichever operating system and any browser●   It supports HTTP/S, JMS, and JDBC protocols●   Colorful reports in HTML●   It doesn’t allow you to distribute the load between machines, but it can execute its tests in different test clouds. It can scale using flood.io or Taurus with BlazeMeter (Taurus provides many facilities for continuous integration)It’s a great tool for when:●   You need to simulate less than 600 concurrent users. This is just a reference number, depending on how much processing your simulation script has, but if it needs to generate more, then you will have to pay for a tool in the cloud. A colleague told me that he managed to execute a script with 4,000 concurrent users with a simple script from just one machine.●   You want to learn about performance tests (it’s very simple and the code is very legible)●   You are interested in maintaining the test code (the language, Scala, and the Gatling’s DSL are pretty focused on facilitating the maintainability of the tests, which is ideal if you are focusing on continuous integration).This tool allows you to carry out a load simulation of concurrent users against a system through the HTTP/S, JMS, or JDBC protocols. The most typical scenario of when you want to use this tool is to simulate users of a web system in order to analyze the bottlenecks and optimize it. For comparison, some very popular alternatives on the market are JMeter and HP Load Runner (to name one opensource tool and one commercial, both are widely used).Gatling is a free and opensource tool. It works on Java, thus it’s suitable for all operating systems. It requires the JDK8 (it’s not enough with the runtime, we need the development kit).The tool has two executables: one to record the tests and the other to execute them. The tests are recorded in Scala, which is a very clean and easy to read language, even upon looking at it for the first time. After each execution, you get a colorful and wordy report.Fundamental Aspects for the Correct Simulation of UsersThe scripts count on fundamental aspects for the correct simulation of users, which for our consideration are:●   Handling of protocol (from the invocations and responses, to the management of headers, cookies, etc.)●   Handling of strings, facilities to parse, regular expressions, and including, localization of elements for xpath, json path, css, and more●   Validations, being that we need to check that the responses are correct●   Parametrization from different sources of data (here I see a very strong point of this tool, since it offers various, easy alternatives to use)●   Handling of dynamic variables, known as variable correlation●   Handling of different scopes of the variables (level of threads, tests, etc.)●   Modularization (facilitating the maintainability and legibility of the scripts) ●   Handling waits (to simulate think times)●   Metrics management (response times, individual ones and group ones, transactions per second, amount of concurrent users, errors, amount of transferred data, etc)●   Management of errors and exceptions●   Flow control (loops, if-then-else)What other things do you consider in the moment of evaluating the scripting language of a load or stress simulation tool?Gatling ReportsRegarding the reports, they are very colorful and complete. Here I’d like to highlight that its reports:●   Are in HTML with easy navigation, with an index and organized●   Graphically show the information in a well grouped and very well-processed and related way●   Include a graphic of the quantity of virtual users during the test●   You can zoom in on the graphics to focus and analyze them with more detail in certain areas.●   Graph the requests per second and the responses per second, including the comparison of the quantity of active users●   You can see each request in detail, in order to refine your analysis.●   Separate the response times for the ones that were “ok” and the ones that failed●   Handle of the concept of percentiles●   Have a log of errors foundWhat other things do you deem important when evaluating the reports of a stress or load simulation tool?In short, we at Abstracta are big fans of Gatling. We are starting to use it in projects as we have received several requests from clients to use it. In the future, I am sure that it will continue to be an important item in our continuous integration toolshed.Have you used Gatling? How does it measure up for you? Read more

The Life of a Serverless Microservice on AWS

In this post, I will demonstrate how you can develop, test, deploy, and operate a production-ready serverless microservice using the AWS ecosystem. The combination of AWS Lambda and Amazon API Gateway allows us to operate a REST endpoint without the need of any virtual machines. We will use Amazon DynamoDB as our database, Amazon CloudWatch for metrics and logs, and AWS CodeCommit and AWS CodePipeline as our delivery pipeline. In the end, you will know how to wire together a bunch of AWS services to run a system in production.The LifeMy idea of "The Life of a Serverless Microservice on AWS" is best described by this figure:A developer is pushing code changes to a repository. This git push triggers the CI & CD pipeline to deploy a new version of the service, which our users consume. The load generated on the system produces logs and metrics that are used by the developer to operate the system. The operational feedback is used to improve the quality of the system.What is Serverless?Serverless or Function as a Service (FaaS) describes the idea that the deployment unit is a single function. A function takes input and returns output. The responsibility of the FaaS user is to develop the function while the FaaS provider's responsible is to execute the function whenever some event happens. The following figure demonstrates this idea.Some possible events:File uploaded.E-Mail received.Database changed.Manual invoked.HTTP API called.Cron.The cool things about serverless architecture are:You only pay when the function is executed.No under/over provisioning.No boot time.No patching.No SSH.No load balancing.Read more about Serverless Architectures if you are interested in the details.What is a Microservice?Imagine a small system where users have a public visible profile page with location information of that user. The idea of a microservice architecture is that you slice your system into smaller units around bounded contexts. I identified three of them:Authentication Service: Handles authentication.Location Service: Manages location information via a private HTTP API. Uses the Authentication Service internally to authenticate requests.Profile Service: Stores and retrieves the profile via a public HTTP API. Makes an internal call to the Location Service to retrieve the location information.Each service gets its own database, and services are only to communicate with each other over well-defined APIs, not the database!Let's get started!The source code and installation instruction can be found at the bottom of this page. Please use the us-east-1 region! We will use services that are not available in other AWS regions at the moment.CodeAWS CodeCommit is a hosted Git repository that uses IAM for access control. You need to upload your public SSH key to your IAM User as shown in the following figure:Creating a repository is simple. Just click on the Create new Repository button in the AWS Management Console.We need a repository for each service. You can then clone the repository locally with the following command. Replace $SSHKeyID with the SSH Key ID of your IAM user and $RepositoryName with the name of your repository.git clone ssh://$SSHKeyID@git-codecommit.us-east-1.amazonaws.com/v1/repos/$RepositoryName` We now have a home for our code.Continuous Integration & Continuous DeliveryAWS CodePipeline is a service to manage a build and deployment pipeline. CodePipeline itself is only responsible triggering integrations to do things like:Build.TestDeploy.We need a pipeline for each service that:Downloads the sources from CodeCommit if something changes there.Runs our test and bundles the code in a zip file for Lambda.Deploys the zip file.Luckily, CodePipeline has native support for downloading sources from CodeCommit. To run our tests, we will use a third-party integration to trigger Solano CI to run our tests and bundle the source files. The deployment step is implemented in a Lambda function that triggers a CloudFormation stack update. A CloudFormation stack is a bunch of AWS resources managed by CloudFormation based on a template that you provide (Infrastructure as Code). Read more about CloudFormation on our blog.The following figure shows the pipeline:The cool thing about CloudFormation is that you can define the pipeline itself in a template. So we get Pipeline as Code.The CloudFormation template that is used for service deployment describes a Lambda function, a DynamoDB database, and an API Gateway. After deployment you will see one CloudFormation stack for each service:We now have a CI & CD pipeline.ServiceWe use a bunch of AWS services to run our microservices.Amazon API GatewayAPI Gateway is a service that offers a configurable REST API as a service. You describe what should happen if a certain HTTP Method (GET, POST,PUT, DELETE, ...) is called on a certain HTTP Resource (e.g. /user). In our case, we want to execute a Lambda function if an HTTP request comes in. API Gateway also takes care of mapping input and output data between formats. The following figure shows how this looks like in the AWS Management Console for the Profile Service.The API Gateway is a fully managed service. You only pay for requests, no under/over provisioning, no boot time, no patching, no SSH, no load balancing. AWS takes care of all those aspects.Read more about API Gateway on our blogAWS LambdaTo run code in AWS Lambda you need to:use one of the supported runtimes (Node.js (JavaScript), Python, JVM (Java, Scala, ...).implement a predefined interface.The interface in abstract terms requires a function that takes an input parameter and returns void, something, or throws an error.We will use the Node.js runtime where a function implementation looks like this:exports.handler = function(event, context, cb) { console.log(JSON.stringify(event)); // TODO do something cb(null, {name: 'Michael'}); }; In Node.js, the function is not expected to return something. Instead, you need to call the callback function cb that is passed into the function as a parameter.The following figure shows how this looks like in the AWS Management Console for the profile service.AWS Lambda is a fully managed service. You only pay for function executions, no under/over provisioning, no boot time, no patching, no SSH, no load balancing. AWS takes care of all those aspects.Read more about Lambda on our blogAmazon DynamoDBDynamoDB is a Key-Value-Store or Document-Store. You can lookup values by their key. DynamoDB replicates across multiple Availability Zones (data centers) and is eventually consistent.The following figure shows how this looks like in the AWS Management Console for the authentication service.Amazon DynamoDB is a 99% managed service. The 1% that is up to you is that you need to provision read and write capacity. When your service makes more request than provisioned, you will see errors. So it is your job to monitor the consumed capacity to increase the provisioned capacity before you run out of capacity.Read more about DynamoDB on our blogRequest FlowThe three services work together in the following way:The user's HTTP request hits API Gateway. API Gateway checks if the request is valid — if so, it invokes the Lambda function. The function makes one or more requests to the database and executes some business logic. The result of the function is then transformed into an HTTP response by API Gateway.We now have an environment to run our microservices.Logs, Metrics, and AlertingA Blackbox is very hard to operate. That's why we need as much information from the inside of the system as possible. AWS CloudWatch is the right place to store and analyze this kind of information:Metrics (numbers).Logs (text).CloudWatch also lets you define alarms on metrics. The following figure demonstrated how the pieces work together.Operational insights that you get out-of-the-box:Lambda writes STDOUTand STDERR to CloudWatch logs.Lambda publishes metrics to CloudWatch about the number of invocations, runtime duration, the number of failures, etc.API Gateway publishes metrics about the number of requests, 4XX and 5XX Response Codes, etc.DynamoDB publishes metrics about consumed capacity, the number of requests, etc.The following figure shows a CloudWatch alarm that is triggered if the number of throttled read requests of the Location Service DynamoDB table is bigger or equal to one. This situation indicates that the provisioned capacity is not sufficient to serve the traffic.With all those metrics and alarms in place, we now can be confident that we receive an alert if our system is not working properly.SummaryYou can run a high-quality system on AWS by only using managed services. This approach frees you from many operational tasks that are not directly related to your service. Think of operating a monitoring system, a log index system, a database, virtual machines, etc. Instead, you can focus on operating and improving your service's code.The following figure shows the overall architecture of our system:Serverless or FaaS does not force you to use a specific framework. As long as you are fine with the interface (a function with input and output), you can do whatever you want inside your function to produce an output with the given input. Read more

China moves closer to adopting controversial cybersecurity law

BEIJING China moved closer on Monday to adopting a controversial cybersecurity law, after parliament held a second reading of the draft rules, which carry significant consequences for domestic and foreign business and threaten greater censorship.China enforces widespread controls over the internet that it has sought to codify in law, and Chinese laws often go through multiple readings and drafts before they are adopted.The draft, presented before the standing committee of the National People's Congress, requires network operators to comply with social morals and accept the supervision of the government and public, official news agency Xinhua said.It also reiterated that Chinese citizens' personal data, as well as "important business data" must be stored domestically, adding that those wishing to provide that information overseas faced a government security evaluation.Parliament has not yet published the full second draft of the cybersecurity law and it is not clear when it may be passed. Cybersecurity has been a particularly irksome area in China's relations with economic partners such as the United States and the European Union, which see many recently proposed rules as unfair to foreign firms.Chinese officials say internet restrictions, including the blocking of popular foreign sites like Google and Facebook, are needed to ensure security against growing threats, such as terrorism. The first draft of the cybersecurity law, published almost a year ago, stiffened user privacy protection from hackers and data resellers but also boosted the government's powers to access and block dissemination of private information records that Chinese law deems illegal.China's broadly-defined regulations have been a source of concern, especially for foreign governments, multinational companies and rights activists, which worry that the government can interpret the law as it sees fit. Chinese companies have also been on the receiving end of government efforts to tighten control of the internet. Regulators last month set limits on the number of lucrative healthcare advertisements carried by Baidu Inc after a student died following an experimental cancer treatment he uncovered by using China's biggest internet search engine. (Reporting by Paul Carsten and Michael Martina; Editing by Clarence Fernandez) Read more

AWS Weekly 2016 #24

Monday, June 13Cloud Academy explained Everything You Ever Wanted to Know about Amazon Kinesis Firehose.Tuesday, June 14AWS SES Blog announced Amazon SES Now Supports Email Headers in Notifications.AWS announced More Details from Service Last Accessed Data.AWS announced AWS Config offers a new rule for assessing license compliance.AWS announced New Edge Location in New Delhi, India for Amazon CloudFront and Amazon Route 53.Server Density Blog wrote about How Spotify and GOV.UK handle on call, and more.Wednesday, June 15AWS announced Standardized Architecture for NIST High-Impact Controls on AWS Featuring Trend Micro Deep Security.AWS announced New AWS Public Data Set - IRS 990 Filing Data.AWS announced AWS CodeDeploy Available in Asia Pacific (Seoul) Region.AWS announced Amazon RDS for PostgreSQL now supports cross-region read replicas.Cloud Academy blogged about How to Deploy Apache Storm on AWS with Storm-Deploy.Thursday, June 16AWS announced New Amazon EC2 Spot Console Now Supports Spot Fleet and Spot BlocksAWS announced Amazon CloudWatch Events Available in the South America (São Paulo) and Asia Pacific (Seoul) RegionsTrek10 wrote about Serverless Secrets.Friday, June 17AWS announced Oracle Repository Creation Utility (RCU) and April PSU Patches are now available for Amazon RDS for Oracle.Is anything missing? Looking forward to your feedback! @andreaswittig or andreas@widdix.de. Read more

Older Post