
At Vandebron we have been organizing a regular hackathon for the last four years. Every three months we organize a two-day event. At first glance this seems quite an investment. Eight days a year, almost losing two working-weeks of productivity for your teams! 

Our company is like any other. Our roadmaps are stuffed, our backlogs are never-ending and pressure for delivering value to our customers is always present. Our ambitions are always higher than what we can handle with the amount of teams and people available. We like to say: ‘the energy transition can’t wait!’, but we sure do have to prioritize our projects very carefully.

However this does not stop us from organizing our quarterly hackathons. Most of the time our regular hackathons are light-weight. People within the company know how it works. We try not to waste too much time in ‘organizing’ the event. We get right to it. 

#### Reasons why you should be organizing (regular) hackathons:
- Fun - this reason does not need much explanation. Working on challenging, fun and creative ideas with uncertain outcome in a not-business-as-usual way. It makes you step out of your daily comfort zone and explore new things. 
- Walk the extra mile - Some team-members will have the energy, enthusiasm and commitment to use their spare time to fuel their curiosity and bring new insights to the workplace. These are the same people that you also expect to walk the extra mile if the team- or company objectives are at stake. This is in that same spare time! But in the end, if you value your teams to continuously think about new ideas, insights and work on out-of-the-box ideas, it is not a weird idea to create this environment within the company.
- Bottled up energy - our people are focused on reaching goals and objectives. Every day, every week and every sprint the focus is clear. This also means that there is not always time for creative or high risk escapades that could hurt the overall team objectives. This might give an unsatisfied feeling to people. If the bottled up energy can not be released, engineers might get frustrated. But maybe even more important, you might be missing opportunities for the company.
- Cross team collaboration - in an agile way of working the concept of the team is very important. At Vandebron we focus on teams staying together for a long period of time. This makes the relationship between individuals stronger, the knowledge of the product deeper and the team as a whole more effective. However, the company is bigger than your team. There might be different ways of connecting with other people within your company, but a hackathon is an ideal way of linking yourself up with people that you can learn from. It can really bring you new insights as an individual, and it will also be an investment for improved cross-team collaboration going forward.
- Learning organisation - as mentioned, hackathons give you an excellent opportunity to learn new things. For yourself, but definitely also for the company. In my experience I often see that topics get selected that have high-risk and high-reward kind of characteristics. These topics can be scary to touch, which make you go out of your comfort zone. This is where you learn the most! These high-risk and high-reward projects are also very likely to fail, meaning that the reward is not as high as expected, or the complexity and risks are even greater than anticipated. At these moments the pressure-cooker of a hackathon is very valuable, because it forces the participants to draw conclusions in a short time-frame. The insights gained from these projects can be used to further steer the roadmap. And last but not least, it supports building a culture of being bold enough to try new things, and fail fast. I’ve noticed this is appreciated by a lot of people within the company and the hackathon contributes to a culture of innovation.

#### Our most important learnings over the years
- Spotlights on - It is good to put teams and their results in the spotlight. Let them take the podium and make sure there is an audience. However don’t make it too much about winning. Ideas that have completely failed are just as important as over-engineered fancy product demos. At Vandebron we like to declare ‘winners’ in different categories: ‘Fun & Original’, ’Impactful’, ‘Exploration & Learning’ and ‘Technical achievement’. 
- Harvest your ideas continuously - during normal work and life you hit those topics that you would like to investigate a bit deeper. But while you stumble upon such a topic you don’t have the time to dive into it. So therefore, write your idea down and publish it in the ‘hackathon-idea-box’ for everyone to see! It might already give you some good conversations during coffee or lunch, and it might already generate you some people that would like to join forces with you during the hackathon. Because rest assured, a new hackathon is always coming up!
- To-theme-or-not-to-theme - we have experimented with adding a theme to a hackathon. It can help the company to generate ideas and action in a certain area of interest. It also helps to generate more focus within the company on a certain persistent challenge that we feel deserves a solution. Although everyone will be working on different sub-topics the full event will be experienced as more correlated and unified. But be careful not to push normal business-projects disguised as hackathon projects to your teams. This goes against the basic concept of a hackathon. At Vandebron we sometimes pick a theme if we would like to motivate people to think about ideas in a certain direction. But most of the time we keep it open.
- Participation is optional. - At Vandebron we have autonomous teams with professionals that can manage their own agenda. As a team and as an individual. We put effort in promoting the hackathon by trying to make people enthusiastic about participating. But in the end people make their own decisions. Sometimes the team and company objectives do need to have priority, but the teams are perfectly able to make this judgement call themselves.
- Magnify impact - show everyone what the impact is they have been making. It is good if people recognize how some projects have become reality and that feedback will be appreciated by the community. It gives people a feeling that the podium of the hackathon is a strong force. And ultimately that is how you also proof the value of organizing a hackathon.

For our next hackathon we are opening our (virtual) doors also for guests, as we are organizing a GreenTech hackathon with other sustainability minded companies (‘Hack the Planet’ and ‘Top Dutch Solar Racing’). You can find more information and sign up via [this link](https://www.vandebron.tech/greentech-hackathon). It is the first time we do it like this, and we sure will learn another thing or two!



In October 2020 D2IQ [announced](https://d2iq.com/blog/d2iq-takes-the-next-step-forward) that they are moving onwards with their Kubernetes offering. Vandebron has been a D2IQ customer for their DCOS offering, we were just in the middle of a migration of our first workloads to DCOS Enterprise. We have evaluated the D2IQ K8s offering and decided to go for another Kubernetes product. We had a few migrations over the years, we migrated from Azure to AWS, we migrated workloads from normal instances to spot instances and all these migrations were done with nearly any downtime. We plan to reduce the downtime to a couple of minutes this migration and this is a real challenge. The first challenge that we will discuss today: We want to pair our Kubernetes clusters to the DCOS/Mesos clusters, while we move a workload it should be able to connect to its dependencies in the DCOS cluster. We use DCOS for our NoSQL databases like Cassandra, internal data that we want to keep internal. Pairing DCOS and Kubernetes clusters enable us to reduce downtime, enabling us to switch back if we run into issues and move faster because it reduces complexity.

## L4LB

The internal layer 4 load balancer DCOS provides is used in the majority of our workloads. When our data scientists schedule a spark driver, they connect to the spark dispatcher through the Layer 4 load balancer. Most of the DCOS frameworks use this Layer 4 load balancer as an internal service discovery tool, with Vandebron we use this layer 4 load balancer to communicate between services. In a default DCOS set up this load balancer responds on domain names like: `spark-dispatcher.marathon.l4lb.thisdcos.directory:7077`

When we ping the spark dispatcher we get the following:

```bash
PING spark-dispatcher.marathon.l4lb.thisdcos.directory (11.155.161.35) 56(84) bytes of data.
64 bytes from 11.155.161.35 (11.155.161.35): icmp_seq=1 ttl=64 time=0.024 ms
```

After some investigation we found out that this IP range is not actually on a network interface, it is a Linux kernel functionality called `IPVS`. With IPVS you can do layer 4 load balancing, you provide the target location and the location you want to respond on.

When we search for the IP from the spark dispatcher with ipvsadm, we get 3 results:

```bash
sudo ipvsadm -L -n |grep --color '11.155.161.35\|$'
TCP  11.155.161.35:80 wlc
  -> 10.2.7.146:16827             Masq    1      0          0
TCP  11.155.161.35:4040 wlc
  -> 10.2.7.146:16826             Masq    1      0          0
TCP  11.155.161.35:7077 wlc
  -> 10.2.7.146:16825             Masq    1      0          0
````

As you can see the IP `11.155.161.35` points towards `10.2.7.146`, even the ports are configured and forwarded. We can add our route with ipvsadm, to understand IPVS a bit better. For example:

```bash
sudo ipvsadm -A -t 1.2.3.4:80 -s wlc # we add the target server and assign the scheduler
sudo ipvsadm -a -r 10.2.7.146:16825 -t 1.2.3.4:80 -m # we configure the real server and target server and configure Masquerading
curl 1.2.3.4:80
{
  "action" : "ErrorResponse",
  "message" : "Missing protocol version. Please submit requests through http://[host]:[port]/v1/submissions/...",
  "serverSparkVersion" : "2.3.4"
}
```

This results in that the spark dispatcher now also is available on `1.2.3.4:80`. As mentioned before we wanted to connect our DCOS and Kubernetes clusters, getting hundreds of entries from ipvsadm and manually adding them one by one didn’t sound appealing to us. Especially if you consider that sometimes services fail and run on a different port or different host after recovery, maintaining this by hand would be a nightmare. We therefore decided to build a tool to sync IPVS entries from DCOS to Kubernetes.

## Stack

Within Vandebron we have our tech stack, we strongly believe it is good to eat your own dog food. When possible and when our use cases are similar we use the same tools as our Developers use. The parts of the stack we will be using are:

- AWS ELB in front of Traefik 1.7
- DCOS
- Kubernetes

Within our platform team, we use Golang as our scripting language. Golang gives us the ability to build binary files with all the required libraries in the binary, we don’t have to install any packages, we do not even need to install Golang on the machine the application will be running on.

In our DCOS cluster we use Traefik 1.7, this version of Traefik only forwards HTTP requests. We decided to use Traefik to expose a JSON endpoint so we can gather the IPVS information from this location.

## ipvs-server

Within our DCOS cluster we will expose the IPVS information through a JSON endpoint. We have built a tool for this to expose this information in multiple ways. In the next section, we are going to discuss some of the concepts and choices we made, we won’t deep dive into Go specifics. We have provided the entire code for this project in the examples directory of our GitHub repo:
<https://github.com/Vandebron/tech-blog>

First, let’s discuss the library we use: <https://github.com/nanobox-io/golang-lvs>. This library in its essence translates to ipvsadm commands, it helped save us time to implement this ourselves. There are some gotcha’s, such as newlines are not filtered out from the output. We solved this by cleaning up some of the data.

In the `childChan` function we create a go channel that is responsible for polling `ipvsadm` every 10 seconds and stores the result in a couple of variables we use in our HTTP endpoints. IPVS is a Linux kernel functionality and should be highly performant, we do not want to trigger kernel panics when the server gets overloaded with requests. We expect that every 10 seconds gives us accurate enough results, we can always lower this interval to ensure faster results. We also added in this function the string manipulation to ensure all the newlines were gone in the JSON output. The newline gave issues when we tried to add the IPVS scheduler entries.

```go
func childChan(c chan bool) {
   fmt.Println("Starting time based IPVS Admin poll")

   pollInterval := 10
   timerCh := time.Tick(time.Duration(pollInterval) * time.Second)
   // Time based loop to generate Global variable
   for range timerCh {
       select {
       // when shutdown is received we break
       case <-c:
           fmt.Println("Received shutdown, stopping timer")
           break
       default:
           var err error
           listIpvs.Save()
           ipvsString = fmt.Sprintln(listIpvs.Services)

           res := &responseObject{
               Services: listIpvs.Services,
           }
 
           ipvsJSONbyte, err := json.Marshal(res)
           if err != nil {
               logToErr.Printf("ERROR: -- Marshal JSON -- %v\n", err)
           }
 
           ipvsString = string(ipvsJSONbyte)
           ipvsJSON = strings.Replace(ipvsString, `\n`, ``, -1)
           if debug != false {
               logToOut.Println("DEBUG: -- ipvsJSON --", ipvsJSON)
           }
       }
   }
}
```

Next is the index handler, we set our headers correctly and print the result as we would receive through ipvsadm. The index is mainly for our platform engineers to debug and verify the output. Thanks to this overview we found much faster that there was a newline hidden in the scheduler output.

```go
func index() http.Handler {
   // Generating the Index
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

       // Only available when debug is on
       if debug != false {
           logToOut.Println("DEBUG: -- index --", ipvsString)
       }
 
       if r.URL.Path != "/" {
           http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
           return
       }
       w.Header().Set("Content-Type", "text/plain; charset=utf-8")
       // Site security testers expect this header to be set
       w.Header().Set("X-Content-Type-Options", "nosniff")
       w.WriteHeader(http.StatusOK)
       fmt.Fprintln(w, ipvsString)
   })
}
```

The JSON endpoint is what we use in the client communicate with the server. 

```go
func jsonz() http.Handler {
   // Generating the Index
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

       // Only available when debug is on
       if debug != false {
           logToOut.Println("DEBUG: -- jsonz --", ipvsJSON)
       }
 
       if r.URL.Path != "/json" {
           http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
           return
       }
       w.Header().Set("Content-Type", "application/json; charset=utf-8")
       // Site security testers expect this header to be set
       w.Header().Set("X-Content-Type-Options", "nosniff")
       w.WriteHeader(http.StatusOK)
       fmt.Fprintln(w, ipvsJSON)
   })
}
```

We ask our Developers often to implement a basic health endpoint, in DCOS we use this to see if a service needs to be restarted. In our application we enable set the statusOK in the index or in the JSON endpoint.

```go
func healthz() http.Handler {
   // Generating the healthz endpoint
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
       if atomic.LoadInt32(&healthy) == 1 {
           w.WriteHeader(http.StatusNoContent)
           return
       }
       w.WriteHeader(http.StatusServiceUnavailable)
   })
}
```

In our logging and tracing functions we want to register the clients that are connecting, this gives us information where calls are coming from. It helps us debugging if we see weird behaviour.

```go
func tracing(nextRequestID func() string) func(http.Handler) http.Handler {
   // Tracing the http requests so its easier to check if server is reached
   return func(next http.Handler) http.Handler {
       return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
           requestID := r.Header.Get("X-Request-Id")
           if requestID == "" {
               requestID = nextRequestID()
           }
           ctx := context.WithValue(r.Context(), requestIDKey, requestID)
           w.Header().Set("X-Request-Id", requestID)
           next.ServeHTTP(w, r.WithContext(ctx))
       })
   }
}

func logging(logToOut *log.Logger) func(http.Handler) http.Handler {
   // Creating logging entry tracing the http requests
   return func(next http.Handler) http.Handler {
       return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
           defer func() {
               requestID, ok := r.Context().Value(requestIDKey).(string)
               if !ok {
                   requestID = "unknown"
               }
               logToOut.Println(requestID, r.Method, r.URL.Path, r.RemoteAddr, r.UserAgent())
           }()
           next.ServeHTTP(w, r)
       })
   }
}
```

IPVS needs to be executed with root privileges, to ensure this is correct we get the userid and print it when starting the server.

```go
// getProcessOwner function is to see who is running the process. It needs to be a sudo / root user
func getProcessOwner() string {
   stdout, err := exec.Command("ps", "-o", "user=", "-p", strconv.Itoa(os.Getpid())).Output()
   if err != nil {
       logToErr.Printf("ERROR: -- getProcessOwner -- %v\n", err)
       os.Exit(1)
   }
   return string(stdout)
}
```

We added the init function to ensure we print results the moment the server starts up, if we would not do this it would take 10 seconds for the go channel to activate

```go
func init() {
   // Placing the Save and val in the init, else we will need to wait for channel to perform its first run
   listIpvs.Save()
   ipvsString = fmt.Sprintln(listIpvs.Services)
}
```

In the main function, we set the configurable flags, such as debugging to show error messages. It proved useful during the creation of this tool to keep track and print output. If we would print the output at every call to our logs, our Elastic cluster would get thousands of logs that add little to no value.

We configure the listen port in the flags, we can use the portIndex from DCOS to assign a random port on the host to listen on. We also provided to print the version we are running. In our versioning, we use a constant to list the application semver version, we also provide the git-commit hash.
When we begin the server we print the version information, the port we listen on and the user running the process. We then start the server process with the go channel, in setting up the go channel we ensure that when the server stops we try to gracefully stop the server within a 30-second timeframe. Since our ipvsadm timer is 10 seconds it should be able to cleanly shutdown within that period.

### Docker build

In the repository, we have included a Dockerfile and a script to build the Dockerfile. In this Dockerfile, we pass the git commit hash to the go install. This way we always get the Git Hash from our GitHub repo and we can use this information in our version output.

### DCOS service.json

In the repository, we have provided the service.json file, since it is opinionated on using Traefik you might need to change it. But in this service.json you see how we set up Traefik, the health check, and port index. Since the Mesos UCR container has fewer abstractions and has fewer limited capabilities. We can run the IPVS server inside a UCR container and get all the output as if we were running this directly as root on the host machine.

## ipvs-client

The IPVS client is the component we use in the Kubernetes environment. The client connects to the server and gets the IPVS entries from the IPVS server inside our DCOS cluster. It then adds these IPVS entries to each node in the Kubernetes cluster. You, therefore, need to run each client per Kubernetes node.

You can find the code from the IPVS client in our repository.

```go
func httpGet(remoteURL string) []byte {
   if debug != false {
       _, err := url.ParseRequestURI(remoteURL)
       if err != nil {
           panic(err)
       }
   }

   req, err := http.NewRequest(http.MethodGet, remoteURL, nil)
   if err != nil {
       logToErr.Fatalf("ERROR: -- new HTTP request -- %v", err)
   }

   ipvsClient := http.Client{
       Timeout: time.Second * 2, // Timeout after 2 seconds
   }
   req.Header.Set("User-Agent", "go-ipvs-get \tversion: "+version+"\t Git Commit: "+gitCommit)
   res, err := ipvsClient.Do(req)
   if err != nil {
       logToErr.Fatalf("ERROR: -- ipvsClient -- %v\n", err)
   }

   if res.Body != nil {
       defer res.Body.Close()
   }

   body, readErr := ioutil.ReadAll(res.Body)
   if readErr != nil {
       logToErr.Fatalf("ERROR: -- body -- %v\n", readErr)
   }

   return body
}
```

In the httpGet function we can debug the URL and check if it is valid. Again we set the correct headers and retrieve the JSON body.

```go
func unmarshal(body []byte) []lvs.Service {

   res := &responseObject{
       Services: listIpvs.Services,
   }

   jsonErr := json.Unmarshal(body, &res)
   if jsonErr != nil {
       logToErr.Fatalf("ERROR: -- Unmarshal -- %v \n", jsonErr)
   }

   if debug != false {
       logToOut.Fatalf("DEBUG: -- res -- %v \n", res.Services)
   }

   r := res.Services

   return r
}
```

In the unmarshal function we unmarshal the JSON and turn it in a slice of lvs.Service.

```go
func addServers(remoteAddr string) {
   body := httpGet(remoteAddr)
   jsonData := unmarshal(body)

   for i, v := range jsonData {
       if debug != false {
           logToOut.Printf("DEBUG: -- range jsonDATA --\n")
           logToOut.Printf("ipvsCount=%v, value=%v", i, v)
       }

       err := lvs.DefaultIpvs.AddService(v)
       if err != nil {
           logToErr.Printf("ERROR: -- AddService -- %v", err)
       }
 
       i++
       ipvsServerCount = float64(i)
   }
}
```

In the addServers function we add the servers to IPVS.

```go
func clientChan(c chan bool) {
   logToOut.Println("Starting time based IPVS Admin add")

   pollInterval := 10
   timerCh := time.Tick(time.Duration(pollInterval) * time.Second)
   // Time based loop to generate Global variable
   for range timerCh {
       select {
       // when shutdown is received we break
       case <-c:
           logToOut.Println("Received shutdown, stopping timer")
           break
       default:

           logToOut.Println("Clearing & Adding servers...")
           // Before we add Servers we need to clear the existing list
           lvs.Clear()
           addServers(remoteAddr)
           if debug != false {
               logToOut.Printf("IPVS servers added:\t%v", ipvsServerCount)
           }
       }
   }
}
```

Like we did in the IPVS server we create a go channel to poll every 10 seconds the server endpoint. We perform this to get at a set interval the IPVS entries.

Since we run the IPVS client as a binary directly on the Kubernetes hosts we build the binary with a few parameters we pass to the go build command. The binary we build with this command we host on an internal s3 bucket, we can download this binary with systemd unit files.

```bash
GOOS=linux
GOARCH=amd64
GIT_COMMIT=$(git rev-list -1 HEAD)

export GOOS
export GOARCH
export GIT_COMMIT

env GOOS=${GOOS} GOARCH=${GOARCH} go build -v -ldflags "-X main.gitCommit=${GIT_COMMIT}" .
```

When we run the IPVS client we can verify if the IPVS routes are added by running the `ipvsadm -L -n` command.

### Unit files

Since IPVS is part of the Linux kernel it is hard to deploy this in a docker container, the capabilities are more restricted in Kubernetes. We decided to deploy the IPVS client on each host machine through a systemd unit file, the main reason was that we ran into restrictions that slowed us down and this is not a permanent solution. By adding the IPVS client on the machines alone does not make it possible for containers to use the IPVS routes. We needed to add NET_ADMIN capabilities to all containers using the l4lb loadbalancer locations and configure `hostNetworking: true` in the Kubernetes pods.

We provided a deployment.yml file that runs a Ubuntu docker container with ipvsadm only installed extra. When the pods are deployed in this deployment you can use kubectl exec to get into the pod and run the `ipvsadm -L -n` command.

## Vacancy at Vandebron

We are looking for a platform engineer in Vandebron. As you can understand this is not a typical scenario we daily run across, but it is part of the workloads that we will support when working on our platform. Within Vandebron we try to use the best technology available, when it is not available we build it. Due to this as platform engineers, we have many interesting challenges and offer engineers to support further than only a strict domain. We support all components of our entire platform, regardless if it is a Linux kernel issue like this, involves setting up and maintaining a NoSQL cluster, or helping the business with something like requesting a certificate.

If you are interested in learning more about this position, take a look at our Vacancy and get in contact with us.
<https://werkenbij.vandebron.nl/>



Cypress is a game-changer in the automation testing world, the way that Cypress was built and its architecture allows us as testers to cover more scenarios.

Cypress is not Selenium; in fact, it is different. And the way to build and design a framework should be different as well.

The most famous design technique in Selenium is the Page Object Model, and many testers use the same design technique with Cypress. Even that Cypress on their official website [recommended](https://www.cypress.io/blog/2019/01/03/stop-using-page-objects-and-start-using-app-actions/) us not to go with that approach.

## Page Object Model

The main benefit of using the page object model Is to make the automation framework maintenance-friendly. We can define a specific page's selectors in a separate file and then use these selectors in our test cases.

```js
class SignInPage {
  visit() {
    cy.visit("/signin");
  }
  getEmailError() {
    return cy.get(`[data-testid=SignInEmailError]`);
  }
  getPasswordError() {
    return cy.get(`[data-testid=SignInPasswordError]`);
  }
  fillEmail(value) {
    const field = cy.get(`[data-testid=SignInEmailField]`);
    field.clear();
    field.type(value);
    return this;
  }
  fillPassword(value) {
    const field = cy.get(`[data-testid=SignInPasswordField]`);
    field.clear();
    field.type(value);
    return this;
  }
  submit() {
    const button = cy.get(`[data-testid=SignInSubmitButton]`);
    button.click();
  }
}
export default SignInPage;
```

The main two downsides using the typical page object model with cypress are:

- Page objects introduce an additional state into the tests, separate from the application’s internal state. This makes understanding the tests and failures harder.
- Page objects make tests slow because they force the tests to always go through the application user interface.

## Component-Based Architecture

On the other hand, a React application is component-based, where a specific page will be built from a collection of components. And components in React can be used on different pages too. So if we want to use the Page Object Model, we may define the same locator twice on different pages.

So having these two facts, At Vandebron, we came up with a new way to design our Cypress Automation framework by creating a separate JavaScript file for every component in our application, inside a folder called `components` within our Cypress project as below:

```js
// Locators
export const getEmailError = () => cy.get(`[data-testid=SignInEmailError]`);
export const getPasswordError = () =>
  cy.get(`[data-testid=SignInPasswordError]`);
export const emailField = () => cy.get(`[data-testid=SignInEmailField]`);
export const passwordField = () => cy.get(`[data-testid=SignInPasswordField]`);
export const submitButton = () => cy.get(`[data-testid=SignInSubmitButton]`);

// Actions
export const visit = () => cy.visit("/signin");
export const performLogin = (email, password) => {
  emailField().clear().type(email);
  passwordField().clear().type(password);
  submitButton().click();
};
```

Having it built this way, we eliminated all the previous problems mentioned earlier; we are not adding any classes, and we are defining objects within our test cases. And the most important part is that we are following the way that Cypress recommends it.

And after defining the component locators and actions, we can import them inside our test case and use them as below:

```js
import LoginComponent from "../components/loginComponent";
import Menu from "../components/Menu";

describe("Test Login Page", () => {
  it("should show an error message if the password in wrong", () => {
    LoginComponent.visit();
    LoginComponent.performLogin("email@gmail.com", "wrongPassword");
    LoginComponent.getPasswordError().should("be.visible");
  });
  it("should show the logout button if the user logged in succesfully", () => {
    LoginComponent.visit();
    LoginComponent.performLogin("email@gmail.com", "correctPassword");
    Menu.LogoutButton().should("be.visible");
  });
});
```

And as you can see, our test cases are readable for anyone! And if any locator changes in any of the components, we can easily fix it in one location and from the same file. And lastly, if a component will be used in different places, we can use the same code.

In the next article, I will talk about how we use Cypress in our manual testing during the sprint and how it saves us tons of time and effort.



In Vandebron we have been using container clusters to host our services since the foundation of our Big Data team. 
Recently our cluster of choice has declared End-Of-Life development stage, so we decided to take a step forward and get a ticket for the Kubernetes boat.

A change in the OS that is used to run your services and applications can look quite challenging and not everyone is on the same experience level. To make everyone comfortable it is a good choice to give everyone the possibility to play with the new tools and learn what can be done and how: **you need a sandbox.**

Our developers are provided with a Macbook and at the moment of writing there some options you can go for when deciding how to set up your playground:

- **Docker CE Kubernetes**: This is the easiest solution since there is a handy button to run your containers into a Kubernetes environment.

- **Vagrant and Virtualbox**: This solution is the one that can give you more control and you can easily create a cluster the size you want, but you need to be handy with VMs, the hypervisor of choice, and Vagrant. It's the old school way to do it but, while it's a chunk your platform engineers can bite, it can be a steep and frustrating process for people that are not used to handle VMs.

- **Multipass + some bash magic glue**: Since Canonical created this tool for macOS, creating an Ubuntu VM became a breeze and you can have a single, easily manageable VM with its networking up and running in less than a minute, without having to handle disks, distros, and stuff. On top of it, the command line interface is straight forward and it has just the basic commands we will need, so wrapping the entire process into a bash script is a piece of cake.

I have found this super cool in-depth [article](https://jyeee.medium.com/kubernetes-on-your-macos-laptop-with-multipass-k3s-and-rancher-2-4-6e9cbf013f58) from Jason Yee (kudos to you bruh) that guided me through the installation of my first single node Kubernetes cluster.

The process is not that long but it involves a lot of copy/pasting and, once learned the basics, I didn't want to go under the same process more times, plus it could be interesting for me as a Platform Engineer, but it may be boring and pointless for developers who just want to have a sandbox replica of what they are working on in the remote environment.
My automator (aka do-it-once-never-do-it-again) spirit kicked in and I decided to wrap every step in a small command-line tool with only 3 options:
- **install**
- **cleanup**
- **help**


### What is happening under the hood

What the script does is substantially automating all the steps needed to:
1. Create a new VM using Multipass (tool released by Canonical)
2. Fetch the VM IP address and adding it to your local `/etc/hosts` file
3. Install k3s (a lightweight distribution of Kubernetes) on top of the VM
4. Install the Kubernetes command-line tools on your laptop
5. Install Helm (the Kubernetes package manager) on your laptop
6. Install cert-manager (certificate manager) package on top of your k3s cluster
7. Install Rancher (a Kubernetes control plane) package on top of your k3s cluster

If you are looking for a more in-depth breakdown of the single steps you can download and inspect [the script](https://gist.githubusercontent.com/nikotrone/50b1a5f8d137411879eb2467e689bfbe/raw/090b4b4323d96ac28d96bbb346e2e657073722e6/bronernetes) (one of the many advantages of [OpenSource](https://en.wikipedia.org/wiki/Open_source) projects) or checkout and read the original [article](https://jyeee.medium.com/kubernetes-on-your-macos-laptop-with-multipass-k3s-and-rancher-2-4-6e9cbf013f58): it explains line by line what the specific commands are doing.

#### 1. Multipass VM
[Multipass](https://multipass.run/) is a tool from Canonical (the company developing and maintaining the Ubuntu Linux distribution) that leverages Hyperkit (macOS feature to handle virtualization) to create and handle a Virtual Machine directly on your Mac.

#### 2. Edit /etc/hosts
Once we have our VM up and running we need to make it available with an easy url that is also gonna be used to generate the SSL certificate, in our case we picked up `rancher.localdev`.
It is important to have a name setup in the beginning since this one will need to match with the certificate so we can use it programmatically.

#### 3. Install K3S
This step is pretty straightforward: just fetch a script that is publicly available on the [k3s official website](https://get.k3s.io) and feed it to your bash.
K3s is a lightweight version of Kubernetes with all the needed dependencies and executable packaged in a convenient installation script. Because of its light nature, it is often used in embedded devices that have a limited amount of resources to offer.

#### 4 & 5. Kubernetes and Helm cli
**Kubernetes cli** (`kubectl`) is used to talk and interact with your Kubernetes cluster. It can be used to manage multiple clusters according to the content of your KUBECONFIG environment variable. 
The variable itself contains just a path to where your cluster configuration is stored, so you can switch from a cluster to another by simply pointing to another file that contains the configuration of another cluster.

**Helm** instead is the "package manager" of Kubernetes: you can use it to add repositories to specific `charts` which are the blueprint that contains a way to install a specific tool on your cluster.
Both of these tools have to be installed and run from your local laptop, either in the case you are managing a local VM or in the case you are interacting with a remote cluster.

#### 6 & 7. cert-manager and Rancher

**Rancher** is the control plane for our cluster: it provides a GUI and an overview of our single node cluster. It offers other goodies like management of multiple clusters, deployed on different locations like AWS Azure and GCP or even on your own hardware, plus certificate deployment and some other handy functionalities.

**cert-manager** is installed via Helm chart and it is the tool used by Rancher to generate and deploy a certificate across the entire cluster.

### How to use it

All the steps will involve the use of a Terminal window
#### Installation
The first thing you need to do is download [this script](https://gist.github.com/nikotrone/50b1a5f8d137411879eb2467e689bfbe) and save it in a folder on your Mac (let's assume `~/bronernetes`) by executing
```bash
    mkdir ~/bronernetes
    cd ~/bronernetes
    curl https://gist.githubusercontent.com/nikotrone/50b1a5f8d137411879eb2467e689bfbe/raw/090b4b4323d96ac28d96bbb346e2e657073722e6/bronernetes > bronernetes
    export PATH=$PATH:$(pwd)
```

Now we have the toolset and you can confirm it works by simply running `bronernetes help`.

#### Spin up Kubernetes
The next step is to run the installation process with the command `bronernetes install`

#### Clean up
When you are done or you just want to hard reset your environment you can just type `bronernetes cleanup` and it will take care of cleaning up the VM you just used, leaving you with a pristine machine, as nothing ever happened :)

### Conclusion

Having a sandbox is very useful to play around with the concepts of a new setup or service and it packs up a huge amount of positive sides. No matter what is the language or the nature of the system you are trying to replicate, it can be challenging and involve a long list of instructions or manual operations and, sometimes, even dedicated hardware. Although with some bash glue, it is possible to automate most of those processes and the investment cost can be enormously beneficial for yourself (less work the next time you do it) and for the other people working with you (they can use the tool, comment and suggest improvements). Most of all, in the case of infrastructure, it helps raise the knowledge of "what's going on here" and documents for the ones interested in taking a trip down the rabbit hole.


At Vandebron we're maintaining a component library called [Windmolen](https://windmolen.netlify.app/) (Dutch for "wind turbine"). And if you've ever built a component library, you probably dealt with optimizing and converting icons before. With SVGO and SVGR you can do this at scale, without compromising the quality or size of your icons.

## The problem

The web is full of icons, and often these icons are rendered from SVG files to ensure you can increase (or decrease) the size of the icons depending on the use case. Designers often create these icons from design tools like Adobe Photoshop or Sketch. Although these icons might look pretty, exporting a SVG out of these tools is often difficult as [this article](https://medium.com/sketch-app-sources/the-best-way-to-export-an-svg-from-sketch-dd8c66bb6ef2) explains. Also, added lot of code in the form of metadata is added to the SVG file. Let's have a look at what a typical SVG file exported out of Sketch looks like:

```svg

<?xml version="1.0" encoding="UTF-8"?>
<svg width="14px" height="14px" viewBox="0 0 14 14" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
    
    <title>last</title>
    <desc>Created with Sketch.</desc>
    <defs></defs>
    <g id="Page-1" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
        <g id="last" transform="translate(2.000000, 0.000000)" fill-rule="nonzero" fill="#666666">
            <polygon id="Fill-2" points="6.6902923 9.6812703 9.3700469 7.0005052 6.6902923 4.3187297 2.37257308 0 0 2.37358354 4.3177192 6.6902923 4.6279322 7.0005052 4.3177192 7.3107182 0 11.6274269 2.37257308 14"></polygon>
        </g>
    </g>
</svg>
```

The SVG file above holds a lot of information about Sketch, such as the `title` of the icon and a `desc`ription. Next to that, there's a lot of elements that could be combined into one element to reduce the file size.

## Optimizing SVGs

What's cool about SVG files is that you can optimize and minify them, without affecting what the SVG looks like. This is something you can try out yourself using the website [SVGOMG](https://jakearchibald.github.io/svgomg/), which is powered by the library SVGO that you'll learn more about later.


You can optimize the SVG file above by following these steps:

1. Go to [https://jakearchibald.github.io/svgomg/](https://jakearchibald.github.io/svgomg/)
2. Click on `Paste markup` an paste the SVG code that you exported from Sketch (a.k.a. the SVG file above)
3. You will see the icon rendered, now you have to either click at the `Copy as a text` or `Download` button to get the optimized SVG file

With these simple steps you've optimized the SVG from over 450 bytes, which is already small, to 173 bytes (a decrease of over 62%!). If you'd open this file in the editor of your choice, you can see a lot of the useless (meta)data from the original file has been deleted. Also, the different elements of the SVG are combined in a single `path` that renders the icon:

```svg

<svg width="14" height="14" xmlns="http://www.w3.org/2000/svg">
  <path d="M8.69 9.681l2.68-2.68-2.68-2.682L4.373 0 2 2.374 6.318 6.69l.31.31-.31.31L2 11.628 4.373 14z" fill-rule="nonzero" fill="#666"/>
</svg>
```

This SVG can be even further optimized by checking the "Prefer viewbox to width/height" in SVGOMG, but let's save that for later when we use SVGO instead.

## Using SVGO

By using SVGOMG you've already experienced what power [SVGO](https://github.com/svg/svgo) has, as SVGOMG is described by its creators as *" SVGO's Missing GUI, aiming to expose the majority if not all the configuration options of SVGO"*. Instead of using the GUI, you can also use SVGO directly from the command line as a CLI-tool or as a Node.js module. For the sake of this article, we'll be using it solely as CLI.

SVGO can be installed globally on your machine, or locally in your project, from npm by running:

```bash
npm i -g svgo

# Yarn equivalent
yarn add -G svgo
```

After doing this you can run `svgo` from the command line and optimize any SVG file instantly. But, you don't want to do this manually on your machine anytime you're adding a new icon to a project (or component library). Therefore, you can also add SVGO to a project locally and add a script to the `package.json` file to optimize all SVGs in a certain directory.

```json
// package.json
{
 // ...
 "scripts": {
     // ...
    "optimize-svg": "svgo --config=.svgo.yml -f ./src/assets/icons"
 }
}
```

The `optimize-svg` script will run SVGO in the directory `src/assets/icons` and optimize all the SVG files based on the settings in `.svgo.yml`. This file is where you can configure the rules for SVGO, as the previously mentioned "Prefer viewbox to width/height":

```yaml
# .svgo.yml
plugins:
  - removeViewBox: false
  - removeDimensions: true # this deletes width/height and adds it to the viewBox
  - removeDoctype: true
  - removeComments: true
  - removeMetadata: true
  - removeEditorsNSData: true
  - cleanupIDs: true
  - removeRasterImages: true
  - removeUselessDefs: true
  - removeUnknownsAndDefaults: true
  - removeUselessStrokeAndFill: true
  - removeHiddenElems: true
  - removeEmptyText: true
  - removeEmptyAttrs: true
  - removeEmptyContainers: true
  - removeUnusedNS: true
  - removeDesc: true
  - prefixIds: false
  - prefixClassNames: false
```
   
From the rules above you'll get an idea about all the redundant and useless lines of code that might be present in your SVG files. But luckily, they will all get removed when you run the command `npm run optimize-svg`.

## Converting SVGs with SVGR

You've now learned how to optimize your SVG files, and are probably wondering how to use these files in a React application. To render an SVG in React, you need to either configure Webpack in a way that it knows how to deal with SVG files or use a library called SVGR. By default, any application created with `create-react-app` can render SVG files as a component, using the following `import` statement:

```jsx
// MyComponent.jsx
import React from 'react';
import { ReactComponent as MySVG } from './something.svg';

const MyComponent = () => {
  return (
    <div>
      <MySVG />
    </div>
  );
}
export default MyComponent;
```

More information about how this is done can be found in [this article](https://blog.logrocket.com/how-to-use-svgs-in-react/), but let me show you how to solve that with SVGR.

With [SVGR](https://react-svgr.com/) you can convert SVG files into React Components, either by adding it to Webpack or by using the SVGR CLI or Node.js module. In the same way, as we optimized the SVGs from the command line with SVGO, we can also convert these icons from the command line with SVGR:

```json
// package.json
{
 // ...
 "scripts": {
     // ...
    "optimize-svg": "svgo --config=.svgo.yml -f ./src/assets/icons",
    "convert-svg": "svgr -d ./src/components/Icon ./src/assets/icons"
 }
}
```

Whenever you run the command `npm run convert-svg` a JSX file will be created for every SVG file that's present in the directory `src/assets/icons`. These JSX files can be found in the directory `src/components/Icons`, together with an `index.js` file that exports all these components from this directory.

An example of such a converted SVG file is:


```jsx
// MySVG.jsx
import * as React from 'react';

const MySVG = (props) => (
  <svg viewBox="0 0 14 14" xmlns="http://www.w3.org/2000/svg" {...props}>
  <path d="M8.69 9.681l2.68-2.68-2.68-2.682L4.373 0 2 2.374 6.318 6.69l.31.31-.31.31L2 11.628 4.373 14z" fill-rule="nonzero" fill="#666"/>
  </svg>
);

export default MySVG;
```

And, as we now have a directory filled with converted SVGs these can be imported into any React component like this:

```jsx
// MyComponent.jsx
import React from 'react';
import MySVG from './MySVG.jsx';

const MyComponent = () => {
  return (
    <div>
      <MySVG />
    </div>
  );
}
export default MyComponent;
```

Often SVGR is used alongside SVGO, so you can even automatically optimize all SVGS that will be converted by SVGR. This is done by adding the flag `--no-svgo true` and point it towards your SVGO configuration file:

```json
// package.json
{
 // ...
 "scripts": {
     // ...
    "convert-svg": "svgr -d ./src/components/Icon ./src/assets/icons --no-svgo true --svgo-config .svgo.yml"
 }
}
```

By running the `convert-svg` script you both optimize and convert all the SVG files in `src/assets/icons` to React components based on optimized SVGs.

## Reading further

The examples in this post are the tip of the metaphorical iceberg on what problems SVGO and SVGR can solve. There are many other features you can enable, such as using them as Node.js modules or enabling TypeScript support. To read further make sure to have a look at the SVGR [playground](https://react-svgr.com/playground/) or [documentation](https://react-svgr.com/docs/getting-started/).


Here at Vandebron, we have several projects which need to compute large amounts of data. To achieve acceptable results, we had to choose a computing tool that should have helped us to build such algorithms.

As you may have read in other articles our main backend language is Scala so the natural choice to build distributed parallel algorithms was indeed Spark.

## What is Spark

We will briefly introduce Spark in the next few lines and then we will dive deep into some of its key concepts.

Spark is an ETL distributed tool. ETL are three phases that describe a general procedure for moving data from a source to a destination.

![ETL Diagram](/images/etlprocess.png "ETL")

- **_Extract_** is the act of retrieving data from a data source which could be a database or a file system.
- **_Transform_** is the core part of an algorithm. As you may know, functional programming is all about transformation. Whenever you write a block of code in Scala you go from an initial data structure to a resulting data structure, the same goes with Spark but the data structures you use are specific Spark structures we will describe later.
- **_Load_** is the final part. Here you need to save (load) the resulting data structure from the transformation phase to a data source. This can either be the same as the extract phase or a different one.
- **_Distributed_**: Spark is meant to be run in a cluster of nodes. Each node runs its own JVM and every Spark data structure can/should be distributed among all the nodes of the cluster (using serialization) to parallelize the computation.

### Spark data structure: RDD, DataFrame, and Dataset

The core of Spark is its _distributed resilient dataset (RDD)_.

![Spark API history](/images/sparkapihistory.png "Spark API history")

An **_RDD_** is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel. _Extracting_ data from a source creates an RDD. Operating on the RDD allows us to _transform_ the data. Writing the RDD _loads_ the data into the end target like a database for example). They are made to be distributed over the cluster to parallelize the computation.

A **_DataFrame_** is an abstraction on top of an RDD. It is the first attempt of Spark (2013) to organize the data inside and RDD with an SQL-like structure. With dataframe, you can actually make a transformation in an SQL fashion. Every element in a dataframe is a Row and you can actually transform a dataframe to another by adding or removing columns.

A **_DataSet_** finally is a further abstraction on top of a dataframe to organize data in an OO fashion (2015). Every element in a dataset is a case class and you can operate transformation in a scala fashion from a case class to another.

## Spark in action

Let’s see now some code samples from our codebase to illustrate in more detail each of the ETL phases.

### Extract

The extraction phase is the first step in which you gather the data from a datasource.

```scala
val allConnections = sparkSession
.read
.jdbc(connectionString, tableName, props)

val selectedConnections = allConnections
.select(ColumnNames.head, ColumnNames.tail: _*)

val p4Connections = selectedConnections
.filter(allConnections("HasP4Day activated").equalTo(1))
.filter(allConnections("HasP4INT activated").equalTo(1))
.as[Connection]

p4Connections.show()
```

For most people the extraction phase is just the first line (the invocation to the read method), they are not wrong because extracting means reading data from a datasource (in this case an SQL server database). I decided to include in this phase also some filtering and projection operations because I think these are not really part of the algorithm, this is still the preparation phase before you actually process the data. We can ultimately say that _preparing the data_ is something in between extraction and transformation therefore it is up to you to decide which phase it belongs to.

### Transform

Transformation phase is the core of the algorithm. Here you actually process your data to reach your final result.

```java scala
usageDF
.groupBy('ConnectionId, window('ReadingDate, "1 day"))
.agg(
    sum('Consumption).as("Consumption"),
    sum('OffPeak_consumption).as("OffPeak_consumption"),
    sum('Peak_consumption).as("Peak_consumption"),
    sum('Production).as("Production"),
    sum('OffPeak_production).as("OffPeak_production"),
    sum('Peak_production).as("Peak_production"),
    first('ReadingDate).as("ReadingDate"),
    first('marketsegment).as("marketsegment"),
    collect_set('Source).as("Sources"),
    collect_set('Tag).as("Tags"),
    max('Last_modified).as("Last_modified")
)
.withColumn(
    "Tag", when(array_contains('Tags, “Interpolated”),
lit(Tag.Interpolated.toString)).otherwise(lit(“Measured”)))
.withColumn("Source",
when(size('Sources) > 1,
lit(Source.Multiple.toString)).otherwise(mkString('Sources)))
.orderBy('ConnectionId, 'ReadingDate)
.drop("window", "sources", "tags")
```

In this specific example, we are processing connection usage data by aggregating it daily. In the `usageDF` we have 15 minutes interval usage data, now we want to show to the user the same data but with a different aggregation interval (1 day). So we group the whole data by connection id and window the reading date by 1 day (A window function calculates a return value for every input row of a table based on a group of rows [Introducing Window Functions in Spark SQL - The Databricks Blog](https://databricks.com/blog/2015/07/15/introducing-window-functions-in-spark-sql.html).

Once the data is grouped we can aggregate it, using the `agg` method which allows us to call the aggregation functions over the dataframe (for example: `sum`, `first`,`max` or `collect_set`). Successively we transform the dataframe to suit our visualization needs, the methods used are self-explanatory and the documentation is very clear. [Getting Started - Spark 3.0.1 Documentation](https://spark.apache.org/docs/latest/sql-getting-started.html)

### Load

The final phase is the one which `save`, `put`, `show` the transformed data into the target data source.

```java scala
dataFrame
.select(columns.head, columns.tail: _*)
.write
.cassandraFormat(tableName, keySpace)
.mode(saveMode)
.save()
```

In this specific case, we will save our dataframe into a Cassandra database. In Spark, methods used to achieve the load phase are called _actions_. It is very important to distinguish Spark actions from the rest because actions are the only ones that trigger Spark to actually perform the whole transformation chain you have defined previously.

If our transformation phase, as we described above, wasn’t followed by an action (for example `save`) nothing would have happened, the software would have simply terminated without doing anything.

## One concept to rule them all

```java scala
val rdd1 = sc.parallelize(1 to 10)
val rdd2 = sc.parallelize(11 to 20)
val rdd2Count = rdd1.map(
x => rdd2.values.count() * x //This will NEVER work!!!!
)
```

_One does not simply use RDD inside another RDD_. (Same goes for Dataframes or Datasets).

This is a very simple concept that leads very often to lots of questions because many people just want to use Spark as a normal scala library. But this is not possible due to the inner distributed nature of Spark and its data structures. We have said that an RDD is a resilient distributed dataset, let’s focus on the word _distributed_, it means that the data inside it is spread across the nodes of the cluster. Every node has its own JVM and it is called _Executor_, except for the master node where your program starts which is called _Driver_:

![Spark cluster overview](/images/spark-cluster-overview.png "Spark cluster overview")

Your code starts from the Driver and a copy is distributed to all executors, this also means that each executor needs to have the same working environment of the Driver, for Scala it is not a problem since it just needs a JVM to run. (but we will see that if you use _pySpark_ you need to take extra care when you distribute your application.) Every Spark data structure you have defined in your code will also be distributed across the executors and every time you perform a transformation it will be performed to each chunk of data in each executor.

Now let’s go back to our example, a `map` is a transformation on `rdd1` this means that block inside will be executed at the executor level, if we need `rdd2` to perform this block Spark should somehow serialize the whole `rdd2` and send it to each executor. You can understand now that _it is really not possible to serialize the whole RDD since it is by its nature already a distributed data structure_. So what can you do to actually perform such computation we showed in the example? The solution is “simple”: _prepare your data in such a way that it will be contained in one single RDD_. To do so you can take advantage of all the transformation functions Spark has to offer such `map` `join` `union` `reduce` etc.

## Next step…

We have explained all the main concepts of Spark and we have shown some real snippets of our codebase. In the next article, I would like to show you a real-life problem we have solved in our company using [_pySpark_](https://spark.apache.org/docs/latest/api/python/index.html). I will show you how to customize Spark infrastructure to correctly parallelize the ETL algorithm you have built.



Two months ago, I started my journey at Vandebron. One of the projects I first dove into was their efforts to build a [component library](https://windmolen.netlify.app/). Something I was already familiar with from previous companies I worked at. 

On the internet, you can find many articles that describe why a reusable component library is a good investment for your development team(s). Although there's much to say about the advantages of component libraries, most articles don't state the (obvious) disadvantages such projects can have. In this post, I'll point out some of our learnings and why you might not need such a reusable component library.

## About component libraries

Often you find yourself repeating the same lines of code to make, for example, a button or the layout of a page look nice, especially when you're working on multiple projects. Or as a designer, you get frustrated every time the styling for a part of the application is off when a new page or project is created. Many companies have already found multiple solutions to preventing themselves from repeating styling, which is the main reason for design inconsistencies. And therefore component libraries were created.

A component library is a collection of all the styled parts (or components) of a website or multiple websites that make it easier for developers to reuse these parts. Also, designers will know for sure that all components in the component library adhere to their designs, and therefore all projects that use these components will conform. Often these libraries consist of different layers of components, for example, offering atoms, molecules, and organisms when an [Atomic Design](https://bradfrost.com/blog/post/atomic-web-design/) pattern is applied. Following this pattern, developers can use the parts to style their templates and pages consistently.

Component libraries are becoming more and more popular with the rise of JavaScript libraries and frameworks like React and Vue. These technologies are very suitable for quickly building interactive components that you can use in your application, and can easily be exposed as a library on NPM or Github Packages. At Vandebron, we're building all our web and mobile applications with React and React Native and are using [Storybook](https://storybook.js.org/) to develop our components in a shared library between the engineering and design teams. This can potentially create a lot of advantages for both the developers and designers, as you can read below.

## Why you *might* need a component library

Before deciding to create a component library for your team or company, you probably want to hear about the advantages such a project can lead to. The main advantages of component libraries are briefly mentioned in the first section above and are often defined as:

- **Reducing code duplication**: With a component library, you can create components that can be shared across multiple websites or applications. This way you no longer have to duplicate styling in different projects. This can seriously decrease the amount of code duplication that you have in your projects, also reducing the number of bugs or design inconsistencies.

- **Preventing design inconsistencies**: By adding all your components and styled parts to the component library you're certain that these will look the same on all the places they're used. Not only will all the components look the same on every page, when designers make a change to one of these components they can be easily updated on all the places they're used.

- **Easier collaborating**: Component libraries make it easier for developers and designers to collaborate on applications and designs, with the component library as the common "playground". By using a tool, like Storybook, you can also make this playground visible to non-technical people and show what components are already available to use for new features.

But these advantages come at a certain price, as I'll explain in the next section.

## Disadvantages of component libraries

Besides the obvious advantages of a component library, it can also have serious disadvantages that are listed below. Whether or not these disadvantages apply to you depends on numerous things that are discussed later on in this article.

- **Increasing complexity**: With all attempts to make code more generic,  an increased level of complexity also comes to play. Reusable components should be easy to extend or customize, which requires you to think about the different use cases beforehand or force you to add many different variations to a component. With every new project that starts to use the component library, you get the risk of increasing the complexity of the library even more.

- **Time-consuming**: Every time you want to add a component to your project, you need to create that component in the component library first and import it locally in the project to test it. Therefore you need to be working in multiple projects at the same time, which requires you to set up a more time-consuming workflow. Also, when you want to use this new component from the library, you have to publish a new version of the library to make the component available.

- **Conflicting dependencies**: When you're using different versions of dependencies across your projects and the component library, you're forced to sync those with each other. Imagine having, for example, an older version of React running in one of your projects that doesn't use a recent React API that you want to use in your component library. In this scenario, you either have to update that project or are unable to keep your component library on par with the latest release of your dependency on React. Both solutions have pros and cons, and would rather be avoided.

As mentioned before, there are reasons why these disadvantages might apply to you that are the team size, the number of teams and projects at the company, development or release lifecycles, and how your source code is organized. It clearly doesn't make sense to invest in a component library if you have just a small amount of people work on just one project, or a sole team is working on all the different projects making it easier to manage code duplication or design inconsistencies.

## Considerations before starting

There are two main alternatives that you need to take into consideration before building a reusable component library, which is (obviously) using or extending an existing component library or sourcing your code in a monorepo. 

- **Existing component libraries:** Using an existing component library is an efficient way to create consistently (web) pages and reduce the amount of complexity of your own project, while also taking advantage of best practices of large open-source projects. Popular examples of component libraries are [Ant Design For React](https://ant.design/docs/react/introduce) or [various implementations](https://material.io/develop) for Google's Material Design. These libraries allow you to move quickly without having all the overhead of creating complex components but limit you to the design guidelines of these component libraries.

- **Monorepo:** If you don't want to take advantage of existing libraries or are very keen to apply your own styling to components across multiple applications without having to copy-paste the code, you can host the source code of applications in a monorepo. With the monorepo approach, you can create a shared folder that includes all the components used by your applications. This makes it possible to apply changes with a simple pull request and import these components from every project in that repository.

Besides these two alternatives, you also need to have proper design guidelines set by your designer(s). When the design guidelines are flexible and fluctuating, you could be structuring components incorrectly with the risk of doing a lot of work that will be omitted once the project evolves.

## To summarize

Component libraries are a great way to reduce the amount of code duplication in your applications, prevent design inconsistencies, and increase collaborations between developers, designers, and different teams. But this comes with increased complexity, slower development cycles, and possible code conflicts between projects. Therefore you should consider if using an existing component library or having a monorepo for your source code is a workable solution. At Vandebron we decided to build our own component library (called [windmolen](https://windmolen.netlify.app/)) and if you'd decide the same, then be sure that your design guidelines are properly structured and mature enough.


Vandebron Engineering & Data

Latest posts

The power of regular hackathons

Migrating from DCOS to Kubernetes, dealing with the l4lb loadbalancer

Cypress.io Component Design Technique for React Applications

How to Spin Up A Kubernetes Cluster On Your Macbook

Optimizing, Converting And Exporting SVG Icons In React

Fueling the Energy Transition With Spark - Part 1

Building native images and compiling with GraalVM and sbt

When (Not) To Build A Reusable Component Library

Links

Vandebron on Github

Windmolen

Work with us