
Cypress is a game-changer in the automation testing world, the way that Cypress was built and its architecture allows us as testers to cover more scenarios.

Cypress is not Selenium; in fact, it is different. And the way to build and design a framework should be different as well.

The most famous design technique in Selenium is the Page Object Model, and many testers use the same design technique with Cypress. Even that Cypress on their official website [recommended](https://www.cypress.io/blog/2019/01/03/stop-using-page-objects-and-start-using-app-actions/) us not to go with that approach.

## Page Object Model

The main benefit of using the page object model Is to make the automation framework maintenance-friendly. We can define a specific page's selectors in a separate file and then use these selectors in our test cases.

```js
class SignInPage {
  visit() {
    cy.visit("/signin");
  }
  getEmailError() {
    return cy.get(`[data-testid=SignInEmailError]`);
  }
  getPasswordError() {
    return cy.get(`[data-testid=SignInPasswordError]`);
  }
  fillEmail(value) {
    const field = cy.get(`[data-testid=SignInEmailField]`);
    field.clear();
    field.type(value);
    return this;
  }
  fillPassword(value) {
    const field = cy.get(`[data-testid=SignInPasswordField]`);
    field.clear();
    field.type(value);
    return this;
  }
  submit() {
    const button = cy.get(`[data-testid=SignInSubmitButton]`);
    button.click();
  }
}
export default SignInPage;
```

The main two downsides using the typical page object model with cypress are:

- Page objects introduce an additional state into the tests, separate from the application’s internal state. This makes understanding the tests and failures harder.
- Page objects make tests slow because they force the tests to always go through the application user interface.

## Component-Based Architecture

On the other hand, a React application is component-based, where a specific page will be built from a collection of components. And components in React can be used on different pages too. So if we want to use the Page Object Model, we may define the same locator twice on different pages.

So having these two facts, At Vandebron, we came up with a new way to design our Cypress Automation framework by creating a separate JavaScript file for every component in our application, inside a folder called `components` within our Cypress project as below:

```js
// Locators
export const getEmailError = () => cy.get(`[data-testid=SignInEmailError]`);
export const getPasswordError = () =>
  cy.get(`[data-testid=SignInPasswordError]`);
export const emailField = () => cy.get(`[data-testid=SignInEmailField]`);
export const passwordField = () => cy.get(`[data-testid=SignInPasswordField]`);
export const submitButton = () => cy.get(`[data-testid=SignInSubmitButton]`);

// Actions
export const visit = () => cy.visit("/signin");
export const performLogin = (email, password) => {
  emailField().clear().type(email);
  passwordField().clear().type(password);
  submitButton().click();
};
```

Having it built this way, we eliminated all the previous problems mentioned earlier; we are not adding any classes, and we are defining objects within our test cases. And the most important part is that we are following the way that Cypress recommends it.

And after defining the component locators and actions, we can import them inside our test case and use them as below:

```js
import LoginComponent from "../components/loginComponent";
import Menu from "../components/Menu";

describe("Test Login Page", () => {
  it("should show an error message if the password in wrong", () => {
    LoginComponent.visit();
    LoginComponent.performLogin("email@gmail.com", "wrongPassword");
    LoginComponent.getPasswordError().should("be.visible");
  });
  it("should show the logout button if the user logged in succesfully", () => {
    LoginComponent.visit();
    LoginComponent.performLogin("email@gmail.com", "correctPassword");
    Menu.LogoutButton().should("be.visible");
  });
});
```

And as you can see, our test cases are readable for anyone! And if any locator changes in any of the components, we can easily fix it in one location and from the same file. And lastly, if a component will be used in different places, we can use the same code.

In the next article, I will talk about how we use Cypress in our manual testing during the sprint and how it saves us tons of time and effort.



In Vandebron we have been using container clusters to host our services since the foundation of our Big Data team. 
Recently our cluster of choice has declared End-Of-Life development stage, so we decided to take a step forward and get a ticket for the Kubernetes boat.

A change in the OS that is used to run your services and applications can look quite challenging and not everyone is on the same experience level. To make everyone comfortable it is a good choice to give everyone the possibility to play with the new tools and learn what can be done and how: **you need a sandbox.**

Our developers are provided with a Macbook and at the moment of writing there some options you can go for when deciding how to set up your playground:

- **Docker CE Kubernetes**: This is the easiest solution since there is a handy button to run your containers into a Kubernetes environment.

- **Vagrant and Virtualbox**: This solution is the one that can give you more control and you can easily create a cluster the size you want, but you need to be handy with VMs, the hypervisor of choice, and Vagrant. It's the old school way to do it but, while it's a chunk your platform engineers can bite, it can be a steep and frustrating process for people that are not used to handle VMs.

- **Multipass + some bash magic glue**: Since Canonical created this tool for macOS, creating an Ubuntu VM became a breeze and you can have a single, easily manageable VM with its networking up and running in less than a minute, without having to handle disks, distros, and stuff. On top of it, the command line interface is straight forward and it has just the basic commands we will need, so wrapping the entire process into a bash script is a piece of cake.

I have found this super cool in-depth [article](https://jyeee.medium.com/kubernetes-on-your-macos-laptop-with-multipass-k3s-and-rancher-2-4-6e9cbf013f58) from Jason Yee (kudos to you bruh) that guided me through the installation of my first single node Kubernetes cluster.

The process is not that long but it involves a lot of copy/pasting and, once learned the basics, I didn't want to go under the same process more times, plus it could be interesting for me as a Platform Engineer, but it may be boring and pointless for developers who just want to have a sandbox replica of what they are working on in the remote environment.
My automator (aka do-it-once-never-do-it-again) spirit kicked in and I decided to wrap every step in a small command-line tool with only 3 options:
- **install**
- **cleanup**
- **help**


### What is happening under the hood

What the script does is substantially automating all the steps needed to:
1. Create a new VM using Multipass (tool released by Canonical)
2. Fetch the VM IP address and adding it to your local `/etc/hosts` file
3. Install k3s (a lightweight distribution of Kubernetes) on top of the VM
4. Install the Kubernetes command-line tools on your laptop
5. Install Helm (the Kubernetes package manager) on your laptop
6. Install cert-manager (certificate manager) package on top of your k3s cluster
7. Install Rancher (a Kubernetes control plane) package on top of your k3s cluster

If you are looking for a more in-depth breakdown of the single steps you can download and inspect [the script](https://gist.githubusercontent.com/nikotrone/50b1a5f8d137411879eb2467e689bfbe/raw/090b4b4323d96ac28d96bbb346e2e657073722e6/bronernetes) (one of the many advantages of [OpenSource](https://en.wikipedia.org/wiki/Open_source) projects) or checkout and read the original [article](https://jyeee.medium.com/kubernetes-on-your-macos-laptop-with-multipass-k3s-and-rancher-2-4-6e9cbf013f58): it explains line by line what the specific commands are doing.

#### 1. Multipass VM
[Multipass](https://multipass.run/) is a tool from Canonical (the company developing and maintaining the Ubuntu Linux distribution) that leverages Hyperkit (macOS feature to handle virtualization) to create and handle a Virtual Machine directly on your Mac.

#### 2. Edit /etc/hosts
Once we have our VM up and running we need to make it available with an easy url that is also gonna be used to generate the SSL certificate, in our case we picked up `rancher.localdev`.
It is important to have a name setup in the beginning since this one will need to match with the certificate so we can use it programmatically.

#### 3. Install K3S
This step is pretty straightforward: just fetch a script that is publicly available on the [k3s official website](https://get.k3s.io) and feed it to your bash.
K3s is a lightweight version of Kubernetes with all the needed dependencies and executable packaged in a convenient installation script. Because of its light nature, it is often used in embedded devices that have a limited amount of resources to offer.

#### 4 & 5. Kubernetes and Helm cli
**Kubernetes cli** (`kubectl`) is used to talk and interact with your Kubernetes cluster. It can be used to manage multiple clusters according to the content of your KUBECONFIG environment variable. 
The variable itself contains just a path to where your cluster configuration is stored, so you can switch from a cluster to another by simply pointing to another file that contains the configuration of another cluster.

**Helm** instead is the "package manager" of Kubernetes: you can use it to add repositories to specific `charts` which are the blueprint that contains a way to install a specific tool on your cluster.
Both of these tools have to be installed and run from your local laptop, either in the case you are managing a local VM or in the case you are interacting with a remote cluster.

#### 6 & 7. cert-manager and Rancher

**Rancher** is the control plane for our cluster: it provides a GUI and an overview of our single node cluster. It offers other goodies like management of multiple clusters, deployed on different locations like AWS Azure and GCP or even on your own hardware, plus certificate deployment and some other handy functionalities.

**cert-manager** is installed via Helm chart and it is the tool used by Rancher to generate and deploy a certificate across the entire cluster.

### How to use it

All the steps will involve the use of a Terminal window
#### Installation
The first thing you need to do is download [this script](https://gist.github.com/nikotrone/50b1a5f8d137411879eb2467e689bfbe) and save it in a folder on your Mac (let's assume `~/bronernetes`) by executing
```bash
    mkdir ~/bronernetes
    cd ~/bronernetes
    curl https://gist.githubusercontent.com/nikotrone/50b1a5f8d137411879eb2467e689bfbe/raw/090b4b4323d96ac28d96bbb346e2e657073722e6/bronernetes > bronernetes
    export PATH=$PATH:$(pwd)
```

Now we have the toolset and you can confirm it works by simply running `bronernetes help`.

#### Spin up Kubernetes
The next step is to run the installation process with the command `bronernetes install`

#### Clean up
When you are done or you just want to hard reset your environment you can just type `bronernetes cleanup` and it will take care of cleaning up the VM you just used, leaving you with a pristine machine, as nothing ever happened :)

### Conclusion

Having a sandbox is very useful to play around with the concepts of a new setup or service and it packs up a huge amount of positive sides. No matter what is the language or the nature of the system you are trying to replicate, it can be challenging and involve a long list of instructions or manual operations and, sometimes, even dedicated hardware. Although with some bash glue, it is possible to automate most of those processes and the investment cost can be enormously beneficial for yourself (less work the next time you do it) and for the other people working with you (they can use the tool, comment and suggest improvements). Most of all, in the case of infrastructure, it helps raise the knowledge of "what's going on here" and documents for the ones interested in taking a trip down the rabbit hole.


At Vandebron we're maintaining a component library called [Windmolen](https://windmolen.netlify.app/) (Dutch for "wind turbine"). And if you've ever built a component library, you probably dealt with optimizing and converting icons before. With SVGO and SVGR you can do this at scale, without compromising the quality or size of your icons.

## The problem

The web is full of icons, and often these icons are rendered from SVG files to ensure you can increase (or decrease) the size of the icons depending on the use case. Designers often create these icons from design tools like Adobe Photoshop or Sketch. Although these icons might look pretty, exporting a SVG out of these tools is often difficult as [this article](https://medium.com/sketch-app-sources/the-best-way-to-export-an-svg-from-sketch-dd8c66bb6ef2) explains. Also, added lot of code in the form of metadata is added to the SVG file. Let's have a look at what a typical SVG file exported out of Sketch looks like:

```svg

<?xml version="1.0" encoding="UTF-8"?>
<svg width="14px" height="14px" viewBox="0 0 14 14" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
    
    <title>last</title>
    <desc>Created with Sketch.</desc>
    <defs></defs>
    <g id="Page-1" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
        <g id="last" transform="translate(2.000000, 0.000000)" fill-rule="nonzero" fill="#666666">
            <polygon id="Fill-2" points="6.6902923 9.6812703 9.3700469 7.0005052 6.6902923 4.3187297 2.37257308 0 0 2.37358354 4.3177192 6.6902923 4.6279322 7.0005052 4.3177192 7.3107182 0 11.6274269 2.37257308 14"></polygon>
        </g>
    </g>
</svg>
```

The SVG file above holds a lot of information about Sketch, such as the `title` of the icon and a `desc`ription. Next to that, there's a lot of elements that could be combined into one element to reduce the file size.

## Optimizing SVGs

What's cool about SVG files is that you can optimize and minify them, without affecting what the SVG looks like. This is something you can try out yourself using the website [SVGOMG](https://jakearchibald.github.io/svgomg/), which is powered by the library SVGO that you'll learn more about later.


You can optimize the SVG file above by following these steps:

1. Go to [https://jakearchibald.github.io/svgomg/](https://jakearchibald.github.io/svgomg/)
2. Click on `Paste markup` an paste the SVG code that you exported from Sketch (a.k.a. the SVG file above)
3. You will see the icon rendered, now you have to either click at the `Copy as a text` or `Download` button to get the optimized SVG file

With these simple steps you've optimized the SVG from over 450 bytes, which is already small, to 173 bytes (a decrease of over 62%!). If you'd open this file in the editor of your choice, you can see a lot of the useless (meta)data from the original file has been deleted. Also, the different elements of the SVG are combined in a single `path` that renders the icon:

```svg

<svg width="14" height="14" xmlns="http://www.w3.org/2000/svg">
  <path d="M8.69 9.681l2.68-2.68-2.68-2.682L4.373 0 2 2.374 6.318 6.69l.31.31-.31.31L2 11.628 4.373 14z" fill-rule="nonzero" fill="#666"/>
</svg>
```

This SVG can be even further optimized by checking the "Prefer viewbox to width/height" in SVGOMG, but let's save that for later when we use SVGO instead.

## Using SVGO

By using SVGOMG you've already experienced what power [SVGO](https://github.com/svg/svgo) has, as SVGOMG is described by its creators as *" SVGO's Missing GUI, aiming to expose the majority if not all the configuration options of SVGO"*. Instead of using the GUI, you can also use SVGO directly from the command line as a CLI-tool or as a Node.js module. For the sake of this article, we'll be using it solely as CLI.

SVGO can be installed globally on your machine, or locally in your project, from npm by running:

```bash
npm i -g svgo

# Yarn equivalent
yarn add -G svgo
```

After doing this you can run `svgo` from the command line and optimize any SVG file instantly. But, you don't want to do this manually on your machine anytime you're adding a new icon to a project (or component library). Therefore, you can also add SVGO to a project locally and add a script to the `package.json` file to optimize all SVGs in a certain directory.

```json
// package.json
{
 // ...
 "scripts": {
     // ...
    "optimize-svg": "svgo --config=.svgo.yml -f ./src/assets/icons"
 }
}
```

The `optimize-svg` script will run SVGO in the directory `src/assets/icons` and optimize all the SVG files based on the settings in `.svgo.yml`. This file is where you can configure the rules for SVGO, as the previously mentioned "Prefer viewbox to width/height":

```yaml
# .svgo.yml
plugins:
  - removeViewBox: false
  - removeDimensions: true # this deletes width/height and adds it to the viewBox
  - removeDoctype: true
  - removeComments: true
  - removeMetadata: true
  - removeEditorsNSData: true
  - cleanupIDs: true
  - removeRasterImages: true
  - removeUselessDefs: true
  - removeUnknownsAndDefaults: true
  - removeUselessStrokeAndFill: true
  - removeHiddenElems: true
  - removeEmptyText: true
  - removeEmptyAttrs: true
  - removeEmptyContainers: true
  - removeUnusedNS: true
  - removeDesc: true
  - prefixIds: false
  - prefixClassNames: false
```
   
From the rules above you'll get an idea about all the redundant and useless lines of code that might be present in your SVG files. But luckily, they will all get removed when you run the command `npm run optimize-svg`.

## Converting SVGs with SVGR

You've now learned how to optimize your SVG files, and are probably wondering how to use these files in a React application. To render an SVG in React, you need to either configure Webpack in a way that it knows how to deal with SVG files or use a library called SVGR. By default, any application created with `create-react-app` can render SVG files as a component, using the following `import` statement:

```jsx
// MyComponent.jsx
import React from 'react';
import { ReactComponent as MySVG } from './something.svg';

const MyComponent = () => {
  return (
    <div>
      <MySVG />
    </div>
  );
}
export default MyComponent;
```

More information about how this is done can be found in [this article](https://blog.logrocket.com/how-to-use-svgs-in-react/), but let me show you how to solve that with SVGR.

With [SVGR](https://react-svgr.com/) you can convert SVG files into React Components, either by adding it to Webpack or by using the SVGR CLI or Node.js module. In the same way, as we optimized the SVGs from the command line with SVGO, we can also convert these icons from the command line with SVGR:

```json
// package.json
{
 // ...
 "scripts": {
     // ...
    "optimize-svg": "svgo --config=.svgo.yml -f ./src/assets/icons",
    "convert-svg": "svgr -d ./src/components/Icon ./src/assets/icons"
 }
}
```

Whenever you run the command `npm run convert-svg` a JSX file will be created for every SVG file that's present in the directory `src/assets/icons`. These JSX files can be found in the directory `src/components/Icons`, together with an `index.js` file that exports all these components from this directory.

An example of such a converted SVG file is:


```jsx
// MySVG.jsx
import * as React from 'react';

const MySVG = (props) => (
  <svg viewBox="0 0 14 14" xmlns="http://www.w3.org/2000/svg" {...props}>
  <path d="M8.69 9.681l2.68-2.68-2.68-2.682L4.373 0 2 2.374 6.318 6.69l.31.31-.31.31L2 11.628 4.373 14z" fill-rule="nonzero" fill="#666"/>
  </svg>
);

export default MySVG;
```

And, as we now have a directory filled with converted SVGs these can be imported into any React component like this:

```jsx
// MyComponent.jsx
import React from 'react';
import MySVG from './MySVG.jsx';

const MyComponent = () => {
  return (
    <div>
      <MySVG />
    </div>
  );
}
export default MyComponent;
```

Often SVGR is used alongside SVGO, so you can even automatically optimize all SVGS that will be converted by SVGR. This is done by adding the flag `--no-svgo true` and point it towards your SVGO configuration file:

```json
// package.json
{
 // ...
 "scripts": {
     // ...
    "convert-svg": "svgr -d ./src/components/Icon ./src/assets/icons --no-svgo true --svgo-config .svgo.yml"
 }
}
```

By running the `convert-svg` script you both optimize and convert all the SVG files in `src/assets/icons` to React components based on optimized SVGs.

## Reading further

The examples in this post are the tip of the metaphorical iceberg on what problems SVGO and SVGR can solve. There are many other features you can enable, such as using them as Node.js modules or enabling TypeScript support. To read further make sure to have a look at the SVGR [playground](https://react-svgr.com/playground/) or [documentation](https://react-svgr.com/docs/getting-started/).


Here at Vandebron, we have several projects which need to compute large amounts of data. To achieve acceptable results, we had to choose a computing tool that should have helped us to build such algorithms.

As you may have read in other articles our main backend language is Scala so the natural choice to build distributed parallel algorithms was indeed Spark.

## What is Spark

We will briefly introduce Spark in the next few lines and then we will dive deep into some of its key concepts.

Spark is an ETL distributed tool. ETL are three phases that describe a general procedure for moving data from a source to a destination.

![ETL Diagram](/images/etlprocess.png "ETL")

- **_Extract_** is the act of retrieving data from a data source which could be a database or a file system.
- **_Transform_** is the core part of an algorithm. As you may know, functional programming is all about transformation. Whenever you write a block of code in Scala you go from an initial data structure to a resulting data structure, the same goes with Spark but the data structures you use are specific Spark structures we will describe later.
- **_Load_** is the final part. Here you need to save (load) the resulting data structure from the transformation phase to a data source. This can either be the same as the extract phase or a different one.
- **_Distributed_**: Spark is meant to be run in a cluster of nodes. Each node runs its own JVM and every Spark data structure can/should be distributed among all the nodes of the cluster (using serialization) to parallelize the computation.

### Spark data structure: RDD, DataFrame, and Dataset

The core of Spark is its _distributed resilient dataset (RDD)_.

![Spark API history](/images/sparkapihistory.png "Spark API history")

An **_RDD_** is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel. _Extracting_ data from a source creates an RDD. Operating on the RDD allows us to _transform_ the data. Writing the RDD _loads_ the data into the end target like a database for example). They are made to be distributed over the cluster to parallelize the computation.

A **_DataFrame_** is an abstraction on top of an RDD. It is the first attempt of Spark (2013) to organize the data inside and RDD with an SQL-like structure. With dataframe, you can actually make a transformation in an SQL fashion. Every element in a dataframe is a Row and you can actually transform a dataframe to another by adding or removing columns.

A **_DataSet_** finally is a further abstraction on top of a dataframe to organize data in an OO fashion (2015). Every element in a dataset is a case class and you can operate transformation in a scala fashion from a case class to another.

## Spark in action

Let’s see now some code samples from our codebase to illustrate in more detail each of the ETL phases.

### Extract

The extraction phase is the first step in which you gather the data from a datasource.

```scala
val allConnections = sparkSession
.read
.jdbc(connectionString, tableName, props)

val selectedConnections = allConnections
.select(ColumnNames.head, ColumnNames.tail: _*)

val p4Connections = selectedConnections
.filter(allConnections("HasP4Day activated").equalTo(1))
.filter(allConnections("HasP4INT activated").equalTo(1))
.as[Connection]

p4Connections.show()
```

For most people the extraction phase is just the first line (the invocation to the read method), they are not wrong because extracting means reading data from a datasource (in this case an SQL server database). I decided to include in this phase also some filtering and projection operations because I think these are not really part of the algorithm, this is still the preparation phase before you actually process the data. We can ultimately say that _preparing the data_ is something in between extraction and transformation therefore it is up to you to decide which phase it belongs to.

### Transform

Transformation phase is the core of the algorithm. Here you actually process your data to reach your final result.

```java scala
usageDF
.groupBy('ConnectionId, window('ReadingDate, "1 day"))
.agg(
    sum('Consumption).as("Consumption"),
    sum('OffPeak_consumption).as("OffPeak_consumption"),
    sum('Peak_consumption).as("Peak_consumption"),
    sum('Production).as("Production"),
    sum('OffPeak_production).as("OffPeak_production"),
    sum('Peak_production).as("Peak_production"),
    first('ReadingDate).as("ReadingDate"),
    first('marketsegment).as("marketsegment"),
    collect_set('Source).as("Sources"),
    collect_set('Tag).as("Tags"),
    max('Last_modified).as("Last_modified")
)
.withColumn(
    "Tag", when(array_contains('Tags, “Interpolated”),
lit(Tag.Interpolated.toString)).otherwise(lit(“Measured”)))
.withColumn("Source",
when(size('Sources) > 1,
lit(Source.Multiple.toString)).otherwise(mkString('Sources)))
.orderBy('ConnectionId, 'ReadingDate)
.drop("window", "sources", "tags")
```

In this specific example, we are processing connection usage data by aggregating it daily. In the `usageDF` we have 15 minutes interval usage data, now we want to show to the user the same data but with a different aggregation interval (1 day). So we group the whole data by connection id and window the reading date by 1 day (A window function calculates a return value for every input row of a table based on a group of rows [Introducing Window Functions in Spark SQL - The Databricks Blog](https://databricks.com/blog/2015/07/15/introducing-window-functions-in-spark-sql.html).

Once the data is grouped we can aggregate it, using the `agg` method which allows us to call the aggregation functions over the dataframe (for example: `sum`, `first`,`max` or `collect_set`). Successively we transform the dataframe to suit our visualization needs, the methods used are self-explanatory and the documentation is very clear. [Getting Started - Spark 3.0.1 Documentation](https://spark.apache.org/docs/latest/sql-getting-started.html)

### Load

The final phase is the one which `save`, `put`, `show` the transformed data into the target data source.

```java scala
dataFrame
.select(columns.head, columns.tail: _*)
.write
.cassandraFormat(tableName, keySpace)
.mode(saveMode)
.save()
```

In this specific case, we will save our dataframe into a Cassandra database. In Spark, methods used to achieve the load phase are called _actions_. It is very important to distinguish Spark actions from the rest because actions are the only ones that trigger Spark to actually perform the whole transformation chain you have defined previously.

If our transformation phase, as we described above, wasn’t followed by an action (for example `save`) nothing would have happened, the software would have simply terminated without doing anything.

## One concept to rule them all

```java scala
val rdd1 = sc.parallelize(1 to 10)
val rdd2 = sc.parallelize(11 to 20)
val rdd2Count = rdd1.map(
x => rdd2.values.count() * x //This will NEVER work!!!!
)
```

_One does not simply use RDD inside another RDD_. (Same goes for Dataframes or Datasets).

This is a very simple concept that leads very often to lots of questions because many people just want to use Spark as a normal scala library. But this is not possible due to the inner distributed nature of Spark and its data structures. We have said that an RDD is a resilient distributed dataset, let’s focus on the word _distributed_, it means that the data inside it is spread across the nodes of the cluster. Every node has its own JVM and it is called _Executor_, except for the master node where your program starts which is called _Driver_:

![Spark cluster overview](/images/spark-cluster-overview.png "Spark cluster overview")

Your code starts from the Driver and a copy is distributed to all executors, this also means that each executor needs to have the same working environment of the Driver, for Scala it is not a problem since it just needs a JVM to run. (but we will see that if you use _pySpark_ you need to take extra care when you distribute your application.) Every Spark data structure you have defined in your code will also be distributed across the executors and every time you perform a transformation it will be performed to each chunk of data in each executor.

Now let’s go back to our example, a `map` is a transformation on `rdd1` this means that block inside will be executed at the executor level, if we need `rdd2` to perform this block Spark should somehow serialize the whole `rdd2` and send it to each executor. You can understand now that _it is really not possible to serialize the whole RDD since it is by its nature already a distributed data structure_. So what can you do to actually perform such computation we showed in the example? The solution is “simple”: _prepare your data in such a way that it will be contained in one single RDD_. To do so you can take advantage of all the transformation functions Spark has to offer such `map` `join` `union` `reduce` etc.

## Next step…

We have explained all the main concepts of Spark and we have shown some real snippets of our codebase. In the next article, I would like to show you a real-life problem we have solved in our company using [_pySpark_](https://spark.apache.org/docs/latest/api/python/index.html). I will show you how to customize Spark infrastructure to correctly parallelize the ETL algorithm you have built.



Two months ago, I started my journey at Vandebron. One of the projects I first dove into was their efforts to build a [component library](https://windmolen.netlify.app/). Something I was already familiar with from previous companies I worked at. 

On the internet, you can find many articles that describe why a reusable component library is a good investment for your development team(s). Although there's much to say about the advantages of component libraries, most articles don't state the (obvious) disadvantages such projects can have. In this post, I'll point out some of our learnings and why you might not need such a reusable component library.

## About component libraries

Often you find yourself repeating the same lines of code to make, for example, a button or the layout of a page look nice, especially when you're working on multiple projects. Or as a designer, you get frustrated every time the styling for a part of the application is off when a new page or project is created. Many companies have already found multiple solutions to preventing themselves from repeating styling, which is the main reason for design inconsistencies. And therefore component libraries were created.

A component library is a collection of all the styled parts (or components) of a website or multiple websites that make it easier for developers to reuse these parts. Also, designers will know for sure that all components in the component library adhere to their designs, and therefore all projects that use these components will conform. Often these libraries consist of different layers of components, for example, offering atoms, molecules, and organisms when an [Atomic Design](https://bradfrost.com/blog/post/atomic-web-design/) pattern is applied. Following this pattern, developers can use the parts to style their templates and pages consistently.

Component libraries are becoming more and more popular with the rise of JavaScript libraries and frameworks like React and Vue. These technologies are very suitable for quickly building interactive components that you can use in your application, and can easily be exposed as a library on NPM or Github Packages. At Vandebron, we're building all our web and mobile applications with React and React Native and are using [Storybook](https://storybook.js.org/) to develop our components in a shared library between the engineering and design teams. This can potentially create a lot of advantages for both the developers and designers, as you can read below.

## Why you *might* need a component library

Before deciding to create a component library for your team or company, you probably want to hear about the advantages such a project can lead to. The main advantages of component libraries are briefly mentioned in the first section above and are often defined as:

- **Reducing code duplication**: With a component library, you can create components that can be shared across multiple websites or applications. This way you no longer have to duplicate styling in different projects. This can seriously decrease the amount of code duplication that you have in your projects, also reducing the number of bugs or design inconsistencies.

- **Preventing design inconsistencies**: By adding all your components and styled parts to the component library you're certain that these will look the same on all the places they're used. Not only will all the components look the same on every page, when designers make a change to one of these components they can be easily updated on all the places they're used.

- **Easier collaborating**: Component libraries make it easier for developers and designers to collaborate on applications and designs, with the component library as the common "playground". By using a tool, like Storybook, you can also make this playground visible to non-technical people and show what components are already available to use for new features.

But these advantages come at a certain price, as I'll explain in the next section.

## Disadvantages of component libraries

Besides the obvious advantages of a component library, it can also have serious disadvantages that are listed below. Whether or not these disadvantages apply to you depends on numerous things that are discussed later on in this article.

- **Increasing complexity**: With all attempts to make code more generic,  an increased level of complexity also comes to play. Reusable components should be easy to extend or customize, which requires you to think about the different use cases beforehand or force you to add many different variations to a component. With every new project that starts to use the component library, you get the risk of increasing the complexity of the library even more.

- **Time-consuming**: Every time you want to add a component to your project, you need to create that component in the component library first and import it locally in the project to test it. Therefore you need to be working in multiple projects at the same time, which requires you to set up a more time-consuming workflow. Also, when you want to use this new component from the library, you have to publish a new version of the library to make the component available.

- **Conflicting dependencies**: When you're using different versions of dependencies across your projects and the component library, you're forced to sync those with each other. Imagine having, for example, an older version of React running in one of your projects that doesn't use a recent React API that you want to use in your component library. In this scenario, you either have to update that project or are unable to keep your component library on par with the latest release of your dependency on React. Both solutions have pros and cons, and would rather be avoided.

As mentioned before, there are reasons why these disadvantages might apply to you that are the team size, the number of teams and projects at the company, development or release lifecycles, and how your source code is organized. It clearly doesn't make sense to invest in a component library if you have just a small amount of people work on just one project, or a sole team is working on all the different projects making it easier to manage code duplication or design inconsistencies.

## Considerations before starting

There are two main alternatives that you need to take into consideration before building a reusable component library, which is (obviously) using or extending an existing component library or sourcing your code in a monorepo. 

- **Existing component libraries:** Using an existing component library is an efficient way to create consistently (web) pages and reduce the amount of complexity of your own project, while also taking advantage of best practices of large open-source projects. Popular examples of component libraries are [Ant Design For React](https://ant.design/docs/react/introduce) or [various implementations](https://material.io/develop) for Google's Material Design. These libraries allow you to move quickly without having all the overhead of creating complex components but limit you to the design guidelines of these component libraries.

- **Monorepo:** If you don't want to take advantage of existing libraries or are very keen to apply your own styling to components across multiple applications without having to copy-paste the code, you can host the source code of applications in a monorepo. With the monorepo approach, you can create a shared folder that includes all the components used by your applications. This makes it possible to apply changes with a simple pull request and import these components from every project in that repository.

Besides these two alternatives, you also need to have proper design guidelines set by your designer(s). When the design guidelines are flexible and fluctuating, you could be structuring components incorrectly with the risk of doing a lot of work that will be omitted once the project evolves.

## To summarize

Component libraries are a great way to reduce the amount of code duplication in your applications, prevent design inconsistencies, and increase collaborations between developers, designers, and different teams. But this comes with increased complexity, slower development cycles, and possible code conflicts between projects. Therefore you should consider if using an existing component library or having a monorepo for your source code is a workable solution. At Vandebron we decided to build our own component library (called [windmolen](https://windmolen.netlify.app/)) and if you'd decide the same, then be sure that your design guidelines are properly structured and mature enough.


Vandebron Engineering & Data

Latest posts

Cypress.io Component Design Technique for React Applications

How to Spin Up A Kubernetes Cluster On Your Macbook

Optimizing, Converting And Exporting SVG Icons In React

Fueling the Energy Transition With Spark - Part 1

Building native images and compiling with GraalVM and sbt

When (Not) To Build A Reusable Component Library

Links

Vandebron on Github

Windmolen

Work with us