
Vandebron is a Dutch green-tech energy company on a mission to accelerate the transition to 100% renewable energy, 100% of the time. As part of [our mission and strategy](https://vandebron.nl/100procentgroen), we are constantly innovating and looking for ways to optimize energy operations and reduce negative impacts when it comes to energy production.

Our new mission: [100% renewable energy, 100% of the time](https://youtu.be/_Yf8jk4gZbI)

### The importance of curtailment and flexibility services

One area where we are currently focusing our efforts is the area of curtailment and flexibility of wind turbines, solar parks, industrial batteries and electric vehicles. [Curtailment](https://vandebron.nl/blog/curtailment-slimmer-omgaan-met-goeie-energie) refers to the practice of reducing the electricity inflow to balance the electricity grid. In other words, it involves adjusting the operation of, for example, a wind turbine in order to match the demand for electricity at any given time.

[This is often necessary](https://vandebron.nl/blog/hoe-houdt-onze-technologie-het-energienet-in-balans) because the output of renewable energy sources can vary significantly due to changes in weather conditions. If the output of these sources exceeds the demand for electricity, it can lead to an excess of electricity on the grid, which can cause stability issues. On the other hand, if the output of wind turbines is too low, it can lead to a deficit of electricity on the grid, which can cause blackouts or other disruptions. To tackle this, we look at our customer’s batteries and electric vehicles offering flexibility capabilities.

### Our journey to finding reliable, secure and energy-efficient hardware and software

To optimize these curtailment and flexibility efforts, we were in need of a gateway device that we could place at the installations of the producers on our platform. To keep it close to our mission, we preferred an ARM-based CPU for its [energy efficiency](https://www.redhat.com/en/topics/linux/ARM-vs-x86) compared to an x86-based CPU. After all, we don’t want to consume all of the produced energy to power an actively cooled NUC… 😉

While gathering our hardware requirements, we concluded there was really only one competitor. Therefore, we partnered up with OnLogic! We chose their [Factor 201 device](https://www.onlogic.com/fr201/), which boasts the ARM-based Raspberry Pi CM4 module packed in a small and beautiful orange industrial chassis. The model also enables a lot of custom configurations. For example, we are able to configure multiple (wireless) networks, add extra SSD storage or optionally mount on DIN rails.

![OnLogic Factor 201](/images/flex-onlogic-factor-201.jpg "OnLogic Factor 201")

To ensure our gateway devices are secure and agile (like us, developers, 😛) we needed them to integrate well into our existing technology landscape based on Kubernetes. After struggling for some time to harden several (lightweight) operating systems and bootstrapping lightweight Kubernetes clusters our eyes fell on a new kid in town: ‘Talos Linux, the Kubernetes Operating system’ built by [Sidero Labs](https://www.siderolabs.com/). Again our predetermined wishlist was covered (even more), and what we got is a minimal OS tailored for Kubernetes, hardened, immutable and ephemeral out-of-the-box. Can you survive even more buzzwords than that? 

Until the present day though, they have fulfilled every promise made on [their website](https://www.talos.dev/). It initially didn’t work on our ARM CM4-based device from OnLogic. But after testing a lot together with their team (thank you!) the [latest release (v1.3.0)](https://www.talos.dev/v1.3/introduction/what-is-new/#raspberry-generic-images) officially supports our ARM devices. Ready for action! Right after the stable release the first batches were shipped and connected to the installations of our producers on the platform.

Overall, Vandebron's use of OnLogic's fabricated gateway devices running Talos Linux demonstrates the potential of IoT computing to drive innovation and sustainability in the renewable energy industry. By leveraging the power of these technologies combined, we are one step closer to achieving our goal of 100% renewable energy, 100% of the time. Care to join our mission? Look for [open positions](https://werkenbij.vandebron.nl/).




### TL;DR
If you want to deploy new Dagster user code respositories, you need to modify and redeploy the whole Dagster system (while they are [presented as separate](https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#separately-deploying-dagster-infrastructure-and-user-code) in the docs). This is undesirable for many reasons, most notably because it slows down a migration or the regular development process. This post presents a way to avoid this and build a fully automated CI/CD-pipeline for (new) user code.

This article assumes that:
* you (plan to) host Dagster on Kubernetes and manage its deployment with Helm and Ansible;
* you want to automate the deployment of new Dagster user code repositories with a CI/CD pipeline automation tool of choice;
* and you want to be able to (re)deploy the whole Dagster system and user code from scratch.

### Why Dagster?

In short Dagster is a tool to build and orchestrate complex data applications in Python. For us, in the end, Dagster improved the development cycle for things like simple cron jobs as well as for complex ML pipelines. Testing the flows locally was never so easy, for instance. And with features like [asset materialization](https://docs.dagster.io/concepts/assets/asset-materializations) and [sensors](https://docs.dagster.io/concepts/partitions-schedules-sensors/sensors), we can trigger downstream jobs based on the change of an external state that an upstream job caused, without these jobs having to know of each other's existence.

However, deployment of new [user code respositories](https://docs.dagster.io/concepts/repositories-workspaces/repositories) caused us some CI/CD related headaches...

### System and user code are separated

Dagster separates the system deployment - the Dagit (UI) web server and the daemons that coordinate the runs - from the user code deployment - the actual data pipeline. In other words: the user code servers run in complete isolation from the system and each other. 

This is a great feature of which the advantages are obvious: user code repositories have their own Python environment, teams can manage these separately, and if a user code server breaks down the system is not impacted. In fact, it even doesn't require a restart when user code is updated!

![Schematic of the Dagster architecture. The user code repositories (green) are separate from the rest of the system (yellow and blue). The right side — irrelevant for now — shows the job runs. Source: https://docs.dagster.io/deployment/overview.](images/dagster-architecture.png)

In Helm terms: there are 2 charts, namely the _system_: `dagster/dagster` ([values.yaml](https://github.com/dagster-io/dagster/blob/master/helm/dagster/values.yaml)), and the _user code_: `dagster/dagster-user-deployments` ([values.yaml](https://github.com/dagster-io/dagster/blob/master/helm/dagster/charts/dagster-user-deployments/values.yaml)). Note that you have to set `dagster-user-deployments.enabled: true` in the `dagster/dagster` values-yaml to enable this.

#### Or are they?

That having said, you might find it peculiar that in the values-yaml of the system deployment, _you need to specify the user code servers_. That looks like this:

```
workspace:
    enabled: true
    servers:
      - host: "k8s-example-user-code-1"
        port: 3030
        name: "user-code-example"
```

**This means system and user deployments are not actually completely separated!**

This implies that, if you want to add a _new_ user code repository, not only do you need to:

1. add the repo to the user code's `values.yaml` (via a PR in the Git repo of your company's platform team, probably);
2. do a helm-upgrade of the corresponding `dagster/dagster-user-deployments` chart;

but because of the not-so-separation, you still need to:

3. add the user code server to the system's `values.yaml` (via that same PR);
4. and do a helm-upgrade of the corresponding `dagster/dagster` chart.

Formally this is the process to go through. If you are fine with this, stop reading here. It's the cleanest solution anyway. But it is quite cumbersome, so...

If you are in a situation in which new repositories can get added multiple times a day - for instance because you are in the middle of a migration to Dagster, or you want a staging environment for every single PR - then read on.

#### Give me more details

How it works is that [for every new repo Dagster spins up a (gRPC) server to host the user code](https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm#user-code-deployment). The separation is clear here. But the Dagster _system_ also needs to know about these user code servers, and it does so through a workspace-yaml file. If you run Dagit locally it relies on a `workspace.yaml` file; on Kubernetes it relies on a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) - a Kubernetes object used to store non-confidential data in key-value pairs, e.g. the content of a file - which they named `dagster-workspace-yaml`.

This workspace-yaml is the connection between the system and the user code. The fact that the charts are designed as such that this workspace-yaml is created and modified through the system deployment rather than the user code deployment is the reason we need to redeploy the system. 

**But what if we could modify this workspace-yaml file ourselves? Can we make the system redeployment obsolete? Short answer: we can.**

### Our solution

_Disclaimer: what we present here is a workaround that we'll keep in place until the moment Dagster releases a version in which the Dagster user code deployment is **actually completely separated** from the system deployment. And it works like a charm._

**Remember: the desired situation is that we do not have to edit the values-yaml files (through a PR) and redeploy all of Dagster for every new repo.**

First of all, we added an extra ConfigMap in Kubernetes that contains the `values.yaml` for the `dagster/dagster-user-deployments` chart. We named it `dagster-user-deployments-values-yaml`. The fact that this is a ConfigMap is crucial to prevent conflicts (see next section).

With the extra ConfigMap in place, these are the steps when a repo gets added:
1. Add the new repo to the `dagster-user-deployments-values-yaml` Configmap.
2. Helm-upgrade the `dagster/dagster-user-deployments` chart with the content of that ConfigMap.
3. Add the server to the `dagster-workspace-yaml` ConfigMap.
4. Do a rolling restart of the `dagster-dagit` and `dagster-daemon` deployment to pull the latest workspace to these services.

**Refresh the workspace in the UI and there it is, your new repo!**

Notes:
* The steps above are completely automatable through your favorite CI/CD pipeline automation tool.
* There is no interaction with a (platform team) Git repo.
* The process, unfortunately, still requires a restart of the system in order to pull the latest workspace-yaml to the system services. The daemon terminates, then restarts, and it might cause a short interruption. Note that this is unavoidable if you add a new repo, no matter how you add it. This could be avoided if a reload of the ConfigMap would be triggered upon a change, [which is possible](https://kubernetes.io/docs/concepts/configuration/configmap/#mounted-configmaps-are-updated-automatically) but not enabled.
* If you want to make changes to an existing repo (not code changes but server setting changes), you only have to do the first step (and _modify_ instead of _add_).

#### How to prevent conflicts

With many of your team members adding new Dagster repositories through an automated CI/CD pipeline, you might face the situation that 2 people are adding a new repo at around the same time. 

When this happens, the `dagster-user-deployments-values-yaml` ConfigMap cannot be uploaded in the first step because Kubernetes demands that you provide the _last-applied-configuration_ when doing an update. If it doesn't match, the upload fails. 

This is perfect as we do not want to overwrite the changes of the conflicting flow. You can optionally build in a retry-mechanism that starts over with pulling the ConfigMap again.

#### How to deploy from scratch

The above does not yet cover how we are able to deploy the Dagster system _and user code_ completely from scratch. Why do we want this? Well, for instance when somebody accidently deletes the `dagster` namespace for instance. Or hell breaks loose in any other physical or non-physical form. Or when we simply want to bump the Dagster version, actually.

The key to this is that we version both the `dagster-user-deployments-values-yaml` and `dagster-workspace-yaml` as a final step to the flow described above (we do it on S3, in a versioned bucket). Whenever we redeploy Dagster (with Ansible) we pull the latest versions and use them to compile both the values-yaml files from it. 

#### How to clean up old repositories

The above described automation _adds_ new repos but doesn't take care of old obsolete repos. The steps for removing a repo are the same for adding one. The exact implementation depends on your situation. You might want to automatically remove PR staging environments after closing a PR, for instance.

### Conclusion

Dagster is an incredibly powerful tool that enabled us to build complex data pipelines with ease. This posts explains how we **streamlined the CI/CD pipeline for user code respositories**, which enabled us to migrate to Dagster very quickly and saves us lots of time on a daily basis.



**A while back, our former technology manager Roy Derks covered the subject of component libraries here on the blog. From a technical perspective, he spoke about when you need one (and when you don’t need one) and what to consider when building one. Since then, obviously a lot has happened at Vandebron. But one of the more interesting things to happen is that design became an integrated part of the digital department, as opposed to previously being attached to marketing. In this new setup, one of the first major projects the design team was involved in was the alignment of our component libraries. And no that’s not a typo, that’s libraries as in the plural form of library. Confusing? I thought so too. In this blog I’ll try to explain further why that was the case, how the work actually helped us bridge the gap between design and development, and dissect the work of unifying those component libraries into one single source of truth and ultimately what’s to become our design system.**

### A bit of a mess
Before we get into it, some background as to where we started out might be useful. As previously mentioned, the design team had just become a part of the digital department and one of the first tasks at hand was the creation of a design system. In the design team, we had previously worked with a certain set of brand guidelines, a style guide if you will, which had not necessarily been translated or aligned to the requirements of a digital product or development environment. Development had also created a set of stylesheets and libraries with reusable components which they used to reduce development time. Having it all separately might sound a bit counter-intuitive, but not very surprising if you consider designers and developers not being in the same department, working on a different timeline, different priorities and so forth. However, this only highlighted the importance of designers and developers working together and the need for a proper design system to help prevent creating a fence between the teams causing unnecessary and ineffective work on both sides. The result of this previous “unsynciness”, a rebrand in 2017, and a re-aligned techstack, was the existence of 3 different libraries and subsequently 3 different sources of truth within the development environment. To add to this, we also had separate design guidelines geared more towards brand/marketing purposes in the design team. Now came the rather massive task of unifying these and eventually, rather than having just a library, _having a system_. 

### Component library ≠ design system
Now, there’s a lot of terminology here that might be difficult to grasp if you’re new to the subject. So I thought I’d clarify what we mean when referring to these, how they fit into the context of our situation, and how many of them we had!

- #### Brand guidelines / style guide (design)
  A set of guidelines and examples outlining all the visual elements of a brand such as logos, color, typography, imagery etc. and subsequently in what - - manner they should be applied. It can also be expanded to include more things brand related such as tone of voice, brand values and so forth. Often with brand guidelines, they are created from a marketing perspective and the digital experience(or product) aspect of how the brand should be applied/represented is usually thought about in the second hand, or not included at all. 

  _Amount: 1_
    
- #### Design kit/library (design)
  A designer resource file with all the available building blocks that make up the digital design language of a brand and/or the user interface of a product. This is usually only visual(no code) and lives in the design software of the designer's choosing. For us this used to be Sketch, but we recently moved to Figma. Can also include documentation and examples of how the different building blocks should be applied and utilized. 

  _Amount: 1_
  
- #### Style sheet (front-end)
  A set of styling properties to be applied when rendering a web page, usually in the format of CSS. This can include things related to the brand guidelines such as font size, colors, etc. but also things related to web layout such as the margins and paddings of different web elements.

  _Amount: 1_

- #### Component library (front-end)
  A set of dynamic web components that can be used in a development environment in order to quickly build user interfaces. This helps to ensure consistency, to avoid rebuilding the same component more than once and to avoid changing said component in more places than one, and subsequently help reduce development time. 

  _Amount: 3_
  
All of the above mentioned things, together with rigorous documentation, amount to what’s called a design system. Having it all combined in a structured way is key to getting the most out of such a system. In our case, most of these things were separate and not necessarily connected to each other. But what stands out most of the things above is probably the fact that we, over time, had amounted to 3 different component libraries. I mentioned earlier how that scenario had transpired so I won’t go into too much detail as to how that happened, but if you’re a developer in a small to medium-sized company and I mention “rebrand” and “new techstack” you can probably figure out how. However complex, this also proved to be an excellent opportunity for our developers and for us in the design team. We finally get to unify our component libraries into one, while simultaneously aligning it with our design kit and expanding the guidelines with new and updated documentation. Thus ensuring that designers and developers speak the same language and share the same single source of truth.

### A guild forms
To kickstart this process we formed a project group(or ‘guild’) composed of 2 designers and 2 developers, each designer and developer from the two consumer-facing scrum teams. The idea was to let the developers work on the migration and unification of the component libraries in collaboration with us designers in the same project, making it easier to align and to create co-ownership of the product. Our first step was to decide on the structure of our component library, this way the developers could slot all the existing, reworked and new components into the right place in the new library. Easy enough right? Well, here comes our first challenge. We initially wanted to take an atomic approach and build our components from the well known and widely used atomic design principles. We also needed to consider the 3 different “product groups” which the library should apply to, all still utilizing the same style properties. 

Vandebron has a wide range of products serving different platforms, with the visual language remaining the same but where the user interface might differ. This requires the top elements of the system(such as colors and typography) to be shared across all products, whereas the lower you get the more product-specific an element becomes. This is the reason why we wanted to structure the system according to the principles of Atomic Design first, in order to assign the components to a hierarchical structure.

![Atomic Design](/images/AtomicDesign.jpg "Atomic Design")

With this approach the atoms would work like design tokens and the molecules would be components general enough that they’d be shared across all product groups, this CORE part of the library would essentially be the stylesheet that impacts all visual aspects of the digital brand experience. Only on organism-level do we start to differentiate what product group the component belongs to. So a change to the CORE parts of the library(atoms or molecules) would impact all components in all product groups.

However, this approach actually made less sense from a development perspective. Not that it wouldn’t work or that the categorization didn’t make sense, but it would require us rewriting all the already existing components. Components that are actively in use. We deemed this approach a bit too high-risk and high-effort for the time being and started looking into alternatives, while still keeping the atomic structure as a more long-term goal. Another thing our initial idea didn’t take into account was the experience of the future main user of the library, **_the developer!_** Organizing a design system after the brand properties and product groups makes a lot of sense from a designers or a marketeers perspective, and it should probably still be presented outwards that way, but a component library is something else(remember?). So based on our development environment and the way we build our websites and apps our developers suggested a different structure:

![Iteration](/images/Iteration.jpg "Iteration")

In this structure, similar to the previous one, the components are instead categorized and sorted by how they should be applied to the page or application that’s being built. Styles, layouts and inputs are general enough to be applied to all product groups whereas from the surface level the components start becoming more specific in their use case. That way, the components can be separated into specific or even several product groups. In this format the components themselves are not as intertwined as in the atomic structure, albeit still connected by the style element. So while it’s a bit more resistant to overall changes the main idea of having the same style properties applying to everything still works, and it helps us designers to better relate and contextualize what we’re designing from more of a development perspective, thus helping bridge the gap between development and design even further. The main insight we drew from this experience is to not let industry standards and certain trends dictate what you should do. Sure they’re important to keep an eye on, but do it with carefulness and always apply an asterisk to it. Figure out what works best for your specific situation and what’s realistic in the short-term vs. in the long-term. There’s no one-size-fits-all.

### Speaking the same language
With the component library migration now underway, we started looking into ways to improve our system from the designers' side of things. As previously mentioned, we had just gone from using Sketch to using Figma and with that came a good opportunity to rebuild, adjust and expand our design kit also. We did that by removing, adding, simplifying and renaming a lot of what was in there since before and with the design kit now adjusted to match the component library we were now also speaking the same language. We can actually now compare this side-by-side with the tools we’re using. In Storybook we have attached the Figma design of every component, simply by activating the feature and pasting the link to its page or artboard in the Figma file. This will refresh in almost real-time if any changes are made so we can easily spot any differences and inconsistencies between what’s live and how the design looks. In Figma, we try to document all our components and give some context as to how it works and should be applied. This is now also directly visible to the developer in the context of the component library. Expanding on our documentation and exposing our digital design guidelines like that has been a great way to create a shared understanding of our designs. Rather than just an image being tossed over a fence, there is now quite literally a direct connection between design and development and therefore also more of a shared ownership.

![Storybook & Figma](/images/StorybookFigma.jpg "Storybook & Figma")

### Further defining the process
As all the alignment on the design side and the migration neared completion, we started seeing a lot of things that could be improved upon or even added to our component library. When we started logging these things down on our project backlog we quickly realized that the scope of our project had quickly been growing into something beyond what was initially intended, and that rather than giving us focus this guild format was instead at risk of creating an isolated bubble of knowledge surrounding the design system. This prompted us to gauge the opportunity and capacity among our development teams to instead tackle these tasks together, either alongside or within their respective day-to-day tasks. In order to do so we needed the buy-in from key stakeholders such as the product owners from the respective development teams. It’s obviously a big ask to get 1 developer from each team to work on improving a component library, especially when they’ve already given us a quarter on migration and have other important business and/or user needs to tend to. So instead, we looked into how we can embed the improvement and further development of our design system into the developers current processes and primary day-to-day work. We structure this by linking our component library improvement/addition tickets to relevant tickets in their respective sprints. In defining the workflow like this, our 2 designer 2 developer guild in effect rendered unnecessary and instead we opened up the co-ownership and contribution to all developers in all customer-facing development teams and in the process of it preventing isolating knowledge too much. In opening up the process like this, another positive side effect we see is the involvement, engagement and subsequent use of our component library going up. With the product designers now also actively a part of the front-end guild meetings, we have an ever bigger forum and bigger opportunity to build a world class component library and design system while also having more hands on deck to work on maintenance and improvements. We still have a long way to go, but all parts are now even more aligned and the future is looking bright!

### What’s next
In the newly formed designer+developer guild, the work of defining requirements and improvements on the design system continues. From the design side we’re also looking to constantly improve on the documentation and the presentation of our system. This is something we imagine we’ll keep on doing continuously and iteratively for as long as it’s needed, if not even forever. After all, “design is never done” and a design system can and should be a living thing constantly evolving along with the products and the brand it serves, and in extension even the promise the brand and it’s products. In our case, that’s to aid in **accelerating the energy transition towards 100% renewable energy**. More on how we exactly do that, and how we always aim to design for impact, in the next blog post. Thanks for reading and stay tuned!


Petter Andersson, Product Designer at Vandebron



_If the type of work mentioned in this blog post sounds interesting to you, [take a look at our job openings here](https://werkenbij.vandebron.nl/l/en/)._ 



# Signing and verfiying SOAP messages with wss4j and Scala

SOAP is not dead. It is an established, XML-based and mature messaging protocol that comes with built-in security mechanisms, integrity checks, content validation and much more. A lot of enterprises and corporations are using it (sadly) still.
Just recently, Vandebron had to implement a SOAP client to communicate with an external party. 
This blog post will explain with code examples how we at Vandebron are signing and verifying SOAP messages for our latest SOAP client implementation. 

For this process, we are using Apache's Web Service Security Library [wss4j](https://ws.apache.org/wss4j/) as it is a proven tool in the WSS context and provides, as a Java library, great interoperability with the programming language Scala.

## Signing SOAP messages

Here we will take a look at the necessary steps to sign a SOAP message like this one:
```xml
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <soapenv:Header/>
  <soapenv:Body>
    <heading>Hello World</heading>
    <body>I am just a test</body>
  </soapenv:Body>
</soapenv:Envelope>
```
To look after signing like this:
```xml
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <soapenv:Header>
    <wsse:Security 
    soapenv:mustUnderstand="1" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
      <ds:Signature 
      Id="SIG-ec946953-2470-4689-ad2f-0c579e1e06e3" xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
        <ds:SignedInfo>
          <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">
            <ec:InclusiveNamespaces PrefixList="soapenv" xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#"/>
          </ds:CanonicalizationMethod>
          <ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/>
          <ds:Reference URI="#id-47817454-f6e2-470c-9109-870e7895e3e0">
            <ds:Transforms>
              <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
            </ds:Transforms>
            <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
            <ds:DigestValue>7KfPcTwDYWtLj4ZVWmWmVqX4IGwbBAAmUPigCdXdk4U=</ds:DigestValue>
          </ds:Reference>
        </ds:SignedInfo>
        <ds:SignatureValue>
          OBnbBWv8S70xDDn5uG++7cTRFa2Uz3D47oxTHuO163Y3/V7H35M1GHXbKaUDOHsgsfx3SdVmVi++ra06cpwJknzqoIQgDV9Qc0ydzfxljCqupPKBnfONDYJtihEE1jtQ0RP7OLzPVNUpgOgHqbLwJu2pRUA05ool+lxIs924OwPVPKyUryoYwWhwY1ttY4P+WY2L3ZqsH3fgoLCyjlvhDEAhsP9PCxsEzPSq3ECC55Nh7nqMoHPj2uNxonuMlPeYbrlMnwyiqEW8s3Sc+WmfiIOgekRE1AdNhpn3ARlO490nObQtXCU/TxeTfbh98TMbQRZWWyT4HuLS3fF6aeyD/Q==
        </ds:SignatureValue>
        <ds:KeyInfo Id="KI-e18395de-9a26-4cad-9501-7c6cf6c7c74a">
          <wsse:SecurityTokenReference wsu:Id="STR-daa47836-f1f9-4d71-95cc-b7bcc6051c84">
            <wsse:KeyIdentifier 
            ValueType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-x509-token-profile-1.0#X509SubjectKeyIdentifier" EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary">
              ox4ajWTdigy9oApTYs97CuCV/4k=
            </wsse:KeyIdentifier>
          </wsse:SecurityTokenReference>
        </ds:KeyInfo>
      </ds:Signature>
    </wsse:Security>
  </soapenv:Header>
  <soapenv:Body 
    wsu:Id="id-47817454-f6e2-470c-9109-870e7895e3e0" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
      <heading>Hello World</heading>
      <body>I am just a test</body>
  </soapenv:Body>
</soapenv:Envelope>
```

For implementing the steps of the blog post you will need:
- a SOAP service you want to send messages to
- documentation of that SOAP service that describes:
    - signature algorithm
    - canonicalization method
    - digest algorithm
    - key identifier type
- a private key with which you will sign your messages
- a certificate that is the counterpart of the private key
- (optional) a pool of trusted certificates

Our private and public key pair are available in the PKCS#12-format (.p12 file extension). Check out [this](https://www.ssl.com/how-to/create-a-pfx-p12-certificate-file-using-openssl/) to learn more about this format and how to achieve it.
The pool of trusted certificates are in the [PKCS#7 format](https://www.ssl.com/guide/pem-der-crt-and-cer-x-509-encodings-and-conversions/) (.p7b file extension).

First we have to setup the necessary dependencies:

```scala
   // in your build.sbt or project/Dependencies.scala
  // enabling signing and signature verification for SOAP messages
  lazy val webServiceSecurity = Seq(
    "org.apache.wss4j" % "wss4j"                    % "2.3.1" pomOnly (),
    "org.apache.wss4j" % "wss4j-ws-security-dom"    % "2.3.1",
    "org.apache.wss4j" % "wss4j-ws-security-common" % "2.3.1"
  )

  libraryDependencies ++= webServiceSecurity
```

Next, we continue with a scala representation of our certificate we are using for signing:

```scala
  import org.apache.wss4j.dom.WSConstants
  
  // algorithm configuration
  object SigningCertificate {
    val CanonicalizationMethodURI: String = "http://www.w3.org/2001/10/xml-exc-c14n#"
    val DigestAlgorithmURI: String        = DigestMethod.SHA256
    val SignatureAlgorithmURI: String     = "http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"
    val KeyIdentifierType: Int             = WSConstants.SKI_KEY_IDENTIFIER
  }

  case class SigningCertificate(keyStore: KeyStore, password: String) {
    require(
      keyStore.aliases().asScala.size == 1,
      s"Certificate of Keystore needs to have one alias but had ${keyStore.aliases().asScala.size}"
    )
    val alias: String = keyStore.aliases().nextElement()

    override def toString: String = s"SigningCertificate(alias=$alias)"
  }
```
In the documentation of the SOAP service that you want to call should stand some information regarding the canonicalization method, signature algorithm, digest algorithm, and the key identifier type. Those are algorithms and information that define the signing process and we explain roughly now.

Before signing a message it has to be canonicalized. "Canonicalization is a method for generating a physical representation, the canonical form, of an XML document that accounts for syntactic changes permitted by the XML specification" (from [here](https://www.di-mgt.com.au/xmldsig-c14n.html)). In our case, the Exclusive XML Canonicalization is used.

The digest algorithm is used to ensure the integrity of the message during the verification of a signature. The algorithm is used to calculate a hash of the signed message. It should be documented in the SOAP service documentation. Here we will use SHA256 as a hashing algorithm.

The signature algorithm describes how the message will be signed. It can be defined in the SOAP service documentation but in the worst case you can read this algorithm from the certificate itself by using [`keytool`](https://docs.oracle.com/en/java/javase/12/tools/keytool.html):
```
$ keytool -list -v -keystore signature.p12
Enter keystore password: ...

[...] # more information about the certificates

Signature algorithm name: SHA256withRSA # thats what we are after!

[...] # more information about the certificates
```
According to the keytool inspection we will use SHA256withRSA (http://www.w3.org/2001/04/xmldsig-more#rsa-sha256) for signing.

Last but not least, in our signature, a `<KeyInfo>` element is included. This element contains information about the public key of the sender (us) and is needed for the signature verification once the message is received (read more [here](https://www.xml.com/pub/a/2001/08/08/xmldsig.html)). Since we have our public key provided we don't need to do much here. The `KeyIdentifierType` describes which form of key identifier is used to present the public key information.

Having all this information about our certificate in place, we build the mechanism to load in our signing certificate. For this, we create the object `KeyStoreBuilder`.

```scala
import java.io.{File, FileInputStream}

object KeyStoreBuilder {

  def loadSigningCertificate(signingCertificate: File, password: String): SigningCertificate = {
    val fis = new FileInputStream(signingCertificate)
    val ks: KeyStore               = KeyStore.getInstance("PKCS12")
    ks.load(fis, password.toCharArray)
    SigningCertificate(ks, password)
  } 
}
```
Bear in mind, that you probably **don't** want to version any sensitive information like private keys and passwords hard-coded or in any environment variables, so a safe mechanism for storing/fetching passwords and certificates (like [Vault](https://www.hashicorp.com/products/vault)) should be in place.

With the signing certificate in place, we can actually start signing a message. The next code example contains quite some Java boilerplate from wss4j that is required to make the signing mechanism work.

To restrict the usage of Java classes to a small portion of our code we will firstly implement a conversion method `.toElem` inside of the companion object `SigningService`:

```scala
  import java.io.StringWriter
  import javax.xml.transform.{OutputKeys, TransformerFactory}
  import javax.xml.transform.dom.DOMSource
  import javax.xml.transform.stream.StreamResult

  import org.w3c.dom.Document

  import scala.xml.Elem

  object SigningService {
    implicit class RichDocument(document: Document) {
      private val tf = TransformerFactory.newInstance()

      def toElem: Elem =
        val transformer = tf.newTransformer()
        transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
        val stringWriter = new StringWriter()
        transformer.transform(new DOMSource(document), new StreamResult(stringWriter))
        scala.xml.XML.loadString(stringWriter.getBuffer.toString)
    }
  }
```
With that, we can convert any `Document` SOAP message representation back to the `scala.xml` supported  `Elem` format.

```scala
class SigningService(signingCertificate: SigningCertificate) {

  // importing our conversion method
  import SigningService.RichDocument

  /**
    * REQUIRED, otherwise it will throw:
    *
    * org.apache.wss4j.common.ext.WSSecurityException:
    * You must initialize the xml-security library correctly before you use it.
    * Call the static method "org.apache.xml.security.Init.init();"
    * to do that before you use any functionality from that library
    */
  org.apache.xml.security.Init.init()
  
  private val documentBuilderFactory = DocumentBuilderFactory.newInstance()
  private val crypto: Merlin = getCrypto

  crypto.setKeyStore(signingCertificate.keyStore)

  def signElement(elem: Elem): Elem = {
    documentBuilderFactory.setNamespaceAware(true)
    // converting Elem to Document (Scala to Java conversion)
    val doc = documentBuilderFactory.newDocumentBuilder().parse(new InputSource(new StringReader(elem.toString())))

    // WSSecHeader wraps around the document we want to sign
    val header = new WSSecHeader(doc)
    header.setMustUnderstand(true)
    header.insertSecurityHeader()

    // start building Signature, use the (wrapper) header-instance
    val builder = new WSSecSignature(header)
    builder.setUserInfo(signingCertificate.alias, signingCertificate.password)

    // setting algorithms
    builder.setSignatureAlgorithm(SigningCertificate.SignatureAlgorithmURI)
    builder.setSigCanonicalization(SigningCertificate.CanonicalizationMethodURI)
    builder.setDigestAlgo(SigningCertificate.DigestAlgorithmURI)
    builder.setKeyIdentifierType(SigningCertificate.KeyIdentifierType)
    builder.setAddInclusivePrefixes(true)

    // signing the document!
    val signedDocument = builder.build(crypto)
    // conversion back to Elem
    signedDocument.toElem
  }

  private def getCrypto: Merlin = {
    val properties = new Properties()
    properties.setProperty("org.apache.wss4j.crypto.provider", "class org.apache.ws.security.components.crypto.Merlin")
    CryptoFactory.getInstance().asInstanceOf[Merlin]
  }
}
```

Wss4j is a library that maintains an internal state during a signing process, but to avoid confusion it can be summarized as:
1. `WSSecHeader` wraps around the document to be signed
2. the WSSecHeader instance `header` will be used as part of the `WSSecSignature`-Builder
3. the WSSecSignature instance `builder` gets configured with all necessary information, which algorithms are used for signing, digesting, canonicalization, which key identifier should be included. Those settings an vary from webservice to webservice.

The actual signing of the document, which is now nested like a matryoshka doll, is happening with the help of an instance of `Crypto`. `Crypto` will contain either a keystore or a truststore or even both. It needs to be specified in the `crypto.properties` file or a runtime which class of Crypto will be used.
 The most common one is [`Merlin`](https://ws.apache.org/wss4j/apidocs/org/apache/wss4j/common/crypto/Merlin.html).
We have decided to specify its configuration during runtime, since it is more visible than a properties file. Nevertheless, the `crypto.properties`-file needs to exist in your `resources` folder neverthless otherwise you will get a following `WSSecurityException`:
```java
  org.apache.wss4j.common.ext.WSSecurityException: No message with ID "resourceNotFound" found in resource bundle "org/apache/xml/security/resource/xmlsecurity"
  [... rest of stacktrace ...]
  Cause: java.nio.file.NoSuchFileException: crypto.properties
```

And that's it! The `KeyStoreBuilder` helps us to load a `SigningCertificate` and the `SigningService` uses this loaded certificate to sign SOAP messages. 
A receiver of our SOAP message has all the necessary information in our signature to verify that this message has not been tampered with and we are the original sender.

This verification is something we should also do on our side for incoming messages. So let's take a look at how we can verify the signature of received messages.

## Verification of SOAP messages

Verifying the signature of incoming messages is equally important to ensure that the connection is secure. A verification process will tell you if the message is coming from a trusted source and has not been tampered with.

As previously mentioned we need our source of truth, a pool of trusted public keys from all parties which will receive our SOAP messages. These build the basis of the trust store.

We will create a `TrustedCertificates` wrapper class in which we will load in the trust store and add this method to the `KeyStoreBuilder`.
```scala
case class TrustedCertificates(keyStore: KeyStore)

object KeyStoreBuilder {

    def loadTrustedCertificate(certificates: Seq[File]): TrustedCertificates = {
    val ks = KeyStore.getInstance(KeyStore.getDefaultType)
    // we just want the keystore to act as a truststore (only containing trusted certificates), so we initialize it empty
    ks.load(null, null)
    val cf = CertificateFactory.getInstance("X.509")
    certificates.foreach { file =>
      CloseableUtil.using(getClass.getResourceAsStream(file.getPath)) { fis =>
        val certPath = cf.generateCertPath(fis, "PKCS7")
        certPath.getCertificates.asScala.toList.foreach { certificate =>
          ks.setCertificateEntry(file.getName, certificate)
        }
      }
    }
    TrustedCertificates(ks)
  }
}
```
This trust store is under the hood also just a KeyStore, without containing a private key that requires a password, that's why we can initialize the KeyStore with `null`-parameters.

Now, the SigningService needs to be extended with this trusted certificates and a `verifySignatureOf`-method:

```scala
import java.io.StringReader
import java.util.Properties
import javax.xml.parsers.DocumentBuilderFactory

import org.apache.wss4j.common.crypto.{ CryptoFactory, Merlin }
import org.apache.wss4j.dom.engine.WSSecurityEngine
import org.xml.sax.InputSource

import scala.util.{Failure, Success, Try}
import scala.xml.Elem

class SigningService(signingCertificate: SigningCertificate, trustedCertificates: TrustedCertificates) {

    private val engine = new WSSecurityEngine()
    private val documentBuilderFactory = DocumentBuilderFactory.newInstance()
    private val crypto: Merlin = getCrypto

    crypto.setKeyStore(signingCertificate.keyStore)
    crypto.setTrustStore(trustedCertificates.keyStore)

    def verifySignatureOf(elem: Elem): Boolean = {
      documentBuilderFactory.setNamespaceAware(true)
      val doc = documentBuilderFactory.newDocumentBuilder().parse(new InputSource(new StringReader(elem.toString())))

      Try(engine.processSecurityHeader(doc, null, null, crypto)) match {
        case Success(_) => true
        case Failure(exception) =>
          // replace with proper logging
          println(
            s"Unsuccessful signature verification, it is most likely that the certificate used for signing is not in our Truststore: ${exception.getMessage}")
          false
      }
  }

  private def getCrypto: Merlin = {
    val properties = new Properties()
    properties.setProperty("org.apache.wss4j.crypto.provider", "class org.apache.ws.security.components.crypto.Merlin")
    CryptoFactory.getInstance().asInstanceOf[Merlin]
  }
}
```

And with that, we have completed our roundtrip of signing and verifying SOAP messages!

Here are gists, articles, and documentation that inspired and helped us to figure out the signing and verification process for our SOAP client. Feel free to check them out!

* * *

### Sources

[WSSecurityVerifier by Luis Wolff](https://gist.github.com/luiswolff/1d388ec8c1d63cfb58974a6f826bc1be) 

[WSSecuritySigner by Luis Wolff](https://gist.github.com/luiswolff/64d15a99fbb5ec4b4e90eec04b09e053)

[Unit Tests from ws-wss4j](https://github.com/apache/ws-wss4j/blob/master/ws-security-dom/src/test/java/org/apache/wss4j/dom/message/SignatureTest.java)

[An Introduction to XML Digital Signatures](https://www.xml.com/pub/a/2001/08/08/xmldsig.html)

[SOAP vs. REST](https://stackify.com/soap-vs-rest/)


At the end of March, we organized a public hackathon to create solutions to tackle climate challenges. After having done internal hackathons, we thought it was time to share our technologies with other innovative people and companies. Together with a group of partners, and enthusiastic participants, spend three full days of (remote) hacking with great results.

### Why organize a public hackathon?

Climate change is one of the many pressing challenges our society is currently facing. At [Vandebron](https://vandebron.nl/), we want to continue finding ways to tackle this immense challenge. That’s why we decided to organize a 3-day GreenTech hackathon that ran from March 31st to April 2nd, 2021. We've been organizing internal hackathons for the past four years, to foster innovation within our company and allow our developers to work on something exciting without any constraints. If you want to read more about why we organize internal hackathons, you can find an article by our CTO [here](https://www.vandebron.tech/blog/power-regular-hackathons). 

By organizing a public hackathon, we hoped to attract a bigger audience, possibly even outside our country, The Netherlands, and attract partners to work together with. We succeeded in both, and together with [Hack the Planet](https://hack-the-planet.io/) and [Top Dutch Solar Racing](https://solarracing.nl/), we wanted to find technological solutions to problems in wildlife conservation and renewable energy. For these three days, all participants got the opportunity to work on challenges from our partners, access their technology and knowledge, and got the chance to win unique prizes. Also, we organized a free event with speakers Florian Dirkse ([The Ocean Cleanup](https://theoceancleanup.com/)), Thijs Suijten (Hack the Planet) and Heleen Klinkert ([Nieuw Groen](https://nieuw-groen.nl/)). 

### Looking back

The event started on March 31st, when all hackathon challenges were presented and the participants could select which challenge they wanted to work on. People from all over The Netherlands (and even beyond) signed up for the hackathon, ranging from students from the University of Amsterdam to young professionals looking for a job. The first challenge the participants could subscribe to was from Vandebron itself, where teams got the opportunity to use a selection of our Electronic Vehicle (EV) data. With this data, they could for example make a forecast on the amount of charging sessions we could expect on a typical day. Second, our partner Hack the Planet presented their challenge that was aimed at thinking of innovative solutions for their project [Hack the Poacher](https://www.hackthepoacher.com/). With Hack the Poacher, they install smart camera traps in African wildlife reservations to detect poachers. The teams could use their camera traps and data to create more solutions to map the poachers or use the camera traps for other needs. Finally, the students from Top Dutch Solar Racing presented a challenge to simulate the race they were supposed to join at the end of the year in Australia. Using their weather and traffic data, the teams could simulate the race and predict how much time they would need to complete the race. After selecting a challenge, all teams started the hackathon and participated in sessions to learn more about the challenges to get started.

All teams continued working on the hackathon challenge on the second day, after a nice warming-up quiz about climate change in the morning. For most teams this second day was when their project started to take shape, and they got a better idea about what they would be presenting on the final day. This second day was also an opportunity for non-technical people to get to know Vandebron and their partners better as we organized inspirational sessions with talks from different speakers in the afternoon. One of the co-founders from The Ocean Cleanup, Florian Dirkse, inspired us with his story behind making a difference in the world. After which, one of our hackathon partners Thijs Suijten, from Hack the Planet, demonstrated how technology can be used for the good. Our third, and final, speaker Heleen Klinkert (Nieuw Groen), showed how we can compensate for our CO2 emissions by storing them in the soil.

On the final day of the hackathon, all teams had to finalize their projects and create a presentation for the closing ceremony. During this ceremony, all participants and partners looked back at the past three days and shared what they had been working on during the hackathon. For every challenge, one team could win and take home several prizes, sponsored by [Marie-Stella-Maris](https://marie-stella-maris.com/), [EV Experience](https://evexperience.nl/), and [Klimaatroute](https://www.klimaatroute.nl/). The first presentations were for the Vandebron challenge about EV forecasts. This challenge was won by not one but two teams as the jury and audience were so impressed by their solutions. Both teams created not only the forecast based on the sample data provided, but also created interactive dashboards. On the challenge for Hack the Planet, the team that won came up with a unique solution to use the camera traps to detect wild animals on the streets. For countries like India, this is a huge problem, as wild animals get stuck in traffic or walk through rural areas. The final winner of the hackathon was a group of students that simulated the Top Dutch Solar Racing trip through Australia and forecasted they could complete the race within 7 days.

### Thanks everyone

I'd like to thank all the participants, prize/challenge partners, and speakers for their efforts during these days. The GreenTech Hackathon 2021 was a huge success thanks to everyone that has been involved. Keep following the [vandebron.tech](https://vandebron.tech) to be updated on future hackathons and events.



At Vandebron we have been organizing a regular hackathon for the last four years. Every three months we organize a two-day event. At first glance this seems quite an investment. Eight days a year, almost losing two working-weeks of productivity for your teams! 

Our company is like any other. Our roadmaps are stuffed, our backlogs are never-ending and pressure for delivering value to our customers is always present. Our ambitions are always higher than what we can handle with the amount of teams and people available. We like to say: ‘the energy transition can’t wait!’, but we sure do have to prioritize our projects very carefully.

However this does not stop us from organizing our quarterly hackathons. Most of the time our regular hackathons are light-weight. People within the company know how it works. We try not to waste too much time in ‘organizing’ the event. We get right to it. 

#### Reasons why you should be organizing (regular) hackathons:
- Fun - this reason does not need much explanation. Working on challenging, fun and creative ideas with uncertain outcome in a not-business-as-usual way. It makes you step out of your daily comfort zone and explore new things. 
- Walk the extra mile - Some team-members will have the energy, enthusiasm and commitment to use their spare time to fuel their curiosity and bring new insights to the workplace. These are the same people that you also expect to walk the extra mile if the team- or company objectives are at stake. This is in that same spare time! But in the end, if you value your teams to continuously think about new ideas, insights and work on out-of-the-box ideas, it is not a weird idea to create this environment within the company.
- Bottled up energy - our people are focused on reaching goals and objectives. Every day, every week and every sprint the focus is clear. This also means that there is not always time for creative or high risk escapades that could hurt the overall team objectives. This might give an unsatisfied feeling to people. If the bottled up energy can not be released, engineers might get frustrated. But maybe even more important, you might be missing opportunities for the company.
- Cross team collaboration - in an agile way of working the concept of the team is very important. At Vandebron we focus on teams staying together for a long period of time. This makes the relationship between individuals stronger, the knowledge of the product deeper and the team as a whole more effective. However, the company is bigger than your team. There might be different ways of connecting with other people within your company, but a hackathon is an ideal way of linking yourself up with people that you can learn from. It can really bring you new insights as an individual, and it will also be an investment for improved cross-team collaboration going forward.
- Learning organisation - as mentioned, hackathons give you an excellent opportunity to learn new things. For yourself, but definitely also for the company. In my experience I often see that topics get selected that have high-risk and high-reward kind of characteristics. These topics can be scary to touch, which make you go out of your comfort zone. This is where you learn the most! These high-risk and high-reward projects are also very likely to fail, meaning that the reward is not as high as expected, or the complexity and risks are even greater than anticipated. At these moments the pressure-cooker of a hackathon is very valuable, because it forces the participants to draw conclusions in a short time-frame. The insights gained from these projects can be used to further steer the roadmap. And last but not least, it supports building a culture of being bold enough to try new things, and fail fast. I’ve noticed this is appreciated by a lot of people within the company and the hackathon contributes to a culture of innovation.

#### Our most important learnings over the years
- Spotlights on - It is good to put teams and their results in the spotlight. Let them take the podium and make sure there is an audience. However don’t make it too much about winning. Ideas that have completely failed are just as important as over-engineered fancy product demos. At Vandebron we like to declare ‘winners’ in different categories: ‘Fun & Original’, ’Impactful’, ‘Exploration & Learning’ and ‘Technical achievement’. 
- Harvest your ideas continuously - during normal work and life you hit those topics that you would like to investigate a bit deeper. But while you stumble upon such a topic you don’t have the time to dive into it. So therefore, write your idea down and publish it in the ‘hackathon-idea-box’ for everyone to see! It might already give you some good conversations during coffee or lunch, and it might already generate you some people that would like to join forces with you during the hackathon. Because rest assured, a new hackathon is always coming up!
- To-theme-or-not-to-theme - we have experimented with adding a theme to a hackathon. It can help the company to generate ideas and action in a certain area of interest. It also helps to generate more focus within the company on a certain persistent challenge that we feel deserves a solution. Although everyone will be working on different sub-topics the full event will be experienced as more correlated and unified. But be careful not to push normal business-projects disguised as hackathon projects to your teams. This goes against the basic concept of a hackathon. At Vandebron we sometimes pick a theme if we would like to motivate people to think about ideas in a certain direction. But most of the time we keep it open.
- Participation is optional. - At Vandebron we have autonomous teams with professionals that can manage their own agenda. As a team and as an individual. We put effort in promoting the hackathon by trying to make people enthusiastic about participating. But in the end people make their own decisions. Sometimes the team and company objectives do need to have priority, but the teams are perfectly able to make this judgement call themselves.
- Magnify impact - show everyone what the impact is they have been making. It is good if people recognize how some projects have become reality and that feedback will be appreciated by the community. It gives people a feeling that the podium of the hackathon is a strong force. And ultimately that is how you also proof the value of organizing a hackathon.

For our next hackathon we are opening our (virtual) doors also for guests, as we are organizing a GreenTech hackathon with other sustainability minded companies (‘Hack the Planet’ and ‘Top Dutch Solar Racing’). You can find more information and sign up via [this link](https://www.vandebron.tech/greentech-hackathon). It is the first time we do it like this, and we sure will learn another thing or two!



In October 2020 D2IQ [announced](https://d2iq.com/blog/d2iq-takes-the-next-step-forward) that they are moving onwards with their Kubernetes offering. Vandebron has been a D2IQ customer for their DCOS offering, we were just in the middle of a migration of our first workloads to DCOS Enterprise. We have evaluated the D2IQ K8s offering and decided to go for another Kubernetes product. We had a few migrations over the years, we migrated from Azure to AWS, we migrated workloads from normal instances to spot instances and all these migrations were done with nearly any downtime. We plan to reduce the downtime to a couple of minutes this migration and this is a real challenge. The first challenge that we will discuss today: We want to pair our Kubernetes clusters to the DCOS/Mesos clusters, while we move a workload it should be able to connect to its dependencies in the DCOS cluster. We use DCOS for our NoSQL databases like Cassandra, internal data that we want to keep internal. Pairing DCOS and Kubernetes clusters enable us to reduce downtime, enabling us to switch back if we run into issues and move faster because it reduces complexity.

## L4LB

The internal layer 4 load balancer DCOS provides is used in the majority of our workloads. When our data scientists schedule a spark driver, they connect to the spark dispatcher through the Layer 4 load balancer. Most of the DCOS frameworks use this Layer 4 load balancer as an internal service discovery tool, with Vandebron we use this layer 4 load balancer to communicate between services. In a default DCOS set up this load balancer responds on domain names like: `spark-dispatcher.marathon.l4lb.thisdcos.directory:7077`

When we ping the spark dispatcher we get the following:

```bash
PING spark-dispatcher.marathon.l4lb.thisdcos.directory (11.155.161.35) 56(84) bytes of data.
64 bytes from 11.155.161.35 (11.155.161.35): icmp_seq=1 ttl=64 time=0.024 ms
```

After some investigation we found out that this IP range is not actually on a network interface, it is a Linux kernel functionality called `IPVS`. With IPVS you can do layer 4 load balancing, you provide the target location and the location you want to respond on.

When we search for the IP from the spark dispatcher with ipvsadm, we get 3 results:

```bash
sudo ipvsadm -L -n |grep --color '11.155.161.35\|$'
TCP  11.155.161.35:80 wlc
  -> 10.2.7.146:16827             Masq    1      0          0
TCP  11.155.161.35:4040 wlc
  -> 10.2.7.146:16826             Masq    1      0          0
TCP  11.155.161.35:7077 wlc
  -> 10.2.7.146:16825             Masq    1      0          0
````

As you can see the IP `11.155.161.35` points towards `10.2.7.146`, even the ports are configured and forwarded. We can add our route with ipvsadm, to understand IPVS a bit better. For example:

```bash
sudo ipvsadm -A -t 1.2.3.4:80 -s wlc # we add the target server and assign the scheduler
sudo ipvsadm -a -r 10.2.7.146:16825 -t 1.2.3.4:80 -m # we configure the real server and target server and configure Masquerading
curl 1.2.3.4:80
{
  "action" : "ErrorResponse",
  "message" : "Missing protocol version. Please submit requests through http://[host]:[port]/v1/submissions/...",
  "serverSparkVersion" : "2.3.4"
}
```

This results in that the spark dispatcher now also is available on `1.2.3.4:80`. As mentioned before we wanted to connect our DCOS and Kubernetes clusters, getting hundreds of entries from ipvsadm and manually adding them one by one didn’t sound appealing to us. Especially if you consider that sometimes services fail and run on a different port or different host after recovery, maintaining this by hand would be a nightmare. We therefore decided to build a tool to sync IPVS entries from DCOS to Kubernetes.

## Stack

Within Vandebron we have our tech stack, we strongly believe it is good to eat your own dog food. When possible and when our use cases are similar we use the same tools as our Developers use. The parts of the stack we will be using are:

- AWS ELB in front of Traefik 1.7
- DCOS
- Kubernetes

Within our platform team, we use Golang as our scripting language. Golang gives us the ability to build binary files with all the required libraries in the binary, we don’t have to install any packages, we do not even need to install Golang on the machine the application will be running on.

In our DCOS cluster we use Traefik 1.7, this version of Traefik only forwards HTTP requests. We decided to use Traefik to expose a JSON endpoint so we can gather the IPVS information from this location.

## ipvs-server

Within our DCOS cluster we will expose the IPVS information through a JSON endpoint. We have built a tool for this to expose this information in multiple ways. In the next section, we are going to discuss some of the concepts and choices we made, we won’t deep dive into Go specifics. We have provided the entire code for this project in the examples directory of our GitHub repo:
<https://github.com/Vandebron/tech-blog>

First, let’s discuss the library we use: <https://github.com/nanobox-io/golang-lvs>. This library in its essence translates to ipvsadm commands, it helped save us time to implement this ourselves. There are some gotcha’s, such as newlines are not filtered out from the output. We solved this by cleaning up some of the data.

In the `childChan` function we create a go channel that is responsible for polling `ipvsadm` every 10 seconds and stores the result in a couple of variables we use in our HTTP endpoints. IPVS is a Linux kernel functionality and should be highly performant, we do not want to trigger kernel panics when the server gets overloaded with requests. We expect that every 10 seconds gives us accurate enough results, we can always lower this interval to ensure faster results. We also added in this function the string manipulation to ensure all the newlines were gone in the JSON output. The newline gave issues when we tried to add the IPVS scheduler entries.

```go
func childChan(c chan bool) {
   fmt.Println("Starting time based IPVS Admin poll")

   pollInterval := 10
   timerCh := time.Tick(time.Duration(pollInterval) * time.Second)
   // Time based loop to generate Global variable
   for range timerCh {
       select {
       // when shutdown is received we break
       case <-c:
           fmt.Println("Received shutdown, stopping timer")
           break
       default:
           var err error
           listIpvs.Save()
           ipvsString = fmt.Sprintln(listIpvs.Services)

           res := &responseObject{
               Services: listIpvs.Services,
           }
 
           ipvsJSONbyte, err := json.Marshal(res)
           if err != nil {
               logToErr.Printf("ERROR: -- Marshal JSON -- %v\n", err)
           }
 
           ipvsString = string(ipvsJSONbyte)
           ipvsJSON = strings.Replace(ipvsString, `\n`, ``, -1)
           if debug != false {
               logToOut.Println("DEBUG: -- ipvsJSON --", ipvsJSON)
           }
       }
   }
}
```

Next is the index handler, we set our headers correctly and print the result as we would receive through ipvsadm. The index is mainly for our platform engineers to debug and verify the output. Thanks to this overview we found much faster that there was a newline hidden in the scheduler output.

```go
func index() http.Handler {
   // Generating the Index
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

       // Only available when debug is on
       if debug != false {
           logToOut.Println("DEBUG: -- index --", ipvsString)
       }
 
       if r.URL.Path != "/" {
           http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
           return
       }
       w.Header().Set("Content-Type", "text/plain; charset=utf-8")
       // Site security testers expect this header to be set
       w.Header().Set("X-Content-Type-Options", "nosniff")
       w.WriteHeader(http.StatusOK)
       fmt.Fprintln(w, ipvsString)
   })
}
```

The JSON endpoint is what we use in the client communicate with the server. 

```go
func jsonz() http.Handler {
   // Generating the Index
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

       // Only available when debug is on
       if debug != false {
           logToOut.Println("DEBUG: -- jsonz --", ipvsJSON)
       }
 
       if r.URL.Path != "/json" {
           http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
           return
       }
       w.Header().Set("Content-Type", "application/json; charset=utf-8")
       // Site security testers expect this header to be set
       w.Header().Set("X-Content-Type-Options", "nosniff")
       w.WriteHeader(http.StatusOK)
       fmt.Fprintln(w, ipvsJSON)
   })
}
```

We ask our Developers often to implement a basic health endpoint, in DCOS we use this to see if a service needs to be restarted. In our application we enable set the statusOK in the index or in the JSON endpoint.

```go
func healthz() http.Handler {
   // Generating the healthz endpoint
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
       if atomic.LoadInt32(&healthy) == 1 {
           w.WriteHeader(http.StatusNoContent)
           return
       }
       w.WriteHeader(http.StatusServiceUnavailable)
   })
}
```

In our logging and tracing functions we want to register the clients that are connecting, this gives us information where calls are coming from. It helps us debugging if we see weird behaviour.

```go
func tracing(nextRequestID func() string) func(http.Handler) http.Handler {
   // Tracing the http requests so its easier to check if server is reached
   return func(next http.Handler) http.Handler {
       return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
           requestID := r.Header.Get("X-Request-Id")
           if requestID == "" {
               requestID = nextRequestID()
           }
           ctx := context.WithValue(r.Context(), requestIDKey, requestID)
           w.Header().Set("X-Request-Id", requestID)
           next.ServeHTTP(w, r.WithContext(ctx))
       })
   }
}

func logging(logToOut *log.Logger) func(http.Handler) http.Handler {
   // Creating logging entry tracing the http requests
   return func(next http.Handler) http.Handler {
       return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
           defer func() {
               requestID, ok := r.Context().Value(requestIDKey).(string)
               if !ok {
                   requestID = "unknown"
               }
               logToOut.Println(requestID, r.Method, r.URL.Path, r.RemoteAddr, r.UserAgent())
           }()
           next.ServeHTTP(w, r)
       })
   }
}
```

IPVS needs to be executed with root privileges, to ensure this is correct we get the userid and print it when starting the server.

```go
// getProcessOwner function is to see who is running the process. It needs to be a sudo / root user
func getProcessOwner() string {
   stdout, err := exec.Command("ps", "-o", "user=", "-p", strconv.Itoa(os.Getpid())).Output()
   if err != nil {
       logToErr.Printf("ERROR: -- getProcessOwner -- %v\n", err)
       os.Exit(1)
   }
   return string(stdout)
}
```

We added the init function to ensure we print results the moment the server starts up, if we would not do this it would take 10 seconds for the go channel to activate

```go
func init() {
   // Placing the Save and val in the init, else we will need to wait for channel to perform its first run
   listIpvs.Save()
   ipvsString = fmt.Sprintln(listIpvs.Services)
}
```

In the main function, we set the configurable flags, such as debugging to show error messages. It proved useful during the creation of this tool to keep track and print output. If we would print the output at every call to our logs, our Elastic cluster would get thousands of logs that add little to no value.

We configure the listen port in the flags, we can use the portIndex from DCOS to assign a random port on the host to listen on. We also provided to print the version we are running. In our versioning, we use a constant to list the application semver version, we also provide the git-commit hash.
When we begin the server we print the version information, the port we listen on and the user running the process. We then start the server process with the go channel, in setting up the go channel we ensure that when the server stops we try to gracefully stop the server within a 30-second timeframe. Since our ipvsadm timer is 10 seconds it should be able to cleanly shutdown within that period.

### Docker build

In the repository, we have included a Dockerfile and a script to build the Dockerfile. In this Dockerfile, we pass the git commit hash to the go install. This way we always get the Git Hash from our GitHub repo and we can use this information in our version output.

### DCOS service.json

In the repository, we have provided the service.json file, since it is opinionated on using Traefik you might need to change it. But in this service.json you see how we set up Traefik, the health check, and port index. Since the Mesos UCR container has fewer abstractions and has fewer limited capabilities. We can run the IPVS server inside a UCR container and get all the output as if we were running this directly as root on the host machine.

## ipvs-client

The IPVS client is the component we use in the Kubernetes environment. The client connects to the server and gets the IPVS entries from the IPVS server inside our DCOS cluster. It then adds these IPVS entries to each node in the Kubernetes cluster. You, therefore, need to run each client per Kubernetes node.

You can find the code from the IPVS client in our repository.

```go
func httpGet(remoteURL string) []byte {
   if debug != false {
       _, err := url.ParseRequestURI(remoteURL)
       if err != nil {
           panic(err)
       }
   }

   req, err := http.NewRequest(http.MethodGet, remoteURL, nil)
   if err != nil {
       logToErr.Fatalf("ERROR: -- new HTTP request -- %v", err)
   }

   ipvsClient := http.Client{
       Timeout: time.Second * 2, // Timeout after 2 seconds
   }
   req.Header.Set("User-Agent", "go-ipvs-get \tversion: "+version+"\t Git Commit: "+gitCommit)
   res, err := ipvsClient.Do(req)
   if err != nil {
       logToErr.Fatalf("ERROR: -- ipvsClient -- %v\n", err)
   }

   if res.Body != nil {
       defer res.Body.Close()
   }

   body, readErr := ioutil.ReadAll(res.Body)
   if readErr != nil {
       logToErr.Fatalf("ERROR: -- body -- %v\n", readErr)
   }

   return body
}
```

In the httpGet function we can debug the URL and check if it is valid. Again we set the correct headers and retrieve the JSON body.

```go
func unmarshal(body []byte) []lvs.Service {

   res := &responseObject{
       Services: listIpvs.Services,
   }

   jsonErr := json.Unmarshal(body, &res)
   if jsonErr != nil {
       logToErr.Fatalf("ERROR: -- Unmarshal -- %v \n", jsonErr)
   }

   if debug != false {
       logToOut.Fatalf("DEBUG: -- res -- %v \n", res.Services)
   }

   r := res.Services

   return r
}
```

In the unmarshal function we unmarshal the JSON and turn it in a slice of lvs.Service.

```go
func addServers(remoteAddr string) {
   body := httpGet(remoteAddr)
   jsonData := unmarshal(body)

   for i, v := range jsonData {
       if debug != false {
           logToOut.Printf("DEBUG: -- range jsonDATA --\n")
           logToOut.Printf("ipvsCount=%v, value=%v", i, v)
       }

       err := lvs.DefaultIpvs.AddService(v)
       if err != nil {
           logToErr.Printf("ERROR: -- AddService -- %v", err)
       }
 
       i++
       ipvsServerCount = float64(i)
   }
}
```

In the addServers function we add the servers to IPVS.

```go
func clientChan(c chan bool) {
   logToOut.Println("Starting time based IPVS Admin add")

   pollInterval := 10
   timerCh := time.Tick(time.Duration(pollInterval) * time.Second)
   // Time based loop to generate Global variable
   for range timerCh {
       select {
       // when shutdown is received we break
       case <-c:
           logToOut.Println("Received shutdown, stopping timer")
           break
       default:

           logToOut.Println("Clearing & Adding servers...")
           // Before we add Servers we need to clear the existing list
           lvs.Clear()
           addServers(remoteAddr)
           if debug != false {
               logToOut.Printf("IPVS servers added:\t%v", ipvsServerCount)
           }
       }
   }
}
```

Like we did in the IPVS server we create a go channel to poll every 10 seconds the server endpoint. We perform this to get at a set interval the IPVS entries.

Since we run the IPVS client as a binary directly on the Kubernetes hosts we build the binary with a few parameters we pass to the go build command. The binary we build with this command we host on an internal s3 bucket, we can download this binary with systemd unit files.

```bash
GOOS=linux
GOARCH=amd64
GIT_COMMIT=$(git rev-list -1 HEAD)

export GOOS
export GOARCH
export GIT_COMMIT

env GOOS=${GOOS} GOARCH=${GOARCH} go build -v -ldflags "-X main.gitCommit=${GIT_COMMIT}" .
```

When we run the IPVS client we can verify if the IPVS routes are added by running the `ipvsadm -L -n` command.

### Unit files

Since IPVS is part of the Linux kernel it is hard to deploy this in a docker container, the capabilities are more restricted in Kubernetes. We decided to deploy the IPVS client on each host machine through a systemd unit file, the main reason was that we ran into restrictions that slowed us down and this is not a permanent solution. By adding the IPVS client on the machines alone does not make it possible for containers to use the IPVS routes. We needed to add NET_ADMIN capabilities to all containers using the l4lb loadbalancer locations and configure `hostNetworking: true` in the Kubernetes pods.

We provided a deployment.yml file that runs a Ubuntu docker container with ipvsadm only installed extra. When the pods are deployed in this deployment you can use kubectl exec to get into the pod and run the `ipvsadm -L -n` command.

## Vacancy at Vandebron

We are looking for a platform engineer in Vandebron. As you can understand this is not a typical scenario we daily run across, but it is part of the workloads that we will support when working on our platform. Within Vandebron we try to use the best technology available, when it is not available we build it. Due to this as platform engineers, we have many interesting challenges and offer engineers to support further than only a strict domain. We support all components of our entire platform, regardless if it is a Linux kernel issue like this, involves setting up and maintaining a NoSQL cluster, or helping the business with something like requesting a certificate.

If you are interested in learning more about this position, take a look at our Vacancy and get in contact with us.
<https://werkenbij.vandebron.nl/>



Cypress is a game-changer in the automation testing world, the way that Cypress was built and its architecture allows us as testers to cover more scenarios.

Cypress is not Selenium; in fact, it is different. And the way to build and design a framework should be different as well.

The most famous design technique in Selenium is the Page Object Model, and many testers use the same design technique with Cypress. Even that Cypress on their official website [recommended](https://www.cypress.io/blog/2019/01/03/stop-using-page-objects-and-start-using-app-actions/) us not to go with that approach.

## Page Object Model

The main benefit of using the page object model Is to make the automation framework maintenance-friendly. We can define a specific page's selectors in a separate file and then use these selectors in our test cases.

```js
class SignInPage {
  visit() {
    cy.visit("/signin");
  }
  getEmailError() {
    return cy.get(`[data-testid=SignInEmailError]`);
  }
  getPasswordError() {
    return cy.get(`[data-testid=SignInPasswordError]`);
  }
  fillEmail(value) {
    const field = cy.get(`[data-testid=SignInEmailField]`);
    field.clear();
    field.type(value);
    return this;
  }
  fillPassword(value) {
    const field = cy.get(`[data-testid=SignInPasswordField]`);
    field.clear();
    field.type(value);
    return this;
  }
  submit() {
    const button = cy.get(`[data-testid=SignInSubmitButton]`);
    button.click();
  }
}
export default SignInPage;
```

The main two downsides using the typical page object model with cypress are:

- Page objects introduce an additional state into the tests, separate from the application’s internal state. This makes understanding the tests and failures harder.
- Page objects make tests slow because they force the tests to always go through the application user interface.

## Component-Based Architecture

On the other hand, a React application is component-based, where a specific page will be built from a collection of components. And components in React can be used on different pages too. So if we want to use the Page Object Model, we may define the same locator twice on different pages.

So having these two facts, At Vandebron, we came up with a new way to design our Cypress Automation framework by creating a separate JavaScript file for every component in our application, inside a folder called `components` within our Cypress project as below:

```js
// Locators
export const getEmailError = () => cy.get(`[data-testid=SignInEmailError]`);
export const getPasswordError = () =>
  cy.get(`[data-testid=SignInPasswordError]`);
export const emailField = () => cy.get(`[data-testid=SignInEmailField]`);
export const passwordField = () => cy.get(`[data-testid=SignInPasswordField]`);
export const submitButton = () => cy.get(`[data-testid=SignInSubmitButton]`);

// Actions
export const visit = () => cy.visit("/signin");
export const performLogin = (email, password) => {
  emailField().clear().type(email);
  passwordField().clear().type(password);
  submitButton().click();
};
```

Having it built this way, we eliminated all the previous problems mentioned earlier; we are not adding any classes, and we are defining objects within our test cases. And the most important part is that we are following the way that Cypress recommends it.

And after defining the component locators and actions, we can import them inside our test case and use them as below:

```js
import LoginComponent from "../components/loginComponent";
import Menu from "../components/Menu";

describe("Test Login Page", () => {
  it("should show an error message if the password in wrong", () => {
    LoginComponent.visit();
    LoginComponent.performLogin("email@gmail.com", "wrongPassword");
    LoginComponent.getPasswordError().should("be.visible");
  });
  it("should show the logout button if the user logged in succesfully", () => {
    LoginComponent.visit();
    LoginComponent.performLogin("email@gmail.com", "correctPassword");
    Menu.LogoutButton().should("be.visible");
  });
});
```

And as you can see, our test cases are readable for anyone! And if any locator changes in any of the components, we can easily fix it in one location and from the same file. And lastly, if a component will be used in different places, we can use the same code.

In the next article, I will talk about how we use Cypress in our manual testing during the sprint and how it saves us tons of time and effort.



In Vandebron we have been using container clusters to host our services since the foundation of our Big Data team. 
Recently our cluster of choice has declared End-Of-Life development stage, so we decided to take a step forward and get a ticket for the Kubernetes boat.

A change in the OS that is used to run your services and applications can look quite challenging and not everyone is on the same experience level. To make everyone comfortable it is a good choice to give everyone the possibility to play with the new tools and learn what can be done and how: **you need a sandbox.**

Our developers are provided with a Macbook and at the moment of writing there some options you can go for when deciding how to set up your playground:

- **Docker CE Kubernetes**: This is the easiest solution since there is a handy button to run your containers into a Kubernetes environment.

- **Vagrant and Virtualbox**: This solution is the one that can give you more control and you can easily create a cluster the size you want, but you need to be handy with VMs, the hypervisor of choice, and Vagrant. It's the old school way to do it but, while it's a chunk your platform engineers can bite, it can be a steep and frustrating process for people that are not used to handle VMs.

- **Multipass + some bash magic glue**: Since Canonical created this tool for macOS, creating an Ubuntu VM became a breeze and you can have a single, easily manageable VM with its networking up and running in less than a minute, without having to handle disks, distros, and stuff. On top of it, the command line interface is straight forward and it has just the basic commands we will need, so wrapping the entire process into a bash script is a piece of cake.

I have found this super cool in-depth [article](https://jyeee.medium.com/kubernetes-on-your-macos-laptop-with-multipass-k3s-and-rancher-2-4-6e9cbf013f58) from Jason Yee (kudos to you bruh) that guided me through the installation of my first single node Kubernetes cluster.

The process is not that long but it involves a lot of copy/pasting and, once learned the basics, I didn't want to go under the same process more times, plus it could be interesting for me as a Platform Engineer, but it may be boring and pointless for developers who just want to have a sandbox replica of what they are working on in the remote environment.
My automator (aka do-it-once-never-do-it-again) spirit kicked in and I decided to wrap every step in a small command-line tool with only 3 options:
- **install**
- **cleanup**
- **help**


### What is happening under the hood

What the script does is substantially automating all the steps needed to:
1. Create a new VM using Multipass (tool released by Canonical)
2. Fetch the VM IP address and adding it to your local `/etc/hosts` file
3. Install k3s (a lightweight distribution of Kubernetes) on top of the VM
4. Install the Kubernetes command-line tools on your laptop
5. Install Helm (the Kubernetes package manager) on your laptop
6. Install cert-manager (certificate manager) package on top of your k3s cluster
7. Install Rancher (a Kubernetes control plane) package on top of your k3s cluster

If you are looking for a more in-depth breakdown of the single steps you can download and inspect [the script](https://gist.githubusercontent.com/nikotrone/50b1a5f8d137411879eb2467e689bfbe/raw/090b4b4323d96ac28d96bbb346e2e657073722e6/bronernetes) (one of the many advantages of [OpenSource](https://en.wikipedia.org/wiki/Open_source) projects) or checkout and read the original [article](https://jyeee.medium.com/kubernetes-on-your-macos-laptop-with-multipass-k3s-and-rancher-2-4-6e9cbf013f58): it explains line by line what the specific commands are doing.

#### 1. Multipass VM
[Multipass](https://multipass.run/) is a tool from Canonical (the company developing and maintaining the Ubuntu Linux distribution) that leverages Hyperkit (macOS feature to handle virtualization) to create and handle a Virtual Machine directly on your Mac.

#### 2. Edit /etc/hosts
Once we have our VM up and running we need to make it available with an easy url that is also gonna be used to generate the SSL certificate, in our case we picked up `rancher.localdev`.
It is important to have a name setup in the beginning since this one will need to match with the certificate so we can use it programmatically.

#### 3. Install K3S
This step is pretty straightforward: just fetch a script that is publicly available on the [k3s official website](https://get.k3s.io) and feed it to your bash.
K3s is a lightweight version of Kubernetes with all the needed dependencies and executable packaged in a convenient installation script. Because of its light nature, it is often used in embedded devices that have a limited amount of resources to offer.

#### 4 & 5. Kubernetes and Helm cli
**Kubernetes cli** (`kubectl`) is used to talk and interact with your Kubernetes cluster. It can be used to manage multiple clusters according to the content of your KUBECONFIG environment variable. 
The variable itself contains just a path to where your cluster configuration is stored, so you can switch from a cluster to another by simply pointing to another file that contains the configuration of another cluster.

**Helm** instead is the "package manager" of Kubernetes: you can use it to add repositories to specific `charts` which are the blueprint that contains a way to install a specific tool on your cluster.
Both of these tools have to be installed and run from your local laptop, either in the case you are managing a local VM or in the case you are interacting with a remote cluster.

#### 6 & 7. cert-manager and Rancher

**Rancher** is the control plane for our cluster: it provides a GUI and an overview of our single node cluster. It offers other goodies like management of multiple clusters, deployed on different locations like AWS Azure and GCP or even on your own hardware, plus certificate deployment and some other handy functionalities.

**cert-manager** is installed via Helm chart and it is the tool used by Rancher to generate and deploy a certificate across the entire cluster.

### How to use it

All the steps will involve the use of a Terminal window
#### Installation
The first thing you need to do is download [this script](https://gist.github.com/nikotrone/50b1a5f8d137411879eb2467e689bfbe) and save it in a folder on your Mac (let's assume `~/bronernetes`) by executing
```bash
    mkdir ~/bronernetes
    cd ~/bronernetes
    curl https://gist.githubusercontent.com/nikotrone/50b1a5f8d137411879eb2467e689bfbe/raw/090b4b4323d96ac28d96bbb346e2e657073722e6/bronernetes > bronernetes
    export PATH=$PATH:$(pwd)
```

Now we have the toolset and you can confirm it works by simply running `bronernetes help`.

#### Spin up Kubernetes
The next step is to run the installation process with the command `bronernetes install`

#### Clean up
When you are done or you just want to hard reset your environment you can just type `bronernetes cleanup` and it will take care of cleaning up the VM you just used, leaving you with a pristine machine, as nothing ever happened :)

### Conclusion

Having a sandbox is very useful to play around with the concepts of a new setup or service and it packs up a huge amount of positive sides. No matter what is the language or the nature of the system you are trying to replicate, it can be challenging and involve a long list of instructions or manual operations and, sometimes, even dedicated hardware. Although with some bash glue, it is possible to automate most of those processes and the investment cost can be enormously beneficial for yourself (less work the next time you do it) and for the other people working with you (they can use the tool, comment and suggest improvements). Most of all, in the case of infrastructure, it helps raise the knowledge of "what's going on here" and documents for the ones interested in taking a trip down the rabbit hole.


At Vandebron we're maintaining a component library called [Windmolen](https://windmolen.netlify.app/) (Dutch for "wind turbine"). And if you've ever built a component library, you probably dealt with optimizing and converting icons before. With SVGO and SVGR you can do this at scale, without compromising the quality or size of your icons.

## The problem

The web is full of icons, and often these icons are rendered from SVG files to ensure you can increase (or decrease) the size of the icons depending on the use case. Designers often create these icons from design tools like Adobe Photoshop or Sketch. Although these icons might look pretty, exporting a SVG out of these tools is often difficult as [this article](https://medium.com/sketch-app-sources/the-best-way-to-export-an-svg-from-sketch-dd8c66bb6ef2) explains. Also, added lot of code in the form of metadata is added to the SVG file. Let's have a look at what a typical SVG file exported out of Sketch looks like:

```svg

<?xml version="1.0" encoding="UTF-8"?>
<svg width="14px" height="14px" viewBox="0 0 14 14" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
    
    <title>last</title>
    <desc>Created with Sketch.</desc>
    <defs></defs>
    <g id="Page-1" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
        <g id="last" transform="translate(2.000000, 0.000000)" fill-rule="nonzero" fill="#666666">
            <polygon id="Fill-2" points="6.6902923 9.6812703 9.3700469 7.0005052 6.6902923 4.3187297 2.37257308 0 0 2.37358354 4.3177192 6.6902923 4.6279322 7.0005052 4.3177192 7.3107182 0 11.6274269 2.37257308 14"></polygon>
        </g>
    </g>
</svg>
```

The SVG file above holds a lot of information about Sketch, such as the `title` of the icon and a `desc`ription. Next to that, there's a lot of elements that could be combined into one element to reduce the file size.

## Optimizing SVGs

What's cool about SVG files is that you can optimize and minify them, without affecting what the SVG looks like. This is something you can try out yourself using the website [SVGOMG](https://jakearchibald.github.io/svgomg/), which is powered by the library SVGO that you'll learn more about later.


You can optimize the SVG file above by following these steps:

1. Go to [https://jakearchibald.github.io/svgomg/](https://jakearchibald.github.io/svgomg/)
2. Click on `Paste markup` an paste the SVG code that you exported from Sketch (a.k.a. the SVG file above)
3. You will see the icon rendered, now you have to either click at the `Copy as a text` or `Download` button to get the optimized SVG file

With these simple steps you've optimized the SVG from over 450 bytes, which is already small, to 173 bytes (a decrease of over 62%!). If you'd open this file in the editor of your choice, you can see a lot of the useless (meta)data from the original file has been deleted. Also, the different elements of the SVG are combined in a single `path` that renders the icon:

```svg

<svg width="14" height="14" xmlns="http://www.w3.org/2000/svg">
  <path d="M8.69 9.681l2.68-2.68-2.68-2.682L4.373 0 2 2.374 6.318 6.69l.31.31-.31.31L2 11.628 4.373 14z" fill-rule="nonzero" fill="#666"/>
</svg>
```

This SVG can be even further optimized by checking the "Prefer viewbox to width/height" in SVGOMG, but let's save that for later when we use SVGO instead.

## Using SVGO

By using SVGOMG you've already experienced what power [SVGO](https://github.com/svg/svgo) has, as SVGOMG is described by its creators as *" SVGO's Missing GUI, aiming to expose the majority if not all the configuration options of SVGO"*. Instead of using the GUI, you can also use SVGO directly from the command line as a CLI-tool or as a Node.js module. For the sake of this article, we'll be using it solely as CLI.

SVGO can be installed globally on your machine, or locally in your project, from npm by running:

```bash
npm i -g svgo

# Yarn equivalent
yarn add -G svgo
```

After doing this you can run `svgo` from the command line and optimize any SVG file instantly. But, you don't want to do this manually on your machine anytime you're adding a new icon to a project (or component library). Therefore, you can also add SVGO to a project locally and add a script to the `package.json` file to optimize all SVGs in a certain directory.

```json
// package.json
{
 // ...
 "scripts": {
     // ...
    "optimize-svg": "svgo --config=.svgo.yml -f ./src/assets/icons"
 }
}
```

The `optimize-svg` script will run SVGO in the directory `src/assets/icons` and optimize all the SVG files based on the settings in `.svgo.yml`. This file is where you can configure the rules for SVGO, as the previously mentioned "Prefer viewbox to width/height":

```yaml
# .svgo.yml
plugins:
  - removeViewBox: false
  - removeDimensions: true # this deletes width/height and adds it to the viewBox
  - removeDoctype: true
  - removeComments: true
  - removeMetadata: true
  - removeEditorsNSData: true
  - cleanupIDs: true
  - removeRasterImages: true
  - removeUselessDefs: true
  - removeUnknownsAndDefaults: true
  - removeUselessStrokeAndFill: true
  - removeHiddenElems: true
  - removeEmptyText: true
  - removeEmptyAttrs: true
  - removeEmptyContainers: true
  - removeUnusedNS: true
  - removeDesc: true
  - prefixIds: false
  - prefixClassNames: false
```
   
From the rules above you'll get an idea about all the redundant and useless lines of code that might be present in your SVG files. But luckily, they will all get removed when you run the command `npm run optimize-svg`.

## Converting SVGs with SVGR

You've now learned how to optimize your SVG files, and are probably wondering how to use these files in a React application. To render an SVG in React, you need to either configure Webpack in a way that it knows how to deal with SVG files or use a library called SVGR. By default, any application created with `create-react-app` can render SVG files as a component, using the following `import` statement:

```jsx
// MyComponent.jsx
import React from 'react';
import { ReactComponent as MySVG } from './something.svg';

const MyComponent = () => {
  return (
    <div>
      <MySVG />
    </div>
  );
}
export default MyComponent;
```

More information about how this is done can be found in [this article](https://blog.logrocket.com/how-to-use-svgs-in-react/), but let me show you how to solve that with SVGR.

With [SVGR](https://react-svgr.com/) you can convert SVG files into React Components, either by adding it to Webpack or by using the SVGR CLI or Node.js module. In the same way, as we optimized the SVGs from the command line with SVGO, we can also convert these icons from the command line with SVGR:

```json
// package.json
{
 // ...
 "scripts": {
     // ...
    "optimize-svg": "svgo --config=.svgo.yml -f ./src/assets/icons",
    "convert-svg": "svgr -d ./src/components/Icon ./src/assets/icons"
 }
}
```

Whenever you run the command `npm run convert-svg` a JSX file will be created for every SVG file that's present in the directory `src/assets/icons`. These JSX files can be found in the directory `src/components/Icons`, together with an `index.js` file that exports all these components from this directory.

An example of such a converted SVG file is:


```jsx
// MySVG.jsx
import * as React from 'react';

const MySVG = (props) => (
  <svg viewBox="0 0 14 14" xmlns="http://www.w3.org/2000/svg" {...props}>
  <path d="M8.69 9.681l2.68-2.68-2.68-2.682L4.373 0 2 2.374 6.318 6.69l.31.31-.31.31L2 11.628 4.373 14z" fill-rule="nonzero" fill="#666"/>
  </svg>
);

export default MySVG;
```

And, as we now have a directory filled with converted SVGs these can be imported into any React component like this:

```jsx
// MyComponent.jsx
import React from 'react';
import MySVG from './MySVG.jsx';

const MyComponent = () => {
  return (
    <div>
      <MySVG />
    </div>
  );
}
export default MyComponent;
```

Often SVGR is used alongside SVGO, so you can even automatically optimize all SVGS that will be converted by SVGR. This is done by adding the flag `--no-svgo true` and point it towards your SVGO configuration file:

```json
// package.json
{
 // ...
 "scripts": {
     // ...
    "convert-svg": "svgr -d ./src/components/Icon ./src/assets/icons --no-svgo true --svgo-config .svgo.yml"
 }
}
```

By running the `convert-svg` script you both optimize and convert all the SVG files in `src/assets/icons` to React components based on optimized SVGs.

## Reading further

The examples in this post are the tip of the metaphorical iceberg on what problems SVGO and SVGR can solve. There are many other features you can enable, such as using them as Node.js modules or enabling TypeScript support. To read further make sure to have a look at the SVGR [playground](https://react-svgr.com/playground/) or [documentation](https://react-svgr.com/docs/getting-started/).


Here at Vandebron, we have several projects which need to compute large amounts of data. To achieve acceptable results, we had to choose a computing tool that should have helped us to build such algorithms.

As you may have read in other articles our main backend language is Scala so the natural choice to build distributed parallel algorithms was indeed Spark.

## What is Spark

We will briefly introduce Spark in the next few lines and then we will dive deep into some of its key concepts.

Spark is an ETL distributed tool. ETL are three phases that describe a general procedure for moving data from a source to a destination.

![ETL Diagram](/images/etlprocess.png "ETL")

- **_Extract_** is the act of retrieving data from a data source which could be a database or a file system.
- **_Transform_** is the core part of an algorithm. As you may know, functional programming is all about transformation. Whenever you write a block of code in Scala you go from an initial data structure to a resulting data structure, the same goes with Spark but the data structures you use are specific Spark structures we will describe later.
- **_Load_** is the final part. Here you need to save (load) the resulting data structure from the transformation phase to a data source. This can either be the same as the extract phase or a different one.
- **_Distributed_**: Spark is meant to be run in a cluster of nodes. Each node runs its own JVM and every Spark data structure can/should be distributed among all the nodes of the cluster (using serialization) to parallelize the computation.

### Spark data structure: RDD, DataFrame, and Dataset

The core of Spark is its _distributed resilient dataset (RDD)_.

![Spark API history](/images/sparkapihistory.png "Spark API history")

An **_RDD_** is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel. _Extracting_ data from a source creates an RDD. Operating on the RDD allows us to _transform_ the data. Writing the RDD _loads_ the data into the end target like a database for example). They are made to be distributed over the cluster to parallelize the computation.

A **_DataFrame_** is an abstraction on top of an RDD. It is the first attempt of Spark (2013) to organize the data inside and RDD with an SQL-like structure. With dataframe, you can actually make a transformation in an SQL fashion. Every element in a dataframe is a Row and you can actually transform a dataframe to another by adding or removing columns.

A **_DataSet_** finally is a further abstraction on top of a dataframe to organize data in an OO fashion (2015). Every element in a dataset is a case class and you can operate transformation in a scala fashion from a case class to another.

## Spark in action

Let’s see now some code samples from our codebase to illustrate in more detail each of the ETL phases.

### Extract

The extraction phase is the first step in which you gather the data from a datasource.

```scala
val allConnections = sparkSession
.read
.jdbc(connectionString, tableName, props)

val selectedConnections = allConnections
.select(ColumnNames.head, ColumnNames.tail: _*)

val p4Connections = selectedConnections
.filter(allConnections("HasP4Day activated").equalTo(1))
.filter(allConnections("HasP4INT activated").equalTo(1))
.as[Connection]

p4Connections.show()
```

For most people the extraction phase is just the first line (the invocation to the read method), they are not wrong because extracting means reading data from a datasource (in this case an SQL server database). I decided to include in this phase also some filtering and projection operations because I think these are not really part of the algorithm, this is still the preparation phase before you actually process the data. We can ultimately say that _preparing the data_ is something in between extraction and transformation therefore it is up to you to decide which phase it belongs to.

### Transform

Transformation phase is the core of the algorithm. Here you actually process your data to reach your final result.

```java scala
usageDF
.groupBy('ConnectionId, window('ReadingDate, "1 day"))
.agg(
    sum('Consumption).as("Consumption"),
    sum('OffPeak_consumption).as("OffPeak_consumption"),
    sum('Peak_consumption).as("Peak_consumption"),
    sum('Production).as("Production"),
    sum('OffPeak_production).as("OffPeak_production"),
    sum('Peak_production).as("Peak_production"),
    first('ReadingDate).as("ReadingDate"),
    first('marketsegment).as("marketsegment"),
    collect_set('Source).as("Sources"),
    collect_set('Tag).as("Tags"),
    max('Last_modified).as("Last_modified")
)
.withColumn(
    "Tag", when(array_contains('Tags, “Interpolated”),
lit(Tag.Interpolated.toString)).otherwise(lit(“Measured”)))
.withColumn("Source",
when(size('Sources) > 1,
lit(Source.Multiple.toString)).otherwise(mkString('Sources)))
.orderBy('ConnectionId, 'ReadingDate)
.drop("window", "sources", "tags")
```

In this specific example, we are processing connection usage data by aggregating it daily. In the `usageDF` we have 15 minutes interval usage data, now we want to show to the user the same data but with a different aggregation interval (1 day). So we group the whole data by connection id and window the reading date by 1 day (A window function calculates a return value for every input row of a table based on a group of rows [Introducing Window Functions in Spark SQL - The Databricks Blog](https://databricks.com/blog/2015/07/15/introducing-window-functions-in-spark-sql.html).

Once the data is grouped we can aggregate it, using the `agg` method which allows us to call the aggregation functions over the dataframe (for example: `sum`, `first`,`max` or `collect_set`). Successively we transform the dataframe to suit our visualization needs, the methods used are self-explanatory and the documentation is very clear. [Getting Started - Spark 3.0.1 Documentation](https://spark.apache.org/docs/latest/sql-getting-started.html)

### Load

The final phase is the one which `save`, `put`, `show` the transformed data into the target data source.

```java scala
dataFrame
.select(columns.head, columns.tail: _*)
.write
.cassandraFormat(tableName, keySpace)
.mode(saveMode)
.save()
```

In this specific case, we will save our dataframe into a Cassandra database. In Spark, methods used to achieve the load phase are called _actions_. It is very important to distinguish Spark actions from the rest because actions are the only ones that trigger Spark to actually perform the whole transformation chain you have defined previously.

If our transformation phase, as we described above, wasn’t followed by an action (for example `save`) nothing would have happened, the software would have simply terminated without doing anything.

## One concept to rule them all

```java scala
val rdd1 = sc.parallelize(1 to 10)
val rdd2 = sc.parallelize(11 to 20)
val rdd2Count = rdd1.map(
x => rdd2.values.count() * x //This will NEVER work!!!!
)
```

_One does not simply use RDD inside another RDD_. (Same goes for Dataframes or Datasets).

This is a very simple concept that leads very often to lots of questions because many people just want to use Spark as a normal scala library. But this is not possible due to the inner distributed nature of Spark and its data structures. We have said that an RDD is a resilient distributed dataset, let’s focus on the word _distributed_, it means that the data inside it is spread across the nodes of the cluster. Every node has its own JVM and it is called _Executor_, except for the master node where your program starts which is called _Driver_:

![Spark cluster overview](/images/spark-cluster-overview.png "Spark cluster overview")

Your code starts from the Driver and a copy is distributed to all executors, this also means that each executor needs to have the same working environment of the Driver, for Scala it is not a problem since it just needs a JVM to run. (but we will see that if you use _pySpark_ you need to take extra care when you distribute your application.) Every Spark data structure you have defined in your code will also be distributed across the executors and every time you perform a transformation it will be performed to each chunk of data in each executor.

Now let’s go back to our example, a `map` is a transformation on `rdd1` this means that block inside will be executed at the executor level, if we need `rdd2` to perform this block Spark should somehow serialize the whole `rdd2` and send it to each executor. You can understand now that _it is really not possible to serialize the whole RDD since it is by its nature already a distributed data structure_. So what can you do to actually perform such computation we showed in the example? The solution is “simple”: _prepare your data in such a way that it will be contained in one single RDD_. To do so you can take advantage of all the transformation functions Spark has to offer such `map` `join` `union` `reduce` etc.

## Next step…

We have explained all the main concepts of Spark and we have shown some real snippets of our codebase. In the next article, I would like to show you a real-life problem we have solved in our company using [_pySpark_](https://spark.apache.org/docs/latest/api/python/index.html). I will show you how to customize Spark infrastructure to correctly parallelize the ETL algorithm you have built.



Two months ago, I started my journey at Vandebron. One of the projects I first dove into was their efforts to build a [component library](https://windmolen.netlify.app/). Something I was already familiar with from previous companies I worked at. 

On the internet, you can find many articles that describe why a reusable component library is a good investment for your development team(s). Although there's much to say about the advantages of component libraries, most articles don't state the (obvious) disadvantages such projects can have. In this post, I'll point out some of our learnings and why you might not need such a reusable component library.

## About component libraries

Often you find yourself repeating the same lines of code to make, for example, a button or the layout of a page look nice, especially when you're working on multiple projects. Or as a designer, you get frustrated every time the styling for a part of the application is off when a new page or project is created. Many companies have already found multiple solutions to preventing themselves from repeating styling, which is the main reason for design inconsistencies. And therefore component libraries were created.

A component library is a collection of all the styled parts (or components) of a website or multiple websites that make it easier for developers to reuse these parts. Also, designers will know for sure that all components in the component library adhere to their designs, and therefore all projects that use these components will conform. Often these libraries consist of different layers of components, for example, offering atoms, molecules, and organisms when an [Atomic Design](https://bradfrost.com/blog/post/atomic-web-design/) pattern is applied. Following this pattern, developers can use the parts to style their templates and pages consistently.

Component libraries are becoming more and more popular with the rise of JavaScript libraries and frameworks like React and Vue. These technologies are very suitable for quickly building interactive components that you can use in your application, and can easily be exposed as a library on NPM or Github Packages. At Vandebron, we're building all our web and mobile applications with React and React Native and are using [Storybook](https://storybook.js.org/) to develop our components in a shared library between the engineering and design teams. This can potentially create a lot of advantages for both the developers and designers, as you can read below.

## Why you *might* need a component library

Before deciding to create a component library for your team or company, you probably want to hear about the advantages such a project can lead to. The main advantages of component libraries are briefly mentioned in the first section above and are often defined as:

- **Reducing code duplication**: With a component library, you can create components that can be shared across multiple websites or applications. This way you no longer have to duplicate styling in different projects. This can seriously decrease the amount of code duplication that you have in your projects, also reducing the number of bugs or design inconsistencies.

- **Preventing design inconsistencies**: By adding all your components and styled parts to the component library you're certain that these will look the same on all the places they're used. Not only will all the components look the same on every page, when designers make a change to one of these components they can be easily updated on all the places they're used.

- **Easier collaborating**: Component libraries make it easier for developers and designers to collaborate on applications and designs, with the component library as the common "playground". By using a tool, like Storybook, you can also make this playground visible to non-technical people and show what components are already available to use for new features.

But these advantages come at a certain price, as I'll explain in the next section.

## Disadvantages of component libraries

Besides the obvious advantages of a component library, it can also have serious disadvantages that are listed below. Whether or not these disadvantages apply to you depends on numerous things that are discussed later on in this article.

- **Increasing complexity**: With all attempts to make code more generic,  an increased level of complexity also comes to play. Reusable components should be easy to extend or customize, which requires you to think about the different use cases beforehand or force you to add many different variations to a component. With every new project that starts to use the component library, you get the risk of increasing the complexity of the library even more.

- **Time-consuming**: Every time you want to add a component to your project, you need to create that component in the component library first and import it locally in the project to test it. Therefore you need to be working in multiple projects at the same time, which requires you to set up a more time-consuming workflow. Also, when you want to use this new component from the library, you have to publish a new version of the library to make the component available.

- **Conflicting dependencies**: When you're using different versions of dependencies across your projects and the component library, you're forced to sync those with each other. Imagine having, for example, an older version of React running in one of your projects that doesn't use a recent React API that you want to use in your component library. In this scenario, you either have to update that project or are unable to keep your component library on par with the latest release of your dependency on React. Both solutions have pros and cons, and would rather be avoided.

As mentioned before, there are reasons why these disadvantages might apply to you that are the team size, the number of teams and projects at the company, development or release lifecycles, and how your source code is organized. It clearly doesn't make sense to invest in a component library if you have just a small amount of people work on just one project, or a sole team is working on all the different projects making it easier to manage code duplication or design inconsistencies.

## Considerations before starting

There are two main alternatives that you need to take into consideration before building a reusable component library, which is (obviously) using or extending an existing component library or sourcing your code in a monorepo. 

- **Existing component libraries:** Using an existing component library is an efficient way to create consistently (web) pages and reduce the amount of complexity of your own project, while also taking advantage of best practices of large open-source projects. Popular examples of component libraries are [Ant Design For React](https://ant.design/docs/react/introduce) or [various implementations](https://material.io/develop) for Google's Material Design. These libraries allow you to move quickly without having all the overhead of creating complex components but limit you to the design guidelines of these component libraries.

- **Monorepo:** If you don't want to take advantage of existing libraries or are very keen to apply your own styling to components across multiple applications without having to copy-paste the code, you can host the source code of applications in a monorepo. With the monorepo approach, you can create a shared folder that includes all the components used by your applications. This makes it possible to apply changes with a simple pull request and import these components from every project in that repository.

Besides these two alternatives, you also need to have proper design guidelines set by your designer(s). When the design guidelines are flexible and fluctuating, you could be structuring components incorrectly with the risk of doing a lot of work that will be omitted once the project evolves.

## To summarize

Component libraries are a great way to reduce the amount of code duplication in your applications, prevent design inconsistencies, and increase collaborations between developers, designers, and different teams. But this comes with increased complexity, slower development cycles, and possible code conflicts between projects. Therefore you should consider if using an existing component library or having a monorepo for your source code is a workable solution. At Vandebron we decided to build our own component library (called [windmolen](https://windmolen.netlify.app/)) and if you'd decide the same, then be sure that your design guidelines are properly structured and mature enough.


Vandebron Engineering & Data

Latest posts

How Vandebron helps balancing the Dutch energy grid together with OnLogic & Talos Linux

The Why and How of Dagster User Code Deployment Automation

The difference between a component library and a design system, and how they can help bridge the gap between design and development

Signing and verifying SOAP messages with wss4j and Scala

Looking back at the Vandebron GreenTech Hackathon 2021

The power of regular hackathons

Migrating from DCOS to Kubernetes, dealing with the l4lb loadbalancer

Cypress.io Component Design Technique for React Applications

How to Spin Up A Kubernetes Cluster On Your Macbook

Optimizing, Converting And Exporting SVG Icons In React

Fueling the Energy Transition With Spark - Part 1

Building native images and compiling with GraalVM and sbt

When (Not) To Build A Reusable Component Library

Links

Vandebron on Github

Windmolen

Work with us