Data Mesh: Building an “Amazon.com” for your Data Products

How to improve the code quality of your dbt models with unit tests and TDD

<p><img src="/images/posts/manchester-office.jpeg" alt="Thoughtworks office views" title="Thoughtworks office views" /></p>
<p>Nine years ago I moved to Manchester to start my journey as a junior software engineer. To mark my ninth work anniversary, I sat down with a nice cup of <a href="https://imgflip.com/i/2h15z3">barista coffee</a> and <strong>gathered my thoughts on what I think are the most important lessons I learned along the way</strong>.</p>
<p>These are the seven things I think <strong>every junior software engineer</strong> starting their career in 2023 should start considering doing <strong>to create a long and fulfilling career in tech</strong>.</p>
<h2>1. Optimize for learning over money</h2>
<blockquote>
<p>Not all experience is created equal</p>
</blockquote>
<p><strong>Not all experience is created equal. Doesn't matter how many years of experience you have. What matters is what you were doing all those years and with whom.</strong> Treat being in the right learning environment a priority for your first couple of jobs. At the beginning of your journey, learning and growing during your working hours is far more important than money.</p>
<p>The knowledge you will gather during the first years of your career will compound and it will allow you to get higher paying jobs in the long term. Of course, you should only do this if you don't need money for paying off debt or supporting a family.</p>
<p><strong>The best career decision I made by far was leaving my comfort zone, move abroad and start my professional journey in a great working environment</strong>. Working with a group of international and passionate people that cared about their craft expedited my learning more than anything else. With remote working, you can now find these type of places without having to leave your home.</p>
<p><strong>Being in a great learning environment during working hours is only the first step. Don't forget to study and train a bit each day.</strong> That daily hour of study will also compound fast over the first years.</p>
<h2>2. Focus on principles and practices</h2>
<p>Invest in learning the principles and practices behind modern software engineering. <strong>Tools and technologies come and go, the principles and practices of our craft last longer.</strong></p>
<p>I was lucky to be in an environment where my colleagues emphasized the importance of good practices over tools. Knowing practices like clean code, refactoring, automated testing, and test-driven development <strong>will make your code a pleasure to work with</strong>.</p>
<h2>3. Be the engineer that everyone wants to work with</h2>
<blockquote>
<p>The degree to which other people want to work with you is a direct indication of how successful you’ll be in your career as an engineer. Be the engineer that everyone wants to work with. - John Allspaw</p>
</blockquote>
<p><strong>Making your code a pleasure to work with is important but that is only half of the job</strong>. Don't become the engineer that nobody wants to work with. Collaborate with others, help that colleague that is stuck, listen to other people's ideas even if they are not your own, and try <strong>be the kind of person everybody wants to be in a team with</strong>.</p>
<h2>4. Keep the balance</h2>
<blockquote>
<p>You cannot be really first rate at your work if your work is all you are. – Anna Quindlen</p>
</blockquote>
<p>Try to adopt the habits and rituals that allow you to have a decent work-life balance if you want to have a long and fulfilling career. <strong>Investing additional time early in your career, like that extra hour studying every day I mentioned, doesn't mean that you should forget other areas of your life</strong> like your health and relationships.</p>
<p><strong>To avoid burning out, you should balance times where you invest more in your career and others where you can just relax and just do your 9 to 5 shift</strong>. These are some of the rules I set:</p>
<ul>
<li>No work related stuff on the weekends</li>
<li>Disconnect from the computer in the evenings</li>
<li>Remove any work related stuff from my smartphone</li>
<li>Reduce my screen time after work</li>
<li>Find hobbies that get me out there, like traveling and MTB</li>
</ul>
<p>Of course, there are periods where we break these rules and put more time into career related projects or self-learning. <strong>That is totally fine as long as you are aware and you are willing to make that compromise for what you are getting out of the time you invest on them.</strong></p>
<h2>5. Feedback is gold, mine it!</h2>
<blockquote>
<p>We all need people who will give us feedback. That’s how we improve. - Bill Gates</p>
</blockquote>
<p>Constantly ask for feedback. <strong>Feedback, especially constructive, will help you identify your blind spots and get to know yourself better, faster. Feedback will boost your growth more than any other thing.</strong></p>
<p>I am lucky to work for a company with a well established feedback culture. I treat feedback as gold. I consolidate and curate the feedback I received in digital notes. <strong>This feedback allows me to spot trends, reflect, and write down some actions I want to take to become a better professional</strong>. I set a reminder every month to go over this list of actions to check if I am improving.</p>
<p>If you are not asking for feedback, start now! Here are <a href="https://www.thoughtworks.com/en-es/insights/blog/feed-feedback">good tips to get started</a>.</p>
<h2>6. Go broad first</h2>
<blockquote>
<p>To be a successful professional these days, I believe that one should go broad first (master the 20% that will allow you to deliver 80% of the tasks) and go in-depth only when there is a big need, a big opportunity, or a strong interest in keeping developing your skills further in a particular area. - Thiago Ghisi</p>
</blockquote>
<p>Don't just stop at the tech and the tools. Learn about product management, agile and lean ideas, the business domain of the problem you are trying to solve and how to work with people and lead a team. Be a product engineer and <a href="https://www.svpg.com/the-most-important-thing/">question what you are helping to build</a>.</p>
<p><strong>My most productive learning experiences came from leaving my comfort zone and pushing myself to wear other hats.</strong> The better you understand the other roles in a team, the better you will be able to collaborate with them and contribute on building a high-performing team.</p>
<p><strong>There are always opportunities to explore and cross the frontiers of your current role.</strong> You can volunteer to do the exploratory testing of a new piece of work, work with the operations team to build observability into your service, or pair with the product manager of your team to understand why the users are not liking the latest feature you shipped.</p>
<p><strong>This can be a bit of a controversial advice, but I believe the best engineers I worked with were well-rounded.</strong> They understood the end-to-end software development process, the product and the business problem they were trying to solve. They were not afraid to speak up and suggest improvements in areas not related with their main role.</p>
<p>Becoming a <a href="https://www.agilemodeling.com/essays/generalizingSpecialists.htm">generalizing specialist</a> allowed me to make an impact quicker when starting a new project from scrach or jumping into an existing one. <strong>Going broad and learning how to be well-rounded is the second best super-skill you can spend time on after learning how to learn.</strong></p>
<h2>7. Share your journey</h2>
<blockquote>
<p>Become a documentarian of what you do. - Austin Kleon</p>
</blockquote>
<p><strong>Build a habit to document, curate, and <a href="https://austinkleon.com/show-your-work/">share your work</a> early in your journey.</strong></p>
<p>My biggest regret is not starting to capture and share what I learned earlier. Don't become the engineer who only consumes, contribute back!. <strong>Write that comment or appreciation in an article you like, publish that article that you were thinking of writing for a year, and sign up for that talk you are so afraid to give.</strong> There is always someone that can find your current challenges and ideas useful.</p>
<p>Writing your ideas and learnings down and sharing them with other people will allow you to get feedback faster and develop a better understanding of a topic. I really like <a href="https://fortelabs.com/blog/just-in-time-pm-4-intermediate-packets/">Thiago Forte's idea of building and sharing &quot;intermediate packets&quot;</a>.</p>
<h2>Conclusion</h2>
<p>These are the lessons that have resonated the most with me over the last nine years. I hope you find them useful to set the foundations for a long and fulfilling career as a software engineer.</p>
<p><strong>What's the most important lesson you've learned since starting in tech as a junior engineer?</strong> Feel free to ping me at <a href="https://twitter.com/portovep">@portovep</a> and share it with all of us.</p>


Seven lessons I wish I knew as a junior software engineer

<p><img src="/images/posts/advent-security-2022.jpeg" alt="Photo by Ales Nesetril on Unsplash" title="Photo by Ales Nesetril on Unsplash" /></p>
<p>At the end of each year I complete what I like to call &quot;an end of the year security review&quot;. The idea is to follow a set of small steps to review and improve my online security and privacy setup.</p>
<p>This year I consolidated all the steps in a checklist and I decided to share it in case you want to use the list as a guide to review your current setup and stay safer on the Internet. Each of the steps I propose are easy to follow and can be completed by anyone with basic digital skills so you could even send the checklist to any of your friends and loved ones.</p>
<p>This is a TL;DR; checklist with the steps we will cover:</p>
<ul>
<li><input type="checkbox" disabled="" /> Use a password manager</li>
<li><input type="checkbox" disabled="" /> Use a multifactor authentication app (MFA)</li>
<li><input type="checkbox" disabled="" /> Enable automatic software updates on your devices</li>
<li><input type="checkbox" disabled="" /> Review your browser extensions</li>
<li><input type="checkbox" disabled="" /> Create a list of your key digital assets</li>
<li><input type="checkbox" disabled="" /> Enable two factor authentication for your key digital assets</li>
<li><input type="checkbox" disabled="" /> Enable storage encryption for your physical devices</li>
<li><input type="checkbox" disabled="" /> Reduce the amount of permissions granted to the apps you use to the minimum</li>
</ul>
<p>Let's begin!</p>
<h2>Use a password manager</h2>
<p>Let's start with the basics. You probably use dozens of digital services and apps. If you are like me, remembering the password of each of them is mission impossible. Using the same password is not an option either. So, the first step is to install a password manager if you don´t already.</p>
<p>If you already have one, the action for us is to improve how we use it by:</p>
<ul>
<li>Making sure we can access our password manager from multiple devices. We don't want to lose all our most important digital assets, passwords and credentials if we lost the only device that allows us access.</li>
<li>If you use the same password manager for both personal and work credentials, make sure you store your work passwords in a <a href="https://www.youtube.com/watch?v=i8esgZ7Kyrg">different folder or vault</a>.</li>
<li>Making sure the password manager password is secure by 2022 password standards. That password is the gate to all your identities on the Internet.</li>
</ul>
<p>I use <a href="https://1password.com/">1Password</a> and I am very happy with it. But there are other <a href="https://cybernews.com/best-password-managers/free-password-managers/">free options out there</a>.</p>
<h2>Use a multifactor authentication app (MFA)</h2>
<p>A password manager is the first step towards a better digital experience. But if we want peace of mind, we should use a multi-factor authentication service. So, step two is to install one on your smartphone.</p>
<p>If you already use an MFA service, the action for us is to improve how we use it by:</p>
<ul>
<li>Making sure we can access the MFA services from at least two devices. This ensures that if we lose our phone, we can still access accounts with MFA enabled.</li>
<li>If the MFA service uses a backup password or recovery code, let's ensure we store it in a secure place and, of course, we remember where.</li>
<li>Let's perform a security drill: let's leave our phones in another room and go to our sofa with our laptops. First, open an incognito window. Second, check if we can still access the digital services where MFA is enabled. Did you have to leave your sofa and go to the other room to grab your phone? If the answer is yes, then congratulations on a successful MFA setup.&quot;</li>
</ul>
<h2>Enable automatic software updates on your devices</h2>
<p>This is a simple one but is easy to forget. Make sure your devices (smartphone, tablet, laptop, smart TV...) have automatic software updates enabled. Automatic updates will ensure potential security patches are installed as soon as they are made available. Here is a typical checklist:</p>
<ul>
<li>Enable automatic operating system updates on MacOS, Windows, Linux, Android, and iOS.</li>
<li>Enable automatic application updates in your desktop, tablet, and Android app stores.</li>
<li>For the advanced users, make sure your OS package manager (e.g. brew in mac) has the option to <a href="https://github.com/Homebrew/homebrew-autoupdate">automatically update packages enabled</a>.</li>
</ul>
<h2>Review your browser extensions</h2>
<p>Browser extensions like ad-blockers improve our experience while browsing the web but they are also a potential attack vector that can compromise our privacy.</p>
<ul>
<li>Make sure you reduce the number of browser extensions to the minimum. For each extension, ask yourself if you use it often enough that it deserves to clutter and slow down your browser. If the answer is no, just installed the extension every time you need it.</li>
<li>Once you get down to the minimum viable extensions, make sure they are up to date and come from trusted sources. I usually look at things like the number of reviews, number of installations, if it is featured by the browser company...</li>
</ul>
<h2>Create a list of your key digital assets</h2>
<p>A key digital asset is an asset that if it gets compromised we will be panicking and running around with our hands in our heads. So, these assets are worth taking extra care of. By creating a list of your digital assets you know how to prioritise what to protect. Here are some of mine. I divide them into two categories:</p>
<p><strong>Physical</strong>:</p>
<ul>
<li>Work laptop</li>
<li>Personal laptop</li>
<li>Smartphone</li>
<li>Portable SSD with personal documents</li>
</ul>
<p><strong>Digital</strong>:</p>
<ul>
<li>Password manager service</li>
<li>Phones numbers</li>
<li>MFA or second-factor authentication service</li>
<li>Personal email accounts
<ul>
<li>I bet that we don't want anyone to be able to access the thousands of personal emails you forgot to delete over the years</li>
</ul>
</li>
<li>Work email account</li>
<li>Cloud file storage services like GDrive, OneDrive, and iCloud</li>
<li>Social accounts like Instagram, Facebook, WhatsApp, Twitter...
<ul>
<li>I bet we don't want anyone to hijack our digital identity</li>
</ul>
</li>
<li>Other important online accounts like bank accounts, Github, etc...</li>
</ul>
<h2>Enable two factor authentication for your key digital assets</h2>
<p>Adding two factor authentication is going to add an extra layer of security to the key digital assets identified in the previous step. For each item, we are going to:</p>
<ul>
<li>Enable a second factor authentication method using the services we configured during day two.</li>
<li>If the online service has a phone number associated with it, make sure is up to date and it is a phone number that you have access to.</li>
<li>For online services with mobile apps (like bank accounts, email, etc), make sure those apps are protected with an additional PIN or biometric authentication mechanism. Some smarphones allow to add this extra layer of security even if the native app doesn't support it.</li>
</ul>
<h2>Enable storage encryption for your physical devices</h2>
<p>Next step is to review if the storage encryption option is enabled in our devices. Storage encryption ensures that our <a href="https://securitystudio.com/data-at-rest-vs-data-in-transit/">data at rest is protected</a> even if our devices get lost or stolen. We can use the list of physical assets created in day five to prioritize the devices that need to be protected.</p>
<p>Some devices already ship with this option enabled, for example my phone running on Android 12 comes with it enabled by default. Other devices like portable SSD disks may ship with third party software that allows you to encrypt them.</p>
<p>In some cases, like when you use FileVault on MacOS, you will have to generate a recovery code. You can save this code in a secure place like your password manager.</p>
<h2>Reduce the number of permissions granted to the apps you use</h2>
<p>Review the app privacy settings for each of our devices to reduce the level of access we give to the apps we have installed is a simple action that will improve our privacy online and will give us control over our data.</p>
<p>We should pay attention to the permissions that grant apps access to the private data we value the most. Some examples are:</p>
<ul>
<li>Our location</li>
<li>Our contact list</li>
<li>Our devices inputs like camera and microphone</li>
<li>Screen recording and input monitoring</li>
</ul>
<h2>Nex steps</h2>
<p>You made it until here, congratulations!. Hope you found it useful. As a next step, I recommend you to do like me and create a reminder in your calendar to schedule your next security review.</p>
<p>If you have any suggestion for additional steps you think can be useful please let me know so I can add them to the list :)</p>


How to stay safe online in eight steps

<h2>Intro</h2>
<p>A few years ago I transitioned to the role of data engineer at work. I started focusing on helping clients build their data management and data platform capabilities. The transition involved a lot of learning and moving outside my comfort zone.</p>
<p>I usually learn better by doing. During this transition, I always felt that I was missing a place where I could put all that I read and learned together, so I started thinking...</p>
<ul>
<li>Wouldn't be awesome if I could have a data engineering playground where I can keep learning trying new tools and developing new data skills?? Similar to the other two playgrounds I created to learn <a href="https://github.com/portovep/pabloporto.me">frontend development</a> and <a href="https://github.com/portovep/continuous-delivery-playground">continuous delivery</a>.</li>
</ul>
<ul>
<li>Wouldn't be even more awesome to build my own data platform where I could store data that is important to me and run analytics over it to get valuable insights in several areas of my life?</li>
</ul>
<p>I created a Notion page and I started drafting what I would like to build and what it may look like. This article is a more advanced and polished version of that page.</p>
<h2>Who is this guide for?</h2>
<ul>
<li>You are familiar with basic data engineering concepts and you want to have a place to put them into practice.</li>
<li>You are starting to learn more about data engineering and want to understand what are the main components of a modern data platform.</li>
<li>You want to learn modern data engineering by applying it to your use cases.</li>
<li>You want to have a place to store all the important data from your digital life and where you can run analytics on top.</li>
</ul>
<p>The idea is to create a playground for anyone looking to improve their data engineering knowledge and build something useful at the same time.</p>
<h2>What we will learn?</h2>
<p>In this series, I will show you how I build my data platform by implementing what is called the <a href="https://continual.ai/post/the-future-of-the-modern-data-stack">modern data stack</a>.</p>
<blockquote>
<p>The Modern Data Stack commonly refers to a collection of technologies that comprise a cloud-native data platform, generally leveraged to reduce the complexity in running a traditional data platform. The goal is to make data actionable by reducing the time it takes for data to become useful to data workers in an organization.</p>
</blockquote>
<p>We’ll go through all the main features you will need to build your modern data platform:</p>
<ul>
<li>Data ingestion</li>
<li>Data modeling and transformation</li>
<li>Data storage</li>
<li>Data visualization and what BI (Personal Intelligence in this case)</li>
<li>Data orchestration</li>
<li>Data quality and observability</li>
<li>Data discovery</li>
</ul>
<p>(DIAGRAM) and explain what I consider an operational source.</p>
<h2>What we will build?</h2>
<p>The goal is to build a data platform that allows us to analyze our data from the apps and tools we use every day. The platform should allow us to easily ingest data coming from new data sources, perform analytical queries against it and create dashboards that will help us make better life decisions in several areas of life: finance, health, etc.</p>
<p>The idea of having a place where you can track and store all your digital data is not new. The <a href="https://en.wikipedia.org/wiki/Quantified_self">quantified self movement</a> is several years old and is trying to achieve just that.</p>
<blockquote>
<p>The <strong>quantified self</strong> refers both to the cultural phenomenon of self-tracking with technology and to a community of users and makers of <a href="https://en.wikipedia.org/wiki/Self-tracking" title="Self-tracking">self-tracking</a> tools who share an interest in &quot;self-knowledge through numbers&quot;</p>
</blockquote>
<p>But you would be asking yourself, couldn't I do most of what you are describing with a simple spreadsheet? Maybe..., but what would be the fun and the learning process then?</p>
<h3>Use cases</h3>
<p>We will be guiding the implementation of the platform by value. What does this mean? It means that instead of building all the platform capabilities up front, we will be building them in thin slices and just what we need to fulfill a given use case.</p>
<p>You can think of a use case as a specific feature or insight that the platform will allow us to get. Some of the initial use cases I have in mind are:</p>
<p><strong>Health domain</strong></p>
<ul>
<li>Track and visualize important health metrics like weight, body mass, and blood pressure in one place. Be able to see how they evolved.</li>
<li>Understand how many workouts and training sessions I performed over the year.</li>
<li>How many kilometers do I cycle per year?</li>
</ul>
<p><strong>Finance domain</strong></p>
<ul>
<li>Track my 4 key financial metrics. The ones I describe in <a href="/blog/the-four-key-metrics-of-personal-finance">this post</a>.</li>
<li>Visualize how my wealth evolved since I began investing.</li>
<li>Automatically ingest payslips so I can track my monthly income.</li>
</ul>
<p><strong>Culture domain</strong></p>
<ul>
<li>Understand how many books and pages I read per year</li>
<li>How many movies do I watch in a given year. How many times did I go to the cinema?</li>
</ul>
<p><strong>Share &amp; contribute domain</strong></p>
<ul>
<li>How many times have I shared something on Twitter?</li>
<li>How many times have I shared a photo of my travels on Instagram?</li>
</ul>
<p>These are some initial ideas. The key here is that the platform should be giving you enough flexibility to implement any analytical use case you have in mind to extract insights from your data.</p>
<h3>Architecture</h3>
<p>These are some of the architectural principles that should guide us as we choose the different tools and technologies that will allow us to build the platform:</p>
<ul>
<li>Based on the modern data stack and in an extract-load-transform (ELT) architecture.</li>
<li>New data source connectors can be created and integrated easily.</li>
<li>The platform can be deployed fully on our personal computers.</li>
<li>The platform can be deployed on the cloud cheaply and with a low environmental impact.</li>
<li>Data privacy and security are first-class concerns.</li>
<li>All tools should be open-source software.</li>
<li>The infrastructure components should be deployed in a serverless offering if possible.</li>
</ul>
<h3>Candidate tech stack</h3>
<ul>
<li>Data ingestion
<ul>
<li>Lambdas</li>
<li>Airbyte</li>
</ul>
</li>
<li>Data storage and processing
<ul>
<li>Postgres Serverless</li>
</ul>
</li>
<li>Data transformation
<ul>
<li>dbt and SQL</li>
</ul>
</li>
<li>Data orchestration
<ul>
<li>Github Actions</li>
</ul>
</li>
<li>Data discovery
<ul>
<li>OpenMetadata ora</li>
<li>Datahub</li>
</ul>
</li>
<li>Data observability and quality
<ul>
<li>Metaplane</li>
</ul>
</li>
<li>Data visualization
<ul>
<li>Apache Superset</li>
<li>Appsmith or Retool</li>
<li>Tremor</li>
</ul>
</li>
<li>Continuous delivery
<ul>
<li>Github Actions</li>
</ul>
</li>
</ul>
<p>If you have any suggestions for better tools for any of the capabilities please let me know!</p>
<h2>I am interested, how do I follow along?</h2>
<p>When I was a kid, I used to go to the local newsstand or paper store and always ask my parents to buy me one of the new installments (fasciculus) that they sell to build train models and other toys like Scalextric.</p>
<p>They forced you to subscribe so you could build the train model or toy in installments. My idea is to follow the same approach and use a newsletter to keep you posted on the project and release the different installments that will allow us to build our modern data platform.</p>
<p>If this sounds interesting to you and I invite you to follow along and subscribe to the newsletter where I will be sharing the progress as I build this thing.</p>
<p>Next installment: Setting up the local development environment.</p>
<p>(Link to repo)</p>


Build your own modern data platform and learn data engineering

<p><img src="/images/posts/money-minimalist.jpeg" alt="Photo by Towfiqu barbhuiya on Unsplash" title="Photo by Towfiqu barbhuiya on Unsplash" /></p>
<p>The book <strong><em>Accelerate: The Science of Lean Software and DevOps</em></strong> introduced the concept of the <a href="https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance">four key metrics</a> for measuring the software delivery performance of technology organizations. Since then, many software development teams have used these metrics as a north star to understand if they are improving their software delivery capability.</p>
<p>As a software engineer I have used these metrics in several projects to analyze potential areas of improvement in our ability to delivery quality software fast. During the last year, I asked myself if I could apply the concept of the 4 key metrics to several areas of my life, including finance. What would be the 4 key metrics for measuring the performance of my personal finances? How could I track them?</p>
<p>In this post I will describe the 4 metrics I choose and the thinking process that let me to pick them.</p>
<h2>Motivation</h2>
<p>Over the last few years, I became more interested in personal finances and began to educate myself on the topic. I like to apply a lean mindset to managing my finances as I don't enjoy spending too many hours tracking expenses and money in and out of my accounts. Since the beginning, I want to have a system that gives me a high-level view of my financial health and performance.</p>
<p>My long-term goal is to create a system that allows me to achieve <a href="https://www.investopedia.com/articles/personal-finance/112015/these-10-habits-will-help-you-reach-financial-freedom.asp">financial freedom</a>. This term could have several meanings, freedom means different things to different people. In my case, I understand financial freedom as a state where money doesn't condition my lifestyle decisions.</p>
<p>Achieving financial freedom doesn't necesarialy mean that I would like to stop working and retire to an isolated island. Still I want to have the flexibility and the privilege to not having to worry too much about money and do the things I like, e.g. continue traveling the world.</p>
<p><img src="/images/posts/money-naval-tweet.png" alt="4 key metrics of personal finance" title="4 key metrics of personal finance" /></p>
<h2>The selection criteria</h2>
<p>I started by reflecting on all the things I learned about personal finances over the years. Mostly focusing on things people always give importance to, like paying off your debt, trying to save every month, having an emergency fund, etc.</p>
<p>I already had a tracking system in place based on a spreadsheet so I went through the metrics I was tracking and I started prioritizing them based on their impact on my long-term financial goals. I also differentiated the ones that are within my control, e.g. spending, from the ones that are not, for example, investment returns as I don't find very useful to track things that are not partially under my control.</p>
<h2>The metrics</h2>
<p>After completing this process, I arrived at the following 4 metrics that I consider key to understanding my financial health:</p>
<ul>
<li><strong>Average Monthly Saving Rate</strong>: Percentage of my total monthly income that I am able to save measured as an average over the last 12 months.</li>
<li><strong>Average Monthly Investment Rate</strong>: Percentage of the total monthly income that I invest measured as an average over the last 12 months. This metrics cover the money that I invest in index funds and other assets.</li>
<li><strong>Total Debt</strong>: Amount of money I owned. In general, prioritizing paying off debts above investing is a good idea. Although there are cases where the opposite is also true.</li>
<li><strong>Emergency Fund Runway</strong>: Number of months with covered basic lifestyle expenses. In other words, how many months I could live out of my saved and easily accessible money? My goal is to have a runaway of 6 months or more based on my average spending per month.</li>
</ul>
<p>I believe all of these metrics are great leading indicators of how I am doing financially. If we take the categorization from the original 4 key metrics by <a href="https://www.devops-research.com/research.html">DORA</a> as a reference, we could also split the metrics into two main categories:</p>
<ul>
<li><strong>Financial performance</strong>: They help you to understand if you are improving the chances to increase your total wealth over time. By increasing your saving and investment rates you increase the likelihood of becoming wealthier over time.</li>
<li><strong>Financial stability</strong>: They help you to understand how financially resilient you are if something unexpected happens, e.g. if you lose your job. Having several months of runway in your emergency fund and no debt to repay will allow you to overcome financial challenges more easily.</li>
</ul>
<blockquote>
<p>Disclaimer: I am not a financial advisor. This post is not meant to be used as financial advice. The metrics that work for me and my current situation may not work for everybody. I encourage you to take them only as a reference and come up with your own.</p>
</blockquote>
<h2>Next steps</h2>
<p>Tracking these metrics already triggered possitive results. They help me reflect on my finances and make adjustments if needed. They give me a lean and practical way to manage my wealth creation process.</p>
<p>I review my finances and manually update the data required to calculate these metrics monthly. Similar to what we do in software delivery teams, I would like to start automating the process of gathering the data to calculate them automatically. I have a few ideas on how to do it. I may try to cover some of them in a future post.</p>
<p>I want to show you a sneak peak of what I am building. The data displayed is not real.</p>
<p><img src="/images/posts/money-four-key-metrics-personal-finance.png" alt="4 key metrics of personal finance" title="4 key metrics of personal finance" /></p>
<p>And you, do you agree with these metrics? What are the financial metrics you track? Feel free to reach out by email or on Twitter. I am always willing to learn more about this topic and improve my current system.</p>


The four key metrics of personal finance

<p>Once in a while, I get tired of the design of my
website or its technical stack and start tinkering around. Having a personal website as a pet project has helped me to stay up to date with modern front-end development.</p>
<p><img src="/images/posts/website-version-5-framed.png" alt="Version 5.0 of the website" title="Version 5.0 of the website" /></p>
<p>Over the last year, I have been working on a full redesign and rewrite of this website. What is now version 5 started as an attempt to learn new technologies, Next.js and TaildwindCSS. I thought this milestone would be a good opportunity to reflect on the different rewrites and redesigns I have done over the years. Thus, I decided to jump into the <a href="https://web.archive.org/">Wayback Machine</a> and write what I like to call a retrospective changelog. Here it goes.</p>
<h2>Changelog</h2>
<h3>Version 1.0 - 2008</h3>
<p><img src="/images/posts/website-version-1-framed.png" alt="Version 1.0 of the website" title="Version 1.0 of the website" /></p>
<p>In the mid 2000s blogging was already very popular. I wanted a quick way to start writing and sharing my ideas so I decided to build a blog.</p>
<p>The setup I went for was quite simple. I choose one of the most known CMS, WordPress, and I hosted the site in one of the popular VPS-based hosting providers of the time. I tried to find a more representative screenshoot of the site using Wayback Machine but I failed.</p>
<table>
<thead>
<tr>
<th>Purpose</th>
<th>Tool</th>
</tr>
</thead>
<tbody>
<tr>
<td>Core</td>
<td>Wordpress</td>
</tr>
<tr>
<td>Content</td>
<td>Wordpress</td>
</tr>
<tr>
<td>Styles</td>
<td>Plain CSS</td>
</tr>
<tr>
<td>Hosting</td>
<td>VPS</td>
</tr>
<tr>
<td>Features</td>
<td>Blog</td>
</tr>
</tbody>
</table>
<h3>Version 2.0 - October 2013</h3>
<p><img src="/images/posts/website-version-2-framed.png" alt="Version 2.0 of the website" title="Version 2.0 of the website" /></p>
<p>In September 2013, I finished my master's in Networks and distributed systems. I took a few months to prepare for job interviews and look for my first job as a software engineer. I had quite a lot of free time so I decided to create a portfolio from scratch to share my CV.</p>
<p>Inspired by <a href="https://www.allthingsdistributed.com/2011/08/jekyll-amazon-s3.html">this post</a> by Amazon CTO Werner Vogels. I choose <a href="https://jamstack.org/generators/cactus/">Cactus</a> as a Python static site generator. I used AWS S3 for serverless hosting as I was starting to learn about cloud technologies back then.</p>
<table>
<thead>
<tr>
<th>Purpose</th>
<th>Tool</th>
</tr>
</thead>
<tbody>
<tr>
<td>Core</td>
<td>Cactus, Python</td>
</tr>
<tr>
<td>Content</td>
<td>HTML, Django template system</td>
</tr>
<tr>
<td>Styles</td>
<td>Bootstrap, Fontawesome</td>
</tr>
<tr>
<td>Hosting</td>
<td>AWS S3</td>
</tr>
<tr>
<td>Features</td>
<td>Portfolio, Online CV</td>
</tr>
</tbody>
</table>
<h3>Version 3.0 - December 2014</h3>
<p>For version 3, I decided to migrate the tech stack to host the website for free on GitHub Pages. I replaced Cactus for <a href="https://jamstack.org/generators/jekyll/">Jekyll</a> as a static site generator. The fact that I was working with Ruby at work at the time influenced my choice.</p>
<p>I also migrated the content from HTML to YAML. I honestly don't remember why I didn't choose another more text-friendly format like Markdown.</p>
<table>
<thead>
<tr>
<th>Purpose</th>
<th>Tool</th>
</tr>
</thead>
<tbody>
<tr>
<td>Core</td>
<td>Jekyll, Ruby</td>
</tr>
<tr>
<td>Content</td>
<td>YAML files</td>
</tr>
<tr>
<td>Styles</td>
<td>Bootstrap, Fontawesome</td>
</tr>
<tr>
<td>Hosting</td>
<td>Github Pages</td>
</tr>
<tr>
<td>Features</td>
<td>Portfolio, Online CV</td>
</tr>
</tbody>
</table>
<h3>Version 4.0 - July 2017</h3>
<p>Version 4 was another tech stack migration. The most time-consuming and complex to date. I was learning React and Redux and I wanted to have a pet project to practice those technologies. I started by creating a web application that offered the same functionality as the Jekyll site.</p>
<p>I managed to learn a lot about creating single-page applications or SPA. I was reading the book <a href="https://www.goodreads.com/en/book/show/35496817">Atomic Design</a> and working with <a href="https://storybook.js.org/">Storybook</a> at work so I also used this version of the website to learn how to create my own design system and a component library. This is another example of how using the site as playground helped me to stay up to date and develop new skills.</p>
<p>From an SEO perspective, it was a terrible decision. Moving from a <a href="https://jamstack.org/">Jamstack</a> architecture to <a href="https://developer.mozilla.org/en-US/docs/Glossary/SPA">SPA</a> made it hard for the SEO crawlers to index my content.</p>
<table>
<thead>
<tr>
<th>Purpose</th>
<th>Tool</th>
</tr>
</thead>
<tbody>
<tr>
<td>Core</td>
<td>React, Redux, Storybook</td>
</tr>
<tr>
<td>Content</td>
<td>JSON</td>
</tr>
<tr>
<td>Styles</td>
<td>Bootstrap, Fontawesome</td>
</tr>
<tr>
<td>Hosting</td>
<td>Github Pages</td>
</tr>
<tr>
<td>Features</td>
<td>Portfolio, Online CV</td>
</tr>
</tbody>
</table>
<h3>Version 5.0 - March 2021</h3>
<p><img src="/images/posts/website-version-5-about-framed.png" alt="Version 5.0 of the website: Making page" title="Version 5.0 of the website: Making page" /></p>
<p>Version 5 is the one you are probably seeing right now. I decided to go back to the Jamstack architectural style of versions 2 and 3 but keeping React and JavaScript as the core technologies.</p>
<p>I choose <a href="https://jamstack.org/generators/next/">NextJS</a> as a static site generator and React framework. I was curious about modern react frameworks so I decided to get some experience with them. The same happened with <a href="https://tailwindcss.com/">TailwindCSS</a>. The design was a bit old so I took the opportunity to do a complete re-design and learn about <a href="https://medium.com/@sascha.wolff/utility-first-css-ridiculously-fast-front-end-development-for-almost-every-design-503130d8fefc">utility-first CSS frameworks</a>.</p>
<p>One of the goals of this version was to build a site that allowed me to go back to blogging. I wanted a workflow that allowed me to quickly create blog posts or notes and share them on the site. I migrate the content to Markdown so I can write and edit the content in minimalist tools like <a href="https://obsidian.md/">Obsidian</a> and easily copy and paste it into the codebase.</p>
<p>I also change the infrastructure. From GitHub Pages to <a href="https://vercel.com/">Vercel</a>. I was quite impressed by the developer experience of Vercel as a platform. I
really recommend to give a try to anyone looking for a simple hosting solution.</p>
<p>I didn't move away from GitHub completely. When I started doing this rewrite, I was quite excited about GitHub Actions. I went to create a continuous deployment pipeline for the blog based on a <a href="https://www.atlassian.com/continuous-delivery/continuous-integration/trunk-based-development">trunk-based development</a> workflow. I don't create pull requests for new changes (except for the ones created by Dependabot). I push directly to master and I trust the automated quality gates I created in the pipeline to do their job. You can checkout the pipeline in action <a href="https://github.com/portovep/pabloporto.me/actions/runs/3392341840">here</a>.</p>
<table>
<thead>
<tr>
<th>Purpose</th>
<th>Tool</th>
</tr>
</thead>
<tbody>
<tr>
<td>Core</td>
<td>NextJS, React, TypeScript</td>
</tr>
<tr>
<td>Content</td>
<td>Markdown</td>
</tr>
<tr>
<td>Styles</td>
<td>TailwindCSS</td>
</tr>
<tr>
<td>Hosting</td>
<td>Vercel</td>
</tr>
<tr>
<td>CI/CD</td>
<td>Github Actions</td>
</tr>
<tr>
<td>Features</td>
<td>Blog, Portfolio, Online CV, Travel tracker</td>
</tr>
</tbody>
</table>
<h2>Take aways</h2>
<p>As you can see, this website has changed quite a lot over the years. Writing this post allowed me to reflect on how much time I put into making these changes both in the design and the code. Was it worth it? Could I spend that time on something more productive like writing more or building an app? Maybe.</p>
<p>I still think the main ROI of these efforts has been the ability to have a playground to learn and develop new front-end skills that I could apply in my day-to-day job. I have many examples of technologies I was able to understand better after applying them to my website: CSS, SaSS, TypeScript, Storybook, and Next.js. TailwindCSS, Atomic Design. GitHub Actions...</p>
<p>I am always checking other people's websites <a href="https://austinkleon.com/steal/">to get inspiration</a>. Do you have one you want to share? Reach out to me on <a href="https://twitter.com/portovep">Twitter</a>. Finally, I want to share some of the personal websites that inspired me to improve mine and write this.</p>
<ul>
<li><a href="https://brianlovin.com/writing/how-my-website-works?utm_source=pocket_reader">brianlovin.com</a></li>
<li><a href="https://kentcdodds.com/blog/how-i-built-a-modern-website-in-2021">kentcdodds.com</a></li>
<li><a href="https://maggieappleton.com/garden-history?utm_source=pocket_reader">maggieappleton.com</a></li>
<li><a href="https://www.taniarascia.com/redesign-version-6/">taniarascia.com</a></li>
</ul>


Redesign and new stack: Version 5.0

<p>Every three years or so my work laptop gets renewed. Every time I have to set it up from strach.
This year I took the opportunity to look at my current setup with the pareto's principle in mind. What are the 20% of the tools and configurations that I leverage or use 80% of the time? In other words, what are the minimum vital few things I should setup to stay productive as a professional software developer.</p>
<h2>Getting Started</h2>
<p>If you are starting from scratch like me, I recommend starting by installing all the latest OS level updates. The first step is to install and upgrade the xcode command line tools.</p>
<pre><code class="language-bash">xcode-select --install
</code></pre>
<h2>The package manager</h2>
<p>One of the frist tools we need is a package manager. For macOS I recommend brew. You can install it with a simple command.</p>
<pre><code class="language-bash">/bin/bash -c &quot;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)&quot;
</code></pre>
<p>Finally, update brew and make sure everything is working fine with the following commands.</p>
<pre><code class="language-bash">brew update
brew doctor
</code></pre>
<h2>Security</h2>
<p>Following the principle of <a href="https://snyk.io/learn/shift-left-security/">shifting left security</a>, lets first configure some sensible defaults to keep our Mac protected.</p>
<ul>
<li>Enable file encryption with FileVault.</li>
<li>Enable the firewall.</li>
<li>Setup a strong password for your OS user and enable touchid if required.</li>
<li>Install your password manager of choise so we can login into services and apps going forward. I use and recommend <a href="https://1password.com/">1Password</a>.</li>
<li>Install any pending security updates.</li>
<li>Enable automatic updates for security patches and apps installed via the App Store.</li>
</ul>
<h3>Setting up automatic updates for brew packages</h3>
<p>Brew has a feature to enable automatic updates called <a href="https://github.com/Homebrew/homebrew-autoupdate">autoupdate</a>. The tool will run brew update in the background once every 24 hours. You can customize the time interval.</p>
<pre><code class="language-bash"># Set auto-updates every 7 days
brew autoupdate start --upgrade 604800
</code></pre>
<h2>The terminal</h2>
<p>As developers, we spend most of the time either in the IDE, the terminal or a browser. There are two important choices to make regarding our terminal setup: the shell and the terminal app you use. Over the years I defaulted to ZSH for the shell and Hyper for the app.</p>
<h3>Setting up Zsh</h3>
<p>The newest versions of macOS come with Zsh installed. On top of that, I manage my Zsh configuration with <a href="https://ohmyz.sh/">OhMyZsh</a>.</p>
<pre><code class="language-bash">sh -c &quot;$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)&quot;
</code></pre>
<p>and I setup the following plugins and theme in <code>.zshrc</code></p>
<pre><code class="language-bash">ZSH_THEME=&quot;gallois&quot;

plugins=(
  git
  autojump
  vagrant
)
</code></pre>
<h3>Setting up Hyper</h3>
<pre><code class="language-bash">brew install --cask hyper
</code></pre>
<p>Next step is to edit the configuration. I increase the font size and leave everything else as default.</p>
<pre><code class="language-bash">vim ~/Library/Application Support/Hyper/.hyper.js
</code></pre>
<h2>The code editor or IDE</h2>
<p>Coming soon..</p>
<h2>The version control system</h2>
<p>Git has become the standard for version control systems.</p>
<p>Setup your global Git config</p>
<pre><code class="language-bash">vim ~/.gitconfig
</code></pre>
<p>This is how mine looks like</p>
<pre><code class="language-bash">[user]
name = &quot;Pablo&quot;
email = &quot;personal email&quot;

[alias]
st = status
lol = log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)&lt;%an&gt;%Creset' --abbrev-commit

[core]
excludesfile = ~/.gitignore_global

[pull]
rebase = false

[includeIf &quot;gitdir:~/Dev/work/&quot;]
path = ~/.gitconfig-work
</code></pre>
<p>I also configure a globa .gitignore.</p>
<pre><code class="language-bash"># ~/.gitignore_global

# OS generated files #
######################
.DS_Store
.DS_Store?

# IDEs #
######################
.idea
</code></pre>
<p>If you use your laptop for both personal and work projects, you can isolate your work config in a separate file. The statement <code>includeIf</code> allows you to specific a different config file that will only apply in the directory &amp; subdirectories set with <code>gitdir</code>. This is an example of my separate file for work.</p>
<pre><code class="language-bash">[user]
   name = &quot;Pablo@work&quot;
   email = &quot;work@email.com&quot;
[core]
   sshCommand = &quot;ssh -i ~/.ssh/work.ssh.key&quot;
</code></pre>
<h2>SSH</h2>
<p>Create a ssh config to simplify how you connect to remote machines (and how you pull code from Github &amp; co).</p>
<pre><code class="language-bash">vim ~/.ssh/config
</code></pre>
<pre><code class="language-bash">Include ssh_config_work

Host github.com
	IdentityFile ~/.ssh/personal_private_key

Host bitbucket.org
	IdentityFile ~/.ssh/personal_private_key
</code></pre>
<p>As we did for the Git config, you can isolate your work config into a separate file using <code>Include</code>. This is how my work ssh config looks like.</p>
<pre><code class="language-bash">#Work

Host github.com-work
	HostName github.com
	AddKeysToAgent yes
	UseKeychain yes
	IdentityFile ~/.ssh/work_private_key

Host work-machine.training
    User ec2-user

</code></pre>
<h2>Runtime version manager</h2>
<p>Dealing with multiple versions of language runtimes is painful. When you work on different projects, you will need to switch versions of the different runtimes: node, java, etc. Installing version managers for each runtime will make your life easier.</p>
<h3>Node</h3>
<p>Node Version Manager or <a href="https://github.com/nvm-sh/nvm">NVM</a> is my preferred choice to manage active Node.js versions.</p>
<pre><code class="language-bash">brew install nvm
</code></pre>
<p>After installing nvm we need to add the following piece of configuration in our ZSH config file.</p>
<pre><code class="language-bash"># ~/.zshrc

export NVM_DIR=&quot;$HOME/.nvm&quot;
  [ -s &quot;/opt/homebrew/opt/nvm/nvm.sh&quot; ] &amp;&amp; \. &quot;/opt/homebrew/opt/nvm/nvm.sh&quot;  # This loads nvm
</code></pre>
<h3>Java</h3>
<p>Coming soon..</p>
<h2>Docker as container runtime</h2>
<p>Since Docker Desktop change its licensing in mid 2021, several alternatives to run Docker in macOS popped up. My favourite so far is <a href="https://github.com/abiosoft/colima">Colima</a>.</p>
<p>With the following steps we can have a fully working Colima-based Docker setup in our Mac:</p>
<pre><code class="language-bash">brew install colima
brew install docker
brew install docker-compose
</code></pre>
<h2>MacOS configuration</h2>
<p>Coming soon..</p>
<h2>Productivity apps</h2>
<p>Other apps that I use to stay productive.</p>
<table>
<thead>
<tr>
<th>Application</th>
<th>Purporse</th>
</tr>
</thead>
<tbody>
<tr>
<td>Rectangle</td>
<td>Window manager</td>
</tr>
<tr>
<td>Flux</td>
<td>Screen eye protection</td>
</tr>
<tr>
<td>1password</td>
<td>Password manager</td>
</tr>
<tr>
<td>Mural</td>
<td>Online whiteboarding</td>
</tr>
<tr>
<td>Figma</td>
<td>Prototyping</td>
</tr>
<tr>
<td>Todoist</td>
<td>Task manager</td>
</tr>
<tr>
<td>Notion</td>
<td>Knowledge base / digital garden</td>
</tr>
<tr>
<td>Spotify</td>
<td>Music</td>
</tr>
<tr>
<td>Chrome</td>
<td>Browser</td>
</tr>
<tr>
<td>Slack</td>
<td>Communication and communities</td>
</tr>
<tr>
<td>AppCleaner</td>
<td>Managing MacOS apps not installed by brew</td>
</tr>
<tr>
<td>Steam</td>
<td>Games</td>
</tr>
<tr>
<td>Obsidian</td>
<td>Secure notes</td>
</tr>
</tbody>
</table>
<p>If you are curious what are my default choices for building frontend, backend and data applications you can check <a href="/stack">my prefered tech stack</a>.</p>
<h2>Conclusions</h2>
<p>This is my current minimal macOS developer setup. I will keep updating this post as I finish setting up my laptop. Stay tuned!</p>


Minimal macOS development setup for 2022

<p>Around a year ago, my team started the journey of modernizing the frontend stack of a back-office web application. The application helps assessors to manage the lifecycle of a student loan.</p>
<p>The web application is part of a system of 30 microservices and interacts with almost all of them. Three cross-functional teams work on the frontend application. Each team is in charge of the UI that interacts with the microservices containing the business logic for their specific domain.</p>
<p>As the teams grew bigger, there was the need to modernize the frontend tech stack into a set of technologies that allowed us to deliver business value faster while providing a consistent user experience.</p>
<p>As we are in the middle of the journey, I wanted to reflect on some of the things I learned.</p>
<h2>The old stack(s)</h2>
<p>At the time the team started thinking about modernizing the stack, the codebase was three years old. The application was built using jQuery and vanilla JavaScript. RequireJS was used as a module loader. At some point in the life of the application, there was an attempt to create individual modules using Knockout, perhaps trying to move towards a micro-frontends pattern.</p>
<p>The tests were written using Jasmine and ran with Karma. The coverage was good as the team was doing TDD.</p>
<p>The team decided to perform a spike to assess React and Redux as the foundation for the new stack. The technical goal was to modernize the stack to increase maintainability and allow new developers from the different teams to start contributing fast.</p>
<h2>Knowing your libraries inside-in</h2>
<p>The web app digitalizes the process of managing the lifecycle of a student loan application. The users are the assessors, responsible to take a paper loan request form and enter all the details into the system so that the system can assess the application. As you can imagine, the UI has a high amount of forms. This led the team to evaluate using a library that abstracts the complexity of managing controlled form inputs in React.</p>
<p>We started using redux-forms without knowing the library well and we pay a high price. In our favor, the library has evolved a lot since we started using and it knows accommodates more different use cases than before.</p>
<h2>Investing in improving the developer experience early on</h2>
<p>One of the key decisions we made was to invest in improving the developer's experience early on in our migration journey. We migrate from Mocha and expect.js to Jest, which includes test doubles, coverage, and assertions out of the box. This simplified the number of libraries the developer had to get used to.</p>
<p>We changed the way we were serving our integrated React app. We were using an old gulp setup but we wanted to benefit from features like hot reloading so we quickly invest some time setting up the web server provided by Webpack.</p>
<p>The old codebase used JSHint as the linter utility. When we start using ESLint for our React code, we end up having two different linter tools for the three different stacks. We invest in migrating the JSHint rules to ESLint to simplify and only have one. The same happened with Karma and Gulp.</p>
<p>In the technical conversations we were having about the process of strangling the old stacks, we referred to them as the legacy stack. As the time passed, we realized that the modernization of the codebase was going to take a while, or even stop at some point like it happened when Knockout was introduced.</p>
<p>We starting to invest time also in improving areas of the code implemented with the old stacks. The first thing we did was re-arrange the folder structure so that the three stacks had clear boundaries and the code was structured similarly. This allows us to have a clear view of which part of the application was built with each stack.</p>
<h2>Bringing the teams up to speed</h2>
<p>By the time we started the migration, most of our team were backend developers with a few developers who had experience with Javascript. We knew a key aspect of the modernization success was to bring people up to speed quickly and enable them to start sharing as much knowledge as possible.</p>
<p>The team started to organize weekly ES6 katas using the Exercism platform. A few of us gathered at 4 pm and split into pairs. We then try to complete the kata suggested for the day, after 50 minutes we gathered again and we shared and discuss all the solutions. This helped a lot to get people comfortable with the new features introduced in ES6 and to some degree to get people excited about the new tech.</p>
<p>Like most projects in ThoughtWorks, we do pair programming when we work on any piece of code that will eventually be deployed to production. This helped a lot. People with a few weeks on the code base will be able to teach people new to React how to get started and what worked for them. They will also help them to navigate the code base and share the patterns we were trying to follow, like the container vs presentation components pattern.</p>
<p>We also organized a mobbing session where the whole team tried to solve a problem together.</p>
<h2>Creating a reusable UI component library</h2>
<p>One of the key aspects of the modernization effort was to get everybody thinking about components, mostly about consistency and reusability.</p>
<p>As we were rewriting the different modules and creating new ones, we spotted and remove some inconsistencies in the UI. This was a joint effort between the BAs, QAs, Dev, and UX people in the team.</p>
<p>It was not enough with having the developers splitting the UI functionality into components. We wanted the business analysts and UX designers to be aware of the components that already existed and how they were called, eventually creating a ubiquitous language based on components.</p>
<p>Some things we did were having the business analysts referencing existing components in the user stories and even collaborating with UX and developers to set the names and purposes for new ones.</p>
<p>Developing components and isolation and discussing how to test them with the QAs allowed us to create smaller stories and built bigger pieces of the UI in incremental steps. The caveat of this is that some stories didn't provide user value directly, they were technical tasks that helped us modernize the application incrementally.</p>
<h3>Storybook</h3>
<p>We started thinking early on how we could visualize our library of components that at the time it was growing bigger and bigger.</p>
<p>We started researching if any libraries allow us to render the component directly from the source code. We found Storybook and we decided to give it a try.</p>
<p>We quickly realized the enormous value this tool brought, not only allowing us to collaborate with QAs, BAs, UX, and PO, but also in testing our components.</p>
<p>We started writing stories to implement individual components and mapping the different acceptance criteria scenarios to stories in the storybook instance of the component. This helped us during desk check to evaluate if the user story was complete, and also the QA to go through the different scenarios and validate when a component was ready.</p>
<h2>Lessons learned</h2>
<p>We are still in the middle of the journey, but these are some of the lessons we learnt along the way:</p>
<ul>
<li>Setting boundaries between different stacks helps to isolate the parts of the code that need to be replaced.</li>
<li>Having tests for the legacy parts help to understand the code and made the rewrite faster.</li>
<li>Rewrites are hard. Make sure you plan for them in advance and bring the business/product stakeholders along.</li>
<li>Invest in improving the developer experience to work with the different stacks.</li>
<li>Don't forget about the human aspect. Different people take different times to get up to speed with new technology, provide spaces to learn, and bring the whole team on a journey.</li>
<li>Creating a common vocabulary with designers and business people to name components fosters adoption across teams.</li>
<li>Tools like Storybook allow for the creation of reusable components and make it easy to maintain component libraries.</li>
</ul>


Modernising a frontend application with React, Redux and Storybook

<p>I recently came across <a href="https://increment.com/development/what-its-like-to-be-a-developer-at/">a really interesting article</a> published on Increment, a digital magazine that shares ideas on how effective software teams work together. The author asks a set of questions to software developers and managers from what can be considered great product companies in the software industry about how is it to be a developer in their teams.</p>
<p>I don’t work for a product company but I thought it would be interesting to answer the same set of questions so that I can see how my current experience in an agile software development team may differ from the ones in the original article.</p>
<p>I work as a software consultant for a company called <a href="https://www.thoughtworks.com/">ThoughtWorks</a>. As part of my job, I usually get involved in the delivery of at least a couple of software products every year, mostly by advising our clients to improve the ways they engineer those products and pairing with their engineering teams to build them.</p>
<p>I am currently working for a government organisation helping to evolve a loan assessment platform. Although all the answers reflect my experience as a developer in this specific project, I think they can be applied to most of the projects I have been involved with recently.</p>
<h2>What are the most common tools that developers use?</h2>
<p>Most of the developers use IntelliJ as IDE for JavaScript, Clojure and Java. We really like the integration with Jest to run individual tests with a simple keyboard shortcut. We use the <a href="https://cursive-ide.com/">Cursive</a> plugin to deal with Clojure’s parenthesis in a productive way. While the IDE is not enforced, having the same one with a shared setup proves more efficient for pair programming.</p>
<p>Git and Gitlab are used for source control, combined with <a href="https://github.com/saki/devkit">DevKit</a> to manage GitLab identities while pair programming. <a href="https://www.thoughtworks.com/mingle/">Mingle</a> is used for both project management and as a knowledge base. We use HipChat as IM and IBM Notes as email solution.</p>
<p>We run our development environment in a Vagrant VM running CentOS (RHEL in production) on a Windows host. Developers use <a href="https://www.gocd.org/">GoCD</a> for CI/CD, Kibana (ELK) for logging, Graphite for metrics and Icinga for monitoring and alerting.</p>
<h2>Which languages do developers code in?</h2>
<p>Most of our backend (micro)services are written in Clojure, with some written in Java 8. We have around 25 in total. New services are usually created in Clojure. We are able to spin up a new one quite quickly by combining automation techniques like a <a href="https://www.thoughtworks.com/radar/techniques/tailored-service-template">service template</a> for <a href="https://leiningen.org/">Leiningen</a> that includes all the common libraries and <a href="https://www.thoughtworks.com/radar/techniques/pipelines-as-code">pipeline as code</a> for GoCD.</p>
<p>For the back office web application we are slowly refactoring a 4 years old single page application with React/Redux using ES6 JavaScript. This frontend was built with a mix of vanilla JavaScript, jQuery and Knockout . We develop new functionality features using the new stack.</p>
<h2>What is the development process like? (the lifecycle of a piece of committed code)</h2>
<p>The requirements for a new product or feature usually come from Government’s policy. These requirements ultimately get split into user stories and assigned to a cross-functional team. The developers help the business analysts (BAs) and quality analysts (QAs) in the team to review the story from a technical perspective. When <a href="https://www.agilealliance.org/glossary/three-amigos/#q=~(filters~(postType~(~'page~'post~'aa_book~'aa_event_session~'aa_experience_report~'aa_glossary~'aa_research_paper~'aa_video)~tags~(~'three*20amigos))~searchTerm~'~sort~false~sortDirection~'asc~page~1)">the three amigos</a> think that the story is ready to be played, it is moved to the next stage in our physical (and digital) Kanban wall, “ready for dev”.</p>
<p>After that, a pair of developers start working on the story. We swap pairs every two or three days in order to share context as much as possible. The pair that started the story may not be the same that completes it, although one of the people usually sticks with the story till completion. When the story is ready, it is desk-checked by the three amigos again. The QA then starts analysing the quality of the functionality built through exploratory testing and at the same time assessing test coverage and cross-functional requirements.</p>
<p>When the QA finishes and the product owner (PO) is available, the story is showcased to the PO to get it signed off, authorising it to be released. If the PO is happy, the story is moved into ready for release, usually being released the next day.</p>
<h2>What is code review like?</h2>
<p>We don’t have many code reviews. We work on the principle that all production code should be delivered by doing pair programming. Developers pair with their peers and always push changes to master (<a href="https://paulhammant.com/2013/04/05/what-is-trunk-based-development/">trunk-based development</a>). We do use additional branches to explore technical solutions, like assessing a new library or a new design.</p>
<p>Sometimes, the tech lead or feature leads review other people code <strong>after they pushed to master</strong> to ensure the technical direction is kept and new changes do not break or impact the design of a service or the system as a whole.</p>
<h2>How is testing done, and what kind of tests are run?</h2>
<p>Developers write unit (we follow TDD), integration, service/functional level tests. They also help the QAs to write automated end to end tests that exercise the whole system, some of them through the UI. We also have <a href="https://storybook.js.org/testing/structural-testing/">snapshot tests</a> for our reusable React components (thanks to Storybook.js and Jest) and some contract tests between microservices and external services using <a href="https://docs.pact.io/">Pact</a>.</p>
<p>In addition to the local development environment, all these tests run in the build and deployment pipelines of our continuous delivery pipeline. The QAs also perform some exploratory testing in a staging environment, mostly through the UI and sometimes through Swagger, the tool we use to document our REST APIs.</p>
<h2>How is code deployed?</h2>
<p>We push to master all the time, which means that each individual commit must be able to be deployed in production without breaking functionality or must be under a <a href="https://martinfowler.com/articles/feature-toggles.html">feature flag</a>. Each commit has a story number and the names of the pair that authored it, that is how we track each piece of work.</p>
<p>As an example, every time a commit of the assessment service is pushed to master, the service build pipeline gets triggered. A GoCD agent builds the service and runs the automated test suite, among other things like linters. If the process completes successfully, a new version of the assessment service is automatically deployed to the first staging environment where end to end tests exercise the system.</p>
<p>Once a user story involving this service is completed and the QA and the PO are happy with the functionality to be deployed, the new version of the assessment service is scheduled to be released in the next release window given by the change management (CM) department of the company. This department has to authorise all the releases we do and they use the story numbers to track and understand the changes being deployed to production.</p>
<p>Given this constrain, quite common in organisations like this, we usually deploy to production only once a day. In most cases this deployment involves not only one, but several services.</p>
<p>When the deployment is authorised, the pair organising promotes the new version of the assessment service and the other services included in the deployment into a pre-production environment and conducts some smoke tests. The new functionality is then deployed during the release window agreed with CM. We then check our monitoring and alerting systems to verify if any of the new changes have broken the system.</p>
<h2>What is an average day-in-the-life of someone on one of the development teams?</h2>
<p>The development team working on the assessment platform is split into smaller teams following <a href="https://medium.com/project-management-learnings/spotify-squad-framework-part-i-8f74bcfcd761">Spotify’s squads model</a>. Each squad is cross-functional, formed by Devs, QAs, BAs and a a developer that acts as technical leader. Each squad owns a piece of the functionality of the platform.</p>
<p>The day starts with each squad daily standup, where people walk the Kanban wall talking about the progress made in each story, what they are planing to do that day and surfacing any blocker that may get in the way.</p>
<p>Most of the time, a developer is pairing with a peer in delivering theuser story functionality. Other times is pairing with a BA analysing the technical details of a piece of work, or with the Tech Lead, helping him or her set the technical direction of a feature, evolving the architecture of the system, etc.</p>
<p>The squad holds showcase sessions every two weeks to share what they have done with the other squads. Developers, QAs and BAs are usually the ones showcasing the progress made. Another ceremony, the team retrospective, is done every two weeks. Developers take an active role on this session reflecting on how the team can work more effectively. The squads also organise technical sessions called “code time” or “tech congress” where developers share technical solutions and discuss technical topics.</p>
<p>We tried to follow the <a href="https://www.thoughtworks.com/insights/blog/there-no-such-thing-devops-team">“you build it, you run it”</a> principle, which implies that some times, as a developer, you will be pairing with someone from other squad to support the production system during working hours. This requires working with the customer support teams in investigating production issues and trying to solve them. An additional developer from the client is on call outside working hours.</p>
<h2>What makes this project a special place to be a developer?</h2>
<p>I will quote one of the answers from the original article.</p>
<blockquote>
<p>There’s a real feeling that while writing code can be a solitary act, making software isn’t. People are really kind and generous with their knowledge, and overwhelmingly willing to help.</p>
</blockquote>
<p>In my team writing code is a collaborative act, thanks to practicing pair programming, one of the core practices we follow at ThoughtWorks. You can have a bad day or lose motivation, but your pair will always try to help you to forget about it and carry on doing great work.</p>
<p>Moreover, I personally like the technical challenges I have faced in this project. Building and evolving a platform based on a microservices architecture on top of very rigid virtualised infrastructure, flaky networks (self-managed data centres..) and in the context of a “risk-adverse” organisation is always a challenge.</p>
<p>I hope you found my answers useful or at least enjoyable to read. I encourage you to do the same exercise and share with us how it is like to be a developer in your current project. And please, if you have any recommendations on how my team can improve their development experience, share them in the comments.</p>
<p>You can find other tools that we use checking <a href="https://stackshare.io/portovep/loan-assessment-platform">this stack</a> on Stackshare.</p>
<p>PS: If you would be interesting to join me in this project, please leave me a PM :).</p>
