[{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/tags/aid/","section":"Tags","summary":"","title":"Aid","type":"tags"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/","section":"Daniel Kawalsky","summary":"","title":"Daniel Kawalsky","type":"page"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/tags/development/","section":"Tags","summary":"","title":"Development","type":"tags"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/categories/policy--development/","section":"Categories","summary":"","title":"Policy \u0026 Development","type":"categories"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/tags/rwanda/","section":"Tags","summary":"","title":"Rwanda","type":"tags"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":" DR Congo loses $750m in mining revenues to corruption | Corruption News\nIn a fair system, every road built and every school opened would push the world forward. Corruption drains that momentum before it can build. Over the past few years, sixteen books and video essays have reshaped how I think about aid and its adversary, corruption. Their common lesson is simple: helping others is enlightened self-interest. Kurzgesagt binds their lessons together, positing, “In a positive-sum world, it’s in your personal best interest that every human on Earth is well off.” When eight billion potential scientists, engineers, and artists can flourish, moonshot cancer cures and Mars colonies edge closer and sooner, and life grows richer and safer for all of us. Yet the moment you try to accelerate that world, whether by funding a clinic, backing a rural school, or supporting disaster relief , you collide with a preventable scourge: corruption. Even if it spares the cash you give, it can still steal the future your gift was meant to buy.\nGiving Anyway: The Net Gain Argument # At this point, the cynic objects: “If a chunk of every donation leaks away, why give at all?” Because, even after the leak, the pool is deeper than before. Over the past two decades, investments in global health have returned roughly ten dollars in economic benefit for every dollar spent. Yes, some aid is lost to corruption, but audits consistently find fraud rates below 1 percent, adding only a few cents of waste to programs that already save a life-year for well under ten dollars. In vaccination campaigns, for example, the gains still overwhelm the losses by orders of magnitude. If a fire brigade misplaces one hose, you don’t abandon firefighting; you fix procurement and keep pumping water.\nThe paragraphs that follow examine that leakage in three settings. This essay traces how corruption repeatedly cuts into that compounding effect. We’ll start with Kenya and the Democratic Republic of Congo, where small-scale and systemic graft undercut basic services and economic growth. Next comes Beirut, where years of political patronage left 3,000 tons of ammonium nitrate unattended until it destroyed part of the city, a reminder that corruption can hamper both development and disaster response. From there, we pivot to the United States, where legalized channels of money in politics erode public trust and slow urgently needed reforms. Finally, we turn to counter-examples: Norway’s oil fund and practical accountability tools (from transparency clauses to public rating systems) that show corruption is resistible. The message throughout is straightforward: the more effectively we seal these leaks, the faster shared progress accelerates.\nHow Corruption Traps Progress # Poor Economics calls the relationship viciously circular: “Poverty causes corruption and corruption causes poverty.” You see this in Kenya, where 1 in 3 shillings from the national budget disappear behind shell companies, while mothers still pay bribes just to keep a child in school. Those billions that “should have gone into development have been stolen.” In the Democratic Republic of Congo, an underground wealth of cobalt, copper, gold, and other minerals crucial to the global economy and modern innovation worth an estimated $24 trillion has translated into malnutrition, broken roads, and conflict because, as one Deutsche Welle reporter puts it, “Congo is resource-endowed but also resource-cursed… the kleptocratic rule refined under Mobutu became part and parcel of how the country is governed.” Under Mobutu, millions meant for clinics and malaria programs vanished into the pockets of officials, leaving villages with crumbling walls where hospitals should have stood.\nThe Knowledge Warehouse\nLebanon offers another brutal example: political factions carved ministries into personal fiefdoms, public services rotted, and for seven years, a cache of ammonium nitrate sat forgotten at Beirut’s port… until it exploded. In seconds, the blast leveled neighborhoods, shattered a city, and killed nearly 200 people. International aid poured in, but trust in the government was so broken that donors insisted on bypassing state channels, and even then, distribution was slow, tangled, and marred by allegations of corruption.\nIn Beirut, as in so many places, theft didn’t just steal the future– it crippled the rescue. In Lebanon, like in Kenya, aid might buy medical supplies or mosquito nets, but corruption can erase the hospitals, the bridges, and the lives they were meant to save.\nLebanon’s capital city rocked by explosion\nCorruption is hardly the only obstacle. Geography, history, even rotten luck all drag on growth, but corruption is the rot that hollows every other effort. When “extractive institutions… do not create the incentives needed for people to save, invest and innovate,” they lock whole societies into what Why Nations Fail calls a vicious circle: elites cling to power, reform stalls, and the poverty they profit from deepens. Poverty, By America shows the same mechanism at work in rich countries: a fair share of aid “never reaches” its target, so relief expands even as the poverty rate stays flat, because “some wish and will it to.” Whether we label the leakage graft, state capture, or legalized bribery, the pattern is identical; value is siphoned off instead of created, and the circle of shared prosperity never quite closes.\nBreaking the Cycle # But nothing in human nature says things must stay this way. Poor Economics reminds us that “careful understanding of the motivations and constraints of everyone… can lead to policies and institutions that are better designed and less likely to be perverted.” In contrast to the cursed riches beneath the DRC sits the oil offshore Norway, which proved that natural wealth doesn’t have to be a trap. From the first barrel of oil Norway procured the government declared that the resource “belonged to the public,” channelled revenues into a transparent sovereign wealth fund, and resisted the siren song of quick political spending. Key to this discipline is the Handlingsregelen (Fiscal Rule), which limits annual government spending from the fund to the expected real return of 3%. The proceeds quietly underpin Norway’s social model, helping to fund healthcare, education, infrastructure, and pensions without exhausting the oil wealth itself. More than just policy, the rule is culturally internalized: Norwegians learn about it in school, the media fiercely monitors it, and political norms punish those who try to erode it. The result is a rare cycle of accountability, where transparency and long-term thinking reinforce each other. Three decades later that fund is the world’s largest, proof that a society can turn finite wealth into “lasting, generational, and widely-felt prosperity.”\nThe headquarters of the Norges Bank in Oslo. Photo: Bloomberg News\nAt the heart of success stories like Norway’s are not natural gifts, but human choices. What separates Oslo from Kinshasa (beyond geography) is not luck, but the presence of institutions that align power with accountability. As Why Nations Fail notes, the richest countries are not those with the best soil or biggest populations but those with “well-managed, honest, and effective institutions” at the national level. But change can also start small: James Long’s TEDx talk shows that even simple feedback loops; public “Yelp” scores for local officials can reduce fraud by double-digit percentages, because “knowing you’re being rated can ignite behavior change.”\nYet even nations with strong institutional traditions, like the United States, are not immune to erosion. Cracks can form (whether through court rulings like Citizens United or the quiet normalization of money-driven politics) and deepen until once-resilient systems start to wither. The same diagnosis applies to wealthy democracies. If a Kenyan policeman extracts cash on the roadside, an American politician can legally receive unlimited support from a super-PAC that “never coordinates” with the campaign. As this Ostonox video drives home, the distinction is cosmetic: “Most of the spending on modern campaigns doesn’t come from the campaigns anymore… it comes from super-PACs funded by the rich to buy politicians.” When policymaking is quietly auctioned, it slows climate action, drug-price relief, and a host of innovations whose spill-over benefits would otherwise circle the globe.\nElon Musk’s PAC pays out third $1 million check to voter\nGiving Is Still Worth It # None of this negates giving; it just clarifies the battlefield. If corruption siphons a share of every development dollar, we owe it to ourselves and to our own enlightened self-interest to attack the leakage directly. That means insisting on transparency clauses in aid contracts, backing whistle-blowers, funding investigative journalism, and championing voting reforms like “democracy vouchers” that drown out big-money influence with many small voices. It also means directing some aid toward building institutions themselves: supporting anti-corruption bodies like Kenya’s Ethics and Anti-Corruption Commission, or funding transparency initiatives like Lebanon’s Anti-Corruption for Trust project. These investments may seem indirect, but they strengthen the foundations that let every later dollar travel farther. And it means seeing progress in proportions, not absolutes. Hans Rosling liked to remind his audiences that “things can be bad and getting better.” The share of humans in extreme poverty has fallen faster in the last thirty years than at any point in history, a momentum worth protecting and accelerating. Corruption remains a challenge, but it is not an immovable one. With each effort to make systems more accountable, we ensure that progress not only continues but gathers speed.\nEgoistic altruism is not naïve idealism; it is realism scaled to a planet of eight billion. Your life improves when a Congolese engineer invents a cheaper battery chemistry, or when a when a Kenyan researcher discovers a new biomarker that helps catch your cancer earlier. Those breakthroughs arrive sooner when the school fees are not swallowed by graft and when the electricity stays on because public funds bought transformers instead of vacation homes. The obligation to fight corruption is thus inseparable from the desire to be safe, prosperous and inspired in your own life.\nThat is why this essay circles back to Kurzgesagt’s line. In a positive-sum world, lifting the worst-off is the smartest selfish play available. Corruption is the tax that turns positive sums into zero sums. Strip it away, whether in Nairobi ministries, Washington committees, or Beirut port authorities, and the arithmetic of shared progress works again. Then your donated aid is no longer a token gesture fighting a tide; it is a catalyst multiplied by functioning institutions, accelerating the day when every human being is well off and, by extension, so are you.\nSources # Books # (2018) Factfulness — Hans Rosling (2011) Poor Economics — Abhijit V. Banerjee \u0026amp; Esther Duflo (2012) Why Nations Fail — Daron Acemoglu \u0026amp; James A. Robinson (2023) Poverty, By America — Matthew Desmond YouTube # (2018) A Selfish Argument for Making the World a Better Place — Egoistic Altruism — Kurzgesagt — In a Nutshell (2021) Why Oil Doesn’t Corrupt Norway — PolyMatter (2024) Why isn’t the DR Congo the world’s richest country? | Mapped Out — DW News (2023) So You Want to Bribe a Politician — Ostonox (2024) The 2024 Nobel Prize in Economics: Explained — Economics Explained (2009) Billions lost in Kenya bribes — 01 Jul 09 — Al Jazeera English (2011) Poor Economics (Interview/Discussion) — TVO Today (2023) MIT Study Reveals Why Africa Is Still Poor — Economics Explained (2016) Fighting Corruption in the Developing World | James D. Long — TEDxUofW (TEDx Talks) (2019) The Economic Failure of Venezuela — Economics Explained (2020) How the Beirut explosion was a government failure — Vox (2022) Undercover In Guyana: Exposing Chinese Business in South America — VICE News ","date":"28 April 2025","externalUrl":null,"permalink":"/why-aid-is-still-worth-it-06ab815a4659/","section":"Posts","summary":"In a fair system, every road built and every school opened would push the world forward. Corruption drains that momentum before it can build. Over the past few years, sixteen books and video essays have reshaped how I…","title":"Why Aid Is Still Worth It","type":"posts"},{"content":" This project began from a simple frustration: existing real estate websites in Rwanda often fail to show exactly where a property is located. In many cases, this makes it impossible for a buyer to understand whether a property is in a desirable neighborhood, close to infrastructure, or in a remote location. This gap led me to wonder: could I build a better, geospatially-aware version of Zillow for Rwanda?\nBefore we get into the thick of it, consider this: in Rwanda, putting property data online shifts the responsibility for accuracy to individual owners, many of whom don’t have the tools or digital know-how to create and maintain good listings. In the U.S., MLS platforms (which Zillow sources its data from) are run by industry groups and only licensed agents can post, usually under standardized contracts like the exclusive right-to-sell. Rwanda’s approach, with e-government services that let anyone list and sell their own property, is good and democratic, but without standardized agreements or professional upkeep, portals struggle to build the same trust and momentum. It’s a tradeoff between openness and reliability.\nThe Existing Landscape of Rwandan Real Estate Platforms # Today, Rwanda has several real estate platforms, including HouseInRwanda.com and HomeRwanda.com. These platforms list properties online, but many of them charge real estate agents to post listings (sometimes with monthly fees), and none of them reliably show the precise location of a property. Most listings give only the sector or village, leaving buyers guessing.\nAlongside these more formal platforms, many brokers use free tools like WhatsApp and Instagram to list and promote properties. Some real estate agents have created Instagram feeds specifically dedicated to showcasing listings, and over time, they’ve built a following of interested buyers. They post photos, pricing, brief details, and contact information. It’s an effective and free alternative, even though these platforms don’t support map-based browsing or structured data entry.\nThe UPI: Rwanda’s Unique Parcel Identifier # Rwanda has an advanced land management system. Every parcel of land in Rwanda is assigned a Unique Parcel Identifier (UPI), a code that encodes its location within Rwanda’s administrative hierarchy: province, district, sector, cell, and village. But the UPI code alone does not provide latitude and longitude. You need a geospatial database that maps each UPI to coordinates to plot them on a map. In some cases, this information exists within government systems like the Rwanda Land Management and Use Authority (RLMUA), but it is not widely available for public platforms.\nThe Stakeholders of Rwanda’s Real Estate Ecosystem # Building a Zillow-like platform in Rwanda involves navigating a complex web of stakeholders, each with their own interests and concerns:\nThe Rwanda Land Management and Use Authority (RLMUA) (or now, just the National Land Authority), which manages land data and UPI information The Rwanda Revenue Authority (RRA), which may be interested in property valuations for tax purposes The Institute of Real Property Valuers (IRPV), which regulates property valuation practices Real estate agents and brokers, who play a critical intermediary role between buyers and sellers Property owners, who sometimes prefer informal transactions Buyers, who increasingly expect transparency, convenience, and online listings Each of these groups presents both potential partnerships and obstacles depending on how a platform engages with their incentives.\nBrokers, Informality, and the Business of Real Estate in Kigali # During my research, I interviewed real estate agents in Kigali. Their responses revealed a complicated, largely informal ecosystem where transparency is both an opportunity and a threat.\nAs one broker put it:\n“Here in Kigali, there’s no regulation around real estate. None. Even though we’ve had meetings with the government, right now there’s no regulation in place. That’s what makes it difficult to manage this business.”\n— Broker A\nAnother challenge is that owners will often contact multiple brokers to market the same property, which fuels competition:\n“Many brokers might know about the same house. For example, in Kigali, an owner might call many brokers to help market their property. In the end, the owner will pay the broker who successfully brings the client.”\n— Broker A\nAnd getting leads on properties is itself a business. Brokers sometimes rely on locals who act as fixers, earning a tiny reward for spotting a new property for sale:\n“I pay locals a Coca-Cola or maybe a few bucks a day to pass along tips about homes for sale. It might not sound like much, but these locals can give you first-hand knowledge of a new property before anyone else knows about it.”\n— Broker B\nIn some cases, brokers keep these local scouts on a kind of informal retainer, offering small daily incentives like soft drinks or modest payments in exchange for real-time intel from neighborhoods. Once a scout alerts the broker about a property, the broker then advertises it through their network or Instagram feed, hoping to close a deal and earn a commission.\nThis use of Instagram to build a lead-generation funnel is surprisingly effective. Agents post in their feed and stories, engage directly with prospective buyers in DMs, and manage follow-ups all from a free app that doesn’t require technical skill to operate.\nWhy Brokers Avoid Sharing UPI or Exact Locations # The resistance to publicly sharing UPI codes or even dropping a pin on a map comes down to one key fear: being cut out of the deal.\n“If someone has the UPI, it’s easy to get all the information about the property owner. Even just having the exact location makes it easier to find the house and contact the owner directly.”\n“One client asked me for the UPI of a house. I shared it with him. After that, he didn’t call me back to go view the house together. Instead, he went there on his own, met the owner, and finalized the deal. I didn’t earn anything from that transaction.”\n— Broker A\nAs a result, brokers have developed norms around how much information to share:\n“Our job is about sharing information, yes, but in a controlled way. So when posting online, putting the sector and village is fine. That’s enough detail for clients to have an idea, but not enough for them to go directly without us.”\n— Broker A\nRwanda’s shift toward digital, self-service platforms like the e-Title system for electronic land registration has empowered citizens to handle property transactions directly, often without needing intermediaries. While this reflects a commendable push for government efficiency and transparency, it also inadvertently undermines the traditional role of real estate agents. When buyers can access official land titles and ownership details online, especially if paired with a property’s exact location or UPI code, they’re more likely to bypass brokers entirely, cutting them out of the negotiation and the commission.\nNavigating This Reality When Building a Platform # This creates a fundamental tension for any Zillow-like platform in Rwanda. Buyers want more transparency. Brokers want to protect their leads. A platform must balance these needs carefully.\nOne option is to allow approximate location information (sector or village) but avoid showing the exact parcel or UPI until later in the transaction flow.\nOffering a Free Platform (And Managing the Risks) # One of the simplest competitive advantages against platforms like HustleRwanda.com would be to offer free listings. However, this also invites spam and scams. Content moderation becomes a significant cost.\nTo mitigate bad actors, platforms could require users to verify their phone number via SMS. But sending SMS messages costs money. WhatsApp-based verification might not be feasible due to limitations in their business messaging platform.\nHosting Costs \u0026amp; Monetization Considerations # Aside from verification, there are also hosting costs to consider. Moderating user-generated content (photos, descriptions, locations) requires not just technical resources but also human oversight. Monetizing a free platform might require a mix of premium features, ads, or partnerships with government bodies or developers.\nLanguage Localization: A Simple Way to Add Value # Many existing real estate platforms list properties only in English. Offering Kinyarwanda and Swahili translations for both platform content and user listings would immediately expand accessibility, especially for local buyers and sellers who are not fluent in English.\nUsing AI for Property Valuation # Finally, an exciting feature would be using AI models to estimate the value of properties, based on location, UPI data, square footage, number of bedrooms/bathrooms, and other features. Scraping existing listings and mapping them to UPI codes could provide the training data needed to build such a model.\nConclusion # Building a Zillow-like platform for Rwanda is not just a technical challenge, it’s a cultural and market design challenge. Balancing transparency with brokers’ business models, managing content moderation, and localizing for language and context will all be essential to success.\nYet, despite the challenges, the opportunity is clear: to build a geospatial, user-friendly, multilingual platform that reflects how real estate really works in Rwanda while gently nudging it toward greater openness and efficiency.\n","date":"15 April 2025","externalUrl":null,"permalink":"/building-a-zillow-like-platform-for-rwanda-opportunities-and-challenges-eb7304e7bb5e/","section":"Posts","summary":"This project began from a simple frustration: existing real estate websites in Rwanda often fail to show exactly where a property is located. In many cases, this makes it impossible for a buyer to understand whether a property…","title":"Consider This Before Building a Zillow-Like Platform for Rwanda","type":"posts"},{"content":"","date":"15 April 2025","externalUrl":null,"permalink":"/tags/product-design/","section":"Tags","summary":"","title":"Product Design","type":"tags"},{"content":"","date":"15 April 2025","externalUrl":null,"permalink":"/tags/real-estate/","section":"Tags","summary":"","title":"Real Estate","type":"tags"},{"content":"","date":"15 April 2025","externalUrl":null,"permalink":"/categories/technology--engineering/","section":"Categories","summary":"","title":"Technology \u0026 Engineering","type":"categories"},{"content":"","date":"24 December 2024","externalUrl":null,"permalink":"/tags/ai/","section":"Tags","summary":"","title":"AI","type":"tags"},{"content":" This blog post is a “how I built this” follow-up to A Chat-First Approach to Digital Government Services: The Irembo Chatbot The code for this blog post can be found in this Github repository: https://github.com/danksky/irembo-chatbot “ChatGPT, generate an image of a digital Rwanda”\nThe goal was simple: to build a system that make IremboGov portal services more accessible to Rwandan citizen by combining conversational AI with a Retrieval-Augmented Generation (RAG) framework.\nThis post outlines the architecture and key components — OpenAI’s models for language processing, Pinecone for vector storage, and LangChain for managing conversational flows. It also digs into the trade-offs and decisions that shaped the project. If you’re an Irembo developer or manager, you might be curious about how this chatbot project came together and what practical takeaways it offers. This write-up is for anyone who is exploring ways to enhance their own systems or just looking to understand the tech better.\nOverview of Components # GPT and Embedding Model # I wanted to optimize my two model types, a GPT model and an embedding model, for three key points: most cost-effectiveness, speed, and accuracy.\nTo experiment for speed and accuracy, I used a script to:\ndownload the page content from the Irembo support article: Frequently Asked Questions About Driving Licenses load the page content into an InMemoryVectorStore invoke a query on my LangChain StateGraph flow with various combinations of embedding and GPT models: The question I asked was: Human: How much do I pay for an application for a provisional driving license, and which country is this document about?\nThe answer I was looking for was: AI: The application for a provisional driving license costs RWF 10,000. This document pertains to Rwanda.\nGPT Model: Why I chose Open AI GPT-4o # I chose GPT-4o (specifically gpt-4o-mini-2024–07–18) for this project because it performs well and is cost-effective for the use case. It was straightforward to integrate the GPT-4 LLM model into conversational flows and retrieval logic with LangChain. This setup ensures that the LLM can respond naturally while leveraging the context provided by the Retrieval-Augmented Generation framework.\nYou can also see in this gist that I tried meta-llama/Llama-3.3–70B-Instruct but it wasn’t free and I didn’t want a paid HuggingFace subscription.\nI also tried CohereForAI/c4ai-command-r7b-12–2024 but it wasn’t accurate enough for me.\nEmbeddings Model: Why I chose Open AI # To enable effective context retrieval from the Pinecone vector database, I experimented with a variety of embedding models, including:\nThe good: Open AI’s text-embedding-3-large got the job done, accurately. The bad: These models simply didn’t work well enough to accurately store the vectors such that the necessary information was retrievable during my testing. The context that was retrieved was wrong, no matter how much I adjusted the k value for my vector database’s similarity_search.\n– HuggingFace’s all-MiniLM-L6-v2\n– Open AI’s text-embedding-3-small\n– Google’s text-embedding-004 The ugly: Mistral AI’s mistral-embed gave me grief, because the Langchain package, langchain-mistralai, was requesting something from HuggingFace and the request was breaking. After testing these options, I ultimately chose OpenAI’s text-embedding-3-large. It provided the most reliable results for retrieving relevant content, so I trusted it to do the same when I changed my vector database from InMemoryVectorStore to PineconeVectorStore.\nPinecone # I chose Pinecone as a scalable and persistent vector database, eliminating the need to rebuild the vector database on each deployment. This persistence allows us to reuse embeddings, reducing the API consumption for OpenAI’s embedding model. Pinecone’s generous free tier will hopefully offer enough capacity to host our demo as users query Irembo’s support articles via the live chatbot.\nSetting up Pinecone involved creating an index optimized for storing embeddings from OpenAI’s text-embedding-3-large model. LangChain’s PineconeVectorStore integration made it straightforward to connect Pinecone to the chatbot, enabling seamless similarity-based retrieval of stored documents.\nState Graph # Graph Setup\nThe state graph forms the backbone of the chatbot, organizing how it processes user input and generates responses. Built using LangChain’s StateGraph, the design ensures that each stage of the interaction is modular and manageable, from understanding the user’s query to retrieving relevant context and generating a final response.\nThe graph includes three primary nodes:\nquery_or_respond: Determines if a query requires retrieval or can be answered directly using the conversational context. This node acts as the decision point. It evaluates the user’s query to decide whether to proceed with retrieval or skip directly to response generation. By ensuring retrieval is only used when necessary, this step optimizes both performance and cost.\nretrieve: Fetches the most relevant information from the Pinecone vector database when needed. If the decision is made to fetch additional context, this node interfaces with the Pinecone database. It performs a similarity search on the stored embeddings to find the most relevant documents. The results are passed along to the next step for integration into the response.\ngenerate: Uses the LLM to create a response, incorporating any retrieved context. At this stage, the LLM generates a complete response. It combines the query, retrieved context (if any), and the conversation history to produce a concise, accurate answer tailored to the user’s needs.\nNode Ordering\nThe nodes are connected through conditional edges to maintain a logical flow. For example:\nquery_or_respond → retrieve → generate: For queries requiring external context.\nquery_or_respond → generate: For straightforward conversational queries.\nBy leveraging LangChain’s add_conditional_edges and checkpointer (using MemorySaver), the system handles transitions seamlessly while preserving conversation history across sessions.\nThis modular design makes the chatbot robust, efficient, and easy to extend. Each node focuses on a specific task, ensuring clarity and maintainability in the overall workflow.\nLangChainTracer\nLangChainTracer was used to debug and monitor the flow of the chatbot during development. It tracked how queries moved through the state graph, providing insights into transitions between nodes like query_or_respond, retrieve, and generate. This made it easier to spot bottlenecks or errors, especially in complex interactions.\nAdding LangChainTracer was simple — just a few lines to set it up as a callback in LangChain’s configuration. This low-effort integration proved invaluable for ensuring that each node performed as expected and that retrieval and generation steps were correctly synchronized. It also allowed for easy performance tuning and iterative improvements throughout the project.\nGradio Interface\nThe chatbot’s interface was built using Gradio’s ChatInterface, which simplifies the process of creating conversational applications. This component allowed me to focus on integrating the backend functionality without needing to build a custom front-end from scratch.\nOne major advantage of using Gradio is its seamless deployment capability with gradio deploy, enabling the app to be published directly to Hugging Face Spaces. This made it easy to share a live, interactive version of the chatbot for demos and testing.\nThe integration of ChatInterface provided a clean, user-friendly way to manage conversations while maintaining support for features like session tracking through Gradio’s request parameter. This ensured that the chatbot could handle user interactions efficiently and maintain context across sessions.\nHow the Pinecone Database Was Populated # To populate the Pinecone vector database with Irembo’s support articles, I developed a Python script that automates the process of scraping, processing, and storing the content. Here’s an overview of the key components and their roles:\nScraping Irembo’s Support Articles\nThe script begins by fetching the sitemap from Irembo’s support portal, which lists all available articles. By parsing this sitemap, the script identifies the URLs of individual support articles for subsequent processing.\nProcessing and Cleaning the Content\nFor each article URL, the script performs the following steps:\nHTML Retrieval: Downloads the HTML content of the article page.\nContent Extraction: Utilizes Beautiful Soup to parse the HTML and extract the main content, specifically targeting elements that contain the article’s text.\nText Cleaning: Processes the extracted text to remove any extraneous whitespace, HTML tags, or irrelevant sections, ensuring that only the meaningful content remains.\nSplitting Documents into Chunks\nTo enhance the efficiency of the vector database and improve the relevance of search results, the cleaned text is divided into smaller chunks. This segmentation allows the system to retrieve specific sections of an article that are most pertinent to a user’s query.\nGenerating Embeddings\nEach text chunk is then converted into a numerical vector representation using OpenAI’s text-embedding-3-large model. These embeddings capture the semantic meaning of the text, enabling effective similarity searches within the vector database.\nUpserting into Pinecone\nThe final step involves inserting (upserting) these embeddings into the Pinecone vector database. Each entry in the database includes:\nID: A unique identifier for the text chunk.\nEmbedding: The vector representation of the text.\nMetadata: Additional information such as the source URL and the original text content, which can be useful for reference and debugging.\nBy automating this pipeline, the system ensures that the Pinecone database remains up-to-date with the latest support articles from Irembo, facilitating accurate and efficient retrieval of information in response to user queries.\nFor a detailed implementation, you can refer to the scrape_irembo.py script in the GitHub repository.\nWhy Not Just Use BotoPress? # BotoPress is an appealing option for quickly building chatbots, especially with its straightforward drag-and-drop interface and built-in integrations with platforms like WhatsApp, Facebook Messenger, and others. It would have streamlined the process of deploying a chatbot to multiple channels without much additional effort.\nHowever, the focus of this project was to deeply explore and customize a Retrieval-Augmented Generation (RAG) workflow. By building the system from scratch, I was able to directly control how data is stored in Pinecone, how retrieval integrates with the chatbot’s conversational logic, and how embeddings are managed. This level of granularity wouldn’t have been possible with BotoPress, which abstracts much of the implementation.\nMoreover, developing this solution allowed me to break down the components of a RAG workflow and understand how systems like BotoPress might function under the hood. While BotoPress could still be considered for future iterations, this hands-on approach provided valuable insights and flexibility for tailoring the chatbot to Irembo’s specific needs.\nLessons Learned # poetry for Dependency Management: Poetry streamlined managing dependencies and environments, ensuring a consistent and reproducible setup.\nExperimentation is key: Testing multiple models (HuggingFace, OpenAI, Mistral, etc.) and tools like Pinecone highlighted the importance of flexibility during development.\nLangChain simplifies complexity: LangChain’s abstractions, like StateGraph and VectorStores, made integrating retrieval and LLM workflows straightforward while keeping the system modular.\nPinecone saves resources: Pinecone’s persistent storage eliminated the need to regenerate embeddings, reducing API usage and speeding up deployments.\nCustom builds offer insight: Building the chatbot manually provided a deeper understanding of RAG workflows, making it easier to customize and optimize.\nConclusion # Building this chatbot was an exercise in understanding and customizing a Retrieval-Augmented Generation (RAG) workflow to meet specific requirements. By combining tools like OpenAI’s models, Pinecone for vector storage, and LangChain for conversational logic, I was able to create a modular, scalable, and cost-effective solution for improving access to IremboGov portal services.\nThe project highlighted the value of experimentation, from testing different embedding models to refining retrieval logic and integrating state graph workflows. It also reinforced the importance of tools like Poetry for dependency management, LangChainTracer for debugging, and Gradio for a seamless user interface.\nThis approach not only delivered a functional chatbot but also provided a deeper understanding of the underlying technology, which can be applied to future projects. If you’re exploring similar solutions or have questions about the tools and techniques discussed here, feel free to reach out — I’d love to hear your thoughts or collaborate further.\n","date":"24 December 2024","externalUrl":null,"permalink":"/building-a-rag-powered-chatbot-for-irembo-48dfada7e1b0/","section":"Posts","summary":"The goal was simple: make IremboGov portal services more accessible to Rwandan citizens with conversational AI and a Retrieval-Augmented Generation framework.","title":"Building a RAG-Powered Chatbot for Irembo","type":"posts"},{"content":"","date":"24 December 2024","externalUrl":null,"permalink":"/series/building-the-irembo-chatbot/","section":"Series","summary":"","title":"Building the Irembo Chatbot","type":"series"},{"content":"","date":"24 December 2024","externalUrl":null,"permalink":"/tags/digital-government/","section":"Tags","summary":"","title":"Digital Government","type":"tags"},{"content":"","date":"24 December 2024","externalUrl":null,"permalink":"/tags/rag/","section":"Tags","summary":"","title":"RAG","type":"tags"},{"content":"","date":"24 December 2024","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"},{"content":"When my friend Claude came to me for advice on registering his newly purchased land, I realized how much of a challenge navigating government services can still be. Claude had bought part of a larger parcel of land to build his house but needed to officially subdivide it and register his ownership. The process involved visits to the National Land Authority (NLA), approvals from notaries, and payments for inspections.\nWhile Rwanda has made significant strides in digitizing public services through platforms like Irembo, there’s still room to make these services easier to use for everyone. This led me to create the Irembo Chatbot, a proof of concept demonstrating how a chat-first approach could simplify access to these essential services.\nMaking Complex Services Intuitive # Irembo offers over 200 government services online, ranging from administering the Certificate of Being Alive to the E-Provisional Driving License.\nWhile these services are digitally accessible, navigating them through a web portal can be intimidating for those less familiar with websites. Many Rwandans are more comfortable with chat interfaces like WhatsApp, which provide an intuitive and familiar way to communicate. The chatbot leverages this familiarity, encouraging users to ask questions conversationally rather than searching through menus and articles.\nHow the Irembo Chatbot Helps # The chatbot offers a chat-first way to interact with government services. Instead of navigating a web portal or consulting intermediaries, users can ask direct questions like:\n“How do I subdivide land and register ownership?”\nThe chatbot provides actionable, step-by-step guidance, explaining what documents are needed, where to go, and how much to pay.\nA Proof of Concept with Big Potential # This project isn’t a replacement for Irembo — it’s a prototype exploring how conversational AI can complement and enhance digital services. By starting with basic Q\u0026amp;A and navigation, the chatbot demonstrates how chat interfaces could:\nSimplify complex processes by turning them into easy-to-follow conversations. Reduce reliance on agents and intermediaries, empowering citizens to self-serve. Provide guidance on-demand, anytime and anywhere. Looking ahead, the chatbot could grow to handle more complex interactions. It could integrate with multiple ministries, like MINEDUC or MINAGRI, to streamline tasks such as checking agricultural subsidies or applying for educational scholarships.\nA Future of Seamless Government Interaction # The Irembo Chatbot shows what’s possible when technology meets accessibility. By making services more intuitive and removing barriers to entry, it represents a small but meaningful step toward a more inclusive digital future for Rwanda.\nThis is just the start. With time and development, the chatbot could evolve into a powerful tool that connects citizens with government services, not just as a resource for answers but as an all-around agent for completing processes. For now, it’s a simple demonstration of what could be — a glimpse at a future where engaging with public services is as easy as sending a message.\nWant to try the chatbot? You can access a live demo here:\nhttps://huggingface.co/spaces/danksky/irembo-chatbot\nWant to know how the chatbot is built and why it’s built that way? Read my article on just that: https://blog.danielkawalsky.com/building-a-rag-powered-chatbot-for-irembo-48dfada7e1b0\nIrembo Auto-Agent\ndanksky-irembo-chatbot.hf.space\n","date":"24 December 2024","externalUrl":null,"permalink":"/a-chat-first-approach-to-digital-government-services-the-irembo-chatbot-62f443053b83/","section":"Posts","summary":"When my friend Claude came to me for advice on registering his newly purchased land, I realized how much of a challenge navigating government services can still be. Claude had bought part of a larger parcel of land…","title":"A Chat-First Approach to Digital Government Services: The Irembo Chatbot","type":"posts"},{"content":"","date":"24 December 2024","externalUrl":null,"permalink":"/tags/public-services/","section":"Tags","summary":"","title":"Public Services","type":"tags"},{"content":"","date":"6 December 2023","externalUrl":null,"permalink":"/tags/lake-kivu/","section":"Tags","summary":"","title":"Lake Kivu","type":"tags"},{"content":"","date":"6 December 2023","externalUrl":null,"permalink":"/tags/travel/","section":"Tags","summary":"","title":"Travel","type":"tags"},{"content":"","date":"6 December 2023","externalUrl":null,"permalink":"/categories/travel--experiences/","section":"Categories","summary":"","title":"Travel \u0026 Experiences","type":"categories"},{"content":"In October 2021, I visited Rwanda’s Lake Kivu. My guide, Claude, from Africa Green Tours, made the trip memorable for me, his only client for the weekend.\nOur first day, we went Island Hopping. We zoomed around the lake on a wooden motor boat, with Claude pointing out different plants, animals, and sharing island stories.\nThat night was the Night Fishing tour. Before we boated out to meet the fisherman in the middle of the lake, we stopped at a village, navigating along a trail of concrete bags laid down on the ground through a cluster of houses, where Claude disappeared into a home full of chatter, emerging with a container of urwagwa, a local banana beer, which would be our gift for the fishermen.\nThough I missed the singing, the experience was still special. We joined fishermen at night on one of their three joined boat, watching them catch sambaza under the light of kerosene lanterns until Claude and I were too drunk or tired to stay on the boat anymore.\nSipping that urwaga, getting lit by the latern light.\nNight Fishing on Lake Kivu - Africa Green Tours\nThe next day, I went kayaking on Lake Kivu. You could see the DRC on the other side of the lake. The water was calm, so I visited islands and some fishermen hanging out on the lakeshore.\nThe hospitality I received was heartwarming. Claude invited me to his home, where I met his family and enjoyed a meal with them. His wife had fried up the sambaza that we’d caught the night before. The backdrop to our conversation was pop music videos, playing from DVDs on a small TV.\nInspired by Claude’s kindness, I later created a website for him, africagreentours.com, to help share these experiences with others.\n","date":"6 December 2023","externalUrl":null,"permalink":"/weekend-at-lake-kivu-0b7a253d5100/","section":"Posts","summary":"In October 2021, I visited Rwanda’s Lake Kivu. My guide, Claude, from Africa Green Tours, made the trip memorable for me, his only client for the weekend.","title":"Weekend at Lake Kivu","type":"posts"},{"content":"","date":"27 November 2023","externalUrl":null,"permalink":"/tags/apis/","section":"Tags","summary":"","title":"APIs","type":"tags"},{"content":" Image generated with DALL·E 3\nThe Problem: # You know the drill. You go for a medical visit, maybe get some tests done, and in a few weeks, a bill for an eye-watering amount lands in your lap. As you scan the itemized list, you can’t help but wonder, “aren’t these doctors supposed to be looking out for me? What sort of billing syndicate are they a part of?”\nThis isn’t just about sticker shock; it’s about rejecting the doctor’s office as an opaque institution where only a few know the true cost of admission.\nFool me once… # You find a mole on your face. Oh nose! After you’ve already been stuck with a bill featuring some surprise line items, you, the oh-so-wise consumer, now know to call ahead to get a quote on a mole removal. You might even call around to several dermatologists to get the best price. Here are a couple of responses to expect:\n“I don’t know.” “It depends on your insurance.” the dismay\nLet’s break down the problem here:\nUnexpected Costs: Unclear pricing leads to financial surprises. Lack of Informed Consent: Without cost information, you can’t make an informed decision. Administrative Burden: The current system is overly bureaucratic. It doesn’t have to be this way. Cue…\nThe Solution: (API-Enabled) Price Estimates: # The solution isn’t in rearranging the deck chairs; it’s about changing the entire game. Enter pricing APIs — a tech-driven answer to the opaque world of healthcare billing. Imagine your healthcare provider could ping your insurance, and in a second, you get a straightforward invoice detailing what you’ll owe. This isn’t science fiction; it’s a practical, feasible solution that could revolutionize how we navigate healthcare costs. So, how does this sound… What if there was a federal law that requires healthcare providers and insurance companies to electronically communicate pricing and individual coverage information to each other so that they can give you accurate quotes prior to care?\nEnter H.R. 4822: Health Care Price Transparency Act of 2023 — this bill aims to meaningfully reform healthcare by making costs transparent, ensuring patients know what they’re paying before they receive care. Among the provisions in this bill is a requirement for healthcare providers to provide a price estimation tool, which allows a user to plug in their insurance plan and get an idea of what a range of services will cost them. And because this legislation applies to any hospital that accepts Medicare, it will apply to all or nearly all hospitals in the United States. [1][2]\nExecution # Of course, if we want to get this system to work for the American patient, the execution of this legislation matters. Passing this legislation sets in motion a public works project, requiring the construction of considerable information system infrastructure. And since this project is commissioned by top-down mandates, we should be wary of how efficiently and sustainably compliant systems will be built. We don’t want this to go the way of HealthCare.gov’s initial launch, which was plagued by a “lack of dynamic alignment between policy and technology development” and “poor project scoping and system requirements analysis.” But you’ll see why the imperative for such legislation is so important when you consider the alternative: waiting around for healthcare providers and insurance companies to respond to healthcare consumers’ demand for more efficient and modern solutions. Fortunately, there’s federal precedent in legislation requiring hospitals to make their service pricing more transparent in the Public Health Service Act, “which requires each hospital operating in the United States to make its standard charges public.”\nWhile it might sound nice that your local hospital would be getting a price estimator widget on their website, I think this is likely to suck. The real good will come when services aggregate the price of services across all healthcare providers in a region. This will truly empower consumers to optimize for cost, quality of care, and convenience when choosing their healthcare. The onus of aggregation will fall on insurance providers, who already act as aggregators for healthcare services and provide such digital services to consumers.\nConclusion: # I don’t usually stan for legislation as a solution to problems. Actually, this blog started out as an opinion piece to broadcast an idea that I had for smarter integration between insurance and healthcare service providers. While researching to substantiate my idea, I came across existing legislation which champions this same idea, and more.\nThe need for pricing transparency in healthcare is undeniable. It’s time to leave behind the days of surprise medical bills and bewildering invoices. Pricing APIs offer a practical solution that can revolutionize healthcare billing, providing clarity to patients, efficiency to providers, and a brighter future for the entire healthcare system.\n","date":"27 November 2023","externalUrl":null,"permalink":"/empowering-healthcare-with-transparency-the-case-for-mandatory-pricing-apis-6b12225aca98/","section":"Posts","summary":"You know the drill. You go for a medical visit, maybe get some tests done, and in a few weeks, a bill for an eye-watering amount lands in your lap. As you scan the itemized list, you can’t…","title":"Empowering Healthcare with Transparency: The Case for Mandatory Pricing APIs","type":"posts"},{"content":"","date":"27 November 2023","externalUrl":null,"permalink":"/tags/healthcare/","section":"Tags","summary":"","title":"Healthcare","type":"tags"},{"content":"","date":"27 November 2023","externalUrl":null,"permalink":"/tags/transparency/","section":"Tags","summary":"","title":"Transparency","type":"tags"},{"content":"","date":"25 April 2022","externalUrl":null,"permalink":"/tags/customer-loyalty/","section":"Tags","summary":"","title":"Customer Loyalty","type":"tags"},{"content":"","date":"25 April 2022","externalUrl":null,"permalink":"/tags/data-privacy/","section":"Tags","summary":"","title":"Data Privacy","type":"tags"},{"content":" Heads up: not what I mean by data silos\nWhen companies entrust customers with control over their own data, they earn actual customer loyalty.\nWhy Companies Won’t Share Data # There are tons of reasons that companies do not want to allow applications to have access to our accounts’ data. An airline wants you to visit their website to see your mileage balance, so they can market airfare and credit cards to you. A bank assumes more liability by allowing other applications to read your balance and account number. A boutique loyalty program might not have the budget to engineer a custom data-sharing solution. Security issues, development costs, and marketing strategies turn companies into data silos, forcing inefficient consumer behaviors and generally keeping life harder for everyone. Businesses’ choice to silo squanders their potential to build an advanced world where businesses and consumers willingly contribute to a healthy data-sharing ecosystem.\nFortunately, some companies have built sophisticated little peep-holes into these silos– the wallet applications of Google, Apple, and AwardWallet deserve some mention– but none of these wallets can do what I want out of a digital wallet. If third-party loyalty services want to have their place in the wallet of my dreams, they need to come to the table with a solution that can work for them, so I’d like to suggest one.\nFirst, let’s talk about how these three wallets work and what that tells us about (1) the companies that store our data and (2) the people who (like me) want it all tidily in one place.\nWhat are Wallet Applications and How Do They Work # Wallet applications offer users the ability to aggregate and pay with their digital assets. Digital assets include points, cryptocurrencies, gift cards, credit cards, and even boarding passes. There are a million and one wallet applications that offer these (or some subset of these) services, so I’ll cover three preeminent wallet developers: Google, Apple, and AwardWallet.\nUsers can manually copy their data from third parties, like from hotels, airlines, and even restaurants into almost any wallet application, but these “smart” wallets use clever ways to automatically populate and update your wallet. Google Pay and Apple Wallet built “Pass” APIs that allow businesses to integrate with their native wallet applications, while AwardWallet takes a hackier approach.\nApple Wallet # Apple Wallet offers the most straightforward way to integrate third-party digital assets. A user can either add a “Pass” to their wallet either directly from the Apple Wallet app, or they can click a special link on their phone to add a Pass into their wallet. With Apple Wallet, you can see your Starbucks gift card balance and pay with your phone, but you won’t be able to see how many United Airlines MileagePlus Rewards points you have. Apple’s PassKit for developers certainly enables airlines to integrate this information, but these companies don’t want to share this information outside of controlled channels. Apple Wallet is also limited to a mobile interface, so maybe loyalty program companies figure that their Pass users don’t want to deal with the clutter.\nWhat Apple Wallet looks like\nGoogle Pay # Google Pay is like Apple Wallet because it allows users to personally add Passes to their wallet, but it also automatically adds accounts. Google has all of the information that arrives in Gmail, so in other words, Google knows your soul. Google can populate your wallet with your United Airlines MileagePlus Rewards points and Marriott Bonvoy points because your account balances arrive every month into your inbox.\nGoogle Pay grabs your loyalty account updates from your inbox.\nBoth Google and Apple built new wallet data sharing standards (Pass APIs) and enticed companies to feed data into their wallets by touting a huge user pool for third parties to tap into. These frameworks are undeniably sophisticated, but they are opinionated; that is, they’re limited to use on mobile devices and add extra complexity for businesses to integrate with those devices.\nAwardWallet # AwardWallet is where things get really interesting. Challenging all of the reasons why companies won’t share digital asset data, AwardWallet’s popularity attests to consumer demand for more control over their own data. The company provides tracking services to its more than 713,378 users despite severe pushback from third-party companies complaining about its unconventional approaches to pulling data into users’ wallets.\nThe AwardWallet dashboard showing credit card, airline, rental, and other loyalty account information\nAwardWallet is like Google since it reads your emails to populate your account balances and to track your travel itineraries, but AwardWallet requires users to opt-in to this feature. If sharing your emails outside of your email service seems sketchy to you, you might be surprised to know that 36% of new AwardWallet users opt-in. The other 64% of those users are just fine providing their various accounts’ usernames and passwords to the wallet service so that it can retrieve information on your behalf. Operating for over 17 years, AwardWallet’s user base has entrusted the online wallet with tracking 4,031,992 (and counting) loyalty accounts.¹\nAwardWallet can track over 680 loyalty accounts.\nThird party digital asset companies hate this one trick to keep track of all your important digital assets in one place. AwardWallet has received several Cease and Desists from major airlines, including American Airlines, Delta, United Airlines, and Southwest Airlines, urging AwardWallet to stop tracking their loyalty programs. In response, the AwardWallet community has rallied against these airlines’ decisions with petitions to allow integrations to resume.\nIn the meantime, loyalty customers are forced to visit several individual websites, juggling passwords and scrolling past advertisements every time they want to check up on their vouchers and points. This friction often means that users lose out on rewards that expire quietly in the night.\nWhile a select few airlines have taken to sending Cease and Desist letters, hundreds of airlines, banks, and credit card issuers are still allowing AwardWallet to fetch information on behalf of their users.\nIn 2016, when I got into credit card churning, this service turned out to be an invaluable tool to keep track of my travel rewards.\n¹ Data courtesy of Alexi Vereschaga, AwardWallet\nWhat Companies Should Do # Companies that provide digital assets should go headless and loosen their control over how their customers interact with their own assets. Just as I trust Mint and Personal Capital integrate with Chase through their secure API to give me my full financial picture, I want AwardWallet to be able to manage all of my rewards and Delta to provide a similar secure API.\nAs part of their data sharing strategy, businesses like Delta could limit which wallets services qualify for integration, and if certain wallet providers manage to prove themselves as responsible custodians of customer data, Delta could graduate them to have elevated access to information.\nA Concrete Example # Here’s what an ideal use case would look like for AwardWallet and Delta:\nAwardWallet applies to Delta as a wallet and flight tracking application. Delta approves AwardWallet’s app registration. Now AwardWallet users can track their Delta rewards and trips. As an AwardWallet user,\nI log into AwardWallet and add my Delta account. Instead of an AwardWallet form prompting me for my sensitive SkyMiles member ID and password credentials (like it does for my credit card account), AwardWallet can redirect me to Delta’s website. I log into Delta, then Delta asks if I would like to share my SkyMiles member ID, mileage balance and expiration date, and (optionally) upcoming trips with AwardWallet. I authorize AwardWallet to have access to my mileage information, but not to my upcoming trips (maybe I’m just paranoid about sharing that). Whenever I log into AwardWallet, my dashboard shows my SkyMiles balance and expiration date, but it has no idea that I’m flying to Cincinnati next Tuesday. How OAuth2.0 Fits In # OAuth 2.0 can be used to implement this formulaic use case by providing a standard way to administer data rights to third-party applications. With OAuth 2.0, businesses can let their customers decide what account and asset information is shared with their wallets.\nAn OAuth 2.0 protected API would be platform agnostic; that is, businesses would not have to build custom solutions to accomodate the likes of Google’s and Apple’s opinionated “Pass” frameworks. Businesses could build their API once and it could work everywhere.\nBut OAuth 2.0 is not the full story. It is a broad characterization of APIs that use the OAuth2.0 framework to restrict which information is shared with individual users. If we really want to foster an ecosystem of responsible data sharing, the most invested businesses should define standard data types, similar to how the Internet Engineering Task Force created GeoJSON to help people convey geospatial information. The models and schemas that airlines, restaurants, and hotels could use might resemble these types defined by Google’s Pass API:\nBoarding passes Event tickets Gift cards Loyalty Offers Transit passes A Google developer guide walks you through some use cases for implementing a Pass.\nThere are a lot of different types of businesses that would need to be involved as stakeholders in this design. The data fields that should be included in a “boarding pass” are pretty consistent among airlines, yet there might be variations in how loyalty programs treat points and vouchers. For example, mileage programs points typically have expiration dates, while credit card points typically do not. Unless there is an inclusive effort towards standardization, a wallet application would have to write custom business logic to handle every unique API’s interpretation of a user’s “rewards balance.”\nThe applications that would consume loyalty rewards data would hardly be limited to wallets. Businesses might find that by providing controlled access to customer assets, they’ll open up new, unexpected revenue streams.\nReality Check # When I reached out to Alexi Vereschaga (who started AwardWallet back in 2004) about AwardWallet building a direct integration with loyalty program providers, he explained,\nYou lost me at “integrating with airlines” ;) This is next to impossible.\nGood APIs are hard to make, and industries such as airlines are laggardly when it comes to building IT. If a community of businesses with loyalty programs did manage to define a new standard API specification, it could easily become “just another standard” that gets outdated quickly.\nhttps://xkcd.com/927/\nI only hope that the businesses running loyalty programs are considering the millions of reasons for and against publishing standard APIs. In any event, I’ll leave this suggestion out there in hopes that one day my rewards can integrate with my trackers in a more modern way.\n","date":"25 April 2022","externalUrl":null,"permalink":"/heres-how-companies-should-share-loyalty-program-data-3a88a848f528/","section":"Posts","summary":"When companies entrust customers with control over their own data, they earn actual customer loyalty.","title":"Here’s How Companies Should Share Loyalty Program Data","type":"posts"},{"content":" When I arrived in Cape Town International Airport from Kenya, a Customs officer told me that I had only 7 days to legally be in South Africa. This was actually my second entry into South Africa during the year, and I had unwittingly started a 90-day timer on my Visitor’s Visa when I arrived 83 days prior and spent four weeks in the country. After my first visit in 2021, I went on a tour of Ethiopia, Rwanda, and Kenya, so I was expecting to be able to return to South Africa with a new 90-day visa based on this statement from the U.S. Department of State’ South African travel website:\n“TOURIST VISA REQUIRED: No, if visiting 90 days or less.”\nAlas, I had been too optimistic about the definition of a “visit.” Life would have been a lot simpler if I had read this on the South African government’s website:\n“On entry to South Africa, a visa is considered to be a visitor’s permit. The permit’s period of validity is calculated from the date of entry into the country\u0026hellip;”\nAnyhow, I was given two options at Customs:\nrisk being banned from South Africa by remaining in South Africa while applying to extend my Visitor’s Visa through the Department of Home Affairs leave South Africa within 7 days, and wait a year I had to figure out a game plan, fast. I couldn’t just go home, since I had to attend a funeral and a wedding in Cape Town in just a few weeks. Luckily, I found out you can stay in South Africa for more than 90 days out of the year if you pursue a third option that Customs did not mention: the Loophole.\nExtending Your Visitor’s Visa # If you want to go by the book, you can try to extend your Visitor’s Visa, but it’s going to be a bumpy ride. If you are American (or a resident of any other country found on this Visa Exemption List), extending your Visitor’s Visa is arguably the most “official” way to remain in South Africa, but I trust this option the least. If you want to know the option that I chose, make sure to read on to “The Loophole” section.\nDepartment of Home Affairs # Before I realized that the Department of Home Affairs would do nothing for me, I went to their office in downtown Cape Town. The Cape Town Department of Home Affairs office makes the dreaded American DMV look efficient. People line up before 5 a.m. to wait for the office to open at 8 a.m. When I was waiting in line, fights broke out in the line between people who had jumped the line and people who had waited.\nMe acting casual at the front of the line into the Department of Home Affairs (a spot in line that I had paid someone for), while fights about line jumping broke out behind me.\nWhen I finally arrived inside of the Department of Home Affairs, several officers told me that the branch does not handle visas whatsoever. If I wanted to extend my Visitor’s Visa, I would have to go to the office of VFS Global.\nVFS Global # The Department of Home Affairs branch has outsourced their visa administration to VFS Global. Even this Department of Home Affairs website says to go to VFS Global.\nYou are technically supposed to submit an application for an extension of your Visitor’s Visa at least 60 days before its expiration (source); however, there are official service providers, including VFS Global, that have state-sanctioned ways to circumvent this rule.\nThey’ll instruct you to assemble a set of forms constituting an “application” for a visa renewal which will be sent off to the Department of Home Affairs in Pretoria. If you are in Cape Town, your only option is to submit this application to VFS Global; do NOT go to the Department of Home Affairs office.\nIn my case, I spent two days at VFS Global because their web portal was unstable due to load shedding, but with only a couple of days left on my visa and no guarantee that my application would be approved, I did not want to risk being banned for overstaying my permitted time in South Africa. Instead of following through with the application, I pursued the Loophole.\nThe Loophole # In order to stay in South Africa on a Visitor’s Visa without applying for a visa extension, follow the Loophole. The Loophole is actually two ways that allow you to leave South Africa and come back during the same year. I tried one way, and it worked! I have not tried the other way, so I cannot vouch for it with personal experience; however both of these methods were explicitly outlined to me and confirmed by multiple VFS Global and Department of Home Affairs officers in Cape Town and Johannesburg:\nLeave South Africa During Your First 90 Days and Come Back After Your First 90 Days # This option is apparently very common for people who try to stay in South Africa with Visitor’s Visas because it is much, much easier. You just need to leave South Africa before your initial 90 days has lapsed and return after your initial visa has expired. This means you can fly to Kenya, Dubai, Norway, etc. and allow your visa to expire, then return! One Department of Home Affairs officer told me not to try to re-enter South Africa from a bordering country, because South Africa is starting to crack down on “border hopping.”\nI received a second visa by going back to America for a week, then returning to South Africa a few days after my first visa had lapsed. I believe I could have come back during the initial 90 days and had the same result, which would have fallen into the following category.\nLeave South Africa and Come Back During Your First 90 Days # If you do not have an extension, you must leave South Africa before your Visitor’s Visa expires. If you go back to a country where you are a Resident (or higher status, e.g. citizenship), you can return to South Africa before your initial 90 days has lapsed and receive a fresh 90 days and a new Visitor’s Visa. This method was explained to me by two Customs officers upon entry to South Africa and verified by a VFS Global agent.\nConclusions # Waste # I won’t answer whether the Loophole should or should not exist in general. I used it to continue contributing to the South African tourism industry while spending valuable time with family. The Loophole was godsend for those reasons, but it has a darkside. Through the Loophole, I wastefully spewed out tons of plane emissions, spent thousands of dollars, and spent dozens of hours that, at the end of the day, I could have saved by just staying in South Africa instead of flying to America for a week.\nRoom to Improve # The magnitude of the rigamarole I encountered during the whole experience reinforced my belief that the South African Department of Home Affairs desperately needs new tech solutions, more transparency and accountability, and fewer chokepoints. They should strive to be more like Estonia, which has embraced entirely remote applications for Digital Nomad Visas.\nVFS Global, which has absorbed so much of the Department of Home Affair’s charter, is infuriatingly disorganized, slow, and outdated. Still, the desperation that pervades the sterile VFS Global office, where South Africans and VFS Global employees are trying their best to tend to everyone’s needs, showed me a unique and somewhat beautiful glimpse of humanity.\nWhy # I spent hours on the Department of Home Affairs’ website and reading through South African immigration and visa laws, and there is nothing online that explains how to extend your time in South Africa on a Visitor’s Visa. While there are other independent websites that “fill in the gaps” for this information, you cannot validate this information on an official government website or in legislative text. This inspired me to share my experience, which I hope helps you.\nDisclaimer # This article is not legal advice. I followed the advice of officers in the Department of Home Affairs and VFS Global. This article is purely to share my personal experience with nebulous South African immigration policies.\n","date":"21 April 2022","externalUrl":null,"permalink":"/how-i-got-another-90-days-in-south-africa-and-how-you-can-too-ac4efd72e266/","section":"Posts","summary":"When I arrived in Cape Town International Airport from Kenya, a Customs officer told me that I had only 7 days to legally be in South Africa. This was actually my second entry into South Africa during the…","title":"How I Got Another 90 Days in South Africa, And How You Can, Too","type":"posts"},{"content":"","date":"21 April 2022","externalUrl":null,"permalink":"/tags/immigration/","section":"Tags","summary":"","title":"Immigration","type":"tags"},{"content":"","date":"21 April 2022","externalUrl":null,"permalink":"/tags/south-africa/","section":"Tags","summary":"","title":"South Africa","type":"tags"},{"content":"","date":"2 January 2022","externalUrl":null,"permalink":"/tags/ethiopia/","section":"Tags","summary":"","title":"Ethiopia","type":"tags"},{"content":" I arrived in Ethiopia just as the city was preparing for the newly elected administration to enter office. Armed soldiers in blue camouflage uniforms were taking up posts at every street corner to make sure there was no funny business during processions in Meskel Square. Dignitaries had come in from all over the world to recognize the new government, so large parts of Kirkos were blocked off from traffic for their security. And as a tourist who was trying to see the sites, that meant I was stuck in taxis many mornings, which led to long conversations with taxi drivers.\nI recorded the celebrations from a passing truck as an excuse to capture a few seconds of footage of the armed soldiers. They shouted at me to leave after this.\nI briefed countless taxi drivers about my family and life in America in return for their stories. Some drivers were very friendly, and several had a bone to pick with me about my country’s meddling in Ethiopia. Biden’s administration was drafting sanctions against Ethiopia for its failure to deescalate the conflict in Tigray and for preventing the delivery of US-funded humanitarian aid to the region. When it comes to seeing things from the Ethiopian nationalist point of view, I can’t thank Tewodros, my impassioned Ride driver, enough for the low-down. Tewodros drove me to the Lucy exhibition, which lives in the National Museum of Ethiopia. After some chit-chat about tourist attractions, a tank blocked our view of the road, prompting Tewodros’ explanation for the sorry state that Addis Ababa appeared in.\nHe pointed out shuttered businesses along the road whose owners were forced to close their shops because of their public support for Tigray. He told me that only a week before I arrived, officials in the central government who are supporters of Tigray were jailed, and that he hoped that their disablement would lead to a sooner victory over the rebellion. Finally, he explained that America was perpetuating this war by threatening sanctions on an already vulnerable Ethiopia while providing vehicles, food, and medical supplies to Tigrayan rebels under the guise of humanitarian aid to slow Ethiopia’s construction of the Grand Ethiopian Renaissance Dam (the same one that former President Trump warned the Egyptians would bomb).\nYou find tanks and USAID vehicles all throughout Addis Ababa.\nMy prior understanding was that Eritrean and Ethiopian forces were collaborating to cut off American aid in order to manufacture a humanitarian crisis and cripple the Tigray region. Towedros’ perspective was a complete reversal of that. I felt an obligation to tell him that he was being fed fake news and that he shouldn’t think of America so negatively, but I literally didn’t know any better. Plus, my source of truth was the Western media. While the rest of Africa was being plundered by Western powers, Ethiopia uniquely resisted colonization twice, so you can understand why their history might have sowed a general distrust of foreign influence. I decided that instead of arguing, I should just look up the validity of his claims.\nWhat is USAID doing in Ethiopia?\nUNITED STATES PROVIDES ADDITIONAL HUMANITARIAN AID AND AIRLIFTS RELIEF SUPPLIES TO HELP PEOPLE AFFECTED BY THE CRISIS IN NORTHERN ETHIOPIA🇺🇸 USAID press release\nIs the United States providing supplies to Tigrayan rebel troops?\nADDIS DIALOGE , EXCLUSIVE INTERVIEW WITH USAID MISSION DIRECTOR TO ETHIOPIA SEAN JONES |\n🇪🇹 Ethiopian Broadcasting Company is a state-run media channel.\n","date":"2 January 2022","externalUrl":null,"permalink":"/my-taxi-driver-introduction-to-the-tigray-war-2eaf42a78b99/","section":"Posts","summary":"I arrived in Ethiopia just as the city was preparing for the newly elected administration to enter office. Armed soldiers in blue camouflage uniforms were taking up posts at every street corner to make sure there was no…","title":"My Taxi Driver Introduction to the Tigray War","type":"posts"},{"content":"","date":"2 January 2022","externalUrl":null,"permalink":"/tags/tigray/","section":"Tags","summary":"","title":"Tigray","type":"tags"},{"content":"","date":"26 December 2021","externalUrl":null,"permalink":"/tags/bitcoin/","section":"Tags","summary":"","title":"Bitcoin","type":"tags"},{"content":"","date":"26 December 2021","externalUrl":null,"permalink":"/tags/cryptocurrency/","section":"Tags","summary":"","title":"Cryptocurrency","type":"tags"},{"content":"","date":"26 December 2021","externalUrl":null,"permalink":"/categories/data-science--finance/","section":"Categories","summary":"","title":"Data Science \u0026 Finance","type":"categories"},{"content":"","date":"26 December 2021","externalUrl":null,"permalink":"/tags/machine-learning/","section":"Tags","summary":"","title":"Machine Learning","type":"tags"},{"content":" If you knew exactly when each price dip for Bitcoin was going to occur, you could make tens of thousands in one day. I don’t know what Bitcoin is about to do, but I thought a machine just might… anyhow, here’s why I won’t make the mistake of algorithmic trading trading again.\nCryptohopper # Cryptohopper is an online auto-trading bot service that automatically make trades between asset pairs (for example, BTC-USD) on various cryptocurrency exchanges, such as Coinbase, Binance, Kraken, you name it. The service they offer is very reliable and secure–ultimately, you are in control of the funds you allocate for Cryptohopper to trade with.\nHere’s where Cryptohopper is dangerous–in order to get respectable gains, you need to employ an algorithm for your bot to make trades that you never would have thought of. People make these algorithms for their full-time jobs. If you make one of these algorithms yourself, you had better know what you are doing. I certainly don’t, so I subscribed to one of the most popular and high-rated algorithms on the site: Profit Hero.\nMy message to you (Source: University of Alaska Fairbanks)\nProfit Hero, like the other most popular algorithms on Cryptohopper, is configured to automatically trade on dozens of cryptocurrencies. These include the top twenty coins ranked by market cap along with dozens of other obscure coins you have probably never heard of.\nToo Good to Be True # I was using Cryptohopper to be my FOMO-prophylactic; that is, I wanted a bot to buy into coins before they became big. But here’s the thing–Cryptohopper is built to be an active-trading platform. A bot might make thousands of trades in a single day. The algorithm I had configured my bot to run was a black box, so I just had to trust that it would deliver the results Cryptohopper’s marketplace had promised about it.\nCryptohopper shows stats about algorithmic signals and strategies. These can be manipulated by developers to show higher profit values by evaluating algorithm performance over a time range during a bullish market.\nThese results reflect the average profit a user can experience while running the algorithm; however, Cryptohopper’s definition of profit can be misleading. When you set up an auto-trading bot, you dump a specific amount of cash into the bot’s reserves, and Cryptohopper will use those reserves to purchase cryptocurrencies on your behalf. By allocating that cash to Cryptohopper, you are essentially investing that money into a mixed and ever-changing bag of cryptocurrencies. What you should compare your “profits” to is the amount of money you would have gained if you had simply invested the same amount of money into a cryptocurrency (or portfolio of cryptocurrencies) and just left it to ride the volatile waves of the market. And when Cryptohopper does not address that point, it fails to educate its users.\nSour Grapes? Nope. # My complaint is not that I lost money when I put funds into Cryptohopper. I did make profits, but when I compared these gains to what I would have gained if I had just bought the same amount of Bitcoin, Ethereum, or Litecoin with the money I had indirectly invested (via Cryptohopper) into a random assortment of obscurecoins, it turned out that my bot had underperformed compared to these markets.\nTo verify this, I wrote a script using the Binance API that evaluated what I would have profited from a buy-and-hold strategy during the same timeframe during which they entrusted a bot to automatically handle my portfolio. I then invited other members of the community to run my script, specifying the assets that they traded… lo and behold, their bots’ smart algorithms were also doing worse for their portfolios than a simple holding strategy.\nA closer look at the algorithm marketplace revealed that many of the most popular and “profitable” algorithms had deceptively started measuring their performance when the cryptocurrency market as a whole was on a bullish tear. The fact that these algorithm providers were attributing potential gains to their algorithms’ intelligence is just not right.\nConclusion # Cryptohopper has built an impressive and democratic tool. Algorithmic trading used to be reserved for well-capitalized firms that could hire engineers to analyze the market and make trades based on signals that only machines would be able to detect. Now, Cryptohopper has made it so anyone with a bit of disposable cash can get started auto-trading today.\nWhile I won’t be using Cryptohopper again anytime soon, I appreciated that it encourages you to jump into the deep end of trading and to start learning about altcoins. You can easily end up owning a collection of volatile assets that might be worth less than shitcoin, but at least you would be participating in the market and a step further in your journey of crypto investment.\n","date":"26 December 2021","externalUrl":null,"permalink":"/my-foray-into-the-world-of-cryptocurrency-auto-trading-b78d4ac85b93/","section":"Posts","summary":"If you knew exactly when each price dip for Bitcoin was going to occur, you could make tens of thousands in one day. I don’t know what Bitcoin is about to do, but I thought a machine just…","title":"My Foray into the World of Cryptocurrency Auto-Trading","type":"posts"},{"content":"","date":"19 December 2021","externalUrl":null,"permalink":"/tags/aviation/","section":"Tags","summary":"","title":"Aviation","type":"tags"},{"content":"","date":"19 December 2021","externalUrl":null,"permalink":"/tags/carbon-offsetting/","section":"Tags","summary":"","title":"Carbon Offsetting","type":"tags"},{"content":"","date":"19 December 2021","externalUrl":null,"permalink":"/categories/climate--sustainability/","section":"Categories","summary":"","title":"Climate \u0026 Sustainability","type":"categories"},{"content":"","date":"19 December 2021","externalUrl":null,"permalink":"/tags/net-zero/","section":"Tags","summary":"","title":"Net Zero","type":"tags"},{"content":"","date":"19 December 2021","externalUrl":null,"permalink":"/series/road-to-personal-net-zero/","section":"Series","summary":"","title":"Road to Personal Net Zero","type":"series"},{"content":" I offset all of the pollution I generated from flying. Here’s how and why. # Offsetting my flight carbon emissions is my first step in reaching Personal Net Zero.\nWhy # My flight emissions are currently far and away the biggest source of my emissions. I typically fly more than five round trips per year and spew out roughly two metric tons of carbon dioxide, which is a lot more than the average American, who falls in the top fifth percentile of global emitters.\nMy personal transportation emissions are way higher than the average American’s. Source: CarbonFootPrince.com\nFlying accounts for the brunt of my personal transportation emissions. Source: CarbonFootPrince.com\nTransportation accounts for the brunt of my annual emissions. Source: Wren.co calculator.\nFlying spews out tremendous amounts of CO2 and threatens to undermine carbon reductions in other sectors, like electrifying passenger vehicles. Although electric planes are in our future, I won’t hold out for their commercial viability and overhaul of the aviation industry.\nHow # I wish there was a way to see every flight I have taken on one central dashboard that I could just request my flight carbon data from. But the airline industry’s customer-facing tech is still in the early 2000s, so I went back to basics with a spreadsheet. In this live spreadsheet, you can track how much carbon I’ve emitted by flying and how much carbon I’ve offset.\nDaniel Kawalsky Flight Carbon Offset Tracker\ndocs.google.com\nWait, how? # Data: Let’s say I was gathering my flights from 2018. I typed this into my email search bar:\nflight after:2017/12/1 before:2019/1/1 and BOOM! I had everything I needed. I asked my parents about a list of flights I knew they booked for me when I was a kid.\nCalculation: I made this Google Sheets template:\nTemplate of Flight Carbon Offset Tracker\ndocs.google.com\nwhich aggregates each of the flight entries from each year into one sheet. I calculated my emissions for each flight using this tool:\nICAO Carbon Emissions Calculator\nwww.icao.int\nThen in the spreadsheet, I track my offset purchases in the “Offsets Overview” sheet, which gives me color-coded feedback on which years’ emissions I have and haven’t offset.\nCompensation: When I set out to offset my lifecycle carbon emissions, I determined that Carbonfund.org is the best service for me to purchase through. I’m going to continue with them.\nThoughts # Q: How can the aviation industry emit less greenhouse gas?\nA: The fastest ways aviation could cut emissions\nQ: Is there a quick way to accurately measure my personal transportation carbon footprint?\nA: Read Calculate your Transportation Emissions from your Location History\nQ: Is there an easier way to gather all of my flight data?\nA: AwardWallet is an awesome tool for tracking rewards, like airline miles, hotel points, credit card points, and more. You can even import all of your flight data from your email into AwardWallet, if you’re cool with that.\n","date":"19 December 2021","externalUrl":null,"permalink":"/road-to-personal-net-zero-offsetting-lifecycle-carbon-emissions-from-flying-cbbc0fecf722/","section":"Posts","summary":"I offset all of the pollution I generated from flying. Here’s how and why.","title":"Road to Personal Net Zero: Offsetting Lifecycle Carbon Emissions from Flying","type":"posts"},{"content":"","date":"12 December 2021","externalUrl":null,"permalink":"/tags/carbon-footprint/","section":"Tags","summary":"","title":"Carbon Footprint","type":"tags"},{"content":" I know I might never reach Net Zero, but here’s to trying. # Photo by MaxPixel\nWhy # Climate change will throw some countries into crises, and it can completely erase some countries from the map. COP26 showed that America will not follow through with its pledge to sufficiently help repair the damages suffered by countries under human-amplified global warming, which America signed off on with a fat John Hancock. On the flip side, we did contribute some 6% of the Least Developed Countries Fund, and maybe next year we’ll come back to COP27 with a bigger check.\nI like to complain about big business and politicians being slow to the mark, but that doesn’t go anywhere, because none of the people I have the attention of are big businessmen or politicians. And who am I to complain while I continue to fly for fun and buy single-use plastics for convenience?\nWith my own hypocrisy staring me in the face, I am setting out to offset my past and present carbon emissions.\nHow # It will be extremely difficult and time-consuming to calculate my exact carbon footprint across every category of housing, transportation, services, food, and goods, but there are myriad decent estimation tools to simplify this.\nTransportation accounts for the brunt of my current annual emissions. Source: Wren.co calculator.\nA nice breakdown of emission sources, further broken down at: PBS\nPast Emissions # The main challenge will be accounting for all of my past emissions and lifestyle. For example, the way I buy and hold onto clothing has changed since I was a kid, when the main reason to get new clothing was my growing body. And whereas before, I used to be shuttled everywhere by my parents, now I take a variety of modes of transportation.\nCompensation # Carbon offsets are controversial tool, and many people even consider the concept of paying someone else to clean up after your mess a distracting ruse. I agree; my philosophy is that you can only really fix the climate problem by disassembling the carbon-emitting system and retrofitting it with sustainable parts in-place. But compensation and and fixing are two different things, and while carbon offsets do reduce the amount of carbon in the atmosphere, I’ll start with cleaning up my own emissions.\nThoughts # Q: Are carbon offsets legit?\nA: Can you really negate your carbon emissions? Carbon offsets, explained.\nQ: What’s a good carbon offset charity?\nA: Carbonfund.org is a 501(c)(3) organization with a fantastic Charity Navigator score.\nQ: How can you donate to fixing the carbon-emitting institutions?\nA: Want to fight climate change effectively? Here’s where to donate your money.\n","date":"12 December 2021","externalUrl":null,"permalink":"/road-to-personal-net-zero-start-here-1805b192f7d5/","section":"Posts","summary":"I know I might never reach Net Zero, but here’s to trying.","title":"Road to Personal Net Zero: Start Here","type":"posts"},{"content":" Calculate your Transportation Emissions from your Location History # Your location history says a lot about your carbon footprint. # Google richly documents your life story in the cloud, and along with the many data points Google tracks is your “Location History,” which is basically a GPS trail of where you’ve brought Google Maps. If you have the app installed on your phone, as long as you have Location History set to “on,” this trail might be a continuous line of everywhere you go. Google infers the mode of transportation you took to get there, too.\nYou can start to see how much value from you, a consumer, they can extract from your data. If you feel like an unpaid Google intern, feel better knowing that you, too, can gain rich insights into your location history.\nWhen you request your data from them, Google will deliver a history of you in a neatly zipped file. You can take this file and head to the Carbon Foot Prince. Carbon Foot Prince is an open-source website that runs calculations on your file completely offline to answer questions like, “how far did I drive in 2020?” and “how much less carbon dioxide did I personally exhaust by staying home during the COVID-19 pandemic?”\nCurious? Try it out!\nRequest your Data # Go to Google Takeout Choose “Deselect all” Reselect “Location History” (scroll or Ctrl/Cmd+F to find it) Export the data in JSON format as a 2GB .zip Wait a while — you might get the zip file by email/Google Drive, etc. immediately, otherwise it could take a few days. Import into Carbon Foot Prince # Go to carbonfootprince.com Click Start (or Learn More, first) Import the .zip file — et voilà! Explore # Calculate and interrogate how “getting around” warms the planet and how to help make movement more sustainable (and equitable) for everyone.\nLet me know if you find it useful! Leave comments here, in the repository, or reach out at contact @ carbonfootprince . com (remove spaces).\n","date":"27 January 2021","externalUrl":null,"permalink":"/calculate-your-carbon-emissions-by-plane-train-and-car-and-more-from-your-location-history-c86852c4d26f/","section":"Posts","summary":"Calculate your Transportation Emissions from your Location History","title":"Calculate your Carbon Emissions by Plane, Train, and Car (and more) from your Location History","type":"posts"},{"content":"","date":"27 January 2021","externalUrl":null,"permalink":"/tags/data/","section":"Tags","summary":"","title":"Data","type":"tags"},{"content":"","date":"21 April 2019","externalUrl":null,"permalink":"/tags/classification/","section":"Tags","summary":"","title":"Classification","type":"tags"},{"content":"","date":"21 April 2019","externalUrl":null,"permalink":"/tags/film/","section":"Tags","summary":"","title":"Film","type":"tags"},{"content":" Why I’m bad with names, but it turned out for the best. # Nobody: The credits: Tom Skerritt, Sigourney Weaver, Veronica Cartwright, Harry Dean Stanton, John Hurt, Ian Holm, Yaphet Kotto, Bolaji Badejo, Helen Horton, Eddie Powell, Gordon Carroll, David Giler, Walter Hill, Ivor…\nBackground # The credit sequence is a sad excuse for an effort to connect people who “touch the film” with those who watch it in its final form. It’s such a dull format that it has become customary for theaters to brighten the room for moviegoers to comfortably leave the theater. I want to curb this custom and get people invested in the countless names that scroll by. In my previous article, I explain why I ran with tagging names with nationality — put simply, strangers instantly connect when they find out they share a common origin story.\nI want to make something like the sample credit sequence below. Let’s call this format “flag-tagged.”\nThe flag-tags are randomly assigned and do not accurately represent ethnonationality.\nThis article is about how one developer’s quick-and-dirty tech solutions can’t change a culture but instead act as a powerful tool to expose and correct naïvete. Whose? Mine, for starters.\nSituation ‘A’ — I am the go-to credits-processing guy.\nSituation ‘B’ — Where I am now, developing a client-side post-production solution.\nI will start off by addressing the hopelessness of my original project. Ideally, I would integrate nationality information into films themselves (flag-tagging) to make the credit sequence, hopefully, more enticing than the bathroom at the end of a movie. Since I don’t run an established editing operation and I’m not given footage as a matter of course for rapid credits-processing, I’m more inclined to write a script to automate flag-tagging by (1) programmatically find names in streamed video and (2) insert animated graphics to retroactively assign nationality information. This is above my paygrade.\nOver time, the credits grow longer and list more distinct roles. [1]\nBut I trudged onward, dual-wielding classification tools and public databases in hopes of revealing an ethnonational narrative in the long history of feature films. The history of feature films is well over 100 years-old, involving roughly 300,000 credited names. It was a huge process of augmenting name data, and the challenges I came across led me to narrow my scope to a small, intriguing subset of people.\nPlease Onomastics Responsibly # From the name Mark Hamill, you cannot definitively gather ethnic nationality, but you can assign a probability that the name was given by parents of a certain ethnonational lineage. This kind of task is perfectly suited for a classification algorithm, which dissects inputs and uses machine learning models to sort them into categories that most closely aligns with their attributes. Let’s see how NamSor predicts the heritage of the name of the famous American actor, which we know to be of English, Irish, Scottish, Welsh (paternal), and Swedish (maternal) descent.\nBaby Name Wizard and Ancestry serve as the base-truth origin for the first and last names. For “Mark Hamill, U.S.A.,” NamSor’s most confident predictions are “British” and then “(German).”\nSee that it correctly guessed British? That’s 1 for 1! Only 299,999 to go!\nNamSor’s Diaspora endpoint drives the ethnonationality classification.\nYet, we can’t rely upon NamSor to be 100% accurate, even if it was a magical API. That’s because NamSor assumes tradition as a rule — that parents name their children to reflect their heritage. NamSor has no idea that names it’s asked to classify might not be names given at birth.\nCall Me By That Name # Historically, Jews, Italians, and Poles in Hollywood changed their names to more pronounceable stage names, sometimes distancing themselves altogether from ethnic groups prone to discrimination by taking on a name with no trace of their born heritage.\nI don’t care what anyone says, Herschlag has a nice ring to it!\nI couldn’t go on labeling people with presumed ethnonationality knowing that some large chunk of the names I had access to through The Movie Database weren’t names that were given at birth. Lucky for me, IMBb had the birth and stage name of every person I was looking for, so I just had to identify every credited person in the 5–7 most popular movies of every year since 1900 (n≈100,000) who changed their name for better marketability in the industry in order to correct the “name-changers.” Without knowing the motivation of any individual’s name-change, and to eliminate false positives, I assumed that they made the switch for reasons other than the following:\nThey changed their name for marriage. (It’s actually more professional for showbiz women to keep their maiden name after marriage. These women went against the grain.) They go by their initials or a nickname. (Consider writer-actor B.J. Novak.) They gained an honorific. (Consider Sir Patrick Stewart.) What remained after that filter were ~7,000 significant name-changes, a substantial 7% of the sample and an interesting group to explore! I had some questions for the data before I dove in, including:\nCan we observe anglicization as a trend in name-changes?\nNo # Hypothesis: since a HUGE portion of the names in the population ended up British, you can observe a pattern of anglicization among name-changes in the American film industry.\nAnalysis: Using NamSor to determine someone’s actual ethnonationality is dumb. Instead, I used NamSor to do what it’s built for; I got “impressions” of the ethnonationality associated with someone’s former and current names to look for patterns in ethnocultural importance to the industry.\nToo much?\nFacts: As it turns out, there is a net-outflux of British-sounding names to elsewhere-sounding names. My hypothesis is WRONG. Check it out.\nWe’re only looking at the eastern hemisphere because ethnonationality, even in melting pots like the USA, is categorized primarily by geographical ties developed over the Modern Era, when populations were concentrated outside of the Americas.\nAbout the above visualization:\n• A red line means the name originates in Britain and terminates elsewhere.\n• A blue line means the name originates elsewhere and terminates in Britain.\n• A white line means there is no association with Britain. • The thickness of the line indicates the number of name-changes recorded in one year from one “ethnic nation” to another.\nThere’s a lot to see here, but I want to point your attention to the red and blue lines, specifically between 1900–1970. Over this period, there appears to be a mass exodus of British-sounding to Portuguese-sounding names. Contrary to the hypothesis, this shows that there is a tremendous de-anglicization effect underway. Maybe because people aimed to distinguish themselves from the showbiz crowd with ethnic-sounding names?\nConcluding points # After I saw my hypothesis was wrong, I felt that my project was at a dead-end. My goal was to deliver a product that could change the way people watch credits and I felt I had turned up with nothing. Even if I developed post-production software to change the visual content of the credit sequence, there are no guarantees that anyone would notice, let alone care that there is more to see after their movie has ended. Instead, I developed an independent website as a platform to present my progress as I explored and understood my dataset. This, of course, meant that anything I published was disassociated from the industry I was poking at. While this made my impact negligible, it forced me to find real value in the data and the tools I had on hand before I exposed something people would find interesting. The fun of exploring an untouched dataset quickly drained away when I found out my hypothesis was wrong, but I believe there might be more to extract from this group of name-changers than I originally thought.\nCould there be a Part III? Only time will tell… In the meantime, thanks for reading! If the population I observed or my analysis process is of any interest to you, please get in touch!\n[1] The Movie Database (~5 most popular feature films of each year)\nI took on this project with Jyotsna Pant, a graduate student, for Computing for Social Good at the University of Southern California.\n","date":"21 April 2019","externalUrl":null,"permalink":"/spicing-up-feature-film-credits-with-classification-part-ii-c715d8375975/","section":"Posts","summary":"Why I’m bad with names, but it turned out for the best.","title":"Spicing up Feature Film Credits with Classification — Part II","type":"posts"},{"content":"","date":"10 April 2019","externalUrl":null,"permalink":"/tags/data-science/","section":"Tags","summary":"","title":"Data Science","type":"tags"},{"content":" Spicing up Feature Film Credits with Classification — Part I # Why you’re better off leaving holes in “people data” unfilled and just leaving classification alone. # When you go to the movies, do you ever wish you could just push a button, skip the story, and watch the closing credits scroll by? If you answered “no,” you’re normal. Credits are boring.\nBut we take credits at the end of a movie for granted; after all, the credit sequence is factored into the running time of films. Giving credits a second thought, it seems bizarre that we cap a film, an artform that uses dramatic visuals and rhetoric to evoke a visceral response, with such a dull format as the credit sequence. Of course, the credits have earned their place in film as a traditional way of paying respect to its creators, sometimes even accompanied by thematic clips to make the last transition less stark. Everything before the credits is meant to be enjoyed passively, while everything after hardly resembles the art that precedes it — the traditional format suggests that you read a list of names and pay respect to each for the role they played in making a movie before it slides out of view. Only with time and repetition does the average moviegoer learn to associate names (especially those below the line) with style, ethos, and privilege.\nActors have the better end of the stick. Since their faces are projected in high detail while their voices project from surround sound speakers, you get familiar with the cast quickly; but, if you watched a movie for the second time, I bet you wouldn’t notice if all of the names of the crew roles were scrambled.\nTo try to close the “familiarity gap,” I thought that if I found the elements in names that help people recognize them, I could make credits more appealing for people. Personally, I remember people’s names better when I share experiences with them. Herein is the first complication: people will never meet most of the people whose names they see in front of them. The next apparent solution would be to bombard people with the public profiles built around these people, e.g. dynamically changing the videos of credit sequences and injecting IMDB biographies for each credited person. This seemed effective but too burdensome.\n“that” = reading the life story of every writer, sound designer, and costume writer of Transformers 7\nNext, I considered identity. If occupation and expertise aren’t interesting enough to keep people around for the credits, I figured a common background would be. Americans seem to light up when they meet someone who shares a common nationality, even if their ancestors immigrated 5 generations ago. I ran with this truism and decided to label every credited person I could find with their nationality. Then, I figured, I will have accounted for even the typical moviegoer who doesn’t care to look beyond the theater experience to “get to know” credited people.\n“Oh my god, I’m Irish, too! “\nI figured, since the “typical moviegoer” can’t know the ethnic background of each member of the credits, the labeling process should reflect the same naive perspective and make an educated geographical estimate based on the name itself. I figured wrong, but before I explain why, I’ll show you how far I took a hunch before it became glaringly obvious that my process was flawed.\nI followed the intuition people use to assume the ethnicity and gender of someone they have never met and turned to machine learning models that associate names with places using survey, census, and poll data.\n“Assume,” “ethnicity,” and “gender” have been used in the same sentence. Exercise caution.\nThere are many tools to tack on identifiers to names, namely NamSor, which draws on massive datasets to provide a best guess of someone’s gender and ethnicity based on solely their name. As sophisticated as NamSor has been engineered to work and as many onomasticians, linguists, and anthropologists as they consult, its (and any) AI classifier cannot operate with 100% accuracy. It’s impossible for two reasons:\nThe name sequence is not long enough. Take, for example, a similar problem of labeling a story with a genre. If the story went, “The person walked,” you’d be equally unconfident about placing it into any genre. But if the story went, “The girl forced trepid steps toward the dingey manor,” you’d presume this is the precursor of a horror story for a number of reasons, not the least of which is that manors only house evil. Much more goes into a name than ethnicity, and much more goes into ethnicity than someone’s name. Just as someone wouldn’t look exclusively to their name to determine how they identify themself, assigning identifiers from the outside-in isn’t a valid means of understanding a population. In the end, my plan to give more value to individual names than what appears on-screen caved under the responsibility to properly represent people. I think I had the right intentions, but my approach was all wrong. Rather than look for how people identified themselves, I thought, “this project calls for really good guessing,” and overlooked better solutions. Classification, as objective as it seems, is problematic when applied to qualitative human characteristics.\nIn my next article, I’ll show how NamSor performed by augmenting personal data with fine-grained attributes, like ethnicity, in terms of diaspora (where movie creators most likely came to the United States from). I’ll also show how this relates to base-truth information. Stay tuned!\n","date":"10 April 2019","externalUrl":null,"permalink":"/the-trouble-with-classification-312cc6fe8b79/","section":"Posts","summary":"Spicing up Feature Film Credits with Classification — Part I","title":"The Trouble with Classification","type":"posts"}]