sontek.net

Use asdf to manage Python, NodeJS, GoLang and more!

February 18, 2022
asdf is a general purpose version manager that can manage versions of most programming language runtimes through a set of plugins.
</article></li><li><article class="blog_hentry__PXeF7"><header><h3 class="entry-title"><a href="/blog/2022/intro_to_just">Automate project workflows with the command runner Just!</a></h3></header><br/><small class="util_lightText__xcqUE"><time dateTime="2022-02-26T00:00:00Z">February 26, 2022</time></small><div><p>I believe every project should have a CLI built around the standard workflows of developing on the project. Things like:</p> <ul> <li>Install dependencies</li> <li>Run tests</li> <li>Run linter and formatters</li> <li>Build project</li> <li>Start / Stop the docker environment</li> </ul> <p>The reason I think this is important is because it makes a nice consistent and discoverable entrypoint fo...</div></article></li><li><article class="blog_hentry__PXeF7"><header><h3 class="entry-title"><a href="/blog/2022/local_kubeadm_cluster">Running a kubernetes cluster locally with kubeadm</a></h3></header><br/><small class="util_lightText__xcqUE"><time dateTime="2022-04-17T00:00:00Z">April 17, 2022</time></small><div><p>I’m going to show you how to get a real kubernetes cluster setup locally on top of virtual machines! I’ll be using multipass but feel free to use virtualbox, proxmox, or whatever your favorite cloud provider is.</p> <p>kubeadm a production ready kubernetes install tool and I prefer to use it over minikube, kind, etc. because it gives you a more real world experience for <em>managing</em> the k...</div></article></li><li><article class="blog_hentry__PXeF7"><header><h3 class="entry-title"><a href="/blog/2023/local_k8s_with_kind">Running a kubernetes cluster locally with kind</a></h3></header><br/><small class="util_lightText__xcqUE"><time dateTime="2023-07-21T00:00:00Z">July 21, 2023</time></small><div><p>Previously I <a href="/blog/2022/local_kubeadm_cluster">showed</a> how to run kubernetes locally with <code>kubeadm</code> and VMs but sometimes that is overkill so I wanted to show how to run <a href="https://kind.sigs.k8s.io/">kind</a> which is "kuberetes in docker".</p> <h1>Creating your first cluster</h1> <p>kind is a very flexible way to run kubernetes locally and allows you to run single ...</div></article></li><li><article class="blog_hentry__PXeF7"><header><h3 class="entry-title"><a href="/blog/old/preparing_cloud_images_with_libvirt">Preparing custom images for OpenStack</a></h3></header><br/><small class="util_lightText__xcqUE"><time dateTime="2014-08-03T00:00:00Z">August 3, 2014</time></small><div><p>This article will show you how to use libvirt to create base images that can be uploaded to OpenStack.</p> <h1>Why would you want to do this?</h1> <p>Linux distributions like Fedora and Ubuntu already ship "cloud" images and most providers also have their own custom images for you to use, but I find it much more comforting to have full control of the software that is installed and I like the ab...</div></article></li></ul></div></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"id":["Linux"],"allTagData":{"Python":{"count":7,"posts":[{"id":["2022","intro_to_asdf"],"path":"2022/intro_to_asdf","contentHtml":"\u003cp\u003e\u003ca href=\"https://asdf-vm.com/\"\u003easdf\u003c/a\u003e is a general purpose version manager that\ncan manage versions of most programming language runtimes through a set\nof plugins.\u003c/p\u003e\n\u003ciframe width=\"854\" height=\"480\" src=\"https://www.youtube.com/embed/RTaqWRj-6Lg\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen\u003e\u003c/iframe\u003e\n\u003cp\u003eWith micro-services being all the rage and the ever changing landscape\nof the development world, it is rare to utilize a single version of\nlanguage runtime. Even when you want to upgrade from one to the other\nyou'll need both usable on your system at the same time.\u003c/p\u003e\n\u003cp\u003eI've used tools like \u003ccode\u003epyenv\u003c/code\u003e and \u003ccode\u003envm\u003c/code\u003e in the past when I needed to change\nversions depending on which project I'm contributing to. But with \u003ccode\u003easdf\u003c/code\u003e\nyou have one tool to rule them all!\u003c/p\u003e\n\u003ch2\u003eGetting Started\u003c/h2\u003e\n\u003cp\u003eThe first thing you need to do when working with \u003ccode\u003easdf\u003c/code\u003e is grab the\nplugins for the languages you are interested in working with. You can list\nwhat plugins are available:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf plugin list all\ngolang *https://github.com/kennyp/asdf-golang.git\ngolangci-lint https://github.com/hypnoglow/asdf-golangci-lint.git\nnodejs *https://github.com/asdf-vm/asdf-nodejs.git\npoetry *https://github.com/asdf-community/asdf-poetry.git\npython *https://github.com/danhper/asdf-python.git\nyarn *https://github.com/twuni/asdf-yarn.git\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOn the left will be the name of the plugin and on the right will be the repository\nwhere it lives. It'll me marked with an asterisk if you already have it installed.\u003c/p\u003e\n\u003cp\u003eTo install a plugin you say \u003ccode\u003easdf plugin add \u0026#x3C;plugin\u003e\u003c/code\u003e to get it installed. You can\nalso provide the repository where you want it pulled from, for example:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf plugin add nodejs https://github.com/asdf-vm/asdf-nodejs.git\n\u003e asdf plugin add python https://github.com/danhper/asdf-python.git\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will not give you any version of those languages, it is only installing the\nplugin that knows how to work with those languages. You are ready to pull down\nany versions you want at that point:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf install nodejs 14.19.0\n\u003e asdf install python 3.9.10\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce you have the versions installed you will be able to view them like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf list\ngolang\n 1.17.7\nnodejs\n --\u003cspan class=\"hljs-built_in\"\u003ehelp\u003c/span\u003e\n 12.22.10\n 14.19.0\n 16.14.0\n 17.5.0\npoetry\n 1.1.13\npython\n 3.9.10\nyarn\n 1.22.17\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eUsing the installed languages\u003c/h2\u003e\n\u003cp\u003eTo activate a specific version of a language you have you have three options:\u003c/p\u003e\n\u003ch3\u003eMake it global\u003c/h3\u003e\n\u003cp\u003eYou can make it global, meaning when you run the tool like \u003ccode\u003epython\u003c/code\u003e it'll use\nthis version for the system:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf global python 3.9.10\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch3\u003eMake it local\u003c/h3\u003e\n\u003cp\u003eYou can make it local, which means it will generate a file in the current\ndirectory named \u003ccode\u003e.tool-versions\u003c/code\u003e and so whenever you change into a directory\nit will activate the versions defined in there.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf \u003cspan class=\"hljs-built_in\"\u003elocal\u003c/span\u003e nodejs 12.22.10\n\u003e \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e .tool-versions \nnodejs 12.22.10\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe great thing about this is you can commit that file to git and then anyone\nwho checks out the project and uses \u003ccode\u003easdf\u003c/code\u003e will have the same versions activated!\u003c/p\u003e\n\u003ch3\u003eTemporary\u003c/h3\u003e\n\u003cp\u003eIf you want to activate a version of a language temporarily you can swap to it\nfor the current shell:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf shell golang 1.17.7\n\u003e \u003cspan class=\"hljs-built_in\"\u003eenv\u003c/span\u003e|grep -i ASDF\nASDF_GOLANG_VERSION=1.17.7\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIt sets an environment variable that will have preference over the file. If you\never wonder what versions a directory is using you can run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf current\ngolang ______ No version \u003cspan class=\"hljs-built_in\"\u003eset\u003c/span\u003e. Run \u003cspan class=\"hljs-string\"\u003e\"asdf \u0026#x3C;global|shell|local\u003e golang \u0026#x3C;version\u003e\"\u003c/span\u003e\nnodejs 12.22.10 .tool-versions\npoetry ______ No version \u003cspan class=\"hljs-built_in\"\u003eset\u003c/span\u003e. Run \u003cspan class=\"hljs-string\"\u003e\"asdf \u0026#x3C;global|shell|local\u003e poetry \u0026#x3C;version\u003e\"\u003c/span\u003e\npython 3.9.10 .tool-versions\nyarn 1.22.17 .tool-versions\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eConclusion\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://asdf-vm.com/\"\u003easdf\u003c/a\u003e is an AWESOME tool to utilize if you find yourself using many\ndifferent languages or many different versions of the same language. You should check it out\nand see if it can improve your workflow.\u003c/p\u003e","category":"Development","date":"2022-02-18T00:00:00Z","tags":["Python","NodeJS","GoLang","Linux"],"title":"Use asdf to manage Python, NodeJS, GoLang and more!"},{"id":["2022","learning_spanish"],"path":"2022/learning_spanish","contentHtml":"\u003cp\u003eI've been living in Puerto Rico for 4 years but two of those have been COVID and so I haven't been able to practice Spanish as much as I'd like. So to speed up my learning I've decided I want to watch a lot of spanish speaking television to start training my ears, but to do this I need a baseline of words I understand to be able to even know what they are saying!\u003c/p\u003e\n\u003cp\u003eLearning through apps like Duolingo, Drops, etc start with weird topics like vegetables that don't get you to a very good baseline for actually understanding daily conversations, so I think consuming TV is a better use of my time.\u003c/p\u003e\n\u003ch2\u003eSubtitles\u003c/h2\u003e\n\u003cp\u003eI've decided the way to understand what the best words to study are is to download every subtitle for every episode of a show I want to watch and then count each word. The more a word is spoken the more important it is for me to know it since I'll be hearing it a lot in the show.\u003c/p\u003e\n\u003cp\u003eI'm going to download subtitles from Netflix. Subtitles in Netflix are in WebVTT format, which looks like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003e\u003cspan class=\"hljs-number\"\u003e248\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e17\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e58.285\u003c/span\u003e --\u003e \u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e18\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01.163\u003c/span\u003e position:\u003cspan class=\"hljs-number\"\u003e50.00\u003c/span\u003e%,middle align:middle size:\u003cspan class=\"hljs-number\"\u003e80.00\u003c/span\u003e% line:\u003cspan class=\"hljs-number\"\u003e79.33\u003c/span\u003e% \nYo de verdad espero que ustedes\nme vean como una amiga, ¿mmm?\n\n\u003cspan class=\"hljs-number\"\u003e249\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e18\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01.247\u003c/span\u003e --\u003e \u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e18\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e02.539\u003c/span\u003e position:\u003cspan class=\"hljs-number\"\u003e50.00\u003c/span\u003e%,middle align:middle size:\u003cspan class=\"hljs-number\"\u003e80.00\u003c/span\u003e% line:\u003cspan class=\"hljs-number\"\u003e84.67\u003c/span\u003e% \nNo como una madrastra.\n\n\u003cspan class=\"hljs-number\"\u003e250\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e18\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e04.250\u003c/span\u003e --\u003e \u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e18\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e06.127\u003c/span\u003e position:\u003cspan class=\"hljs-number\"\u003e50.00\u003c/span\u003e%,middle align:middle size:\u003cspan class=\"hljs-number\"\u003e80.00\u003c/span\u003e% line:\u003cspan class=\"hljs-number\"\u003e84.67\u003c/span\u003e% \nYo nunca te vi como una madrastra.\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIt gives you a start time, end time, and the text on the screen. So my first process was parsing this format and just turning it into a list of words using https://github.com/glut23/webvtt-py.\u003c/p\u003e\n\u003ch3\u003eDummy parsing\u003c/h3\u003e\n\u003cp\u003eWhat I basically did was \u003ccode\u003etext.split(\" \")\u003c/code\u003e and started counting the words. This approach was quick and painless but it had a few downs falls. Some words \u003cem\u003elook\u003c/em\u003e the same when in reality they are not and so this meant I'd have to study every meaning of a word even if it was more rare.\u003c/p\u003e\n\u003cp\u003eAn example of this is the word \"como\", you can say:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eHaz como te digo: \"Do as I say\", where como means \"as\"\u003c/li\u003e\n\u003cli\u003ecomo tacos todos los dias: \"I eat tacos every day\", where como is a conjugated form of the verb \"to eat\"\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI need to know which version of a word is being used so I can count it properly.\u003c/p\u003e\n\u003ch3\u003eRegular Expressions are always the answer\u003c/h3\u003e\n\u003cp\u003eI couldn't figure out what the word was without it being in a complete sentence, but subtitles are fragments. They are split up into timings for displaying on the screen but they don't include entire sentences. For example, it might look like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003e\u003cspan class=\"hljs-number\"\u003e23\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e21.960\u003c/span\u003e --\u003e \u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e23.520\u003c/span\u003e position:\u003cspan class=\"hljs-number\"\u003e50.00\u003c/span\u003e%,middle align:middle size:\u003cspan class=\"hljs-number\"\u003e80.00\u003c/span\u003e% line:\u003cspan class=\"hljs-number\"\u003e84.67\u003c/span\u003e% \nSolo las que luchan por ellos\n\n\u003cspan class=\"hljs-number\"\u003e24\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e23.680\u003c/span\u003e --\u003e \u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e25.680\u003c/span\u003e position:\u003cspan class=\"hljs-number\"\u003e50.00\u003c/span\u003e%,middle align:middle size:\u003cspan class=\"hljs-number\"\u003e80.00\u003c/span\u003e% line:\u003cspan class=\"hljs-number\"\u003e84.67\u003c/span\u003e% \nconsiguen sus sueños.\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eI want to detect the start of a sentence and the end of a sentence and then combine it, so that you end up with \"Solo las que luchan por ellos consiguen sus sueños.\". My first thought was a regular expression on punctuation. This worked well \u003cem\u003emost\u003c/em\u003e of the time but there were enough exceptions to the rule that it broke often on generated a lot of broken sentences:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAbbreviations like \"EE. UU\" for estados unidos (united states)\u003c/li\u003e\n\u003cli\u003eEllipsis\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSplitting on spaces also didn't work for identifying the parts of speech since I needed the context around the word.\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/regex-extraction.png\"\u003e\n\u003c/center\u003e\n\u003ch2\u003eNatural Language Processing\u003c/h2\u003e\n\u003cp\u003eSo to solve my pain I decided to grab https://spacy.io/ and do some NLP on the subtitles so that I could identify the proper parts of speech and get an accurate representation of the words I needed to learn.\u003c/p\u003e\n\u003cp\u003eThe way spaCy works is you can send it a sentence and it'll return you a set of tokens:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e spacy\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003enlp = spacy.load(\u003cspan class=\"hljs-string\"\u003e\"es_core_news_sm\"\u003c/span\u003e)\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e[x.pos_ \u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e x \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e nlp(\u003cspan class=\"hljs-string\"\u003e\"Hola, como estas?\"\u003c/span\u003e)]\n[\u003cspan class=\"hljs-string\"\u003e'PROPN'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'PUNCT'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'SCONJ'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'PRON'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'PUNCT'\u003c/span\u003e]\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo now I could identify the parts of speech and pull sentences together through end of sentence punctation. The first thing I did was generate a CSV of sentences that looked like this:\u003c/p\u003e\n\u003ctable\u003e\n\u003ctbody\u003e\u003ctr\u003e\n\u003cth\u003esentence\u003c/th\u003e\n\u003cth\u003estart\u003c/th\u003e\n\u003cth\u003eend\u003c/th\u003e\n\u003cth\u003eshow\u003c/th\u003e\n\u003cth\u003efile\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eSi no, le voy a cortar todos los deditos\u003c/td\u003e\n\u003ctd\u003e00:00:20.605\u003c/td\u003e\n\u003ctd\u003e00:00:24.125\u003c/td\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003eEl marginal S02E02 WEBRip Netflix es[cc].vtt\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\u003c/table\u003e\n\u003cp\u003eOnce I had a CSV of sentences I could send those back through spaCy for NLP and then start counting words, to generate another CSV:\u003c/p\u003e\n\u003ctable\u003e\n\u003ctbody\u003e\u003ctr\u003e\n\u003cth\u003eword\u003c/th\u003e\n\u003cth\u003epos\u003c/th\u003e\n\u003cth\u003eshow\u003c/th\u003e\n\u003cth\u003efile\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003ea\u003c/td\u003e\n\u003ctd\u003eADP\u003c/td\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003eEl marginal S02E02 WEBRip Netflix es[cc].vtt\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003ecortar\u003c/td\u003e\n\u003ctd\u003eVERB\u003c/td\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003eEl marginal S02E02 WEBRip Netflix es[cc].vtt\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003etodos\u003c/td\u003e\n\u003ctd\u003ePRON\u003c/td\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003eEl marginal S02E02 WEBRip Netflix es[cc].vtt\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\u003c/table\u003e\n\u003cp\u003eFrom there I had all the data I needed! So now it was time to start doing some data analysis!\u003c/p\u003e\n\u003ch2\u003eData analysis\u003c/h2\u003e\n\u003cp\u003eUsing a jupyter notebook ( https://jupyter.org/ ) I grabbed pandas ( https://pandas.pydata.org/ ) and read in my CSVs to start analyzing the results.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-javascript\"\u003e\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e numpy \u003cspan class=\"hljs-keyword\"\u003eas\u003c/span\u003e np\n\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e pandas \u003cspan class=\"hljs-keyword\"\u003eas\u003c/span\u003e pd\n\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e matplotlib.\u003cspan class=\"hljs-property\"\u003epyplot\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eas\u003c/span\u003e plt\npd.\u003cspan class=\"hljs-title function_\"\u003eset_option\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'display.max_rows'\u003c/span\u003e, \u003cspan class=\"hljs-number\"\u003e1000\u003c/span\u003e)\nwords = pd.\u003cspan class=\"hljs-title function_\"\u003eread_csv\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'word_data.csv.gz'\u003c/span\u003e, compression=\u003cspan class=\"hljs-string\"\u003e'gzip'\u003c/span\u003e, delimiter=\u003cspan class=\"hljs-string\"\u003e','\u003c/span\u003e)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe words dataframe is built up out of the second table I showed above with just words and their parts of speech. I started off grouping the dataset by the word so I could get a count for how many times it was spoken in every series I parsed:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003egrouped_result\u003c/span\u003e = (words.groupby(words.word).size() \n .sort_values(\u003cspan class=\"hljs-attr\"\u003eascending\u003c/span\u003e=\u003cspan class=\"hljs-literal\"\u003eFalse\u003c/span\u003e) \n .reset_index(\u003cspan class=\"hljs-attr\"\u003ename\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e)\n .drop_duplicates(\u003cspan class=\"hljs-attr\"\u003esubset\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e))\n\ngrouped_result.head(300)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWhich returned a list of words and their count:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003e\t\u003cspan class=\"hljs-type\"\u003eword\u003c/span\u003e\tcount\n\u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\tque\t\u003cspan class=\"hljs-number\"\u003e94430\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\tno\t\u003cspan class=\"hljs-number\"\u003e75931\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e\ta\t\u003cspan class=\"hljs-number\"\u003e70968\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e3\u003c/span\u003e\tde\t\u003cspan class=\"hljs-number\"\u003e67982\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e4\u003c/span\u003e\tser\t\u003cspan class=\"hljs-number\"\u003e64226\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e5\u003c/span\u003e\tla\t\u003cspan class=\"hljs-number\"\u003e52143\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e6\u003c/span\u003e\ty\t\u003cspan class=\"hljs-number\"\u003e44390\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e7\u003c/span\u003e\testar\t\u003cspan class=\"hljs-number\"\u003e37819\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e8\u003c/span\u003e\tel\t\u003cspan class=\"hljs-number\"\u003e35920\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow I wanted to identify where my diminishing returns would be. Is there a set of words that I must learn because they are spoken so often that I wouldn't understand a conversation if they weren't in my vocabulary?\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/diminishing_returns.png\"\u003e\n\u003c/center\u003e\n\u003cp\u003eAs you can see in this chart, the usage for words drops off at around the ~200 mark. So there are basically 150 words I \u003cem\u003emust\u003c/em\u003e know and then the rest are equally important. I wasn't quite happy with this because some parts of speech are higher priority than others, for example I think having a strong understanding of the popular verbs will go a long way. So I also wanted to identify what are the most important verbs to learn:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003egrouped_verbs\u003c/span\u003e = (words[words.pos == \u003cspan class=\"hljs-string\"\u003e'VERB'\u003c/span\u003e].groupby([\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'pos'\u003c/span\u003e]).size() \n .sort_values(\u003cspan class=\"hljs-attr\"\u003eascending\u003c/span\u003e=\u003cspan class=\"hljs-literal\"\u003eFalse\u003c/span\u003e) \n .reset_index(\u003cspan class=\"hljs-attr\"\u003ename\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e)\n .drop_duplicates(\u003cspan class=\"hljs-attr\"\u003esubset\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e))\n\ngrouped_verbs.head(50)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWhich got me this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\t\u003cspan class=\"hljs-string\"\u003eword\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003epos\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ecount\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003etener\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e22072\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ehacer\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e14946\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eir\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e12570\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e3\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003edecir\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e11314\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e4\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003equerer\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e11083\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e5\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ever\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e10269\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e6\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eestar\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e9780\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e7\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003esaber\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e8704\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e8\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eser\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e7674\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e9\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003edar\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e5722\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e10\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003epasar\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e5528\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e11\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ehablar\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e5355\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e12\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003evenir\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e5145\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e13\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ecreer\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e4895\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e14\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003esalir\u003c/span\u003e \t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e3395\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eVerbs had a slightly different drop-off pattern when I targeted them directly:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/diminishing_verbs.png\"\u003e\n\u003c/center\u003e\n\u003cp\u003eI get a big bang for my buck by learning those top 40 verbs. Nouns on the other hand are much more spread out and most are evenly distributed:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-string\"\u003eword\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003epos\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ecount\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003egracias\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e4676\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003efavor\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e4625\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eseñor\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e4116\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e3\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003everdad\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e3566\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e4\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003evida\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2673\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e5\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ehombre\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2601\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e6\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003emadre\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2597\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e7\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003evez\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2537\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e8\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003etiempo\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2492\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e9\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ehijo\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2215\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/diminishing_nouns.png\"\u003e\n\u003c/center\u003e\n\u003cp\u003eSo then I thought to myself... How much of a show would I understand if I just learned these most important words? So I started by excluding some of the easy parts of speech and focused on the most important:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003efind_important_words\u003c/span\u003e = (words[~words.pos.isin([\u003cspan class=\"hljs-string\"\u003e'PRON'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'CONJ'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'ADP'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'ADV'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'SCONJ'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'AUX'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'INTJ'\u003c/span\u003e])].groupby([\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'pos'\u003c/span\u003e]).size() \n .sort_values(\u003cspan class=\"hljs-attr\"\u003eascending\u003c/span\u003e=\u003cspan class=\"hljs-literal\"\u003eFalse\u003c/span\u003e) \n .reset_index(\u003cspan class=\"hljs-attr\"\u003ename\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e)\n .drop_duplicates(\u003cspan class=\"hljs-attr\"\u003esubset\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e))\n\nfind_important_words.head(50)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe top 20 were all verbs except for \u003ccode\u003ebueno\u003c/code\u003e and \u003ccode\u003egracias\u003c/code\u003e. So now with my list of what I considered \"important words\" I plotted it to find what amount of words I wanted to learn:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/important_words.png\"\u003e\n\u003c/center\u003e\n\u003cp\u003eIt looks like 200 learned words would give me a reasonable amount of understanding for a series, so I decided to calculate how much of a series I would understand if I learned just those first 200 words:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003epercentages\u003c/span\u003e = {}\n\nfor show_name in words\u003cspan class=\"hljs-section\"\u003e['media']\u003c/span\u003e.drop_duplicates().values:\n \u003cspan class=\"hljs-attr\"\u003ewords_in_show\u003c/span\u003e = (words[words.media == show_name].groupby(words.word).size() \n .sort_values(\u003cspan class=\"hljs-attr\"\u003eascending\u003c/span\u003e=\u003cspan class=\"hljs-literal\"\u003eFalse\u003c/span\u003e) \n .reset_index(\u003cspan class=\"hljs-attr\"\u003ename\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e)\n .drop_duplicates(\u003cspan class=\"hljs-attr\"\u003esubset\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e))\n \n \u003cspan class=\"hljs-attr\"\u003etotal_words_handled\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\n for word in grouped_result\u003cspan class=\"hljs-section\"\u003e['word']\u003c/span\u003e\u003cspan class=\"hljs-section\"\u003e[:200]\u003c/span\u003e:\n \u003cspan class=\"hljs-attr\"\u003evalues\u003c/span\u003e = words_in_show[words_in_show.word == word][\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e].values\n\n if values.size \u003e 0:\n total_words_handled += values\u003cspan class=\"hljs-section\"\u003e[0]\u003c/span\u003e\n\n percentages\u003cspan class=\"hljs-section\"\u003e[show_name]\u003c/span\u003e = total_words_handled / words_in_show.sum().loc\u003cspan class=\"hljs-section\"\u003e['count']\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow I had a table that would show me what percentage of the spoken words were covered by the first 200 words in my list:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003ep_df\u003c/span\u003e = pd.DataFrame(percentages.items(), columns=[\u003cspan class=\"hljs-string\"\u003e'show'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'percentage'\u003c/span\u003e])\n\u003cspan class=\"hljs-attr\"\u003ep_df\u003c/span\u003e = p_df.sort_values(by=\u003cspan class=\"hljs-string\"\u003e'percentage'\u003c/span\u003e)\np_df\u003cspan class=\"hljs-section\"\u003e['percentage']\u003c/span\u003e = p_df\u003cspan class=\"hljs-section\"\u003e['percentage']\u003c/span\u003e * 100\n\u003cspan class=\"hljs-attr\"\u003epd.options.display.float_format\u003c/span\u003e = \u003cspan class=\"hljs-string\"\u003e'{:,.2f}%'\u003c/span\u003e.format\np_df\n\u003c/code\u003e\u003c/pre\u003e\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\u003ctable\u003e\n\u003ctbody\u003e\u003ctr\u003e\n\u003cth\u003eShow\u003c/th\u003e\n\u003cth\u003ePercentage\u003c/th\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eVerónica\u003c/td\u003e\n\u003ctd\u003e64.24%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl ciudadano ilustre\u003c/td\u003e\n\u003ctd\u003e65.28%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl Chapo\u003c/td\u003e\n\u003ctd\u003e66.68%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eNeruda\u003c/td\u003e\n\u003ctd\u003e66.89%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa casa de papel\u003c/td\u003e\n\u003ctd\u003e67.56%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl Ministerio del Tiempo\u003c/td\u003e\n\u003ctd\u003e68.03%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eClub de Cuervos\u003c/td\u003e\n\u003ctd\u003e68.19%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003e68.47%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eIngobernable\u003c/td\u003e\n\u003ctd\u003e68.59%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003ePablo Escobar\u003c/td\u003e\n\u003ctd\u003e70.20%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eFariña\u003c/td\u003e\n\u003ctd\u003e70.95\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa Reina del Sur\u003c/td\u003e\n\u003ctd\u003e71.52%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eGran Hotel\u003c/td\u003e\n\u003ctd\u003e73.15%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLas chicas del cable\u003c/td\u003e\n\u003ctd\u003e73.58%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eÉlite\u003c/td\u003e\n\u003ctd\u003e73.78%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa Piloto\u003c/td\u003e\n\u003ctd\u003e74.03%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl bar\u003c/td\u003e\n\u003ctd\u003e74.07%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa casa de las flores\u003c/td\u003e\n\u003ctd\u003e75.40%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eTarde para la ira\u003c/td\u003e\n\u003ctd\u003e75.59%\u003c/td\u003e\n\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\n\u003cp\u003eBut living in Puerto Rico, one thing I've realized is speed of speech is also important. I have a much easier time speaking with people from Colombia and Mexico than I do with Puerto Ricans because they speak so much faster. So even though I could understand 75% of \"Tarde para la ira\" if I learned the 200 words, I want to make sure they are speaking at a pace I could understand as well.\u003c/p\u003e\n\u003cp\u003eSo I loaded up the other CSV file that was the full sentences and added a \"time per word\" column:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-css\"\u003esentences = pd\u003cspan class=\"hljs-selector-class\"\u003e.read_csv\u003c/span\u003e('sentences\u003cspan class=\"hljs-selector-class\"\u003e.csv\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.gz\u003c/span\u003e', compression='gzip', delimiter=',', parse_dates=\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'start'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'end'\u003c/span\u003e]\u003c/span\u003e)\nsentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'total_time'\u003c/span\u003e]\u003c/span\u003e = (sentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'end'\u003c/span\u003e]\u003c/span\u003e - sentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'start'\u003c/span\u003e]\u003c/span\u003e)\u003cspan class=\"hljs-selector-class\"\u003e.dt\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.total_seconds\u003c/span\u003e()\nsentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'word_count'\u003c/span\u003e]\u003c/span\u003e = sentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'sentence'\u003c/span\u003e]\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.str\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.split\u003c/span\u003e()\u003cspan class=\"hljs-selector-class\"\u003e.str\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.len\u003c/span\u003e()\nsentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'time_per_word'\u003c/span\u003e]\u003c/span\u003e = sentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'total_time'\u003c/span\u003e]\u003c/span\u003e / sentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'word_count'\u003c/span\u003e]\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen I was able to have a speed rating for each show:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-scss\"\u003esentence_group = sentences\u003cspan class=\"hljs-selector-class\"\u003e.groupby\u003c/span\u003e([sentences.media])\nsentence_group\u003cspan class=\"hljs-selector-class\"\u003e.time_per_word\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.mean\u003c/span\u003e()\u003cspan class=\"hljs-selector-class\"\u003e.reset_index\u003c/span\u003e()\u003cspan class=\"hljs-selector-class\"\u003e.sort_values\u003c/span\u003e('time_per_word')\n\u003c/code\u003e\u003c/pre\u003e\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\u003ctable\u003e\n\u003ctbody\u003e\u003ctr\u003e\n\u003cth\u003emedia\u003c/th\u003e\n\u003cth\u003etime_per_word\u003c/th\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eGran Hotel\u003c/td\u003e\n\u003ctd\u003e0.58\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl Chapo\u003c/td\u003e\n\u003ctd\u003e0.59\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLas chicas del cable\u003c/td\u003e\n\u003ctd\u003e0.61\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eÉlite\u003c/td\u003e\n\u003ctd\u003e0.63\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eIngobernable\u003c/td\u003e\n\u003ctd\u003e0.64\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl Ministerio del Tiempo\u003c/td\u003e\n\u003ctd\u003e0.64\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eFariña\u003c/td\u003e\n\u003ctd\u003e0.65\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl ciudadano ilustre\u003c/td\u003e\n\u003ctd\u003e0.67\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eNeruda\u003c/td\u003e\n\u003ctd\u003e0.68\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa Piloto\u003c/td\u003e\n\u003ctd\u003e0.69\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa casa de papel\u003c/td\u003e\n\u003ctd\u003e0.70\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl bar\u003c/td\u003e\n\u003ctd\u003e0.70\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eVerónica\u003c/td\u003e\n\u003ctd\u003e0.72\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa Reina del Sur\u003c/td\u003e\n\u003ctd\u003e0.75\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eClub de Cuervos\u003c/td\u003e\n\u003ctd\u003e0.76\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003e0.76\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003ePablo Escobar\u003c/td\u003e\n\u003ctd\u003e0.77\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eTarde para la ira\u003c/td\u003e\n\u003ctd\u003e0.77\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa casa de las flores\u003c/td\u003e\n\u003ctd\u003e0.81\u003c/td\u003e\n\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\n\u003cp\u003eLuckily the two series that have the least amount of vocabulary also speak the slowest! So these will be the series I start with. The final question I wanted to answer is \"What are the top words I'm missing for a series\". Since I'll know 75% of the series from the top 200 words, I'm hoping there are some top words from a specific series that I can also learn to get an even higher understanding.\u003c/p\u003e\n\u003cp\u003eFirst, find which words are in each show but not in the top 200:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003emissing_words_by_show\u003c/span\u003e = {}\n\nfor show_name in words\u003cspan class=\"hljs-section\"\u003e['media']\u003c/span\u003e.drop_duplicates().values:\n \u003cspan class=\"hljs-attr\"\u003ewords_in_show\u003c/span\u003e = (words[words.media == show_name].groupby(words.word).size() \n .sort_values(\u003cspan class=\"hljs-attr\"\u003eascending\u003c/span\u003e=\u003cspan class=\"hljs-literal\"\u003eFalse\u003c/span\u003e) \n .reset_index(\u003cspan class=\"hljs-attr\"\u003ename\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e)\n .drop_duplicates(\u003cspan class=\"hljs-attr\"\u003esubset\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e))\n \n \u003cspan class=\"hljs-attr\"\u003efrequency_words\u003c/span\u003e = grouped_result[\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e][:\u003cspan class=\"hljs-number\"\u003e200\u003c/span\u003e]\n\n \u003cspan class=\"hljs-attr\"\u003emissing_words\u003c/span\u003e = words_in_show[~words_in_show.word.isin(frequency_words.values)]\n missing_words_by_show\u003cspan class=\"hljs-section\"\u003e[show_name]\u003c/span\u003e = missing_words\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen we were able to grab them per show:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-css\"\u003emissing_words_by_show\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'La casa de las flores'\u003c/span\u003e]\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.head\u003c/span\u003e(\u003cspan class=\"hljs-number\"\u003e50\u003c/span\u003e)\n\nword\tcount\n\u003cspan class=\"hljs-number\"\u003e31\u003c/span\u003e\tmamá\t\u003cspan class=\"hljs-number\"\u003e252\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e70\u003c/span\u003e\tflorerí\u003cspan class=\"hljs-selector-tag\"\u003ea\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e87\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e98\u003c/span\u003e\tperdón\t\u003cspan class=\"hljs-number\"\u003e56\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e102\u003c/span\u003e\tsea\t\u003cspan class=\"hljs-number\"\u003e54\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e116\u003c/span\u003e\tademás\t\u003cspan class=\"hljs-number\"\u003e44\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e126\u003c/span\u003e\tahorita\t\u003cspan class=\"hljs-number\"\u003e40\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e132\u003c/span\u003e\tcárcel\t\u003cspan class=\"hljs-number\"\u003e38\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e133\u003c/span\u003e\tfiesta\t\u003cspan class=\"hljs-number\"\u003e38\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo adding those few words to my vocabulary will also give me a better understanding of the series.\u003c/p\u003e\n\u003ch2\u003eConclusion\u003c/h2\u003e\n\u003cp\u003eI believe a data-driven approach to language learning will be an effective way to get me speaking better spanish. It was a ton of fun to play with spaCy, pandas, and jupyter as well!\u003c/p\u003e\n\u003cp\u003eI'll improve the data analysis over time as well but I do believe this is a pretty good starting point!\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/meme.png\"\u003e\n\u003c/center\u003e","category":"Development","date":"2022-04-30T00:00:00Z","tags":["Python","Pandas","NLP"],"title":"How to speak spanish like a colombian drug lord!"},{"id":["old","caesar_cipher_in_python"],"path":"old/caesar_cipher_in_python","contentHtml":"\u003cp\u003eI'm currently teaching my wife to code and one of the problems that we\nworked on to teach her some fundamental programming concepts was\nre-implementing the caesar cipher in python. It was fun not only to code\nbut to also start sending each other \"secret\" messages!\u003c/p\u003e\n\u003cp\u003eThe caesar cipher is a rather simple encoding, you just shift the\nalphabet a certain amount of characters. For example, if you are using a\nshift of 2:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003ea\u003c/span\u003e =\u003e c\n\u003cspan class=\"hljs-attr\"\u003eb\u003c/span\u003e =\u003e d\n\u003cspan class=\"hljs-attr\"\u003ey\u003c/span\u003e =\u003e a\n\u003cspan class=\"hljs-attr\"\u003ez\u003c/span\u003e =\u003e b\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eUsing this as an interview type question would provide a few interesting\nproblems and give you a good perspective on how good a developers\nproblem solving skills are and how knowledgeable they are in the\nlanguage of their choice.\u003c/p\u003e\n\u003cp\u003eThe first issue is to handle the beginning and end of the alphabet, if\nyou are encoding 'z' then you will have to start your shift on a. The\nsecond problem is to only encode letters since there was no ascii table\nto define in what order characters are shifted back in those times.\u003c/p\u003e\n\u003cp\u003eWithout using too much of the built in python niceties you could do\nsomething similar to this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-comment\"\u003e#!/usr/bin/python\u003c/span\u003e\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003edecode_shift_letter\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003ecurrent_ord, start, end, shift\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e current_ord - shift \u0026#x3C; start:\n new_ord = (current_ord + \u003cspan class=\"hljs-number\"\u003e26\u003c/span\u003e) - shift\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003echr\u003c/span\u003e(new_ord)\n \u003cspan class=\"hljs-keyword\"\u003eelse\u003c/span\u003e:\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003echr\u003c/span\u003e(current_ord-shift)\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003eencode_shift_letter\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003ecurrent_ord, start, end, shift\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e current_ord + shift \u003e end:\n new_ord = (current_ord - \u003cspan class=\"hljs-number\"\u003e26\u003c/span\u003e) + shift\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003echr\u003c/span\u003e(new_ord)\n \u003cspan class=\"hljs-keyword\"\u003eelse\u003c/span\u003e:\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003echr\u003c/span\u003e(current_ord+shift)\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003edecode\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003e\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e modify_input(\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift, decode_shift_letter)\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003eencode\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003e\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e modify_input(\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift, encode_shift_letter)\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003emodify_input\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003e\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift, shift_letter\u003c/span\u003e):\n new_sentence = \u003cspan class=\"hljs-string\"\u003e''\u003c/span\u003e\n\n \u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e letter \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e:\n \u003cspan class=\"hljs-comment\"\u003e# we only encode letters, random characters like +!%$ are not encoded.\u003c/span\u003e\n \u003cspan class=\"hljs-comment\"\u003e# Lower and Capital letters are not stored near each other on the\u003c/span\u003e\n \u003cspan class=\"hljs-comment\"\u003e# ascii table\u003c/span\u003e\n lower_start = \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'a'\u003c/span\u003e)\n lower_end = \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'z'\u003c/span\u003e)\n upper_start = \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'A'\u003c/span\u003e)\n upper_end = \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'Z'\u003c/span\u003e)\n current_ord = \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(letter)\n\n \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e current_ord \u003e= lower_start \u003cspan class=\"hljs-keyword\"\u003eand\u003c/span\u003e current_ord \u0026#x3C;= lower_end:\n new_sentence += shift_letter(current_ord, lower_start, lower_end, shift)\n \u003cspan class=\"hljs-keyword\"\u003eelif\u003c/span\u003e current_ord \u003e= upper_start \u003cspan class=\"hljs-keyword\"\u003eand\u003c/span\u003e current_ord \u0026#x3C;= upper_end:\n new_sentence += shift_letter(current_ord, upper_start, upper_end, shift)\n \u003cspan class=\"hljs-keyword\"\u003eelse\u003c/span\u003e:\n new_sentence += letter\n\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e new_sentence\n\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003eget_shift\u003c/span\u003e():\n \u003cspan class=\"hljs-keyword\"\u003etry\u003c/span\u003e:\n shift = \u003cspan class=\"hljs-built_in\"\u003eint\u003c/span\u003e(raw_input(\u003cspan class=\"hljs-string\"\u003e'What shift would you like to use?\\n'\u003c/span\u003e))\n \u003cspan class=\"hljs-keyword\"\u003eexcept\u003c/span\u003e ValueError:\n \u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'Shift must be a number'\u003c/span\u003e\n shift = get_shift()\n\n \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003enot\u003c/span\u003e (shift \u003e \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eand\u003c/span\u003e shift \u0026#x3C;= \u003cspan class=\"hljs-number\"\u003e25\u003c/span\u003e):\n \u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'Shift must be between 1 and 25'\u003c/span\u003e\n shift = get_shift()\n\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e shift\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003emain\u003c/span\u003e():\n \u003cspan class=\"hljs-keyword\"\u003etry\u003c/span\u003e:\n task = \u003cspan class=\"hljs-built_in\"\u003eint\u003c/span\u003e(raw_input(\u003cspan class=\"hljs-string\"\u003e'1) Encode \\n'\u003c/span\u003e+ \\\n \u003cspan class=\"hljs-string\"\u003e'2) Decode \\n'\u003c/span\u003e))\n \u003cspan class=\"hljs-keyword\"\u003eexcept\u003c/span\u003e ValueError:\n \u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'Invalid task, try again!'\u003c/span\u003e\n main()\n\n shift = get_shift()\n \u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e = raw_input(\u003cspan class=\"hljs-string\"\u003e'What message would you like to %s\\n'\u003c/span\u003e % (\u003cspan class=\"hljs-string\"\u003e'Encode'\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e task == \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eelse\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'Decode'\u003c/span\u003e))\n\n \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e task == \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e:\n \u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e encode(\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift)\n \u003cspan class=\"hljs-keyword\"\u003eelif\u003c/span\u003e task == \u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e:\n \u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e decode(\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift)\n\n\u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e __name__ == \u003cspan class=\"hljs-string\"\u003e'__main__'\u003c/span\u003e:\n main()\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis would prove that you are a decent problem solver and have enough of\nthe language to get things done but if you want to prove you have\nmastered the python language you might take advantage of some slicing\nand some methods out of the string module and change your code to look\nsomething like:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e string \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e letters, maketrans\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003edecode\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003e\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e modify_input(\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, -shift)\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003eencode\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003e\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e modify_input(\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift)\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003emodify_input\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003e\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift\u003c/span\u003e):\n trans = maketrans(letters, letters[shift:] + letters[:shift])\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e.translate(trans)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eDo get more information on string.letters and string.maketrans you can\nvisit their documentation\n[here](\u003ca href=\"http://docs.python.org/library/string.html\"\u003ehttp://docs.python.org/library/string.html\u003c/a\u003e)\u003c/p\u003e","category":"Development","date":"2010-11-15T00:00:00Z","tags":["Python","Interviews"],"title":"Caesar Cipher in Python"},{"id":["old","convert_a_string_to_an_integer_in_python"],"path":"old/convert_a_string_to_an_integer_in_python","contentHtml":"\u003cp\u003eA fun interview question some developers like to ask is to have you\nconvert ascii characters to an integer without using built in methods\nlike string.atoi or int().\u003c/p\u003e\n\u003cp\u003eSo using python the obvious ways to convert a string to an integer are\nthese:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-built_in\"\u003eint\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'1234'\u003c/span\u003e)\n\u003cspan class=\"hljs-number\"\u003e1234\u003c/span\u003e\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e string\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003estring.atoi(\u003cspan class=\"hljs-string\"\u003e'1234'\u003c/span\u003e)\n\u003cspan class=\"hljs-number\"\u003e1234\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe interesting thing here is finding out where on the ascii character\ntable the number is. Luckily python has this already built in with the\nord method:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-built_in\"\u003ehelp\u003c/span\u003e(\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e)\n\n\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(...)\n \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(c) -\u003e integer\n\n Return the integer ordinal of a one-character string.\n\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'1'\u003c/span\u003e)\n\u003cspan class=\"hljs-number\"\u003e49\u003c/span\u003e\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'2'\u003c/span\u003e)\n\u003cspan class=\"hljs-number\"\u003e50\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can see that the numbers are grouped together on the ascii table, so\nyou just have to grab '0' as the base and subtract the rest:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'1'\u003c/span\u003e)-\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'0'\u003c/span\u003e)\n\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo if we have the string '1234', we can get each of the individual\nnumbers by looping over it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003enum_string = \u003cspan class=\"hljs-string\"\u003e'1234'\u003c/span\u003e\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003enum_list = []\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003ebase = \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'0'\u003c/span\u003e)\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e num \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e num_string:\n\u003cspan class=\"hljs-meta\"\u003e... \u003c/span\u003e num_list.append(\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(num) - base)\n...\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e num_list\n[\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e, \u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e, \u003cspan class=\"hljs-number\"\u003e3\u003c/span\u003e, \u003cspan class=\"hljs-number\"\u003e4\u003c/span\u003e]\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003ebut now how to we combine all these together to get 1234? You can't\njust add them up because you'll just get 1+2+3+4 = 10.\u003c/p\u003e\n\u003cp\u003eSo, we have to get 1000 + 200 + 30 + 4, which is a simple problem to\nsolve. Its just number times 10 to the nth power, so the final solution\nis:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003enum = \u003cspan class=\"hljs-string\"\u003e'1234'\u003c/span\u003e\nnew_num = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\nbase = \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'0'\u003c/span\u003e)\n\n\u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e i,n \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003eenumerate\u003c/span\u003e(\u003cspan class=\"hljs-built_in\"\u003ereversed\u003c/span\u003e(num)):\n new_num += (\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(n) - base) * (\u003cspan class=\"hljs-number\"\u003e10\u003c/span\u003e**i)\n\n\u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e new_num\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis code is a little verbose though, lets make it a dirty nasty one\nliner!\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-built_in\"\u003esum\u003c/span\u003e([(\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(n)-\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'0'\u003c/span\u003e)) * (\u003cspan class=\"hljs-number\"\u003e10\u003c/span\u003e ** i) \u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e i,n \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003eenumerate\u003c/span\u003e(\u003cspan class=\"hljs-built_in\"\u003ereversed\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'1234'\u003c/span\u003e))])\n\u003cspan class=\"hljs-number\"\u003e1234\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e","category":"Development","date":"2010-10-28T00:00:00Z","tags":["Python","Interviews"],"title":"Convert a string to an integer in Python"},{"id":["old","tips_and_tricks_for_the_python_interpreter"],"path":"old/tips_and_tricks_for_the_python_interpreter","contentHtml":"\u003cp\u003eI have seen a lot of people switch over to using ipython, bpython, etc\nto get auto-complete support without realizing that the standard\ninterpreter does have this functionality.\u003c/p\u003e\n\u003cp\u003eTo enable auto-complete support in the python interpreter you need to\ncreate a python startup file that enables readline support. A python\nstartup file is just a bunch of python code that gets executed at\nstartup of the interpreter. To do this you just setup PYTHONSTARTUP in\nyour ~/.bashrc and then create a ~/.pythonrc.py file:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-comment\"\u003e#.bashrc\u003c/span\u003e\nPYTHONSTARTUP=~/.pythonrc.py\nexport PYTHONSTARTUP\n\n\u003cspan class=\"hljs-comment\"\u003e#.pythonrc.py\u003c/span\u003e\n\u003cspan class=\"hljs-keyword\"\u003etry\u003c/span\u003e:\n \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e readline\n\u003cspan class=\"hljs-keyword\"\u003eexcept\u003c/span\u003e ImportError:\n \u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e\"Module readline not available.\"\u003c/span\u003e)\n\u003cspan class=\"hljs-keyword\"\u003eelse\u003c/span\u003e:\n \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e rlcompleter\n readline.parse_and_bind(\u003cspan class=\"hljs-string\"\u003e\"tab: complete\"\u003c/span\u003e)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow when you are in python you have tab completion on importing, calling\nmethods on a module, etc.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e o\n\u003cspan class=\"hljs-built_in\"\u003eobject\u003c/span\u003e( \u003cspan class=\"hljs-built_in\"\u003eoct\u003c/span\u003e( \u003cspan class=\"hljs-built_in\"\u003eopen\u003c/span\u003e( \u003cspan class=\"hljs-keyword\"\u003eor\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e( os\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eI always end up using the pretty print module for viewing long lists and\nstrings in the interpreter so I prefer to just use it by default:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-comment\"\u003e# Enable Pretty Printing for stdout\u003c/span\u003e\n\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e pprint\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003emy_displayhook\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003evalue\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e value \u003cspan class=\"hljs-keyword\"\u003eis\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003enot\u003c/span\u003e \u003cspan class=\"hljs-literal\"\u003eNone\u003c/span\u003e:\n \u003cspan class=\"hljs-keyword\"\u003etry\u003c/span\u003e:\n \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e __builtin__\n __builtin__._ = value\n \u003cspan class=\"hljs-keyword\"\u003eexcept\u003c/span\u003e ImportError:\n __builtins__._ = value\n\n pprint.pprint(value)\n\nsys.displayhook = my_displayhook\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIt is also very useful to be able to load up your favorite editor to\nedit lines of code from the interpreter, you can do this by adding the\nfollowing into your ~/.pythonrc.py:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e os\n\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e sys\n\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e code \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e InteractiveConsole\n\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e tempfile \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e mkstemp\n\nEDITOR = os.environ.get(\u003cspan class=\"hljs-string\"\u003e'EDITOR'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'vi'\u003c/span\u003e)\nEDIT_CMD = \u003cspan class=\"hljs-string\"\u003e'\\e'\u003c/span\u003e\n\n\u003cspan class=\"hljs-keyword\"\u003eclass\u003c/span\u003e \u003cspan class=\"hljs-title class_\"\u003eEditableBufferInteractiveConsole\u003c/span\u003e(\u003cspan class=\"hljs-title class_ inherited__\"\u003eInteractiveConsole\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003e__init__\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003eself, *args, **kwargs\u003c/span\u003e):\n self.last_buffer = [] \u003cspan class=\"hljs-comment\"\u003e# This holds the last executed statement\u003c/span\u003e\n InteractiveConsole.__init__(self, *args, **kwargs)\n\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003erunsource\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003eself, source, *args\u003c/span\u003e):\n self.last_buffer = [ source.encode(\u003cspan class=\"hljs-string\"\u003e'latin-1'\u003c/span\u003e) ]\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e InteractiveConsole.runsource(self, source, *args)\n\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003eraw_input\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003eself, *args\u003c/span\u003e):\n line = InteractiveConsole.raw_input(self, *args)\n \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e line == EDIT_CMD:\n fd, tmpfl = mkstemp(\u003cspan class=\"hljs-string\"\u003e'.py'\u003c/span\u003e)\n os.write(fd, \u003cspan class=\"hljs-string\"\u003eb'\\n'\u003c/span\u003e.join(self.last_buffer))\n os.close(fd)\n os.system(\u003cspan class=\"hljs-string\"\u003e'%s %s'\u003c/span\u003e % (EDITOR, tmpfl))\n line = \u003cspan class=\"hljs-built_in\"\u003eopen\u003c/span\u003e(tmpfl).read()\n os.unlink(tmpfl)\n tmpfl = \u003cspan class=\"hljs-string\"\u003e''\u003c/span\u003e\n lines = line.split( \u003cspan class=\"hljs-string\"\u003e'\\n'\u003c/span\u003e )\n \u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e i \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003erange\u003c/span\u003e(\u003cspan class=\"hljs-built_in\"\u003elen\u003c/span\u003e(lines) - \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e): self.push( lines[i] )\n line = lines[-\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e]\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e line\n\nc = EditableBufferInteractiveConsole(\u003cspan class=\"hljs-built_in\"\u003elocals\u003c/span\u003e=\u003cspan class=\"hljs-built_in\"\u003elocals\u003c/span\u003e())\nc.interact(banner=\u003cspan class=\"hljs-string\"\u003e''\u003c/span\u003e)\n\n\u003cspan class=\"hljs-comment\"\u003e# Exit the Python shell on exiting the InteractiveConsole\u003c/span\u003e\nsys.exit()\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eFor Django developers when you load up the ./manage.py shell it is nice\nto have access to all your models and settings for testing:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-comment\"\u003e# If we're working with a Django project, set up the environment\u003c/span\u003e\n\u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'DJANGO_SETTINGS_MODULE'\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e os.environ:\n \u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e django.db.models.loading \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e get_models\n \u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e django.test.client \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e Client\n \u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e django.test.utils \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e setup_test_environment, teardown_test_environment\n \u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e django.conf \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e settings \u003cspan class=\"hljs-keyword\"\u003eas\u003c/span\u003e S\n\n \u003cspan class=\"hljs-keyword\"\u003eclass\u003c/span\u003e \u003cspan class=\"hljs-title class_\"\u003eDjangoModels\u003c/span\u003e(\u003cspan class=\"hljs-title class_ inherited__\"\u003eobject\u003c/span\u003e):\n \u003cspan class=\"hljs-string\"\u003e\"\"\"Loop through all the models in INSTALLED_APPS and import them.\"\"\"\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003e__init__\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003eself\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e m \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e get_models():\n \u003cspan class=\"hljs-built_in\"\u003esetattr\u003c/span\u003e(self, m.__name__, m)\n\n A = DjangoModels()\n C = Client()\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAfter these tweaks the python interpreter is a lot more powerful and you\nreally lose the need for the more interactive shells like ipython and\nbpython. All of these settings work in both python2 and python3.\u003c/p\u003e\n\u003cp\u003eIf you want to see my complete ~/.pythonrc.py you can get it on\n\u003ca href=\"https://github.com/sontek/dotfiles/blob/master/_pythonrc.py\"\u003egithub\u003c/a\u003e\u003c/p\u003e","category":"Development","date":"2010-12-28T00:00:00Z","tags":["Python"],"title":"Tips and Tricks for the Python Interpreter"},{"id":["old","turning_vim_into_a_modern_python_ide"],"path":"old/turning_vim_into_a_modern_python_ide","contentHtml":"\u003cp\u003eTL;DR:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-shell\"\u003e\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003egit \u003cspan class=\"hljs-built_in\"\u003eclone\u003c/span\u003e https://github.com/sontek/dotfiles.git\u003c/span\u003e\n\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003e\u003cspan class=\"hljs-built_in\"\u003ecd\u003c/span\u003e dotfiles\u003c/span\u003e\n\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003e./install.sh vim\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eIntro\u003c/h1\u003e\n\u003cp\u003eBack in 2008, I wrote the article \u003ca href=\"http://sontek.net/python-with-a-modular-ide-vim\"\u003ePython with a modular IDE\n(Vim)\u003c/a\u003e. Years later, I\nhave people e-mailing me and commenting daily asking for more\ninformation, even though most of the information in it is outdated. Here\nis the modern way to work with Python and Vim to achieve the perfect\nenvironment.\u003c/p\u003e\n\u003cp\u003eBecause one of the most important parts about a development environment\nis the ability to easily reproduce across machines, we are going to\nstore our vim configuration in git:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-shell\"\u003e\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003e\u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e ~/.vim/\u003c/span\u003e\n\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003e\u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e ~/.vim/{\u003cspan class=\"hljs-built_in\"\u003eautoload\u003c/span\u003e,bundle}\u003c/span\u003e\n\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003e\u003cspan class=\"hljs-built_in\"\u003ecd\u003c/span\u003e ~/.vim/\u003c/span\u003e\n\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003egit init\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe purpose of the autoload directory is to automatically load the vim\nplugin \u003ca href=\"https://github.com/tpope/vim-pathogen\"\u003ePathogen\u003c/a\u003e, which we'll\nthen use to load all other plugins that are located in the bundle\ndirectory. So download pathogen and put it in your autoload folder.\u003c/p\u003e\n\u003cp\u003eYou'll need to add the following to your ~/.vimrc so that pathogen\nwill be loaded properly. Filetype detection must be off when you run the\ncommands so its best to execute them first:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-r\"\u003efiletype off\n\u003cspan class=\"hljs-built_in\"\u003ecall\u003c/span\u003e pathogen\u003cspan class=\"hljs-comment\"\u003e#runtime_append_all_bundles()\u003c/span\u003e\n\u003cspan class=\"hljs-built_in\"\u003ecall\u003c/span\u003e pathogen\u003cspan class=\"hljs-comment\"\u003e#helptags()\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow lets add all of the vim plugins we plan on using as submodules to\nour git repository:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003egit submodule add http://github.com/tpope/vim-fugitive.git bundle/fugitive\ngit submodule add https://github.com/msanders/snipmate.vim.git bundle/snipmate\ngit submodule add https://github.com/tpope/vim-surround.git bundle/surround\ngit submodule add https://github.com/tpope/vim-git.git bundle/git\ngit submodule add https://github.com/ervandew/supertab.git bundle/supertab\ngit submodule add https://github.com/sontek/minibufexpl.vim.git bundle/minibufexpl\ngit submodule add https://github.com/wincent/Command-T.git bundle/command-t\ngit submodule add https://github.com/mitechie/pyflakes-pathogen.git\ngit submodule add https://github.com/mileszs/ack.vim.git bundle/ack\ngit submodule add https://github.com/sjl/gundo.vim.git bundle/gundo\ngit submodule add https://github.com/fs111/pydoc.vim.git bundle/pydoc\ngit submodule add https://github.com/vim-scripts/pep8.git bundle/pep8\ngit submodule add https://github.com/alfredodeza/pytest.vim.git bundle/py.test\ngit submodule add https://github.com/reinh/vim-makegreen bundle/makegreen\ngit submodule add https://github.com/vim-scripts/TaskList.vim.git bundle/tasklist\ngit submodule add https://github.com/vim-scripts/The-NERD-tree.git bundle/nerdtree\ngit submodule add https://github.com/sontek/rope-vim.git bundle/ropevim\ngit submodule init\ngit submodule update\ngit submodule foreach git submodule init\ngit submodule foreach git submodule update\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThats it! Now that we've got our vim configuration in git!\u003c/p\u003e\n\u003cp\u003eNow lets look at how to use each of these plugins to improve the power\nof vim:\u003c/p\u003e\n\u003ch1\u003eBasic Editing and Debugging\u003c/h1\u003e\n\u003ch2\u003eCode Folding\u003c/h2\u003e\n\u003cp\u003eLets first enable code folding. This makes it a lot easier to organize\nyour code and hide portions that you aren't interested in working on.\nThis is quite easy for Python, since whitespace is required.\u003c/p\u003e\n\u003cp\u003eIn your ~/.vimrc just add:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003eset \u003cspan class=\"hljs-attr\"\u003efoldmethod\u003c/span\u003e=indent\nset \u003cspan class=\"hljs-attr\"\u003efoldlevel\u003c/span\u003e=\u003cspan class=\"hljs-number\"\u003e99\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen you will be able to be inside a method and type 'za' to open and\nclose a fold.\u003c/p\u003e\n\u003ch2\u003eWindow Splits\u003c/h2\u003e\n\u003cp\u003eSometimes code folding isn't enough; you may need to start opening up\nmultiple windows and working on multiple files at once or different\nlocations within the same file. To do this in vim, you can use these\nshortcuts:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003eVertical Split :\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eCtrl+w\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e+\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ev\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eHorizontal Split:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eCtrl+w\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e+\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003es\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eClose current windows:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eCtrl+w\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e+\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eq\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eI also like to bind Ctrl+\u0026#x3C;movement\u003e keys to move around the windows,\ninstead of using Ctrl+w + \u0026#x3C;movement\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003emap \u0026#x3C;c-j\u003e \u0026#x3C;c-w\u003ej\nmap \u0026#x3C;c-k\u003e \u0026#x3C;c-w\u003ek\nmap \u0026#x3C;c-l\u003e \u0026#x3C;c-w\u003el\nmap \u0026#x3C;c-h\u003e \u0026#x3C;c-w\u003eh\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/krj0l.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003ch2\u003eSnippets\u003c/h2\u003e\n\u003cp\u003eThe next tweak that really speeds up development is using snipmate.\nWe've already included it in our bundle/ folder so its already enabled.\nTry opening up a python file and typing 'def\u0026#x3C;tab\u003e'. It should stub\nout a method definition for you and allow you to tab through and fill\nout the arguments, doc string, etc.\u003c/p\u003e\n\u003cp\u003eI also like to create my own snippets folder to put in some custom\nsnippets:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-shell\"\u003e\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003e\u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e ~/.vim/snippets\u003c/span\u003e\n\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003evim ~/.vim/snippets/python.snippets\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003ePut this in the file:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003esnippet pdb\n \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e pdb; pdb.\u003cspan class=\"hljs-built_in\"\u003eset_trace\u003c/span\u003e()\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow you can type pdb\u0026#x3C;tab\u003e and it'll insert your breakpoint!\u003c/p\u003e\n\u003ch2\u003eTask lists\u003c/h2\u003e\n\u003cp\u003eAnother really useful thing is to mark some of your code as TODO or\nFIXME! I know we all like to think we write perfect code, but sometimes\nyou just have to settle and leave a note for yourself to come back\nlater. One of the plugins we included was the tasklist plugin that will\nallow us to search all open buffers for things to fix. Just add a\nmapping to open it in ~/.vimrc:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003emap \u0026#x3C;leader\u003etd \u0026#x3C;Plug\u003eTaskList\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow you can hit \u0026#x3C;leader\u003etd to open your task list and hit 'q' to\nclose it. You can also hit enter on the task to jump to the buffer and\nline that it is placed on.\u003c/p\u003e\n\u003ch2\u003eRevision History\u003c/h2\u003e\n\u003cp\u003eThe final basic editing tweak I suggest everyone start utilizing is the\nGundo plugin. It'll allow you to view diff's of every save on a file\nyou've made and allow you to quickly revert back and forth:\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/2NrPS.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003cp\u003eJust bind a key in your .vimrc to toggle the Gundo window:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ruby\"\u003emap \u0026#x3C;leader\u003eg \u003cspan class=\"hljs-symbol\"\u003e:GundoToggle\u0026#x3C;CR\u003e\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eSyntax Highlighting and Validation\u003c/h1\u003e\n\u003cp\u003eSimply enable syntax highlighting in your ~/.vimrc:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-vbnet\"\u003esyntax \u003cspan class=\"hljs-keyword\"\u003eon\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\" syntax highlighing\nfiletype on \"\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003etry\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eto\u003c/span\u003e detect filetypes\nfiletype plugin indent \u003cspan class=\"hljs-keyword\"\u003eon\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\" enable loading indent file for filetype\n\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eBecause we enabled pyflakes when we added it as a submodule in\n~/.vim/bundle, it will notify you about unused imports and invalid\nsyntax. It will save you a lot of time saving and running just to find\nout you missed a colon. I like to tell it not use the quickfix window:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003elet g:\u003cspan class=\"hljs-attr\"\u003epyflakes_use_quickfix\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/ZfjFe.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003ch2\u003ePep8\u003c/h2\u003e\n\u003cp\u003eThe final plugin that really helps validate your code is the pep8\nplugin, it'll make sure your code is consistent across all projects.\nAdd a key mapping to your ~/.vimrc and then you'll be able to jump to\neach of the pep8 violations in the quickfix window:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003elet g:\u003cspan class=\"hljs-attr\"\u003epep8_map\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'\u0026#x3C;leader\u003e8'\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/VU9AB.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003ch1\u003eTab Completion and Documentation\u003c/h1\u003e\n\u003cp\u003eVim has many different code completion options. We are going to use the\nSuperTab plugin to check the context of the code you are working on and\nchoose the best for the situation. We've already enabled the SuperTab\nplugin in the bundle/ folder, so we just have to configure it to be\ncontext sensitive and to enable omni code completion in your ~/.vimrc:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003eau FileType python set \u003cspan class=\"hljs-attr\"\u003eomnifunc\u003c/span\u003e=pythoncomplete\u003cspan class=\"hljs-comment\"\u003e#Complete\u003c/span\u003e\nlet g:\u003cspan class=\"hljs-attr\"\u003eSuperTabDefaultCompletionType\u003c/span\u003e = \u003cspan class=\"hljs-string\"\u003e\"context\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we just enable the menu and pydoc preview to get the most useful\ninformation out of the code completion:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003eset \u003cspan class=\"hljs-attr\"\u003ecompleteopt\u003c/span\u003e=menuone,longest,preview\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/g4lxP.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003cp\u003eWe also enabled the pydoc plugin at the beginning with all the\nsubmodules; that gives us the ability to hit \u0026#x3C;leader\u003epw when our\ncursor is on a module and have a new window open with the whole\ndocumentation page for it.\u003c/p\u003e\n\u003ch1\u003eCode Navigation\u003c/h1\u003e\n\u003ch2\u003eBuffers\u003c/h2\u003e\n\u003cp\u003eThe most important part about navigating code within vim, is to\ncompletely understand how to use buffers. There is no reason to use\ntabs. Open files with :e \u0026#x3C;filename\u003e to place in a buffer. We already\ninstalled the minibufexpl plugin, so you will already visually see every\nbuffer opened. You can also get a list of them doing :buffers.\u003c/p\u003e\n\u003cp\u003eYou can switch between the buffers using b\u0026#x3C;number\u003e, such as :b1 for\nthe first buffer. You can also use its name to match, so you can type :b\nmod\u0026#x3C;tab\u003e to autocomplete opening the models.py buffer. You need to\nmake sure you are using the minibufexpl from my github since it has\npatches that make it much better to work with.\u003c/p\u003e\n\u003cp\u003eTo close a buffer you use :bd or :bw.\u003c/p\u003e\n\u003ch2\u003eFuzzy Text File Search\u003c/h2\u003e\n\u003cp\u003eTo make finding and opening files within your project even easier, we\nare going to use the command-t plugin. It does have some parts that need\nto be compiled, so its not already installed by adding it as a\nsubmodule. Go to your ~/.vim/bundle/command-t folder and run 'rake\nmake'. Yes you need ruby installed. By default, command-t is bound to\n\u0026#x3C;leader\u003et. This will use fuzzy text matching to find any file in your\nproject.\u003c/p\u003e\n\u003cp\u003eIt also supports searching only through opened buffers, instead of files\nusing \u0026#x3C;leader\u003eb.\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/hUcSl.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003ch2\u003eFile Browser\u003c/h2\u003e\n\u003cp\u003eNERD Tree is a project file browser. I must admit I used this heavily\nback when I was migrating from Visual Studio and used to the Solution\nExplorer, but I rarely use it anymore. Command-T is usually all you'll\nneed. It is useful when you are getting to know a new codebase for the\nfirst time though. Lets bind a shortcut key for opening it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ruby\"\u003emap \u0026#x3C;leader\u003en \u003cspan class=\"hljs-symbol\"\u003e:NERDTreeToggle\u0026#x3C;CR\u003e\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/R4ZzQ.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003ch2\u003eRefactoring and Go to definition\u003c/h2\u003e\n\u003cp\u003eRopevim is also a great tool that will allow you to navigate around your\ncode. It supports automatically inserting import statements, goto\ndefinition, refactoring, and code completion. You'll really want to\nread up on everything it does, but the two big things I use it for is to\njump to function or class definitions quickly and to rename things\n(including all their references).\u003c/p\u003e\n\u003cp\u003eFor instance, if you are using django and you place your cursor over the\nclass models.Model you reference and then called :RopeGotoDefintion, it\nwould jump you straight to the django library to that class definition.\nWe already have it installed in our bundles, so we bind it to a key to\nuse it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ruby\"\u003emap \u0026#x3C;leader\u003ej \u003cspan class=\"hljs-symbol\"\u003e:RopeGotoDefinition\u0026#x3C;CR\u003e\u003c/span\u003e\nmap \u0026#x3C;leader\u003er \u003cspan class=\"hljs-symbol\"\u003e:RopeRename\u0026#x3C;CR\u003e\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eSearching\u003c/h2\u003e\n\u003cp\u003eThe final tool that really speeds up navigating your code is the Ack\nplugin. Ack is similar to grep, but much better in my opinion. You can\nfuzzy text search for anything in your code (variable name, class,\nmethod, etc) and it'll give you a list of files and line numbers where\nthey are defined so you can quickly cycle through them. Just bind the\nsearching to a key:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-php-template\"\u003e\u003cspan class=\"xml\"\u003enmap \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eleader\u003c/span\u003e\u003e\u003c/span\u003ea \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eEsc\u003c/span\u003e\u003e\u003c/span\u003e:Ack!\n\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe use ! at the end of it so it doesn't open the first result\nautomatically.\u003c/p\u003e\n\u003ch1\u003eIntegration with Git\u003c/h1\u003e\n\u003cp\u003eWe installed 2 plugins, git.vim and fugitive, that give us all the\nintegration we need. Git.vim will provide us syntax highlighting for git\nconfiguration files; fugitive provides a great interface for interacting\nwith git including getting diffs, status updates, committing, and moving\nfiles.\u003c/p\u003e\n\u003cp\u003eFugitive also allows you to view what branch you are working in directly\nfrom vim. Add this to your statusline in ~/.vimrc:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-shell\"\u003e\u003cspan class=\"hljs-meta prompt_\"\u003e%\u003c/span\u003e\u003cspan class=\"bash\"\u003e{fugitive\u003cspan class=\"hljs-comment\"\u003e#statusline()}\u003c/span\u003e\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe big commands you need to know:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eGblame\u003c/strong\u003e: This allows you to view a line by line comparison of who\nthe last person to touch that line of code is.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eGwrite\u003c/strong\u003e: This will stage your file for commit, basically doing\ngit add \u0026#x3C;filename\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eGread\u003c/strong\u003e: This will basically run a git checkout \u0026#x3C;filename\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eGcommit\u003c/strong\u003e: This will just run git commit. Since its in a vim\nbuffer, you can use keyword completion (Ctrl-N), like\ntest_all\u0026#x3C;Ctrl-N\u003e to find the method name in your buffer and\ncomplete it for the commit message. You can also use + and - on the\nfilenames in the message to stage/unstage them for the commit.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/NuRRj.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003ch1\u003eTest Integration\u003c/h1\u003e\n\u003ch2\u003edjango nose\u003c/h2\u003e\n\u003cp\u003eTest runner integration really depends on the testing library you are\nusing and what type of tests you are running but we included a great\ngeneric plugin called MakeGreen that executes off of vim's makeprg\nvariable. So for instance, if you are using django with django-nose you\ncould define a shortcut key in your ~/.vimrc like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003emap \u0026#x3C;leader\u003edt :set makeprg=python\\ manage.py\\ test\\|:call \u003cspan class=\"hljs-built_in\"\u003eMakeGreen\u003c/span\u003e()\u0026#x3C;CR\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will just give you a green bar at the bottom of vim if your test\npassed or a red bar with the message of the failed test if it doesn't.\nVery simple.\u003c/p\u003e\n\u003ch2\u003epy.test\u003c/h2\u003e\n\u003cp\u003eI also included the py.test vim plugin for those who prefer it. This\nplugin has a lot more functionality including executing individual tests\nby class, file, or method. You can also cycle through the individual\nassertion errors. I have the following bindings:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-php-template\"\u003e\u003cspan class=\"xml\"\u003e\" Execute the tests\nnmap \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003esilent\u003c/span\u003e\u003e\u003c/span\u003e\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eLeader\u003c/span\u003e\u003e\u003c/span\u003etf \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eEsc\u003c/span\u003e\u003e\u003c/span\u003e:Pytest file\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eCR\u003c/span\u003e\u003e\u003c/span\u003e\nnmap \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003esilent\u003c/span\u003e\u003e\u003c/span\u003e\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eLeader\u003c/span\u003e\u003e\u003c/span\u003etc \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eEsc\u003c/span\u003e\u003e\u003c/span\u003e:Pytest class\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eCR\u003c/span\u003e\u003e\u003c/span\u003e\nnmap \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003esilent\u003c/span\u003e\u003e\u003c/span\u003e\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eLeader\u003c/span\u003e\u003e\u003c/span\u003etm \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eEsc\u003c/span\u003e\u003e\u003c/span\u003e:Pytest method\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eCR\u003c/span\u003e\u003e\u003c/span\u003e\n\" cycle through test errors\nnmap \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003esilent\u003c/span\u003e\u003e\u003c/span\u003e\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eLeader\u003c/span\u003e\u003e\u003c/span\u003etn \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eEsc\u003c/span\u003e\u003e\u003c/span\u003e:Pytest next\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eCR\u003c/span\u003e\u003e\u003c/span\u003e\nnmap \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003esilent\u003c/span\u003e\u003e\u003c/span\u003e\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eLeader\u003c/span\u003e\u003e\u003c/span\u003etp \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eEsc\u003c/span\u003e\u003e\u003c/span\u003e:Pytest previous\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eCR\u003c/span\u003e\u003e\u003c/span\u003e\nnmap \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003esilent\u003c/span\u003e\u003e\u003c/span\u003e\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eLeader\u003c/span\u003e\u003e\u003c/span\u003ete \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eEsc\u003c/span\u003e\u003e\u003c/span\u003e:Pytest error\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eCR\u003c/span\u003e\u003e\u003c/span\u003e\n\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/RAE7v.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003ch1\u003eVirtualenv\u003c/h1\u003e\n\u003cp\u003eVim doesn't realize that you are in a virtualenv so it wont give you\ncode completion for libraries only installed there. Add the following\nscript to your ~/.vimrc to fix it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\" Add the virtualenv's site-packages to vim path\npy \u0026#x3C;\u0026#x3C; EOF\nimport os.path\nimport sys\nimport vim\nif 'VIRTUAL_ENV' in os.environ:\n \u003cspan class=\"hljs-attr\"\u003eproject_base_dir\u003c/span\u003e = os.environ[\u003cspan class=\"hljs-string\"\u003e'VIRTUAL_ENV'\u003c/span\u003e]\n sys.path.insert(0, project_base_dir)\n \u003cspan class=\"hljs-attr\"\u003eactivate_this\u003c/span\u003e = os.path.join(project_base_dir, \u003cspan class=\"hljs-string\"\u003e'bin/activate_this.py'\u003c/span\u003e)\n execfile(activate_this, dict(\u003cspan class=\"hljs-attr\"\u003e__file__\u003c/span\u003e=activate_this))\nEOF\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eDjango\u003c/h1\u003e\n\u003cp\u003eThe only true django tweak I make is before I open vim I'll export the\nDJANGO_SETTINGS_MODULE environment so that I get code completion for\ndjango modules as well:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003eexport \u003cspan class=\"hljs-attr\"\u003eDJANGO_SETTINGS_MODULE\u003c/span\u003e=project.settings\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eRandom Tips\u003c/h1\u003e\n\u003cp\u003eIf you want to find a new color scheme just go to\n\u003ca href=\"http://code.google.com/p/vimcolorschemetest/\"\u003ehttp://code.google.com/p/vimcolorschemetest/\u003c/a\u003e to preview a large\nselection.\u003c/p\u003e\n\u003cp\u003eJohn Anderson \u0026#x3C;\u003ca href=\"mailto:sontek@gmail.com\"\u003esontek@gmail.com\u003c/a\u003e\u003e 2011\u003c/p\u003e","category":"Development","date":"2011-05-07T00:00:00Z","tags":["Vim","Python"],"title":"Turning Vim into a modern Python IDE"},{"id":["old","writing_tests_for_pyramid_and_sqlalchemy"],"path":"old/writing_tests_for_pyramid_and_sqlalchemy","contentHtml":"\u003cp\u003eTL;DR: Putting it all together, the full code can be found here:\n\u003ca href=\"https://gist.github.com/1420255\"\u003ehttps://gist.github.com/1420255\u003c/a\u003e\u003c/p\u003e\n\u003ch1\u003eIntro\u003c/h1\u003e\n\u003cp\u003ePyramid's documentation doesn't cover the preferred way to test with\nSQLAlchemy, because Pyramid tries to stay out of your way and allow you\nto make your own decisions. However, I feel i'ts necessary to document\nwhat I think is the best way to test.\u003c/p\u003e\n\u003cp\u003eWhen I first started writing tests with SQLAlchemy I found plenty of\nexamples of how to to get started by doing something like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e db \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e session \u003cspan class=\"hljs-comment\"\u003e# probably a contextbound sessionmaker\u003c/span\u003e\n\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e db \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e model\n\n\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e sqlalchemy \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e create_engine\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003esetup\u003c/span\u003e():\n engine = create_engine(\u003cspan class=\"hljs-string\"\u003e'sqlite:///test.db'\u003c/span\u003e)\n session.configure(bind=engine)\n model.metadata.create_all(engine)\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003eteardown\u003c/span\u003e():\n model.metadata.drop_all(engine)\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003etest_something\u003c/span\u003e():\n \u003cspan class=\"hljs-keyword\"\u003epass\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eI have seen this done so many times, but I feel there is so much wrong\nwith it! So let's establish some base rules when testing:\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cul\u003e\n\u003cli\u003eAlways test your system like it would be used in production.\nSQLite does not enforce the same rules or have the same features\nas Postgres or MySQL and will allow tests to pass that would\notherwise fail in production.\u003c/li\u003e\n\u003cli\u003eTests should be fast! You should be writing tests for all your\ncode. This is the main reason people do test against SQLite, but\nwe can't violate rule number one. We have to make sure tests\nagainst Postgres are fast, so we shouldn't be tearing down and\nrecreating tables for every single test.\u003c/li\u003e\n\u003cli\u003eYou should be able to execute in parallel to speed up when you\nhave thousands of tests. Dropping and creating tables per test\nwould not work in a parallel environment.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eFor an example, I have a project with 600+ tests and it would take 2 and\nhalf minutes to execute running against SQLite. But when we swapped our\ntest configuration to execute against Postgres, testing took well over\nan hour. That is unacceptable!\u003c/p\u003e\n\u003cp\u003eBut running them in parallel will give us a huge speed up. Check out the\nresults of the tests running in single proc mode vs using all 4 cores:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e$ \u003cspan class=\"hljs-attr\"\u003epy.test\u003c/span\u003e\n======= 616 passed in 143.67 \u003cspan class=\"hljs-attr\"\u003eseconds\u003c/span\u003e =======\n\n$ py.test \u003cspan class=\"hljs-attr\"\u003e-n4\u003c/span\u003e\n======= 616 passed in 68.12 \u003cspan class=\"hljs-attr\"\u003eseconds\u003c/span\u003e =======\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eThe right way\u003c/h1\u003e\n\u003cp\u003eSo what is the proper way to setup your tests? You should initialize the\ndatabase when you start your test runner and then use transactions to\nrollback any data changes your tests made. This allows you to keep a\nclean database for each test in a very efficient way.\u003c/p\u003e\n\u003cp\u003eIn py.test, you just have to create a file called conftest.py that looks\nsimilar to:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e os\n\nROOT_PATH = os.path.dirname(__file__)\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003epytest_sessionstart\u003c/span\u003e():\n \u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e py.test \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e config\n\n \u003cspan class=\"hljs-comment\"\u003e# Only run database setup on master (in case of xdist/multiproc mode)\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003enot\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003ehasattr\u003c/span\u003e(config, \u003cspan class=\"hljs-string\"\u003e'slaveinput'\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e models \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e initialize_sql\n \u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e pyramid.config \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e Configurator\n \u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e paste.deploy.loadwsgi \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e appconfig\n \u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e sqlalchemy \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e engine_from_config\n \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e os\n\n ROOT_PATH = os.path.dirname(__file__)\n settings = appconfig(\u003cspan class=\"hljs-string\"\u003e'config:'\u003c/span\u003e + os.path.join(ROOT_PATH, \u003cspan class=\"hljs-string\"\u003e'test.ini'\u003c/span\u003e))\n engine = engine_from_config(settings, prefix=\u003cspan class=\"hljs-string\"\u003e'sqlalchemy.'\u003c/span\u003e)\n\n \u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'Creating the tables on the test database %s'\u003c/span\u003e % engine\n\n config = Configurator(settings=settings)\n initialize_sql(settings, config)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWith py.test, when you are running in parallel mode, the\npytest_sessionstart hook gets fired for each node, so we check that we\nare on the master node. Then we just grab our test.ini configuration\nfile and execute the initialize_sql function.\u003c/p\u003e\n\u003cp\u003eNow that you have your initial test configuration finished, you have to\ndefine a base test class that does the transaction management in setUp\nand teardown.\u003c/p\u003e\n\u003cp\u003eFirst, lets setup the Base testing class what will manage our\ntransactions:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e unittest\n\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e pyramid \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e testing\n\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e paste.deploy.loadwsgi \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e appconfig\n\n\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e webtest \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e TestApp\n\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e mock \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e Mock\n\n\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e sqlalchemy \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e engine_from_config\n\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e sqlalchemy.orm \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e sessionmaker\n\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e app.db \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e Session\n\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e app.db \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e Entity \u003cspan class=\"hljs-comment\"\u003e# base declarative object\u003c/span\u003e\n\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e app \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e main\n\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e os\nhere = os.path.dirname(__file__)\nsettings = appconfig(\u003cspan class=\"hljs-string\"\u003e'config:'\u003c/span\u003e + os.path.join(here, \u003cspan class=\"hljs-string\"\u003e'../../'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'test.ini'\u003c/span\u003e))\n\n\u003cspan class=\"hljs-keyword\"\u003eclass\u003c/span\u003e \u003cspan class=\"hljs-title class_\"\u003eBaseTestCase\u003c/span\u003e(unittest.TestCase):\n\u003cspan class=\"hljs-meta\"\u003e @classmethod\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003esetUpClass\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003ecls\u003c/span\u003e):\n cls.engine = engine_from_config(settings, prefix=\u003cspan class=\"hljs-string\"\u003e'sqlalchemy.'\u003c/span\u003e)\n cls.Session = sessionmaker()\n\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003esetUp\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003eself\u003c/span\u003e):\n connection = self.engine.connect()\n\n \u003cspan class=\"hljs-comment\"\u003e# begin a non-ORM transaction\u003c/span\u003e\n self.trans = connection.begin()\n\n \u003cspan class=\"hljs-comment\"\u003e# bind an individual Session to the connection\u003c/span\u003e\n Session.configure(bind=connection)\n self.session = self.Session(bind=connection)\n Entity.session = self.session\n\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003etearDown\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003eself\u003c/span\u003e):\n \u003cspan class=\"hljs-comment\"\u003e# rollback - everything that happened with the\u003c/span\u003e\n \u003cspan class=\"hljs-comment\"\u003e# Session above (including calls to commit())\u003c/span\u003e\n \u003cspan class=\"hljs-comment\"\u003e# is rolled back.\u003c/span\u003e\n testing.tearDown()\n self.trans.rollback()\n self.session.close()\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis base test case will wrap all your sessions in an external\ntransaction so that you still have the ability to call flush/commit/etc\nand it will still be able to rollback any data changes you make.\u003c/p\u003e\n\u003ch1\u003eUnit Tests\u003c/h1\u003e\n\u003cp\u003eNow there are a few different types of tests you will want to run.\nFirst, you will want to do unit tests, which are small tests that only\ntest 1 thing at a time. This means you will skip the routes, templates,\netc. So let's setup our Unit Test Base class:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-keyword\"\u003eclass\u003c/span\u003e \u003cspan class=\"hljs-title class_\"\u003eUnitTestBase\u003c/span\u003e(\u003cspan class=\"hljs-title class_ inherited__\"\u003eBaseTestCase\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003esetUp\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003eself\u003c/span\u003e):\n self.config = testing.setUp(request=testing.DummyRequest())\n \u003cspan class=\"hljs-built_in\"\u003esuper\u003c/span\u003e(UnitTestBase, self).setUp()\n\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003eget_csrf_request\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003eself, post=\u003cspan class=\"hljs-literal\"\u003eNone\u003c/span\u003e\u003c/span\u003e):\n csrf = \u003cspan class=\"hljs-string\"\u003e'abc'\u003c/span\u003e\n\n \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003enot\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eu'csrf_token'\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e post.keys():\n post.update({\n \u003cspan class=\"hljs-string\"\u003e'csrf_token'\u003c/span\u003e: csrf\n })\n\n request = testing.DummyRequest(post)\n\n request.session = Mock()\n csrf_token = Mock()\n csrf_token.return_value = csrf\n\n request.session.get_csrf_token = csrf_token\n\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e request\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe built in a utility function to help us test requests that require a\ncsrf token as well. Here is how we would use this class:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-keyword\"\u003eclass\u003c/span\u003e \u003cspan class=\"hljs-title class_\"\u003eTestViews\u003c/span\u003e(\u003cspan class=\"hljs-title class_ inherited__\"\u003eUnitTestBase\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003etest_login_fails_empty\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003eself\u003c/span\u003e):\n \u003cspan class=\"hljs-string\"\u003e\"\"\" Make sure we can't login with empty credentials\"\"\"\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e app.accounts.views \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e LoginView\n self.config.add_route(\u003cspan class=\"hljs-string\"\u003e'index'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'/'\u003c/span\u003e)\n self.config.add_route(\u003cspan class=\"hljs-string\"\u003e'dashboard'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'/'\u003c/span\u003e)\n\n request = testing.DummyRequest(post={\n \u003cspan class=\"hljs-string\"\u003e'submit'\u003c/span\u003e: \u003cspan class=\"hljs-literal\"\u003eTrue\u003c/span\u003e,\n })\n\n view = LoginView(request)\n response = view.post()\n errors = response[\u003cspan class=\"hljs-string\"\u003e'errors'\u003c/span\u003e]\n\n \u003cspan class=\"hljs-keyword\"\u003eassert\u003c/span\u003e errors[\u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e].node.name == \u003cspan class=\"hljs-string\"\u003eu'csrf_token'\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003eassert\u003c/span\u003e errors[\u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e].msg == \u003cspan class=\"hljs-string\"\u003eu'Required'\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003eassert\u003c/span\u003e errors[\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e].node.name == \u003cspan class=\"hljs-string\"\u003eu'Username'\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003eassert\u003c/span\u003e errors[\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e].msg == \u003cspan class=\"hljs-string\"\u003eu'Required'\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003eassert\u003c/span\u003e errors[\u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e].node.name == \u003cspan class=\"hljs-string\"\u003eu'Password'\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003eassert\u003c/span\u003e errors[\u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e].msg == \u003cspan class=\"hljs-string\"\u003eu'Required'\u003c/span\u003e\n\n\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003etest_login_succeeds\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003eself\u003c/span\u003e):\n \u003cspan class=\"hljs-string\"\u003e\"\"\" Make sure we can login \"\"\"\u003c/span\u003e\n admin = User(username=\u003cspan class=\"hljs-string\"\u003e'sontek'\u003c/span\u003e, password=\u003cspan class=\"hljs-string\"\u003e'temp'\u003c/span\u003e, kind=\u003cspan class=\"hljs-string\"\u003eu'admin'\u003c/span\u003e)\n admin.activated = \u003cspan class=\"hljs-literal\"\u003eTrue\u003c/span\u003e\n self.session.add(admin)\n self.session.flush()\n\n \u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e app.accounts.views \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e LoginView\n self.config.add_route(\u003cspan class=\"hljs-string\"\u003e'index'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'/'\u003c/span\u003e)\n self.config.add_route(\u003cspan class=\"hljs-string\"\u003e'dashboard'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'/dashboard'\u003c/span\u003e)\n\n request = self.get_csrf_request(post={\n \u003cspan class=\"hljs-string\"\u003e'submit'\u003c/span\u003e: \u003cspan class=\"hljs-literal\"\u003eTrue\u003c/span\u003e,\n \u003cspan class=\"hljs-string\"\u003e'Username'\u003c/span\u003e: \u003cspan class=\"hljs-string\"\u003e'sontek'\u003c/span\u003e,\n \u003cspan class=\"hljs-string\"\u003e'Password'\u003c/span\u003e: \u003cspan class=\"hljs-string\"\u003e'temp'\u003c/span\u003e,\n })\n\n view = LoginView(request)\n response = view.post()\n\n \u003cspan class=\"hljs-keyword\"\u003eassert\u003c/span\u003e response.status_int == \u003cspan class=\"hljs-number\"\u003e302\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eIntegration Tests\u003c/h1\u003e\n\u003cp\u003eThe second type of test you will want to write is an integration test.\nThis will integrate with the whole web framework and actually hit the\ndefine routes, render the templates, and actually test the full stack of\nyour application.\u003c/p\u003e\n\u003cp\u003eLuckily this is pretty easy to do with Pyramid using WebTest:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-keyword\"\u003eclass\u003c/span\u003e \u003cspan class=\"hljs-title class_\"\u003eIntegrationTestBase\u003c/span\u003e(\u003cspan class=\"hljs-title class_ inherited__\"\u003eBaseTestCase\u003c/span\u003e):\n\u003cspan class=\"hljs-meta\"\u003e @classmethod\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003esetUpClass\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003ecls\u003c/span\u003e):\n cls.app = main({}, **settings)\n \u003cspan class=\"hljs-built_in\"\u003esuper\u003c/span\u003e(IntegrationTestBase, cls).setUpClass()\n\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003esetUp\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003eself\u003c/span\u003e):\n self.app = TestApp(self.app)\n self.config = testing.setUp()\n \u003cspan class=\"hljs-built_in\"\u003esuper\u003c/span\u003e(IntegrationTestBase, self).setUp()\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIn setUpClass, we run the main function of the applications\n__init__.py that sets up the WSGI application and then we wrap it in\na TestApp that gives us the ability to call get/post on it.\u003c/p\u003e\n\u003cp\u003eHere is an example of it in use:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-keyword\"\u003eclass\u003c/span\u003e \u003cspan class=\"hljs-title class_\"\u003eTestViews\u003c/span\u003e(\u003cspan class=\"hljs-title class_ inherited__\"\u003eIntegrationTestBase\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003etest_get_login\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003eself\u003c/span\u003e):\n \u003cspan class=\"hljs-string\"\u003e\"\"\" Call the login view, make sure routes are working \"\"\"\u003c/span\u003e\n res = self.app.get(\u003cspan class=\"hljs-string\"\u003e'/login'\u003c/span\u003e)\n self.assertEqual(res.status_int, \u003cspan class=\"hljs-number\"\u003e200\u003c/span\u003e)\n\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003etest_empty_login\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003eself\u003c/span\u003e):\n \u003cspan class=\"hljs-string\"\u003e\"\"\" Empty login fails \"\"\"\u003c/span\u003e\n res = self.app.post(\u003cspan class=\"hljs-string\"\u003e'/login'\u003c/span\u003e, {\u003cspan class=\"hljs-string\"\u003e'submit'\u003c/span\u003e: \u003cspan class=\"hljs-literal\"\u003eTrue\u003c/span\u003e})\n\n \u003cspan class=\"hljs-keyword\"\u003eassert\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"There was a problem with your submission\"\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e res.body\n \u003cspan class=\"hljs-keyword\"\u003eassert\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"Required\"\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e res.body\n \u003cspan class=\"hljs-keyword\"\u003eassert\u003c/span\u003e res.status_int == \u003cspan class=\"hljs-number\"\u003e200\u003c/span\u003e\n\n \u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003etest_valid_login\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003eself\u003c/span\u003e):\n \u003cspan class=\"hljs-string\"\u003e\"\"\" Call the login view, make sure routes are working \"\"\"\u003c/span\u003e\n admin = User(username=\u003cspan class=\"hljs-string\"\u003e'sontek'\u003c/span\u003e, password=\u003cspan class=\"hljs-string\"\u003e'temp'\u003c/span\u003e, kind=\u003cspan class=\"hljs-string\"\u003eu'admin'\u003c/span\u003e)\n admin.activated = \u003cspan class=\"hljs-literal\"\u003eTrue\u003c/span\u003e\n self.session.add(admin)\n self.session.flush()\n\n res = self.app.get(\u003cspan class=\"hljs-string\"\u003e'/login'\u003c/span\u003e)\n\n csrf = res.form.fields[\u003cspan class=\"hljs-string\"\u003e'csrf_token'\u003c/span\u003e][\u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e].value\n\n res = self.app.post(\u003cspan class=\"hljs-string\"\u003e'/login'\u003c/span\u003e,\n {\n \u003cspan class=\"hljs-string\"\u003e'submit'\u003c/span\u003e: \u003cspan class=\"hljs-literal\"\u003eTrue\u003c/span\u003e,\n \u003cspan class=\"hljs-string\"\u003e'Username'\u003c/span\u003e: \u003cspan class=\"hljs-string\"\u003e'sontek'\u003c/span\u003e,\n \u003cspan class=\"hljs-string\"\u003e'Password'\u003c/span\u003e: \u003cspan class=\"hljs-string\"\u003e'temp'\u003c/span\u003e,\n \u003cspan class=\"hljs-string\"\u003e'csrf_token'\u003c/span\u003e: csrf\n }\n )\n\n \u003cspan class=\"hljs-keyword\"\u003eassert\u003c/span\u003e res.status_int == \u003cspan class=\"hljs-number\"\u003e302\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eProblems with this approach\u003c/h1\u003e\n\u003cp\u003eIf a test causes an error that will prevent the transaction from rolling\nback, such as closing the engine, then this approach will leave your\ndatabase in a state that might cause other tests to fail.\u003c/p\u003e\n\u003cp\u003eIf this happens tracing the root cause could be difficult but you should\nbe able to just look at the first failed test unless you are running the\ntests in parallel.\u003c/p\u003e\n\u003cp\u003eIf you are good about writing and running your tests regularly you\nshould be able to catch individual tests causing issues like this fairly\nquickly.\u003c/p\u003e\n\u003ch1\u003eResources\u003c/h1\u003e\n\u003cp\u003e\u003ca href=\"http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/testing.html\"\u003ehttp://docs.pylonsproject.org/projects/pyramid/en/latest/narr/testing.html\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"http://www.sqlalchemy.org/docs/orm/session.html#joining-a-session-into-an-external-transaction\"\u003ehttp://www.sqlalchemy.org/docs/orm/session.html#joining-a-session-into-an-external-transaction\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eJohn Anderson \u0026#x3C;\u003ca href=\"mailto:sontek@gmail.com\"\u003esontek@gmail.com\u003c/a\u003e\u003e 2011\u003c/p\u003e","category":"Python","date":"2011-12-01T00:00:00Z","tags":["Python"],"title":"Writing tests for Pyramid and SQLAlchemy"}]},"NodeJS":{"count":1,"posts":[{"id":["2022","intro_to_asdf"],"path":"2022/intro_to_asdf","contentHtml":"\u003cp\u003e\u003ca href=\"https://asdf-vm.com/\"\u003easdf\u003c/a\u003e is a general purpose version manager that\ncan manage versions of most programming language runtimes through a set\nof plugins.\u003c/p\u003e\n\u003ciframe width=\"854\" height=\"480\" src=\"https://www.youtube.com/embed/RTaqWRj-6Lg\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen\u003e\u003c/iframe\u003e\n\u003cp\u003eWith micro-services being all the rage and the ever changing landscape\nof the development world, it is rare to utilize a single version of\nlanguage runtime. Even when you want to upgrade from one to the other\nyou'll need both usable on your system at the same time.\u003c/p\u003e\n\u003cp\u003eI've used tools like \u003ccode\u003epyenv\u003c/code\u003e and \u003ccode\u003envm\u003c/code\u003e in the past when I needed to change\nversions depending on which project I'm contributing to. But with \u003ccode\u003easdf\u003c/code\u003e\nyou have one tool to rule them all!\u003c/p\u003e\n\u003ch2\u003eGetting Started\u003c/h2\u003e\n\u003cp\u003eThe first thing you need to do when working with \u003ccode\u003easdf\u003c/code\u003e is grab the\nplugins for the languages you are interested in working with. You can list\nwhat plugins are available:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf plugin list all\ngolang *https://github.com/kennyp/asdf-golang.git\ngolangci-lint https://github.com/hypnoglow/asdf-golangci-lint.git\nnodejs *https://github.com/asdf-vm/asdf-nodejs.git\npoetry *https://github.com/asdf-community/asdf-poetry.git\npython *https://github.com/danhper/asdf-python.git\nyarn *https://github.com/twuni/asdf-yarn.git\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOn the left will be the name of the plugin and on the right will be the repository\nwhere it lives. It'll me marked with an asterisk if you already have it installed.\u003c/p\u003e\n\u003cp\u003eTo install a plugin you say \u003ccode\u003easdf plugin add \u0026#x3C;plugin\u003e\u003c/code\u003e to get it installed. You can\nalso provide the repository where you want it pulled from, for example:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf plugin add nodejs https://github.com/asdf-vm/asdf-nodejs.git\n\u003e asdf plugin add python https://github.com/danhper/asdf-python.git\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will not give you any version of those languages, it is only installing the\nplugin that knows how to work with those languages. You are ready to pull down\nany versions you want at that point:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf install nodejs 14.19.0\n\u003e asdf install python 3.9.10\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce you have the versions installed you will be able to view them like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf list\ngolang\n 1.17.7\nnodejs\n --\u003cspan class=\"hljs-built_in\"\u003ehelp\u003c/span\u003e\n 12.22.10\n 14.19.0\n 16.14.0\n 17.5.0\npoetry\n 1.1.13\npython\n 3.9.10\nyarn\n 1.22.17\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eUsing the installed languages\u003c/h2\u003e\n\u003cp\u003eTo activate a specific version of a language you have you have three options:\u003c/p\u003e\n\u003ch3\u003eMake it global\u003c/h3\u003e\n\u003cp\u003eYou can make it global, meaning when you run the tool like \u003ccode\u003epython\u003c/code\u003e it'll use\nthis version for the system:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf global python 3.9.10\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch3\u003eMake it local\u003c/h3\u003e\n\u003cp\u003eYou can make it local, which means it will generate a file in the current\ndirectory named \u003ccode\u003e.tool-versions\u003c/code\u003e and so whenever you change into a directory\nit will activate the versions defined in there.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf \u003cspan class=\"hljs-built_in\"\u003elocal\u003c/span\u003e nodejs 12.22.10\n\u003e \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e .tool-versions \nnodejs 12.22.10\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe great thing about this is you can commit that file to git and then anyone\nwho checks out the project and uses \u003ccode\u003easdf\u003c/code\u003e will have the same versions activated!\u003c/p\u003e\n\u003ch3\u003eTemporary\u003c/h3\u003e\n\u003cp\u003eIf you want to activate a version of a language temporarily you can swap to it\nfor the current shell:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf shell golang 1.17.7\n\u003e \u003cspan class=\"hljs-built_in\"\u003eenv\u003c/span\u003e|grep -i ASDF\nASDF_GOLANG_VERSION=1.17.7\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIt sets an environment variable that will have preference over the file. If you\never wonder what versions a directory is using you can run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf current\ngolang ______ No version \u003cspan class=\"hljs-built_in\"\u003eset\u003c/span\u003e. Run \u003cspan class=\"hljs-string\"\u003e\"asdf \u0026#x3C;global|shell|local\u003e golang \u0026#x3C;version\u003e\"\u003c/span\u003e\nnodejs 12.22.10 .tool-versions\npoetry ______ No version \u003cspan class=\"hljs-built_in\"\u003eset\u003c/span\u003e. Run \u003cspan class=\"hljs-string\"\u003e\"asdf \u0026#x3C;global|shell|local\u003e poetry \u0026#x3C;version\u003e\"\u003c/span\u003e\npython 3.9.10 .tool-versions\nyarn 1.22.17 .tool-versions\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eConclusion\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://asdf-vm.com/\"\u003easdf\u003c/a\u003e is an AWESOME tool to utilize if you find yourself using many\ndifferent languages or many different versions of the same language. You should check it out\nand see if it can improve your workflow.\u003c/p\u003e","category":"Development","date":"2022-02-18T00:00:00Z","tags":["Python","NodeJS","GoLang","Linux"],"title":"Use asdf to manage Python, NodeJS, GoLang and more!"}]},"GoLang":{"count":1,"posts":[{"id":["2022","intro_to_asdf"],"path":"2022/intro_to_asdf","contentHtml":"\u003cp\u003e\u003ca href=\"https://asdf-vm.com/\"\u003easdf\u003c/a\u003e is a general purpose version manager that\ncan manage versions of most programming language runtimes through a set\nof plugins.\u003c/p\u003e\n\u003ciframe width=\"854\" height=\"480\" src=\"https://www.youtube.com/embed/RTaqWRj-6Lg\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen\u003e\u003c/iframe\u003e\n\u003cp\u003eWith micro-services being all the rage and the ever changing landscape\nof the development world, it is rare to utilize a single version of\nlanguage runtime. Even when you want to upgrade from one to the other\nyou'll need both usable on your system at the same time.\u003c/p\u003e\n\u003cp\u003eI've used tools like \u003ccode\u003epyenv\u003c/code\u003e and \u003ccode\u003envm\u003c/code\u003e in the past when I needed to change\nversions depending on which project I'm contributing to. But with \u003ccode\u003easdf\u003c/code\u003e\nyou have one tool to rule them all!\u003c/p\u003e\n\u003ch2\u003eGetting Started\u003c/h2\u003e\n\u003cp\u003eThe first thing you need to do when working with \u003ccode\u003easdf\u003c/code\u003e is grab the\nplugins for the languages you are interested in working with. You can list\nwhat plugins are available:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf plugin list all\ngolang *https://github.com/kennyp/asdf-golang.git\ngolangci-lint https://github.com/hypnoglow/asdf-golangci-lint.git\nnodejs *https://github.com/asdf-vm/asdf-nodejs.git\npoetry *https://github.com/asdf-community/asdf-poetry.git\npython *https://github.com/danhper/asdf-python.git\nyarn *https://github.com/twuni/asdf-yarn.git\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOn the left will be the name of the plugin and on the right will be the repository\nwhere it lives. It'll me marked with an asterisk if you already have it installed.\u003c/p\u003e\n\u003cp\u003eTo install a plugin you say \u003ccode\u003easdf plugin add \u0026#x3C;plugin\u003e\u003c/code\u003e to get it installed. You can\nalso provide the repository where you want it pulled from, for example:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf plugin add nodejs https://github.com/asdf-vm/asdf-nodejs.git\n\u003e asdf plugin add python https://github.com/danhper/asdf-python.git\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will not give you any version of those languages, it is only installing the\nplugin that knows how to work with those languages. You are ready to pull down\nany versions you want at that point:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf install nodejs 14.19.0\n\u003e asdf install python 3.9.10\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce you have the versions installed you will be able to view them like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf list\ngolang\n 1.17.7\nnodejs\n --\u003cspan class=\"hljs-built_in\"\u003ehelp\u003c/span\u003e\n 12.22.10\n 14.19.0\n 16.14.0\n 17.5.0\npoetry\n 1.1.13\npython\n 3.9.10\nyarn\n 1.22.17\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eUsing the installed languages\u003c/h2\u003e\n\u003cp\u003eTo activate a specific version of a language you have you have three options:\u003c/p\u003e\n\u003ch3\u003eMake it global\u003c/h3\u003e\n\u003cp\u003eYou can make it global, meaning when you run the tool like \u003ccode\u003epython\u003c/code\u003e it'll use\nthis version for the system:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf global python 3.9.10\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch3\u003eMake it local\u003c/h3\u003e\n\u003cp\u003eYou can make it local, which means it will generate a file in the current\ndirectory named \u003ccode\u003e.tool-versions\u003c/code\u003e and so whenever you change into a directory\nit will activate the versions defined in there.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf \u003cspan class=\"hljs-built_in\"\u003elocal\u003c/span\u003e nodejs 12.22.10\n\u003e \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e .tool-versions \nnodejs 12.22.10\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe great thing about this is you can commit that file to git and then anyone\nwho checks out the project and uses \u003ccode\u003easdf\u003c/code\u003e will have the same versions activated!\u003c/p\u003e\n\u003ch3\u003eTemporary\u003c/h3\u003e\n\u003cp\u003eIf you want to activate a version of a language temporarily you can swap to it\nfor the current shell:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf shell golang 1.17.7\n\u003e \u003cspan class=\"hljs-built_in\"\u003eenv\u003c/span\u003e|grep -i ASDF\nASDF_GOLANG_VERSION=1.17.7\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIt sets an environment variable that will have preference over the file. If you\never wonder what versions a directory is using you can run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf current\ngolang ______ No version \u003cspan class=\"hljs-built_in\"\u003eset\u003c/span\u003e. Run \u003cspan class=\"hljs-string\"\u003e\"asdf \u0026#x3C;global|shell|local\u003e golang \u0026#x3C;version\u003e\"\u003c/span\u003e\nnodejs 12.22.10 .tool-versions\npoetry ______ No version \u003cspan class=\"hljs-built_in\"\u003eset\u003c/span\u003e. Run \u003cspan class=\"hljs-string\"\u003e\"asdf \u0026#x3C;global|shell|local\u003e poetry \u0026#x3C;version\u003e\"\u003c/span\u003e\npython 3.9.10 .tool-versions\nyarn 1.22.17 .tool-versions\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eConclusion\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://asdf-vm.com/\"\u003easdf\u003c/a\u003e is an AWESOME tool to utilize if you find yourself using many\ndifferent languages or many different versions of the same language. You should check it out\nand see if it can improve your workflow.\u003c/p\u003e","category":"Development","date":"2022-02-18T00:00:00Z","tags":["Python","NodeJS","GoLang","Linux"],"title":"Use asdf to manage Python, NodeJS, GoLang and more!"}]},"Linux":{"count":5,"posts":[{"id":["2022","intro_to_asdf"],"path":"2022/intro_to_asdf","contentHtml":"\u003cp\u003e\u003ca href=\"https://asdf-vm.com/\"\u003easdf\u003c/a\u003e is a general purpose version manager that\ncan manage versions of most programming language runtimes through a set\nof plugins.\u003c/p\u003e\n\u003ciframe width=\"854\" height=\"480\" src=\"https://www.youtube.com/embed/RTaqWRj-6Lg\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen\u003e\u003c/iframe\u003e\n\u003cp\u003eWith micro-services being all the rage and the ever changing landscape\nof the development world, it is rare to utilize a single version of\nlanguage runtime. Even when you want to upgrade from one to the other\nyou'll need both usable on your system at the same time.\u003c/p\u003e\n\u003cp\u003eI've used tools like \u003ccode\u003epyenv\u003c/code\u003e and \u003ccode\u003envm\u003c/code\u003e in the past when I needed to change\nversions depending on which project I'm contributing to. But with \u003ccode\u003easdf\u003c/code\u003e\nyou have one tool to rule them all!\u003c/p\u003e\n\u003ch2\u003eGetting Started\u003c/h2\u003e\n\u003cp\u003eThe first thing you need to do when working with \u003ccode\u003easdf\u003c/code\u003e is grab the\nplugins for the languages you are interested in working with. You can list\nwhat plugins are available:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf plugin list all\ngolang *https://github.com/kennyp/asdf-golang.git\ngolangci-lint https://github.com/hypnoglow/asdf-golangci-lint.git\nnodejs *https://github.com/asdf-vm/asdf-nodejs.git\npoetry *https://github.com/asdf-community/asdf-poetry.git\npython *https://github.com/danhper/asdf-python.git\nyarn *https://github.com/twuni/asdf-yarn.git\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOn the left will be the name of the plugin and on the right will be the repository\nwhere it lives. It'll me marked with an asterisk if you already have it installed.\u003c/p\u003e\n\u003cp\u003eTo install a plugin you say \u003ccode\u003easdf plugin add \u0026#x3C;plugin\u003e\u003c/code\u003e to get it installed. You can\nalso provide the repository where you want it pulled from, for example:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf plugin add nodejs https://github.com/asdf-vm/asdf-nodejs.git\n\u003e asdf plugin add python https://github.com/danhper/asdf-python.git\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will not give you any version of those languages, it is only installing the\nplugin that knows how to work with those languages. You are ready to pull down\nany versions you want at that point:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf install nodejs 14.19.0\n\u003e asdf install python 3.9.10\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce you have the versions installed you will be able to view them like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf list\ngolang\n 1.17.7\nnodejs\n --\u003cspan class=\"hljs-built_in\"\u003ehelp\u003c/span\u003e\n 12.22.10\n 14.19.0\n 16.14.0\n 17.5.0\npoetry\n 1.1.13\npython\n 3.9.10\nyarn\n 1.22.17\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eUsing the installed languages\u003c/h2\u003e\n\u003cp\u003eTo activate a specific version of a language you have you have three options:\u003c/p\u003e\n\u003ch3\u003eMake it global\u003c/h3\u003e\n\u003cp\u003eYou can make it global, meaning when you run the tool like \u003ccode\u003epython\u003c/code\u003e it'll use\nthis version for the system:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf global python 3.9.10\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch3\u003eMake it local\u003c/h3\u003e\n\u003cp\u003eYou can make it local, which means it will generate a file in the current\ndirectory named \u003ccode\u003e.tool-versions\u003c/code\u003e and so whenever you change into a directory\nit will activate the versions defined in there.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf \u003cspan class=\"hljs-built_in\"\u003elocal\u003c/span\u003e nodejs 12.22.10\n\u003e \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e .tool-versions \nnodejs 12.22.10\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe great thing about this is you can commit that file to git and then anyone\nwho checks out the project and uses \u003ccode\u003easdf\u003c/code\u003e will have the same versions activated!\u003c/p\u003e\n\u003ch3\u003eTemporary\u003c/h3\u003e\n\u003cp\u003eIf you want to activate a version of a language temporarily you can swap to it\nfor the current shell:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf shell golang 1.17.7\n\u003e \u003cspan class=\"hljs-built_in\"\u003eenv\u003c/span\u003e|grep -i ASDF\nASDF_GOLANG_VERSION=1.17.7\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIt sets an environment variable that will have preference over the file. If you\never wonder what versions a directory is using you can run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e\u003e asdf current\ngolang ______ No version \u003cspan class=\"hljs-built_in\"\u003eset\u003c/span\u003e. Run \u003cspan class=\"hljs-string\"\u003e\"asdf \u0026#x3C;global|shell|local\u003e golang \u0026#x3C;version\u003e\"\u003c/span\u003e\nnodejs 12.22.10 .tool-versions\npoetry ______ No version \u003cspan class=\"hljs-built_in\"\u003eset\u003c/span\u003e. Run \u003cspan class=\"hljs-string\"\u003e\"asdf \u0026#x3C;global|shell|local\u003e poetry \u0026#x3C;version\u003e\"\u003c/span\u003e\npython 3.9.10 .tool-versions\nyarn 1.22.17 .tool-versions\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eConclusion\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://asdf-vm.com/\"\u003easdf\u003c/a\u003e is an AWESOME tool to utilize if you find yourself using many\ndifferent languages or many different versions of the same language. You should check it out\nand see if it can improve your workflow.\u003c/p\u003e","category":"Development","date":"2022-02-18T00:00:00Z","tags":["Python","NodeJS","GoLang","Linux"],"title":"Use asdf to manage Python, NodeJS, GoLang and more!"},{"id":["2022","intro_to_just"],"path":"2022/intro_to_just","contentHtml":"\u003cp\u003eI believe every project should have a CLI built around the standard workflows of developing\non the project. Things like:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eInstall dependencies\u003c/li\u003e\n\u003cli\u003eRun tests\u003c/li\u003e\n\u003cli\u003eRun linter and formatters\u003c/li\u003e\n\u003cli\u003eBuild project\u003c/li\u003e\n\u003cli\u003eStart / Stop the docker environment\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe reason I think this is important is because it makes a nice consistent and discoverable\nentrypoint for understanding how you should work in the project. If you only provide the\ninstructions in the \u003ccode\u003eREADME\u003c/code\u003e then you have to remember to update those docs every time you\nadd a new command. Those docs aren't easily testable either.\u003c/p\u003e\n\u003cp\u003eMost of my career the command runner of choice for my projects as been \u003ccode\u003eGNU Make\u003c/code\u003e but it was\ndefinitely the wrong tool for the job. It is a build tool that I bent into shape to work\nas a command runner for me. These days I use the tool \u003ca href=\"https://github.com/casey/just\"\u003ejust\u003c/a\u003e.\u003c/p\u003e\n\u003ch2\u003eIntro to just\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/casey/just\"\u003eJust\u003c/a\u003e is a modern command runner with a similar syntax to \u003ccode\u003emake\u003c/code\u003e\nthat provides a nice way for building out your project CLI! You create a file named \u003ccode\u003ejustfile\u003c/code\u003e\nat the root of your project and then the basic syntax is:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-make\"\u003e\u003cspan class=\"hljs-section\"\u003ehelp:\u003c/span\u003e\n @just --list\n\n\u003cspan class=\"hljs-comment\"\u003e# My first command\u003c/span\u003e\n\u003cspan class=\"hljs-section\"\u003efirst:\u003c/span\u003e\n echo \u003cspan class=\"hljs-string\"\u003e\"Any commands you want to run go here!\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe first \u003ccode\u003ehelp\u003c/code\u003e line defines a command \"help\" for your CLI and it lists out all the other available\ncommans. I always put this line first because \u003ccode\u003ejust\u003c/code\u003e runs the first command in the file if a specific\ncommand isn't requested. The output of this file looks like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ just\nAvailable recipes:\n first \u003cspan class=\"hljs-comment\"\u003e# My first command\u003c/span\u003e\n \u003cspan class=\"hljs-built_in\"\u003ehelp\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eHaving help automatically generated is fantastic! Its also really helpful that it adds the comment\nto the command so that each command is self-documenting. If you run the \u003ccode\u003efirst\u003c/code\u003e command you'll notice\nit also has a feature where it prints out the commands being ran so the user knows exactly what is\nhappening:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ just first\n\u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"Any commands you want to run go here!\"\u003c/span\u003e\nAny commands you want to run go here!\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis doesn't always make sense though, so you can quickly remove that behavior by putting an \u003ccode\u003e@\u003c/code\u003e in front\nof any of the commands, like I did for the \u003ccode\u003ehelp\u003c/code\u003e command above. You can also declare dependencies if\nyou have re-usable parts of your workflow that many of your commands need.\u003c/p\u003e\n\u003cp\u003eFor example, you might want to check versions of things like \u003ccode\u003enode\u003c/code\u003e and \u003ccode\u003epython\u003c/code\u003e before running the install\nof their dependencies. So you could do something like:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-make\"\u003e\u003cspan class=\"hljs-section\"\u003ehelp:\u003c/span\u003e\n @just --list\n\nnode_version := \u003cspan class=\"hljs-string\"\u003e\"v17.6.0\"\u003c/span\u003e\n\n\u003cspan class=\"hljs-comment\"\u003e# Verify system dependencies\u003c/span\u003e\n\u003cspan class=\"hljs-section\"\u003echeck-dependencies:\u003c/span\u003e\n @if [ ! \u003cspan class=\"hljs-string\"\u003e\"$(node --version)\"\u003c/span\u003e = {{ node_version }} ]; \\\n then \\\n echo \u003cspan class=\"hljs-string\"\u003e\"Missing node version: {{ node_version }}\"\u003c/span\u003e; \\\n exit 1; \\\n fi\n\n\u003cspan class=\"hljs-comment\"\u003e# Install frontend\u003c/span\u003e\n\u003cspan class=\"hljs-section\"\u003einstall: check-dependencies\u003c/span\u003e\n @echo \u003cspan class=\"hljs-string\"\u003e\"yarn install\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003ewhich ends up with a CLI that looks like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ just\nAvailable recipes:\n check-dependencies \u003cspan class=\"hljs-comment\"\u003e# Verify system dependencies\u003c/span\u003e\n \u003cspan class=\"hljs-built_in\"\u003ehelp\u003c/span\u003e\n install \u003cspan class=\"hljs-comment\"\u003e# Install frontend\u003c/span\u003e\n\n❯ just install\nMissing node version: v17.6.0\nerror: Recipe `check-dependencies` failed on line 12 with \u003cspan class=\"hljs-built_in\"\u003eexit\u003c/span\u003e code 1\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis opens up a lot of possibilities! In the above \u003ccode\u003ejustfile\u003c/code\u003e you'll notice I'm using a multi-line\ncommand but I have \u003ccode\u003e\\\u003c/code\u003e at the end of each line. This is because \u003ccode\u003ejust\u003c/code\u003e by default is going to run\neach new line in their own shell. So this just makes all those lines run in the same shell.\u003c/p\u003e\n\u003cp\u003eYou do not have to use this syntax though. Just is \u003ccode\u003epolyglot\u003c/code\u003e and can run commands from any language\nyou would like.\u003c/p\u003e\n\u003ch3\u003ePolyglot\u003c/h3\u003e\n\u003cp\u003eIf you want to use a bash script as one of your commands, you can do so by adding a shebang at the top:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-make\"\u003e\u003cspan class=\"hljs-section\"\u003echeck-dependencies:\u003c/span\u003e\n \u003cspan class=\"hljs-comment\"\u003e#!/usr/bin/env bash\u003c/span\u003e\n set -euxo pipefail\n if [ ! \u003cspan class=\"hljs-string\"\u003e\"$(node --version)\"\u003c/span\u003e = {{ node_version }} ];\n then\n echo \u003cspan class=\"hljs-string\"\u003e\"Missing node version: {{ node_version }}\"\u003c/span\u003e\n exit 1\n fi\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow the entire command is using a bash script to execute! This gets really interesting if you want to start\nusing things like python, so if you'd like to change the dependency checker above to python:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003echeck-dependencies:\n \u003cspan class=\"hljs-comment\"\u003e#!/usr/bin/env python3\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e subprocess\n result = subprocess.run(\n [\u003cspan class=\"hljs-string\"\u003e'node'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'--version'\u003c/span\u003e],\n stdout=subprocess.PIPE\n )\n \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e result != \u003cspan class=\"hljs-string\"\u003e\"{{ node_version }}\"\u003c/span\u003e:\n \u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003ef\"Missing node version: {{ node_version }}\"\u003c/span\u003e)\n exit(\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can even tell \u003ccode\u003ejust\u003c/code\u003e that you want to use a specific language for all commands!\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003eset shell := [\u003cspan class=\"hljs-string\"\u003e\"python3\"\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e\"-c\"\u003c/span\u003e]\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis not only affects the commands you have in your recipe but also anything inside\nbackticks! So something like:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-make\"\u003e`print(\u003cspan class=\"hljs-string\"\u003e\"Rust is the best programming language\"\u003c/span\u003e)`\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIt would run through python instead of the shell.\u003c/p\u003e\n\u003ch3\u003eEnviornment Files\u003c/h3\u003e\n\u003cp\u003eOne of the other modern things \u003ccode\u003ejust\u003c/code\u003e adds to your workflow is the ability to utilize dotenv\nfiles. So for example if you want to define which port you launch your http server on, you can\ncreate a file called \u003ccode\u003e.env\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eWEBSERVER_PORT=9000\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eand then utilize it in your \u003ccode\u003ejustfile\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-make\"\u003eset dotenv-load\n\n\u003cspan class=\"hljs-section\"\u003ehttp:\u003c/span\u003e\n @echo \u003cspan class=\"hljs-string\"\u003e\"Starting webserver in current directory\"\u003c/span\u003e\n python3 -m http.server $WEBSERVER_PORT\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWhen you run \u003ccode\u003ejust http\u003c/code\u003e it'll launch the http server on port 9000. One important line\nin this file is \u003ccode\u003eset dotenv-load\u003c/code\u003e, it will not load the \u003ccode\u003e.env\u003c/code\u003e file without you telling it to.\u003c/p\u003e\n\u003ch2\u003eDon't use language specific scripts!\u003c/h2\u003e\n\u003cp\u003eI'n not a fan of language specific command runners like \u003ccode\u003epackage.json\u003c/code\u003e in the node community.\u003c/p\u003e\n\u003cp\u003eIt always frustrates me when I start working on a project that heavily uses \u003ccode\u003escripts\u003c/code\u003e in their\npackage.json instead of using a real command runner. \u003ccode\u003ejson\u003c/code\u003e is not a great format for writing\ndiscoverable CLI commands. For example if you wanted to write a \u003ccode\u003enext.js\u003c/code\u003e build script:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-json\"\u003e \u003cspan class=\"hljs-attr\"\u003e\"scripts\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e:\u003c/span\u003e \u003cspan class=\"hljs-punctuation\"\u003e{\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"predeploy\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"yarn build \u0026#x26;\u0026#x26; yarn export \u0026#x26;\u0026#x26; touch dist/.nojekyll \u0026#x26;\u0026#x26; echo sontek.net \u003e dist/CNAME\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e,\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"deploy\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"gh-pages -d dist -t true\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e,\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"build\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"next build\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e,\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"export\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"next export -o dist/\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e,\u003c/span\u003e\n \u003cspan class=\"hljs-punctuation\"\u003e}\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e,\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eCombining all those commands is really messy and not easily understandable through \u003ccode\u003eyarn run\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ yarn run\nyarn run v1.22.17\ninfo Commands available from binary scripts: autoprefixer, browserslist, css-blank-pseudo, css-has-pseudo, css-prefers-color-scheme, cssesc, esparse, esvalidate, extract-zip, gh-pages, gh-pages-clean, js-yaml, loose-envify, nanoid, next, prettier, resolve, rimraf, semver, svgo, uvu\ninfo Project commands\n - build\n next build\n - deploy\n gh-pages -d dist -t \u003cspan class=\"hljs-literal\"\u003etrue\u003c/span\u003e\n - \u003cspan class=\"hljs-built_in\"\u003eexport\u003c/span\u003e\n next \u003cspan class=\"hljs-built_in\"\u003eexport\u003c/span\u003e -o dist/\n - predeploy\n yarn build \u0026#x26;\u0026#x26; yarn \u003cspan class=\"hljs-built_in\"\u003eexport\u003c/span\u003e \u0026#x26;\u0026#x26; \u003cspan class=\"hljs-built_in\"\u003etouch\u003c/span\u003e dist/.nojekyll \u0026#x26;\u0026#x26; \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e sontek.net \u003e dist/CNAME\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eI'd much rather have this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ just\nAvailable recipes:\n build \u003cspan class=\"hljs-comment\"\u003e# Build frontend assets\u003c/span\u003e\n deploy \u003cspan class=\"hljs-comment\"\u003e# Deploy assets to cloudfront\u003c/span\u003e\n \u003cspan class=\"hljs-built_in\"\u003eexport\u003c/span\u003e \u003cspan class=\"hljs-comment\"\u003e# Export to static assets (no SSR)\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eConclusion\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/casey/just\"\u003eJust\u003c/a\u003e is a wonderful tool for building project specific CLIs without much effort. It is\na great replacement for \u003ccode\u003eMake\u003c/code\u003e if you are using it as a command runner and it has most of the features you'd need.\u003c/p\u003e\n\u003cp\u003eI recommend adding a \u003ccode\u003ejustfile\u003c/code\u003e to your projects today! If you'd like to see a real world example of how to use \u003ccode\u003ejust\u003c/code\u003e,\nyou can check out the one I use to maintain my \u003ca href=\"https://github.com/sontek/homies/blob/master/justfile\"\u003ehome directory\u003c/a\u003e!\u003c/p\u003e","category":"Development","date":"2022-02-26T00:00:00Z","tags":["Linux"],"title":"Automate project workflows with the command runner Just!"},{"id":["2022","local_kubeadm_cluster"],"path":"2022/local_kubeadm_cluster","contentHtml":"\u003cp\u003eI’m going to show you how to get a real kubernetes cluster setup locally on top of virtual\nmachines! I’ll be using multipass but feel free to use virtualbox, proxmox, or whatever your\nfavorite cloud provider is.\u003c/p\u003e\n\u003cp\u003ekubeadm a production ready kubernetes install tool and I prefer to use it over minikube, kind,\netc. because it gives you a more real world experience for \u003cem\u003emanaging\u003c/em\u003e the kubernetes cluster.\nThis isn’t important if you are a user of the cluster but if you have to run your own this is\na great way to gain some daily experience.\u003c/p\u003e\n\u003cp\u003eThe kubernetes documentation on kubeadm is great and you can find it \u003ca href=\"https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eThe differences between this blog and the kubernetes docs is that they leave a lot of decisions\nup to the reader such as:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003echoosing a container runtime\u003c/li\u003e\n\u003cli\u003eSelecting and installing a CNI (container network interface)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI’m going to be opinionated and make specific technology decisions such as using containerd and\ncilium so that you don't have to think about those decisions.\u003c/p\u003e\n\u003ch2\u003eGetting your Virtual Machines setup!\u003c/h2\u003e\n\u003cp\u003eThe minimum requirements for a control plane node in kubernetes is 2gb of RAM and 2 CPUs. Since\nwe actually want to be able to schedule workloads on the workers afterwards we are going to setup\na cluster that looks like this:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eControl Plane: 2gb RAM, 2 CPU\u003c/li\u003e\n\u003cli\u003eWorker: 4gb RAM, 2 CPU\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSince we’ll be using multipass to launch the nodes, we can do that now:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ multipass launch -c 2 -m 4G -d 10G -n controlplane 22.04\n❯ multipass launch -c 2 -m 4G -d 10G -n worker 22.04\n❯ multipass list\nName State IPv4 Image\ncontrolplane Running 192.168.64.7 Ubuntu 22.04 LTS\nworker Running 192.168.64.8 Ubuntu 22.04 LTS\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we can start working on our controlplane first, lets shell in:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ multipass shell controlplane\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eLets first add the kubernetes repo to the system so we have access to all the kubernetes tools:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb http://apt.kubernetes.io/ kubernetes-xenial main\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/kubernetes.list\n\n❯ curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg|sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/k8s.gpg\n❯ sudo apt-get update \u0026#x26;\u0026#x26; sudo apt-get upgrade -y\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow that our system is setup, we can move on to getting a container runtime.\u003c/p\u003e\n\u003ch2\u003eGetting your Container Runtime!\u003c/h2\u003e\n\u003cp\u003eBefore we start pulling in kubernetes components we need to get a container runtime setup on the\nmachine. We we are going to use containerd for this purpose. You can view the docs of for it\n\u003ca href=\"https://github.com/containerd/containerd/blob/main/docs/getting-started.md\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eWhich will download the latest binary and set it up. I’m going to walk you through how to do it\nusing the version packaged with Ubuntu which could be older than the latest release.\u003c/p\u003e\n\u003cp\u003eFirst thing we want to do is configure the networking to allow iptables to manage:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/modules-load.d/k8s.conf\noverlay\nbr_netfilter\nEOF\u003c/span\u003e\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/sysctl.d/k8s.conf\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nnet.ipv4.ip_forward = 1\nEOF\u003c/span\u003e\n\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe also need to disable some default systemd settings for \u003ccode\u003erp_filter\u003c/code\u003e because\nthey are not compatible with cilium. See the bug report\n\u003ca href=\"https://github.com/cilium/cilium/commit/cabc6581b8128681f4ed23f8d6dc463180eea61e\"\u003ehere\u003c/a\u003e\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo sed -i -e \u003cspan class=\"hljs-string\"\u003e'/net.ipv4.conf.*.rp_filter/d'\u003c/span\u003e $(grep -ril \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e /etc/sysctl.d/ /usr/lib/sysctl.d/)\n❯ sudo sysctl -a | grep \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e | awk \u003cspan class=\"hljs-string\"\u003e'{print $1\" = 0\"}'\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e -a /etc/sysctl.d/1000-cilium.conf\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen we need to refresh sysctl so those settings are applied:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo systemctl restart systemd-modules-load\n❯ sudo sysctl --system\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou should see it applying all the changes:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e* Applying /etc/sysctl.d/k8s.conf ...\n\u003cspan class=\"hljs-attr\"\u003enet.bridge.bridge-nf-call-ip6tables\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.bridge.bridge-nf-call-iptables\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.ip_forward\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIf you do not, the netfilter module may not have loaded properly:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ lsmod |grep br_netfilter\nbr_netfilter 28672 0\nbridge 176128 1 br_netfilter\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou want to make sure \u003ccode\u003erp_filter\u003c/code\u003e is \u003ccode\u003e0\u003c/code\u003e everywhere as well for cilium:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e❯ sudo sysctl -a | grep '\\.rp_filter'\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.all.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.cilium_host.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.cilium_net.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.cilium_vxlan.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.default.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.enp0s1.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.lo.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.lxc0965b7b545f7.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.lxcb05ffd84ab74.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow lets pull down the container runtime we’ll be using which is containerd.\u003c/p\u003e\n\u003cp\u003eUbuntu ships with a very old version of containerd so you need to upgrade to\nthe version shipped from the docker repos:\nYou can find which versions are available by running:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker.gpg\n❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb https://download.docker.com/linux/ubuntu \u003cspan class=\"hljs-subst\"\u003e$(lsb_release -cs)\u003c/span\u003e stable\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/docker.list\n❯ sudo apt-get update\n\u003c/code\u003e\u003c/pre\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-cache madison containerd.io\ncontainerd.io | 1.6.8-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.6.7-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.6.6-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.6.4-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.5.11-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.5.10-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe are going to use the latest version available which was 1.6.8-1\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-get install containerd.io=1.6.8-1 -y\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen we'll setup a configuration that enables containerd to use the systemd\ncgroup. We are hard coding this config instead of using \u003ccode\u003econtainerd config default\u003c/code\u003e\nbecause that currently has had a \u003ca href=\"https://github.com/containerd/containerd/issues/4574\"\u003ebug\u003c/a\u003e\nfor many years that generates an invalid config.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/containerd/config.toml\nversion = 2\n[plugins]\n [plugins.\"io.containerd.grpc.v1.cri\"]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc]\n runtime_type = \"io.containerd.runc.v2\"\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc.options]\n SystemdCgroup = true\nEOF\u003c/span\u003e\n\n❯ sudo systemctl restart containerd.service\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can verify its running with ctr:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo ctr --address /var/run/containerd/containerd.sock containers list\nCONTAINER IMAGE RUNTIME\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow that this is working we can move on to getting kubernetes installed!\u003c/p\u003e\n\u003ch2\u003eUsing kubeadm!\u003c/h2\u003e\n\u003cp\u003eNow we need to get the kubernetes tools installed onto the system. I’m going to be using 1.23\nbut to find the latest version you can run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-cache madison kubeadm|\u003cspan class=\"hljs-built_in\"\u003ehead\u003c/span\u003e -n2\n kubeadm | 1.23.5-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages\n kubeadm | 1.23.4-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen install the version you want, we install kubelet and kubeadm here to make\nsure the versions align:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-get install kubeadm=1.23.5-00 kubelet=1.23.5-00 kubectl=1.23.5-00 -y\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will pull in a few tools, including an alternative to \u003ccode\u003ectr\u003c/code\u003e that we used earlier called\n\u003ccode\u003ecrictl\u003c/code\u003e. You can check that it is available to you doing this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo crictl --runtime-endpoint=unix:///var/run/containerd/containerd.sock ps\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe can finally init our cluster:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo kubeadm init\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce that finishes running it should give you some tips setup your configuration, it should look like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e -p \u003cspan class=\"hljs-variable\"\u003e$HOME\u003c/span\u003e/.kube\n❯ sudo \u003cspan class=\"hljs-built_in\"\u003ecp\u003c/span\u003e -i /etc/kubernetes/admin.conf \u003cspan class=\"hljs-variable\"\u003e$HOME\u003c/span\u003e/.kube/config\n❯ sudo \u003cspan class=\"hljs-built_in\"\u003echown\u003c/span\u003e $(\u003cspan class=\"hljs-built_in\"\u003eid\u003c/span\u003e -u):$(\u003cspan class=\"hljs-built_in\"\u003eid\u003c/span\u003e -g) \u003cspan class=\"hljs-variable\"\u003e$HOME\u003c/span\u003e/.kube/config\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can run those on the master node for now, but later I'll show you how to move\nthe config to your host computer.\u003c/p\u003e\n\u003cp\u003eNow you should be able to check that your node is not ready yet:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get nodes\nNAME STATUS ROLES AGE VERSION\ncontrolplane NotReady control-plane,master 4m16s v1.23.5\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cem\u003eNote\u003c/em\u003e: If you recieve \"The connecto to the server was refused\" error,\nThe cluster starting up and getting all the dependencies running could take\na bit of time. So if you aren't able to communicate right away you can check\nwhich pods are up and running with \u003ccode\u003ecrictl\u003c/code\u003e. You'll need \u003ccode\u003ekube-apiserver\u003c/code\u003e up\nand running. If it isn't you can check:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo crictl --runtime-endpoint=unix:///var/run/containerd/containerd.sock ps -a\nCONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD\n8322192c4605c bd8cc6d582470 36 seconds ago Running kube-proxy 4 344c4f7fffbe8 kube-proxy-drm46\n30ce27c40adb2 81a4a8a4ac639 2 minutes ago Exited kube-controller-manager 4 3a819c3a864b2 kube-controller-manager-controlplane\n7709fd5e92898 bd8cc6d582470 2 minutes ago Exited kube-proxy 3 7cc6922c82015 kube-proxy-drm46\n10432b81d7c61 3767741e7fba7 2 minutes ago Exited kube-apiserver 4 e64ddf3679d98 kube-apiserver-controlplane\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003ewhich will show you pods that have exited. You can grab the container ID for\nkube-apiserver and read its logs:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo crictl --runtime-endpoint=unix:///var/run/containerd/containerd.sock logs 10432b81d7c61\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThere are a few ways to figure out why the node isn’t ready yet. Usually I would check the\n\u003ccode\u003ekubelet\u003c/code\u003e logs first:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo journalctl -flu kubelet\n-- Logs begin at Sun 2022-04-17 19:22:19 AST. --\nApr 17 20:53:15 controlplane kubelet[19727]: E0417 20:53:15.951350 19727 kubelet.go:2347] \u003cspan class=\"hljs-string\"\u003e\"Container runtime network not ready\"\u003c/span\u003e networkReady=\u003cspan class=\"hljs-string\"\u003e\"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized\"\u003c/span\u003e\nApr 17 20:53:20 controlplane kubelet[19727]: E0417 20:53:20.952148 19727 kubelet.go:2347] \u003cspan class=\"hljs-string\"\u003e\"Container runtime network not ready\"\u003c/span\u003e networkReady=\u003cspan class=\"hljs-string\"\u003e\"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIt is clear the problem is that we are missing the CNI. The other way you can find out what is\ngoing on is describing the node:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl describe node controlplane\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will have a lot of information but if you scroll through there looking at \u003ccode\u003eReason\u003c/code\u003e you\nmight see something useful. In this case under \u003ccode\u003eLease\u003c/code\u003e you would see:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl describe node controlplane|grep NotReady\nReady False Sun, 17 Apr 2022 20:53:37 -0400 Sun, 17 Apr 2022 20:43:07 -0400 KubeletNotReady container runtime network not ready: NetworkReady=\u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialize\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eLets get our CNI installed, we’ll be using cilium!\u003c/p\u003e\n\u003ch2\u003eSetting up your CNI!\u003c/h2\u003e\n\u003cp\u003eCilium has great documentation over \u003ca href=\"https://docs.cilium.io/en/v1.9/gettingstarted/k8s-install-kubeadm/\"\u003ehere\u003c/a\u003e,\nbut I’ll walk you through it anyways. I do recommend checking out their documentation so you\nare familiar with it. We will use \u003ccode\u003ehelm\u003c/code\u003e to pull down the version of cilium we want:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl -fsSL https://baltocdn.com/helm/signing.asc | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/helm.gpg\n\n❯ sudo apt-get install apt-transport-https --\u003cspan class=\"hljs-built_in\"\u003eyes\u003c/span\u003e\n\n❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb https://baltocdn.com/helm/stable/debian/ all main\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/helm-stable-debian.list\n\n❯ sudo apt-get update\n❯ sudo apt-get install helm\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we can install cilium! It is \u003cem\u003every\u003c/em\u003e important that you pay attention to the\ncompatibility of cilium with the version of kubernetes you are intstalling. Check\nthe compatibility list \u003ca href=\"https://docs.cilium.io/en/v1.12/concepts/kubernetes/compatibility/\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ helm repo add cilium https://helm.cilium.io/\n❯ helm repo update\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce the repo is added you can list the versions available:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ helm search repo -l|\u003cspan class=\"hljs-built_in\"\u003ehead\u003c/span\u003e -n8\nNAME \tCHART VERSION\tAPP VERSION\tDESCRIPTION\ncilium/cilium \t1.12.1 \t1.12.1 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.12.0 \t1.12.0 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.8 \t1.11.8 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.7 \t1.11.7 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.6 \t1.11.6 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.5 \t1.11.5 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.4 \t1.11.4 \teBPF-based Networking, Security, and Observability\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo we want \u003ccode\u003e1.11.4\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ helm install cilium cilium/cilium --namespace kube-system --version 1.11.4\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow our node should be ready!\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get node\nNAME STATUS ROLES AGE VERSION\ncontrolplane Ready control-plane,master 24m v1.23.5\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eTime to join our worker to the cluster!\u003c/p\u003e\n\u003ch2\u003eJoining a worker to the cluster!\u003c/h2\u003e\n\u003cp\u003eWe have to go through the same steps as the controlplane to get the point that we have a\ncontainer runtime and \u003ccode\u003ekubeadm\u003c/code\u003e. I’m not going to talk about the commands a second time but\nI’ll re-iterate them here for ease of following along.\u003c/p\u003e\n\u003cp\u003eFirst open up another shell and connect to the worker:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ multipass shell worker\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow run the following commands:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb http://apt.kubernetes.io/ kubernetes-xenial main\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/kubernetes.list\n❯ curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg|sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/k8s.gpg\n❯ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker.gpg\n❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb https://download.docker.com/linux/ubuntu \u003cspan class=\"hljs-subst\"\u003e$(lsb_release -cs)\u003c/span\u003e stable\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/docker.list\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/modules-load.d/k8s.conf\noverlay\nbr_netfilter\nEOF\u003c/span\u003e\n\n❯ sudo sed -i -e \u003cspan class=\"hljs-string\"\u003e'/net.ipv4.conf.*.rp_filter/d'\u003c/span\u003e $(grep -ril \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e /etc/sysctl.d/ /usr/lib/sysctl.d/)\n❯ sudo sysctl -a | grep \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e | awk \u003cspan class=\"hljs-string\"\u003e'{print $1\" = 0\"}'\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e -a /etc/sysctl.d/1000-cilium.conf\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/sysctl.d/k8s.conf\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nnet.ipv4.ip_forward = 1\nEOF\u003c/span\u003e\n\n❯ sudo systemctl restart systemd-modules-load\n❯ sudo sysctl --system\n\n❯ sudo apt-get update \u0026#x26;\u0026#x26; sudo apt-get upgrade -y\n❯ sudo apt-get install containerd.io=1.6.8-1 -y\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/containerd/config.toml\nversion = 2\n[plugins]\n [plugins.\"io.containerd.grpc.v1.cri\"]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc]\n runtime_type = \"io.containerd.runc.v2\"\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc.options]\n SystemdCgroup = true\nEOF\u003c/span\u003e\n\n❯ sudo systemctl restart containerd.service\n❯ sudo apt-get install kubeadm=1.23.5-00 kubelet=1.23.5-00 kubectl=1.23.5-00 -y\n\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eFrom there we should be ready to join the cluster. When we ran \u003ccode\u003ekubeadm init\u003c/code\u003e previously it\nprinted a join command out that we could use but I’m going to show you how to do it if you\nwere coming back later and no longer had that token.\u003c/p\u003e\n\u003cp\u003eBack on the \u003cem\u003econtroplane\u003c/em\u003e node run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubeadm token create --print-join-command\nkubeadm \u003cspan class=\"hljs-built_in\"\u003ejoin\u003c/span\u003e 192.168.64.7:6443 --token wxs197.cco6mjj9ricvu8ov --discovery-token-ca-cert-hash sha256:bd01c065240fa76f30a02ecb70a8cea6e329c9678994d4da1f6ccac7694b97fb\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow copy that command and run it with \u003ccode\u003esudo\u003c/code\u003e on the worker:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo kubeadm \u003cspan class=\"hljs-built_in\"\u003ejoin\u003c/span\u003e 192.168.64.7:6443 --token wxs197.cco6mjj9ricvu8ov --discovery-token-ca-cert-hash sha256:bd01c065240fa76f30a02ecb70a8cea6e329c9678994d4da1f6ccac7694b97fb\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAfter this completes it’ll take a minute or two for everything to be synced up but if you go\nback to the master node you should have 2 ready nodes now:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get nodes\nNAME STATUS ROLES AGE VERSION\ncontrolplane Ready control-plane,master 46m v1.23.5\nworker Ready \u0026#x3C;none\u003e 79s v1.23.5\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eAccessing the cluster outside of the VMs!\u003c/h2\u003e\n\u003cp\u003eNow the final part is to get the \u003ccode\u003eadmin.conf\u003c/code\u003e as a kubeconfig on your machine so you can control\nit from outside of the cluster. To do this we can use scp\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003emultipass transfer controlplane:/home/ubuntu/.kube/config local.config\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNormally kubernetes configuration is in ~/.kube/config but I like to maint a separate file for\neach cluster and then I set the \u003ccode\u003eKUBECONFIG\u003c/code\u003e env var to access it.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003eexport\u003c/span\u003e KUBECONFIG=local.config\n❯ kubectl get nodes\nNAME STATUS ROLES AGE VERSION\ncontrolplane Ready control-plane,master 56m v1.23.5\nworker Ready \u0026#x3C;none\u003e 11m v1.23.5\n\u003c/code\u003e\u003c/pre\u003e","category":"Kubernetes","date":"2022-04-17T00:00:00Z","tags":["Linux","Kubernetes","DevOps","SRE"],"title":"Running a kubernetes cluster locally with kubeadm"},{"id":["2023","local_k8s_with_kind"],"path":"2023/local_k8s_with_kind","contentHtml":"\u003cp\u003ePreviously I \u003ca href=\"/blog/2022/local_kubeadm_cluster\"\u003eshowed\u003c/a\u003e how to run kubernetes\nlocally with \u003ccode\u003ekubeadm\u003c/code\u003e and VMs but sometimes that is overkill so I wanted to\nshow how to run \u003ca href=\"https://kind.sigs.k8s.io/\"\u003ekind\u003c/a\u003e which is \"kuberetes in\ndocker\".\u003c/p\u003e\n\u003ch1\u003eCreating your first cluster\u003c/h1\u003e\n\u003cp\u003ekind is a very flexible way to run kubernetes locally and allows you to run\nsingle node or multinode clusters while having the flexibility to use all\nthe features of kubernetes success as ingress.\u003c/p\u003e\n\u003cp\u003eTo create your first cluster it is as simple as running:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kind create cluster \n\nCreating cluster \u003cspan class=\"hljs-string\"\u003e\"kind\"\u003c/span\u003e ...\n ✓ Ensuring node image (kindest/node:v1.27.3) 🖼 \n ✓ Preparing nodes 📦 \n ✓ Writing configuration 📜 \n ✓ Starting control-plane 🕹️ \n ✓ Installing CNI 🔌 \n ✓ Installing StorageClass 💾 \nSet kubectl context to \u003cspan class=\"hljs-string\"\u003e\"kind-kind\"\u003c/span\u003e\nYou can now use your cluster with:\n\nkubectl cluster-info --context kind-kind\n\nHave a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/\u003cspan class=\"hljs-comment\"\u003e#community 🙂\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou now have a functioning kubernetes cluster and you\ncan view what it created:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ k get node\nNAME STATUS ROLES AGE VERSION\nkind-control-plane Ready control-plane 4m26s v1.27.3\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can also verify that it is running inside docker:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n1c3ba74dc29b kindest/node:v1.27.3 \u003cspan class=\"hljs-string\"\u003e\"/usr/local/bin/entr…\"\u003c/span\u003e 3 minutes ago Up 3 minutes 127.0.0.1:59327-\u003e6443/tcp kind-control-plane\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eMaking the cluster useful\u003c/h1\u003e\n\u003cp\u003eThere are a few things you'll notice with the command we ran originally:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eIt grabbed the latest kubernetes version available\u003c/li\u003e\n\u003cli\u003eIt is running a single node cluster\u003c/li\u003e\n\u003cli\u003eNo ingress available\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eLuckily kind makes it really easy to customize your local cluster to be what\nyou want it to be by using a \u003ccode\u003eYAML\u003c/code\u003e configuration.\u003c/p\u003e\n\u003cp\u003eCreate the configuration:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eCluster\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ekind.x-k8s.io/v1alpha4\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enodes:\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003econtrol-plane\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ekubeadmConfigPatches:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e|\n kind: InitConfiguration\n nodeRegistration:\n kubeletExtraArgs:\n node-labels: \"ingress-ready=true\"\n\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003eextraPortMappings:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003econtainerPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e80\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ehostPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e80\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eprotocol:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eTCP\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003econtainerPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e443\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ehostPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e443\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eprotocol:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eTCP\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eworker\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eworker\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eworker\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWith this we've now generated a 4 node cluster where we have a single\ncontrol-plane and three workers. Then we defined some extra configuration on\nthe control-plane:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ekubeadmConfigPatches\u003c/strong\u003e: We want to change the default configuration the\ncluster uses so it'll tag the nodes with the \u003ccode\u003eingress-ready\u003c/code\u003e label so the\ncontroller will use them.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eextraPortMappings\u003c/strong\u003e: allow the local host to make requests to the Ingress controller over ports 80/443\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003enode-labels\u003c/strong\u003e: only allow the ingress controller to run on specific node(s) matching the label selector\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSo now we can create the new cluster with the configuration. Save that config\nas \u003ccode\u003ekind_config.yml\u003c/code\u003e and then run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kind create cluster --image kindest/node:v1.25.11 --config kind_config.yml --name kind-multinode\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis time I've added a few additional flags on the commandline. \u003ccode\u003e--image\u003c/code\u003e\nallows us to use a different version of kubernetes and \u003ccode\u003e--name\u003c/code\u003e allows us to\nmake more than one cluster. So if you didn't destroy the first cluster you'll\nsee we have two of them now:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kind get clusters\nkind\nkind-multinode\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003ebut \u003ccode\u003ekind\u003c/code\u003e will swap the to the newest cluster by default:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl config current-context\nkind-kind-multinode\n\n❯ kubectl get node\nNAME STATUS ROLES AGE VERSION\nkind-multinode-control-plane Ready control-plane 107s v1.25.11\nkind-multinode-worker Ready \u0026#x3C;none\u003e 88s v1.25.11\nkind-multinode-worker2 Ready \u0026#x3C;none\u003e 88s v1.25.11\nkind-multinode-worker3 Ready \u0026#x3C;none\u003e 88s v1.25.11\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we need to get the \u003ccode\u003eingress-nginx\u003c/code\u003e controller installed so we can start\nusing our cluster with ingress:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe manifests contains \u003ccode\u003ekind\u003c/code\u003e specific patches to forward the hostPorts to the ingress controller, set taint tolerations and schedule it to the custom labelled node.\u003c/p\u003e\n\u003cp\u003eThis will take a little bit of time to get up and running, you can monitor it\nby running:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003ekubectl \u003cspan class=\"hljs-built_in\"\u003ewait\u003c/span\u003e --namespace ingress-nginx \\\n --\u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e=condition=ready pod \\\n --selector=app.kubernetes.io/component=controller \\\n --\u003cspan class=\"hljs-built_in\"\u003etimeout\u003c/span\u003e=90s\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eor just manually check the status:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get all -n ingress-nginx\nNAME READY STATUS RESTARTS AGE\npod/ingress-nginx-admission-create-bbmlc 0/1 Completed 0 68s\npod/ingress-nginx-admission-patch-qlnr8 0/1 Completed 2 68s\npod/ingress-nginx-controller-5f748f78c8-6tc6b 0/1 ContainerCreating 0 68s\n\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nservice/ingress-nginx-controller NodePort 10.96.228.248 \u0026#x3C;none\u003e 80:31771/TCP,443:31759/TCP 68s\nservice/ingress-nginx-controller-admission ClusterIP 10.96.180.126 \u0026#x3C;none\u003e 443/TCP 68s\n\nNAME READY UP-TO-DATE AVAILABLE AGE\ndeployment.apps/ingress-nginx-controller 0/1 1 0 68s\n\nNAME DESIRED CURRENT READY AGE\nreplicaset.apps/ingress-nginx-controller-5f748f78c8 1 1 0 68s\n\nNAME COMPLETIONS DURATION AGE\njob.batch/ingress-nginx-admission-create 1/1 22s 68s\njob.batch/ingress-nginx-admission-patch 1/1 35s 68s\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce \u003ccode\u003eingress-nginx-controller\u003c/code\u003e is in \u003ccode\u003eRunning\u003c/code\u003e state you are read to go!\u003c/p\u003e\n\u003ch1\u003eDeploying your first app\u003c/h1\u003e\n\u003cp\u003eTo prove that the cluster is working correctly we will deploy\n\u003ca href=\"https://github.com/Kong/httpbin\"\u003ehttpbin\u003c/a\u003e which is a nice little API server\nso we can prove everything is working.\u003c/p\u003e\n\u003cp\u003eCreate a \u003ccode\u003ehttbin.yml\u003c/code\u003e file and paste this into it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-meta\"\u003e---\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ev1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eService\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003elabels:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eservice:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eports:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttp\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eport:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8000\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003etargetPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8080\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eselector:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n\u003cspan class=\"hljs-meta\"\u003e---\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eapps/v1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eDeployment\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ereplicas:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eselector:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematchLabels:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eversion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ev1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003etemplate:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003elabels:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eversion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ev1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003econtainers:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003eimage:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edocker.io/mccutchen/go-httpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eimagePullPolicy:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eIfNotPresent\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eports:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003econtainerPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8080\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis is creating a couple of Kubernetes resources:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003eService\u003c/code\u003e: This is exposing the port to the ingress\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eDeployment\u003c/code\u003e: This is actually launching the service\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSo we are not using the ingress yet but we can prove that we can launch the\nservice at least. So apply those manifests:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl apply -f httpbin.yml \nservice/httpbin created\ndeployment.apps/httpbin created\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou should see two pods come up. You should wait for them to get into the\n\u003ccode\u003eRunning\u003c/code\u003e status:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get pod -o wide\nNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES\nhttpbin-5c5494967-2z5wz 1/1 Running 0 48s 10.244.3.3 kind-multinode-worker3 \u0026#x3C;none\u003e \u0026#x3C;none\u003e\nhttpbin-5c5494967-9lf47 1/1 Running 0 72s 10.244.1.2 kind-multinode-worker \u0026#x3C;none\u003e \u0026#x3C;none\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe can now use port forwarding to access it. \u003ccode\u003ehttpbin\u003c/code\u003e is exposed on \u003ccode\u003e8000\u003c/code\u003e so\nlets create port \u003ccode\u003e9000\u003c/code\u003e on our host that forwards to it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl port-forward service/httpbin 9000:8000\nForwarding from 127.0.0.1:9000 -\u003e 80\nForwarding from [::1]:9000 -\u003e 80\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can access it via:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl localhost:9000/get \n{\n \u003cspan class=\"hljs-string\"\u003e\"args\"\u003c/span\u003e: {},\n \u003cspan class=\"hljs-string\"\u003e\"headers\"\u003c/span\u003e: {\n \u003cspan class=\"hljs-string\"\u003e\"Accept\"\u003c/span\u003e: [\n \u003cspan class=\"hljs-string\"\u003e\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8\"\u003c/span\u003e\n ],\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eUsing Ingress\u003c/h1\u003e\n\u003cp\u003eNow to use the ingress rather than port forwarding we create one additional\nresource, the \u003ccode\u003eIngress\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003enetworking.k8s.io/v1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eIngress\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin-ingress\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eannotations:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003enginx.ingress.kubernetes.io/rewrite-target:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e/$2\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eingressClassName:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003enginx\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003erules:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ehttp:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epaths:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e/httpbin(/|$)(.*)\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epathType:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eImplementationSpecific\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ebackend:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eservice:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eport:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003enumber:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8000\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThere are a few critical options here. The first is the annotation to rewrite\nthe path so it doesn't include \u003ccode\u003e/httpbin/\u003c/code\u003e when it sends the request to the\nservice and then the \u003ccode\u003epath\u003c/code\u003e and \u003ccode\u003epathType\u003c/code\u003e so it knows which paths to send to\nwhich service.\u003c/p\u003e\n\u003cp\u003eNow you should be able to hit your local host and get routed to your\nkubernetes service:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl localhost/httpbin/get\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSuccess! Now you have a multinode kubernetes cluster that has an ingress\ncontroller!\u003c/p\u003e\n\u003ch1\u003eNext Steps\u003c/h1\u003e\n\u003cp\u003eThe cluster can be used like a production cluster now for local\ndevelopment! You could setup Grafana, ArgoCD, etc. to run\ninside the cluster.\u003c/p\u003e","category":"Kubernetes","date":"2023-07-21T00:00:00Z","tags":["Linux","Kubernetes","DevOps","SRE"],"title":"Running a kubernetes cluster locally with kind"},{"id":["old","preparing_cloud_images_with_libvirt"],"path":"old/preparing_cloud_images_with_libvirt","contentHtml":"\u003cp\u003eThis article will show you how to use libvirt to create base images that\ncan be uploaded to OpenStack.\u003c/p\u003e\n\u003ch1\u003eWhy would you want to do this?\u003c/h1\u003e\n\u003cp\u003eLinux distributions like Fedora and Ubuntu already ship \"cloud\" images\nand most providers also have their own custom images for you to use, but\nI find it much more comforting to have full control of the software that\nis installed and I like the ability to easily apply new security patches\nto base images.\u003c/p\u003e\n\u003cp\u003eI wouldn't use images to replace config management (CM) with something\nlike \u003ca href=\"http://www.saltstack.com/\"\u003eSalt\u003c/a\u003e or\n\u003ca href=\"http://www.ansible.com/\"\u003eAnsible\u003c/a\u003e but they are nice to give sane system\ndefaults in things like \u003ccode\u003egrub.conf\u003c/code\u003e, \u003ccode\u003esysctl.conf\u003c/code\u003e, and shipping a Chef\nor Salt agent so that your CM engine can communicate with your server\nright away.\u003c/p\u003e\n\u003ch1\u003eSetting up your environment\u003c/h1\u003e\n\u003cp\u003eThe first thing you need to do is get a minimal install disk for the\nLinux distribution you want to use. I prefer using Fedora netinst disks\nbut another popular option is Ubuntu Server.\u003c/p\u003e\n\u003cp\u003eTo get the latest Fedora here, you can choose \"netinst\" under Direct\nDownloads: \u003ca href=\"http://fedoraproject.org/en/get-fedora-all\"\u003ehttp://fedoraproject.org/en/get-fedora-all\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eTo get the latest Ubuntu you can go here:\n\u003ca href=\"http://www.ubuntu.com/download/server\"\u003ehttp://www.ubuntu.com/download/server\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eOnce you have acquired your distribution of choice you just need to\nverify that you have \u003ccode\u003evirt-install\u003c/code\u003e and \u003ccode\u003evirt-viewer\u003c/code\u003e installed:\u003c/p\u003e\n\u003cp\u003eFedora:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eyum install virt-install virt-viewer\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eUbuntu:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eapt-get install virtinst virt-viewer\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIf you prefer a graphical user interface, you may use \u003ccode\u003evirt-manager\u003c/code\u003e\ninstead, but I try to keep everything in the CLI; that way it can be\nrepeated easily.\u003c/p\u003e\n\u003ch1\u003ePreparing your disk\u003c/h1\u003e\n\u003cp\u003eNow that you have a base ISO and the tools necessary, let's get started\nby creating a disk to install the virtual server into. Resizing an image\nisn't an impossible task but it is much easier to choose a reasonably\nsized disk for the purpose it will be used for.\u003c/p\u003e\n\u003cp\u003eI primarily use 8 GB disks -- that way we can fit all the system\ncomponents required as well as our own web applications. Any large files\nshould be placed in a SAN or something like Dreamhost's dreamobjects.\u003c/p\u003e\n\u003cp\u003eThe other big decision you must make upfront is what disk format you\nwant to use -- the trade-off is disk space vs performance. The two\nprimary formats are qcow2 (QEMU Copy on Write) and Raw. qcow2 is great\nif you have limited disk space and don't want to allocate the full 8 GB\nup front. Raw is preferred if you want the best performance.\u003c/p\u003e\n\u003cp\u003eIf you choose qcow2, you'll also need to make sure you have \u003ccode\u003eqemu-img\u003c/code\u003e:\u003c/p\u003e\n\u003cp\u003eFedora:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eyum install qemu-img\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eUbuntu:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eapt-get install qemu-utils\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eCreate a raw disk:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003efallocate -l 8192M server.img\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eCreate a qcow2 disk:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eqemu-img create -f qcow2 server.qcow2 8G\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eInstalling your distribution onto the disk\u003c/h1\u003e\n\u003cp\u003eWe will use the \u003ccode\u003evirt-install\u003c/code\u003e command to get the distribution installed\nonto the disk image.\u003c/p\u003e\n\u003cp\u003eTo install Fedora on a qcow2 disk image:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003evirt-install --name base_server --ram 1024 --cdrom=./Fedora-20-x86_64-netinst.iso \\\n--disk path=./server.qcow2,format=qcow2\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eTo install Ubuntu Server on a raw disk image:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003evirt-install --name base_server --ram 1024 --cdrom=./ubuntu-12.04.4-server-amd64.iso \\\n--disk path=./server.img,format=raw\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou should follow the standard install steps that you normally would\nwhen setting up your distribution. But here are some tips for each:\u003c/p\u003e\n\u003cp\u003eFedora:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eChoose minimal install -- by default it selects \"GNOME\".\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eUbuntu:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eBe sure to select OpenSSH server -- it won't install it by\ndefault.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eOn Ubuntu 12.04, there is a bug that makes it hang after running\n\u003ccode\u003efsck\u003c/code\u003e. You will need to edit grub to get it to boot, hit _\u003ca href=\"\"\u003ee\u003c/a\u003e at\nthe boot prompt and add \"nomodeset\" on the linux line. You will\nknow that you need to do this if your boot hangs on fsck:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003efsck from util-linux 2.20.1\n/dev/mapper/ubuntu--vg-root: clean, 57106/441504 files, 286779/1764352 blocks\n/dev/sda1: clean, 230/62248 files, 39833/248832 blocks\n\u003c/code\u003e\u003c/pre\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch1\u003ePreparing image for openstack\u003c/h1\u003e\n\u003cp\u003eTo prepare a virtual machine for the cloud, you will need to install the\n\u003ccode\u003ecloud-init\u003c/code\u003e package, which allows the cloud providers to inject certain\nsystem settings when creating servers based on the image. These are\nthings like hostname and ssh keys.\u003c/p\u003e\n\u003cp\u003eOn Fedora:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eyum install cloud-init\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOn Ubuntu:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eapt-get install cloud-init\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen you need to just configure \u003ccode\u003ecloud-init\u003c/code\u003e by editing\n\u003ccode\u003e/etc/cloud/cloud.cfg\u003c/code\u003e and update the \u003ccode\u003edatasources_list\u003c/code\u003e section to\ninclude EC2. OpenStack uses EC2 metadata for \u003ccode\u003ecloud-init\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003eYou should also verify the user setting in this same config and define\nthe user you plan to use, it will be where the \u003ccode\u003eauthorized_keys\u003c/code\u003e file is\nsetup for when the cloud provider injects your SSH key into the server.\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003ecloud-init\u003c/code\u003e will not create the user for you, it will just assign the\nSSH keypair and reset the password. So make sure the user defined in\n\u003ccode\u003ecloud.cfg\u003c/code\u003e is also created on the system.\u003c/p\u003e\n\u003cp\u003eOnce you have your \u003ccode\u003ecloud-init\u003c/code\u003e settings the way you want them, just\nshutdown and run the \u003ccode\u003evirt-sysprep\u003c/code\u003e command.\u003c/p\u003e\n\u003cp\u003eOn the guest machine:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eshutdown -h now\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOn the host machine:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003evirt-sysprep -d base_server\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eUploading your image to OpenStack\u003c/h1\u003e\n\u003cp\u003eUsing the glance API it is very straightforward to upload the image to\nOpenStack. Just run the following command:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eglance image-create --name base_server --disk-format=qcow2 \\\n--container-format=bare --is-public=True --file server.qcow2 --progress\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce the image upload completes you will be able to use it immediately\nwithin nova. You can reference it by name or by the id from [glance\nimage-list]{.title-ref}.\u003c/p\u003e\n\u003cp\u003eTo create your first instance from the image:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003enova boot --flavor m1.tiny --image base_server --key-name devops \\\n--security-groups free_for_all test_server\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eObviously the security groups, key name, and flavors are based on your\ninstallation of OpenStack but can all easily be queried from the nova\nAPI:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003enova flavor-list\nnova secgroup-list\nnova keypair-list\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAnd you are done! You'll be able to re-use your new image as a base for\nall new instances you spin up in openstack!\u003c/p\u003e","category":"DevOps","date":"2014-08-03T00:00:00Z","tags":["Linux"],"title":"Preparing custom images for OpenStack"}]},"Pandas":{"count":1,"posts":[{"id":["2022","learning_spanish"],"path":"2022/learning_spanish","contentHtml":"\u003cp\u003eI've been living in Puerto Rico for 4 years but two of those have been COVID and so I haven't been able to practice Spanish as much as I'd like. So to speed up my learning I've decided I want to watch a lot of spanish speaking television to start training my ears, but to do this I need a baseline of words I understand to be able to even know what they are saying!\u003c/p\u003e\n\u003cp\u003eLearning through apps like Duolingo, Drops, etc start with weird topics like vegetables that don't get you to a very good baseline for actually understanding daily conversations, so I think consuming TV is a better use of my time.\u003c/p\u003e\n\u003ch2\u003eSubtitles\u003c/h2\u003e\n\u003cp\u003eI've decided the way to understand what the best words to study are is to download every subtitle for every episode of a show I want to watch and then count each word. The more a word is spoken the more important it is for me to know it since I'll be hearing it a lot in the show.\u003c/p\u003e\n\u003cp\u003eI'm going to download subtitles from Netflix. Subtitles in Netflix are in WebVTT format, which looks like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003e\u003cspan class=\"hljs-number\"\u003e248\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e17\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e58.285\u003c/span\u003e --\u003e \u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e18\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01.163\u003c/span\u003e position:\u003cspan class=\"hljs-number\"\u003e50.00\u003c/span\u003e%,middle align:middle size:\u003cspan class=\"hljs-number\"\u003e80.00\u003c/span\u003e% line:\u003cspan class=\"hljs-number\"\u003e79.33\u003c/span\u003e% \nYo de verdad espero que ustedes\nme vean como una amiga, ¿mmm?\n\n\u003cspan class=\"hljs-number\"\u003e249\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e18\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01.247\u003c/span\u003e --\u003e \u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e18\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e02.539\u003c/span\u003e position:\u003cspan class=\"hljs-number\"\u003e50.00\u003c/span\u003e%,middle align:middle size:\u003cspan class=\"hljs-number\"\u003e80.00\u003c/span\u003e% line:\u003cspan class=\"hljs-number\"\u003e84.67\u003c/span\u003e% \nNo como una madrastra.\n\n\u003cspan class=\"hljs-number\"\u003e250\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e18\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e04.250\u003c/span\u003e --\u003e \u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e18\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e06.127\u003c/span\u003e position:\u003cspan class=\"hljs-number\"\u003e50.00\u003c/span\u003e%,middle align:middle size:\u003cspan class=\"hljs-number\"\u003e80.00\u003c/span\u003e% line:\u003cspan class=\"hljs-number\"\u003e84.67\u003c/span\u003e% \nYo nunca te vi como una madrastra.\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIt gives you a start time, end time, and the text on the screen. So my first process was parsing this format and just turning it into a list of words using https://github.com/glut23/webvtt-py.\u003c/p\u003e\n\u003ch3\u003eDummy parsing\u003c/h3\u003e\n\u003cp\u003eWhat I basically did was \u003ccode\u003etext.split(\" \")\u003c/code\u003e and started counting the words. This approach was quick and painless but it had a few downs falls. Some words \u003cem\u003elook\u003c/em\u003e the same when in reality they are not and so this meant I'd have to study every meaning of a word even if it was more rare.\u003c/p\u003e\n\u003cp\u003eAn example of this is the word \"como\", you can say:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eHaz como te digo: \"Do as I say\", where como means \"as\"\u003c/li\u003e\n\u003cli\u003ecomo tacos todos los dias: \"I eat tacos every day\", where como is a conjugated form of the verb \"to eat\"\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI need to know which version of a word is being used so I can count it properly.\u003c/p\u003e\n\u003ch3\u003eRegular Expressions are always the answer\u003c/h3\u003e\n\u003cp\u003eI couldn't figure out what the word was without it being in a complete sentence, but subtitles are fragments. They are split up into timings for displaying on the screen but they don't include entire sentences. For example, it might look like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003e\u003cspan class=\"hljs-number\"\u003e23\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e21.960\u003c/span\u003e --\u003e \u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e23.520\u003c/span\u003e position:\u003cspan class=\"hljs-number\"\u003e50.00\u003c/span\u003e%,middle align:middle size:\u003cspan class=\"hljs-number\"\u003e80.00\u003c/span\u003e% line:\u003cspan class=\"hljs-number\"\u003e84.67\u003c/span\u003e% \nSolo las que luchan por ellos\n\n\u003cspan class=\"hljs-number\"\u003e24\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e23.680\u003c/span\u003e --\u003e \u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e25.680\u003c/span\u003e position:\u003cspan class=\"hljs-number\"\u003e50.00\u003c/span\u003e%,middle align:middle size:\u003cspan class=\"hljs-number\"\u003e80.00\u003c/span\u003e% line:\u003cspan class=\"hljs-number\"\u003e84.67\u003c/span\u003e% \nconsiguen sus sueños.\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eI want to detect the start of a sentence and the end of a sentence and then combine it, so that you end up with \"Solo las que luchan por ellos consiguen sus sueños.\". My first thought was a regular expression on punctuation. This worked well \u003cem\u003emost\u003c/em\u003e of the time but there were enough exceptions to the rule that it broke often on generated a lot of broken sentences:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAbbreviations like \"EE. UU\" for estados unidos (united states)\u003c/li\u003e\n\u003cli\u003eEllipsis\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSplitting on spaces also didn't work for identifying the parts of speech since I needed the context around the word.\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/regex-extraction.png\"\u003e\n\u003c/center\u003e\n\u003ch2\u003eNatural Language Processing\u003c/h2\u003e\n\u003cp\u003eSo to solve my pain I decided to grab https://spacy.io/ and do some NLP on the subtitles so that I could identify the proper parts of speech and get an accurate representation of the words I needed to learn.\u003c/p\u003e\n\u003cp\u003eThe way spaCy works is you can send it a sentence and it'll return you a set of tokens:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e spacy\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003enlp = spacy.load(\u003cspan class=\"hljs-string\"\u003e\"es_core_news_sm\"\u003c/span\u003e)\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e[x.pos_ \u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e x \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e nlp(\u003cspan class=\"hljs-string\"\u003e\"Hola, como estas?\"\u003c/span\u003e)]\n[\u003cspan class=\"hljs-string\"\u003e'PROPN'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'PUNCT'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'SCONJ'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'PRON'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'PUNCT'\u003c/span\u003e]\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo now I could identify the parts of speech and pull sentences together through end of sentence punctation. The first thing I did was generate a CSV of sentences that looked like this:\u003c/p\u003e\n\u003ctable\u003e\n\u003ctbody\u003e\u003ctr\u003e\n\u003cth\u003esentence\u003c/th\u003e\n\u003cth\u003estart\u003c/th\u003e\n\u003cth\u003eend\u003c/th\u003e\n\u003cth\u003eshow\u003c/th\u003e\n\u003cth\u003efile\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eSi no, le voy a cortar todos los deditos\u003c/td\u003e\n\u003ctd\u003e00:00:20.605\u003c/td\u003e\n\u003ctd\u003e00:00:24.125\u003c/td\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003eEl marginal S02E02 WEBRip Netflix es[cc].vtt\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\u003c/table\u003e\n\u003cp\u003eOnce I had a CSV of sentences I could send those back through spaCy for NLP and then start counting words, to generate another CSV:\u003c/p\u003e\n\u003ctable\u003e\n\u003ctbody\u003e\u003ctr\u003e\n\u003cth\u003eword\u003c/th\u003e\n\u003cth\u003epos\u003c/th\u003e\n\u003cth\u003eshow\u003c/th\u003e\n\u003cth\u003efile\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003ea\u003c/td\u003e\n\u003ctd\u003eADP\u003c/td\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003eEl marginal S02E02 WEBRip Netflix es[cc].vtt\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003ecortar\u003c/td\u003e\n\u003ctd\u003eVERB\u003c/td\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003eEl marginal S02E02 WEBRip Netflix es[cc].vtt\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003etodos\u003c/td\u003e\n\u003ctd\u003ePRON\u003c/td\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003eEl marginal S02E02 WEBRip Netflix es[cc].vtt\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\u003c/table\u003e\n\u003cp\u003eFrom there I had all the data I needed! So now it was time to start doing some data analysis!\u003c/p\u003e\n\u003ch2\u003eData analysis\u003c/h2\u003e\n\u003cp\u003eUsing a jupyter notebook ( https://jupyter.org/ ) I grabbed pandas ( https://pandas.pydata.org/ ) and read in my CSVs to start analyzing the results.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-javascript\"\u003e\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e numpy \u003cspan class=\"hljs-keyword\"\u003eas\u003c/span\u003e np\n\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e pandas \u003cspan class=\"hljs-keyword\"\u003eas\u003c/span\u003e pd\n\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e matplotlib.\u003cspan class=\"hljs-property\"\u003epyplot\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eas\u003c/span\u003e plt\npd.\u003cspan class=\"hljs-title function_\"\u003eset_option\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'display.max_rows'\u003c/span\u003e, \u003cspan class=\"hljs-number\"\u003e1000\u003c/span\u003e)\nwords = pd.\u003cspan class=\"hljs-title function_\"\u003eread_csv\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'word_data.csv.gz'\u003c/span\u003e, compression=\u003cspan class=\"hljs-string\"\u003e'gzip'\u003c/span\u003e, delimiter=\u003cspan class=\"hljs-string\"\u003e','\u003c/span\u003e)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe words dataframe is built up out of the second table I showed above with just words and their parts of speech. I started off grouping the dataset by the word so I could get a count for how many times it was spoken in every series I parsed:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003egrouped_result\u003c/span\u003e = (words.groupby(words.word).size() \n .sort_values(\u003cspan class=\"hljs-attr\"\u003eascending\u003c/span\u003e=\u003cspan class=\"hljs-literal\"\u003eFalse\u003c/span\u003e) \n .reset_index(\u003cspan class=\"hljs-attr\"\u003ename\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e)\n .drop_duplicates(\u003cspan class=\"hljs-attr\"\u003esubset\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e))\n\ngrouped_result.head(300)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWhich returned a list of words and their count:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003e\t\u003cspan class=\"hljs-type\"\u003eword\u003c/span\u003e\tcount\n\u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\tque\t\u003cspan class=\"hljs-number\"\u003e94430\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\tno\t\u003cspan class=\"hljs-number\"\u003e75931\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e\ta\t\u003cspan class=\"hljs-number\"\u003e70968\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e3\u003c/span\u003e\tde\t\u003cspan class=\"hljs-number\"\u003e67982\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e4\u003c/span\u003e\tser\t\u003cspan class=\"hljs-number\"\u003e64226\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e5\u003c/span\u003e\tla\t\u003cspan class=\"hljs-number\"\u003e52143\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e6\u003c/span\u003e\ty\t\u003cspan class=\"hljs-number\"\u003e44390\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e7\u003c/span\u003e\testar\t\u003cspan class=\"hljs-number\"\u003e37819\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e8\u003c/span\u003e\tel\t\u003cspan class=\"hljs-number\"\u003e35920\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow I wanted to identify where my diminishing returns would be. Is there a set of words that I must learn because they are spoken so often that I wouldn't understand a conversation if they weren't in my vocabulary?\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/diminishing_returns.png\"\u003e\n\u003c/center\u003e\n\u003cp\u003eAs you can see in this chart, the usage for words drops off at around the ~200 mark. So there are basically 150 words I \u003cem\u003emust\u003c/em\u003e know and then the rest are equally important. I wasn't quite happy with this because some parts of speech are higher priority than others, for example I think having a strong understanding of the popular verbs will go a long way. So I also wanted to identify what are the most important verbs to learn:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003egrouped_verbs\u003c/span\u003e = (words[words.pos == \u003cspan class=\"hljs-string\"\u003e'VERB'\u003c/span\u003e].groupby([\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'pos'\u003c/span\u003e]).size() \n .sort_values(\u003cspan class=\"hljs-attr\"\u003eascending\u003c/span\u003e=\u003cspan class=\"hljs-literal\"\u003eFalse\u003c/span\u003e) \n .reset_index(\u003cspan class=\"hljs-attr\"\u003ename\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e)\n .drop_duplicates(\u003cspan class=\"hljs-attr\"\u003esubset\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e))\n\ngrouped_verbs.head(50)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWhich got me this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\t\u003cspan class=\"hljs-string\"\u003eword\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003epos\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ecount\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003etener\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e22072\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ehacer\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e14946\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eir\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e12570\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e3\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003edecir\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e11314\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e4\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003equerer\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e11083\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e5\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ever\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e10269\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e6\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eestar\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e9780\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e7\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003esaber\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e8704\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e8\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eser\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e7674\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e9\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003edar\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e5722\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e10\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003epasar\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e5528\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e11\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ehablar\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e5355\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e12\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003evenir\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e5145\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e13\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ecreer\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e4895\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e14\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003esalir\u003c/span\u003e \t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e3395\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eVerbs had a slightly different drop-off pattern when I targeted them directly:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/diminishing_verbs.png\"\u003e\n\u003c/center\u003e\n\u003cp\u003eI get a big bang for my buck by learning those top 40 verbs. Nouns on the other hand are much more spread out and most are evenly distributed:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-string\"\u003eword\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003epos\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ecount\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003egracias\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e4676\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003efavor\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e4625\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eseñor\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e4116\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e3\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003everdad\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e3566\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e4\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003evida\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2673\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e5\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ehombre\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2601\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e6\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003emadre\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2597\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e7\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003evez\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2537\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e8\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003etiempo\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2492\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e9\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ehijo\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2215\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/diminishing_nouns.png\"\u003e\n\u003c/center\u003e\n\u003cp\u003eSo then I thought to myself... How much of a show would I understand if I just learned these most important words? So I started by excluding some of the easy parts of speech and focused on the most important:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003efind_important_words\u003c/span\u003e = (words[~words.pos.isin([\u003cspan class=\"hljs-string\"\u003e'PRON'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'CONJ'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'ADP'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'ADV'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'SCONJ'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'AUX'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'INTJ'\u003c/span\u003e])].groupby([\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'pos'\u003c/span\u003e]).size() \n .sort_values(\u003cspan class=\"hljs-attr\"\u003eascending\u003c/span\u003e=\u003cspan class=\"hljs-literal\"\u003eFalse\u003c/span\u003e) \n .reset_index(\u003cspan class=\"hljs-attr\"\u003ename\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e)\n .drop_duplicates(\u003cspan class=\"hljs-attr\"\u003esubset\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e))\n\nfind_important_words.head(50)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe top 20 were all verbs except for \u003ccode\u003ebueno\u003c/code\u003e and \u003ccode\u003egracias\u003c/code\u003e. So now with my list of what I considered \"important words\" I plotted it to find what amount of words I wanted to learn:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/important_words.png\"\u003e\n\u003c/center\u003e\n\u003cp\u003eIt looks like 200 learned words would give me a reasonable amount of understanding for a series, so I decided to calculate how much of a series I would understand if I learned just those first 200 words:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003epercentages\u003c/span\u003e = {}\n\nfor show_name in words\u003cspan class=\"hljs-section\"\u003e['media']\u003c/span\u003e.drop_duplicates().values:\n \u003cspan class=\"hljs-attr\"\u003ewords_in_show\u003c/span\u003e = (words[words.media == show_name].groupby(words.word).size() \n .sort_values(\u003cspan class=\"hljs-attr\"\u003eascending\u003c/span\u003e=\u003cspan class=\"hljs-literal\"\u003eFalse\u003c/span\u003e) \n .reset_index(\u003cspan class=\"hljs-attr\"\u003ename\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e)\n .drop_duplicates(\u003cspan class=\"hljs-attr\"\u003esubset\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e))\n \n \u003cspan class=\"hljs-attr\"\u003etotal_words_handled\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\n for word in grouped_result\u003cspan class=\"hljs-section\"\u003e['word']\u003c/span\u003e\u003cspan class=\"hljs-section\"\u003e[:200]\u003c/span\u003e:\n \u003cspan class=\"hljs-attr\"\u003evalues\u003c/span\u003e = words_in_show[words_in_show.word == word][\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e].values\n\n if values.size \u003e 0:\n total_words_handled += values\u003cspan class=\"hljs-section\"\u003e[0]\u003c/span\u003e\n\n percentages\u003cspan class=\"hljs-section\"\u003e[show_name]\u003c/span\u003e = total_words_handled / words_in_show.sum().loc\u003cspan class=\"hljs-section\"\u003e['count']\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow I had a table that would show me what percentage of the spoken words were covered by the first 200 words in my list:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003ep_df\u003c/span\u003e = pd.DataFrame(percentages.items(), columns=[\u003cspan class=\"hljs-string\"\u003e'show'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'percentage'\u003c/span\u003e])\n\u003cspan class=\"hljs-attr\"\u003ep_df\u003c/span\u003e = p_df.sort_values(by=\u003cspan class=\"hljs-string\"\u003e'percentage'\u003c/span\u003e)\np_df\u003cspan class=\"hljs-section\"\u003e['percentage']\u003c/span\u003e = p_df\u003cspan class=\"hljs-section\"\u003e['percentage']\u003c/span\u003e * 100\n\u003cspan class=\"hljs-attr\"\u003epd.options.display.float_format\u003c/span\u003e = \u003cspan class=\"hljs-string\"\u003e'{:,.2f}%'\u003c/span\u003e.format\np_df\n\u003c/code\u003e\u003c/pre\u003e\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\u003ctable\u003e\n\u003ctbody\u003e\u003ctr\u003e\n\u003cth\u003eShow\u003c/th\u003e\n\u003cth\u003ePercentage\u003c/th\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eVerónica\u003c/td\u003e\n\u003ctd\u003e64.24%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl ciudadano ilustre\u003c/td\u003e\n\u003ctd\u003e65.28%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl Chapo\u003c/td\u003e\n\u003ctd\u003e66.68%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eNeruda\u003c/td\u003e\n\u003ctd\u003e66.89%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa casa de papel\u003c/td\u003e\n\u003ctd\u003e67.56%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl Ministerio del Tiempo\u003c/td\u003e\n\u003ctd\u003e68.03%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eClub de Cuervos\u003c/td\u003e\n\u003ctd\u003e68.19%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003e68.47%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eIngobernable\u003c/td\u003e\n\u003ctd\u003e68.59%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003ePablo Escobar\u003c/td\u003e\n\u003ctd\u003e70.20%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eFariña\u003c/td\u003e\n\u003ctd\u003e70.95\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa Reina del Sur\u003c/td\u003e\n\u003ctd\u003e71.52%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eGran Hotel\u003c/td\u003e\n\u003ctd\u003e73.15%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLas chicas del cable\u003c/td\u003e\n\u003ctd\u003e73.58%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eÉlite\u003c/td\u003e\n\u003ctd\u003e73.78%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa Piloto\u003c/td\u003e\n\u003ctd\u003e74.03%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl bar\u003c/td\u003e\n\u003ctd\u003e74.07%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa casa de las flores\u003c/td\u003e\n\u003ctd\u003e75.40%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eTarde para la ira\u003c/td\u003e\n\u003ctd\u003e75.59%\u003c/td\u003e\n\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\n\u003cp\u003eBut living in Puerto Rico, one thing I've realized is speed of speech is also important. I have a much easier time speaking with people from Colombia and Mexico than I do with Puerto Ricans because they speak so much faster. So even though I could understand 75% of \"Tarde para la ira\" if I learned the 200 words, I want to make sure they are speaking at a pace I could understand as well.\u003c/p\u003e\n\u003cp\u003eSo I loaded up the other CSV file that was the full sentences and added a \"time per word\" column:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-css\"\u003esentences = pd\u003cspan class=\"hljs-selector-class\"\u003e.read_csv\u003c/span\u003e('sentences\u003cspan class=\"hljs-selector-class\"\u003e.csv\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.gz\u003c/span\u003e', compression='gzip', delimiter=',', parse_dates=\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'start'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'end'\u003c/span\u003e]\u003c/span\u003e)\nsentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'total_time'\u003c/span\u003e]\u003c/span\u003e = (sentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'end'\u003c/span\u003e]\u003c/span\u003e - sentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'start'\u003c/span\u003e]\u003c/span\u003e)\u003cspan class=\"hljs-selector-class\"\u003e.dt\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.total_seconds\u003c/span\u003e()\nsentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'word_count'\u003c/span\u003e]\u003c/span\u003e = sentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'sentence'\u003c/span\u003e]\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.str\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.split\u003c/span\u003e()\u003cspan class=\"hljs-selector-class\"\u003e.str\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.len\u003c/span\u003e()\nsentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'time_per_word'\u003c/span\u003e]\u003c/span\u003e = sentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'total_time'\u003c/span\u003e]\u003c/span\u003e / sentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'word_count'\u003c/span\u003e]\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen I was able to have a speed rating for each show:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-scss\"\u003esentence_group = sentences\u003cspan class=\"hljs-selector-class\"\u003e.groupby\u003c/span\u003e([sentences.media])\nsentence_group\u003cspan class=\"hljs-selector-class\"\u003e.time_per_word\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.mean\u003c/span\u003e()\u003cspan class=\"hljs-selector-class\"\u003e.reset_index\u003c/span\u003e()\u003cspan class=\"hljs-selector-class\"\u003e.sort_values\u003c/span\u003e('time_per_word')\n\u003c/code\u003e\u003c/pre\u003e\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\u003ctable\u003e\n\u003ctbody\u003e\u003ctr\u003e\n\u003cth\u003emedia\u003c/th\u003e\n\u003cth\u003etime_per_word\u003c/th\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eGran Hotel\u003c/td\u003e\n\u003ctd\u003e0.58\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl Chapo\u003c/td\u003e\n\u003ctd\u003e0.59\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLas chicas del cable\u003c/td\u003e\n\u003ctd\u003e0.61\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eÉlite\u003c/td\u003e\n\u003ctd\u003e0.63\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eIngobernable\u003c/td\u003e\n\u003ctd\u003e0.64\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl Ministerio del Tiempo\u003c/td\u003e\n\u003ctd\u003e0.64\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eFariña\u003c/td\u003e\n\u003ctd\u003e0.65\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl ciudadano ilustre\u003c/td\u003e\n\u003ctd\u003e0.67\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eNeruda\u003c/td\u003e\n\u003ctd\u003e0.68\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa Piloto\u003c/td\u003e\n\u003ctd\u003e0.69\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa casa de papel\u003c/td\u003e\n\u003ctd\u003e0.70\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl bar\u003c/td\u003e\n\u003ctd\u003e0.70\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eVerónica\u003c/td\u003e\n\u003ctd\u003e0.72\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa Reina del Sur\u003c/td\u003e\n\u003ctd\u003e0.75\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eClub de Cuervos\u003c/td\u003e\n\u003ctd\u003e0.76\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003e0.76\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003ePablo Escobar\u003c/td\u003e\n\u003ctd\u003e0.77\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eTarde para la ira\u003c/td\u003e\n\u003ctd\u003e0.77\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa casa de las flores\u003c/td\u003e\n\u003ctd\u003e0.81\u003c/td\u003e\n\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\n\u003cp\u003eLuckily the two series that have the least amount of vocabulary also speak the slowest! So these will be the series I start with. The final question I wanted to answer is \"What are the top words I'm missing for a series\". Since I'll know 75% of the series from the top 200 words, I'm hoping there are some top words from a specific series that I can also learn to get an even higher understanding.\u003c/p\u003e\n\u003cp\u003eFirst, find which words are in each show but not in the top 200:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003emissing_words_by_show\u003c/span\u003e = {}\n\nfor show_name in words\u003cspan class=\"hljs-section\"\u003e['media']\u003c/span\u003e.drop_duplicates().values:\n \u003cspan class=\"hljs-attr\"\u003ewords_in_show\u003c/span\u003e = (words[words.media == show_name].groupby(words.word).size() \n .sort_values(\u003cspan class=\"hljs-attr\"\u003eascending\u003c/span\u003e=\u003cspan class=\"hljs-literal\"\u003eFalse\u003c/span\u003e) \n .reset_index(\u003cspan class=\"hljs-attr\"\u003ename\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e)\n .drop_duplicates(\u003cspan class=\"hljs-attr\"\u003esubset\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e))\n \n \u003cspan class=\"hljs-attr\"\u003efrequency_words\u003c/span\u003e = grouped_result[\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e][:\u003cspan class=\"hljs-number\"\u003e200\u003c/span\u003e]\n\n \u003cspan class=\"hljs-attr\"\u003emissing_words\u003c/span\u003e = words_in_show[~words_in_show.word.isin(frequency_words.values)]\n missing_words_by_show\u003cspan class=\"hljs-section\"\u003e[show_name]\u003c/span\u003e = missing_words\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen we were able to grab them per show:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-css\"\u003emissing_words_by_show\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'La casa de las flores'\u003c/span\u003e]\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.head\u003c/span\u003e(\u003cspan class=\"hljs-number\"\u003e50\u003c/span\u003e)\n\nword\tcount\n\u003cspan class=\"hljs-number\"\u003e31\u003c/span\u003e\tmamá\t\u003cspan class=\"hljs-number\"\u003e252\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e70\u003c/span\u003e\tflorerí\u003cspan class=\"hljs-selector-tag\"\u003ea\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e87\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e98\u003c/span\u003e\tperdón\t\u003cspan class=\"hljs-number\"\u003e56\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e102\u003c/span\u003e\tsea\t\u003cspan class=\"hljs-number\"\u003e54\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e116\u003c/span\u003e\tademás\t\u003cspan class=\"hljs-number\"\u003e44\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e126\u003c/span\u003e\tahorita\t\u003cspan class=\"hljs-number\"\u003e40\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e132\u003c/span\u003e\tcárcel\t\u003cspan class=\"hljs-number\"\u003e38\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e133\u003c/span\u003e\tfiesta\t\u003cspan class=\"hljs-number\"\u003e38\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo adding those few words to my vocabulary will also give me a better understanding of the series.\u003c/p\u003e\n\u003ch2\u003eConclusion\u003c/h2\u003e\n\u003cp\u003eI believe a data-driven approach to language learning will be an effective way to get me speaking better spanish. It was a ton of fun to play with spaCy, pandas, and jupyter as well!\u003c/p\u003e\n\u003cp\u003eI'll improve the data analysis over time as well but I do believe this is a pretty good starting point!\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/meme.png\"\u003e\n\u003c/center\u003e","category":"Development","date":"2022-04-30T00:00:00Z","tags":["Python","Pandas","NLP"],"title":"How to speak spanish like a colombian drug lord!"}]},"NLP":{"count":1,"posts":[{"id":["2022","learning_spanish"],"path":"2022/learning_spanish","contentHtml":"\u003cp\u003eI've been living in Puerto Rico for 4 years but two of those have been COVID and so I haven't been able to practice Spanish as much as I'd like. So to speed up my learning I've decided I want to watch a lot of spanish speaking television to start training my ears, but to do this I need a baseline of words I understand to be able to even know what they are saying!\u003c/p\u003e\n\u003cp\u003eLearning through apps like Duolingo, Drops, etc start with weird topics like vegetables that don't get you to a very good baseline for actually understanding daily conversations, so I think consuming TV is a better use of my time.\u003c/p\u003e\n\u003ch2\u003eSubtitles\u003c/h2\u003e\n\u003cp\u003eI've decided the way to understand what the best words to study are is to download every subtitle for every episode of a show I want to watch and then count each word. The more a word is spoken the more important it is for me to know it since I'll be hearing it a lot in the show.\u003c/p\u003e\n\u003cp\u003eI'm going to download subtitles from Netflix. Subtitles in Netflix are in WebVTT format, which looks like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003e\u003cspan class=\"hljs-number\"\u003e248\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e17\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e58.285\u003c/span\u003e --\u003e \u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e18\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01.163\u003c/span\u003e position:\u003cspan class=\"hljs-number\"\u003e50.00\u003c/span\u003e%,middle align:middle size:\u003cspan class=\"hljs-number\"\u003e80.00\u003c/span\u003e% line:\u003cspan class=\"hljs-number\"\u003e79.33\u003c/span\u003e% \nYo de verdad espero que ustedes\nme vean como una amiga, ¿mmm?\n\n\u003cspan class=\"hljs-number\"\u003e249\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e18\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01.247\u003c/span\u003e --\u003e \u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e18\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e02.539\u003c/span\u003e position:\u003cspan class=\"hljs-number\"\u003e50.00\u003c/span\u003e%,middle align:middle size:\u003cspan class=\"hljs-number\"\u003e80.00\u003c/span\u003e% line:\u003cspan class=\"hljs-number\"\u003e84.67\u003c/span\u003e% \nNo como una madrastra.\n\n\u003cspan class=\"hljs-number\"\u003e250\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e18\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e04.250\u003c/span\u003e --\u003e \u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e18\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e06.127\u003c/span\u003e position:\u003cspan class=\"hljs-number\"\u003e50.00\u003c/span\u003e%,middle align:middle size:\u003cspan class=\"hljs-number\"\u003e80.00\u003c/span\u003e% line:\u003cspan class=\"hljs-number\"\u003e84.67\u003c/span\u003e% \nYo nunca te vi como una madrastra.\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIt gives you a start time, end time, and the text on the screen. So my first process was parsing this format and just turning it into a list of words using https://github.com/glut23/webvtt-py.\u003c/p\u003e\n\u003ch3\u003eDummy parsing\u003c/h3\u003e\n\u003cp\u003eWhat I basically did was \u003ccode\u003etext.split(\" \")\u003c/code\u003e and started counting the words. This approach was quick and painless but it had a few downs falls. Some words \u003cem\u003elook\u003c/em\u003e the same when in reality they are not and so this meant I'd have to study every meaning of a word even if it was more rare.\u003c/p\u003e\n\u003cp\u003eAn example of this is the word \"como\", you can say:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eHaz como te digo: \"Do as I say\", where como means \"as\"\u003c/li\u003e\n\u003cli\u003ecomo tacos todos los dias: \"I eat tacos every day\", where como is a conjugated form of the verb \"to eat\"\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI need to know which version of a word is being used so I can count it properly.\u003c/p\u003e\n\u003ch3\u003eRegular Expressions are always the answer\u003c/h3\u003e\n\u003cp\u003eI couldn't figure out what the word was without it being in a complete sentence, but subtitles are fragments. They are split up into timings for displaying on the screen but they don't include entire sentences. For example, it might look like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003e\u003cspan class=\"hljs-number\"\u003e23\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e21.960\u003c/span\u003e --\u003e \u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e23.520\u003c/span\u003e position:\u003cspan class=\"hljs-number\"\u003e50.00\u003c/span\u003e%,middle align:middle size:\u003cspan class=\"hljs-number\"\u003e80.00\u003c/span\u003e% line:\u003cspan class=\"hljs-number\"\u003e84.67\u003c/span\u003e% \nSolo las que luchan por ellos\n\n\u003cspan class=\"hljs-number\"\u003e24\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e23.680\u003c/span\u003e --\u003e \u003cspan class=\"hljs-number\"\u003e00\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e01\u003c/span\u003e:\u003cspan class=\"hljs-number\"\u003e25.680\u003c/span\u003e position:\u003cspan class=\"hljs-number\"\u003e50.00\u003c/span\u003e%,middle align:middle size:\u003cspan class=\"hljs-number\"\u003e80.00\u003c/span\u003e% line:\u003cspan class=\"hljs-number\"\u003e84.67\u003c/span\u003e% \nconsiguen sus sueños.\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eI want to detect the start of a sentence and the end of a sentence and then combine it, so that you end up with \"Solo las que luchan por ellos consiguen sus sueños.\". My first thought was a regular expression on punctuation. This worked well \u003cem\u003emost\u003c/em\u003e of the time but there were enough exceptions to the rule that it broke often on generated a lot of broken sentences:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAbbreviations like \"EE. UU\" for estados unidos (united states)\u003c/li\u003e\n\u003cli\u003eEllipsis\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSplitting on spaces also didn't work for identifying the parts of speech since I needed the context around the word.\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/regex-extraction.png\"\u003e\n\u003c/center\u003e\n\u003ch2\u003eNatural Language Processing\u003c/h2\u003e\n\u003cp\u003eSo to solve my pain I decided to grab https://spacy.io/ and do some NLP on the subtitles so that I could identify the proper parts of speech and get an accurate representation of the words I needed to learn.\u003c/p\u003e\n\u003cp\u003eThe way spaCy works is you can send it a sentence and it'll return you a set of tokens:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e spacy\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003enlp = spacy.load(\u003cspan class=\"hljs-string\"\u003e\"es_core_news_sm\"\u003c/span\u003e)\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e[x.pos_ \u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e x \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e nlp(\u003cspan class=\"hljs-string\"\u003e\"Hola, como estas?\"\u003c/span\u003e)]\n[\u003cspan class=\"hljs-string\"\u003e'PROPN'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'PUNCT'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'SCONJ'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'PRON'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'PUNCT'\u003c/span\u003e]\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo now I could identify the parts of speech and pull sentences together through end of sentence punctation. The first thing I did was generate a CSV of sentences that looked like this:\u003c/p\u003e\n\u003ctable\u003e\n\u003ctbody\u003e\u003ctr\u003e\n\u003cth\u003esentence\u003c/th\u003e\n\u003cth\u003estart\u003c/th\u003e\n\u003cth\u003eend\u003c/th\u003e\n\u003cth\u003eshow\u003c/th\u003e\n\u003cth\u003efile\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eSi no, le voy a cortar todos los deditos\u003c/td\u003e\n\u003ctd\u003e00:00:20.605\u003c/td\u003e\n\u003ctd\u003e00:00:24.125\u003c/td\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003eEl marginal S02E02 WEBRip Netflix es[cc].vtt\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\u003c/table\u003e\n\u003cp\u003eOnce I had a CSV of sentences I could send those back through spaCy for NLP and then start counting words, to generate another CSV:\u003c/p\u003e\n\u003ctable\u003e\n\u003ctbody\u003e\u003ctr\u003e\n\u003cth\u003eword\u003c/th\u003e\n\u003cth\u003epos\u003c/th\u003e\n\u003cth\u003eshow\u003c/th\u003e\n\u003cth\u003efile\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003ea\u003c/td\u003e\n\u003ctd\u003eADP\u003c/td\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003eEl marginal S02E02 WEBRip Netflix es[cc].vtt\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003ecortar\u003c/td\u003e\n\u003ctd\u003eVERB\u003c/td\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003eEl marginal S02E02 WEBRip Netflix es[cc].vtt\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003etodos\u003c/td\u003e\n\u003ctd\u003ePRON\u003c/td\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003eEl marginal S02E02 WEBRip Netflix es[cc].vtt\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\u003c/table\u003e\n\u003cp\u003eFrom there I had all the data I needed! So now it was time to start doing some data analysis!\u003c/p\u003e\n\u003ch2\u003eData analysis\u003c/h2\u003e\n\u003cp\u003eUsing a jupyter notebook ( https://jupyter.org/ ) I grabbed pandas ( https://pandas.pydata.org/ ) and read in my CSVs to start analyzing the results.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-javascript\"\u003e\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e numpy \u003cspan class=\"hljs-keyword\"\u003eas\u003c/span\u003e np\n\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e pandas \u003cspan class=\"hljs-keyword\"\u003eas\u003c/span\u003e pd\n\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e matplotlib.\u003cspan class=\"hljs-property\"\u003epyplot\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eas\u003c/span\u003e plt\npd.\u003cspan class=\"hljs-title function_\"\u003eset_option\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'display.max_rows'\u003c/span\u003e, \u003cspan class=\"hljs-number\"\u003e1000\u003c/span\u003e)\nwords = pd.\u003cspan class=\"hljs-title function_\"\u003eread_csv\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'word_data.csv.gz'\u003c/span\u003e, compression=\u003cspan class=\"hljs-string\"\u003e'gzip'\u003c/span\u003e, delimiter=\u003cspan class=\"hljs-string\"\u003e','\u003c/span\u003e)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe words dataframe is built up out of the second table I showed above with just words and their parts of speech. I started off grouping the dataset by the word so I could get a count for how many times it was spoken in every series I parsed:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003egrouped_result\u003c/span\u003e = (words.groupby(words.word).size() \n .sort_values(\u003cspan class=\"hljs-attr\"\u003eascending\u003c/span\u003e=\u003cspan class=\"hljs-literal\"\u003eFalse\u003c/span\u003e) \n .reset_index(\u003cspan class=\"hljs-attr\"\u003ename\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e)\n .drop_duplicates(\u003cspan class=\"hljs-attr\"\u003esubset\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e))\n\ngrouped_result.head(300)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWhich returned a list of words and their count:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003e\t\u003cspan class=\"hljs-type\"\u003eword\u003c/span\u003e\tcount\n\u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\tque\t\u003cspan class=\"hljs-number\"\u003e94430\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\tno\t\u003cspan class=\"hljs-number\"\u003e75931\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e\ta\t\u003cspan class=\"hljs-number\"\u003e70968\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e3\u003c/span\u003e\tde\t\u003cspan class=\"hljs-number\"\u003e67982\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e4\u003c/span\u003e\tser\t\u003cspan class=\"hljs-number\"\u003e64226\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e5\u003c/span\u003e\tla\t\u003cspan class=\"hljs-number\"\u003e52143\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e6\u003c/span\u003e\ty\t\u003cspan class=\"hljs-number\"\u003e44390\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e7\u003c/span\u003e\testar\t\u003cspan class=\"hljs-number\"\u003e37819\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e8\u003c/span\u003e\tel\t\u003cspan class=\"hljs-number\"\u003e35920\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow I wanted to identify where my diminishing returns would be. Is there a set of words that I must learn because they are spoken so often that I wouldn't understand a conversation if they weren't in my vocabulary?\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/diminishing_returns.png\"\u003e\n\u003c/center\u003e\n\u003cp\u003eAs you can see in this chart, the usage for words drops off at around the ~200 mark. So there are basically 150 words I \u003cem\u003emust\u003c/em\u003e know and then the rest are equally important. I wasn't quite happy with this because some parts of speech are higher priority than others, for example I think having a strong understanding of the popular verbs will go a long way. So I also wanted to identify what are the most important verbs to learn:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003egrouped_verbs\u003c/span\u003e = (words[words.pos == \u003cspan class=\"hljs-string\"\u003e'VERB'\u003c/span\u003e].groupby([\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'pos'\u003c/span\u003e]).size() \n .sort_values(\u003cspan class=\"hljs-attr\"\u003eascending\u003c/span\u003e=\u003cspan class=\"hljs-literal\"\u003eFalse\u003c/span\u003e) \n .reset_index(\u003cspan class=\"hljs-attr\"\u003ename\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e)\n .drop_duplicates(\u003cspan class=\"hljs-attr\"\u003esubset\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e))\n\ngrouped_verbs.head(50)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWhich got me this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\t\u003cspan class=\"hljs-string\"\u003eword\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003epos\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ecount\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003etener\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e22072\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ehacer\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e14946\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eir\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e12570\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e3\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003edecir\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e11314\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e4\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003equerer\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e11083\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e5\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ever\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e10269\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e6\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eestar\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e9780\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e7\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003esaber\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e8704\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e8\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eser\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e7674\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e9\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003edar\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e5722\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e10\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003epasar\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e5528\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e11\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ehablar\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e5355\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e12\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003evenir\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e5145\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e13\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ecreer\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e4895\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e14\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003esalir\u003c/span\u003e \t\u003cspan class=\"hljs-string\"\u003eVERB\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e3395\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eVerbs had a slightly different drop-off pattern when I targeted them directly:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/diminishing_verbs.png\"\u003e\n\u003c/center\u003e\n\u003cp\u003eI get a big bang for my buck by learning those top 40 verbs. Nouns on the other hand are much more spread out and most are evenly distributed:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-string\"\u003eword\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003epos\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ecount\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003egracias\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e4676\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003efavor\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e4625\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eseñor\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e4116\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e3\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003everdad\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e3566\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e4\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003evida\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2673\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e5\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ehombre\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2601\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e6\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003emadre\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2597\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e7\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003evez\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2537\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e8\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003etiempo\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2492\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e9\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003ehijo\u003c/span\u003e\t\u003cspan class=\"hljs-string\"\u003eNOUN\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e2215\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/diminishing_nouns.png\"\u003e\n\u003c/center\u003e\n\u003cp\u003eSo then I thought to myself... How much of a show would I understand if I just learned these most important words? So I started by excluding some of the easy parts of speech and focused on the most important:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003efind_important_words\u003c/span\u003e = (words[~words.pos.isin([\u003cspan class=\"hljs-string\"\u003e'PRON'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'CONJ'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'ADP'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'ADV'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'SCONJ'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'AUX'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'INTJ'\u003c/span\u003e])].groupby([\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'pos'\u003c/span\u003e]).size() \n .sort_values(\u003cspan class=\"hljs-attr\"\u003eascending\u003c/span\u003e=\u003cspan class=\"hljs-literal\"\u003eFalse\u003c/span\u003e) \n .reset_index(\u003cspan class=\"hljs-attr\"\u003ename\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e)\n .drop_duplicates(\u003cspan class=\"hljs-attr\"\u003esubset\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e))\n\nfind_important_words.head(50)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe top 20 were all verbs except for \u003ccode\u003ebueno\u003c/code\u003e and \u003ccode\u003egracias\u003c/code\u003e. So now with my list of what I considered \"important words\" I plotted it to find what amount of words I wanted to learn:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/important_words.png\"\u003e\n\u003c/center\u003e\n\u003cp\u003eIt looks like 200 learned words would give me a reasonable amount of understanding for a series, so I decided to calculate how much of a series I would understand if I learned just those first 200 words:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003epercentages\u003c/span\u003e = {}\n\nfor show_name in words\u003cspan class=\"hljs-section\"\u003e['media']\u003c/span\u003e.drop_duplicates().values:\n \u003cspan class=\"hljs-attr\"\u003ewords_in_show\u003c/span\u003e = (words[words.media == show_name].groupby(words.word).size() \n .sort_values(\u003cspan class=\"hljs-attr\"\u003eascending\u003c/span\u003e=\u003cspan class=\"hljs-literal\"\u003eFalse\u003c/span\u003e) \n .reset_index(\u003cspan class=\"hljs-attr\"\u003ename\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e)\n .drop_duplicates(\u003cspan class=\"hljs-attr\"\u003esubset\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e))\n \n \u003cspan class=\"hljs-attr\"\u003etotal_words_handled\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\n for word in grouped_result\u003cspan class=\"hljs-section\"\u003e['word']\u003c/span\u003e\u003cspan class=\"hljs-section\"\u003e[:200]\u003c/span\u003e:\n \u003cspan class=\"hljs-attr\"\u003evalues\u003c/span\u003e = words_in_show[words_in_show.word == word][\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e].values\n\n if values.size \u003e 0:\n total_words_handled += values\u003cspan class=\"hljs-section\"\u003e[0]\u003c/span\u003e\n\n percentages\u003cspan class=\"hljs-section\"\u003e[show_name]\u003c/span\u003e = total_words_handled / words_in_show.sum().loc\u003cspan class=\"hljs-section\"\u003e['count']\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow I had a table that would show me what percentage of the spoken words were covered by the first 200 words in my list:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003ep_df\u003c/span\u003e = pd.DataFrame(percentages.items(), columns=[\u003cspan class=\"hljs-string\"\u003e'show'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'percentage'\u003c/span\u003e])\n\u003cspan class=\"hljs-attr\"\u003ep_df\u003c/span\u003e = p_df.sort_values(by=\u003cspan class=\"hljs-string\"\u003e'percentage'\u003c/span\u003e)\np_df\u003cspan class=\"hljs-section\"\u003e['percentage']\u003c/span\u003e = p_df\u003cspan class=\"hljs-section\"\u003e['percentage']\u003c/span\u003e * 100\n\u003cspan class=\"hljs-attr\"\u003epd.options.display.float_format\u003c/span\u003e = \u003cspan class=\"hljs-string\"\u003e'{:,.2f}%'\u003c/span\u003e.format\np_df\n\u003c/code\u003e\u003c/pre\u003e\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\u003ctable\u003e\n\u003ctbody\u003e\u003ctr\u003e\n\u003cth\u003eShow\u003c/th\u003e\n\u003cth\u003ePercentage\u003c/th\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eVerónica\u003c/td\u003e\n\u003ctd\u003e64.24%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl ciudadano ilustre\u003c/td\u003e\n\u003ctd\u003e65.28%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl Chapo\u003c/td\u003e\n\u003ctd\u003e66.68%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eNeruda\u003c/td\u003e\n\u003ctd\u003e66.89%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa casa de papel\u003c/td\u003e\n\u003ctd\u003e67.56%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl Ministerio del Tiempo\u003c/td\u003e\n\u003ctd\u003e68.03%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eClub de Cuervos\u003c/td\u003e\n\u003ctd\u003e68.19%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003e68.47%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eIngobernable\u003c/td\u003e\n\u003ctd\u003e68.59%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003ePablo Escobar\u003c/td\u003e\n\u003ctd\u003e70.20%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eFariña\u003c/td\u003e\n\u003ctd\u003e70.95\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa Reina del Sur\u003c/td\u003e\n\u003ctd\u003e71.52%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eGran Hotel\u003c/td\u003e\n\u003ctd\u003e73.15%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLas chicas del cable\u003c/td\u003e\n\u003ctd\u003e73.58%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eÉlite\u003c/td\u003e\n\u003ctd\u003e73.78%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa Piloto\u003c/td\u003e\n\u003ctd\u003e74.03%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl bar\u003c/td\u003e\n\u003ctd\u003e74.07%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa casa de las flores\u003c/td\u003e\n\u003ctd\u003e75.40%\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eTarde para la ira\u003c/td\u003e\n\u003ctd\u003e75.59%\u003c/td\u003e\n\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\n\u003cp\u003eBut living in Puerto Rico, one thing I've realized is speed of speech is also important. I have a much easier time speaking with people from Colombia and Mexico than I do with Puerto Ricans because they speak so much faster. So even though I could understand 75% of \"Tarde para la ira\" if I learned the 200 words, I want to make sure they are speaking at a pace I could understand as well.\u003c/p\u003e\n\u003cp\u003eSo I loaded up the other CSV file that was the full sentences and added a \"time per word\" column:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-css\"\u003esentences = pd\u003cspan class=\"hljs-selector-class\"\u003e.read_csv\u003c/span\u003e('sentences\u003cspan class=\"hljs-selector-class\"\u003e.csv\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.gz\u003c/span\u003e', compression='gzip', delimiter=',', parse_dates=\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'start'\u003c/span\u003e, \u003cspan class=\"hljs-string\"\u003e'end'\u003c/span\u003e]\u003c/span\u003e)\nsentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'total_time'\u003c/span\u003e]\u003c/span\u003e = (sentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'end'\u003c/span\u003e]\u003c/span\u003e - sentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'start'\u003c/span\u003e]\u003c/span\u003e)\u003cspan class=\"hljs-selector-class\"\u003e.dt\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.total_seconds\u003c/span\u003e()\nsentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'word_count'\u003c/span\u003e]\u003c/span\u003e = sentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'sentence'\u003c/span\u003e]\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.str\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.split\u003c/span\u003e()\u003cspan class=\"hljs-selector-class\"\u003e.str\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.len\u003c/span\u003e()\nsentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'time_per_word'\u003c/span\u003e]\u003c/span\u003e = sentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'total_time'\u003c/span\u003e]\u003c/span\u003e / sentences\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'word_count'\u003c/span\u003e]\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen I was able to have a speed rating for each show:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-scss\"\u003esentence_group = sentences\u003cspan class=\"hljs-selector-class\"\u003e.groupby\u003c/span\u003e([sentences.media])\nsentence_group\u003cspan class=\"hljs-selector-class\"\u003e.time_per_word\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.mean\u003c/span\u003e()\u003cspan class=\"hljs-selector-class\"\u003e.reset_index\u003c/span\u003e()\u003cspan class=\"hljs-selector-class\"\u003e.sort_values\u003c/span\u003e('time_per_word')\n\u003c/code\u003e\u003c/pre\u003e\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\u003ctable\u003e\n\u003ctbody\u003e\u003ctr\u003e\n\u003cth\u003emedia\u003c/th\u003e\n\u003cth\u003etime_per_word\u003c/th\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eGran Hotel\u003c/td\u003e\n\u003ctd\u003e0.58\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl Chapo\u003c/td\u003e\n\u003ctd\u003e0.59\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLas chicas del cable\u003c/td\u003e\n\u003ctd\u003e0.61\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eÉlite\u003c/td\u003e\n\u003ctd\u003e0.63\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eIngobernable\u003c/td\u003e\n\u003ctd\u003e0.64\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl Ministerio del Tiempo\u003c/td\u003e\n\u003ctd\u003e0.64\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eFariña\u003c/td\u003e\n\u003ctd\u003e0.65\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl ciudadano ilustre\u003c/td\u003e\n\u003ctd\u003e0.67\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eNeruda\u003c/td\u003e\n\u003ctd\u003e0.68\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa Piloto\u003c/td\u003e\n\u003ctd\u003e0.69\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa casa de papel\u003c/td\u003e\n\u003ctd\u003e0.70\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl bar\u003c/td\u003e\n\u003ctd\u003e0.70\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eVerónica\u003c/td\u003e\n\u003ctd\u003e0.72\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa Reina del Sur\u003c/td\u003e\n\u003ctd\u003e0.75\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eClub de Cuervos\u003c/td\u003e\n\u003ctd\u003e0.76\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eEl marginal\u003c/td\u003e\n\u003ctd\u003e0.76\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003ePablo Escobar\u003c/td\u003e\n\u003ctd\u003e0.77\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eTarde para la ira\u003c/td\u003e\n\u003ctd\u003e0.77\u003c/td\u003e\n\u003c/tr\u003e\u003ctr\u003e\n\u003ctd\u003eLa casa de las flores\u003c/td\u003e\n\u003ctd\u003e0.81\u003c/td\u003e\n\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\n\u003cp\u003eLuckily the two series that have the least amount of vocabulary also speak the slowest! So these will be the series I start with. The final question I wanted to answer is \"What are the top words I'm missing for a series\". Since I'll know 75% of the series from the top 200 words, I'm hoping there are some top words from a specific series that I can also learn to get an even higher understanding.\u003c/p\u003e\n\u003cp\u003eFirst, find which words are in each show but not in the top 200:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003emissing_words_by_show\u003c/span\u003e = {}\n\nfor show_name in words\u003cspan class=\"hljs-section\"\u003e['media']\u003c/span\u003e.drop_duplicates().values:\n \u003cspan class=\"hljs-attr\"\u003ewords_in_show\u003c/span\u003e = (words[words.media == show_name].groupby(words.word).size() \n .sort_values(\u003cspan class=\"hljs-attr\"\u003eascending\u003c/span\u003e=\u003cspan class=\"hljs-literal\"\u003eFalse\u003c/span\u003e) \n .reset_index(\u003cspan class=\"hljs-attr\"\u003ename\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'count'\u003c/span\u003e)\n .drop_duplicates(\u003cspan class=\"hljs-attr\"\u003esubset\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e))\n \n \u003cspan class=\"hljs-attr\"\u003efrequency_words\u003c/span\u003e = grouped_result[\u003cspan class=\"hljs-string\"\u003e'word'\u003c/span\u003e][:\u003cspan class=\"hljs-number\"\u003e200\u003c/span\u003e]\n\n \u003cspan class=\"hljs-attr\"\u003emissing_words\u003c/span\u003e = words_in_show[~words_in_show.word.isin(frequency_words.values)]\n missing_words_by_show\u003cspan class=\"hljs-section\"\u003e[show_name]\u003c/span\u003e = missing_words\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen we were able to grab them per show:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-css\"\u003emissing_words_by_show\u003cspan class=\"hljs-selector-attr\"\u003e[\u003cspan class=\"hljs-string\"\u003e'La casa de las flores'\u003c/span\u003e]\u003c/span\u003e\u003cspan class=\"hljs-selector-class\"\u003e.head\u003c/span\u003e(\u003cspan class=\"hljs-number\"\u003e50\u003c/span\u003e)\n\nword\tcount\n\u003cspan class=\"hljs-number\"\u003e31\u003c/span\u003e\tmamá\t\u003cspan class=\"hljs-number\"\u003e252\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e70\u003c/span\u003e\tflorerí\u003cspan class=\"hljs-selector-tag\"\u003ea\u003c/span\u003e\t\u003cspan class=\"hljs-number\"\u003e87\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e98\u003c/span\u003e\tperdón\t\u003cspan class=\"hljs-number\"\u003e56\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e102\u003c/span\u003e\tsea\t\u003cspan class=\"hljs-number\"\u003e54\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e116\u003c/span\u003e\tademás\t\u003cspan class=\"hljs-number\"\u003e44\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e126\u003c/span\u003e\tahorita\t\u003cspan class=\"hljs-number\"\u003e40\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e132\u003c/span\u003e\tcárcel\t\u003cspan class=\"hljs-number\"\u003e38\u003c/span\u003e\n\u003cspan class=\"hljs-number\"\u003e133\u003c/span\u003e\tfiesta\t\u003cspan class=\"hljs-number\"\u003e38\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo adding those few words to my vocabulary will also give me a better understanding of the series.\u003c/p\u003e\n\u003ch2\u003eConclusion\u003c/h2\u003e\n\u003cp\u003eI believe a data-driven approach to language learning will be an effective way to get me speaking better spanish. It was a ton of fun to play with spaCy, pandas, and jupyter as well!\u003c/p\u003e\n\u003cp\u003eI'll improve the data analysis over time as well but I do believe this is a pretty good starting point!\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/learning_spanish/meme.png\"\u003e\n\u003c/center\u003e","category":"Development","date":"2022-04-30T00:00:00Z","tags":["Python","Pandas","NLP"],"title":"How to speak spanish like a colombian drug lord!"}]},"Kubernetes":{"count":2,"posts":[{"id":["2022","local_kubeadm_cluster"],"path":"2022/local_kubeadm_cluster","contentHtml":"\u003cp\u003eI’m going to show you how to get a real kubernetes cluster setup locally on top of virtual\nmachines! I’ll be using multipass but feel free to use virtualbox, proxmox, or whatever your\nfavorite cloud provider is.\u003c/p\u003e\n\u003cp\u003ekubeadm a production ready kubernetes install tool and I prefer to use it over minikube, kind,\netc. because it gives you a more real world experience for \u003cem\u003emanaging\u003c/em\u003e the kubernetes cluster.\nThis isn’t important if you are a user of the cluster but if you have to run your own this is\na great way to gain some daily experience.\u003c/p\u003e\n\u003cp\u003eThe kubernetes documentation on kubeadm is great and you can find it \u003ca href=\"https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eThe differences between this blog and the kubernetes docs is that they leave a lot of decisions\nup to the reader such as:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003echoosing a container runtime\u003c/li\u003e\n\u003cli\u003eSelecting and installing a CNI (container network interface)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI’m going to be opinionated and make specific technology decisions such as using containerd and\ncilium so that you don't have to think about those decisions.\u003c/p\u003e\n\u003ch2\u003eGetting your Virtual Machines setup!\u003c/h2\u003e\n\u003cp\u003eThe minimum requirements for a control plane node in kubernetes is 2gb of RAM and 2 CPUs. Since\nwe actually want to be able to schedule workloads on the workers afterwards we are going to setup\na cluster that looks like this:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eControl Plane: 2gb RAM, 2 CPU\u003c/li\u003e\n\u003cli\u003eWorker: 4gb RAM, 2 CPU\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSince we’ll be using multipass to launch the nodes, we can do that now:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ multipass launch -c 2 -m 4G -d 10G -n controlplane 22.04\n❯ multipass launch -c 2 -m 4G -d 10G -n worker 22.04\n❯ multipass list\nName State IPv4 Image\ncontrolplane Running 192.168.64.7 Ubuntu 22.04 LTS\nworker Running 192.168.64.8 Ubuntu 22.04 LTS\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we can start working on our controlplane first, lets shell in:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ multipass shell controlplane\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eLets first add the kubernetes repo to the system so we have access to all the kubernetes tools:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb http://apt.kubernetes.io/ kubernetes-xenial main\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/kubernetes.list\n\n❯ curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg|sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/k8s.gpg\n❯ sudo apt-get update \u0026#x26;\u0026#x26; sudo apt-get upgrade -y\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow that our system is setup, we can move on to getting a container runtime.\u003c/p\u003e\n\u003ch2\u003eGetting your Container Runtime!\u003c/h2\u003e\n\u003cp\u003eBefore we start pulling in kubernetes components we need to get a container runtime setup on the\nmachine. We we are going to use containerd for this purpose. You can view the docs of for it\n\u003ca href=\"https://github.com/containerd/containerd/blob/main/docs/getting-started.md\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eWhich will download the latest binary and set it up. I’m going to walk you through how to do it\nusing the version packaged with Ubuntu which could be older than the latest release.\u003c/p\u003e\n\u003cp\u003eFirst thing we want to do is configure the networking to allow iptables to manage:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/modules-load.d/k8s.conf\noverlay\nbr_netfilter\nEOF\u003c/span\u003e\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/sysctl.d/k8s.conf\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nnet.ipv4.ip_forward = 1\nEOF\u003c/span\u003e\n\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe also need to disable some default systemd settings for \u003ccode\u003erp_filter\u003c/code\u003e because\nthey are not compatible with cilium. See the bug report\n\u003ca href=\"https://github.com/cilium/cilium/commit/cabc6581b8128681f4ed23f8d6dc463180eea61e\"\u003ehere\u003c/a\u003e\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo sed -i -e \u003cspan class=\"hljs-string\"\u003e'/net.ipv4.conf.*.rp_filter/d'\u003c/span\u003e $(grep -ril \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e /etc/sysctl.d/ /usr/lib/sysctl.d/)\n❯ sudo sysctl -a | grep \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e | awk \u003cspan class=\"hljs-string\"\u003e'{print $1\" = 0\"}'\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e -a /etc/sysctl.d/1000-cilium.conf\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen we need to refresh sysctl so those settings are applied:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo systemctl restart systemd-modules-load\n❯ sudo sysctl --system\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou should see it applying all the changes:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e* Applying /etc/sysctl.d/k8s.conf ...\n\u003cspan class=\"hljs-attr\"\u003enet.bridge.bridge-nf-call-ip6tables\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.bridge.bridge-nf-call-iptables\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.ip_forward\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIf you do not, the netfilter module may not have loaded properly:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ lsmod |grep br_netfilter\nbr_netfilter 28672 0\nbridge 176128 1 br_netfilter\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou want to make sure \u003ccode\u003erp_filter\u003c/code\u003e is \u003ccode\u003e0\u003c/code\u003e everywhere as well for cilium:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e❯ sudo sysctl -a | grep '\\.rp_filter'\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.all.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.cilium_host.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.cilium_net.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.cilium_vxlan.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.default.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.enp0s1.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.lo.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.lxc0965b7b545f7.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.lxcb05ffd84ab74.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow lets pull down the container runtime we’ll be using which is containerd.\u003c/p\u003e\n\u003cp\u003eUbuntu ships with a very old version of containerd so you need to upgrade to\nthe version shipped from the docker repos:\nYou can find which versions are available by running:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker.gpg\n❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb https://download.docker.com/linux/ubuntu \u003cspan class=\"hljs-subst\"\u003e$(lsb_release -cs)\u003c/span\u003e stable\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/docker.list\n❯ sudo apt-get update\n\u003c/code\u003e\u003c/pre\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-cache madison containerd.io\ncontainerd.io | 1.6.8-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.6.7-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.6.6-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.6.4-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.5.11-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.5.10-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe are going to use the latest version available which was 1.6.8-1\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-get install containerd.io=1.6.8-1 -y\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen we'll setup a configuration that enables containerd to use the systemd\ncgroup. We are hard coding this config instead of using \u003ccode\u003econtainerd config default\u003c/code\u003e\nbecause that currently has had a \u003ca href=\"https://github.com/containerd/containerd/issues/4574\"\u003ebug\u003c/a\u003e\nfor many years that generates an invalid config.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/containerd/config.toml\nversion = 2\n[plugins]\n [plugins.\"io.containerd.grpc.v1.cri\"]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc]\n runtime_type = \"io.containerd.runc.v2\"\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc.options]\n SystemdCgroup = true\nEOF\u003c/span\u003e\n\n❯ sudo systemctl restart containerd.service\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can verify its running with ctr:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo ctr --address /var/run/containerd/containerd.sock containers list\nCONTAINER IMAGE RUNTIME\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow that this is working we can move on to getting kubernetes installed!\u003c/p\u003e\n\u003ch2\u003eUsing kubeadm!\u003c/h2\u003e\n\u003cp\u003eNow we need to get the kubernetes tools installed onto the system. I’m going to be using 1.23\nbut to find the latest version you can run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-cache madison kubeadm|\u003cspan class=\"hljs-built_in\"\u003ehead\u003c/span\u003e -n2\n kubeadm | 1.23.5-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages\n kubeadm | 1.23.4-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen install the version you want, we install kubelet and kubeadm here to make\nsure the versions align:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-get install kubeadm=1.23.5-00 kubelet=1.23.5-00 kubectl=1.23.5-00 -y\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will pull in a few tools, including an alternative to \u003ccode\u003ectr\u003c/code\u003e that we used earlier called\n\u003ccode\u003ecrictl\u003c/code\u003e. You can check that it is available to you doing this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo crictl --runtime-endpoint=unix:///var/run/containerd/containerd.sock ps\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe can finally init our cluster:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo kubeadm init\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce that finishes running it should give you some tips setup your configuration, it should look like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e -p \u003cspan class=\"hljs-variable\"\u003e$HOME\u003c/span\u003e/.kube\n❯ sudo \u003cspan class=\"hljs-built_in\"\u003ecp\u003c/span\u003e -i /etc/kubernetes/admin.conf \u003cspan class=\"hljs-variable\"\u003e$HOME\u003c/span\u003e/.kube/config\n❯ sudo \u003cspan class=\"hljs-built_in\"\u003echown\u003c/span\u003e $(\u003cspan class=\"hljs-built_in\"\u003eid\u003c/span\u003e -u):$(\u003cspan class=\"hljs-built_in\"\u003eid\u003c/span\u003e -g) \u003cspan class=\"hljs-variable\"\u003e$HOME\u003c/span\u003e/.kube/config\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can run those on the master node for now, but later I'll show you how to move\nthe config to your host computer.\u003c/p\u003e\n\u003cp\u003eNow you should be able to check that your node is not ready yet:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get nodes\nNAME STATUS ROLES AGE VERSION\ncontrolplane NotReady control-plane,master 4m16s v1.23.5\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cem\u003eNote\u003c/em\u003e: If you recieve \"The connecto to the server was refused\" error,\nThe cluster starting up and getting all the dependencies running could take\na bit of time. So if you aren't able to communicate right away you can check\nwhich pods are up and running with \u003ccode\u003ecrictl\u003c/code\u003e. You'll need \u003ccode\u003ekube-apiserver\u003c/code\u003e up\nand running. If it isn't you can check:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo crictl --runtime-endpoint=unix:///var/run/containerd/containerd.sock ps -a\nCONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD\n8322192c4605c bd8cc6d582470 36 seconds ago Running kube-proxy 4 344c4f7fffbe8 kube-proxy-drm46\n30ce27c40adb2 81a4a8a4ac639 2 minutes ago Exited kube-controller-manager 4 3a819c3a864b2 kube-controller-manager-controlplane\n7709fd5e92898 bd8cc6d582470 2 minutes ago Exited kube-proxy 3 7cc6922c82015 kube-proxy-drm46\n10432b81d7c61 3767741e7fba7 2 minutes ago Exited kube-apiserver 4 e64ddf3679d98 kube-apiserver-controlplane\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003ewhich will show you pods that have exited. You can grab the container ID for\nkube-apiserver and read its logs:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo crictl --runtime-endpoint=unix:///var/run/containerd/containerd.sock logs 10432b81d7c61\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThere are a few ways to figure out why the node isn’t ready yet. Usually I would check the\n\u003ccode\u003ekubelet\u003c/code\u003e logs first:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo journalctl -flu kubelet\n-- Logs begin at Sun 2022-04-17 19:22:19 AST. --\nApr 17 20:53:15 controlplane kubelet[19727]: E0417 20:53:15.951350 19727 kubelet.go:2347] \u003cspan class=\"hljs-string\"\u003e\"Container runtime network not ready\"\u003c/span\u003e networkReady=\u003cspan class=\"hljs-string\"\u003e\"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized\"\u003c/span\u003e\nApr 17 20:53:20 controlplane kubelet[19727]: E0417 20:53:20.952148 19727 kubelet.go:2347] \u003cspan class=\"hljs-string\"\u003e\"Container runtime network not ready\"\u003c/span\u003e networkReady=\u003cspan class=\"hljs-string\"\u003e\"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIt is clear the problem is that we are missing the CNI. The other way you can find out what is\ngoing on is describing the node:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl describe node controlplane\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will have a lot of information but if you scroll through there looking at \u003ccode\u003eReason\u003c/code\u003e you\nmight see something useful. In this case under \u003ccode\u003eLease\u003c/code\u003e you would see:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl describe node controlplane|grep NotReady\nReady False Sun, 17 Apr 2022 20:53:37 -0400 Sun, 17 Apr 2022 20:43:07 -0400 KubeletNotReady container runtime network not ready: NetworkReady=\u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialize\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eLets get our CNI installed, we’ll be using cilium!\u003c/p\u003e\n\u003ch2\u003eSetting up your CNI!\u003c/h2\u003e\n\u003cp\u003eCilium has great documentation over \u003ca href=\"https://docs.cilium.io/en/v1.9/gettingstarted/k8s-install-kubeadm/\"\u003ehere\u003c/a\u003e,\nbut I’ll walk you through it anyways. I do recommend checking out their documentation so you\nare familiar with it. We will use \u003ccode\u003ehelm\u003c/code\u003e to pull down the version of cilium we want:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl -fsSL https://baltocdn.com/helm/signing.asc | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/helm.gpg\n\n❯ sudo apt-get install apt-transport-https --\u003cspan class=\"hljs-built_in\"\u003eyes\u003c/span\u003e\n\n❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb https://baltocdn.com/helm/stable/debian/ all main\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/helm-stable-debian.list\n\n❯ sudo apt-get update\n❯ sudo apt-get install helm\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we can install cilium! It is \u003cem\u003every\u003c/em\u003e important that you pay attention to the\ncompatibility of cilium with the version of kubernetes you are intstalling. Check\nthe compatibility list \u003ca href=\"https://docs.cilium.io/en/v1.12/concepts/kubernetes/compatibility/\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ helm repo add cilium https://helm.cilium.io/\n❯ helm repo update\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce the repo is added you can list the versions available:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ helm search repo -l|\u003cspan class=\"hljs-built_in\"\u003ehead\u003c/span\u003e -n8\nNAME \tCHART VERSION\tAPP VERSION\tDESCRIPTION\ncilium/cilium \t1.12.1 \t1.12.1 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.12.0 \t1.12.0 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.8 \t1.11.8 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.7 \t1.11.7 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.6 \t1.11.6 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.5 \t1.11.5 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.4 \t1.11.4 \teBPF-based Networking, Security, and Observability\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo we want \u003ccode\u003e1.11.4\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ helm install cilium cilium/cilium --namespace kube-system --version 1.11.4\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow our node should be ready!\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get node\nNAME STATUS ROLES AGE VERSION\ncontrolplane Ready control-plane,master 24m v1.23.5\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eTime to join our worker to the cluster!\u003c/p\u003e\n\u003ch2\u003eJoining a worker to the cluster!\u003c/h2\u003e\n\u003cp\u003eWe have to go through the same steps as the controlplane to get the point that we have a\ncontainer runtime and \u003ccode\u003ekubeadm\u003c/code\u003e. I’m not going to talk about the commands a second time but\nI’ll re-iterate them here for ease of following along.\u003c/p\u003e\n\u003cp\u003eFirst open up another shell and connect to the worker:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ multipass shell worker\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow run the following commands:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb http://apt.kubernetes.io/ kubernetes-xenial main\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/kubernetes.list\n❯ curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg|sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/k8s.gpg\n❯ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker.gpg\n❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb https://download.docker.com/linux/ubuntu \u003cspan class=\"hljs-subst\"\u003e$(lsb_release -cs)\u003c/span\u003e stable\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/docker.list\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/modules-load.d/k8s.conf\noverlay\nbr_netfilter\nEOF\u003c/span\u003e\n\n❯ sudo sed -i -e \u003cspan class=\"hljs-string\"\u003e'/net.ipv4.conf.*.rp_filter/d'\u003c/span\u003e $(grep -ril \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e /etc/sysctl.d/ /usr/lib/sysctl.d/)\n❯ sudo sysctl -a | grep \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e | awk \u003cspan class=\"hljs-string\"\u003e'{print $1\" = 0\"}'\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e -a /etc/sysctl.d/1000-cilium.conf\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/sysctl.d/k8s.conf\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nnet.ipv4.ip_forward = 1\nEOF\u003c/span\u003e\n\n❯ sudo systemctl restart systemd-modules-load\n❯ sudo sysctl --system\n\n❯ sudo apt-get update \u0026#x26;\u0026#x26; sudo apt-get upgrade -y\n❯ sudo apt-get install containerd.io=1.6.8-1 -y\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/containerd/config.toml\nversion = 2\n[plugins]\n [plugins.\"io.containerd.grpc.v1.cri\"]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc]\n runtime_type = \"io.containerd.runc.v2\"\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc.options]\n SystemdCgroup = true\nEOF\u003c/span\u003e\n\n❯ sudo systemctl restart containerd.service\n❯ sudo apt-get install kubeadm=1.23.5-00 kubelet=1.23.5-00 kubectl=1.23.5-00 -y\n\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eFrom there we should be ready to join the cluster. When we ran \u003ccode\u003ekubeadm init\u003c/code\u003e previously it\nprinted a join command out that we could use but I’m going to show you how to do it if you\nwere coming back later and no longer had that token.\u003c/p\u003e\n\u003cp\u003eBack on the \u003cem\u003econtroplane\u003c/em\u003e node run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubeadm token create --print-join-command\nkubeadm \u003cspan class=\"hljs-built_in\"\u003ejoin\u003c/span\u003e 192.168.64.7:6443 --token wxs197.cco6mjj9ricvu8ov --discovery-token-ca-cert-hash sha256:bd01c065240fa76f30a02ecb70a8cea6e329c9678994d4da1f6ccac7694b97fb\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow copy that command and run it with \u003ccode\u003esudo\u003c/code\u003e on the worker:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo kubeadm \u003cspan class=\"hljs-built_in\"\u003ejoin\u003c/span\u003e 192.168.64.7:6443 --token wxs197.cco6mjj9ricvu8ov --discovery-token-ca-cert-hash sha256:bd01c065240fa76f30a02ecb70a8cea6e329c9678994d4da1f6ccac7694b97fb\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAfter this completes it’ll take a minute or two for everything to be synced up but if you go\nback to the master node you should have 2 ready nodes now:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get nodes\nNAME STATUS ROLES AGE VERSION\ncontrolplane Ready control-plane,master 46m v1.23.5\nworker Ready \u0026#x3C;none\u003e 79s v1.23.5\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eAccessing the cluster outside of the VMs!\u003c/h2\u003e\n\u003cp\u003eNow the final part is to get the \u003ccode\u003eadmin.conf\u003c/code\u003e as a kubeconfig on your machine so you can control\nit from outside of the cluster. To do this we can use scp\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003emultipass transfer controlplane:/home/ubuntu/.kube/config local.config\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNormally kubernetes configuration is in ~/.kube/config but I like to maint a separate file for\neach cluster and then I set the \u003ccode\u003eKUBECONFIG\u003c/code\u003e env var to access it.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003eexport\u003c/span\u003e KUBECONFIG=local.config\n❯ kubectl get nodes\nNAME STATUS ROLES AGE VERSION\ncontrolplane Ready control-plane,master 56m v1.23.5\nworker Ready \u0026#x3C;none\u003e 11m v1.23.5\n\u003c/code\u003e\u003c/pre\u003e","category":"Kubernetes","date":"2022-04-17T00:00:00Z","tags":["Linux","Kubernetes","DevOps","SRE"],"title":"Running a kubernetes cluster locally with kubeadm"},{"id":["2023","local_k8s_with_kind"],"path":"2023/local_k8s_with_kind","contentHtml":"\u003cp\u003ePreviously I \u003ca href=\"/blog/2022/local_kubeadm_cluster\"\u003eshowed\u003c/a\u003e how to run kubernetes\nlocally with \u003ccode\u003ekubeadm\u003c/code\u003e and VMs but sometimes that is overkill so I wanted to\nshow how to run \u003ca href=\"https://kind.sigs.k8s.io/\"\u003ekind\u003c/a\u003e which is \"kuberetes in\ndocker\".\u003c/p\u003e\n\u003ch1\u003eCreating your first cluster\u003c/h1\u003e\n\u003cp\u003ekind is a very flexible way to run kubernetes locally and allows you to run\nsingle node or multinode clusters while having the flexibility to use all\nthe features of kubernetes success as ingress.\u003c/p\u003e\n\u003cp\u003eTo create your first cluster it is as simple as running:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kind create cluster \n\nCreating cluster \u003cspan class=\"hljs-string\"\u003e\"kind\"\u003c/span\u003e ...\n ✓ Ensuring node image (kindest/node:v1.27.3) 🖼 \n ✓ Preparing nodes 📦 \n ✓ Writing configuration 📜 \n ✓ Starting control-plane 🕹️ \n ✓ Installing CNI 🔌 \n ✓ Installing StorageClass 💾 \nSet kubectl context to \u003cspan class=\"hljs-string\"\u003e\"kind-kind\"\u003c/span\u003e\nYou can now use your cluster with:\n\nkubectl cluster-info --context kind-kind\n\nHave a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/\u003cspan class=\"hljs-comment\"\u003e#community 🙂\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou now have a functioning kubernetes cluster and you\ncan view what it created:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ k get node\nNAME STATUS ROLES AGE VERSION\nkind-control-plane Ready control-plane 4m26s v1.27.3\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can also verify that it is running inside docker:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n1c3ba74dc29b kindest/node:v1.27.3 \u003cspan class=\"hljs-string\"\u003e\"/usr/local/bin/entr…\"\u003c/span\u003e 3 minutes ago Up 3 minutes 127.0.0.1:59327-\u003e6443/tcp kind-control-plane\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eMaking the cluster useful\u003c/h1\u003e\n\u003cp\u003eThere are a few things you'll notice with the command we ran originally:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eIt grabbed the latest kubernetes version available\u003c/li\u003e\n\u003cli\u003eIt is running a single node cluster\u003c/li\u003e\n\u003cli\u003eNo ingress available\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eLuckily kind makes it really easy to customize your local cluster to be what\nyou want it to be by using a \u003ccode\u003eYAML\u003c/code\u003e configuration.\u003c/p\u003e\n\u003cp\u003eCreate the configuration:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eCluster\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ekind.x-k8s.io/v1alpha4\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enodes:\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003econtrol-plane\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ekubeadmConfigPatches:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e|\n kind: InitConfiguration\n nodeRegistration:\n kubeletExtraArgs:\n node-labels: \"ingress-ready=true\"\n\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003eextraPortMappings:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003econtainerPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e80\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ehostPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e80\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eprotocol:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eTCP\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003econtainerPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e443\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ehostPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e443\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eprotocol:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eTCP\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eworker\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eworker\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eworker\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWith this we've now generated a 4 node cluster where we have a single\ncontrol-plane and three workers. Then we defined some extra configuration on\nthe control-plane:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ekubeadmConfigPatches\u003c/strong\u003e: We want to change the default configuration the\ncluster uses so it'll tag the nodes with the \u003ccode\u003eingress-ready\u003c/code\u003e label so the\ncontroller will use them.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eextraPortMappings\u003c/strong\u003e: allow the local host to make requests to the Ingress controller over ports 80/443\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003enode-labels\u003c/strong\u003e: only allow the ingress controller to run on specific node(s) matching the label selector\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSo now we can create the new cluster with the configuration. Save that config\nas \u003ccode\u003ekind_config.yml\u003c/code\u003e and then run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kind create cluster --image kindest/node:v1.25.11 --config kind_config.yml --name kind-multinode\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis time I've added a few additional flags on the commandline. \u003ccode\u003e--image\u003c/code\u003e\nallows us to use a different version of kubernetes and \u003ccode\u003e--name\u003c/code\u003e allows us to\nmake more than one cluster. So if you didn't destroy the first cluster you'll\nsee we have two of them now:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kind get clusters\nkind\nkind-multinode\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003ebut \u003ccode\u003ekind\u003c/code\u003e will swap the to the newest cluster by default:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl config current-context\nkind-kind-multinode\n\n❯ kubectl get node\nNAME STATUS ROLES AGE VERSION\nkind-multinode-control-plane Ready control-plane 107s v1.25.11\nkind-multinode-worker Ready \u0026#x3C;none\u003e 88s v1.25.11\nkind-multinode-worker2 Ready \u0026#x3C;none\u003e 88s v1.25.11\nkind-multinode-worker3 Ready \u0026#x3C;none\u003e 88s v1.25.11\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we need to get the \u003ccode\u003eingress-nginx\u003c/code\u003e controller installed so we can start\nusing our cluster with ingress:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe manifests contains \u003ccode\u003ekind\u003c/code\u003e specific patches to forward the hostPorts to the ingress controller, set taint tolerations and schedule it to the custom labelled node.\u003c/p\u003e\n\u003cp\u003eThis will take a little bit of time to get up and running, you can monitor it\nby running:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003ekubectl \u003cspan class=\"hljs-built_in\"\u003ewait\u003c/span\u003e --namespace ingress-nginx \\\n --\u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e=condition=ready pod \\\n --selector=app.kubernetes.io/component=controller \\\n --\u003cspan class=\"hljs-built_in\"\u003etimeout\u003c/span\u003e=90s\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eor just manually check the status:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get all -n ingress-nginx\nNAME READY STATUS RESTARTS AGE\npod/ingress-nginx-admission-create-bbmlc 0/1 Completed 0 68s\npod/ingress-nginx-admission-patch-qlnr8 0/1 Completed 2 68s\npod/ingress-nginx-controller-5f748f78c8-6tc6b 0/1 ContainerCreating 0 68s\n\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nservice/ingress-nginx-controller NodePort 10.96.228.248 \u0026#x3C;none\u003e 80:31771/TCP,443:31759/TCP 68s\nservice/ingress-nginx-controller-admission ClusterIP 10.96.180.126 \u0026#x3C;none\u003e 443/TCP 68s\n\nNAME READY UP-TO-DATE AVAILABLE AGE\ndeployment.apps/ingress-nginx-controller 0/1 1 0 68s\n\nNAME DESIRED CURRENT READY AGE\nreplicaset.apps/ingress-nginx-controller-5f748f78c8 1 1 0 68s\n\nNAME COMPLETIONS DURATION AGE\njob.batch/ingress-nginx-admission-create 1/1 22s 68s\njob.batch/ingress-nginx-admission-patch 1/1 35s 68s\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce \u003ccode\u003eingress-nginx-controller\u003c/code\u003e is in \u003ccode\u003eRunning\u003c/code\u003e state you are read to go!\u003c/p\u003e\n\u003ch1\u003eDeploying your first app\u003c/h1\u003e\n\u003cp\u003eTo prove that the cluster is working correctly we will deploy\n\u003ca href=\"https://github.com/Kong/httpbin\"\u003ehttpbin\u003c/a\u003e which is a nice little API server\nso we can prove everything is working.\u003c/p\u003e\n\u003cp\u003eCreate a \u003ccode\u003ehttbin.yml\u003c/code\u003e file and paste this into it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-meta\"\u003e---\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ev1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eService\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003elabels:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eservice:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eports:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttp\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eport:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8000\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003etargetPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8080\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eselector:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n\u003cspan class=\"hljs-meta\"\u003e---\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eapps/v1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eDeployment\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ereplicas:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eselector:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematchLabels:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eversion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ev1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003etemplate:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003elabels:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eversion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ev1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003econtainers:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003eimage:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edocker.io/mccutchen/go-httpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eimagePullPolicy:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eIfNotPresent\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eports:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003econtainerPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8080\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis is creating a couple of Kubernetes resources:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003eService\u003c/code\u003e: This is exposing the port to the ingress\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eDeployment\u003c/code\u003e: This is actually launching the service\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSo we are not using the ingress yet but we can prove that we can launch the\nservice at least. So apply those manifests:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl apply -f httpbin.yml \nservice/httpbin created\ndeployment.apps/httpbin created\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou should see two pods come up. You should wait for them to get into the\n\u003ccode\u003eRunning\u003c/code\u003e status:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get pod -o wide\nNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES\nhttpbin-5c5494967-2z5wz 1/1 Running 0 48s 10.244.3.3 kind-multinode-worker3 \u0026#x3C;none\u003e \u0026#x3C;none\u003e\nhttpbin-5c5494967-9lf47 1/1 Running 0 72s 10.244.1.2 kind-multinode-worker \u0026#x3C;none\u003e \u0026#x3C;none\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe can now use port forwarding to access it. \u003ccode\u003ehttpbin\u003c/code\u003e is exposed on \u003ccode\u003e8000\u003c/code\u003e so\nlets create port \u003ccode\u003e9000\u003c/code\u003e on our host that forwards to it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl port-forward service/httpbin 9000:8000\nForwarding from 127.0.0.1:9000 -\u003e 80\nForwarding from [::1]:9000 -\u003e 80\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can access it via:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl localhost:9000/get \n{\n \u003cspan class=\"hljs-string\"\u003e\"args\"\u003c/span\u003e: {},\n \u003cspan class=\"hljs-string\"\u003e\"headers\"\u003c/span\u003e: {\n \u003cspan class=\"hljs-string\"\u003e\"Accept\"\u003c/span\u003e: [\n \u003cspan class=\"hljs-string\"\u003e\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8\"\u003c/span\u003e\n ],\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eUsing Ingress\u003c/h1\u003e\n\u003cp\u003eNow to use the ingress rather than port forwarding we create one additional\nresource, the \u003ccode\u003eIngress\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003enetworking.k8s.io/v1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eIngress\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin-ingress\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eannotations:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003enginx.ingress.kubernetes.io/rewrite-target:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e/$2\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eingressClassName:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003enginx\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003erules:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ehttp:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epaths:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e/httpbin(/|$)(.*)\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epathType:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eImplementationSpecific\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ebackend:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eservice:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eport:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003enumber:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8000\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThere are a few critical options here. The first is the annotation to rewrite\nthe path so it doesn't include \u003ccode\u003e/httpbin/\u003c/code\u003e when it sends the request to the\nservice and then the \u003ccode\u003epath\u003c/code\u003e and \u003ccode\u003epathType\u003c/code\u003e so it knows which paths to send to\nwhich service.\u003c/p\u003e\n\u003cp\u003eNow you should be able to hit your local host and get routed to your\nkubernetes service:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl localhost/httpbin/get\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSuccess! Now you have a multinode kubernetes cluster that has an ingress\ncontroller!\u003c/p\u003e\n\u003ch1\u003eNext Steps\u003c/h1\u003e\n\u003cp\u003eThe cluster can be used like a production cluster now for local\ndevelopment! You could setup Grafana, ArgoCD, etc. to run\ninside the cluster.\u003c/p\u003e","category":"Kubernetes","date":"2023-07-21T00:00:00Z","tags":["Linux","Kubernetes","DevOps","SRE"],"title":"Running a kubernetes cluster locally with kind"}]},"DevOps":{"count":4,"posts":[{"id":["2022","local_kubeadm_cluster"],"path":"2022/local_kubeadm_cluster","contentHtml":"\u003cp\u003eI’m going to show you how to get a real kubernetes cluster setup locally on top of virtual\nmachines! I’ll be using multipass but feel free to use virtualbox, proxmox, or whatever your\nfavorite cloud provider is.\u003c/p\u003e\n\u003cp\u003ekubeadm a production ready kubernetes install tool and I prefer to use it over minikube, kind,\netc. because it gives you a more real world experience for \u003cem\u003emanaging\u003c/em\u003e the kubernetes cluster.\nThis isn’t important if you are a user of the cluster but if you have to run your own this is\na great way to gain some daily experience.\u003c/p\u003e\n\u003cp\u003eThe kubernetes documentation on kubeadm is great and you can find it \u003ca href=\"https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eThe differences between this blog and the kubernetes docs is that they leave a lot of decisions\nup to the reader such as:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003echoosing a container runtime\u003c/li\u003e\n\u003cli\u003eSelecting and installing a CNI (container network interface)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI’m going to be opinionated and make specific technology decisions such as using containerd and\ncilium so that you don't have to think about those decisions.\u003c/p\u003e\n\u003ch2\u003eGetting your Virtual Machines setup!\u003c/h2\u003e\n\u003cp\u003eThe minimum requirements for a control plane node in kubernetes is 2gb of RAM and 2 CPUs. Since\nwe actually want to be able to schedule workloads on the workers afterwards we are going to setup\na cluster that looks like this:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eControl Plane: 2gb RAM, 2 CPU\u003c/li\u003e\n\u003cli\u003eWorker: 4gb RAM, 2 CPU\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSince we’ll be using multipass to launch the nodes, we can do that now:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ multipass launch -c 2 -m 4G -d 10G -n controlplane 22.04\n❯ multipass launch -c 2 -m 4G -d 10G -n worker 22.04\n❯ multipass list\nName State IPv4 Image\ncontrolplane Running 192.168.64.7 Ubuntu 22.04 LTS\nworker Running 192.168.64.8 Ubuntu 22.04 LTS\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we can start working on our controlplane first, lets shell in:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ multipass shell controlplane\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eLets first add the kubernetes repo to the system so we have access to all the kubernetes tools:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb http://apt.kubernetes.io/ kubernetes-xenial main\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/kubernetes.list\n\n❯ curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg|sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/k8s.gpg\n❯ sudo apt-get update \u0026#x26;\u0026#x26; sudo apt-get upgrade -y\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow that our system is setup, we can move on to getting a container runtime.\u003c/p\u003e\n\u003ch2\u003eGetting your Container Runtime!\u003c/h2\u003e\n\u003cp\u003eBefore we start pulling in kubernetes components we need to get a container runtime setup on the\nmachine. We we are going to use containerd for this purpose. You can view the docs of for it\n\u003ca href=\"https://github.com/containerd/containerd/blob/main/docs/getting-started.md\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eWhich will download the latest binary and set it up. I’m going to walk you through how to do it\nusing the version packaged with Ubuntu which could be older than the latest release.\u003c/p\u003e\n\u003cp\u003eFirst thing we want to do is configure the networking to allow iptables to manage:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/modules-load.d/k8s.conf\noverlay\nbr_netfilter\nEOF\u003c/span\u003e\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/sysctl.d/k8s.conf\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nnet.ipv4.ip_forward = 1\nEOF\u003c/span\u003e\n\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe also need to disable some default systemd settings for \u003ccode\u003erp_filter\u003c/code\u003e because\nthey are not compatible with cilium. See the bug report\n\u003ca href=\"https://github.com/cilium/cilium/commit/cabc6581b8128681f4ed23f8d6dc463180eea61e\"\u003ehere\u003c/a\u003e\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo sed -i -e \u003cspan class=\"hljs-string\"\u003e'/net.ipv4.conf.*.rp_filter/d'\u003c/span\u003e $(grep -ril \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e /etc/sysctl.d/ /usr/lib/sysctl.d/)\n❯ sudo sysctl -a | grep \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e | awk \u003cspan class=\"hljs-string\"\u003e'{print $1\" = 0\"}'\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e -a /etc/sysctl.d/1000-cilium.conf\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen we need to refresh sysctl so those settings are applied:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo systemctl restart systemd-modules-load\n❯ sudo sysctl --system\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou should see it applying all the changes:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e* Applying /etc/sysctl.d/k8s.conf ...\n\u003cspan class=\"hljs-attr\"\u003enet.bridge.bridge-nf-call-ip6tables\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.bridge.bridge-nf-call-iptables\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.ip_forward\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIf you do not, the netfilter module may not have loaded properly:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ lsmod |grep br_netfilter\nbr_netfilter 28672 0\nbridge 176128 1 br_netfilter\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou want to make sure \u003ccode\u003erp_filter\u003c/code\u003e is \u003ccode\u003e0\u003c/code\u003e everywhere as well for cilium:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e❯ sudo sysctl -a | grep '\\.rp_filter'\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.all.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.cilium_host.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.cilium_net.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.cilium_vxlan.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.default.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.enp0s1.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.lo.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.lxc0965b7b545f7.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.lxcb05ffd84ab74.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow lets pull down the container runtime we’ll be using which is containerd.\u003c/p\u003e\n\u003cp\u003eUbuntu ships with a very old version of containerd so you need to upgrade to\nthe version shipped from the docker repos:\nYou can find which versions are available by running:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker.gpg\n❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb https://download.docker.com/linux/ubuntu \u003cspan class=\"hljs-subst\"\u003e$(lsb_release -cs)\u003c/span\u003e stable\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/docker.list\n❯ sudo apt-get update\n\u003c/code\u003e\u003c/pre\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-cache madison containerd.io\ncontainerd.io | 1.6.8-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.6.7-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.6.6-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.6.4-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.5.11-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.5.10-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe are going to use the latest version available which was 1.6.8-1\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-get install containerd.io=1.6.8-1 -y\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen we'll setup a configuration that enables containerd to use the systemd\ncgroup. We are hard coding this config instead of using \u003ccode\u003econtainerd config default\u003c/code\u003e\nbecause that currently has had a \u003ca href=\"https://github.com/containerd/containerd/issues/4574\"\u003ebug\u003c/a\u003e\nfor many years that generates an invalid config.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/containerd/config.toml\nversion = 2\n[plugins]\n [plugins.\"io.containerd.grpc.v1.cri\"]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc]\n runtime_type = \"io.containerd.runc.v2\"\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc.options]\n SystemdCgroup = true\nEOF\u003c/span\u003e\n\n❯ sudo systemctl restart containerd.service\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can verify its running with ctr:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo ctr --address /var/run/containerd/containerd.sock containers list\nCONTAINER IMAGE RUNTIME\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow that this is working we can move on to getting kubernetes installed!\u003c/p\u003e\n\u003ch2\u003eUsing kubeadm!\u003c/h2\u003e\n\u003cp\u003eNow we need to get the kubernetes tools installed onto the system. I’m going to be using 1.23\nbut to find the latest version you can run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-cache madison kubeadm|\u003cspan class=\"hljs-built_in\"\u003ehead\u003c/span\u003e -n2\n kubeadm | 1.23.5-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages\n kubeadm | 1.23.4-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen install the version you want, we install kubelet and kubeadm here to make\nsure the versions align:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-get install kubeadm=1.23.5-00 kubelet=1.23.5-00 kubectl=1.23.5-00 -y\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will pull in a few tools, including an alternative to \u003ccode\u003ectr\u003c/code\u003e that we used earlier called\n\u003ccode\u003ecrictl\u003c/code\u003e. You can check that it is available to you doing this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo crictl --runtime-endpoint=unix:///var/run/containerd/containerd.sock ps\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe can finally init our cluster:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo kubeadm init\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce that finishes running it should give you some tips setup your configuration, it should look like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e -p \u003cspan class=\"hljs-variable\"\u003e$HOME\u003c/span\u003e/.kube\n❯ sudo \u003cspan class=\"hljs-built_in\"\u003ecp\u003c/span\u003e -i /etc/kubernetes/admin.conf \u003cspan class=\"hljs-variable\"\u003e$HOME\u003c/span\u003e/.kube/config\n❯ sudo \u003cspan class=\"hljs-built_in\"\u003echown\u003c/span\u003e $(\u003cspan class=\"hljs-built_in\"\u003eid\u003c/span\u003e -u):$(\u003cspan class=\"hljs-built_in\"\u003eid\u003c/span\u003e -g) \u003cspan class=\"hljs-variable\"\u003e$HOME\u003c/span\u003e/.kube/config\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can run those on the master node for now, but later I'll show you how to move\nthe config to your host computer.\u003c/p\u003e\n\u003cp\u003eNow you should be able to check that your node is not ready yet:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get nodes\nNAME STATUS ROLES AGE VERSION\ncontrolplane NotReady control-plane,master 4m16s v1.23.5\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cem\u003eNote\u003c/em\u003e: If you recieve \"The connecto to the server was refused\" error,\nThe cluster starting up and getting all the dependencies running could take\na bit of time. So if you aren't able to communicate right away you can check\nwhich pods are up and running with \u003ccode\u003ecrictl\u003c/code\u003e. You'll need \u003ccode\u003ekube-apiserver\u003c/code\u003e up\nand running. If it isn't you can check:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo crictl --runtime-endpoint=unix:///var/run/containerd/containerd.sock ps -a\nCONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD\n8322192c4605c bd8cc6d582470 36 seconds ago Running kube-proxy 4 344c4f7fffbe8 kube-proxy-drm46\n30ce27c40adb2 81a4a8a4ac639 2 minutes ago Exited kube-controller-manager 4 3a819c3a864b2 kube-controller-manager-controlplane\n7709fd5e92898 bd8cc6d582470 2 minutes ago Exited kube-proxy 3 7cc6922c82015 kube-proxy-drm46\n10432b81d7c61 3767741e7fba7 2 minutes ago Exited kube-apiserver 4 e64ddf3679d98 kube-apiserver-controlplane\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003ewhich will show you pods that have exited. You can grab the container ID for\nkube-apiserver and read its logs:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo crictl --runtime-endpoint=unix:///var/run/containerd/containerd.sock logs 10432b81d7c61\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThere are a few ways to figure out why the node isn’t ready yet. Usually I would check the\n\u003ccode\u003ekubelet\u003c/code\u003e logs first:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo journalctl -flu kubelet\n-- Logs begin at Sun 2022-04-17 19:22:19 AST. --\nApr 17 20:53:15 controlplane kubelet[19727]: E0417 20:53:15.951350 19727 kubelet.go:2347] \u003cspan class=\"hljs-string\"\u003e\"Container runtime network not ready\"\u003c/span\u003e networkReady=\u003cspan class=\"hljs-string\"\u003e\"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized\"\u003c/span\u003e\nApr 17 20:53:20 controlplane kubelet[19727]: E0417 20:53:20.952148 19727 kubelet.go:2347] \u003cspan class=\"hljs-string\"\u003e\"Container runtime network not ready\"\u003c/span\u003e networkReady=\u003cspan class=\"hljs-string\"\u003e\"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIt is clear the problem is that we are missing the CNI. The other way you can find out what is\ngoing on is describing the node:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl describe node controlplane\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will have a lot of information but if you scroll through there looking at \u003ccode\u003eReason\u003c/code\u003e you\nmight see something useful. In this case under \u003ccode\u003eLease\u003c/code\u003e you would see:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl describe node controlplane|grep NotReady\nReady False Sun, 17 Apr 2022 20:53:37 -0400 Sun, 17 Apr 2022 20:43:07 -0400 KubeletNotReady container runtime network not ready: NetworkReady=\u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialize\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eLets get our CNI installed, we’ll be using cilium!\u003c/p\u003e\n\u003ch2\u003eSetting up your CNI!\u003c/h2\u003e\n\u003cp\u003eCilium has great documentation over \u003ca href=\"https://docs.cilium.io/en/v1.9/gettingstarted/k8s-install-kubeadm/\"\u003ehere\u003c/a\u003e,\nbut I’ll walk you through it anyways. I do recommend checking out their documentation so you\nare familiar with it. We will use \u003ccode\u003ehelm\u003c/code\u003e to pull down the version of cilium we want:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl -fsSL https://baltocdn.com/helm/signing.asc | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/helm.gpg\n\n❯ sudo apt-get install apt-transport-https --\u003cspan class=\"hljs-built_in\"\u003eyes\u003c/span\u003e\n\n❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb https://baltocdn.com/helm/stable/debian/ all main\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/helm-stable-debian.list\n\n❯ sudo apt-get update\n❯ sudo apt-get install helm\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we can install cilium! It is \u003cem\u003every\u003c/em\u003e important that you pay attention to the\ncompatibility of cilium with the version of kubernetes you are intstalling. Check\nthe compatibility list \u003ca href=\"https://docs.cilium.io/en/v1.12/concepts/kubernetes/compatibility/\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ helm repo add cilium https://helm.cilium.io/\n❯ helm repo update\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce the repo is added you can list the versions available:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ helm search repo -l|\u003cspan class=\"hljs-built_in\"\u003ehead\u003c/span\u003e -n8\nNAME \tCHART VERSION\tAPP VERSION\tDESCRIPTION\ncilium/cilium \t1.12.1 \t1.12.1 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.12.0 \t1.12.0 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.8 \t1.11.8 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.7 \t1.11.7 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.6 \t1.11.6 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.5 \t1.11.5 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.4 \t1.11.4 \teBPF-based Networking, Security, and Observability\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo we want \u003ccode\u003e1.11.4\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ helm install cilium cilium/cilium --namespace kube-system --version 1.11.4\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow our node should be ready!\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get node\nNAME STATUS ROLES AGE VERSION\ncontrolplane Ready control-plane,master 24m v1.23.5\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eTime to join our worker to the cluster!\u003c/p\u003e\n\u003ch2\u003eJoining a worker to the cluster!\u003c/h2\u003e\n\u003cp\u003eWe have to go through the same steps as the controlplane to get the point that we have a\ncontainer runtime and \u003ccode\u003ekubeadm\u003c/code\u003e. I’m not going to talk about the commands a second time but\nI’ll re-iterate them here for ease of following along.\u003c/p\u003e\n\u003cp\u003eFirst open up another shell and connect to the worker:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ multipass shell worker\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow run the following commands:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb http://apt.kubernetes.io/ kubernetes-xenial main\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/kubernetes.list\n❯ curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg|sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/k8s.gpg\n❯ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker.gpg\n❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb https://download.docker.com/linux/ubuntu \u003cspan class=\"hljs-subst\"\u003e$(lsb_release -cs)\u003c/span\u003e stable\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/docker.list\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/modules-load.d/k8s.conf\noverlay\nbr_netfilter\nEOF\u003c/span\u003e\n\n❯ sudo sed -i -e \u003cspan class=\"hljs-string\"\u003e'/net.ipv4.conf.*.rp_filter/d'\u003c/span\u003e $(grep -ril \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e /etc/sysctl.d/ /usr/lib/sysctl.d/)\n❯ sudo sysctl -a | grep \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e | awk \u003cspan class=\"hljs-string\"\u003e'{print $1\" = 0\"}'\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e -a /etc/sysctl.d/1000-cilium.conf\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/sysctl.d/k8s.conf\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nnet.ipv4.ip_forward = 1\nEOF\u003c/span\u003e\n\n❯ sudo systemctl restart systemd-modules-load\n❯ sudo sysctl --system\n\n❯ sudo apt-get update \u0026#x26;\u0026#x26; sudo apt-get upgrade -y\n❯ sudo apt-get install containerd.io=1.6.8-1 -y\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/containerd/config.toml\nversion = 2\n[plugins]\n [plugins.\"io.containerd.grpc.v1.cri\"]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc]\n runtime_type = \"io.containerd.runc.v2\"\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc.options]\n SystemdCgroup = true\nEOF\u003c/span\u003e\n\n❯ sudo systemctl restart containerd.service\n❯ sudo apt-get install kubeadm=1.23.5-00 kubelet=1.23.5-00 kubectl=1.23.5-00 -y\n\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eFrom there we should be ready to join the cluster. When we ran \u003ccode\u003ekubeadm init\u003c/code\u003e previously it\nprinted a join command out that we could use but I’m going to show you how to do it if you\nwere coming back later and no longer had that token.\u003c/p\u003e\n\u003cp\u003eBack on the \u003cem\u003econtroplane\u003c/em\u003e node run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubeadm token create --print-join-command\nkubeadm \u003cspan class=\"hljs-built_in\"\u003ejoin\u003c/span\u003e 192.168.64.7:6443 --token wxs197.cco6mjj9ricvu8ov --discovery-token-ca-cert-hash sha256:bd01c065240fa76f30a02ecb70a8cea6e329c9678994d4da1f6ccac7694b97fb\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow copy that command and run it with \u003ccode\u003esudo\u003c/code\u003e on the worker:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo kubeadm \u003cspan class=\"hljs-built_in\"\u003ejoin\u003c/span\u003e 192.168.64.7:6443 --token wxs197.cco6mjj9ricvu8ov --discovery-token-ca-cert-hash sha256:bd01c065240fa76f30a02ecb70a8cea6e329c9678994d4da1f6ccac7694b97fb\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAfter this completes it’ll take a minute or two for everything to be synced up but if you go\nback to the master node you should have 2 ready nodes now:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get nodes\nNAME STATUS ROLES AGE VERSION\ncontrolplane Ready control-plane,master 46m v1.23.5\nworker Ready \u0026#x3C;none\u003e 79s v1.23.5\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eAccessing the cluster outside of the VMs!\u003c/h2\u003e\n\u003cp\u003eNow the final part is to get the \u003ccode\u003eadmin.conf\u003c/code\u003e as a kubeconfig on your machine so you can control\nit from outside of the cluster. To do this we can use scp\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003emultipass transfer controlplane:/home/ubuntu/.kube/config local.config\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNormally kubernetes configuration is in ~/.kube/config but I like to maint a separate file for\neach cluster and then I set the \u003ccode\u003eKUBECONFIG\u003c/code\u003e env var to access it.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003eexport\u003c/span\u003e KUBECONFIG=local.config\n❯ kubectl get nodes\nNAME STATUS ROLES AGE VERSION\ncontrolplane Ready control-plane,master 56m v1.23.5\nworker Ready \u0026#x3C;none\u003e 11m v1.23.5\n\u003c/code\u003e\u003c/pre\u003e","category":"Kubernetes","date":"2022-04-17T00:00:00Z","tags":["Linux","Kubernetes","DevOps","SRE"],"title":"Running a kubernetes cluster locally with kubeadm"},{"id":["2023","aws_from_scratch_apply_before_merge"],"path":"2023/aws_from_scratch_apply_before_merge","contentHtml":"\u003cp\u003eFollowing this article will get you setup with an AWS Root account that can be\nmanaged through through Terraform Cloud with OIDC and github actions. As a best practice you\nshould not keep long-lived access keys in your CI/CD pipelines when\ndeploying to AWS, instead you should use OIDC (OpenID Connect) to securely\ndeploy to AWS when using Terraform Cloud or Github Actions.\u003c/p\u003e\n\u003ciframe width=\"854\" height=\"480\" src=\"https://www.youtube.com/embed/3oZd1m8_KIo\" title=\"AWS From Scratch - Preparing your account to be managed by IaC via Terraform and Github Actions\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture;web-share\" allowfullscreen\u003e\u003c/iframe\u003e\n\u003ch1\u003eTL;DR\u003c/h1\u003e\n\u003cp\u003eDownload all the terraform from the blog post here:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/sontek/aws-apply-before-merge\"\u003ehttps://github.com/sontek/aws-apply-before-merge\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/sontek/aws-terraform-github-actions\"\u003ehttps://github.com/sontek/aws-terraform-github-actions\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch1\u003eHow does OIDC work\u003c/h1\u003e\n\u003cp\u003eOIDC enables us to request a short-lived access token directly from AWS. We\njust have to create trust relationship that controls which workflows are able\nto request the access tokens.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eNo need to duplicate AWS credentials as long-lived GitHub secrets.\u003c/li\u003e\n\u003cli\u003eSince we are using a short-lived access token that is only valid for a single\njob there is no reason to worry about rotating secrets.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe following diagram gives an overview of how we can use Terraform Cloud's\nOIDC provider to integrate with AWS:\u003c/p\u003e\n\u003cdiv class=\"remark-mermaid remark-mermaid-default\"\u003e\u003csvg aria-roledescription=\"flowchart-v2\" role=\"graphics-document document\" viewBox=\"-8 -8 843.078125 320\" style=\"max-width: 843.078px; background-color: transparent;\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xmlns=\"http://www.w3.org/2000/svg\" width=\"100%\" id=\"mermaid-1691159159024\"\u003e\u003cstyle\u003e#mermaid-1691159159024{font-family:\"trebuchet ms\",verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-1691159159024 .error-icon{fill:#552222;}#mermaid-1691159159024 .error-text{fill:#552222;stroke:#552222;}#mermaid-1691159159024 .edge-thickness-normal{stroke-width:2px;}#mermaid-1691159159024 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-1691159159024 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-1691159159024 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-1691159159024 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-1691159159024 .marker{fill:#333333;stroke:#333333;}#mermaid-1691159159024 .marker.cross{stroke:#333333;}#mermaid-1691159159024 svg{font-family:\"trebuchet ms\",verdana,arial,sans-serif;font-size:16px;}#mermaid-1691159159024 .label{font-family:\"trebuchet ms\",verdana,arial,sans-serif;color:#333;}#mermaid-1691159159024 .cluster-label text{fill:#333;}#mermaid-1691159159024 .cluster-label span{color:#333;}#mermaid-1691159159024 .label text,#mermaid-1691159159024 span{fill:#333;color:#333;}#mermaid-1691159159024 .node rect,#mermaid-1691159159024 .node circle,#mermaid-1691159159024 .node ellipse,#mermaid-1691159159024 .node polygon,#mermaid-1691159159024 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-1691159159024 .node .label{text-align:center;}#mermaid-1691159159024 .node.clickable{cursor:pointer;}#mermaid-1691159159024 .arrowheadPath{fill:#333333;}#mermaid-1691159159024 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-1691159159024 .flowchart-link{stroke:#333333;fill:none;}#mermaid-1691159159024 .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-1691159159024 .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-1691159159024 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-1691159159024 .cluster text{fill:#333;}#mermaid-1691159159024 .cluster span{color:#333;}#mermaid-1691159159024 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\"trebuchet ms\",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-1691159159024 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-1691159159024 :root{--mermaid-font-family:\"trebuchet ms\",verdana,arial,sans-serif;}\u003c/style\u003e\u003cg\u003e\u003cmarker orient=\"auto\" markerHeight=\"12\" markerWidth=\"12\" markerUnits=\"userSpaceOnUse\" refY=\"5\" refX=\"10\" viewBox=\"0 0 12 20\" class=\"marker flowchart\" id=\"flowchart-pointEnd\"\u003e\u003cpath style=\"stroke-width: 1; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" d=\"M 0 0 L 10 5 L 0 10 z\"\u003e\u003c/path\u003e\u003c/marker\u003e\u003cmarker orient=\"auto\" markerHeight=\"12\" markerWidth=\"12\" markerUnits=\"userSpaceOnUse\" refY=\"5\" refX=\"0\" viewBox=\"0 0 10 10\" class=\"marker flowchart\" id=\"flowchart-pointStart\"\u003e\u003cpath style=\"stroke-width: 1; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" d=\"M 0 5 L 10 10 L 10 0 z\"\u003e\u003c/path\u003e\u003c/marker\u003e\u003cmarker orient=\"auto\" markerHeight=\"11\" markerWidth=\"11\" markerUnits=\"userSpaceOnUse\" refY=\"5\" refX=\"11\" viewBox=\"0 0 10 10\" class=\"marker flowchart\" id=\"flowchart-circleEnd\"\u003e\u003ccircle style=\"stroke-width: 1; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" r=\"5\" cy=\"5\" cx=\"5\"\u003e\u003c/circle\u003e\u003c/marker\u003e\u003cmarker orient=\"auto\" markerHeight=\"11\" markerWidth=\"11\" markerUnits=\"userSpaceOnUse\" refY=\"5\" refX=\"-1\" viewBox=\"0 0 10 10\" class=\"marker flowchart\" id=\"flowchart-circleStart\"\u003e\u003ccircle style=\"stroke-width: 1; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" r=\"5\" cy=\"5\" cx=\"5\"\u003e\u003c/circle\u003e\u003c/marker\u003e\u003cmarker orient=\"auto\" markerHeight=\"11\" markerWidth=\"11\" markerUnits=\"userSpaceOnUse\" refY=\"5.2\" refX=\"12\" viewBox=\"0 0 11 11\" class=\"marker cross flowchart\" id=\"flowchart-crossEnd\"\u003e\u003cpath style=\"stroke-width: 2; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" d=\"M 1,1 l 9,9 M 10,1 l -9,9\"\u003e\u003c/path\u003e\u003c/marker\u003e\u003cmarker orient=\"auto\" markerHeight=\"11\" markerWidth=\"11\" markerUnits=\"userSpaceOnUse\" refY=\"5.2\" refX=\"-1\" viewBox=\"0 0 11 11\" class=\"marker cross flowchart\" id=\"flowchart-crossStart\"\u003e\u003cpath style=\"stroke-width: 2; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" d=\"M 1,1 l 9,9 M 10,1 l -9,9\"\u003e\u003c/path\u003e\u003c/marker\u003e\u003cg class=\"root\"\u003e\u003cg class=\"clusters\"\u003e\u003c/g\u003e\u003cg class=\"edgePaths\"\u003e\u003cpath marker-end=\"url(#flowchart-pointEnd)\" style=\"fill:none;\" class=\"edge-thickness-normal edge-pattern-solid flowchart-link LS-AWS LE-Token\" id=\"L-AWS-Token-0\" d=\"M185.671875,134.5L181.50520833333334,134.5C177.33854166666666,134.5,169.00520833333334,134.5,152.378260969764,145.875C135.75131360619469,157.25,110.83075221238937,180,98.37047151548673,191.375L85.91019081858407,202.75\"\u003e\u003c/path\u003e\u003cpath marker-end=\"url(#flowchart-pointEnd)\" style=\"fill:none;\" class=\"edge-thickness-normal edge-pattern-solid flowchart-link LS-Token LE-Terraform\" id=\"L-Token-Terraform-0\" d=\"M85.91019081858407,235.75L98.37047151548673,247.125C110.83075221238937,258.5,135.75131360619469,281.25,164.46289638643069,292.625C193.17447916666666,304,225.67708333333334,304,258.1796875,304C290.6822916666667,304,323.1848958333333,304,359.3528645833333,304C395.5208333333333,304,435.3541666666667,304,475.1875,304C515.0208333333334,304,554.8541666666666,304,585.6956312991642,296.7916666666667C616.5370959316618,289.5833333333333,638.3866918633236,275.1666666666667,649.3114898291544,267.9583333333333L660.2362877949853,260.75\"\u003e\u003c/path\u003e\u003cpath marker-end=\"url(#flowchart-pointEnd)\" style=\"fill:none;\" class=\"edge-thickness-normal edge-pattern-solid flowchart-link LS-Terraform LE-JWT\" id=\"L-Terraform-JWT-0\" d=\"M660.2362877949853,177.75L649.3114898291544,170.54166666666666C638.3866918633236,163.33333333333334,616.5370959316618,148.91666666666666,601.4456312991642,141.70833333333334C586.3541666666666,134.5,578.0208333333334,134.5,573.8541666666666,134.5L569.6875,134.5\"\u003e\u003c/path\u003e\u003cpath marker-end=\"url(#flowchart-pointEnd)\" style=\"fill:none;\" class=\"edge-thickness-normal edge-pattern-solid flowchart-link LS-JWT LE-AWS\" id=\"L-JWT-AWS-0\" d=\"M380.6875,134.5L376.5208333333333,134.5C372.3541666666667,134.5,364.0208333333333,134.5,355.6875,134.5C347.3541666666667,134.5,339.0208333333333,134.5,334.8541666666667,134.5L330.6875,134.5\"\u003e\u003c/path\u003e\u003c/g\u003e\u003cg class=\"edgeLabels\"\u003e\u003cg class=\"edgeLabel\"\u003e\u003cg transform=\"translate(0, 0)\" class=\"label\"\u003e\u003cforeignObject height=\"0\" width=\"0\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"edgeLabel\"\u003e\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"edgeLabel\"\u003e\u003cg transform=\"translate(0, 0)\" class=\"label\"\u003e\u003cforeignObject height=\"0\" width=\"0\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"edgeLabel\"\u003e\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"edgeLabel\"\u003e\u003cg transform=\"translate(0, 0)\" class=\"label\"\u003e\u003cforeignObject height=\"0\" width=\"0\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"edgeLabel\"\u003e\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"edgeLabel\"\u003e\u003cg transform=\"translate(0, 0)\" class=\"label\"\u003e\u003cforeignObject height=\"0\" width=\"0\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"edgeLabel\"\u003e\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"nodes\"\u003e\u003cg transform=\"translate(621.1171875, 169.75)\" class=\"root\"\u003e\u003cg class=\"clusters\"\u003e\u003cg id=\"Terraform\" class=\"cluster default\"\u003e\u003crect height=\"83\" width=\"206.890625\" y=\"8\" x=\"-0.9296875\" ry=\"0\" rx=\"0\" style=\"\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-0.9296875, 8)\" class=\"cluster-label\"\u003e\u003cforeignObject height=\"18\" width=\"206.890625\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eTerraform Cloud Workflow #2\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"edgePaths\"\u003e\u003c/g\u003e\u003cg class=\"edgeLabels\"\u003e\u003c/g\u003e\u003cg class=\"nodes\"\u003e\u003cg transform=\"translate(102.515625, 49.5)\" id=\"flowchart-OIDCProvider-23\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"119.03125\" y=\"-16.5\" x=\"-59.515625\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-52.015625, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"104.03125\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eOIDC Provider\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003cg transform=\"translate(178.171875, -8)\" class=\"root\"\u003e\u003cg class=\"clusters\"\u003e\u003cg id=\"AWS\" class=\"cluster default\"\u003e\u003crect height=\"269\" width=\"145.015625\" y=\"8\" x=\"8\" ry=\"0\" rx=\"0\" style=\"\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(51.4609375, 8)\" class=\"cluster-label\"\u003e\u003cforeignObject height=\"18\" width=\"58.09375\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eAWS #1\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"edgePaths\"\u003e\u003c/g\u003e\u003cg class=\"edgeLabels\"\u003e\u003c/g\u003e\u003cg class=\"nodes\"\u003e\u003cg transform=\"translate(80.5078125, 59.5)\" id=\"flowchart-OIDC-20\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"95.015625\" y=\"-16.5\" x=\"-47.5078125\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-40.0078125, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"80.015625\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eOIDC Trust\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg transform=\"translate(80.5078125, 142.5)\" id=\"flowchart-Roles-21\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"55.90625\" y=\"-16.5\" x=\"-27.953125\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-20.453125, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"40.90625\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eRoles\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg transform=\"translate(80.5078125, 225.5)\" id=\"flowchart-Resources-22\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"91.484375\" y=\"-16.5\" x=\"-45.7421875\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-38.2421875, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"76.484375\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eResources\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003cg transform=\"translate(67.8359375, 219.25)\" id=\"flowchart-Token-25\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"135.671875\" y=\"-16.5\" x=\"-67.8359375\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-60.3359375, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"120.671875\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eAccess Token #4\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg transform=\"translate(475.1875, 134.5)\" id=\"flowchart-JWT-28\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"189\" y=\"-16.5\" x=\"-94.5\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-87, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"174\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eJWT \u0026#x26; Cloud Role ID #3\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003c/svg\u003e\u003c/div\u003e\n\u003col\u003e\n\u003cli\u003eIn AWS, create an OIDC trust between a role and our terraform cloud\nworkflow(s) that need access to the cloud.\u003c/li\u003e\n\u003cli\u003eEvery time a job runs, TFC's OIDC Provider auto-generates an OIDC token.\nThis token contains multiple claims to establish a security-hardened and\nverifiable identity about the specific workflow that is trying to authenticate.\u003c/li\u003e\n\u003cli\u003eRequest this token from TFC's OIDC provider, and present it to AWS\u003c/li\u003e\n\u003cli\u003eOnce AWS successfully validates the claims presented in the token, it then\nprovides a short-lived cloud access token that is available only for the duration\nof the job.\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch1\u003eWhat does this post accomplish\u003c/h1\u003e\n\u003cul\u003e\n\u003cli\u003eSetup a root AWS account that is managed througuh terraform\u003c/li\u003e\n\u003cli\u003eSetup OIDC authentication with Terraform Cloud so it can talk to AWS\u003c/li\u003e\n\u003cli\u003eSetup Github Actions authentication with Terraform Cloud so we can run plan\nand apply through the CI/CD pipeline.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch1\u003eSetup AWS Access\u003c/h1\u003e\n\u003cp\u003eIt is very bad practice to use the root account for much of anything but for\nbootstrapping the account it is necessary, so the first step is to get your\n\u003ccode\u003eAWS_ACCESS_KEY_ID\u003c/code\u003e and \u003ccode\u003eAWS_SECRET_ACCESS_KEY\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003eTo do this click your account and choose \u003ccode\u003eSecurity Credentials\u003c/code\u003e in the top\nright:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_root_account/security_credentials.png\" height=\"200\"\u003e\n\u003c/center\u003e\n\u003cp\u003eThen choose \u003ccode\u003eCreate Access key\u003c/code\u003e:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_root_account/create_access_token.png\" width=\"200\"\u003e\n\u003c/center\u003e\n\u003cp\u003eYou need to set these environment variables in your shell so that your local\nshell has access to AWS. After you set them you can verify you set them correct\nby running:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ aws sts get-caller-identity\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eand you should get a result similar to:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-json\"\u003e\u003cspan class=\"hljs-punctuation\"\u003e{\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"UserId\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e,\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"Account\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"888888888888\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e,\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"Arn\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"arn:aws:iam::888888888888:root\"\u003c/span\u003e\n\u003cspan class=\"hljs-punctuation\"\u003e}\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eBootstrap\u003c/h2\u003e\n\u003cp\u003eBefore you can manage any of your accounts through Terraform Cloud you'll need\nbootstrap some core infrastructure like OIDC so Terraform Cloud can authenticate\nsecurely and manage AWS Resources on your behalf.\u003c/p\u003e\n\u003cp\u003eI personally prefer doing this in two repositories:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ccode\u003einfra-bootstrap\u003c/code\u003e: This repository does the bare minimum to hook up terraform\ncloud with your AWS account and stores the state in git. Its the only infra\nthat will not be controlled by your CI/CD pipeline.ccccccug\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ccode\u003einfra\u003c/code\u003e: The actual repository where all the rest of your AWS resources are\nmanaged. It will store state in Terraform Cloud and you can introduce a\nCI/CD pipeline for approving changes.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNote\u003c/strong\u003e: This repository will be generated with the terraform code.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAfter manually creating the git repository \u003ccode\u003einfra-boostrap\u003c/code\u003e in your Github\naccount We will need 3 providers to bootstrap the account \u003ccode\u003eaws\u003c/code\u003e, \u003ccode\u003egithub\u003c/code\u003e, and\n\u003ccode\u003etfe\u003c/code\u003e.\u003c/p\u003e\n\u003ch3\u003eVariables\u003c/h3\u003e\n\u003cp\u003eCreate a \u003ccode\u003e1-variables.tf\u003c/code\u003e where we can define the variables we'll need\nfor creating these resources.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_aws_audience\"\u003c/span\u003e {\n type = string\n default = \u003cspan class=\"hljs-string\"\u003e\"aws.workload.identity\"\u003c/span\u003e\n description = \u003cspan class=\"hljs-string\"\u003e\"The audience value to use in run identity tokens\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_hostname\"\u003c/span\u003e {\n type = string\n default = \u003cspan class=\"hljs-string\"\u003e\"app.terraform.io\"\u003c/span\u003e\n description = \u003cspan class=\"hljs-string\"\u003e\"The hostname of the TFC or TFE instance you'd like to use with AWS\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_project_name\"\u003c/span\u003e {\n type = string\n default = \u003cspan class=\"hljs-string\"\u003e\"Default Project\"\u003c/span\u003e\n description = \u003cspan class=\"hljs-string\"\u003e\"The project under which a workspace will be created\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_organization_name\"\u003c/span\u003e {\n type = string\n description = \u003cspan class=\"hljs-string\"\u003e\"The name of your Terraform Cloud organization\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_organization_owner\"\u003c/span\u003e {\n type = string\n description = \u003cspan class=\"hljs-string\"\u003e\"The owner of the TFC organization\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_workspaces\"\u003c/span\u003e {\n type = list(string)\n description = \u003cspan class=\"hljs-string\"\u003e\"The list of TFC workspaces\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_organization\"\u003c/span\u003e {\n description = \u003cspan class=\"hljs-string\"\u003e\"The organization the repositories are owned by\"\u003c/span\u003e\n type = string\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_repo_name\"\u003c/span\u003e {\n description = \u003cspan class=\"hljs-string\"\u003e\"The name of the git reppository we'll create for managing infra\"\u003c/span\u003e\n type = string\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_default_branch\"\u003c/span\u003e {\n description = \u003cspan class=\"hljs-string\"\u003e\"The default branch to utilize\"\u003c/span\u003e\n type = string\n default = \u003cspan class=\"hljs-string\"\u003e\"main\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_root_account_id\"\u003c/span\u003e {\n description = \u003cspan class=\"hljs-string\"\u003e\"The AWS root account we want to apply these changes to\"\u003c/span\u003e\n type = string\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe will use these variables in the later modules but they are mostly metadata\naround the terraform and github accounts you'll need to setup manually.\u003c/p\u003e\n\u003ch3\u003eProviders\u003c/h3\u003e\n\u003cp\u003eCreate a file called \u003ccode\u003e2-providers.tf\u003c/code\u003e and define the providers:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e {\n required_providers {\n tfe = {\n source = \u003cspan class=\"hljs-string\"\u003e\"hashicorp/tfe\"\u003c/span\u003e\n version = \u003cspan class=\"hljs-string\"\u003e\"0.41.0\"\u003c/span\u003e\n }\n\n aws = {\n source = \u003cspan class=\"hljs-string\"\u003e\"hashicorp/aws\"\u003c/span\u003e\n version = \u003cspan class=\"hljs-string\"\u003e\"4.58.0\"\u003c/span\u003e\n }\n\n github = {\n source = \u003cspan class=\"hljs-string\"\u003e\"integrations/github\"\u003c/span\u003e\n version = \u003cspan class=\"hljs-string\"\u003e\"5.18.3\"\u003c/span\u003e\n }\n }\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eprovider\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws\"\u003c/span\u003e {\n region = \u003cspan class=\"hljs-string\"\u003e\"us-east-1\"\u003c/span\u003e\n\n \u003cspan class=\"hljs-comment\"\u003e# Root account, all other accounts should be provisioned\u003c/span\u003e\n \u003cspan class=\"hljs-comment\"\u003e# via pull requests\u003c/span\u003e\n allowed_account_ids = [var.aws_root_account_id]\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eprovider\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github\"\u003c/span\u003e {\n owner = var.github_organization\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe key things there are we define \u003ccode\u003eallowed_account_ids\u003c/code\u003e to prevent us from\nworking against any account that isn't the root and we are using one of the\nvariables we defines earlier.\u003c/p\u003e\n\u003ch3\u003eGithub\u003c/h3\u003e\n\u003cp\u003eWe will utilize \u003ccode\u003eterraform\u003c/code\u003e to create the second git repository where the rest\nof the infrastructure will go. Create a file called \u003ccode\u003e3-github.tf\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_repository\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"repo\"\u003c/span\u003e {\n name = var.github_repo_name\n description = \u003cspan class=\"hljs-string\"\u003e\"Infrastructure Repository\"\u003c/span\u003e\n visibility = \u003cspan class=\"hljs-string\"\u003e\"private\"\u003c/span\u003e\n auto_init = true\n has_issues = true\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_branch_default\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"default\"\u003c/span\u003e {\n repository = github_repository.repo.name\n branch = var.github_default_branch\n}\n\n\u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_team\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"owners\"\u003c/span\u003e {\n name = \u003cspan class=\"hljs-string\"\u003e\"owners\"\u003c/span\u003e\n organization = tfe_organization.organization.name\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_team_token\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_actions_token\"\u003c/span\u003e {\n team_id = \u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e.tfe_team.owners.id\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_actions_secret\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_secret\"\u003c/span\u003e {\n repository = github_repository.repo.name\n secret_name = \u003cspan class=\"hljs-string\"\u003e\"TFE_TOKEN\"\u003c/span\u003e\n plaintext_value = tfe_team_token.github_actions_token.token\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eoutput\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"repository_id\"\u003c/span\u003e {\n value = github_repository.repo.id\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will generate a new repository in your account called \u003ccode\u003einfra\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003eFor the terraform provider to have access to github you need to create a new\npersonal access token with full \u003ccode\u003erepo\u003c/code\u003e access and set it as an environment\nvariable named \u003ccode\u003eGITHUB_TOKEN\u003c/code\u003e.\u003c/p\u003e\n\u003ch3\u003eTerraform Cloud\u003c/h3\u003e\n\u003cp\u003eNow we need to setup dynamic credentials so the terraform cloud agent is\nallowed to take actions on your behalf. To do this we'll setup an IAM\nrole and an OIDC provider. Create a file called \u003ccode\u003e4-tfc.tf\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_organization\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"organization\"\u003c/span\u003e {\n name = var.tfc_organization_name\n email = var.tfc_organization_owner\n}\n\n/* AWS will use this TLS certificate to verify that requests for dynamic\ncredentials come from Terraform Cloud.*/\n\u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tls_certificate\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_certificate\"\u003c/span\u003e {\n url = \u003cspan class=\"hljs-string\"\u003e\"https://\u003cspan class=\"hljs-variable\"\u003e${var.tfc_hostname}\u003c/span\u003e\"\u003c/span\u003e\n}\n\n/* sets up an OIDC \u003cspan class=\"hljs-keyword\"\u003eprovider\u003c/span\u003e in AWS with Terraform Cloud's TLS certificate,\nthe SHA1 fingerprint from the TLS certificate\n*/\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_openid_connect_provider\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_provider\"\u003c/span\u003e {\n url = \u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e.tls_certificate.tfc_certificate.url\n client_id_list = [var.tfc_aws_audience]\n thumbprint_list = [\n \u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e.tls_certificate.tfc_certificate.certificates[\u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e].sha1_fingerprint\n ]\n}\n\n/* Policy to allow TFC to assume the AWS IAM role in our account */\n\u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_policy_document\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"assume_role\"\u003c/span\u003e {\n statement {\n effect = \u003cspan class=\"hljs-string\"\u003e\"Allow\"\u003c/span\u003e\n\n principals {\n type = \u003cspan class=\"hljs-string\"\u003e\"Federated\"\u003c/span\u003e\n identifiers = [aws_iam_openid_connect_provider.tfc_provider.arn]\n }\n condition {\n test = \u003cspan class=\"hljs-string\"\u003e\"StringEquals\"\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e = \u003cspan class=\"hljs-string\"\u003e\"\u003cspan class=\"hljs-variable\"\u003e${var.tfc_hostname}\u003c/span\u003e:aud\"\u003c/span\u003e\n\n values = [\n \u003cspan class=\"hljs-string\"\u003e\"\u003cspan class=\"hljs-variable\"\u003e${\u003cspan class=\"hljs-meta\"\u003eone(aws_iam_openid_connect_provider.tfc_provider.client_id_list)\u003c/span\u003e}\u003c/span\u003e\"\u003c/span\u003e\n ]\n }\n\n condition {\n test = \u003cspan class=\"hljs-string\"\u003e\"StringLike\"\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e = \u003cspan class=\"hljs-string\"\u003e\"\u003cspan class=\"hljs-variable\"\u003e${var.tfc_hostname}\u003c/span\u003e:sub\"\u003c/span\u003e\n\n values = [\n for workspace in var.tfc_workspaces : \u003cspan class=\"hljs-string\"\u003e\"organization:\u003cspan class=\"hljs-variable\"\u003e${tfe_organization.organization.name}\u003c/span\u003e:project:\u003cspan class=\"hljs-variable\"\u003e${var.tfc_project_name}\u003c/span\u003e:workspace:\u003cspan class=\"hljs-variable\"\u003e${workspace}\u003c/span\u003e:run_phase:*\"\u003c/span\u003e\n ]\n }\n actions = [\u003cspan class=\"hljs-string\"\u003e\"sts:AssumeRoleWithWebIdentity\"\u003c/span\u003e]\n }\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_role\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e {\n name = \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e\n assume_role_policy = \u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e.aws_iam_policy_document.assume_role.json\n}\n\n/* Policy for what the TFC agent is allowed to do */\n\u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_policy_document\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e {\n version = \u003cspan class=\"hljs-string\"\u003e\"2012-10-17\"\u003c/span\u003e\n\n statement {\n actions = [\u003cspan class=\"hljs-string\"\u003e\"*\"\u003c/span\u003e]\n effect = \u003cspan class=\"hljs-string\"\u003e\"Allow\"\u003c/span\u003e\n resources = [\u003cspan class=\"hljs-string\"\u003e\"*\"\u003c/span\u003e]\n }\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_policy\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e {\n name = \u003cspan class=\"hljs-string\"\u003e\"tfc-agent-access-policy\"\u003c/span\u003e\n description = \u003cspan class=\"hljs-string\"\u003e\"Access policy for the TFC agent\"\u003c/span\u003e\n policy = \u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e.aws_iam_policy_document.tfc-agent.json\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_role_policy_attachment\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-access-attach\"\u003c/span\u003e {\n role = aws_iam_role.tfc-agent.name\n policy_arn = aws_iam_policy.tfc-agent.arn\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_workspace\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"workspaces\"\u003c/span\u003e {\n count = length(var.tfc_workspaces)\n name = var.tfc_workspaces[count.index]\n organization = tfe_organization.organization.name\n\n working_directory = var.tfc_workspaces[count.index]\n}\n\n/* These variables tell the agent to use dynamic credentials */\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_variable\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-auth\"\u003c/span\u003e {\n count = length(var.tfc_workspaces)\n key = \u003cspan class=\"hljs-string\"\u003e\"TFC_AWS_PROVIDER_AUTH\"\u003c/span\u003e\n value = true\n category = \u003cspan class=\"hljs-string\"\u003e\"env\"\u003c/span\u003e\n workspace_id = tfe_workspace.workspaces[count.index].id\n description = \u003cspan class=\"hljs-string\"\u003e\"Enable dynamic auth on the TFC agents\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_variable\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-role\"\u003c/span\u003e {\n count = length(var.tfc_workspaces)\n key = \u003cspan class=\"hljs-string\"\u003e\"TFC_AWS_RUN_ROLE_ARN\"\u003c/span\u003e\n value = aws_iam_role.tfc-agent.arn\n category = \u003cspan class=\"hljs-string\"\u003e\"env\"\u003c/span\u003e\n workspace_id = tfe_workspace.workspaces[count.index].id\n description = \u003cspan class=\"hljs-string\"\u003e\"Tell TFC what Role to run as\"\u003c/span\u003e\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis module is dynamic because there is one piece that will require a\nmanual oauth setup for github. So the first pass will apply without it\nand then later on we'll create it and run the apply again.\u003c/p\u003e\n\u003ch2\u003eApplying the changes\u003c/h2\u003e\n\u003cp\u003eNow we just need to define our settings for the module and we'll get our\ninfrastructure applied. Create a file called \u003ccode\u003esettings.auto.tfvars\u003c/code\u003e and\npopulate it with the content for your account. This is an example of what\nthis should look like:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003etfc_organization_name = \u003cspan class=\"hljs-string\"\u003e\"sontek\"\u003c/span\u003e\ntfc_organization_owner = \u003cspan class=\"hljs-string\"\u003e\"john@sontek.net\"\u003c/span\u003e\n\n\u003cspan class=\"hljs-comment\"\u003e# The workspaces you want to create and be able to manage with IaC\u003c/span\u003e\ntfc_workspaces = [\n \u003cspan class=\"hljs-string\"\u003e\"root\"\u003c/span\u003e\n]\n\u003cspan class=\"hljs-comment\"\u003e# this can be your username\u003c/span\u003e\ngithub_organization = \u003cspan class=\"hljs-string\"\u003e\"sontek\"\u003c/span\u003e\ngithub_repo_name = \u003cspan class=\"hljs-string\"\u003e\"sontek-infra\"\u003c/span\u003e\naws_root_account_id = \u003cspan class=\"hljs-string\"\u003e\"888888888888\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ terraform login\n❯ terraform init\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eand you should see:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs\"\u003eTerraform has been successfully initialized!\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow lets run our plan:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e❯ \u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e plan\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou should see a result:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-vbnet\"\u003e\u003cspan class=\"hljs-symbol\"\u003ePlan:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e10\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eto\u003c/span\u003e add, \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eto\u003c/span\u003e change, \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eto\u003c/span\u003e destroy.\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eApply it to make those resources:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e❯ \u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e apply\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAt this point it:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eCreated a terraform cloud organization\u003c/li\u003e\n\u003cli\u003eCreated a terraform cloud workspace\u003c/li\u003e\n\u003cli\u003eCreated a git repository\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch1\u003eVerify TFC can talk to AWS\u003c/h1\u003e\n\u003cp\u003eTo verify that TFC can communicate with AWS through the dynamic credentials,\nlets clone the \u003cem\u003eNEW\u003c/em\u003e repository we just generated and make some dummy resources. After\nyou've cloned the repository lets make a folder for the workspace \u003ccode\u003eroot\u003c/code\u003e that we\ndefined in bootstrap:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e root\n❯ \u003cspan class=\"hljs-built_in\"\u003ecd\u003c/span\u003e root\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow create a \u003ccode\u003e1-providers.tf\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e {\n cloud {\n organization = \u003cspan class=\"hljs-string\"\u003e\"sontek\"\u003c/span\u003e\n\n workspaces {\n name = \u003cspan class=\"hljs-string\"\u003e\"root\"\u003c/span\u003e\n }\n }\n\n required_providers {\n aws = {\n source = \u003cspan class=\"hljs-string\"\u003e\"hashicorp/aws\"\u003c/span\u003e\n version = \u003cspan class=\"hljs-string\"\u003e\"4.58.0\"\u003c/span\u003e\n }\n\n tfe = {\n source = \u003cspan class=\"hljs-string\"\u003e\"hashicorp/tfe\"\u003c/span\u003e\n version = \u003cspan class=\"hljs-string\"\u003e\"0.42.0\"\u003c/span\u003e\n }\n }\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eprovider\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws\"\u003c/span\u003e {\n region = \u003cspan class=\"hljs-string\"\u003e\"us-east-1\"\u003c/span\u003e\n\n default_tags {\n tags = {\n Owner = \u003cspan class=\"hljs-string\"\u003e\"john@sontek.net\"\u003c/span\u003e\n Env = \u003cspan class=\"hljs-string\"\u003e\"Root\"\u003c/span\u003e\n Service = \u003cspan class=\"hljs-string\"\u003e\"BusinessOperations\"\u003c/span\u003e\n }\n }\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cstrong\u003eNOTE\u003c/strong\u003e: You should replace \u003ccode\u003eorganization\u003c/code\u003e, \u003ccode\u003eworkspaces.name\u003c/code\u003e, and\n\u003ccode\u003etags.Owner\u003c/code\u003e to be your own values.\u003c/p\u003e\n\u003cp\u003eNow create a small resource to prove everything is working, we'll use SQS for\nthis. Create a file called \u003ccode\u003e2-sqs.tf\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_sqs_queue\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"example-sqs\"\u003c/span\u003e {\n name = \u003cspan class=\"hljs-string\"\u003e\"example-sqs\"\u003c/span\u003e\n message_retention_seconds = \u003cspan class=\"hljs-number\"\u003e86400\u003c/span\u003e\n receive_wait_time_seconds = \u003cspan class=\"hljs-number\"\u003e10\u003c/span\u003e\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIf you run the plan you should see the resource it wants to create:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ terraform init\n❯ terraform plan\n\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eand you should see the run is executing in terraform cloud:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003eRunning plan in Terraform Cloud. Output will stream here. Pressing Ctrl-C\nwill stop streaming the logs, but will \u003cspan class=\"hljs-keyword\"\u003enot\u003c/span\u003e stop the plan running remotely.\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can click the link it provides to see the logs. Now lets apply this\nresource to see it all working:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e❯ \u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e apply\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou should get a response like:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-string\"\u003eApply\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ecomplete!\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003eResources:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eadded,\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003echanged,\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edestroyed.\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo Terraform Cloud has full access to create AWS resources! The final step\nis to get github running the plan/apply on pull requests. Commit these files\nto your repository and we'll remove them in a pull request. Create a\n\u003ccode\u003e.gitignore\u003c/code\u003e file in the root:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e.\u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e*\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eand commit all the files:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ git add *\n❯ git commit -m \u003cspan class=\"hljs-string\"\u003e\"initial infra\"\u003c/span\u003e\n❯ git push origin \u003cspan class=\"hljs-built_in\"\u003ehead\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eGithub Actions\u003c/h1\u003e\n\u003cp\u003eThe two most popular workflows when using terraform are:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eApply after Merge\u003c/strong\u003e: This is the default for things like\n\u003ca href=\"https://terraform.io\"\u003eterraform cloud\u003c/a\u003e and most github actions.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eApply before Merge\u003c/strong\u003e: This is the default for things like\n\u003ca href=\"https://www.runatlantis.io/\"\u003eAtlantis\u003c/a\u003e.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI don't like apply-after-merge. There are a lot of ways where a \u003ccode\u003eplan\u003c/code\u003e\ncan succeed but an \u003ccode\u003eapply\u003c/code\u003e will fail and you end up with broken configuration\nin \u003ccode\u003emain\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003eSo in this article I'll show you how to implement \u003cstrong\u003eapply-before-merge\u003c/strong\u003e with\ngithub actions.\u003c/p\u003e\n\u003cp\u003eAll of these changes will be in the \u003ccode\u003einfra\u003c/code\u003e repository that was generated from\n\u003ccode\u003ebootstrap\u003c/code\u003e. We are done with the bootstrap at this point.\u003c/p\u003e\n\u003cp\u003eFirst, lets setup the \u003ccode\u003e.github\u003c/code\u003e folder, the end result we want is:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e.github/\n└── workflows\n ├── on-apply-finished.yml\n ├── on-pull-request-labeled.yml\n └── on-pull-request.yml\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo create the folders:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e -p .github/workflows\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eOn Pull Request\u003c/h1\u003e\n\u003cp\u003eThe first flow we'll create is the \u003ccode\u003eterraform plan\u003c/code\u003e workflow which should be\nran whenever a pull request is opened. Create the file\n\u003ccode\u003e.github/workflows/on-pull-request.yml\u003c/code\u003e and put this content in it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yml\"\u003e\u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003epr_build\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003eon:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epull_request:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ebranches:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003emain\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003eenv:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eTERRAFORM_CLOUD_TOKENS:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eapp.terraform.io=${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.TFE_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eGITHUB_TOKEN:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.GITHUB_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003ejobs:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eterraform_validate:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eruns-on:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eubuntu-22.04\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003estrategy:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efail-fast:\u003c/span\u003e \u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematrix:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efolder:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eroot\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esteps:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eCheckout\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eactions/checkout@v3\u003c/span\u003e\n\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eterraform\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003evalidate\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edflook/terraform-validate@v1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkspace:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\n \u003cspan class=\"hljs-attr\"\u003eterraform_fmt:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eruns-on:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eubuntu-22.04\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003estrategy:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efail-fast:\u003c/span\u003e \u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematrix:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efolder:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eroot\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esteps:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eactions/checkout@v3\u003c/span\u003e\n\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eterraform\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003efmt\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edflook/terraform-fmt-check@v1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkspace:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\n \u003cspan class=\"hljs-attr\"\u003eterraform_plan:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eruns-on:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eubuntu-22.04\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epermissions:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003econtents:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eread\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epull-requests:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ewrite\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003estrategy:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efail-fast:\u003c/span\u003e \u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematrix:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efolder:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eroot\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esteps:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eactions/checkout@v3\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eterraform\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eplan\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edflook/terraform-plan@v1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkspace:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis creates three jobs:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eterraform_validate\u003c/strong\u003e: This validates the terraform via \u003ccode\u003eterraform validate\u003c/code\u003e\ncommand to make sure that it is correct and doesn't have duplicate resources\nor anything like that.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eterraform_fmt\u003c/strong\u003e: This verifies that the terraform is well formatted by\nrunning the \u003ccode\u003eterraform fmt\u003c/code\u003e command.`\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eterraform_plan\u003c/strong\u003e: This runs the \u003ccode\u003eterraform\u003c/code\u003e plan and comments on the PR a\ndiff of the changes for you to verify.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eTo verify this is working, lets delete \u003ccode\u003eroot/2-sqs.tf\u003c/code\u003e, then lets push a branch\nand make a pull request to see the result so far:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003erm\u003c/span\u003e root/2-sqs.tf\n❯ git add .github/ root/\n❯ git checkout -b apply-before-merge\n❯ git commit -m \u003cspan class=\"hljs-string\"\u003e\"Implemented on-pull-request\"\u003c/span\u003e\n❯ git push origin \u003cspan class=\"hljs-built_in\"\u003ehead\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAfter you make the pull request you should 3 checks on it and a comment that\nshows the plan:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/github_comment.png\" width=\"400\"\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/github_checks.png\" width=\"400\"\u003e\n\u003c/center\u003e\n\u003ch1\u003eApply on Label\u003c/h1\u003e\n\u003cp\u003eSo now that the plan is working we need some way to \u003ccode\u003eapply\u003c/code\u003e the changes. I've\nfound the best way to do this is via a label rather than a comment because of\nthe way github actions work. Their event based actions like \u003ccode\u003eon-comment\u003c/code\u003e aren't\nexecuted in the context of a pull-request.\u003c/p\u003e\n\u003cp\u003eSince we will be using a label to signal a plan is ready to be applied lets\ncreate a new file \u003ccode\u003e.github/workflows/on-pull-request-labeled.yml\u003c/code\u003e and provide\nthis content:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003epr_apply\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003eon:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epull_request:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003etypes:\u003c/span\u003e [ \u003cspan class=\"hljs-string\"\u003elabeled\u003c/span\u003e ]\n\n\u003cspan class=\"hljs-attr\"\u003eenv:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eTERRAFORM_CLOUD_TOKENS:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eapp.terraform.io=${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.TFE_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eGITHUB_TOKEN:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.GITHUB_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003ejobs:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eterraform_apply:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eif:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003egithub.event.label.name\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e==\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'tfc-apply'\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eruns-on:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eubuntu-22.04\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epermissions:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003econtents:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eread\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epull-requests:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ewrite\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003estrategy:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efail-fast:\u003c/span\u003e \u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematrix:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efolder:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eroot\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esteps:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eactions/checkout@v3\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edflook/terraform-apply@v1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkspace:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will fire whenever a pull request is labeled with the \u003ccode\u003etfc-apply\u003c/code\u003e label.\nYou will need to create this label for the repository.\u003c/p\u003e\n\u003cp\u003eIt will run the \u003ccode\u003eapply\u003c/code\u003e and update the previous plan comment to let you\nknow the status.\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/tfc_applying.png\" width=\"400\"\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/tfc_applying_comment.png\" width=\"400\"\u003e\n\u003c/center\u003e\n\u003ch1\u003eMerge on Apply\u003c/h1\u003e\n\u003cp\u003eOne thing you'll notice is that the pull request stayed open even after the\ninfrastructure is applied and we don't want that. We want any changes that have\nmade it into the environment to be merged into \u003ccode\u003emain\u003c/code\u003e automatically. To do\nthis we'll create our final action.\u003c/p\u003e\n\u003cp\u003eCreate a new file \u003ccode\u003e.github/workflows/on-apply-finished.yml\u003c/code\u003e with this content:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003epr_merge\u003c/span\u003e\n\n\u003cspan class=\"hljs-comment\"\u003e# Only trigger, when the build workflow succeeded\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eon:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkflow_run:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkflows:\u003c/span\u003e [\u003cspan class=\"hljs-string\"\u003epr_apply\u003c/span\u003e]\n \u003cspan class=\"hljs-attr\"\u003etypes:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ecompleted\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003ejobs:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003emerge:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eif:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003egithub.event.workflow_run.conclusion\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e==\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'success'\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eruns-on:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eubuntu-22.04\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epermissions:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003econtents:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ewrite\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epull-requests:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ewrite\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003echecks:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eread\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003estatuses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eread\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eactions:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eread\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eoutputs:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epullRequestNumber:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esteps.workflow-run-info.outputs.pullRequestNumber\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esteps:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"Get information about the current run\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003epotiuk/get-workflow-origin@v1_5\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eid:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eworkflow-run-info\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003etoken:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.GITHUB_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esourceRunId:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003egithub.event.workflow_run.id\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003emerge\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ea\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003epull\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003erequest\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eafter\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eterraform\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eapply\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esudo-bot/action-pull-request-merge@v1.2.0\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003egithub-token:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.GITHUB_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003enumber:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esteps.workflow-run-info.outputs.pullRequestNumber\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will wait until the \u003ccode\u003epr_apply\u003c/code\u003e job completes and as long as it was\nsuccessful it'll merge the branch!\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNOTE\u003c/strong\u003e: As I mentioned earlier, the event based actions do not run in the\ncontext of the pull request which means you cannot test changes to them during\nthe PR either. You must merge the \u003ccode\u003eon-apply-finished.yml\u003c/code\u003e file to \u003ccode\u003emain\u003c/code\u003e\nbefore it starts working.\u003c/p\u003e\n\u003ch1\u003eBranch Protection\u003c/h1\u003e\n\u003cp\u003eThe final step to the process is to make sure you go to your github settings\nand make sure these status checks are required before merging. Branch protection\nis a feature that will prevent merging changes into a branch unless all\nrequired checks are passing.\u003c/p\u003e\n\u003cp\u003eGo to \u003ccode\u003eSettings\u003c/code\u003e -\u003e \u003ccode\u003eBranches\u003c/code\u003e -\u003e \u003ccode\u003eBranch Protection\u003c/code\u003e and add a branch\nprotection rule:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/branch_protection.png\" width=\"500\"\u003e\n\u003c/center\u003e\n\u003cp\u003eYou want to enable the following settings:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eBranch Name\u003c/strong\u003e: main\u003c/li\u003e\n\u003cli\u003e✅ Require a pull request before merging\u003c/li\u003e\n\u003cli\u003e✅ Require status checks to pass before merging\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThen for \u003ccode\u003eStatus checks that are required.\u003c/code\u003e select all of the ones we've\ncreated:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/required_checks.png\" height=\"200\"\u003e\n\u003c/center\u003e\n\u003ch1\u003eHelpful Resources\u003c/h1\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://developer.hashicorp.com/terraform/tutorials/cloud/dynamic-credentials?product_intent=terraform\"\u003eTerraform Dynamic Credentials Tutorial\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://developer.hashicorp.com/terraform/cloud-docs/workspaces/dynamic-provider-credentials/aws-configuration\"\u003eTerraform docs on Dynamic Credentials\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#understanding-the-oidc-token\"\u003eGithub's understanding OIDC\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e","category":"AWS","date":"2023-06-23T00:00:00Z","tags":["AWS","DevOps","SRE"],"title":"AWS From Scratch with Terraform - Setting up your Root Account for IaC with Terraform Cloud and Github actions."},{"id":["2023","local_k8s_with_kind"],"path":"2023/local_k8s_with_kind","contentHtml":"\u003cp\u003ePreviously I \u003ca href=\"/blog/2022/local_kubeadm_cluster\"\u003eshowed\u003c/a\u003e how to run kubernetes\nlocally with \u003ccode\u003ekubeadm\u003c/code\u003e and VMs but sometimes that is overkill so I wanted to\nshow how to run \u003ca href=\"https://kind.sigs.k8s.io/\"\u003ekind\u003c/a\u003e which is \"kuberetes in\ndocker\".\u003c/p\u003e\n\u003ch1\u003eCreating your first cluster\u003c/h1\u003e\n\u003cp\u003ekind is a very flexible way to run kubernetes locally and allows you to run\nsingle node or multinode clusters while having the flexibility to use all\nthe features of kubernetes success as ingress.\u003c/p\u003e\n\u003cp\u003eTo create your first cluster it is as simple as running:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kind create cluster \n\nCreating cluster \u003cspan class=\"hljs-string\"\u003e\"kind\"\u003c/span\u003e ...\n ✓ Ensuring node image (kindest/node:v1.27.3) 🖼 \n ✓ Preparing nodes 📦 \n ✓ Writing configuration 📜 \n ✓ Starting control-plane 🕹️ \n ✓ Installing CNI 🔌 \n ✓ Installing StorageClass 💾 \nSet kubectl context to \u003cspan class=\"hljs-string\"\u003e\"kind-kind\"\u003c/span\u003e\nYou can now use your cluster with:\n\nkubectl cluster-info --context kind-kind\n\nHave a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/\u003cspan class=\"hljs-comment\"\u003e#community 🙂\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou now have a functioning kubernetes cluster and you\ncan view what it created:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ k get node\nNAME STATUS ROLES AGE VERSION\nkind-control-plane Ready control-plane 4m26s v1.27.3\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can also verify that it is running inside docker:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n1c3ba74dc29b kindest/node:v1.27.3 \u003cspan class=\"hljs-string\"\u003e\"/usr/local/bin/entr…\"\u003c/span\u003e 3 minutes ago Up 3 minutes 127.0.0.1:59327-\u003e6443/tcp kind-control-plane\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eMaking the cluster useful\u003c/h1\u003e\n\u003cp\u003eThere are a few things you'll notice with the command we ran originally:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eIt grabbed the latest kubernetes version available\u003c/li\u003e\n\u003cli\u003eIt is running a single node cluster\u003c/li\u003e\n\u003cli\u003eNo ingress available\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eLuckily kind makes it really easy to customize your local cluster to be what\nyou want it to be by using a \u003ccode\u003eYAML\u003c/code\u003e configuration.\u003c/p\u003e\n\u003cp\u003eCreate the configuration:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eCluster\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ekind.x-k8s.io/v1alpha4\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enodes:\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003econtrol-plane\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ekubeadmConfigPatches:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e|\n kind: InitConfiguration\n nodeRegistration:\n kubeletExtraArgs:\n node-labels: \"ingress-ready=true\"\n\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003eextraPortMappings:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003econtainerPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e80\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ehostPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e80\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eprotocol:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eTCP\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003econtainerPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e443\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ehostPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e443\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eprotocol:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eTCP\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eworker\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eworker\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eworker\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWith this we've now generated a 4 node cluster where we have a single\ncontrol-plane and three workers. Then we defined some extra configuration on\nthe control-plane:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ekubeadmConfigPatches\u003c/strong\u003e: We want to change the default configuration the\ncluster uses so it'll tag the nodes with the \u003ccode\u003eingress-ready\u003c/code\u003e label so the\ncontroller will use them.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eextraPortMappings\u003c/strong\u003e: allow the local host to make requests to the Ingress controller over ports 80/443\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003enode-labels\u003c/strong\u003e: only allow the ingress controller to run on specific node(s) matching the label selector\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSo now we can create the new cluster with the configuration. Save that config\nas \u003ccode\u003ekind_config.yml\u003c/code\u003e and then run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kind create cluster --image kindest/node:v1.25.11 --config kind_config.yml --name kind-multinode\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis time I've added a few additional flags on the commandline. \u003ccode\u003e--image\u003c/code\u003e\nallows us to use a different version of kubernetes and \u003ccode\u003e--name\u003c/code\u003e allows us to\nmake more than one cluster. So if you didn't destroy the first cluster you'll\nsee we have two of them now:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kind get clusters\nkind\nkind-multinode\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003ebut \u003ccode\u003ekind\u003c/code\u003e will swap the to the newest cluster by default:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl config current-context\nkind-kind-multinode\n\n❯ kubectl get node\nNAME STATUS ROLES AGE VERSION\nkind-multinode-control-plane Ready control-plane 107s v1.25.11\nkind-multinode-worker Ready \u0026#x3C;none\u003e 88s v1.25.11\nkind-multinode-worker2 Ready \u0026#x3C;none\u003e 88s v1.25.11\nkind-multinode-worker3 Ready \u0026#x3C;none\u003e 88s v1.25.11\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we need to get the \u003ccode\u003eingress-nginx\u003c/code\u003e controller installed so we can start\nusing our cluster with ingress:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe manifests contains \u003ccode\u003ekind\u003c/code\u003e specific patches to forward the hostPorts to the ingress controller, set taint tolerations and schedule it to the custom labelled node.\u003c/p\u003e\n\u003cp\u003eThis will take a little bit of time to get up and running, you can monitor it\nby running:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003ekubectl \u003cspan class=\"hljs-built_in\"\u003ewait\u003c/span\u003e --namespace ingress-nginx \\\n --\u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e=condition=ready pod \\\n --selector=app.kubernetes.io/component=controller \\\n --\u003cspan class=\"hljs-built_in\"\u003etimeout\u003c/span\u003e=90s\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eor just manually check the status:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get all -n ingress-nginx\nNAME READY STATUS RESTARTS AGE\npod/ingress-nginx-admission-create-bbmlc 0/1 Completed 0 68s\npod/ingress-nginx-admission-patch-qlnr8 0/1 Completed 2 68s\npod/ingress-nginx-controller-5f748f78c8-6tc6b 0/1 ContainerCreating 0 68s\n\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nservice/ingress-nginx-controller NodePort 10.96.228.248 \u0026#x3C;none\u003e 80:31771/TCP,443:31759/TCP 68s\nservice/ingress-nginx-controller-admission ClusterIP 10.96.180.126 \u0026#x3C;none\u003e 443/TCP 68s\n\nNAME READY UP-TO-DATE AVAILABLE AGE\ndeployment.apps/ingress-nginx-controller 0/1 1 0 68s\n\nNAME DESIRED CURRENT READY AGE\nreplicaset.apps/ingress-nginx-controller-5f748f78c8 1 1 0 68s\n\nNAME COMPLETIONS DURATION AGE\njob.batch/ingress-nginx-admission-create 1/1 22s 68s\njob.batch/ingress-nginx-admission-patch 1/1 35s 68s\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce \u003ccode\u003eingress-nginx-controller\u003c/code\u003e is in \u003ccode\u003eRunning\u003c/code\u003e state you are read to go!\u003c/p\u003e\n\u003ch1\u003eDeploying your first app\u003c/h1\u003e\n\u003cp\u003eTo prove that the cluster is working correctly we will deploy\n\u003ca href=\"https://github.com/Kong/httpbin\"\u003ehttpbin\u003c/a\u003e which is a nice little API server\nso we can prove everything is working.\u003c/p\u003e\n\u003cp\u003eCreate a \u003ccode\u003ehttbin.yml\u003c/code\u003e file and paste this into it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-meta\"\u003e---\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ev1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eService\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003elabels:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eservice:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eports:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttp\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eport:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8000\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003etargetPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8080\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eselector:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n\u003cspan class=\"hljs-meta\"\u003e---\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eapps/v1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eDeployment\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ereplicas:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eselector:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematchLabels:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eversion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ev1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003etemplate:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003elabels:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eversion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ev1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003econtainers:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003eimage:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edocker.io/mccutchen/go-httpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eimagePullPolicy:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eIfNotPresent\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eports:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003econtainerPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8080\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis is creating a couple of Kubernetes resources:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003eService\u003c/code\u003e: This is exposing the port to the ingress\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eDeployment\u003c/code\u003e: This is actually launching the service\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSo we are not using the ingress yet but we can prove that we can launch the\nservice at least. So apply those manifests:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl apply -f httpbin.yml \nservice/httpbin created\ndeployment.apps/httpbin created\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou should see two pods come up. You should wait for them to get into the\n\u003ccode\u003eRunning\u003c/code\u003e status:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get pod -o wide\nNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES\nhttpbin-5c5494967-2z5wz 1/1 Running 0 48s 10.244.3.3 kind-multinode-worker3 \u0026#x3C;none\u003e \u0026#x3C;none\u003e\nhttpbin-5c5494967-9lf47 1/1 Running 0 72s 10.244.1.2 kind-multinode-worker \u0026#x3C;none\u003e \u0026#x3C;none\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe can now use port forwarding to access it. \u003ccode\u003ehttpbin\u003c/code\u003e is exposed on \u003ccode\u003e8000\u003c/code\u003e so\nlets create port \u003ccode\u003e9000\u003c/code\u003e on our host that forwards to it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl port-forward service/httpbin 9000:8000\nForwarding from 127.0.0.1:9000 -\u003e 80\nForwarding from [::1]:9000 -\u003e 80\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can access it via:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl localhost:9000/get \n{\n \u003cspan class=\"hljs-string\"\u003e\"args\"\u003c/span\u003e: {},\n \u003cspan class=\"hljs-string\"\u003e\"headers\"\u003c/span\u003e: {\n \u003cspan class=\"hljs-string\"\u003e\"Accept\"\u003c/span\u003e: [\n \u003cspan class=\"hljs-string\"\u003e\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8\"\u003c/span\u003e\n ],\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eUsing Ingress\u003c/h1\u003e\n\u003cp\u003eNow to use the ingress rather than port forwarding we create one additional\nresource, the \u003ccode\u003eIngress\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003enetworking.k8s.io/v1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eIngress\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin-ingress\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eannotations:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003enginx.ingress.kubernetes.io/rewrite-target:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e/$2\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eingressClassName:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003enginx\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003erules:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ehttp:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epaths:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e/httpbin(/|$)(.*)\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epathType:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eImplementationSpecific\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ebackend:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eservice:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eport:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003enumber:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8000\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThere are a few critical options here. The first is the annotation to rewrite\nthe path so it doesn't include \u003ccode\u003e/httpbin/\u003c/code\u003e when it sends the request to the\nservice and then the \u003ccode\u003epath\u003c/code\u003e and \u003ccode\u003epathType\u003c/code\u003e so it knows which paths to send to\nwhich service.\u003c/p\u003e\n\u003cp\u003eNow you should be able to hit your local host and get routed to your\nkubernetes service:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl localhost/httpbin/get\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSuccess! Now you have a multinode kubernetes cluster that has an ingress\ncontroller!\u003c/p\u003e\n\u003ch1\u003eNext Steps\u003c/h1\u003e\n\u003cp\u003eThe cluster can be used like a production cluster now for local\ndevelopment! You could setup Grafana, ArgoCD, etc. to run\ninside the cluster.\u003c/p\u003e","category":"Kubernetes","date":"2023-07-21T00:00:00Z","tags":["Linux","Kubernetes","DevOps","SRE"],"title":"Running a kubernetes cluster locally with kind"},{"id":["2023","nuking_aws_account"],"path":"2023/nuking_aws_account","contentHtml":"\u003cp\u003eWhen you're an SRE/DevOps engineer you'll end up making AWS accounts and\ncreate a lot of cruft in your sandbox and development accounts. AWS\ndoes not make it easy to clear these up but there is a tool called\n\u003ca href=\"https://github.com/rebuy-de/aws-nuke\"\u003eaws-nuke\u003c/a\u003e that will do it for you!\u003c/p\u003e\n\u003ch1\u003eSafe Guards\u003c/h1\u003e\n\u003cp\u003eaws-nuke has a few safeguards in place to prevent inadvertent data loss.\nThe first of which is it requires you to alias the targetted account. I\nlike to put \u003ccode\u003enuke\u003c/code\u003e in the alias to make it clear.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eaws iam create-account-alias --account-alias nuke-\u0026#x3C;account\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe second safe-guard is the config takes a key called \u003ccode\u003eaccount-blocklist\u003c/code\u003e\nthat will guarantee nuke will not run against it no matter what.\u003c/p\u003e\n\u003cp\u003eThe final safety switch it has is it will not take any action by default,\nit will only execute a dry-run. You need to run the CLI with\n\u003ccode\u003e--no-dry-run\u003c/code\u003e if you want it to take action.\u003c/p\u003e\n\u003ch1\u003eGetting Started\u003c/h1\u003e\n\u003cp\u003eYou configure \u003ccode\u003eaws-nuke\u003c/code\u003e with YAML, so the first step is to define that:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-makefile\"\u003e\u003cspan class=\"hljs-section\"\u003eregions:\u003c/span\u003e\n - us-east-1\n - global\n\n\u003cspan class=\"hljs-section\"\u003eaccount-blocklist:\u003c/span\u003e\n - \u003cspan class=\"hljs-string\"\u003e\"888888888888\"\u003c/span\u003e \u003cspan class=\"hljs-comment\"\u003e# production\u003c/span\u003e\n\n\u003cspan class=\"hljs-section\"\u003eaccounts:\u003c/span\u003e\n \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e: {} \u003cspan class=\"hljs-comment\"\u003e# nuke-\u0026#x3C;account\u003e\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will prevent us from nuking our production account and target all resources\nin the account we actually want to nuke.\u003c/p\u003e\n\u003cp\u003eYou might want to have it nuke \u003cem\u003eALL REGIONS\u003c/em\u003e in AWS since you may not know which\nregions resources are deployed in. To do this you can query the regions from AWS:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ aws ec2 describe-regions --all-regions --query \u003cspan class=\"hljs-string\"\u003e\"Regions[*].RegionName\"\u003c/span\u003e --output text | xargs -n 1 | \u003cspan class=\"hljs-built_in\"\u003esort\u003c/span\u003e\n\naf-south-1\nap-east-1\nap-northeast-1\nap-northeast-2\nap-northeast-3\nap-south-1\nap-south-2\nap-southeast-1\nap-southeast-2\nap-southeast-3\nap-southeast-4\nca-central-1\neu-central-1\neu-central-2\neu-north-1\neu-south-1\neu-south-2\neu-west-1\neu-west-2\neu-west-3\nme-central-1\nme-south-1\nsa-east-1\nus-east-1\nus-east-2\nus-west-1\nus-west-2\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWhich would give you an updated config of:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003eregions:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eglobal\u003c/span\u003e \u003cspan class=\"hljs-comment\"\u003e# Global resources like IAM\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eaf-south-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-east-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-northeast-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-northeast-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-northeast-3\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-south-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-south-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-southeast-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-southeast-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-southeast-3\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-southeast-4\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eca-central-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-central-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-central-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-north-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-south-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-south-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-west-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-west-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-west-3\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eme-central-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eme-south-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esa-east-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eus-east-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eus-east-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eus-west-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eus-west-2\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eI personally don't recommend targetting all AWS regions at the same time. It\nwill generate a lot of output and be slow. You could do it if necessary but\nmost people only have a few regions they use and so they can set those directly.\nFor example it, maybe you only use \u003ccode\u003eus-\u003c/code\u003e based regions?\u003c/p\u003e\n\u003cp\u003eSo lets run the dry-run and see what it wants to nuke:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ aws-nuke -c nuke.yaml\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis should output something like:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eDo you really want to nuke the account with the ID 777777777777 and the \u003cspan class=\"hljs-built_in\"\u003ealias\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'nuke-sandbox'\u003c/span\u003e?\nDo you want to \u003cspan class=\"hljs-built_in\"\u003econtinue\u003c/span\u003e? Enter account \u003cspan class=\"hljs-built_in\"\u003ealias\u003c/span\u003e to \u003cspan class=\"hljs-built_in\"\u003econtinue\u003c/span\u003e.\n\u003e nuke-sandbox\nus-east-1 - EC2Subnet - subnet-0cd9975a443a6304b - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2Subnet - subnet-0be39d02e399a371c - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2Subnet - subnet-02d7017bd4730ea63 - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2Subnet - subnet-0ec04b28c32708ab2 - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2Subnet - subnet-0eea1b4be084840ed - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2Subnet - subnet-05a294cc04736012e - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2RouteTable - rtb-0abda0e94015064ca - [DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e] - would remove\nus-east-1 - EC2DefaultSecurityGroupRule - sgr-0368525f77bf566ac - [SecurityGroupId: \u003cspan class=\"hljs-string\"\u003e\"sg-0a59900b52ced5e10\"\u003c/span\u003e] - would remove\nus-east-1 - EC2DefaultSecurityGroupRule - sgr-0890a837ed6148729 - [SecurityGroupId: \u003cspan class=\"hljs-string\"\u003e\"sg-0a59900b52ced5e10\"\u003c/span\u003e] - would remove\nus-east-1 - EC2InternetGatewayAttachment - igw-0acfb474f1fd71375 -\u003e vpc-0be5d310ab44c239a - [DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e] - would remove\nus-east-1 - SQSQueue - https://sqs.us-east-1.amazonaws.com/777777777777/example-sqs - would remove\nglobal - IAMSAMLProvider - arn:aws:iam::777777777777:saml-provider/AWSSSO_254abb4071f10b25_DO_NOT_DELETE - would remove\nglobal - IAMOpenIDConnectProvider - arn:aws:iam::777777777777:oidc-provider/app.terraform.io - [Arn: \u003cspan class=\"hljs-string\"\u003e\"arn:aws:iam::777777777777:oidc-provider/app.terraform.io\"\u003c/span\u003e] - would remove\nglobal - IAMPolicy - arn:aws:iam::777777777777:policy/tfc-agent-access-policy - [ARN: \u003cspan class=\"hljs-string\"\u003e\"arn:aws:iam::777777777777:policy/tfc-agent-access-policy\"\u003c/span\u003e, Name: \u003cspan class=\"hljs-string\"\u003e\"tfc-agent-access-policy\"\u003c/span\u003e, Path: \u003cspan class=\"hljs-string\"\u003e\"/\"\u003c/span\u003e, PolicyID: \u003cspan class=\"hljs-string\"\u003e\"ANPA2T6PZOBNWI76TKQRF\"\u003c/span\u003e] - would remove\nglobal - IAMRole - tfc-agent - [CreateDate: \u003cspan class=\"hljs-string\"\u003e\"2023-04-02T17:55:23Z\"\u003c/span\u003e, LastUsedDate: \u003cspan class=\"hljs-string\"\u003e\"2023-06-22T13:45:02Z\"\u003c/span\u003e, Name: \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e, Path: \u003cspan class=\"hljs-string\"\u003e\"/\"\u003c/span\u003e] - would remove\nglobal - IAMRolePolicyAttachment - tfc-agent -\u003e tfc-agent-access-policy - [PolicyArn: \u003cspan class=\"hljs-string\"\u003e\"arn:aws:iam::777777777777:policy/tfc-agent-access-policy\"\u003c/span\u003e, PolicyName: \u003cspan class=\"hljs-string\"\u003e\"tfc-agent-access-policy\"\u003c/span\u003e, RoleCreateDate: \u003cspan class=\"hljs-string\"\u003e\"2023-04-02T17:55:23Z\"\u003c/span\u003e, RoleLastUsed: \u003cspan class=\"hljs-string\"\u003e\"2023-06-22T13:45:02Z\"\u003c/span\u003e, RoleName: \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e, RolePath: \u003cspan class=\"hljs-string\"\u003e\"/\"\u003c/span\u003e] - would remove\nScan complete: 85 total, 19 nukeable, 66 filtered.\n\nThe above resources would be deleted with the supplied configuration. Provide --no-dry-run to actually destroy resources.\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis is great, it fully scanned the account and found every resource to delete!\nIt even wants to delete the DefaultVPC which is usually a good idea. The one\nresource that should catch your eye that you probably do not want to delete:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003earn:aws:iam::777777777777:saml-provider/AWSSSO_254abb4071f10b25_DO_NOT_DELETE\u003c/code\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAWS clearly doesn't want us to delete that!\u003c/p\u003e\n\u003ch1\u003eFilters\u003c/h1\u003e\n\u003cp\u003eTo prevent nuke from deleting resources you want to keep you can define presets\nthat you use on each account. So with our SSO example we want to prevent it\nfrom deleting those resources in a preset.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003epresets:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esso:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efilters:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMSAMLProvider:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"regex\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"AWSSSO_.*_DO_NOT_DELETE\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRole:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"glob\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"AWSReservedSSO_*\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRolePolicyAttachment:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"glob\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"AWSReservedSSO_*\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can see in this example I'm targetting specific resource types and then\nmatching them with both \u003ccode\u003eregex\u003c/code\u003e and \u003ccode\u003eglob\u003c/code\u003e filter types. These are super\npowerful but a lot of times the simpler filters can be used. I start with\n\u003ccode\u003econtains\u003c/code\u003e filter and then go from there:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003econtains\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eAWSReservedSSO\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen the other thing you may have noticed is that I was repeating\n\u003ccode\u003eAWSReservedSSO\u003c/code\u003e multiple times. To reduce that you can use standard YAML\nanchors. So the final config for your preset would look like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003epresets:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esso:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efilters:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e_DEFAULT_FILTERS:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e\u0026#x26;DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"contains\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"DO_NOT_DELETE\"\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"contains\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"AWSReservedSSO\"\u003c/span\u003e\n\n \u003cspan class=\"hljs-attr\"\u003eIAMSAMLProvider:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRole:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRolePolicyAttachment:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we can use that preset in our accounts configuration:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003eaccounts:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"777777777777\":\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"presets\":\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esso\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo your final config should look something like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003eregions:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eus-east-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eglobal\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003eaccount-blocklist:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"888888888888\"\u003c/span\u003e \u003cspan class=\"hljs-comment\"\u003e# production\u003c/span\u003e\n\n\n\u003cspan class=\"hljs-attr\"\u003epresets:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esso:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efilters:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e_DEFAULT_FILTERS:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e\u0026#x26;DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"contains\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"DO_NOT_DELETE\"\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"contains\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"AWSReservedSSO\"\u003c/span\u003e\n\n \u003cspan class=\"hljs-attr\"\u003eIAMSAMLProvider:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRole:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRolePolicyAttachment:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003eaccounts:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"777777777777\":\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"presets\":\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esso\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWhen you run this you should see now the resources we want to keep are filtered\nout:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eglobal - IAMSAMLProvider - arn:aws:iam::777777777777:saml-provider/AWSSSO_254abb4071f10b25_DO_NOT_DELETE - filtered by config\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce you are ready and have your filters in place you can run it for real!\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ aws-nuke -c nuke.yaml --no-dry-run\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eNext steps\u003c/h1\u003e\n\u003cp\u003eOne final note about it is that it does not understand the relationship between\nresources and so it could try deleting an EBS volume that is still in use by an\nEC2 instance. There isn't a great solution for this outside of running nuke a\nfew times.\u003c/p\u003e\n\u003cp\u003eThe tool is well documented and so you can find the rest of information going to\n\u003ca href=\"https://github.com/rebuy-de/aws-nuke\"\u003ehttps://github.com/rebuy-de/aws-nuke\u003c/a\u003e!\u003c/p\u003e","category":"AWS","date":"2023-06-22T00:00:00Z","tags":["AWS","DevOps","SRE"],"title":"Wiping an AWS Account with aws-nuke"}]},"SRE":{"count":4,"posts":[{"id":["2022","local_kubeadm_cluster"],"path":"2022/local_kubeadm_cluster","contentHtml":"\u003cp\u003eI’m going to show you how to get a real kubernetes cluster setup locally on top of virtual\nmachines! I’ll be using multipass but feel free to use virtualbox, proxmox, or whatever your\nfavorite cloud provider is.\u003c/p\u003e\n\u003cp\u003ekubeadm a production ready kubernetes install tool and I prefer to use it over minikube, kind,\netc. because it gives you a more real world experience for \u003cem\u003emanaging\u003c/em\u003e the kubernetes cluster.\nThis isn’t important if you are a user of the cluster but if you have to run your own this is\na great way to gain some daily experience.\u003c/p\u003e\n\u003cp\u003eThe kubernetes documentation on kubeadm is great and you can find it \u003ca href=\"https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eThe differences between this blog and the kubernetes docs is that they leave a lot of decisions\nup to the reader such as:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003echoosing a container runtime\u003c/li\u003e\n\u003cli\u003eSelecting and installing a CNI (container network interface)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI’m going to be opinionated and make specific technology decisions such as using containerd and\ncilium so that you don't have to think about those decisions.\u003c/p\u003e\n\u003ch2\u003eGetting your Virtual Machines setup!\u003c/h2\u003e\n\u003cp\u003eThe minimum requirements for a control plane node in kubernetes is 2gb of RAM and 2 CPUs. Since\nwe actually want to be able to schedule workloads on the workers afterwards we are going to setup\na cluster that looks like this:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eControl Plane: 2gb RAM, 2 CPU\u003c/li\u003e\n\u003cli\u003eWorker: 4gb RAM, 2 CPU\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSince we’ll be using multipass to launch the nodes, we can do that now:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ multipass launch -c 2 -m 4G -d 10G -n controlplane 22.04\n❯ multipass launch -c 2 -m 4G -d 10G -n worker 22.04\n❯ multipass list\nName State IPv4 Image\ncontrolplane Running 192.168.64.7 Ubuntu 22.04 LTS\nworker Running 192.168.64.8 Ubuntu 22.04 LTS\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we can start working on our controlplane first, lets shell in:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ multipass shell controlplane\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eLets first add the kubernetes repo to the system so we have access to all the kubernetes tools:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb http://apt.kubernetes.io/ kubernetes-xenial main\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/kubernetes.list\n\n❯ curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg|sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/k8s.gpg\n❯ sudo apt-get update \u0026#x26;\u0026#x26; sudo apt-get upgrade -y\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow that our system is setup, we can move on to getting a container runtime.\u003c/p\u003e\n\u003ch2\u003eGetting your Container Runtime!\u003c/h2\u003e\n\u003cp\u003eBefore we start pulling in kubernetes components we need to get a container runtime setup on the\nmachine. We we are going to use containerd for this purpose. You can view the docs of for it\n\u003ca href=\"https://github.com/containerd/containerd/blob/main/docs/getting-started.md\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eWhich will download the latest binary and set it up. I’m going to walk you through how to do it\nusing the version packaged with Ubuntu which could be older than the latest release.\u003c/p\u003e\n\u003cp\u003eFirst thing we want to do is configure the networking to allow iptables to manage:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/modules-load.d/k8s.conf\noverlay\nbr_netfilter\nEOF\u003c/span\u003e\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/sysctl.d/k8s.conf\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nnet.ipv4.ip_forward = 1\nEOF\u003c/span\u003e\n\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe also need to disable some default systemd settings for \u003ccode\u003erp_filter\u003c/code\u003e because\nthey are not compatible with cilium. See the bug report\n\u003ca href=\"https://github.com/cilium/cilium/commit/cabc6581b8128681f4ed23f8d6dc463180eea61e\"\u003ehere\u003c/a\u003e\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo sed -i -e \u003cspan class=\"hljs-string\"\u003e'/net.ipv4.conf.*.rp_filter/d'\u003c/span\u003e $(grep -ril \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e /etc/sysctl.d/ /usr/lib/sysctl.d/)\n❯ sudo sysctl -a | grep \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e | awk \u003cspan class=\"hljs-string\"\u003e'{print $1\" = 0\"}'\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e -a /etc/sysctl.d/1000-cilium.conf\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen we need to refresh sysctl so those settings are applied:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo systemctl restart systemd-modules-load\n❯ sudo sysctl --system\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou should see it applying all the changes:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e* Applying /etc/sysctl.d/k8s.conf ...\n\u003cspan class=\"hljs-attr\"\u003enet.bridge.bridge-nf-call-ip6tables\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.bridge.bridge-nf-call-iptables\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.ip_forward\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIf you do not, the netfilter module may not have loaded properly:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ lsmod |grep br_netfilter\nbr_netfilter 28672 0\nbridge 176128 1 br_netfilter\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou want to make sure \u003ccode\u003erp_filter\u003c/code\u003e is \u003ccode\u003e0\u003c/code\u003e everywhere as well for cilium:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e❯ sudo sysctl -a | grep '\\.rp_filter'\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.all.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.cilium_host.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.cilium_net.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.cilium_vxlan.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.default.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.enp0s1.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.lo.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.lxc0965b7b545f7.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enet.ipv4.conf.lxcb05ffd84ab74.rp_filter\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow lets pull down the container runtime we’ll be using which is containerd.\u003c/p\u003e\n\u003cp\u003eUbuntu ships with a very old version of containerd so you need to upgrade to\nthe version shipped from the docker repos:\nYou can find which versions are available by running:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker.gpg\n❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb https://download.docker.com/linux/ubuntu \u003cspan class=\"hljs-subst\"\u003e$(lsb_release -cs)\u003c/span\u003e stable\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/docker.list\n❯ sudo apt-get update\n\u003c/code\u003e\u003c/pre\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-cache madison containerd.io\ncontainerd.io | 1.6.8-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.6.7-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.6.6-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.6.4-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.5.11-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\ncontainerd.io | 1.5.10-1 | https://download.docker.com/linux/ubuntu jammy/stable arm64 Packages\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe are going to use the latest version available which was 1.6.8-1\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-get install containerd.io=1.6.8-1 -y\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen we'll setup a configuration that enables containerd to use the systemd\ncgroup. We are hard coding this config instead of using \u003ccode\u003econtainerd config default\u003c/code\u003e\nbecause that currently has had a \u003ca href=\"https://github.com/containerd/containerd/issues/4574\"\u003ebug\u003c/a\u003e\nfor many years that generates an invalid config.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/containerd/config.toml\nversion = 2\n[plugins]\n [plugins.\"io.containerd.grpc.v1.cri\"]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc]\n runtime_type = \"io.containerd.runc.v2\"\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc.options]\n SystemdCgroup = true\nEOF\u003c/span\u003e\n\n❯ sudo systemctl restart containerd.service\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can verify its running with ctr:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo ctr --address /var/run/containerd/containerd.sock containers list\nCONTAINER IMAGE RUNTIME\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow that this is working we can move on to getting kubernetes installed!\u003c/p\u003e\n\u003ch2\u003eUsing kubeadm!\u003c/h2\u003e\n\u003cp\u003eNow we need to get the kubernetes tools installed onto the system. I’m going to be using 1.23\nbut to find the latest version you can run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-cache madison kubeadm|\u003cspan class=\"hljs-built_in\"\u003ehead\u003c/span\u003e -n2\n kubeadm | 1.23.5-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages\n kubeadm | 1.23.4-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen install the version you want, we install kubelet and kubeadm here to make\nsure the versions align:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo apt-get install kubeadm=1.23.5-00 kubelet=1.23.5-00 kubectl=1.23.5-00 -y\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will pull in a few tools, including an alternative to \u003ccode\u003ectr\u003c/code\u003e that we used earlier called\n\u003ccode\u003ecrictl\u003c/code\u003e. You can check that it is available to you doing this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo crictl --runtime-endpoint=unix:///var/run/containerd/containerd.sock ps\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe can finally init our cluster:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo kubeadm init\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce that finishes running it should give you some tips setup your configuration, it should look like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e -p \u003cspan class=\"hljs-variable\"\u003e$HOME\u003c/span\u003e/.kube\n❯ sudo \u003cspan class=\"hljs-built_in\"\u003ecp\u003c/span\u003e -i /etc/kubernetes/admin.conf \u003cspan class=\"hljs-variable\"\u003e$HOME\u003c/span\u003e/.kube/config\n❯ sudo \u003cspan class=\"hljs-built_in\"\u003echown\u003c/span\u003e $(\u003cspan class=\"hljs-built_in\"\u003eid\u003c/span\u003e -u):$(\u003cspan class=\"hljs-built_in\"\u003eid\u003c/span\u003e -g) \u003cspan class=\"hljs-variable\"\u003e$HOME\u003c/span\u003e/.kube/config\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can run those on the master node for now, but later I'll show you how to move\nthe config to your host computer.\u003c/p\u003e\n\u003cp\u003eNow you should be able to check that your node is not ready yet:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get nodes\nNAME STATUS ROLES AGE VERSION\ncontrolplane NotReady control-plane,master 4m16s v1.23.5\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cem\u003eNote\u003c/em\u003e: If you recieve \"The connecto to the server was refused\" error,\nThe cluster starting up and getting all the dependencies running could take\na bit of time. So if you aren't able to communicate right away you can check\nwhich pods are up and running with \u003ccode\u003ecrictl\u003c/code\u003e. You'll need \u003ccode\u003ekube-apiserver\u003c/code\u003e up\nand running. If it isn't you can check:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo crictl --runtime-endpoint=unix:///var/run/containerd/containerd.sock ps -a\nCONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD\n8322192c4605c bd8cc6d582470 36 seconds ago Running kube-proxy 4 344c4f7fffbe8 kube-proxy-drm46\n30ce27c40adb2 81a4a8a4ac639 2 minutes ago Exited kube-controller-manager 4 3a819c3a864b2 kube-controller-manager-controlplane\n7709fd5e92898 bd8cc6d582470 2 minutes ago Exited kube-proxy 3 7cc6922c82015 kube-proxy-drm46\n10432b81d7c61 3767741e7fba7 2 minutes ago Exited kube-apiserver 4 e64ddf3679d98 kube-apiserver-controlplane\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003ewhich will show you pods that have exited. You can grab the container ID for\nkube-apiserver and read its logs:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo crictl --runtime-endpoint=unix:///var/run/containerd/containerd.sock logs 10432b81d7c61\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThere are a few ways to figure out why the node isn’t ready yet. Usually I would check the\n\u003ccode\u003ekubelet\u003c/code\u003e logs first:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo journalctl -flu kubelet\n-- Logs begin at Sun 2022-04-17 19:22:19 AST. --\nApr 17 20:53:15 controlplane kubelet[19727]: E0417 20:53:15.951350 19727 kubelet.go:2347] \u003cspan class=\"hljs-string\"\u003e\"Container runtime network not ready\"\u003c/span\u003e networkReady=\u003cspan class=\"hljs-string\"\u003e\"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized\"\u003c/span\u003e\nApr 17 20:53:20 controlplane kubelet[19727]: E0417 20:53:20.952148 19727 kubelet.go:2347] \u003cspan class=\"hljs-string\"\u003e\"Container runtime network not ready\"\u003c/span\u003e networkReady=\u003cspan class=\"hljs-string\"\u003e\"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIt is clear the problem is that we are missing the CNI. The other way you can find out what is\ngoing on is describing the node:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl describe node controlplane\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will have a lot of information but if you scroll through there looking at \u003ccode\u003eReason\u003c/code\u003e you\nmight see something useful. In this case under \u003ccode\u003eLease\u003c/code\u003e you would see:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl describe node controlplane|grep NotReady\nReady False Sun, 17 Apr 2022 20:53:37 -0400 Sun, 17 Apr 2022 20:43:07 -0400 KubeletNotReady container runtime network not ready: NetworkReady=\u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialize\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eLets get our CNI installed, we’ll be using cilium!\u003c/p\u003e\n\u003ch2\u003eSetting up your CNI!\u003c/h2\u003e\n\u003cp\u003eCilium has great documentation over \u003ca href=\"https://docs.cilium.io/en/v1.9/gettingstarted/k8s-install-kubeadm/\"\u003ehere\u003c/a\u003e,\nbut I’ll walk you through it anyways. I do recommend checking out their documentation so you\nare familiar with it. We will use \u003ccode\u003ehelm\u003c/code\u003e to pull down the version of cilium we want:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl -fsSL https://baltocdn.com/helm/signing.asc | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/helm.gpg\n\n❯ sudo apt-get install apt-transport-https --\u003cspan class=\"hljs-built_in\"\u003eyes\u003c/span\u003e\n\n❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb https://baltocdn.com/helm/stable/debian/ all main\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/helm-stable-debian.list\n\n❯ sudo apt-get update\n❯ sudo apt-get install helm\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we can install cilium! It is \u003cem\u003every\u003c/em\u003e important that you pay attention to the\ncompatibility of cilium with the version of kubernetes you are intstalling. Check\nthe compatibility list \u003ca href=\"https://docs.cilium.io/en/v1.12/concepts/kubernetes/compatibility/\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ helm repo add cilium https://helm.cilium.io/\n❯ helm repo update\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce the repo is added you can list the versions available:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ helm search repo -l|\u003cspan class=\"hljs-built_in\"\u003ehead\u003c/span\u003e -n8\nNAME \tCHART VERSION\tAPP VERSION\tDESCRIPTION\ncilium/cilium \t1.12.1 \t1.12.1 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.12.0 \t1.12.0 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.8 \t1.11.8 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.7 \t1.11.7 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.6 \t1.11.6 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.5 \t1.11.5 \teBPF-based Networking, Security, and Observability\ncilium/cilium \t1.11.4 \t1.11.4 \teBPF-based Networking, Security, and Observability\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo we want \u003ccode\u003e1.11.4\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ helm install cilium cilium/cilium --namespace kube-system --version 1.11.4\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow our node should be ready!\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get node\nNAME STATUS ROLES AGE VERSION\ncontrolplane Ready control-plane,master 24m v1.23.5\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eTime to join our worker to the cluster!\u003c/p\u003e\n\u003ch2\u003eJoining a worker to the cluster!\u003c/h2\u003e\n\u003cp\u003eWe have to go through the same steps as the controlplane to get the point that we have a\ncontainer runtime and \u003ccode\u003ekubeadm\u003c/code\u003e. I’m not going to talk about the commands a second time but\nI’ll re-iterate them here for ease of following along.\u003c/p\u003e\n\u003cp\u003eFirst open up another shell and connect to the worker:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ multipass shell worker\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow run the following commands:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb http://apt.kubernetes.io/ kubernetes-xenial main\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/kubernetes.list\n❯ curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg|sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/k8s.gpg\n❯ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker.gpg\n❯ \u003cspan class=\"hljs-built_in\"\u003eecho\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"deb https://download.docker.com/linux/ubuntu \u003cspan class=\"hljs-subst\"\u003e$(lsb_release -cs)\u003c/span\u003e stable\"\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e /etc/apt/sources.list.d/docker.list\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/modules-load.d/k8s.conf\noverlay\nbr_netfilter\nEOF\u003c/span\u003e\n\n❯ sudo sed -i -e \u003cspan class=\"hljs-string\"\u003e'/net.ipv4.conf.*.rp_filter/d'\u003c/span\u003e $(grep -ril \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e /etc/sysctl.d/ /usr/lib/sysctl.d/)\n❯ sudo sysctl -a | grep \u003cspan class=\"hljs-string\"\u003e'\\.rp_filter'\u003c/span\u003e | awk \u003cspan class=\"hljs-string\"\u003e'{print $1\" = 0\"}'\u003c/span\u003e | sudo \u003cspan class=\"hljs-built_in\"\u003etee\u003c/span\u003e -a /etc/sysctl.d/1000-cilium.conf\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/sysctl.d/k8s.conf\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nnet.ipv4.ip_forward = 1\nEOF\u003c/span\u003e\n\n❯ sudo systemctl restart systemd-modules-load\n❯ sudo sysctl --system\n\n❯ sudo apt-get update \u0026#x26;\u0026#x26; sudo apt-get upgrade -y\n❯ sudo apt-get install containerd.io=1.6.8-1 -y\n\n❯ \u003cspan class=\"hljs-built_in\"\u003ecat\u003c/span\u003e \u0026#x3C;\u0026#x3C;\u003cspan class=\"hljs-string\"\u003eEOF | sudo tee /etc/containerd/config.toml\nversion = 2\n[plugins]\n [plugins.\"io.containerd.grpc.v1.cri\"]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc]\n runtime_type = \"io.containerd.runc.v2\"\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc.options]\n SystemdCgroup = true\nEOF\u003c/span\u003e\n\n❯ sudo systemctl restart containerd.service\n❯ sudo apt-get install kubeadm=1.23.5-00 kubelet=1.23.5-00 kubectl=1.23.5-00 -y\n\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eFrom there we should be ready to join the cluster. When we ran \u003ccode\u003ekubeadm init\u003c/code\u003e previously it\nprinted a join command out that we could use but I’m going to show you how to do it if you\nwere coming back later and no longer had that token.\u003c/p\u003e\n\u003cp\u003eBack on the \u003cem\u003econtroplane\u003c/em\u003e node run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubeadm token create --print-join-command\nkubeadm \u003cspan class=\"hljs-built_in\"\u003ejoin\u003c/span\u003e 192.168.64.7:6443 --token wxs197.cco6mjj9ricvu8ov --discovery-token-ca-cert-hash sha256:bd01c065240fa76f30a02ecb70a8cea6e329c9678994d4da1f6ccac7694b97fb\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow copy that command and run it with \u003ccode\u003esudo\u003c/code\u003e on the worker:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ sudo kubeadm \u003cspan class=\"hljs-built_in\"\u003ejoin\u003c/span\u003e 192.168.64.7:6443 --token wxs197.cco6mjj9ricvu8ov --discovery-token-ca-cert-hash sha256:bd01c065240fa76f30a02ecb70a8cea6e329c9678994d4da1f6ccac7694b97fb\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAfter this completes it’ll take a minute or two for everything to be synced up but if you go\nback to the master node you should have 2 ready nodes now:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get nodes\nNAME STATUS ROLES AGE VERSION\ncontrolplane Ready control-plane,master 46m v1.23.5\nworker Ready \u0026#x3C;none\u003e 79s v1.23.5\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eAccessing the cluster outside of the VMs!\u003c/h2\u003e\n\u003cp\u003eNow the final part is to get the \u003ccode\u003eadmin.conf\u003c/code\u003e as a kubeconfig on your machine so you can control\nit from outside of the cluster. To do this we can use scp\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003emultipass transfer controlplane:/home/ubuntu/.kube/config local.config\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNormally kubernetes configuration is in ~/.kube/config but I like to maint a separate file for\neach cluster and then I set the \u003ccode\u003eKUBECONFIG\u003c/code\u003e env var to access it.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003eexport\u003c/span\u003e KUBECONFIG=local.config\n❯ kubectl get nodes\nNAME STATUS ROLES AGE VERSION\ncontrolplane Ready control-plane,master 56m v1.23.5\nworker Ready \u0026#x3C;none\u003e 11m v1.23.5\n\u003c/code\u003e\u003c/pre\u003e","category":"Kubernetes","date":"2022-04-17T00:00:00Z","tags":["Linux","Kubernetes","DevOps","SRE"],"title":"Running a kubernetes cluster locally with kubeadm"},{"id":["2023","aws_from_scratch_apply_before_merge"],"path":"2023/aws_from_scratch_apply_before_merge","contentHtml":"\u003cp\u003eFollowing this article will get you setup with an AWS Root account that can be\nmanaged through through Terraform Cloud with OIDC and github actions. As a best practice you\nshould not keep long-lived access keys in your CI/CD pipelines when\ndeploying to AWS, instead you should use OIDC (OpenID Connect) to securely\ndeploy to AWS when using Terraform Cloud or Github Actions.\u003c/p\u003e\n\u003ciframe width=\"854\" height=\"480\" src=\"https://www.youtube.com/embed/3oZd1m8_KIo\" title=\"AWS From Scratch - Preparing your account to be managed by IaC via Terraform and Github Actions\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture;web-share\" allowfullscreen\u003e\u003c/iframe\u003e\n\u003ch1\u003eTL;DR\u003c/h1\u003e\n\u003cp\u003eDownload all the terraform from the blog post here:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/sontek/aws-apply-before-merge\"\u003ehttps://github.com/sontek/aws-apply-before-merge\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/sontek/aws-terraform-github-actions\"\u003ehttps://github.com/sontek/aws-terraform-github-actions\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch1\u003eHow does OIDC work\u003c/h1\u003e\n\u003cp\u003eOIDC enables us to request a short-lived access token directly from AWS. We\njust have to create trust relationship that controls which workflows are able\nto request the access tokens.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eNo need to duplicate AWS credentials as long-lived GitHub secrets.\u003c/li\u003e\n\u003cli\u003eSince we are using a short-lived access token that is only valid for a single\njob there is no reason to worry about rotating secrets.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe following diagram gives an overview of how we can use Terraform Cloud's\nOIDC provider to integrate with AWS:\u003c/p\u003e\n\u003cdiv class=\"remark-mermaid remark-mermaid-default\"\u003e\u003csvg aria-roledescription=\"flowchart-v2\" role=\"graphics-document document\" viewBox=\"-8 -8 843.078125 320\" style=\"max-width: 843.078px; background-color: transparent;\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xmlns=\"http://www.w3.org/2000/svg\" width=\"100%\" id=\"mermaid-1691159159024\"\u003e\u003cstyle\u003e#mermaid-1691159159024{font-family:\"trebuchet ms\",verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-1691159159024 .error-icon{fill:#552222;}#mermaid-1691159159024 .error-text{fill:#552222;stroke:#552222;}#mermaid-1691159159024 .edge-thickness-normal{stroke-width:2px;}#mermaid-1691159159024 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-1691159159024 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-1691159159024 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-1691159159024 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-1691159159024 .marker{fill:#333333;stroke:#333333;}#mermaid-1691159159024 .marker.cross{stroke:#333333;}#mermaid-1691159159024 svg{font-family:\"trebuchet ms\",verdana,arial,sans-serif;font-size:16px;}#mermaid-1691159159024 .label{font-family:\"trebuchet ms\",verdana,arial,sans-serif;color:#333;}#mermaid-1691159159024 .cluster-label text{fill:#333;}#mermaid-1691159159024 .cluster-label span{color:#333;}#mermaid-1691159159024 .label text,#mermaid-1691159159024 span{fill:#333;color:#333;}#mermaid-1691159159024 .node rect,#mermaid-1691159159024 .node circle,#mermaid-1691159159024 .node ellipse,#mermaid-1691159159024 .node polygon,#mermaid-1691159159024 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-1691159159024 .node .label{text-align:center;}#mermaid-1691159159024 .node.clickable{cursor:pointer;}#mermaid-1691159159024 .arrowheadPath{fill:#333333;}#mermaid-1691159159024 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-1691159159024 .flowchart-link{stroke:#333333;fill:none;}#mermaid-1691159159024 .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-1691159159024 .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-1691159159024 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-1691159159024 .cluster text{fill:#333;}#mermaid-1691159159024 .cluster span{color:#333;}#mermaid-1691159159024 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\"trebuchet ms\",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-1691159159024 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-1691159159024 :root{--mermaid-font-family:\"trebuchet ms\",verdana,arial,sans-serif;}\u003c/style\u003e\u003cg\u003e\u003cmarker orient=\"auto\" markerHeight=\"12\" markerWidth=\"12\" markerUnits=\"userSpaceOnUse\" refY=\"5\" refX=\"10\" viewBox=\"0 0 12 20\" class=\"marker flowchart\" id=\"flowchart-pointEnd\"\u003e\u003cpath style=\"stroke-width: 1; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" d=\"M 0 0 L 10 5 L 0 10 z\"\u003e\u003c/path\u003e\u003c/marker\u003e\u003cmarker orient=\"auto\" markerHeight=\"12\" markerWidth=\"12\" markerUnits=\"userSpaceOnUse\" refY=\"5\" refX=\"0\" viewBox=\"0 0 10 10\" class=\"marker flowchart\" id=\"flowchart-pointStart\"\u003e\u003cpath style=\"stroke-width: 1; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" d=\"M 0 5 L 10 10 L 10 0 z\"\u003e\u003c/path\u003e\u003c/marker\u003e\u003cmarker orient=\"auto\" markerHeight=\"11\" markerWidth=\"11\" markerUnits=\"userSpaceOnUse\" refY=\"5\" refX=\"11\" viewBox=\"0 0 10 10\" class=\"marker flowchart\" id=\"flowchart-circleEnd\"\u003e\u003ccircle style=\"stroke-width: 1; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" r=\"5\" cy=\"5\" cx=\"5\"\u003e\u003c/circle\u003e\u003c/marker\u003e\u003cmarker orient=\"auto\" markerHeight=\"11\" markerWidth=\"11\" markerUnits=\"userSpaceOnUse\" refY=\"5\" refX=\"-1\" viewBox=\"0 0 10 10\" class=\"marker flowchart\" id=\"flowchart-circleStart\"\u003e\u003ccircle style=\"stroke-width: 1; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" r=\"5\" cy=\"5\" cx=\"5\"\u003e\u003c/circle\u003e\u003c/marker\u003e\u003cmarker orient=\"auto\" markerHeight=\"11\" markerWidth=\"11\" markerUnits=\"userSpaceOnUse\" refY=\"5.2\" refX=\"12\" viewBox=\"0 0 11 11\" class=\"marker cross flowchart\" id=\"flowchart-crossEnd\"\u003e\u003cpath style=\"stroke-width: 2; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" d=\"M 1,1 l 9,9 M 10,1 l -9,9\"\u003e\u003c/path\u003e\u003c/marker\u003e\u003cmarker orient=\"auto\" markerHeight=\"11\" markerWidth=\"11\" markerUnits=\"userSpaceOnUse\" refY=\"5.2\" refX=\"-1\" viewBox=\"0 0 11 11\" class=\"marker cross flowchart\" id=\"flowchart-crossStart\"\u003e\u003cpath style=\"stroke-width: 2; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" d=\"M 1,1 l 9,9 M 10,1 l -9,9\"\u003e\u003c/path\u003e\u003c/marker\u003e\u003cg class=\"root\"\u003e\u003cg class=\"clusters\"\u003e\u003c/g\u003e\u003cg class=\"edgePaths\"\u003e\u003cpath marker-end=\"url(#flowchart-pointEnd)\" style=\"fill:none;\" class=\"edge-thickness-normal edge-pattern-solid flowchart-link LS-AWS LE-Token\" id=\"L-AWS-Token-0\" d=\"M185.671875,134.5L181.50520833333334,134.5C177.33854166666666,134.5,169.00520833333334,134.5,152.378260969764,145.875C135.75131360619469,157.25,110.83075221238937,180,98.37047151548673,191.375L85.91019081858407,202.75\"\u003e\u003c/path\u003e\u003cpath marker-end=\"url(#flowchart-pointEnd)\" style=\"fill:none;\" class=\"edge-thickness-normal edge-pattern-solid flowchart-link LS-Token LE-Terraform\" id=\"L-Token-Terraform-0\" d=\"M85.91019081858407,235.75L98.37047151548673,247.125C110.83075221238937,258.5,135.75131360619469,281.25,164.46289638643069,292.625C193.17447916666666,304,225.67708333333334,304,258.1796875,304C290.6822916666667,304,323.1848958333333,304,359.3528645833333,304C395.5208333333333,304,435.3541666666667,304,475.1875,304C515.0208333333334,304,554.8541666666666,304,585.6956312991642,296.7916666666667C616.5370959316618,289.5833333333333,638.3866918633236,275.1666666666667,649.3114898291544,267.9583333333333L660.2362877949853,260.75\"\u003e\u003c/path\u003e\u003cpath marker-end=\"url(#flowchart-pointEnd)\" style=\"fill:none;\" class=\"edge-thickness-normal edge-pattern-solid flowchart-link LS-Terraform LE-JWT\" id=\"L-Terraform-JWT-0\" d=\"M660.2362877949853,177.75L649.3114898291544,170.54166666666666C638.3866918633236,163.33333333333334,616.5370959316618,148.91666666666666,601.4456312991642,141.70833333333334C586.3541666666666,134.5,578.0208333333334,134.5,573.8541666666666,134.5L569.6875,134.5\"\u003e\u003c/path\u003e\u003cpath marker-end=\"url(#flowchart-pointEnd)\" style=\"fill:none;\" class=\"edge-thickness-normal edge-pattern-solid flowchart-link LS-JWT LE-AWS\" id=\"L-JWT-AWS-0\" d=\"M380.6875,134.5L376.5208333333333,134.5C372.3541666666667,134.5,364.0208333333333,134.5,355.6875,134.5C347.3541666666667,134.5,339.0208333333333,134.5,334.8541666666667,134.5L330.6875,134.5\"\u003e\u003c/path\u003e\u003c/g\u003e\u003cg class=\"edgeLabels\"\u003e\u003cg class=\"edgeLabel\"\u003e\u003cg transform=\"translate(0, 0)\" class=\"label\"\u003e\u003cforeignObject height=\"0\" width=\"0\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"edgeLabel\"\u003e\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"edgeLabel\"\u003e\u003cg transform=\"translate(0, 0)\" class=\"label\"\u003e\u003cforeignObject height=\"0\" width=\"0\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"edgeLabel\"\u003e\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"edgeLabel\"\u003e\u003cg transform=\"translate(0, 0)\" class=\"label\"\u003e\u003cforeignObject height=\"0\" width=\"0\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"edgeLabel\"\u003e\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"edgeLabel\"\u003e\u003cg transform=\"translate(0, 0)\" class=\"label\"\u003e\u003cforeignObject height=\"0\" width=\"0\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"edgeLabel\"\u003e\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"nodes\"\u003e\u003cg transform=\"translate(621.1171875, 169.75)\" class=\"root\"\u003e\u003cg class=\"clusters\"\u003e\u003cg id=\"Terraform\" class=\"cluster default\"\u003e\u003crect height=\"83\" width=\"206.890625\" y=\"8\" x=\"-0.9296875\" ry=\"0\" rx=\"0\" style=\"\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-0.9296875, 8)\" class=\"cluster-label\"\u003e\u003cforeignObject height=\"18\" width=\"206.890625\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eTerraform Cloud Workflow #2\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"edgePaths\"\u003e\u003c/g\u003e\u003cg class=\"edgeLabels\"\u003e\u003c/g\u003e\u003cg class=\"nodes\"\u003e\u003cg transform=\"translate(102.515625, 49.5)\" id=\"flowchart-OIDCProvider-23\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"119.03125\" y=\"-16.5\" x=\"-59.515625\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-52.015625, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"104.03125\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eOIDC Provider\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003cg transform=\"translate(178.171875, -8)\" class=\"root\"\u003e\u003cg class=\"clusters\"\u003e\u003cg id=\"AWS\" class=\"cluster default\"\u003e\u003crect height=\"269\" width=\"145.015625\" y=\"8\" x=\"8\" ry=\"0\" rx=\"0\" style=\"\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(51.4609375, 8)\" class=\"cluster-label\"\u003e\u003cforeignObject height=\"18\" width=\"58.09375\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eAWS #1\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"edgePaths\"\u003e\u003c/g\u003e\u003cg class=\"edgeLabels\"\u003e\u003c/g\u003e\u003cg class=\"nodes\"\u003e\u003cg transform=\"translate(80.5078125, 59.5)\" id=\"flowchart-OIDC-20\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"95.015625\" y=\"-16.5\" x=\"-47.5078125\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-40.0078125, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"80.015625\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eOIDC Trust\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg transform=\"translate(80.5078125, 142.5)\" id=\"flowchart-Roles-21\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"55.90625\" y=\"-16.5\" x=\"-27.953125\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-20.453125, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"40.90625\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eRoles\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg transform=\"translate(80.5078125, 225.5)\" id=\"flowchart-Resources-22\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"91.484375\" y=\"-16.5\" x=\"-45.7421875\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-38.2421875, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"76.484375\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eResources\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003cg transform=\"translate(67.8359375, 219.25)\" id=\"flowchart-Token-25\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"135.671875\" y=\"-16.5\" x=\"-67.8359375\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-60.3359375, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"120.671875\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eAccess Token #4\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg transform=\"translate(475.1875, 134.5)\" id=\"flowchart-JWT-28\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"189\" y=\"-16.5\" x=\"-94.5\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-87, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"174\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eJWT \u0026#x26; Cloud Role ID #3\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003c/svg\u003e\u003c/div\u003e\n\u003col\u003e\n\u003cli\u003eIn AWS, create an OIDC trust between a role and our terraform cloud\nworkflow(s) that need access to the cloud.\u003c/li\u003e\n\u003cli\u003eEvery time a job runs, TFC's OIDC Provider auto-generates an OIDC token.\nThis token contains multiple claims to establish a security-hardened and\nverifiable identity about the specific workflow that is trying to authenticate.\u003c/li\u003e\n\u003cli\u003eRequest this token from TFC's OIDC provider, and present it to AWS\u003c/li\u003e\n\u003cli\u003eOnce AWS successfully validates the claims presented in the token, it then\nprovides a short-lived cloud access token that is available only for the duration\nof the job.\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch1\u003eWhat does this post accomplish\u003c/h1\u003e\n\u003cul\u003e\n\u003cli\u003eSetup a root AWS account that is managed througuh terraform\u003c/li\u003e\n\u003cli\u003eSetup OIDC authentication with Terraform Cloud so it can talk to AWS\u003c/li\u003e\n\u003cli\u003eSetup Github Actions authentication with Terraform Cloud so we can run plan\nand apply through the CI/CD pipeline.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch1\u003eSetup AWS Access\u003c/h1\u003e\n\u003cp\u003eIt is very bad practice to use the root account for much of anything but for\nbootstrapping the account it is necessary, so the first step is to get your\n\u003ccode\u003eAWS_ACCESS_KEY_ID\u003c/code\u003e and \u003ccode\u003eAWS_SECRET_ACCESS_KEY\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003eTo do this click your account and choose \u003ccode\u003eSecurity Credentials\u003c/code\u003e in the top\nright:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_root_account/security_credentials.png\" height=\"200\"\u003e\n\u003c/center\u003e\n\u003cp\u003eThen choose \u003ccode\u003eCreate Access key\u003c/code\u003e:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_root_account/create_access_token.png\" width=\"200\"\u003e\n\u003c/center\u003e\n\u003cp\u003eYou need to set these environment variables in your shell so that your local\nshell has access to AWS. After you set them you can verify you set them correct\nby running:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ aws sts get-caller-identity\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eand you should get a result similar to:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-json\"\u003e\u003cspan class=\"hljs-punctuation\"\u003e{\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"UserId\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e,\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"Account\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"888888888888\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e,\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"Arn\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"arn:aws:iam::888888888888:root\"\u003c/span\u003e\n\u003cspan class=\"hljs-punctuation\"\u003e}\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eBootstrap\u003c/h2\u003e\n\u003cp\u003eBefore you can manage any of your accounts through Terraform Cloud you'll need\nbootstrap some core infrastructure like OIDC so Terraform Cloud can authenticate\nsecurely and manage AWS Resources on your behalf.\u003c/p\u003e\n\u003cp\u003eI personally prefer doing this in two repositories:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ccode\u003einfra-bootstrap\u003c/code\u003e: This repository does the bare minimum to hook up terraform\ncloud with your AWS account and stores the state in git. Its the only infra\nthat will not be controlled by your CI/CD pipeline.ccccccug\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ccode\u003einfra\u003c/code\u003e: The actual repository where all the rest of your AWS resources are\nmanaged. It will store state in Terraform Cloud and you can introduce a\nCI/CD pipeline for approving changes.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNote\u003c/strong\u003e: This repository will be generated with the terraform code.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAfter manually creating the git repository \u003ccode\u003einfra-boostrap\u003c/code\u003e in your Github\naccount We will need 3 providers to bootstrap the account \u003ccode\u003eaws\u003c/code\u003e, \u003ccode\u003egithub\u003c/code\u003e, and\n\u003ccode\u003etfe\u003c/code\u003e.\u003c/p\u003e\n\u003ch3\u003eVariables\u003c/h3\u003e\n\u003cp\u003eCreate a \u003ccode\u003e1-variables.tf\u003c/code\u003e where we can define the variables we'll need\nfor creating these resources.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_aws_audience\"\u003c/span\u003e {\n type = string\n default = \u003cspan class=\"hljs-string\"\u003e\"aws.workload.identity\"\u003c/span\u003e\n description = \u003cspan class=\"hljs-string\"\u003e\"The audience value to use in run identity tokens\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_hostname\"\u003c/span\u003e {\n type = string\n default = \u003cspan class=\"hljs-string\"\u003e\"app.terraform.io\"\u003c/span\u003e\n description = \u003cspan class=\"hljs-string\"\u003e\"The hostname of the TFC or TFE instance you'd like to use with AWS\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_project_name\"\u003c/span\u003e {\n type = string\n default = \u003cspan class=\"hljs-string\"\u003e\"Default Project\"\u003c/span\u003e\n description = \u003cspan class=\"hljs-string\"\u003e\"The project under which a workspace will be created\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_organization_name\"\u003c/span\u003e {\n type = string\n description = \u003cspan class=\"hljs-string\"\u003e\"The name of your Terraform Cloud organization\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_organization_owner\"\u003c/span\u003e {\n type = string\n description = \u003cspan class=\"hljs-string\"\u003e\"The owner of the TFC organization\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_workspaces\"\u003c/span\u003e {\n type = list(string)\n description = \u003cspan class=\"hljs-string\"\u003e\"The list of TFC workspaces\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_organization\"\u003c/span\u003e {\n description = \u003cspan class=\"hljs-string\"\u003e\"The organization the repositories are owned by\"\u003c/span\u003e\n type = string\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_repo_name\"\u003c/span\u003e {\n description = \u003cspan class=\"hljs-string\"\u003e\"The name of the git reppository we'll create for managing infra\"\u003c/span\u003e\n type = string\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_default_branch\"\u003c/span\u003e {\n description = \u003cspan class=\"hljs-string\"\u003e\"The default branch to utilize\"\u003c/span\u003e\n type = string\n default = \u003cspan class=\"hljs-string\"\u003e\"main\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_root_account_id\"\u003c/span\u003e {\n description = \u003cspan class=\"hljs-string\"\u003e\"The AWS root account we want to apply these changes to\"\u003c/span\u003e\n type = string\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe will use these variables in the later modules but they are mostly metadata\naround the terraform and github accounts you'll need to setup manually.\u003c/p\u003e\n\u003ch3\u003eProviders\u003c/h3\u003e\n\u003cp\u003eCreate a file called \u003ccode\u003e2-providers.tf\u003c/code\u003e and define the providers:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e {\n required_providers {\n tfe = {\n source = \u003cspan class=\"hljs-string\"\u003e\"hashicorp/tfe\"\u003c/span\u003e\n version = \u003cspan class=\"hljs-string\"\u003e\"0.41.0\"\u003c/span\u003e\n }\n\n aws = {\n source = \u003cspan class=\"hljs-string\"\u003e\"hashicorp/aws\"\u003c/span\u003e\n version = \u003cspan class=\"hljs-string\"\u003e\"4.58.0\"\u003c/span\u003e\n }\n\n github = {\n source = \u003cspan class=\"hljs-string\"\u003e\"integrations/github\"\u003c/span\u003e\n version = \u003cspan class=\"hljs-string\"\u003e\"5.18.3\"\u003c/span\u003e\n }\n }\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eprovider\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws\"\u003c/span\u003e {\n region = \u003cspan class=\"hljs-string\"\u003e\"us-east-1\"\u003c/span\u003e\n\n \u003cspan class=\"hljs-comment\"\u003e# Root account, all other accounts should be provisioned\u003c/span\u003e\n \u003cspan class=\"hljs-comment\"\u003e# via pull requests\u003c/span\u003e\n allowed_account_ids = [var.aws_root_account_id]\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eprovider\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github\"\u003c/span\u003e {\n owner = var.github_organization\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe key things there are we define \u003ccode\u003eallowed_account_ids\u003c/code\u003e to prevent us from\nworking against any account that isn't the root and we are using one of the\nvariables we defines earlier.\u003c/p\u003e\n\u003ch3\u003eGithub\u003c/h3\u003e\n\u003cp\u003eWe will utilize \u003ccode\u003eterraform\u003c/code\u003e to create the second git repository where the rest\nof the infrastructure will go. Create a file called \u003ccode\u003e3-github.tf\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_repository\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"repo\"\u003c/span\u003e {\n name = var.github_repo_name\n description = \u003cspan class=\"hljs-string\"\u003e\"Infrastructure Repository\"\u003c/span\u003e\n visibility = \u003cspan class=\"hljs-string\"\u003e\"private\"\u003c/span\u003e\n auto_init = true\n has_issues = true\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_branch_default\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"default\"\u003c/span\u003e {\n repository = github_repository.repo.name\n branch = var.github_default_branch\n}\n\n\u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_team\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"owners\"\u003c/span\u003e {\n name = \u003cspan class=\"hljs-string\"\u003e\"owners\"\u003c/span\u003e\n organization = tfe_organization.organization.name\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_team_token\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_actions_token\"\u003c/span\u003e {\n team_id = \u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e.tfe_team.owners.id\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_actions_secret\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_secret\"\u003c/span\u003e {\n repository = github_repository.repo.name\n secret_name = \u003cspan class=\"hljs-string\"\u003e\"TFE_TOKEN\"\u003c/span\u003e\n plaintext_value = tfe_team_token.github_actions_token.token\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eoutput\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"repository_id\"\u003c/span\u003e {\n value = github_repository.repo.id\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will generate a new repository in your account called \u003ccode\u003einfra\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003eFor the terraform provider to have access to github you need to create a new\npersonal access token with full \u003ccode\u003erepo\u003c/code\u003e access and set it as an environment\nvariable named \u003ccode\u003eGITHUB_TOKEN\u003c/code\u003e.\u003c/p\u003e\n\u003ch3\u003eTerraform Cloud\u003c/h3\u003e\n\u003cp\u003eNow we need to setup dynamic credentials so the terraform cloud agent is\nallowed to take actions on your behalf. To do this we'll setup an IAM\nrole and an OIDC provider. Create a file called \u003ccode\u003e4-tfc.tf\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_organization\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"organization\"\u003c/span\u003e {\n name = var.tfc_organization_name\n email = var.tfc_organization_owner\n}\n\n/* AWS will use this TLS certificate to verify that requests for dynamic\ncredentials come from Terraform Cloud.*/\n\u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tls_certificate\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_certificate\"\u003c/span\u003e {\n url = \u003cspan class=\"hljs-string\"\u003e\"https://\u003cspan class=\"hljs-variable\"\u003e${var.tfc_hostname}\u003c/span\u003e\"\u003c/span\u003e\n}\n\n/* sets up an OIDC \u003cspan class=\"hljs-keyword\"\u003eprovider\u003c/span\u003e in AWS with Terraform Cloud's TLS certificate,\nthe SHA1 fingerprint from the TLS certificate\n*/\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_openid_connect_provider\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_provider\"\u003c/span\u003e {\n url = \u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e.tls_certificate.tfc_certificate.url\n client_id_list = [var.tfc_aws_audience]\n thumbprint_list = [\n \u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e.tls_certificate.tfc_certificate.certificates[\u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e].sha1_fingerprint\n ]\n}\n\n/* Policy to allow TFC to assume the AWS IAM role in our account */\n\u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_policy_document\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"assume_role\"\u003c/span\u003e {\n statement {\n effect = \u003cspan class=\"hljs-string\"\u003e\"Allow\"\u003c/span\u003e\n\n principals {\n type = \u003cspan class=\"hljs-string\"\u003e\"Federated\"\u003c/span\u003e\n identifiers = [aws_iam_openid_connect_provider.tfc_provider.arn]\n }\n condition {\n test = \u003cspan class=\"hljs-string\"\u003e\"StringEquals\"\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e = \u003cspan class=\"hljs-string\"\u003e\"\u003cspan class=\"hljs-variable\"\u003e${var.tfc_hostname}\u003c/span\u003e:aud\"\u003c/span\u003e\n\n values = [\n \u003cspan class=\"hljs-string\"\u003e\"\u003cspan class=\"hljs-variable\"\u003e${\u003cspan class=\"hljs-meta\"\u003eone(aws_iam_openid_connect_provider.tfc_provider.client_id_list)\u003c/span\u003e}\u003c/span\u003e\"\u003c/span\u003e\n ]\n }\n\n condition {\n test = \u003cspan class=\"hljs-string\"\u003e\"StringLike\"\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e = \u003cspan class=\"hljs-string\"\u003e\"\u003cspan class=\"hljs-variable\"\u003e${var.tfc_hostname}\u003c/span\u003e:sub\"\u003c/span\u003e\n\n values = [\n for workspace in var.tfc_workspaces : \u003cspan class=\"hljs-string\"\u003e\"organization:\u003cspan class=\"hljs-variable\"\u003e${tfe_organization.organization.name}\u003c/span\u003e:project:\u003cspan class=\"hljs-variable\"\u003e${var.tfc_project_name}\u003c/span\u003e:workspace:\u003cspan class=\"hljs-variable\"\u003e${workspace}\u003c/span\u003e:run_phase:*\"\u003c/span\u003e\n ]\n }\n actions = [\u003cspan class=\"hljs-string\"\u003e\"sts:AssumeRoleWithWebIdentity\"\u003c/span\u003e]\n }\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_role\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e {\n name = \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e\n assume_role_policy = \u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e.aws_iam_policy_document.assume_role.json\n}\n\n/* Policy for what the TFC agent is allowed to do */\n\u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_policy_document\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e {\n version = \u003cspan class=\"hljs-string\"\u003e\"2012-10-17\"\u003c/span\u003e\n\n statement {\n actions = [\u003cspan class=\"hljs-string\"\u003e\"*\"\u003c/span\u003e]\n effect = \u003cspan class=\"hljs-string\"\u003e\"Allow\"\u003c/span\u003e\n resources = [\u003cspan class=\"hljs-string\"\u003e\"*\"\u003c/span\u003e]\n }\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_policy\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e {\n name = \u003cspan class=\"hljs-string\"\u003e\"tfc-agent-access-policy\"\u003c/span\u003e\n description = \u003cspan class=\"hljs-string\"\u003e\"Access policy for the TFC agent\"\u003c/span\u003e\n policy = \u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e.aws_iam_policy_document.tfc-agent.json\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_role_policy_attachment\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-access-attach\"\u003c/span\u003e {\n role = aws_iam_role.tfc-agent.name\n policy_arn = aws_iam_policy.tfc-agent.arn\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_workspace\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"workspaces\"\u003c/span\u003e {\n count = length(var.tfc_workspaces)\n name = var.tfc_workspaces[count.index]\n organization = tfe_organization.organization.name\n\n working_directory = var.tfc_workspaces[count.index]\n}\n\n/* These variables tell the agent to use dynamic credentials */\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_variable\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-auth\"\u003c/span\u003e {\n count = length(var.tfc_workspaces)\n key = \u003cspan class=\"hljs-string\"\u003e\"TFC_AWS_PROVIDER_AUTH\"\u003c/span\u003e\n value = true\n category = \u003cspan class=\"hljs-string\"\u003e\"env\"\u003c/span\u003e\n workspace_id = tfe_workspace.workspaces[count.index].id\n description = \u003cspan class=\"hljs-string\"\u003e\"Enable dynamic auth on the TFC agents\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_variable\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-role\"\u003c/span\u003e {\n count = length(var.tfc_workspaces)\n key = \u003cspan class=\"hljs-string\"\u003e\"TFC_AWS_RUN_ROLE_ARN\"\u003c/span\u003e\n value = aws_iam_role.tfc-agent.arn\n category = \u003cspan class=\"hljs-string\"\u003e\"env\"\u003c/span\u003e\n workspace_id = tfe_workspace.workspaces[count.index].id\n description = \u003cspan class=\"hljs-string\"\u003e\"Tell TFC what Role to run as\"\u003c/span\u003e\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis module is dynamic because there is one piece that will require a\nmanual oauth setup for github. So the first pass will apply without it\nand then later on we'll create it and run the apply again.\u003c/p\u003e\n\u003ch2\u003eApplying the changes\u003c/h2\u003e\n\u003cp\u003eNow we just need to define our settings for the module and we'll get our\ninfrastructure applied. Create a file called \u003ccode\u003esettings.auto.tfvars\u003c/code\u003e and\npopulate it with the content for your account. This is an example of what\nthis should look like:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003etfc_organization_name = \u003cspan class=\"hljs-string\"\u003e\"sontek\"\u003c/span\u003e\ntfc_organization_owner = \u003cspan class=\"hljs-string\"\u003e\"john@sontek.net\"\u003c/span\u003e\n\n\u003cspan class=\"hljs-comment\"\u003e# The workspaces you want to create and be able to manage with IaC\u003c/span\u003e\ntfc_workspaces = [\n \u003cspan class=\"hljs-string\"\u003e\"root\"\u003c/span\u003e\n]\n\u003cspan class=\"hljs-comment\"\u003e# this can be your username\u003c/span\u003e\ngithub_organization = \u003cspan class=\"hljs-string\"\u003e\"sontek\"\u003c/span\u003e\ngithub_repo_name = \u003cspan class=\"hljs-string\"\u003e\"sontek-infra\"\u003c/span\u003e\naws_root_account_id = \u003cspan class=\"hljs-string\"\u003e\"888888888888\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ terraform login\n❯ terraform init\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eand you should see:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs\"\u003eTerraform has been successfully initialized!\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow lets run our plan:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e❯ \u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e plan\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou should see a result:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-vbnet\"\u003e\u003cspan class=\"hljs-symbol\"\u003ePlan:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e10\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eto\u003c/span\u003e add, \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eto\u003c/span\u003e change, \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eto\u003c/span\u003e destroy.\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eApply it to make those resources:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e❯ \u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e apply\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAt this point it:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eCreated a terraform cloud organization\u003c/li\u003e\n\u003cli\u003eCreated a terraform cloud workspace\u003c/li\u003e\n\u003cli\u003eCreated a git repository\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch1\u003eVerify TFC can talk to AWS\u003c/h1\u003e\n\u003cp\u003eTo verify that TFC can communicate with AWS through the dynamic credentials,\nlets clone the \u003cem\u003eNEW\u003c/em\u003e repository we just generated and make some dummy resources. After\nyou've cloned the repository lets make a folder for the workspace \u003ccode\u003eroot\u003c/code\u003e that we\ndefined in bootstrap:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e root\n❯ \u003cspan class=\"hljs-built_in\"\u003ecd\u003c/span\u003e root\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow create a \u003ccode\u003e1-providers.tf\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e {\n cloud {\n organization = \u003cspan class=\"hljs-string\"\u003e\"sontek\"\u003c/span\u003e\n\n workspaces {\n name = \u003cspan class=\"hljs-string\"\u003e\"root\"\u003c/span\u003e\n }\n }\n\n required_providers {\n aws = {\n source = \u003cspan class=\"hljs-string\"\u003e\"hashicorp/aws\"\u003c/span\u003e\n version = \u003cspan class=\"hljs-string\"\u003e\"4.58.0\"\u003c/span\u003e\n }\n\n tfe = {\n source = \u003cspan class=\"hljs-string\"\u003e\"hashicorp/tfe\"\u003c/span\u003e\n version = \u003cspan class=\"hljs-string\"\u003e\"0.42.0\"\u003c/span\u003e\n }\n }\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eprovider\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws\"\u003c/span\u003e {\n region = \u003cspan class=\"hljs-string\"\u003e\"us-east-1\"\u003c/span\u003e\n\n default_tags {\n tags = {\n Owner = \u003cspan class=\"hljs-string\"\u003e\"john@sontek.net\"\u003c/span\u003e\n Env = \u003cspan class=\"hljs-string\"\u003e\"Root\"\u003c/span\u003e\n Service = \u003cspan class=\"hljs-string\"\u003e\"BusinessOperations\"\u003c/span\u003e\n }\n }\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cstrong\u003eNOTE\u003c/strong\u003e: You should replace \u003ccode\u003eorganization\u003c/code\u003e, \u003ccode\u003eworkspaces.name\u003c/code\u003e, and\n\u003ccode\u003etags.Owner\u003c/code\u003e to be your own values.\u003c/p\u003e\n\u003cp\u003eNow create a small resource to prove everything is working, we'll use SQS for\nthis. Create a file called \u003ccode\u003e2-sqs.tf\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_sqs_queue\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"example-sqs\"\u003c/span\u003e {\n name = \u003cspan class=\"hljs-string\"\u003e\"example-sqs\"\u003c/span\u003e\n message_retention_seconds = \u003cspan class=\"hljs-number\"\u003e86400\u003c/span\u003e\n receive_wait_time_seconds = \u003cspan class=\"hljs-number\"\u003e10\u003c/span\u003e\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIf you run the plan you should see the resource it wants to create:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ terraform init\n❯ terraform plan\n\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eand you should see the run is executing in terraform cloud:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003eRunning plan in Terraform Cloud. Output will stream here. Pressing Ctrl-C\nwill stop streaming the logs, but will \u003cspan class=\"hljs-keyword\"\u003enot\u003c/span\u003e stop the plan running remotely.\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can click the link it provides to see the logs. Now lets apply this\nresource to see it all working:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e❯ \u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e apply\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou should get a response like:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-string\"\u003eApply\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ecomplete!\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003eResources:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eadded,\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003echanged,\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edestroyed.\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo Terraform Cloud has full access to create AWS resources! The final step\nis to get github running the plan/apply on pull requests. Commit these files\nto your repository and we'll remove them in a pull request. Create a\n\u003ccode\u003e.gitignore\u003c/code\u003e file in the root:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e.\u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e*\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eand commit all the files:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ git add *\n❯ git commit -m \u003cspan class=\"hljs-string\"\u003e\"initial infra\"\u003c/span\u003e\n❯ git push origin \u003cspan class=\"hljs-built_in\"\u003ehead\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eGithub Actions\u003c/h1\u003e\n\u003cp\u003eThe two most popular workflows when using terraform are:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eApply after Merge\u003c/strong\u003e: This is the default for things like\n\u003ca href=\"https://terraform.io\"\u003eterraform cloud\u003c/a\u003e and most github actions.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eApply before Merge\u003c/strong\u003e: This is the default for things like\n\u003ca href=\"https://www.runatlantis.io/\"\u003eAtlantis\u003c/a\u003e.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI don't like apply-after-merge. There are a lot of ways where a \u003ccode\u003eplan\u003c/code\u003e\ncan succeed but an \u003ccode\u003eapply\u003c/code\u003e will fail and you end up with broken configuration\nin \u003ccode\u003emain\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003eSo in this article I'll show you how to implement \u003cstrong\u003eapply-before-merge\u003c/strong\u003e with\ngithub actions.\u003c/p\u003e\n\u003cp\u003eAll of these changes will be in the \u003ccode\u003einfra\u003c/code\u003e repository that was generated from\n\u003ccode\u003ebootstrap\u003c/code\u003e. We are done with the bootstrap at this point.\u003c/p\u003e\n\u003cp\u003eFirst, lets setup the \u003ccode\u003e.github\u003c/code\u003e folder, the end result we want is:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e.github/\n└── workflows\n ├── on-apply-finished.yml\n ├── on-pull-request-labeled.yml\n └── on-pull-request.yml\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo create the folders:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e -p .github/workflows\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eOn Pull Request\u003c/h1\u003e\n\u003cp\u003eThe first flow we'll create is the \u003ccode\u003eterraform plan\u003c/code\u003e workflow which should be\nran whenever a pull request is opened. Create the file\n\u003ccode\u003e.github/workflows/on-pull-request.yml\u003c/code\u003e and put this content in it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yml\"\u003e\u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003epr_build\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003eon:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epull_request:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ebranches:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003emain\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003eenv:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eTERRAFORM_CLOUD_TOKENS:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eapp.terraform.io=${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.TFE_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eGITHUB_TOKEN:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.GITHUB_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003ejobs:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eterraform_validate:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eruns-on:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eubuntu-22.04\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003estrategy:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efail-fast:\u003c/span\u003e \u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematrix:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efolder:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eroot\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esteps:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eCheckout\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eactions/checkout@v3\u003c/span\u003e\n\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eterraform\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003evalidate\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edflook/terraform-validate@v1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkspace:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\n \u003cspan class=\"hljs-attr\"\u003eterraform_fmt:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eruns-on:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eubuntu-22.04\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003estrategy:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efail-fast:\u003c/span\u003e \u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematrix:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efolder:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eroot\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esteps:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eactions/checkout@v3\u003c/span\u003e\n\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eterraform\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003efmt\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edflook/terraform-fmt-check@v1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkspace:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\n \u003cspan class=\"hljs-attr\"\u003eterraform_plan:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eruns-on:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eubuntu-22.04\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epermissions:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003econtents:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eread\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epull-requests:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ewrite\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003estrategy:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efail-fast:\u003c/span\u003e \u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematrix:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efolder:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eroot\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esteps:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eactions/checkout@v3\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eterraform\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eplan\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edflook/terraform-plan@v1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkspace:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis creates three jobs:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eterraform_validate\u003c/strong\u003e: This validates the terraform via \u003ccode\u003eterraform validate\u003c/code\u003e\ncommand to make sure that it is correct and doesn't have duplicate resources\nor anything like that.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eterraform_fmt\u003c/strong\u003e: This verifies that the terraform is well formatted by\nrunning the \u003ccode\u003eterraform fmt\u003c/code\u003e command.`\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eterraform_plan\u003c/strong\u003e: This runs the \u003ccode\u003eterraform\u003c/code\u003e plan and comments on the PR a\ndiff of the changes for you to verify.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eTo verify this is working, lets delete \u003ccode\u003eroot/2-sqs.tf\u003c/code\u003e, then lets push a branch\nand make a pull request to see the result so far:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003erm\u003c/span\u003e root/2-sqs.tf\n❯ git add .github/ root/\n❯ git checkout -b apply-before-merge\n❯ git commit -m \u003cspan class=\"hljs-string\"\u003e\"Implemented on-pull-request\"\u003c/span\u003e\n❯ git push origin \u003cspan class=\"hljs-built_in\"\u003ehead\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAfter you make the pull request you should 3 checks on it and a comment that\nshows the plan:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/github_comment.png\" width=\"400\"\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/github_checks.png\" width=\"400\"\u003e\n\u003c/center\u003e\n\u003ch1\u003eApply on Label\u003c/h1\u003e\n\u003cp\u003eSo now that the plan is working we need some way to \u003ccode\u003eapply\u003c/code\u003e the changes. I've\nfound the best way to do this is via a label rather than a comment because of\nthe way github actions work. Their event based actions like \u003ccode\u003eon-comment\u003c/code\u003e aren't\nexecuted in the context of a pull-request.\u003c/p\u003e\n\u003cp\u003eSince we will be using a label to signal a plan is ready to be applied lets\ncreate a new file \u003ccode\u003e.github/workflows/on-pull-request-labeled.yml\u003c/code\u003e and provide\nthis content:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003epr_apply\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003eon:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epull_request:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003etypes:\u003c/span\u003e [ \u003cspan class=\"hljs-string\"\u003elabeled\u003c/span\u003e ]\n\n\u003cspan class=\"hljs-attr\"\u003eenv:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eTERRAFORM_CLOUD_TOKENS:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eapp.terraform.io=${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.TFE_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eGITHUB_TOKEN:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.GITHUB_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003ejobs:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eterraform_apply:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eif:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003egithub.event.label.name\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e==\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'tfc-apply'\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eruns-on:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eubuntu-22.04\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epermissions:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003econtents:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eread\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epull-requests:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ewrite\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003estrategy:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efail-fast:\u003c/span\u003e \u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematrix:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efolder:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eroot\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esteps:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eactions/checkout@v3\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edflook/terraform-apply@v1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkspace:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will fire whenever a pull request is labeled with the \u003ccode\u003etfc-apply\u003c/code\u003e label.\nYou will need to create this label for the repository.\u003c/p\u003e\n\u003cp\u003eIt will run the \u003ccode\u003eapply\u003c/code\u003e and update the previous plan comment to let you\nknow the status.\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/tfc_applying.png\" width=\"400\"\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/tfc_applying_comment.png\" width=\"400\"\u003e\n\u003c/center\u003e\n\u003ch1\u003eMerge on Apply\u003c/h1\u003e\n\u003cp\u003eOne thing you'll notice is that the pull request stayed open even after the\ninfrastructure is applied and we don't want that. We want any changes that have\nmade it into the environment to be merged into \u003ccode\u003emain\u003c/code\u003e automatically. To do\nthis we'll create our final action.\u003c/p\u003e\n\u003cp\u003eCreate a new file \u003ccode\u003e.github/workflows/on-apply-finished.yml\u003c/code\u003e with this content:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003epr_merge\u003c/span\u003e\n\n\u003cspan class=\"hljs-comment\"\u003e# Only trigger, when the build workflow succeeded\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eon:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkflow_run:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkflows:\u003c/span\u003e [\u003cspan class=\"hljs-string\"\u003epr_apply\u003c/span\u003e]\n \u003cspan class=\"hljs-attr\"\u003etypes:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ecompleted\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003ejobs:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003emerge:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eif:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003egithub.event.workflow_run.conclusion\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e==\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'success'\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eruns-on:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eubuntu-22.04\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epermissions:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003econtents:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ewrite\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epull-requests:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ewrite\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003echecks:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eread\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003estatuses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eread\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eactions:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eread\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eoutputs:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epullRequestNumber:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esteps.workflow-run-info.outputs.pullRequestNumber\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esteps:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"Get information about the current run\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003epotiuk/get-workflow-origin@v1_5\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eid:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eworkflow-run-info\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003etoken:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.GITHUB_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esourceRunId:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003egithub.event.workflow_run.id\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003emerge\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ea\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003epull\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003erequest\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eafter\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eterraform\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eapply\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esudo-bot/action-pull-request-merge@v1.2.0\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003egithub-token:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.GITHUB_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003enumber:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esteps.workflow-run-info.outputs.pullRequestNumber\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will wait until the \u003ccode\u003epr_apply\u003c/code\u003e job completes and as long as it was\nsuccessful it'll merge the branch!\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNOTE\u003c/strong\u003e: As I mentioned earlier, the event based actions do not run in the\ncontext of the pull request which means you cannot test changes to them during\nthe PR either. You must merge the \u003ccode\u003eon-apply-finished.yml\u003c/code\u003e file to \u003ccode\u003emain\u003c/code\u003e\nbefore it starts working.\u003c/p\u003e\n\u003ch1\u003eBranch Protection\u003c/h1\u003e\n\u003cp\u003eThe final step to the process is to make sure you go to your github settings\nand make sure these status checks are required before merging. Branch protection\nis a feature that will prevent merging changes into a branch unless all\nrequired checks are passing.\u003c/p\u003e\n\u003cp\u003eGo to \u003ccode\u003eSettings\u003c/code\u003e -\u003e \u003ccode\u003eBranches\u003c/code\u003e -\u003e \u003ccode\u003eBranch Protection\u003c/code\u003e and add a branch\nprotection rule:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/branch_protection.png\" width=\"500\"\u003e\n\u003c/center\u003e\n\u003cp\u003eYou want to enable the following settings:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eBranch Name\u003c/strong\u003e: main\u003c/li\u003e\n\u003cli\u003e✅ Require a pull request before merging\u003c/li\u003e\n\u003cli\u003e✅ Require status checks to pass before merging\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThen for \u003ccode\u003eStatus checks that are required.\u003c/code\u003e select all of the ones we've\ncreated:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/required_checks.png\" height=\"200\"\u003e\n\u003c/center\u003e\n\u003ch1\u003eHelpful Resources\u003c/h1\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://developer.hashicorp.com/terraform/tutorials/cloud/dynamic-credentials?product_intent=terraform\"\u003eTerraform Dynamic Credentials Tutorial\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://developer.hashicorp.com/terraform/cloud-docs/workspaces/dynamic-provider-credentials/aws-configuration\"\u003eTerraform docs on Dynamic Credentials\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#understanding-the-oidc-token\"\u003eGithub's understanding OIDC\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e","category":"AWS","date":"2023-06-23T00:00:00Z","tags":["AWS","DevOps","SRE"],"title":"AWS From Scratch with Terraform - Setting up your Root Account for IaC with Terraform Cloud and Github actions."},{"id":["2023","local_k8s_with_kind"],"path":"2023/local_k8s_with_kind","contentHtml":"\u003cp\u003ePreviously I \u003ca href=\"/blog/2022/local_kubeadm_cluster\"\u003eshowed\u003c/a\u003e how to run kubernetes\nlocally with \u003ccode\u003ekubeadm\u003c/code\u003e and VMs but sometimes that is overkill so I wanted to\nshow how to run \u003ca href=\"https://kind.sigs.k8s.io/\"\u003ekind\u003c/a\u003e which is \"kuberetes in\ndocker\".\u003c/p\u003e\n\u003ch1\u003eCreating your first cluster\u003c/h1\u003e\n\u003cp\u003ekind is a very flexible way to run kubernetes locally and allows you to run\nsingle node or multinode clusters while having the flexibility to use all\nthe features of kubernetes success as ingress.\u003c/p\u003e\n\u003cp\u003eTo create your first cluster it is as simple as running:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kind create cluster \n\nCreating cluster \u003cspan class=\"hljs-string\"\u003e\"kind\"\u003c/span\u003e ...\n ✓ Ensuring node image (kindest/node:v1.27.3) 🖼 \n ✓ Preparing nodes 📦 \n ✓ Writing configuration 📜 \n ✓ Starting control-plane 🕹️ \n ✓ Installing CNI 🔌 \n ✓ Installing StorageClass 💾 \nSet kubectl context to \u003cspan class=\"hljs-string\"\u003e\"kind-kind\"\u003c/span\u003e\nYou can now use your cluster with:\n\nkubectl cluster-info --context kind-kind\n\nHave a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/\u003cspan class=\"hljs-comment\"\u003e#community 🙂\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou now have a functioning kubernetes cluster and you\ncan view what it created:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ k get node\nNAME STATUS ROLES AGE VERSION\nkind-control-plane Ready control-plane 4m26s v1.27.3\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can also verify that it is running inside docker:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n1c3ba74dc29b kindest/node:v1.27.3 \u003cspan class=\"hljs-string\"\u003e\"/usr/local/bin/entr…\"\u003c/span\u003e 3 minutes ago Up 3 minutes 127.0.0.1:59327-\u003e6443/tcp kind-control-plane\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eMaking the cluster useful\u003c/h1\u003e\n\u003cp\u003eThere are a few things you'll notice with the command we ran originally:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eIt grabbed the latest kubernetes version available\u003c/li\u003e\n\u003cli\u003eIt is running a single node cluster\u003c/li\u003e\n\u003cli\u003eNo ingress available\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eLuckily kind makes it really easy to customize your local cluster to be what\nyou want it to be by using a \u003ccode\u003eYAML\u003c/code\u003e configuration.\u003c/p\u003e\n\u003cp\u003eCreate the configuration:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eCluster\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ekind.x-k8s.io/v1alpha4\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003enodes:\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003econtrol-plane\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ekubeadmConfigPatches:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e|\n kind: InitConfiguration\n nodeRegistration:\n kubeletExtraArgs:\n node-labels: \"ingress-ready=true\"\n\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003eextraPortMappings:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003econtainerPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e80\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ehostPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e80\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eprotocol:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eTCP\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003econtainerPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e443\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ehostPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e443\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eprotocol:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eTCP\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eworker\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eworker\u003c/span\u003e\n\u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003erole:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eworker\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWith this we've now generated a 4 node cluster where we have a single\ncontrol-plane and three workers. Then we defined some extra configuration on\nthe control-plane:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ekubeadmConfigPatches\u003c/strong\u003e: We want to change the default configuration the\ncluster uses so it'll tag the nodes with the \u003ccode\u003eingress-ready\u003c/code\u003e label so the\ncontroller will use them.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eextraPortMappings\u003c/strong\u003e: allow the local host to make requests to the Ingress controller over ports 80/443\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003enode-labels\u003c/strong\u003e: only allow the ingress controller to run on specific node(s) matching the label selector\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSo now we can create the new cluster with the configuration. Save that config\nas \u003ccode\u003ekind_config.yml\u003c/code\u003e and then run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kind create cluster --image kindest/node:v1.25.11 --config kind_config.yml --name kind-multinode\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis time I've added a few additional flags on the commandline. \u003ccode\u003e--image\u003c/code\u003e\nallows us to use a different version of kubernetes and \u003ccode\u003e--name\u003c/code\u003e allows us to\nmake more than one cluster. So if you didn't destroy the first cluster you'll\nsee we have two of them now:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kind get clusters\nkind\nkind-multinode\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003ebut \u003ccode\u003ekind\u003c/code\u003e will swap the to the newest cluster by default:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl config current-context\nkind-kind-multinode\n\n❯ kubectl get node\nNAME STATUS ROLES AGE VERSION\nkind-multinode-control-plane Ready control-plane 107s v1.25.11\nkind-multinode-worker Ready \u0026#x3C;none\u003e 88s v1.25.11\nkind-multinode-worker2 Ready \u0026#x3C;none\u003e 88s v1.25.11\nkind-multinode-worker3 Ready \u0026#x3C;none\u003e 88s v1.25.11\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we need to get the \u003ccode\u003eingress-nginx\u003c/code\u003e controller installed so we can start\nusing our cluster with ingress:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe manifests contains \u003ccode\u003ekind\u003c/code\u003e specific patches to forward the hostPorts to the ingress controller, set taint tolerations and schedule it to the custom labelled node.\u003c/p\u003e\n\u003cp\u003eThis will take a little bit of time to get up and running, you can monitor it\nby running:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003ekubectl \u003cspan class=\"hljs-built_in\"\u003ewait\u003c/span\u003e --namespace ingress-nginx \\\n --\u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e=condition=ready pod \\\n --selector=app.kubernetes.io/component=controller \\\n --\u003cspan class=\"hljs-built_in\"\u003etimeout\u003c/span\u003e=90s\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eor just manually check the status:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get all -n ingress-nginx\nNAME READY STATUS RESTARTS AGE\npod/ingress-nginx-admission-create-bbmlc 0/1 Completed 0 68s\npod/ingress-nginx-admission-patch-qlnr8 0/1 Completed 2 68s\npod/ingress-nginx-controller-5f748f78c8-6tc6b 0/1 ContainerCreating 0 68s\n\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nservice/ingress-nginx-controller NodePort 10.96.228.248 \u0026#x3C;none\u003e 80:31771/TCP,443:31759/TCP 68s\nservice/ingress-nginx-controller-admission ClusterIP 10.96.180.126 \u0026#x3C;none\u003e 443/TCP 68s\n\nNAME READY UP-TO-DATE AVAILABLE AGE\ndeployment.apps/ingress-nginx-controller 0/1 1 0 68s\n\nNAME DESIRED CURRENT READY AGE\nreplicaset.apps/ingress-nginx-controller-5f748f78c8 1 1 0 68s\n\nNAME COMPLETIONS DURATION AGE\njob.batch/ingress-nginx-admission-create 1/1 22s 68s\njob.batch/ingress-nginx-admission-patch 1/1 35s 68s\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce \u003ccode\u003eingress-nginx-controller\u003c/code\u003e is in \u003ccode\u003eRunning\u003c/code\u003e state you are read to go!\u003c/p\u003e\n\u003ch1\u003eDeploying your first app\u003c/h1\u003e\n\u003cp\u003eTo prove that the cluster is working correctly we will deploy\n\u003ca href=\"https://github.com/Kong/httpbin\"\u003ehttpbin\u003c/a\u003e which is a nice little API server\nso we can prove everything is working.\u003c/p\u003e\n\u003cp\u003eCreate a \u003ccode\u003ehttbin.yml\u003c/code\u003e file and paste this into it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-meta\"\u003e---\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ev1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eService\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003elabels:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eservice:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eports:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttp\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eport:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8000\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003etargetPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8080\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eselector:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n\u003cspan class=\"hljs-meta\"\u003e---\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eapps/v1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eDeployment\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ereplicas:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eselector:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematchLabels:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eversion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ev1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003etemplate:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003elabels:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eapp:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eversion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ev1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003econtainers:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003eimage:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edocker.io/mccutchen/go-httpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eimagePullPolicy:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eIfNotPresent\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eports:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003econtainerPort:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8080\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis is creating a couple of Kubernetes resources:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003eService\u003c/code\u003e: This is exposing the port to the ingress\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eDeployment\u003c/code\u003e: This is actually launching the service\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSo we are not using the ingress yet but we can prove that we can launch the\nservice at least. So apply those manifests:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl apply -f httpbin.yml \nservice/httpbin created\ndeployment.apps/httpbin created\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou should see two pods come up. You should wait for them to get into the\n\u003ccode\u003eRunning\u003c/code\u003e status:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl get pod -o wide\nNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES\nhttpbin-5c5494967-2z5wz 1/1 Running 0 48s 10.244.3.3 kind-multinode-worker3 \u0026#x3C;none\u003e \u0026#x3C;none\u003e\nhttpbin-5c5494967-9lf47 1/1 Running 0 72s 10.244.1.2 kind-multinode-worker \u0026#x3C;none\u003e \u0026#x3C;none\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe can now use port forwarding to access it. \u003ccode\u003ehttpbin\u003c/code\u003e is exposed on \u003ccode\u003e8000\u003c/code\u003e so\nlets create port \u003ccode\u003e9000\u003c/code\u003e on our host that forwards to it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ kubectl port-forward service/httpbin 9000:8000\nForwarding from 127.0.0.1:9000 -\u003e 80\nForwarding from [::1]:9000 -\u003e 80\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can access it via:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl localhost:9000/get \n{\n \u003cspan class=\"hljs-string\"\u003e\"args\"\u003c/span\u003e: {},\n \u003cspan class=\"hljs-string\"\u003e\"headers\"\u003c/span\u003e: {\n \u003cspan class=\"hljs-string\"\u003e\"Accept\"\u003c/span\u003e: [\n \u003cspan class=\"hljs-string\"\u003e\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8\"\u003c/span\u003e\n ],\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eUsing Ingress\u003c/h1\u003e\n\u003cp\u003eNow to use the ingress rather than port forwarding we create one additional\nresource, the \u003ccode\u003eIngress\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003eapiVersion:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003enetworking.k8s.io/v1\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003ekind:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eIngress\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003emetadata:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin-ingress\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eannotations:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003enginx.ingress.kubernetes.io/rewrite-target:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e/$2\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003espec:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eingressClassName:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003enginx\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003erules:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ehttp:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epaths:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e/httpbin(/|$)(.*)\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epathType:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eImplementationSpecific\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ebackend:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eservice:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ehttpbin\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eport:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003enumber:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e8000\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThere are a few critical options here. The first is the annotation to rewrite\nthe path so it doesn't include \u003ccode\u003e/httpbin/\u003c/code\u003e when it sends the request to the\nservice and then the \u003ccode\u003epath\u003c/code\u003e and \u003ccode\u003epathType\u003c/code\u003e so it knows which paths to send to\nwhich service.\u003c/p\u003e\n\u003cp\u003eNow you should be able to hit your local host and get routed to your\nkubernetes service:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ curl localhost/httpbin/get\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSuccess! Now you have a multinode kubernetes cluster that has an ingress\ncontroller!\u003c/p\u003e\n\u003ch1\u003eNext Steps\u003c/h1\u003e\n\u003cp\u003eThe cluster can be used like a production cluster now for local\ndevelopment! You could setup Grafana, ArgoCD, etc. to run\ninside the cluster.\u003c/p\u003e","category":"Kubernetes","date":"2023-07-21T00:00:00Z","tags":["Linux","Kubernetes","DevOps","SRE"],"title":"Running a kubernetes cluster locally with kind"},{"id":["2023","nuking_aws_account"],"path":"2023/nuking_aws_account","contentHtml":"\u003cp\u003eWhen you're an SRE/DevOps engineer you'll end up making AWS accounts and\ncreate a lot of cruft in your sandbox and development accounts. AWS\ndoes not make it easy to clear these up but there is a tool called\n\u003ca href=\"https://github.com/rebuy-de/aws-nuke\"\u003eaws-nuke\u003c/a\u003e that will do it for you!\u003c/p\u003e\n\u003ch1\u003eSafe Guards\u003c/h1\u003e\n\u003cp\u003eaws-nuke has a few safeguards in place to prevent inadvertent data loss.\nThe first of which is it requires you to alias the targetted account. I\nlike to put \u003ccode\u003enuke\u003c/code\u003e in the alias to make it clear.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eaws iam create-account-alias --account-alias nuke-\u0026#x3C;account\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe second safe-guard is the config takes a key called \u003ccode\u003eaccount-blocklist\u003c/code\u003e\nthat will guarantee nuke will not run against it no matter what.\u003c/p\u003e\n\u003cp\u003eThe final safety switch it has is it will not take any action by default,\nit will only execute a dry-run. You need to run the CLI with\n\u003ccode\u003e--no-dry-run\u003c/code\u003e if you want it to take action.\u003c/p\u003e\n\u003ch1\u003eGetting Started\u003c/h1\u003e\n\u003cp\u003eYou configure \u003ccode\u003eaws-nuke\u003c/code\u003e with YAML, so the first step is to define that:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-makefile\"\u003e\u003cspan class=\"hljs-section\"\u003eregions:\u003c/span\u003e\n - us-east-1\n - global\n\n\u003cspan class=\"hljs-section\"\u003eaccount-blocklist:\u003c/span\u003e\n - \u003cspan class=\"hljs-string\"\u003e\"888888888888\"\u003c/span\u003e \u003cspan class=\"hljs-comment\"\u003e# production\u003c/span\u003e\n\n\u003cspan class=\"hljs-section\"\u003eaccounts:\u003c/span\u003e\n \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e: {} \u003cspan class=\"hljs-comment\"\u003e# nuke-\u0026#x3C;account\u003e\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will prevent us from nuking our production account and target all resources\nin the account we actually want to nuke.\u003c/p\u003e\n\u003cp\u003eYou might want to have it nuke \u003cem\u003eALL REGIONS\u003c/em\u003e in AWS since you may not know which\nregions resources are deployed in. To do this you can query the regions from AWS:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ aws ec2 describe-regions --all-regions --query \u003cspan class=\"hljs-string\"\u003e\"Regions[*].RegionName\"\u003c/span\u003e --output text | xargs -n 1 | \u003cspan class=\"hljs-built_in\"\u003esort\u003c/span\u003e\n\naf-south-1\nap-east-1\nap-northeast-1\nap-northeast-2\nap-northeast-3\nap-south-1\nap-south-2\nap-southeast-1\nap-southeast-2\nap-southeast-3\nap-southeast-4\nca-central-1\neu-central-1\neu-central-2\neu-north-1\neu-south-1\neu-south-2\neu-west-1\neu-west-2\neu-west-3\nme-central-1\nme-south-1\nsa-east-1\nus-east-1\nus-east-2\nus-west-1\nus-west-2\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWhich would give you an updated config of:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003eregions:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eglobal\u003c/span\u003e \u003cspan class=\"hljs-comment\"\u003e# Global resources like IAM\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eaf-south-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-east-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-northeast-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-northeast-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-northeast-3\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-south-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-south-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-southeast-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-southeast-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-southeast-3\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-southeast-4\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eca-central-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-central-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-central-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-north-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-south-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-south-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-west-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-west-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-west-3\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eme-central-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eme-south-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esa-east-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eus-east-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eus-east-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eus-west-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eus-west-2\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eI personally don't recommend targetting all AWS regions at the same time. It\nwill generate a lot of output and be slow. You could do it if necessary but\nmost people only have a few regions they use and so they can set those directly.\nFor example it, maybe you only use \u003ccode\u003eus-\u003c/code\u003e based regions?\u003c/p\u003e\n\u003cp\u003eSo lets run the dry-run and see what it wants to nuke:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ aws-nuke -c nuke.yaml\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis should output something like:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eDo you really want to nuke the account with the ID 777777777777 and the \u003cspan class=\"hljs-built_in\"\u003ealias\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'nuke-sandbox'\u003c/span\u003e?\nDo you want to \u003cspan class=\"hljs-built_in\"\u003econtinue\u003c/span\u003e? Enter account \u003cspan class=\"hljs-built_in\"\u003ealias\u003c/span\u003e to \u003cspan class=\"hljs-built_in\"\u003econtinue\u003c/span\u003e.\n\u003e nuke-sandbox\nus-east-1 - EC2Subnet - subnet-0cd9975a443a6304b - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2Subnet - subnet-0be39d02e399a371c - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2Subnet - subnet-02d7017bd4730ea63 - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2Subnet - subnet-0ec04b28c32708ab2 - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2Subnet - subnet-0eea1b4be084840ed - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2Subnet - subnet-05a294cc04736012e - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2RouteTable - rtb-0abda0e94015064ca - [DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e] - would remove\nus-east-1 - EC2DefaultSecurityGroupRule - sgr-0368525f77bf566ac - [SecurityGroupId: \u003cspan class=\"hljs-string\"\u003e\"sg-0a59900b52ced5e10\"\u003c/span\u003e] - would remove\nus-east-1 - EC2DefaultSecurityGroupRule - sgr-0890a837ed6148729 - [SecurityGroupId: \u003cspan class=\"hljs-string\"\u003e\"sg-0a59900b52ced5e10\"\u003c/span\u003e] - would remove\nus-east-1 - EC2InternetGatewayAttachment - igw-0acfb474f1fd71375 -\u003e vpc-0be5d310ab44c239a - [DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e] - would remove\nus-east-1 - SQSQueue - https://sqs.us-east-1.amazonaws.com/777777777777/example-sqs - would remove\nglobal - IAMSAMLProvider - arn:aws:iam::777777777777:saml-provider/AWSSSO_254abb4071f10b25_DO_NOT_DELETE - would remove\nglobal - IAMOpenIDConnectProvider - arn:aws:iam::777777777777:oidc-provider/app.terraform.io - [Arn: \u003cspan class=\"hljs-string\"\u003e\"arn:aws:iam::777777777777:oidc-provider/app.terraform.io\"\u003c/span\u003e] - would remove\nglobal - IAMPolicy - arn:aws:iam::777777777777:policy/tfc-agent-access-policy - [ARN: \u003cspan class=\"hljs-string\"\u003e\"arn:aws:iam::777777777777:policy/tfc-agent-access-policy\"\u003c/span\u003e, Name: \u003cspan class=\"hljs-string\"\u003e\"tfc-agent-access-policy\"\u003c/span\u003e, Path: \u003cspan class=\"hljs-string\"\u003e\"/\"\u003c/span\u003e, PolicyID: \u003cspan class=\"hljs-string\"\u003e\"ANPA2T6PZOBNWI76TKQRF\"\u003c/span\u003e] - would remove\nglobal - IAMRole - tfc-agent - [CreateDate: \u003cspan class=\"hljs-string\"\u003e\"2023-04-02T17:55:23Z\"\u003c/span\u003e, LastUsedDate: \u003cspan class=\"hljs-string\"\u003e\"2023-06-22T13:45:02Z\"\u003c/span\u003e, Name: \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e, Path: \u003cspan class=\"hljs-string\"\u003e\"/\"\u003c/span\u003e] - would remove\nglobal - IAMRolePolicyAttachment - tfc-agent -\u003e tfc-agent-access-policy - [PolicyArn: \u003cspan class=\"hljs-string\"\u003e\"arn:aws:iam::777777777777:policy/tfc-agent-access-policy\"\u003c/span\u003e, PolicyName: \u003cspan class=\"hljs-string\"\u003e\"tfc-agent-access-policy\"\u003c/span\u003e, RoleCreateDate: \u003cspan class=\"hljs-string\"\u003e\"2023-04-02T17:55:23Z\"\u003c/span\u003e, RoleLastUsed: \u003cspan class=\"hljs-string\"\u003e\"2023-06-22T13:45:02Z\"\u003c/span\u003e, RoleName: \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e, RolePath: \u003cspan class=\"hljs-string\"\u003e\"/\"\u003c/span\u003e] - would remove\nScan complete: 85 total, 19 nukeable, 66 filtered.\n\nThe above resources would be deleted with the supplied configuration. Provide --no-dry-run to actually destroy resources.\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis is great, it fully scanned the account and found every resource to delete!\nIt even wants to delete the DefaultVPC which is usually a good idea. The one\nresource that should catch your eye that you probably do not want to delete:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003earn:aws:iam::777777777777:saml-provider/AWSSSO_254abb4071f10b25_DO_NOT_DELETE\u003c/code\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAWS clearly doesn't want us to delete that!\u003c/p\u003e\n\u003ch1\u003eFilters\u003c/h1\u003e\n\u003cp\u003eTo prevent nuke from deleting resources you want to keep you can define presets\nthat you use on each account. So with our SSO example we want to prevent it\nfrom deleting those resources in a preset.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003epresets:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esso:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efilters:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMSAMLProvider:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"regex\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"AWSSSO_.*_DO_NOT_DELETE\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRole:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"glob\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"AWSReservedSSO_*\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRolePolicyAttachment:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"glob\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"AWSReservedSSO_*\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can see in this example I'm targetting specific resource types and then\nmatching them with both \u003ccode\u003eregex\u003c/code\u003e and \u003ccode\u003eglob\u003c/code\u003e filter types. These are super\npowerful but a lot of times the simpler filters can be used. I start with\n\u003ccode\u003econtains\u003c/code\u003e filter and then go from there:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003econtains\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eAWSReservedSSO\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen the other thing you may have noticed is that I was repeating\n\u003ccode\u003eAWSReservedSSO\u003c/code\u003e multiple times. To reduce that you can use standard YAML\nanchors. So the final config for your preset would look like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003epresets:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esso:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efilters:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e_DEFAULT_FILTERS:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e\u0026#x26;DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"contains\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"DO_NOT_DELETE\"\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"contains\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"AWSReservedSSO\"\u003c/span\u003e\n\n \u003cspan class=\"hljs-attr\"\u003eIAMSAMLProvider:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRole:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRolePolicyAttachment:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we can use that preset in our accounts configuration:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003eaccounts:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"777777777777\":\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"presets\":\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esso\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo your final config should look something like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003eregions:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eus-east-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eglobal\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003eaccount-blocklist:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"888888888888\"\u003c/span\u003e \u003cspan class=\"hljs-comment\"\u003e# production\u003c/span\u003e\n\n\n\u003cspan class=\"hljs-attr\"\u003epresets:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esso:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efilters:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e_DEFAULT_FILTERS:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e\u0026#x26;DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"contains\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"DO_NOT_DELETE\"\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"contains\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"AWSReservedSSO\"\u003c/span\u003e\n\n \u003cspan class=\"hljs-attr\"\u003eIAMSAMLProvider:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRole:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRolePolicyAttachment:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003eaccounts:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"777777777777\":\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"presets\":\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esso\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWhen you run this you should see now the resources we want to keep are filtered\nout:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eglobal - IAMSAMLProvider - arn:aws:iam::777777777777:saml-provider/AWSSSO_254abb4071f10b25_DO_NOT_DELETE - filtered by config\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce you are ready and have your filters in place you can run it for real!\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ aws-nuke -c nuke.yaml --no-dry-run\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eNext steps\u003c/h1\u003e\n\u003cp\u003eOne final note about it is that it does not understand the relationship between\nresources and so it could try deleting an EBS volume that is still in use by an\nEC2 instance. There isn't a great solution for this outside of running nuke a\nfew times.\u003c/p\u003e\n\u003cp\u003eThe tool is well documented and so you can find the rest of information going to\n\u003ca href=\"https://github.com/rebuy-de/aws-nuke\"\u003ehttps://github.com/rebuy-de/aws-nuke\u003c/a\u003e!\u003c/p\u003e","category":"AWS","date":"2023-06-22T00:00:00Z","tags":["AWS","DevOps","SRE"],"title":"Wiping an AWS Account with aws-nuke"}]},"AWS":{"count":2,"posts":[{"id":["2023","aws_from_scratch_apply_before_merge"],"path":"2023/aws_from_scratch_apply_before_merge","contentHtml":"\u003cp\u003eFollowing this article will get you setup with an AWS Root account that can be\nmanaged through through Terraform Cloud with OIDC and github actions. As a best practice you\nshould not keep long-lived access keys in your CI/CD pipelines when\ndeploying to AWS, instead you should use OIDC (OpenID Connect) to securely\ndeploy to AWS when using Terraform Cloud or Github Actions.\u003c/p\u003e\n\u003ciframe width=\"854\" height=\"480\" src=\"https://www.youtube.com/embed/3oZd1m8_KIo\" title=\"AWS From Scratch - Preparing your account to be managed by IaC via Terraform and Github Actions\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture;web-share\" allowfullscreen\u003e\u003c/iframe\u003e\n\u003ch1\u003eTL;DR\u003c/h1\u003e\n\u003cp\u003eDownload all the terraform from the blog post here:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/sontek/aws-apply-before-merge\"\u003ehttps://github.com/sontek/aws-apply-before-merge\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/sontek/aws-terraform-github-actions\"\u003ehttps://github.com/sontek/aws-terraform-github-actions\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch1\u003eHow does OIDC work\u003c/h1\u003e\n\u003cp\u003eOIDC enables us to request a short-lived access token directly from AWS. We\njust have to create trust relationship that controls which workflows are able\nto request the access tokens.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eNo need to duplicate AWS credentials as long-lived GitHub secrets.\u003c/li\u003e\n\u003cli\u003eSince we are using a short-lived access token that is only valid for a single\njob there is no reason to worry about rotating secrets.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe following diagram gives an overview of how we can use Terraform Cloud's\nOIDC provider to integrate with AWS:\u003c/p\u003e\n\u003cdiv class=\"remark-mermaid remark-mermaid-default\"\u003e\u003csvg aria-roledescription=\"flowchart-v2\" role=\"graphics-document document\" viewBox=\"-8 -8 843.078125 320\" style=\"max-width: 843.078px; background-color: transparent;\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xmlns=\"http://www.w3.org/2000/svg\" width=\"100%\" id=\"mermaid-1691159159024\"\u003e\u003cstyle\u003e#mermaid-1691159159024{font-family:\"trebuchet ms\",verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-1691159159024 .error-icon{fill:#552222;}#mermaid-1691159159024 .error-text{fill:#552222;stroke:#552222;}#mermaid-1691159159024 .edge-thickness-normal{stroke-width:2px;}#mermaid-1691159159024 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-1691159159024 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-1691159159024 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-1691159159024 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-1691159159024 .marker{fill:#333333;stroke:#333333;}#mermaid-1691159159024 .marker.cross{stroke:#333333;}#mermaid-1691159159024 svg{font-family:\"trebuchet ms\",verdana,arial,sans-serif;font-size:16px;}#mermaid-1691159159024 .label{font-family:\"trebuchet ms\",verdana,arial,sans-serif;color:#333;}#mermaid-1691159159024 .cluster-label text{fill:#333;}#mermaid-1691159159024 .cluster-label span{color:#333;}#mermaid-1691159159024 .label text,#mermaid-1691159159024 span{fill:#333;color:#333;}#mermaid-1691159159024 .node rect,#mermaid-1691159159024 .node circle,#mermaid-1691159159024 .node ellipse,#mermaid-1691159159024 .node polygon,#mermaid-1691159159024 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-1691159159024 .node .label{text-align:center;}#mermaid-1691159159024 .node.clickable{cursor:pointer;}#mermaid-1691159159024 .arrowheadPath{fill:#333333;}#mermaid-1691159159024 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-1691159159024 .flowchart-link{stroke:#333333;fill:none;}#mermaid-1691159159024 .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-1691159159024 .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-1691159159024 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-1691159159024 .cluster text{fill:#333;}#mermaid-1691159159024 .cluster span{color:#333;}#mermaid-1691159159024 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\"trebuchet ms\",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-1691159159024 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-1691159159024 :root{--mermaid-font-family:\"trebuchet ms\",verdana,arial,sans-serif;}\u003c/style\u003e\u003cg\u003e\u003cmarker orient=\"auto\" markerHeight=\"12\" markerWidth=\"12\" markerUnits=\"userSpaceOnUse\" refY=\"5\" refX=\"10\" viewBox=\"0 0 12 20\" class=\"marker flowchart\" id=\"flowchart-pointEnd\"\u003e\u003cpath style=\"stroke-width: 1; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" d=\"M 0 0 L 10 5 L 0 10 z\"\u003e\u003c/path\u003e\u003c/marker\u003e\u003cmarker orient=\"auto\" markerHeight=\"12\" markerWidth=\"12\" markerUnits=\"userSpaceOnUse\" refY=\"5\" refX=\"0\" viewBox=\"0 0 10 10\" class=\"marker flowchart\" id=\"flowchart-pointStart\"\u003e\u003cpath style=\"stroke-width: 1; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" d=\"M 0 5 L 10 10 L 10 0 z\"\u003e\u003c/path\u003e\u003c/marker\u003e\u003cmarker orient=\"auto\" markerHeight=\"11\" markerWidth=\"11\" markerUnits=\"userSpaceOnUse\" refY=\"5\" refX=\"11\" viewBox=\"0 0 10 10\" class=\"marker flowchart\" id=\"flowchart-circleEnd\"\u003e\u003ccircle style=\"stroke-width: 1; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" r=\"5\" cy=\"5\" cx=\"5\"\u003e\u003c/circle\u003e\u003c/marker\u003e\u003cmarker orient=\"auto\" markerHeight=\"11\" markerWidth=\"11\" markerUnits=\"userSpaceOnUse\" refY=\"5\" refX=\"-1\" viewBox=\"0 0 10 10\" class=\"marker flowchart\" id=\"flowchart-circleStart\"\u003e\u003ccircle style=\"stroke-width: 1; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" r=\"5\" cy=\"5\" cx=\"5\"\u003e\u003c/circle\u003e\u003c/marker\u003e\u003cmarker orient=\"auto\" markerHeight=\"11\" markerWidth=\"11\" markerUnits=\"userSpaceOnUse\" refY=\"5.2\" refX=\"12\" viewBox=\"0 0 11 11\" class=\"marker cross flowchart\" id=\"flowchart-crossEnd\"\u003e\u003cpath style=\"stroke-width: 2; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" d=\"M 1,1 l 9,9 M 10,1 l -9,9\"\u003e\u003c/path\u003e\u003c/marker\u003e\u003cmarker orient=\"auto\" markerHeight=\"11\" markerWidth=\"11\" markerUnits=\"userSpaceOnUse\" refY=\"5.2\" refX=\"-1\" viewBox=\"0 0 11 11\" class=\"marker cross flowchart\" id=\"flowchart-crossStart\"\u003e\u003cpath style=\"stroke-width: 2; stroke-dasharray: 1, 0;\" class=\"arrowMarkerPath\" d=\"M 1,1 l 9,9 M 10,1 l -9,9\"\u003e\u003c/path\u003e\u003c/marker\u003e\u003cg class=\"root\"\u003e\u003cg class=\"clusters\"\u003e\u003c/g\u003e\u003cg class=\"edgePaths\"\u003e\u003cpath marker-end=\"url(#flowchart-pointEnd)\" style=\"fill:none;\" class=\"edge-thickness-normal edge-pattern-solid flowchart-link LS-AWS LE-Token\" id=\"L-AWS-Token-0\" d=\"M185.671875,134.5L181.50520833333334,134.5C177.33854166666666,134.5,169.00520833333334,134.5,152.378260969764,145.875C135.75131360619469,157.25,110.83075221238937,180,98.37047151548673,191.375L85.91019081858407,202.75\"\u003e\u003c/path\u003e\u003cpath marker-end=\"url(#flowchart-pointEnd)\" style=\"fill:none;\" class=\"edge-thickness-normal edge-pattern-solid flowchart-link LS-Token LE-Terraform\" id=\"L-Token-Terraform-0\" d=\"M85.91019081858407,235.75L98.37047151548673,247.125C110.83075221238937,258.5,135.75131360619469,281.25,164.46289638643069,292.625C193.17447916666666,304,225.67708333333334,304,258.1796875,304C290.6822916666667,304,323.1848958333333,304,359.3528645833333,304C395.5208333333333,304,435.3541666666667,304,475.1875,304C515.0208333333334,304,554.8541666666666,304,585.6956312991642,296.7916666666667C616.5370959316618,289.5833333333333,638.3866918633236,275.1666666666667,649.3114898291544,267.9583333333333L660.2362877949853,260.75\"\u003e\u003c/path\u003e\u003cpath marker-end=\"url(#flowchart-pointEnd)\" style=\"fill:none;\" class=\"edge-thickness-normal edge-pattern-solid flowchart-link LS-Terraform LE-JWT\" id=\"L-Terraform-JWT-0\" d=\"M660.2362877949853,177.75L649.3114898291544,170.54166666666666C638.3866918633236,163.33333333333334,616.5370959316618,148.91666666666666,601.4456312991642,141.70833333333334C586.3541666666666,134.5,578.0208333333334,134.5,573.8541666666666,134.5L569.6875,134.5\"\u003e\u003c/path\u003e\u003cpath marker-end=\"url(#flowchart-pointEnd)\" style=\"fill:none;\" class=\"edge-thickness-normal edge-pattern-solid flowchart-link LS-JWT LE-AWS\" id=\"L-JWT-AWS-0\" d=\"M380.6875,134.5L376.5208333333333,134.5C372.3541666666667,134.5,364.0208333333333,134.5,355.6875,134.5C347.3541666666667,134.5,339.0208333333333,134.5,334.8541666666667,134.5L330.6875,134.5\"\u003e\u003c/path\u003e\u003c/g\u003e\u003cg class=\"edgeLabels\"\u003e\u003cg class=\"edgeLabel\"\u003e\u003cg transform=\"translate(0, 0)\" class=\"label\"\u003e\u003cforeignObject height=\"0\" width=\"0\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"edgeLabel\"\u003e\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"edgeLabel\"\u003e\u003cg transform=\"translate(0, 0)\" class=\"label\"\u003e\u003cforeignObject height=\"0\" width=\"0\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"edgeLabel\"\u003e\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"edgeLabel\"\u003e\u003cg transform=\"translate(0, 0)\" class=\"label\"\u003e\u003cforeignObject height=\"0\" width=\"0\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"edgeLabel\"\u003e\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"edgeLabel\"\u003e\u003cg transform=\"translate(0, 0)\" class=\"label\"\u003e\u003cforeignObject height=\"0\" width=\"0\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"edgeLabel\"\u003e\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"nodes\"\u003e\u003cg transform=\"translate(621.1171875, 169.75)\" class=\"root\"\u003e\u003cg class=\"clusters\"\u003e\u003cg id=\"Terraform\" class=\"cluster default\"\u003e\u003crect height=\"83\" width=\"206.890625\" y=\"8\" x=\"-0.9296875\" ry=\"0\" rx=\"0\" style=\"\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-0.9296875, 8)\" class=\"cluster-label\"\u003e\u003cforeignObject height=\"18\" width=\"206.890625\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eTerraform Cloud Workflow #2\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"edgePaths\"\u003e\u003c/g\u003e\u003cg class=\"edgeLabels\"\u003e\u003c/g\u003e\u003cg class=\"nodes\"\u003e\u003cg transform=\"translate(102.515625, 49.5)\" id=\"flowchart-OIDCProvider-23\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"119.03125\" y=\"-16.5\" x=\"-59.515625\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-52.015625, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"104.03125\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eOIDC Provider\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003cg transform=\"translate(178.171875, -8)\" class=\"root\"\u003e\u003cg class=\"clusters\"\u003e\u003cg id=\"AWS\" class=\"cluster default\"\u003e\u003crect height=\"269\" width=\"145.015625\" y=\"8\" x=\"8\" ry=\"0\" rx=\"0\" style=\"\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(51.4609375, 8)\" class=\"cluster-label\"\u003e\u003cforeignObject height=\"18\" width=\"58.09375\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eAWS #1\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003cg class=\"edgePaths\"\u003e\u003c/g\u003e\u003cg class=\"edgeLabels\"\u003e\u003c/g\u003e\u003cg class=\"nodes\"\u003e\u003cg transform=\"translate(80.5078125, 59.5)\" id=\"flowchart-OIDC-20\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"95.015625\" y=\"-16.5\" x=\"-47.5078125\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-40.0078125, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"80.015625\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eOIDC Trust\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg transform=\"translate(80.5078125, 142.5)\" id=\"flowchart-Roles-21\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"55.90625\" y=\"-16.5\" x=\"-27.953125\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-20.453125, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"40.90625\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eRoles\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg transform=\"translate(80.5078125, 225.5)\" id=\"flowchart-Resources-22\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"91.484375\" y=\"-16.5\" x=\"-45.7421875\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-38.2421875, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"76.484375\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eResources\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003cg transform=\"translate(67.8359375, 219.25)\" id=\"flowchart-Token-25\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"135.671875\" y=\"-16.5\" x=\"-67.8359375\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-60.3359375, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"120.671875\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eAccess Token #4\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003cg transform=\"translate(475.1875, 134.5)\" id=\"flowchart-JWT-28\" class=\"node default default\"\u003e\u003crect height=\"33\" width=\"189\" y=\"-16.5\" x=\"-94.5\" ry=\"0\" rx=\"0\" style=\"\" class=\"basic label-container\"\u003e\u003c/rect\u003e\u003cg transform=\"translate(-87, -9)\" style=\"\" class=\"label\"\u003e\u003cforeignObject height=\"18\" width=\"174\"\u003e\u003cdiv style=\"display: inline-block; white-space: nowrap;\" xmlns=\"http://www.w3.org/1999/xhtml\"\u003e\u003cspan class=\"nodeLabel\"\u003eJWT \u0026#x26; Cloud Role ID #3\u003c/span\u003e\u003c/div\u003e\u003c/foreignObject\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003c/g\u003e\u003c/svg\u003e\u003c/div\u003e\n\u003col\u003e\n\u003cli\u003eIn AWS, create an OIDC trust between a role and our terraform cloud\nworkflow(s) that need access to the cloud.\u003c/li\u003e\n\u003cli\u003eEvery time a job runs, TFC's OIDC Provider auto-generates an OIDC token.\nThis token contains multiple claims to establish a security-hardened and\nverifiable identity about the specific workflow that is trying to authenticate.\u003c/li\u003e\n\u003cli\u003eRequest this token from TFC's OIDC provider, and present it to AWS\u003c/li\u003e\n\u003cli\u003eOnce AWS successfully validates the claims presented in the token, it then\nprovides a short-lived cloud access token that is available only for the duration\nof the job.\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch1\u003eWhat does this post accomplish\u003c/h1\u003e\n\u003cul\u003e\n\u003cli\u003eSetup a root AWS account that is managed througuh terraform\u003c/li\u003e\n\u003cli\u003eSetup OIDC authentication with Terraform Cloud so it can talk to AWS\u003c/li\u003e\n\u003cli\u003eSetup Github Actions authentication with Terraform Cloud so we can run plan\nand apply through the CI/CD pipeline.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch1\u003eSetup AWS Access\u003c/h1\u003e\n\u003cp\u003eIt is very bad practice to use the root account for much of anything but for\nbootstrapping the account it is necessary, so the first step is to get your\n\u003ccode\u003eAWS_ACCESS_KEY_ID\u003c/code\u003e and \u003ccode\u003eAWS_SECRET_ACCESS_KEY\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003eTo do this click your account and choose \u003ccode\u003eSecurity Credentials\u003c/code\u003e in the top\nright:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_root_account/security_credentials.png\" height=\"200\"\u003e\n\u003c/center\u003e\n\u003cp\u003eThen choose \u003ccode\u003eCreate Access key\u003c/code\u003e:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_root_account/create_access_token.png\" width=\"200\"\u003e\n\u003c/center\u003e\n\u003cp\u003eYou need to set these environment variables in your shell so that your local\nshell has access to AWS. After you set them you can verify you set them correct\nby running:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ aws sts get-caller-identity\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eand you should get a result similar to:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-json\"\u003e\u003cspan class=\"hljs-punctuation\"\u003e{\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"UserId\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e,\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"Account\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"888888888888\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e,\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"Arn\"\u003c/span\u003e\u003cspan class=\"hljs-punctuation\"\u003e:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"arn:aws:iam::888888888888:root\"\u003c/span\u003e\n\u003cspan class=\"hljs-punctuation\"\u003e}\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eBootstrap\u003c/h2\u003e\n\u003cp\u003eBefore you can manage any of your accounts through Terraform Cloud you'll need\nbootstrap some core infrastructure like OIDC so Terraform Cloud can authenticate\nsecurely and manage AWS Resources on your behalf.\u003c/p\u003e\n\u003cp\u003eI personally prefer doing this in two repositories:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ccode\u003einfra-bootstrap\u003c/code\u003e: This repository does the bare minimum to hook up terraform\ncloud with your AWS account and stores the state in git. Its the only infra\nthat will not be controlled by your CI/CD pipeline.ccccccug\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ccode\u003einfra\u003c/code\u003e: The actual repository where all the rest of your AWS resources are\nmanaged. It will store state in Terraform Cloud and you can introduce a\nCI/CD pipeline for approving changes.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNote\u003c/strong\u003e: This repository will be generated with the terraform code.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAfter manually creating the git repository \u003ccode\u003einfra-boostrap\u003c/code\u003e in your Github\naccount We will need 3 providers to bootstrap the account \u003ccode\u003eaws\u003c/code\u003e, \u003ccode\u003egithub\u003c/code\u003e, and\n\u003ccode\u003etfe\u003c/code\u003e.\u003c/p\u003e\n\u003ch3\u003eVariables\u003c/h3\u003e\n\u003cp\u003eCreate a \u003ccode\u003e1-variables.tf\u003c/code\u003e where we can define the variables we'll need\nfor creating these resources.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_aws_audience\"\u003c/span\u003e {\n type = string\n default = \u003cspan class=\"hljs-string\"\u003e\"aws.workload.identity\"\u003c/span\u003e\n description = \u003cspan class=\"hljs-string\"\u003e\"The audience value to use in run identity tokens\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_hostname\"\u003c/span\u003e {\n type = string\n default = \u003cspan class=\"hljs-string\"\u003e\"app.terraform.io\"\u003c/span\u003e\n description = \u003cspan class=\"hljs-string\"\u003e\"The hostname of the TFC or TFE instance you'd like to use with AWS\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_project_name\"\u003c/span\u003e {\n type = string\n default = \u003cspan class=\"hljs-string\"\u003e\"Default Project\"\u003c/span\u003e\n description = \u003cspan class=\"hljs-string\"\u003e\"The project under which a workspace will be created\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_organization_name\"\u003c/span\u003e {\n type = string\n description = \u003cspan class=\"hljs-string\"\u003e\"The name of your Terraform Cloud organization\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_organization_owner\"\u003c/span\u003e {\n type = string\n description = \u003cspan class=\"hljs-string\"\u003e\"The owner of the TFC organization\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_workspaces\"\u003c/span\u003e {\n type = list(string)\n description = \u003cspan class=\"hljs-string\"\u003e\"The list of TFC workspaces\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_organization\"\u003c/span\u003e {\n description = \u003cspan class=\"hljs-string\"\u003e\"The organization the repositories are owned by\"\u003c/span\u003e\n type = string\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_repo_name\"\u003c/span\u003e {\n description = \u003cspan class=\"hljs-string\"\u003e\"The name of the git reppository we'll create for managing infra\"\u003c/span\u003e\n type = string\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_default_branch\"\u003c/span\u003e {\n description = \u003cspan class=\"hljs-string\"\u003e\"The default branch to utilize\"\u003c/span\u003e\n type = string\n default = \u003cspan class=\"hljs-string\"\u003e\"main\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_root_account_id\"\u003c/span\u003e {\n description = \u003cspan class=\"hljs-string\"\u003e\"The AWS root account we want to apply these changes to\"\u003c/span\u003e\n type = string\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe will use these variables in the later modules but they are mostly metadata\naround the terraform and github accounts you'll need to setup manually.\u003c/p\u003e\n\u003ch3\u003eProviders\u003c/h3\u003e\n\u003cp\u003eCreate a file called \u003ccode\u003e2-providers.tf\u003c/code\u003e and define the providers:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e {\n required_providers {\n tfe = {\n source = \u003cspan class=\"hljs-string\"\u003e\"hashicorp/tfe\"\u003c/span\u003e\n version = \u003cspan class=\"hljs-string\"\u003e\"0.41.0\"\u003c/span\u003e\n }\n\n aws = {\n source = \u003cspan class=\"hljs-string\"\u003e\"hashicorp/aws\"\u003c/span\u003e\n version = \u003cspan class=\"hljs-string\"\u003e\"4.58.0\"\u003c/span\u003e\n }\n\n github = {\n source = \u003cspan class=\"hljs-string\"\u003e\"integrations/github\"\u003c/span\u003e\n version = \u003cspan class=\"hljs-string\"\u003e\"5.18.3\"\u003c/span\u003e\n }\n }\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eprovider\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws\"\u003c/span\u003e {\n region = \u003cspan class=\"hljs-string\"\u003e\"us-east-1\"\u003c/span\u003e\n\n \u003cspan class=\"hljs-comment\"\u003e# Root account, all other accounts should be provisioned\u003c/span\u003e\n \u003cspan class=\"hljs-comment\"\u003e# via pull requests\u003c/span\u003e\n allowed_account_ids = [var.aws_root_account_id]\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eprovider\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github\"\u003c/span\u003e {\n owner = var.github_organization\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe key things there are we define \u003ccode\u003eallowed_account_ids\u003c/code\u003e to prevent us from\nworking against any account that isn't the root and we are using one of the\nvariables we defines earlier.\u003c/p\u003e\n\u003ch3\u003eGithub\u003c/h3\u003e\n\u003cp\u003eWe will utilize \u003ccode\u003eterraform\u003c/code\u003e to create the second git repository where the rest\nof the infrastructure will go. Create a file called \u003ccode\u003e3-github.tf\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_repository\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"repo\"\u003c/span\u003e {\n name = var.github_repo_name\n description = \u003cspan class=\"hljs-string\"\u003e\"Infrastructure Repository\"\u003c/span\u003e\n visibility = \u003cspan class=\"hljs-string\"\u003e\"private\"\u003c/span\u003e\n auto_init = true\n has_issues = true\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_branch_default\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"default\"\u003c/span\u003e {\n repository = github_repository.repo.name\n branch = var.github_default_branch\n}\n\n\u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_team\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"owners\"\u003c/span\u003e {\n name = \u003cspan class=\"hljs-string\"\u003e\"owners\"\u003c/span\u003e\n organization = tfe_organization.organization.name\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_team_token\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_actions_token\"\u003c/span\u003e {\n team_id = \u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e.tfe_team.owners.id\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"github_actions_secret\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_secret\"\u003c/span\u003e {\n repository = github_repository.repo.name\n secret_name = \u003cspan class=\"hljs-string\"\u003e\"TFE_TOKEN\"\u003c/span\u003e\n plaintext_value = tfe_team_token.github_actions_token.token\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eoutput\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"repository_id\"\u003c/span\u003e {\n value = github_repository.repo.id\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will generate a new repository in your account called \u003ccode\u003einfra\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003eFor the terraform provider to have access to github you need to create a new\npersonal access token with full \u003ccode\u003erepo\u003c/code\u003e access and set it as an environment\nvariable named \u003ccode\u003eGITHUB_TOKEN\u003c/code\u003e.\u003c/p\u003e\n\u003ch3\u003eTerraform Cloud\u003c/h3\u003e\n\u003cp\u003eNow we need to setup dynamic credentials so the terraform cloud agent is\nallowed to take actions on your behalf. To do this we'll setup an IAM\nrole and an OIDC provider. Create a file called \u003ccode\u003e4-tfc.tf\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_organization\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"organization\"\u003c/span\u003e {\n name = var.tfc_organization_name\n email = var.tfc_organization_owner\n}\n\n/* AWS will use this TLS certificate to verify that requests for dynamic\ncredentials come from Terraform Cloud.*/\n\u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tls_certificate\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_certificate\"\u003c/span\u003e {\n url = \u003cspan class=\"hljs-string\"\u003e\"https://\u003cspan class=\"hljs-variable\"\u003e${var.tfc_hostname}\u003c/span\u003e\"\u003c/span\u003e\n}\n\n/* sets up an OIDC \u003cspan class=\"hljs-keyword\"\u003eprovider\u003c/span\u003e in AWS with Terraform Cloud's TLS certificate,\nthe SHA1 fingerprint from the TLS certificate\n*/\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_openid_connect_provider\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc_provider\"\u003c/span\u003e {\n url = \u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e.tls_certificate.tfc_certificate.url\n client_id_list = [var.tfc_aws_audience]\n thumbprint_list = [\n \u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e.tls_certificate.tfc_certificate.certificates[\u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e].sha1_fingerprint\n ]\n}\n\n/* Policy to allow TFC to assume the AWS IAM role in our account */\n\u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_policy_document\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"assume_role\"\u003c/span\u003e {\n statement {\n effect = \u003cspan class=\"hljs-string\"\u003e\"Allow\"\u003c/span\u003e\n\n principals {\n type = \u003cspan class=\"hljs-string\"\u003e\"Federated\"\u003c/span\u003e\n identifiers = [aws_iam_openid_connect_provider.tfc_provider.arn]\n }\n condition {\n test = \u003cspan class=\"hljs-string\"\u003e\"StringEquals\"\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e = \u003cspan class=\"hljs-string\"\u003e\"\u003cspan class=\"hljs-variable\"\u003e${var.tfc_hostname}\u003c/span\u003e:aud\"\u003c/span\u003e\n\n values = [\n \u003cspan class=\"hljs-string\"\u003e\"\u003cspan class=\"hljs-variable\"\u003e${\u003cspan class=\"hljs-meta\"\u003eone(aws_iam_openid_connect_provider.tfc_provider.client_id_list)\u003c/span\u003e}\u003c/span\u003e\"\u003c/span\u003e\n ]\n }\n\n condition {\n test = \u003cspan class=\"hljs-string\"\u003e\"StringLike\"\u003c/span\u003e\n \u003cspan class=\"hljs-keyword\"\u003evariable\u003c/span\u003e = \u003cspan class=\"hljs-string\"\u003e\"\u003cspan class=\"hljs-variable\"\u003e${var.tfc_hostname}\u003c/span\u003e:sub\"\u003c/span\u003e\n\n values = [\n for workspace in var.tfc_workspaces : \u003cspan class=\"hljs-string\"\u003e\"organization:\u003cspan class=\"hljs-variable\"\u003e${tfe_organization.organization.name}\u003c/span\u003e:project:\u003cspan class=\"hljs-variable\"\u003e${var.tfc_project_name}\u003c/span\u003e:workspace:\u003cspan class=\"hljs-variable\"\u003e${workspace}\u003c/span\u003e:run_phase:*\"\u003c/span\u003e\n ]\n }\n actions = [\u003cspan class=\"hljs-string\"\u003e\"sts:AssumeRoleWithWebIdentity\"\u003c/span\u003e]\n }\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_role\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e {\n name = \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e\n assume_role_policy = \u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e.aws_iam_policy_document.assume_role.json\n}\n\n/* Policy for what the TFC agent is allowed to do */\n\u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_policy_document\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e {\n version = \u003cspan class=\"hljs-string\"\u003e\"2012-10-17\"\u003c/span\u003e\n\n statement {\n actions = [\u003cspan class=\"hljs-string\"\u003e\"*\"\u003c/span\u003e]\n effect = \u003cspan class=\"hljs-string\"\u003e\"Allow\"\u003c/span\u003e\n resources = [\u003cspan class=\"hljs-string\"\u003e\"*\"\u003c/span\u003e]\n }\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_policy\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e {\n name = \u003cspan class=\"hljs-string\"\u003e\"tfc-agent-access-policy\"\u003c/span\u003e\n description = \u003cspan class=\"hljs-string\"\u003e\"Access policy for the TFC agent\"\u003c/span\u003e\n policy = \u003cspan class=\"hljs-keyword\"\u003edata\u003c/span\u003e.aws_iam_policy_document.tfc-agent.json\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_iam_role_policy_attachment\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-access-attach\"\u003c/span\u003e {\n role = aws_iam_role.tfc-agent.name\n policy_arn = aws_iam_policy.tfc-agent.arn\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_workspace\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"workspaces\"\u003c/span\u003e {\n count = length(var.tfc_workspaces)\n name = var.tfc_workspaces[count.index]\n organization = tfe_organization.organization.name\n\n working_directory = var.tfc_workspaces[count.index]\n}\n\n/* These variables tell the agent to use dynamic credentials */\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_variable\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-auth\"\u003c/span\u003e {\n count = length(var.tfc_workspaces)\n key = \u003cspan class=\"hljs-string\"\u003e\"TFC_AWS_PROVIDER_AUTH\"\u003c/span\u003e\n value = true\n category = \u003cspan class=\"hljs-string\"\u003e\"env\"\u003c/span\u003e\n workspace_id = tfe_workspace.workspaces[count.index].id\n description = \u003cspan class=\"hljs-string\"\u003e\"Enable dynamic auth on the TFC agents\"\u003c/span\u003e\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfe_variable\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"tfc-role\"\u003c/span\u003e {\n count = length(var.tfc_workspaces)\n key = \u003cspan class=\"hljs-string\"\u003e\"TFC_AWS_RUN_ROLE_ARN\"\u003c/span\u003e\n value = aws_iam_role.tfc-agent.arn\n category = \u003cspan class=\"hljs-string\"\u003e\"env\"\u003c/span\u003e\n workspace_id = tfe_workspace.workspaces[count.index].id\n description = \u003cspan class=\"hljs-string\"\u003e\"Tell TFC what Role to run as\"\u003c/span\u003e\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis module is dynamic because there is one piece that will require a\nmanual oauth setup for github. So the first pass will apply without it\nand then later on we'll create it and run the apply again.\u003c/p\u003e\n\u003ch2\u003eApplying the changes\u003c/h2\u003e\n\u003cp\u003eNow we just need to define our settings for the module and we'll get our\ninfrastructure applied. Create a file called \u003ccode\u003esettings.auto.tfvars\u003c/code\u003e and\npopulate it with the content for your account. This is an example of what\nthis should look like:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003etfc_organization_name = \u003cspan class=\"hljs-string\"\u003e\"sontek\"\u003c/span\u003e\ntfc_organization_owner = \u003cspan class=\"hljs-string\"\u003e\"john@sontek.net\"\u003c/span\u003e\n\n\u003cspan class=\"hljs-comment\"\u003e# The workspaces you want to create and be able to manage with IaC\u003c/span\u003e\ntfc_workspaces = [\n \u003cspan class=\"hljs-string\"\u003e\"root\"\u003c/span\u003e\n]\n\u003cspan class=\"hljs-comment\"\u003e# this can be your username\u003c/span\u003e\ngithub_organization = \u003cspan class=\"hljs-string\"\u003e\"sontek\"\u003c/span\u003e\ngithub_repo_name = \u003cspan class=\"hljs-string\"\u003e\"sontek-infra\"\u003c/span\u003e\naws_root_account_id = \u003cspan class=\"hljs-string\"\u003e\"888888888888\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow run:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ terraform login\n❯ terraform init\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eand you should see:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs\"\u003eTerraform has been successfully initialized!\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow lets run our plan:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e❯ \u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e plan\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou should see a result:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-vbnet\"\u003e\u003cspan class=\"hljs-symbol\"\u003ePlan:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e10\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eto\u003c/span\u003e add, \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eto\u003c/span\u003e change, \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eto\u003c/span\u003e destroy.\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eApply it to make those resources:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e❯ \u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e apply\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAt this point it:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eCreated a terraform cloud organization\u003c/li\u003e\n\u003cli\u003eCreated a terraform cloud workspace\u003c/li\u003e\n\u003cli\u003eCreated a git repository\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch1\u003eVerify TFC can talk to AWS\u003c/h1\u003e\n\u003cp\u003eTo verify that TFC can communicate with AWS through the dynamic credentials,\nlets clone the \u003cem\u003eNEW\u003c/em\u003e repository we just generated and make some dummy resources. After\nyou've cloned the repository lets make a folder for the workspace \u003ccode\u003eroot\u003c/code\u003e that we\ndefined in bootstrap:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e root\n❯ \u003cspan class=\"hljs-built_in\"\u003ecd\u003c/span\u003e root\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow create a \u003ccode\u003e1-providers.tf\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e {\n cloud {\n organization = \u003cspan class=\"hljs-string\"\u003e\"sontek\"\u003c/span\u003e\n\n workspaces {\n name = \u003cspan class=\"hljs-string\"\u003e\"root\"\u003c/span\u003e\n }\n }\n\n required_providers {\n aws = {\n source = \u003cspan class=\"hljs-string\"\u003e\"hashicorp/aws\"\u003c/span\u003e\n version = \u003cspan class=\"hljs-string\"\u003e\"4.58.0\"\u003c/span\u003e\n }\n\n tfe = {\n source = \u003cspan class=\"hljs-string\"\u003e\"hashicorp/tfe\"\u003c/span\u003e\n version = \u003cspan class=\"hljs-string\"\u003e\"0.42.0\"\u003c/span\u003e\n }\n }\n}\n\n\u003cspan class=\"hljs-keyword\"\u003eprovider\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws\"\u003c/span\u003e {\n region = \u003cspan class=\"hljs-string\"\u003e\"us-east-1\"\u003c/span\u003e\n\n default_tags {\n tags = {\n Owner = \u003cspan class=\"hljs-string\"\u003e\"john@sontek.net\"\u003c/span\u003e\n Env = \u003cspan class=\"hljs-string\"\u003e\"Root\"\u003c/span\u003e\n Service = \u003cspan class=\"hljs-string\"\u003e\"BusinessOperations\"\u003c/span\u003e\n }\n }\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cstrong\u003eNOTE\u003c/strong\u003e: You should replace \u003ccode\u003eorganization\u003c/code\u003e, \u003ccode\u003eworkspaces.name\u003c/code\u003e, and\n\u003ccode\u003etags.Owner\u003c/code\u003e to be your own values.\u003c/p\u003e\n\u003cp\u003eNow create a small resource to prove everything is working, we'll use SQS for\nthis. Create a file called \u003ccode\u003e2-sqs.tf\u003c/code\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e\u003cspan class=\"hljs-keyword\"\u003eresource\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"aws_sqs_queue\"\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"example-sqs\"\u003c/span\u003e {\n name = \u003cspan class=\"hljs-string\"\u003e\"example-sqs\"\u003c/span\u003e\n message_retention_seconds = \u003cspan class=\"hljs-number\"\u003e86400\u003c/span\u003e\n receive_wait_time_seconds = \u003cspan class=\"hljs-number\"\u003e10\u003c/span\u003e\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eIf you run the plan you should see the resource it wants to create:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ terraform init\n❯ terraform plan\n\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eand you should see the run is executing in terraform cloud:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003eRunning plan in Terraform Cloud. Output will stream here. Pressing Ctrl-C\nwill stop streaming the logs, but will \u003cspan class=\"hljs-keyword\"\u003enot\u003c/span\u003e stop the plan running remotely.\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can click the link it provides to see the logs. Now lets apply this\nresource to see it all working:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e❯ \u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e apply\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou should get a response like:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-string\"\u003eApply\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ecomplete!\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003eResources:\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eadded,\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003echanged,\u003c/span\u003e \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edestroyed.\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo Terraform Cloud has full access to create AWS resources! The final step\nis to get github running the plan/apply on pull requests. Commit these files\nto your repository and we'll remove them in a pull request. Create a\n\u003ccode\u003e.gitignore\u003c/code\u003e file in the root:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-hcl\"\u003e.\u003cspan class=\"hljs-keyword\"\u003eterraform\u003c/span\u003e*\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eand commit all the files:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ git add *\n❯ git commit -m \u003cspan class=\"hljs-string\"\u003e\"initial infra\"\u003c/span\u003e\n❯ git push origin \u003cspan class=\"hljs-built_in\"\u003ehead\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eGithub Actions\u003c/h1\u003e\n\u003cp\u003eThe two most popular workflows when using terraform are:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eApply after Merge\u003c/strong\u003e: This is the default for things like\n\u003ca href=\"https://terraform.io\"\u003eterraform cloud\u003c/a\u003e and most github actions.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eApply before Merge\u003c/strong\u003e: This is the default for things like\n\u003ca href=\"https://www.runatlantis.io/\"\u003eAtlantis\u003c/a\u003e.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI don't like apply-after-merge. There are a lot of ways where a \u003ccode\u003eplan\u003c/code\u003e\ncan succeed but an \u003ccode\u003eapply\u003c/code\u003e will fail and you end up with broken configuration\nin \u003ccode\u003emain\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003eSo in this article I'll show you how to implement \u003cstrong\u003eapply-before-merge\u003c/strong\u003e with\ngithub actions.\u003c/p\u003e\n\u003cp\u003eAll of these changes will be in the \u003ccode\u003einfra\u003c/code\u003e repository that was generated from\n\u003ccode\u003ebootstrap\u003c/code\u003e. We are done with the bootstrap at this point.\u003c/p\u003e\n\u003cp\u003eFirst, lets setup the \u003ccode\u003e.github\u003c/code\u003e folder, the end result we want is:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e.github/\n└── workflows\n ├── on-apply-finished.yml\n ├── on-pull-request-labeled.yml\n └── on-pull-request.yml\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo create the folders:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e -p .github/workflows\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eOn Pull Request\u003c/h1\u003e\n\u003cp\u003eThe first flow we'll create is the \u003ccode\u003eterraform plan\u003c/code\u003e workflow which should be\nran whenever a pull request is opened. Create the file\n\u003ccode\u003e.github/workflows/on-pull-request.yml\u003c/code\u003e and put this content in it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yml\"\u003e\u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003epr_build\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003eon:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epull_request:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ebranches:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003emain\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003eenv:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eTERRAFORM_CLOUD_TOKENS:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eapp.terraform.io=${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.TFE_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eGITHUB_TOKEN:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.GITHUB_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003ejobs:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eterraform_validate:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eruns-on:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eubuntu-22.04\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003estrategy:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efail-fast:\u003c/span\u003e \u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematrix:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efolder:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eroot\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esteps:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eCheckout\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eactions/checkout@v3\u003c/span\u003e\n\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eterraform\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003evalidate\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edflook/terraform-validate@v1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkspace:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\n \u003cspan class=\"hljs-attr\"\u003eterraform_fmt:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eruns-on:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eubuntu-22.04\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003estrategy:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efail-fast:\u003c/span\u003e \u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematrix:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efolder:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eroot\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esteps:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eactions/checkout@v3\u003c/span\u003e\n\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eterraform\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003efmt\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edflook/terraform-fmt-check@v1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkspace:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\n \u003cspan class=\"hljs-attr\"\u003eterraform_plan:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eruns-on:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eubuntu-22.04\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epermissions:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003econtents:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eread\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epull-requests:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ewrite\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003estrategy:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efail-fast:\u003c/span\u003e \u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematrix:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efolder:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eroot\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esteps:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eactions/checkout@v3\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eterraform\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eplan\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edflook/terraform-plan@v1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkspace:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis creates three jobs:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eterraform_validate\u003c/strong\u003e: This validates the terraform via \u003ccode\u003eterraform validate\u003c/code\u003e\ncommand to make sure that it is correct and doesn't have duplicate resources\nor anything like that.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eterraform_fmt\u003c/strong\u003e: This verifies that the terraform is well formatted by\nrunning the \u003ccode\u003eterraform fmt\u003c/code\u003e command.`\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eterraform_plan\u003c/strong\u003e: This runs the \u003ccode\u003eterraform\u003c/code\u003e plan and comments on the PR a\ndiff of the changes for you to verify.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eTo verify this is working, lets delete \u003ccode\u003eroot/2-sqs.tf\u003c/code\u003e, then lets push a branch\nand make a pull request to see the result so far:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ \u003cspan class=\"hljs-built_in\"\u003erm\u003c/span\u003e root/2-sqs.tf\n❯ git add .github/ root/\n❯ git checkout -b apply-before-merge\n❯ git commit -m \u003cspan class=\"hljs-string\"\u003e\"Implemented on-pull-request\"\u003c/span\u003e\n❯ git push origin \u003cspan class=\"hljs-built_in\"\u003ehead\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAfter you make the pull request you should 3 checks on it and a comment that\nshows the plan:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/github_comment.png\" width=\"400\"\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/github_checks.png\" width=\"400\"\u003e\n\u003c/center\u003e\n\u003ch1\u003eApply on Label\u003c/h1\u003e\n\u003cp\u003eSo now that the plan is working we need some way to \u003ccode\u003eapply\u003c/code\u003e the changes. I've\nfound the best way to do this is via a label rather than a comment because of\nthe way github actions work. Their event based actions like \u003ccode\u003eon-comment\u003c/code\u003e aren't\nexecuted in the context of a pull-request.\u003c/p\u003e\n\u003cp\u003eSince we will be using a label to signal a plan is ready to be applied lets\ncreate a new file \u003ccode\u003e.github/workflows/on-pull-request-labeled.yml\u003c/code\u003e and provide\nthis content:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003epr_apply\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003eon:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epull_request:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003etypes:\u003c/span\u003e [ \u003cspan class=\"hljs-string\"\u003elabeled\u003c/span\u003e ]\n\n\u003cspan class=\"hljs-attr\"\u003eenv:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eTERRAFORM_CLOUD_TOKENS:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eapp.terraform.io=${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.TFE_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eGITHUB_TOKEN:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.GITHUB_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003ejobs:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eterraform_apply:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eif:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003egithub.event.label.name\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e==\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'tfc-apply'\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eruns-on:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eubuntu-22.04\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epermissions:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003econtents:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eread\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epull-requests:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ewrite\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003estrategy:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efail-fast:\u003c/span\u003e \u003cspan class=\"hljs-literal\"\u003efalse\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ematrix:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efolder:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eroot\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esteps:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eactions/checkout@v3\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003edflook/terraform-apply@v1\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epath:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkspace:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ematrix.folder\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will fire whenever a pull request is labeled with the \u003ccode\u003etfc-apply\u003c/code\u003e label.\nYou will need to create this label for the repository.\u003c/p\u003e\n\u003cp\u003eIt will run the \u003ccode\u003eapply\u003c/code\u003e and update the previous plan comment to let you\nknow the status.\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/tfc_applying.png\" width=\"400\"\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/tfc_applying_comment.png\" width=\"400\"\u003e\n\u003c/center\u003e\n\u003ch1\u003eMerge on Apply\u003c/h1\u003e\n\u003cp\u003eOne thing you'll notice is that the pull request stayed open even after the\ninfrastructure is applied and we don't want that. We want any changes that have\nmade it into the environment to be merged into \u003ccode\u003emain\u003c/code\u003e automatically. To do\nthis we'll create our final action.\u003c/p\u003e\n\u003cp\u003eCreate a new file \u003ccode\u003e.github/workflows/on-apply-finished.yml\u003c/code\u003e with this content:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003epr_merge\u003c/span\u003e\n\n\u003cspan class=\"hljs-comment\"\u003e# Only trigger, when the build workflow succeeded\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eon:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkflow_run:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eworkflows:\u003c/span\u003e [\u003cspan class=\"hljs-string\"\u003epr_apply\u003c/span\u003e]\n \u003cspan class=\"hljs-attr\"\u003etypes:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ecompleted\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003ejobs:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003emerge:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eif:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003egithub.event.workflow_run.conclusion\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e==\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'success'\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eruns-on:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eubuntu-22.04\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epermissions:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003econtents:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ewrite\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epull-requests:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ewrite\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003echecks:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eread\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003estatuses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eread\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eactions:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eread\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eoutputs:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003epullRequestNumber:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esteps.workflow-run-info.outputs.pullRequestNumber\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esteps:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"Get information about the current run\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003epotiuk/get-workflow-origin@v1_5\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eid:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eworkflow-run-info\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003etoken:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.GITHUB_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esourceRunId:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003egithub.event.workflow_run.id\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003ename:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003emerge\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ea\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003epull\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003erequest\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eafter\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eterraform\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eapply\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003euses:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esudo-bot/action-pull-request-merge@v1.2.0\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003ewith:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003egithub-token:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esecrets.GITHUB_TOKEN\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003enumber:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e${{\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esteps.workflow-run-info.outputs.pullRequestNumber\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e}}\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will wait until the \u003ccode\u003epr_apply\u003c/code\u003e job completes and as long as it was\nsuccessful it'll merge the branch!\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNOTE\u003c/strong\u003e: As I mentioned earlier, the event based actions do not run in the\ncontext of the pull request which means you cannot test changes to them during\nthe PR either. You must merge the \u003ccode\u003eon-apply-finished.yml\u003c/code\u003e file to \u003ccode\u003emain\u003c/code\u003e\nbefore it starts working.\u003c/p\u003e\n\u003ch1\u003eBranch Protection\u003c/h1\u003e\n\u003cp\u003eThe final step to the process is to make sure you go to your github settings\nand make sure these status checks are required before merging. Branch protection\nis a feature that will prevent merging changes into a branch unless all\nrequired checks are passing.\u003c/p\u003e\n\u003cp\u003eGo to \u003ccode\u003eSettings\u003c/code\u003e -\u003e \u003ccode\u003eBranches\u003c/code\u003e -\u003e \u003ccode\u003eBranch Protection\u003c/code\u003e and add a branch\nprotection rule:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/branch_protection.png\" width=\"500\"\u003e\n\u003c/center\u003e\n\u003cp\u003eYou want to enable the following settings:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eBranch Name\u003c/strong\u003e: main\u003c/li\u003e\n\u003cli\u003e✅ Require a pull request before merging\u003c/li\u003e\n\u003cli\u003e✅ Require status checks to pass before merging\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThen for \u003ccode\u003eStatus checks that are required.\u003c/code\u003e select all of the ones we've\ncreated:\u003c/p\u003e\n\u003ccenter\u003e\n\u003cimg src=\"/images/posts/aws_apply_before_merge/required_checks.png\" height=\"200\"\u003e\n\u003c/center\u003e\n\u003ch1\u003eHelpful Resources\u003c/h1\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://developer.hashicorp.com/terraform/tutorials/cloud/dynamic-credentials?product_intent=terraform\"\u003eTerraform Dynamic Credentials Tutorial\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://developer.hashicorp.com/terraform/cloud-docs/workspaces/dynamic-provider-credentials/aws-configuration\"\u003eTerraform docs on Dynamic Credentials\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#understanding-the-oidc-token\"\u003eGithub's understanding OIDC\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e","category":"AWS","date":"2023-06-23T00:00:00Z","tags":["AWS","DevOps","SRE"],"title":"AWS From Scratch with Terraform - Setting up your Root Account for IaC with Terraform Cloud and Github actions."},{"id":["2023","nuking_aws_account"],"path":"2023/nuking_aws_account","contentHtml":"\u003cp\u003eWhen you're an SRE/DevOps engineer you'll end up making AWS accounts and\ncreate a lot of cruft in your sandbox and development accounts. AWS\ndoes not make it easy to clear these up but there is a tool called\n\u003ca href=\"https://github.com/rebuy-de/aws-nuke\"\u003eaws-nuke\u003c/a\u003e that will do it for you!\u003c/p\u003e\n\u003ch1\u003eSafe Guards\u003c/h1\u003e\n\u003cp\u003eaws-nuke has a few safeguards in place to prevent inadvertent data loss.\nThe first of which is it requires you to alias the targetted account. I\nlike to put \u003ccode\u003enuke\u003c/code\u003e in the alias to make it clear.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eaws iam create-account-alias --account-alias nuke-\u0026#x3C;account\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe second safe-guard is the config takes a key called \u003ccode\u003eaccount-blocklist\u003c/code\u003e\nthat will guarantee nuke will not run against it no matter what.\u003c/p\u003e\n\u003cp\u003eThe final safety switch it has is it will not take any action by default,\nit will only execute a dry-run. You need to run the CLI with\n\u003ccode\u003e--no-dry-run\u003c/code\u003e if you want it to take action.\u003c/p\u003e\n\u003ch1\u003eGetting Started\u003c/h1\u003e\n\u003cp\u003eYou configure \u003ccode\u003eaws-nuke\u003c/code\u003e with YAML, so the first step is to define that:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-makefile\"\u003e\u003cspan class=\"hljs-section\"\u003eregions:\u003c/span\u003e\n - us-east-1\n - global\n\n\u003cspan class=\"hljs-section\"\u003eaccount-blocklist:\u003c/span\u003e\n - \u003cspan class=\"hljs-string\"\u003e\"888888888888\"\u003c/span\u003e \u003cspan class=\"hljs-comment\"\u003e# production\u003c/span\u003e\n\n\u003cspan class=\"hljs-section\"\u003eaccounts:\u003c/span\u003e\n \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e: {} \u003cspan class=\"hljs-comment\"\u003e# nuke-\u0026#x3C;account\u003e\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will prevent us from nuking our production account and target all resources\nin the account we actually want to nuke.\u003c/p\u003e\n\u003cp\u003eYou might want to have it nuke \u003cem\u003eALL REGIONS\u003c/em\u003e in AWS since you may not know which\nregions resources are deployed in. To do this you can query the regions from AWS:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ aws ec2 describe-regions --all-regions --query \u003cspan class=\"hljs-string\"\u003e\"Regions[*].RegionName\"\u003c/span\u003e --output text | xargs -n 1 | \u003cspan class=\"hljs-built_in\"\u003esort\u003c/span\u003e\n\naf-south-1\nap-east-1\nap-northeast-1\nap-northeast-2\nap-northeast-3\nap-south-1\nap-south-2\nap-southeast-1\nap-southeast-2\nap-southeast-3\nap-southeast-4\nca-central-1\neu-central-1\neu-central-2\neu-north-1\neu-south-1\neu-south-2\neu-west-1\neu-west-2\neu-west-3\nme-central-1\nme-south-1\nsa-east-1\nus-east-1\nus-east-2\nus-west-1\nus-west-2\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWhich would give you an updated config of:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003eregions:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eglobal\u003c/span\u003e \u003cspan class=\"hljs-comment\"\u003e# Global resources like IAM\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eaf-south-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-east-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-northeast-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-northeast-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-northeast-3\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-south-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-south-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-southeast-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-southeast-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-southeast-3\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eap-southeast-4\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eca-central-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-central-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-central-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-north-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-south-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-south-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-west-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-west-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eeu-west-3\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eme-central-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eme-south-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esa-east-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eus-east-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eus-east-2\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eus-west-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eus-west-2\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eI personally don't recommend targetting all AWS regions at the same time. It\nwill generate a lot of output and be slow. You could do it if necessary but\nmost people only have a few regions they use and so they can set those directly.\nFor example it, maybe you only use \u003ccode\u003eus-\u003c/code\u003e based regions?\u003c/p\u003e\n\u003cp\u003eSo lets run the dry-run and see what it wants to nuke:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ aws-nuke -c nuke.yaml\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis should output something like:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eDo you really want to nuke the account with the ID 777777777777 and the \u003cspan class=\"hljs-built_in\"\u003ealias\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'nuke-sandbox'\u003c/span\u003e?\nDo you want to \u003cspan class=\"hljs-built_in\"\u003econtinue\u003c/span\u003e? Enter account \u003cspan class=\"hljs-built_in\"\u003ealias\u003c/span\u003e to \u003cspan class=\"hljs-built_in\"\u003econtinue\u003c/span\u003e.\n\u003e nuke-sandbox\nus-east-1 - EC2Subnet - subnet-0cd9975a443a6304b - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2Subnet - subnet-0be39d02e399a371c - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2Subnet - subnet-02d7017bd4730ea63 - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2Subnet - subnet-0ec04b28c32708ab2 - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2Subnet - subnet-0eea1b4be084840ed - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2Subnet - subnet-05a294cc04736012e - [DefaultForAz: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e, OwnerID: \u003cspan class=\"hljs-string\"\u003e\"777777777777\"\u003c/span\u003e] - would remove\nus-east-1 - EC2RouteTable - rtb-0abda0e94015064ca - [DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e] - would remove\nus-east-1 - EC2DefaultSecurityGroupRule - sgr-0368525f77bf566ac - [SecurityGroupId: \u003cspan class=\"hljs-string\"\u003e\"sg-0a59900b52ced5e10\"\u003c/span\u003e] - would remove\nus-east-1 - EC2DefaultSecurityGroupRule - sgr-0890a837ed6148729 - [SecurityGroupId: \u003cspan class=\"hljs-string\"\u003e\"sg-0a59900b52ced5e10\"\u003c/span\u003e] - would remove\nus-east-1 - EC2InternetGatewayAttachment - igw-0acfb474f1fd71375 -\u003e vpc-0be5d310ab44c239a - [DefaultVPC: \u003cspan class=\"hljs-string\"\u003e\"true\"\u003c/span\u003e] - would remove\nus-east-1 - SQSQueue - https://sqs.us-east-1.amazonaws.com/777777777777/example-sqs - would remove\nglobal - IAMSAMLProvider - arn:aws:iam::777777777777:saml-provider/AWSSSO_254abb4071f10b25_DO_NOT_DELETE - would remove\nglobal - IAMOpenIDConnectProvider - arn:aws:iam::777777777777:oidc-provider/app.terraform.io - [Arn: \u003cspan class=\"hljs-string\"\u003e\"arn:aws:iam::777777777777:oidc-provider/app.terraform.io\"\u003c/span\u003e] - would remove\nglobal - IAMPolicy - arn:aws:iam::777777777777:policy/tfc-agent-access-policy - [ARN: \u003cspan class=\"hljs-string\"\u003e\"arn:aws:iam::777777777777:policy/tfc-agent-access-policy\"\u003c/span\u003e, Name: \u003cspan class=\"hljs-string\"\u003e\"tfc-agent-access-policy\"\u003c/span\u003e, Path: \u003cspan class=\"hljs-string\"\u003e\"/\"\u003c/span\u003e, PolicyID: \u003cspan class=\"hljs-string\"\u003e\"ANPA2T6PZOBNWI76TKQRF\"\u003c/span\u003e] - would remove\nglobal - IAMRole - tfc-agent - [CreateDate: \u003cspan class=\"hljs-string\"\u003e\"2023-04-02T17:55:23Z\"\u003c/span\u003e, LastUsedDate: \u003cspan class=\"hljs-string\"\u003e\"2023-06-22T13:45:02Z\"\u003c/span\u003e, Name: \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e, Path: \u003cspan class=\"hljs-string\"\u003e\"/\"\u003c/span\u003e] - would remove\nglobal - IAMRolePolicyAttachment - tfc-agent -\u003e tfc-agent-access-policy - [PolicyArn: \u003cspan class=\"hljs-string\"\u003e\"arn:aws:iam::777777777777:policy/tfc-agent-access-policy\"\u003c/span\u003e, PolicyName: \u003cspan class=\"hljs-string\"\u003e\"tfc-agent-access-policy\"\u003c/span\u003e, RoleCreateDate: \u003cspan class=\"hljs-string\"\u003e\"2023-04-02T17:55:23Z\"\u003c/span\u003e, RoleLastUsed: \u003cspan class=\"hljs-string\"\u003e\"2023-06-22T13:45:02Z\"\u003c/span\u003e, RoleName: \u003cspan class=\"hljs-string\"\u003e\"tfc-agent\"\u003c/span\u003e, RolePath: \u003cspan class=\"hljs-string\"\u003e\"/\"\u003c/span\u003e] - would remove\nScan complete: 85 total, 19 nukeable, 66 filtered.\n\nThe above resources would be deleted with the supplied configuration. Provide --no-dry-run to actually destroy resources.\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis is great, it fully scanned the account and found every resource to delete!\nIt even wants to delete the DefaultVPC which is usually a good idea. The one\nresource that should catch your eye that you probably do not want to delete:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003earn:aws:iam::777777777777:saml-provider/AWSSSO_254abb4071f10b25_DO_NOT_DELETE\u003c/code\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAWS clearly doesn't want us to delete that!\u003c/p\u003e\n\u003ch1\u003eFilters\u003c/h1\u003e\n\u003cp\u003eTo prevent nuke from deleting resources you want to keep you can define presets\nthat you use on each account. So with our SSO example we want to prevent it\nfrom deleting those resources in a preset.\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003epresets:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esso:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efilters:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMSAMLProvider:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"regex\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"AWSSSO_.*_DO_NOT_DELETE\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRole:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"glob\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"AWSReservedSSO_*\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRolePolicyAttachment:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"glob\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"AWSReservedSSO_*\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can see in this example I'm targetting specific resource types and then\nmatching them with both \u003ccode\u003eregex\u003c/code\u003e and \u003ccode\u003eglob\u003c/code\u003e filter types. These are super\npowerful but a lot of times the simpler filters can be used. I start with\n\u003ccode\u003econtains\u003c/code\u003e filter and then go from there:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003econtains\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eAWSReservedSSO\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen the other thing you may have noticed is that I was repeating\n\u003ccode\u003eAWSReservedSSO\u003c/code\u003e multiple times. To reduce that you can use standard YAML\nanchors. So the final config for your preset would look like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003epresets:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esso:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efilters:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e_DEFAULT_FILTERS:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e\u0026#x26;DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"contains\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"DO_NOT_DELETE\"\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"contains\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"AWSReservedSSO\"\u003c/span\u003e\n\n \u003cspan class=\"hljs-attr\"\u003eIAMSAMLProvider:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRole:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRolePolicyAttachment:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we can use that preset in our accounts configuration:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003eaccounts:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"777777777777\":\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"presets\":\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esso\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo your final config should look something like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003eregions:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eus-east-1\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eglobal\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003eaccount-blocklist:\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"888888888888\"\u003c/span\u003e \u003cspan class=\"hljs-comment\"\u003e# production\u003c/span\u003e\n\n\n\u003cspan class=\"hljs-attr\"\u003epresets:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003esso:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003efilters:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e_DEFAULT_FILTERS:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e\u0026#x26;DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"contains\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"DO_NOT_DELETE\"\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-attr\"\u003etype:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"contains\"\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003evalue:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\"AWSReservedSSO\"\u003c/span\u003e\n\n \u003cspan class=\"hljs-attr\"\u003eIAMSAMLProvider:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRole:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003eIAMRolePolicyAttachment:\u003c/span\u003e \u003cspan class=\"hljs-meta\"\u003e*DEFAULT\u003c/span\u003e\n\n\u003cspan class=\"hljs-attr\"\u003eaccounts:\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"777777777777\":\u003c/span\u003e\n \u003cspan class=\"hljs-attr\"\u003e\"presets\":\u003c/span\u003e\n \u003cspan class=\"hljs-bullet\"\u003e-\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003esso\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWhen you run this you should see now the resources we want to keep are filtered\nout:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003eglobal - IAMSAMLProvider - arn:aws:iam::777777777777:saml-provider/AWSSSO_254abb4071f10b25_DO_NOT_DELETE - filtered by config\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eOnce you are ready and have your filters in place you can run it for real!\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003e❯ aws-nuke -c nuke.yaml --no-dry-run\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eNext steps\u003c/h1\u003e\n\u003cp\u003eOne final note about it is that it does not understand the relationship between\nresources and so it could try deleting an EBS volume that is still in use by an\nEC2 instance. There isn't a great solution for this outside of running nuke a\nfew times.\u003c/p\u003e\n\u003cp\u003eThe tool is well documented and so you can find the rest of information going to\n\u003ca href=\"https://github.com/rebuy-de/aws-nuke\"\u003ehttps://github.com/rebuy-de/aws-nuke\u003c/a\u003e!\u003c/p\u003e","category":"AWS","date":"2023-06-22T00:00:00Z","tags":["AWS","DevOps","SRE"],"title":"Wiping an AWS Account with aws-nuke"}]},"Interviews":{"count":2,"posts":[{"id":["old","caesar_cipher_in_python"],"path":"old/caesar_cipher_in_python","contentHtml":"\u003cp\u003eI'm currently teaching my wife to code and one of the problems that we\nworked on to teach her some fundamental programming concepts was\nre-implementing the caesar cipher in python. It was fun not only to code\nbut to also start sending each other \"secret\" messages!\u003c/p\u003e\n\u003cp\u003eThe caesar cipher is a rather simple encoding, you just shift the\nalphabet a certain amount of characters. For example, if you are using a\nshift of 2:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\u003cspan class=\"hljs-attr\"\u003ea\u003c/span\u003e =\u003e c\n\u003cspan class=\"hljs-attr\"\u003eb\u003c/span\u003e =\u003e d\n\u003cspan class=\"hljs-attr\"\u003ey\u003c/span\u003e =\u003e a\n\u003cspan class=\"hljs-attr\"\u003ez\u003c/span\u003e =\u003e b\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eUsing this as an interview type question would provide a few interesting\nproblems and give you a good perspective on how good a developers\nproblem solving skills are and how knowledgeable they are in the\nlanguage of their choice.\u003c/p\u003e\n\u003cp\u003eThe first issue is to handle the beginning and end of the alphabet, if\nyou are encoding 'z' then you will have to start your shift on a. The\nsecond problem is to only encode letters since there was no ascii table\nto define in what order characters are shifted back in those times.\u003c/p\u003e\n\u003cp\u003eWithout using too much of the built in python niceties you could do\nsomething similar to this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-comment\"\u003e#!/usr/bin/python\u003c/span\u003e\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003edecode_shift_letter\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003ecurrent_ord, start, end, shift\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e current_ord - shift \u0026#x3C; start:\n new_ord = (current_ord + \u003cspan class=\"hljs-number\"\u003e26\u003c/span\u003e) - shift\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003echr\u003c/span\u003e(new_ord)\n \u003cspan class=\"hljs-keyword\"\u003eelse\u003c/span\u003e:\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003echr\u003c/span\u003e(current_ord-shift)\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003eencode_shift_letter\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003ecurrent_ord, start, end, shift\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e current_ord + shift \u003e end:\n new_ord = (current_ord - \u003cspan class=\"hljs-number\"\u003e26\u003c/span\u003e) + shift\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003echr\u003c/span\u003e(new_ord)\n \u003cspan class=\"hljs-keyword\"\u003eelse\u003c/span\u003e:\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003echr\u003c/span\u003e(current_ord+shift)\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003edecode\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003e\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e modify_input(\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift, decode_shift_letter)\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003eencode\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003e\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e modify_input(\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift, encode_shift_letter)\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003emodify_input\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003e\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift, shift_letter\u003c/span\u003e):\n new_sentence = \u003cspan class=\"hljs-string\"\u003e''\u003c/span\u003e\n\n \u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e letter \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e:\n \u003cspan class=\"hljs-comment\"\u003e# we only encode letters, random characters like +!%$ are not encoded.\u003c/span\u003e\n \u003cspan class=\"hljs-comment\"\u003e# Lower and Capital letters are not stored near each other on the\u003c/span\u003e\n \u003cspan class=\"hljs-comment\"\u003e# ascii table\u003c/span\u003e\n lower_start = \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'a'\u003c/span\u003e)\n lower_end = \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'z'\u003c/span\u003e)\n upper_start = \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'A'\u003c/span\u003e)\n upper_end = \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'Z'\u003c/span\u003e)\n current_ord = \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(letter)\n\n \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e current_ord \u003e= lower_start \u003cspan class=\"hljs-keyword\"\u003eand\u003c/span\u003e current_ord \u0026#x3C;= lower_end:\n new_sentence += shift_letter(current_ord, lower_start, lower_end, shift)\n \u003cspan class=\"hljs-keyword\"\u003eelif\u003c/span\u003e current_ord \u003e= upper_start \u003cspan class=\"hljs-keyword\"\u003eand\u003c/span\u003e current_ord \u0026#x3C;= upper_end:\n new_sentence += shift_letter(current_ord, upper_start, upper_end, shift)\n \u003cspan class=\"hljs-keyword\"\u003eelse\u003c/span\u003e:\n new_sentence += letter\n\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e new_sentence\n\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003eget_shift\u003c/span\u003e():\n \u003cspan class=\"hljs-keyword\"\u003etry\u003c/span\u003e:\n shift = \u003cspan class=\"hljs-built_in\"\u003eint\u003c/span\u003e(raw_input(\u003cspan class=\"hljs-string\"\u003e'What shift would you like to use?\\n'\u003c/span\u003e))\n \u003cspan class=\"hljs-keyword\"\u003eexcept\u003c/span\u003e ValueError:\n \u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'Shift must be a number'\u003c/span\u003e\n shift = get_shift()\n\n \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003enot\u003c/span\u003e (shift \u003e \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eand\u003c/span\u003e shift \u0026#x3C;= \u003cspan class=\"hljs-number\"\u003e25\u003c/span\u003e):\n \u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'Shift must be between 1 and 25'\u003c/span\u003e\n shift = get_shift()\n\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e shift\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003emain\u003c/span\u003e():\n \u003cspan class=\"hljs-keyword\"\u003etry\u003c/span\u003e:\n task = \u003cspan class=\"hljs-built_in\"\u003eint\u003c/span\u003e(raw_input(\u003cspan class=\"hljs-string\"\u003e'1) Encode \\n'\u003c/span\u003e+ \\\n \u003cspan class=\"hljs-string\"\u003e'2) Decode \\n'\u003c/span\u003e))\n \u003cspan class=\"hljs-keyword\"\u003eexcept\u003c/span\u003e ValueError:\n \u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'Invalid task, try again!'\u003c/span\u003e\n main()\n\n shift = get_shift()\n \u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e = raw_input(\u003cspan class=\"hljs-string\"\u003e'What message would you like to %s\\n'\u003c/span\u003e % (\u003cspan class=\"hljs-string\"\u003e'Encode'\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e task == \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eelse\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e'Decode'\u003c/span\u003e))\n\n \u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e task == \u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e:\n \u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e encode(\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift)\n \u003cspan class=\"hljs-keyword\"\u003eelif\u003c/span\u003e task == \u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e:\n \u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e decode(\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift)\n\n\u003cspan class=\"hljs-keyword\"\u003eif\u003c/span\u003e __name__ == \u003cspan class=\"hljs-string\"\u003e'__main__'\u003c/span\u003e:\n main()\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis would prove that you are a decent problem solver and have enough of\nthe language to get things done but if you want to prove you have\nmastered the python language you might take advantage of some slicing\nand some methods out of the string module and change your code to look\nsomething like:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-keyword\"\u003efrom\u003c/span\u003e string \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e letters, maketrans\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003edecode\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003e\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e modify_input(\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, -shift)\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003eencode\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003e\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift\u003c/span\u003e):\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e modify_input(\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift)\n\n\u003cspan class=\"hljs-keyword\"\u003edef\u003c/span\u003e \u003cspan class=\"hljs-title function_\"\u003emodify_input\u003c/span\u003e(\u003cspan class=\"hljs-params\"\u003e\u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e, shift\u003c/span\u003e):\n trans = maketrans(letters, letters[shift:] + letters[:shift])\n \u003cspan class=\"hljs-keyword\"\u003ereturn\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003einput\u003c/span\u003e.translate(trans)\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eDo get more information on string.letters and string.maketrans you can\nvisit their documentation\n[here](\u003ca href=\"http://docs.python.org/library/string.html\"\u003ehttp://docs.python.org/library/string.html\u003c/a\u003e)\u003c/p\u003e","category":"Development","date":"2010-11-15T00:00:00Z","tags":["Python","Interviews"],"title":"Caesar Cipher in Python"},{"id":["old","convert_a_string_to_an_integer_in_python"],"path":"old/convert_a_string_to_an_integer_in_python","contentHtml":"\u003cp\u003eA fun interview question some developers like to ask is to have you\nconvert ascii characters to an integer without using built in methods\nlike string.atoi or int().\u003c/p\u003e\n\u003cp\u003eSo using python the obvious ways to convert a string to an integer are\nthese:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-built_in\"\u003eint\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'1234'\u003c/span\u003e)\n\u003cspan class=\"hljs-number\"\u003e1234\u003c/span\u003e\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e string\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003estring.atoi(\u003cspan class=\"hljs-string\"\u003e'1234'\u003c/span\u003e)\n\u003cspan class=\"hljs-number\"\u003e1234\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe interesting thing here is finding out where on the ascii character\ntable the number is. Luckily python has this already built in with the\nord method:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-built_in\"\u003ehelp\u003c/span\u003e(\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e)\n\n\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(...)\n \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(c) -\u003e integer\n\n Return the integer ordinal of a one-character string.\n\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'1'\u003c/span\u003e)\n\u003cspan class=\"hljs-number\"\u003e49\u003c/span\u003e\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'2'\u003c/span\u003e)\n\u003cspan class=\"hljs-number\"\u003e50\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eYou can see that the numbers are grouped together on the ascii table, so\nyou just have to grab '0' as the base and subtract the rest:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'1'\u003c/span\u003e)-\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'0'\u003c/span\u003e)\n\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eSo if we have the string '1234', we can get each of the individual\nnumbers by looping over it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003enum_string = \u003cspan class=\"hljs-string\"\u003e'1234'\u003c/span\u003e\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003enum_list = []\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003ebase = \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'0'\u003c/span\u003e)\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e num \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e num_string:\n\u003cspan class=\"hljs-meta\"\u003e... \u003c/span\u003e num_list.append(\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(num) - base)\n...\n\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e num_list\n[\u003cspan class=\"hljs-number\"\u003e1\u003c/span\u003e, \u003cspan class=\"hljs-number\"\u003e2\u003c/span\u003e, \u003cspan class=\"hljs-number\"\u003e3\u003c/span\u003e, \u003cspan class=\"hljs-number\"\u003e4\u003c/span\u003e]\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003ebut now how to we combine all these together to get 1234? You can't\njust add them up because you'll just get 1+2+3+4 = 10.\u003c/p\u003e\n\u003cp\u003eSo, we have to get 1000 + 200 + 30 + 4, which is a simple problem to\nsolve. Its just number times 10 to the nth power, so the final solution\nis:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003enum = \u003cspan class=\"hljs-string\"\u003e'1234'\u003c/span\u003e\nnew_num = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\nbase = \u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'0'\u003c/span\u003e)\n\n\u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e i,n \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003eenumerate\u003c/span\u003e(\u003cspan class=\"hljs-built_in\"\u003ereversed\u003c/span\u003e(num)):\n new_num += (\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(n) - base) * (\u003cspan class=\"hljs-number\"\u003e10\u003c/span\u003e**i)\n\n\u003cspan class=\"hljs-built_in\"\u003eprint\u003c/span\u003e new_num\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis code is a little verbose though, lets make it a dirty nasty one\nliner!\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-python\"\u003e\u003cspan class=\"hljs-meta\"\u003e\u003e\u003e\u003e \u003c/span\u003e\u003cspan class=\"hljs-built_in\"\u003esum\u003c/span\u003e([(\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(n)-\u003cspan class=\"hljs-built_in\"\u003eord\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'0'\u003c/span\u003e)) * (\u003cspan class=\"hljs-number\"\u003e10\u003c/span\u003e ** i) \u003cspan class=\"hljs-keyword\"\u003efor\u003c/span\u003e i,n \u003cspan class=\"hljs-keyword\"\u003ein\u003c/span\u003e \u003cspan class=\"hljs-built_in\"\u003eenumerate\u003c/span\u003e(\u003cspan class=\"hljs-built_in\"\u003ereversed\u003c/span\u003e(\u003cspan class=\"hljs-string\"\u003e'1234'\u003c/span\u003e))])\n\u003cspan class=\"hljs-number\"\u003e1234\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e","category":"Development","date":"2010-10-28T00:00:00Z","tags":["Python","Interviews"],"title":"Convert a string to an integer in Python"}]},"Vim":{"count":1,"posts":[{"id":["old","turning_vim_into_a_modern_python_ide"],"path":"old/turning_vim_into_a_modern_python_ide","contentHtml":"\u003cp\u003eTL;DR:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-shell\"\u003e\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003egit \u003cspan class=\"hljs-built_in\"\u003eclone\u003c/span\u003e https://github.com/sontek/dotfiles.git\u003c/span\u003e\n\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003e\u003cspan class=\"hljs-built_in\"\u003ecd\u003c/span\u003e dotfiles\u003c/span\u003e\n\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003e./install.sh vim\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eIntro\u003c/h1\u003e\n\u003cp\u003eBack in 2008, I wrote the article \u003ca href=\"http://sontek.net/python-with-a-modular-ide-vim\"\u003ePython with a modular IDE\n(Vim)\u003c/a\u003e. Years later, I\nhave people e-mailing me and commenting daily asking for more\ninformation, even though most of the information in it is outdated. Here\nis the modern way to work with Python and Vim to achieve the perfect\nenvironment.\u003c/p\u003e\n\u003cp\u003eBecause one of the most important parts about a development environment\nis the ability to easily reproduce across machines, we are going to\nstore our vim configuration in git:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-shell\"\u003e\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003e\u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e ~/.vim/\u003c/span\u003e\n\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003e\u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e ~/.vim/{\u003cspan class=\"hljs-built_in\"\u003eautoload\u003c/span\u003e,bundle}\u003c/span\u003e\n\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003e\u003cspan class=\"hljs-built_in\"\u003ecd\u003c/span\u003e ~/.vim/\u003c/span\u003e\n\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003egit init\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe purpose of the autoload directory is to automatically load the vim\nplugin \u003ca href=\"https://github.com/tpope/vim-pathogen\"\u003ePathogen\u003c/a\u003e, which we'll\nthen use to load all other plugins that are located in the bundle\ndirectory. So download pathogen and put it in your autoload folder.\u003c/p\u003e\n\u003cp\u003eYou'll need to add the following to your ~/.vimrc so that pathogen\nwill be loaded properly. Filetype detection must be off when you run the\ncommands so its best to execute them first:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-r\"\u003efiletype off\n\u003cspan class=\"hljs-built_in\"\u003ecall\u003c/span\u003e pathogen\u003cspan class=\"hljs-comment\"\u003e#runtime_append_all_bundles()\u003c/span\u003e\n\u003cspan class=\"hljs-built_in\"\u003ecall\u003c/span\u003e pathogen\u003cspan class=\"hljs-comment\"\u003e#helptags()\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow lets add all of the vim plugins we plan on using as submodules to\nour git repository:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-bash\"\u003egit submodule add http://github.com/tpope/vim-fugitive.git bundle/fugitive\ngit submodule add https://github.com/msanders/snipmate.vim.git bundle/snipmate\ngit submodule add https://github.com/tpope/vim-surround.git bundle/surround\ngit submodule add https://github.com/tpope/vim-git.git bundle/git\ngit submodule add https://github.com/ervandew/supertab.git bundle/supertab\ngit submodule add https://github.com/sontek/minibufexpl.vim.git bundle/minibufexpl\ngit submodule add https://github.com/wincent/Command-T.git bundle/command-t\ngit submodule add https://github.com/mitechie/pyflakes-pathogen.git\ngit submodule add https://github.com/mileszs/ack.vim.git bundle/ack\ngit submodule add https://github.com/sjl/gundo.vim.git bundle/gundo\ngit submodule add https://github.com/fs111/pydoc.vim.git bundle/pydoc\ngit submodule add https://github.com/vim-scripts/pep8.git bundle/pep8\ngit submodule add https://github.com/alfredodeza/pytest.vim.git bundle/py.test\ngit submodule add https://github.com/reinh/vim-makegreen bundle/makegreen\ngit submodule add https://github.com/vim-scripts/TaskList.vim.git bundle/tasklist\ngit submodule add https://github.com/vim-scripts/The-NERD-tree.git bundle/nerdtree\ngit submodule add https://github.com/sontek/rope-vim.git bundle/ropevim\ngit submodule init\ngit submodule update\ngit submodule foreach git submodule init\ngit submodule foreach git submodule update\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThats it! Now that we've got our vim configuration in git!\u003c/p\u003e\n\u003cp\u003eNow lets look at how to use each of these plugins to improve the power\nof vim:\u003c/p\u003e\n\u003ch1\u003eBasic Editing and Debugging\u003c/h1\u003e\n\u003ch2\u003eCode Folding\u003c/h2\u003e\n\u003cp\u003eLets first enable code folding. This makes it a lot easier to organize\nyour code and hide portions that you aren't interested in working on.\nThis is quite easy for Python, since whitespace is required.\u003c/p\u003e\n\u003cp\u003eIn your ~/.vimrc just add:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003eset \u003cspan class=\"hljs-attr\"\u003efoldmethod\u003c/span\u003e=indent\nset \u003cspan class=\"hljs-attr\"\u003efoldlevel\u003c/span\u003e=\u003cspan class=\"hljs-number\"\u003e99\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThen you will be able to be inside a method and type 'za' to open and\nclose a fold.\u003c/p\u003e\n\u003ch2\u003eWindow Splits\u003c/h2\u003e\n\u003cp\u003eSometimes code folding isn't enough; you may need to start opening up\nmultiple windows and working on multiple files at once or different\nlocations within the same file. To do this in vim, you can use these\nshortcuts:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-yaml\"\u003e\u003cspan class=\"hljs-attr\"\u003eVertical Split :\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eCtrl+w\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e+\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003ev\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eHorizontal Split:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eCtrl+w\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e+\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003es\u003c/span\u003e\n\u003cspan class=\"hljs-attr\"\u003eClose current windows:\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eCtrl+w\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e+\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003eq\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eI also like to bind Ctrl+\u0026#x3C;movement\u003e keys to move around the windows,\ninstead of using Ctrl+w + \u0026#x3C;movement\u003e:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003emap \u0026#x3C;c-j\u003e \u0026#x3C;c-w\u003ej\nmap \u0026#x3C;c-k\u003e \u0026#x3C;c-w\u003ek\nmap \u0026#x3C;c-l\u003e \u0026#x3C;c-w\u003el\nmap \u0026#x3C;c-h\u003e \u0026#x3C;c-w\u003eh\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/krj0l.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003ch2\u003eSnippets\u003c/h2\u003e\n\u003cp\u003eThe next tweak that really speeds up development is using snipmate.\nWe've already included it in our bundle/ folder so its already enabled.\nTry opening up a python file and typing 'def\u0026#x3C;tab\u003e'. It should stub\nout a method definition for you and allow you to tab through and fill\nout the arguments, doc string, etc.\u003c/p\u003e\n\u003cp\u003eI also like to create my own snippets folder to put in some custom\nsnippets:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-shell\"\u003e\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003e\u003cspan class=\"hljs-built_in\"\u003emkdir\u003c/span\u003e ~/.vim/snippets\u003c/span\u003e\n\u003cspan class=\"hljs-meta prompt_\"\u003e$ \u003c/span\u003e\u003cspan class=\"bash\"\u003evim ~/.vim/snippets/python.snippets\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003ePut this in the file:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003esnippet pdb\n \u003cspan class=\"hljs-keyword\"\u003eimport\u003c/span\u003e pdb; pdb.\u003cspan class=\"hljs-built_in\"\u003eset_trace\u003c/span\u003e()\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow you can type pdb\u0026#x3C;tab\u003e and it'll insert your breakpoint!\u003c/p\u003e\n\u003ch2\u003eTask lists\u003c/h2\u003e\n\u003cp\u003eAnother really useful thing is to mark some of your code as TODO or\nFIXME! I know we all like to think we write perfect code, but sometimes\nyou just have to settle and leave a note for yourself to come back\nlater. One of the plugins we included was the tasklist plugin that will\nallow us to search all open buffers for things to fix. Just add a\nmapping to open it in ~/.vimrc:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003emap \u0026#x3C;leader\u003etd \u0026#x3C;Plug\u003eTaskList\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow you can hit \u0026#x3C;leader\u003etd to open your task list and hit 'q' to\nclose it. You can also hit enter on the task to jump to the buffer and\nline that it is placed on.\u003c/p\u003e\n\u003ch2\u003eRevision History\u003c/h2\u003e\n\u003cp\u003eThe final basic editing tweak I suggest everyone start utilizing is the\nGundo plugin. It'll allow you to view diff's of every save on a file\nyou've made and allow you to quickly revert back and forth:\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/2NrPS.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003cp\u003eJust bind a key in your .vimrc to toggle the Gundo window:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ruby\"\u003emap \u0026#x3C;leader\u003eg \u003cspan class=\"hljs-symbol\"\u003e:GundoToggle\u0026#x3C;CR\u003e\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eSyntax Highlighting and Validation\u003c/h1\u003e\n\u003cp\u003eSimply enable syntax highlighting in your ~/.vimrc:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-vbnet\"\u003esyntax \u003cspan class=\"hljs-keyword\"\u003eon\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\" syntax highlighing\nfiletype on \"\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003etry\u003c/span\u003e \u003cspan class=\"hljs-keyword\"\u003eto\u003c/span\u003e detect filetypes\nfiletype plugin indent \u003cspan class=\"hljs-keyword\"\u003eon\u003c/span\u003e \u003cspan class=\"hljs-string\"\u003e\" enable loading indent file for filetype\n\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eBecause we enabled pyflakes when we added it as a submodule in\n~/.vim/bundle, it will notify you about unused imports and invalid\nsyntax. It will save you a lot of time saving and running just to find\nout you missed a colon. I like to tell it not use the quickfix window:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003elet g:\u003cspan class=\"hljs-attr\"\u003epyflakes_use_quickfix\u003c/span\u003e = \u003cspan class=\"hljs-number\"\u003e0\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/ZfjFe.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003ch2\u003ePep8\u003c/h2\u003e\n\u003cp\u003eThe final plugin that really helps validate your code is the pep8\nplugin, it'll make sure your code is consistent across all projects.\nAdd a key mapping to your ~/.vimrc and then you'll be able to jump to\neach of the pep8 violations in the quickfix window:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003elet g:\u003cspan class=\"hljs-attr\"\u003epep8_map\u003c/span\u003e=\u003cspan class=\"hljs-string\"\u003e'\u0026#x3C;leader\u003e8'\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/VU9AB.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003ch1\u003eTab Completion and Documentation\u003c/h1\u003e\n\u003cp\u003eVim has many different code completion options. We are going to use the\nSuperTab plugin to check the context of the code you are working on and\nchoose the best for the situation. We've already enabled the SuperTab\nplugin in the bundle/ folder, so we just have to configure it to be\ncontext sensitive and to enable omni code completion in your ~/.vimrc:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003eau FileType python set \u003cspan class=\"hljs-attr\"\u003eomnifunc\u003c/span\u003e=pythoncomplete\u003cspan class=\"hljs-comment\"\u003e#Complete\u003c/span\u003e\nlet g:\u003cspan class=\"hljs-attr\"\u003eSuperTabDefaultCompletionType\u003c/span\u003e = \u003cspan class=\"hljs-string\"\u003e\"context\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eNow we just enable the menu and pydoc preview to get the most useful\ninformation out of the code completion:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003eset \u003cspan class=\"hljs-attr\"\u003ecompleteopt\u003c/span\u003e=menuone,longest,preview\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/g4lxP.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003cp\u003eWe also enabled the pydoc plugin at the beginning with all the\nsubmodules; that gives us the ability to hit \u0026#x3C;leader\u003epw when our\ncursor is on a module and have a new window open with the whole\ndocumentation page for it.\u003c/p\u003e\n\u003ch1\u003eCode Navigation\u003c/h1\u003e\n\u003ch2\u003eBuffers\u003c/h2\u003e\n\u003cp\u003eThe most important part about navigating code within vim, is to\ncompletely understand how to use buffers. There is no reason to use\ntabs. Open files with :e \u0026#x3C;filename\u003e to place in a buffer. We already\ninstalled the minibufexpl plugin, so you will already visually see every\nbuffer opened. You can also get a list of them doing :buffers.\u003c/p\u003e\n\u003cp\u003eYou can switch between the buffers using b\u0026#x3C;number\u003e, such as :b1 for\nthe first buffer. You can also use its name to match, so you can type :b\nmod\u0026#x3C;tab\u003e to autocomplete opening the models.py buffer. You need to\nmake sure you are using the minibufexpl from my github since it has\npatches that make it much better to work with.\u003c/p\u003e\n\u003cp\u003eTo close a buffer you use :bd or :bw.\u003c/p\u003e\n\u003ch2\u003eFuzzy Text File Search\u003c/h2\u003e\n\u003cp\u003eTo make finding and opening files within your project even easier, we\nare going to use the command-t plugin. It does have some parts that need\nto be compiled, so its not already installed by adding it as a\nsubmodule. Go to your ~/.vim/bundle/command-t folder and run 'rake\nmake'. Yes you need ruby installed. By default, command-t is bound to\n\u0026#x3C;leader\u003et. This will use fuzzy text matching to find any file in your\nproject.\u003c/p\u003e\n\u003cp\u003eIt also supports searching only through opened buffers, instead of files\nusing \u0026#x3C;leader\u003eb.\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/hUcSl.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003ch2\u003eFile Browser\u003c/h2\u003e\n\u003cp\u003eNERD Tree is a project file browser. I must admit I used this heavily\nback when I was migrating from Visual Studio and used to the Solution\nExplorer, but I rarely use it anymore. Command-T is usually all you'll\nneed. It is useful when you are getting to know a new codebase for the\nfirst time though. Lets bind a shortcut key for opening it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ruby\"\u003emap \u0026#x3C;leader\u003en \u003cspan class=\"hljs-symbol\"\u003e:NERDTreeToggle\u0026#x3C;CR\u003e\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/R4ZzQ.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003ch2\u003eRefactoring and Go to definition\u003c/h2\u003e\n\u003cp\u003eRopevim is also a great tool that will allow you to navigate around your\ncode. It supports automatically inserting import statements, goto\ndefinition, refactoring, and code completion. You'll really want to\nread up on everything it does, but the two big things I use it for is to\njump to function or class definitions quickly and to rename things\n(including all their references).\u003c/p\u003e\n\u003cp\u003eFor instance, if you are using django and you place your cursor over the\nclass models.Model you reference and then called :RopeGotoDefintion, it\nwould jump you straight to the django library to that class definition.\nWe already have it installed in our bundles, so we bind it to a key to\nuse it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ruby\"\u003emap \u0026#x3C;leader\u003ej \u003cspan class=\"hljs-symbol\"\u003e:RopeGotoDefinition\u0026#x3C;CR\u003e\u003c/span\u003e\nmap \u0026#x3C;leader\u003er \u003cspan class=\"hljs-symbol\"\u003e:RopeRename\u0026#x3C;CR\u003e\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2\u003eSearching\u003c/h2\u003e\n\u003cp\u003eThe final tool that really speeds up navigating your code is the Ack\nplugin. Ack is similar to grep, but much better in my opinion. You can\nfuzzy text search for anything in your code (variable name, class,\nmethod, etc) and it'll give you a list of files and line numbers where\nthey are defined so you can quickly cycle through them. Just bind the\nsearching to a key:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-php-template\"\u003e\u003cspan class=\"xml\"\u003enmap \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eleader\u003c/span\u003e\u003e\u003c/span\u003ea \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eEsc\u003c/span\u003e\u003e\u003c/span\u003e:Ack!\n\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eWe use ! at the end of it so it doesn't open the first result\nautomatically.\u003c/p\u003e\n\u003ch1\u003eIntegration with Git\u003c/h1\u003e\n\u003cp\u003eWe installed 2 plugins, git.vim and fugitive, that give us all the\nintegration we need. Git.vim will provide us syntax highlighting for git\nconfiguration files; fugitive provides a great interface for interacting\nwith git including getting diffs, status updates, committing, and moving\nfiles.\u003c/p\u003e\n\u003cp\u003eFugitive also allows you to view what branch you are working in directly\nfrom vim. Add this to your statusline in ~/.vimrc:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-shell\"\u003e\u003cspan class=\"hljs-meta prompt_\"\u003e%\u003c/span\u003e\u003cspan class=\"bash\"\u003e{fugitive\u003cspan class=\"hljs-comment\"\u003e#statusline()}\u003c/span\u003e\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe big commands you need to know:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eGblame\u003c/strong\u003e: This allows you to view a line by line comparison of who\nthe last person to touch that line of code is.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eGwrite\u003c/strong\u003e: This will stage your file for commit, basically doing\ngit add \u0026#x3C;filename\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eGread\u003c/strong\u003e: This will basically run a git checkout \u0026#x3C;filename\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eGcommit\u003c/strong\u003e: This will just run git commit. Since its in a vim\nbuffer, you can use keyword completion (Ctrl-N), like\ntest_all\u0026#x3C;Ctrl-N\u003e to find the method name in your buffer and\ncomplete it for the commit message. You can also use + and - on the\nfilenames in the message to stage/unstage them for the commit.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/NuRRj.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003ch1\u003eTest Integration\u003c/h1\u003e\n\u003ch2\u003edjango nose\u003c/h2\u003e\n\u003cp\u003eTest runner integration really depends on the testing library you are\nusing and what type of tests you are running but we included a great\ngeneric plugin called MakeGreen that executes off of vim's makeprg\nvariable. So for instance, if you are using django with django-nose you\ncould define a shortcut key in your ~/.vimrc like this:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-arduino\"\u003emap \u0026#x3C;leader\u003edt :set makeprg=python\\ manage.py\\ test\\|:call \u003cspan class=\"hljs-built_in\"\u003eMakeGreen\u003c/span\u003e()\u0026#x3C;CR\u003e\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis will just give you a green bar at the bottom of vim if your test\npassed or a red bar with the message of the failed test if it doesn't.\nVery simple.\u003c/p\u003e\n\u003ch2\u003epy.test\u003c/h2\u003e\n\u003cp\u003eI also included the py.test vim plugin for those who prefer it. This\nplugin has a lot more functionality including executing individual tests\nby class, file, or method. You can also cycle through the individual\nassertion errors. I have the following bindings:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-php-template\"\u003e\u003cspan class=\"xml\"\u003e\" Execute the tests\nnmap \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003esilent\u003c/span\u003e\u003e\u003c/span\u003e\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eLeader\u003c/span\u003e\u003e\u003c/span\u003etf \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eEsc\u003c/span\u003e\u003e\u003c/span\u003e:Pytest file\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eCR\u003c/span\u003e\u003e\u003c/span\u003e\nnmap \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003esilent\u003c/span\u003e\u003e\u003c/span\u003e\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eLeader\u003c/span\u003e\u003e\u003c/span\u003etc \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eEsc\u003c/span\u003e\u003e\u003c/span\u003e:Pytest class\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eCR\u003c/span\u003e\u003e\u003c/span\u003e\nnmap \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003esilent\u003c/span\u003e\u003e\u003c/span\u003e\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eLeader\u003c/span\u003e\u003e\u003c/span\u003etm \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eEsc\u003c/span\u003e\u003e\u003c/span\u003e:Pytest method\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eCR\u003c/span\u003e\u003e\u003c/span\u003e\n\" cycle through test errors\nnmap \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003esilent\u003c/span\u003e\u003e\u003c/span\u003e\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eLeader\u003c/span\u003e\u003e\u003c/span\u003etn \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eEsc\u003c/span\u003e\u003e\u003c/span\u003e:Pytest next\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eCR\u003c/span\u003e\u003e\u003c/span\u003e\nnmap \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003esilent\u003c/span\u003e\u003e\u003c/span\u003e\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eLeader\u003c/span\u003e\u003e\u003c/span\u003etp \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eEsc\u003c/span\u003e\u003e\u003c/span\u003e:Pytest previous\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eCR\u003c/span\u003e\u003e\u003c/span\u003e\nnmap \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003esilent\u003c/span\u003e\u003e\u003c/span\u003e\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eLeader\u003c/span\u003e\u003e\u003c/span\u003ete \u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eEsc\u003c/span\u003e\u003e\u003c/span\u003e:Pytest error\u003cspan class=\"hljs-tag\"\u003e\u0026#x3C;\u003cspan class=\"hljs-name\"\u003eCR\u003c/span\u003e\u003e\u003c/span\u003e\n\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003cimg src=\"http://i.imgur.com/RAE7v.png\" alt=\"image\"\u003e\u003c/p\u003e\n\u003ch1\u003eVirtualenv\u003c/h1\u003e\n\u003cp\u003eVim doesn't realize that you are in a virtualenv so it wont give you\ncode completion for libraries only installed there. Add the following\nscript to your ~/.vimrc to fix it:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003e\" Add the virtualenv's site-packages to vim path\npy \u0026#x3C;\u0026#x3C; EOF\nimport os.path\nimport sys\nimport vim\nif 'VIRTUAL_ENV' in os.environ:\n \u003cspan class=\"hljs-attr\"\u003eproject_base_dir\u003c/span\u003e = os.environ[\u003cspan class=\"hljs-string\"\u003e'VIRTUAL_ENV'\u003c/span\u003e]\n sys.path.insert(0, project_base_dir)\n \u003cspan class=\"hljs-attr\"\u003eactivate_this\u003c/span\u003e = os.path.join(project_base_dir, \u003cspan class=\"hljs-string\"\u003e'bin/activate_this.py'\u003c/span\u003e)\n execfile(activate_this, dict(\u003cspan class=\"hljs-attr\"\u003e__file__\u003c/span\u003e=activate_this))\nEOF\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eDjango\u003c/h1\u003e\n\u003cp\u003eThe only true django tweak I make is before I open vim I'll export the\nDJANGO_SETTINGS_MODULE environment so that I get code completion for\ndjango modules as well:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class=\"hljs language-ini\"\u003eexport \u003cspan class=\"hljs-attr\"\u003eDJANGO_SETTINGS_MODULE\u003c/span\u003e=project.settings\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1\u003eRandom Tips\u003c/h1\u003e\n\u003cp\u003eIf you want to find a new color scheme just go to\n\u003ca href=\"http://code.google.com/p/vimcolorschemetest/\"\u003ehttp://code.google.com/p/vimcolorschemetest/\u003c/a\u003e to preview a large\nselection.\u003c/p\u003e\n\u003cp\u003eJohn Anderson \u0026#x3C;\u003ca href=\"mailto:sontek@gmail.com\"\u003esontek@gmail.com\u003c/a\u003e\u003e 2011\u003c/p\u003e","category":"Development","date":"2011-05-07T00:00:00Z","tags":["Vim","Python"],"title":"Turning Vim into a modern Python IDE"}]}}},"__N_SSG":true},"page":"/blog/tags/[...id]","query":{"id":["Linux"]},"buildId":"JtefKWhIxZ_l05YEYkE82","isFallback":false,"gsp":true,"scriptLoader":[]}</script></body></html>