AWS From Scratch with Terraform - Setting up your Root Account for IaC  with Terraform Cloud and Github actions.

Wiping an AWS Account with aws-nuke

Running a kubernetes cluster locally with kubeadm

Running a kubernetes cluster locally with kind

<a href="https://asdf-vm.com/">asdf</a> is a general purpose version manager that
can manage versions of most programming language runtimes through a set
of plugins.
<iframe width="854" height="480" src="https://www.youtube.com/embed/RTaqWRj-6Lg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
With micro-services being all the rage and the ever changing landscape
of the development world, it is rare to utilize a single version of
language runtime. Even when you want to upgrade from one to the other
you will need both usable on your system at the same time.
I have used tools like <code>pyenv</code> and <code>nvm</code> in the past when I needed to change
versions depending on which project I am contributing to. But with <code>asdf</code>
you have one tool to rule them all!
<h2>Getting Started</h2>
The first thing you need to do when working with <code>asdf</code> is grab the
plugins for the languages you are interested in working with. You can list
what plugins are available:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">> asdf plugin list all
golang *https://github.com/kennyp/asdf-golang.git
golangci-lint https://github.com/hypnoglow/asdf-golangci-lint.git
nodejs *https://github.com/asdf-vm/asdf-nodejs.git
poetry *https://github.com/asdf-community/asdf-poetry.git
python *https://github.com/danhper/asdf-python.git
yarn *https://github.com/twuni/asdf-yarn.git</code></pre></figure>
On the left will be the name of the plugin and on the right will be the repository
where it lives. It will be marked with an asterisk if you already have it installed.
To install a plugin you say <code>asdf plugin add &#x3C;plugin></code> to get it installed. You can
also provide the repository where you want it pulled from, for example:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">> asdf plugin add nodejs https://github.com/asdf-vm/asdf-nodejs.git
> asdf plugin add python https://github.com/danhper/asdf-python.git</code></pre></figure>
This will not give you any version of those languages, it is only installing the
plugin that knows how to work with those languages. You are ready to pull down
any versions you want at that point:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">> asdf install nodejs 14.19.0
> asdf install python 3.9.10</code></pre></figure>
Once you have the versions installed you will be able to view them like this:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">> asdf list
golang
 1.17.7
nodejs
 --help
 12.22.10
 14.19.0
 16.14.0
 17.5.0
poetry
 1.1.13
python
 3.9.10
yarn
 1.22.17</code></pre></figure>
<h2>Using the installed languages</h2>
To activate a specific version of a language you have three options:
<h3>Make it global</h3>
You can make it global, meaning when you run the tool like <code>python</code> it will use
this version for the system:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">> asdf global python 3.9.10</code></pre></figure>
<h3>Make it local</h3>
You can make it local, which means it will generate a file in the current
directory named <code>.tool-versions</code> and so whenever you change into a directory
it will activate the versions defined in there.
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">> asdf local nodejs 12.22.10
> cat .tool-versions 
nodejs 12.22.10</code></pre></figure>
The great thing about this is you can commit that file to git and then anyone
who checks out the project and uses <code>asdf</code> will have the same versions activated!
<h3>Temporary</h3>
If you want to activate a version of a language temporarily you can swap to it
for the current shell:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">> asdf shell golang 1.17.7
> env|grep -i ASDF
ASDF_GOLANG_VERSION=1.17.7</code></pre></figure>
It sets an environment variable that will have preference over the file. If you
ever wonder what versions a directory is using you can run:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">> asdf current
golang ______ No version set. Run "asdf &#x3C;global|shell|local> golang &#x3C;version>"
nodejs 12.22.10 .tool-versions
poetry ______ No version set. Run "asdf &#x3C;global|shell|local> poetry &#x3C;version>"
python 3.9.10 .tool-versions
yarn 1.22.17 .tool-versions</code></pre></figure>
<h2>Conclusion</h2>
<a href="https://asdf-vm.com/">asdf</a> is an AWESOME tool to utilize if you find yourself using many
different languages or many different versions of the same language. You should check it out
and see if it can improve your workflow.

Use asdf to manage Python, NodeJS, GoLang and more!

I'm currently teaching my wife to code and one of the problems that we
worked on to teach her some fundamental programming concepts was
re-implementing the caesar cipher in python. It was fun not only to code
but to also start sending each other "secret" messages!
The caesar cipher is a rather simple encoding, you just shift the
alphabet a certain amount of characters. For example, if you are using a
shift of 2:
<pre><code>a => c
b => d
y => a
z => b
</code></pre>
Using this as an interview type question would provide a few interesting
problems and give you a good perspective on how good a developers
problem solving skills are and how knowledgeable they are in the
language of their choice.
The first issue is to handle the beginning and end of the alphabet, if
you are encoding 'z' then you will have to start your shift on a. The
second problem is to only encode letters since there was no ascii table
to define in what order characters are shifted back in those times.
Without using too much of the built in python niceties you could do
something similar to this:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="python" data-theme="github-dark-dimmed"><code data-language="python" data-theme="github-dark-dimmed" style="display: grid;">#!/usr/bin/python
def decode_shift_letter(current_ord, start, end, shift):
 if current_ord - shift &#x3C; start:
 new_ord = (current_ord + 26) - shift
 return chr(new_ord)
 else:
 return chr(current_ord-shift)
 
def encode_shift_letter(current_ord, start, end, shift):
 if current_ord + shift > end:
 new_ord = (current_ord - 26) + shift
 return chr(new_ord)
 else:
 return chr(current_ord+shift)
 
def decode(input, shift):
 return modify_input(input, shift, decode_shift_letter)
 
def encode(input, shift):
 return modify_input(input, shift, encode_shift_letter)
 
def modify_input(input, shift, shift_letter):
 new_sentence = ''
 
 for letter in input:
 # we only encode letters, random characters like +!%$ are not encoded.
 # Lower and Capital letters are not stored near each other on the
 # ascii table
 lower_start = ord('a')
 lower_end = ord('z')
 upper_start = ord('A')
 upper_end = ord('Z')
 current_ord = ord(letter)
 
 if current_ord >= lower_start and current_ord &#x3C;= lower_end:
 new_sentence += shift_letter(current_ord, lower_start, lower_end, shift)
 elif current_ord >= upper_start and current_ord &#x3C;= upper_end:
 new_sentence += shift_letter(current_ord, upper_start, upper_end, shift)
 else:
 new_sentence += letter
 
 return new_sentence
 
 
def get_shift():
 try:
 shift = int(raw_input('What shift would you like to use?\n'))
 except ValueError:
 print 'Shift must be a number'
 shift = get_shift()
 
 if not (shift > 0 and shift &#x3C;= 25):
 print 'Shift must be between 1 and 25'
 shift = get_shift()
 
 return shift
 
def main():
 try:
 task = int(raw_input('1) Encode \n'+ \
 '2) Decode \n'))
 except ValueError:
 print 'Invalid task, try again!'
 main()
 
 shift = get_shift()
 input = raw_input('What message would you like to %s\n' % ('Encode' if task == 1 else 'Decode'))
 
 if task == 1:
 print encode(input, shift)
 elif task == 2:
 print decode(input, shift)
 
if __name__ == '__main__':
 main()</code></pre></figure>
This would prove that you are a decent problem solver and have enough of
the language to get things done but if you want to prove you have
mastered the python language you might take advantage of some slicing
and some methods out of the string module and change your code to look
something like:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="python" data-theme="github-dark-dimmed"><code data-language="python" data-theme="github-dark-dimmed" style="display: grid;">from string import letters, maketrans
 
def decode(input, shift):
 return modify_input(input, -shift)
 
def encode(input, shift):
 return modify_input(input, shift)
 
def modify_input(input, shift):
 trans = maketrans(letters, letters[shift:] + letters[:shift])
 return input.translate(trans)</code></pre></figure>
Do get more information on string.letters and string.maketrans you can
visit their documentation
[here](<a href="http://docs.python.org/library/string.html">http://docs.python.org/library/string.html</a>)

Caesar Cipher in Python

A fun interview question some developers like to ask is to have you
convert ascii characters to an integer without using built in methods
like string.atoi or int().
So using python the obvious ways to convert a string to an integer are
these:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="python" data-theme="github-dark-dimmed"><code data-language="python" data-theme="github-dark-dimmed" style="display: grid;">>>> int('1234')
1234
>>> import string
>>> string.atoi('1234')
1234</code></pre></figure>
The interesting thing here is finding out where on the ascii character
table the number is. Luckily python has this already built in with the
ord method:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="python" data-theme="github-dark-dimmed"><code data-language="python" data-theme="github-dark-dimmed" style="display: grid;">>>> help(ord)
 
ord(...)
 ord(c) -> integer
 
 Return the integer ordinal of a one-character string.
 
>>> ord('1')
49
>>> ord('2')
50</code></pre></figure>
You can see that the numbers are grouped together on the ascii table, so
you just have to grab '0' as the base and subtract the rest:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="python" data-theme="github-dark-dimmed"><code data-language="python" data-theme="github-dark-dimmed" style="display: grid;">>>> ord('1')-ord('0')
1</code></pre></figure>
So if we have the string '1234', we can get each of the individual
numbers by looping over it:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="python" data-theme="github-dark-dimmed"><code data-language="python" data-theme="github-dark-dimmed" style="display: grid;">>>> num_string = '1234'
>>> num_list = []
>>> base = ord('0')
>>> for num in num_string:
... num_list.append(ord(num) - base)
...
>>> print num_list
[1, 2, 3, 4]</code></pre></figure>
but now how to we combine all these together to get 1234? You can't
just add them up because you'll just get 1+2+3+4 = 10.
So, we have to get 1000 + 200 + 30 + 4, which is a simple problem to
solve. Its just number times 10 to the nth power, so the final solution
is:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="python" data-theme="github-dark-dimmed"><code data-language="python" data-theme="github-dark-dimmed" style="display: grid;">num = '1234'
new_num = 0
base = ord('0')
 
for i,n in enumerate(reversed(num)):
 new_num += (ord(n) - base) * (10**i)
 
print new_num</code></pre></figure>
This code is a little verbose though, lets make it a dirty nasty one
liner!
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="python" data-theme="github-dark-dimmed"><code data-language="python" data-theme="github-dark-dimmed" style="display: grid;">>>> sum([(ord(n)-ord('0')) * (10 ** i) for i,n in enumerate(reversed('1234'))])
1234</code></pre></figure>

Convert a string to an integer in Python

eksclustergames.com walk through!

Automate project workflows with the command runner Just!

This article will show you how to use libvirt to create base images that
can be uploaded to OpenStack.
<h1>Why would you want to do this?</h1>
Linux distributions like Fedora and Ubuntu already ship "cloud" images
and most providers also have their own custom images for you to use, but
I find it much more comforting to have full control of the software that
is installed and I like the ability to easily apply new security patches
to base images.
I wouldn't use images to replace config management (CM) with something
like <a href="http://www.saltstack.com/">Salt</a> or
<a href="http://www.ansible.com/">Ansible</a> but they are nice to give sane system
defaults in things like <code>grub.conf</code>, <code>sysctl.conf</code>, and shipping a Chef
or Salt agent so that your CM engine can communicate with your server
right away.
<h1>Setting up your environment</h1>
The first thing you need to do is get a minimal install disk for the
Linux distribution you want to use. I prefer using Fedora netinst disks
but another popular option is Ubuntu Server.
To get the latest Fedora here, you can choose "netinst" under Direct
Downloads: <a href="http://fedoraproject.org/en/get-fedora-all">http://fedoraproject.org/en/get-fedora-all</a>
To get the latest Ubuntu you can go here:
<a href="http://www.ubuntu.com/download/server">http://www.ubuntu.com/download/server</a>
Once you have acquired your distribution of choice you just need to
verify that you have <code>virt-install</code> and <code>virt-viewer</code> installed:
Fedora:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">yum install virt-install virt-viewer</code></pre></figure>
Ubuntu:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">apt-get install virtinst virt-viewer</code></pre></figure>
If you prefer a graphical user interface, you may use <code>virt-manager</code>
instead, but I try to keep everything in the CLI; that way it can be
repeated easily.
<h1>Preparing your disk</h1>
Now that you have a base ISO and the tools necessary, let's get started
by creating a disk to install the virtual server into. Resizing an image
isn't an impossible task but it is much easier to choose a reasonably
sized disk for the purpose it will be used for.
I primarily use 8 GB disks -- that way we can fit all the system
components required as well as our own web applications. Any large files
should be placed in a SAN or something like Dreamhost's dreamobjects.
The other big decision you must make upfront is what disk format you
want to use -- the trade-off is disk space vs performance. The two
primary formats are qcow2 (QEMU Copy on Write) and Raw. qcow2 is great
if you have limited disk space and don't want to allocate the full 8 GB
up front. Raw is preferred if you want the best performance.
If you choose qcow2, you'll also need to make sure you have <code>qemu-img</code>:
Fedora:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">yum install qemu-img</code></pre></figure>
Ubuntu:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">apt-get install qemu-utils</code></pre></figure>
Create a raw disk:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">fallocate -l 8192M server.img</code></pre></figure>
Create a qcow2 disk:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">qemu-img create -f qcow2 server.qcow2 8G</code></pre></figure>
<h1>Installing your distribution onto the disk</h1>
We will use the <code>virt-install</code> command to get the distribution installed
onto the disk image.
To install Fedora on a qcow2 disk image:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">virt-install --name base_server --ram 1024 --cdrom=./Fedora-20-x86_64-netinst.iso \
--disk path=./server.qcow2,format=qcow2</code></pre></figure>
To install Ubuntu Server on a raw disk image:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">virt-install --name base_server --ram 1024 --cdrom=./ubuntu-12.04.4-server-amd64.iso \
--disk path=./server.img,format=raw</code></pre></figure>
You should follow the standard install steps that you normally would
when setting up your distribution. But here are some tips for each:
Fedora:
<ul>
<li>Choose minimal install -- by default it selects "GNOME".</li>
</ul>
Ubuntu:
<ul>
<li>
Be sure to select OpenSSH server -- it won't install it by
default.
</li>
<li>
On Ubuntu 12.04, there is a bug that makes it hang after running
<code>fsck</code>. You will need to edit grub to get it to boot, hit _<a href="">e</a> at
the boot prompt and add "nomodeset" on the linux line. You will
know that you need to do this if your boot hangs on fsck:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">fsck from util-linux 2.20.1
/dev/mapper/ubuntu--vg-root: clean, 57106/441504 files, 286779/1764352 blocks
/dev/sda1: clean, 230/62248 files, 39833/248832 blocks</code></pre></figure>
</li>
</ul>
<h1>Preparing image for openstack</h1>
To prepare a virtual machine for the cloud, you will need to install the
<code>cloud-init</code> package, which allows the cloud providers to inject certain
system settings when creating servers based on the image. These are
things like hostname and ssh keys.
On Fedora:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">yum install cloud-init</code></pre></figure>
On Ubuntu:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">apt-get install cloud-init</code></pre></figure>
Then you need to just configure <code>cloud-init</code> by editing
<code>/etc/cloud/cloud.cfg</code> and update the <code>datasources_list</code> section to
include EC2. OpenStack uses EC2 metadata for <code>cloud-init</code>.
You should also verify the user setting in this same config and define
the user you plan to use, it will be where the <code>authorized_keys</code> file is
setup for when the cloud provider injects your SSH key into the server.
<code>cloud-init</code> will not create the user for you, it will just assign the
SSH keypair and reset the password. So make sure the user defined in
<code>cloud.cfg</code> is also created on the system.
Once you have your <code>cloud-init</code> settings the way you want them, just
shutdown and run the <code>virt-sysprep</code> command.
On the guest machine:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">shutdown -h now</code></pre></figure>
On the host machine:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">virt-sysprep -d base_server</code></pre></figure>
<h1>Uploading your image to OpenStack</h1>
Using the glance API it is very straightforward to upload the image to
OpenStack. Just run the following command:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">glance image-create --name base_server --disk-format=qcow2 \
--container-format=bare --is-public=True --file server.qcow2 --progress</code></pre></figure>
Once the image upload completes you will be able to use it immediately
within nova. You can reference it by name or by the id from [glance
image-list]{.title-ref}.
To create your first instance from the image:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">nova boot --flavor m1.tiny --image base_server --key-name devops \
--security-groups free_for_all test_server</code></pre></figure>
Obviously the security groups, key name, and flavors are based on your
installation of OpenStack but can all easily be queried from the nova
API:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="bash" data-theme="github-dark-dimmed"><code data-language="bash" data-theme="github-dark-dimmed" style="display: grid;">nova flavor-list
nova secgroup-list
nova keypair-list</code></pre></figure>
And you are done! You'll be able to re-use your new image as a base for
all new instances you spin up in openstack!

Preparing custom images for OpenStack

I have been living in Puerto Rico for 4 years but two of those have been COVID and so I have not been able to practice Spanish as much as I would like. So to speed up my learning I have decided I want to watch a lot of spanish speaking television to start training my ears, but to do this I need a baseline of words I understand to be able to even know what they are saying!
Learning through apps like Duolingo, Drops, etc start with weird topics like vegetables that do not get you to a very good baseline for actually understanding daily conversations, so I think consuming TV is a better use of my time.
<h2>Subtitles</h2>
I have decided the way to understand what the best words to study are is to download every subtitle for every episode of a show I want to watch and then count each word. The more a word is spoken the more important it is for me to know it since I will be hearing it a lot in the show.
I am going to download subtitles from Netflix. Subtitles in Netflix are in WebVTT format, which looks like this:
<pre><code>248
00:17:58.285 --> 00:18:01.163 position:50.00%,middle align:middle size:80.00% line:79.33% 
Yo de verdad espero que ustedes
me vean como una amiga, ¿mmm?

249
00:18:01.247 --> 00:18:02.539 position:50.00%,middle align:middle size:80.00% line:84.67% 
No como una madrastra.

250
00:18:04.250 --> 00:18:06.127 position:50.00%,middle align:middle size:80.00% line:84.67% 
Yo nunca te vi como una madrastra.
</code></pre>
It gives you a start time, end time, and the text on the screen. So my first process was parsing this format and just turning it into a list of words using https://github.com/glut23/webvtt-py.
<h3>Dummy parsing</h3>
What I basically did was <code>text.split(" ")</code> and started counting the words. This approach was quick and painless but it had a few downsides. Some words look the same when in reality they are not and so this meant I would have to study every meaning of a word even if it was more rare.
An example of this is the word "como", you can say:
<ul>
<li>Haz como te digo: "Do as I say", where como means "as"</li>
<li>como tacos todos los dias: "I eat tacos every day", where como is a conjugated form of the verb "to eat"</li>
</ul>
I need to know which version of a word is being used so I can count it properly.
<h3>Regular Expressions are always the answer</h3>
I could not figure out what the word was without it being in a complete sentence, but subtitles are fragments. They are split up into timings for displaying on the screen but they do not include entire sentences. For example, it might look like this:
<pre><code>23
00:01:21.960 --> 00:01:23.520 position:50.00%,middle align:middle size:80.00% line:84.67% 
Solo las que luchan por ellos

24
00:01:23.680 --> 00:01:25.680 position:50.00%,middle align:middle size:80.00% line:84.67% 
consiguen sus sueños.
</code></pre>
I want to detect the start of a sentence and the end of a sentence and then combine it, so that you end up with "Solo las que luchan por ellos consiguen sus sueños.". My first thought was a regular expression on punctuation. This worked well most of the time but there were enough exceptions to the rule that it broke often and generated a lot of broken sentences:
<ul>
<li>Abbreviations like "EE. UU" for estados unidos (united states)</li>
<li>Ellipsis</li>
</ul>
Splitting on spaces also did not work for identifying the parts of speech since I needed the context around the word.
<center>
<img src="/images/posts/learning_spanish/regex-extraction.png">
</center>
<h2>Natural Language Processing</h2>
So to solve my pain I decided to grab https://spacy.io/ and do some NLP on the subtitles so that I could identify the proper parts of speech and get an accurate representation of the words I needed to learn.
The way spaCy works is you can send it a sentence and it will return you a set of tokens:
<pre><code>>>> import spacy
>>> nlp = spacy.load("es_core_news_sm")
>>> [x.pos_ for x in nlp("Hola, como estas?")]
['PROPN', 'PUNCT', 'SCONJ', 'PRON', 'PUNCT']
</code></pre>
So now I could identify the parts of speech and pull sentences together through end of sentence punctuation. The first thing I did was generate a CSV of sentences that looked like this:
<table>
<tbody><tr>
<th>sentence</th>
<th>start</th>
<th>end</th>
<th>show</th>
<th>file</th>
</tr>
<tr>
<td>Si no, le voy a cortar todos los deditos</td>
<td>00:00:20.605</td>
<td>00:00:24.125</td>
<td>El marginal</td>
<td>El marginal S02E02 WEBRip Netflix es[cc].vtt</td>
</tr>
</tbody></table>
Once I had a CSV of sentences I could send those back through spaCy for NLP and then start counting words, to generate another CSV:
<table>
<tbody><tr>
<th>word</th>
<th>pos</th>
<th>show</th>
<th>file</th>
</tr>
<tr>
<td>a</td>
<td>ADP</td>
<td>El marginal</td>
<td>El marginal S02E02 WEBRip Netflix es[cc].vtt</td>
</tr>
<tr>
<td>cortar</td>
<td>VERB</td>
<td>El marginal</td>
<td>El marginal S02E02 WEBRip Netflix es[cc].vtt</td>
</tr>
<tr>
<td>todos</td>
<td>PRON</td>
<td>El marginal</td>
<td>El marginal S02E02 WEBRip Netflix es[cc].vtt</td>
</tr>
</tbody></table>
From there I had all the data I needed! So now it was time to start doing some data analysis!
<h2>Data analysis</h2>
Using a jupyter notebook ( https://jupyter.org/ ) I grabbed pandas ( https://pandas.pydata.org/ ) and read in my CSVs to start analyzing the results.
<pre><code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
pd.set_option('display.max_rows', 1000)
words = pd.read_csv('word_data.csv.gz', compression='gzip', delimiter=',')
</code></pre>
The words dataframe is built up out of the second table I showed above with just words and their parts of speech. I started off grouping the dataset by the word so I could get a count for how many times it was spoken in every series I parsed:
<pre><code>grouped_result = (words.groupby(words.word).size() 
 .sort_values(ascending=False) 
 .reset_index(name='count')
 .drop_duplicates(subset='word'))

grouped_result.head(300)
</code></pre>
Which returned a list of words and their count:
<pre><code>	word	count
0	que	94430
1	no	75931
2	a	70968
3	de	67982
4	ser	64226
5	la	52143
6	y	44390
7	estar	37819
8	el	35920
</code></pre>
Now I wanted to identify where my diminishing returns would be. Is there a set of words that I must learn because they are spoken so often that I would not understand a conversation if they were not in my vocabulary?
<center>
<img src="/images/posts/learning_spanish/diminishing_returns.png">
</center>
As you can see in this chart, the usage for words drops off at around the ~200 mark. So there are basically 150 words I must know and then the rest are equally important. I was not quite happy with this because some parts of speech are higher priority than others, for example I think having a strong understanding of the popular verbs will go a long way. So I also wanted to identify what are the most important verbs to learn:
<pre><code>grouped_verbs = (words[words.pos == 'VERB'].groupby(['word', 'pos']).size() 
 .sort_values(ascending=False) 
 .reset_index(name='count')
 .drop_duplicates(subset='word'))

grouped_verbs.head(50)
</code></pre>
Which got me this:
<pre><code>	word	pos	count
0	tener	VERB	22072
1	hacer	VERB	14946
2	ir	VERB	12570
3	decir	VERB	11314
4	querer	VERB	11083
5	ver	VERB	10269
6	estar	VERB	9780
7	saber	VERB	8704
8	ser	VERB	7674
9	dar	VERB	5722
10	pasar	VERB	5528
11	hablar	VERB	5355
12	venir	VERB	5145
13	creer	VERB	4895
14	salir 	VERB	3395
</code></pre>
Verbs had a slightly different drop-off pattern when I targeted them directly:
<center>
<img src="/images/posts/learning_spanish/diminishing_verbs.png">
</center>
I get a big bang for my buck by learning those top 40 verbs. Nouns on the other hand are much more spread out and most are evenly distributed:
<pre><code>word	pos	count
0	gracias	NOUN	4676
1	favor	NOUN	4625
2	señor	NOUN	4116
3	verdad	NOUN	3566
4	vida	NOUN	2673
5	hombre	NOUN	2601
6	madre	NOUN	2597
7	vez	NOUN	2537
8	tiempo	NOUN	2492
9	hijo	NOUN	2215
</code></pre>
<center>
<img src="/images/posts/learning_spanish/diminishing_nouns.png">
</center>
So then I thought to myself... How much of a show would I understand if I just learned these most important words? So I started by excluding some of the easy parts of speech and focused on the most important:
<pre><code>find_important_words = (words[~words.pos.isin(['PRON', 'CONJ', 'ADP', 'ADV', 'SCONJ', 'AUX', 'INTJ'])].groupby(['word', 'pos']).size() 
 .sort_values(ascending=False) 
 .reset_index(name='count')
 .drop_duplicates(subset='word'))

find_important_words.head(50)
</code></pre>
The top 20 were all verbs except for <code>bueno</code> and <code>gracias</code>. So now with my list of what I considered "important words" I plotted it to find what amount of words I wanted to learn:
<center>
<img src="/images/posts/learning_spanish/important_words.png">
</center>
It looks like 200 learned words would give me a reasonable amount of understanding for a series, so I decided to calculate how much of a series I would understand if I learned just those first 200 words:
<pre><code>percentages = {}

for show_name in words['media'].drop_duplicates().values:
 words_in_show = (words[words.media == show_name].groupby(words.word).size() 
 .sort_values(ascending=False) 
 .reset_index(name='count')
 .drop_duplicates(subset='word'))
 
 total_words_handled = 0

 for word in grouped_result['word'][:200]:
 values = words_in_show[words_in_show.word == word]['count'].values

 if values.size > 0:
 total_words_handled += values[0]

 percentages[show_name] = total_words_handled / words_in_show.sum().loc['count']
</code></pre>
Now I had a table that would show me what percentage of the spoken words were covered by the first 200 words in my list:
<pre><code>p_df = pd.DataFrame(percentages.items(), columns=['show', 'percentage'])
p_df = p_df.sort_values(by='percentage')
p_df['percentage'] = p_df['percentage'] * 100
pd.options.display.float_format = '{:,.2f}%'.format
p_df
</code></pre>




















<table>
<tbody><tr>
<th>Show</th>
<th>Percentage</th>
</tr><tr>
<td>Verónica</td>
<td>64.24%</td>
</tr><tr>
<td>El ciudadano ilustre</td>
<td>65.28%</td>
</tr><tr>
<td>El Chapo</td>
<td>66.68%</td>
</tr><tr>
<td>Neruda</td>
<td>66.89%</td>
</tr><tr>
<td>La casa de papel</td>
<td>67.56%</td>
</tr><tr>
<td>El Ministerio del Tiempo</td>
<td>68.03%</td>
</tr><tr>
<td>Club de Cuervos</td>
<td>68.19%</td>
</tr><tr>
<td>El marginal</td>
<td>68.47%</td>
</tr><tr>
<td>Ingobernable</td>
<td>68.59%</td>
</tr><tr>
<td>Pablo Escobar</td>
<td>70.20%</td>
</tr><tr>
<td>Fariña</td>
<td>70.95</td>
</tr><tr>
<td>La Reina del Sur</td>
<td>71.52%</td>
</tr><tr>
<td>Gran Hotel</td>
<td>73.15%</td>
</tr><tr>
<td>Las chicas del cable</td>
<td>73.58%</td>
</tr><tr>
<td>Élite</td>
<td>73.78%</td>
</tr><tr>
<td>La Piloto</td>
<td>74.03%</td>
</tr><tr>
<td>El bar</td>
<td>74.07%</td>
</tr><tr>
<td>La casa de las flores</td>
<td>75.40%</td>
</tr><tr>
<td>Tarde para la ira</td>
<td>75.59%</td>
</tr></tbody></table>
But living in Puerto Rico, one thing I have realized is speed of speech is also important. I have a much easier time speaking with people from Colombia and Mexico than I do with Puerto Ricans because they speak so much faster. So even though I could understand 75% of "Tarde para la ira" if I learned the 200 words, I want to make sure they are speaking at a pace I could understand as well.
So I loaded up the other CSV file that was the full sentences and added a "time per word" column:
<pre><code>sentences = pd.read_csv('sentences.csv.gz', compression='gzip', delimiter=',', parse_dates=['start', 'end'])
sentences['total_time'] = (sentences['end'] - sentences['start']).dt.total_seconds()
sentences['word_count'] = sentences['sentence'].str.split().str.len()
sentences['time_per_word'] = sentences['total_time'] / sentences['word_count']
</code></pre>
Then I was able to have a speed rating for each show:
<pre><code>sentence_group = sentences.groupby([sentences.media])
sentence_group.time_per_word.mean().reset_index().sort_values('time_per_word')
</code></pre>




















<table>
<tbody><tr>
<th>media</th>
<th>time_per_word</th>
</tr><tr>
<td>Gran Hotel</td>
<td>0.58</td>
</tr><tr>
<td>El Chapo</td>
<td>0.59</td>
</tr><tr>
<td>Las chicas del cable</td>
<td>0.61</td>
</tr><tr>
<td>Élite</td>
<td>0.63</td>
</tr><tr>
<td>Ingobernable</td>
<td>0.64</td>
</tr><tr>
<td>El Ministerio del Tiempo</td>
<td>0.64</td>
</tr><tr>
<td>Fariña</td>
<td>0.65</td>
</tr><tr>
<td>El ciudadano ilustre</td>
<td>0.67</td>
</tr><tr>
<td>Neruda</td>
<td>0.68</td>
</tr><tr>
<td>La Piloto</td>
<td>0.69</td>
</tr><tr>
<td>La casa de papel</td>
<td>0.70</td>
</tr><tr>
<td>El bar</td>
<td>0.70</td>
</tr><tr>
<td>Verónica</td>
<td>0.72</td>
</tr><tr>
<td>La Reina del Sur</td>
<td>0.75</td>
</tr><tr>
<td>Club de Cuervos</td>
<td>0.76</td>
</tr><tr>
<td>El marginal</td>
<td>0.76</td>
</tr><tr>
<td>Pablo Escobar</td>
<td>0.77</td>
</tr><tr>
<td>Tarde para la ira</td>
<td>0.77</td>
</tr><tr>
<td>La casa de las flores</td>
<td>0.81</td>
</tr></tbody></table>
Luckily the two series that have the least amount of vocabulary also speak the slowest! So these will be the series I start with. The final question I wanted to answer is "What are the top words I am missing for a series". Since I will know 75% of the series from the top 200 words, I am hoping there are some top words from a specific series that I can also learn to get an even higher understanding.
First, find which words are in each show but not in the top 200:
<pre><code>missing_words_by_show = {}

for show_name in words['media'].drop_duplicates().values:
 words_in_show = (words[words.media == show_name].groupby(words.word).size() 
 .sort_values(ascending=False) 
 .reset_index(name='count')
 .drop_duplicates(subset='word'))
 
 frequency_words = grouped_result['word'][:200]

 missing_words = words_in_show[~words_in_show.word.isin(frequency_words.values)]
 missing_words_by_show[show_name] = missing_words
</code></pre>
Then we were able to grab them per show:
<pre><code>missing_words_by_show['La casa de las flores'].head(50)

word	count
31	mamá	252
70	florería	87
98	perdón	56
102	sea	54
116	además	44
126	ahorita	40
132	cárcel	38
133	fiesta	38
</code></pre>
So adding those few words to my vocabulary will also give me a better understanding of the series.
<h2>Conclusion</h2>
I believe a data-driven approach to language learning will be an effective way to get me speaking better spanish. It was a ton of fun to play with spaCy, pandas, and jupyter as well!
I will improve the data analysis over time as well but I do believe this is a pretty good starting point!
<center>
<img src="/images/posts/learning_spanish/meme.png">
</center>

How to speak spanish like a colombian drug lord!

TL;DR:
<pre><code>$ git clone https://github.com/sontek/dotfiles.git
$ cd dotfiles
$ ./install.sh vim
</code></pre>
<h1>Intro</h1>
Back in 2008, I wrote the article <a href="http://sontek.net/python-with-a-modular-ide-vim">Python with a modular IDE
(Vim)</a>. Years later, I
have people e-mailing me and commenting daily asking for more
information, even though most of the information in it is outdated. Here
is the modern way to work with Python and Vim to achieve the perfect
environment.
Because one of the most important parts about a development environment
is the ability to easily reproduce across machines, we are going to
store our vim configuration in git:
<pre><code>$ mkdir ~/.vim/
$ mkdir ~/.vim/{autoload,bundle}
$ cd ~/.vim/
$ git init
</code></pre>
The purpose of the autoload directory is to automatically load the vim
plugin <a href="https://github.com/tpope/vim-pathogen">Pathogen</a>, which we'll
then use to load all other plugins that are located in the bundle
directory. So download pathogen and put it in your autoload folder.
You'll need to add the following to your ~/.vimrc so that pathogen
will be loaded properly. Filetype detection must be off when you run the
commands so its best to execute them first:
<pre><code>filetype off
call pathogen#runtime_append_all_bundles()
call pathogen#helptags()
</code></pre>
Now lets add all of the vim plugins we plan on using as submodules to
our git repository:
<pre><code>git submodule add http://github.com/tpope/vim-fugitive.git bundle/fugitive
git submodule add https://github.com/msanders/snipmate.vim.git bundle/snipmate
git submodule add https://github.com/tpope/vim-surround.git bundle/surround
git submodule add https://github.com/tpope/vim-git.git bundle/git
git submodule add https://github.com/ervandew/supertab.git bundle/supertab
git submodule add https://github.com/sontek/minibufexpl.vim.git bundle/minibufexpl
git submodule add https://github.com/wincent/Command-T.git bundle/command-t
git submodule add https://github.com/mitechie/pyflakes-pathogen.git
git submodule add https://github.com/mileszs/ack.vim.git bundle/ack
git submodule add https://github.com/sjl/gundo.vim.git bundle/gundo
git submodule add https://github.com/fs111/pydoc.vim.git bundle/pydoc
git submodule add https://github.com/vim-scripts/pep8.git bundle/pep8
git submodule add https://github.com/alfredodeza/pytest.vim.git bundle/py.test
git submodule add https://github.com/reinh/vim-makegreen bundle/makegreen
git submodule add https://github.com/vim-scripts/TaskList.vim.git bundle/tasklist
git submodule add https://github.com/vim-scripts/The-NERD-tree.git bundle/nerdtree
git submodule add https://github.com/sontek/rope-vim.git bundle/ropevim
git submodule init
git submodule update
git submodule foreach git submodule init
git submodule foreach git submodule update
</code></pre>
Thats it! Now that we've got our vim configuration in git!
Now lets look at how to use each of these plugins to improve the power
of vim:
<h1>Basic Editing and Debugging</h1>
<h2>Code Folding</h2>
Lets first enable code folding. This makes it a lot easier to organize
your code and hide portions that you aren't interested in working on.
This is quite easy for Python, since whitespace is required.
In your ~/.vimrc just add:
<pre><code>set foldmethod=indent
set foldlevel=99
</code></pre>
Then you will be able to be inside a method and type 'za' to open and
close a fold.
<h2>Window Splits</h2>
Sometimes code folding isn't enough; you may need to start opening up
multiple windows and working on multiple files at once or different
locations within the same file. To do this in vim, you can use these
shortcuts:
<pre><code>Vertical Split : Ctrl+w + v
Horizontal Split: Ctrl+w + s
Close current windows: Ctrl+w + q
</code></pre>
I also like to bind Ctrl+&#x3C;movement> keys to move around the windows,
instead of using Ctrl+w + &#x3C;movement>:
<pre><code>map &#x3C;c-j> &#x3C;c-w>j
map &#x3C;c-k> &#x3C;c-w>k
map &#x3C;c-l> &#x3C;c-w>l
map &#x3C;c-h> &#x3C;c-w>h
</code></pre>
<img src="http://i.imgur.com/krj0l.png" alt="image">
<h2>Snippets</h2>
The next tweak that really speeds up development is using snipmate.
We've already included it in our bundle/ folder so its already enabled.
Try opening up a python file and typing 'def&#x3C;tab>'. It should stub
out a method definition for you and allow you to tab through and fill
out the arguments, doc string, etc.
I also like to create my own snippets folder to put in some custom
snippets:
<pre><code>$ mkdir ~/.vim/snippets
$ vim ~/.vim/snippets/python.snippets
</code></pre>
Put this in the file:
<pre><code>snippet pdb
 import pdb; pdb.set_trace()
</code></pre>
Now you can type pdb&#x3C;tab> and it'll insert your breakpoint!
<h2>Task lists</h2>
Another really useful thing is to mark some of your code as TODO or
FIXME! I know we all like to think we write perfect code, but sometimes
you just have to settle and leave a note for yourself to come back
later. One of the plugins we included was the tasklist plugin that will
allow us to search all open buffers for things to fix. Just add a
mapping to open it in ~/.vimrc:
<pre><code>map &#x3C;leader>td &#x3C;Plug>TaskList
</code></pre>
Now you can hit &#x3C;leader>td to open your task list and hit 'q' to
close it. You can also hit enter on the task to jump to the buffer and
line that it is placed on.
<h2>Revision History</h2>
The final basic editing tweak I suggest everyone start utilizing is the
Gundo plugin. It'll allow you to view diff's of every save on a file
you've made and allow you to quickly revert back and forth:
<img src="http://i.imgur.com/2NrPS.png" alt="image">
Just bind a key in your .vimrc to toggle the Gundo window:
<pre><code>map &#x3C;leader>g :GundoToggle&#x3C;CR>
</code></pre>
<h1>Syntax Highlighting and Validation</h1>
Simply enable syntax highlighting in your ~/.vimrc:
<pre><code>syntax on " syntax highlighing
filetype on " try to detect filetypes
filetype plugin indent on " enable loading indent file for filetype
</code></pre>
Because we enabled pyflakes when we added it as a submodule in
~/.vim/bundle, it will notify you about unused imports and invalid
syntax. It will save you a lot of time saving and running just to find
out you missed a colon. I like to tell it not use the quickfix window:
<pre><code>let g:pyflakes_use_quickfix = 0
</code></pre>
<img src="http://i.imgur.com/ZfjFe.png" alt="image">
<h2>Pep8</h2>
The final plugin that really helps validate your code is the pep8
plugin, it'll make sure your code is consistent across all projects.
Add a key mapping to your ~/.vimrc and then you'll be able to jump to
each of the pep8 violations in the quickfix window:
<pre><code>let g:pep8_map='&#x3C;leader>8'
</code></pre>
<img src="http://i.imgur.com/VU9AB.png" alt="image">
<h1>Tab Completion and Documentation</h1>
Vim has many different code completion options. We are going to use the
SuperTab plugin to check the context of the code you are working on and
choose the best for the situation. We've already enabled the SuperTab
plugin in the bundle/ folder, so we just have to configure it to be
context sensitive and to enable omni code completion in your ~/.vimrc:
<pre><code>au FileType python set omnifunc=pythoncomplete#Complete
let g:SuperTabDefaultCompletionType = "context"
</code></pre>
Now we just enable the menu and pydoc preview to get the most useful
information out of the code completion:
<pre><code>set completeopt=menuone,longest,preview
</code></pre>
<img src="http://i.imgur.com/g4lxP.png" alt="image">
We also enabled the pydoc plugin at the beginning with all the
submodules; that gives us the ability to hit &#x3C;leader>pw when our
cursor is on a module and have a new window open with the whole
documentation page for it.
<h1>Code Navigation</h1>
<h2>Buffers</h2>
The most important part about navigating code within vim, is to
completely understand how to use buffers. There is no reason to use
tabs. Open files with :e &#x3C;filename> to place in a buffer. We already
installed the minibufexpl plugin, so you will already visually see every
buffer opened. You can also get a list of them doing :buffers.
You can switch between the buffers using b&#x3C;number>, such as :b1 for
the first buffer. You can also use its name to match, so you can type :b
mod&#x3C;tab> to autocomplete opening the models.py buffer. You need to
make sure you are using the minibufexpl from my github since it has
patches that make it much better to work with.
To close a buffer you use :bd or :bw.
<h2>Fuzzy Text File Search</h2>
To make finding and opening files within your project even easier, we
are going to use the command-t plugin. It does have some parts that need
to be compiled, so its not already installed by adding it as a
submodule. Go to your ~/.vim/bundle/command-t folder and run 'rake
make'. Yes you need ruby installed. By default, command-t is bound to
&#x3C;leader>t. This will use fuzzy text matching to find any file in your
project.
It also supports searching only through opened buffers, instead of files
using &#x3C;leader>b.
<img src="http://i.imgur.com/hUcSl.png" alt="image">
<h2>File Browser</h2>
NERD Tree is a project file browser. I must admit I used this heavily
back when I was migrating from Visual Studio and used to the Solution
Explorer, but I rarely use it anymore. Command-T is usually all you'll
need. It is useful when you are getting to know a new codebase for the
first time though. Lets bind a shortcut key for opening it:
<pre><code>map &#x3C;leader>n :NERDTreeToggle&#x3C;CR>
</code></pre>
<img src="http://i.imgur.com/R4ZzQ.png" alt="image">
<h2>Refactoring and Go to definition</h2>
Ropevim is also a great tool that will allow you to navigate around your
code. It supports automatically inserting import statements, goto
definition, refactoring, and code completion. You'll really want to
read up on everything it does, but the two big things I use it for is to
jump to function or class definitions quickly and to rename things
(including all their references).
For instance, if you are using django and you place your cursor over the
class models.Model you reference and then called :RopeGotoDefintion, it
would jump you straight to the django library to that class definition.
We already have it installed in our bundles, so we bind it to a key to
use it:
<pre><code>map &#x3C;leader>j :RopeGotoDefinition&#x3C;CR>
map &#x3C;leader>r :RopeRename&#x3C;CR>
</code></pre>
<h2>Searching</h2>
The final tool that really speeds up navigating your code is the Ack
plugin. Ack is similar to grep, but much better in my opinion. You can
fuzzy text search for anything in your code (variable name, class,
method, etc) and it'll give you a list of files and line numbers where
they are defined so you can quickly cycle through them. Just bind the
searching to a key:
<pre><code>nmap &#x3C;leader>a &#x3C;Esc>:Ack!
</code></pre>
We use ! at the end of it so it doesn't open the first result
automatically.
<h1>Integration with Git</h1>
We installed 2 plugins, git.vim and fugitive, that give us all the
integration we need. Git.vim will provide us syntax highlighting for git
configuration files; fugitive provides a great interface for interacting
with git including getting diffs, status updates, committing, and moving
files.
Fugitive also allows you to view what branch you are working in directly
from vim. Add this to your statusline in ~/.vimrc:
<pre><code>%{fugitive#statusline()}
</code></pre>
The big commands you need to know:
<ul>
<li>Gblame: This allows you to view a line by line comparison of who
the last person to touch that line of code is.</li>
<li>Gwrite: This will stage your file for commit, basically doing
git add &#x3C;filename></li>
<li>Gread: This will basically run a git checkout &#x3C;filename></li>
<li>Gcommit: This will just run git commit. Since its in a vim
buffer, you can use keyword completion (Ctrl-N), like
test_all&#x3C;Ctrl-N> to find the method name in your buffer and
complete it for the commit message. You can also use + and - on the
filenames in the message to stage/unstage them for the commit.</li>
</ul>
<img src="http://i.imgur.com/NuRRj.png" alt="image">
<h1>Test Integration</h1>
<h2>django nose</h2>
Test runner integration really depends on the testing library you are
using and what type of tests you are running but we included a great
generic plugin called MakeGreen that executes off of vim's makeprg
variable. So for instance, if you are using django with django-nose you
could define a shortcut key in your ~/.vimrc like this:
<pre><code>map &#x3C;leader>dt :set makeprg=python\ manage.py\ test\|:call MakeGreen()&#x3C;CR>
</code></pre>
This will just give you a green bar at the bottom of vim if your test
passed or a red bar with the message of the failed test if it doesn't.
Very simple.
<h2>py.test</h2>
I also included the py.test vim plugin for those who prefer it. This
plugin has a lot more functionality including executing individual tests
by class, file, or method. You can also cycle through the individual
assertion errors. I have the following bindings:
<pre><code>" Execute the tests
nmap &#x3C;silent>&#x3C;Leader>tf &#x3C;Esc>:Pytest file&#x3C;CR>
nmap &#x3C;silent>&#x3C;Leader>tc &#x3C;Esc>:Pytest class&#x3C;CR>
nmap &#x3C;silent>&#x3C;Leader>tm &#x3C;Esc>:Pytest method&#x3C;CR>
" cycle through test errors
nmap &#x3C;silent>&#x3C;Leader>tn &#x3C;Esc>:Pytest next&#x3C;CR>
nmap &#x3C;silent>&#x3C;Leader>tp &#x3C;Esc>:Pytest previous&#x3C;CR>
nmap &#x3C;silent>&#x3C;Leader>te &#x3C;Esc>:Pytest error&#x3C;CR>
</code></pre>
<img src="http://i.imgur.com/RAE7v.png" alt="image">
<h1>Virtualenv</h1>
Vim doesn't realize that you are in a virtualenv so it wont give you
code completion for libraries only installed there. Add the following
script to your ~/.vimrc to fix it:
<pre><code>" Add the virtualenv's site-packages to vim path
py &#x3C;&#x3C; EOF
import os.path
import sys
import vim
if 'VIRTUAL_ENV' in os.environ:
 project_base_dir = os.environ['VIRTUAL_ENV']
 sys.path.insert(0, project_base_dir)
 activate_this = os.path.join(project_base_dir, 'bin/activate_this.py')
 execfile(activate_this, dict(__file__=activate_this))
EOF
</code></pre>
<h1>Django</h1>
The only true django tweak I make is before I open vim I'll export the
DJANGO_SETTINGS_MODULE environment so that I get code completion for
django modules as well:
<pre><code>export DJANGO_SETTINGS_MODULE=project.settings
</code></pre>
<h1>Random Tips</h1>
If you want to find a new color scheme just go to
<a href="http://code.google.com/p/vimcolorschemetest/">http://code.google.com/p/vimcolorschemetest/</a> to preview a large
selection.
John Anderson &#x3C;<a href="mailto:sontek@gmail.com">sontek@gmail.com</a>> 2011

Turning Vim into a modern Python IDE

I have seen a lot of people switch over to using ipython, bpython, etc
to get auto-complete support without realizing that the standard
interpreter does have this functionality.
To enable auto-complete support in the python interpreter you need to
create a python startup file that enables readline support. A python
startup file is just a bunch of python code that gets executed at
startup of the interpreter. To do this you just setup PYTHONSTARTUP in
your ~/.bashrc and then create a ~/.pythonrc.py file:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="python" data-theme="github-dark-dimmed"><code data-language="python" data-theme="github-dark-dimmed" style="display: grid;">#.bashrc
PYTHONSTARTUP=~/.pythonrc.py
export PYTHONSTARTUP
 
#.pythonrc.py
try:
 import readline
except ImportError:
 print("Module readline not available.")
else:
 import rlcompleter
 readline.parse_and_bind("tab: complete")</code></pre></figure>
Now when you are in python you have tab completion on importing, calling
methods on a module, etc.
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="python" data-theme="github-dark-dimmed"><code data-language="python" data-theme="github-dark-dimmed" style="display: grid;">>>> import o
object( oct( open( or ord( os</code></pre></figure>
I always end up using the pretty print module for viewing long lists and
strings in the interpreter so I prefer to just use it by default:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="python" data-theme="github-dark-dimmed"><code data-language="python" data-theme="github-dark-dimmed" style="display: grid;"># Enable Pretty Printing for stdout
import pprint
def my_displayhook(value):
 if value is not None:
 try:
 import __builtin__
 __builtin__._ = value
 except ImportError:
 __builtins__._ = value
 
 pprint.pprint(value)
 
sys.displayhook = my_displayhook</code></pre></figure>
It is also very useful to be able to load up your favorite editor to
edit lines of code from the interpreter, you can do this by adding the
following into your ~/.pythonrc.py:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="python" data-theme="github-dark-dimmed"><code data-language="python" data-theme="github-dark-dimmed" style="display: grid;">import os
import sys
from code import InteractiveConsole
from tempfile import mkstemp
 
EDITOR = os.environ.get('EDITOR', 'vi')
EDIT_CMD = '\e'
 
class EditableBufferInteractiveConsole(InteractiveConsole):
 def __init__(self, *args, **kwargs):
 self.last_buffer = [] # This holds the last executed statement
 InteractiveConsole.__init__(self, *args, **kwargs)
 
 def runsource(self, source, *args):
 self.last_buffer = [ source.encode('latin-1') ]
 return InteractiveConsole.runsource(self, source, *args)
 
 def raw_input(self, *args):
 line = InteractiveConsole.raw_input(self, *args)
 if line == EDIT_CMD:
 fd, tmpfl = mkstemp('.py')
 os.write(fd, b'\n'.join(self.last_buffer))
 os.close(fd)
 os.system('%s %s' % (EDITOR, tmpfl))
 line = open(tmpfl).read()
 os.unlink(tmpfl)
 tmpfl = ''
 lines = line.split( '\n' )
 for i in range(len(lines) - 1): self.push( lines[i] )
 line = lines[-1]
 return line
 
c = EditableBufferInteractiveConsole(locals=locals())
c.interact(banner='')
 
# Exit the Python shell on exiting the InteractiveConsole
sys.exit()</code></pre></figure>
For Django developers when you load up the ./manage.py shell it is nice
to have access to all your models and settings for testing:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="python" data-theme="github-dark-dimmed"><code data-language="python" data-theme="github-dark-dimmed" style="display: grid;"># If we're working with a Django project, set up the environment
if 'DJANGO_SETTINGS_MODULE' in os.environ:
 from django.db.models.loading import get_models
 from django.test.client import Client
 from django.test.utils import setup_test_environment, teardown_test_environment
 from django.conf import settings as S
 
 class DjangoModels(object):
 """Loop through all the models in INSTALLED_APPS and import them."""
 def __init__(self):
 for m in get_models():
 setattr(self, m.__name__, m)
 
 A = DjangoModels()
 C = Client()</code></pre></figure>
After these tweaks the python interpreter is a lot more powerful and you
really lose the need for the more interactive shells like ipython and
bpython. All of these settings work in both python2 and python3.
If you want to see my complete ~/.pythonrc.py you can get it on
<a href="https://github.com/sontek/dotfiles/blob/master/_pythonrc.py">github</a>

Tips and Tricks for the Python Interpreter

Writing tests for Pyramid and SQLAlchemy

Having every github action run on every pull request will end up slowing you
down and sometimes even discourage you from making changes. For example, if you
see an error in the <code>README.md</code> but you know you will have to wait for the entire
test suite to run you may choose not to push the change.
<h1>Path Filtering</h1>
To address this problem, Github has a feature called <a href="https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore">path filtering</a>
which works really well! You can include and exclude paths with a simple syntax:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="yaml" data-theme="github-dark-dimmed"><code data-language="yaml" data-theme="github-dark-dimmed" style="display: grid;">on:
 pull_request:
 paths:
 - '*'
 - '*/**'
 - '!README.md'
 - '!.gitignore'</code></pre></figure>
Which will run on any changes except for <code>README.md</code> and <code>.gitignore</code>. There is
even a simpler format to this by using <code>paths-ignore</code>:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="yaml" data-theme="github-dark-dimmed"><code data-language="yaml" data-theme="github-dark-dimmed" style="display: grid;">on:
 pull_request:
 paths-ignore:
 - 'README.md'
 - '.gitignore'</code></pre></figure>
Although the path filtering feature of github works, I do not recommend using
it because it has some critical issues that I will talk about below!
<h1>Branch Protection / Required Checks</h1>
Path Filtering has one major flaw which is that it skips the run of the job
completely which means if you are using <a href="https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches">branch protection</a>
your PR will be stuck in <code>pending</code> state.
To resolve this we will have to run the job, but skip the actions inside it that
are slow conditionally. There are a few opensource actions out there that make
this easy but the one I recommend is:
<a href="https://github.com/tj-actions/changed-files">https://github.com/tj-actions/changed-files</a>
It creates a lot of useful outputs you can use in your conditional out of the box
which makes your action easy to write.
<h1>Skipping a required check</h1>
So using <code>tj-actions/changed-files</code> to skip a required check, it will look very
similar to the path filtering from github. You first define what files you care
about:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="yaml" data-theme="github-dark-dimmed"><code data-language="yaml" data-theme="github-dark-dimmed" style="display: grid;">jobs:
 terraform-lint: 
 runs-on: ubuntu-22.04
 steps:
 - uses: actions/checkout@v3
 - name: Check if files changed
 id: changed-files-yaml
 uses: tj-actions/changed-files@v40
 with:
 files_yaml: |
 src:
 - '*'
 - '*/**'
 - '!README.md'
 - '!.gitignore'</code></pre></figure>
The <code>src</code> is just a key selection I made here. You can have many groups of
files like <code>tests</code>, <code>docs</code>, etc. and conditionally do things if each of them
changed.
So from our example, this step will generate output we can use <code>steps.changed-files-yaml.outputs.src_any_changed</code>
that is either <code>true</code> or <code>false</code>. To use this we can use the <code>if</code> conditional
block on our jobs.
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="yaml" data-theme="github-dark-dimmed"><code data-language="yaml" data-theme="github-dark-dimmed" style="display: grid;">steps:
 - if: steps.changed-files-yaml.outputs.src_any_changed == 'true'
 run: |
 earthly \
 +ci-terraform-lint \
 --TERRAFORM_VERSION=${{ env.RTX_TERRAFORM_VERSION }}</code></pre></figure>
This will then skip linting if we did not change any terraform and the job will
be marked successful!
<center>
<img src="/images/posts/github_skip_required_checks/skipped_checks.png">
</center>
This can be confusing sometimes and the people you work with (or future you!)
may wonder why these critical checks are being skipped. I like to include
a message with the decision that was made so it is obvious that we wanted to skip
them:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="yaml" data-theme="github-dark-dimmed"><code data-language="yaml" data-theme="github-dark-dimmed" style="display: grid;">- if: steps.changed-files-yaml.outputs.src_any_changed == 'false'
 run: |
 echo "No terraform files changed"</code></pre></figure>
Which will output:
<pre><code>✅ "No terraform files changed"
</code></pre>
Right next to the skipped job which is exactly what we wanted to see! So the end
result would look something like this:
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#22272e;color:#adbac7" tabindex="0" data-language="yaml" data-theme="github-dark-dimmed"><code data-language="yaml" data-theme="github-dark-dimmed" style="display: grid;">on:
 pull_request:
 
jobs:
 terraform-lint: 
 runs-on: ubuntu-22.04
 steps:
 - uses: actions/checkout@v3
 - name: Check if real files changed
 id: changed-files-yaml
 uses: tj-actions/changed-files@v40
 with:
 files_yaml: |
 src:
 - '*'
 - '*/**'
 - '!README.md'
 - '!.gitignore'
 
 - if: steps.changed-files-yaml.outputs.src_any_changed == 'true'
 run: |
 earthly \
 +ci-terraform-lint \
 --TERRAFORM_VERSION=${{ env.RTX_TERRAFORM_VERSION }}
 
 - if: steps.changed-files-yaml.outputs.src_any_changed == 'false'
 run: |
 echo "No terraform files changed"</code></pre></figure>
<h1>Conclusion</h1>
Hopefully with this you can start speeding up your github actions and making the
developer experience enjoyable!

Challenge 1