AWS From Scratch with Terraform - Setting up your Root Account for IaC  with Terraform Cloud and Github actions.

When you're an SRE/DevOps engineer you'll end up making AWS accounts and
create a lot of cruft in your sandbox and development accounts. AWS
does not make it easy to clear these up but there is a tool called
<a href="https://github.com/rebuy-de/aws-nuke">aws-nuke</a> that will do it for you!
<h1>Safe Guards</h1>
aws-nuke has a few safeguards in place to prevent inadvertent data loss.
The first of which is it requires you to alias the targetted account. I
like to put <code>nuke</code> in the alias to make it clear.
<pre><code class="hljs language-bash">aws iam create-account-alias --account-alias nuke-&#x3C;account>
</code></pre>
The second safe-guard is the config takes a key called <code>account-blocklist</code>
that will guarantee nuke will not run against it no matter what.
The final safety switch it has is it will not take any action by default,
it will only execute a dry-run. You need to run the CLI with
<code>--no-dry-run</code> if you want it to take action.
<h1>Getting Started</h1>
You configure <code>aws-nuke</code> with YAML, so the first step is to define that:
<pre><code class="hljs language-makefile">regions:
 - us-east-1
 - global

account-blocklist:
 - "888888888888" # production

accounts:
 "777777777777": {} # nuke-&#x3C;account>
</code></pre>
This will prevent us from nuking our production account and target all resources
in the account we actually want to nuke.
You might want to have it nuke ALL REGIONS in AWS since you may not know which
regions resources are deployed in. To do this you can query the regions from AWS:
<pre><code class="hljs language-bash">❯ aws ec2 describe-regions --all-regions --query "Regions[*].RegionName" --output text | xargs -n 1 | sort

af-south-1
ap-east-1
ap-northeast-1
ap-northeast-2
ap-northeast-3
ap-south-1
ap-south-2
ap-southeast-1
ap-southeast-2
ap-southeast-3
ap-southeast-4
ca-central-1
eu-central-1
eu-central-2
eu-north-1
eu-south-1
eu-south-2
eu-west-1
eu-west-2
eu-west-3
me-central-1
me-south-1
sa-east-1
us-east-1
us-east-2
us-west-1
us-west-2
</code></pre>
Which would give you an updated config of:
<pre><code class="hljs language-yaml">regions:
 - global # Global resources like IAM
 - af-south-1
 - ap-east-1
 - ap-northeast-1
 - ap-northeast-2
 - ap-northeast-3
 - ap-south-1
 - ap-south-2
 - ap-southeast-1
 - ap-southeast-2
 - ap-southeast-3
 - ap-southeast-4
 - ca-central-1
 - eu-central-1
 - eu-central-2
 - eu-north-1
 - eu-south-1
 - eu-south-2
 - eu-west-1
 - eu-west-2
 - eu-west-3
 - me-central-1
 - me-south-1
 - sa-east-1
 - us-east-1
 - us-east-2
 - us-west-1
 - us-west-2
</code></pre>
I personally don't recommend targetting all AWS regions at the same time. It
will generate a lot of output and be slow. You could do it if necessary but
most people only have a few regions they use and so they can set those directly.
For example it, maybe you only use <code>us-</code> based regions?
So lets run the dry-run and see what it wants to nuke:
<pre><code class="hljs language-bash">❯ aws-nuke -c nuke.yaml
</code></pre>
This should output something like:
<pre><code class="hljs language-bash">Do you really want to nuke the account with the ID 777777777777 and the alias 'nuke-sandbox'?
Do you want to continue? Enter account alias to continue.
> nuke-sandbox
us-east-1 - EC2Subnet - subnet-0cd9975a443a6304b - [DefaultForAz: "true", DefaultVPC: "true", OwnerID: "777777777777"] - would remove
us-east-1 - EC2Subnet - subnet-0be39d02e399a371c - [DefaultForAz: "true", DefaultVPC: "true", OwnerID: "777777777777"] - would remove
us-east-1 - EC2Subnet - subnet-02d7017bd4730ea63 - [DefaultForAz: "true", DefaultVPC: "true", OwnerID: "777777777777"] - would remove
us-east-1 - EC2Subnet - subnet-0ec04b28c32708ab2 - [DefaultForAz: "true", DefaultVPC: "true", OwnerID: "777777777777"] - would remove
us-east-1 - EC2Subnet - subnet-0eea1b4be084840ed - [DefaultForAz: "true", DefaultVPC: "true", OwnerID: "777777777777"] - would remove
us-east-1 - EC2Subnet - subnet-05a294cc04736012e - [DefaultForAz: "true", DefaultVPC: "true", OwnerID: "777777777777"] - would remove
us-east-1 - EC2RouteTable - rtb-0abda0e94015064ca - [DefaultVPC: "true"] - would remove
us-east-1 - EC2DefaultSecurityGroupRule - sgr-0368525f77bf566ac - [SecurityGroupId: "sg-0a59900b52ced5e10"] - would remove
us-east-1 - EC2DefaultSecurityGroupRule - sgr-0890a837ed6148729 - [SecurityGroupId: "sg-0a59900b52ced5e10"] - would remove
us-east-1 - EC2InternetGatewayAttachment - igw-0acfb474f1fd71375 -> vpc-0be5d310ab44c239a - [DefaultVPC: "true"] - would remove
us-east-1 - SQSQueue - https://sqs.us-east-1.amazonaws.com/777777777777/example-sqs - would remove
global - IAMSAMLProvider - arn:aws:iam::777777777777:saml-provider/AWSSSO_254abb4071f10b25_DO_NOT_DELETE - would remove
global - IAMOpenIDConnectProvider - arn:aws:iam::777777777777:oidc-provider/app.terraform.io - [Arn: "arn:aws:iam::777777777777:oidc-provider/app.terraform.io"] - would remove
global - IAMPolicy - arn:aws:iam::777777777777:policy/tfc-agent-access-policy - [ARN: "arn:aws:iam::777777777777:policy/tfc-agent-access-policy", Name: "tfc-agent-access-policy", Path: "/", PolicyID: "ANPA2T6PZOBNWI76TKQRF"] - would remove
global - IAMRole - tfc-agent - [CreateDate: "2023-04-02T17:55:23Z", LastUsedDate: "2023-06-22T13:45:02Z", Name: "tfc-agent", Path: "/"] - would remove
global - IAMRolePolicyAttachment - tfc-agent -> tfc-agent-access-policy - [PolicyArn: "arn:aws:iam::777777777777:policy/tfc-agent-access-policy", PolicyName: "tfc-agent-access-policy", RoleCreateDate: "2023-04-02T17:55:23Z", RoleLastUsed: "2023-06-22T13:45:02Z", RoleName: "tfc-agent", RolePath: "/"] - would remove
Scan complete: 85 total, 19 nukeable, 66 filtered.

The above resources would be deleted with the supplied configuration. Provide --no-dry-run to actually destroy resources.
</code></pre>
This is great, it fully scanned the account and found every resource to delete!
It even wants to delete the DefaultVPC which is usually a good idea. The one
resource that should catch your eye that you probably do not want to delete:
<ul>
<li><code>arn:aws:iam::777777777777:saml-provider/AWSSSO_254abb4071f10b25_DO_NOT_DELETE</code></li>
</ul>
AWS clearly doesn't want us to delete that!
<h1>Filters</h1>
To prevent nuke from deleting resources you want to keep you can define presets
that you use on each account. So with our SSO example we want to prevent it
from deleting those resources in a preset.
<pre><code class="hljs language-yaml">presets:
 sso:
 filters:
 IAMSAMLProvider:
 - type: "regex"
 value: "AWSSSO_.*_DO_NOT_DELETE"
 IAMRole:
 - type: "glob"
 value: "AWSReservedSSO_*"
 IAMRolePolicyAttachment:
 - type: "glob"
 value: "AWSReservedSSO_*"
</code></pre>
You can see in this example I'm targetting specific resource types and then
matching them with both <code>regex</code> and <code>glob</code> filter types. These are super
powerful but a lot of times the simpler filters can be used. I start with
<code>contains</code> filter and then go from there:
<pre><code class="hljs language-yaml"> - type: contains
 value: AWSReservedSSO
</code></pre>
Then the other thing you may have noticed is that I was repeating
<code>AWSReservedSSO</code> multiple times. To reduce that you can use standard YAML
anchors. So the final config for your preset would look like this:
<pre><code class="hljs language-yaml">presets:
 sso:
 filters:
 _DEFAULT_FILTERS: &#x26;DEFAULT
 - type: "contains"
 value: "DO_NOT_DELETE"
 - type: "contains"
 value: "AWSReservedSSO"

 IAMSAMLProvider: *DEFAULT
 IAMRole: *DEFAULT
 IAMRolePolicyAttachment: *DEFAULT
</code></pre>
Now we can use that preset in our accounts configuration:
<pre><code class="hljs language-yaml">accounts:
 "777777777777":
 "presets":
 - sso
</code></pre>
So your final config should look something like this:
<pre><code class="hljs language-yaml">regions:
 - us-east-1
 - global

account-blocklist:
 - "888888888888" # production


presets:
 sso:
 filters:
 _DEFAULT_FILTERS: &#x26;DEFAULT
 - type: "contains"
 value: "DO_NOT_DELETE"
 - type: "contains"
 value: "AWSReservedSSO"

 IAMSAMLProvider: *DEFAULT
 IAMRole: *DEFAULT
 IAMRolePolicyAttachment: *DEFAULT

accounts:
 "777777777777":
 "presets":
 - sso
</code></pre>
When you run this you should see now the resources we want to keep are filtered
out:
<pre><code class="hljs language-bash">global - IAMSAMLProvider - arn:aws:iam::777777777777:saml-provider/AWSSSO_254abb4071f10b25_DO_NOT_DELETE - filtered by config
</code></pre>
Once you are ready and have your filters in place you can run it for real!
<pre><code class="hljs language-bash">❯ aws-nuke -c nuke.yaml --no-dry-run
</code></pre>
<h1>Next steps</h1>
One final note about it is that it does not understand the relationship between
resources and so it could try deleting an EBS volume that is still in use by an
EC2 instance. There isn't a great solution for this outside of running nuke a
few times.
The tool is well documented and so you can find the rest of information going to
<a href="https://github.com/rebuy-de/aws-nuke">https://github.com/rebuy-de/aws-nuke</a>!

Wiping an AWS Account with aws-nuke

Running a kubernetes cluster locally with kubeadm

Previously I <a href="/blog/2022/local_kubeadm_cluster">showed</a> how to run kubernetes
locally with <code>kubeadm</code> and VMs but sometimes that is overkill so I wanted to
show how to run <a href="https://kind.sigs.k8s.io/">kind</a> which is "kuberetes in
docker".
<h1>Creating your first cluster</h1>
kind is a very flexible way to run kubernetes locally and allows you to run
single node or multinode clusters while having the flexibility to use all
the features of kubernetes success as ingress.
To create your first cluster it is as simple as running:
<pre><code class="hljs language-bash">❯ kind create cluster 

Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼 
 ✓ Preparing nodes 📦 
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
</code></pre>
You now have a functioning kubernetes cluster and you
can view what it created:
<pre><code class="hljs language-bash">❯ k get node
NAME STATUS ROLES AGE VERSION
kind-control-plane Ready control-plane 4m26s v1.27.3
</code></pre>
You can also verify that it is running inside docker:
<pre><code class="hljs language-bash">❯ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1c3ba74dc29b kindest/node:v1.27.3 "/usr/local/bin/entr…" 3 minutes ago Up 3 minutes 127.0.0.1:59327->6443/tcp kind-control-plane
</code></pre>
<h1>Making the cluster useful</h1>
There are a few things you'll notice with the command we ran originally:
<ul>
<li>It grabbed the latest kubernetes version available</li>
<li>It is running a single node cluster</li>
<li>No ingress available</li>
</ul>
Luckily kind makes it really easy to customize your local cluster to be what
you want it to be by using a <code>YAML</code> configuration.
Create the configuration:
<pre><code class="hljs language-yaml">kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
 kubeadmConfigPatches:
 - |
 kind: InitConfiguration
 nodeRegistration:
 kubeletExtraArgs:
 node-labels: "ingress-ready=true"
 extraPortMappings:
 - containerPort: 80
 hostPort: 80
 protocol: TCP
 - containerPort: 443
 hostPort: 443
 protocol: TCP
- role: worker
- role: worker
- role: worker
</code></pre>
With this we've now generated a 4 node cluster where we have a single
control-plane and three workers. Then we defined some extra configuration on
the control-plane:
<ul>
<li>kubeadmConfigPatches: We want to change the default configuration the
cluster uses so it'll tag the nodes with the <code>ingress-ready</code> label so the
controller will use them.</li>
<li>extraPortMappings: allow the local host to make requests to the Ingress controller over ports 80/443</li>
<li>node-labels: only allow the ingress controller to run on specific node(s) matching the label selector</li>
</ul>
So now we can create the new cluster with the configuration. Save that config
as <code>kind_config.yml</code> and then run:
<pre><code class="hljs language-bash">❯ kind create cluster --image kindest/node:v1.25.11 --config kind_config.yml --name kind-multinode
</code></pre>
This time I've added a few additional flags on the commandline. <code>--image</code>
allows us to use a different version of kubernetes and <code>--name</code> allows us to
make more than one cluster. So if you didn't destroy the first cluster you'll
see we have two of them now:
<pre><code class="hljs language-bash">❯ kind get clusters
kind
kind-multinode
</code></pre>
but <code>kind</code> will swap the to the newest cluster by default:
<pre><code class="hljs language-bash">❯ kubectl config current-context
kind-kind-multinode

❯ kubectl get node
NAME STATUS ROLES AGE VERSION
kind-multinode-control-plane Ready control-plane 107s v1.25.11
kind-multinode-worker Ready &#x3C;none> 88s v1.25.11
kind-multinode-worker2 Ready &#x3C;none> 88s v1.25.11
kind-multinode-worker3 Ready &#x3C;none> 88s v1.25.11
</code></pre>
Now we need to get the <code>ingress-nginx</code> controller installed so we can start
using our cluster with ingress:
<pre><code class="hljs language-bash">❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
</code></pre>
The manifests contains <code>kind</code> specific patches to forward the hostPorts to the ingress controller, set taint tolerations and schedule it to the custom labelled node.
This will take a little bit of time to get up and running, you can monitor it
by running:
<pre><code class="hljs language-bash">kubectl wait --namespace ingress-nginx \
 --for=condition=ready pod \
 --selector=app.kubernetes.io/component=controller \
 --timeout=90s
</code></pre>
or just manually check the status:
<pre><code class="hljs language-bash">❯ kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-bbmlc 0/1 Completed 0 68s
pod/ingress-nginx-admission-patch-qlnr8 0/1 Completed 2 68s
pod/ingress-nginx-controller-5f748f78c8-6tc6b 0/1 ContainerCreating 0 68s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller NodePort 10.96.228.248 &#x3C;none> 80:31771/TCP,443:31759/TCP 68s
service/ingress-nginx-controller-admission ClusterIP 10.96.180.126 &#x3C;none> 443/TCP 68s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 0/1 1 0 68s

NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-5f748f78c8 1 1 0 68s

NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 22s 68s
job.batch/ingress-nginx-admission-patch 1/1 35s 68s
</code></pre>
Once <code>ingress-nginx-controller</code> is in <code>Running</code> state you are read to go!
<h1>Deploying your first app</h1>
To prove that the cluster is working correctly we will deploy
<a href="https://github.com/Kong/httpbin">httpbin</a> which is a nice little API server
so we can prove everything is working.
Create a <code>httbin.yml</code> file and paste this into it:
<pre><code class="hljs language-yaml">---
apiVersion: v1
kind: Service
metadata:
 name: httpbin
 labels:
 app: httpbin
 service: httpbin
spec:
 ports:
 - name: http
 port: 8000
 targetPort: 8080
 selector:
 app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: httpbin
spec:
 replicas: 2
 selector:
 matchLabels:
 app: httpbin
 version: v1
 template:
 metadata:
 labels:
 app: httpbin
 version: v1
 spec:
 containers:
 - image: docker.io/mccutchen/go-httpbin
 imagePullPolicy: IfNotPresent
 name: httpbin
 ports:
 - containerPort: 8080
</code></pre>
This is creating a couple of Kubernetes resources:
<ul>
<li><code>Service</code>: This is exposing the port to the ingress</li>
<li><code>Deployment</code>: This is actually launching the service</li>
</ul>
So we are not using the ingress yet but we can prove that we can launch the
service at least. So apply those manifests:
<pre><code class="hljs language-bash">❯ kubectl apply -f httpbin.yml 
service/httpbin created
deployment.apps/httpbin created
</code></pre>
You should see two pods come up. You should wait for them to get into the
<code>Running</code> status:
<pre><code class="hljs language-bash">❯ kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
httpbin-5c5494967-2z5wz 1/1 Running 0 48s 10.244.3.3 kind-multinode-worker3 &#x3C;none> &#x3C;none>
httpbin-5c5494967-9lf47 1/1 Running 0 72s 10.244.1.2 kind-multinode-worker &#x3C;none> &#x3C;none>
</code></pre>
We can now use port forwarding to access it. <code>httpbin</code> is exposed on <code>8000</code> so
lets create port <code>9000</code> on our host that forwards to it:
<pre><code class="hljs language-bash">❯ kubectl port-forward service/httpbin 9000:8000
Forwarding from 127.0.0.1:9000 -> 80
Forwarding from [::1]:9000 -> 80
</code></pre>
You can access it via:
<pre><code class="hljs language-bash">❯ curl localhost:9000/get 
{
 "args": {},
 "headers": {
 "Accept": [
 "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"
 ],
}
</code></pre>
<h1>Using Ingress</h1>
Now to use the ingress rather than port forwarding we create one additional
resource, the <code>Ingress</code>:
<pre><code class="hljs language-yaml">apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: httpbin-ingress
 annotations:
 nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
 ingressClassName: nginx
 rules:
 - http:
 paths:
 - path: /httpbin(/|$)(.*)
 pathType: ImplementationSpecific
 backend:
 service:
 name: httpbin
 port:
 number: 8000
</code></pre>
There are a few critical options here. The first is the annotation to rewrite
the path so it doesn't include <code>/httpbin/</code> when it sends the request to the
service and then the <code>path</code> and <code>pathType</code> so it knows which paths to send to
which service.
Now you should be able to hit your local host and get routed to your
kubernetes service:
<pre><code class="hljs language-bash">❯ curl localhost/httpbin/get
</code></pre>
Success! Now you have a multinode kubernetes cluster that has an ingress
controller!
<h1>Next Steps</h1>
The cluster can be used like a production cluster now for local
development! You could setup Grafana, ArgoCD, etc. to run
inside the cluster.

Running a kubernetes cluster locally with kind

<a href="https://asdf-vm.com/">asdf</a> is a general purpose version manager that
can manage versions of most programming language runtimes through a set
of plugins.
<iframe width="854" height="480" src="https://www.youtube.com/embed/RTaqWRj-6Lg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
With micro-services being all the rage and the ever changing landscape
of the development world, it is rare to utilize a single version of
language runtime. Even when you want to upgrade from one to the other
you'll need both usable on your system at the same time.
I've used tools like <code>pyenv</code> and <code>nvm</code> in the past when I needed to change
versions depending on which project I'm contributing to. But with <code>asdf</code>
you have one tool to rule them all!
<h2>Getting Started</h2>
The first thing you need to do when working with <code>asdf</code> is grab the
plugins for the languages you are interested in working with. You can list
what plugins are available:
<pre><code class="hljs language-bash">> asdf plugin list all
golang *https://github.com/kennyp/asdf-golang.git
golangci-lint https://github.com/hypnoglow/asdf-golangci-lint.git
nodejs *https://github.com/asdf-vm/asdf-nodejs.git
poetry *https://github.com/asdf-community/asdf-poetry.git
python *https://github.com/danhper/asdf-python.git
yarn *https://github.com/twuni/asdf-yarn.git
</code></pre>
On the left will be the name of the plugin and on the right will be the repository
where it lives. It'll me marked with an asterisk if you already have it installed.
To install a plugin you say <code>asdf plugin add &#x3C;plugin></code> to get it installed. You can
also provide the repository where you want it pulled from, for example:
<pre><code class="hljs language-bash">> asdf plugin add nodejs https://github.com/asdf-vm/asdf-nodejs.git
> asdf plugin add python https://github.com/danhper/asdf-python.git
</code></pre>
This will not give you any version of those languages, it is only installing the
plugin that knows how to work with those languages. You are ready to pull down
any versions you want at that point:
<pre><code class="hljs language-bash">> asdf install nodejs 14.19.0
> asdf install python 3.9.10
</code></pre>
Once you have the versions installed you will be able to view them like this:
<pre><code class="hljs language-bash">> asdf list
golang
 1.17.7
nodejs
 --help
 12.22.10
 14.19.0
 16.14.0
 17.5.0
poetry
 1.1.13
python
 3.9.10
yarn
 1.22.17
</code></pre>
<h2>Using the installed languages</h2>
To activate a specific version of a language you have you have three options:
<h3>Make it global</h3>
You can make it global, meaning when you run the tool like <code>python</code> it'll use
this version for the system:
<pre><code class="hljs language-bash">> asdf global python 3.9.10
</code></pre>
<h3>Make it local</h3>
You can make it local, which means it will generate a file in the current
directory named <code>.tool-versions</code> and so whenever you change into a directory
it will activate the versions defined in there.
<pre><code class="hljs language-bash">> asdf local nodejs 12.22.10
> cat .tool-versions 
nodejs 12.22.10
</code></pre>
The great thing about this is you can commit that file to git and then anyone
who checks out the project and uses <code>asdf</code> will have the same versions activated!
<h3>Temporary</h3>
If you want to activate a version of a language temporarily you can swap to it
for the current shell:
<pre><code class="hljs language-bash">> asdf shell golang 1.17.7
> env|grep -i ASDF
ASDF_GOLANG_VERSION=1.17.7
</code></pre>
It sets an environment variable that will have preference over the file. If you
ever wonder what versions a directory is using you can run:
<pre><code class="hljs language-bash">> asdf current
golang ______ No version set. Run "asdf &#x3C;global|shell|local> golang &#x3C;version>"
nodejs 12.22.10 .tool-versions
poetry ______ No version set. Run "asdf &#x3C;global|shell|local> poetry &#x3C;version>"
python 3.9.10 .tool-versions
yarn 1.22.17 .tool-versions
</code></pre>
<h2>Conclusion</h2>
<a href="https://asdf-vm.com/">asdf</a> is an AWESOME tool to utilize if you find yourself using many
different languages or many different versions of the same language. You should check it out
and see if it can improve your workflow.

Use asdf to manage Python, NodeJS, GoLang and more!

I'm currently teaching my wife to code and one of the problems that we
worked on to teach her some fundamental programming concepts was
re-implementing the caesar cipher in python. It was fun not only to code
but to also start sending each other "secret" messages!
The caesar cipher is a rather simple encoding, you just shift the
alphabet a certain amount of characters. For example, if you are using a
shift of 2:
<pre><code class="hljs language-ini">a => c
b => d
y => a
z => b
</code></pre>
Using this as an interview type question would provide a few interesting
problems and give you a good perspective on how good a developers
problem solving skills are and how knowledgeable they are in the
language of their choice.
The first issue is to handle the beginning and end of the alphabet, if
you are encoding 'z' then you will have to start your shift on a. The
second problem is to only encode letters since there was no ascii table
to define in what order characters are shifted back in those times.
Without using too much of the built in python niceties you could do
something similar to this:
<pre><code class="hljs language-python">#!/usr/bin/python
def decode_shift_letter(current_ord, start, end, shift):
 if current_ord - shift &#x3C; start:
 new_ord = (current_ord + 26) - shift
 return chr(new_ord)
 else:
 return chr(current_ord-shift)

def encode_shift_letter(current_ord, start, end, shift):
 if current_ord + shift > end:
 new_ord = (current_ord - 26) + shift
 return chr(new_ord)
 else:
 return chr(current_ord+shift)

def decode(input, shift):
 return modify_input(input, shift, decode_shift_letter)

def encode(input, shift):
 return modify_input(input, shift, encode_shift_letter)

def modify_input(input, shift, shift_letter):
 new_sentence = ''

 for letter in input:
 # we only encode letters, random characters like +!%$ are not encoded.
 # Lower and Capital letters are not stored near each other on the
 # ascii table
 lower_start = ord('a')
 lower_end = ord('z')
 upper_start = ord('A')
 upper_end = ord('Z')
 current_ord = ord(letter)

 if current_ord >= lower_start and current_ord &#x3C;= lower_end:
 new_sentence += shift_letter(current_ord, lower_start, lower_end, shift)
 elif current_ord >= upper_start and current_ord &#x3C;= upper_end:
 new_sentence += shift_letter(current_ord, upper_start, upper_end, shift)
 else:
 new_sentence += letter

 return new_sentence


def get_shift():
 try:
 shift = int(raw_input('What shift would you like to use?\n'))
 except ValueError:
 print 'Shift must be a number'
 shift = get_shift()

 if not (shift > 0 and shift &#x3C;= 25):
 print 'Shift must be between 1 and 25'
 shift = get_shift()

 return shift

def main():
 try:
 task = int(raw_input('1) Encode \n'+ \
 '2) Decode \n'))
 except ValueError:
 print 'Invalid task, try again!'
 main()

 shift = get_shift()
 input = raw_input('What message would you like to %s\n' % ('Encode' if task == 1 else 'Decode'))

 if task == 1:
 print encode(input, shift)
 elif task == 2:
 print decode(input, shift)

if __name__ == '__main__':
 main()
</code></pre>
This would prove that you are a decent problem solver and have enough of
the language to get things done but if you want to prove you have
mastered the python language you might take advantage of some slicing
and some methods out of the string module and change your code to look
something like:
<pre><code class="hljs language-python">from string import letters, maketrans

def decode(input, shift):
 return modify_input(input, -shift)

def encode(input, shift):
 return modify_input(input, shift)

def modify_input(input, shift):
 trans = maketrans(letters, letters[shift:] + letters[:shift])
 return input.translate(trans)
</code></pre>
Do get more information on string.letters and string.maketrans you can
visit their documentation
[here](<a href="http://docs.python.org/library/string.html">http://docs.python.org/library/string.html</a>)

Caesar Cipher in Python

A fun interview question some developers like to ask is to have you
convert ascii characters to an integer without using built in methods
like string.atoi or int().
So using python the obvious ways to convert a string to an integer are
these:
<pre><code class="hljs language-python">>>> int('1234')
1234
>>> import string
>>> string.atoi('1234')
1234
</code></pre>
The interesting thing here is finding out where on the ascii character
table the number is. Luckily python has this already built in with the
ord method:
<pre><code class="hljs language-python">>>> help(ord)

ord(...)
 ord(c) -> integer

 Return the integer ordinal of a one-character string.

>>> ord('1')
49
>>> ord('2')
50
</code></pre>
You can see that the numbers are grouped together on the ascii table, so
you just have to grab '0' as the base and subtract the rest:
<pre><code class="hljs language-python">>>> ord('1')-ord('0')
1
</code></pre>
So if we have the string '1234', we can get each of the individual
numbers by looping over it:
<pre><code class="hljs language-python">>>> num_string = '1234'
>>> num_list = []
>>> base = ord('0')
>>> for num in num_string:
... num_list.append(ord(num) - base)
...
>>> print num_list
[1, 2, 3, 4]
</code></pre>
but now how to we combine all these together to get 1234? You can't
just add them up because you'll just get 1+2+3+4 = 10.
So, we have to get 1000 + 200 + 30 + 4, which is a simple problem to
solve. Its just number times 10 to the nth power, so the final solution
is:
<pre><code class="hljs language-python">num = '1234'
new_num = 0
base = ord('0')

for i,n in enumerate(reversed(num)):
 new_num += (ord(n) - base) * (10**i)

print new_num
</code></pre>
This code is a little verbose though, lets make it a dirty nasty one
liner!
<pre><code class="hljs language-python">>>> sum([(ord(n)-ord('0')) * (10 ** i) for i,n in enumerate(reversed('1234'))])
1234
</code></pre>

Convert a string to an integer in Python

I believe every project should have a CLI built around the standard workflows of developing
on the project. Things like:
<ul>
<li>Install dependencies</li>
<li>Run tests</li>
<li>Run linter and formatters</li>
<li>Build project</li>
<li>Start / Stop the docker environment</li>
</ul>
The reason I think this is important is because it makes a nice consistent and discoverable
entrypoint for understanding how you should work in the project. If you only provide the
instructions in the <code>README</code> then you have to remember to update those docs every time you
add a new command. Those docs aren't easily testable either.
Most of my career the command runner of choice for my projects as been <code>GNU Make</code> but it was
definitely the wrong tool for the job. It is a build tool that I bent into shape to work
as a command runner for me. These days I use the tool <a href="https://github.com/casey/just">just</a>.
<h2>Intro to just</h2>
<a href="https://github.com/casey/just">Just</a> is a modern command runner with a similar syntax to <code>make</code>
that provides a nice way for building out your project CLI! You create a file named <code>justfile</code>
at the root of your project and then the basic syntax is:
<pre><code class="hljs language-make">help:
 @just --list

# My first command
first:
 echo "Any commands you want to run go here!"
</code></pre>
The first <code>help</code> line defines a command "help" for your CLI and it lists out all the other available
commans. I always put this line first because <code>just</code> runs the first command in the file if a specific
command isn't requested. The output of this file looks like this:
<pre><code class="hljs language-bash">❯ just
Available recipes:
 first # My first command
 help
</code></pre>
Having help automatically generated is fantastic! Its also really helpful that it adds the comment
to the command so that each command is self-documenting. If you run the <code>first</code> command you'll notice
it also has a feature where it prints out the commands being ran so the user knows exactly what is
happening:
<pre><code class="hljs language-bash">❯ just first
echo "Any commands you want to run go here!"
Any commands you want to run go here!
</code></pre>
This doesn't always make sense though, so you can quickly remove that behavior by putting an <code>@</code> in front
of any of the commands, like I did for the <code>help</code> command above. You can also declare dependencies if
you have re-usable parts of your workflow that many of your commands need.
For example, you might want to check versions of things like <code>node</code> and <code>python</code> before running the install
of their dependencies. So you could do something like:
<pre><code class="hljs language-make">help:
 @just --list

node_version := "v17.6.0"

# Verify system dependencies
check-dependencies:
 @if [ ! "$(node --version)" = {{ node_version }} ]; \
 then \
 echo "Missing node version: {{ node_version }}"; \
 exit 1; \
 fi

# Install frontend
install: check-dependencies
 @echo "yarn install"
</code></pre>
which ends up with a CLI that looks like this:
<pre><code class="hljs language-bash">❯ just
Available recipes:
 check-dependencies # Verify system dependencies
 help
 install # Install frontend

❯ just install
Missing node version: v17.6.0
error: Recipe `check-dependencies` failed on line 12 with exit code 1
</code></pre>
This opens up a lot of possibilities! In the above <code>justfile</code> you'll notice I'm using a multi-line
command but I have <code>\</code> at the end of each line. This is because <code>just</code> by default is going to run
each new line in their own shell. So this just makes all those lines run in the same shell.
You do not have to use this syntax though. Just is <code>polyglot</code> and can run commands from any language
you would like.
<h3>Polyglot</h3>
If you want to use a bash script as one of your commands, you can do so by adding a shebang at the top:
<pre><code class="hljs language-make">check-dependencies:
 #!/usr/bin/env bash
 set -euxo pipefail
 if [ ! "$(node --version)" = {{ node_version }} ];
 then
 echo "Missing node version: {{ node_version }}"
 exit 1
 fi
</code></pre>
Now the entire command is using a bash script to execute! This gets really interesting if you want to start
using things like python, so if you'd like to change the dependency checker above to python:
<pre><code class="hljs language-python">check-dependencies:
 #!/usr/bin/env python3
 import subprocess
 result = subprocess.run(
 ['node', '--version'],
 stdout=subprocess.PIPE
 )
 if result != "{{ node_version }}":
 print(f"Missing node version: {{ node_version }}")
 exit(1)
</code></pre>
You can even tell <code>just</code> that you want to use a specific language for all commands!
<pre><code class="hljs language-arduino">set shell := ["python3", "-c"]
</code></pre>
This not only affects the commands you have in your recipe but also anything inside
backticks! So something like:
<pre><code class="hljs language-make">`print("Rust is the best programming language")`
</code></pre>
It would run through python instead of the shell.
<h3>Enviornment Files</h3>
One of the other modern things <code>just</code> adds to your workflow is the ability to utilize dotenv
files. So for example if you want to define which port you launch your http server on, you can
create a file called <code>.env</code>:
<pre><code class="hljs language-bash">WEBSERVER_PORT=9000
</code></pre>
and then utilize it in your <code>justfile</code>:
<pre><code class="hljs language-make">set dotenv-load

http:
 @echo "Starting webserver in current directory"
 python3 -m http.server $WEBSERVER_PORT
</code></pre>
When you run <code>just http</code> it'll launch the http server on port 9000. One important line
in this file is <code>set dotenv-load</code>, it will not load the <code>.env</code> file without you telling it to.
<h2>Don't use language specific scripts!</h2>
I'n not a fan of language specific command runners like <code>package.json</code> in the node community.
It always frustrates me when I start working on a project that heavily uses <code>scripts</code> in their
package.json instead of using a real command runner. <code>json</code> is not a great format for writing
discoverable CLI commands. For example if you wanted to write a <code>next.js</code> build script:
<pre><code class="hljs language-json"> "scripts": {
 "predeploy": "yarn build &#x26;&#x26; yarn export &#x26;&#x26; touch dist/.nojekyll &#x26;&#x26; echo sontek.net > dist/CNAME",
 "deploy": "gh-pages -d dist -t true",
 "build": "next build",
 "export": "next export -o dist/",
 },
</code></pre>
Combining all those commands is really messy and not easily understandable through <code>yarn run</code>:
<pre><code class="hljs language-bash">❯ yarn run
yarn run v1.22.17
info Commands available from binary scripts: autoprefixer, browserslist, css-blank-pseudo, css-has-pseudo, css-prefers-color-scheme, cssesc, esparse, esvalidate, extract-zip, gh-pages, gh-pages-clean, js-yaml, loose-envify, nanoid, next, prettier, resolve, rimraf, semver, svgo, uvu
info Project commands
 - build
 next build
 - deploy
 gh-pages -d dist -t true
 - export
 next export -o dist/
 - predeploy
 yarn build &#x26;&#x26; yarn export &#x26;&#x26; touch dist/.nojekyll &#x26;&#x26; echo sontek.net > dist/CNAME
</code></pre>
I'd much rather have this:
<pre><code class="hljs language-bash">❯ just
Available recipes:
 build # Build frontend assets
 deploy # Deploy assets to cloudfront
 export # Export to static assets (no SSR)
</code></pre>
<h2>Conclusion</h2>
<a href="https://github.com/casey/just">Just</a> is a wonderful tool for building project specific CLIs without much effort. It is
a great replacement for <code>Make</code> if you are using it as a command runner and it has most of the features you'd need.
I recommend adding a <code>justfile</code> to your projects today! If you'd like to see a real world example of how to use <code>just</code>,
you can check out the one I use to maintain my <a href="https://github.com/sontek/homies/blob/master/justfile">home directory</a>!

Automate project workflows with the command runner Just!

This article will show you how to use libvirt to create base images that
can be uploaded to OpenStack.
<h1>Why would you want to do this?</h1>
Linux distributions like Fedora and Ubuntu already ship "cloud" images
and most providers also have their own custom images for you to use, but
I find it much more comforting to have full control of the software that
is installed and I like the ability to easily apply new security patches
to base images.
I wouldn't use images to replace config management (CM) with something
like <a href="http://www.saltstack.com/">Salt</a> or
<a href="http://www.ansible.com/">Ansible</a> but they are nice to give sane system
defaults in things like <code>grub.conf</code>, <code>sysctl.conf</code>, and shipping a Chef
or Salt agent so that your CM engine can communicate with your server
right away.
<h1>Setting up your environment</h1>
The first thing you need to do is get a minimal install disk for the
Linux distribution you want to use. I prefer using Fedora netinst disks
but another popular option is Ubuntu Server.
To get the latest Fedora here, you can choose "netinst" under Direct
Downloads: <a href="http://fedoraproject.org/en/get-fedora-all">http://fedoraproject.org/en/get-fedora-all</a>
To get the latest Ubuntu you can go here:
<a href="http://www.ubuntu.com/download/server">http://www.ubuntu.com/download/server</a>
Once you have acquired your distribution of choice you just need to
verify that you have <code>virt-install</code> and <code>virt-viewer</code> installed:
Fedora:
<pre><code class="hljs language-bash">yum install virt-install virt-viewer
</code></pre>
Ubuntu:
<pre><code class="hljs language-bash">apt-get install virtinst virt-viewer
</code></pre>
If you prefer a graphical user interface, you may use <code>virt-manager</code>
instead, but I try to keep everything in the CLI; that way it can be
repeated easily.
<h1>Preparing your disk</h1>
Now that you have a base ISO and the tools necessary, let's get started
by creating a disk to install the virtual server into. Resizing an image
isn't an impossible task but it is much easier to choose a reasonably
sized disk for the purpose it will be used for.
I primarily use 8 GB disks -- that way we can fit all the system
components required as well as our own web applications. Any large files
should be placed in a SAN or something like Dreamhost's dreamobjects.
The other big decision you must make upfront is what disk format you
want to use -- the trade-off is disk space vs performance. The two
primary formats are qcow2 (QEMU Copy on Write) and Raw. qcow2 is great
if you have limited disk space and don't want to allocate the full 8 GB
up front. Raw is preferred if you want the best performance.
If you choose qcow2, you'll also need to make sure you have <code>qemu-img</code>:
Fedora:
<pre><code class="hljs language-bash">yum install qemu-img
</code></pre>
Ubuntu:
<pre><code class="hljs language-bash">apt-get install qemu-utils
</code></pre>
Create a raw disk:
<pre><code class="hljs language-bash">fallocate -l 8192M server.img
</code></pre>
Create a qcow2 disk:
<pre><code class="hljs language-bash">qemu-img create -f qcow2 server.qcow2 8G
</code></pre>
<h1>Installing your distribution onto the disk</h1>
We will use the <code>virt-install</code> command to get the distribution installed
onto the disk image.
To install Fedora on a qcow2 disk image:
<pre><code class="hljs language-bash">virt-install --name base_server --ram 1024 --cdrom=./Fedora-20-x86_64-netinst.iso \
--disk path=./server.qcow2,format=qcow2
</code></pre>
To install Ubuntu Server on a raw disk image:
<pre><code class="hljs language-bash">virt-install --name base_server --ram 1024 --cdrom=./ubuntu-12.04.4-server-amd64.iso \
--disk path=./server.img,format=raw
</code></pre>
You should follow the standard install steps that you normally would
when setting up your distribution. But here are some tips for each:
Fedora:
<ul>
<li>Choose minimal install -- by default it selects "GNOME".</li>
</ul>
Ubuntu:
<ul>
<li>
Be sure to select OpenSSH server -- it won't install it by
default.
</li>
<li>
On Ubuntu 12.04, there is a bug that makes it hang after running
<code>fsck</code>. You will need to edit grub to get it to boot, hit _<a href="">e</a> at
the boot prompt and add "nomodeset" on the linux line. You will
know that you need to do this if your boot hangs on fsck:
<pre><code class="hljs language-bash">fsck from util-linux 2.20.1
/dev/mapper/ubuntu--vg-root: clean, 57106/441504 files, 286779/1764352 blocks
/dev/sda1: clean, 230/62248 files, 39833/248832 blocks
</code></pre>
</li>
</ul>
<h1>Preparing image for openstack</h1>
To prepare a virtual machine for the cloud, you will need to install the
<code>cloud-init</code> package, which allows the cloud providers to inject certain
system settings when creating servers based on the image. These are
things like hostname and ssh keys.
On Fedora:
<pre><code class="hljs language-bash">yum install cloud-init
</code></pre>
On Ubuntu:
<pre><code class="hljs language-bash">apt-get install cloud-init
</code></pre>
Then you need to just configure <code>cloud-init</code> by editing
<code>/etc/cloud/cloud.cfg</code> and update the <code>datasources_list</code> section to
include EC2. OpenStack uses EC2 metadata for <code>cloud-init</code>.
You should also verify the user setting in this same config and define
the user you plan to use, it will be where the <code>authorized_keys</code> file is
setup for when the cloud provider injects your SSH key into the server.
<code>cloud-init</code> will not create the user for you, it will just assign the
SSH keypair and reset the password. So make sure the user defined in
<code>cloud.cfg</code> is also created on the system.
Once you have your <code>cloud-init</code> settings the way you want them, just
shutdown and run the <code>virt-sysprep</code> command.
On the guest machine:
<pre><code class="hljs language-bash">shutdown -h now
</code></pre>
On the host machine:
<pre><code class="hljs language-bash">virt-sysprep -d base_server
</code></pre>
<h1>Uploading your image to OpenStack</h1>
Using the glance API it is very straightforward to upload the image to
OpenStack. Just run the following command:
<pre><code class="hljs language-bash">glance image-create --name base_server --disk-format=qcow2 \
--container-format=bare --is-public=True --file server.qcow2 --progress
</code></pre>
Once the image upload completes you will be able to use it immediately
within nova. You can reference it by name or by the id from [glance
image-list]{.title-ref}.
To create your first instance from the image:
<pre><code class="hljs language-bash">nova boot --flavor m1.tiny --image base_server --key-name devops \
--security-groups free_for_all test_server
</code></pre>
Obviously the security groups, key name, and flavors are based on your
installation of OpenStack but can all easily be queried from the nova
API:
<pre><code class="hljs language-bash">nova flavor-list
nova secgroup-list
nova keypair-list
</code></pre>
And you are done! You'll be able to re-use your new image as a base for
all new instances you spin up in openstack!

Preparing custom images for OpenStack

How to speak spanish like a colombian drug lord!

I have seen a lot of people switch over to using ipython, bpython, etc
to get auto-complete support without realizing that the standard
interpreter does have this functionality.
To enable auto-complete support in the python interpreter you need to
create a python startup file that enables readline support. A python
startup file is just a bunch of python code that gets executed at
startup of the interpreter. To do this you just setup PYTHONSTARTUP in
your ~/.bashrc and then create a ~/.pythonrc.py file:
<pre><code class="hljs language-python">#.bashrc
PYTHONSTARTUP=~/.pythonrc.py
export PYTHONSTARTUP

#.pythonrc.py
try:
 import readline
except ImportError:
 print("Module readline not available.")
else:
 import rlcompleter
 readline.parse_and_bind("tab: complete")
</code></pre>
Now when you are in python you have tab completion on importing, calling
methods on a module, etc.
<pre><code class="hljs language-python">>>> import o
object( oct( open( or ord( os
</code></pre>
I always end up using the pretty print module for viewing long lists and
strings in the interpreter so I prefer to just use it by default:
<pre><code class="hljs language-python"># Enable Pretty Printing for stdout
import pprint
def my_displayhook(value):
 if value is not None:
 try:
 import __builtin__
 __builtin__._ = value
 except ImportError:
 __builtins__._ = value

 pprint.pprint(value)

sys.displayhook = my_displayhook
</code></pre>
It is also very useful to be able to load up your favorite editor to
edit lines of code from the interpreter, you can do this by adding the
following into your ~/.pythonrc.py:
<pre><code class="hljs language-python">import os
import sys
from code import InteractiveConsole
from tempfile import mkstemp

EDITOR = os.environ.get('EDITOR', 'vi')
EDIT_CMD = '\e'

class EditableBufferInteractiveConsole(InteractiveConsole):
 def __init__(self, *args, **kwargs):
 self.last_buffer = [] # This holds the last executed statement
 InteractiveConsole.__init__(self, *args, **kwargs)

 def runsource(self, source, *args):
 self.last_buffer = [ source.encode('latin-1') ]
 return InteractiveConsole.runsource(self, source, *args)

 def raw_input(self, *args):
 line = InteractiveConsole.raw_input(self, *args)
 if line == EDIT_CMD:
 fd, tmpfl = mkstemp('.py')
 os.write(fd, b'\n'.join(self.last_buffer))
 os.close(fd)
 os.system('%s %s' % (EDITOR, tmpfl))
 line = open(tmpfl).read()
 os.unlink(tmpfl)
 tmpfl = ''
 lines = line.split( '\n' )
 for i in range(len(lines) - 1): self.push( lines[i] )
 line = lines[-1]
 return line

c = EditableBufferInteractiveConsole(locals=locals())
c.interact(banner='')

# Exit the Python shell on exiting the InteractiveConsole
sys.exit()
</code></pre>
For Django developers when you load up the ./manage.py shell it is nice
to have access to all your models and settings for testing:
<pre><code class="hljs language-python"># If we're working with a Django project, set up the environment
if 'DJANGO_SETTINGS_MODULE' in os.environ:
 from django.db.models.loading import get_models
 from django.test.client import Client
 from django.test.utils import setup_test_environment, teardown_test_environment
 from django.conf import settings as S

 class DjangoModels(object):
 """Loop through all the models in INSTALLED_APPS and import them."""
 def __init__(self):
 for m in get_models():
 setattr(self, m.__name__, m)

 A = DjangoModels()
 C = Client()
</code></pre>
After these tweaks the python interpreter is a lot more powerful and you
really lose the need for the more interactive shells like ipython and
bpython. All of these settings work in both python2 and python3.
If you want to see my complete ~/.pythonrc.py you can get it on
<a href="https://github.com/sontek/dotfiles/blob/master/_pythonrc.py">github</a>

Tips and Tricks for the Python Interpreter

Turning Vim into a modern Python IDE

TL;DR: Putting it all together, the full code can be found here:
<a href="https://gist.github.com/1420255">https://gist.github.com/1420255</a>
<h1>Intro</h1>
Pyramid's documentation doesn't cover the preferred way to test with
SQLAlchemy, because Pyramid tries to stay out of your way and allow you
to make your own decisions. However, I feel i'ts necessary to document
what I think is the best way to test.
When I first started writing tests with SQLAlchemy I found plenty of
examples of how to to get started by doing something like this:
<pre><code class="hljs language-python">from db import session # probably a contextbound sessionmaker
from db import model

from sqlalchemy import create_engine

def setup():
 engine = create_engine('sqlite:///test.db')
 session.configure(bind=engine)
 model.metadata.create_all(engine)

def teardown():
 model.metadata.drop_all(engine)

def test_something():
 pass
</code></pre>
I have seen this done so many times, but I feel there is so much wrong
with it! So let's establish some base rules when testing:
<blockquote>
<ul>
<li>Always test your system like it would be used in production.
SQLite does not enforce the same rules or have the same features
as Postgres or MySQL and will allow tests to pass that would
otherwise fail in production.</li>
<li>Tests should be fast! You should be writing tests for all your
code. This is the main reason people do test against SQLite, but
we can't violate rule number one. We have to make sure tests
against Postgres are fast, so we shouldn't be tearing down and
recreating tables for every single test.</li>
<li>You should be able to execute in parallel to speed up when you
have thousands of tests. Dropping and creating tables per test
would not work in a parallel environment.</li>
</ul>
</blockquote>
For an example, I have a project with 600+ tests and it would take 2 and
half minutes to execute running against SQLite. But when we swapped our
test configuration to execute against Postgres, testing took well over
an hour. That is unacceptable!
But running them in parallel will give us a huge speed up. Check out the
results of the tests running in single proc mode vs using all 4 cores:
<pre><code class="hljs language-ini">$ py.test
======= 616 passed in 143.67 seconds =======

$ py.test -n4
======= 616 passed in 68.12 seconds =======
</code></pre>
<h1>The right way</h1>
So what is the proper way to setup your tests? You should initialize the
database when you start your test runner and then use transactions to
rollback any data changes your tests made. This allows you to keep a
clean database for each test in a very efficient way.
In py.test, you just have to create a file called conftest.py that looks
similar to:
<pre><code class="hljs language-python">import os

ROOT_PATH = os.path.dirname(__file__)

def pytest_sessionstart():
 from py.test import config

 # Only run database setup on master (in case of xdist/multiproc mode)
 if not hasattr(config, 'slaveinput'):
 from models import initialize_sql
 from pyramid.config import Configurator
 from paste.deploy.loadwsgi import appconfig
 from sqlalchemy import engine_from_config
 import os

 ROOT_PATH = os.path.dirname(__file__)
 settings = appconfig('config:' + os.path.join(ROOT_PATH, 'test.ini'))
 engine = engine_from_config(settings, prefix='sqlalchemy.')

 print 'Creating the tables on the test database %s' % engine

 config = Configurator(settings=settings)
 initialize_sql(settings, config)
</code></pre>
With py.test, when you are running in parallel mode, the
pytest_sessionstart hook gets fired for each node, so we check that we
are on the master node. Then we just grab our test.ini configuration
file and execute the initialize_sql function.
Now that you have your initial test configuration finished, you have to
define a base test class that does the transaction management in setUp
and teardown.
First, lets setup the Base testing class what will manage our
transactions:
<pre><code class="hljs language-python">import unittest
from pyramid import testing
from paste.deploy.loadwsgi import appconfig

from webtest import TestApp
from mock import Mock

from sqlalchemy import engine_from_config
from sqlalchemy.orm import sessionmaker
from app.db import Session
from app.db import Entity # base declarative object
from app import main
import os
here = os.path.dirname(__file__)
settings = appconfig('config:' + os.path.join(here, '../../', 'test.ini'))

class BaseTestCase(unittest.TestCase):
 @classmethod
 def setUpClass(cls):
 cls.engine = engine_from_config(settings, prefix='sqlalchemy.')
 cls.Session = sessionmaker()

 def setUp(self):
 connection = self.engine.connect()

 # begin a non-ORM transaction
 self.trans = connection.begin()

 # bind an individual Session to the connection
 Session.configure(bind=connection)
 self.session = self.Session(bind=connection)
 Entity.session = self.session

 def tearDown(self):
 # rollback - everything that happened with the
 # Session above (including calls to commit())
 # is rolled back.
 testing.tearDown()
 self.trans.rollback()
 self.session.close()
</code></pre>
This base test case will wrap all your sessions in an external
transaction so that you still have the ability to call flush/commit/etc
and it will still be able to rollback any data changes you make.
<h1>Unit Tests</h1>
Now there are a few different types of tests you will want to run.
First, you will want to do unit tests, which are small tests that only
test 1 thing at a time. This means you will skip the routes, templates,
etc. So let's setup our Unit Test Base class:
<pre><code class="hljs language-python">class UnitTestBase(BaseTestCase):
 def setUp(self):
 self.config = testing.setUp(request=testing.DummyRequest())
 super(UnitTestBase, self).setUp()

 def get_csrf_request(self, post=None):
 csrf = 'abc'

 if not u'csrf_token' in post.keys():
 post.update({
 'csrf_token': csrf
 })

 request = testing.DummyRequest(post)

 request.session = Mock()
 csrf_token = Mock()
 csrf_token.return_value = csrf

 request.session.get_csrf_token = csrf_token

 return request
</code></pre>
We built in a utility function to help us test requests that require a
csrf token as well. Here is how we would use this class:
<pre><code class="hljs language-python">class TestViews(UnitTestBase):
 def test_login_fails_empty(self):
 """ Make sure we can't login with empty credentials"""
 from app.accounts.views import LoginView
 self.config.add_route('index', '/')
 self.config.add_route('dashboard', '/')

 request = testing.DummyRequest(post={
 'submit': True,
 })

 view = LoginView(request)
 response = view.post()
 errors = response['errors']

 assert errors[0].node.name == u'csrf_token'
 assert errors[0].msg == u'Required'
 assert errors[1].node.name == u'Username'
 assert errors[1].msg == u'Required'
 assert errors[2].node.name == u'Password'
 assert errors[2].msg == u'Required'


 def test_login_succeeds(self):
 """ Make sure we can login """
 admin = User(username='sontek', password='temp', kind=u'admin')
 admin.activated = True
 self.session.add(admin)
 self.session.flush()

 from app.accounts.views import LoginView
 self.config.add_route('index', '/')
 self.config.add_route('dashboard', '/dashboard')

 request = self.get_csrf_request(post={
 'submit': True,
 'Username': 'sontek',
 'Password': 'temp',
 })

 view = LoginView(request)
 response = view.post()

 assert response.status_int == 302
</code></pre>
<h1>Integration Tests</h1>
The second type of test you will want to write is an integration test.
This will integrate with the whole web framework and actually hit the
define routes, render the templates, and actually test the full stack of
your application.
Luckily this is pretty easy to do with Pyramid using WebTest:
<pre><code class="hljs language-python">class IntegrationTestBase(BaseTestCase):
 @classmethod
 def setUpClass(cls):
 cls.app = main({}, **settings)
 super(IntegrationTestBase, cls).setUpClass()

 def setUp(self):
 self.app = TestApp(self.app)
 self.config = testing.setUp()
 super(IntegrationTestBase, self).setUp()
</code></pre>
In setUpClass, we run the main function of the applications
__init__.py that sets up the WSGI application and then we wrap it in
a TestApp that gives us the ability to call get/post on it.
Here is an example of it in use:
<pre><code class="hljs language-python">class TestViews(IntegrationTestBase):
 def test_get_login(self):
 """ Call the login view, make sure routes are working """
 res = self.app.get('/login')
 self.assertEqual(res.status_int, 200)

 def test_empty_login(self):
 """ Empty login fails """
 res = self.app.post('/login', {'submit': True})

 assert "There was a problem with your submission" in res.body
 assert "Required" in res.body
 assert res.status_int == 200

 def test_valid_login(self):
 """ Call the login view, make sure routes are working """
 admin = User(username='sontek', password='temp', kind=u'admin')
 admin.activated = True
 self.session.add(admin)
 self.session.flush()

 res = self.app.get('/login')

 csrf = res.form.fields['csrf_token'][0].value

 res = self.app.post('/login',
 {
 'submit': True,
 'Username': 'sontek',
 'Password': 'temp',
 'csrf_token': csrf
 }
 )

 assert res.status_int == 302
</code></pre>
<h1>Problems with this approach</h1>
If a test causes an error that will prevent the transaction from rolling
back, such as closing the engine, then this approach will leave your
database in a state that might cause other tests to fail.
If this happens tracing the root cause could be difficult but you should
be able to just look at the first failed test unless you are running the
tests in parallel.
If you are good about writing and running your tests regularly you
should be able to catch individual tests causing issues like this fairly
quickly.
<h1>Resources</h1>
<a href="http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/testing.html">http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/testing.html</a>
<a href="http://www.sqlalchemy.org/docs/orm/session.html#joining-a-session-into-an-external-transaction">http://www.sqlalchemy.org/docs/orm/session.html#joining-a-session-into-an-external-transaction</a>
John Anderson &#x3C;<a href="mailto:sontek@gmail.com">sontek@gmail.com</a>> 2011