chewbranca.com

---
layout: post
title:  "Emacs + EDTS + (big)CouchDB... a journey"
date:   2014-03-14 21:05:46
categories: tech
tags: emacs edts couchdb bigcouch erlang
---


If you use Emacs to write Erlang, you might have heard of
[EDTS](https://github.com/tjarvstrand/edts). It's a recent Erlang
development suite for Emacs, and it has some impressive features, but
it can be a beast to get setup and working properly with CouchDB and
the BigCouch merge branch.

Current CouchDB master's folder hierarchy is non standard for Erlang
applications, and as a result it does not work well with EDTS. There
is work being done in the BigCouch merge branch and the RCouch merge
branch to restructure things in a standard OTP compliant
hierarchy. This guide is for Emacs + EDTS + the
[BigCouch merge branch](https://github.com/apache/couchdb/tree/1843-feature-bigcouch).

If you just want to load a vanilla VM with no running applications,
then just follow the standard EDTS config, but that's boring. This
guide will get you setup connecting Emacs/EDTS to your local BigCouch
dev cluster, and provides all the nice EDTS features like automatic
code reloading in the running VM, Emacs repl with tab completion and
documentation, show callers and everything else in EDTS.


*WARNING* this is quite a hack, and involves a number of changes and
 updates. I hope that by writing this up I'll get some feedback on how
 to do things cleaner, but it works, and that's the important part ;-)


## Steps

We'll need to update the EDTS start script, make an .edts config file,
and use a hacked together function for spinning up the repl.

Also worth noting is that this setup expects you to manually run
`./rel/boot_dev_cluster.sh` in the BigCouch branch to spin up the
VM. If you forget to do that, EDTS will spin up a VM for you and then
when you try to boot the dev cluster you'll get errors from trying to
run multiple Erlang VMs with the same name.

### Update EDTS start script

We need to update the base EDTS VM start script to use `-name` instead
of `-sname` so that it can talk to the CouchDB nodes. I also like to
add `-hidden` so that th EDTS VM doesn't show up in the list of nodes
in the three node dev cluster. Here's my diff to the [start script](https://github.com/tjarvstrand/edts/blob/master/start):


{% highlight diff %}
diff --git a/start b/start
index de68605..c744e72 100755
--- a/start
+++ b/start
@@ -27,8 +27,9 @@ PLUGIN_DIR=$EDTS_HOME/plugins

cd $EDTS_HOME
exec $ERL \
-    -sname edts \
+    -name edts@127.0.0.1 \
     -edts project_data_dir "\"$PROJDIR\"" \
     -edts plugin_dir "\"$PLUGIN_DIR\"" \
     -pa $EDTS_HOME/{lib,plugins}/*/ebin \
-    -s edts_app
+    -s edts_app \
+    -hidden
{% endhighlight %}

### Add a .edts config

EDTS works by having a .edts config in the top level of your
project. We need to let the config know where to find all the
projects, and we also need to specify the name of the VM. It's
important that the name matches the name of the local dev cluster node
you want to connect to, as EDTS will connect to the existing VM we
start with the dev script, rather than starting a new VM. This way any
changes you make will be immediately reflected in the running VM,
allowing you to curl again or work in remsh without having to manually
reload the modules or dev server. Here's my config, it's pretty
simple:

{% highlight common-lisp %}
:name "dev1"
:lib-dirs '("src")
{% endhighlight %}

Now when you open a `.erl` file within the CouchDB source folder, EDTS
will start and then you will connect to the running dev server. At
this point you'll be able to save files and have them loaded in the
VM, but you won't have a repl available to you. For some reason EDTS
does not start a repl for nodes in remsh's into.

### Opening an inferior Erlang remsh

I cobbled together a little function to spin up an inferior Erlang
shell and remsh into the dev cluster. This works, and let's you
interact with the running VM, but you lose out on all the niceties of
the EDTS shell, like tab completion and inline documentation
boxes.


{% highlight common-lisp %}
;; adapted from: https://github.com/adbl/tools-emacs/blob/master/david.emacs
(defun erlang-shell-connect-to-node (name)
  (interactive "MNode name to connect to: ")
  (let* ((inferior-erlang-machine-options
          (list "-hidden"
                "-name" (format "emacs-remsh-%s" name)
                "-remsh" (format "%s@127.0.0.1" name))))
    (erlang-shell-display)))
{% endhighlight %}

### Hacking together an EDTS remsh

I decided I really wanted to have the proper EDTS repl, so I dove in
and hacked one out today. I updated
[edts-shell.el](https://github.com/tjarvstrand/edts/blob/master/elisp/edts/edts-shell.el)
with a new hardcoded function to connect to dev1 through a remsh. It's
ugly, hardcoded, and it even throws an error at you when you run it,
but it works!

{% highlight common-lisp %}
(defun edts-shell-dev1 (&optional pwd switch-to)
  "Start an interactive erlang shell."
  (interactive '(nil t))
  (edts-ensure-server-started)
  (let*((buffer-name "*dev1-shell")
        (node-name   "dev1@127.0.0.1")
        ;; (command     (list "erl" "-name dev1-remsh -hidden -remsh" node-name))
        (command (list "/Users/russell/src/dotfiles/emacs.d/plugins/edts/start_dev1"))
        (root        (expand-file-name (or pwd default-directory))))
    (let ((buffer (edts-shell-make-comint-buffer
                   buffer-name
                   node-name
                   root
                   command)))
      (edts-init-node-when-ready node-name node-name root nil)
      (when switch-to (switch-to-buffer buffer))
      buffer)))
{% endhighlight %}

I was having trouble properly passing the Erlang VM command line
options, so I switched to using a little shell script to start the
VM. Ideally the commented out command declaration should do the trick,
but I was having issues with it. Here's my start_dev1 script:

{% highlight bash %}
#/usr/bin/env bash

exec /usr/local/bin/erl -name dev1-remsh -hidden -remsh dev1@127.0.0.1
{% endhighlight %}

Hopefully someone has a cleaner approach for this, but it works for
now.

## Results

With this in place I've got a solid Emacs + EDTS dev environment for
hacking on (big)CouchDB. If you have any feedback or suggestions,
please don't be shy as this is definitely a work in progress and I see
it as an minimal working version, not a finished configuration. I
would also like to better understand interacting with running VMs with
EDTS, it seems to be designed with the assumption you're working
locally with a VM started by EDTS, not a running environment. I'm
curious if the author of EDTS has any suggestions on better approach
here.

## Gotchas

Some random things to keep in mind.

### Single node interaction only

Right now you connect to an individual node, in this case dev1. The
local dev cluster spins up dev2 and dev3 as well, but this does not
connect to those nodes, and does not load code with `nl` to upgrade
all nodes with your modifications. I would like to fix this to at
least do an `nl`. You'll occasionally run into whacky behavior on the
dev cluster, just restart it and run `M-x edts-project-node-init` to
reconnect.

### Disconnected from dev1 VM

If you restart the dev server, you'll lose the connection from EDTS
and will get an error when it tries to push your changes. I just do
`M-x edts-project-node-init` to get back up and running.

### Show source under cursor doesn't go into Erlang/OTP source

I was having issues following code into the full OTP source, and I
tracked this down to the compile info stored in the beams pointing to
a directory that did not exist on my system. I'm not sure what
happened there, I was using erlbrew but ended up going back to erln8
and now I can follow code down into the Erlang/OTP source.

### Built in manual

Don't forget to generate the manual with `M-x edts-man-setup RET` so
you can go directly to the man pages inside emacs and also get inline
function docs, very handy!

---
layout: post
title:  "On the Viability of Erlang Releases and CouchDB"
date:   2014-05-07 20:17:51
categories: tech
tags: erlang releases couchdb deployment
---


There has been some discussion on what versions of Erlang CouchDB
should support, and what versions of Erlang are detrimental to
use. Sadly there were some pretty substantial problems in the R15 line
and even parts of R16 that are landmines for CouchDB. This post will
describe the current state of things and make some potential
recommendations on approach.

## Scheduler Collapse

It was discovered by Basho that R15* and R16B are susceptible to
scheduler collapse. There's quite a bit of discussion and
information in several threads [1] [2] [3] [4] [5].

So what is scheduler collapse? Erlang schedulers can be put to sleep
when there is not sufficient work to occupy all schedulers, which
saves on CPU and power consumption. When the schedulers that are still
running go through enough reductions to pass the work balancing
threshold, they can trigger a rebalance of work that will wake up
sleeping schedulers. The other mechanism for sharing scheduler load is
work stealing. A scheduler that does not have any work to do can
steal work from other schedulers. However a scheduler that has gone
to sleep cannot steal work, it has to be woken up separately.

Now the real problem of scheduler collapse occurs when you take
sleeping schedulers and long running NIFs and BIFs that do not report
an appropriate amount of reductions. When you have NIFs and BIFs that
don't report an appropriate amount of reductions, you can get into a
situation where a long running function call will only show up as
taking one reduction, and never hit the work balance threshold,
causing that scheduler to be blocked during the operation and no
additional schedulers getting woken up.

I keep mentioning "NIFs and BIFs" because it's important to note that
it is _not_ just user defined NIFs that are problematic, but also a
number of Erlang BIFs that don't properly report reductions.
Particularly relevant to CouchDB are the BIFs `term_to_binary` and
`binary_to_term` which do _not_ behave properly, and each report a
single reduction count, regardless of the size of the value passed to
them. Given that every write CouchDB makes goes through
`term_to_binary`, this is definitely not good.

This problem is systemic to all versions of R15 and R16B. In R16B01,
two changes were made to alleviate the problem. First, in `OTP-11163`
`term_to_binary` now uses an appropriate amount of reductions and will
yield back to the scheduler. The second important change was the
introduction of the `+sfwi` (Scheduler Forced Wakeup Interval) flag
[6] which allows you to specify a time interval for a new watchdog
process to check scheduler run queues and wake up sleeping schedulers
if need be. These two changes help significantly, although from what I
understand, they do not fully eliminate scheduler collapse.

*NOTE*: the `+sfwi` is _not_ enabled by default, you must specify a
greater than zero time interval to enable this. *WE NEED TO ENABLE
THIS SETTING.* We should figure out a way to conditionally add this
to vm.args or some such.

On a side note, Basho runs R15B01 because they backported the `+sfwi`
feature to R15B01 [7] [8]. They recommend running with `+sfwi 500` for
a 500ms interval. It might be worth testing out different values, but
500 seems like a good starting point. For Riak 2.0, they will be
building against R16B03-1 and 17.0 as their set of patches to R16B02
landed in R16B03-1 [9] [10].

## R16B01 and the breaking of monitors

So R16B01 sorted out the scheduler collapse issues, but unfortunately
it also broke monitors, which immediately disqualifies this release as
something we should recommend to users. The issues was fixed in
`OTP-11225` in R16B02.

## R16B02 and R16B03*

I don't know of any catastrophic problems on the order of those
described above in either of these releases. Basho fixed a number of
unrelated bugs in R16B02 [9] [10] that have since landed in R16B03-1,
which indicates we should probably prefer R16B03-1 over R16B02. R16B03
is also disqualified because it broke SSL and `erl_syntax`, resulting
in the patched R16B03-1.

## R14

R14B01, R14B03, and R14B04 are known good stable releases of Erlang,
and in my opinion the only known stable releases > R13 that don't
present issues for CouchDB (I think R16B02/R16B03-1 are too new to
declare stable yet). As for R14B02, there are some bad `ets` issues
with that release.

It's worth pointing out that there are two known bugs in R14B01, as
Robert Newson explains:

```
There are two bugs in R14B01 that we do encounter, however. 1) Another
32/64 bit oops causes the vm to attempt to allocate huge amounts of
ram (terabytes, or more) if it ever tries to allocate more than 2gib
of ram at once. When this happens, the vm dies and is restarted. It’s
annoying, but infrequent. 2) Sometimes when closing a file, the
underlying file descriptor is *not* closed, though the erlang process
exits. This is rare but still quite annoying.
```

## Erlang 17.0

The 17.0 release brings in a number of interesting changes to help the
scheduler collapse situation. `OTP-11648` improves reduction cost and
yielding of `term_to_binary`. It also utilizes `OTP-11388` which
allows for NIFs and BIFs to have more control over when and how they
are garbage collected (we should do some investigation on the
usefulness of this for NIFs like Jiffy).

The 17.0 release also updates `binary_to_term` in `OTP-11535` to
behave properly with reductions and yielding similar to
`term_to_binary`. This marks the 17.0 release as an important one for
CouchDB as now `term_to_binary` and `binary_to_term` both behave
properly.

## Dirty Schedulers

One other interesting item introduced in the 17.0 release is the
concept of dirty schedulers [12] [13]. This is an experimental feature
providing CPU and I/O schedulers specifically for NIFs that are known
to take longer that 1ms to run. In general, we want to make sure the
NIFs we use will yield and report reductions properly, but for
situations where that isn't feasible, we may want to look into using
dirty schedulers down the road when it's a non experimental feature.


## Recommendations for CouchDB

In my opinion we need to take the Erlang release issues more seriously
than we currently do and provide strong recommendations to users on
what versions of Erlang we support. I suggest we loosely take an
approach similar to Debian, and make three recommendations:

  * OldStable: [R14B01, R14B03, R14B04 (NOTE: _not_ R14B02)]
  * Unstable: [R16B03-1 recommended, R16B02 acceptable]
  * Experimental: [17.0]

I'm not suggesting permanently having three Erlang releases
recommended like this, but it currently seems appropriate. I think
long term we should target 17.x as our preferred Erlang release, and
then make a CouchDB 3.0 release that is backwards incompatible with
anything less than 17.0 so that we can switch over to using maps.

The narrowness of the acceptable releases list is going to cause some
problems. Debian Wheezy runs R15B01, which as established above, is
not good to run with unless you have the `+sfwi` patch, and I'm sure
there are many other distros running R15 and R16B or R16B01. I think
it would be useful to users to have a set of packages with a proper
Erlang CouchDB release allowing us to bless specific versions of
Erlang and bundle it together, but I know this idea goes against the
recent change in stance on working with distributions, and I don't
know the ASF stance on this issue well enough to comment on the
legality of it. That said, it does seem like the logical approach
until we get a range of stable releases spread out through the
distros.

## Work to be done

We need to make sure that all NIFs we use that could potentially take
longer than 1ms to run properly yield and report reductions. For
Jiffy, there is already a good start on this work [11]. We'll want to
look into what needs to be done for the rest of the NIFs.


## Wrapping up


There's quite a bit of information here, and plenty more in the
footnotes, so I hope this gives a good overview of the current state
of Erlang releases and helps us to make informed decisions on what
approach to take with Erlang releases.





### Footnotes

[1] [http://comments.gmane.org/gmane.comp.lang.erlang.bugs/3564](http://comments.gmane.org/gmane.comp.lang.erlang.bugs/3564)

[2] [http://erlang.org/pipermail/erlang-questions/2013-April/073490.html](http://erlang.org/pipermail/erlang-questions/2013-April/073490.html)

[3] [http://erlang.org/pipermail/erlang-questions/2012-October/069503.html](http://erlang.org/pipermail/erlang-questions/2012-October/069503.html)

[4] [http://erlang.org/pipermail/erlang-questions/2012-October/069585.html](http://erlang.org/pipermail/erlang-questions/2012-October/069585.html)

[5] [http://permalink.gmane.org/gmane.comp.lang.erlang.bugs/3573](http://permalink.gmane.org/gmane.comp.lang.erlang.bugs/3573)

[6] [http://erlang.org/pipermail/erlang-patches/2013-June/004109.html](http://erlang.org/pipermail/erlang-patches/2013-June/004109.html)

[7] [https://gist.github.com/evanmcc/a599f4c6374338ed672e](https://gist.github.com/evanmcc/a599f4c6374338ed672e)

[8] [http://data.story.lu/2013/06/23/riak-1-3-2-released](http://data.story.lu/2013/06/23/riak-1-3-2-released)

[9] [https://github.com/basho/otp/compare/erlang:maint...OTP_R16B02_basho4](https://github.com/basho/otp/compare/erlang:maint...OTP_R16B02_basho4)

[10] [https://groups.google.com/forum/#!topic/nosql-databases/XpFKVeUBdn0](https://groups.google.com/forum/#!topic/nosql-databases/XpFKVeUBdn0)

[11] [https://github.com/davisp/jiffy/pull/49](https://github.com/davisp/jiffy/pull/49)

[12] [https://github.com/erlang/otp/commit/c1c03ae4ee50e58b7669ea88ec4d29c6b2b67c7b](https://github.com/erlang/otp/commit/c1c03ae4ee50e58b7669ea88ec4d29c6b2b67c7b)

[13] [http://www.erlang.org/doc/man/erl_nif.html#dirty_nifs](http://www.erlang.org/doc/man/erl_nif.html#dirty_nifs)

---
layout: post
title:  "Rewriting the CouchDB HTTP layer"
date:   2014-08-17 10:19:23
categories: tech
tags: erlang couchdb http chttpd2
---

With the light at the end of tunnel on the BigCouch merge, I thought
it was time to get the conversation going on cleaning up the current
HTTP stack duality. We've got a good opportunity to do some major
cleanup, remove duplication, and really start more clearly separating
the various components of CouchDB.

## Primary objectives

    * Consolidate down to one HTTP layer
    * Isolate HTTP functionality
    * Separate HTTP server from HTTP resources
    * Easy plugin integration
    * Build clustered/local API


### Consolidate down to one HTTP layer

We currently have two HTTP layers, `couch_httpd` and `chttpd`. This
was a useful construct when BigCouch was a separate application where
isolating the clustered layer from the local layer was necessary, and
quite useful.

This is no longer the case, and we can significantly reduce code
duplication by consolidating down to one http layer. There are a
number of places in the two apps where the code is nearly identical,
except one calls out to `fabric` and the other calls out for
`couch_*`. For instance, compare `couch_httpd_db:couch_doc_open/4` [1]
with `chttpd_db:couch_doc_open/4` [2]. These are completely identical
aside from whether it goes through the clustered layer, `fabric`, or
through the local layer `couch_db`.

There are plenty of other places with similar duplication. This is
obviously ripe with opportunity to refactor and introduce some higher
level abstractions to make the HTTP layer function independently of the
document/database level APIs.


### Isolate HTTP functionality

I don't think `couch_doc_open/4` has any business existing in
the HTTP layer, we should move all non HTTP logic out. IMO the HTTP
layer should only concern itself with:

    1. Receiving the HTTP requests
    2. Extracting out the request data into a standard data structure
    3. Dispatch requests to the appropriate internal APIs
    4. Forward the response

Anything that doesn't fit into those four steps should be ripped out
and moved elsewhere. For instance, the primary logic for determining the
database redundancy and shard values is done in `chttpd_db` [3]. I
would greatly prefer to see this logic in a database API.

The more we can isolate HTTP logic from database logic the
better. Once they are fully decoupled, then the HTTP layer is merely
one particular client interface on top of the core database. We also
get all the benefits of isolation for testing and what not.

Along these lines, I think we greatly overuse the #http{} record for
passing around request data, and instead you extract the body, and
then combine all of the user supplied headers and query string params
into a standard options list. This we can we completely separate
making database requests from the representation of the client
request.


### Separate HTTP server from HTTP resources.

I think everything I've said so far is pretty clear cut in terms of
it's _the_ logical thing to do, but separating the HTTP server from
the HTTP endpoints is less clearly defined. However, we do have
precedence for this and there are a number of solid benefits.

First, let me explain what I mean here. There are two pieces to an
HTTP stack, first there's the core HTTP engine that handles receiving
and responding to requests and other things along those lines, and
second there's the places where you supply your business logic and
figure what content to send to the user.

CouchDB has a handful of places using this aproach, where instead of
defining all the logic in the HTTP stack directly, we have auxilary
modules defined within the appropriate applications that specify how
any HTTP requests for that application are handled. A good clean
example of this approach is `couch_mrview_http` [4].


### Easy plugin integration

One big advantage of the above separation of HTTP resources is that it
provides a standard way of plugins hooking in new HTTP endpoints. The
more we can treat the "core" CouchDB applications as plugins, the more
easily it is to isolate and replace various parts of the stack.


### Build clustered/local API

The above example of `couch_doc_open/4` is a clear cut case where
we want to abstract the process of loading a document. Not all places
are as easily abstractable, but this is a great example of why I think
we should have a standard API on top of clustered and local layers,
where deciding which to use is based on a local/clustered flag, or
some other heuristic.

I've been toying around with the idea of making a request object of
some sort, is something like `couch_req:make(ReqBody, ReqOptions)`
that you can then pass to `couch_doc_api` or some such, but I don't
have any strong opinions on this.


## Where I've gotten so far: chttpd2, a proof of concept

I've hacked out an experimental WebMachine [5] based rewrite of the
HTTP stack called `chttpd2` [6]. This PoC follows the same ideas I've
outlined above, so I'll run back through the previous outlined items
and explain how `chttpd2` handles it.


### Consolidate down to one HTTP layer

Right now I'm not doing anything special here, I still think building
an API layer that handles deciding whether to make a clustered or
local request is the proper approach, so I've not included any logic
in the HTTP stack for doing so.


### Isolate HTTP functionality

I've got a solid separation of functionality in `chttpd2`. If you
notice the current codebase in [6], there is zero logic for actually
handling any particular CouchDB requests. Rather those are self
contained within the appropriate sub applications. I've started this
for `couchdb-couch` [7] and `couchdb-config` [8]. Here's a simple
example of the new welcome resource [9].

As you can see, there is zero database logic in the welcome request
module. In fact, I started moving all the random logic in the current
HTTP layer to a temporary module I'm calling `couch_api` [10]. As you
can see from that module, it removes all the logic that was previously
nested in `couch_httpd_misc_handlers` [11]. More complicated examples
for creating a database and viewing database info are in [12], and an
all dbs example is in [13]. Also I've done similar things for
`couchdb-couch` as mentioned above in [8].


### Easy plugin integration

As I mentioned above, by making it easy to plugin in new HTTP
endpoints, we also make it easier for plugins to do the same. On that
front I've made it so each application can optionally declare a
`couch_dispatch` function describing what endpoints it can handle, and
then `chttpd2` will go and find all of those to figure out how to
dispatch requests [14]. And for example, here's how the
`couchdb-couch` endpoints are declared [15].


### Build clustered/local API

I have not started on this front, and have only built these endpoints
for interacting with the clustered layer for simplicity as this is
just a proof of concept I hacked together. However, as I mentioned
above I've started moving all the logic out of the HTTP layer into
more appropriate places. I've made similar changes to `couch-config`
by moving all of the logic from [16] into the `couch-config`
application itself.


### Why WebMachine?

I find WebMachine [5] to be one of the more interesting HTTP stacks for
building webapps. In particular I like how they have a specific flow
chart [17] and coordinate point corresponds to a particular definition
of the `webmachine_decision_core:decision/1` function.

That said I think Cowboy [19] has more momentum and might be a better
long term project to tie ourselves too.

Also, if we decide to go the WebMachine route, we'll need to
restructure a fair bit of the current HTTP layer, making a number of
breaking changes. I'm a strong -1 for coercing WebMachine into the
current haphazard CouchDB API. WebMachine is very opinionated on how
you structure your API (for good reason!) and I think going against
that is a mistake.

So if we wanted to just do a drop in replacement of the current
CouchDB API, then Cowboy is the way to go. Although one of these days
we should clean up the HTTP API.


# Conclusion

I hope this can start a good discussion on a game plan for the HTTP
layer. Like I said, this is a proof of concept that I hacked out, so
I'm not attached to the code or the use of WebMachine, but I do think
it's a good representation of the ideas outlined above.

Looking forward to hearing your thoughts and comments!



#### Footnotes

[1] https://github.com/apache/couchdb-couch/blob/master/src/couch_httpd_db.erl#L805-L823

[2] https://github.com/apache/couchdb-chttpd/blob/master/src/chttpd_db.erl#L886-L904

[3] https://github.com/apache/couchdb-chttpd/blob/master/src/chttpd_db.erl#L203-L205

[4] https://github.com/apache/couchdb-couch-mrview/blob/master/src/couch_mrview_http.erl


[5] https://github.com/basho/webmachine

[6] https://github.com/chewbranca/chttpd2/tree/initial-branch

[7] https://github.com/apache/couchdb-couch/tree/2073-feature-webmachine-http-engine

[8] https://github.com/apache/couchdb-config/tree/2073-feature-webmachine-http-engine

[9] https://github.com/apache/couchdb-couch/blob/2073-feature-webmachine-http-engine/src/couch_httpr_welcome.erl

[10]
https://github.com/apache/couchdb-couch/blob/2073-feature-webmachine-http-engine/src/couch_api.erl

[11] https://github.com/apache/couchdb-couch/blob/master/src/couch_httpd_misc_handlers.erl#L32-L45

[12] https://github.com/apache/couchdb-couch/blob/2073-feature-webmachine-http-engine/src/couch_httpr_db.erl

[13] https://github.com/apache/couchdb-couch/blob/2073-feature-webmachine-http-engine/src/couch_httpr_dbs.erl

[14] https://github.com/chewbranca/chttpd2/blob/initial-branch/src/chttpd2_config.erl#L26-L33

[15] https://github.com/apache/couchdb-couch/blob/2073-feature-webmachine-http-engine/src/couch.erl#L68-L73

[16] https://github.com/apache/couchdb-couch/blob/master/src/couch_httpd_misc_handlers.erl#L155-L249


[17] https://raw.githubusercontent.com/basho/webmachine/develop/docs/http-headers-status-v3.png

[18] https://github.com/basho/webmachine/blob/develop/src/webmachine_decision_core.erl#L158-L595

[19] https://github.com/ninenines/cowboy

---
layout: post
title:  "Does this thing still work?"
date:   2018-04-20 09:35:51
categories: general
tags: blog
---

Is the mic on? Does this thing still work? It sure has been a while
since I've updated this site. Truth be told, I don't really care for
the current technology I'm using to power this site. I've got a
handful of grand ambitions on what I want a personal "homepage" to be,
but they all require a considerable chunk of development time, which I
haven't invested in yet. As a result, I typically don't bother
updating this site, but the net result of that is that it stays
stagnant and I don't create new content. Well, I've decided to take a
different approach and just start generating the content and be
accepting that this isn't quite what I want it to be, but I can always
fix it later. I mean hey, 301 redirects are a thing, I can migrate
this content to a new site later. So here goes my attempt at
resurrecting chewbranca.com in 2018!

---
layout: post
title:  "Lisp Game Jam - Day one update 01"
date:   2018-04-20 09:35:51
categories: lisp-game-jam
tags: gamedev lua love2d lisp game-jam fennel
---

I'm very excited to be diving into the [2018 Lisp Game
Jam](https://itch.io/jam/lisp-game-jam-2018)! My plan is to make a
Diablo-esque pvp deathmatch game! I stumbled upon the awesome open
source [Flare Game](https://github.com/clintbellanger/flare-game) that
is a single player action RPG inspired by Diablo. What really
intrigued me about the game is all of the artwork.

I haven't had great luck finding open source isometric game art,
especially with a variety of characters and animations, so when I
found all of the open source artwork in Flare Game, I started dancing
and getting ideas for this game. The Flare Game repo itself that I
linked to above is actually just the game assets, containing the
artwork, animation definitions, maps, map tiles, sounds, etc, and the
engine is in the [Flare
Engine](https://github.com/clintbellanger/flare-engine) repo. This is
perfect for me, because I'm rolling my own engine on top of [Love
2D](http://love2d.org/) using Lua, so I just need the art assets.

I've built a library around the Flare Game assets, called
[LFG](https://github.com/chewbranca/lfg), short for Lua Flare
Game. The idea with this library is to take care of parsing the
various animation definitions and making usable characters out of the
sprite sheets.

What does that actually mean? Well, let's take a look. A sprite sheet
is a series of animation frames combined into a single image that you
then load in your game engine to render. So for instance, I've taken
a sprite sheet from Flare Game like this minotaur sprite sheet:

![Minotaur Sprite Sheet](/img/flare-game/minotaur.png)

and using LFG I've turned it into animations like:

![Minotaur Animation](/img/lfg_minotaur.gif)

and that's based on Flare Game's animation definition for the [Minotaur
Animation](https://github.com/clintbellanger/flare-game/blob/master/art_src/animation_defs/characters/minotaur.txt)
which looks like:

```
#image=

render_size=128,128
render_offset=64,96

[stance]
position=0
frames=8
duration=1600ms
type=looped

[run]
position=8
frames=8
duration=533ms
type=looped

[swing]
position=16
frames=8
duration=1066ms
type=play_once
...
```

Those animation definition files give you the information to know what
frames correspond to a particular animation, and then the 8 rows of
the sprite sheet correspond to the 8 cardinal directions. So to make
an animation, you figure out which cardinal direction the player is
facing, select the appropriate row, and then select the appropriate
sprite tiles for the desired animation. Then you divide the animation
duration by the number of frames in the animation, and you play each
frame for that long to animate the character.

The idea of the LFG library is to provide a simple library interface
around these Flare Game assets, so you can do things like `local
minotaur = lfg.get_character("Minotaur")` and have the full set of
animations at your disposal. The LFG library also takes care of player
controls, movement, circling strafing, and it has rudimentary
projectiles and collision detection in place. I'm in the process of
adding networking support to it as well. In general, LFG's goal is to
make it easy to build Love 2D games using the wonderful assets from
Flare Game.

## Lisp Game Jam 2018

So how does this all fit together for the Lisp Game Jam? Well, like I
said, I'm going to build a Diabo-esque pvp deathmatch game using the
Flare Game assets! This is the first video game I've ever made, so
I've been having a blast diving in and researching how to build games
and how game engines work and game networking and all that fun
stuff. It's been a bit tortuous not being able to actually start on
the game itself until today, but it took a chunk of work to get LFG to
where it provides higher level abstractions around the Flare Game
assets so you can just grab the different characters and use them in
your game. A handful of the Flare Game character mobs are currently
supported, for instance:

![Random Mobs](/img/lfg_rand_mobs.gif)


"But I thought you said this is a *Lisp* game jam? Lua ain't a Lisp."
Correct! That's where the awesome [Fennel
Lang](https://github.com/bakpakin/fennel) comes in! Fennel is a
lightweight Lisp that compiles down to Lua code with minimal
overhead. So my plan for the game jam is to make a new Love 2D based
game that is written in Fennel, and uses LFG to provide the characters
and game world! The entire point of this game is for pvp deathmatch,
so getting networking in place quickly is a priority for today. I'm
going to focus on LAN only game play, because low latency internet
based gaming is serious undertaking. Maybe in version 2 :D

My plan is to post updates to this blog as development progresses so
folks can see it come together. Happy hacking!

Flare Game Credits

Art

Blarumyrran
Brandon Morris "Augmentality" | http://www.youtube.com/brandon75689
Clint Bellanger | http://clintbellanger.net
Holly Daniel | https://www.odesk.com/users/~~822174c5679d5273
Justin Jacobs | http://dorkster.github.io/
Justin Nichol | http://blackflaggames.org/
Jessica "Zeldyn" Cox | http://zeldyn.deviantart.com/
remaxim | http://opengameart.org/users/remaxim
Sarah Benalene | http://benalene.com/
Scrittl
Stefan Beller | http://github.com/stefanbeller

Additional Art
  Thanks to these artists for releasing reusable open-licensed art

Aare
Bart K | OpenGameArt.org
Blender Foundation | apricot.blender.org
Cori Samuel
D. Sharon Pruitt
Hythlodaeus
Iwan "qubodup" Gabovitch
Lamoot
Lattice
Ljudbank
Lorc
MaximB
Michael Baradari
Mike Koenig
Mikodrak
Misha
Mumu
Naraphim
p0ss
Renderwahn
rubberduck
Samuel Moxham
Sindwiller
Spookymodem
Stephan
TiZiana
Vwolfdog
Yughues
Zuendholz

Content

Clint Bellanger | http://clintbellanger.net
Justin Jacobs | http://dorkster.github.io/
Stefan Beller | http://github.com/stefanbeller
Thane "pennomi" Brimhall | http://github.com/pennomi

Distributions

Ablu
Feillyne
Jan-Hendrik (hennr) Peters
Manuel A. Fernandez Montecelo
miffe
... and many others

Translations

(be) Belarussian by Mikhail Karalevich
(cs) Czech by Nikita Vanku (Zaraka)
(de) German by Thomas 'CruzR' Glamsch, Chris Oelmueller, Janet Hunt, Stefan Beller, and Wuzzy2
(el) Greek by Yannis Anthymidis and Michael Papageorgiou
(es) Spanish by Juan Pablo 'morris989' Tamayo and Carlos Sanchez
(fi) Finnish by Timo Sievänen
(fr) French by Quentin 'acieroid' Stievenart, Bonbadil, Morgan Strauss, and Christoph J. Thompson
(gd) Scottish Gaelic by GunChleoc
(gl) Galacian by Adrian Chaves Fernandez (Gallaecio)
(it) Italian by Giovanni Dalla Torre and Andrea Ranaldi
(ja) Japanese by Paul Wortmann, sujiniku
(nb) Norwegian Bokmal by Hans Joachim Desserud
(nl) Dutch by Bas Doodeman
(pl) Polish by Paweł Puszczyński
(ru) Russian by Sergey Basalaev and Evgen Pavlov
(sk) Slovak by Miro Jánošík
(sv) Swedish by Andreas Berheim Brudin
(uk) Ukrainian by Igor Paliychuk
(zh) Chinese by lonsine

Full Credits

See the full credits listing including per-file attribution online:
https://github.com/clintbellanger/flare-engine/wiki/Credits
https://github.com/clintbellanger/flare-game/wiki/Credits

# All assets in this folder are from Flare Game and retain the Flare Game license.

The Flare Game README is copied below, and the CREDITS.txt file is in this repo
as wlel.




# Flare

Flare is a single-player 2D action RPG with fast-paced action and a dark fantasy style.

It is built on the FLARE engine (Free/Libre Action Roleplaying Engine). Its repo is at https://github.com/clintbellanger/flare-engine

Flare uses simple file formats (INI style config files) for most of the game data, allowing anyone to easily modify game contents. Open formats are preferred (png, ogg). Maps are edited using Tiled.

For best results, please use the latest version of the FLARE engine with the game data in this repository. Using an older version of the engine could result in some game elements not working correctly.

![Flare screenshot](distribution/screenshot1.jpg)

The flare-game content contains depictions of violence, blood, and mild gore.

## Copyright and License

Flare (the game) is Copyright ©2010-2013 Clint Bellanger. Contributors retain copyrights to their original contributions.

The Flare Engine is released under GPL version 3 or later.

All of Flare's art and data files are released under CC-BY-SA 3.0. Later versions are permitted.

The Liberation Sans fonts version 2 are released under the SIL Open Font License, Version 1.1.

The GNU Unifont font is released under GPL v2, with the exception that embedding the font in a document does not in itself bind that document to the terms of the GPL.


## Links

* Homepage  http://flarerpg.org
* Repo      https://github.com/clintbellanger/flare-game
* Binaries  https://sourceforge.net/projects/flare-game/
* Forums    http://opengameart.org/forums/flare
* Email     clintbellanger@gmail.com

## The Simplest Setup

If you are building and running Flare Game from source, this is the simplest way to get started on all operating systems. Here you build and play Flare in a local folder instead of installing it to a standard OS folder (e.g. /usr/local or Program Files).

* Clone the flare-engine repo and build the executable.
* Clone the flare-game repo into your user directory.
* Copy the flare-engine "default" mod into the flare-game mods folder.
* Put the flare executable inside the flare-game folder
* The executable and the "mods" folder should be in the same directory
* Run Flare

## Building and Running (non-Linux)

See the Flare-Engine repo for instructions on how to build/install Flare:

https://github.com/clintbellanger/flare-engine

The mods folder should be in the same directory as the flare executable when building from source.

Flare Game is installed as a set of mods for Flare Engine. Place the contents of the "mods" folder inside Flare Engine's mods folder.

Then enable these mods in Flare's Configuration screen.

    fantasycore
    empyrean_campaign


### Building and Running on Linux

Clone the repos:

	git clone https://github.com/clintbellanger/flare-engine
	git clone https://github.com/clintbellanger/flare-game


Move to the flare-engine repo and build:
For instructions more in detail see the INSTALL file in the
flare-engine repo.

	cd flare-engine
	cmake . && make


From this point we have two options:

Option 1
Install the game system-wide (requires root access):

	sudo make install
	cd ../flare-game
	cmake . && sudo make install
	cd ~ #this step is important because we don't want to load mods from the repo folders
	flare #alternatively, launch Flare from the desktop application menu


Option 2
Run the game without installing:

	cd ../flare-game
	ln -s ../flare-engine/flare flare
	cd mods
	ln -s ../../flare-engine/mods/default default
	cd ..
	./flare



## Settings

Settings are stored in one of these places:

    $XDG_CONFIG_HOME/flare
    $HOME/.config/flare
    ./config

Here you can enable fullscreen, change the game resolution, enable mouse-move, and change keybindings. The settings files are created the first time you run Flare.

## Save Files

Save files are stored in one of these places:

    $XDG_DATA_HOME/flare
    $HOME/.local/share/flare
    ./saves

If permissions are correct, the game is automatically saved when you exit.

## Packaging and Distributing

If you are packaging Flare Game for release (e.g. on an operating system's software repo), we suggest creating two packages.

* flare-engine the package that contains the single engine reused by several games
* flare-game, a package that requires flare-engine that only contains this game data

When distributing flare-game you may elect to omit these folders which are not used at runtime.

* art_src contains the raw files (e.g. Blender files) used to generate Flare's art.
* tiled contains the Tiled-native map files used to export Flare's maps